Forums

Hi,

It's great having ssh access to PythonAnywhere, but it would even more awesomer if I can also have sftp access.

Is this something on the roadmap (hint, hint)...

Cheers, Robert

Actually, ignore my post. I used SCP over SSH, and all was well.

I always thought SCP == SFTP, but apparently not. We'll look into that...

No, SFTP is quite a different protocol that was added later than the original SCP one. Its operation is similar, however, in that it first opens an SSH connection and then spawns a daemon on the remote machine to which it connects via SSH and communicates with the SFTP protocol. I can't remember everything that you need to do to get it working but I don't remember it being tricky - you need to make sure that sftp-server is installed and you need to add a line to sshd_config like this (if it's not already there):

Subsystem sftp /usr/lib/openssh/sftp-server

I'm pretty sure those steps are necessary but I can't recall if they're sufficient. Anyway, most distributions will already have this configured so you can probably just copy one of those.

Hm, in our current setup I think it might be quite hard to get the security / sandboxing / isolation to work with user chroots... May have to wait until we move to LXC...

Really don't worry... almost every SFTP client supports SCP as a fallback.

Yes, I doubt it's a particularly big deal. I'm curious as to why SFTP is any harder to get working than SCP, though - don't they work in a similar way, just with a different protocol?

Since we thought it was the same thing as SCP, we didn't do anything to enable it. I'll raise a ticket for us to have a look at it since it may not work with our modifications to the SSH login process.

Err... Has anybody actually spotted any problems with SFTP? I just tried it and it works fine for me - looking at the debug logging it seems to be using the proper SFTP subsystem and not some sort of SCP fallback option. At least basic get and put seem to work.

That'll teach me to take people's word for something... (^_^)

EDIT:

Looks like it's using /usr/lib/openssh/sftp-server (or its symlink /usr/lib/sftp-server).

Yup. Looks like WinSCP and Filezilla both connect happily and work well.

Sorry I didn't see this topic until now. I would have added right away that I've been using @"c:\Program Files (x86)\PuTTY\psftp.exe" -i c:\Users\pathto\certificate\pa.ppk a2j@ssh.pythonanywhere.com in a batch file on my path for quite some time. It works great!!

I forgot to mention that I named the file paftp.cmd.

So paftp.cmd a_file pushes a_file up to your account? Nice!

I love it...and it can pick up where it left off if the connection breaks for larger files, very nice when you have Comcast...☺

Handy!

Are you aware that rsync can even resume partial transfers? Rsync is fab. Unfortunately it's not yet supported here (it requires the rsync binary to be available at the destination).

While I'm on the subject, one tool I started using myself recently was rdiff-backup which uses Rsync's low-level rdiff utility to perform efficient backups over an SSH connection, where the destination folder ends up as a clone of the target but with additional reverse-diff snapshots going back in time. It makes efficient offsite backups wonderfully simple.

Now if only you could use that with Glacier!!

Well, there's also Duplicity which is effectively the same thing except that it also encrypts the backups for storage on an untrusted service (Amazon S3, Google Drive, etc.), but I haven't tried that yet. You could create the encrypted data locally and upload it to any service you like, or you might be able to find a FUSE back-end - I know they exist for S3 (haven't tried them, though), but I'm not sure about Glacier.

I think Duplicity might use forward-diffs, however, so more like a traditional level 0 plus incrementals situation. I'll let you know once I've had a play, if you're interested.

EDIT: Hmm, according to the Wikipedia page, Duplicity already supports Amazon S3 directly. In fact, looking at the man page it has a sort of pluggable back-end system and it also supports Rackspace open cloud, FTP(S), Google Docs (do they mean Google Drive?), WebDAV and Ubuntu One, as well as the obvious rysnc, scp, sftp and local directory options.

Having now had chance to quickly test out Duplicity, I have objectively ascertained that it is, in fact, da bomb.

Essentially it builds a compressed multivolume tar archive of your data, but each volume is also encrypted with GPG. When you do further backups, it makes those incremental, using rdiff to find a fairly minimal set of diffs, and dumps those in place too. You can effectively use it like rsync, but as opposed to just supporting local and SSH-based destinations, it also supports all the ones I listed above - I did a dump out to Google drive without a hitch (aside from having to generate an application-specific password because it doesn't support 2-factor auth directly).

This is kinda tangential to the main thread here so I won't go on (for a change!), but it's definitely worth investigating if you want to use a third party storage service for offsite backups.

One more reason to stop using Windows...☺

Actually there's a Duplicity clone that someone reimplemented in C# called Duplicati which runs on Windows, and also adds a pretty GUI on top and attempts to offer a more friendly interface (Duplicity isn't really aimed at the novice user).

It also runs on Linux and OSX, presumably via Mono, but since I have previous little I need to back up on my Windows machine I didn't bother exploring any further. Fair play to the chap, though, he's re-implemented the rsync algorithm in C# and done a ton of other work on it - I would suggest taking a look if you need truly cross-platform backups. It's not compatible with Duplicity (simply inspired by it), and I don't believe it currently saves any permissions or other meta-info (although it appears to be planned).

EDIT

As an aside, spurred by a conversation with a friend where I ended up covering a lot of this again, I wrote a blog post about backup solutions that people might find interesting.

That's funny, I know about Duplicati, but I didn't connect it when you mentioned Duplicity. I really need to review and fix what I'm personally doing with backups.

In case it helps anyone, I use Ipswitch WS_FTP Pro (a Windows client) here and it's fine with:

Server type: SFTP/SSH

Host name: ssh.pythonanywhere.com UserID: jgmdavies Password: (my password) Account: jgmdavies

Everything else at default settings. Get a reasonable Explorer type view of local and remote folders, including .local etc. on the PA side.

Jim

Thanks, Jim -- that's useful!

Thanks Jim - that helped me setting up Filezilla access to site. @giles - would you consider adding that info to your SSH page in the help wiki? For Filezilla I used these settings:

host: ssh.pythonanywhere.com
port: blank
logon type: normal
user: <user name for pythonanywhere>
password: <password for pythonanywhere>

Good idea! It was actually already there, but hidden away on a different page. So I've linked to that one from the SSH page.

Just to update this for others searching around. I set my droidedit up to use the SSH connection as SFTP and it works fine. Not SUPER fast, but nothing to complain about.