Amazon Transfer for SFTP if a hosted SFTP service announced at AWS re:Invent yesterday and I've been playing around with it to see how easy it is to set up. I've also managed to integrate it into VS Code for absolutely no other reason than I could.

Its always quite surprising to me how much Secure File Transfer Protocol (SFTP) is used by businesses. I would encounter it all the time with retail customers, who would often get data feeds from partners delivered over SFTP. They'd have scheduled tasks that would process files on an hourly or nightly basis, and if you asked why they were still using SFTP, often the answer was that it would be too much hassle to get third parties to change to something else. It wasn't broke so why fix it.

Factors, like those described above, means that you end up with an SFTP server running somewhere, requiring updates, patching and management.

Amazon Transfer for SFTP is an excellent concept for companies that find themselves in a similar position. You don't have to move away from SFTP, but instead, it creates an SFTP interface over an AWS S3 (Simple Storage Service) bucket.

Not only is an S3 bucket an inexpensive, durable and easy way to store this sort of data you also have all the advantages that come with S3 like Webhooks that fire when content is added or modified. These could be used to kick off a processing job. It is incredibly simple to have an AWS Lambda function (Serverless Compute) to be called when an S3 bucket has objects created.

Amazon Transfer can be a seen a simple way to modernise legacy workflows and bring them kicking and screaming into 2018.

Setup

To set it up the documentation does a good job of explaining all you need to know. If you follow along, you should be able to be up and running in about 20 minutes.

The most confusing part about the set up for me, being new to AWS, was the IAM (Identity and Access Management) security set up. After following along with the docs I got the following error:

Reading directory .: received failure with description 'Unable to load AWS credentials from any provider in the chain

The root of my issue was Trust Relationships. The way that authentication works in Amazon Transfer for SFTP is that in the console you create a user and then associate the user with an IAM Role. This role contains the permissions to read and write to an S3 bucket. However, under the hood, the Amazon Transfer service is going to be using the role rather than a logged in user. Therefore, in the Policy, you need to say that it's ok for the transfer service to use the policy and also act on behalf of the user.

The following JSON is what I used in my trust relationship; you should note two important things. The Principal is transfer.amazonaws.com the reason I was getting the above error was that I had s3.amazonaws.com as the principle. Also, make sure you have an Action element set to "sts.AssumeRole" this permits the service to assume the role and use its permissions.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "transfer.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

SSH Key Gen

Another stumbling block on Windows is the generation of the SSH public and private keys. For Windows users, the simplest method is to download and install Git for Windows. It comes with ssh-keygen as part of the bundle and ready to use from the commandline or PowerShell. Also if you have Windows Subsystem for Linux you could just use BASH and run ssh-keygen from there.

Visual Studio Code

While it is probably best to use an SFTP client like FileZilla I stumbled across a plugin on the Visual Studio marketplace which allows you to use SFTP directly from within Visual Studio Code and I'm glad to report that it works with The Amazon Transfer for SFTP service.

You can get hold of the plugin here  It's had over 1.3 million downloads so I think it is safe to assume that a fair few people are still using SFTP.

Once you have installed the plugin, open a folder and type SFTP:Config in the Command Palette (Ctrl + Shift + p) this creates  a base config file. My finished file looked something like this:

{
    "protocol": "sftp",
    "host": "s-6ad9e96bc9ca40ec8.server.transfer.eu-west-1.amazonaws.com",
    "port": 22,
    "username": "thebeebs",
    "privateKeyPath": "/Users/marti/keys",
    "remotePath": "/thebeebssftpdemo"
}

The privateKeypath points to the private key file were generated earlier and are used by the service to authenticate the session.

If you now add a file to the folder in Visual Studio Code, you can right-click the file and choose Upload, this sends the file to the SFTP service, and the file ultimately ends up in the S3 bucket.

The plugin supports more advanced features such as upload on saving a fileand watchers that upload files based upon rules you define in the config. For a full set of features check out it's GitHub https://github.com/liximomo/vscode-sftp