Support for Explicit FTP over SSL/TLS

Octavian -

Yesterday we've rolled out support for FTPS (Explicit FTP over SSL/TLS) to the Pipe Platform.

This means that our customers can now transfer recording files to their final storage location using the popular FTP protocol while also enjoying the added security benefits of SSL/TLS encryption.

FTP, or File Transfer Protocol, is a widely-used protocol for transferring files over the Internet. We've been supporting FTP since the early days of the Pipe Platform. However, it has a significant drawback in that it is not secure. All data transferred using FTP is sent in plaintext, meaning that anyone who has access to the transmitted data can intercept and read it. This is a major concern for businesses and individuals who need to transfer sensitive information like personal recordings.

The FTPS option is now available under the FTP section in the Pipe account dashboard and it is the default option.

Explicit FTP over SSL/TLS option in the Pipe Account Dashboard

The Pipe Platform now supports several secure methods for pushing the recording data directly to your storage solution:

  1. SSH File Transfer Protocol
  2. the (Amazon) S3 protocol
  3. Dropbox
  4. new: Explicit FTP over SSL/TLS

Note regarding (cloud) firewalls

We're using passive mode exclusively, which means the data connection - FTP uses two TCP connections, an initial control connection, and a data connection - is established by us towards the passive ports indicated by your FTP server through the initial control connection. If you're using a firewall, you'll have to explicitly open the passive ports configured in your FTP server for the 2nd data connection to go through.

However, our experience with the cloud firewalls of DigitalOcean and Hetzner is that with non-encrypted plain FTP, it is enough to open just port 21, while with FTPS, you must explicitly open the passive ports. My understanding is that these firewalls will inspect/read the unencrypted traffic and immediately open/prepare the firewall for the 2nd connection when they see the client-server discussion about opening the 2nd connection (`PASV` FTP command). This is not possible with FTPS since the communication is encrypted.

We had to explicitly allow both the main port (21) and the range of passive ports in our Hetzner cloud firewall for the FTPS connections to go through all the way to our FTPS server.

Updates to the video_copied_ftp webhook

Since this webhook is now sent for successful storage attempts over

  1. plain FTP
  2. explicit FTP over TLS/SSL
  3. SSH file transfer protocol

We've extended the body and added one new key:value pair to make it easier to distinguish between these requests:

  1. protocol : ftp or sftp

Testing details/audit

The code that pushes files through FTP(S) is written in PHP. Since the PHP FTP implementation is not explicit in using a 2nd secure connection for ftp_put along with the control connection opened by ftp_ssl_connect and since the "secure data channel is not enabled by default," we decided to take a deeper look.

We've tested with real FTPS connections from our infrastructure to a test server running vsftpd with various SSL/TLS configurations. One important configuration was to set up vsftpd to log all received FTP commands and responses to the log file. This gave us a clear view of all FTP commands and responses exchanged between our infrastructure and the destination FTP server. We used the log to confirm that the two FTPS commands that separately secure the control and data connections ( AUTH TLS and PROT respectively) are issued.

For example, in the log below, created as a result of the FTPS test available from https://dashboard.addpipe.com/ftp, AUTH TLS is sent by our infrastructure towards the destination FTP server to encrypt the control connection (the username and password are exchanged after this command). The FTP server responds with "234 Proceed with negotiation." Immediately after, the PHP implementation of ftp_ssl_connect sends the PROT FTP command which is used to specify the level of protection used on the data connection. PROT P is issued meaning the "Protection" level is requested and the server responds with "200 PROT now Private" meaning the data channel will also be encrypted. This is what we were looking for!

Thu Jan 12 09:31:14 2023 [pid 13707] FTP response: Client "::ffff:65.21.xx.yy", "220 (vsFTPd 3.0.3)"
Thu Jan 12 09:31:14 2023 [pid 13707] FTP command: Client "::ffff:65.21.xx.yy", "AUTH TLS"
Thu Jan 12 09:31:14 2023 [pid 13707] FTP response: Client "::ffff:65.21.xx.yy", "234 Proceed with negotiation."
Thu Jan 12 09:31:14 2023 [pid 13707] FTP command: Client "::ffff:65.21.xx.yy", "PBSZ 0"
Thu Jan 12 09:31:14 2023 [pid 13707] FTP response: Client "::ffff:65.21.xx.yy", "200 PBSZ set to 0."
Thu Jan 12 09:31:14 2023 [pid 13707] FTP command: Client "::ffff:65.21.xx.yy", "PROT P"
Thu Jan 12 09:31:14 2023 [pid 13707] FTP response: Client "::ffff:65.21.xx.yy", "200 PROT now Private."
Thu Jan 12 09:31:14 2023 [pid 13707] FTP command: Client "::ffff:65.21.xx.yy", "USER ftpuser"
Thu Jan 12 09:31:14 2023 [pid 13707] [ftpuser] FTP response: Client "::ffff:65.21.xx.yy", "331 Please specify the password."
Thu Jan 12 09:31:14 2023 [pid 13707] [ftpuser] FTP command: Client "::ffff:65.21.xx.yy", "PASS <password>"
Thu Jan 12 09:31:14 2023 [pid 13706] [ftpuser] OK LOGIN: Client "::ffff:65.21.xx.yy"
Thu Jan 12 09:31:14 2023 [pid 13708] [ftpuser] FTP response: Client "::ffff:65.21.xx.yy", "230 Login successful."
Thu Jan 12 09:31:14 2023 [pid 13708] [ftpuser] FTP command: Client "::ffff:65.21.xx.yy", "PWD"
Thu Jan 12 09:31:14 2023 [pid 13708] [ftpuser] FTP response: Client "::ffff:65.21.xx.yy", "257 "/" is the current directory"
Thu Jan 12 09:31:14 2023 [pid 13708] [ftpuser] FTP command: Client "::ffff:65.21.xx.yy", "PASV"
Thu Jan 12 09:31:14 2023 [pid 13708] [ftpuser] FTP response: Client "::ffff:65.21.xx.yy", "227 Entering Passive Mode (65,109,184,141,23,189)."
Thu Jan 12 09:31:14 2023 [pid 13708] [ftpuser] FTP command: Client "::ffff:65.21.xx.yy", "TYPE I"
Thu Jan 12 09:31:14 2023 [pid 13708] [ftpuser] FTP response: Client "::ffff:65.21.xx.yy", "200 Switching to Binary mode."
Thu Jan 12 09:31:14 2023 [pid 13708] [ftpuser] FTP command: Client "::ffff:65.21.xx.yy", "STOR addpipe-ftp-test.test"
Thu Jan 12 09:31:14 2023 [pid 13708] [ftpuser] FTP response: Client "::ffff:65.21.xx.yy", "150 Ok to send data."
Thu Jan 12 09:31:14 2023 [pid 13708] [ftpuser] OK UPLOAD: Client "::ffff:65.21.xx.yy", "/addpipe-ftp-test.test", 60 bytes, 9.59Kbyte/sec
Thu Jan 12 09:31:14 2023 [pid 13708] [ftpuser] FTP response: Client "::ffff:65.21.xx.yy", "226 Transfer complete."
Thu Jan 12 09:31:14 2023 [pid 13708] [ftpuser] FTP command: Client "::ffff:65.21.xx.yy", "DELE addpipe-ftp-test.test"
Thu Jan 12 09:31:14 2023 [pid 13708] [ftpuser] OK DELETE: Client "::ffff:65.21.xx.yy", "/addpipe-ftp-test.test"
Thu Jan 12 09:31:14 2023 [pid 13708] [ftpuser] FTP response: Client "::ffff:65.21.xx.yy", "250 Delete operation successful."
Thu Jan 12 09:31:14 2023 [pid 13708] [ftpuser] FTP command: Client "::ffff:65.21.xx.yy", "QUIT"
Thu Jan 12 09:31:14 2023 [pid 13708] [ftpuser] FTP response: Client "::ffff:65.21.xx.yy", "221 Goodbye."

Furthermore, we've also looked at the source code for PHP's ftp_ssl_connect and ftp_put and learned that when opening a secure FTP control connection using ftp_ssl_connect, AUTH TLS is always requested before AUTH SSL, which is as it should be.