You should stop using SFTP as it is an outdated technology that also comes with its share of disadvantages. The main problems with SFTP are the following:
-
You have to keep the servers secure (network security, access control, …). It adds extra loads on top of your list of duties.
-
SFTP servers only do a SFTP-to-bucket dump which is unnecessary if we would use buckets directly.
-
As for Google Cloud Platform, SFTP servers mount GCP buckets as files systems to a Docker container. This is something that can be done but depending on the load sent to the SFTP can become a problem (volume limitations).
Note: SFTP can be used as an interface to pass on requests between the user and the bucket it mirrors. It uses user-friendly command lines, see SFTP basic CLI commands survival kit.
Alternative to SFTP
Beside providing a friendly interface to our customers (e.g. if they use FileZilla), why using SFTP is a no go? Preaching against it sounds like we are fighting against hell. Why Data Software Engineers so adamant about it? At the end, do we now why it is an issue? It is not the best practice but is it that hurtful?
To give you the short answer: there is nothing wrong with SFTP in general, it is just an outdated technology.
On the contrary, there are multiple benefits using buckets directly:
-
Bucket security is done by GCP (of course, we have to do the access control – which is not so hard).
-
The interface is also easy since you can see the bucket contents in the GCP console (no need for external tools like FileZilla).
-
Buckets can handle a high amount of parallel reads.
It is not always possible though
As stated in the Fundamentals of Data Engineering book published by O’REILLY:
Data engineers rightfully cringe at the mention of SFTP. Regardless, SFTP is still a practical reality for many businesses. They work with partner businesses that consume or provide data using SFTP and are unwilling to rely on other standards.
To conclude, try to get ride of SFTP legacies but don’t get your knickers in a twist regarding this. 👟