rsync data transfers are very slow
Problem
Data transfers with rsync are very slow. Is there a faster way?
Solution
Enable "Allow External Connection" on the bucket and upload directly to Data Hub with an S3 tool like rclone or cyberduck. This is much faster.
For directories with many files (rather than one 320 GB single file), uploading in parallel with rclone is faster still.
rclone copy /source/path your-datahub-config:bucket-name --transfers=8 --s3-no-check-bucket --inplace