Recently, I encountered an issue that required speeding up the upload of large files to S3 from the front end only.

To address this, we can consider two strategies: implementing parallel uploads and enabling S3 acceleration. Let's explore how these methods can be utilized to enhance upload speeds.

Parallel Uploads

For parallel uploads, the multipart upload API and presigned URLs from S3 are key tools. This approach involves segmenting a file into various parts and independently uploading each segment. After all segments are uploaded, they are merged to complete the upload. Presigned URLs offer temporary access for uploading directly to the bucket, with configurable expiration. This method ensures data is uploaded directly to S3, bypassing the backend.

Workflow is like this:

• To initiate a parallel upload using S3, start with the CreateMultipartUpload API to obtain an upload ID. This ID is crucial as it associates each uploaded chunk as part of the same file.

• Next, acquire a set of presigned URLs from the backend using the GetPresignedUrl function. These URLs, each with a specified expiration, allow for direct uploads from the frontend. We can get multiple of these URLs to upload chunks in parallel.

• Upload each chunk. Each part must be uploaded with the provided upload ID and a sequential part ID. Make sure partid is correct for each segment, so that S3 can combine all these part in correct way.

• After all parts are uploaded, execute the CompleteMultipartUpload command. This signals S3 to merge all the uploaded chunks associated with the given upload ID, finalizing the file upload.

S3 Acceleration

S3 Acceleration is a service offered by S3 to enhance the speed of uploads and downloads for a bucket. To leverage this, it must be activated for the desired bucket, and it incurs additional charges. The service improves transfer speeds by utilizing edge locations, thereby optimizing the upload process.