At Experiential, we take on existing projects that are seeing successful user adoption and make them scalable. One fruitful place to start is determining how an application or suite of services handles data. Today, we’ll dive specifically into file handling.
Here’s a typical example: Uploading an image to cloud storage (AWS S3) as part of a social media app. There are three methods of file upload handling we’ll compare:
- Batching: Upload to API, Load into Memory, Then Upload to S3
- Streaming: Upload to API, Forward to S3 as a Stream
- Direct Storage Upload: API Pre-signs an Access Key, Client Uploads Directly to S3
While we reference specific vendors and technologies based on our own proven experiences, these principles apply to multiple cloud or storage providers.
The Case for Batching
Batching can seem like the most natural starting point for a social media platform handling user-generated content. It’s straightforward for developers to implement and offers complete control over the upload process. Files land on the API server first, providing an opportunity to validate content, apply security scans, or even add metadata before sending it to S3.
This method can be appealing in the early stages of building a platform because you know exactly what’s in the uploaded file content and can intercept bad actors or errors before they reach your storage. While it’s resource-intensive and might not scale infinitely, it's simplicity and centralized oversight make batching a reasonable choice when your traffic is low, or your API needs to process uploads anyway.
When to Batch
- Proof of concepts
- Simplicity in low-traffic environments
- Scenarios requiring immediate validation of the entire file, such as virus scanning
Batching Architecture
The Case for Streaming
As the platform grows, batching's limitations surface: memory costs, slower uploads, and latency. Streaming provides an elegant middle ground. Instead of holding onto a file until it’s completely uploaded, the API can forward chunks directly to S3 as soon as they arrive. File streaming reduces server memory usage while allowing the API to act as a gatekeeper, inspecting or enhancing the file in real time.
Streaming is like the scalable, “pro” version of batching. It delivers faster uploads for end users while maintaining enough control to enforce policies or transformations. Although it’s a bit more complex to implement, streaming can be a powerful, enterprise-ready approach for a social media app that balances user experience with security and compliance.
When to Stream
- Stream when batching is too slow or costly, but your API still needs file access for tasks like reading metadata or image classification.
- Stream when you need faster performance with less complex logging and error handling than direct storage uploads.
Streaming Architecture
The Case for Direct Storage Upload
Direct storage upload becomes the clear frontrunner when your platform reaches a scale where every millisecond counts. The API uses a cryptographic method to pre-sign keys accepted by the storage provider (S3). By generating pre-signed URLs with the temporary keys as query parameters, you hand the heavy lifting directly to the client, allowing uploads to bypass the API entirely. This approach minimizes server costs and eliminates the API as a bottleneck.
For social media platforms that thrive on high-volume uploads, this direct-to-S3 strategy reduces latency and improves reliability. However, direct storage upload does shift more responsibility to the client, and you lose some visibility during the upload process. Pairing it with post-upload callbacks can sometimes mitigate this. If scalability and performance are the ultimate goals, direct storage upload is the method to beat.
When to Use Direct Storage Upload
- Speed at Scale. The client makes a direct or nearly direct connection to cloud storage. Best performance for files that do not need any immediate manipulation before being stored or retrieved
- Simplified performant architecture when you have a global user base
- Scenarios with lower regulatory or security risks. We sometimes fall back to API streaming for additional controls in these environments.
Direct Storage Upload Architecture
Method Comparison