Save big on S3 traffic costs with a caching proxy
This proxy component is part of a series of proxies that are modular in design and require almost zero configuration. The series currently includes an SSL terminator, forward proxy and a caching proxy. Over time, I will add a a rate limiter, a loadbalancer/router, and a CDN proxy/image resizer.
Learn how to reduce S3 data transfer expenses by deploying your own caching proxy. This guide shows you the docker compose file with minimal configuration.
Utilizing AWS S3 for storing your images and other static files is beneficial due to its scalability and reliability. Using cloud storage eliminates the need for provisioning large disk space on your own servers.
However, the cost of data transfer can be a concern, especially with AWS charging $0.09 per gigabyte, potentially leading to $90 in monthly expenses for each terabyte of data transfer.
To mitigate these costs, implementing a caching proxy allows for serving cached files, which can significantly reduce expenses. For instance, using a $5 per month LightSail server, which includes 2TB of traffic, you could save up to $90 monthly if even 50% of your data is served from cache, with greater savings possible at higher cache hit rates.
You can establish a self-managed proxy chain using a concise setup of only 46 lines of code with my modular docker-based caching components.
This proxy chain is comprised of three elements:
• An SSL terminator to secure cached file delivery via HTTPS.
• A caching proxy that manages the caching process.
• A forward proxy that forwards requests to S3 or another backend, and also is capable of handling challenges like SNI certificates and changing load balancer ip addresses.
Before you begin, you will need:
• An S3 bucket filled with static files that allows public read access
• A Linux server, such as AWS LightSail, which offers a sufficient blend of traffic and storage within its pricing
• Docker installed on your Linux server
• Your own SSL server certificate (recommended)
Once you configure a basic Docker Compose file on your server, your proxy will be operational, enabling the serving of cached S3 files. This method allows you to benefit from the traffic costs associated with your server (e.g., approximately $2.50/TB for LightSail) rather than the higher costs of S3 direct access ($90/TB, which is 36 times more costly).
View the docker compose file to create the proxy chain at this github gist.
This post outlines the strategies and thought processes behind my DevOps projects. It reflects my dedication to adopting best practices in DevOps and staying aligned with the latest technological advancements.