While going through our AWS bills, I noticed a trend. Our database costs were going up linearly.
I looked at it positively. Our user base is increasing, leading to more activity. And more activity means more costs, right?
So I asked our DB to list its most popular queries. Here's what I saw on top:
UPDATE user_activities SET last_seen = CURRENT_TIMESTAMP WHERE user_id = ?
This query runs every time a client sends a request. For our system, that's a million queries per month.
That explains the bills 😛
--- What did we do?
The first thing we did was comment out the query line. To save money, you know.
Then we started brainstorming. One of the ideas was using AWS kinesis to store events, and Athena to process them later. But we did something much simpler.
We persisted the data as a log line in the server's local file system. Periodically, we aggregate these logs, sample them, and persist them in the database.
With a sampling factor of 1:50, we reduced our costs significantly. With aggregation and batched writes, we reduced costs even further.
As a startup, a 200$ cost reduction per month is worth fighting for!
--- Generalised solution
The idea is simple:
1. Persist data in a durable store (quickly).
2. Replicate this data into a different store, optimized for reads.
3. Clients can now read this data (eventually).
We call it Deferred Visibility. It is a deliberate delay in processing updates, trading off high consistency for low latency and fast persistence.
This technique is also part of popular solutions like WAL, CQRS, and CDC. We got inspired by WAL in particular!
To learn more about design patterns, try the System Design Course at InterviewReady.
Cheers!
#SystemDesign #DesignPatterns #CostOptimisations
I looked at it positively. Our user base is increasing, leading to more activity. And more activity means more costs, right?
So I asked our DB to list its most popular queries. Here's what I saw on top:
UPDATE user_activities SET last_seen = CURRENT_TIMESTAMP WHERE user_id = ?
This query runs every time a client sends a request. For our system, that's a million queries per month.
That explains the bills 😛
--- What did we do?
The first thing we did was comment out the query line. To save money, you know.
Then we started brainstorming. One of the ideas was using AWS kinesis to store events, and Athena to process them later. But we did something much simpler.
We persisted the data as a log line in the server's local file system. Periodically, we aggregate these logs, sample them, and persist them in the database.
With a sampling factor of 1:50, we reduced our costs significantly. With aggregation and batched writes, we reduced costs even further.
As a startup, a 200$ cost reduction per month is worth fighting for!
--- Generalised solution
The idea is simple:
1. Persist data in a durable store (quickly).
2. Replicate this data into a different store, optimized for reads.
3. Clients can now read this data (eventually).
We call it Deferred Visibility. It is a deliberate delay in processing updates, trading off high consistency for low latency and fast persistence.
This technique is also part of popular solutions like WAL, CQRS, and CDC. We got inspired by WAL in particular!
To learn more about design patterns, try the System Design Course at InterviewReady.
Cheers!
#SystemDesign #DesignPatterns #CostOptimisations
