ByteByteGo

Top 12 Tips for API Security
.
.
- Use HTTPS
- Use OAuth2
- Use WebAuthn
- Use Leveled API Keys
- Authorization
- Rate Limiting
- API Versioning
- Whitelisting
- Check OWASP API Security Risks
- Use API Gateway
- Error Handling
- Input Validation

–
Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): https://bit.ly/3KCnWXq

#systemdesign #coding #interviewtips
.

Netflix Tech Stack - (CI/CD Pipeline)

Planing: Netflix Engineering uses JIRA for planning and Confluence for documentation.

Coding: Java is the primary programming language for the backend service, while other languages are used for different use cases.

Build: Gradle is mainly used for building, and Gradle plugins are built to support various use cases.

Packaging: Package and dependencies are packed into an Amazon Machine Image (AMI) for release.

Testing: Testing emphasizes the production culture's focus on building chaos tools.

Deployment: Netflix uses its self-built Spinnaker for canary rollout deployment.

Monitoring: The monitoring metrics are centralized in Atlas, and Kayenta is used to detect anomalies.

Incident report: Incidents are dispatched according to priority, and PagerDuty is used for incident handling.

—
Get a Free System Design PDF (158 pages) by subscribing to our weekly newsletter today: https://bit.ly/42Ex9oZ

#systemdesign #coding #interviewtips
.

Recently discovered a cheat sheet that covers many design patterns, intended to jog your memory about how different patterns work.

—
Get a Free System Design PDF (158 pages) by subscribing to our weekly newsletter today: https://lnkd.in/g9wAgcke

#systemdesign #coding #interviewtips
.

IBM MQ -> RabbitMQ -> Kafka ->Pulsar, How do message queue architectures evolve? 
 
🔹 IBM MQ 
IBM MQ was launched in 1993. It was originally called MQSeries and was renamed WebSphere MQ in 2002. It was renamed to IBM MQ in 2014. IBM MQ is a very successful product widely used in the financial sector. Its revenue still reached 1 billion dollars in 2020. 
 
🔹 RabbitMQ 
RabbitMQ architecture differs from IBM MQ and is more similar to Kafka concepts. The producer publishes a message to an exchange with a specified exchange type. It can be direct, topic, or fanout. The exchange then routes the message into the queues based on different message attributes and the exchange type. The consumers pick up the message accordingly. 
 
🔹 Kafka 
In early 2011, LinkedIn open sourced Kafka, which is a distributed event streaming platform. It was named after Franz Kafka. As the name suggested, Kafka is optimized for writing. It offers a high-throughput, low-latency platform for handling real-time data feeds. It provides a unified event log to enable event streaming and is widely used in internet companies. 
 
Kafka defines producer, broker, topic, partition, and consumer. Its simplicity and fault tolerance allow it to replace previous products like AMQP-based message queues. 
 
🔹 Pulsar 
Pulsar, developed originally by Yahoo, is an all-in-one messaging and streaming platform. Compared with Kafka, Pulsar incorporates many useful features from other products and supports a wide range of capabilities. Also, Pulsar architecture is more cloud-native, providing better support for cluster scaling and partition migration, etc. 
 
There are two layers in Pulsar architecture: the serving layer and the persistent layer. Pulsar natively supports tiered storage, where we can leverage cheaper object storage like AWS S3 to persist messages for a longer term. 
 
Over to you: which message queues have you used? 
 
– 
Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): https://bit.ly/3KCnWXq 
 
#systemdesign #coding #interviewtips 
.

What does API gateway do?
.
.
The diagram below shows the detail.

Step 1 - The client sends an HTTP request to the API gateway.

Step 2 - The API gateway parses and validates the attributes in the HTTP request.

Step 3 - The API gateway performs allow-list/deny-list checks.

Step 4 - The API gateway talks to an identity provider for authentication and authorization.

Step 5 - The rate limiting rules are applied to the request. If it is over the limit, the request is rejected.

Steps 6 and 7 - Now that the request has passed basic checks, the API gateway finds the relevant service to route to by path matching.

Step 8 - The API gateway transforms the request into the appropriate protocol and sends it to backend microservices.

Steps 9-12: The API gateway can handle errors properly, and deals with faults if the error takes a longer time to recover (circuit break). It can also leverage ELK (Elastic-Logstash-Kibana) stack for logging and monitoring. We sometimes cache data in the API gateway.

Over to you: 1) What’s the difference between a load balancer and an API gateway?
2) Do we need to use different API gateways for PC, mobile and browser separately?

– 
Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): https://bit.ly/3KCnWXq 
 
#systemdesign #coding #interviewtips 
.

Would it be nice if the code we wrote automatically turned into architecture diagrams?

I recently discovered a Github repo that does exactly this: Diagram as Code for prototyping cloud system architectures.

𝐖𝐡𝐚𝐭 𝐝𝐨𝐞𝐬 𝐢𝐭 𝐝𝐨?
- Draw the cloud system architecture in Python code.
- Diagrams can also be rendered directly inside the Jupyter Notebooks.
- No design tools are needed. 
- Supports the following providers: AWS, Azure, GCP, Kubernetes, Alibaba Cloud, Oracle Cloud, etc. 
 
Github repo: mingrammer/diagrams

– 
Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): https://bit.ly/3KCnWXq 
 
#systemdesign #coding #interviewtips 
.

Would it be nice if the code we wrote automatically turned into architecture diagrams?

A Github repo that does exactly this: Diagram as Code for prototyping cloud system architectures.

𝐖𝐡𝐚𝐭 𝐝𝐨𝐞𝐬 𝐢𝐭 𝐝𝐨?
- Draw the cloud system architecture in Python code.
- Diagrams can also be rendered directly inside the Jupyter Notebooks.
- No design tools are needed. 
- Supports the following providers: AWS, Azure, GCP, Kubernetes, Alibaba Cloud, Oracle Cloud, etc.

Github repo link in the comment.

--
Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): https://bit.ly/3KCnWXq

#systemdesign #coding #interviewtips
.

Visualizing a SQL query
.
.
SQL statements are executed by the database system in several steps, including:

- Parsing the SQL statement and checking its validity
- Transforming the SQL into an internal representation, such as relational algebra
- Optimizing the internal representation and creating an execution plan that utilizes index information

Executing the plan and returning the results
- The execution of SQL is highly complex and involves many considerations, such as:
- The use of indexes and caches
- The order of table joins
- Concurrency control
- Transaction management

Over to you: what is your favorite SQL statement?

–
Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): https://bit.ly/3KCnWXq

#systemdesign #coding #interviewtips
.

REST API Vs. GraphQL 
 
When it comes to API design, REST and GraphQL each have their own strengths and weaknesses. 
 
REST 
- Uses standard HTTP methods like GET, POST, PUT, DELETE for CRUD operations. 
- Works well when you need simple, uniform interfaces between separate services/applications. 
- Caching strategies are straightforward to implement. 
- The downside is it may require multiple roundtrips to assemble related data from separate endpoints. 
 
GraphQL 
- Provides a single endpoint for clients to query for precisely the data they need. 
- Clients specify the exact fields required in nested queries, and the server returns optimized payloads containing just those fields. 
- Supports Mutations for modifying data and Subscriptions for real-time notifications. 
- Great for aggregating data from multiple sources and works well with rapidly evolving frontend requirements. 
- However, it shifts complexity to the client side and can allow abusive queries if not properly safeguarded 
- Caching strategies can be more complicated than REST. 
 
The best choice between REST and GraphQL depends on the specific requirements of the application and development team. GraphQL is a good fit for complex or frequently changing frontend needs, while REST suits applications where simple and consistent contracts are preferred.

– 
Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): https://bit.ly/3KCnWXq 
 
#systemdesign #coding #interviewtips 
.

Which HTTP status codes are most common?
.
.
The response codes for HTTP are divided into five categories:

Informational (100-199)
Success (200-299)
Redirection (300-399)
Client Error (400-499)
Server Error (500-599)

These codes are defined in RFC 9110. To save you from reading the entire document (which is about 200 pages), here is a summary of the most common ones:

Over to you: HTTP status code 401 is for Unauthorized. Can you explain the difference between authentication and authorization, and which one does code 401 check for?

–
Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): https://bit.ly/42Ex9oZ

#systemdesign #coding #interviewtips
.

What is DevSecOps?
.
.
DevSecOps emerged as a natural evolution of DevOps practices with a focus on integrating security into the software development and deployment process. The term “DevSecOps“ represents the convergence of Development (Dev), Security (Sec), and Operations (Ops) practices, emphasizing the importance of security throughout the software development lifecycle.

The diagram below shows the important concepts in DevSecOps.

1 . Automated Security Checks
2 . Continuous Monitoring
3 . CI/CD Automation
4 . Infrastructure as Code (IaC)
5 . Container Security
6 . Secret Management
7 . Threat Modeling
8. Quality Assurance (QA) Integration
9 . Collaboration and Communication
10 . Vulnerability Management

– 
Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): https://bit.ly/3KCnWXq 
 
#systemdesign #coding #interviewtips 
.

Domain-Driven Design (DDD)
.
.
DDD was introduced in Eric Evans’ classic book “Domain-Driven Design: Tackling Complexity in the Heart of Software”. It explained a methodology to model a complex business. There is a lot of content in this book, so I'll summarize the basics.

The composition of domain objects:
- Entity: a domain object that has ID and life cycle. 
- Value Object: a domain object without ID. It is used to describe the property of Entity.
- Aggregate: a collection of Entities that are bounded together by Aggregate Root (which is also an entity). It is the unit of storage.

The life cycle of domain objects:
- Repository: storing and loading the Aggregate.
- Factory: handling the creation of the Aggregate.

Behavior of domain objects:
- Domain Service: orchestrate multiple Aggregate.
- Domain Event: a description of what has happened to the Aggregate. The publication is made public so others can consume and reconstruct it.

Congratulations on getting this far. Now you know the basics of DDD. If you want to learn more, I highly recommend the book. It might help to simplify the complexity of software modeling.

–
Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): https://bit.ly/42Ex9oZ

#systemdesign #coding #interviewtips
.

7 must-know strategies to scale your database. 
 
1 - Indexing: 
Check the query patterns of your application and create the right indexes. 
 
2 - Materialized Views: 
Pre-compute complex query results and store them for faster access. 
 
3 - Denormalization: 
Reduce complex joins to improve query performance. 
 
4 - Vertical Scaling 
Boost your database server by adding more CPU, RAM, or storage. 
 
5 - Caching 
Store frequently accessed data in a faster storage layer to reduce database load. 
 
6 - Replication 
Create replicas of your primary database on different servers for scaling the reads. 
 
7 - Sharding 
Split your database tables into smaller pieces and spread them across servers. Used for scaling the writes as well as the reads. 
 
Over to you: What other strategies do you use for scaling your databases?

-- 
Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): https://bit.ly/3KCnWXq

#systemdesign #coding #interviewtips 
.

Where do we cache data?
.
.
This diagram illustrates where we cache data in a typical architecture.

There are multiple layers along the flow.

Client apps: HTTP responses can be cached by the browser. We request data over HTTP for the first time, and it is returned with an expiry policy in the HTTP header; we request data again, and the client app tries to retrieve the data from the browser cache first.

CDN: CDN caches static web resources. The clients can retrieve data from a CDN node nearby.

Load Balancer: The load Balancer can cache resources as well.

Messaging infra: Message brokers store messages on disk first, and then consumers retrieve them at their own pace. Depending on the retention policy, the data is cached in Kafka clusters for a period of time.

Services: There are multiple layers of cache in a service. If the data is not cached in the CPU cache, the service will try to retrieve the data from memory. Sometimes the service has a second-level cache to store data on disk.

Distributed Cache: Distributed cache like Redis hold key-value pairs for multiple services in memory. It provides much better read/write performance than the database.

Full-text Search: we sometimes need to use full-text searches like Elastic Search for document search or log search. A copy of data is indexed in the search engine as well.

Database: Even in the database, we have different levels of caches:

- WAL(Write-ahead Log): data is written to WAL first before building the B tree index
- Bufferpool: A memory area allocated to cache query results
- Materialized View: Pre-compute query results and store them in the database tables for better query performance

Transaction log: record all the transactions and database updates

Replication Log: used to record the replication state in a database cluster

Over to you: With the data cached at so many levels, how can we guarantee the sensitive user data is completely erased from the systems?

--
Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): https://bit.ly/42Ex9oZ

#systemdesign #coding #interviewtips
.

How is data sent over the internet? What does that have to do with the OSI model? How does TCP/IP fit into this? 
 
7 Layers in the OSI model are: 
1. Physical Layer 
2. Data Link Layer 
3. Network Layer 
4. Transport Layer 
5. Session Layer 
6. Presentation Layer 
7. Application Layer

-- 
Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): https://bit.ly/3KCnWXq 
 
#systemdesign #coding #interviewtips 
.

How does Docker work?

The diagram below shows the architecture of Docker and how it works when we run “docker build”, “docker pull” and “docker run”.

There are 3 components in Docker architecture:

🔹 Docker client
The docker client talks to the Docker daemon.

🔹 Docker host
The Docker daemon listens for Docker API requests and manages Docker objects such as images, containers, networks, and volumes.

🔹 Docker registry
A Docker registry stores Docker images. Docker Hub is a public registry that anyone can use.

Let’s take the “docker run” command as an example.
1. Docker pulls the image from the registry.
2. Docker creates a new container.
3. Docker allocates a read-write filesystem to the container.
4. Docker creates a network interface to connect the container to the default network.
5. Docker starts the container.

–
Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): https://bit.ly/3KCnWXq

#systemdesign #coding #interviewtips
.

What distinguishes MVC, MVP, MVVM, MVVM-C, and VIPER architecture patterns from each other?

These architecture patterns are among the most commonly used in app development, whether on iOS or Android platforms. Developers have introduced them to overcome the limitations of earlier patterns. So, how do they differ?

- MVC, the oldest pattern, dates back almost 50 years
- Every pattern has a “view“ (V) responsible for displaying content and receiving user input
- Most patterns include a “model“ (M) to manage business data
- “Controller,“ “presenter,“ and “view-model“ are translators that mediate between the view and the model (“entity“ in the VIPER pattern)
- These translators can be quite complex to write, so various patterns have been proposed to make them more maintainable

--
Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): https://bit.ly/3KCnWXq

#systemdesign #coding #interviewtips
.

Authentication in REST APIs acts as the crucial gateway, ensuring that solely authorized users or applications gain access to the API's resources. 
 
Some popular authentication methods for REST APIs include: 
 
1. Basic Authentication: 
Involves sending a username and password with each request, but can be less secure without encryption. 
 
When to use: 
Suitable for simple applications where security and encryption aren’t the primary concern or when used over secured connections. 
 
2. Token Authentication: 
Uses generated tokens, like JSON Web Tokens (JWT), exchanged between client and server, offering enhanced security without sending login credentials with each request. 
 
When to use: 
Ideal for more secure and scalable systems, especially when avoiding sending login credentials with each request is a priority. 
 
3. OAuth Authentication: 
Enables third-party limited access to user resources without revealing credentials by issuing access tokens after user authentication. 
 
When to use: 
Ideal for scenarios requiring controlled access to user resources by third-party applications or services. 
 
4. API Key Authentication: 
Assigns unique keys to users or applications, sent in headers or parameters; while simple, it might lack the security features of token-based or OAuth methods. 
 
When to use: 
Convenient for straightforward access control in less sensitive environments or for granting access to certain functionalities without the need for user-specific permissions. 
 
Over to you: 
Which REST API authentication method do you find most effective in ensuring both security and usability for your applications?

--
Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): https://bit.ly/bbg-social

#systemdesign #coding #interviewtips 
.

Top 6 Load Balancing Algorithms.

🔹 Static Algorithms
1. Round robin
The client requests are sent to different service instances in sequential order. The services are usually required to be stateless.

2. Sticky round-robin
This is an improvement of the round-robin algorithm. If Alice’s first request goes to service A, the following requests go to service A as well.

3. Weighted round-robin
The admin can specify the weight for each service. The ones with a higher weight handle more requests than others.

4. Hash
This algorithm applies a hash function on the incoming requests’ IP or URL. The requests are routed to relevant instances based on the hash function result.

🔹 Dynamic Algorithms
5. Least connections
A new request is sent to the service instance with the least concurrent connections.

6. Least response time
A new request is sent to the service instance with the fastest response time.

--
Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): https://bit.ly/3KCnWXq

#systemdesign #coding #interviewtips
.

Have you heard of Domain-Driven Design (DDD), a major software design approach?

DDD was introduced in Eric Evans’ classic book “Domain-Driven Design: Tackling Complexity in the Heart of Software”. It explained a methodology to model a complex business. In this book, there is a lot of content, so I'll summarize the basics.

𝐓𝐡𝐞 𝐜𝐨𝐦𝐩𝐨𝐬𝐢𝐭𝐢𝐨𝐧 𝐨𝐟 𝐝𝐨𝐦𝐚𝐢𝐧 𝐨𝐛𝐣𝐞𝐜𝐭𝐬:
🔹Entity: a domain object that has ID and life cycle. 
🔹Value Object: a domain object without ID. It is used to describe the property of Entity.
🔹Aggregate: a collection of Entities that are bounded together by Aggregate Root (which is also an entity). It is the unit of storage.

𝐓𝐡𝐞 𝐥𝐢𝐟𝐞 𝐜𝐲𝐜𝐥𝐞 𝐨𝐟 𝐝𝐨𝐦𝐚𝐢𝐧 𝐨𝐛𝐣𝐞𝐜𝐭𝐬:
🔹Repository: storing and loading the Aggregate.
🔹Factory: handling the creation of the Aggregate.

𝐁𝐞𝐡𝐚𝐯𝐢𝐨𝐫 𝐨𝐟 𝐝𝐨𝐦𝐚𝐢𝐧 𝐨𝐛𝐣𝐞𝐜𝐭𝐬:
🔹Domain Service: orchestrate multiple Aggregate.
🔹Domain Event: a description of what has happened to the Aggregate. The publication is made public so others can consume and reconstruct it.

Congratulations on getting this far. Now you know the basics of DDD. If you want to learn more, I highly recommend the book. It might help to simplify the complexity of software modeling.

Over to you: do you know how to check the equality of two Value Objects? How about two Entities?

– 
Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): https://bit.ly/3KCnWXq 
 
#systemdesign #coding #interviewtips
.

8 Key Data Structures That Power Modern Databases
.
.
🔹Skiplist: a common in-memory index type. Used in Redis
🔹Hash index: a very common implementation of the “Map” data structure (or “Collection”)
🔹SSTable: immutable on-disk “Map” implementation
🔹LSM tree: Skiplist + SSTable. High write throughput
🔹B-tree: disk-based solution. Consistent read/write performance
🔹Inverted index: used for document indexing. Used in Lucene
🔹Suffix tree: for string pattern search
🔹R-tree: multi-dimension search, such as finding the nearest neighbor

– 
Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): https://bit.ly/3KCnWXq 
 
#systemdesign #coding #interviewtips 
.