ALL Blog career CODE CRAFT System Design Interviews

System Design Interview Concepts – Load Balancing

Load balancing is among the founding pillars of designing distributed techniques. A load balancer merely distributes a set of requested operations (database write requests, cache queries) successfully throughout a set of servers.

Right here’s an instance of a shopper software accessing some server on the web without Load Balancing. In this case, the shopper software connects to the online server instantly.

No load balancing between Servers

There are two major problems with this mannequin for web sites servicing a really high number of requests:

  1. Single Level of Failure: If one thing happens to the Net server, the complete service becomes unavailable for certain time period. This is unacceptable for majority of online retailers and service suppliers.
  2. Overloaded Servers: The variety of requests that your  net server can entertain is often capped. There’s only a lot RAM and CPU you possibly can add to a single server. So as your enterprise grows, you’ll soon saturate your server’s potential to entertain requests. The one approach for you to service the growing variety of requests is so as to add a number of additional servers and have a load balancer distribute the requests to your cluster of servers.

The picture under exhibits how including a load balancer in entrance of your net servers may also help alleviate the above two points. Now you’ll be able to add any number of net servers behind your load balancer and make sure that even when one of the servers goes offline, your system as an entire continues to be servicing requests. Moreover, because you can now spread out the requests throughout a number of servers, your latency on every request will go down because each server shouldn’t be bottlenecked on RAM/Disk/CPU anymore.

Simple Load Balancing Scenario

Load balancing across multiple net entrance ends

The subsequent obvious question is what sort of workloads can you load stability? That’s, the place all can you place load balancers to realize excessive scalability ?

Load balancers are sometimes placed between:

  1. The shopper software/consumer and the online server
  2. The Net Server and the Software/Job servers
  3. The Software servers and the Cache Servers
  4. The Cache Servers the Database Servers
Where to place Load Balancers.

The place to put Load Balancers?

Observe that introducing load balancers at each of those 4 software layers might or is probably not mandatory depending on your system necessities. Load balancing at every layer will increase availability, efficiency and fault tolerance, nevertheless it also introduces extra complexity in the system. More complexity often interprets to extra value and maintenance overhead in the long run. There isn’t any free lunch -something to all the time bear in mind whereas designing any distributed system.

Load balancing could be achieved in three ways:

  1. Through the use of Software program Load balancers in shoppers which requests knowledge from an inventory of servers
  2. Through the use of Software Load balancers in providers layer
  3. Through the use of Hardware Load balancers in providers layer

Software Load Balancers in Shoppers

That is in all probability the most cost effective solution to implement load balancing . On this case, all logic in your load balancing resides on the shopper software. On startup, the shopper software (Eg. A mobile phone app) is provided with an inventory of net servers / software servers it will possibly talk with. The shopper app picks the primary one in the record and requests knowledge from the server. If a failure is detected persistently (after a configurable variety of retries), it marks the first server as unavailable and picks another server from the record to request knowledge from.

Software Load Balancers in Providers

Software load balancers are items of software that receives a set of requests and routes them in accordance with a algorithm. In contrast to hardware load balancers, software program load balancers do not require any particular

Sort of hardware – they are often put in on any home windows or linux machines. One has the choice of both utilizing an off-the-shelf software load balancer like HA Proxy or writing their very own custom software for

Load balancing particular workload varieties. For instance, when designing the Authentication platform for Microsoft Office365, we wrote a customized load balancer to load stability Lively Directory Queries.

Hardware Load Balancers

Hardware Load balancer gadget (HLD) is a physical gadget used to distribute net visitors throughout a cluster of network servers. HLDs principally current a digital server handle to the surface world and when shopper purposes attempt to connect, it will ahead the connection on probably the most applicable real server doing bi-directional community handle translation (NAT). HLDs, also referred to as Layer 4-7 Routers are sometimes capable of load stability not only HTTP and HTTPS visitors, but in addition TCP and UDP traffics. For example, TCP visitors to a database cluster might be unfold across all servers by a HLD.

The load balancer might management exactly which server acquired which connection and employed “health monitors” of increasing complexity to ensure that the appliance server (a real, bodily server) was responding as wanted; if not, it might routinely stop sending visitors to that server till it produced the desired response (indicating that the server was functioning correctly).

HLDs, while remarkably flexible when it comes to the sort and scale of load balancing they perform, are expensive to accumulate and configure. Because of this cause, most on-line service providers use HLDs at the first entry level of consumer requests into their infrastructure and then use inner software program load balancers to route knowledge behind their infrastructure wall.

For instance, SharePoint on-line (again in 2012) had one F5 Massive-IP Hardware Load Balancer in front of our net servers and used software load balancing in it’s software servers for load balancing across lively listing situations and databases.

Using a Load Balancer because the gateway to your inner cluster of net servers has the next advantages:

Facilitate Zero-downtime rolling updates to net servers: This is accomplished by effectively taking an internet server (due for maintainance) out of the load balancer pool, waiting for all of the lively connections to “drain i.e. service requeests in progress” after which safely shutting down the server. This manner, no shopper requests in flight are dropped and you may carry out patching/maiantenance on the internet servers with out affecting you excessive availability SLA.

Facilitate fast improve in capacity: Including extra net servers to DNS for load balacing purposes takes time to propagate. DNS is principally an Ultimately Constant system. Nevertheless, with Load balancers (hardware or software), as quickly as you add a brand new server, it will possibly begin servicing the shopper requests instantly. Thus, you’ll be able to improve your capacity on the flick of a change (properly virtually 🙂 ).

Enhance Fault Tolerance: Load balancers allow a fault net server occasion to be instantly taken out of rotation by removing it from the load balancer pool. That is a lot better than having to remove the server from DNS which takes time and during that window, the DNS will still be sending visitors to the faulty net server which can fail the shopper requests.

Scale back load on net servers by way of SSL termination: SSL offloading ( a.okay.a SSL termination) is a load balancer function that permits you to handle all SSL encryption/ decryption work on the load balancer and use un-encrypted connections internally between the load balancer and net servers. This removes a big load of the online servers who not have to absorb the overhead of visitors encryption/decryption. It’s additionally potential to offer SSL acceleration utilizing specialised hardware put in on the load balancer. Please take a look at https://kemptechnologies.com/solutions/ssl-acceleration-solutions/

Facilitate Simply In Time Load Balancing: If your net servers are hosted in the cloud by way of AWS or Azure, you possibly can add new workloads (net servers and entrance ends) relying on the load your system is experiencing. Should you use the elastic load balancer (ELB) in AWS or the cloud load balancer in Azure, the scaling can happen mechanically and just in time to accomodate your growing/reducing visitors. This automated load balancing has three benefits – no downtime and low latency on your clients, no IT upkeep for the load balancer because it’s hosted in AWS or Azure cloud and price savings as a result of the system scales down mechanically when visitors reduces.

Whether or not you’re utilizing a software program or hardware load balancer, it needs to determine which backend server to forward the request to. Totally different techniques may require alternative ways of selecting servers from the load balancer — hence the necessity for different load balancing algorithms. A number of the widespread load balancing algorithms are given under:

  • Round Robin: Requests are distributed across the group of servers sequentially.
  • Weighted Spherical Robin: similar as spherical robin but some servers get a much bigger share of the overall workload based mostly on some criteria.
  • Least Connections:  A brand new request is shipped to the server with the fewest current connections to shoppers. The relative computing capability of each server is factored into determining which one has the least connections. If a webnode fails and is taken out of service the distribution modifications. As long as all servers are operating a given shopper IP tackle will all the time go to the identical net server.
  • Quickest Response: The load balancer frequently pings the servers and mainatains a map of servers with least response occasions. Visitors is routed to the servers in clusters with least response occasions.
  • IP Hash: The IP tackle of the shopper is used to determine which server receives the request.
  • URL hash: That is like source IP hash, besides hashing is completed on the URL of the request. Helpful when load balancing in entrance of proxy caches, as requests for a given object will all the time go to just one backend cache. This avoids cache duplication, having the identical object stored in a number of / all caches, and increases effective capability of the backend caches.
  • Consistent Hashing: Take a look at the Constant Hashing submit for a detailed rationalization. Also, right here’s a analysis paper from Google explaining how Vimeo solved their load balancing drawback using a variant of this system.

In abstract, you’ll virtually ALWAYS be asked some kind of scalability question in system design interviews for which you’ll want to use a load balancer. The important thing things to recollect from this article are:

  • Load balancing allows elastic scalability and redundancy (you’ll be able to have many copies of the identical knowledge). Elastic scalability improves efficiency and throughput of knowledge. Redundancy improves availability and in addition helps in backup/restore of service in case a couple of servers fail.
  • Load balancers might be placed at any software layer – check with the section above for particulars.
  • Load balancers may be carried out in hardware or software program. Many corporations uses both at totally different scale points of their system.

Load balancing is likely one of the founding pillars of designing distributed techniques. A load balancer merely distributes a set of requested operations (database write requests, cache queries) successfully across a set of servers.

Here’s an example of a shopper software accessing some server on the internet without Load Balancing. In this case, the shopper software connects to the online server instantly.

No Load Balancing

No load balancing between Servers

There are two essential problems with this mannequin for websites servicing a really excessive number of requests:

  1. Single Level of Failure: If one thing happens to the Net server, the whole service becomes unavailable for sure time period. That is unacceptable for majority of on-line retailers and repair suppliers.
  2. Overloaded Servers: The number of requests that your  net server can entertain is often capped. There’s solely a lot RAM and CPU you’ll be able to add to a single server. In order your small business grows, you’ll soon saturate your server’s potential to entertain requests. The only approach so that you can service the growing number of requests is so as to add a number of additional servers and have a load balancer distribute the requests to your cluster of servers.

The image under exhibits how adding a load balancer in front of your net servers will help alleviate the above two points. Now you possibly can add any number of net servers behind your load balancer and make sure that even when one of the servers goes offline, your system as an entire continues to be servicing requests. Moreover, because now you can spread out the requests across a number of servers, your latency on each request will go down because every server shouldn’t be bottlenecked on RAM/Disk/CPU anymore.

Simple Load Balancing Scenario

Load balancing across multiple net front ends

The subsequent obvious query is what sort of workloads are you able to load stability? That is, the place all are you able to place load balancers to realize excessive scalability ?

Load balancers are sometimes positioned between:

  1. The shopper software/consumer and the online server
  2. The Net Server and the Software/Job servers
  3. The Software servers and the Cache Servers
  4. The Cache Servers the Database Servers

Where to place Load Balancers.

Where to put Load Balancers?

Word that introducing load balancers at every of those 4 software layers might or is probably not needed relying on your system necessities. Load balancing at each layer will increase availability, efficiency and fault tolerance, however it additionally introduces more complexity in the system. Extra complexity often translates to extra value and upkeep overhead in the long term. There isn’t a free lunch -something to all the time take note whereas designing any distributed system.

Load balancing might be achieved in three ways:

  1. Through the use of Software Load balancers in shoppers which requests knowledge from an inventory of servers
  2. Through the use of Software Load balancers in providers layer
  3. Through the use of Hardware Load balancers in providers layer

Software Load Balancers in Shoppers

That is in all probability the most cost effective strategy to implement load balancing . In this case, all logic in your load balancing resides on the shopper software. On startup, the shopper software (Eg. A cell phone app) is supplied with an inventory of net servers / software servers it could possibly communicate with. The shopper app picks the first one within the record and requests knowledge from the server. If a failure is detected persistently (after a configurable variety of retries), it marks the first server as unavailable and picks another server from the record to request knowledge from.

Software Load Balancers in Providers

Software load balancers are items of software program that receives a set of requests and routes them based on a set of rules. In contrast to hardware load balancers, software program load balancers do not require any particular

Sort of hardware – they can be installed on any windows or linux machines. One has the option of either utilizing an off-the-shelf software program load balancer like HA Proxy or writing their own customized software for

Load balancing specific workload varieties. For instance, when designing the Authentication platform for Microsoft Workplace365, we wrote a custom load balancer to load stability Lively Directory Queries.

Hardware Load Balancers

Hardware Load balancer system (HLD) is a bodily gadget used to distribute net visitors across a cluster of community servers. HLDs principally present a virtual server tackle to the surface world and when shopper purposes attempt to connect, it will forward the connection on probably the most applicable actual server doing bi-directional community tackle translation (NAT). HLDs, also referred to as Layer 4-7 Routers are sometimes capable of load stability not only HTTP and HTTPS visitors, but in addition TCP and UDP traffics. For instance, TCP visitors to a database cluster may be unfold across all servers by a HLD.

The load balancer might control exactly which server acquired which connection and employed “health monitors” of accelerating complexity to make sure that the appliance server (a real, bodily server) was responding as needed; if not, it will routinely cease sending visitors to that server till it produced the specified response (indicating that the server was functioning correctly).

HLDs, whereas remarkably flexible when it comes to the sort and scale of load balancing they perform, are costly to accumulate and configure. Due to this cause, most on-line service suppliers use HLDs on the first entry level of consumer requests into their infrastructure and then use inner software program load balancers to route knowledge behind their infrastructure wall.

For example, SharePoint on-line (again in 2012) had one F5 Huge-IP Hardware Load Balancer in entrance of our net servers and used software program load balancing in it’s software servers for load balancing across lively directory situations and databases.

Using a Load Balancer because the gateway to your inner cluster of net servers has the next advantages:Facilitate Zero-downtime rolling updates to net servers: That is achieved by successfully taking an internet server (due for maintainance) out of the load balancer pool, ready for all the lively connections to “drain i.e. service requeests in progress” and then safely shutting down the server. This manner, no shopper requests in flight are dropped and you may carry out patching/maiantenance on the internet servers without affecting you high availability SLA.Facilitate fast improve in capacity: Adding more net servers to DNS for load balacing functions takes time to propagate. DNS is principally an Ultimately Consistent system. Nevertheless, with Load balancers (hardware or software), as soon as you add a brand new server, it could actually begin servicing the shopper requests immediately. Thus, you’ll be able to improve your capacity at the flick of a change (properly virtually 🙂 ).Enhance Fault Tolerance: Load balancers allow a fault net server instance to be instantly taken out of rotation by removing it from the load balancer pool. That is a lot better than having to take away the server from DNS which takes time and through that window, the DNS will nonetheless be sending visitors to the defective net server which can fail the shopper requests. Scale back load on net servers via SSL termination: SSL offloading ( a.okay.a SSL termination) is a load balancer function that lets you deal with all SSL encryption/ decryption work on the load balancer and use un-encrypted connections internally between the load balancer and net servers. This removes a big load of the online servers who not have to absorb the overhead of visitors encryption/decryption. It’s also attainable to offer SSL acceleration utilizing specialised hardware installed on the load balancer. Please take a look at https://kemptechnologies.com/solutions/ssl-acceleration-solutions/Facilitate Just In Time Load Balancing: If your net servers are hosted in the cloud by way of AWS or Azure, you’ll be able to add new workloads (net servers and entrance ends) relying on the load your system is experiencing. In case you use the elastic load balancer (ELB) in AWS or the cloud load balancer in Azure, the scaling can occur mechanically and simply in time to accomodate your growing/reducing visitors. This automated load balancing has three advantages – no downtime and low latency on your clients, no IT maintenance for the load balancer because it’s hosted in AWS or Azure cloud and price savings as a result of the system scales down routinely when visitors reduces.

Whether or not you’re utilizing a software program or hardware load balancer, it needs to determine which backend server to forward the request to. Totally different techniques may require alternative ways of choosing servers from the load balancer — therefore the necessity for various load balancing algorithms. A number of the widespread load balancing algorithms are given under:

  • Spherical Robin: Requests are distributed across the group of servers sequentially.
  • Weighted Round Robin: similar as spherical robin however some servers get a much bigger share of the general workload based mostly on some standards.
  • Least Connections:  A new request is shipped to the server with the fewest present connections to shoppers. The relative computing capacity of every server is factored into figuring out which one has the least connections. If a webnode fails and is taken out of service the distribution modifications. As long as all servers are operating a given shopper IP tackle will all the time go to the identical net server.
  • Fastest Response: The load balancer repeatedly pings the servers and mainatains a map of servers with least response occasions. Visitors is routed to the servers in clusters with least response occasions.
  • IP Hash: The IP tackle of the shopper is used to determine which server receives the request.
  • URL hash: That is like supply IP hash, besides hashing is completed on the URL of the request. Useful when load balancing in front of proxy caches, as requests for a given object will all the time go to just one backend cache. This avoids cache duplication, having the identical object stored in several / all caches, and will increase efficient capacity of the backend caches.
  • Constant Hashing: Take a look at the Constant Hashing submit for an in depth rationalization. Also, right here’s a analysis paper from Google explaining how Vimeo solved their load balancing drawback utilizing a variant of this system.

In summary, you’ll virtually ALWAYS be requested some type of scalability query in system design interviews for which you’ll want to use a load balancer. The key issues to remember from this article are:

  • Load balancing allows elastic scalability and redundancy (you’ll be able to have many copies of the identical knowledge). Elastic scalability improves efficiency and throughput of knowledge. Redundancy improves availability and in addition helps in backup/restore of service in case a couple of servers fail.
  • Load balancers could be positioned at any software program layer – check with the section above for particulars.
  • Load balancers may be carried out in hardware or software. Many corporations makes use of each at totally different scale points of their system.