Preface
The idea here is to build a working architecture from first principles. I'm still learning on how to build stuff without the obvious bottlenecks, and this is an attempt at that. There’s benefits and tradeoffs to what I’ve built, and I’ve tried discussing them out below.
What and why?
The idea behind why we need a load balancer is relatively simple. We have an application backend server (this could also be a database server), which can handle X amount of throughput, and only has finite hardware resources. This means that eventually, we would hit a point where the hardware for that one backend server is not enough, and would need to distribute load between multiple backend servers. That’s where the concept of a load balancer comes in. However, beyond this knowledge, how is load balancing actually done? I read about one of the approaches for load balancing: consistent hashing, and this write-up explores how consistent hashing works not just in theory, but with a first principles implementation written in Golang.
Round Robin & IP Hashing LB algorithms
One of the simpler load balancing algorithms is a round robin. As the name suggests, if we had 3 backend application servers, and 3 requests, each request would go to each of the backend servers. For the 4th request in, the request would go to the 1st server again.
As displayed in the image, the Requests R1, R2, and R3 go to Servers S1, S2, and S3, and the Request R4 goes to S1.
Whereas for IP Hashing, all requests coming from the same IP address are mapped to a single server. For instance, if I, Aneesh, connected to this application server, all my network requests as long as I’m on this same IP address will go to this backend server.
As seen in the image, both requests from R1 (assuming R1 is a user here) are sent to Server I, since they both come from the same user (given the user R1’s IP remains the same for both those requests).
Connection Churn - the problem with Round Robin & IP Hashing
Case 1: Assume one of the servers went down due to network failures. Now, I’m left with 2 other application servers. What would happen here is that all our load that existed on server 2 would get pushed onto the next, server S3. This is an issue, because there is a sudden increase in the load of S3, which would add onto the already existing load that’s meant to be going on S3, and also, all incoming requests that were meant to be going to S2 would now go to S3, which is a problem.
Case 2: If using the IP Hashing algorithm, how it works on the load balancer level is that the IP addresses are hashed (any hashing algorithm could work in theory: murmurhash, SHA256, etc.), and performed a modulo on by the number of servers that exist. If we had IP address ABC, and on hashing, we got back a value D, performing D % X where X is the number of servers would give us a server to send the request to.
Assume we added a new server S4. Since now X, which represents the number of servers changed, our load balancing algorithm which would’ve earlier for an IP address XYZ given server 3, may now give server 4 because our value of X changed. And if we have active/persistent connections, the hash values will be recalculated across all servers, potentially reassigning several clients (almost as much as n clients, where n represents the number of clients) to new servers and breaking active connections.
Since load balancing also exists on databases, and that could determine on what node our data is stored, if we add another node, we would be referring to moving almost ALL data from one node to another, which is highly unoptimal.
The intuition to the solution
Since the issue here is that if we add or remove a server, the connection churn means that a lot of connections/data gets reassigned, and in some cases, all to a single server whichever was the next one. Hence, what we’re looking for is a way to reduce the number of reassigned connections and avoid all the load going to a single server if one goes down.
Addressing the problem, our solution would somehow want that if I had X connections on S2, I only want a portion of X to go to S3 if S2 went down. That’s possible if the other portions go to S1, and maybe S4 if it exists.
This is where consistent hashing comes into the picture.
The theory & diagram behind Consistent Hashing
Consistent hashing involves a circle representation of the servers we have. This circle representation is only theoretical, and the actual implementation involves using hash functions in a hash space. But I’ll get to that in a bit.
In this theoretical representation with a circle, we have servers present at different points in the circle. When we receive a request, using our request id, we get an arbitrary point on this circle (how that’s done using hash functions is what I’ll get to later), and that request is placed on that arbitrary point. Once these requests are placed at the arbitrary point, the request is resolved by the server node that exists closest to the right.
The above image shows a bunch of requests which through some black box get to a point within this circle, all between any of the servers on the circle. Here, requests 1 and 2 would be resolved by S1 (first to the right), requests 5 and 6 would be resolved by S2, and requests 4, 3, and 7 by S3.
Adding a server node
Let’s say I added a node in between S3 and S4.
We through our blackbox (which will be hash functions which we’ll get to) concluded that S5’s position on the circle is in between S3 and S4. Since we earlier mentioned that in the ring, all the requests on arbitrary positions would be resolved by the server closest to the right of it, S5 now resolves request 3 and 4 instead of S4. This is a lot better than our brute force solution, since we were initially talking of moving almost the entire data/set of connections if we added a new node.
Removing a server node
But what if removed a node? Our initial problem still exists, where if we removed a node, we would end up sending all our connections to the next right one, consequently increasing all our load onto that server node.
What we see above is a connection churn issue once again, where if we removed S4 (which could happen for N reasons, like the node going down, or whatever else), all our requests that were meant to go to server 4, now go to server 1, beyond the requests that are already going to server 1.
This is where the concept of virtual servers/virtual nodes comes in.
What do these virtual servers mean?
Let’s say I had 4 Servers: S1, S2, S3, and S4. These virtual servers are in essence just “pointers” to one of the servers S1, S2, S3, and S4. Let’s say I had 2 virtual servers for each server, hence V1S1, V2S1, V1S2, V2S2, and so on, where V represents a virtual server. These virtual servers, just like the original ones are in diagrammatic representation like the one below placed on the circle/ring at different points.
Keeping this structure above, once we run the the black box against a request ID, we eventually get a random spot on the circle, and once again, the request is mapped to the virtual/physical server that is present closest next to the spot. Note that the virtual servers are just a concept, which means that if we had our request id go through the black box, and it landed at the below spot, the request would end up at the physical server 2, and the virtual server is just the conceptual understanding.
Why this works
Imagine Server 2 went down. With our system earlier that did not involve virtual servers, if you observe the right side of the image below ignoring the virtual servers, all connections in the gap between S1 and S2 would end up at S3, creating a very unoptimal situation since all load shifts to the node S3 (note that since we’re currently talking about our previous system, make sure to ignore virtual nodes). But in our new system with virtual nodes, only the connections that exist between the gap of virtual server S4 on the right, and physical server S2 move to virtual server S1, and the connections on the left between the gap of virtual server S1 and virtual server S2 move to S4. This also shows how our load on S2, which is now down got distributed between S4, and S1, through the concept of virtual servers.
Coding it with hash functions
The building block to be able to code out the above system are hash functions, which is what we referred to as our “black box” above so far. Assume our hashing algorithm is SHA256, one of the most popular hashing algorithms. The range of values SHA256 can produce can be anywhere in between 0 and 2**256 - 1. This range is what represents our maximum hash space. This hash space is a representation of our circle, meaning a server node can live anywhere within this hash space. This means that we need to store the positions where these server nodes exist at within the hash space. We also need to store the server node ids itself, so that we know that which node is present at which position in the hash space.
This leads us to a map implementation, where our key is the server node, and our value is where it is stored in the hash space.
But how do we find out where within this hash space is the server going to be present? How do virtual nodes go into the hash space?
Since our maximum hash space can be 2**256 - 1, we could start with something significantly smaller, and for the sake of an example, we’ll say 100. This value means that there are a 100 total slots for our servers and requests to exist within the hash space.
Let’s say we have our first server, S1, with a server ID of iwant2sleep. On sending this through a hash function, let’s say SHA256, we can get any value from 0 to 2**256 - 1, and to give us a value within our hash space, we would mod this value by the total hash space we have available, in our case, 100.
func hashKey(totalHashSpace int, toHashKey string) int {
sum := sha256.Sum256([]byte(toHashKey))
hashInt := new(big.Int)
hashInt.SetBytes(sum[:])
return int(hashInt.Mod(hashInt, big.NewInt(int64(totalHashSpace))).Int64())
}
This function would return a value, that represents where our S1/any other server we add within our hash space of size 100 is present. The integer returned here is what we’ll store in our map where we store our server id with the hashspace.
serverToHashLocationMap := map[string]interface{}{}
serverToHashLocationMap["iwant2sleep"] = 72 //assumimg our hashKey func returns 72, which is the location of our server within our 100 sized hashspace.
But what about virtual servers? Technically, virtual servers have the same server IDs as the physical ones since virtual servers are just a concept, and since hash functions give the same output each time for the same input, how do we place virtual servers in the hash space. This is where multiple hash functions come into play.
Using multiple hash functions
Let’s say we have k = 3 (virtual + real instances) of each server.
S1 (server id : iwant2sleep) + 2 virtual instances
S2 (server id : iwant2sleep2)+ 2 virtual instances
S3 (server id : iwant2sleep3)+ 2 virtual instances
servers := []string{"iwant2sleep", "iwant2sleep2", "iwant2sleep3"}
algorithms := []string{"murmurhash", "md5", "sha256"};
for _, algorithm := range algorithms {
for _, server := range servers {
position := hash(algorithm, server);
serverToHashLocationMap[server] = position
}
}
We loop through each of these, and use 3 different hash functions (number of hash functions = k), and each of these virtual instances, despite having the same server id would now return a different value in the hash space, hence getting random positions within the hash space.
Resolving Requests to Server with Linear Search
Now that we’ve assigned our server nodes to random places, what happens when we get a request?
The request ID is passed through the hashKey(totalHashSpace, requestID) and once we get a response, that’s where our request is located in the hash space. And as discussed earlier, we need to get the closest server to the right of the request id, and hence, iterate over our Map, and find the smallest greater value than our request id, which is then what resolves the request.
requestInHashSpace := hashKey(totalHashSpace, requestID)
for _, value := range serverToHashLocationMap {
if value > requestInHashSpace {
return value
}
}
Essentially, consistent hashing is now reduced to a linear search problem now, lol. To make this more efficient, we can perform the same with binary search, which reduces the time complexity from O(n) to O(log n).