Distributed Systems

Chubby Lock Service


What is Chubby Lock Service

  • Chubby lock service provides coarse-grained locking as well as reliable (though low-volume) storage for a loosely coupled distributed system
  • The design emphasis is on availability and reliability, as opposed to high performance

What is the purpose of Chubby Lock Service

  • It is intended for use within a loosely-coupled distributed system consisting of moderately large numbers of small machines connected by a high-speed network.
  • The purpose of the lock service is to allow its clients to synchronize their activities and to agree on basic information about their environment.
  • Chubby helps developers deal with coarse-grained synchronization within their systems, and in particular to deal with the problem of electing a leader from among a set of otherwise equivalent servers.
  • The Google File System uses a Chubby lock to appoint a GFS master server, and Bigtable uses Chubby in several ways: to elect a master, to allow the master to discover the servers it controls, and to permit clients to find the master. In addition, both GFS and Bigtable use Chubby as a well-known and available location to store a small amount of meta-data.

Chubby Lock Service System Architecture

  • Chubby has two main components that communicate via RPC: a server, and a library that client applications link against. All communication between Chubby clients and the servers is mediated by the client library.
  • A Chubby cell consists of a small set of servers (typically five) known as replicas, placed so as to reduce the likelihood of correlated failure (for example, in different racks). The replicas use a distributed consensus protocol to elect a master; the master must obtain votes from a majority of the replicas, plus promises that those replicas will not elect a different master for an interval of a few seconds known as the master lease. The master lease is periodically renewed by the replicas provided the master continues to win a majority of the vote.
  • The replicas maintain copies of a simple database, but only the master initiates reads and writes of this database. All other replicas simply copy updates from the master, sent using the consensus protocol.
  • Clients find the master by sending master location requests to the replicas listed in the DNS. Non-master replicas respond to such requests by returning the identity of the master.
  • Write requests are propagated via the consensus protocol to all replicas; such requests are acknowledged when the write has reached a majority of the replicas in the cell.
  • Read requests are satisfied by the master alone; this is safe provided the master lease has not expired, as no other master can possibly exist. If a master fails, the other replicas run the election protocol when their master leases expire; a new master will typically be elected in a few seconds.
  • If a replica fails and does not recover for a few hours, a simple replacement system selects a fresh machine from a free pool and starts the lock server binary on it. It then updates the DNS tables, replacing the IP address of the failed replica with that of the new one.
  • The current master polls the DNS periodically and eventually notices the change. It then updates the list of the cell’s members in the cell’s database; this list is kept consistent across all the members via the normal replication protocol. In the meantime, the new replica obtains a recent copy of the database from a combination of backups stored on file servers and updates from active replicas. Once the new replica has processed a request that the current master is waiting to commit, the replica is permitted to vote in the elections for new master.

Lessons Learnt while Designing Chubby

  • Developers rarely consider availability We find that our developers rarely think about failure probabilities, and are inclined to treat a service like Chubby as though it were always available.
  • Developers also fail to appreciate the difference between a service being up, and that service being available to their applications.
  • Chubby provides an event that allows clients to detect when a master fail-over has taken place. Unfortunately, many developers chose to crash their applications on receiving this event, thus decreasing the availability of their systems substantially

Chubby Lock Service Summary

  • Chubby is a distributed lock service intended for coarse grained synchronization of activities within Google’s distributed systems. It has found wider use as a name service and repository for configuration information.
  • Chubby is based on distributed consensus among a few replicas for fault tolerance, consistent client-side caching to reduce server load while retaining simple semantics, timely notification of updates, and a familiar file system interface.
  • Chubby has become Google’s primary internal name service; it is a common rendezvous mechanism for systems such as MapReduce, the storage systems GFS and Bigtable use Chubby to elect a primary from redundant replicas; and it is a standard repository for files that require high availability, such as access control lists.

How useful was this post?

Click on a star to rate it!

Average rating 0 / 5. Vote count: 0

No votes so far! Be the first to rate this post.

As you found this post useful...

Follow us on social media!

We are sorry that this post was not useful for you!

Let us improve this post!

Tell us how we can improve this post?

0 0 votes
Article Rating
Notify of
Inline Feedbacks
View all comments