Distributed Systems, System Design, Uncategorized

How to build experience with large scale distributed systems


If you are like many software engineers/ development managers/software architects at a company and the projects that you had been part of doesn’t have much scope for distributed systems engineering. How do you grow this experience so as to interview for other companies and perform well? You have gone through multiple books and blogs, Grokking the system design interview, Martin Klepmann DIA book and his youtube videos, but without real work, theory doesn’t seem to be sufficient.

Let me try answering above question and in the process help you. Yes, it is easier to change job to another company if you have already worked at a place where you are working on large scale distributed systems. But it is not impossible to transition to very large scale distributed systems based product companies (Facebook, Apple, Amazon, Netflix, Google, Microsoft, Atlassian, Twitter, AirBnb) from a more traditional workplace where you are typically working on SAAS based products.

Designing large scale distributed systems is more often than not begins with a thought process.

Step 1 Understanding and deriving the requirement

  • Think of any large scale distributed system application like a messaging service, a cache service, twitter, facebook, Uber, etc
  • Ask yourself a lot of questions about the requirement for any of the above app that you are thinking of designing . No question is stupid. Don’t be afraid of asking even simple questions.
  • Scope the requirement into a very simple MVP requirement that you think can be designed within an hour or so.
  • From the above activity derive the basic functional requirement of the application. What is the basic minimum functionality that you like to build?
  • Derive what are the non functional requirements (think in terms of scale). Try to derive what are the system requirements in terms of latency, availability, reliability, scalability and fault tolerance.

Step 2 Solutioning the above requirement

  • Come up with a very simple end to end solution through APIs and methods.
  • Think about input and output parameters. That would help you think in terms of objects, schema design, etc. Come up with models based on parameters.
  • Based on the above, identify right kind of storage systems, queue systems, etc. Think if it is read heavy or write heavy, batch messaging or streaming based, consistency heavy or availability heavy.
  • Think in terms of fault tolerance, how does your system handle when some nodes go down, etc.
  • Calculate the cost of systems (number of machines, CPU, storage, etc) for your application. Think of ways how you can optimize or reduce storage cost.

Keep doing the above exercise for multiple applications, products. This will help train your mind to come up with system design solutions to scalable applications based on first principles. You can validate your solutions with a simple google search about system design of system X.

Do not spend a lot of time on youtube or any video courses. While you may get some ideas, most of times you will be overwhelmed with a lot of information and a waste of time.

Once you are comfortable with above exercises (Should not take more than 4 weeks), start giving actual interviews at these companies. Many software engineers, managers, architects keep preparing for the interviews till eternity and feel they are never ready. Nobody is 100% ready for next job switch or interview ever. Do not apply through LinkedIn. Your chances of interview as well as getting a job would be much higher if you apply through references. Find folks in your network who can help you get an interview scheduled at these places. In case you don’t have one, drop me your CV at anil.ojblabs@gmail.com.

In case you do not want to give actual interviews (which I highly recommend you do), start giving mock interviews at services like Pramp.com. It will not only help you hone your interview and system design skills, many times you might land up great interviews and jobs through this.

One mistake that many engineers/managers/architects do during the interviews is that they start kind of lying about the systems that they have built. That is a sure shot recipe for disaster. Because as soon as the interviewer will start grilling you on the same, your lies will fall flat and you would be shown the door. If you have built SAAS based b2b applications, say it so and describe the problems that you have solved there. Don’t coat it into a cloud native application. Be very honest and upfront during all your interviews and trust me, it will get you past a lot of loud/cocky liar candidates.

Another mistake that a lot of engineers do for interview preparation is they put too much focus on system design of distributed applications. With the above preparation and giving mock and actual interviews you will automatically start getting better on system design skills. Remember that system design is one of the skills being evaluated. The other skills like problem solving (coding), technical leadership, code review skills, people management and mentoring are equally important. And you would have honed these skills in your current traditional organization and bring out examples of the same during interviews

How useful was this post?

Click on a star to rate it!

Average rating 0 / 5. Vote count: 0

No votes so far! Be the first to rate this post.

As you found this post useful...

Follow us on social media!

We are sorry that this post was not useful for you!

Let us improve this post!

Tell us how we can improve this post?

0 0 votes
Article Rating
Notify of
Inline Feedbacks
View all comments