[Techie Tuesday] Meet Prashant Malik, the man who co-built Cassandra, the database that powers Facebook, Instagram, Netflix, and Apple

It was 2006, and Facebook, the new kid on the block, was making waves across universities like Harvard and Yale. The social media platform had started to grab eyeballs, and was looking to get more users on board. 

But it faced a major obstacle. The software database needed to be upgraded in order to accommodate more users. The challenge was attractive enough to get Prashant Malik, a core member of the Microsoft Windows code developers’ team, to join Facebook. 

From working on developing Facebook’s search engine, Prashant soon went on to co-build Cassandra, NoSQL database management systems designed to handle large amounts of data. And it is the core on which Facebook, Instagram, Netflix, and Apple run even today. 

ALSO READ

[Techie Tuesday] From building a group messaging product on SMS to leading Gojek’s Tech Centre ...

The son of a banker and an accountant, Prashant had no engineers in the family. However, he says he discovered his love for coding in Class 9. Beginning with writing code in BASIC, Prashant says he would crack the algorithms in few seconds. He would program simple pages and thus discovered his love for the subject. 

When he got into IIT Delhi, choosing Computer Science was a no brainer. While he did have the option of choosing Electronics and Communications, Prashant knew he wanted to code.

“I wasn’t the cream of the class, but I liked to work in the lab. The subject where I scored the highest was when I had to code and there was no theory. Getting things done and being in the thick of things and building things is what I really liked doing,” he says. 

After he passed out of IIT Delhi in 1995, he worked in India. “There was a company that was into telecommunications network. They were building networks and they had systems that were client server-based architecture. I used to like that kind of stuff and took up the job in Bengaluru. But I realised that almost my entire class had left for the US. So after nearly three years, I went to the US to work for Microsoft,” says Prashant. 

From Microsoft to Facebook

For him, Facebook after Microsoft was a big shift. While at Microsoft, Prashant was already awarded several patents in client architecture and security systems, and had a private cubicle and a view of the mountains at the Redmond, Seattle, and a house in the suburbs. 

Though things were going smoothly, the offer from Facebook was irresistible for its challenge even though he joined the startup with a 30 percent salary cut. It was a one-floor open office above an Indian restaurant. Many in his circle thought the startup was a fad, and that Prashant was not in his right mind to have left Microsoft.  

Speaking of what got him to make the shift, Prashant says, 

“After meeting Mark Zuckerberg and the core team at Facebook, I realised the dream they had was infectious. The first thing that hit me when I met Mark was he is too young. And then there were this bunch of young guys who were dreaming big. They were talking about Facebook being ubiquitous and everyone in the world using it. It was hard to first take them seriously, but when I started logically debating Mark and the other guys, it hit me that these guys are really dreaming big, and have a big picture. I like big dreams, and that is what attracted me to Facebook.” 

The startup didn’t even have close to a 100 million users then. But in 2006, joining a startup was also the next big thing. 

Many of Prashant’s friends from IIT Delhi had joined Google in the early days. And in 2004, when Google went public, he thought he had missed out and wanted to get into the "next hottest startup."

Once he joined Facebook, Prashant hit the ground running. His first task was to build the search system. Here he was left to his devices. There was no form of code review or checks like he was used to at Microsoft. “I had to review others’ codes if at all, and suddenly I was building things from scratch,” says Prashant. 

ALSO READ

[Techie Tuesday] Swapan Rajdev of Haptik, one of the largest conversational AI platforms, has o...

The Big Data problem 

It was then he realised there were a lot of projects that were stuck because of the lack of a database. And Google was the only company which had a database, called BigTable, that could scale immensely. Until then, Google was the only company that was dealing with Big Data, but now Facebook too started getting increasing volumes of data as it was scaling fast. 

It wasn’t the kind of action that Prashant was used to at Microsoft. He says, everything was planned and there were phases of growth there. But Facebook had an addictive energy and adrenaline pump. 

“Within six months of my joining in 2007, we opened Facebook to the world. Anybody with an email ID could now join Facebook. Those were the days when we would end the day with 10 million new users, and begin the next day with 15 to 20 million users. As the days passed, people were coming in hordes, systems kept breaking, and we had to build batches over and over again, every six hours,” says Prashant. 

At that point in time, there was only MySQL, which didn’t scale that fast. Prashant says, Mark kept mentioning there was a need for a system like BigTable, and they started building on a similar architecture. 

Soon, Prashant was joined by Avinash Lakshman, who had worked on the Dynamo Project at Amazon, which hadn’t seen production scale, and was theoretical in nature. But the core concepts were strong.

“When both of us got together, something magical happened. We thought why not bring BigTable on top of Dynamo? This led to the beginning of building Cassandra. We had named it after an oracle in Greek mythology. We thought it would become bigger than Oracle. And I don’t think anywhere else in the world we could build that kind of project as nobody would trust two guys to go and build something this big. But Facebook had that DNA - to give people super challenging projects,” recollects Prashant. 

Learning to code for failure 

His training at Microsoft helped a lot while building Cassandra. “The critical reviews at Microsoft would happen with Dave Cutler. And when someone like Dave works with you, you realise you are nowhere, and you need to learn a lot,” says Prashant. 

During his early days at Microsoft, he would be told to use a problem statement or go to statement. He recollects, 

“I was self-critical and thought that it isn’t a good idea to use these kind of statements. But I learnt I should think of errors every time I am writing a programme. It is even more important to think of all the failure than getting your programme right. That got ingrained in my brain, and it was a strong message. It was important for a program to not fail rather than it being correct. You should think of all kinds of failures, especially in systems programming and core code development. This helped me write the most complex programs. When you are writing distributed programs, there is no real methodology there because it is very complex in nature. And this kind of thinking helps you a lot. The essence of distributed systems is that when you code you keep thinking of what can go wrong.” 

It was this training that aided them in building Cassandra. Prashant would spend days in office, and he and Avinash would spend nights on the streets discussing design ideas. “We would write codes on walls and would draw wireframes. Nothing (else) mattered for eight months to a year. It was a meditation of sorts,” says Prashant. 

Initially, when Prashant and Avinash started building Cassandra, they were building it for Facebook’s inbox search. It was a big success, and everyone was pleased that at a low budget they were able to crack something that could scale. 

But people were still sceptical if it could take on big load, and if it could scale to several gigabytes of data. Prashant says that they were confident it would work.  

“We had thought of every possible failure, worked for every loophole, thought about multiple data centres, machines, and clusters. We thought of things that weren’t common 13 years back, but could still cause a problem. We had thought through every single thing, and that made the design beautiful,” says Prashant. 

ALSO READ

[Techie Tuesday] From Microsoft, Facebook, PayPal, Coinbase to Lambda - Namrata Ganatra's journ...

Open sourcing Cassandra 

Cassandra was open sourced in 2008, making it one of the first such database of its size to do so. Prashant says, he had no inkling that it would become so popular. Developers from all over the world soon started using Cassandra because everyone had started facing the Big Data problem. 

“Today, 70 to 80 percent of the companies in Silicon Valley use Cassandra. Netflix, Apple, Walmart, and Instagram are built on Cassandra,” says Prashant. 

Soon, many VCs started approaching Prashant on opening companies on the basis of Cassandra. But Prashant says he was happy at Facebook. 

After Cassandra, Prashant took the next big project at Facebook - building the entire backend system of the Messenger platform. The entire email and messages chart of Facebook needed a backend. It was one of the largest teams at Facebook. While the team didn’t open source this, it still is one of the largest distributed systems and has scaled to billions of users. 

“It is the system that is used even today, and we had built it when the user base at Facebook was 100 million. Today, there are over 2.2 billion Facebook users, and they use the same system. This was again because of my learning at Microsoft and building Cassandra. It is important to think of the smallest things while designing and architecting a system,” says Prashant. 

He adds that it is important to think if the systems can still function if two nodes go down, and what happens when the nodes come back up. This led them to build something called as Hinted Handoff - if there are three nodes - A, B, and C, and if C goes down, all the data that is going to C, will be captured by A and B. And when C is back, A and B hand it over to C. 

The Limeroad journey 

In 2012, Prashant felt it was time to get into the next phase of his journey. He also wanted to come back to India, and do something of his own. 

“As you start achieving things, it is human nature to strive for more, and some of my friends at Google had started their own companies,” he says. 

Prashant had an idea to build a bot that had all the functionalities of AWS, which would help people build AWS like functionalities in their private data centres. He started executing it, and after Facebook, Prashant started helping and investing in a lot of startups. 

But the idea of going back to India was stronger than starting up in the US. He returned to India in 2013, but was still not sure what to do. 

“I was a deep technology guy, and most of the things in India then were on ecommerce and consumer facing. Fortunately, at Facebook, I was granted a few patents with Mark Zuckerberg on Social Networks, and I wanted to leverage on that. After talking to investors, I got to know of Limeroad - a social ecommerce platform,” says Prashant. 

He joined Limeroad as the CTO and co-founder in 2013. Soon, Prashant built the tech team and the entire backend technology for Limeroad. But it was a huge shift.

“Silicon Valley is the hub of startups, and I was fortunate enough to be at Facebook, the hottest startup then. When I built my team for messaging, I could get people from the likes of Google. But when I came to India, I realised that building a team wasn’t that easy. People wanted to be in Bengaluru and not come to Gurugram. Initially, we had a team in Bengaluru, and I was able to convince everyone to come to Gurugram. I was lucky enough to convince a lot of people. We built a great tech team at Limeroad, " Prashant says.

In early 2013, he built the automated messaging systems for warehousing at Limeroad. It was ML trained to send notifications and messages. There were no external libraries, he and his team had built it internally to have complete control of personalisation.

ALSO READ

[Techie Tuesday] From selling his startup to Facebook to building ‘the mother of Alexa’ - Kumar...

The investor journey

Prashant also started investing in a lot of companies. Today, he advices many startups including Zomato, and has invested in UrbanClap, and a few other startups. Prashant is also the first person to write a cheque for Shadowfax. 

After his Facebook stint, Prashant says he again felt the need to get into deep tech. “I thought why not invest in companies and advise them. I took a break from Limeroad, and this is what I have been doing for a year and half now. I travel to Singapore and Israel where I can find deep tech companies. I have invested in seven Israeli companies and advising a few others,” says Prashant. 

This, he says, helps him get a good perspective of technology that is happening around the world. “It also helps me satisfy my craving for deep tech and scale, and also gives me the freedom to play with different ideas,” adds Prashant. 

Hiring and advice to techies 

Today, deep tech startups are growing in India. Prashant says there is a big difference between deep tech and consumer companies. He adds that consumer companies are about building products that can be associated with the consumer, and for deep tech companies, it is a good idea to have experience in that particular area and then start out. 

“I could have never built something like Cassandra if I didn’t have the training at Microsoft. Experience in the area matters for deep tech. My advice to deep tech startup founders - get experience in the area and then attack the problem,” he says.

Prashant says that while looking for people, more than the qualification, he looks at their passion and the kind of code they have made live. “I have always been a doer, and I really respect people who have built good systems and have architected stuff. Open sourcing is a great test,” says Prashant. 

Advising all new techies, Prashant says: “Everyone who is graduating today wants to build a startup. While that is great and positive for the ecosystem, before jumping in, get to know a startup. Work for a startup, get a feel of the culture and get the DNA, and then grow that DNA. Building startups are hard, and there are several ups and downs, and there are times when you want to give it all up. If you have worked for a startup and have been part of it, you will understand the path and will be better equipped to handle things.”

(Edited by Megha Reddy)