Vijayender Karnaty: Mining Meaningful Insights from Massive Datasets

Vijayender Karnaty was drawn to Artificial Intelligence in stages. During his time at the Indian Institute of Technology - Kanpur (IIT), two topics caught his attention: information theory and distributed systems, both of which are intrinsically linked to AI.

Information theory, which Vijayender calls a beautiful measure of uncertainty, is a branch of mathematics that deals with the quantification, storage, and communication of information. Distributed systems, a collection of independent computers or nodes connected by a network, caught Vijayender’s eye because of how these machines work together as one to achieve a goal.

The pivotal turn of a page

Another pivotal moment Vijayender recalls is reading Jeffrey Ullman’s 2011 book titled ‘Mining of Massive Datasets’. “The book fascinated me. There are certain problems that become solvable just because of the sheer amount of data we had. That’s what started my journey in this industry,” he says.

Vijayender started his career at Cisco, but made an industry switch, joining the search team at Target with the primary intent of working on massive datasets. “Eventually, I started asking deeper questions, like how do we gain meaningful insights from large datasets? I believe that one of the best aspects of this particular industry is taking data at scale, trying to solve problems and create clear and impactful solutions,” he says.

Leveraging AI for meaningful experiences in ecommerce

As a Director, Data Science, at Nykaa, Vijayender’s primary responsibility is to create meaningful and relevant search experiences for customers, enabling them to discover products they are looking for. The main focus for Vijayender’s team is discovery - the process by which customers explore and find products, beyond what they initially thought to buy. As an ecommerce platform, Nykaa is committed to creating a frictionless discovery process for customers.

“One of Nykaa’s biggest advantages is that it is the largest player in the beauty online vertical, and has significant data. It allows us to make data-driven decisions when it comes to three key verticals: search, recommendations, and ads”, says Vijayender.

These three verticals are a big part of discovery, and Nykaa is investing heavily in AI to enhance them. According to Vijayender, Nykaa is consistently searching for AI technologies to ease customer discovery on the site. Significant resources are invested in crafting the right selection of products that would include the customer’s brand of choice, while encouraging them to explore similar products based on analysing their historical data. Nykaa takes these particular learnings and applies them to creating relevant search experiences.

Furthermore, Nykaa leverages AI to understand customer affinities to brands and categories and carefully curates personalized widgets to delight them.

One of the biggest challenges of leveraging AI at Nykaa was setting expectations of model iterations with various stakeholders. Vijayender shared that the team strives to give balanced, realistic and tangible expectations about the various phases of these AI-driven projects. “Things can steer away from the direction we want. Getting it wrong at this particular stage erodes trust from our stakeholders. So we need to really understand the subphases in an ML project, be it exploration or prototyping,” he says. He adds that each stage has dropout rates and “we have to keep delivering to customers and improving”. “So balancing this from a teamwork perspective is one of the key things we do.”

Nykaa will continue to invest in AI as the company believes that it is an important lever to stay relevant and ahead of the competition. Having the advantage of a large pool of data, the company is looking to harness it in meaningful ways.

Vijayender takes personal pride in semantic search, technology that interprets both intent and meaning of a query, as opposed to merely matching keywords. He highlights another AI endeavour: the core ranking of models across different verticals, be it search recommendations or ads that match customer expectations.

The team at Nykaa is also experimenting with different advances in AI, including GenAI. “We are always looking at how we can take upcoming trends, experiment with them, and create working prototypes for the company,” he says.

AWS and Nykaa: Driving digital transformation

Amazon Web Services and Nykaa have built a deep technological partnership to drive digital transformation, customer experience and data strategies. The tech giant has been a primary platform for all Nykaa’s compute needs. Additionally, AWS has also assisted Nykaa when it comes to its data foundation, as Nykaa’s data lake is currently hosted on AWS. Its data processing workflows leverage Amazon’s Elastic MapReduce, which converts, transforms, and hosts the data. Nykaa has also used Sagemaker, a cloud-based, machine-learning platform that allows for the creation, training, and deployment of ML models.

"In our journey, we initially relied on batch inference. As we move towards more dynamic, real-time inference, one of our key constraints is delivering performant predictions. We also need to support a diverse set of compute options—some backed by GPUs and others by CPUs,” Vijayender says.

This is where Amazon SageMaker has been essential. We use it to host our real-time inference models, and it abstracts away all the complexity of the underlying hardware. Ultimately, it brings infrastructure to our data scientists as a service, which they can use to host their models directly."

He adds that Amazon’s Sagemaker was of real use to host real time inference, abstracting out all the complexity of hardware, providing infrastructure as a service to data scientists.

These are just a few areas where AWS services have assisted Nykaa. Vijayender also highlighted the use of Amazon Bedrock, which Nykaa is using to build agentic solutions.

AI Ethics: Transparency, trust, and the collective soul

Vijayender examines ethical AI through two distinct lenses: as an organization and as an individual. On an organizational level, he believes that there is a ‘collective soul’ of people who translate the intention of the company. This collective soul is typically aligned on the ethical guardrails of the technology they use.

He believes that organizations that leverage data and AI for business and growth should prioritize privacy. Security, anonymisation, and privacy measures should be baked into an organisation’s data foundation at the very beginning. This will help mitigate risks. The second guardrail companies should consider is fairness when it comes to models.

“We are asking models to come up with the right solutions and make the right decisions based on data from a majority of our customers. We should also ensure that models should make the same kind of decisions for the marginalised or underrepresented. We will have to incorporate fairness and bias into our modelling strategies. That is an area we all need to be conscious of and invest in whenever we take a model into production,” Vijayender says.

Continuing on this theme, Vijayender emphasised the importance of inclusive, high-quality data in AI development. “Brands must ensure all customer segments are represented, and models should be trained on quality data and applied responsibly,” he said.

On a personal note, Vijayender reflected on how individuals must adapt to the rapid evolution of AI and Large Language Models. “What once seemed like science fiction is now becoming reality. Naturally, there’s apprehension about where it’s all headed,” he acknowledged. To address these concerns, Vijayender stressed the need for transparency. Large organisations developing AI models should clearly disclose the types of data used and the goals behind their training.

“Making this information public to some extent will help everyone understand what’s happening and guide AI development in the right direction,” he explained. Transparency, he believes, will be driven by regulation and governance, ultimately fostering trust. Clear communication, openness, and addressing concerns directly will help people embrace new technologies while staying aware of their potential risks.

In his downtime, Vijayender enjoys cycling through the countryside, seeking out green landscapes. A passionate reader, he continues to explore the works of his favourite author, Isaac Asimov.

The next big thing in AI

Vijayender believes AI has been limited to the digital world. However, he sees exciting possibilities in bot AI and robotics.

“Just like computers have helped the common man realize the actual implementations of his/her thoughts and helped us create something new everyday, I would like to see the same thing happen with respect to robotics,” he says.