India's tech leaders debate the real challenge with AI: It's brilliant at solving problems but terrible at asking them

Everyone sees the AI outputs: faster code, smarter dashboards, and instant insights. But behind those outcomes lies a far more complex layer of decisions, trade-offs, and engineering realities that rarely make it to the spotlight.

At TechSparks 2025, Oracle brought together a room full of technology leaders to examine that layer with unusual candour. This closed-door roundtable, titled ‘How CTOs Can Future-Proof Their Tech Stacks For An AI-Driven Future’, gathered senior voices: Prashant Khare (Oracle), Kiran Reddy Singam (BigHaat), Rishabh Jain (Vahan), Gaurav Kapatia (Newton School), Mayank Satnalika (CloudSEK), Manish Kumar (KredX), Megha Agarwal (Table Space), Sandeep Pandey (AppsForBharat), Nitin Jain (ShareChat), Hemant Mangla (Namma Yatri), Ram Sellaratnam (Ibus Network), Kuldeep Singh (MediBuddy), Anshul Sharma (Scaler), Prakash Revanna (Ace Turtle) and Pankaj Sharma (Log9 Materials), guided by moderator Rishabh Mansur of YourStory.

This session led to a candid examination of where AI adoption is actually struggling, and why the industry might be looking at the problem all wrong.

The productivity wins everyone's already banking

Leaders agreed that the first wave of AI adoption has already delivered visible results. Teams have shrunk in size while output has grown. Coding agents now contribute a large share of engineering work. AI systems have rapidly accelerated workflows that once demanded extensive manual effort.

Yet beneath these productivity wins, participants cautioned against a narrow view of AI. Many organisations, they observed, risk using AI as a microscope, zooming in on local efficiencies while missing broader strategic opportunities.

Several examples reinforced this point. AI has already proven its potential in large-scale applications such as predicting battery life with remarkable accuracy by analyzing millions of kilometres of driving data. In healthcare, advanced models trained on deep datasets can differentiate between healing tissue and cancer recurrence, offering glimpses of breakthroughs that move far beyond operational optimization.

The sentiment was clear. The transformative power of AI emerges only when companies look beyond isolated use cases and examine the larger systems in which their data and decisions operate.

The question nobody's asking

The conversation took a sharp turn when participants confronted a fundamental limitation. Current AI models excel at solving problems but fail spectacularly at asking the right questions. This gap becomes most visible in engineering teams. While AI accelerates problem-solving, it does not inherently build understanding. Leaders debated whether this risks creating a generation of prompt-dependent developers. Others argued that expectations from new graduates are already significantly higher than in the past, and AI simply helps them match the pace of modern engineering.

The consensus settled on a balanced middle ground. AI can amplify capability, but learning still requires review, guidance, and mentorship. Senior engineers must ensure junior developers understand why something works, not just how to generate an output.

When accuracy isn't negotiable

The roundtable seamlessly transitioned into a more challenging discussion, focusing on an important question: What outcomes can we realistically embrace?

Organizations often assume that higher accuracy is always better. But in AI-driven systems, accuracy is directly linked to cost, liability, and risk.

A slight delay in a delivery service may be inconvenient, but a false positive in fraud detection can freeze a customer’s funds and create severe fallout. In healthcare, even small error margins become unacceptable.

This led to a deeper understanding. Accuracy is not universal. It must be defined by the consequences of failure within each business context.

Some companies are already building systems that reflect this reality. For instance, AI copilots in telemedicine now assist doctors by prompting questions and generating notes, but the clinician retains complete control. The system is designed to empower, not replace.

Meanwhile, leaders acknowledged that accountability ultimately sits with humans. When AI makes a mistake, organizations carry the responsibility. This mirrors how human errors function in traditional systems, reinforcing that AI must be implemented with oversight, not blind trust.

The data collection dilemma

The conversation then shifted to another uncomfortable territory: How much data should AI systems actually collect? Participants discussed the growing unease among enterprise clients who fear that their data may be used for model training without transparency. Partnerships between tech companies and telecom networks were cited as examples of collaborations that blur the boundaries between distribution and data access.

Leaders acknowledged that modern AI systems often rely on behavioral data rather than simple demographic attributes. How users navigate applications can be far more revealing than who they are. This level of insight introduces both opportunity and risk.

In several sectors, organizations now treat data as a product in itself, but maintaining flow, governance, and privacy simultaneously is becoming increasingly complex. The group agreed that trust frameworks and clear governance models will be essential for enterprises to adopt AI at scale.

The telescope, not the microscope

One of the most striking insights of the session reframed how organizations should think about AI.

Participants noted that companies tend to use AI internally to analyse their own operations. But the bigger opportunity is external. AI can model competitive threats, identify market shifts, forecast regulatory changes, and reveal blind spots humans overlook.

Business history is full of disruptions that leaders did not anticipate, even though the data was available. The roundtable reflected on such examples as reminders that AI should help organizations scan horizons, not just optimize internal workflows.

There was strong support for a new kind of AI-driven decision support system. A strategic copilot that sits in leadership discussions, combining internal data with external intelligence to challenge assumptions, test hypotheses, and offer objective perspectives. The group emphasized that AI should not replace strategic judgment, but it can elevate it by eliminating blind spots.

The cost of complexity

The final phase of the conversation returned to a theme that appeared repeatedly: complexity must earn its place.

Leaders noted that every additional data source, feature or dimension increases system cost, not just model accuracy. The temptation to build intricate architectures is strong, especially in organizations that want to showcase AI maturity.

Yet several participants shared that they had once built highly sophisticated models that were technically impressive but financially impractical. The lesson was simple. The question is not what AI can achieve, but what outcomes justify the investment.

Some use cases demand high accuracy and richer data. Others work perfectly well with lighter models and leaner pipelines.

The group agreed that matching the level of sophistication to the problem is one of the most important decisions for AI-first organizations.

What happens next

As the roundtable concluded, one thing became clear: the easy wins from AI are largely captured. The hard questions are just beginning.

Organizations must define acceptable error rates based on business criticality rather than chasing perfect accuracy. They need to balance data collection with privacy concerns, navigating an evolving regulatory landscape. And they must resist the temptation to use AI only for incremental operational improvements while ignoring strategic blind spots.

Companies must resist the urge to build complicated RAG systems and complex architectures as their first solution. Break problems into smaller chunks, use existing tools creatively, and only build sophisticated systems when simpler approaches fail. Efficiency might be lower initially, but you'll learn what actually works before committing to expensive, difficult-to-maintain infrastructure.

The conversation revealed an industry in transition. The question is no longer whether to adopt AI, but how to deploy it wisely, where to accept its limitations, and when to trust human judgment over algorithmic outputs. For India's tech leaders, that tension will define the next phase of innovation.