Can AI save us from AI? This startup is building digital tools to detect deepfakes

Generative AI may be revolutionising countless fields, from scientific research to everyday tasks, but it has also powered bad actors with tools for deception and fraud. Digvijay Singh and Amitabh Kumar witnessed this challenge firsthand. 

Singh, an ML engineer and product manager who worked at HyperVerge, a company specialising in identity verification and fraud prevention, saw how bad actors exploited AI across loan apps, banking platforms, and mobile games. 

“With easy availability of AI tools, platforms that relied on images or videos to build trust, whether for product, age or identity verification, are now left vulnerable,” Singh explains. 

Kumar, founder of multiple online safety ventures and a trust and safety consultant to platforms like Meta, noticed the same vulnerabilities. He points out that, unlike established cybersecurity solutions such as antivirus software, the industry lacked comparable tools to combat AI-generated threats.

This realisation led them to found Contrails AI in 2023 in Bengaluru, building what they describe as the “safety belt” for the generative AI revolution. 

The name "Contrails" references aeroplane trails, visually striking but harmful to the climate. If planes flew at different altitudes, they'd avoid creating these trails and reduce pollution. Similarly, harmful content on social media generates engagement but damages the digital ecosystem. 

“Contrails AI aims to identify and manage these 'digital contrails', protecting online communities,” Singh tells YourStory. 

How it works 

Contrails AI is building a platform to detect deepfakes, synthetic media manipulation, and harmful content before it can cause damage. 

The platform focuses on two core areas: synthetic media manipulation and harmful content. It catches deepfakes, face swaps, lip-sync manipulation, audio spoofing, and subtler alterations, such as a broken table leg digitally restored on an online marketplace. The system also flags NSFW content, child abuse imagery, terrorism material, and xenophobic content.

Contrails AI works with a forensic approach, analysing minute pixel-level irregularities rather than whole images. Using deep learning and transformer models, it detects fake media, policy violations, and security risks in real time across video, image, audio, and text. The founders say Contrails AI’s algorithms work across modalities and can handle the complexity and scale of different global contexts. 

The system connects to client platforms through an API and flags suspicious content for human review. Clients can set up workflows to choose what data to scan, how detailed reports should be, and response formats for specific requirements. 

If the required data isn't available through the client's infrastructure, the system requests it from users, then moves to the next verification step. All flagged cases can be reviewed through a centralised dashboard.

Clients and ideation 

The trust and safety startup primarily targets B2B clients, platforms with user-generated content vulnerable to exploitation. This includes marketplaces like eBay and OLX, gaming platforms requiring age assurance, dating apps where images build trust, food delivery services where sellers manipulate product photos, and financial services needing KYC verification.

The B2B model operates on volume-based pricing, with monthly or annual contracts based on content processing volumes. The company plans to shift toward ROI-based pricing, charging clients based on fraud prevented or costs saved. 

“Trust and safety is no longer a back-office concern; it has become a boardroom priority,” Kumar says. The founders are counting on market adoption to accelerate once regulations are strengthened. “Deepfakes have evolved through six or seven generations since we started in 2023,” he adds. 

For B2C, Contrails AI targets high-net-worth (HNI) individuals, business owners, and celebrities whose personal image carries significant brand value. The startup is piloting a 24/7 web monitoring service to detect deepfakes and unauthorised likeness use across social media, ads, and marketplaces. 

With regulations like the United States' TAKE IT DOWN Act (which allows victims to request removal of non-consensual intimate imagery) recently signed into law, and Delhi High Court orders protecting celebrities, these individuals have legal backing. “This makes them natural customers,” Kumar says.

The startup is also expanding into AI-generated music detection for platforms like Spotify, with both services operating on volume-based contracts.

To power these solutions, Contrails AI has developed its model by identifying potential failure points. The team tested various parameters in digital media—face colour, eye colour, and lighting—to locate weaknesses.

When the AI struggled with certain inputs, the team improved training data to address those gaps. Pilot programmes launched in early 2024, and client data revealed where threats actually originate: specific Twitter handles, Telegram channels, and other platforms.

The platform handles sensitive personally identifiable information (PII), including facial data, with GDPR and SOC 2 compliance, secure storage, and strict retention limits. The system is modular, building tailored models for different industries and their specific challenges.

Funding and expansion 

Contrails AI has raised $1 million in pre-seed funding from Huddle Ventures and the Indian Angel Network (IAN) Group, with additional backing from Ajai Chowdhry (HCL), Gaurav Mangla (Pickrr), and Sreeraman Girija (Fynd). The startup plans to work with clients that need domain-specific solutions and, after setting up its first few focus areas, expand to new regions based on client demand. 

The new funding will help scale the company’s technology for wider real-world use. It is currently running pilot programmes with marketplaces, content platforms, and companies in the BFSI sector. The investment will also speed up these pilots across the US and EU in sectors like media, finance, and digital platforms. 

“We’ve already expanded from India to the US, and if a client needs EU coverage, we’ll add support for European languages and compliance standards,” Singh says.

The startup is addressing the growing threat of “AI slop”, which is an explosion of fake and malicious content online that could push the web toward a “dead internet”, where most content is counterfeit or harmful.

Singh adds that bad actors operate at the cutting edge, ignoring data privacy, jurisdiction, and regulations, giving them “cheat codes”. "Technologies fighting these threats must evolve just as rapidly," he emphasises.