X implements measures to prevent Grok from generating sexualised AI content

Microblogging platform X has implemented technological measures to prevent Grok from allowing the generation of images of real people in revealing clothing in jurisdictions where it is illegal, after it faced a heavy backlash over obscene deepfakes generated by the AI chatbot.

This restriction applies to all users, including paid subscribers, X said in a post on its official 'Safety' handle.

Further, image creation and the ability to edit images via the Grok account on the X platform are now only available to paid subscribers. This, it said, adds an extra layer of protection by helping to ensure that individuals who attempt to abuse the Grok account to violate the law or platform's policies can be held accountable.

"We now geoblock the ability of all users to generate images of real people in bikinis, underwear, and similar attire via the Grok account and in Grok in X in those jurisdictions where it's illegal," the post said.

X said it has implemented technological measures to prevent the Grok account from allowing the editing of images of real people in revealing clothing, such as bikinis.

"This restriction applies to all users, including paid subscribers," it added.

X asserted that this does not change its existing safety protocol that all AI prompts and generated content posted to X must strictly adhere to its stipulated rules.

"However content is created or whether users are free or paid subscribers, our Safety team are working around the clock to add additional safeguards, take swift and decisive action to remove violating and illegal content, permanently suspend accounts where appropriate, and collaborate with local governments and law enforcement as necessary," X said.

X said it remains committed to providing a safe platform for everyone and continues to have zero tolerance for any forms of child sexual exploitation, non-consensual nudity, and unwanted sexual content.

"We take action to remove high-priority violative content, including Child Sexual Abuse Material (CSAM) and non-consensual nudity, taking appropriate action against accounts that violate our X rules," the post said.

It added: "We also report accounts seeking Child Sexual Exploitation materials to law enforcement authorities as necessary."

X recently admitted its mistake and removed about 3,500 pieces of content and deleted over 600 accounts after the IT Ministry raised concerns over obscene content linked to Elon Musk-backed AI chatbot Grok.

X had assured authorities it would comply with Indian laws.

There has been mounting pressure on Grok from governments worldwide, including India, as regulators intensify scrutiny of the generative AI engine over content moderation, data safety and non-consensual sexually-explicit images that have flooded X over the last few days.

On January 2, the IT Ministry had pulled up X and directed it to immediately remove all vulgar, obscene and unlawful content generated by Grok or face action under the law.

In that directive, the ministry asked the US-based social media firm to submit a detailed action taken report (ATR) spelling out specific technical and organisational measures adopted or proposed in relation to the Grok application; the role and oversight exercised by the Chief Compliance Officer; actions taken against offending content, users and accounts; as well as mechanisms to ensure compliance with the mandatory reporting requirement under Indian laws.

The IT Ministry's missive had noted that Grok AI, developed by X and integrated on the platform, is being misused by users to create fake accounts to host, generate, publish or share obscene images or videos of women in a derogatory or vulgar manner.

"Importantly, this is not limited to creation of fake accounts but also targets women who host or publish their images or videos, through prompts, image manipulation and synthetic outputs," the ministry had said, asserting that such conduct reflects a serious failure of platform-level safeguards and enforcement mechanisms, and amounts to gross misuse of artificial intelligence (AI) technologies in violation of stipulated laws.

The government had made it clear to X that compliance with the IT Act and rules is not optional, and that the statutory exemptions under section 79 of the IT Act (which deals with safe harbour and immunity from liability for online intermediaries) are conditional upon strict observance of due diligence obligations.

The government warned X in clear terms that any failure to observe due diligence obligations shall result in the loss of the exemption from liability (under section 79 of the IT Act), and that the platform will also be liable for consequential action under other laws, including the IT Act and Bharatiya Nyaya Sanhita.

It asked X to enforce user terms of service and AI usage restrictions, including ensuring strong deterrent measures such as suspension, termination and other enforcement actions against violating users and accounts.

X was also asked to remove or disable access "without delay" to all content already generated or disseminated in violation of applicable laws, in strict compliance with the timelines prescribed under the IT Rules, 2021, without, as such, vitiating the evidence.

The initial response by X to the notice, though detailed, was seen as a mere reiteration by the platform that it respects Indian laws and stipulated guidelines, and that India is a big market for the platform. In its response, X also outlined the strict content takedown policies it abides by when it comes to misleading posts, and those related to non-consensual sexualised images.

Although that reply was long and detailed, the IT Ministry was of the opinion that it "missed" key information including takedown details and specific action that has been taken on Grok AI obscene content issue, and to prevent it in future.

The response fell short of expectation, prompting the government to toughen its stance and ask for more details, including specific action taken on obscene content linked to its Grok AI and measures to prevent a repeat in future.

X had then accepted its mistake, and told the government that it will comply with Indian laws and regulations, sources said.

The European Commission ordered X to preserve Grok-related documents under the Digital Services Act as part of an inquiry into whether harmful content and platform practices broke EU rules, signalling possible further enforcement action.

UK ministers described the phenomenon as appalling, adding they would consider using the Online Safety Act to hold platforms to account. Other European capitals raised similar concerns about privacy and child protection.

This episode exposes hard choices for regulators and platforms alike. Technical mitigations such as watermarking, stricter filtering of requests that aim to sexualise or impersonate real people, and limits on image editing are necessary but not sufficient. 

(With inputs from PTI)