AI In The Courtroom: Fabricated Precedents, Synthetic Authority And The Supreme Court’S Draft AI Regulations
This article examines AI-generated fabricated legal precedents appearing before Indian courtsand tribunals, and the Supreme Court's response through its draft Regulations for Use ofArtificial Intelligence in Courts, 2026. Drawing on recent cases Gummadi Usha Rani, GreenopolisWelfare Association, KMG Wires, Buckeye Trust, JSK Industries, and Deepak Raheja, it showshow nonexistent or misquoted citations have entered pleadings and orders, implicating bothadvocates and authorities. Three principles emerge: mandatory verification, continued humanresponsibility, and AI as an aid rather than legal authority. The draft Regulations formalise thesethrough disclosure obligations, while comparable developments in the US, China, and the EUunderscore that AI cannot replace professional judgment.
Introduction
Artificial Intelligence has quickly moved from being a topic of academic discussion to becoming a practical tool in legal work. Law firms in India and abroad are now using AI for legal research, document review, drafting support, case summaries and knowledge management. Courts and tribunals are also becoming increasingly conscious of its use by lawyers, litigants and public authorities. This shift is understandable. Legal work often involves strict timelines, large volumes of material and the need for quick analysis, making technology a natural aid.
At the same time, the use of AI in legal practice has raised a serious concern. Generative AI can produce content that appears polished, structured and convincing, even when it is completely false. In law, this risk becomes particularly dangerous when AI generates case laws, citations or extracts from judgments that do not actually exist. A fabricated precedent is not a minor drafting mistake. It can affect the very basis of legal reasoning, because every submission before a court depends on the authenticity of the authority relied upon.
This problem arises from the way generative AI functions. Unlike a legal database, which retrieves existing judgments and materials, an AI system generates responses based on patterns in data. It may therefore produce an answer that looks complete and reliable, even where there is no real source behind it. In legal practice, this can result in fictional case names, invented citations, wrong paragraph references and fabricated extracts which closely resemble genuine judgments. Unless these outputs are independently verified, they may be mistaken for valid legal authority.
Indian courts have already started dealing with this issue. In recent matters, courts have found that pleadings, notices and assessments contained citations which were fabricated, wrongly attributed or incapable of verification. The judicial approach has been clear: AI may assist research, but it cannot replace professional responsibility. The duty to verify the correctness of legal material continues to rest on the person relying on it.
This concern has now become more significant in light of the Supreme Court of India’s draft Regulations for Use of Artificial Intelligence in Courts, 2026. The draft framework does not reject the use of AI. Instead, it seeks to regulate its use by placing human judgment, transparency, accountability, data protection and judicial independence at the centre of AI adoption. It also recognises the risk of hallucination, where AI produces plausible but false or unsupported material, including incorrect legal precedents or legal principles.
AI Generated False Precedents before Indian Courts
One of the clearest illustrations of this issue is found in Gummadi Usha Rani & Anr v. Sure Mallikarjuna Rao & Anr. In that matter, a trial court relied upon four judgments that were later shown to be entirely nonexistent. The issue was carried to the Andhra Pradesh High Court, which confirmed that the citations were imaginary. The matter eventually reached the Supreme Court, which stayed further proceedings based on the impugned material and treated the use of such fabricated judgments as an “institutional concern.” The importance of this observation lies in its character. The Supreme Court did not regard the matter as a routine defect capable of being brushed aside. It recognized that when invented precedents enter the judicial process, the integrity of adjudication itself is placed at risk.
A similar concern arose before the Delhi High Court in Greenopolis Welfare Association v. Narendra Singh and Ors. There, the respondents pointed out that the petition relied on judgments which could not be traced and also misquoted portions of an actual decision by referring to paragraph numbers that did not exist in the reported judgment. Once the defect was exposed, the petition was withdrawn. The Court permitted the withdrawal, while recording the issue and leaving open the possibility of appropriate steps being taken. This matter is significant because it demonstrates the mixed manner in which AI generated inaccuracies may appear. The problem is not always limited to wholly imaginary cases. In some instances, real decisions are blended with false quotations, invented paragraph references or distorted propositions of law, thereby making the defect more difficult to detect.
The Bombay High Court addressed the same problem in the KMG Wires Pvt Ltd vs NFAC[1] matter, but in the context of revenue adjudication. The Court found that an income tax assessment had relied on case laws that did not exist and quashed the assessment. It emphasised that authorities exercising quasi-judicial powers are under a duty to verify the correctness of all material on which they rely, especially in an era where AI tools may generate convincing but false legal content. This decision is particularly important because it extends the discussion beyond the conduct of advocates and squarely places responsibility on administrative authorities as well. The misuse of AI in legal reasoning is thus not confined to courtroom advocacy. It can enter official decision making and directly affect the rights of citizens.
In Buckeye Trust v. PCIT-1 Bangalore, the Income Tax Appellate Tribunal recalled its own earlier order after realizing that it had relied upon authorities cited by a representative which later turned out to be fabricated. That development is noteworthy because it shows that once false precedent enters the judicial record, the consequences cannot always be cured by mere explanation. The order itself may need to be revisited, since adjudication founded upon nonexistent authority cannot be allowed to stand uncorrected.
The Delhi High Court again dealt with the issue in JSK Industries Ltd. v. Union of India & Ors[2]., where a show cause notice under the GST regime was found to contain citations that were either incorrect or incapable of verification. The Court cautioned tax and GST authorities to exercise utmost care while using AI generated material and made it clear that full responsibility for the contents of such notices would continue to lie with the issuing authority. The significance of this case lies in the Court’s clear distinction between AI as an aid and AI as an authority. The Court recognised that such tools may have limited utility in preliminary research, but they cannot be treated as a substitute for legal verification.
The issue has also reached the Supreme Court in Deepak Raheja & Anr. vs. Omkara Assets Reconstruction Pvt. Ltd[3]., where allegations were made that a rejoinder contained more than one hundred nonexistent or AI fabricated citations. The scale of the allegation itself shows how serious the matter has become. What was once seen as an occasional problem now appears capable of affecting an entire body of pleadings.
Apart from false citations in written filings, courts have also expressed concern over real time dependence on AI during oral hearings. The Punjab and Haryana High Court warned lawyers against the repeated use of mobile phones during proceedings to obtain answers through online search tools or AI based systems, stressing that courtroom assistance cannot replace proper legal preparation. This observation is important because it connects the present issue with a wider professional concern. The use of AI is not objectionable merely because it produces errors. It becomes objectionable when it encourages a casual attitude toward preparation, verification and responsibility
Taken together, these cases reveal a clear pattern. Indian courts are not rejecting the use of technology in legal work. What they are rejecting is the unverified use of machine generated legal material in proceedings that depend for their legitimacy on authentic authority.
What Indian Courts Are Effectively Emphasising
A review of these decisions shows that Indian courts are taking a clear and consistent position on the use of AI in legal proceedings. The first principle is verification. A citation cannot be accepted only because it appears to be properly drafted or because it has been generated by an AI tool. If a party, lawyer or authority relies on a judgment, the correctness and existence of that judgment must be independently checked.
The second principle is human responsibility. Courts have not treated AI as the real actor. Their concern has been with the person or authority that relied on the AI-generated material without proper verification. A machine does not sign pleadings, issue notices, make submissions or pass orders. These remain human acts, and responsibility for them must remain with the person using the material.
The third principle is the distinction between assistance and reliance. AI may assist in research, summarisation or identifying possible authorities, but it cannot become a source of legal truth. Its output can only be a starting point. It cannot replace professional judgment or the duty to verify the law before placing it before a court.
The harm caused by fabricated precedents is also not limited to one case or one party. Courts function on the assumption that the authorities cited before them are genuine. When imaginary judgments enter the record, the damage is institutional. It affects the trust on which the judicial process depends.
This judicial approach now finds support in the Supreme Court’s draft Regulations for Use of Artificial Intelligence in Courts, 2026, which place human oversight, accountability, transparency and verification at the centre of AI use in court processes. The draft framework therefore gives formal shape to what Indian courts have already been emphasising: AI may assist, but responsibility must remain human.
Synthetic Legal Authority and the Draft AI Regulations
The Supreme Court’s draft Regulations for Use of Artificial Intelligence in Courts, 2026 give direct relevance to the problem of fabricated precedents. The draft recognizes that AI systems may produce “hallucinations”, meaning outputs that appear plausible or coherent but are factually incorrect, fabricated, misleading or unsupported by verifiable material. Importantly, this includes the fabrication or misstatement of case precedents, statutory provisions, rules or legal principles
This recognition is important because fabricated precedents are not always obvious errors. An AI-generated citation or extract may look authentic at first glance. It may contain a case name, court name, citation, paragraph number and legal proposition, all in a format that resembles a real judgment. Unless independently checked, such material can enter pleadings, notices, submissions or orders without immediate detection.
The draft regulations also refer to synthetic data or synthetic information, including text-based material generated or modified by AI systems. In the context of legal proceedings, a fabricated precedent may therefore be understood as synthetic legal authority. It carries the appearance of law, but has no actual legal foundation. This is what makes the issue more serious than an ordinary research mistake.
The draft seeks to address this risk through disclosure and verification. Where AI is used in preparing or submitting documents, pleadings or evidence, the AI-assisted nature of the material is required to be disclosed. The Court may also ask what AI system was used, the extent of AI assistance taken and the steps adopted to verify the accuracy of the AI-generated content. If such material is later found to be fabricated, false, misleading or inaccurate because of its AI-generated character, responsibility remains with the person submitting it. The fact that the error came from AI cannot be used as a defense.
This concern also connects with the wider framework under the Information Technology Act, 2000 and the Information Technology (Intermediary Guidelines and Digital Media Ethics Code) Rules, 2021 specifically in the amendment. Recent regulatory discussions under this framework have focused on AI-generated and synthetically generated information, particularly content that appears authentic but may be artificially created, modified or misleading. Although this framework is mainly directed at digital platforms and intermediaries, the underlying concern is relevant to legal proceedings as well, since courts depend on the authenticity of the material placed before them.
The Digital Personal Data Protection Act, 2023 is relevant from a different angle. It does not directly deal with fabricated precedents or false legal citations. However, it reinforces the broader need for responsibility, safeguards and careful handling of digital information. This becomes important where lawyers, courts or public authorities use AI tools in connection with pleadings, case records, client documents, personal data or court-related material.
Taken together, the Supreme Court’s draft AI Regulations, the IT Act framework and the DPDP Act point towards the same larger principle: technology may be used, but it must be used with accountability. For court proceedings, however, the Supreme Court’s draft Regulations are now the most direct reference point. They bring the concern of fabricated precedents within a formal framework of human responsibility, disclosure, verification and institutional oversight.
Comparative Developments and the Wider Institutional Concern
The concerns now being expressed in India are not unique. The Deloitte Australia incident, where AI generated material attracted criticism after serious inaccuracies came to light, illustrates that even highly organized professional environments are not immune from the risks of unchecked AI output. The significance of this example lies in its institutional context, demonstrating that the issue cannot be attributed solely to individual carelessness but is also linked to the persuasive fluency of the technology itself.
Developments in other jurisdictions reflect a similar pattern. In the United States, lawyers have faced consequences for submitting court documents containing fabricated authorities generated through AI tools. The principle emerging from such cases is consistent with the position now taking shape in India, namely that professional obligations remain unchanged regardless of the use of technological assistance. In a different regulatory setting, China’s approach toward controlling unverified or misleading content further indicates a broader concern with artificial authenticity in the public domain.
China has adopted a more structured and preventive regulatory approach. Its “Deep Synthesis Provisions,” which came into force in January 2023, specifically regulate AI generated content including text, images, audio and video. These rules require service providers to clearly label synthetic content, verify user identities, monitor generated outputs and prevent the dissemination of false or harmful information. In addition, China has introduced further measures on AI content labelling and generative AI services, with a strong emphasis on traceability, accountability and platform responsibility. Although these regulations are primarily aimed at online content, they reflect a recognition of the risks posed by AI generated material that appears authentic, particularly its potential to mislead users and influence perception.[4]
In the European Union, the recently adopted AI Act represents a comprehensive attempt to regulate artificial intelligence through a risk based framework. One of its key features is the requirement of transparency for AI generated content, including obligations to disclose or label synthetic or manipulated material such as deepfakes. This is complemented by other regulatory instruments such as the Digital Services Act, which together aim to ensure accountability and trust in digital systems. The European approach focuses less on prohibition and more on ensuring that users are aware when they are interacting with AI generated or altered content.[5]
These developments, when viewed together, indicate that the misuse of AI in legal and institutional settings forms part of a wider global challenge. While the methods differ, a common concern runs across jurisdictions, namely the ability of AI to generate content that appears structured, credible and authoritative despite lacking a real foundation. The question, therefore, is no longer whether AI will be used in professional practice, but how legal systems will respond to the increasing presence of such content while preserving standards of truth, authenticity and accountability.
Conclusion
Recent Indian decisions and the Supreme Court’s draft AI Regulations point in the same direction: AI may assist legal work, but it cannot dilute the duty of verification. Courts have consistently emphasized that responsibility for the correctness of legal material remains with the person relying on it, even where the material has been generated or assisted by technology.
The concern becomes more serious in the case of AI-generated false precedents. Such material may not appear incorrect on its face. It may carry the structure of a genuine judgment, with a case name, citation, paragraph reference and legal proposition. Unless carefully verified, it can enter pleadings, notices, submissions or orders and affect the reliability of the adjudicatory process.
The Supreme Court’s draft Regulations for Use of Artificial Intelligence in Courts, 2026 now give this concern a formal institutional context. By recognizing hallucination, synthetic information, disclosure obligations, verification requirements and human oversight, the draft framework makes it clear that AI can be used only as an aid and not as a substitute for human responsibility. The draft also makes it clear that where AI-generated material is found to be false, fabricated or misleading, the responsibility remains with the person submitting or relying on it.
The broader regulatory developments under the Information Technology Act, 2000, the Information Technology (Intermediary Guidelines and Digital Media Ethics Code) Rules, 2021 and the Digital Personal Data Protection Act, 2023 also point in the same direction. They reflect a growing legal concern with digital systems, synthetic content, safeguards, accountability and responsible handling of information.
The position that emerges is therefore clear. AI may assist legal research, drafting, summarization and court administration, but it cannot replace professional judgment or judicial responsibility. The final obligation remains human. This recognition is significant for legal practice to verify legal material, disclose AI assistance where required, ensure accuracy and protect the integrity of legal proceedings.
