Is AI Hallucinating, or Did the Human Make It Hallucinate?
By Cody Heimerdinger, CPA, Director
Artificial intelligence has developed a reputation for “hallucinating.” Ask an AI system a question, and every once in a while it may confidently provide a case that does not exist, cite a nonexistent source, misstate a fact, or connect two concepts that should never have been connected in the first place. But after reading Chris Gaetano’s recent Accounting Today article, “False info, real problems: Can accountants tame AI hallucinations?”, I found myself coming back to a slightly different question:
Is AI hallucinating, or did the human make it hallucinate?
Gaetano’s article highlights several high-profile examples in which major professional firms released work containing factual errors, fabricated citations, or other AI-generated inaccuracies. More importantly, the article moves beyond simply blaming the technology. Several professionals interviewed for the article point instead to overreliance, failures in review, and weaknesses in quality-control processes.
One observation particularly resonated with me: when an AI-generated error gets through multiple levels of review and ultimately reaches a client or the public, that is no longer simply an AI problem. It is also a human and control-design problem.
For accountants and tax professionals, I think that distinction matters.
AI Can Hallucinate, but So Can Its Instructions
There is no question that generative AI can produce false information on its own. Current AI systems are designed to generate likely responses based upon patterns and context. They are not databases that simply retrieve guaranteed facts.
As Gaetano’s article explains, hallucination risk appears to be an inherent characteristic of generative AI, which means completely eliminating it may not be realistic.
But there is another problem that receives less attention.
What happens when the information supplied to AI is incomplete, inaccurate, misleading, or misunderstood by the person asking the question?
Consider a simple tax example.
Suppose someone asks an AI system: “The partnership allocated $2.5 million of additional taxable income to the taxpayer. What is the federal tax consequence?” The AI might calculate an estimated tax liability, discuss the applicable income tax rate, address estimated-tax implications, and produce an extremely professional-looking answer. There is only one problem. What if there was never $2.5 million of taxable income? What if the actual adjustment involved a partnership capital account, basis item, liability adjustment, or another non-income partnership-related item?
The AI’s analysis might be completely logical based upon the premise it was given. The premise was simply wrong. That is not quite the traditional definition of an AI hallucination. It is more like: Garbage in equals extraordinarily persuasive garbage out. And that may be one of the greater risks for professionals using AI.
The Most Dangerous Answer Is the One That Sounds Right
AI is remarkably good at producing something that sounds authoritative. That is part of its usefulness. It is also part of its risk.
A poorly written answer naturally invites skepticism. A well-written answer with technical terminology, footnotes, Internal Revenue Code citations, and a confident conclusion can have exactly the opposite effect.
As discussed in the Accounting Today article, humans can begin to implicitly trust AI-generated work because the output is so convincing. Once that happens, reviewers may unconsciously shift from verifying the answer to merely confirming that the answer looks reasonable. Those are not the same thing.
Tax professionals already understand this concept outside of AI. A beautifully prepared tax memorandum does not become correct because it is twenty pages long. A taxpayer does not establish a tax position because an accountant said so. And a conclusion does not become authoritative simply because Bloomberg, Google, ChatGPT, or even another CPA, produced it.
The question always comes back to: What is the authority?
AI Is Not Tax Authority
This distinction is particularly important in tax practice. AI can help locate authority. It can summarize authority. It can compare authorities, explain competing interpretations, draft arguments, identify issues, and help a professional think through how a statute or regulation may apply to a particular fact pattern. But AI itself is not authority.
Neither, for that matter, is a tax research article, continuing education presentation, treatise, newsletter, or professional opinion standing alone.
Treasury regulations addressing the substantial-authority standard specifically recognize authorities such as provisions of the Internal Revenue Code, regulations, revenue rulings, revenue procedures, court decisions, treaties, certain congressional materials, and various other governmental pronouncements. The regulations also make an important distinction: conclusions found in treatises, legal periodicals, legal opinions, or opinions of tax professionals are not themselves authority, although the underlying authorities upon which those conclusions rely may support a position. That concept translates remarkably well to AI.
Do not stop at the answer. Follow the answer back to its source.
A Simplified Hierarchy of Tax Authority
There is no perfect universal ranking because the weight of tax authority depends on the particular issue, jurisdiction, relevance of the authority, subsequent developments, and the similarity of the underlying facts.
Treas. Reg. §1.6662-4 itself provides that the weight given to an authority depends upon its relevance, persuasiveness, and the type of document providing the authority.
Still, professionals can think about federal tax research using a simplified framework:
Highest-Level Primary Authority
Internal Revenue Code and other federal statutes
Congress writes the law. Tax analysis generally begins with the statutory language.
⬇
Treasury Regulations
Regulations interpret and implement the Code. Final Treasury Regulations ordinarily carry considerably greater authority than informal IRS guidance.
⬇
Judicial Authority
U.S. Supreme Court, Courts of Appeals, Tax Court, Court of Federal Claims, and District Court decisions interpret the law. The weight of a case depends heavily upon the court involved and whether the decision is controlling in the taxpayer’s jurisdiction. The IRS itself recognizes that Supreme Court decisions take precedence over lower-court decisions.
⬇
Published Administrative Guidance
Revenue Rulings, Revenue Procedures, Notices, Announcements, and other guidance published through the Internal Revenue Bulletin can provide important administrative authority. Revenue rulings and procedures may be relied upon as precedent in appropriate circumstances, although they do not carry the same force and effect as Treasury Regulations.
⬇
Other Governmental Materials
Depending upon the issue, legislative history, Joint Committee explanations, private letter rulings, technical advice memoranda, and other governmental pronouncements may provide varying degrees of support or insight.
⬇
Secondary Sources
Tax services, research platforms, articles, treatises, CPE materials, newsletters, professional commentary, and AI. These sources can be extraordinarily useful. But generally, their job is to lead us to authority, not replace it. The IRS’s own examination guidance similarly divides federal tax research into statutory, administrative, and judicial primary authority. Even IRS.gov Is Not Always “The Law”
This can surprise taxpayers, and sometimes practitioners. An IRS publication, FAQ, form instruction, or webpage may be excellent practical guidance, but it does not necessarily carry the same authoritative weight as the Code, regulations, or published administrative guidance.
The IRS itself distinguishes formal guidance published in the Internal Revenue Bulletin from more informal guidance such as publications, instructions, forms, and webpages. It specifically notes that the law ultimately controls the taxpayer’s liability.
That is another important lesson for AI research. Finding a source is only step one. The next question should be:
What kind of source did we find?
The Human Still Owns the Conclusion
I frequently think of AI as an extremely capable accounting intern. Give the intern a well-defined assignment, accurate facts, relevant documents, and clear instructions, and you may receive excellent work.
Give that same intern incomplete facts and vaguely ask, “What do you think happened here?” and you may receive something much less reliable. And even if the intern produces outstanding work, the CPA reviewing it still has a job to do.
That means asking:
- Were the underlying facts stated correctly?
- Did AI make assumptions that were never provided?
- Are factual statements distinguishable from inference?
- Does the cited authority actually exist?
- Does the authority say what the AI claims it says?
- Is there higher or contrary authority?
- Is the authority still current?
- Does it actually apply to these particular facts?
The professional’s role therefore does not disappear because AI becomes better. Arguably, professional judgment becomes even more valuable.
A Better AI Control: Risk-Based Verification
There is also a practical limitation to telling professionals that every single sentence generated by AI must be independently researched from scratch.
If we duplicate every step AI performs, we eliminate much of the efficiency that made the technology useful in the first place.
Gaetano’s article recognizes this dilemma as well. Firms have to identify where AI output requires critical challenge rather than pretending every use carries identical risk.
For an accounting firm, that might mean thinking about AI use along a spectrum. What is low risk versus what requires more review? Verify the facts, read the authority, identify contrary authority, and apply professional judgment. That is not fundamentally different from the quality-control system accounting firms have always needed. AI simply changes where some of the risks arise.
Maybe “AI Hallucination” Is the Wrong Question
AI hallucinations are real. They are not going away. But when inaccurate AI-generated information ultimately reaches a client, perhaps asking whether the AI hallucinated does not go far enough.
We should also ask:
- Were the facts supplied to it correct?
- Did the person using it understand the issue well enough to frame the question correctly?
- Did someone verify the important conclusions against authoritative sources?
- And did the review process challenge the answer, or simply admire how convincing it sounded?
AI will make mistakes. So will humans. And sometimes humans will provide AI with bad facts and receive a remarkably polished explanation of something that never happened. The solution is therefore not avoiding AI. It is understanding what AI is exceptionally good at, and what responsibility still belongs to the professional sitting behind the keyboard. Because at the end of the day, AI may help write the analysis. It does not sign the tax return. We do.