The widespread adoption of artificial intelligence (AI) in management practices is drawing parallels to the mechanics of the 2008 mortgage crisis, creating a bubble that could have dire consequences. Company documents have become the focal point for experts, investors, and shareholders, who scrutinize strategic reports and employee performance. However, the quality of a document does not guarantee the credibility of the expert information it presents. This raises concerns about what can be termed "toxic" documents.
While there are no formal requirements for documents to be internally consistent, the idea seems self-evident in frameworks like ISO 9001 and ITIL. As generative models are increasingly adopted in business processes, documentation, and decision-making for the sake of efficiency, they introduce hidden risks. Materials generated by AI may seem polished on the surface but can harbor severe factual inaccuracies, logical flaws, and what is often referred to as "hallucinations."
Experts are expected to integrate their knowledge with insights from relevant stakeholders to create context-driven expertise. However, AI lacks the capability to maintain coherence across related documents, leading to potential contradictions within documents representative of the same process. When documents fail to embody true expertise, organizations lose their ability to differentiate between fact and plausible fiction.
This leads us to define critical terms: an expert document should accurately reflect an expert’s understanding of a process, whereas a toxic document can contradict other system documents, lack verifiable links to reality, or falsely simulate coherence without factual grounding.
Programmers have faced the challenge of context, design quality, and AI's limitations for years. They have even developed complex validation pipelines and new tools to manage the quality of AI-generated outputs. Yet, despite these efforts, projects continue to fail at unexpected junctures, not due to mere statistical errors, but due to inherent limitations in the large, interconnected texts produced by AI.
The core understanding among programmers highlights that their work involves reading and writing text, which must be consistent as it describes action sequences. Inconsistent action descriptions become a barrier for computers to process programs effectively. This fundamental insight equips programmers with a degree of immunity against the shortcomings of contemporary AI tools.
The corporate world has relied heavily on Key Performance Indicators (KPIs) for over three decades. These metrics drive employee evaluations based on quantifiable outputs such as lines of code written, tasks completed, or documents produced. While the logic behind KPIs is sound, their use often overlooks comprehensive coherence, as they operate under the assumption that an expert equates to genuine expertise.
On the flip side, AI provides a faster way to generate "artifacts" of metrics, compelling employees to choose between spending significant time on thorough analysis or quickly producing surface-level results for the sake of high productivity bonuses. This separation of expertise from the expert becomes problematic.
Goodhart's Law posits that once a measure becomes a target, it loses its integrity as a measure. Just as the assumption that individuals taking out mortgages would reliably pay them back fueled the growth of unsound mortgage-backed securities before the 2008 crisis, the unchallenged premise that AI-generated documents represent expertise is equally dangerous.
The superficiality of documents that appear competent yet lack true expertise poses the risk of transforming these documents into "points of entropy" within a company's knowledge base. As investors are drawn to high performance metrics, they may mistakenly believe that a company’s assets are backed by real expertise, while in reality, a considerable portion of the documentation may be toxic.
Currently, there is no cheap or scalable method for filtering toxic documents from those that reflect true expertise. The cost of creating plausible-sounding text is diminishing, while the expense of verification remains high. This situation is reminiscent of the false security offered by AAA-rated mortgage-backed securities.
The propagation of toxic documents can significantly undermine an organization’s integrity. They act as sources of noise in informational theory, masking genuine signals and leading to cascading errors. As these flawed documents are used as inputs for further analysis and documentation, the very structure of organizational knowledge begins to deteriorate.
In conclusion, the implications for the market and competitors are significant. Organizations that fail to recognize the dangers of AI-generated documentation risk inadvertently undermining their credibility and value, while competitors who prioritize rigorous validation and contextual expertise may gain a distinct advantage.
Informational material. 18+.
