By Asro 
Three years ago, Jonathan Frankle, then Chief Scientist at MosaicML, sat down with the Invisible Machines podcast to discuss the "data mixology" of large language models (LLMs). He likened the process of training models to a culinary endeavor—the careful calibration of ingredients required to bake a cake, the ladder of mini-cupcakes one must climb before mastering the pâtisserie, and the sobering reality that academic benchmarks like HellaSwag often failed to reflect the actual desires of ChatGPT users.
Today, Frankle returns to the show as Chief AI Scientist at Databricks. His demeanor remains as rigorous as ever, but his diagnosis has evolved. He no longer talks about the nuances of the baking process; he talks about the power grid. His central thesis: we have successfully harnessed the raw, volatile power of nuclear fusion in the form of LLMs, but we have entirely neglected the power lines—the infrastructure of specification, testing, and application—required to deliver that energy to the end user.
The State of the Industry: A Chronology of "Data Mixology"
To understand where we are, one must look at the trajectory of the past three years. In 2021 and 2022, the industry was obsessed with the "mixology" of data—what went into the model and how it was curated. This was the era of discovery, where researchers realized that the quality of the training set was just as vital as the size of the parameter count.
However, as the models grew, the focus shifted from research to deployment. Enterprises began clamoring for "AI-native" workflows, leading to the current state of affairs: a market flooded with massive, highly capable models that companies are struggling to control. We have moved from the "science experiment" phase to the "integration" phase, yet our software engineering practices have remained tethered to the methodologies of the pre-transformer era.
Frankle argues that we are currently in the "Fortran days" of artificial intelligence. We have the raw computational engine, but we lack the high-level languages—the "Java" or "C" of the AI world—that would allow practitioners to define, edit, and verify machine behavior with the precision required for mission-critical enterprise applications.
Fusion Without Infrastructure: The Engineering Void
The primary challenge identified by Frankle is the absence of a "grid" for AI. In traditional software engineering, the relationship between intent and execution is bridged by compilers, debuggers, and unit tests. In the world of generative AI, that bridge is essentially a "black box" prompt.
The Myth of the Benchmark
Frankle is particularly critical of the industry’s reliance on static benchmarks. He views them as a "cop-out" for organizations that have not yet defined their own business requirements. Building a comprehensive evaluation set—a battery of tests that measures whether an AI performs according to a specific business goal—is infinitely more difficult and more valuable than chasing a high score on a leaderboard.
"The hard work," Frankle notes, "is translating intention into something testable before you even argue about which model to use." When organizations rely on public benchmarks rather than proprietary evaluation protocols, they are essentially outsourcing their quality control to third-party testers who do not understand their specific operational needs.
The Necessity of Specification
The current paradigm is one of "prompting as programming." But as Frankle points out, if you are working exclusively through a chat interface, you are not really building software—you are simply performing a high-stakes version of "natural language parameter tuning." Without a formal specification, you are at the mercy of the model’s internal uncertainty. We need tools that allow for modular design, where reasoning and knowledge can be decoupled, and where the output can be verified against a ground truth.
Supporting Data: Context Windows and the "Garbage In, Garbage Out" Reality
A major point of contention in current AI strategy is the obsession with "long context windows"—the ability to stuff vast amounts of documentation into a single prompt for the model to process. While many enterprises view this as the "silver bullet" that eliminates the need for fine-tuning, Frankle remains skeptical.
The Performance Decay
The science, he notes, hasn’t changed as much as the marketing suggests. In many instances, performance actually degrades as the context window expands. The model becomes prone to "distractors," struggling to weigh relevant data against the noise of a million-token dump. The lesson of data mixology remains absolute: if you ingest garbage, you will output garbage. Whether that garbage arrives through pre-training, Retrieval-Augmented Generation (RAG), or prompt stuffing, the model is still limited by the quality of the input.
The Multimodal Complexity
The rush toward multimodal models—which process video, images, and audio—only compounds the issue. Video files are essentially massive, high-density token streams. Without a disciplined approach to curation, organizations are effectively training their agents on unverified, unstable data, leading to what Frankle describes as "trillion-parameter uncertainty."
Official Perspectives: The Professionalization of AI
Frankle’s transition from MosaicML to Databricks has provided him with a unique vantage point: he is no longer just talking to researchers; he is talking to the leadership teams of over 12,000 enterprises. This scale has confirmed his earlier suspicions. The companies that are succeeding are not those chasing the latest "AGI-in-a-box" hype; they are those that treat AI as a rigorous engineering discipline.
Prompts as Parameters
One of the most profound realizations discussed in the interview is that modern prompt engineering is simply a form of weight optimization. Whether an organization uses a visual back-end to construct prompts or writes them in plain English, they are fundamentally modifying the behavior of a system in ways that are often opaque and difficult to reproduce.
This leads to a significant concern: the "Marshall McLuhan loop." We are wrapping old media—traditional document management and internal processes—inside new media (the LLM), but we haven’t fundamentally changed the nature of the information flow. We are building sophisticated user interfaces for prompt construction, but we have yet to build the unit tests and regression suites that would make these systems reliable.
Implications for the Future: The SEO-fication of Knowledge
Perhaps the most startling takeaway from Frankle’s analysis is the changing nature of content creation. Text on the internet is no longer just for human consumption; it is now a training signal for the next generation of AI agents.
The New Cottage Industry
Frankle predicts the rise of an "LLM optimization" industry, a successor to the SEO (Search Engine Optimization) era. Companies will need to treat their internal data like a curated library. Static FAQ pages, canonical truth documents, and the deliberate separation of "stable facts" from "tentative drafts" will become strategic imperatives.
The danger, as highlighted by Robb, is the "dump and pray" approach: organizations that indiscriminately ingest decades of outdated or contradictory internal documents into their AI systems are essentially manufacturing falsehoods at scale. Until we reach a point where knowledge can be cleanly separated from the "reasoner"—where a company can plug its official guidelines into a faithful, predictable model—organizations will remain vulnerable to the hallucinations of their own internal databases.
Conclusion: Measuring Success Beyond the Hype
The "mixologist mindset" that Jonathan Frankle championed in 2023 remains the guiding light for his work in 2024. The metaphor may have evolved from cupcakes to nuclear fusion, but the fundamental methodology has not. To succeed in the era of generative AI, organizations must:
- Measure success with rigor: Forget the leaderboards and focus on internal, application-specific evaluations.
- Start small: Avoid the temptation to ingest the entire company knowledge base at once.
- Build the infrastructure: Prioritize the creation of unit tests, integration tests, and regression testing protocols.
- Value impact over citation: A tool that solves a mundane, high-volume task is infinitely more valuable than a theoretical model that excels at academic benchmarks but fails in production.
Frankle’s career, marked by early research into the ethics of facial recognition and the lottery ticket hypothesis, has always been defined by a commitment to reality over rhetoric. As we navigate the "Fortran days" of AI, his advice remains a necessary counterweight to the industry’s enthusiasm: stop looking for the next explosion and start building the power lines. Until we do, the true potential of the AI revolution will remain stranded in the laboratory, too dangerous to use and too complex to control.
Lina Hope 
Siti Muinah 
Iffa Jayyana 