Our system excelled at one task: converting natural-language questions into API calls. Analysts, account managers, and operations leads used it to bypass the manual effort of gathering data from multiple dashboards and BI tools. By typing simple English requests like “Compile a report on sales volume for January through March 2026 for the Northeast region, broken down by city,” the system generated structured JSON API calls effortlessly.
Built initially on Claude Sonnet 3.5, the system smoothly upgraded through versions 3.7 and 4.0. However, when updating to Sonnet 4.5, unexpected behavior arose: the model embedded API call parameters in the description field or started returning clarifying questions. This broke downstream systems expecting a consistent JSON structure with no need for human clarification, exposing a critical gap in our engineering assumptions.
The challenge lies in the unpredictable “blast radius” of AI model changes—unlike traditional software components, you cannot fully anticipate how an LLM upgrade will affect outputs. Our post-mortem showed our prompt lacked explicit constraints, leading to the model interpreting instructions too liberally. The solution involves adopting an “evals-first” architecture, where an extensive suite of automated tests define acceptable model behavior and gate changes rigorously.
This approach demands significant effort to create and maintain but is essential for safely deploying AI systems dependent on evolving language models. The next frontier in AI engineering is developing robust evaluation frameworks that serve as formal system specifications, bridging the gap between passing smoke tests and guaranteeing predictable, safe production behavior.
Authors: Vijay Sagar Gullapalli, Founding AI Engineer at Adopt AI, and Sarat Mahavratayajula, Senior Software Engineer at Sherwin-Williams.
