Agent in a Bottle Explained: Automating Cheap AI Workloads

What the Paper Does

Large Language Models (LLMs) are highly capable but expensive when applied to millions of repetitive tasks. The paper introduces the concept of "bottling": the ability of an LLM agent to autonomously transform its general reasoning into a task-specific, low-cost "artifact," such as a small distilled model or a Python script. To evaluate this, the authors created BOTTLED, a benchmark that tests whether agents can take an unlabeled workload and a fixed budget to produce a solution that balances quality and cost. The BOTTLED benchmark requires agents to complete a workload using a fixed budget for time, computation, and API usage. (source) (Source: S1)

How It Works (In Plain Terms)

Instead of a human engineer manually writing code or training a student model, the "Agent in a Bottle" approach treats the LLM as an autonomous developer. The process follows these steps:

1. Analysis: The agent receives a large, unlabeled dataset (the workload).
2. Strategy Selection: The agent decides whether to write a programmatic rule-based script or to use its own outputs to train a smaller, cheaper model (distillation).
3. Execution: Under a strict "bottling budget" for time and tokens, the agent generates the artifact.
4. Deployment: The resulting artifact—not the expensive frontier model—is used to process the millions of remaining data points.

This shifts the LLM's role from a "worker" doing every task to a "manager" building a specialized tool for the task.

What the Authors Report

The researchers tested ten different models across three distinct tasks. Their findings suggest that being a "smart" model does not automatically make a model a good "bottler." A model's high accuracy on individual tasks does not guarantee it will be effective at creating scalable artifacts. (source) (Source: S1)

Key findings include:

• Performance Gaps: In 48 out of 60 experimental runs, the bottled artifacts performed worse than the lower bound of the original model's 95% confidence interval for zero-shot performance. The vast majority of bottling attempts resulted in performance lower than the expected range of the model's standard performance. (source) (Source: S1)
• Baseline Comparison: More than half of the agent-created artifacts (31 of 60) failed to beat simple, standard distillation baselines using the same token budget. More than half of the experimental bottling runs were less effective than standard small-model distillation methods. (source) (Source: S1)
• Success Stories: Despite these failures, high-tier models like Opus 5 showed significant promise. In query-product relevance tasks, Opus 5 retained 82% of its accuracy while reducing costs by an estimated 657 times. The Opus 5 model achieved a cost reduction of over 600 times while maintaining the majority of its performance on specific classification tasks. (source) (Source: S1)
• Efficiency vs. Specialized Models: The agent-bottled solutions were competitive with "system one" models like Jev, which are specifically designed for cheap, high-speed inference. Autonomous bottling can produce results that are competitive with specialized models designed for high-efficiency inference. (source) (Source: S1)

Limitations and Open Questions

• Reliability: The authors note that strong zero-shot performance (answering a single question well) is not a reliable predictor of bottling success. Models that achieve similar scores on initial tasks can show widely different results when attempting to bottle those capabilities. (source) (Source: S1)
• Budget Constraints: The study used fixed compute and API budgets; it is unclear if significantly higher budgets would allow weaker models to bottle effectively.
• Task Variety: While the benchmark covers classification and extraction, the "bottling" success rate for highly creative or subjective tasks remains an open question.

3 Possible Business Uses

1. High-Volume Data Categorization: An enterprise could use a frontier model to "bottle" a classifier for millions of product SKUs, moving from expensive API calls to a self-hosted small model at a fraction of the cost.
2. Legacy System Migration: Agents could be tasked with writing Python scripts to extract structured data from millions of old documents, replacing manual regex writing with autonomous "programmatic bottling."
3. Dynamic Routing: Using the "routing" workflow, a bottled artifact could handle 90% of common customer support queries, only escalating the most complex cases to the expensive frontier model. Routing workflows can improve efficiency by sending simple tasks to smaller models while reserving capable models for difficult questions. (source) (Source: S2)

What to Check Before a Pilot

• Workload Volume: Bottling requires an upfront investment in "bottling tokens." Ensure your workload is large enough (e.g., millions of instances) to justify the initial cost of creating the artifact.
• Quality Thresholds: Determine the minimum acceptable Macro-F1 or accuracy score. The research shows that artifacts often lose 10-20% of the teacher model's performance. In testing, the Opus 5 model was able to reach nearly 94% of the performance of the specialized Jev model at a significantly lower projected cost. (source) (Source: S1)
• Infrastructure Compatibility: Verify if your production environment can host the resulting artifact (e.g., a PyTorch small model or a standalone Python script).

Next Step / Demo

Review the BOTTLED benchmark results for the model closest to your current stack. If using high-tier models like Opus 5, consider a small-scale pilot on a repetitive data extraction task to measure the actual amortized cost savings versus quality loss.

*AI-assisted article, reviewed before publication.*

Sources and review

• 2610.08775 Agent in a Bottle: Can LLM Agents Turn Their Capabilities Into Cheap, Scalable Artifacts?
• Building Effective AI Agents \ Anthropic

AI-assisted research and writing, reviewed by the Stellitron editorial team before publication. Source snapshots and claim checks retained internally. Proposed workflows are not deployed systems.

Try a workflow demo · Discuss a pilot