FDE-Gym Explained: Testing AI for Real-World Delivery
Discover how FDE-Gym evaluates AI agents on 95 complex delivery tasks across 18 sectors to bridge the gap between lab performance and customer ROI.
Applying research like this to a real workflow? See agent workflows by industry or watch the agents work on real cases.
What the Paper Does
Researchers introduced FDE-Gym, a benchmark designed to evaluate how well automated systems handle the "last mile" of delivery. Unlike traditional tests that use well-defined instructions, this framework places agents in simulated customer environments where requirements are often ambiguous or unstated. The benchmark includes 95 end-to-end tasks derived from 20 real-world scenarios across 18 different industry sectors, such as finance and logistics.
The FDE-Gym benchmark consists of 95 delivery tasks spanning 20 scenarios and 18 different industry sectors. (source) Source: S1
How It Works
The FDE-Gym environment simulates the role of a Forward-Deployed Engineer (FDE). It uses scripted stakeholders that the agent must interview to uncover hidden constraints. The system measures performance based on a "Business Uplift Score," which compares the agent's solution to a human-expert reference solution.
To reflect real-world costs, the benchmark applies penalties for:
- Stakeholder Interaction: Excessive messaging or unnecessary questions.
- Trial Runs: Repeatedly testing solutions in the production-like environment.
- Waiting Time: Delays in processing or decision-making.
Performance in the benchmark is graded by comparing system improvements to a reference solution while penalizing excessive stakeholder interaction and trial runs. (source) Source: S1
What the Authors Report
The authors evaluated 13 different model configurations and found a significant performance gap. Even the highest-performing configuration struggled when faced with the reality of unstated customer needs.
The most successful model configuration tested only managed to resolve 14 out of the 95 tasks in a realistic setting. (source) Source: S1
Interestingly, the researchers noted that simply gathering information was not enough to guarantee success. Even when the best-performing model successfully identified every necessary fact from stakeholders, it failed to reach a high business uplift score on more than half of those tasks. (source) Source: S1
Limitations and Open Questions
The study identifies a lack of "engineering intuition" in current architectures as a potential hypothesis for why agents fail even when they have all the facts. There are also questions regarding the hardware and API budget required to run these 95 tasks at scale. While the benchmark uses simulated environments, it remains a hypothesis whether these simulations are a perfect proxy for actual return on investment (ROI) in live production settings.
Furthermore, while the benchmark tracks "incidents" to measure safety, it does not explicitly detail alignment with specific cybersecurity frameworks like NIST CSF 2.0, though such frameworks are often used by organizations to manage risk during system modifications. The NIST Cybersecurity Framework 2.0 provides guidance and resources for organizations to manage and reduce their cybersecurity risks. (source) Source: S2
3 Possible Business Uses
- Vendor Selection: Use FDE-Gym scores to compare how different service providers handle ambiguous project requirements before signing a contract.
- Internal Training: Deploy the FDE-Gym online platform to help human engineers practice discovery skills and minimize interaction penalties.
- Deployment Risk Assessment: Run proposed automated workflows through the 18 sector-specific scenarios to identify potential "incidents" before they occur in your live environment.
What to Check Before a Pilot
Before initiating a pilot using Stellitron Intelligence or similar frameworks, verify the specific cost-per-task by accounting for the waiting time and trial run penalties defined in the benchmark. Ensure your team has defined the "Business Uplift" metrics relevant to your specific sector to allow for normalized scoring. Finally, review the hardware requirements for running the end-to-end delivery tasks to ensure your infrastructure can support the necessary reasoning tokens.
Next Step
Organizations interested in evaluating their delivery capabilities can access the FDE-Gym dataset and code on the project's official website to begin benchmarking their current automated agents against the 20 customer scenarios.
Sources and review
Reviewed by the Stellitron editorial team before publication. Source snapshots and claim checks retained internally. Proposed workflows are not deployed systems.
Recorded source
The archived text is presented as originally stored. A source link does not mean every statement in the generated analysis is supported by it.
Possible applications
No stored applications came with this analysis. Watch how an agent would handle a document like this below, or tell us about your workflow.
No specific applications were recorded for this archived analysis.
Tell us the workflow, the data you can share, and who must approve external actions, and we’ll come back with a reviewed pilot proposal.
Talk to us about itHow an agent would handle it
One synthetic document, read the way this agent would: what it flags, the evidence it quotes, and where a person decides.
An agent we could build · Evidence extraction agent
Research paper feasibility scan
The document arrives. Nothing has been read yet.
- Title: Structured tool use for document-heavy workflows
- Claim to verify: fewer manual review steps (simulation only)
- Data: synthetic documents, no live system
- Code: linked repository (licence to check)
- Fit: an accounts-payable matching pilot
- Open question: grounding on scanned PDFs
Ask for real-data evaluation before a pilot
A person makes the final call