Part 6 of 9 — Agent Development Lifecycle (ADLC)
Software you deploy and forget. Agents you deploy and watch. The day you stop watching is the day they start drifting.
That sentence is the one I find myself repeating most often when I'm explaining to engineering leaders why their existing DevOps practice — which they have correctly invested in for years — is not, by itself, sufficient for building agents that work in production. Agents are software. But agents are also probabilistic, context-dependent, model-version-sensitive, and prone to behavioral drift in ways that traditional services are not. Treating them like microservices and walking away is how you wake up in 90 days to discover that an agent has quietly degraded by 15% on the metrics that matter.
For an entertainment and hospitality company, the analogy that lands hardest is live theater. Every show is a performance. The set, the cast, the script, the lighting, the cueing — all of it has to come together every night, and a missed cue is visible to every guest in the house. Agents in production are the same. Every interaction is a performance. There is no curtain to draw if something goes wrong.
This article is about the three capabilities that make up the Agent Development Lifecycle (ADLC) focus area, and the engineering discipline Aurelia built to ship and maintain agents the way they ship and maintain a Broadway production: tested, rehearsed, watched, and ready to recover from anything.
The Three Capabilities of ADLC
Three capabilities, all heavy:
Build & Deploy — the automation, CI/CD pipelines, and engineering practices for building, testing, and deploying agent systems
Validation (Testing) — the testing of agent interactions and outcomes, including scenario testing, adversarial testing, and ongoing eval discipline
Support — the processes to detect, respond to, and resolve issues with production agents, including drift detection, on-call coverage, and incident response
These look like familiar engineering disciplines. They are not. The shape is the same. The substance is different.
What Level 1 Looks Like
Low-maturity ADLC has a recognizable pattern:
"Deploying an agent" means a human-in-the-loop release process with manual prompt updates and ad-hoc rollouts
The eval suite is a folder of test cases someone wrote during the original POC
"Validation" before release means a manager spot-checks 10 example interactions
Drift is detected by guests complaining — or worse, by the front-line cast members hearing the complaints and not having a way to escalate them
When an agent fails in production, the response involves Slack messages, screenshots, and the original developer being paged at midnight just before the East Coast parks open
There is no CI/CD pipeline specifically designed for agent artifacts (prompts, model versions, retrieval indexes, tool definitions)
The team that built the agent is the team that operates the agent, and they have not slept well in two months
If that sounds like a portrait, it is, drawn from real teams I have walked into.
Aurelia's Starting Point
When I first sat down with Aurelia's engineering team, they had 14 agents in production and zero shared eval framework. Each agent had its own test suite, written by its own developer, in its own format. Releases were done manually, with prompt changes pushed via the same process used for application code despite the prompt being a meaningfully different artifact. Drift detection consisted of two product managers who looked at usage dashboards on Tuesday mornings.
The MTTR (mean time to remediate) for agent incidents was 14 days. By any standard for production software in 2026, this was unsustainable. It was particularly unsustainable for a company whose flagship park sees 175,000 guests on a peak day, whose airline operates 1,200 daily flight legs, and whose streaming service serves 88 million households. A 14-day remediation window in that context is not a metric. It is a confession.
The CDO and CTO together knew it. The question was where to start.
The Big Rocks Aurelia Had to Move
Rock 1: Treating prompts, contexts, and tools as first-class deployable artifacts
The first realization was that agent systems are made of more than code. They are made of: model versions, prompt templates, retrieval indexes, tool definitions, system instructions, eval datasets, and configuration. Each of these is a deployable artifact with its own lifecycle. Treating prompts as code, retrieval indexes as code, tool schemas as code — that conceptual shift required real engineering work.
Aurelia built an agent CI/CD pipeline that handled all of these as versioned, testable, deployable units. A prompt change went through the same review, eval, staging, canary, and rollback process as a code change. A retrieval index update went through eval before it could be promoted to production. Tool definitions had typed schemas, and changes to those schemas required compatibility checks.
This took about 4 months to build properly. It was the foundation everything else sat on.
Rock 2: Eval as a discipline, not a folder
The eval suite is the most underinvested part of most agent programs. People write 50 test cases during development, run them on release, and consider it done. That is not eval. That is a smoke test wearing a costume.
Aurelia's eval discipline was built around three principles:
Golden datasets: every agent had a curated, version-controlled set of canonical inputs and expected behaviors, growing over time as new edge cases were discovered. The Production Operations agent that ran inside Aurelia Pictures and Lantern Animation eventually had 1,800 golden test cases covering everything from talent scheduling conflicts to post-production handoff disputes.
Adversarial testing: every agent was put through a standing red-team process that tried to break it. The red team's incentive structure was tuned so that finding new failure modes was rewarded, not punished. For guest-facing agents, the red team also ran "social engineering" scenarios — guests trying to extract information they should not get, guests trying to manipulate the agent into making promises the company couldn't keep.
Production-grounded eval: the eval suite was continuously updated from real production interactions that had been flagged (by guests, by cast members, by the supervision team, or by automated drift detection) as worth examining. The folder grew because the world surfaced new things the agent needed to handle.
The discipline is not glamorous. It is also the difference between an agent that holds up at scale and one that quietly degrades.
Rock 3: Agent on-call as a real operational muscle
Most engineering teams know how to run on-call rotations for traditional services. Almost no teams have a developed practice for agent on-call. The skills are different: you need someone who can read agent traces, distinguish a model regression from a retrieval failure from a tool integration issue, and decide between rollback, hotfix, and live patch — all under time pressure.
Aurelia established an agent on-call rotation across the COE engineering team and the embedded BU pods. They built runbooks for common incident classes (retrieval index corruption, tool API outage, model provider degradation, prompt regression after release, downstream system error cascade — for example, what happens when the Guest Concierge agent's connection to the dining reservation system fails on a peak Saturday at the Kingdom).
They invested in observability tooling that surfaced agent-specific signals — token usage, retrieval recall, tool call success rate, response latency distribution — alongside traditional service metrics.
A specific operational shift: agent on-call coverage was anchored to the company's busiest hours, not to the engineering team's working hours. The peak hours for the Guest Concierge agent were Saturday and Sunday between 10am and 8pm local time at each park. The peak hours for the Yield agent were the first business hour of each market day. The peak hours for Aurelia Skies' crew rostering agent were 0500 every day. On-call rotations were designed around those peaks, not around the convenience of the team in Burbank.
Within 6 months of standing up agent on-call, MTTR dropped from 14 days to 6 hours. The fact that it could drop that far was a measure of how immature the original posture had been.
What "Leading" Looks Like
Eighteen months in, Aurelia's ADLC maturity looks like this:
A unified agent CI/CD pipeline handles all agent artifacts (prompts, indexes, tools, configurations) with deployment, eval gates, canary rollouts, and automated rollback
Every production agent has a golden dataset of 500+ test cases, a documented eval scorecard, and an SLA on eval pass rate that is checked weekly
A red-team process runs continuously, with results fed back into eval datasets within the same release cycle
Agent on-call is a defined role with a rotating roster across about 75 engineers, with coverage anchored to operational peaks
MTTR for agent incidents is under 6 hours; the worst-case tail (the 95th percentile) is under 24 hours
Deployment frequency for agent updates is 3-4 times per week; model upgrades follow a separate, more cautious cadence with bake-in periods in staging that mimic peak operational load
A standing post-incident review process treats every agent incident as an organizational learning opportunity, not a blame event — modeled, deliberately, after the post-show notes process the company's Live division uses after every Broadway preview
The headline difference between Aurelia at month 1 and month 18 is not technological. It is operational. The team learned how to run agents like production software, and the production software is what gets the Real People Impact to land.
The RPI at the Other End
The cast members who relied on Aurelia's agents — the front-desk hosts at the resorts, the gate agents at Aurelia Skies, the stateroom managers at Aurelia Voyages, the line producers at the studios — did not care about the eval framework. They did not know it existed. What they cared about was that the agent was reliable. When it answered, it answered correctly. When it didn't know, it said so. When something changed, they noticed it before they got embarrassed in front of a guest.
That trust — the day-to-day, accumulating sense that the agent could be relied on — is what eval discipline buys you. It does not feel like impact when you're investing in it. It feels like impact when, six months later, the front-desk host says, "I trust this thing now," and means it. The Net Promoter Score among Aurelia cast members for the agents they used daily climbed from 12 to 64 over the course of the maturity build-out. The biggest single factor in that move was reliability.
That is the RPI signal for ADLC. Not the eval pass rate. Whether the cast member on the other end has stopped second-guessing.
What to Do This Week
If you want to test where your ADLC maturity sits, ask three questions of your engineering team:
If we needed to roll back our most critical agent's last release in the next 10 minutes — say, in the middle of a peak operational window — could we do it cleanly, with full audit, and without losing any in-flight transactions?
What is the size of our golden eval dataset for our most critical agent, and when was the last addition to it from a production-flagged case?
Who is on agent on-call right now, and do they have a runbook for the three most common incident classes?
If those questions make your engineering team uncomfortable, you have the start of a roadmap.
Next article: Technology Platform. The cost of tool sprawl, and what Aurelia consolidated to start moving at platform speed across 11 business units.
This is Part 6 of a 9-part series on agentic enterprise maturity.