Preparing for Event Driven Agents

In the last post we walked through what a team needs in place before moving from request/response to Event Driven Architecture: eventual consistency patterns, schema governance, idempotency, testing discipline, infrastructure ownership, event visibility, and distributed tracing. Every one of those items assumed something we never bothered to state out loud. It assumed the consumer was a deterministic function. Same event in, same effect out, every time.

That assumption is now breaking. Teams that spent the last two years building event backbones are starting to hang AI agents off those topics, and an agent is not a normal consumer. It reasons, it calls tools, it holds state, and given the same input twice it may do two different things. The good news is that EDA is the right foundation for agents. Loose coupling, replay, partitioned ordering, and a durable log are exactly what a network of autonomous workers needs.

The harder news is in how most teams are wiring them up. An agent that gets its context handed to it out of band, reaches enterprise systems through a side door, and is audited by a separate tool is not part of the fabric. It is a bolt-on, and it will be governed like one. The design goal is to make agents first-class participants in the same governed event fabric the services already use, subject to the same contracts, the same authorization, and the same telemetry. Getting there means every item on the readiness checklist needs a second pass.

A traditional consumer is a pure-ish function over a payload. An agent consumes the event, decides what the event means, chooses a plan, and picks which tools to invoke. The branching is no longer visible in code. It lives in a prompt, a model version, and whatever context got assembled at that moment.

This changes the unit of reliability. In classic EDA we ask whether the message was processed. With agents we have to ask whether the message was processed correctly, and correctness is now a judgment, not an assertion. Plan for a confidence dimension on every agent output, and plan for what the system does when confidence is low. In practice that means a review topic and a human in the loop, not a silent retry.

Consumer idempotency in a Kafka or SQS world is usually solved with a processed-message table keyed on message ID. That still works and is still required. It is no longer sufficient.

Consider an agent that receives EnrollmentHoldPlaced and decides to email the student, open a case, and update a record. A duplicate delivery hits your dedupe table and is dropped. Fine. But a retry after a partial failure is different. The agent may re-plan and this time take four actions instead of three, or word the outreach differently, or pick a different tool. The message was idempotent. The effect was not.

The pattern that holds up is effect-level idempotency. Every side-effecting tool call carries a deterministic idempotency key derived from the triggering event ID plus the action type, and the tool, not the agent, enforces uniqueness. Treat the agent’s plan as advisory and the execution layer as the guardrail.

Schema Registry validates shape. It does not validate meaning, and meaning is what an agent consumes. An event that passes Avro validation can still be uninterpretable to a reasoning consumer if the field is called stat_cd and the value is R2.

Event-driven agent systems need two contracts, not one. The structural contract is the schema you already govern. The semantic contract is the description, enumeration, and business meaning attached to those fields. Push that into the registry as documentation and enumerated value descriptions rather than letting it live in prompt strings scattered across five repositories. When R2 changes meaning next term, you want one place to change it.

The underlying claim is worth stating plainly: events cannot function as a universal interoperability primitive unless they carry enough context for the consumer to interpret them without help. A deterministic service gets that context from code its developers wrote. An agent has no such fallback. Whatever meaning is not in the event, or reachable through the metadata attached to it, is simply not available at reasoning time.

The mechanism that makes this sustainable is the data contract pattern applied across every surface at once: one declarative definition per data product rather than a separate one per platform. A single contract names the schema, the topic, the table, the serving interfaces, the policy bindings, and the semantic context, and the platform provisions from it:

Done this way, semantics travel with the event instead of sitting in a registry nobody updates, and the same governance applies whether the consumer is an API client, an analytics query, or an agent. It also changes the incentive. Teams publish into the governed ecosystem because it saves them from hand-building CRUD services, event publishers, and contract enforcement, not because a compliance function told them to.

The same versioning discipline applies to the agent itself. A prompt revision or a model upgrade is a behavioral deployment. Version it, changelog it, and be able to answer which agent version produced a given output six months later. Treat the persona and tool manifest as release artifacts subject to the same review gate as a schema change.

The invisible spaghetti problem gets worse before it gets better. In the previous post, the challenge was tracing which service subscribed to which event across repositories. With agents, subscription is only half the picture. You also need to know what the agent is permitted to do once it consumes.

The multi-agent patterns worth knowing map cleanly onto primitives you already operate:

Orchestrator and workers. A coordinating agent publishes work to a partitioned topic, and worker agents consume as a consumer group. The consumer group protocol handles rebalancing, scaling, and recovery, so the orchestrator never maintains a worker registry. Key on the entity, student ID or order ID, so stateful work for the same subject lands on the same worker in sequence.

Hierarchy. Apply the same decomposition recursively. Each non-leaf agent is the orchestrator of its own subtree, publishing objectives downward and consuming results upward. Depth becomes a topology decision rather than a code change.

Shared blackboard. A compacted topic as a shared working memory that multiple agents post to and read from. This is how you give a set of agents shared context without every agent calling every other agent and without a central state service becoming the bottleneck.

The Event Portal recommendation from the last post now needs an agent registry beside it: which agents exist, what they subscribe to, what tools they hold, what they are allowed to write.

Temporal assertions and Testcontainers still apply. Spin up the broker, publish, assert eventual state. But you cannot assert exact equality on an agent output, and a test suite that does will be flaky for reasons that have nothing to do with timing.

Split the suite. Deterministic integration tests cover the plumbing: the event was consumed, the tool was invoked with the right idempotency key, the output landed on the right topic, the DLQ caught the malformed payload. Evaluation covers the judgment: a golden set of representative events with expected outcomes scored on task completion, tool selection accuracy, and hallucination rate. Run evals in CI with a threshold, and treat a regression the way you would treat a failing integration test.

Record and replay is the bridge between the two. Capture real model responses from a run, store them as fixtures, and replay them so your plumbing tests stay deterministic and cheap.

OpenTelemetry across the board was the right call for EDA and it is the right call here. Extend the span model. A useful agent trace carries the triggering event offset and partition, the model and prompt version, the retrieved context, the delegated authority the agent was acting under, the tool calls attempted with their results, the prompt-context boundary, token usage, and the final action. Without the reasoning inside the trace, you can see that an agent did something strange but never why.

The silent disappearance failure mode from the last post gets a sibling: the confident wrong answer. The event flows, the agent responds, nothing errors, and the outcome is incorrect. Only outcome-level monitoring catches that. Instrument business results, not just pipeline health.

Inference cost as a scaling property. In classic EDA, adding a consumer to a topic is close to free. Add an agent and every event on that topic now has a per-message inference cost. A fan-out that was architecturally elegant becomes a line item. Filter and enrich upstream with stream processing so agents only see events they should actually reason about. Shifting work left is now a cost decision as much as a latency one.

Model provider rate limits as a first-class failure mode. Your broker will happily deliver ten thousand events per second into an agent whose provider caps you at a fraction of that. Backpressure, bounded concurrency per consumer, and a graceful degradation path are not optimizations. Design them at the same time you design the partition count.

Everything that flows into an agent flows into a prompt, and in regulated environments that is a new exposure surface. Field-level encryption and masking in the pipeline matters more than it did when the consumer was a service that only touched the three fields it needed. Data lineage now has to extend past the topic into the model call and the vector store. Retention policy has to account for context that was embedded and stored somewhere else.

There is a harder problem here than masking. When an agent acts on behalf of a user, it is a distinct principal holding delegated authority, and the access decision has to evaluate three things at once: what the agent is, what relationship the user has to the resource, and what purpose the agent is acting under. Role-based access control answers none of that cleanly. Zanzibar-style relationship models, where access derives from tuples connecting users, organizational units, and resources, hold up better, with OPA-style policy-as-code layered on top to constrain by purpose, sensitivity, environment, and workflow state.

Put an enforcement point on the outbound path as well as the inbound one. An agent can retrieve exactly what it is entitled to see and still compose a message that should not go out, to that recipient, on that channel, at that moment. A policy check on the way out can deny the transmission and hand the refusal back to the agent, which then regenerates a compliant response, for example pointing the recipient at a self-service portal rather than restating the sensitive detail in the message body. That is autonomous remediation inside a governance boundary, and it only works if policy evaluation sits in the execution path rather than in a quarterly review.

The upside is that the log gives you something no synchronous architecture does: a complete, replayable, immutable record of what the agent saw and when it saw it. For any audit conversation about an autonomous decision, that record is the whole argument.

The same advice from the last post holds, with more weight behind it. Pick a non-critical flow. Put a single agent on a single topic with read-only tools, and run it in shadow mode against the existing process before it touches anything. Instrument first, then widen the tool permissions, then add the second agent. The organizations that get burned are the ones that give a multi-agent system write access to production systems before they can trace a single decision end to end.

We spent the last cycle learning to stop thinking in request/response. This cycle is about learning to build systems where the consumer thinks for itself, and building them so that we can still explain, replay, and trust every decision they make.

If you want a single number to start the conversation with your platform team, use this one: of the resources your agents can already reach, what fraction have policy, metadata, and audit actually attached? In most enterprises the honest answer is uncomfortable, and it is the gap worth closing first.