A simulation agent system is not a chat box bolted onto simulation software. It compiles a complex task into a structured plan, lets agents with distinct responsibilities share world state, call constrained tools and execute inside a physical environment, and saves every decision, action, response and score as a replayable trajectory.
The Synrene simulation agent cluster organises scenes, data, simulation, evaluation, decision-making and learning feedback into one execution chain. What the customer submits is not a question awaiting an answer but a simulation task that has to be planned, scheduled, verified, recovered and delivered.
Why an agent cluster
A real simulation task involves semantic understanding, asset retrieval, spatial layout, physics settling, sensor configuration, task scheduling, result scoring and fault recovery all at once. Having a single model handle every stage in one inference pass lets the uncertainty of natural language pass straight into the physical world, and blurs which stage is responsible when the task fails.
The agent cluster resolves that by dividing responsibility:
- Plannerbreaks a business goal into scene, data, run and evaluation plans.
- Orchestratormanages dependencies, events, resources, ordering, timeouts, retries and termination conditions.
- Worker agentshandle asset, scene, sensor, simulation, data and reporting tasks separately.
- Shared world stateholds objects, relations, constraints, versions, snapshots, events and evaluation results in one place.
- Criticsinspect intermediate artefacts and run results from a visual, physical and safety standpoint.
- The simulation environmentresponds physically and behaviourally to agent actions, which closes the loop continuously.
What the cluster finally produces is a set of executable scenes, a complete run trajectory, evaluation evidence, a failure index and data usable for post-training — not a passage of unverifiable free text.
The solution and what it covers
1. Task compilation: from business language to a Run Spec
The task compiler takes a natural-language task, reference imagery, maps and assets, business rules, safety constraints, sensor configuration and acceptance criteria, and emits a structured Run Spec.
The Run Spec states explicitly:
- task goals, stages, dependencies, success conditions and stop conditions;
- scene objects, spatial relations, initial state and environment variables;
- agent roles, permissions, tools, resources and collaboration relationships;
- observation, action, event, state and data interfaces;
- the inspection rules for the Visual, Physics and Safety critics;
- simulation run parameters, random seeds, snapshot points and recovery strategy;
- evaluation metrics, ground-truth sources and delivery data requirements.
Compilation first checks object references, role dependencies, tool parameters, constraint conflicts and termination conditions, and only then hands the task to the cluster. Natural language expresses intent; the structured contract constrains execution.
2. Planner and Orchestrator
The Planner decomposes towards the task outcome, deciding which scenes, data, simulation runs and evaluation steps are needed and establishing the dependencies between stages. For a complex task it can split one goal into parallel subtasks, prerequisite validation and result aggregation.
The Orchestrator turns the plan into a schedulable execution graph:
- selects worker agents by capability and permission;
- manages serial, parallel, conditional and looping tasks;
- maintains resource occupancy, priority, timeouts and stop conditions;
- listens for tool, simulation, critic and human-approval events;
- triggers retries, alternative workers, snapshot rollback or replanning;
- aggregates results across roles and decides whether to enter the next stage.
The Planner decides "what has to be achieved and how it decomposes"; the Orchestrator decides "when it runs, who runs it, and what happens when it fails". Separating the two lets task strategy and run control be configured independently.
3. Specialist worker agents
Worker agents are organised by specialism, and each receives only the context, tools and permissions its own task needs:
- Asset agent: retrieval, filtering, version locking and dependency checks;
- Scene agent: scene graph, object relations, layout and scene variants;
- Physics agent: colliders, mass, inertia, joints, contacts and settling conditions;
- Sensor agent: camera, LiDAR, radar and IMU configuration plus capture parameters;
- Run agent: starting, monitoring, pausing, resuming and terminating simulation tasks;
- Data agent: capture, annotation, cleaning, quality inspection, versioning and distribution;
- Evaluation agent: metric computation, failure clustering, version comparison and report generation;
- Recovery agent: chooses retry, substitution, rollback or replanning from the structured error.
Every worker returns a structured result, status, evidence and error. A natural-language description never stands in for an executable artefact.
4. Shared world state
Shared world state is the common source of truth the cluster collaborates around. It holds:
- the scene graph, objects, parts, relations, poses and spatial constraints;
- asset, scene, sensor, policy, tool and software versions;
- agent roles, task assignment, resource occupancy and collaboration relationships;
- physical state, sensor state, task state and evaluation state;
- run snapshots, events, tool results, critic conclusions and errors;
- success, failure, recovery, human intervention and counterfactual branches.
Agents update world state through controlled reads and writes. Each update carries version, origin and time information; conflicting writes are handled by resource locks, conditional updates or Orchestrator arbitration. The cluster therefore works around one set of world facts instead of several roles each inferring a mutually contradictory scene.
5. MCP tool contracts
MCP exposes asset retrieval, scene generation, PCG, 3DGS, rendering, sensors, simulation runs, data processing and evaluation as structured tools. Each tool declares:
- name, responsibility and the roles it suits;
- JSON Schema input and output;
- permission, resource and runtime environment requirements;
- synchronous call or asynchronous task semantics;
- timeout, cancellation, idempotency and retry rules;
- status, progress, result and structured error;
- run trace, version and audit information.
A lightweight operation can run as an in-process tool, a local compute task can go to a sidecar, and a long-running or heavy task can be submitted to a remote queue. Agents always work against the stable tool contract; the concrete executor attaches through an adapter layer.
6. Visual, Physics and Safety critics
A critic's job is to inspect and rule, not to keep generating:
- Visual Criticchecks whether objects appear, whether class and appearance match, whether spatial relations satisfy the task, and whether rendering and observation meet the scene semantics.
- Physics Criticchecks interpenetration, floating objects, support, collision, joints, mass, inertia, contact and whether the motion result is physically feasible.
- Safety Criticchecks exclusion zones, distances, permissions, task rules, resource limits, risk events and stop conditions.
A critic emits pass, reject or a correction request, together with the offending object, the time, the evidence, the rule and a suggested action. The Orchestrator then chooses to repair, rerun, roll back, switch worker or terminate.
7. Simulator-in-the-loop and event scheduling
An agent's plan has to enter the simulated world for closed-loop validation. The execution cycle consists of:
observe the world → update shared state → Planner or policy decides → worker calls a tool or emits an action → simulation advances → critics and ground truth score it → the next event fires
The event scheduler handles, in one place:
- simulation time, agent task time and external message time;
- sensor sampling, policy decisions, control actions and environment responses;
- tool completion, state changes, critic results and human approvals;
- timeouts, faults, task stage transitions, snapshots and termination;
- parallel execution across roles and conflict ordering on a shared resource.
Every action is bound to the version of world state it read, which stops an agent continuing from a stale snapshot.
8. Observability, recovery and memory
The system builds one trace per run, recording task decomposition, agent decisions, tool calls, parameters, results, world state changes, simulation events, critic conclusions and recovery actions.
Observability covers:
- correlation identifiers at the run, stage, agent, tool, simulation and critic levels;
- state snapshots, the event timeline, inputs and outputs, and versions;
- latency, resources, retries, errors and termination reasons;
- the first failure, the first state divergence and the downstream tasks affected.
Recovery is chosen by failure type:
- a transient tool error gets a bounded retry;
- an unsuitable worker means switching role or executor;
- a substandard local artefact means returning to the nearest checkpoint to correct it;
- an anomalous world state means restoring a snapshot;
- an unexecutable plan goes back to the Planner for redecomposition;
- a triggered safety constraint stops the related actions and preserves the evidence in place.
Memory has three layers:
- Run memoryholds the state, events and intermediate results of this task;
- Project memoryholds customer terminology, assets, rules, interfaces and evaluation conventions;
- Experience memoryholds reusable success patterns, failure patterns, repair actions and tool choices.
Anything written to long-term memory must pass structured extraction, source association and a permission check, so an incidental description from one run is never frozen into a rule.
9. Closed-loop and counterfactual rollouts
A closed-loop rollout records the agent's continuous observations, decisions, actions and feedback in a dynamic environment. Participants in the environment respond to the agent's behaviour and task state changes with execution results, which makes it possible to evaluate long-term consequences beyond a single-step output.
A counterfactual rollout branches from the same snapshot and changes only the designated variable:
- substitute the policy, Planner, Orchestrator or worker;
- adjust role division, collaboration relationships and tool permissions;
- change the intent, events or physical parameters of environment participants;
- inject sensor, communication, actuator or tool faults;
- modify task constraints, resource limits or stop conditions.
Branches share the same prior history, and the system compares task outcome, key events, role trajectories, critic conclusions and ground-truth metrics. The customer can use that to analyse whether a given failure came from the policy, the collaboration, the tools, the environment or the physical execution.
10. Trajectory recovery, post-training and deployment adaptation
Once cleaned, segmented, annotated and versioned, run trajectories convert into:
- demonstrations of successful tasks, tool calls and role collaboration;
- preference pairs of success against failure from the same state;
- process supervision data carrying critic feedback and constraint events;
- failure actions, recovery actions and correction trajectories;
- evaluation sets composed of fixed scenes, failure conditions and metrics;
- policy data composed of observations, actions, feedback, termination and environment state.
The deployment adaptation layer aligns state, action, tool, model and communication interfaces between simulation and the real system. When a policy or agent version is updated, regression runs first against the fixed scenes, the failure index and the safety rules, and delivery then follows the deployment protocol.
The run loop
A customer task runs along this chain:
- Submit the task: business goal, reference material, scene resources, policy interface and acceptance criteria.
- Compile the spec: produce the Run Spec, scene graph, role graph, tool inventory, event graph and evaluation rules.
- Plan the division of work: the Planner decomposes the task, the Orchestrator assigns workers, dependencies and resources.
- Initialise the world: load assets, the machine, the environment, sensors, agents and shared state.
- Execute and inspect: workers call MCP tools; the Visual, Physics and Safety critics inspect stage artefacts.
- Run the loop: policy and agents enter simulator-in-the-loop, and the environment keeps responding to actions.
- Recover from faults: the system retries, substitutes, rolls back or replans based on events, errors and snapshots.
- Compare counterfactuals: run branches with different policies, roles, events and constraints from a key snapshot.
- Recover trajectories: organise success, failure, recovery and branch trajectories, and build the failure index.
- Adapt for delivery: emit evaluation evidence, post-training data and deployment interface mappings.
What the customer receives
Scene package
The scene package contains the structured task, the scene graph, the asset and version inventory, roles and their collaboration relationships, sensor configuration, events, constraints, random seeds, tool contracts, evaluation rules and stop conditions. It can be used directly for reproduction and batch runs.
Run trajectory
The run trajectory holds world state, agent decisions, tool calls, simulation feedback, critic conclusions, recovery actions, key snapshots and the termination reason. A customer can view the full execution chain by run, agent, tool, object or event.
Evaluation report
The evaluation report covers task outcome, role collaboration, tool execution, visual consistency, physical feasibility, safety constraints, key events and version differences, each linked to the corresponding trajectory and world snapshot.
Failure index
The failure index is organised by task stage, scene factor, role, policy, tool, critic, fault and root-cause label. Each entry carries a reproduction entry point, the key time window, the first state divergence, the evidence and the relevant versions.
Post-training data
Post-training data is delivered by purpose as demonstration trajectories, preference pairs, process supervision, correction data, evaluation sets and policy rollouts, with a data manifest, provenance, versions, quality results and training interface notes.
Typical applications
- Natural-language scene orchestration: compile a business description into a scene that is executable in objects, relations, assets, physics and evaluation alike.
- Multi-agent task rehearsal: validate role division, resource contention, communication, collaboration, failover and task outcome.
- Closed-loop policy evaluation: put a decision or control policy into an environment that responds to its actions, and inspect long-term behaviour and safety constraints.
- Failure reproduction and counterfactual analysis: compare alternative policies, role configurations, tool results and environment responses from a run snapshot.
- Automated data production: let agents organise scene, simulation, capture, quality inspection, versioning and distribution tasks.
- Post-training data generation: turn success, failure, recovery and critic feedback into a traceable data product.
Closing
The value of a simulation agent cluster is not the number of roles but the fact that tasks, state, tools, the physical world, evaluation rules and learning feedback are connected into an executable loop. Task compilation makes the input verifiable, the Planner and Orchestrator make collaboration schedulable, MCP makes tools constrainable, the critics and simulator-in-the-loop make results pass visual, physical and safety inspection, observability, recovery and memory make a run traceable, and rollout recovery turns every execution into an evaluation and post-training asset.
References
- IAWG World Generator: structured scene intent, asset matching, physics settling and USD output
- Decision Simulation Platform: dynamic AI digital twin, closed-loop evaluation and counterfactual rollouts.
- SAGE, arXiv:2602.10116v2
- DriveArena: A Closed-loop Generative Simulation Platform for Autonomous Driving
- Real-is-Sim: research on a dynamic digital twin running through training, evaluation and deployment as one loop.
- RoaD: research on organising simulation rollouts into demonstrations that feed back into policy learning.
- AI model post-training: from supervised fine-tuning to agentic RL.