Drone threats have grown significantly in complexity and scale over recent years. Commercially available unmanned aerial systems, once associated purely with photography and recreation, are now routinely adapted for surveillance and offensive applications in both state and non-state contexts. In response, defence agencies and private contractors have invested heavily in AI-driven detection systems capable of identifying, classifying, and tracking aerial threats in real time.
Every serious detection model needs training data. For any deep learning system, that means tens of thousands of labelled video frames showing the precise targets the model must learn to recognise — across varied lighting, distances, altitudes, and approach angles.
Here is the fundamental problem: the scenarios a counter-drone AI most needs to learn from are precisely the ones that cannot be safely or legally filmed.
The data acquisition constraint: Filming within restricted airspace requires permits, coordination with aviation authorities, and safety oversight. Replicating representative threat trajectories — high-speed low-altitude approaches, aggressive dive profiles — is not feasible in a civilian test environment. Real footage acquisition is dangerous at worst, and prohibitively expensive at best.
The result is that most counter-drone AI systems are trained on sparse, incomplete data — and their operational performance reflects that gap. FX-Sim is a different approach entirely. Rather than filming the physical world, we built a photorealistic virtual replica of a deployed surveillance system and generate training data synthetically, automatically, at any scale, for any scenario.
The term “digital twin” is used loosely in technology marketing, so a clear definition is warranted. A digital twin is not a 3D model, a visualisation tool, or a simulation in the conventional sense. It is a behavioural replica of a physical system — one that mirrors the real system’s inputs, outputs, and operational characteristics closely enough to substitute for it in testing and data generation workflows.
The methodology is well-established across high-stakes industries. Aerospace engineers use digital twins to validate engine behaviour without physical test flights. Automotive manufacturers use them to certify advanced driver assistance systems across millions of synthetic driving scenarios before real-world trials begin. Surgical robotics researchers train autonomous instruments in virtual operating environments where real-world failure would be unacceptable.
FX-Sim is a digital twin of a counter-drone pan-tilt-zoom (PTZ) camera system. The hardware interfaces, communication protocols, video stream formats, and environmental conditions are all faithfully replicated in a virtual environment. The aerial models flying through it are reconstructed from documented specifications. The physics are accurate. The data generated is structured to be operationally equivalent to footage collected from a live field installation.
FX-Sim is built around a photorealistic 3D environment rendered in Unreal Engine 5. Within that environment, two virtual cameras are mounted on a motorised PTZ rig — one simulating a standard daylight optical sensor, one simulating a thermal infrared sensor. Both stream live video over RTSP, the same Real Time Streaming Protocol used by virtually every commercial IP security camera in deployment today.
The virtual camera system is controlled through a physical joystick using the Pelco-D RS-485 protocol — the same standard used in real deployed PTZ installations. An operator working with FX-Sim through the joystick has an experience mechanically identical to operating a live field system. Pan, tilt, and zoom commands are transmitted in hardware and applied to the virtual rig in real time. The only difference is that everything behind the stream is synthetic.
Flying through the scene are three-dimensional aerial models spanning multiple threat categories:
Each model follows physically plausible flight paths with configurable approach vectors, altitude profiles, and speed curves. The asset library is designed to be extensible as new threat profiles are documented.
The system is architecturally indistinguishable from a real camera installation — except everything inside it is synthetic.
Acquiring labelled drone footage in the real world is constrained at every step. Filming within restricted airspace requires coordination with aviation authorities. Replicating high-risk flight profiles is not feasible in any civilian context. And even where footage can be legally collected, manual annotation at scale is a significant operational bottleneck: drawing precise bounding boxes on tens of thousands of frames is expensive, time-consuming, and prone to inconsistency.
FX-Sim eliminates the annotation bottleneck entirely through automatic labelling. Because the rendering engine has precise knowledge of every object’s 3D position in the scene and every camera’s geometric parameters, it computes exact bounding box coordinates for every aerial object in every frame — in real time, with zero human input. Each frame is exported alongside a structured label file ready for direct ingestion into standard object detection training pipelines.
The result is a large-scale dataset spanning multiple object classes, varied environment types — open terrain, semi-urban, coastal, and arid — and a range of lighting conditions from midday to low-light with active thermal imaging enabled. No labelling pipeline. No annotation tooling.
Detection accuracy is the metric that appears in AI performance reports. What rarely gets discussed is the equally critical problem of false positive rate in operational environments.
A counter-drone system deployed at a critical infrastructure site does not operate in a controlled environment. It operates in shared airspace — where commercial aircraft pass overhead, where birds with similar visual silhouettes cross the camera frame, where utility helicopters travel nearby corridors. A model trained exclusively to detect drones will inevitably trigger on these objects. In operational practice, false alarms degrade operator trust, consume response capacity, and lead system administrators to reduce detection sensitivity — precisely the failure state that undermines the purpose of the system.
An industry-wide pattern: Many deployed counter-drone detection systems are threshold-tuned to suppress false positives — effectively reducing their sensitivity to genuine threats in order to manage the noise introduced by aircraft, birds, and other aerial objects. This is a direct consequence of training data that contains only positive drone examples and no contextual non-threat classes.
FX-Sim addresses this at the data level. The training dataset explicitly includes labelled non-drone classes — commercial aircraft, helicopters, and birds of prey — treated as distinct detection categories. By training a multi-class detector rather than a binary classifier, the model learns to distinguish genuine drone profiles from visually similar aerial objects without any manual threshold tuning. The result is a detection system that is simultaneously more sensitive to real threats and more resistant to false alarms.
The following is an honest accounting of FX-Sim’s implementation status — grounded in what is working today, not what is on the roadmap.
Live stereo camera streaming is operational over RTSP at 30 frames per second, with simultaneous output from both the daylight and thermal channels. The PTZ joystick integration is complete — hardware pan, tilt, and zoom commands are transmitted via the Pelco-D protocol and applied to the virtual camera rig in real time. Initial aerial models are imported, rigged with accurate geometry, and animated on spline-defined trajectories through the scene.
The dataset generation strategy covers multiple object classes across several distinct environment archetypes and a comprehensive range of lighting conditions, including active night scenarios with thermal imaging enabled. The pipeline is structured so that any new environment, lighting preset, or aerial model can be introduced and rendered into the dataset without changes to the labelling workflow.
The next milestones are the full dataset generation run across all configured environments and lighting conditions, followed by initial baseline training using the YOLOv8 object detection framework.
FX-Sim is one implementation of a general methodology. The underlying principle — build a high-fidelity digital twin of a real-world sensing system and use it to generate training data for scenarios that cannot be safely or legally captured in reality — applies across a wide range of domains where AI is being deployed in high-stakes environments.
Autonomous vehicles use simulated environments to generate rare edge-case scenarios that would be dangerous or impossible to stage on real roads. Surgical robotics programmes train instrument manipulation models in virtual operating theatres before committing to physical trials. Industrial inspection AI learns to identify structural fault patterns using synthetic replicas of infrastructure where real defect footage is sparse, sensitive, or simply does not exist in usable form.
In each case, the value proposition is the same as FX-Sim’s: remove the physical, legal, and safety constraints that limit real-world data acquisition, and replace them with a simulation environment capable of generating data on demand at any scale. The difference between an AI system trained on a few hundred labelled examples and one trained on a large, diverse, well-structured dataset is not incremental — it is often the threshold between a research prototype and a system capable of operational deployment.
The constraint on counter-drone AI is not algorithmic. It is the data — and digital twins are precisely how that constraint gets removed.
The next phase of FX-Sim closes the loop between data generation and model deployment. Once the initial detection model is trained on the synthetic dataset, it will be integrated directly back into FX-Sim — running inference on the same RTSP stream it was trained to analyse. An aerial object appearing in the virtual environment will be detected, classified, and annotated in real time by the model that the environment itself produced.
This closed loop — simulation generating data, data training the model, model running inside the simulation — serves as the final validation step before any real-world deployment. It enables the system to be stress-tested, accuracy benchmarked, and failure modes characterised entirely in a synthetic environment, before operational footage is required.
