Synthetic data for better, cost-efficient satellite data-based AI models

Background & Context

Recent advancements in satellite technology, particularly in higher temporal and spatial resolution, have made it possible to monitor the Earth with unprecedented detail and frequency. This has opened the door to entirely new use cases employing AI - from near-real-time infrastructure monitoring to precision agriculture and dynamic risk assessment. These advancements are allowing NTT DATA to address complex challenges and create significant new value for clients across industries. Furthermore, satellite technology continues to evolve rapidly, and we expect a significant expansion of opportunities in the long term.

Despite the many success stories and consistent pipeline of opportunities regarding satellite data, we have also identified significant challenges in acquiring and utilizing large volumes of satellite data for AI development, which are generally essential for robust, high-performing AI. These challenges are driven by the high costs of data acquisition, the difficulty in obtaining specific data due to infrequent appearances of target objects or target situations, and the high costs associated with data annotation.

To remain competitive and meet growing demands, identifying a reliable solution to streamline and reduce the costs of satellite data based AI development became essential. The Satellite Team at the Innovation Center identified synthetic data as the best solution to our challenges and Bifrost AI as a potential partner due to their synthetic data generation technology.

This collaboration aimed not only to test the impact on AI development of Bifrost AI's synthetic data technology, but also to explore its integration into development workflows and business processes.

About Bifrost AI

Bifrost AI, headquartered in San Francisco and founded in 2020 by Charles Wong and Aravind Kandiah, is transforming how physical AI is built by making high-quality data accessible at scale through advanced synthetic data generation.

Its foundation is a physically-accurate 3D simulation engine, designed to be fully programmatically accessible. This enables users to generate realistic, tailored datasets that replicate rare, complex, or high-risk scenarios - the kinds of conditions that are often prohibitively expensive or impossible to capture in the real world at the volume AI systems require.

This programmatic approach offers a level of precision, automation, and customization that outpaces traditional 3D simulators and the current wave of 2D generative AI tools. For organizations like ours, it creates new opportunities to accelerate AI development, unlock the full potential of satellite data, and move faster toward strategic goals.

Examples of synthetic images generated by Bifrost AI's tool

Collaboration Scope & Objectives

The joint collaboration was defined with the following core objectives:

• Evaluate the usability of Bifrost AI's technology and determine how easily it could be adopted into our existing AI development workflow;
• Assess the extent to which Bifrost AI's simulation platform could be used to customize synthetic data to address specific requirements and scenarios;
• Analyze the direct impact of synthetic data generated through Bifrost AI's technology on the performance of developed AI.

The collaboration was designed as a proof of concept over a period of 3 months and the target AI models were object detection and change detection models, where the change detection model targeted a specific type of change, rather than general change.

PoC Approach

Technical Setup: The PoC leveraged AI models already developed by NTT DATA and the satellite image datasets that each model had been originally developed on.

We have taken the following common approach in the preparation of our new datasets including synthetic images and in the training and evaluation of the updated AI models for each scenario:

1. Prepare the original dataset used for each model development, with the same training, validation and test data distribution
2. Place target objects or change detection target objects in the original dataset images and generate new, synthetic images
3. Retrain the untrained model on the new training dataset
4. Test the new model on the original test dataset (unchanged, not containing synthetic images)

Data details: We have used 2487 images as the original dataset for the object detection model, and 414 images for the change detection model, both datasets consisting of high-resolution optical satellite images.

Evaluation: The evaluation metrics involved recall, mean average precision (mAP) and F1-score values, all typical metrics in the evaluation of image based AI models.

Results & Key Findings

The PoC proved the value of synthetic data usage in the training of AI models, showing significant improvements in AI performance obtained with a small number of synthetic images:

• for object detection, 86 images have been updated to include synthetic objects
• for change detection, 70 images have been updated to include synthetic change

With these small dataset updates we have obtained the following results:

• Object detection: the mAP increased by almost 10 percentage points, reaching 0.89, while recall of object detection increased by 4 percentage points, reaching 0.93
• Change detection: the overall F1-score increased by almost 3 percentage points, with precision reaching 0.7 and recall reaching 0.57.

Qualitative insights we have obtained through this PoC include:

• Ease of use and smooth adoption of Bifrost AI's technology by the data scientist team due to the programmatic aspect of the synthetic data generation platform
• High customization of the position and appearance of objects and target changes, as well as ease in matching the appearance of synthetic object with the original image to create highly realistic, aspect-wise homogeneous images
• The output of annotations alongside synthetic images by the Bifrost AI platform implied that no additional annotation effort was required, the tool overall offering performance increase benefits at no additional data purchase and annotation cost.

Technical Insights & Lessons Learned

For the object detection scenario we have observed a very high similarity between the original and synthetic images, generally the synthetic images being indistinguishable from the original ones. On the other hand, in the update of the change detection model, we have understood that synthetic images can still have a positive impact on model performance despite visual discrepancies between the real and the synthetic images showing change. Our assessment of the results showed that improvement to the change integration within Bifrost AI's platform could indeed induce a higher improvement in performance, but the current integration method can still bring improvement.

During the project we have also attempted to approach an object detection model for which we have used additional satellite images, rather than the original dataset's images. Due to budget constraints, the new satellite images had a lower resolution than the original dataset, and despite the easy integration of objects in the appropriate resolution in these new images (the synthetic objects had themselves a lower resolution in the new images), the results did not show improvement with the usage of synthetic images, due to the fact that the test dataset had included original images and objects in higher resolution.

Strategic & Future Outlook

The collaboration between the NTT DATA and Bifrost AI highlights how combining deep AI expertise and cutting-edge simulation innovation can produce measurable technical and strategic benefits.