Simulators are a great way to kick-start modern AI enabled machine vision projects

Collecting data from the real world can be - and usually is expensive and time consuming. You need to wait for suitable conditions, which can be nearly impossible to get to, or might happen very rarely. On top of that, you need to have possibly expensive equipment, cameras for video, microphones for audio, and maybe even something fancier to collect the data that is needed.

But what if you didn't need any of that equipment? What if you could just say, give me 500 pictures of a parking lot at 1920x1080 resolution, when it is snowing, the clock is 7:49am on a January morning in 2011 and most of the cars in the parking lot are either red or yellow, all covered in snow? Well, that's exactly what I've done, using only a computer, without any other equipment.

What is the data used for?

This simulator is used to create images of a parking lot in different conditions. That is the training data for a computer vision algorithm, used to detect vehicles on a parking lot. The output is a list of JPG images and a list of ground truths in XML. The images show what the camera is seeing in the simulation each frame. The XML contains a list of bounding boxes, which are the ground truths, telling where all the vehicles are in the image. The ground truths can then be used to determine the accuracy (or the error rate) of the learning algorithm's predictions to see how well it is performing.

How does the simulator work?

I modeled the parking lot environment using the Unity game engine, and an open source 3D modeling program called Blender. To use the simulation, you first create a simulation configuration file. Every parameter is configurable there. These include, for example, the camera positions and rotations, how many and how fast the vehicles appear in the parking lot, the weather and its parameters, as well as the resolution of the output images.

Based on these parameters, a list of events is created for the span of the whole simulation. The list contains every event when a vehicle either entered or exited the parking lot. When the simulation is running, it takes every action that will happen during the current simulation step and executes them. After that, each camera saves the image it sees, and the bounding boxes of the vehicles it sees. These will be saved to disk to be used later.

The simulator also includes a configurator, where you can move the cameras around, rotate them and change their properties, such as horizontal and vertical field of views. This is so that you can quickly test which positions and angles provide the most coverage for the parking lot. You can then export this as a JSON configuration file and use it as the simulation configuration when loading a new simulation.

This simulator helped us find better positions for the cameras, which we otherwise would maybe not have tried. It gave us an idea of what the cameras would see in the real world, without having to install the cameras, which would've cost money and time tweaking their positions in the real world. So, simulators may not only increase the quantity of data, but also the quality of it by making the planning phase easier.

Simulator data sunny Simulator data rain Simulator data snow

Why does this matter?

The need for simulators to create data varies from project to project. But when you need consistency, reliability and parameterization, simulators are a strong candidate for the method of obtaining data.

Imagine a situation where you need data about what the lighting of an area looks like during a total solar eclipse. Here are a few problems with manual collection of data, applicable to the example:

  • Frequency of the event: You need to wait for a total solar eclipse to happen (For the curious: they happen approximately once every 18 months).
  • Frequency of suitable conditions: You need to be wherever the eclipse happens that time, and it may very well be on the opposite side of the planet. Traveling there costs a lot of money, depending on where it is. If the environment there doesn't work for what you need, for example, the eclipse lands on an ocean, you will need to wait for another one. Same applies, if it is cloudy at the time of totality.
  • Dependent on your equipment: Your equipment must work. Forgot to charge the batteries of your camera? Missed the bus for the few-hour event? Unfortunate, but there's nothing that can be done: 18 more months of waiting.
  • Amount of data: The amount of data may be so huge that it is simply not practical to collect it all manually. Take the parking lot for example, it would not be practical at all to go around parking lots taking pictures of them 1-by-1, waiting for it to rain or snow, waiting for different kinds of cars to enter and exit, etc.

With a simulator, you can generate as many perfect-condition situations as you want. The whole situation can be customized, the location, weather, environment, etc. Another benefit is that anyone can then collect the data. They don't need to know how to use a camera or any other required equipment, it's just a matter of configuring and starting the simulation. This will cut down on training costs, and any other costs that occur due to waiting for suitable conditions.

But as everything in the universe, this too comes back to balance: Are the benefits of a simulator greater than manually collecting the data? Creating a simulator may be better for the long-term, but is it worth the initial cost of creating it?

So, in conclusion, using a simulator to generate the data for a machine learning algorithm might be a good idea, but it really depends on your project. Here are a few questions to think about:

  • Do you need a lot of data?
  • How fast do you need the data?
  • Is the data difficult or impractical to get manually?
  • Is the initial cost of building a simulator worth the time saved?

Here you can read more about the AI solutions offered by Wapice.