MIT, in collaboration with Toyota Research Institute, has developed a data driven simulation system with which autonomous cars can learn to drive in real world conditions and navigate through worse case scenarios.

Called Virtual Image Synthesis and Transformation for Autonomy (VISTA), the photorealistic simulator makes use of a small data set of driving trajectories by human drivers.

From real data, it then synthesises new trajectories in line with the road appearance, distance and motion of all objects in the scene.

Researchers first collected video data from a human driving down a few roads and then fed this into the simulation engine. The engine projects every pixel for every frame into a type of 3D point cloud.

A virtual vehicle is placed inside that world. When the car makes a steering command, this engine synthesises a new trajectory through the point cloud. This data is synthesised based on the steering curve and the car’s orientation and velocity.

The engine uses a new trajectory to create a photorealistic scene. For that, it deploys a convolutional neural network to estimate a depth map.

This map features information associated with the distance of objects from the control system’s viewpoint.

The simulation system brings together the depth map and a technique that can understand the camera’s orientation through a 3D scene. This process helps pinpoint the car’s exact location and the relative distance it stands from everything within the virtual simulator.

Using that information, the system readjusts the original pixels to recreate a 3D representation of the world from the car’s new viewpoint.

The motion of the pixels is also tracked by the engine to capture the movement of the moving objects in the scene.

MIT’s CSAIL director Daniela Rus said: “This is equivalent to providing the vehicle with an infinite number of possible trajectories. Because when we collect physical data, we get data from the specific trajectory the car will follow. But we can modify that trajectory to cover all possible ways of and environments of driving. That’s really powerful.”