MIT Advanced Vehicle Technology Study (MIT-AVT)

As part of the MIT Advanced Vehicle Technology Study (MIT-AVT), researchers are instrumenting vehicles with varying levels of automation to deeply analyze driver behavior and system interaction. A primary focus is placed on a Tesla Model S, which serves as a testbed for advanced instrumentation designed to capture comprehensive data regarding human-machine dynamics. This setup involves three distinct cameras strategically positioned within the vehicle: one monitoring the driver's face, another utilizing a fisheye lens to track the entire body including hand placement, and a third forward-facing camera attached to the windshield observing the external roadway environment. The internal instrumentation is critical for understanding the nuanced states of the driver beyond simple presence or absence. The facial camera captures where the driver is looking, their overall state, emotional condition, and cognitive load. Simultaneously, the body-worn fisheye camera provides essential data on whether hands are off the wheel and if the driver's posture is aligned correctly with safety protocols. These internal metrics complement the forward-facing camera, which records everything in the external environment such as other vehicles, lane markings, and road characteristics, allowing researchers to correlate internal physiological states with external driving conditions. This multi-camera approach enables the study of how people interact with automation over hundreds of thousands of miles of real-world driving across different vehicle models. To date, the MIT-AVT has collected data from 275,000 miles involving Tesla Model S vehicles, Land Rover Evoque vehicles, and Volvo S90s. The sheer volume of raw data generated is immense; researchers are processing approximately 3.5 billion video frames consisting purely of raw pixels. This massive dataset represents a unique opportunity to observe human interaction with artificial intelligence systems in diverse real-world scenarios rather than controlled laboratory settings. To transform this deluge of raw visual information into actionable insights, the team employs computer vision and deep learning methods. These advanced technologies are used to convert every single frame from the face, body, and forward-facing cameras into meaningful knowledge about driver behavior. By analyzing these frames, researchers can derive a profound understanding of what drivers are actually doing while interacting with autonomous systems. This process bridges the gap between raw pixel data and behavioral science, revealing how artificial intelligence can play a pivotal role in maintaining safety and providing an enjoyable driving experience. Ultimately, the study aims to leverage this converted knowledge to refine future automation technologies and ensure they align with human needs and capabilities. The ability to touch every frame of video allows for a granular analysis that was previously impossible, offering unprecedented clarity into how humans adapt to and utilize AI-driven vehicles. As the project continues to expand its dataset across multiple manufacturers, the findings will likely inform critical safety standards and user interface designs, ensuring that autonomous systems evolve in harmony with human drivers rather than replacing them entirely or creating unsafe gaps in trust and control.