Arrow Electronics and NVIDIA have collaborated on a first-ever AI-based steering system that uses face detection and a Jetson AGX Orin ™ module to control a race car.

Arrow Electronics and NVIDIA: AI-based steering system

A new era begins

On a windy, winter day at a private desert racetrack, the Arrow SAM Car Project entered a new era. For the first time, quadriplegic racecar driver Sam Schmidt drove at competitive speeds using a unique AI-based steering system.

The steering system – which Schmidt controls through head motions – has been developed by Arrow engineers using the latest NVIDIA Jetson AGX Orin module.

The test on the two-mile road course at the Thermal Club near Palm Springs, Calif. displayed the SAM Car’s strength as a mobile engineering lab, as well as a demonstration platform for innovative technologies that can extend mobility – and opportunity – for the disabled community.

The successful test proved that Five Years Out, the head-tracking technology could have everyday applications far beyond the racetrack where speeds might be slower, but safety and reliability are paramount.

“The SAM Car has evolved into a technology platform that really personifies our noble purpose, which is to enable the benefit of technology for as many people as possible,” said Murdoch Fitzgerald, vice president of global engineering and design services at Arrow.


How it began

Launched a decade ago, the SAM Car Project modifies performance cars with electronic interfaces to allow Schmidt to drive at racetrack speeds using head movements and breath and voice commands.

Schmidt was a champion IndyCar Series driver who was paralyzed from the shoulders down in a racing crash in 2000. Arrow helped him realize his dream of racing again with demonstration laps at the 2014 Indy 500. Since then, he has completed dozens of high-speed demonstration drives including amateur races and rallies, the Pikes Peak International Hill Climb in Colorado, and the Goodwood Festival of Speed in England.

In 2016, the SAM car was approved for legal street driving and Schmidt received a restricted driver’s license. Since then, he has driven using head controls on public streets from San Francisco to New York City.

The SAM program prototypes new mobility technologies and continues to innovate and enhance new technologies to offer ever-higher levels of control and decision-making for people with disabilities.

“For me, an autonomous car is just a bigger, faster wheelchair,” Schmidt said. “The SAM technology enables me to remain as the driver, to make decisions about where to go and how to get there. Having more freedom and choice makes a positive difference not only in the car, but in many aspects of the life of a person with a physical disability.”


Steering into the future

The first-of-its kind steering system featured in the latest SAM Car – a 2021 McLaren 720S Spyder – was built using a Jetson AGX Orin module made by NVIDIA, an Arrow supplier. The supercharged Jetson module was developed to accelerate machine-learning models. It is the only platform capable of delivering the high accuracy and low latency needed for SAM’s steering system at speeds that often top 150 mph and, in some demonstrations, reach as high as 213 mph.

Other applications for the Jetson AGX Orin include controlling a new Mars scientific rover by NASA, 3-D vision for robotic and drone farming, and a smart lamp to remotely detect falls and other health emergencies in senior communities.

To create the facial recognition system, Arrow commissioned Hollywood artists to mold Schmidt’s head and create an animatronic model. Thousands of images of the model were captured at different angles.

A low-profile stereo camera pair located on the dash captures Schmidt’s movements in real time and feeds the data to the Jetson AGX Orin module. Using a chain of Convolutional Neural Networks, the system processes the information and adjusts the steering angle automatically – within a few thousandths of a second. The entire process is repeated 60 times per second.

If Schmidt keeps his head in the correct position and moves left and right smoothly, the SAM Car steers as normally as if he were gripping a steering wheel.

“With the AI, it’s more controllable,” remarked Schmidt. “There’s no lag. It’s instantaneous for high speed.”

The new AI steering replaces the original steering modifications which used an older, infrared motion-capture based technology and featured as many as four infrared cameras. Those cameras required the driver to wear special reflective markers on a racing helmet or a pair of sunglasses, which were captured by the cameras for the system to measure. The new AI-based system eliminates the need for these markers and reduces the number of cameras required.

Layering capabilities

Known for producing high-powered processors for AI data centers, NVIDIA proved to be the perfect collaborator to support the SAM Car’s transition to machine learning. The company’s Jetson platform is flexible and user-friendly, allowing Arrow to integrate the suppliers’ components into the system – again enabling the SAM CAR to serve as a demonstration model for new technologies.

“We provide a fantastic platform with Jetson to be able to build a very reliable system with high performance, but Arrow really layered on the capability to make this solution happen,” said Geoff Fancher, head of worldwide distribution at NVIDIA.

While Arrow customized the Jetson platform for the SAM Car, the technology could prove useful in a variety of hands-free mobility applications.

For example, a farmer could use the technology to plow fields, while also monitoring soil conditions on a tablet. Likewise, warehouse workers could operate forklifts and perform inventory at the same time.

For people with disabilities, the technology could provide enhanced mobility – wheelchairs, for example, could be easier to control thanks to head-tracking-based guidance systems and hazard avoidance.

“We’re really hoping this technology becomes available to the general public so that people with disabilities can have technology enhance whatever their functional impairment is,” said Fancher.