A Brief History of Automated Driving — Part Three: Toward Product Development
Updated: Oct 5, 2020
I have been working on autonomous vehicles and driver assistance systems for 23 years. During this time, I have had several touchpoints with legal and regulatory aspects and with functional safety certification. I helped write the SAE levels of automated driving, and I cover these topics in my Stanford class as well. This post is an attempt to summarize my insights into a concise summary. Comments, additions, input for missing pieces, and suggestions are welcome. Contact email@example.com. Thanks.
From Driver Assistance to Robocars—Evolution or Revolution?
Driver assistance systems evolved from simple, smart distance keeping features to sophisticated assistance systems over the past twenty years:
Adaptive Cruise Control (ACC) was launched as a single driver assistance feature in luxury-class vehicles in the late 1990s. ACC back then was typically based on a single radar sensor, initially specifically designed for this functionality, to maintain a constant time gap to the preceding vehicle while not exceeding a preset maximum velocity.
A couple of years later, Forward Collision Warning (FCW) was added, still integrated in the radar, still driven by only a single radar sensor. Now for the first time, two functions were driven by a single sensor.
In the 2000s, Lane Departure Warning (LDW) systems were introduced. These are typically based on cameras (Citroen had a system based on infrared sensors), which detect the white lane markers in the camera image. Over the years, these evolved from warning systems to systems that applied a certain limited amount of torque to keep the vehicle in the lane, then called Lane Keeping Support (LKS) or Lane Keeping Assist or Lane Centering Support.
In 2013, Mercedes launched for the first time a system in which longitudinal and lateral control were integrated. This was the first level 2 system ever introduced into a production vehicle. It was called "Distronic Plus with Steering Assist and Stop&Go Pilot." Tesla launched a system with similar functionality in 2015 and gave it the much leaner—though controversial—name Autopilot. Many carmakers have introduced level 2 systems since then.
Audi developed the first level 3 system for launch in 2017. The system was an extension of their level 2 driver assistance system and was perceived by customers to provide very little extra value as it was usable in certain environments only. It did not become available though and the manufacturer claimed legal restrictions in Germany as the reason.
In parallel, advances in machine learning in the past ten years has led to a major breakthrough in computer vision, which has provided the field of vehicle automation with the ability to significantly better understand street scenes using cameras. This led to significant advances in the development of level 4 systems a couple of years ago and resulted in the over-optimistic expectation voiced by many executives, investors, and media that level 4 vehicles (a.k.a. robocars) are practically ready to be deployed and that by today we'd have fleets of robocars running on the streets.
Commercially, robocars will go hand in hand with a revolution of mobility business models. Personally-owned vehicles are purchased with the expectation that vehicles will be able to be driven by their owner to—more or less—any reasonably reachable destination. That creates the buyer expectation that—if those vehicles were highly automated—the vehicle automation system in such vehicles would be able to operate on pretty much any drivable road. While level 4 systems have made an incredible amount of progress in the past decade (as outlined in this blog post series), it's a long shot to a level 5 vehicle—defined as a level 4 vehicle that works in all driving modes, i.e., on all roads under all reasonable conditions.
For (most) individuals owning a level 1, 2, or 5 vehicles may make sense. Level 3 vehicles may not make sense at all for passenger vehicles (I'll come to that later) and level 4 vehicles—defined as (potentially) driverless vehicles, capable of operating within a certain Operational Design Domain (ODD) only—make a lot of sense for fleet operators (and the likes of an Uber or Lyft), but not so much for an individual. Who would want to own a robocar that only works in suburban Silicon Valley, but not all the way up to San Francisco or down to LA? That also puts today's carmakers in an interesting position. Will they offer transportation services? Or do they partner with the rideshare operators? Do they rely on technology from Argo or Waymo, or partner with a Tier 1 supplier or with an ecosystem of start-ups, or better build their own technology including ecosystem from scratch? We just saw in June that building a vehicle plus automation plus ecosystem is too much even for a well-funded start-up. On the other hand, automakers today are selling millions of vehicles with driver assistance. How many money-making vehicles do Argo and Waymo operate today?
A Deeper Dive into the History of Driver Assistance
Vehicle automation was first introduced into series production vehicles in the late 1990s through driver assistance systems, now also known as Level 1 systems.
Adaptive Cruise Control (ACC) is an extension of Cruise Control to maintain a set distance to preceding vehicles using most commonly a millimeter-wave FMCW (frequency-modulated-continuous-wave) radar sensor. Lidar sensors were also used in some very early ACC systems in Japan but soon replaced by radars. The radar directly measures distance and relative velocity of all reflecting objects and indirectly estimates their angular offset. Objects irrelevant for ACC are ignored, and the target object, which is typically a preceding vehicle in the same lane of traffic, is selected. A controller then adjusts the relative velocity to the preceding vehicle such that a certain time gap is kept constant.
Forward Collision Warning (FCW) introduced a few years later, is based on the same sensor and makes drivers aware of impending collisions by means of escalating warning levels. Forward collision prevention systems for low-speed applications using low-range and low-resolution inexpensive lidar sensors entered the market around 2010, e.g., Volvo City Safety.
Lateral guidance systems using cameras and computer vision algorithms to detect lane markers in the camera image were introduced into production vehicles about ten years ago with systems such as Lane Departure Warning (LDW) or Lane Keeping Support (LKS), also called Lane Centering.
In 2013, Daimler released the first production passenger vehicle with an SAE Level 2 (a.k.a. Partial Automation) assistance system. Level 2 systems still require constant supervision by the driver who may need to take over control of the vehicle more or less instantaneously. This combines longitudinal distance keeping with lateral lane-centering and was launched under the somewhat complicated name "Distronic Plus with Steering Assist." Shortly after, other automakers released similar level 2 systems, e.g., Tesla with their Autopilot system; however, all of them still require constant supervision by the driver. In 2020, Consumer Reports ranked GM's Super Cruise system as the most advanced driver assistance system available.
Audi announced that the 2017 Audi A8 would be the first production automobile to have been developed especially for conditional automated driving (SAE level 3). The system never became available in the US or Europe, where the manufacturer cited legal restrictions as the reason. Recently Audi announced that the system will be discontinued in the 2021 mid-cycle refresh.
Toward Automated Driving Product Development
The DARPA Urban Challenge—as described in more detail in the previous post—marked the transition from academic research to self-driving product development through three distinct outcomes:
1. Industry attention. The DARPA prize money attracted top-notch researchers, who attracted leading automotive manufacturers (Volkswagen with Stanford, GM with CMU), large automotive suppliers (Continental and Mobileye both with CMU, Bosch with Stanford), chipmakers (Intel with both top teams to be on the safe side, NXP with Stanford), and Google (also with both top teams) as sponsors. Images: Stanford Racing Team and Tartan Racing.
2. High-resolution automotive lidar. Researchers used simple industrial lidar sensors up to the 2005 Grand Challenge (Omron had built a simple automotive lidar in the late 1990s for Adaptive Cruise Control systems, but ultimately gave that up in favor of much more cost-efficient radar). Stanford mounted five separate single-beam lidar sensors at different vertical angles to achieve some vertical resolution mainly to compensate the pitch movement of the vehicle. The Hall brothers decided to build their own 64-beam high-resolution lidar sensor for