AV Safety Assessments Require Analysis of Variable Complexity Across Driving Environments
Note: Woven Planet became Woven by Toyota on April 1, 2023. Note: This blog was originally published on January 31, 2020 by CARMERA, which is now part of Woven Planet. This post was updated on December 13, 2022.
By Ro Gupta, VP of Automated Mapping
This year, CES proved once again that it is the only truly global car show, and one of the best indicators of where autonomous technologies are in the hype cycle, Gartner’s now ubiquitous rubric for assessing the maturity of emerging technologies. At this year’s show, we saw an industry committed to laying the path toward the so-called “slope of enlightenment” — that all-important phase when new technologies begin to be truly understood and, ultimately, adopted.
Key paving stones in this path are education and safety, (perhaps why our partners at PAVE chose the acronym :). At CES — much like at last summer’s AVS Symposium and in recent moves coming out of Washington — we saw a sharper focus from all stakeholders on emerging safety norms and standards. In November, the National Transportation Safety Board recommended that AV companies be required to conduct safety self-assessments prior to testing any vehicles on the open road. This is an important step toward a pre- AND post-test safety verification standard but it’s incumbent upon both the industry and regulators to actually develop an actionable framework for these safety checks.
As previously noted, key tentpoles to a meaningful pre-test safety assessment will include evaluating simulation and closed-track testing performance; and binary, bright-line checks for core safeguards like HD maps, driver monitoring systems, and/or remote operation. But in addition to these practices, AV developers should consider reporting the risk and complexity specific to the roads they intend to drive. The narrow, pedestrian-ladened streets of San Francisco present different challenges than the sunbaked straightaways of Texas highways.
Most of the leading developers already evaluate complexity differences for internal purposes, and sharing some of that information publicly could help regulators better understand the risk variability across domains and help build a common lexicon for describing different driving environments.
Understanding Unique Road Complexities
A road isn’t a road isn’t a road. Each driving environment comes with its own unique challenges, and a careful examination of the specific road complexities is key to ensuring safe AV testing. Tackling the challenge of complexity can be broken down into four basic steps:
Recognizing the complexity problem. This is already well underway, with leading thinkers like Phil Koopman, chief technology officer at Edge Case Research and a professor at Carnegie Mellon, working on describing various Operational Design Domains (ODD) — a fancy term that essentially means “driving environments” — and with companies like Waymo and Uber ATG expanding their internal analysis to new cities like Los Angeles and Washington, DC.
Building or acquiring tools to help. Factors like pedestrian density, traffic patterns, weather conditions, construction frequency, and condition of the road all contribute to driving complexity. Some AV companies are already able to evaluate these factors, while others could create or outsource tools that can help — from bespoke road scans to third-party road intelligence data.
Sharing complexity information with regulators and peers. While companies must guard proprietary data, there’s a growing recognition that greater education and cooperation around safety are expected by both government and society. An instructive parallel is the airline industry, where safety information has long been shared among even the fiercest of rivals. At the end of the day, we need to remember that the reputational costs of a single accident are shared across the industry. As NTSB Chairman Robert Sumwalt noted, “Any company’s crash affects the public confidence. Anybody’s crash is everybody’s crash.”
Creating greater standardization. Shared safety learnings will ultimately coalesce around key principles, including those being developed for the ISO SOTIF and UL 4600 standards. Regulators, insurers, and industry bodies could codify these into a rubric, similar to the Society of Automotive Engineers’ levels of autonomy. Even if more informal, this knowledge accretion benefits everyone, giving AV leaders an opportunity to burnish their credibility, while providing other stakeholders an ODD yardstick they can easily understand.
As we enter the new decade, we appear closer to the “verification/trust sandwich” becoming a structural norm in safety protocols for AV. And as this approach becomes a reality, transparency around road environment and driving complexity will be integral to every step of that process.
Join us!
We’re looking for talented engineers to help us on this journey. If you’d like to join us, we’re hiring!
Thanks to Edge Case Research CEO, Michael Wagner; PAVE Executive Director, Tara Andringa Dunfee; and former Waymo CBO, Shaun Stewart, for reviewing drafts of this.