SF Standard – WTF happened to Waymo during the blackout? Industry experts have some guesses
The power outage left robotaxis frozen in intersections. City leaders want answers.
Editors note: contrary to Waymo’s usual claims, this serious problem cannot be fixed with a software update. The problem was that Waymos are not truly autonomous. When they encounter an unusual situation, like traffic signals being out, then they need the help of an actual human. But this help is provided via communication over the 5G cellular system. When the power goes out, then eventually the cellular system goes out because the backup batteries that power the cell phone towers run out of juice. This is exactly what happened on December 21st. No magic software can fix Waymos depending on humans to deal with unusual situations.
When the next major earthquake occurs in San Francisco, and it will, then this Waymo problem will happen again and emergency responders will be blocked.
See original article by Sam Mondros and Max Harrison-Caldwell at SF Standard
It was the night the Waymos went haywire.
A power outage that began Saturday afternoon at a PG&E substation threw the city into darkened chaos, the mayhem compounded by the behavior of Waymo’s fleet of hundreds of robotaxis. The autonomous vehicles stalled in intersections, blocked arterial roads, and, according to at least one city official, impeded emergency vehicles. Amid the tumult of the biggest blackout in decades, robotaxis apparently added to the problems.
On Sunday at 7:00 p.m., Waymo announced it would resume its robotaxi service. But now comes the fallout. On Monday, District 5 Supervisor Bilal Mahmood called for a hearing into Waymo’s meltdown, and experts began to weigh in on what, exactly, went wrong. Waymo, meanwhile, has shared few details.
The first 311 call about a stalled Waymo came moments after the blackout started at 2:30 p.m. A caller filed a complaint about an immobile robotaxi at Divisadero Street and Geary Boulevard, where the traffic light had gone dark, just 15 minutes after the San Francisco Fire Department was called to a fire at a PG&E substation.
The 311 complaints flooded in into Saturday evening. Callers filed 20 reports about Waymos impeding traffic between 4:30 p.m. and just after 10 p.m. The map of reported locations mirrors PG&E’s map of the outage.
“I was stuck for almost 15 minutes [waiting] to cross one intersection because Waymo blocked the bus,” one call transcript(opens in new tab) says. “The entire lane could not move because the Waymo decided to block it and refused to let the bus through.”
In one instance, according to Mahmood, paralyzed Waymos hindered first responders’ ability to address two fires, at the substation and in Chinatown. An SFFD spokesperson could not confirm the incidents but acknowledged that the department had dealt with calls related to autonomous vehicles throughout the day.
Mahmood said the outage led to a “compounding loop of unfortunate incidents” that brought gridlock to the city’s streets. Fires caused outages, which hampered autonomous vehicles, which hampered first responders.
“The largest thing this points to with PG&E and Waymo is that this created a disruption that neither company was able to handle with their protocols,” Mahmood said. “We’re calling for a hearing to understand what went wrong technically and operationally and what changes they’re going to make to make sure this doesn’t happen again.”
Waymo said Sunday that its robotaxis are “designed to treat nonfunctional signals as four-way stops,” noting that “the sheer scale of the outage led to instances where vehicles remained stationary longer than usual to confirm the state of the affected intersections.” The company said “the majority of active trips were successfully completed before vehicles were safely returned to depots or pulled over.”
Transit experts suggest that the company’s systems were unprepared — or not programmed — for a failure of this magnitude, in which city signals, electricity, and cellular networks were all compromised at once.
There are 800 to 1,000 Waymo vehicles in operation in San Francisco, more than any other city. This makes users in the city de facto beta-testers for the technology and the first to feel any failures. (Zoox, the Amazon subsidiary that has begun operating robotaxis in San Francisco, did not respond to requests for comment about whether it had service problems as a result of the blackout.)
AV vehicles, like aircraft and other safety-critical systems, are programmed with a “fail-safe” mode: instructions on how to shut down in a system failure in ways that do not compromise human safety. It appears that many Waymo vehicles, when confronted with the chaos of failed traffic signals and aggressive drivers, simply stopped in place.
“A bad outcome is if the vehicles continue to drive aimlessly and cause damage,” said Spencer Penn, a former product manager of simulation at Waymo. “If anything, these vehicles took the safe option and pulled over — it could have been a conscious decision to turn the Waymos off, or it could have been by design. There are always different levels of fail-safe.”
However, many robotaxis didn’t pull over but stopped mid-traffic. Sources from robotics and AV companies speculated that issues with cellular service during the outage could have hindered Waymo’s remote response system, which allows human engineers to observe problems and intervene. Without cell service, this fallback feature might not have been available.
A Waymo spokesperson seemed to confirm this theory Monday, saying the cars can request information from a remote assistance team.
“In limited instances on Saturday night, the [robotaxi] Driver sought additional context,” the spokesperson wrote in an email. (The company refers to its cars’ autonomous driving systems as “Drivers.”) “However, the unprecedented scale of the power outage and its impact on local connectivity resulted in some delays to those confirmations.”
The cascading effects were predictable to robotics experts. Rodney Brooks, founder of iRobot, said he imagined Waymos were more likely to stack up and create serious problems at the busiest intersections. Indeed, many of the 311 calls were about intersections on Fell Street, Oak Street, and Market Street.
“If there is just one Waymo, stalled drivers would go around it, but once a few are stacked up, that looks dangerous, and people decide to call 311,” he said. Brooks, a professor emeritus of robotics at the Massachusetts Institute of Technology, is based in San Francisco.
“It’s true with any technology,” he said. “You think you have a bunch of problems solved, and then there’s an edge case — something you hadn’t considered.”
“Edge cases” are tech-speak for uncommon situations that an automated system must be programmed to navigate.
“Somehow, they didn’t consider this case of massive power failure,” Brooks said. “It would be really bad if we had an earthquake and emergency vehicles were trying to get through. One would hope it’s going to get fixed.”
Waymo said Sunday it is “focused on rapidly integrating the lessons learned from this event.” This may be the one silver lining of the weekend’s fiasco: The company has millions of new data points that should become grist for the next software update, presumably making it better prepared for the next blackout, fire, or earthquake — all of which have happened in San Francisco in recent decades.
On Monday, city leaders called for more accountability from the company — which may have removed its cars from city streets Saturday in response to requests from Mayor Daniel Lurie, rather than of its own initiative.
Lurie stated Monday that he called a Waymo CEO on Saturday asking to have the robotaxis removed immediately, after he received reports that they were exacerbating traffic and obstructing emergency vehicles.
“What we need from them is them to be in better touch with us and have a direct line of communication with us during events like that,” Lurie said.
Mahmood said he plans to call for a hearing at the Jan. 6 meeting of the Board of Supervisors.
Meanwhile, Waymo’s roadmap for 2026 includes expanding to more freeways, airports, and cities in California and other western states.
See original article by Sam Mondros and Max Harrison-Caldwell at SF Standard
