Autonomous vehicle reporting data is driving AV innovation right off the road

At the end of every calendar year, the complaints from autonomous vehicle companies start piling up. This annual tradition is the result of a requirement by the California Department of Motor Vehicles that AV companies deliver “disengagement reports” by January 1 of each year showing the number of times an AV operator had to disengage the vehicle’s autonomous driving function while testing the vehicle.

However, all disengagement reports have one thing in common: their usefulness is ubiquitously criticized by those who have to submit them. The CEO and founder of a San Francisco-based self-driving car company publicly stated that disengagement reporting is “woefully inadequate … to give a meaningful signal about whether an AV is ready for commercial deployment.” The CEO of a self-driving technology startup called the metrics “misguided.” Waymo stated in a tweet that the metric “does not provide relevant insights” into its self-driving technology or “distinguish its performance from others in the self-driving space.”

Why do AV companies object so strongly to California’s disengagement reports? They argue the metric is misleading based on lack of context due to the AV companies’ varied testing strategies. I would argue that a lack of guidance regarding the language used to describe the disengagements also makes the data misleading. Furthermore, the metric incentivizes testing in less difficult circumstances and favors real-world testing over more insightful virtual testing.

Understanding California reporting metrics

To test an autonomous vehicle on public roads in California, an AV company must obtain an AV Testing Permit. As of June 22, 2020, there were 66 Autonomous Vehicle Testing Permit holders in California and 36 of those companies reported autonomous vehicle testing in California in 2019. Only five of those companies have permits to transport passengers.

To operate on California public roads, each permitted company must report any collision that results in property damage, bodily injury, or death within 10 days of the incident.

There have been 24 autonomous vehicle collision reports in 2020 thus far. However, though the majority of those incidents occurred in autonomous mode, accidents were almost exclusively the result of the autonomous vehicle being rear-ended. In California, rear-end collisions are almost always deemed the fault of the rear-ending driver.

The usefulness of collision data is evident — consumers and regulators are most concerned with the safety of autonomous vehicles for pedestrians and passengers. If an AV company reports even one accident resulting in substantial damage to the vehicle or harm to a pedestrian or passenger while the vehicle operates in autonomous mode, the implications and repercussions for the company (and potentially the entire AV industry) are substantial.

However, the usefulness of disengagement reporting data is much more questionable. The California DMV requires AV operators to report the number and details of disengagements while testing on California public roads by January 1 of each year. The DMV defines this as “how often their vehicles disengaged from autonomous mode during tests (whether because of technical failure or situations requiring the test driver/operator to take manual control of the vehicle to operate safely).”

Operators must also track how often their vehicles disengaged from autonomous mode, and whether that disengagement was the result of software malfunction, human error, or at the option of the vehicle operator.

AV companies have kept a tight lid on measurable metrics, often only sharing limited footage of demonstrations performed under controlled settings and very little data, if any. Some companies have shared the occasional “annual safety report,” which reads more like a promotional deck than a source of data on AV performance. Furthermore, there are almost no reporting requirements for companies doing public testing in any other state. California’s disengagement reports are the exception.

This AV information desert means that disengagement reporting in California has often been treated as our only source of information on AVs. The public is forced to judge AV readiness and relative performance based on this disengagement data, which is incomplete at best and misleading at worst.

Disengagement reporting data offers no context

Most AV companies claim that disengagement reporting data is a poor metric for judging advancement in the AV industry due to a lack of context for the numbers: knowing where those miles were driven and the purpose of those trips is essential to understanding the data in disengagement reports.

Some in the AV industry have complained that miles driven in sparsely populated areas with arid climates and few intersections are miles dissimilar from miles driven in a city like San Francisco, Pittsburgh, or Atlanta. As a result, the number of disengagements reported by companies that test in the former versus the latter geography are incomparable.

It’s also important to understand that disengagement reporting requirements influence AV companies’ decisions on where and how to test. A test that requires substantial disengagements, even while safe, would be discouraged, as it would make the company look less ready for commercial deployment than its competitors. In reality, such testing may result in the most commercially ready vehicle. Indeed, some in the AV industry have accused competitors of manipulating disengagement reporting metrics by easing the difficulty of miles driven over time to look like real progress.

Furthermore, while data can look particularly good when manipulated by easy drives and clear roads, data can look particularly bad when it’s being used strategically to improve AV software.

Let’s consider an example provided by Jack Stewart, a reporter for NPR’s Marketplace covering transportation:

“Say a company rolls out a brand-new build of their software, and they’re testing that in California because it’s near their headquarters. That software could be extra buggy at the beginning, and you could see a bunch of disengagements, but that same company could be running a commercial service somewhere like Arizona, where they don’t have to collect these reports.

That service could be running super smoothly. You don’t really get a picture of a company’s overall performance just by looking at this one really tight little metric. It was a nice idea of California some years ago to start collecting some information, but it’s not really doing what it was originally intended to do nowadays.”

Disengagement reports lack prescriptive language

The disengagement reports are also misleading due to a lack of guidance and uniformity in the language used to describe the disengagements. For example, while AV companies used a variety of language, “perception discrepancies” was the most common term used to describe the reason for a disengagement — however, it’s not clear that the term “perception discrepancies” has a set meaning.

Several operators used the phrase “perception discrepancy” to describe a failure to detect an object correctly. Valeo North America described a similar error as “false detection of object.” Toyota Research Institute almost exclusively described their disengagements vaguely as “Safety Driver proactive disengagement,” the meaning of which is “any kind of disengagement.” Whereas, Pony.ai described each instance of disengagement with particularity.

Many other operators reported disengagements that were “planned testing disengagements” or that were described with such insufficient particularity as to be virtually meaningless.

For example, “planned disengagements” could mean the testing of intentionally created malfunctions, or it could simply mean the software is so nascent and unsophisticated that the company expected the disengagement. Similarly, “perception discrepancy” could mean anything from precautionary disengagements to disengagements due to extremely hazardous software malfunctions. “Perception discrepancy,” “planned disengagement” or any number of other vague descriptions of disengagements make comparisons across AV operators virtually impossible.

So, for example, while it appears that a San Francisco-based AV company’s disengagements were exclusively precautionary, the lack of guidance on how to describe disengagements and the many vague descriptions provided by AV companies have cast a shadow over disengagement descriptions, calling them all into question.

Regulations discourage virtual testing

Today, the software of AV companies is the real product. The hardware and physical components — lidar, sensors, etc. — of AV vehicles have become so uniform, they’re practically off-the-shelf. The real component that is being tested is software. It’s well known that software bugs are best found by running the software as often as possible; road testing simply can’t reach the sheer numbers necessary to find all the bugs. What can reach those numbers is virtual testing.

However, the regulations discourage virtual testing as the lower reported road miles would seem to imply that a company is not road-ready.

Jack Stewart of NPR’s Marketplace expressed a similar point of view:

“There are things that can be relatively bought off the shelf and, more so these days, there are just a few companies that you can go to and pick up the hardware that you need. It’s the software, and it’s how many miles that software has driven both in simulation and on the real roads without any incident.”

So, where can we find the real data we need to compare AV companies? One company runs over 30,000 instances daily through its end-to-end, three-dimensional simulation environment. Another company runs millions of off-road tests a day through its internal simulation tool, running driving models that include scenarios that it can’t test on roads involving pedestrians, lane merging, and parked cars. Waymo drives 20 million miles a day in its Carcraft simulation platform — the equivalent of over 100 years of real-world driving on public roads.

One CEO estimated that a single virtual mile can be just as insightful as 1,000 miles collected on the open road.

Jonathan Karmel, Waymo’s product lead for simulation and automation, similarly explained that Carcraft provides “the most interesting miles and useful information.”

Where we go from here

Clearly there are issues with disengagement reports — both in relying on the data therein and in the negative incentives they create for AV companies. However, there are voluntary steps that the AV industry can take to combat some of these issues:

  1. Prioritize and invest in virtual testing. Developing and operating a robust system of virtual testing may present a high expense to AV companies, but it also presents the opportunity to dramatically shorten the pathway to commercial deployment through the ability to test more complex, higher risk, and higher number scenarios.
  2. Share data from virtual testing. Voluntary disclosure of virtual testing data will reduce reliance on disengagement reports by the public. Commercial readiness will be pointless unless AV companies have provided the public with reliable data on AV readiness for a sustained period.
  3. Seek the greatest value from on-road miles. AV companies should continue using on-road testing in California, but they should use those miles to fill in the gaps from virtual testing. They should seek the greatest value possible out of those slower miles, accept the higher percentage of disengagements they will be required to report, and when reporting on those miles, describe their context in particularity.

With these steps, AV companies can lessen the pain of California’s disengagement reporting data and advance more quickly to an AV-ready future.

Boston Dynamics CEO Rob Playter is coming to Disrupt 2020 to talk robotics and automation

Back in January, Robert Playter became the CEO of Boston Dynamics. It was a momentous occasion, marking the company’s first new CEO since its founding in the early 1990s when the company was founded by Marc Raibert. The move came during what was already a transitional period for the company which is why we are excited to chat with him at Disrupt 2020.

Following its sale to Softbank, Boston Dynamics had recently begun early sales of Spot, its first commercial product. In April of last year, the company made its own acquisition, picking up Bay Area-based Kinema Systems to help design a visioning system for its own warehouse robotics like Handle.

Of course, much of this pre-dates the current COVID-19 pandemic, which has made automation and robotics an even more hot button issue than it has been in the years prior. Over the course of the last few months, Spot has been seen employed in a factotum of different jobs, as everyone from construction companies to health care facilities to baseball teams look to the quadrupedal robot for help.

Playter will be making his first public speaking engagement as CEO at our first online-only Disrupt this October. His appearance comes after several from Boston Dynamics founder (and Playter’s predecessor as CEO) Marc Raibert. Most recently, Raibert made a return appearance at our TC Sessions: Robotics event last April to show off the commercial version of Spot.

He will join us to discuss the challenges and opportunities in transforming Boston Dynamics into commercial venture at Disrupt 2020 on September 14-18. Get a front-row seat with your Digital Pro Pass for just $245 or with a Digital Startup Alley Exhibitor Package. Prices increase on Friday, so grab your tickets now!

What’s ahead for no-code and low-code startups?

Since The Exchange last checked in on the world of low- and no-code startup funding, several more interesting rounds in the niche have bubbled up.

This week, TechCrunch covered a startup called Hevo raising $8 million, and Paragon, which raised a $2.5 million seed round. Hevo is a “data pipeline startup” that helps “clients’ employees to integrate data from more than 150 different sources — including enterprise software from Salesforce and Oracle  without requiring a technical background, we reported.

Paragon, part of Y Combinator’s Winter 2020 batch, is a developer productivity-focused service that “makes it easier for non-technical people to be able to build out integrations using our visual workflow editor” according to its co-founder Brandon Foo. Paragon wants to “bring the benefits of low code to product and engineering teams and make it easier to build products without writing manual code for every single integration” to help “streamline the product development process,” Foo added.

And there are more rounds worth highlighting in the space since we last looked, like $4 million for Enduvo (no-code AR/VR), a $3.45 million extension for the fast-growing Turbo Systems (a no-code “engagement platform”), and a seed round for CloudWorx (no-code IoT), among others.

The trend that we noted last week that no-code and low-code startups are raising lots of capital is still hot.

But startups aren’t the only companies working in this space: Apple has long had a foot in the domain via its subsidiary Claris, which rebranded to that name last year after running under the FileMaker moniker. At the time, Claris CEO Brad Freitag told TechCrunch that his company’s vision was to make “powerful technology accessible to everyone.”

That wasn’t merely cliché: Claris’ best-known product, FileMaker, helps users build low-code apps, and its second product is called Connect, a service that helps users link APIs using low-code tooling.

Given that Claris has been in the no-code, low-code space for longer than most, TechCrunch caught up with Freitag again to chat about recent growth in the market category, what he thinks of the low-code terminology, and, of course, his take on startups in the niche.

The growth of no-code and low-code

NASA successfully launches its Mars 2020 Perseverance rover using an Atlas V rocket

NASA has launched one of its most crucial science missions to date, the Mars 2020 mission that carries its Perseverance robotic rover. This rover, a successor to the Curiosity robotic explorer, is equipped with sensors specifically designed to help it hopefully fund evidence of ancient, microbiotic life on Mars.

Mars 2020 departed from Cape Canaveral in Florida at 7:50 AM EDT (4:50 PM PDT). Perseverance was loaded atop a United Launch Alliance (ULA) Atlas V rocket, which had a good liftoff and deployed its second stage which put the spacecraft into a parking orbit as it readies to depart on its trip towards Mars, which will see it arrive in February 2021.

Once at Mars, the lander vehicle will take Perseverance down to the planet’s surface on February 18, 2021, to a target landing zone found in what’s known as Jezero Crater. This location on Mars was once a lake, long ago when the atmosphere on Mars was quite different than the dry, dusty and cold environment we know today. This has been chosen specifically because it’s a prime spot for finding any evidence of microbiological life that might exist, since it contains one of the best-preserved deposits of a river delta on Mars.

NASA scientists don’t expect to be able to confirm the existence of life on Mars using the instruments on Perseverance, however – they think they can find strong indications that the conditions and materials necessary for life once existed, but the ultimate proof could come from the ambitious Mars sample return mission being planned for 2026. This would involve NASA launching a return rocket to the red planet, which will carry a rocket that can take off from the Mars surface with samples collected by Perseverance on board. That would then meet up with a rover to be launched by the European Space Agency (ESA) which would then make the trip all the way back to Earth for scientists to study.

In addition to its contained, radioactive nuclear battery power source, environment sensors, cameras and a suite of other instruments to help pick up any preserved evidence of ancient life, Perseverance is equipped with microphones. This is the first time that microphones are making the trip to the surface of another world, and it means we could hear what it sounds like on the surface of another world, something we’ve never done before.

Perseverance also carries the Mars Ingenuity helicopter, a small drone designed for first-ever self-powered flight, which is also designed to warm itself to survive the cold Martian night. It is set to hopefully make up to five flights in 30 days, could include color photos – the first ever taken from an aerial vantage point.

This is a great first step for this historic Mars 2020 mission, and now we’ll wait and watch for other significant milestones, including next in around two weeks when the spacecraft fires its engines for its departure from Earth’s orbit and begins the long trip to Mars.

Petit Qoobo cat pillow set for December US release, following crowdfunding campaign

A lot has been written about increased interest around automation and robotics during the pandemic — I know because I’ve done a lot of the writing. Most of these discussions tend to revolve around things like logistics, delivery and food preparation. But there’s also a compelling case to be made for companionship. And for that reason, the Petit Qoobo crowdfunding campaign couldn’t have launched at a better time.

The smaller, more portable of the quirky cat pillow goes up on Indiegogo in the U.S. tomorrow, following a successful campaign in its native Japan that netted the project $125,000. This time out, Yukai Engineering is hoping to net an additional $50,000. Prices will range from $60 to $80, depending on tiers and all of the good stuff. It’s expected to start shipping in December, making it a pretty solid holiday gift for allergy-prone animal lovers.

“The crowdfunding success in Japan really goes to show how people are increasingly turning to robots for emotional comfort and how it’s becoming ‘normal’ to ‘adopt’ robots into their lives,” CEO Shunsuke Aoki said in a release. “People also seem to be embracing this new human-robot relationship more while in self-quarantine.”

The latest version of the robot cat pillow includes the standard wagging tail, which speeds us you pet it. It will also start wagging in response to voices and sound and offer a subtle heartbeat sensation. Robotic companions have, of course, been a phenomenon in Japan for some time, owing in part to an aging population. The long stretches of isolation brought up by social distancing could certainly prove to be another key driver in their adoption.

According to the company, sales of the original Qoobo are up up ~30-40% versus the same two-month period last year. Qoobo has been around for a few years now, so it seems entirely likely that much of that renewed interest has been driven by the intense sense of isolation social distancing can engender.

Ford to use Boston Dynamics’ dog-like robots to map their manufacturing facilities

Ford is going to employ two of Boston Dynamics’ ‘Spot’ robots, which are four-legged, dog-like walking robots that weigh roughly 70 lbs each, to help them update the original engineering plans for one of the transmission manufacturing plans. The plants, Ford explains, have undergone any number of changes since their original construction, and it’s difficult to know if the plans they have match up with the reality of the plants as they exist today. The Spot robots, with their laser scanning and imaging capabilities, will be able to produce highly-detailed and accurate maps that Ford engineers can then use to modernize and retool the facility.

There are a few benefits that Ford hopes to realize by employing the Spot robots in place of humans to map the facility: First, they should save a considerable amount of time, since they replace a time-intensive process of setting up a tripod with a laser scanner at various points throughout the facility and spending a while at each location manually capturing the environment. The Spot dogs are roving and scanning continuously, providing a reduction of up to 50% in terms of actual time to complete the facility scan.

The robot dogs are also equipped with five cameras as well as laser scanners, and can operate for up to two hours travelling at around 3 mph continuously. The data they collect can then be synthesized for a more complete overall picture, and because of their small size and nimble navigation capabilities, they can map areas of the plant that aren’t necessarily reachable by people attempting to do the same job.

This is a pilot program that Ford is conducting, using two Spot robots leased by Boston Dynamics . But if it works out the way they seem to think it will, you can imagine that the automaker might seek to expand the program to cover other efforts at more of its manufacturing facilities.

Dexterity exits stealth with $56.2M raised for its collaborative warehouse robots

Dexterity emerged from stealth this week to announced its full-stack solution aimed at creating collaborative robotics systems. The hardware-software system is designed for a variety of different tasks, including bin picking and box packing, targeted at warehouse fulfillment and logistics needs.

Image Credits: Dexterity

The Bay Area-based startup has already built up significant support from the investment world, with $56.2 million raised to date, from a long list of backers, including Kleiner Perkins, Lightspeed Venture Partners, Obvious Ventures, Pacific West Bank, B37 Ventures, Presidio (Sumitomo) Ventures, Blackhorn Ventures, Liquid 2 Ventures and Stanford StartX.

Image Credits: Dexterity

The company was founded back in 2017 as an extension of CEO Samir Menon’s Stanford thesis, described by Dexterity thusly, “Menon worked on a control theory framework to describe how the human brain controls and coordinates the body, which serves as a model to distill human skill into mathematical programs that control robots in a graceful human-like manner.”

Part of the company’s appeal appear to be the versatility of the robotics, which are designed to work alongside their human counterparts and operate collaboratively. Among the early adopters for the system are unnamed an unnamed “global food manufacturer,” “ a worldwide package delivery provider” and Japan’s Kawasaki Heavy Industries.

Dexterity says it’s also seen a boost from the push for essential services during the COVID-19 pandemic, like so many others in the robotics and automation fields, stating that its systems have been involved with the shipping of “half a million units of packaged food.”

Amazon’s Scout robot deliveries expand to additional cities in Georgia and Tennessee

One thing’s for certain about Amazon’s Scout robot: it’s as much of a brand ambassador as it is an experiment in the future of last mile delivery. After debuting early last year, the company has limited to Scout to select markets — namely Irvine, California and Snohomish County, which neighbors King County, home of Amazon’s corporate HQ. Among other things, the robot is a kind of six-wheeled rolling billboard for Amazon’s services.

It also, according to Amazon, has been a useful tool as the company’s essential worker status has allowed it to maintain operations during the COVID-19 shutdown. Accompanied by human “Scout Ambassadors,” the cooler-sized robots have also continued to work throughout the pandemic.

And starting this week, Scout will expand operations to two cities in the American Southeast: Atlanta, Georgia and Franklin, Tennessee. The first you no doubt know. The second is significantly smaller, with a population of around 80,000, situated directly south of Nashville. In both cases, Scout’s deliveries will continue to be fairly modest, targeting “select customers” in those cities.

www.erinleeallender.com

Amazon — like all companies in the robotic delivery game — need to be deliberate in their expansion plans. Some larger cities, including New York and San Francisco, have been less than welcoming to the concept, owing to already-crowded urban sidewalks. Amazon’s expansion has largely been targeted on more residential communities, though Atlanta is certainly an exception as the company determines how Scout manages different terrain.

The company is also quick to allay safety concerns in the announcement, noting, “Amazon Scout delivery devices are built to be inherently safe. They’re the size of a small cooler and move at a walking pace. Each delivery device can navigate around pets, pedestrians, and other objects (including surfboards!) in its path.” Likely many councils in cities with higher populations will continue to approach the topic with caution

Amazon is also using Scout to help push additional community outreach including plans to support robotics and STEM activities in the new cities.

From farm to phone: A paradigm shift in grocery

In the blink of an eye, millennials, moms and grandparents alike have abandoned the decades-old practice of wandering dusty grocery aisles for the convenient and novel use of online grocery. While Instacart, Amazon Fresh and others have been offering an alternative to brick-and-mortar grocery for years, it is the pandemic that has classified them as essential businesses and more than ever afforded them a clear competitive advantage.

But these past couple months have seen not only drastic changes in consumer behavior, but also fundamental shifts in the business models adopted by grocers worldwide. These shifts are not temporary — indeed, they are here to stay, corona-catalyzed and permanent.

Fulfillment innovation can drive efficiency and cost savings

For the consumer, online grocery generally starts and ends the same way: They place their order on an app or website, and hours later it shows up at their door. But the ways those orders are being fulfilled run the gamut.

The most widely known approach comes from Instacart, which relies on hundreds of thousands of human shoppers fulfilling customers’ online grocery orders by shopping side-by-side with regular brick-and-mortar customers. The model clearly works for Instacart, which is valued at nearly $14 billion after its latest raise.

However, this model is far from ideal. Even pre-COVID, shoppers were known to crowd out regular customers, not to mention introduce high delivery costs and the element of human error to the fulfillment process.

One obvious solution has become the central fulfillment center, or CFC. CFCs are large, standalone warehouses — often serving distinct geographies — that can supply both brick-and-mortar stores and online grocery deliveries. As order volumes rise and consumers demand faster and faster delivery times, innovation has already been infused into the CFC model.

Some grocers, notably Kroger, believe that introducing robotic automation into CFCs via solutions such as Ocado can create economies of scale for fulfillment. These CFCs deploy fulfillment robots, controlled by air-traffic control tech, that run along a grid system and move goods via categorized crates. Kroger is continuing its investment in the model, recently announcing three new Ocado-automated CFCs in the West, Pacific Northwest and Great Lakes regions of the United States. The smallest location is over 150,000 square feet.

While Kroger remains uniquely attached to the CFC model, Albertsons/Safeway, Walmart and many others prefer the microfulfillment center (MFC). MFCs, typically far smaller in size (think ~10,000 square feet), are automated warehouses carved out of the back of existing stores that drive faster fulfillment times in a smaller geographic area, allowing chain stores to use their numerous geographic locations to act as effective fulfillment/delivery hubs for e-grocery coverage.

CMU and Facebook AI Research use machine learning to teach robots to navigate by recognizing objects

Carnegie Mellon today showed off new research into the world of robotic navigation. With help from the team at Facebook AI Research (FAIR), the university has designed a semantic navigation that helps robots navigate around by recognizing familiar objects.

The SemExp system, which beat out Samsung to take first place in a recent Habitat ObjectNav Challenge, utilizes machine learning to train the system to recognize objects. That goes beyond simple superficial traits, however. In the example given by CMU, the robot is able to distinguish an end table from a kitchen table, and thus extrapolate in which room it’s located. That should be more straightforward, however, with a fridge, which is both pretty distinct and is largely restricted to a singe room.

“Common sense says that if you’re looking for a refrigerator, you’d better go to the kitchen,” Machine Learning PhD student Devendra S. Chaplot said in a release. “Classical robotic navigation systems, by contrast, explore a space by building a map showing obstacles. The robot eventually gets to where it needs to go, but the route can be circuitous.”

CMU notes that this isn’t the first attempt to apply semantic navigation to robotics, but previous efforts have relied too heavily on having to memorize where objects were in specific areas, rather than tying an object to where it was likely to be.