In another UK self-driving first, Project Encode recently demonstrated transfer of control across three states – manual driving, autonomous driving and teleoperation – in live vehicle tests in Oxford and London.
Backed by the DfT’s Centre for Connected and Autonomous Vehicles (CCAV) – like CAVForth – and Innovate UK, partners in the project included technology specialist StreetDrone, IoT security company Angoka, Coventry University and Oxfordshire County Council.
Manual to self-driving to remote operation
The aim was to illustrate how automation can be progressively introduced into industrial settings, with autonomous systems managing the more straightforward vehicle operations and remote drivers stepping in to handle more complex tasks as necessary.
Project Encode demonstration video
The consortium says this proof of concept – delivering transferable responsibility for vehicle control in a cyber-secure context – is central to advancing the application of driverless and teleoperated vehicles across logistics networks.
Official comments
StreetDrone CEO, Mike Potts, said: “The success of this trial, conducted not in a controlled environment but out on the public highway, is blending autonomous technologies with teleoperation to prove an advanced level of technology readiness that can deliver much-needed efficiencies into the supply chain.
“Where tasks are too complex for autonomous technologies, teleoperations steps in. This integration provides a ‘ready-now’ solution and it has been a sight to behold.”
Project Encode – manual driving, self-driving and teleoperation
Dr Giedre Sabaliauskaite, Associate Professor at Coventry University’s Systems Security Group, part of the Centre for Future Transport and Cities (CFTC), added: “It is very important security assurance processes are addressed through the design and engineering cycle.
“This demonstration through the Encode project offers an opportunity to establish a rigorous assurance cycle, ultimately for wider public acceptance.”
Changes to The Highway Code move Britain “closer to a self-driving revolution”.
In a major development for connected and automated mobility (CAM) in the UK, on 20 April 2022 the government set out changes to The Highway Code to move Britain “closer to a self-driving revolution”.
The announcement, by Department for Transport (DfT), the Centre for Connected and Autonomous Vehicles (CCAV), and Trudy Harrison MP, Parliamentary Under-Secretary at the DfT, certainly had the wow factor.
Self-driving safety
The bullet points at the start stated the important overriding aim to “ensure the first self-driving vehicles are introduced safely on UK roads”.
The planned changes to The Highway Code are therefore intended to “clarify drivers’ responsibilities in self-driving vehicles, including when a driver must be ready to take back control”.
Eyebrows were raised at the line: “While travelling in self-driving mode, motorists must be ready to resume control in a timely way if they are prompted to – such as when they approach motorway exits.”
More hyperbolic self-driving headlines
But the announcement ran into real trouble with this: “The plans also include a change to current regulation, allowing drivers to view content that is not related to driving on built-in display screens, while the self-driving vehicle is in control. It will, however, still be illegal to use mobile phones in self-driving mode, given the greater risk they pose in distracting drivers.”
The national press went into meltdown and we’ll look at this in more detail in a special edition of Hyperbolic Self-driving Headlines. But it was enough to prompt senior CAM industry figures to come out in defence of the technology.
Back to the announcement itself. This section is worth rereading: “Britain’s first vehicles approved for self-driving could be ready for use later this year. Vehicles will undergo rigorous testing and only be approved as self-driving when they have met stringent standards.”
At the time of publication the page lists, er, no vehicles. However, it does include the line “You must insure your self-driving vehicle as self-driving”.
New webpage to check if a vehicle is listed as self-driving for use in Great Britain
So, the question remains: Will cars equipped with Automated Lane Keeping Systems (ALKS) be the first to make the grade and be official recognised as “automated”?
Glassbrook also noted a 2025 target for having a full regulatory framework in place to support the widespread deployment of self-driving technology.
Official comments
The announcement – which asserted that self-driving vehicles could create 38,000 new, high-skilled jobs in Britain by 2035 – was accompanied by statements from Transport Minister Trudy Harrison, the RAC’s Steve Gooding and the SMMT’s Mike Hawes.
Transport Minister Harrison said: “This is a major milestone in our safe introduction of self-driving vehicles, which will revolutionise the way we travel, making our future journeys greener, safer and more reliable.
“This exciting technology is developing at pace right here in Great Britain and we’re ensuring we have strong foundations in place for drivers when it takes to our roads.
“In doing so, we can help improve travel for all while boosting economic growth across the nation and securing Britain’s place as a global science superpower.”
Steve Gooding, director of the RAC Foundation, said: “The Highway Code has been updated a number of times in recent years. These latest additions will help us all understand what we must and must not do as we move forward to an environment where cars drive themselves.
“The final part of the jigsaw is to ensure these amendments are widely communicated to, and understood by, vehicle owners. Vehicle manufacturers and sellers will have a vital role to play in ensuring their customers fully appreciate the capabilities of the cars they buy and the rules that govern them.”
Mike Hawes, Chief Executive of The Society of Motor Manufacturers and Traders, said: “Amending The Highway Code to reflect the pace of technological change will help clarify what motorists can and can’t do when a self-driving feature is engaged, so promoting its safe use.”
As you can imagine, there was reaction to the news from across the automotive industry, including the service and repair sector.
Neil Atherton, Sales and Marketing Director at Autoglass, rightly drew attention to the need for sensor recalibration.
“Much has been made of the dawn of fully autonomous vehicles, but in reality many of these technologies are already in our vehicles,” he said.
“Advanced driver assistance systems (ADAS), especially the windscreen fitted sensors that monitor road conditions and hazards in real time, will need to be fitted in all new cars rolling off the production line later this year, and drivers and the wider automotive industry need to be ready to use and maintain them properly.
“Calibration and recalibration of these technologies is absolutely critical for their effective operation.”
Watch what happens when SFPD pull a driverless Cruise robotaxi on April Fools’ Day 2022
A video has gone viral of San Francisco police pulling over a Cruise robotaxi only to find it completely devoid of humans, a truly driverless self-driving car. “Ain’t nobody in it!” the officer says.
Then, with comic timing, when the officer turns his back, the car – a modified Chevrolet Bolt – drives off, pulling over again on the other side of an intersection, with the police car in pursuit, lights flashing.
Watch police pull a self-driving car
Video of police stopping self-driving car in San Francisco by Brandon Melim via ViralHog
On the night in question – hilariously 1st April 2022, April Fools’ Day (honestly, you couldn’t make it up!) – officers pulled the car because it was driving at night without headlights. Cruise later said this was due to human error.
The video was posted to Instagram by b.rad916 with the comment: “Confused SFPD pulling over an autonomous vehicle in the Richmond District!! Then it tries to take off!!”, followed by the grinning squinting face and police car emojis.
Police stop self-driving car in San Francisco by Brandon Melim via ViralHog
He later explained: “My friends and I were walking home from dinner down Clement Street when we heard police sirens. We didn’t think much of it but when I looked over, I noticed there was nobody driving, so I pulled out my phone and started recording.
“When the officers got the Cruise autonomous car to pull over, they approached the vehicle and the windows rolled down. Turned out they initiated the stop because it was driving without its headlights on.
“I thought it was funny that the officers were so confused and found it funny (they were laughing and pacing back-and-forth). I also thought it was strange that the cops think it’s necessary for a robot car to need headlights to see. I’m sure the cameras and AI are advanced enough to navigate safely at night and can see better than the human eye.”
What about other cars seeing it though? Anyway, mirth abounded with amused onlookers, presumably quite familiar with these revolutionary cars of the future, laughing and joking… as if they’d been waiting for something like this to happen.
The best comments heard on the video include: “Ain’t nobody in it”, “This is crazy”, “Are you serious? How does that happen?”, “Oh my god I have to watch this”, “Can you send that to me please?”, “Guys… oh my god, finally”, “So it stops when pulled over? Automatically?”, “We’ve got a code 7 here” – apparently a reference to police radio code for out of service to eat – and, simply, “What the f***”.
Official responses
The San Francisco Police Department reportedly confirmed: “On Friday, April 1, 2022, at approximately 10:00pm officers observed a vehicle travelling without activated headlights at Clement Street and 8th Avenue.
“Officers stopped behind the vehicle and discovered that there was no driver in the vehicle and no other occupant was present. During this contact officers affected a traffic stop. The vehicle moved forward but stopped again to yield for the officers.
“During the stop officers made contact with the remote operator of the driverless vehicle. Upon the officer’s notification a maintenance team responded to the vehicle’s location and took control of the vehicle. No citation was issued during the traffic stop.”
A pretty lenient response, basically: no ticket on this occasion, be on your way.
Cruise responded on Twitter on 10 April: “Chiming in with more details: our AV yielded to the police vehicle, then pulled over to the nearest safe location for the traffic stop, as intended. An officer contacted Cruise personnel and no citation was issued.”
Before adding: “We work closely with the SFPD on how to interact with our vehicles, including a dedicated phone number for them to call in situations like this.”
Of the 100+ features I’ve done for Cars of the Future, the one I find myself quoting most is last summer’s Letters from America: Partners for Automated Vehicle Education (PAVE). In particular, the evidence that live self-driving vehicle demonstration events are highly effective in boosting public trust.
The organisation with a mission to “inform the public about automated vehicles” is expanding fast – launching PAVE Canada in February and now, PAVE Europe.
Partners for Automated Vehicle Education (PAVE) in Europe
On 17 March 2022, at the Autonomy Paris sustainable mobility conference, PAVE Europe announced six “Founding Members”, and a very prestigious bunch they are too. In alphabetical order: financial services provider, Achmea; autonomous vehicle technology specialists, EasyMile and Mobileye; the world’s largest reinsurer, Swiss Re; Cologne-based safety tester TÜV Rheinland; and the company born of Google’s self-driving car project, Waymo.
From a UK perspective, we note the involvement of Lukas Nekermann, MD of London-based Neckermann Strategic Advisors and author of the influential 2015 book, The Mobility Revolution. Both he and Frederic John, who together co-authored 2020’s Being Driven, are credited as the “co-initiators” of PAVE Europe.
We caught up with Tara Andringa, Executive Director of PAVE, to find out more.
Tara Andringa, Executive Director of Partners for Automated Vehicle Education
TA: “We’ve achieved a lot since your last article – made progress on projects we spoke about and launched some brand new ones too.
“Our weekly virtual panels, which began during Covid, have continued to go extremely well. We thought there might be some Zoom fatigue once people got back to work, but in fact they’ve gotten more popular. We’ve had more than 16,000 people register for the panels live, and we have had an additional 20,000 views on YouTube.
“The virtual panels are a great way to spread the conversation about AVs and in January we won a prestigious award for them – a National Communications Award for Outstanding Public Education at the Transportation Research Board (TRB) conference in Washington.
Partners for Automated Vehicle Education at the TRB conference 2021
Automated Vehicle Survey
“Another major success last fall was our survey work. We put together a powerhouse group with the Massachusetts Institute of Technology (MIT) and JD Power to track longitudinally how views change over time. We released the first results in November and we’re going to make it an annual thing.”
The headline result in 2021 was that only 37% of respondents correctly identified the description of a fully automated self-driving vehicle (according to SAE International’s definition) from seven possibilities. 55% of respondents selected descriptions aligned with driver assistance technology.
The results led Lisa Boor, senior manager of global automotive at J.D. Power, to quote the robot from Lost In Space, describing “a ‘Danger, Will Robinson’ moment for the fully automated self-driving vehicle industry”, with “a significant gap between actual and perceived AV knowledge.”
TA: “Another thing we mentioned last time was our public sector workshop with the state of Ohio. We’ve tried to really build out that program, working a lot on state engagement here in the United States. We help to educate public sector officials, to give them the information they need to make choices for their communities and educate their constituencies.”
Which brings us to your international expansion.
TA: “Yes. We’ve realised that, while every government is taking a different regulatory approach, the public confusion and misperceptions are a global phenomenon. So, we’ve decided to expand PAVE’s mission elsewhere, so each country or continent will have its own chapter or sister organisation.
“Public sector engagement is such an important part of this. Each government is looking at different policy options, but this is a global industry. Our new Canadian and European members will be able to network with our US members, to collaborate, but also do their own thing, run their own events tailored to local needs.”
The first quarter of 2022 has seen two giant leaps forward for self-driving in America. First, in February, General Motors–backed Cruise started offering robotaxi rides to the public in San Francisco… with no safety driver.
Then, in March, The National Highway Traffic Safety Administration (NHTSA) removed the necessity for autonomous vehicles to have manual controls including, notably, a steering wheel.
Cruise self-driving robotaxi
Cruise posted a video showing consumers’ reactions to riding in a truly driverless taxi – they ranged from “This is so cool” to “Just weird”, “Slightly scary” to “A lot smoother than I was expecting”, and probably most astutely: “I am literally witnessing the future”.
General Motors (GM) chief executive, Mary Barra, told shareholders: “This major milestone brings Cruise even closer to offering its first paid rides and generating $50bn in annual revenue by the end of the decade.”
Make no mistake, this is a significant development: A household-name US vehicle manufacturer (VM) operating a driverless taxi with no safety driver in a popular global tourist destination.
Not just any old city either – the streets of San Francisco, so closely associated with the iconic high speed car chase from the Steve McQueen film Bullitt. For fans of burning rubber and squealing brakes, it will be hard to take, but that was 1968, over half a century ago. V8 Ford Mustangs and Dodge Chargers are history. Self-driving cars are the future.
If you need further convincing, you need only look to the historic NHTSA announcement, on 10 March 2022, eliminating the need for manufacturers to equip fully autonomous vehicles with a steering wheel.
It’s something we were speculating about at Cars of the Future just last summer – when we looked at Audi’s Grandsphere concept car, with a steering wheel which folds neatly away when in hands-free mode. It’s also a startling indicator of just how rapidly this industry is moving.
Audi Grandsphere self-driving concept
US self-driving law change
The legislative change follows lobbying by General Motors and updates the Federal Motor Vehicle Safety Standards related to occupant protection in vehicles with an automated driving system (ADS).
NHTSA Deputy Administrator, Steven Cliff, commented: “As the driver changes from a person to a machine in ADS-equipped vehicles, the need to keep the humans safe remains the same and must be integrated from the beginning.”
America is surging ahead in self-driving and if the UK wants to remain “at the forefront of this change”, as the Government says, we’d better get our skates on.
Neil Kennett reviews the CAM Innovators self-driving industry event in London, March 2022
As my first industry do in London for two years, the Zenzic Connected and Automated Mobility (CAM) Innovators event 2022 was always going to be memorable. Actually, it was much better than that. It was a fantastic day packed with astute analysis and exciting announcements about self-driving in the UK.
It was also a reminder of the shared vision – the belief that we’re on the cusp of something momentous, that this technology can deliver seismic safety and societal benefits. And this is no pipedream. Thanks to a lot of hard work over many years by an array of seriously talented people, there’s a detailed Roadmap of exactly how we’ll get there.
Let’s talk self-driving
For starters, we couldn’t have wished for a more impressive venue – The Institution of Engineering and Technology (IET) on The Embankment, near Waterloo Bridge. Passing the statue of Michael Faraday, the father of electromagnetism, I bumped into a former colleague before I’d even reached the front door. How nice to see Tom Flisher of Thatcham – a real live human – after all the remote communications of the pandemic.
Hands up, I missed the morning sessions on cyber resilience, vehicle to everything (V2X) and the Interoperable Simulation project. Catching the fast train in to London to attend a real world event is, admittedly, more time consuming and expensive than clicking into a Teams meeting.
We’ll look at the Interoperable Simulation project in more detail another day as it’s a prime example of joined-up thinking, designed to enable seamless testing across the CAM Testbed UK facilities.
The main reason for attending, I thought, was to hear about the latest six UK-based companies selected for Zenzic’s CAM Scale-Up Programme – a business accelerator for almost ready-for-market products and services that can “meet required safety standards and operate in real-world environments”. There’s also the small matter of sharing £500,000 of government funding.
Of the six winners announced in October 2021, four are London-based: geolocation solution provider Albora; Intelligent CCTV designer Exeros; sensor fusion system developer Grayscale AI; and insurance claims visualiser Xtract 360. The other two are: Cambridge-based vulnerable road user safety specialist R4DAR; and Cardiff-based real-time movement experts Route Konnect.
Each will be supported by the UK government – via the Department for Transport’s Centre for Connected and Autonomous Vehicles (CCAV) – and innovation platform Plug and Play. They’ll get time at the testbeds, benefit from introductions to corporate partners (including Honda, Thales and Vodafone), and gain access to a global network investor platform. Watch this space for in-depth profiles.
The curious among you will have noted the “I thought” a couple of paragraphs ago. Of course, hearing from these exceptional innovators was great, but the best was yet to come.
Wired editor and futurist Jeremy White talks self-driving at the CAM Innovators event 2022
Following an entertaining whip through automotive history with Wired editor and futurist Jeremy White – who urged the self-driving industry to “hurry up!” and make connected and automated mobility a reality – we adjourned to the Haslett & Flowers room for networking drinks.
And that’s where the magic happened: Talking shop and shooting the breeze with people I’d just met, connected with on Linkedin, interviewed on Zoom, been on mute with for hours. That’s where you hear the backstories and inspirations, discover obscure but pertinent bits of information, and see early signs of the next big things.
A maelstrom of tech wizards and engineers, CEOs and interns, the odd safety campaigner and motoring hack, most cautious about over-promising but overwhelmingly excited and optimistic about the fast-approaching road transport revolution.
That’s what self-driving industry events are all about. That’s what we’ve been missing.
Relationship between driverless cars, the media and consumer confidence reflected in five hyperbolic headlines.
Self-driving experts talk constantly of the need to earn public trust, but driverless cars continue to divide opinion. Indeed, recent surveys have shown that people are becoming more, not less, wary of them.
Just this week, in Inverness, Scotland, where an autonomous bus trial is due to start later this year, The Inverness Courier reported significant resistance to the idea. 69% of respondents to its survey of local residents said they would refuse to get on a driverless bus.
What we need, of course, is for the media to convey an informed and nuanced safety message. Hmm! To illustrate the scale of the task, here’s a list of our top five hyperbolic headlines:
On a mission to drive more sensible debate about self-driving, Carsofthefuture.co.uk has renewed its media partnership with Reuters Events for the Auto Tech 2022 digital conference on 14-15 June.
The prestigious two-day online event will enable technology providers and automotive companies to meet and do business with vehicle manufacturers (VMs) including Audi, BMW, Cadillac, Daimler, Fisker, Ford, GM, Honda, Hyundai, Nissan, Opel and Toyota.
Confirmed speakers include Mercedes Benz Mobility chief executive Franz Reiner, Hyundai Motor Company chief safety officer Brian Latouf, Polestar chief operating officer Dennis Nobelius and Lucid Motors vice president of software validation Margaret Burgraff.
Self-driving and AI at Auto Tech 2022
As part of The Key Steps Towards Safer Roads programme on 15 June, Carsofthefuture.co.uk editor Neil Kennett will moderate two sessions: 1) “The Growing Presence of AI”, with Sammy Omari, vice president of autonomy at Motional, and; 2) “Where are we on the journey to full automation?”, with Xinzhou Wu, head of Xpeng Motors’ Autonomous Driving Centre.
The former will cover the value of artificial intelligence in testing cutting-edge systems and its role in autonomous vehicle (AV) decision making. The latter will cover autonomous driving, in-car connectivity and advanced driver assistance systems (ADAS), evaluating the progress made and exploring when carmakers expect to introduce fully automated features.
Carsofthefuture.co.uk editor, Neil Kennett, said: “I’m delighted to renew our partnership with Reuters and look forward to lively discussions about these phenomenal but controversial technologies. It’s a shame, given everything Tesla’s done for electric cars, that so many hyperbolic headlines are caused by its confusingly-named Full Self-Driving (FSD) package. It simply isn’t self-driving as the rest of the industry understands it.
“Conflating assisted and automated driving is dangerous, because it risks drivers misunderstanding what their cars are capable of. News of so-called driverless car crashes then dents consumer confidence – the last thing the industry needs at such a crucial time in terms of public perception. These are safety-critical issues and utmost clarity is vital. For the near future at least, the best advice is that drivers need to be alert at all times.”
Nabil Awan, automotive conference producer at Reuters Events, added: “The moves towards greater connectivity and autonomy that we are seeing will lead to safer roads while also deeply transforming the auto industry as we know it today. Our unique Auto Tech 2022 event will give innovators and technology providers a chance to discuss the latest advances and come away with valuable intelligence with which to drive the evolution of the sector.”
Carsofthefuture editor will moderate sessions on self-driving and AI at Auto Tech 2022
A few messages flew back and forth, and it transpired that he’s an expert in measuring driver behaviour, particularly driver-vehicle interactions in ADAS-equipped and self-driving vehicles. That was music to my ears, so we arranged a Zoom. What follows is the highly insightful result.
Lucas Noldus Ph.D., Founder of Noldus Information Technology
LN: “The future starts here. The world is changing. We see people living longer and there are more and more interactive devices – telephones, tablets, dashboards – with which we can interact, leading to greater risk of distraction while driving. I know personally how tempting it is to use these devices, always trying to keep your eyes on the road.
“We already have fascinating developments in connected driving and now, with self-driving, the role of the driver changes significantly. That has triggered research institutes, universities, OEMs and tier one suppliers to pay more attention to the user experience for both drivers and passengers.
“All these experiences are important because how people perceive the safety and comfort will influence their buying decisions, and their recommendations to other potential users.
“For autonomous driving, how far will we go towards level five? What happens at the intermediate stages? Over the coming decades, driving tasks will gradually diminish but, until full autonomy, the driver will have to remain on standby, ready to take over in certain situations. How will the vehicle know the driver is available? How quickly can he take over? These are the topics we’re involved in as a technology company.
“We make tools to allow automotive researchers to keep the human in the loop. Traditionally, automotive research focused exclusively on improving the vehicle – better engines, drivetrains etc. Until recently, nobody paid much attention to the human being (with a brain, skeletal system, muscles, motor functions), who needs to process information through his sensory organs, draw the right conclusions and take actions.
“Now, these aspects are getting more attention, especially in relation to reduced capacity, whether due to a distracting device, drugs, alcohol or neurodegeneration. As you get older your response time becomes longer, your eyesight and hearing abilities reduce, as does the speed at which you can process information.
“These are the challenges that researchers in automotive are looking at concerning the role of the driver, now and in the future. If the automated or semi-automated system wants to give control back to the driver because its AI algorithms decide a situation is too complex, can the driver safely take over while he’s been doing something like reading or taking a nap? How many milliseconds does the brain need to be alert again?
NK: “Draft legislation seems to be proceeding on a 10-second rule, but some studies say at least 30 seconds is required.”
LN: “Situational awareness – that’s a key word in this business. Not only where am I geographically, but in what situation. Oh, I’m in a situation where the road surface is very wet, there’s a vehicle just in front of me, the exit I need is near and I’m in the wrong lane. Understanding a situation like that takes time.
“If we take a helicopter view, from our perspective as a technology company, what should be measured to understand the driver behaviour? Which sensors should we use to pick up that information? If we use a microphone, a video camera, a heartbeat monitor and a link to the ECU, how do we synchronise that?
“That’s not trivial because one sensor may be sending the sampling at 300Hz and another at 25 frames per second. That’s something my company has specialised in over the years. We’re very good at merging data from different sources, whether it’s a driving simulator continuously spitting out data, a real car, or sensors mounted in the infrastructure.
“You then need to analyse that data and pull out meaningful quantitative units that give you actionable insights. Generating large matrices is no big deal, making sense of that information is the real challenge.
“For example, in dashboard design, a manufacturer might be comparing two or three displays of road quality. A driver behaviour study with our tools will give the designer a clear answer on which design leads to the least cognitive workload, the least confusion.
Noldus DriveLab
“This same technical challenge can be applied to a vast number of design objectives. The vehicle manufacturer might be looking to make incremental improvements to, say, the readability of the dashboard under certain light conditions. Or they might be working on a completely new feature, like an intelligent personal in-car assistant. A number of brands are working on that, but the concept is still relatively new.
“You cannot test every scenario on the road, it’s just too dangerous, so we work with simulator manufacturers too. On the road or in the lab, we can measure a driver’s actions with eye-tracker, audio, video, face-reader and physiology in one.”
NK: “Back to LinkedIn again, I saw a post by Perry McCarthy, the F1 driver and original Stig on Top Gear, who said something like: Simulators are getting so good these days, when you make a mistake they drop three tonnes of bricks on your legs!”
LN: “You have so-called high fidelity and low fidelity simulators – the higher the fidelity, the closer you get to the real vehicle behaviour on the road, and there are all sorts of metrics to benchmark responsiveness.
“You have simple fixed-base simulators right up to motion-based simulators which can rotate, pitch and roll, move forward, backwards, sideways and up and down. For the best ones you’re talking about 10 million euros.
“We work with OEMs, tier1 suppliers, research institutes and simulator manufacturers to build-in our DriveLab software platform. We also advise on what sensors are recommended depending on what aspects of driver behaviour they want to study.
“We try to capture all the driver-vehicle interactions, so if he pushes a pedal, changes gear or turns the steering wheel, that’s all recorded and fed into the data stream. We can also record their body motion, facial expression, what they’re saying and how they’re saying it – it all tells us something about their mental state.
Multi-camera eye tracker (Smart Eye)
“Eye tracking measures the point of gaze – what your pupils are focused on. In a vehicle, that might be the left, right and rear mirrors, through the windscreen or windows, around the interior, even looking back over your shoulders. To capture all that you need multiple eye-tracking cameras. If you just want to look at, for example, how the driver perceives distance to the car in front, you can do with just two cameras rather than six.
“Eye tracking generates all sorts of data. How long the eyes have been looking at something is called dwell time. Then there’s what direction the eyes are looking in and how fast the eyes move from one fixed position to another – that’s the saccade. People doing eye tracking research measure saccades in milliseconds.
“Another important metric is pupil diameter. If the light intensity goes up, the pupil diameter decreases. Given a stable light condition, the diameter of your pupil says something about the cognitive load to your brain – the harder you have to think, the wider your pupils will open. If you’re tired, your blink rate will go up. There’s a normal natural blink rate to refresh the fluid on your eyes with a fully awake person, but if you’re falling asleep the blink rate changes. It’s a very useful instrument.
“Then there’s body worn sensors that measure physiology. It’s harder to do in-car, but in a lab people don’t mind wearing electromyography (EMG) sensors to measure muscle tension. If you’re a designer and you want to know how easy it is for an 80-year-old lady to operate a gearshift, you need to know how much muscle power she has to exert.
“We also measure the pulse rate with a technique called photoplethysmography (PPG), like in a sports watch. From the PPG signal you can derive the heart rate (HR). However, a more accurate method is an electrocardiogram (ECG), which is based on the electrical activity of the heart.
GSR (EDA) measurement
“Further still, we measure galvanic skin response (GSR), also called electrodermal activity (EDA), the level of sweating of your skin. The more nervous you get, the more you sweat. If you’re a bit late braking approaching a traffic jam, your GSR level will jump up. A few body parts are really good for capturing GSR – the wrist, palm, fingers, and the foot.
“We also measure oxygen saturation in the blood with near infrared spectroscopy (NIRS) and brain activity with an electroencephalogram (EEG). Both EEG and NIRS show which brain region is activated.
“Another incredibly useful technique is face reading. Simply by pointing a video camera at someone’s face we can plot 500 points – the surroundings of the eyebrows, the eyelids, the nose, chin, mouth, lips. We feed this into a neural network model and classify it against a database of tens of thousands of annotated images, allowing us to identify basic emotions – happy, sad, angry, surprised, disgusted, scared or neutral. You can capture that from one photograph. For other states, like boredom or confusion, you need a series of images.
“These days we can even capture the heart rate just by looking at the face – tiny changes in colour resulting from the pulsation of the blood vessels in the skin. This field of face reading is evolving every year and I dare to claim that we are leading the pack with our tool.
“Doing this in the lab is one thing, doing it in a real car is another challenge, being able to keep your focus on the driver’s face and deal with variable backgrounds. Of course, cars also drive at night so the next question is can you do all this in darkness? We turned our company van into an instrumented vehicle and my sons agreed to be the guinea pigs.
“It took some work – overcoming the issue of light striking the face and causing sharp shadows, for instance – but we can now use infrared illuminators with our FaceReader software to make these measurements in full darkness.
“The turning of the head is also very important in studying distraction, for example, if the driver looks sideways for too long, or nods their head in sleepiness. When something shocks someone, we see the face change and the blood pressure rise, and these readings are synchronised in DriveLab.
“It is well proven that even things like changing radio station can be very distracting. Taking your eyes off the road for just a few seconds is dangerous. As we move to more and more connected devices, touchscreens and voice commands, minimising distraction is vital to ensure safety.”
NK: “I absolutely love this tech but what I actually drive is a 7-year-old Suzuki Swift Sport with a petrol engine and a manual gearbox, and I quite like it that way”
LN: “I’m doing research on cars of the future with my software but I am personally driving a 30-year old soft-top Saab 900. That’s my ultimate relaxation, getting away from high tech for a moment.
“At Noldus, we’re constantly pushing the boundaries of research, working with top level organisations in automotive – Bosch, Cat, Daimler, Fiat, Honda, Isuzu, Land Rover, Mazda, Nissan, Scania, Skoda, Toyota, Valeo and Volvo, to name just a few – and also with the Netherlands Aerospace Centre (NLR) and the Maritime Research Institute Netherlands (MARIN).
“Our aim is make it so that the client doesn’t have to worry about things like hardware to software connections – we do that for them so they can focus on their research or design challenge.”
Ahead of a flagship product launch later this week, Bill McKinley, Automotive Strategic Planner at Keysight Technologies, gives his thoughts on self-driving and the fast-changing connected and autonomous vehicle (CAV) landscape.
Avid readers may remember that Bill was on the panel I hosted at the Small Cells World Summit in May. He’s got 30+ years’ experience in wireless communications and his current focus is developing test solutions for the automotive sector.
BM: “The UK, in line with other nations around the world, is investing heavily in connectivity and electrification – both the vehicles themselves and the charging infrastructure. Connected vehicles have been demonstrated to enhance safety via cellular vehicle to everything (C-V2X) and dedicated short-range communication (DSRC).
“These technologies allow for more efficient driving, for example, by routing to avoid accidents or poor road conditions. They also enable higher levels of automation, all of which can lead to an improved overall driving experience.
“It is likely that the first fully automated vehicles will be delivery vehicles, controlled environment shuttle type services, and buses on specific routes. With the gradual introduction of robotaxis, we will no doubt start to see Mobility as a Service (MaaS) become more common over the next 10-15 years.
“Keysight was the first test and measurement company to be awarded Global Certification Forum (GCF) validation for C-V2X RF conformance. We have industry leading validated test cases for the C-V2X protocol conformance test, and we were the first to be awarded OmniAir Qualified Test Equipment (OQTE) status.
“Cybersecurity will play a critical role in connected mobility and Keysight is working with leading companies and organisations in this space to develop solutions to ensure vehicular communications remain safe and robust against attacks.
“Clearly, the main risks associated with self-driving vehicles are around the safety aspects, which in turn will heavily influence public acceptance of the technology. We are all very familiar with some of the headlines about Tesla vehicles.
“It remains incredibly challenging to overcome the complexities of urban automated driving, but things are improving all the time. Our autonomous driving emulator (ADE) system is designed with this in mind – to test many autonomous drive systems in a rich and authentic environment within the lab, before moving testing out into the field.”
More on that to follow soon. For further info see keysight.com
