China Automotive Multimodal Interaction Development Research Report, 2022
Multimodal interaction research: more hardware entered the interaction, immersive cockpit experience is continuously enhanced
ResearchInChina's “China Automotive Multimodal Interaction Development Research Report, 2022” conducts analysis and research from three aspects: the installation status of mainstream interaction modes, the application of mainstream vehicle models’ interaction modes, and cockpit interaction solutions of suppliers.
1. Guided by the "third space" concept, multimodal interaction is being deeply applied to intelligent cockpit with five main features.
(1) With the trend of large screen, multi-screen and smart surface materials, touch interaction has gradually expanded its application range
The large screen of center console makes touch control the mainstream interaction mode. For example, Mercedes-Benz EQS and XPeng P7 have almost no physical buttons on the center console, but all done by touch.
Multi-screen cockpits make touch range from the front to the rear, from the center console / co-pilot infotainment extended to the doors, windows, seats and other components. For example, Li Auto L9 eliminates the traditional instrument panel and replaces it with a small TouchBar above the steering wheel; in addition, it also carries a co-pilot screen and rear audio/video screen to achieve five-screen interaction.
(2) Voice interaction evolves from passive to active, personalized and emotional demands will be met
See-and-speak, continuous dialogue, voice source location, wake-up free and other voice technology has been widely equipped in the new cars launched in 2022, the voice interaction mode tends to be more natural.
Personalized experience is currently the focus of the voice function, and intelligent EV brands such as NIO/XPeng/Li Auto are mainly optimized in voice customization, image customization, skill customization, etc.
In the future, the concept of emotional companionship and electronic pets is expected to be realized with the help of voice function.
(3) Face recognition algorithm promotes DMS, OMS and IMS scale installation
Face recognition-based ID login and user habit presets have been implemented in NIO ET7/ET5, XPeng P7/G9, AITO M5/M7, Neta S, and Voyah Dreamer. Among them, AITO can automatically login to Huawei account through face recognition, linking schedule, navigation information, call records, music and video membership rights, third-party application data, and automatic switching of driving information to achieve information flowing.
Face payment function application is still relatively small, which has been installed on Toyota Harrier Premium Edition, and XPeng P7 will be achieved it through OTA upgrade.
Face recognition-based DMS, OMS, and IMS are being installed on a large scale and will promote in-cockpit camera equipment.
(4) Gesture recognition function is single, installed as a complementary interaction mode
Currently, gesture recognition mainly applied to multimedia switching, volume control, phone answering, lighting control, etc., mainly installed as a supplementary interaction mode.
In the future, gesture recognition is expected to be combined with ADAS functions to achieve vehicle summoning, starting and parking based on exterior visual perception.
(5) Fingerprint, iris, vein, heart rate and other biometric applications in the car are still in the exploration stage
Fingerprint is expected to be applied in user ID recording and in-vehicle payment scenarios.
Iris/eye tracking will enhance the accuracy of in-cockpit monitoring such as DMS, and have imagination space in the future in intention prediction and active recommendation.
Vein/heart rate, etc., will be applied under the concept of in-vehicle health.

2. Multimodal recognition and large & multi-screen, AR-HUD, AR/VR, ambient light, high-quality audio and other hardware interaction to be enhanced, immersive cockpit experience continues to grow
(1) Li Auto L9, creating a multimodal interaction experience through five-screen, voice and gestures
Features of Li Auto L9 interaction:
Replaces the traditional instrument screen with HUD + safe driving interaction screen.
Enhances the audiovisual experience through 15.7-inch center console screen + 15.7-inch co-pilot screen + HUD + safe driving interaction screen + 15.7-inch rear entertainment screen with 3K HD resolution and high color reproduction on the automotive screen.
6-voice-zone recognition interaction (AISpeech ) + self-developed speech engine (in cooperation with Microsoft) + 3D ToF sensor (gesture interaction, cockpit monitoring), to achieve multimodal interaction.
Voice + gesture integration, such as point at the sunshade and say "open this", that is, open the sunshade.
Connect to Nintendo Switch via Type-C and rear entertainment screen to create cockpit gaming scenarios.
Two Qualcomm Snapdragon 8155 chips, 24GB RAM + 256G high-speed storage, and dual-5G operator switching to provide computing power and network support.

(2) NIO, Li Auto, Audi and others take the lead in AR/VR glasses installation for an immersive cockpit experience
Jointly developed with NREAL, the NIO AR glasses can project a 6m viewing distance and 201-inch-equivalent screen (optional price RMB 2,299). VR glasses are jointly developed with NOLO, and equipped with ultra-thin Pancake optical lenses for binocular 4K display.

Li Auto AR glasses are provided by Leiniao Technology, a subsidiary of TCL, and the product was available on Li Auto Store from August 2022. Leiniao Air adopts the polarization Birdbath + MicroOLED technology solution and is directly connected to the rear audio and video system of Li Auto L9. Plugging the cable at the legs of the glasses into Li Auto's rear-row DP jack provides users with a 140-inch giant screen (4m distance) viewing experience.

Audi VR glasses will be first launched in USA in 2023. The hardware is supported by HTC and integrated into Holoride Pioneers' Pack in-vehicle entertainment system for games and video scenarios.
3. Chip, algorithm and system integrator work together to create active cockpit interaction based on multimodal interaction
(1) Chip companies, represented by Horizon, integrate multimodal interaction into intelligent driving solution
Horizon Halo, a cockpit solution built by Horizon based on Journey 2 and Journey 3, can integrate vision, voice and other sensor data to achieve active interaction. Among them, Halo 3.0 can provide a complete set of AI solutions including DMS, face detection, behavior detection, gesture recognition, child behavior detection, multimodal voice interaction and other functions for front and rear row users.

(2) iFLYTEK, Cerence, etc. entered with voice, and SenseTime, ArcSoft, etc. entered with vision to achieve the integration of superior modes with other modes to create an overall cockpit solution
iFLYTEK builds a multimodal system based on "listening, speaking, seeing and displaying" all-link technology, realizing that the vehicle can process the fusion of voice, image, live body and other information throughout the car-using cycle of getting on - driving - getting off, so as to understand passengers' information more actively and deeply, and thus actively care for them, push relevant contents/services, and change vehicle settings to explore disruptive interaction experience.

Based on its strengths in voice, Cerence will integrate vehicle data (fatigue monitoring, mobile phone interconnection, entertainment system, air conditioner, fuel, charging, seat, GPS, air quality) with in-vehicle multimodal interaction (voice, speech synthesis, text input, eye tracking, gesture recognition, emotion recognition, biometrics) to create immersive cockpit interaction in the future.

(3) ADAYO, Desay SV and other Tier1, combining multimodal interaction with scenarios to create a personalized cockpit experience
Based on high computing power AI chip, through visual, auditory, touch, multimodal front-fusion, multimodal rear-fusion and other recognition technologies, ADAYO multimodal interaction system can realize 4 major categories and 70+ human-machine interaction scenarios, including voice + gesture to open the window or air conditioner, see-and-speak, greeting when getting into the car, soothing children crying, voice control, face start engine, eye control to light up screen, driver behavior monitoring and dangerous behavior reminder, etc., providing users with active emotional multimodal interaction scenario experience and meeting the personalized needs of thousands of users.

China Automotive Digital Key Research Report, 2023
Automotive Digital Key Research: the pace of mobile phones replacing physical keys quickens amid the booming market
"China Automotive Digital Key Research Report, 2023" released by ResearchInChina co...
Automotive Camera Tier2 Suppliers Research Report, 2022-2023
1. The automotive camera market maintains a pattern of "one superpower and several great powers".
Automotive cameras are used to focus the light reflected from the target onto the CIS after refractio...
Emerging Carmaker Strategy Research Report, 2023 - NIO
Emerging carmaker strategy research: NIO is deploying battery swap and sub-brands for the knockout match in 2023.In 2022, the sales surged by 32.3% year on year, being concentrated in first-tier citie...
Nissan CASE (Connectivity, Automation, Sharing and Electrification) Layout Research Report, 2022-2023
Nissan CASE research: two leverages for Dongfeng Nissan to turn the tables. Introduction: since 2020, the declining sales of Dongfeng Nissan have exposed its problems in brand influence and product co...
China Automotive Gesture Interaction Development Research Report,2022-2023
Vehicle gesture interaction research: in 2022, the installations rocketed by 315.6% year on year.China Automotive Gesture Interaction Development Research Report, 2022-2023 released by ResearchInChina...
Automotive Power Management Integrated Circuits (PMIC) Industry Report, 2023
Automotive PMIC research: the process of domestic automotive PMICs replacing foreign ones in China in the “crisis of chip shortage”.
Automotive power management integrated circuits (PMIC) find broad ...
Automotive Cockpit SoC Research Report, 2023
Cockpit SoC research in 2023: Can X86 solutions returning to cockpit SoC challenge the “ARM+Google” mobile solution?
This report highlights the research on the products and plans of 9 overseas and 8 ...
AI Foundation Model and Autonomous Driving Intelligent Computing Center Research Report, 2023
New infrastructures for autonomous driving: AI foundation models and intelligent computing centers are emerging.
In recent years, the boom of artificial intelligence has actuated autonomous driving, ...
Automotive Microcontroller Unit (MCU) Industry Report, 2023
MCU Industry Research: Automotive high-end MCU will be still in short supply, and how OEMs can break the situation.
ResearchInChina has released "Automotive Microcontroller Unit (MCU) Industry Repor...
Global and China Fuel Cell Market and Trend Research Report, 2023
Fuel Cell Industry Research: Hydrogen energy has been put on the national agenda with scenario application being rolled out.The hydrogen energy industry has been included into the national energy stra...
Global and China Automotive Smart Antenna Research Report, 2022-2023
Smart antenna research: the integration of automotive antennas and intelligent connected terminals tends to accelerate.
The development trend of automotive antennas: tend to be intelligent, diversif...
Chinese Independent OEMs’ Telematics System and Entertainment Ecosystem Research Report, 2022
Vehicle telematics system research 1: the control scope is expected to expand to the entire vehicle.From January to December 2022, Chinese independent OEMs installed telematics systems in 6.42 million...
China Autonomous Shuttle Market Report, 2022-2023
Autonomous Shuttle Research: application scenarios further extend amidst policy promotion and continuous exploration
Autonomous shuttles are roughly categorized into minibuses and robobuses. Minibuse...
Intelligent Cockpit Domain Control Unit (DCU) and Head Unit Dismantling Report, 2023 (1)
Dismantling of Head Unit and Cockpit Domain Control Unit (DCU) of NIO, Toyota and Great Wall Motor The report highlights the dismantling of Toyota’s MT2712-based head unit, Fisker’s Intel A2960-based ...
Automotive Vision Algorithm Industry Research Report, 2023
Research on automotive vision algorithms: focusing on urban scenarios, BEV evolves into three technology routes.1. What is BEV?
BEV (Bird's Eye View), also known as God's Eye View, is an end-to-end t...
ADAS Domain Controller Key Component Trends Report, 2022
ResearchInChina researched and summarized China’s current mainstream high computing power ADAS domain controller products such as Huawei MDC and DJI ADAS domain controller prototype, and technical inf...
Automotive High-precision Positioning Research Report, 2023
High Precision Positioning Research: four forms of mass-produced integrated high-precision positioning products
With the continuous development of autonomous driving, the demand for high-precis...
Automotive AUTOSAR Platform Research Report, 2023
AUTOSAR research: CP + AP integration, ecosystem construction, and localization will be the key directions.
AUTOSAR standard technology keeps upgrading, and the willingness to build open cooperation ...