China Automotive Multimodal Interaction Development Research Report, 2022
  • Dec.2022
  • Hard Copy
  • USD $3,800
  • Pages:170
  • Single User License
    (PDF Unprintable)       
  • USD $3,600
  • Code: YSJ116
  • Enterprise-wide License
    (PDF Printable & Editable)       
  • USD $5,400
  • Hard Copy + Single User License
  • USD $4,000

Multimodal interaction research: more hardware entered the interaction, immersive cockpit experience is continuously enhanced

ResearchInChina's “China Automotive Multimodal Interaction Development Research Report, 2022” conducts analysis and research from three aspects: the installation status of mainstream interaction modes, the application of mainstream vehicle models’ interaction modes, and cockpit interaction solutions of suppliers.

1. Guided by the "third space" concept, multimodal interaction is being deeply applied to intelligent cockpit with five main features.

(1) With the trend of large screen, multi-screen and smart surface materials, touch interaction has gradually expanded its application range

The large screen of center console makes touch control the mainstream interaction mode. For example, Mercedes-Benz EQS and XPeng P7 have almost no physical buttons on the center console, but all done by touch.
Multi-screen cockpits make touch range from the front to the rear, from the center console / co-pilot infotainment extended to the doors, windows, seats and other components. For example, Li Auto L9 eliminates the traditional instrument panel and replaces it with a small TouchBar above the steering wheel; in addition, it also carries a co-pilot screen and rear audio/video screen to achieve five-screen interaction.

(2) Voice interaction evolves from passive to active, personalized and emotional demands will be met
See-and-speak, continuous dialogue, voice source location, wake-up free and other voice technology has been widely equipped in the new cars launched in 2022, the voice interaction mode tends to be more natural.
Personalized experience is currently the focus of the voice function, and intelligent EV brands such as NIO/XPeng/Li Auto are mainly optimized in voice customization, image customization, skill customization, etc.
In the future, the concept of emotional companionship and electronic pets is expected to be realized with the help of voice function.

(3) Face recognition algorithm promotes DMS, OMS and IMS scale installation

Face recognition-based ID login and user habit presets have been implemented in NIO ET7/ET5, XPeng P7/G9, AITO M5/M7, Neta S, and Voyah Dreamer. Among them, AITO can automatically login to Huawei account through face recognition, linking schedule, navigation information, call records, music and video membership rights, third-party application data, and automatic switching of driving information to achieve information flowing.

Face payment function application is still relatively small, which has been installed on Toyota Harrier Premium Edition, and XPeng P7 will be achieved it through OTA upgrade.

Face recognition-based DMS, OMS, and IMS are being installed on a large scale and will promote in-cockpit camera equipment.

(4) Gesture recognition function is single, installed as a complementary interaction mode
Currently, gesture recognition mainly applied to multimedia switching, volume control, phone answering, lighting control, etc., mainly installed as a supplementary interaction mode.
In the future, gesture recognition is expected to be combined with ADAS functions to achieve vehicle summoning, starting and parking based on exterior visual perception.

(5) Fingerprint, iris, vein, heart rate and other biometric applications in the car are still in the exploration stage
Fingerprint is expected to be applied in user ID recording and in-vehicle payment scenarios.
Iris/eye tracking will enhance the accuracy of in-cockpit monitoring such as DMS, and have imagination space in the future in intention prediction and active recommendation.
Vein/heart rate, etc., will be applied under the concept of in-vehicle health.


2. Multimodal recognition and large & multi-screen, AR-HUD, AR/VR, ambient light, high-quality audio and other hardware interaction to be enhanced, immersive cockpit experience continues to grow

(1) Li Auto L9, creating a multimodal interaction experience through five-screen, voice and gestures

Features of Li Auto L9 interaction:
Replaces the traditional instrument screen with HUD + safe driving interaction screen.
Enhances the audiovisual experience through 15.7-inch center console screen + 15.7-inch co-pilot screen + HUD + safe driving interaction screen + 15.7-inch rear entertainment screen with 3K HD resolution and high color reproduction on the automotive screen.
6-voice-zone recognition interaction (AISpeech ) + self-developed speech engine (in cooperation with Microsoft) + 3D ToF sensor (gesture interaction, cockpit monitoring), to achieve multimodal interaction.
Voice + gesture integration, such as point at the sunshade and say "open this", that is, open the sunshade.
Connect to Nintendo Switch via Type-C and rear entertainment screen to create cockpit gaming scenarios.
Two Qualcomm Snapdragon 8155 chips, 24GB RAM + 256G high-speed storage, and dual-5G operator switching to provide computing power and network support.


 (2) NIO, Li Auto, Audi and others take the lead in AR/VR glasses installation for an immersive cockpit experience

Jointly developed with NREAL, the NIO AR glasses can project a 6m viewing distance and 201-inch-equivalent screen (optional price RMB 2,299). VR glasses are jointly developed with NOLO, and equipped with ultra-thin Pancake optical lenses for binocular 4K display.


Li Auto AR glasses are provided by Leiniao Technology, a subsidiary of TCL, and the product was available on Li Auto Store from August 2022. Leiniao Air adopts the polarization Birdbath + MicroOLED technology solution and is directly connected to the rear audio and video system of Li Auto L9. Plugging the cable at the legs of the glasses into Li Auto's rear-row DP jack provides users with a 140-inch giant screen (4m distance) viewing experience.


Audi VR glasses will be first launched in USA in 2023. The hardware is supported by HTC and integrated into Holoride Pioneers' Pack in-vehicle entertainment system for games and video scenarios.
3. Chip, algorithm and system integrator work together to create active cockpit interaction based on multimodal interaction

(1) Chip companies, represented by Horizon, integrate multimodal interaction into intelligent driving solution

Horizon Halo, a cockpit solution built by Horizon based on Journey 2 and Journey 3, can integrate vision, voice and other sensor data to achieve active interaction. Among them, Halo 3.0 can provide a complete set of AI solutions including DMS, face detection, behavior detection, gesture recognition, child behavior detection, multimodal voice interaction and other functions for front and rear row users.


(2) iFLYTEK, Cerence, etc. entered with voice, and SenseTime, ArcSoft, etc. entered with vision to achieve the integration of superior modes with other modes to create an overall cockpit solution

iFLYTEK builds a multimodal system based on "listening, speaking, seeing and displaying" all-link technology, realizing that the vehicle can process the fusion of voice, image, live body and other information throughout the car-using cycle of getting on - driving - getting off, so as to understand passengers' information more actively and deeply, and thus actively care for them, push relevant contents/services, and change vehicle settings to explore disruptive interaction experience.


Based on its strengths in voice, Cerence will integrate vehicle data (fatigue monitoring, mobile phone interconnection, entertainment system, air conditioner, fuel, charging, seat, GPS, air quality) with in-vehicle multimodal interaction (voice, speech synthesis, text input, eye tracking, gesture recognition, emotion recognition, biometrics) to create immersive cockpit interaction in the future.


 (3) ADAYO, Desay SV and other Tier1, combining multimodal interaction with scenarios to create a personalized cockpit experience

Based on high computing power AI chip, through visual, auditory, touch, multimodal front-fusion, multimodal rear-fusion and other recognition technologies, ADAYO multimodal interaction system can realize 4 major categories and 70+ human-machine interaction scenarios, including voice + gesture to open the window or air conditioner, see-and-speak, greeting when getting into the car, soothing children crying, voice control, face start engine, eye control to light up screen, driver behavior monitoring and dangerous behavior reminder, etc., providing users with active emotional multimodal interaction scenario experience and meeting the personalized needs of thousands of users.


1 Overview of Multimodal Interaction
1.1 Definition of Multimodal Interaction
1.2 Intelligent Cockpit Multimodal Interaction Development Roadmap
1.3 Multimodal Interaction Features of Representative Models
1.4 Touch Interaction
1.5 Voice Interaction
1.5.1 Comparison of In-vehicle Voice Functions of Major Brands
1.5.2 Voice Assistant Visualization Will Warm the Interaction
1.6 Gesture Interaction
1.6.1 Gesture Interaction Technology Route
1.6.2 Gesture Interaction Function of Representative Models
1.6.3 Gesture Interaction Trend
1.7 Facial Recognition
1.7.1 Facial Recognition Functions of Representative Models
1.7.2 In-cabin Monitoring based on Facial Recognition 
1.8 Biometrics
1.9 Multimodal Recognition and Hardware Interaction
1.9.1 Exterior Headlight Interaction
1.9.2 Interior Ambient Light Interaction
1.9.3 AR/VR Interaction

2 Industry Chain and Development Trends
2.1 Multimodal Interaction Industry Chain
2.2 Multimodal Perception Algorithms
2.2.1 Voice Algorithms
2.2.2 Vision Algorithms
2.3 Multimodal Interaction Applications Require Significantly More Arithmetic Power on Cockpit Chip
2.4 Multimodal Interaction Security Strategy
2.4.1 Policy Environment
2.4.2 Multimodal Interaction Laws and Regulations
2.4.3 In-cabin Information Security Strategy of OEMs
2.5 Development Trends

3 Multimodal Interaction Solution for Benchmark Models
3.1 Emerging Automaker Brands
3.1.1 NIO ET7 
3.1.2 NIO ET5 
3.1.3 XPeng P7 
3.1.4 XPeng G9 
3.1.5 Li Auto ONE 
3.1.6 Li Auto L9 
3.1.7 Neta S
3.1.8 AITO M5
3.1.9 AITO M7 
3.1.10 Aiways U6 
3.1.11 Voyah Dreamer
3.1.12 NIUTRON NV 
3.1.13 IM L7
3.1.14 HiPhi Z 
3.1.15 ZEEKR 009
3.1.16 Avatr 11 
3.1.17 Rising Auto R7 
3.1.18 DeepAI SL03 
3.1.19 Leap Motor C01 
3.1.20 Tesla Model Y
3.2 Conventional Joint Venture Brands
3.2.1 New Ford Explorer
3.2.2 BMW iX
3.2.3 Mercedes-Benz EQS 
3.2.4 Toyota Harrier
3.2.5 Denza D9
3.2.6 Lotus ELETRE 
3.3 Conventional Independent Brands 
3.3.1 Jetour DaSheng i-DM
3.3.2 BYD Seal
3.3.3 OShan Z6 
3.3.4 GAC Trumpchi Emkoo
3.3.5 Boyue L 

4 Multimodal Interaction Solution Providers
4.1.1 Profile
4.1.2 Intelligent Vehicle Product Layout 
4.1.3 Multimodal Interaction System
4.1.4 Intelligent Cockpit Solutions
4.1.5 Intelligent Interaction Development Routes
4.1.6 Multimodal Interaction Becomes the Key Direction of Xunfei Super Brain 2030 Plan
4.1.7 AI Virtual Human Interaction Platform
4.1.8 Major Customers
4.2 Cerence
4.2.1 Profile
4.2.2 SSE Technology
4.2.3 Cockpit Interaction Solutions
4.2.4 Co-Pilot Customers
4.2.5 Product Development Routes
4.2.6 Partners
4.3 AISpeech
4.3.1 Profile
4.3.2 Multimodal Interaction Solutions
4.4 VW-Mobvoi
4.4.1 Profile
4.4.2 OEM Intelligent Connected Solutions and Multimodal Interaction Principles
4.5 Tencent
4.5.1 Profile
4.5.2 Intelligent Cockpit Solutions
4.5.3 Interaction Technology Applications
4.5.4 Intelligent Voice Interaction System 
4.5.5 Multimodal Interaction System
4.6 Baidu
4.6.1 Profile
4.6.2 Voice Core Technology
4.6.3 Intelligent Cockpit Solutions
4.6.4 Multimodal Interaction Solutions
4.6.5 Intelligent Cockpit Partners
4.7 Banma
4.7.1 Profile
4.7.2 Intelligent Cockpit Solutions
4.7.3 Features of Banma Luoshen OS Interaction 
4.7.4 Voice Technology
4.7.5 Voice Assistant “XiaoGenBan”
4.7.6 Supporting OEMs
4.8 SenseTime
4.8.1 Profile
4.8.2 SenseAuto Platform
4.8.3 SenseAuto Cabin 
4.8.4 Major Customers
4.9 ArcSoft
4.9.1 Profile
4.9.2 In-cabin Monitoring Solutions
4.9.3 Business Models
4.9.4 Partners
4.10 Hikvision
4.10.1 Profile
4.10.2 Intelligent Cockpit Solutions
4.11 Horizon
4.11.1 Profile
4.11.2 Chip Product System
4.11.3 Multimodal Interaction Solutions
4.11.4 In-vehicle OS - TogetherOS
4.11.5 Product Development and Business Models
4.12 ADAYO
4.12.1 Profile
4.12.2 Business Layout
4.12.3 Multimodal Interaction System
4.12.4 Partners
4.12.5 Products and Supporting Customers
4.13 Desay SV
4.13.1 Profile
4.13.2 The Fourth Generation Cockpit Solution


China Automotive Fragrance and Air Purification Systems Research Report, 2023

Automotive fragrance and air purification systems: together to create a comfortable and healthy cockpitTechnology trend: intelligence of fragrance system and integration of air purification system In...

Global and China Solid State Battery Industry Report, 2023

Solid state battery research: semi-solid state battery has come out, is all-solid state battery still far away?In recent years, the new energy vehicle market has been booming, and the penetration of n...

Global and China Passenger Car T-Box Market Report, 2023

T-Box industry research: the market will be worth RMB10 billion and the integration trend is increasingly clear. ResearchInChina released "Global and China Passenger Car T-Box Market Report, 2023", w...

Analysis Report on Auto Shanghai 2023

Analysis on 75 Trends at Auto Shanghai 2023: Unprecedented Prosperity of Intelligent Cockpits and Intelligent Driving Ecology After analyzing the intelligent innovation trends at the Auto Shanghai 20...

Chinese Emerging Carmakers’ Telematics System and Entertainment Ecosystem Research Report, 2022-2023

Telematics service research (III): emerging carmakers work on UI design, interaction, and entertainment ecosystem to improve user cockpit experience. ResearchInChina released Chinese Emerging Carmake...

China Passenger Car Cockpit-Parking Industry Report, 2023

Cockpit-parking integration research: cockpit-parking vs. driving-parking, which one is the optimal solution for cockpit-driving integration?Cockpit-parking vs. driving-parking, which one is the optim...

Automotive Sensor Chip Industry Report, 2023

Sensor chip industry research: driven by the "more weight on perception" route, sensor chips are entering a new stage of rapid iterative evolution. At the Auto Shanghai 2023, "more weight on percepti...

Automotive Electronics OEM/ODM/EMS Industry Report, 2023

Automotive electronics OEM/ODM/EMS research: amid the disruption in the division of labor mode in the supply chain, which auto parts will be covered by OEM/ODM/EMS mode?  Consumer electronic manu...

China Automotive Smart Glass Research Report, 2023

Smart glass research: the automotive smart dimming canopy market valued at RMB127 million in 2022 has a promising future.Smart dimming glass is a new type of special optoelectronic glass formed by com...

Automotive Ultrasonic Radar and OEM Parking Roadmap Development Research Report, 2023

Automotive Ultrasonic Radar Research: as a single vehicle is expected to carry 7 units in 2025, ultrasonic radars will evolve to the second generation.    As a single vehicle is expec...

Autonomous Driving SoC Research Report, 2023

Research on autonomous driving SoC: driving-parking integration boosts the industry, and computing in memory (CIM) and chiplet bring technological disruption.  “Autonomous Driving SoC Research ...

China ADAS Redundant System Strategy Research Report, 2023

Redundant System Research: The Last Line of Safety for Intelligent VehiclesRedundant design refers to a technology adding more than one set of functional channels, components or parts that enable the ...

Intelligent Steering Key Components Report, 2023

Research on intelligent steering key components: four development trends of intelligent steering The automotive chassis consists of four major systems: transmission system, steering system, driving ...

Automotive Digital Instrument Cluster Operating System Report, 2023

Digital Instrument Cluster Operating System Report: QNX commanded 71% of the Chinese intelligent vehicle cluster operating system market. Amid the trend for the integration of digital cluster and cen...

800V High Voltage Platform Research Report, 2023

How to realize the commercialization of 800V will play a crucial part in the strategy of OEMs. As new energy vehicles and battery technology boom, charging and battery swapping in the new energy vehi...

Automotive Intelligent Cockpit Platform Research Report, 2023

Intelligent cockpit platform research: the boundaries between vehicles and PCs are blurring, and there are several feasible paths for cockpit platforms. Automotive Intelligent Cockpit Platform Resea...

Global and China Automotive Wireless Communication Module Industry Report,2023

Vehicle communication module research: 5G R16+C-V2X module, smart SiP module and other new products spring up.   In 2022, 4G modules swept 84.3% of the vehicle communication module market....

Intelligent Vehicle Cockpit-Driving Integration Research Report, 2023

Cockpit-Driving Integration Research: many companies are making layout and may implement it during 2024-2025.  1. What is the real cockpit-driving integration? At present, automotive electroni...

2005- All Rights Reserved 京ICP备05069564号-1 京公网安备1101054484号