Home>Manufacturing>Automotive
China Automotive Multimodal Interaction Development Research Report, 2024
  • Dec.2024
  • Hard Copy
  • USD $4,000
  • Pages:270
  • Single User License
    (PDF Unprintable)       
  • USD $3,800
  • Code: LYX011
  • Enterprise-wide License
    (PDF Printable & Editable)       
  • USD $5,700
  • Hard Copy + Single User License
  • USD $4,200
      

Multimodal interaction research: AI foundation models deeply integrate into the cockpit, helping perceptual intelligence evolve into cognitive intelligence

China Automotive Multimodal Interaction Development Research Report, 2024 released by ResearchInChina combs through the interaction modes of mainstream cockpits, the application of interaction modes in key vehicle models launched in 2024, and the cockpit interaction solutions of OEMs/suppliers, and summarizes the development trends of cockpit multimodal interaction fusion.

1. Voice recognition dominates cockpit interaction, and integrates with multiple modes to create a new interaction experience.

Among current cockpit interaction applications, voice interaction is used most widely and most frequently in intelligent cockpits. According to the latest statistics from ResearchInChina, from January to August 2024, the automate voice systems were installed in about 11 million vehicles, a year-on-year increase of 10.9%, with an installation rate of 83%. Li Tao, General Manager of Baidu Apollo's intelligent cockpit business, pointed out that "the frequency of people using cockpits has increased from 3-5 times a day at the beginning to double digits today, and has even reached nearly three digits on some models with leading voice interaction technology."  

The frequent use of voice recognition function not only greatly optimizes user interactive experience, but also promotes the development trend of fusing with other interactive modes such as touch and face recognition. For example, the full-cabin memory function of NIO Banyan 2.4.0 is based on face recognition, and NOMI actively greets occupants who have recorded information (e.g., "Good morning, Doudou"); Zeekr 7X integrates voice recognition with eye contact to enable the driver to see and speak to control, and tilt his/her head to control the car via voice. 

多模态1.png 

2. BYD launched palm vein recognition, and Sterra in-cabin health monitoring debuted

Compared with the mature interaction modes such as voice and face recognition, biometric technologies such as fingerprint, vein, and heart rate are still in the early stage of exploration and development, but they are gradually being mass-produced and applied. For example, BYD launched a palm vein recognition function in 2024, which can realize convenient vehicle unlocking; Genesis and Mercedes-Benz introduced fingerprint recognition systems in the 2025 Genesis GV70 and 2025 Mercedes-Benz EQE BEV respectively, allowing users to complete a range of operations such as identification, vehicle start and payment only with fingerprints; in addition, Exeed Sterra still uses visual perception technology provided by ArcSoft in new ET model, realizing in-cabin intelligent health monitoring function, and outputting health reports for users including five major physical indicators, i.e., heart rate, blood pressure, blood oxygen saturation, respiratory rate and heart rate variability.   

Introduction of biometric technology not only improves driving convenience, but also significantly enhances the safety protection performance of vehicles, effectively preventing potential safety hazards such as tired driving and car theft. In the future, these biometric technologies will be more widely integrated into the development of intelligent and connected vehicles, providing drivers with a safer and more personalized mobility experience.

Case 1: Fingerprint recognition system of Genesis 2025 GV70 allows users to quickly apply personalized settings (seats, positions, etc.) through fingerprint authentication, and also supports vehicle start/drive. In addition, there are personalized linkage functions such as easy to use, fingerprint payment, and valet mode.

多模态2.png

Case 2: BYD's palm vein recognition system uses a camera to read palm vein data for recognition at a distance of 8-20cm, 360 degrees horizontally and 15 degrees vertically. It uses professional image acquisition module to obtain images of vein patterns, extracts characteristics through algorithms and stores them, and finally realizes identification and recognition. In the future, it may be first installed in high-end brand Yangwang models.

多模态3.png

Case 3: Exeed Sterra ET model is equipped with DHS intelligent health monitoring function. Based on advanced visual multimodal algorithm, it can analyze health status in real time according to the surface of the human body, measure the five major physical indicators of heart rate, blood pressure, blood oxygen saturation, respiratory rate and heart rate variability, and output a health report.

多模态4.png

3. AI foundation models lead cockpit interaction innovation, and perceptual intelligence evolves into cognitive intelligence

China Society of Automotive Engineers clearly defines and classifies intelligent cockpits in its jointly released white paper. The classification system is based on capabilities achieved by intelligent cockpits, comprehensively considers the three dimensions of human-machine interaction capabilities, scenario expansion capabilities, and connected service capabilities, and subdivides intelligent cockpits into five levels from L0 to L4.  

With the wide adoption of AI foundation models in intelligent cockpits, HMI capabilities have crossed the boundary of L1 perceptual intelligence and entered a new stage of L2 cognitive intelligence.

Specifically, in the stage of perceptual intelligence, intelligent cockpit mainly relies on the in-cabin sensor system, such as cameras, microphones and touch screens, to capture and identify the behavior, voice and gesture information of driver and passengers, and then convert the information into machine-recognizable data. However, limited by established rules and algorithm framework, the cockpit interaction system in this stage still lacks the capability of independent decision and self-optimization, which is mainly reflected in the passive response to input information.

After entering the cognitive intelligence stage, intelligent cockpits can comprehensively analyze multiple data types such as voice, vision and touch by virtue of powerful multimodal processing capabilities of foundation model technology. This feature makes intelligent cockpits highly intelligent and humanized, able to actively think and serve, as well as keenly perceive actual needs of the driver and passengers, providing users with personalized HMI services. perceives

多模态5.png

Case 1: SenseAuto introduced an intelligent cockpit AI foundation model product, A New Member For U, at the 2024 SenseAuto AI DAY. It can be regarded as the "Jarvis" on the vehicle, which can weigh up occupants’ words and observe their expressions, actively think, serve, and plan. For example, on the road, it can actively turn up the air conditioner temperature and lower music volume for the sleeping children in the rear seat, and adjust the chassis and driving mode to the comfort mode to create a more comfortable sleeping environment. In addition, it can actively detect the physical condition of occupants, find the nearest hospital for the sick ones, and plan the route.

多模态6.png

Case 2: NOMI Agents, NIO's multi-agent framework, uses AI foundation models to reconstruct NOMI's cognition and complex task processing capabilities, allowing it to learn to use tools, for example, calling search, navigation, and reservation services. Meanwhile, according to complexity and time span of the task, NOMI is able to perform complex planning and scheduling. For example, among NOMI's six core multi-agent functions, "NOMI DJ" recommends a playlist that suits the context to users based on their needs, and actively creates an atmosphere; "NOMI Exploration" understands based on spatial orientation, matches map data and world knowledge, and answers children's questions, for example, "what is the tower on the side?". 

多模态7.png

1 Overview of Cockpit Multimodal Interaction
1.1 Definition of Multimodal Interaction
1.2 Multimodal Interaction Development System
1.3 Multimodal Interaction Industry Chain
1.3.1 Multimodal Interaction Industry Chain - Chip Vendors
1.3.2 Multimodal Interaction Industry Chain - Algorithm Providers
1.3.3 Multimodal Interaction Industry Chain - System Integrators
1.4 Multimodal Interaction Policy Environment
1.4.1 Summary of Laws and Regulations Related to Network Data Security of Intelligent Connected Vehicle
1.4.2 Multimodal Interaction Laws and Regulations (1)
1.4.2 Multimodal Interaction Laws and Regulations (2)
1.4.2 Multimodal Interaction Laws and Regulations (3)

2 Cockpit Single-modal Interaction
2.1 Installation of Cockpit Modal Interaction System 
2.1.1 Installations & Installation Rate of In-vehicle Voice Recognition, 2024
2.1.2 Installations & Installation Rate of In-vehicle Voiceprint Recognition, 2024
2.1.3 Installations & Installation Rate of Exterior Voice Recognition, 2024
2.1.4 Installations & Installation Rate of In-vehicle Gesture Recognition, 2024
2.1.5 Installations & Installation Rate of In-vehicle Face Recognition (FACE ID), 2024
2.1.6 Installations & Installation Rate of In-vehicle DMS, 2024
2.1.7 Installations & Installation Rate of In-vehicle OMS, 2024
2.2 Haptic Interaction
2.2.1 Haptic Interaction Development Route
2.2.2 Application Cases of Haptic Interaction in Vehicle Models
2.2.3 Haptic Feedback Technology
2.2.4 Summary of Haptic Interaction Suppliers
2.3 Auditory Interaction
2.3.1 Voice Recognition Development Route
2.3.2 Application Cases of Voice Recognition in Vehicle Models
2.3.3 Application Cases of Voiceprint Recognition in Vehicle Models
2.3.4 Application Cases of External Voice Recognition in Vehicle Models
2.3.5 Summary of Voice Interaction Suppliers
2.4 Visual Interaction
2.4.1 Gesture Recognition Development Route
2.4.2 Application Cases of Gesture Recognition in Vehicle Models
2.4.3 Facial Recognition Development Route
2.4.4 Application Cases of Face Recognition in Vehicle Models
2.4.5 Application Case of Line of Sight Recognition Vehicle Models
2.4.6 Application Case of Lip Movement Recognition Vehicle Models
2.4.7 Summary of Visual Interaction Suppliers (1) - Gesture Recognition
2.4.7 Summary of Visual Interaction Suppliers (2) - Face Recognition
2.4.7 Summary of Visual Interaction Supplier (3) - Lip Movement Recognition
2.5 Olfactory Interaction
2.5.1 Olfactory Interaction Development Route
2.5.2 Application Cases of Olfactory Interaction in Vehicle Models
2.5.3 Summary of Automotive Smart Fragrance/Air Purification Suppliers
2.6 Other Biometric Functions
2.6.1 Iris Recognition Development Route
2.6.2 Application Case of Iris Recognition Vehicle Models
2.6.3 Iris Recognition AR/VR Applications
2.6.4 Solutions of Iris Recognition Suppliers 
2.6.5 Summary of Iris Recognition Suppliers
2.6.6 Fingerprint Recognition Development Route
2.6.7 Application Cases of Fingerprint Recognition in Vehicle Models
2.6.8 Summary of Fingerprint Recognition Suppliers
2.6.9 Vein Recognition Development Route
2.6.10 Application Cases of Vein Recognition in Vehicle Models
2.6.11 Summary of Vein Recognition Suppliers
2.6.12 Heart Rate Recognition Development Route
2.6.13 Application Case of Heart Rate Recognition Vehicle Models
2.6.14 Summary of Heart Rate Recognition Suppliers
2.6.15 Electromyography Recognition Development Route
2.6.16 Introduction to Electromyography Recognition Equipment
2.6.17 Application of Electromyography Recognition Vehicle Models 
2.6.18 Summary of Electromyography Recognition Suppliers 

3 Cockpit Multimodal Interaction Solutions of OEMs
3.1 SAIC
3.1.1 Z-ONE Galaxy Full-stack Solution
3.1.2 Rising Intelligent Cockpit Solution
3.1.3 IM Intelligent Cockpit Solution
3.1.4 IM Generative Foundation Model
3.1.5 Multimodal Interaction OTA Content Summary (1): Rising Auto
3.1.5 Multimodal Interaction OTA Content Summary (2): IM Motors

3.2 BYD
3.2.1 Intelligent cockpit Solution
3.2.2 In-cabin Unique Multimodal Interactive Applications
3.2.3 Xuanji AI Foundation Model
3.2.4 Multimodal Interaction OTA Content Summary (1): BYD Dynasty & Ocean
3.2.4 Multimodal Interaction OTA Content Summary (2): Denza
3.2.4 Multimodal Interaction OTA Content Summary (3): Fangchengbao & Yangwang

3.3 Changan Automobile
3.3.1 Changan Intelligent Cockpit Solution
3.3.2 Nevo Intelligent Cockpit Solution
3.3.3 Deepal Intelligent Cockpit Solution
3.3.4 Avatr Intelligent Cockpit Solution
3.3.5 Automotive Foundation Model: Xinghai Model
3.3.6 Multimodal Interaction OTA Content Summary (1): Changan
3.3.6 Multimodal Interaction OTA Content Summary (2): Avatr
3.3.6 Multimodal Interaction OTA Content Summary (3): Deepal

3.4 GAC
3.4.1 Intelligent Cockpit Solution
3.4.2 ADiGO SENSE AI Foundation Model
3.4.3 Multimodal Interaction OTA Content Summary

3.5 Geely
3.5.1 Geely Intelligent Cockpit Solution
3.5.2 Zeekr Intelligent Cockpit Solution
3.5.3 Jiyue Intelligent Cockpit Solution
3.5.4 Xingrui AI Foundation Model
3.5.5 Kr AI Foundation Model
3.5.6 Multimodal Interaction OTA Content Summary (1): Geely
3.5.6 Multimodal Interaction OTA Content Summary (2): Zeekr
3.5.6 Multimodal Interaction OTA Content Summary (3): Jiyue

3.7 NIO
3.7.1 Intelligent Cockpit Solution
3.7.2 ONVO Intelligent Cockpit Solution
3.7.3 In-cabin Unique Multimodal Interactive Applications
3.7.4 Multimodal Perception Model: NOMI GPT
3.7.5 Multimodal Interaction OTA Content Summary

3.8 Xpeng Motors
3.8.1 Intelligent Cockpit Solution
3.8.2 In-cabin Unique Multimodal Interactive Applications
3.8.3 Automotive Large Language Model: XGPT
3.8.4 Multimodal Interaction OTA Content Summary

3.9 Li Auto
3.9.1 Intelligent cockpit Solution
3.9.2 In-cabin Unique Multimodal Interactive Applications
3.9.3 Intelligent Cockpit
3.9.4 Multimodal Interaction OTA Content Summary

3.10 Leapmotor
3.10.1 Intelligent Cockpit Solution (1)
3.10.1 Intelligent Cockpit Solution (2)
3.10.2 Voice Foundation Model: Tongyi
3.10.3 Multimodal Interaction OTA Content Summary

3.11 Xiaomi Auto
3.11.1 Intelligent Cockpit Solution
3.11.2 Car-side Large Model: MiLM
3.11.3 Sound Foundation Model is Installed in Cars
3.11.4 Multimodal Interaction OTA Content Summary (1)
3.11.4 Multimodal Interaction OTA Content Summary (2)

3.12 BMW
3.12.1 Intelligent Cockpit Solution (1)
3.12.1 Intelligent Cockpit Solution (2)
3.12.2 In-cabin Unique Multimodal Interactive Applications

3.13 Mercedes-Benz
3.13.1 Intelligent Cockpit Solution
3.13.2 In-cabin Unique Multimodal Interactive Applications
3.13.3 Cooperation Dynamics of Cockpit Foundation Model 

3.14 Volkswagen
3.14.1 Intelligent Cockpit Solution
3.14.2 Upgrade Trends of Haptic Interaction System
3.14.3 Upgrade Trends of Voice Interaction System

4 Cockpit Multimodal Interaction Solutions of Suppliers
4.1 Desay SV
4.1.1 Profile
4.1.2 Multimodal Interaction Solution (1)
4.1.2 Multimodal Interaction Solution (2)

4.2 Joyson Electronics
4.2.1 Profile
4.2.2 Evolution of Joynext Intelligent Cockpit
4.2.3 Multimodal Interaction Layout
4.2.4 Features of Joynext Intelligent Cockpit Interaction (1)
4.2.4 Features of Joynext Intelligent Cockpit Interaction (2)

4.3 SenseTime
4.3.1 I Profile
4.3.2 SenseAuto Intelligent Cockpit Product System
4.3.3 SenseAuto Intelligent Cockpit Products
4.3.4 SenseNova Model Empowers Cockpit Interaction
4.3.5 SenseAuto Multimodal Interaction Application Case

4.4 iFLYTEK
4.4.1 Profile
4.4.2 Full-Stack Intelligent Interaction Technology
4.4.3 Features of Multimodal Perception System
4.4.4 Spark Cognitive Foundation Model
4.4.5 Spark Foundation Model Enables Cockpit Interaction
4.4.6 Multimodal Interaction Becomes the Key Direction of iFlytek Super Brain 2030 Plan

4.5 ThunderSoft
4.5.1 Profile
4.5.2 Cockpit Interaction Features
4.5.3 Rubik Model Enables Cockpit Interaction
4.5.4 Vehicle Operating System

4.6 AISpeech
4.6.1 Profile
4.6.2 Features of Multimodal Interaction Solution
4.6.3 Multimodal Interaction Products
4.6.4 Language Foundation Model

4.7 Huawei
4.7.1 Profile
4.7.2 Multimodal Interaction History
4.7.3 Harmony OS 4.0 Intelligent cockpit
4.7.4 New-generation HarmonySpace cockpit
4.7.5 HarmonySpace Interaction Features (1)
4.7.5 HarmonySpace Interaction Features (2)
4.7.5 HarmonySpace Interaction Features (3)
4.7.5 HarmonySpace Interaction Features (4)
4.7.5 HarmonySpace Interaction Features (5)
4.7.5 HarmonySpace Interaction Features (6)
4.7.6 HarmonyOS NEXT Interaction Features
4.7.7 Pangu Foundation Model

4.8 Baidu
4.8.1 Profile
4.8.2 Interaction Features of AI Native Operating System
4.8.3 ERNIE Bot Empowers Baidu Smart Cabin
4.8.4 Interaction Features of Baidu Smart Cabin Model 2.0

4.9 Tencent
4.9.1 Profile
4.9.2 Cockpit Interaction Features (1)
4.9.2 Cockpit Interaction Features (2)

4.10 NavInfo
4.10.1 Profile
4.10.2 Cockpit Interaction Features
4.10.3 Introduction to AutoChips
4.10.4 Intelligent Cockpit Domain Control SoC Chip of AutoChips
4.10.5 Application of AutoChips In-cabin Monitoring Function

4.11 Continental
4.11.1 Profile
4.11.2 Multimodal Product Layout
4.11.3 Cockpit Interaction Features
4.11.4 Multimodal Interaction Products (1)
4.11.4 Multimodal Interaction Products (2)

4.12 MediaTek
4.12.1 Profile
4.12.2 Cockpit Interaction Features

5 Application Cases of Multimodal Interaction Solutions in Benchmarking Vehicle Models
5.1 Cases of Traditional Brands
5.1.1 Yangwang U9
5.1.2 IM L6
5.1.3 Geely Galaxy E8
5.1.4 Zeekr 7X
5.1.5 Jiyue 07
5.1.6 Changan UNI-Z
5.1.7 Changan Deepal G318
5.1.8 Avatr 07
5.1.9 Dongfeng eπ007
5.1.10 ARCFOX αS5
5.1.11 Exeed Sterra ET
5.2 Cases of Emerging Brands
5.2.1 Xiaomi SU7
5.2.2 Luxeed R7
5.2.3 STELATO S9
5.2.4 Li Auto MEGA Ultra
5.2.5 Xpeng MONA 03
5.2.6 ONVO L60
5.2.7 Leapmotor C16
5.3 Cases of Joint Venture Brands
5.3.1 Volvo EX30
5.3.2 Lotus EMEYA
5.3.3 2024 Buick E5
5.3.4 2025 BMW i4
5.3.5 2025 Mercedes-Benz All-electric EQE
5.3.6 2025 Genesis GV70
 
6 Summary and Development Trends of Multimodal Interaction
6.1 Fusion Application of Multimodal Interaction in Intelligent Cockpits
6.2 Trend 1
6.3 Trend 2 (1): Cockpit Interaction Carriers Expand, and Interaction Range Extends outside the Vehicle 
6.3 Trend 2 (2)
6.3 Trend 2 (3)
6.4 Trend 3 (1)
6.4 Trend 3 (2)
6.4 Trend 3 (3)
6.5 Trend 4
 

Automotive MEMS (Micro Electromechanical System) Sensor Research Report, 2025

Automotive MEMS Research: A single vehicle packs 100+ MEMS sensors, and the pace of product innovation and localization are becoming much faster. MEMS (Micro Electromechanical System) is a micro devi...

Intelligent Vehicle Cockpit-driving Integration (Cockpit-driving-parking) Industry Report, 2024-2025

Cockpit-driving integration is gaining momentum, and single-chip solutions are on the horizon   The Intelligent Vehicle Cockpit-driving Integration (Cockpit-driving-parking) Industry Repor...

Automotive TSP and Application Service Research Report, 2024-2025

TSP Research: In-vehicle connectivity services expand in the direction of cross-domain integration, all-scenario integration and cockpit-driving integration TSP (Telematics Service Provider) is mainl...

Autonomous Driving Domain Controller and Central Control Unit (CCU) Industry Report, 2024-2025

Autonomous Driving Domain Controller Research: One Board/One Chip Solution Will Have Profound Impacts on the Automotive Supply Chain Three development stages of autonomous driving domain controller:...

Global and China Range Extended Electric Vehicle (REEV) and Plug-in Hybrid Electric Vehicle (PHEV) Research Report, 2024-2025

Research on REEV and PHEV: Head in the direction of high thermal efficiency and large batteries, and there is huge potential for REEVs to go overseas In 2024, hybrid vehicles grew faster than batter...

Automotive AI Agent Product Development and Commercialization Research Report, 2024

Automotive AI Agent product development: How to enable “cockpit endorser” via foundation models? According to OPEN AI’s taxonomy of AI (a total of 5 levels), AI Agent is at L3 in the AI development ...

China ADAS Redundant System Strategy Research Report, 2024

Redundant system strategy research: develop towards integrated redundant designADAS redundant system definition framework For autonomous vehicles, safety is the primary premise. Only when ADAS is ful...

Smart Car OTA Industry Report, 2024-2025

Automotive OTA research: With the arrival of the national mandatory OTA standards, OEMs are accelerating their pace in compliance and full life cycle operations The rising OTA installations facilitat...

End-to-end Autonomous Driving Industry Report, 2024-2025

End-to-end intelligent driving research: How Li Auto becomes a leader from an intelligent driving follower There are two types of end-to-end autonomous driving: global (one-stage) and segmented (two-...

China Smart Door and Electric Tailgate Market Research Report, 2024

Smart door research: The market is worth nearly RMB50 billion in 2024, with diverse door opening technologies  This report analyzes and studies the installation, market size, competitive landsc...

Commercial Vehicle Intelligent Chassis Industry Report, 2024

Commercial vehicle intelligent chassis research: 20+ OEMs deploy chassis-by-wire, and electromechanical brake (EMB) policies are expected to be implemented in 2025-2026 The Commercial Vehicle Intell...

Automotive Smart Surface Industry Report, 2024

Research on automotive smart surface: "Plastic material + touch solution" has become mainstream, and sales of smart surface models soared by 105.1% year on year In this report, smart surface refers t...

China Automotive Multimodal Interaction Development Research Report, 2024

Multimodal interaction research: AI foundation models deeply integrate into the cockpit, helping perceptual intelligence evolve into cognitive intelligence China Automotive Multimodal Interaction Dev...

Automotive Vision Industry Report, 2024

Automotive Vision Research: 90 million cameras are installed annually, and vision-only solutions lower the threshold for intelligent driving. The cameras installed in new vehicles in China will hit 90...

Automotive Millimeter-wave (MMW) Radar Industry Report, 2024

Radar research: the pace of mass-producing 4D imaging radars quickens, and the rise of domestic suppliers speeds up. At present, high-level intelligent driving systems represented by urban NOA are fa...

Chinese Independent OEMs’ ADAS and Autonomous Driving Report, 2024

OEM ADAS research: adjust structure, integrate teams, and compete in D2D, all for a leadership in intelligent driving  In recent years, China's intelligent driving market has experienced escala...

Research Report on Overseas Layout of Chinese Passenger Car OEMs and Supply Chain Companies, 2024

Research on overseas layout of OEMs: There are sharp differences among regions. The average unit price of exports to Europe is 3.7 times that to Southeast Asia. The Research Report on Overseas Layou...

In-vehicle Payment and ETC Market Research Report, 2024

Research on in-vehicle payment and ETC: analysis on three major application scenarios of in-vehicle payment In-vehicle payment refers to users selecting and purchasing goods or services in the car an...

Contact us | Copyright | Privacy policy | Buy Reports | Member Service | Mailing List
2005- www.researchinchina.com All Rights Reserved 京ICP备05069564号-1 京公网安备1101054484号