China Automotive Multimodal Interaction Development Research Report, 2023
China Automotive Multimodal Interaction Development Research Report, 2023 released by ResearchInChina combs through the interaction modes of mainstream cockpits, the application of interaction modes in key vehicle models launched in 2023, the cockpit interaction solutions of suppliers, and the multimodal interaction fusion trends.
By sorting out the interaction modes and functions of new models rolled out in the previous year, it can be seen that active, anthropomorphic and natural interaction has become the main trend. In terms of interaction mode, in single-modal interaction, the control scope of mainstream interactions such as touch and voice has expanded from inside to outside cars, and the application cases of novel interactions like fingerprint and electromyography in cars have begun to increase; in multimodal fusion interaction, multiple fusion interactions, for example, voice + head posture/face/lip language, and face + emotion/smell, are being available to cars, aiming to create more active and natural human-vehicle interaction.
1. Single-modal interaction develops in depth.
Haptic interaction: cockpits more tend to have large and multiple screens. The wider application of smart surface materials in cockpits also allows for extension of the haptic sensing scope to doors, windows, seats and other components, and haptic feedback technology is gradually introduced;
Voice interaction: enabled by large AI models, the voice interaction function becomes more intelligent and emotional. The introduction of lip movement recognition, voiceprint recognition and other technologies into cars brings higher accuracy of voice interaction and expands the control scope from inside to outside cars;
Visual interaction: the scope of face/gesture recognition based on visual technology begins to expand to body recognition, including head posture, arm movements, and body actions, etc.;
Olfactory interaction: the olfactory interaction function, which was originally often used to purify the air and remove odors, can now enable cockpit sterilization and disinfection, and supports the linkage of the fragrance system with cockpit scenes/seasons.
Case 1: car control by voice extends from inside to outside cars.
Typical models: Changan Nevo A07, Jiyue 01
Typical functions: voice outside the car to control doors, windows, parking assist, etc.
Changan Nevo A07 adopts iFlytek's latest technology XTTS 4.0. The voice of the car voice assistant is more natural and anthropomorphic, and can express multiple emotions such as happiness, regret, and confusion. It supports saying towards the outside of the car (the content can be user-defined). In addition, the trunk, windows, music, air conditioning, pull-out/parking and other functions can also be controlled by voice outside the car.
Equipped with "SIMO" voice assistant, Jiyue 01 supports fully offline voice control in all zones, and allows for online voice interaction in the full process with weak network or without network. It enables recognition in 500 milliseconds and response in 700 milliseconds. Outside the car, the voiceprint recognition technology allows the driver and passengers to voice to operate air conditioning audio, lights, windows, doors, rear tailgate, charging cover and other functions, and supports voice parking outside the car.
![交互 1.png](/UpLoads/Article/2023/交互%201.png)
Case 2: voiceprint recognition finds wider application.
Typical models: Li L7, Hycan A06/V09
Typical functions: identify drivers and passengers to provide targeted services
All Li Auto’s L series models support voiceprint recognition function. After passengers register their voiceprints, "Lixiang Classmate" can identify who the passenger is, call the nicknames designated by different passengers, and perform vehicle control according to the positions of different passengers memorized via their voiceprint.
The voiceprint recognition VOICE ID of Hycan A06/V09 can clearly identify valid users and commands, and will become the entrance to HYCAN ID, allowing users to access rich smart ecosystems and use 100+ entertainment applications. Moreover based on voiceprint recognition technology, the system will actively block other disturbing sounds to improve the accuracy of recognition at the driver’s seat.
![交互 2.png](/UpLoads/Article/2023/交互%202.png)
Case 3: myoelectric interaction comes into commercial use in cars.
Typical model: Voyah Passion
Typical function: micro-gesture control inside and outside the car
In April 2023, Voyah Passion and FlectoThink introduced a myoelectric interaction fusion solution enabled through a myoelectric bracelet. A multi-channel myoelectric sensor and a high-precision amplifier that are installed inside the bracelet can collect rich myoelectric signals in real time and generate algorithms, and transmit them to the computing terminal to generate a personalized AI gesture model, which is then integrated with Voyah’s vehicle platforms. By connecting the bracelet with in-car Bluetooth, users can control the car with micro-gestures, including 60+ gestures to control the trunk and windows, for example. Additionally the bracelet can also be seamlessly connected to the car gaming system. The gesture recognition feature of the myoelectric bracelet allows users to control characters of games (e.g., Subway Surfers) more naturally and intuitively.
![交互 3.png](/UpLoads/Article/2023/交互%203.png)
![交互 4.png](/UpLoads/Article/2023/交互%204.png)
2. Multimodal fusion creates active interaction.
Currently multimodal fusion enabled by automakers includes but is not limited to voice + lip motion recognition, voice + face recognition, voice + gesture recognition, voice + head posture, face + emotion recognition, face + eye tracking, and fragrance + face + voice recognition. Wherein multimodal voice interaction is mainstream, and supports models mentioned above, like Changan Nevo A07, Jiyue 01, Li L7, and Hycan A06/V09.
Case 1: voice + head posture interaction: WEY Blue Mountain DHT PHEV combines voice and head posture, offering a simple and intuitive interaction mode.
When the driver engages in a voice conversation, the camera in the cockpit of Blue Mountain captures the driver's head movements, and allows the driver to give yes/no reply by nodding/shaking head. For example, when voicing to control navigation, the driver can select a planned route scheme by nodding/shaking head.
![交互 5.png](/UpLoads/Article/2023/交互%205.png)
Case 2: face + emotion recognition: LIVAN 7 and ARCFOX Kaola among other models integrate emotion recognition technology into the face recognition function to provide active interaction and enhance interaction experience.
The multimodal intelligent recognition Face-ID system of LIVAN 7 supports lip movement recognition and emotion recognition, and can remember the personalized settings of vehicle functions such as voice, seats, rearview mirrors, ambient light and trunk, that correspond to the associated accounts. It can also select the appropriate music according to the user’s expression.
Directly facing the rear row, the camera on the B-pillar of ARCFOX Kaola can monitor a child in real time. For example, when the child smiles, a snapshot will be taken automatically and sent to the center console screen; when the child cries, soothing music will be automatically played and the surface of the smart seat will make a respiratory rhythm to calm him/her down. In addition, the camera can also be linked with the in-car radar to determine whether the child is asleep or not. If the child is asleep, the sleep mode will be automatically opened, the seat ventilation will be turned on, the air-conditioning temperature will be adjusted appropriately, and the audio and ambient lighting will be linked, producing a rhythmic effect.
![交互 6.png](/UpLoads/Article/2023/交互%206.png)
Case 3: face + smell: NIO EC7, LIVAN 7 and other models realize the linkage between the driver monitoring system and the fragrance system to improve driving safety.
When NIO EC7 detects the driver's tiredness, it will automatically release a refreshing fragrance to ensure driving safety;
When the camera on the A-pillar of LIVAN 7 detects a drowsy driver, it will automatically release a refreshing fragrance and give a voice prompt.
3. Foundation models and multimodal fusion will facilitate the introduction of AI Agent into cars.
Large AI models are evolving from the single-modal to the multi-modal and multi-task fusion. Compared with the single-modal that can only process one type of data such as text, image and speech, the multimodal can process and understand multiple types of data, including vision, hearing and language, thus better understanding and generating complex information.
As multimodal foundation models continue to develop, their capabilities will also be significantly improved. This improvement gives AI Agent higher capabilities of perception and environment understanding to achieve more intelligent, automatic decisions and actions, and also creates new possibilities for its application in automotive, providing a broader prospect for future intelligent development.
![交互 7.png](/UpLoads/Article/2023/交互%207.png)
The Spark Cockpit OS developed by iFlytek based on the Spark Model supports multiple interaction modes such as voice, gesture, eye tracking and DMS/OMS. The Spark Car Assistant enables multi-intent recognition by deep understanding of the context, providing more natural human-machine interaction. The iFlytek Spark Model, first mounted on the model EXEED Sterra ES, will bring five new experiences: Vehicle Function Tutor, Empathy Partner, Knowledge Encyclopedia, Travel Planning Expert, and Physical Health Consultant.
![交互 8.png](/UpLoads/Article/2023/交互%208.png)
AITO M9, to be launched in December 2023, has HarmonyOS 4 IVI system built in. Xiaoyi, the intelligent assistant in HarmonyOS 4, has been connected to Huawei Pangu Model, which includes natural language model, visual model, and multi-modal model. The combination of HarmonyOS 4 + Xiaoyi + Pangu Model further enhances ecosystem capabilities such as cooperation of devices, and AI scenarios, and provides diverse interaction modes, including voice recognition, gesture control, and touch control, using multimodal interaction technology.
Chinese OEMs (Passenger Car) Going Overseas Report, 2024--Germany
Keywords of Chinese OEMs going to Germany: electric vehicles, cost performance, intelligence, ecological construction, localization
The European Union's temporary tariffs on electric vehicles in Chi...
Analysis on DJI Automotive’s Autonomous Driving Business, 2024
Research on DJI Automotive: lead the NOA market by virtue of unique technology route.
In 2016, DJI Automotive’s internal technicians installed a set of stereo sensors + vision fusion positioning syst...
BYD’s Layout in Electrification, Connectivity, Intelligence and Sharing and Strategy Analysis Report, 2023-2024
Insight: BYD deploys vehicle-mounted drones, and the autonomous driving charging robot market is expected to boom.
BYD and Dongfeng M-Hero make cross-border layout of drones.
In recent years,...
Great Wall Motor’s Layout in Electrification, Connectivity, Intelligence and Sharing and Strategy Analysis Report, 2023-2024
Great Wall Motor (GWM) benchmarks IT giants and accelerates “Process and Digital Transformation”.
In 2022, Great Wall Motor (GWM) hoped to use Haval H6's huge user base to achieve new energy transfo...
Cockpit AI Agent Research Report, 2024
Cockpit AI Agent: Autonomous scenario creation becomes the first step to personalize cockpits
In AI Foundation Models’ Impacts on Vehicle Intelligent Design and Development Research Report, 2024, Res...
Leading Chinese Intelligent Cockpit Tier 1 Supplier Research Report, 2024
Cockpit Tier1 Research: Comprehensively build a cockpit product matrix centered on users' hearing, speaking, seeing, writing and feeling.
ResearchInChina released Leading Chinese Intelligent Cockpit ...
Global and China Automotive Wireless Communication Module Market Report, 2024
Communication module and 5G research: 5G module installation rate reaches new high, 5G-A promotes vehicle application acceleration
5G automotive communication market has exploded, and 5G FWA is evolv...
ADAS and Autonomous Driving Tier 1 Suppliers Research Report, 2024 – Chinese Companies
ADAS Tier1s Research: Suppliers enter intense competition while exploring new businesses such as robotics
In China's intelligent driving market, L2 era is dominated by foreign suppliers. Entering era...
Automotive Gateway Industry Report, 2024
Automotive gateway research: 10BASE-T1S and CAN-XL will bring more flexible gateway deployment solutions
ResearchInChina released "Automotive Gateway Industry Report, 2024", analyzing and researching...
Global and China Electronic Rearview Mirror Industry Report, 2024
Research on electronic rearview mirrors: electronic internal rearview mirrors are growing rapidly, and electronic external rearview mirrors are facing growing pains
ResearchInChina released "Global a...
Next-generation Zonal Communication Network Topology and Chip Industry Research Report, 2024
The in-vehicle communication architecture plays a connecting role in automotive E/E architecture. With the evolution of automotive E/E architecture, in-vehicle communication technology is also develop...
Autonomous Delivery Industry Research Report, 2024
Autonomous Delivery Research: Foundation Models Promote the Normal Application of Autonomous Delivery in Multiple Scenarios
Autonomous Delivery Industry Research Report, 2024 released by ResearchInCh...
Global Autonomous Driving Policies & Regulations and Automotive Market Access Research Report, 2024
Intelligent driving regulations and vehicles going overseas: research on regional markets around the world and access strategies. "Going out”: discussion about regional markets aroun...
China Passenger Car HUD Industry Report, 2024
HUD research: AR-HUD accounted for 21.1%; LBS and optical waveguide solutions are about to be mass-produced. The automotive head-up display system (HUD) uses the principle of optics to display s...
Ecological Domain and Automotive Hardware Expansion Research Report, 2024
Automotive Ecological Domain Research: How Will OEM Ecology and Peripheral Hardware Develop? Ecological Domain and Automotive Hardware Expansion Research Report, 2024 released by ResearchInChina ...
C-V2X and CVIS Industry Research Report, 2024
C-V2X and CVIS Research: In 2023, the OEM scale will exceed 270,000 units, and large-scale verification will start.The pilot application of "vehicle-road-cloud integration” commenced, and C-V2X entere...
Automotive Intelligent Cockpit Platform Configuration Strategy and Industry Research Report, 2024
According to the evolution trends and functions, the cockpit platform has gradually evolved into technical paths such as cockpit-only, cockpit integrated with other domains, cockpit-parking integratio...
Analysis on Huawei's Electrification, Connectivity, Intelligence and Sharing,2023-2024
Analysis on Huawei's Electrification, Connectivity, Intelligence and Sharing: Comprehensive layout in eight major fields and upgrade of Huawei Smart Selection
The “Huawei Intelligent Driving Business...