China Automotive Multimodal Interaction Development Research Report, 2021
Our China Automotive Multimodal Interaction Development Research Report, 2021 combs through and summarizes the interaction modes of mainstream auto brands.
The current automotive human-computer interaction (HMI) is characterized by the following:
1. In terms of interaction modes, in addition to conventional interaction methods such as touch control, button and knob, voice and mobile phone have become favored options in automotive HMI. Moreover, face, gesture, and car lamp have also become popular interaction options. Some brands even have begun to try new interaction ways, such as eye tracking, voiceprint, and fingerprint.
2. As concerns layout, joint-venture brands remain conservative in interaction modes. They seldom use new interaction methods except for voice and mobile phone. Yet luxury brands are pioneers of multimodal interaction. One example is Mercedes-Benz which has equipped its new-generation MBUX system with multiple new interactions like face, gesture, eye, and fingerprint. By comparison, Chinese leading brands are more active in new interaction modes, for example, Changan Auto has eyesight wake-up and Great Wall Motor offers voiceprint recognition.
Some HMI modes of mainstream auto brands inside and outside China:
![多模态1_副本.png](/UpLoads/Article/2021/多模态1_副本.png)
Voice as a conventional interaction mode offers improvements in functions.
Besides the original capabilities of voice interaction, features like continuous dialogue, voice area lock, tone switch, and user-defined voice have become next orientations of mainstream brands.
1. Continuous dialogue and voice area lock have been used by several brands.
After a wake-up, a voice interaction system with the continuous dialogue capability needs no repeated wake-up within a certain period of time, which makes the response to instructions more coherent; the voice area lock feature with the ability to locate the voice command issuer, executes commands like “open window” and “adjust air conditioning temperature” more accurately. The brands using the two features include Geely, Changan, Great Wall, Chery, SAIC Roewe, GAC Trumpchi, Xpeng, NIO and Lixiang.
Geely Xingyue L - the “driver geek mode” of Galaxy OS frees the driver of wake-up in 24 hours, and enables lock of four voice areas.
![多模态2_副本.png](/UpLoads/Article/2021/多模态2_副本.png)
Xpeng Motors – the voice assistant Little P delivers voice functions in all scenarios and allows 30s continuous dialogue and dual-voice area lock.
![多模态3_副本.png](/UpLoads/Article/2021/多模态3_副本.png)
2. Tone switch refers to the availability of other speech options the voice system provides for users in addition to default voices and sounds, such as, voices of celebrities and game characters. The brands with this feature include Geely, Changan, Great Wall, BYD and Roewe, as well as emerging carmakers like NIO, Xpeng and Lixiang. For instance, Banma Zhixing Venus System mounted on SAIC Roewe RX5 offers five tone options: affinity female voice, intellectual female voice, magnetic male voice, innocent child voice, and passionate female voice.
3. User-defined voice feature allows customization of voice tones and voice functions.
“Voice tone customization” means that the system applies the uploaded voices of the user’s own or his/her families’ in scenarios such as voice response and navigation broadcast. Models carrying this capability include Geely, Great Wall, SAIC Roewe, GAC Trumpchi and R Car.
“Voice function customization” means that users can define a voice command as well as corresponding operation, which brings the voice system into full play and meets the personalized needs of users. The typical brand using this capability is Xpeng Motors.
Geely Xingyue L - Galaxy OS allows users to define voice. The customized human voice is available to all scenarios including navigation and response.
Xpeng Motors - Little P Customization allows Xpeng car owners to define voice commands and corresponding operation via their smartphone APP.
Gesture has become the third new interaction mode behind voice and face.
Gesture interaction uses the in-car camera to judge a gesture and send the data to the vehicle system control unit which will then call out the function corresponding to the gesture. Gesture control is more accurate and quicker, and easier to operate than voice control.
Brands using gesture interaction include BMW, Mercedes-Benz, Changan, Great Wall, SAIC Roewe, GAC Trumpchi, VOYAH, R Car, BAIC ARCFOX, and Xpeng, of which Great Wall and Xpeng support gesture recognition outside car.
WEY Mocha – enable gesture control over ignition, forward/backward movement, suspension, and flameout outside the car.
Changan Ford EVOS - the new-generation SYNC+ 2.0 allows gesture control over muting, music play, interface split, and back to the home interface, etc.
![多模态4_副本.png](/UpLoads/Article/2021/多模态4_副本.png)
Light interaction is available to brands like Mercedes-Benz, HiPhi and IM Motors.
Light interaction sends a message out by way of flashing, light projection and combination headlight. Light interaction provides channels for communication with traffic participants outside the car, and enables autonomous vehicles to communicate with the surroundings at a time of implementing autonomous driving. Now brands including Mercedes-Benz, Audi, Volkswagen, Great Wall, BYD, HiPhi, IM and Xpeng have packed this interaction mode.
Mercedes-Benz - Projector headlights project warning signs, assisted markings and other information related to driving assistance systems onto the road ahead.
IM L7 - Projection headlights enable the projection of interactive alerts, navigation, vehicle information among others, offering lane-level guidance with HD maps.; combination headlights provide several patterns to show the status of users in the cockpit; interactive taillights enable display of words and emoticons, as well as information, for example, thanks, pedestrian warning and state of charge.
![多模态5_副本.png](/UpLoads/Article/2021/多模态5_副本.png)
As well as combination headlights, some brands like BAIC and Samsung are trying to add displays at the front and rear of the body to send messages to pedestrians.
BAIC Lite 300 – LED displays at the front and rear of the body show words, emoticons, animations, state of charge and other content.
Multimodal fusion holds a future trend.
Single-modal interaction has a limited role to play in improvement of interaction experience. Multimodal fusion is obviously a feasible and efficient option for better interaction experience in future. Brands that are developing this feature are Mercedes-Benz, BMW, Huawei and Changan.
1. Voice + eye sight
Voice interaction shows little ability to recognize fuzzy pronouns like “this” and “that”. In this case, the addition of eye tracking for locking direction, plus voice commands, can make the response to commands much quicker.
Mercedes-Benz MBUX Travel Knowledge – a driver who fixes his/her gaze on a building says “I want to know the purpose of this building”, and he/she will acquire related information. Cerence Look provides technical support.
![多模态6.png](/UpLoads/Article/2021/多模态6.png)
2. Voice + lip movement recognition
Accurate acquisition of commands is a key step in the voice interaction process. Complex cockpit environment, noise and high personnel density make it harder to recognize voice. The new voice area lock feature added recently comes as a solution to the problem. Yet for its limitations, the single-model voice interaction technology has a limited effect in improving interaction experience.
The combination with lip movement recognition renders voice recognition far more accurate in voice interaction. Lip movements vary over language. In a noisy cockpit environment, combining lip movements, the system can ensure high voice recognition accuracy, even as the voice sounds low.
Chinese OEMs (Passenger Car) Going Overseas Report, 2024--Germany
Keywords of Chinese OEMs going to Germany: electric vehicles, cost performance, intelligence, ecological construction, localization
The European Union's temporary tariffs on electric vehicles in Chi...
Analysis on DJI Automotive’s Autonomous Driving Business, 2024
Research on DJI Automotive: lead the NOA market by virtue of unique technology route.
In 2016, DJI Automotive’s internal technicians installed a set of stereo sensors + vision fusion positioning syst...
BYD’s Layout in Electrification, Connectivity, Intelligence and Sharing and Strategy Analysis Report, 2023-2024
Insight: BYD deploys vehicle-mounted drones, and the autonomous driving charging robot market is expected to boom.
BYD and Dongfeng M-Hero make cross-border layout of drones.
In recent years,...
Great Wall Motor’s Layout in Electrification, Connectivity, Intelligence and Sharing and Strategy Analysis Report, 2023-2024
Great Wall Motor (GWM) benchmarks IT giants and accelerates “Process and Digital Transformation”.
In 2022, Great Wall Motor (GWM) hoped to use Haval H6's huge user base to achieve new energy transfo...
Cockpit AI Agent Research Report, 2024
Cockpit AI Agent: Autonomous scenario creation becomes the first step to personalize cockpits
In AI Foundation Models’ Impacts on Vehicle Intelligent Design and Development Research Report, 2024, Res...
Leading Chinese Intelligent Cockpit Tier 1 Supplier Research Report, 2024
Cockpit Tier1 Research: Comprehensively build a cockpit product matrix centered on users' hearing, speaking, seeing, writing and feeling.
ResearchInChina released Leading Chinese Intelligent Cockpit ...
Global and China Automotive Wireless Communication Module Market Report, 2024
Communication module and 5G research: 5G module installation rate reaches new high, 5G-A promotes vehicle application acceleration
5G automotive communication market has exploded, and 5G FWA is evolv...
ADAS and Autonomous Driving Tier 1 Suppliers Research Report, 2024 – Chinese Companies
ADAS Tier1s Research: Suppliers enter intense competition while exploring new businesses such as robotics
In China's intelligent driving market, L2 era is dominated by foreign suppliers. Entering era...
Automotive Gateway Industry Report, 2024
Automotive gateway research: 10BASE-T1S and CAN-XL will bring more flexible gateway deployment solutions
ResearchInChina released "Automotive Gateway Industry Report, 2024", analyzing and researching...
Global and China Electronic Rearview Mirror Industry Report, 2024
Research on electronic rearview mirrors: electronic internal rearview mirrors are growing rapidly, and electronic external rearview mirrors are facing growing pains
ResearchInChina released "Global a...
Next-generation Zonal Communication Network Topology and Chip Industry Research Report, 2024
The in-vehicle communication architecture plays a connecting role in automotive E/E architecture. With the evolution of automotive E/E architecture, in-vehicle communication technology is also develop...
Autonomous Delivery Industry Research Report, 2024
Autonomous Delivery Research: Foundation Models Promote the Normal Application of Autonomous Delivery in Multiple Scenarios
Autonomous Delivery Industry Research Report, 2024 released by ResearchInCh...
Global Autonomous Driving Policies & Regulations and Automotive Market Access Research Report, 2024
Intelligent driving regulations and vehicles going overseas: research on regional markets around the world and access strategies. "Going out”: discussion about regional markets aroun...
China Passenger Car HUD Industry Report, 2024
HUD research: AR-HUD accounted for 21.1%; LBS and optical waveguide solutions are about to be mass-produced. The automotive head-up display system (HUD) uses the principle of optics to display s...
Ecological Domain and Automotive Hardware Expansion Research Report, 2024
Automotive Ecological Domain Research: How Will OEM Ecology and Peripheral Hardware Develop? Ecological Domain and Automotive Hardware Expansion Research Report, 2024 released by ResearchInChina ...
C-V2X and CVIS Industry Research Report, 2024
C-V2X and CVIS Research: In 2023, the OEM scale will exceed 270,000 units, and large-scale verification will start.The pilot application of "vehicle-road-cloud integration” commenced, and C-V2X entere...
Automotive Intelligent Cockpit Platform Configuration Strategy and Industry Research Report, 2024
According to the evolution trends and functions, the cockpit platform has gradually evolved into technical paths such as cockpit-only, cockpit integrated with other domains, cockpit-parking integratio...
Analysis on Huawei's Electrification, Connectivity, Intelligence and Sharing,2023-2024
Analysis on Huawei's Electrification, Connectivity, Intelligence and Sharing: Comprehensive layout in eight major fields and upgrade of Huawei Smart Selection
The “Huawei Intelligent Driving Business...