OEMs and suppliers are rapidly developing next-generation in-car virtual assistants powered by advanced Large Language Models (LLMs). These new virtual assistants aim to deliver an enhanced in-vehicle experience that is more seamless. However, implementation is important, or the virtual assistant’s capabilities will fall short of users’ expectations.
In this edition of SBD Explores, we delve into the opportunities and challenges in creating these sophisticated virtual assistants, examining their implications for both consumers and OEMs.
What is happening?
The new wave of generative artificial intelligence (GenAI) and large language models (LLM) have had an immediate impact on in-car virtual assistants. With GenAI, OEMs can integrate additional
services and embed their brand DNA more deeply into vehicles. GenAI can also deliver more natural speech.
Several OEMs are actively exploring the integration of ChatGPT into their vehicles. Currently, Mercedes-Benz and DS Automobiles are running beta programs that are open to vehicle owners. The DS program is capped at 20,000 participants. GM and Stellantis have announced interest in incorporating ChatGPT into their vehicles' virtual assistants.
Other OEMs, especially in the Chinese market where ChatGPT isn't available, are focusing on developing their own inhouse GenAI model.
Suppliers of traditional virtual assistant are working to incorporate the latest LLM technology into their forthcoming products.
Tech-giants such as Amazon and Google are repositioning their virtual assistant product teams, aiming to substitute their current AI engines with new ones powered by LLMs.
Why does it matter?
The next-generation of virtual assistants, powered by Large Language Models (LLMs), will be a key feature of tomorrow's vehicles. They will be able to impact many different systems within the vehicle.
This innovation transforms the driving experience by offering natural and precise conversation at a human level. The advanced intelligence of LLMs not only increases usage of virtual assistants in vehicles, but also builds consumer trust.
These new types of assistant are more than just a gateway to complex vehicle settings. They serve as a hub that seamlessly integrates services like table reservations, ticket bookings, and payments. This ability opens opportunities for OEMs to create new revenue streams by incorporating more services.
With time, the new generation of assistants will evolve into a trusted agent, representing both the car and the OEM. It will learn to understand user behavior and offer timely and proactive support.
While integrating LLMs into mobile devices like smartphones remains a challenge, today's smart vehicles are ideal hosts for LLMs. Equipped with the necessary computing capacity, sensors, connections, and data, they are well-prepared to carry true intelligence.
Where next?
To build an engine for a next-gen virtual assistant, four core foundations, a model, hardware, data, and knowledge are required. Capabilities build on these core elements to allow for features.
Model – The future intelligence in cars isn't just a voice assistant, but a digital agent capable of perceiving the world in various ways. Therefore, a multi-modal model is essential. To ensure smooth on-device inference, the model needs to be optimized and compressed to fit into a Car system on chip.
Hardware – This includes an AI chip with sufficient computing power that can operate with lower power consumption, e.g. Qualcomm Snapdragon 8 Gen 3. Hardware also encompasses sensors like cameras, microphone arrays, and motion detectors for enhanced perception.
Data – A “smart” model must be trained using a large amount of data (corpus) and fine-tuned with domain-specific data, such as vehicle knowledge. Accumulating this data might take years.
Knowledge – OEMs can either purchase, build, or co-develop the system with partners. In any case, intensive knowledge in UX, vehicle domain expertise, data science, and ML engineering is required.
What to watch out for?
The architecture of future virtual assistant systems is complex. The key components are shown below:
Multi-modal model is required for future systems as the virtual assistant should not only understand voice commands but also interpret non-voice cues such as touch, gestures, and facial expressions. This will give a more comprehensive interaction experience.
Virtual avatars should be used to enhance user engagement and to help build familiarity with the system. They will be a focus of OEMs as they look to create a unique brand experience.
Knowledge module is one of the key components that OEMs can use to differentiate from others. The more information a virtual assistant has at its disposal, the more use cases it can support.
Action modules will grow substantially with the introduction of LLMs and multi-modal models as virtual assistants develop.
Dialogue management is a necessary module for privacy protection and bias management. User privacy will remain a large concern for owners as virtual assistants develop.
How should you react?
Prioritize
Prioritize key virtual assistant features that can benefit the consumer directly and differentiate from other brands.
Fine-tune
Choose correct GenAI model and spend most effort on fine-tuning the model using domain-specific data.
Iterate
Acquaint team with the machine learning update cycle and iteratively enhance both core model and its downstream applications.
Interested in finding out more?
Most of our work is helping clients go deeper into new challenges and opportunities through custom projects. If you would like to discuss recent projects we've completed relating to Automotive AI applications, contact us today!
Also, be sure to view our related content:
