Digital Lexicon Hub

AI models that process and combine more than one type of data, such as text, images, audio and video, as input or output.

Much business information sits in screenshots, scanned documents, calls and videos, so multimodal models let teams automate work that text-only models could not reach, such as reading invoices or checking product photos.

A support team sends customer screenshots of error messages to a multimodal model, which extracts the error code and screen context as JSON, matches it to the knowledge base and drafts a reply for an agent to approve.

Accuracy varies by modality and drops on small text, dense charts and noisy audio, and models can describe details that are not in an image. Images and recordings often contain personal data, raising GDPR and consent questions.

Multimodal systems encode each modality into vectors and combine them by early fusion in a shared representation space, mid fusion at intermediate stages, or late fusion of separate model outputs. Open research problems include representation, alignment across modalities, reasoning, generation, transference and quantification. Many current Generative AI (GenAI) products, such as Google Gemini, extend a Large Language Model (LLM) with vision and audio encoders.