Multimodal AI: Images, Audio and Video

One model, many senses

Until recently, AI meant text in, text out. Multimodal is a fancy word for a simple idea: more than one modality. A modality is just a kind of input or output — text, image, audio, video. A multimodal model handles several of them in one package: it can read a photo, transcribe a podcast, describe a video, or draw a picture from a sentence.

You have probably already met it without noticing:

• Photograph a paper receipt, and your expense report fills itself in.
• Record a voice memo while walking the dog, and it arrives as a tidy text note with the ums removed.
• Point your camera at a menu in a language you do not read, and read it in your own language.
• Upload a screenshot of a confusing error message and ask what it means.
• Get a photo described automatically so a blind colleague can read it.

For you as an engineer this changes the shape of what you can build: a support bot that reads screenshots, a meeting app that writes its own notes, a catalog that tags products from photos.

The trick behind it is simpler than it sounds. Images, audio and video are all turned into tokens — the same kind of units the model already processes as text. An image becomes a grid of small patches, audio becomes a sequence of sound fragments. Once everything is tokens, one model can reason across all of them.

How an image becomes an input

Worth spelling out from zero, because “send an image to the model” sounds more mysterious than it is.

A normal text request has one part: your message. A multimodal request has several parts in the same message — a piece of text and a piece of image. That is the entire difference. You do not upload to a special image endpoint and you do not get a special image answer: you make the same call you always make, with one more item in it.

The image itself travels one of two ways: as a URL the model provider can fetch, or as base64 — the file’s bytes encoded as a long string of ordinary characters you paste straight into the JSON. An image costs tokens like text does: typically several hundred to a couple of thousand, depending on its size. Audio is similar in spirit: you send the file to a transcription endpoint, and the transcript comes back as text.

Here is a real request with a picture in it:

1from openai import OpenAI
2import base64
3 
4client = OpenAI()
5 
6# The image is base64-encoded: the file's bytes written out as
7# ordinary characters, so it can travel inside JSON.
8with open(#98c379">"checkout.png", "rb") as f:
9  image_b64 = base64.b64encode(f.read()).decode()
10 
11response = client.chat.completions.create(
12  model=#98c379">"gpt-4o-mini",
13  messages=[
14      {
15          #98c379">"role": "user",
16          # content is a LIST of parts: text plus image.
17          # That list is the whole trick.
18          #98c379">"content": [
19              {
20                  #98c379">"type": "text",
21                  #98c379">"text": "This is my checkout page. Why might users drop off?",
22              },
23              {
24                  #98c379">"type": "image_url",
25                  #98c379">"image_url": {
26                      # a data: URL works, so does a normal https:// link
27                      #98c379">"url": f"data:image/png;base64,{image_b64}"
28                  },
29              },
30          ],
31      }
32  ],
33)
34 
35print(response.choices[0].message.content)

Notice what you did not do: no OCR library, no image preprocessing, no separate “vision” API. One call, one list of parts.

Images: two directions

Image work comes in two flavors, and it helps to keep them separate:

• Understanding (image in, text out). You send an image with a question and the model answers — like showing a photo to a friend and asking what they see. Everyday uses: read the total off a photo of a receipt, explain a screenshot of an error, list the ingredients from a label, describe a product photo for an online catalog, check that a UI matches a design. The OpenAI Vision capability (available through the chat API with an image attached) is the easiest way to start.
• Generation (text in, image out). You describe what you want and get a picture: DALL-E, Midjourney, Stable Diffusion, Flux. Uses: a birthday card no shop sells, an illustration for a blog post, a mockup of a product that does not exist yet, a new profile picture. Recent models also do editing — upload a photo of your living room and ask what it would look like with a different sofa, or remove a stranger from the background.

A practical pattern is combining both: generate an image, then send it back for understanding to check that it actually matches the request.

Image understanding in action

Here is what a vision question-and-answer exchange looks like:

No OCR library, no cropping pipeline — one API call with the image attached. That is the whole appeal. And notice the second reply: the model did not re-read the image from a database, it simply still had those tokens in the conversation, which is exactly why images are expensive to keep around.

Audio: listening and speaking

Audio splits into two jobs, and the abbreviations are worth learning because you see them everywhere:

• Speech-to-text (STT) — turning someone talking into written words. Whisper (open source, from OpenAI) is the workhorse: run it locally or via API and it transcribes dozens of languages with timestamps. Typical uses: notes from a meeting you could not attend, written subtitles for a video, dictating a message instead of typing it on a phone.
• Text-to-speech (TTS) — the other direction: written words read aloud in a natural-sounding voice. Models like ElevenLabs or OpenAI TTS power voice assistants, audiobook production, and listening to an article while you cook.

The classic pipeline chains them: record a meeting, transcribe it with Whisper, pass the transcript to a language model for a summary, then read the summary aloud with TTS. Three API calls, one very useful product.

Video

Video understanding is the youngest area. Most current approaches do the obvious thing: pull still frames out of the video and look at them as images, or listen to the audio track and reason over that. Uses: find the moment in a two-hour recording where a topic was mentioned, cut highlights from a stream, generate chapter titles, moderate uploads. Native video models exist, but the practical pattern today is still frames plus audio.

Being honest about the limits

Multimodal demos are shot in ideal conditions. Real inputs are not, and it is cheaper to know the failure modes now than after you ship:

• Small or dense text is where vision breaks first. A screenshot heading is easy; a 9-px error message, a serial number, or a dense table is where models confidently misread digits. For exact text extraction, a real OCR library still beats a general vision model — use the model for meaning, not for transcription.
• Counting and precise measurement are unreliable. “How many people are in this photo?” and “how wide is this gap in pixels?” are the classic failures. Models estimate well and count badly.
• Models infer plausibly, which means they invent. Asked about a blurry label, a model will produce a confident guess rather than “I cannot read it”. Ask for a confidence score, ask it to quote what it actually sees, and verify anything that matters.
• Audio degrades with reality. Accents, background noise, overlapping speakers and jargon all raise the error rate. Whisper is strong, not magic: check word error rate on your audio, not on a demo clip.
• Generation models are bad at text and at exactness. Letters come out misspelled, hands have the wrong number of fingers, and “put the logo in the same place” is not a request they follow reliably. Use generation for illustration and ideation, not for producing final assets with text in them or matching an exact layout.
• Video is not understood, it is sampled. The model saw a handful of frames, so fast action, causality and “what happened first” are weak. If the order of events matters, transcribe and work from the transcript.

None of this makes the feature useless — it makes it a feature you have to evaluate, like any other model call.

Choosing the right modality

Most multimodal failures are planning failures: someone grabbed an image model for a task that was really an audio problem. Walk the decision:

APIs and open source

A quick map of the landscape:

• OpenAI Vision — image understanding via the chat API; images attach right to the message.
• DALL-E and friends — image generation. DALL-E, Midjourney, Flux and Stable Diffusion cover most needs from hosted API to self-hosted.
• Whisper — open-source speech recognition, free to run, strong across languages.
• Hugging Face — the open-source shelf: thousands of multimodal models (image captioning, OCR, TTS like Coqui, vision models like LLaVA) you can download and run yourself when data privacy or cost matters.

Rule of thumb: prototype with a hosted API in an afternoon; move to open-source models when volume, privacy or latency demands it.

Check yourself

Where to go from here

You now know how to give software eyes and ears: understand images, generate them, transcribe and produce speech, and pick the right modality for a task. Later in the roadmap: LLM evaluations, including how to measure whether a multimodal feature actually works, and observability for watching it in production.