When Generative AI Can See the Question

Some questions become awkward as soon as I try to turn them into text. There is a line on a diagram, a symbol beside it and something about their position that matters. Describing all of that can take longer than asking the actual question. By the time the description is complete, I have nearly written a small and rather unsatisfactory replacement for the picture.

That is part of what makes multimodal generative AI interesting to me. An image can be part of the question. A video can provide the sequence being discussed. The answer may be text, or, with an appropriate model, a new image that helps explain an idea. The conversation gains material that would otherwise have to be flattened into a description first.

I find the possibilities more useful when I separate them. Looking at an existing image, reasoning about what it supports and generating a new image are different activities. They can appear in one interface while asking quite different things of the model. A convincing result in one activity doesn't tell me everything about the others.

A picture joins the conversation

Google's image understanding documentation describes tasks such as captioning, classification and answering questions about images. These are examples of visual information becoming input to a language-based interaction. The exact supported features depend on the selected model and interface; multimodal is a broad category rather than one fixed bundle of abilities.

Imagine a made-up drawing of a home heating system. The reader wants to understand why one pipe returns to the same tank. With an image as context, the discussion can refer to that particular line rather than an elaborate verbal reconstruction. The question becomes more grounded in what the reader is actually looking at.

There is a useful difference between asking “What is this system?” and “Which visible features support this interpretation?” The first invites a label. The second asks for an explanation that can be checked against the image. I would often want both, especially if the drawing contains an unfamiliar convention or lacks a legend.

A model might identify a plausible arrangement while overlooking an important symbol. That possibility doesn't erase the benefit of being able to ask about the image. It suggests a better interaction: point to the ambiguous part, ask what remains uncertain and compare the answer with the source's own explanation when one exists. The image can keep the conversation specific.

The new convenience is substantial. A student, a curious reader or someone approaching a diagram from another field can ask a question before knowing all the vocabulary. The answer still needs examination, but the threshold for beginning the investigation is lower. I like tools that let curiosity start before the glossary is finished.

Detail sets a boundary

In the heating example, a tiny arrow could determine the direction of flow. If the arrow is unreadable in the input, a plausible explanation of the rest of the system doesn't recover it. The model may know what similar drawings usually show. The original drawing may be an exception. That is exactly why the tiny detail matters.

I would distinguish a visible observation from an inference: “a line connects these parts” and “this appears to be the return path” deserve different confidence. A useful answer can explain the likely interpretation while keeping the uncertain part available for inspection. It needn't choose between pretending to see everything and refusing to discuss anything.

A closer crop may help when the original contains more detail than the submitted view. So may a higher-quality original or a second view. But enlarging a low-resolution file doesn't establish what a missing mark actually was. More pixels can make an image easier to display without supplying new evidence about the scene that produced it.

This distinction becomes particularly important when asking a model to read text. A familiar word can tempt any reader to complete a blurred ending. In a casual description, that guess might be harmless. In an identifier or a measurement, one guessed character can change the answer. The task determines how much uncertainty can be tolerated.

I find it helpful to ask what part of a conclusion would change if the difficult detail were different. If nothing changes, an approximate reading may be enough. If the conclusion reverses, the detail deserves another source. The value of visual AI grows when it helps locate that hinge, rather than only supplying a smooth description of the whole page.

Video brings time into it

The video understanding documentation describes capabilities including questions about video content and references to timestamps. It also discusses processing and sampling choices. Those details matter because a video is a sequence, while the representation supplied for analysis may select or compress parts of that sequence.

Imagine an invented clip of someone assembling a small shelf. A reader wants to find where a bracket first appears. A model that can discuss the clip might help narrow the relevant interval, saving the reader from repeatedly watching the whole thing. A timestamp becomes a useful route back to the source rather than merely a decorative detail in the answer.

Now suppose the important action is a quick movement lasting only a fraction of a second. A broad summary of the clip and a reliable account of that movement are different tasks. Whether the event survives the processing route matters. A description that confidently fills the gap can be more troublesome than a clear statement that the event wasn't established.

There are temporal questions beyond recognising objects. Did the bracket appear before the screw was tightened? Did a hand obscure the connection at the decisive moment? Was the relevant action outside the frame? Each question needs evidence about order and visibility. A sensible general story about assembling shelves won't settle what happened in this particular clip.

For that reason, I like answers that make revisiting the evidence easy. An approximate interval can be useful if it is labelled as approximate and leads the reader to the right scene. The appeal is a collaboration between search and close attention: let the model help find the passage, then examine the part on which the conclusion depends.

A timestamp is useful when it takes me back to something I can inspect.

Generating a picture asks something else

Google's image generation guide describes generating images and editing images through supported models. That opens a different kind of conversation. The aim might be to illustrate an explanation, explore a composition or transform a reference. Success depends on the requested task, not simply on whether the output looks polished.

For the imagined heating diagram, I might want a simplified teaching image with the two flow directions clearly distinguished. Here the generated image is an explanation I have asked to create. It should be judged for clarity and technical consistency. It should not be treated as new evidence about the original installation, which the model has never inspected beyond the supplied material.

This makes the role of the output important. A conceptual drawing can simplify. A documentary image is expected to preserve what was there. A proposed design is allowed to show something that doesn't exist yet. The same attractive visual style can serve all three roles while hiding the differences unless the surrounding text makes them clear.

I would check a generated technical illustration in a rather unglamorous order. Do the connections make sense? Are labels consistent? Is an arrow pointing in the intended direction? Beautiful lighting can wait. A picture that explains the wrong mechanism with exceptional confidence is still a picture I would need to correct.

There is real creative value here. Being able to propose a different layout, ask for a simpler view or compare two visual explanations can make an abstract idea easier to discuss. The interesting benefit is the shorter distance between a thought and something visible enough to criticise. A draft that gives us a better disagreement has already done useful work.

A neat shape for an uncertain answer

Google's structured output documentation explains how supported responses can follow a supplied JSON schema. That can make results easier for another program to consume. The shape of an answer and the truth of its contents remain separate questions: a date can fit a field perfectly while referring to the wrong event.

In the shelf example, a structured response might hold an event description, an approximate interval and whether the relevant action is fully visible. Those fields could help organise a review. But their design needs room for absence and uncertainty. If every field must contain a confident-looking value, the format can encourage false precision rather than clear communication.

A field called observed action should not quietly contain a guess about what happened behind an obstruction. A separate interpretation field could carry a hypothesis, with the reason it remains uncertain. This is a proposed design for the example, rather than a claim that a particular API automatically supplies reliable confidence or enforces those meanings.

I am interested in the point where a rich visual conversation meets a smaller, more explicit record. A person may want a narrative explanation. A later tool may need a few named fields. Both can be useful, provided the conversion preserves the parts that matter. An uncertain observation shouldn't become certain merely because it has been assigned a column.

A modest example helps expose the issue. If the original clip never shows whether the screw is fully tightened, a good record can say that the step isn't confirmed. It can still identify the interval where the hand obscures the joint. The result is limited, but it gives the next inspection a sensible place to begin.

The kind of progress I want to see

Multimodal AI makes more of a question available to the model. That can be transformative for situations where the difficult part is pointing at what one means. A diagram, a page or a sequence of frames can carry the context that a text-only conversation would otherwise spend several paragraphs trying to reconstruct.

I also like the possibility of moving in both directions. Start with an image, discuss a mechanism, then generate a simpler illustration to test the explanation. The generated illustration becomes a draft for further thought. If it exposes an inconsistency, that is a reason to revise the idea, not just ask for a more flattering picture.

The useful comparison depends on the task. A quick visual description, careful document reading and an exact account of a brief action should not share one undifferentiated score in my head. I would want examples that resemble the question being asked, with the awkward details left in. An easy photograph can show a capability without describing its entire boundary.

There is also something pleasantly direct about asking a technical question in ordinary language while keeping the visual beside it. A reader can say which part is confusing and request another explanation. The interaction can adapt before either side has committed to a long formal description. That responsiveness is a genuine reason to be curious about the technology.

What I want from the next demonstration is a clear view of that exchange: the original input, the question, the interpretation and any uncertainty that remains. If a new image is generated, I want to know what role it is meant to play. With those distinctions visible, the interesting possibilities become easier to appreciate. The picture can finally join the conversation without being asked to prove things it never showed.

— John