How AI image generation works

A diffusion model is trained on millions or billions of image-text pairs. In doing so, it doesn't store individual images. It learns statistical relationships between terms and visual features.

Imagine the model should later generate "a cat wearing a hat." To do that, it must have learned during training what cats look like — fur texture, eye shape, posture — and, independently, what hats look like — shape, brim, material. Both concepts are stored as separate patterns that can later be combined in any way.

This knowledge lives in the model's weights — billions of numerical values that are adjusted step by step during training. The model no longer knows any original images — only the learned statistical patterns.

When a prompt is entered, it activates the matching learned patterns. The prompt acts like a weighting of certain features. The model combines the activated patterns so that an image emerges that statistically matches the description as closely as possible.

It does not select an existing image — it computes a new one that combines the desired properties with high probability.

The control is indirect: what's influenced isn't individual pixels, but the probabilities with which certain structures appear together.

The term comes from the technical process of how the image is formed. The model starts with random noise — comparable to image static — and removes that noise step by step. Across many small computational steps, an increasingly clear image structure emerges.

The computation doesn't happen on the pixels of the finished image, but in latent space — a heavily compressed, mathematical representation of the image that the model can process much faster. Only at the very end does it become a visible image.

The starting noise is only seemingly random. It's computed from a starting number, the seed. Same seed, same prompt, same settings — and on the same machine with the same software, you get the same image. A different seed with the same prompt gives a different variant of the same motif.

The seed makes a result repeatable. The prompt decides how widely the variants spread across different seeds: the more precisely and consistently it's formulated, the more stable the motif stays.

This is one of the most important points for understanding prompts — and one of the most frequently misunderstood.

Before the diffusion process begins, a text encoder translates the prompt into numbers: a sequence of vectors, one for each token. Each of these vectors already carries the context of the entire prompt — the meaning isn't built word by word in sequence, but as a whole, all at once.

A helpful way to picture it: the prompt isn't read like a sentence you work through word by word. It's more like a single photograph taken of the entire text at once — with all the relationships between the words already captured.

Which text encoder does this depends on the model. SDXL uses two CLIP encoders, trained on short image captions and good with keywords. Flux.1 combines CLIP-L with T5, an encoder from the language-model world that captures whole sentences and their relationships. Flux.2 goes further still and uses a full language model (Qwen 3) as its encoder. That's why keyword lists work best for SDXL and written-out sentences work best for Flux — more on this in the articles on prompt architecture.

These vectors then accompany every single step of the diffusion process. At each step — from the noisy starting image to the finished result — the model "looks" at the full prompt again and decides which areas of the emerging image should be influenced by which parts of the prompt.

The mechanism behind this is called attention. It lets the model "weight" how strongly a given word in the prompt influences a given area of the image. In the example "a black-and-white cat wearing a top hat in a retro café," attention ensures that "black-and-white" and "top hat" act on the cat, while "retro café" shapes the surroundings — even though these words sit far apart in the sentence.

What this doesn't mean: the model has no memory in the human sense, and it doesn't read the prompt sequentially the way a person reads a sentence. There's no note-taking, no remembering, no understanding. There's a mathematical representation of the entire prompt, which gets consulted again at every computational step.

In practice, this means: the entire prompt acts on the entire image, at every step. Word order does play a role — but differently than in a sentence read left to right. It influences how much weight attention assigns to particular terms.

The calculations in each diffusion step are carried out by a neural network — a computational model made up of many interconnected units, organized in layers. An input layer, several processing layers, an output layer. Each unit processes numerical values, weights them, sums them, and passes the result on.

In SDXL this network is a so-called U-Net, in Flux a transformer. For the principle it makes no difference: at each step, both estimate what the current state would look like with slightly less noise. In ComfyUI the old name has stuck anyway — the Flux network also lives in the models/unet/ folder.

A helpful way to picture it: a production line. At the start, raw material arrives — the noisy image state plus the prompt vectors. The material passes through several stations where it's processed and altered. Each station makes decisions based on its trained settings — the weights. At the end of the line stands an image that's slightly less noisy than at the start.

What attention mechanisms add to this production line: each station doesn't just have the material in front of it, but also has access at any time to the complete prompt — like a board hanging above the entire production line that every station can read at any moment. There's no handing off from station to station, no note that travels along — instead, a shared reference that everyone accesses simultaneously.