AI · Prompt Architecture
Prompt Architecture: The Foundation
Two encoders, two logics — and why that changes everything: structure, order, wording.
When a prompt doesn't deliver what you had in mind, it's rarely because you chose the wrong words. Most of the time it's because you spoke the wrong language — not in the sense of English or German, but in the sense of the logic a model uses to process text.
There are two fundamentally different ways AI image models read a prompt. One knows no grammar but understands weighting. The other understands sentences, relationships, and syntactic dependencies. Which logic a model uses depends on which encoder it was trained with — and that affects everything: structure, order, wording.
Before you learn how SDXL and Flux work in detail, it's worth understanding this foundation. It turns two seemingly independent topics into a single coherent system.
Why prompt in English
Before we get into encoder logic, a question almost everyone asks: do I have to prompt in English?
Technically no — both encoders accept other languages. But there's a good reason to do it anyway.
CLIP and T5 were trained predominantly on English-language data: millions of image-text pairs whose captions were in English. English terms are anchored as direct tokens in the encoder's vocabulary — "cobblestone" is a token with a clear visual meaning, shaped by thousands of training images. A non-English equivalent, by contrast, gets broken into subword tokens internally or approximated through weaker associations that weren't part of the actual image-text training.
This means a non-English word costs more tokens for less precise activation. The encoder "knows" it less well — not because it doesn't understand it, but because it has seen it less often in connection with the corresponding images. Every deviation from the training vocabulary is another opportunity to become less precise.
This isn't an argument against your own language — it's an argument for speaking the machine's language when precision matters. And in professional image production, it always matters.
CLIP and T5 — two encoder philosophies
CLIP was originally trained to match images with short text descriptions — like alt text under photos: "woman in red coat", "modern kitchen interior", "abstract blue painting". These descriptions aren't sentences. They're lists of concepts. CLIP learned to recognize and weight concepts — but not to understand grammar. Sentence structure is irrelevant to CLIP. What counts: which concepts appear, in what order, and with what emphasis.
T5 was trained as a language model — on real texts with real sentences, subjects, verbs, and relationships between words. T5 understands the difference between "the light falls through the windows" and "the windows through which the light falls" — CLIP doesn't. For T5, the syntactic role of an element is decisive: what is a subject gets more weight than what appears in a subordinate clause.
This has direct consequences for your prompt.
| CLIP | T5 | |
|---|---|---|
| Base unit | Token / concept | Word in sentence context |
| Grammar | No | Yes |
| Order | Earlier tokens = weighted more | Syntactic role matters more |
| Weighting syntax | Works: (word:1.3) | Ineffective |
| Optimal input | Keyword list, comma-separated | Full sentences |
| Negative prompt | Use actively | Barely effective |
Which model uses which encoder?
SDXL (Juggernaut XL and others): exclusively CLIP-based. Two CLIP encoders internally (G and L), but both follow the same concept-list logic. Keyword structure and weighting syntax are the main control instruments here.
Flux.1 dev: uses both encoders simultaneously — CLIP-L and T5XXL. That's why the ComfyUI node CLIPTextEncodeFlux has two separate input fields: one for the keyword list (CLIP-L), one for the full sentence (T5XXL).
SD 3.5, Ideogram 4, newer models: tend toward T5 logic, often just one input field — full sentences work better than keyword lists.
The direction is clear: models are moving away from pure CLIP logic toward genuine language understanding. SDXL keyword structure remains relevant because the ecosystem (LoRAs, checkpoints, fine-tunes) runs very deep — but it's no longer a standard, just one of two encoding logics.
Prompt portability
If you build a Flux prompt cleanly using two-field logic, you automatically have two more prompts ready:
The clip_l field is your SDXL prompt — directly transferable, add weighting syntax where needed. The t5xxl field is your prompt for T5-based single-field models (SD 3.5, Ideogram 4) — directly transferable.
One development process, three models covered. That makes Flux not a special case, but the sensible starting point.
The 8 categories of an image idea
Before you write anything — in whatever language — it helps to break the image idea down into categories. These 8 categories cover everything a prompt can describe:
Not every category needs content every time — you weight according to the image idea. But when a prompt isn't working, it's worth systematically checking the 8 categories: which ones are overdeveloped? Which are missing entirely?
A common observation from practice: people who think photographically naturally fill the material category heavily — surfaces, textures, materiality. The light category, by contrast, often stays too generic ("natural daylight", "soft light"), even though it's the one that decides between a generic and a convincing result.
Why prompts fail — four diagnostic questions
When a result doesn't work, there are four possible causes. They require different solutions and should be checked in this order:
1. Concept conflict
Two elements in the prompt demand something physically or visually incompatible. Example: "harsh midday sun" and "moody fog" simultaneously. The model averages between both — the result is indecisive, not wrong. Solution: remove or sharpen one of the elements.
2. Misallocated specification
One category is overdeveloped (e.g. five adjectives for the material surface), a more important one stays underspecified (e.g. light described in a single word). The model distributes its "attention" according to the prompt — not according to your actual priority. Solution: rebalance the categories.
3. Wrong encoder style
SDXL keyword lists in Flux, or full Flux sentences in SDXL. Often still works reasonably well, but suboptimally. This is the most common cause of inconsistency: sometimes good, sometimes poor results with seemingly similar prompts. Solution: adapt wording to encoder logic.
4. Model limit
Some things a particular model simply can't do well — regardless of the prompt. Hands, text in the image, very specific spatial relations are known weaknesses. Solution: stop optimizing the prompt; use compositing or a different model.
Check for model limits last, not first. Most bad results are not a model problem.
A special case: Canny-guided generation
When working with ControlNet Canny — using a Blender render as a geometric guide — an additional rule applies: composition, camera angle, and subject placement are already determined by the Canny render. These categories no longer need to be described in the prompt.
This is not a minor point. Redundant composition instructions in the prompt can conflict with the Canny guide — the model then tries to follow both simultaneously, and the compromise is worse than no instruction at all. More on this in the RAY-L section.