What is a model

At the core sits the neural network from the last article, with all the weights that emerged during training from billions of image-text pairs. It does the actual work — the step-by-step denoising. When someone talks about "SDXL," "Flux.1 dev," or "Flux.1 Krea dev," they usually mean this network.

On its own, though, it doesn't produce an image. It needs two helpers: a text encoder that translates the prompt into vectors, and a VAE that converts the result from latent space back into pixels. How these three parts end up on your drive depends on the model family.

With SDXL, everything is bundled into one file: diffusion model, text encoder, and VAE together in a checkpoint of about 6–7 GB. You load one file, and the model is ready to work. In ComfyUI it lives in the models/checkpoints/ folder and is loaded via the "Load Checkpoint" node. This applies to the entire older Stable Diffusion line — SD 1.5 and SDXL, plus every model built on them. Newer model families like Flux or Qwen-Image, on the other hand, usually ship as separate files.

The name comes from the training process itself: while a model is being trained, intermediate states — checkpoints — are saved regularly, to preserve progress or to allow comparing different training stages later. One of these intermediate states is eventually released as the finished, published model. The term has stuck as the common name for "the model file" — strictly speaking, though, only where everything sits in one file.

With Flux.1, the parts ship separately: the diffusion model (about 24 GB at full precision), two text encoders, and the VAE. So the diffusion model isn't a checkpoint. It lives in models/unet/ and is loaded via "Load Diffusion Model" — not via "Load Checkpoint." A Flux file in the checkpoints folder won't be found by the right loader.

There are all-in-one Flux packages in checkpoint format, but they're the exception. RAY-L works with the separate files — which is why the panel field is called "Diffusion (UNet)" and not "Checkpoint."

Flux.1 Krea dev is the sibling of Flux.1 dev: both come from the same raw base and differ only in post-training — Krea was trained toward a more natural, more photographic visual language. Krea only replaces the diffusion model — text encoders, VAE, and ControlNet stay the same as with dev. For RAY-L, Krea is the recommended model; how the two differ is covered in the next article and in detail in 2.3.3 — Flux.1 Krea dev.

The text encoder translates the prompt into the vectors the diffusion model works with. With SDXL it sits inside the checkpoint — you never see it as a separate file.

With Flux.1 there are two: CLIP-L (about 250 MB), which mainly captures short keywords, and T5XXL, which understands whole sentences. Both live in models/clip/ (in newer ComfyUI versions also text_encoders/). T5XXL is a large network in its own right: about 10 GB in fp16, just over 5 GB as fp8_scaled, smaller still as GGUF.

Text encoders belong to the architecture. Flux.1 dev and Krea share the same ones; the SDXL encoders don't fit Flux and vice versa.

The same diffusion model often comes in several versions that differ only in how precisely the weights are stored. fp16 is the full version, fp8 halves the size. Even smaller are GGUF files: quantizations in levels like Q8, Q6, Q4, or Q3. The lower the number, the smaller the file — and the further the result drifts from the original.

Important for Apple Silicon: fp8 doesn't run there. On the Mac, the choice is therefore fp16 or GGUF. GGUF files also go in models/unet/, but in ComfyUI they need their own loader from the ComfyUI-GGUF custom node. RAY-L supports them from v0.9.1. Formats and loaders in detail are covered in 2.3.2 — Flux.1 dev; which level runs how fast on which machine is shown in the timing matrix in 2.3.3 — Flux.1 Krea dev.

A finished diffusion model can do a great deal — but not everything equally well. If you want to reliably produce a particular visual style, a recurring face, or a specific product design, the model would essentially need to be taught that specialized discipline. Retraining the entire model for that would be disproportionate — it would require enormous compute and huge amounts of data, just to sharpen one narrow capability.

This is exactly the problem a LoRA (Low-Rank Adaptation) solves. A LoRA isn't a standalone model, but a small, retrained add-on package that sits on top of an existing diffusion model. Instead of relearning all the billions of weights, LoRA training computes only much smaller additional weights that shift the existing knowledge in a specific direction. Technically, this means: instead of completely altering one enormous weight matrix, two much smaller matrices are learned, whose combination produces a low-rank approximation of the desired change — hence "low-rank." The result is a file ranging from a few megabytes to low hundreds of megabytes, instead of several gigabytes. In ComfyUI it lives in models/loras/.

What a LoRA can't do: it doesn't create new foundational knowledge. A LoRA for a particular architectural style works because the model already knows what facades, materials, and proportions fundamentally are — the LoRA shifts that existing knowledge in a specific aesthetic direction. Depending on the training goal, the effect leans either toward precision ("a face" reliably becomes "this specific face") or toward stylistic coloring ("an image" becomes "an image in this particular visual language"). Both are the same mechanism — a targeted weight shift — just with different effects depending on the training data.

This property also explains why LoRA training works with a surprisingly small number of images — often 30 to 50 stylistically consistent shots are enough, instead of the millions needed for a full model training run. No new understanding needs to be built; an existing tendency in the model just needs to be deliberately reinforced.

A diffusion model can be combined with different LoRAs — a different add-on depending on the task, or even several at once with different weighting. The base model always stays the same; only the specialization changes.

Because a LoRA shifts specific weights, it's tied to the architecture it was trained for: an SDXL LoRA has no effect on Flux. LoRAs for Flux.1 dev, on the other hand, load and work on Krea and vice versa, because both share the same architecture. How well a style LoRA gets along with Krea's own aesthetic is a different question.

Your own LoRAs can also be trained on small, self-assembled image sets — relevant, for example, for a particular visual style or for keeping a series of images consistent throughout. This topic deserves its own tutorial and will be covered in more depth elsewhere.

The VAE (variational autoencoder) was already briefly mentioned in the last article: the result of the diffusion process initially exists in latent space — a compressed, mathematical representation that the neural network works with. This representation isn't an image in the usual sense; it first has to be translated.

The VAE handles that translation: a separate, smaller network that converts the compressed latent-space representation into actual pixel values. Without this final step, there would be no visible image — only its mathematical representation.

With SDXL, the VAE is included in the checkpoint and loads automatically; if needed, it can be replaced by a separately loaded VAE. With Flux.1 it's always a separate file: ae.safetensors, about 335 MB, in models/vae/. The VAE also belongs to the architecture — each model family structures its latent space differently, so the Flux VAE doesn't fit SDXL and vice versa.

On top of all this comes one more category of loadable model files: ControlNet models. Unlike a LoRA, they don't change what gets generated — they impose an additional, structural constraint, for example based on edge maps, depth information, or poses derived from a reference image or a 3D scene.

For RAY-L, this category is central: ControlNet Canny is the bridge through which Blender geometry influences the generated image. How that works in detail — and why Canny specifically, rather than another ControlNet variant — follows in its own article.

ControlNet models are tied to the architecture too: SDXL needs an SDXL Canny, Flux.1 a Flux Canny, both in models/controlnet/. This is exactly what determines which models RAY-L can support — for Flux.2 Klein there is so far no ComfyUI-capable ControlNet Canny, and Flux.2 dev does have one, but is ruled out by its render time.