AI · Setup & Infrastructure

Stability Matrix

What a launcher is, why you need one — and why local generation is the right approach.

If you've worked with AI image generation before, you probably know this kind of interface: a text field for the prompt, maybe an option to upload a reference image, click "Generate", see the result. Adobe Firefly, Midjourney, the AI generator in Canva — they all work this way. The complexity underneath is completely hidden. The model runs on someone else's servers, the infrastructure is invisible, and that's intentional.

Local image generation looks different at first contact. Before a single image is produced, you need to download several gigabytes of model files, configure and connect components, and ComfyUI's interface isn't a text field — it's a network of nodes. That's not a bad user experience. It's a different one, reflecting a different intent.

The question is: is the time this setup costs worth it?

For someone who occasionally needs an image for a presentation, probably not. For someone working professionally with image generation — or planning to — the answer is clearly yes. And the reason has little to do with quality and a lot to do with time.

Anyone who has built a cloud workflow has invested in multiple proprietary systems. Each of them evolves at its own pace: pricing models change, features disappear, models get replaced. A professional working with Midjourney v6 today will be recalibrating their workflow in a year — not because they want to, but because the system has moved. Without any lever to pull.

ComfyUI is the alternative. Not one model, but infrastructure for all models. When Flux.1 gets replaced by Flux.2, you swap one node. When a new ControlNet appears, you download a file. The workflow stays. The logic stays. What changes is the one component that changed — not the whole system.

Once you understand ComfyUI, you don't need to learn a new system again. You just need to integrate a new model. That's the difference between a one-time investment and running on a treadmill — where you never really get anywhere because the ground keeps moving under you.

There's also something cloud generators fundamentally can't offer: the ability to extend the system. Training your own LoRAs, building ControlNet workflows, keeping visual styles consistent across a whole production run — all of that requires access to the infrastructure, not just the interface.

Why a launcher

Before you launch ComfyUI for the first time, there's a practical question: how do the models get onto your machine, and where do they go?

AI image generation isn't a single program. It's a combination of components: the inference frontend (ComfyUI), the actual diffusion model (the checkpoint), the VAE, encoders, ControlNet models, LoRAs. Each of these is its own file, often several gigabytes, often spread across different platforms — Hugging Face, Civitai, GitHub. Without a management tool, you quickly end up with a folder full of unlabeled safetensors files where you can't remember which version was for what three weeks later.

Stability Matrix is that management tool. A launcher and package manager for local AI image generation — not a model itself, not an interface, but the layer underneath: installation, management, updates, isolation.

What Stability Matrix does — and what it doesn't

Stability Matrix installs and manages inference frontends like ComfyUI, Automatic1111, or InvokeAI as isolated instances. Each instance has its own Python environment, its own packages, its own configuration. Updating ComfyUI won't break the Automatic1111 installation sitting next to it.

It manages models: checkpoints, VAEs, LoRAs, ControlNet models, embeddings, upscalers — all in a central directory shared by every installed frontend. Download a model once, available everywhere.

It downloads directly from Hugging Face and Civitai, with version control. You can see which version you have, whether a newer one exists, and pin a specific version — which matters more than it sounds, because newer model versions aren't always better for your use case.

What Stability Matrix doesn't do: generate images, run workflows, process prompts. That's ComfyUI — the next chapter.

Installation

Stability Matrix runs on Windows, macOS, and Linux.

Windows: Download the current version as a .exe installer from the official GitHub page: github.com/LykosAI/StabilityMatrix/releases

Run the installer. On first launch, Stability Matrix asks for a data directory — where all models, frontends, and configurations will be stored. This folder needs to be on a drive with enough space. A single Flux.1 dev checkpoint is 23 GB; SDXL checkpoints are 6–7 GB. A realistic setup with two or three models, a few LoRAs, and ControlNet models fills up 60–80 GB quickly.

macOS (Apple Silicon): Download the .dmg package from the same GitHub page. Open it and drag the app to your Applications folder. On first launch, macOS may warn that the app is from an unverified developer — you can open it anyway under System Settings → Privacy & Security. The data directory works identically to Windows. An external SSD is recommended on macOS since internal storage on Apple Silicon is often limited.

Installing ComfyUI

After the first launch, Stability Matrix shows the Packages section. This is where frontends are installed.

"Add Package" → select "ComfyUI" → install. Stability Matrix downloads ComfyUI, sets up an isolated Python environment, and configures the directory structure automatically. No manual Python installation, no path configuration.

Once installed, ComfyUI appears in the package list. "Launch" starts it — Stability Matrix automatically opens the browser with the ComfyUI interface at http://127.0.0.1:8188. More on that shortly.

Understanding models — before you download anything

Worth pausing here for a moment.

An AI image model isn't a single thing. It's a combination of components with different roles. Which components those are — and whether they're embedded in the checkpoint or loaded separately — depends on the model. SDXL and Flux.1 dev look different in this regard, and understanding that difference matters before you start downloading files.

An SDXL setup consists of:

  • Checkpoint — the actual diffusion model, trained on millions of image-text pairs. The checkpoint contains the U-Net weights, but not necessarily its own VAE or CLIP encoder — that depends on the specific checkpoint.
  • VAE (Variational Autoencoder) — translates between the model's latent space (where diffusion happens) and visible pixel images. Juggernaut XL recommends an external VAE because the embedded one is suboptimal for photorealistic output. The SDXL VAE from Stability AI (sdxl_vae.safetensors) is the standard choice.
  • CLIP encoder — processes the text prompt. SDXL uses two CLIP encoders simultaneously (CLIP-G and CLIP-L), which are usually already embedded in most checkpoints.
  • ControlNet model — optional, but required for RAY-L. A separate model that binds image generation to an input structure — in the RAY-L workflow: edge data from Blender. ControlNet models are checkpoint-specific: one trained for SDXL won't work with Flux.

A Flux.1 dev setup consists of:

  • Checkpoint — the diffusion model, here using a Transformer architecture (DiT) instead of U-Net. Flux.1 dev is significantly larger than SDXL at ~23 GB.
  • VAE — Flux includes its own VAE, usually embedded in the checkpoint but also available separately.
  • Two encoders — a key difference from SDXL: Flux uses both CLIP-L and T5XXL simultaneously. T5XXL is a language encoder that understands actual sentence structure — much larger than CLIP (the T5XXL fp16 variant alone is ~10 GB). Why this matters for prompting is covered in 2.4.3 — Prompts for Flux.
  • ControlNet model — for RAY-L: a ControlNet Canny model trained for Flux. This is a different model from the SDXL ControlNet — they're not interchangeable.

These architectural differences explain why Stability Matrix distinguishes between checkpoint, VAE, CLIP, T5, and ControlNet as separate categories. There are no universal model files that work for everything.

Downloading models

The Model Browser in Stability Matrix lets you search and download models directly from Hugging Face and Civitai — with automatic placement into the correct directory.

For an SDXL setup (Juggernaut XL):

  1. Checkpoint: search for "Juggernaut XL" on Civitai, select the current version. The .safetensors file lands automatically in the Checkpoints directory.
  2. VAE: "sdxl-vae-fp16-fix" on Hugging Face — this VAE fixes a known color cast in the original SDXL VAE during fp16 inference.
  3. ControlNet Canny for SDXL: "controlnet-canny-sdxl-1.0" from Hugging Face (diffusers format) or the corresponding .safetensors variant.

For a Flux.1 dev setup:

  1. Checkpoint: Flux.1 dev from Hugging Face (black-forest-labs/FLUX.1-dev) — requires a free account on huggingface.co and accepting Black Forest Labs' license terms on the model page. The download is only accessible after this. The flux1-dev.safetensors (~23 GB) goes into the Checkpoints directory.
  2. T5XXL encoder: t5xxl_fp16.safetensors or t5xxl_fp8_e4m3fn.safetensors (smaller, marginally less precise) — from Hugging Face, into the Text Encoders directory.
  3. CLIP-L encoder: clip_l.safetensors — also from Hugging Face, same directory.
  4. VAE: ae.safetensors (the Flux-native VAE) — often in the same repository as the checkpoint.
  5. ControlNet Canny for Flux: flux-controlnet-canny-v3.safetensors from InstantX on Hugging Face — into the ControlNet directory.

Why T5XXL separately? Some Flux checkpoints have the encoders embedded, others don't. The practical advantage of keeping it separate: T5XXL can be shared between models — anyone later using Flux.2 or other T5-based models can reuse the same encoder file.

Directory structure

Stability Matrix stores all content for an installed frontend in a Packages/ directory. For ComfyUI, this creates the following structure:

[SM-folder]/
└── Packages/
    └── ComfyUI/
        └── models/
            ├── checkpoints/     ← SDXL and Flux checkpoints
            ├── vae/             ← VAE files
            ├── controlnet/      ← ControlNet models
            ├── loras/           ← LoRA files
            ├── clip/            ← CLIP encoders
            └── text_encoders/   ← T5XXL and other encoders

Where [SM-folder] is located depends on your operating system and installation choice — the structure beneath it is identical across all platforms. ComfyUI reads these directories directly — no manual path configuration needed as long as Stability Matrix is used as the launcher.

Updates — when yes, when no

AI models and frontends evolve quickly, and Stability Matrix reliably shows available updates. That doesn't mean you should install them.

ComfyUI updates can break existing workflows when node APIs change. The rule of thumb: update when you know what the new version brings — or when a specific problem needs fixing. Not before.

The same applies to models: new checkpoint versions aren't automatically better for your use case. Stability Matrix allows multiple versions to be installed in parallel — compare before deleting the old one.

fp16, fp8, GGUF — weight formats

Model files often come in multiple variants: fp16, fp8_e4m3fn, GGUF Q4_K_M. These are different precision formats with direct impact on file size, VRAM requirements, and output quality.

fp16 (16-bit floating point) is the standard. Maximum quality, maximum VRAM demand. T5XXL in fp16 is ~10 GB.

fp8 (8-bit floating point) roughly halves file size with a marginal quality loss — barely noticeable in practice for most use cases. Flux.1 dev in fp8 is ~12 GB instead of ~23 GB. On Apple Silicon MPS, fp8 inference requires a custom patch (patch_mps_fp8_SM.py).

GGUF is a format from the LLM world that bundles quantized models at different precision levels (Q4, Q5, Q8…) in a single file. Flux has GGUF variants that run on CPU+RAM instead of GPU — useful with limited VRAM, but significantly slower.

For getting started with an Nvidia GPU: checkpoints in fp16 (if VRAM allows), T5XXL in fp8 (good compromise), ControlNet models in fp16.