Diffussion models

What a difussion model is

It is a generative approach that learns to create data (like images or audio) by reversing a gradual noise-adding process. It starts with random noise and iteratively denoises it, guided by learned probabilities, to produce realistic samples that match the training distribution.

Steps in the Diffusion Model process

1. Noise addition (Training phase): The used base model learns by gradually adding random noise to images until they become pure noise.
2. Denoising (Inference phase): Starting from random noise, the model reverses that process step by step, removing noise while shaping the image.
3. Guidance: Text or conditioning (via CLIP embeddings) directs the generation toward the desired prompt.
4. Decoding: The final latent result is converted back to an image using the VAE.

Basic concepts

Safetensor (LoRA)

A safetensor is a small adapter that modify the main model’s weights at runtime (like LoRAs). It is a secure and fast file format for storing tensors, particularly large tensors like neural network weights.

Acceleration LoRA

Acceleration LoRAs are small, specialized LoRA adapters designed to speed up generation by improving how the model handles denoising or motion.

They optimize internal calculations, allowing fewer steps to reach similar quality.

Quantization (Q)

If a model is quantified it is a compressed version of the original model where the numbers are stored with fewer bits (less precision). It saves memory and runs faster, at the cost of some quality. The number indicates how many bits per weight he model uses.

The higher number of the quantization is, higher quality the result will have and higher memory requirements will the model have.

Floating point (fp)

It is a way of storing real numbers (like 3.14159) with high precision. Used for model weights and calculations.

The higher number of fp is, higher quality will have the result and higher memory requirements will the model have.

VAE (Variational AutoEncoder)

A VAE is a tool that learns to shrink images into simple codes and then rebuild them, keeping their main details. It converts images to and from compressed latent space, allowing diffusion models to work efficiently without directly processing full-resolution pixels.

A latent space is a compressed mathematical representation of an image, where complex visual details are encoded and stored as simple numerical patterns that capture the essence of the image in fewer dimensions.

Model sampling

Iterative denoising process that transforms random noise into coherent images or video frames.

noise level checkpoints

For WAN model, there are high and low are different checkpoints of the WAN model that are designed to handle different noise levels or phases in the diffusion process (start from noise and gradually refine into frames).

• A “high noise” model checkpoint is used for the earlier, high-noise phase of the generation process: when the latent is still very noisy, you need the model to do large corrections, heavy denoising, broad scene layout, motion establishment.

• A “low noise” model checkpoint is used for the later phase, when the latent is already mostly refined and only fine-details, motion smoothing and subtle transitions remain.

GGUF

GGUF is a file format for large language models (LLMs) that optimizes them for consumer hardware by allowing them to be compressed and run efficiently on standard CPUs. They are converted and quantized versions of large diffusion or language models.

Checkpoint

A checkpoint is a container. Inside it are the unet (gguf) model which is the one doing the actual diffusion, the clip encoder that translates your prompts and the vae that translates images to/from latent space.

Clip embeddings (comfyUI "loader" node)

Loads a text-image encoder model that converts written prompts into numerical embeddings understandable by the diffusion model.

Sampler

Algorithm controlling how each diffusion step removes noise (e.g., Euler, DPM++), balancing speed, stability, and visual quality.

Using Difussion models locally

Install ComfyUI

Text to image tutorial

Image to video tutorial