Diffussion models

What a difussion model is

It is a generative approach that learns to create data (like images or audio) by reversing a gradual noise-adding process. It starts with random noise and iteratively denoises it, guided by learned probabilities, to produce realistic samples that match the training distribution.

basic concepts

Safetensor (LoRA)

A safetensor is a small adapter that modify the main model’s weights at runtime (like LoRAs). It is a secure and fast file format for storing tensors, particularly large tensors like neural network weights.

Quantification (Q)

If a model is quantified it is a compressed version of the original model where the numbers are stored with fewer bits (less precision). It saves memory and runs faster, at the cost of some quality.

The higher number of the quantification is, lower quality the result will have and lower memory requirements will have.

Floating point (fp)

It is a way of storing real numbers (like 3.14159) with high precision. Used for model weights and calculations.

The higher number of fp is, higher quality will have the result and higher memory requirements will have.

VAE

Converts images to and from compressed latent space, allowing diffusion models to work efficiently without directly processing full-resolution pixels.

A latent space is a compressed mathematical representation of an image where complex visual details are stored as simple numerical patterns, it captures the essence of the image in fewer dimensions

Model sampling

Iterative denoising process that transforms random noise into coherent images or video frames.

noise level checkpoints

For WAN model, there are high and low are different checkpoints of the WAN model that are designed to handle different noise levels or phases in the diffusion process (start from noise and gradually refine into frames).

• A “high noise” model checkpoint is used for the earlier, high-noise phase of the generation process: when the latent is still very noisy, you need the model to do large corrections, heavy denoising, broad scene layout, motion establishment.

• A “low noise” model checkpoint is used for the later phase, when the latent is already mostly refined and only fine-details, motion smoothing and subtle transitions remain.

GGUF

GGUF is a file format for large language models (LLMs) that optimizes them for consumer hardware by allowing them to be compressed and run efficiently on standard CPUs. They are converted and quantized versions of large diffusion or language models.

Clip loader (comfyUI node)

Loads a text-image encoder model that converts written prompts into numerical embeddings understandable by the diffusion model.

Sampler

Algorithm controlling how each diffusion step removes noise (e.g., Euler, DPM++), balancing speed, stability, and visual quality.

Using Difussion models locally

Text to image tutorial

Image to video tutorial