Chameleon: Dynamic Format Adapter for Efficient Diffusion

TL;DR — Every diffusion PTQ method we compare against fixes the number format (almost always INT8) and only tunes scales around it. Chameleon keeps the bit-width fixed and makes the format itself a choice, made per weight channel and per (layer, timestep bucket) activation tensor from two cheap statistics. It gets the best FID in all six backbone × bit-width settings we test.

SDXL · UNet

SDXL fire hydrant image quantized with Q-Diffusion at W4A8 — Q-Diffusion

SDXL fire hydrant image quantized with Chameleon at W4A8 — Chameleon

SDXL-Turbo · 1-step

SDXL-Turbo fire hydrant image quantized with MixDQ at W4A8 — MixDQ

SDXL-Turbo fire hydrant image quantized with Chameleon at W4A8 — Chameleon

PixArt-α · DiT

PixArt-alpha fire hydrant image quantized with Q-DiT at W4A8 — Q-DiT

PixArt-alpha fire hydrant image quantized with Chameleon at W4A8 — Chameleon

Each per-architecture PTQ baseline next to Chameleon at 4-bit weights (W4A8), from the COCO-2014 caption “An old city fire hydrant has ‘Love was found here’ painted on it.”

Abstract

A very popular way to enable image generation on memory-constrained accelerators using modern spatiotemporal generative models is post-training quantization (PTQ). However, every existing diffusion PTQ scheme shares an unstated commitment: the number format is fixed in advance, and only the scale, zero point, or the per-layer bit-width is allowed to move. This is unfortunate, because at a fixed bit-width the best format depends on the distribution being encoded, and that distribution differs from tensor to tensor: across weight channels, across layers, and – in a denoising model – along the diffusion timestep, where activations slide from heavy-tailed and noise-dominated at t→T to tightly clustered and structured at t→0. One format for the whole network is therefore mispriced for most of its tensors, and re-tuning a scale cannot repair a format mismatch.

In this work, we propose a PTQ framework, which we call Chameleon, that holds the bit-width fixed and treats the number format itself as a discrete variable, chosen per weight channel and per (layer, timestep bucket) activation tensor. Chameleon draws activation formats from a palette of {INT8, FP8 E4M3, FP8 E5M2, MXFP8, MXINT8} and weight formats from {INT8, MXINT8} at 8 bits or {INT4, NF4, FP4 E2M1, MXINT4, MXFP4} at 4 bits, choosing activation formats ahead of time from two cheap statistics – empirical activation kurtosis and closed-form diffusion SNR ᾱ_t/(1−ᾱ_t) – stored in a per-(layer, timestep bucket) lookup table, and weight formats offline by reconstruction error. An architectural fork adapts the same selection layer to each model family, so each model exercises the palette as far as its own statistics warrant: weights and per-(layer, bucket) activations on multi-step SDXL, weights alone where a single sampler step leaves no timestep axis.

We evaluate Chameleon across three diffusion regimes – multi-step SDXL, single-step SDXL-Turbo, and the PixArt-α Diffusion Transformer – on COCO-2014. Chameleon achieves the best FID for all six backbone × bit-width settings, while preserving semantic alignment with the text prompts (CLIP within 0.24 of the FP16 reference in every setting, and the best of all quantized methods at W4A8). Adapting the format, rather than only its scale, recovers distributional fidelity at 4-bit weights while holding prompt adherence at the FP16 level.

Why change the format, not just the scale?

A trained network's weights are static, so their statistics can be measured once, offline, per channel. Activations are different: the input to a layer depends on the timestep, and its distribution changes dramatically along the schedule. Early in sampling (t→T) activations are noise-dominated and heavy-tailed, with kurtosis κ above 5; near the end (t→0) they carry structured signal and κ often drops below 3. Re-scaling INT8 cannot make it tolerate heavy tails, and re-scaling FP8 E5M2 cannot give its logarithmic bins the density to render fine texture. The fix is to change formats along the schedule.

Phase 1 · heavy tails

FP8 E5M2

κ > 5 or SNR < 0.2. Noise-dominated activations need dynamic range.

Phase 2 · settling

FP8 E4M3

Everything in between: more precision, less range.

Phase 3 · clean signal

INT8-asym

κ ≤ 3 and SNR > 2. Near-Gaussian, tightly clustered values.

UNet shortcuts

MXFP8 E4M3

Skip concatenations span a 58× peak-to-median channel range at the median layer (up to 163×); one shared exponent per 32 values absorbs it.

The routing rule, applied per (layer, timestep bucket). The shortcut rule is checked first; the thresholds (κ: 3, 5; SNR: 0.2, 2) are fixed a priori and shared across every model, not tuned per model.

The Chameleon framework

Chameleon framework diagram: offline analysis builds an adaptive weight palette and a per-layer, per-timestep-bucket activation format lookup table; a load-time architectural fork routes to UNet, few-step distilled, or diffusion transformer execution paths; each path runs a fake-quantization dispatch.

Figure 2. Offline, a one-pass calibrator accumulates per-(layer, bucket) activation moments for the empirical kurtosis κ_l(t), while the diffusion SNR(t) is read from the noise schedule. The router writes the chosen format into an integer lookup table; weights are selected offline. At inference, one fake-quantization module reads the LUT per layer per step and dispatches to the matching routine.

Activations: a per-(layer, bucket) lookup table

Calibration is a single pass over 128 COCO prompts with the schedule split into B = 10 timestep buckets. Each instrumented layer keeps seven running scalars per bucket (power sums plus min/max), from which kurtosis follows in closed form. SNR costs nothing: it is read off the noise schedule with no forward pass. At inference the format choice is one indexed read per layer per step.

Weights: per-channel format by SQNR

Weights are static, so selection runs offline at finer grain. Each output channel is round-tripped through every format in the palette, and the format with the highest signal-to-quantization-noise ratio wins. Even within one SDXL conv layer, weight kurtosis varies enough across output channels that a tensor-wise choice would be a compromise. The metadata cost is one byte per output channel, about 1 MB across the UNet.

One selection layer, three model families

A model loader inspects the topology once and forks to a UNet, few-step, or DiT path. The format choice sits underneath the machinery each prior method already uses, so Chameleon composes with them rather than replacing them.

SDXL UNet execution path showing per-bucket activation format shares, the MXFP8 shortcut topology rule, and per-output-channel weight format selection.

UNet path. Activations follow the routing rule with static, ahead-of-time scales (Q-Diffusion style). The conv layers right after each skip concatenation (up_blocks.*.resnets.*.conv1) are pinned to MXFP8 E4M3. SDXL uses every axis of the palette: per-channel weights, per-layer activations, three temporal regimes, and the topological MXFP8 pin.

Results

6 / 6

backbone × bit-width settings with the best FID

−7.0

FID vs. Q-Diffusion on SDXL at W4A8 (21.41 → 14.43)

≤ 0.24

CLIP gap to the FP16 reference in every setting

< 1 h

calibration for all three backbones on one A100, no gradients

Backbone	Method	Bits	FID ↓	CLIP ↑
SDXL50 steps, CFG 7.5, 1024 × 1024	FP16	W16A16	16.16	26.88
	Q-Diffusion	W8A8	14.70	26.77
	PTQ4DM	W8A8	14.65	26.68
	Chameleon	W8A8	14.23	26.70
	Q-Diffusion	W4A8	21.41	26.27
	PTQ4DM	W4A8	21.82	26.21
	Chameleon	W4A8	14.43	26.64
SDXL-Turbo1 step, no CFG, 512 × 512	FP16	W16A16	21.63	26.63
	MixDQ	W8A8	21.43	26.64
	Chameleon	W8A8	20.93	26.62
	MixDQ	W4A8	24.50	25.88
	Chameleon	W4A8	21.29	26.85
PixArt-α20 steps, CFG 4.5, 1024 × 1024	FP16	W16A16	27.63	25.99
	Q-DiT	W8A8	27.69	25.99
	Chameleon	W8A8	24.33	25.81
	Q-DiT	W4A8	23.53	25.73
	Chameleon	W4A8	22.24	26.09

COCO-2014, 24,576 captions, one image each. Clean-FID against the full val2014 set; CLIP-score with ViT-L/14 (cosine × 100). Best FID per backbone, and best CLIP at W4A8, in bold. Resolutions and samplers differ per family, so FID is comparable within a backbone but not across. All numbers use fake quantization.

Qualitative comparison at W4A8

SDXL FP16 reference images for four prompts — FP16 reference

SDXL images quantized with Q-Diffusion at W4A8 — Q-Diffusion

SDXL images quantized with Chameleon at W4A8 — Chameleon

Four prompts per backbone, in reading order: a cherry-red convertible, an orange tabby cat, a golden retriever puppy, and a black sports car. Seeds are fixed across columns, so the same prompt starts from the same latent.

Backbone	FP16	PTQ baseline	Chameleon
SDXL	522	605 (Q-Diffusion)	662
SDXL-Turbo	439	605 (MixDQ)	725
PixArt-α	1287	1030 (Q-DiT)	1057

High-frequency detail, measured as the variance of the Laplacian over 400 prompt-matched images per configuration (higher = more detail). Chameleon keeps more detail than the PTQ baseline on every backbone. The FP16 column is a reference, not a target to exceed.

What each piece is worth

Routing (SDXL W4A8)	FID ↓	Δ
κ and SNR (shipped)	19.39	–
κ gates perturbed (3,5)→(3.5,5.5)	19.41	+0.02
κ only (SNR gate off)	19.60	+0.21
SNR only (κ frozen)	19.68	+0.29
no routing, FP8 E4M3 everywhere	20.29	+0.90

Palette (SDXL W4A8)	FID ↓	Δ
full palette (shipped)	19.39	–
drop NF4	19.46	+0.07
drop MXFP8 (shortcut rule off)	19.54	+0.15
drop MXFP4 E2M1	19.77	+0.38
INT8-asym everywhere	180.51	+161.12

5,000-image ablations on SDXL at W4A8 (a separate run, so the baseline differs from the main table). Two independent calibrations of the same configuration differ by 0.04 FID. Routing is worth 0.90 FID over the best single fixed format, the two statistics carry complementary information, and moving the thresholds barely changes FID. Forcing INT8 everywhere collapses output entirely; no choice of INT8 scale recovers it.

Scope and limitations

No efficiency claims yet. All results use fake quantization on an A100. The numbers describe what the format assignment costs in fidelity; latency and memory savings on native FP8 and MX kernels are future work.
Hardware support. FP8 and MX formats need Hopper- or Blackwell-class accelerators; on INT-only NPUs the palette reduces to its INT members. Per-channel mixed formats would run as same-format sub-GEMMs after a channel permutation, or with weights de-quantized in registers.
Image generation only. Video DiTs and sub-4-bit weights (which would need quantization-aware training) are out of scope.

BibTeX

@misc{sanyal2026chameleon,
  title        = {Chameleon: Dynamic Format Adapter for Efficient Diffusion},
  author       = {Sanyal, Arnab and Chinchali, Sandeep},
  year         = {2026},
  howpublished = {Under review at the International Conference on Learning Representations (ICLR) 2027}
}

Chameleon: Dynamic Format Adapterfor Efficient Diffusion