Researchers at the Hebrew University of Jerusalem have published a membership-inference method that separates whether a diffusion model memorized a training image from whether that image was simply easy to reconstruct in the first place — the confound that has made prior auditing attacks unreliable. The paper, by Michael Joseph and Yedid Hoshen, reports that at the optimal log-SNR operating point, member and non-member loss distributions still overlap by 45%, which is why a simple loss threshold flags easy unseen images as members and misses hard training images.
The paper's two methods, Reference-Model Calibration (RMC) and Temporal Self-Calibration (TSC), both calibrate by comparing a loss value against a reference rather than reading it in isolation. RMC applies when a pretrained checkpoint is available: it takes a fine-tuned model's loss on a candidate image and subtracts the loss the earlier, pretrained checkpoint produced on the same image. Since image difficulty carries over almost unchanged between the two checkpoints — the paper reports a Pearson correlation of r=0.989 between pretrained and fine-tuned log-losses on Stable Diffusion fine-tuned on MS-COCO — the subtraction cancels out difficulty and leaves behind the residual drop caused specifically by memorization during fine-tuning.
TSC solves the case where no earlier checkpoint exists, which the paper notes is the more common real-world situation and the only option for auditing pretraining data. It uses a single model's own denoising trajectory as its reference, comparing the loss at a chosen diffusion timestep against the average loss at two symmetric neighboring timesteps — a three-point stencil the authors describe as measuring deviation from the "chord" connecting the two reference points. Because image difficulty evolves smoothly across nearby noise levels while memorization is localized to a narrow window in the trajectory, the stencil cancels the shared difficulty term and isolates the membership signal.
On operational numbers: on CIFAR-10, TSC reaches 62.3% TPR@1%FPR with 15 model queries per candidate, more than double the strongest baseline tested, PIAN, at 30.90% TPR@1%FPR with 2 queries. On Tiny ImageNet and STL-10-U, TSC's TPR@1%FPR comes in lower, at 12.27% and 9.57% respectively, showing the method's gains are strongest on simpler image distributions. For fine-tuned Stable Diffusion v1.4 under an uncaptioned threat model — no text prompts supplied to the attacker — TSC outperforms all baselines on MS-COCO (29.16% TPR@1%FPR) and Flickr30k (30.28%), but loses to PIA on the Pokémon fine-tune, which the authors attribute to that dataset's limited scale and low visual diversity. RMC, where a reference checkpoint is available, needs only 2 queries per candidate to hit 90.63% TPR@1%FPR on Pokémon, 96.60% on MS-COCO and 96.86% on Flickr30k, versus 64.75%, 44.04% and 47.14% for CLiD at 15 queries with ground-truth captions.
The paper also tests robustness across nine separately trained CIFAR-100 DDPM++ models spanning all combinations of x-, v-, and epsilon-prediction heads and loss targets. TSC wins every configuration, beating the strongest baseline by 23.84 percentage points in TPR@1%FPR on average; under x-loss specifically, existing baselines stay below 7% TPR while TSC reaches 29–31%. The authors show the location of peak membership leakage along the denoising trajectory shifts with the training objective — toward noisier timesteps for x-loss, cleaner timesteps for epsilon-loss — which is why methods tuned to a fixed timestep degrade on alternative parameterizations while TSC's self-calibration does not.
The method has real limits the paper states directly. Both RMC and TSC still require access to the model's denoising outputs at chosen noise levels, which closes off any API that exposes only final generated images. RMC specifically needs the actual pretrained checkpoint the audited model was fine-tuned from — if the training lineage is unknown or that checkpoint was never released, the method doesn't apply. The paper also notes RMC measures the change in loss between two checkpoints and does not separately determine whether an image was already present in the reference model's own pretraining corpus, so an auditor using RMC is only validating what changed during the specific fine-tuning run, not the full training history. Code is available at the authors' GitHub repository (MichaelJoseph777/TSC).
For teams building audit tooling around diffusion models, the actionable move is to stop treating raw reconstruction loss as a membership score and instead log losses at multiple timesteps (or against a saved earlier checkpoint) by default, since the calibration step that turns an unreliable detector into a near-perfect one is just an extra subtraction, not a new model.