Only 2 of 13 Algorithms in CircuitKIT Achieve Production Status

CircuitKIT v1.0.0, developed by Lexsi Labs and released under the LSAL v1.1 source-available license, integrates mechanistic interpretability into a Python pipeline. It consumes a model and a task, then outputs a pruned HuggingFace checkpoint and a six-pillar faithfulness score. Only two of its thirteen discovery algorithms are production-validated, with the rest in experimental or research stages. The library defines a circuit as the minimal attention-head and MLP subgraph responsible for a specific behavior, addressing the gap left by most interpretation tools that stop at attribution heatmaps. It exports an intervened checkpoint that can be reloaded, benchmarked, and deployed.

The stack is built around a typed, serializable representation, replacing the usual patchwork of one-off repos, hand-authored contrastive prompts, and custom pruning scripts. Users instantiate a Pipeline with a model from the architecture registry—options include Llama-3, Gemma, Qwen, or GPT-2—and a declarative task such as "ioi." The pipeline runs discovery, evaluates the result across six pillars, and optionally applies structural weight pruning, circuit-aware 3- or 4-bit mixed-precision quantization, activation steering, ROME/MEMIT knowledge editing, or circuit-restricted LoRA that updates only the components inside the subgraph. The entire process is accessible through a Python API, CLI, or YAML config, with the GPT-2 quickstart running on CPU in minutes without requiring a GPU.

Operationally, on GPT-2 IOI, the default EAP-IG algorithm achieves a P1 causal patching score of approximately 0.9, above the 0.75 threshold considered strong for a circuit. Below 0.6, the circuit is deemed unfaithful. For memory-constrained runs, a bfloat16 precision setting, three integrated-gradient steps, a batch size of one, and reduced example counts can cut peak VRAM by roughly 80 percent. This optimization is crucial as CircuitKIT is task-specific; the exported checkpoint is a behavior-specialized subgraph, not a general-purpose compressed model.

There is no production evidence yet of CircuitKIT running inside a live serving stack or undergoing an enterprise safety audit at scale. Architects would need to see a case study on a production-weight model—such as a 70 B-parameter Llama-3 deployment—before building a pipeline around it. This would include end-to-end latency of the discovery pass, dollar cost per circuit, and downstream regression metrics on the pruned checkpoint versus the full model. The current release has gaps: only EAP and EAP-IG are marked stable; ACDC and IBCircuit are experimental, and the remaining nine algorithms are research-grade. Pillar 6, generalization, is implemented but flagged as unreliable across model families until v1.1. The LSAL license adds procurement friction—research, evaluation, and audit uses are free, but commercial deployment of exported checkpoints requires a separate license, potentially slowing enterprise adoption compared to an Apache-2.0 equivalent.

Architects should treat the discovered circuit as the scope of intervention—prune, quantize, and fine-tune strictly within that subgraph to build task-specialized checkpoints without affecting the rest of the weights.