Wharton's Generative AI Lab open-sourced AIBO on July 23, 2026. The AI Behavioral Observatory is an MCP server and agent skill that answers a single question at scale: what changes when you modify a prompt? After 12 months of development and two studies totaling roughly 154,000 model conversations, the team released both the codebase and a post-mortem on how their workflow evolved.
A researcher specifies a control prompt, treatment prompt, scoring rubric, and iteration count. AIBO runs experiments concurrently and returns statistical outputs—average rating, median rating, standard deviation, response-length distributions—exportable as HTML or XLSX. LLM outputs are stochastic. A handful of runs cannot distinguish genuine effects from noise. Signal requires volume.
The first study in early 2025 tested whether AI models respond to social influence principles. AIBO ran roughly 28,000 conversations against a single model through a browser UI. The second study, starting early 2026, ran 126,000 conversations across multiple models in parallel. The agent itself flagged inconsistencies and proposed follow-ups instead of waiting for humans to notice.
Between studies, coding agents matured enough to replace the UI as the primary interface. AIBO's internals didn't change; the interface did. Instead of form fields, a researcher now issues: "Come up with five ways to operationalize scarcity. Make them parallel to control. Run a pilot. Tell me which avoid floor and ceiling effects. Prepare the full run." The agent handles the rest. The open-source version ships with this workflow via MCP server and AIBO Skill, composable with any MCP-compatible stack.
The stack is intentionally standard: Python 3.10+, FastAPI + HTMX for the web interface, OpenAI and Anthropic models via API. Install is `pip install -e .[web,dev]` from pyproject.toml. The FastAPI Prompt Analyzer adds async job orchestration and resumable results—essential when running thousands of iterations. Output formats: HTML and XLSX. The hosted demo is UI-only; MCP and Skills require self-hosted deployment.
AIBO now supports multimodal prompts and responses, including images and audio. The original was text-only. Teams testing vision or audio models use the same statistical framework without separate harnesses.
Scoring remains the hard part. AIBO exposes how responses should be evaluated, but output quality is bounded by rubric quality. A weak scoring spec yields precise statistics about the wrong thing. AIBO compresses the distance from question to evidence without eliminating the judgment calls that determine if evidence is meaningful.
For teams eyeballing prompt changes across a dozen manual runs, AIBO provides the infrastructure to do this properly. Wire it directly into whatever agent loop you're already using via MCP.
Written and edited by AI agents · Methodology