Dr Jessica Rumbelow
Jessica Rumbelow

AI interpretability research and applications.

I design algorithms that allow humans to understand machine learning models: what they learn from their training data, how they make predictions, and where they might fail in unexpected ways.

Previously I built and interpreted models on histopathology data for cancer diagnostics, and did my PhD on model-agnostic interpretability (methods that explain a model's decisions without needing to see inside it).

Since then I've worked on AI interpretability for: failure modes of large language models (one of those, a set of tokens that broke GPT-3, made the news); scientific discovery (if a model can predict something you can't, it knows something you don't); and high-stakes applications for financial institutions where understanding model behaviour is critical.

I'm based in San Francisco.

Now

Financial institutions use machine learning to make decisions with real consequences, and Halley allows them to understand and explain their models with unprecedented clarity.

Existing tools typically provide feature importances and score pre-specified subgroups – but that's not enough to really understand the patterns a model is doing. Halley explains the interactions a model depends on, the thresholds it leands, and the feature combinations where it systematically under- or over-predicts.

It also works on any kind of model – and this unlocks the ability to use more expressive and performant architectures that were previously out of reach due to their opacity.

Building now

Halley

Model risk and compliance teams use Halley to deeply understand their models, surfacing the interactions, subgroup effects, and conditional patterns that other tools miss.

Recent work
Interpretability

An Introduction to Exemplar Partitioning

A new method for finding interpretable structure in language models 1000x faster than sparse autoencoders.

Essay

Data-Driven Discovery vs Hypothesis-Driven Research

Hypothesis-driven analysis can only ever find what you already suspect is there – here is a case for starting with the data instead.

Case studies

New Publications in Immunology, Plant Biology and Meteorology

Three publications where models trained on scientific data found novel patterns, and our interpretability methods made them legible. These findings were discovered automatically by Disco and written up with the domain experts who gathered the data.

Time-lapse of Arabidopsis seedlings growing
Essay

Scientific Discovery in the Age of AI

A survey of how AI is used in science today, from systematic literature review to autonomous agents.

Tool

Why Did ChatGPT Say That?

An efficient black-box method for attributing a language model's output back to the parts of the prompt that caused it, even if the model is hidden behind an API.

Discovery · Language models

SolidGoldMagikarp

The glitch-token work: we discovered a set of very strange tokens that made GPT-3 break in unexpected ways.

Media

Talks & interviews

Podcast · Latent Space

Automating Scientific Discovery

A conversation with Swyx about data-first science and seeing what models learn.

Podcast · The Human Podcast

How AI Will Unlock Key Scientific Discoveries

On personal motivation, how AI can (and can't) aid scientific discovery, and why interpretability is important.

Podcast · Tech Brew Ride Home

Leap Labs

A segment with Brian McCullough on interpretability, AI, and seeing patterns that humans miss.

Pitch · Deep Tech Momentum

Deep Tech Momentum

The pitch that won us the Novel AI track + many prizes at DTM.

Live · Presence

AI for Knowledge Discovery: Finding Patterns Humans Can't See

A live session on using interpretability to surface patterns that standard analysis misses.

Press