BuddhaBERT

Active

NLPContemplative ScienceFrom-Scratch Training

Hypothesis

Train the same small language model on different reading lists — and separately, change its internal wiring along contemplative lines — and see whether either one changes how it represents experience, attention and awareness. Not just which words it knows, but how it holds them.

Overview

Figure 1. BuddhaBERT architecture — a small from-scratch transformer with three contemplative modifications: attention smoothing, an equanimity penalty on confidence variance, and a periodically-reset prediction head.

Contemplative traditions have spent a very long time describing subjective experience from the inside — the structure of attention, the dynamics of awareness, what happens to a mind that grasps at things. Those texts are not merely about consciousness; they are written from within it.

This experiment asks a simple question about that material: does what a model reads change how it thinks, or only what it knows?

How it is set up

Four small models, trained from scratch. Two reading lists — contemplative texts, or general Wikipedia. Two architectures — unmodified, or carrying three changes meant to make the model structurally contemplative. Every combination gets trained, so the reading list and the wiring can be told apart rather than confounded.

Being small is the point, not a limitation. A model we can see inside is worth more here than a model that performs better. That reasoning is set out in The Descaling Hypothesis.

What we found

The factorial has been trained and measured. Three findings, stated with their caveats attached, because the caveats matter as much as the results.

The contemplative models are more even-handed in their confidence — but mostly for an uninteresting reason. They are simply less confident about everything, and that mechanically shrinks the spread we were measuring. Correct for it and only one of the four — the model that got both the contemplative wiring and the contemplative reading list — is genuinely more even-handed. The other modified model shows no real effect at all. So the property is real, but narrow, and it needs both ingredients.

Their attention is measurably more spread out — but the modification forces that directly. The mechanism blends roughly half the attention toward uniform from the first training step, and the part that was supposed to learn how much to blend barely moved. So the effect is real and consistent across every layer, but it was imposed by construction rather than discovered by the model. That is worth knowing and worth saying plainly: it is not evidence of anything the model worked out for itself.

There is a real cost. The contemplative models are roughly one and a half to two times worse at predicting text. Forcing attention to stay spread out trades focus away.

These measurements are directional rather than publication-grade — the prompt sets used were smaller than the design calls for, and the statistical tests have not been run yet. We are reporting them at that strength and no higher.

What has not been done

The part that motivated the experiment in the first place — ask the four models the same interesting questions and see whether the answers differ in interesting ways — has not been done yet. Everything measured so far is an internal statistic. Nobody has yet sat down with the four models, put the same prompts to each, and read the answers side by side.

That work is specified and queued: thirty-six prompts across six categories, five samples each, with the four models' outputs shuffled and unlabelled so they can be scored blind. It is the next thing to happen here, and it is the piece closest to the original question.

One further gap worth naming: a third reading list — deliberately contrary material, to test the effect from the other direction — is designed but has not been acquired. The factorial has run as a two-by-two, not the fuller design.

What a negative result would mean

If the contemplative reading list turns out to produce only surface vocabulary changes with no structural difference, that is informative and we will report it as such. It would suggest that architectures of this kind cannot internalise the relational structure of contemplative concepts, which points at a real constraint rather than a disappointment. Given that the two effects found so far are one confounded and one imposed by construction, that outcome is genuinely on the table.

Two operating modes

Figure 2. Mode A (retrieval-augmented inference) is a fast prototype built on existing embeddings. Mode B (trained models) is the from-scratch factorial described above, run as a two-by-two of architecture against reading list.

Methodology

Build a contemplative reading list: Pali Canon suttas (Satipatthana, Anapanasati, Dhammapada, Kalama, Fire Sermon, Metta and others), the Heart Sutra, and transcribed dharma talks from Alan Watts, Ram Dass and Joseph Goldstein.
Build a matched control reading list of general-interest Wikipedia writing — science, history, geography — so that any difference can be traced to the contemplative material specifically.
Train small models from scratch, roughly 14 million parameters each, rather than adjusting a large pretrained one. Small enough that we can actually measure what is happening inside.
Cross that against three changes to the model internals: spread its attention evenly instead of letting it focus sharply, penalise it for being far more confident on familiar topics than unfamiliar ones, and periodically wipe its output layer so it cannot get attached to its own predictions. Four models result: modified or not, contemplative reading list or not.
Measure four behavioural properties across all four: how evenly its confidence is spread across topics, how much it degrades under provocative prompts, how spread out its attention is, and how stable it stays when the same question is asked different ways.
Read the four models side by side on a fixed set of prompts, scored blind, to test whether the reading list changes what the model actually says — not only what its internal statistics look like.

Status