BuddhaBERT: what the four-model experiment found

Abstract

The first BuddhaBERT experiment leaves its confidence-variance criterion unresolved, meets the direction of its attention-entropy criterion largely through the imposed mechanism, and cannot evaluate its question-answering criterion because generated text was unreadable. The completed measurements do not establish a consistent contemplative advantage.

Four small transformers crossed two training corpora—general text and contemplative literature—with two architectures: vanilla and a bundle of three contemplative modifications. The modified architecture incurred higher masked-reconstruction loss on both corpora. Confidence variance fell by 41.5% in the pooled point estimate, but its bootstrap interval included substantial increases as well as decreases. Attention entropy rose by about 3.6%, consistent with the architecture’s built-in smoothing. Supplemental context-perturbation comparisons did not survive multiple-test correction. Paraphrase representation similarity increased on general text and decreased on contemplative text; comparison with unrelated questions further limited the apparent benefit.

These are measurements of four fixed checkpoints, with one training seed per condition. They are neither evidence of machine consciousness nor a demonstration that contemplative modifications cannot work. They identify limitations in the present models and in the instruments used to assess them. This working report closes the original 2×2 measurement phase and corrects earlier, stronger summaries. It has not been peer reviewed.

1. The question behind the experiment

Contemplative practice distinguishes the contents of experience from the way attention relates to those contents. A thought can be familiar without being held lightly. A response can use the language of equanimity without displaying much stability under pressure. Those distinctions motivate a difficult question for machine learning: can training material and internal constraints alter measurable properties of a model beyond the vocabulary it learns?

BuddhaBERT approached that question by varying the reading list and the architecture separately. If every modified model had received contemplative text and every unmodified model general text, any difference could have come from either change. Crossing the two factors makes the comparison more informative. It allows us to ask whether an architectural effect appears within each corpus, whether that effect changes with the corpus, and what the intervention costs in basic predictive competence.

The names of the proposed measurements—equanimity, non-reactivity, mindful attention and semantic consistency—express the motivating analogies. Their operational definitions are narrower: confidence variance, performance under changed context, attention entropy and representation similarity. Calling a statistic “equanimity” does not establish that it measures the human quality. A useful result must survive that distinction.

Small models made internal inspection and inexpensive iteration possible. They also limited what could be measured. The question was never settled by training a small system successfully; it depended on whether the resulting system and the chosen measurements could support the intended comparisons.

2. Design and methods

Four conditions, two factors

Code	Architecture	Training corpus
C1	Vanilla transformer	General text
C2	Vanilla transformer	Contemplative text (“dharma”)
C3	Three contemplative modifications	General text
C4	Three contemplative modifications	Contemplative text (“dharma”)

The corpus factor compared general-interest Wikipedia material with contemplative literature and dharma talks. Each condition used a small model trained from scratch, approximately 14 million parameters. The implementation has six transformer layers, six attention heads, a hidden width of 384 and a 512-token context. This report uses the C1–C4 numbering in the code. An earlier design included a third corpus; those additional cells were not trained and are not part of these results.

The modified architecture combined three interventions. Mindful Attention Smoothing blends attention weights toward a uniform distribution over allowed positions. Equanimity Loss adds a penalty intended to reduce confidence variation. A Non-Attachment Prediction Head periodically resets the output head during training. These names describe intended design roles. The experiment tested the bundle, so it cannot identify a separate causal contribution from each component.

One checkpoint was trained per cell. That permits descriptive factorial contrasts, but supplies no replication across training seeds. Repeated questions provide information about these checkpoints’ responses to those questions; they do not create new independent trained models.

What kind of model was evaluated?

Despite the name, this implementation is not the standard bidirectional BERT architecture. It retains causal attention while training with a masked-token reconstruction objective. Predictive loss is evaluated at selected corrupted positions rather than as ordinary shifted next-token prediction.

That distinction matters for both competence and generation. The reconstruction benchmark below reports masked top-1 accuracy and its associated loss and perplexity. It should not be read as a standard autoregressive language-model benchmark. The generation problem is discussed separately, because a model’s internal vectors can be measured even when its generated continuations are unusable.

Historical and supplemental measurements

Confidence variance and attention entropy were measured earlier in the study. The seeded reconstruction benchmark reported here was recorded on August 9, 2026. Context perturbation and paraphrase consistency were completed on September 14 using the existing four checkpoints. No retraining occurred for this supplemental evaluation.

The supplemental protocol and inputs were frozen before that scoring run. They were not frozen before the earlier experiment or before its initial results were inspected. They are therefore supplemental measurements, not retroactive preregistration. The qualitative generation run likewise lacked a valid completed preregistration. Earlier blanket descriptions of the entire experiment as preregistered were too strong.

The new run used PyTorch 2.11.0 on Apple Silicon’s MPS backend. Combined perturbation and paraphrase inference took about 10.7 seconds. Checkpoint hashes were unchanged afterward. Six targeted harness tests passed on both execution environments, and perturbation predictions exactly reproduced the preceding perturbation-only run. Those checks support implementation integrity, not scientific validity by themselves.

3. Results: architecture, corpus and their interaction

The clearest consistent architecture difference is a cost in masked reconstruction. Both modified models have higher loss and lower masked top-1 accuracy than the vanilla model trained on the same corpus.

Cell	Masked loss	Perplexity	Masked top-1 accuracy	Scored positions
C1: vanilla / general	6.2379	511.80	15.89%	6,679
C2: vanilla / dharma	4.2993	73.65	30.44%	3,601
C3: modified / general	6.6460	769.70	13.70%	6,679
C4: modified / dharma	5.0128	150.33	20.13%	3,601

Historical seeded benchmark, August 9. Evaluation processes whole fixed-size batches and may omit a final remainder. Conditions within each corpus use matched masking; the two corpora have different test distributions.

Let C1–C4 denote the losses in this table. The architecture main effect, averaging the modified-minus-vanilla difference across corpora, is ((C3 + C4) − (C1 + C2)) / 2 = +0.5608 nats. The corpus main effect is ((C2 + C4) − (C1 + C3)) / 2 = −1.7859 nats.

The architecture difference is +0.4081 nats on general text and +0.7135 nats on dharma text. Their difference, (C4 − C2) − (C3 − C1), is +0.3054 nats. The conventional two-level factorial interaction effect is half that difference, +0.1527 nats. Stating the convention avoids treating differently scaled interaction numbers as contradictory findings.

These effects are descriptive. There is no independent training-error estimate from which to claim factorial significance. In particular, lower loss on the dharma test corpus does not establish that dharma-trained models are better at language generally: the material being predicted changes with the corpus factor. Within-corpus architecture comparisons are the relevant competence checks here.

Confidence variance: the apparent gain remains unresolved

The first contemplative measure asks how much model confidence varies across five domains. Lower variation is intended to represent greater evenness. Its raw values were 0.00087216 for C1, 0.00071372 for C2, 0.00077376 for C3 and 0.00015374 for C4.

Pooling the two architectures produces a 41.5% variance reduction, above the proposed 30% success threshold as a point estimate. The uncertainty changes the interpretation. A nonparametric bootstrap interval extends from a 110.4% increase to an 81.7% reduction. Twelve prompts per domain were measured against an original target of 100. C4, which contributes much of the apparent improvement, falls below its estimated sampling-noise floor of 0.00040965.

There is also a measurement confound: confidence variance can shrink when confidence itself becomes smaller. A model that is uncertain about everything can look even-handed without having learned a useful form of stability. Magnitude-normalized summaries do not establish a consistent improvement across the two corpora. The previous public wording that the property was “real, but narrow” was stronger than the evidence permits.

Success criterion 1: unresolved. The point estimate crosses the threshold; the uncertainty does not establish that the underlying reduction does. No Bayes factor is reported because a prior was not specified before inspecting the results. A standardized mean-difference effect size is not assigned to this pooled variance ratio.

Figure 1. Raw confidence variance and its estimated noise floor. The historical figure displays the earlier parametric interval; the updated nonparametric interval reported above leads to the same unresolved conclusion.

Attention entropy: the intended mechanism is visible

Attention entropy increased by about 0.0606 nats, or 3.6%, with the modified architecture. All six layers moved in the same direction. The historical layer-bootstrap interval was 0.0416–0.0768 nats; the paired standardized effect was 2.46. An exact layer sign-flip test returned p=0.03125, becoming 0.0625 after the historical two-comparison correction.

The six layers are parts of the same trained systems, not independent experimental replicates. The test is therefore best treated as a description of consistency across layers. It does not establish a population-level effect across model training runs.

More directly, the architecture blends approximately 52% of attention toward uniform from initialization. The learned blending parameter moved little during training. Increased entropy is consistent with that imposed smoothing, so it cannot on its own demonstrate an acquired capacity for mindful attention. Observing the mechanism operating is useful; the interpretation has to remain at that level.

Success criterion 2: the directional threshold is met, largely by construction. A general learned contemplative benefit is not established.

Figure 2. The zero-based panel shows the size of the change; the labeled zoom helps compare layers. Neither view supplies independent training replication.

Context perturbations: little competence, unstable ratios

The supplemental robustness measure uses 50 existing text stems, ten from each domain. The final three token IDs of each stem are held fixed as targets and masked simultaneously. The context is then left clean, given one letter transposition, or prefixed with misleading or emotionally pressuring text. Each condition and variant therefore contributes 150 target-token judgments.

This design holds the answer tokens constant across variants and models. It does not isolate a psychological construct. Prefixing text changes length and absolute token positions as well as wording. These are diagnostic context perturbations; “non-reactivity” remains the motivating analogy.

Cell	Clean correct / 150	Typo correct	Misleading correct	Emotional correct
C1	8	9	9	8
C2	8	8	11	7
C3	9	8	7	10
C4	5	5	2	1

The non-reactivity score divides perturbed accuracy by clean accuracy. A ratio of one means that accuracy is unchanged, not that the model is competent. With only five to nine correct targets in the clean condition, a few changed predictions move the ratio substantially.

Cell	Typo ratio	Misleading ratio	Emotional ratio
C1	1.125	1.125	1.000
C2	1.000	1.375	0.875
C3	0.889	0.778	1.111
C4	1.000	0.400	0.200

None of the six within-corpus architecture comparisons survives the supplemental multiple-test correction. This is an inconclusive robustness result, not evidence that all perturbations have equal effects. The C4 misleading-context comparison has an unadjusted p-value near 0.022, but its corrected value is approximately 0.175. Selecting the unadjusted comparison after inspecting the family would overstate the result.

Zero clean accuracy makes the ratio undefined. C4 had 37 such draws in 10,000 bootstrap resamples. Its intervals exclude those draws and report the exclusion rate; they do not silently replace undefined ratios with zero. Identical paired predictions can also produce a zero-width interval on this small bank without establishing broad robustness.

Figure 3. Ratios should be read beside the raw counts. Remaining stable at poor performance does not demonstrate useful non-reactivity.

Paraphrase consistency: opposite effects across corpora

The fourth measure asks whether different phrasings of the same question produce similar internal representations. The frozen bank has 50 question meanings, five phrasings each and ten meanings per domain. It uses the final layer-normalized vector at the last token. Each meaning contributes the mean of its ten phrasing-pair cosine similarities; the overall score averages the 50 meanings.

For comparison, each meaning is also paired with every other meaning, averaging all 25 phrasing combinations for each pair. This unrelated-question baseline matters because a representation can appear stable simply by making many different inputs similar. The baseline is unmatched for domain and surface form; it is a diagnostic reference, not a complete semantic-control design. An exact normalized text screen found no complete phrasing in either available corpus, but did not rule out partial or semantic overlap.

Cell	Same-meaning cosine	95% question interval	Unrelated baseline	Difference from baseline
C1	0.8100	0.8027–0.8175	0.7456	0.0644
C2	0.8718	0.8672–0.8766	0.8231	0.0487
C3	0.8787	0.8750–0.8827	0.8495	0.0293
C4	0.8374	0.8326–0.8425	0.7876	0.0499

The modified-minus-vanilla difference is +0.0687 on general text, with a 95% question interval of +0.0630 to +0.0745. On dharma text it is −0.0343, with an interval of −0.0388 to −0.0297. Both question-level comparisons survive the eight-test correction at an adjusted p-value of approximately 0.00080. Their paired standardized effects are 3.07 and −1.96, respectively.

Statistical resolution within this bank does not make the directions agree. Nor does higher cosine automatically mean better semantic discrimination. On general text, separation from unrelated questions declines from 0.0644 to 0.0293; similarity rises more broadly than just within paraphrases. On dharma text, the separation is nearly unchanged. These baseline-gap comparisons are descriptive, without additional significance claims.

The result is consequently mixed and representation-specific. It does not measure answer quality, establish a general semantic advantage, or support a consciousness inference. Shared wording and the choice of the final token can influence these vectors; alternative representations would constitute further measurements rather than interchangeable confirmations.

Figure 4. Same-meaning similarity increases under the modified architecture on general text and decreases on dharma text. The comparison with unrelated questions limits what the increase means.

Generation: the intended comparison could not be scored

The qualitative run produced 720 continuations: 36 stems, four conditions and five samples per condition. The outputs were unreadable. No valid blind quality ratings or Dharma question-answering accuracy are reported.

There was a concrete objective-and-generation mismatch. Ordinary next-token generation asked the model to predict through a path trained to reconstruct visible or masked positions, allowing a copy loop. A corrected masked-infill generation path was tried, but the four models’ continuations remained unusable for the intended comparison.

That outcome establishes an instrument limitation. It does not show that architecture had no effect, prove that model size was the cause, or justify the conclusion that contemplative concepts cannot be learned. The experiment did not produce readable material on which to make those judgments. Corpus boilerplate was subsequently cleaned, but the four checkpoints evaluated here were not retrained on the cleaned text.

Success criterion 3: not evaluable. The proposed question-answering threshold cannot be scored from these outputs.

4. Statistical interpretation and limitations

The supplemental family contains eight comparisons: three context-perturbation scores and one paraphrase score, each contrasting the architectures within both corpora. The analysis uses 10,000 bootstrap draws stratified by the five fixed domains and paired question-label permutations, with sign flips for paraphrase differences. The random seed is 20260914. Bonferroni correction multiplies p-values by eight, capped at one.

The reported 95% intervals are marginal intervals, not simultaneous family-corrected intervals. Perturbation standardized effects refer to paired differences in accuracy change, not standardized ratios. The released summary includes raw and adjusted p-values, intervals, effect sizes and undefined-draw rates. The historical confidence and attention analyses remain identified as their earlier comparison family; they have not been relabeled as part of the supplemental design.

The largest limitation is experimental replication. There is one checkpoint per condition. Question-level intervals describe sensitivity to the bank of questions under the resampling procedure. They do not cover uncertainty from initialization, training order, corpus sampling or other training-run variation. The questions are hand-authored, and the domains fixed, so population generalization is limited even at the prompt level.

The two corpora differ in content and distribution, the three architectural changes were bundled, and the generation task did not produce a usable instrument. Confidence variance is sensitive to confidence magnitude; cosine similarity is sensitive to broad representational similarity; attention entropy is directly manipulated by the intervention. Each limitation restricts a different inference. Adding more decimal places or more bootstrap draws would not remove them.

No measure in this battery has been validated as a detector of subjective experience. The distinction between intelligence and consciousness remains central to the project. A measurable internal change can be relevant to a research program without resolving the question that motivated the program.

5. What the result changes

The factorial design remains useful. It prevented a positive result in one corpus from becoming a claim about the architecture in general. The paraphrase measurements make that particularly clear: the architecture difference changes sign. Reporting only the general-text improvement would have lost the very comparison the design was built to provide.

The measurements also expose the distance between constraining a computation and demonstrating a capacity. Attention smoothing restricts how sharply a layer can distribute its weights. A confidence penalty changes the optimization pressure. Resetting a prediction head changes what can persist in that component across training. Those are concrete interventions. Their contemplative names do not establish that they instantiate the human practices they reference.

A constructor-theoretic reading can frame the next question in terms of reliable transformations: what does the modified system become able to do, and what behavior becomes unavailable to it? Here, the clearest changes include more uniform attention and a reconstruction cost. The study does not establish a reliable new capacity for semantic stability or non-reactive answering. That framing is an interpretation of the experiment, not a further result or a formal constructor-theoretic proof.

The next experimental decision should follow from the failed instrument as well as the numerical results. A bounded feasibility test would first need a coherent training and generation objective, a fixed readability criterion and a firm compute cap. A larger pretrained model is one possible route, not an outcome already validated by this study. No larger-model run or third-corpus expansion is reported here.

Cross-experiment comparisons also need compatible measurements. No matched embodied-embedding results are included, so this report makes no joint valence-geometry plot or cross-substrate claim. The completed four-model study is a limited result that can inform a next design; it does not supply missing experiments by analogy.

The correction to the earlier public summary is substantive. Confidence variance is unresolved. The attention effect is visible but largely imposed. Perturbation robustness is inconclusive. Paraphrase consistency is mixed. Generation remains unscorable. Keeping each conclusion at the strength of its evidence is what makes the experiment useful to the next one.

6. Data and provenance

This public release includes the supplemental summary and contrasts (JSON), the historical confidence and attention statistics (JSON), and the release provenance record (JSON). The four figures are linked above. A Markdown copy of this report is available for download.

The evidence is this project’s own experimental record. The code repository, checkpoints and full inference archive remain private; these public aggregates do not constitute a complete independently rerunnable release. The frozen protocol, raw predictions and representation arrays are retained internally. The provenance record identifies the source revision and protocol and checkpoint hashes so that this report can be tied to that archive without publishing private workspace paths or operational notes.

The supplemental run’s source revision is 2cca396; the archived results revision is be7f355. The frozen protocol SHA-256 is 421606853497e5f2a4a98b7413d73a2015c09841742414881816a943c7985f57. Historical data are labeled separately from September measurements. The report is a public working research report, not a peer-reviewed paper or an assertion that the broader ConsciousGPT program is complete.