Reading a neurotechnology paper

Neuroscience and neurotechnology papers are frequently reported far beyond what they claim. A few structural checks resolve most of the gap.

How many subjects

The first number to find and frequently the hardest to locate.

Implanted BCI studies typically involve one to five participants. That is appropriate for the stage of the field and it bounds what can be concluded.

A result from a single participant demonstrates possibility. It does not establish that the approach works generally, and the reporting frequently implies otherwise.

For imaging and behavioural studies, small samples are a chronic problem in the field, and effect sizes from small samples are systematically inflated.

Within-subject or between-subject

Determines what the comparison actually is.

Within-subject designs compare the same person under different conditions, which controls for individual variation and is far more powerful at small sample sizes.

Between-subject designs compare groups, which requires larger samples to detect the same effect.

What the control condition was

The single most informative methodological detail.

For stimulation studies, was there a sham condition? Sham stimulation — electrodes applied, brief ramp-up, then nothing — is the standard, and studies without it cannot separate the intervention from expectation.

Was the sham credible? Participants can frequently tell whether they received real stimulation, which undermines blinding. Good studies report whether participants guessed correctly.

Was the experimenter blinded? Unblinded assessment of a subjective outcome is unreliable.

Was the analysis preregistered

Increasingly reported and worth looking for.

Neuroimaging analysis involves many defensible choices — preprocessing steps, region definitions, statistical thresholds — and the number of possible analysis paths is very large.

Exploring them until something is significant produces findings that do not replicate. Preregistration commits to the analysis in advance and substantially raises the value of a positive result.

Where a study is exploratory, that is legitimate and should be stated. The problem is exploratory analysis presented as confirmatory.

Correction for multiple comparisons

Specific to imaging and electrophysiology.

A brain image contains many voxels; an electrode array many channels. Testing each independently at a conventional threshold guarantees false positives.

Papers should state how they corrected. Uncorrected results across a whole brain volume should be read as hypothesis-generating at best.

Effect size, not just significance

A statistically significant effect can be trivially small.

For a clinical claim, the question is whether the effect is large enough to matter to a patient — which is a different threshold from whether it is distinguishable from zero.

Papers reporting only p-values without effect sizes and confidence intervals are giving you less than half the information.

What the outcome measure was

Frequently where the gap between paper and headline opens.

A study may report improvement on a specific laboratory task. The headline reports improved memory, or improved focus, or restored function.

Task performance in a controlled setting and functional benefit in daily life are different claims, and transfer between them is the exception rather than the rule.

Look for whether any real-world outcome was measured at all.

Duration and follow-up

For anything implanted or chronic, the timescale is the finding.

Results at three months say little about a device intended to last decades. Long-term follow-up data is rarer and considerably more informative.

Conflicts and funding

Standard practice to declare, and worth reading.

A study of a device conducted by its developers is not thereby invalid — they have the expertise and access — and it warrants more weight on independent replication.

Where the paper and the press release diverge

Institutional press releases are written to attract coverage and are frequently the actual source of overstated reporting rather than the journalists.

Studies comparing press releases with the papers they describe have found that exaggeration in coverage correlates strongly with exaggeration in the release.

Which means that where a claim seems stronger than the methods support, the divergence frequently occurred before any journalist was involved.

The remedy is to find the paper. Abstracts are freely available, most methods sections are readable with patience, and authors are generally more careful in their own text than the summary around it.

Where a study is behind a paywall, corresponding authors will usually send a copy on request, and preprint servers increasingly carry a version.

The quick version

Find the sample size. Find the control condition. Find the effect size. Find the follow-up duration. Check whether the outcome measured is the outcome claimed.

Five things, all in the methods and results sections, and they resolve the great majority of overstated reporting without requiring any domain expertise.