12: Reporting and referencing

Write what was done, share the results, and cite the tools.

Introduction

A result is only useful if others can tell how it was made, and make it again. evoparam keeps a record of every step (tutorial 07). This last tutorial turns that record into:

1. the methods paragraph of a paper;

2. a report to share (HTML, PDF, Markdown) and a full JSON file;

3. the citations of evoparam and of the methods it used;

4. a way to make the result again later.

import os
import tempfile

import evoparam
from evoparam import fit_response, gen_response, reports

times, signal = gen_response([[25, -150, 10], [150, -80, 80]], seed=3)
fit = fit_response(times, signal, preset="ccep", delta=20, random_state=0)

1. The methods paragraph

reports.methods_text writes, in plain English, every step that was recorded: the preset and its version, the model, how bumps were chosen, the windows of the names, how the uncertainty was computed, every setting changed from the preset, the random seed, and the citations.

print(reports.methods_text(fit))

Evoked responses were analysed with evoparam (version 0.1.0.dev0) using the preset 'ccep' v1.2; validation: ds004080 (REQ-113, REQ-114, REQ-135). In the window 9 to 500 ms, each waveform was modelled as an exponential background plus up to 6 components (skewed gaussian and gaussian; full width at half maximum 3 to 250 ms, both polarities). Amplitudes and background coefficients were solved by variable projection and the other parameters by bounded least squares (linear loss, 3 starts). Components were added one at a time and kept when the BIC decreased by at least 20, computed on residuals whitened by an autoregressive model of the noise of the baseline. A final model holding two overlapping components of opposite sign was refitted from starts on the extrema of the data and replaced when the refit was better; a component of the refitted model that no longer lowered the criterion by the same amount was removed. Uncertainty: Jacobian-based (Wald) intervals at 95 %. Onset and offset are the times where a component reaches 10 % of its peak amplitude. Components were labelled by latency window: N1 (9 to 50 ms, negative), N2 (50 to 300 ms, negative) and delayed (100 to 1000 ms, either polarity). A response was said to have an N1 when the fitted curve (background and components) had a negative peak in 9 to 50 ms deeper than 3.4 times the noise standard deviation estimated by the fit. Classical measures (peak, mean amplitude, area and fractional-area latencies) were computed on the same window. Settings changed from the preset (times in s): delta = 20. Random draws used the seed 0; the result can be regenerated from its recorded provenance with evoparam.reproduce. Software: Bellec Y.V. evoparam: parameterizing evoked neural responses (version 0.1.0). https://github.com/yannbellec/evoparam

The setting we changed (delta=20) is written in it: “kept when the BIC decreased by at least 20”. This is why it is safer to give settings to evoparam than to change the results by hand afterwards: what is given is recorded.

With real data, run through reports.run_pipeline (tutorial 08), the same function also describes the reading, the artifact removal, the re-referencing, the baseline and the test of a response.

Note

Adapt the wording to your paper, but keep the numbers: they are the ones that were used. cite=False leaves out the citations, if you add them yourself.

2. Reports to share

Three report formats hold the methods text, a table of the results, a summary of the quality of the fits, and the steps of the record:

• reports.html(result, path): one HTML file, with the figures inside it, that opens in any browser without internet;

• reports.pdf(result, path): the same as a PDF;

• reports.markdown(result): the same as Markdown text.

reports.to_json(result, path) writes everything (every number, setting and step) in one JSON file, which any language can read.

folder = tempfile.mkdtemp()

page = os.path.join(folder, "my_fit.html")
reports.html(fit, page)
print(f"HTML report: {os.path.getsize(page) / 1000:.0f} kB")

everything = os.path.join(folder, "my_fit.json")
reports.to_json(fit, everything)
print(f"JSON file: {os.path.getsize(everything) / 1000:.0f} kB")

HTML report: 73 kB
JSON file: 52 kB

With many electrodes (a group fit, or the result of run_pipeline), the reports show one figure per electrode, up to max_channels. To store results next to a BIDS dataset, use evoparam.io.write_bids_derivatives.

3. Citations

evoparam.cite() prints how to cite evoparam (with its version); bibtex=True gives an entry for a reference manager. The methods text also cites the published methods that the steps used (for example the false-discovery-rate correction of the test of a response).

text = evoparam.cite()

Bellec Y.V. evoparam: parameterizing evoked neural responses (version 0.1.0). https://github.com/yannbellec/evoparam

entry = evoparam.cite(bibtex=True)

@software{bellec_evoparam,
  title = {evoparam: parameterizing evoked neural responses},
  author = {Bellec, Yann V.},
  version = {0.1.0},
  url = {https://github.com/yannbellec/evoparam},
  license = {Apache-2.0},
}

Also cite the data: for the recording of tutorials 08 and 11, van Blooijs et al. (2023), Nature Neuroscience, and its OpenNeuro identifier (ds004080).

4. Making the result again

A saved fit (fit.save, tutorial 07) holds its full record: the version of evoparam, every setting, the seed, and where the data came from. evoparam.reproduce(path) runs the fit again from that record and compares every number with the saved ones. When the data were not read from a file (for example a curve made in Python, as here), give them with data=.

The record also lists the computer’s versions of Python and of the main packages, so that a difference in a later run can be traced to its cause.

saved = os.path.join(folder, "my_fit.zip")
fit.save(saved)
print("saved with evoparam", fit.provenance.package_version)

saved with evoparam 0.1.0.dev0

Conclusion

evoparam writes the methods paragraph from what was actually done, makes reports to share and a complete JSON file, says how to cite it, and keeps what is needed to make a result again. Report every setting you changed, each number with its 95 % range (tutorial 06), and cite both the tools and the data.

This is the end of the tutorials: from what the model is (01) to how to report it (12).