References

[rougier]

Unsorted references

• Writing Software Specifications K. Hinsen, Computing in Science and Engineering, 2015.
• A long journey into reproducible computational neuroscience M. Topalidou, A. Leblois, T. Boraud and N.P. Rougier, Frontiers in Computational Neuroscience, 2015.
• Computational science: shifting the focus from tools to models K. Hinsen, F1000Research, 2014.
• Best Practices for Scientific Computing Greg Wilson , D. A. Aruliah, C. Titus Brown, Neil P. Chue Hong, Matt Davis, Richard T. Guy, Steven H. D. Haddock, Kathryn D. Huff, Ian M. Mitchell, Mark D. Plumbley, Ben Waugh, Ethan P. White, Paul Wilson, PLOS Biology, 2014.
• Learning from the Past: Approaches for Reproducibility in Computational Neuroscience S.M. Crook, A.P. Davison, H.E. Plesser, 20 Years of Computational Neuroscience, 2013
• Four aspects to make science open “by design” and not as an after-thought Y.O. Halchenko, M.Hanke, GigaScience, 2015.

Replicability vs. reproducibility

• Replicability vs. reproducibility — or is it the other way around? Mark Liberman (2015)
• Open code for open science? Steve M. Easterbrook, Nature Geoscience 7, 779–781 (2014)

Similar initiative

• Push journal - Research & applied theory in writing with sources
• IPOL Journal - Image Processing On Line
• PSRI - Political Science Replication Initiative

[rougier]

On the issue of different words for reproducibility/replicability/repeatability/etc.

Via Simon Hodson on twitter:
Steps and distinctions in #reproducibility discussion from @CaroleAnneGoble #atir

[khinsen]

Interesting. I wonder about the distinction between repeat and replicate. How can you have "the same" setup in different labs? Put differently, what's the definition of "the same"? Anyway, the distinction hardly matters for computational work, where differences between physical machines are the least of our problems.

[oliviaguest]

Are we interested in drawing (attention to) the distinction between software used for research, but not an implementation of a theory, and software that is the experiment, that is an implementation of a model/theory?

[khinsen]

The distinction is useful to point out, but the software we care about always implements some model, as otherwise there would be no results to reproduce.

BTW, in my field, 99% of all scientific software implements models. I'd like to see this change, but I don't expect it to happen any time soon.

[oliviaguest]

I'm curious what you mean by model because I think everybody (at least in my field) has their own definition. And does your field discriminate between model and simulation as different concepts?

[jsta]

@oliviaguest Are you referring to software such as programming languages? They are in most instances not the subject of the research itself.

[oliviaguest]

@jsta No, I'm referring to the difference between code as research software VS code as research itself. An example for the former is a script written to run an experiment (let's say it runs an eye-tracking experiment, and then analyses the data) VS an example for the latter, a script written to implement a theory or model (let's say a piece of code that implements ACT-R).

Something that might shed light on what I might be getting at more broadly (why distinctions such as what a model is and is not are important, etc.) is this paper I co-authored: Cooper, R. P., Guest, O. (2014). Implementations are not specifications: specification, replication and experimentation in computational cognitive modeling. Cognitive Systems Research, 27, 42-49.

[khinsen]

@oliviaguest For me a model is a formalized combination of assumptions and hypotheses that can be tested by comparing with observations. That means that everything that influences this comparison is part of the model. A model is typically an assembly of many ingredients such as differential equations, numerical parameters, and algorithms. Simulations are just one such ingredient, and thus part of the model.

Note that this is my own view which not everyone in my field shares. Many physicists do distinguish between the "pure" model (equations and parameters) and the "dirty" computation (simulations). I see this as a form of snobbism that will disappear with the generation of "pure" theoreticians.

With those definitions, there isn't much software that does not include some piece of a model. One type is workflows and equivalent scripts, as you point out in your reply to @jsta. Another type is low-level code for data management such as HDF5. Compilers can be lumped with that as well. Scientists rarely write this second type of code, and the workflow layer makes up for a tiny part of the total code base.

[oliviaguest]

Have not read this yet, but it seems very relevant:

Software search is not a science, even among scientists
Michael Hucka, Matthew Graham
http://arxiv.org/abs/1605.02265

[khinsen]

An interesting study, but I don't see much of a link to re[.*]ability, other than that it's a criterion scientists should have in mind when choosing software, but apparently don't.

[oliviaguest]

I thought this was what remixable and reusable were about?

[khinsen]

Well possible - I am rather sure that I don't have a clear understanding of remixable at this time, and only a vague one of reusable. For example, does reusable imply anything about the quality of the results? Or is it sufficient that someone else can legally grab a copy and get it to run, potentially producing garbage?

[oliviaguest]

No idea, I personally have trouble with most of those words.

[oliviaguest]

This might be of interest to some of you: CONTEST: How do we ensure that research is reproducible?
Entries so far are here: https://thewinnower.com/keywords/ljafreproducibility

[khinsen]

And here is mine: https://thewinnower.com/papers/4770-verifiable-research-the-missing-link-between-replicability-and-reproducibility

[oliviaguest]

Oh, nice! Will hopefully submit and link to mine soon enough.

[oliviaguest]

This is a useful reference especially with respect to #5

Goodman, S. N., Fanelli, D., & Ioannidis, J. P. (2016). What does research reproducibility mean?. Science translational medicine, 8(341), 341ps12-341ps12. http://stm.sciencemag.org/content/8/341/341ps12.full