Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

EBRAINS Live Papers - Interactive Resource Sheets for Computational Studies in Neuroscience

EBRAINS Live Papers - Interactive Resource Sheets for Computational Studies in Neuroscience We present here an online platform for sharing resources underlying publications in neuroscience. It enables authors to easily upload and distribute digital resources, such as data, code, and notebooks, in a structured and systematic way. Interactivity is a prominent feature of the Live Papers, with features to download, visualise or simulate data, models and results presented in the corresponding publications. The resources are hosted on reliable data storage servers to ensure long term availability and easy accessibility. All data are managed via the EBRAINS Knowledge Graph, thereby helping maintain data provenance, and enabling tight integration with tools and services offered under the EBRAINS ecosystem. Keywords Computational neuroscience · Data retrieval · Data provenance · Documentation · Experimental data Introduction users to replicate at least a single figure from the correspond- ing publication. It is not intended to tackle issues regard- A major shortcoming in computational approaches to neu- ing access to experimental data, and therefore does not, in roscience, such as modelling/simulation and data analysis, general, hold information about data used to tune, validate is the absence of a widely used and well-defined system for or simulate the model. Similarly, OSB is a model repository sharing code and data that would enable researchers to easily that focuses on model formats and simulator-independent access the data resources employed in published studies and model representations, but not on prioritising the data under- understand in detail the provenance of published results and lying the models. NeuroMorpho.org is a curated inventory figures. This impedes community-based, collaborative mod- of digitally reconstructed neurons associated with publica- elling efforts, reduces the utility of published models within tions, but doesn’t link with other data such as electrophysi- the neuroscience community, and hinders the reproducibility ological recordings or models of the reconstructed neurons. of data analyses. More general services such as Figshare (Hahnel 2013), Open Existing tools and services, such as ModelDB (Hines et al. Science Framework (Foster and Deardorff  2017), or (Dillen 2004), Open Source Brain (OSB) (Gleeson et al. 2012), and et al. 2019) have much less structured records, with limited NeuroMorpho.org (Ascoli et al. 2007), make models, code domain-specific metadata. Finally, applications/infrastruc - and/or data available to the scientific community, with each tures specifically addressing reproducibility and transpar - attempting to fulfil a specific need. For example, ModelDB ency in scientific publishing (e.g., eLife RDS, Galaxy, Repro is a repository of published models with code that allows Zip) require possibly demanding ad hoc installations, con- figurations and maintenance operations (Konkol et al. 2020). EBRAINS Live Papers are interactive documents bring- * Andrew P. Davison ing together code, models, and data. They can be stand- andrew.davison@cnrs.fr alone publications, but in general complement published Université Paris-Saclay, CNRS, Institut des Neurosciences scientific articles. Interactivity is a prominent feature of the Paris-Saclay, Saclay 91400, France “Live Papers” with several integrated tools and services that Institute of Biophysics, National Research Council, allow users to download, visualise or simulate data, models Palermo 90143, Italy and results presented in the corresponding publications. By Blue Brain Project, École polytechnique fédérale de bringing together the various resources underlying compu- Lausanne (EPFL), Campus Biotech, Geneva 1202, tational approaches in neuroscience publications (whether Switzerland Vol.:(0123456789) 1 3 Neuroinformatics modelling/simulation or data analysis), they provide a more are (i) long-term, archival-quality data storage, provided complete picture of the original researchers’ workspace. By through an agreement with the Swiss National Supercom- virtue of being developed within the EBRAINS infrastruc- puting Centre; (ii) a state-of-the-art metadata store, the ture, Live Papers take advantage of EBRAINS platforms Knowledge Graph; (iii) a wide range of tools for modelling, and services for data and model integration and are able to simulation, and data analysis at different scales and levels of effectively leverage the Knowledge Graph (KG) database abstraction; (iv) facilitated access to high-performance com- for storing all information. The KG, being a graph-based puting systems; (v) collaborative workspaces (the Collabo- database, interlinks all data units thereby readily offering a ratory) with fine-grained access control for the documents, high degree of data provenance. code and data for a given project. It is this infrastructure, Below we describe the various features and function- fulfilling the requirements outlined above, that makes the alities currently made available through the Live Papers Live Paper platform possible. platform, and also how authors can distribute their own The Human Brain Project will be completed in 2023, but resources by creating and publishing live papers. Our goal the tools and services developed under it will continue to is to demonstrate to the scientific community the utility of exist as part of EBRAINS. It will continue to be funded by building live papers to complement their publications, and the participating countries, with national nodes being set to develop them not post facto, but rather as a tool to support up in multiple European countries, and with further fund- their manuscript submissions by making available informa- ing from the EC until at least 2026, to ensure continued tion that could assist reviewers in making a more informed availability of these services. Live Papers is one of several evaluation (Bailey et al. 2016). Computational studies in services in the EBRAINS ecosystem, and their sustainability neuroscience, being potentially fully deterministic, lack- is the primary objective of EBRAINS AISBL. ing the intrinsic variability of experimental studies, should be easily reproducible. In reality, this is often not the case, Live Paper Resources which to a large extent is due to the black box situation aris- ing from missing or incomplete access to data and docu- The live papers allow for diverse types of resources to be mentation. With the concept of Live Papers presented here, presented, with practically no limitations. For computational we hope to take significant strides towards resolving these modelling studies, we have found that the most common problems. resources being distributed comprise the following: Electrophysiological Recordings Overview Data obtained from experimental recordings often lay the In this section we begin with a brief overview of the foundations for modeling studies. These recordings can be EBRAINS infrastructure, and move on to describing the obtained from a variety of different experimental techniques various features and functionalities that are currently avail- and protocols, such as intracellular recordings via sharp able for live papers. microelectrodes or patch clamp setup, calcium or dye imag- ing studies, and so on. The Live Paper platform provides EBRAINS Infrastructure for listing and access of all kinds of experimental data. A typical characteristic of experimental data is that they are An online platform for sharing digital resources and pub- often recorded as time series. The platform integrates an lishing interactive, scholarly documents places several online visualiser for such data, as seen in Fig.  1, without requirements on the underlying infrastructure. Resources requiring users to download them. It handles all data for- and documents must be persistent—reliably available over a mats supported by the Neo library (Garcia et al. 2014). The long time period. Tools for visualising or re-using resources Live Paper platform currently integrates with the EBRAINS (models, code, data) should be available, reliable, give repro- Knowledge Graph and the Allen Brain Atlas (Jones et al.  ducible results, and be underpinned by adequate computa- 2009) to enable authors to link to previously registered elec- tional resources. Accurate and complete metadata must be trophysiological recordings from these databases. available. EBRAINS is a new digital research infrastructure devel- • Neuronal Morphologies oped under the Human Brain Project and now on the Euro- pean Strategy Forum on Research Infrastructures (ESFRI) Computational studies involving modeling of single roadmap. It aims to bring together an extensive range of neurons or networks of neurons typically employ digitally data, alongside tools to help analyse and utilise these data. reconstructed neuronal morphologies for configuring Among the key tools and services provided by EBRAINS the anatomical structure of model neurons. In the case of 1 3 Neuroinformatics Fig. 1 The electrophysiology resource section embeds an interactive visualiser for each electrophysiology data file. This enables users to visual- ise and to interact with the data live from within the document neuronal circuits and networks, a multitude of such neuronal It is almost always non-trivial to reconstruct a model from morphologies might be employed. As cellular biophysics scratch solely based on the model description provided in is greatly affected by the anatomical architecture, it is publications. Without access to the model source code, all essential to have access to such data to ensure reproducibility inferences and conclusions have to be evaluated at face value of results. The live paper tool allows authors to list all without any scientific appraisal. This has encouraged most the morphologies associated with their study, and allow scientific publishers to ensure that the source code underly - interested users to download these. Authors may also link ing modeling studies be made publicly accessible, or at least to established repositories such as NeuroMorpho.Org available on request. and the Allen Brain Atlas. The platform also integrates a Live papers allow authors to distribute the source code 3D morphology viewer that users can use to explore and for their models. As shown in Figs. 2 and 3, they also ena- examine the morphologies (Bakker and Tiesinga 2016). ble importing and linking to corresponding entries in other neuroscience repositories such as ModelDB, OSB and Bio- Model Source Code Models, in addition to the EBRAINS Knowledge Graph. Additionally, live papers are closely integrated with the One of the fundamental requirements to ensure reproducibil- EBRAINS Model Catalog and KG. This permits authors ity of a computational study is access to the original source to link their models to corresponding entries in the model code used by the authors to arrive at the reported results. catalog, which then provides additional information on the 1 3 Neuroinformatics Fig. 2 Widgets allow live paper creators to import resources from established neurosci- ence repositories. The figure illustrates how models can be imported from the EBRAINS Knowledge Graph, ModelDB, OSB or BioModels. Filters are available for shortlisting models based on specified criteria, or alternatively selected based on their identifiers models, including outcomes of any validation tests they enables users to launch these within the EBRAINS Collabora- might have undertaken using the EBRAINS Validation tory by simply clicking on each of them. This allows users to Framework. readily explore and execute the contents of these notebooks, without requiring any further setup, and can help in reproduc- • Jupyter Notebooks ing simulations and analysis from the original study. Jupyter Notebook is a free, open-source interactive web appli- Live papers are highly flexible and can easily accommo- date other types of resources, not covered above. Authors cation that supports several programming languages. Over the past few years, notebooks have become the foremost choice can create listings of any kind of data, thereby enabling access to the scientific community for their perusal and fur - for researchers to demonstrate their computational work. It allows them to aggregate software code, simulated outputs, ther reuse in scientific studies. documentation and other relevant resources into a single document (Perkel 2018). A highlight of Jupyter notebooks is Live Paper Builder Tool their ability to capture output, and thereby act as the equiva- lent of laboratory notebooks for computational researchers. A Initially, the process of creating live papers involved the Jupyter notebook basically creates a snapshot of the actions authors having to download an HTML page template and of the original authors and the outcomes they witnessed, and editing this as required to incorporate all the data resources. in principle should enable any other user to follow the same This was somewhat arduous and required some knowledge of footsteps to arrive at the same results. Well designed Jupy- web development. Such an approach was evidently a barrier ter notebooks can also serve as documentation of a study for to wider adoption of the live paper concept, and so we have the better understanding of reviewers and readers. The Live now developed a live paper builder tool that allows authors Paper platform allows authors to add Jupyter Notebooks, and to construct live papers by simply entering information into 1 3 Neuroinformatics Fig. 3 Based on the specified filters, data from the specified neuroscience repository, here ModelDB, are retrieved and tabulated. Authors are provided with relevant metadata for these entries, and are able to add these items to the live paper a form based interface. Fig. 4 shows a snapshot of the user JSON format, and to save the project to the KG. This allows interface for entering the required information. users to develop the live papers across multiple sessions and/ Where the live paper is associated with a traditional pub- or enhance them over time, by either loading the previously lication or preprint, metadata about the associated publica- downloaded project files, or via selecting from a list of live tion can be automatically extracted from the PDF version paper projects from the KG that the user has access to. Once or entered manually. The tool provides widgets for furnish- the live paper development is completed, users can raise ing information on the various kinds of resources discussed requests, from within the tool, for them to be published on earlier. Additionally, it also provides more advanced users the platform. the ability to add custom functionality by specifying custom HTML or Markdown content. The latter ensures that the Provenance Tracking live paper builder tool does not limit the complexity of live papers that can be developed using it. The EBRAINS KG is a multi-modal metadata store that As seen in the lower toolbar in Fig. 4, the tool allows aggregates information from various fields of brain research authors to preview changes, to download the resultant into a single interconnected network of data. It integrates HTML file as well as the project file, with all data saved in experimental data, models, software and other related 1 3 Neuroinformatics Fig. 4 The browser-based live paper builder tool allows users to fill and is very intuitive. Live papers with any level of complexity can be information in a form-based interface to create live papers. This pro- developed from within this tool cess does not require any programming or web development skills resources into a graph based database, and links these data also allows for connecting datasets to relevant software tools units by their relationships to each other. This enables iden- that can help with analysing and visualising the contained tifying all relevant resources associated with a particular data. research object. For example, for a given electrophysi- The Live Paper platform stores all the information pro- ological recording stored in the KG, it would be possible to vided within the live papers in the KG. While creating the retrieve all metadata associated with that recording, such as live paper, users have the option of selecting existing KG information on species, cell type, data modality, and addi- resources (e.g. models, experimental recordings) or, alter- tionally other linked research objects such as experimen- natively, specifying new resource units not currently avail- tal protocols, software, and models that have employed the able in the KG. We encourage the former approach, and given recording in their development or validation. The KG accordingly guide users to tools/processes for registering 1 3 Neuroinformatics resources on the KG, such as the EBRAINS Model Catalog on the journal’s website, or prior to publication hosted on a for registering new models and the EBRAINS Data Cura- preprint server such as bioRxiv (Sever et al. 2019). The live tion Service. paper itself can be created by any of the original authors on the study, or by a third person, in which case the publication Limiting Visibility: Password‑protection of the live paper will require an approval from one of the original authors. It is understandable that authors might prefer to have their Figure 5 presents the steps involved in the live paper crea- resources kept private prior to publication. At the same tion process in the form of a flowchart. The first step towards time, the resources made available in the live paper are creating a live paper is to apply for an EBRAINS account. Inter- often valuable to potential reviewers for a better assessment ested users with a current affiliation to an academic institution of the manuscript under consideration. To allow authors to can directly create an account at: https:// ebrai ns. eu/ regis ter/. develop and share live papers with reviewers, without risk- Users without such an affiliation may request an account. Once ing reviewer anonymity by requiring reviewers to create an they have an EBRAINS account, they can access the live paper account, live papers offer the possibility of restricting their builder tool at: https://ebr ains. eu/ ser vice/ liv e-papers/ . On open- accessibility via password-protection. Authors can share the ing the tool, users are given an option to either begin creation of live paper URL and the associated password with journal a new live paper, or to continue working on an existing project editors, who can then pass them on to reviewers. This could - by loading a previously downloaded project file or selecting a also help address the data availability requirements posed live paper project saved in the KG. by journal publishers at the time of manuscript submissions. When starting a new live paper project, users are given Password-protected live papers are not listed on the Live the option to upload the PDF file of an associated manu - Paper platform, and can only be accessed via their direct script. The tool then attempts to auto-extract all the nec- URLs and assigned password. To further ensure preservation essary publication related information from the uploaded of reviewer anonymity, the web server logs for the platform file. We employ GROBID (Lopez et al.  2009), a tool for do not store any information allowing the identification of extracting metadata from scholarly publications, for this individual users, and are anyway not accessible to live paper purpose. As another option, users can specify the DOI of authors. the published article, and the tool will retrieve the associ- ated metadata. Note that this second method is typically Issuing DOIs more accurate, but can retrieve limited info using the DOI. Alternatively, authors can manually enter all this information The EBRAINS platform issues Digital Object Identifi- pertaining to the publication. ers (DOIs) for curated research objects that are published Widgets are provided for listing the different types of through its Data and Knowledge Services, including experi- resources discussed earlier. Each widget requests, for each mental datasets and Live Papers. Before publication, live item in the listing, information such as the download URL papers go through a quality control process; described in for the resource, the label to be used in the listing, and the following section. Issuance of a DOI further assists the optionally other resource-type specific information, such as citation of published data and models, thereby incentivising the URL to a Model Catalog entry for model source code authors to publish and share such resources, and ensuring resources. An example of such a widget for listing morphol- that they are duly acknowledged when these resources are ogies is illustrated in Fig.  6. The widgets generally offer reused. multiple input formats, with an eye to support both manual entry of information, as well as to assist copying over com- mon programming constructs, such as lists and dictionaries. Live Paper Life Cycle We have recently incorporated a spreadsheet-based input tool that makes it simpler to enter multiple entries at once Below we outline different phases of the live paper life cycle, via the GUI. The user does not need to carry out any addi- from development to publication and usage. tional tasks to integrate the visualisation tools such as the morphology viewer and neural activity visualiser. Authors For Authors: Development to Publication Phase are also requested to select a licensing policy to apply to the resources that they have listed on the live paper. A live paper can be developed as a stand-alone resource, or Live papers, during their development phase, are not pub- as a supplement to an existing or future publication. In this licly accessible. They have their access restricted to users second case, the requirement is for the availability of an belonging to a group, as defined by membership of a user- associated manuscript. This may be an already published selected workspace (known as a ‘Collab’) in the EBRAINS manuscript, in which case authors could point to the article Collaboratory. The authors, at the time of saving the live 1 3 Neuroinformatics Fig. 5 Flowchart depicting the steps involved in creating a live paper. Live papers can either be initiated from scratch, or have the publication related metadata extracted and auto-populated from the associated publication. Live papers are often devel- oped and updated over multiple sessions, and can therefore be saved to KG at any time, or also downloaded locally paper in the EBRAINS KG, therefore need to specify a Col- should be noted that live papers are not peer-reviewed lab for the live paper being developed; this could be one of from a scientific perspective. For now, the live paper their existing Collabs or they can create a new one. Mem- curation process primarily involves verifying that all bers of the Collab with administrator permissions can add or contained resources are actually accessible and that remove team members, and can therefore control visibility of these are hosted on reliable data storage repositories. the under-development live paper. During the development Resources hosted on authors’ or universities’ own phase, all the data resources are controlled by the authors, websites are copied to the EBRAINS archival data including their storage locations. The live papers can con- repository, to ensure long term accessibility and tinuously be updated in this phase. availability of these resources. The URLs within the live Once completed, the authors can submit the live papers are automatically updated to ref lect these new paper for publication, after which it will undergo a storage locations. Resources hosted on other established curation process for the purposes of quality control. neuroscience data repositories, such as ModelDB, Open Understandably, it is not feasible to verify that a given Source Brain, BioModels, NeuroMorpho.org and the live paper contains every single resource employed in Allen Brain Atlas, are not duplicated, but instead we link an associated publication. This aspect is potentially best directly to the corresponding entries in these repositories. assisted by reviewers during the peer-review process of Fig. 7 illustrates, through a f lowchart, the steps involved that publication, and we discuss this further later. It in publishing a live paper. 1 3 Neuroinformatics Fig. 6 A widget for allowing users to input a collection of neuronal morphologies. The data can be input either by manually filling the fields, or by directly updating the underly- ing JSON content. Each widget additionally allows users to select an icon, specify a title and description, and offers the option of grouping items into categories. Widgets also enable the import of data from well- known neuroscience reposito- ries Fig. 7 Flowchart depicting the workflow for processing a live paper once all the contents have been finalised. Authors can request to make the live paper password-protected, whereby access is restricted, or to have it published publicly 1 3 Neuroinformatics Once the live paper metadata and contents have been repositories, reuse these to arrive at further inferences and verified, we publish the live papers on the platform. This conclusions, without really being aware of the model’s ori- makes it available to everyone. Once published, the process gin, scope or limitations. This is very often attributed to the to make further changes to the live papers by the authors will difficulty in locating or accessing the data underlying model require to again undergo the curation process to verify any development. changes. Hence it is strongly encouraged to request publica- Another common problem is with regards to difficulty tion only once the contents are finalised. It should be noted in reproducing results reported in publications. This can be that password-protected live papers, described earlier, are simply owing to unavailability of the model source code or not considered as “published” and therefore do not undergo lack of simulation/protocol specific details being provided the curation process until submitted for publication. in the publications, that are needed to allow reproducibility. An extreme case of this problem was posed by ReScience For Users: Post‑publication Phase C, a journal that encourages testing the reproducibility/rep- licability of computational methods based solely on the cor- Published live papers are categorized by year and listed on responding published article. In 2020, they launched a “Ten the Live Paper platform. All published live papers can be Years Reproducibility Challenge” where scientists were accessed freely by the scientific community. These can be asked to reproduce their own computational work published freely accessed without an EBRAINS account, although at least 10 years earlier (Perkel 2020). The purpose of this certain integrated tools for re-running analyses or simula- challenge was to highlight the difficulties involved in even tions might only be available for registered users owing to reproducing one’s own work (let alone that of others), the the need for accessing EBRAINS computing resources. For need for reliable storage of all relevant modelling resources, example: certain resources, such as Jupyter notebooks, can and proper documentation. be easily copied over to the user’s own workspace in the It is common practice in current times for journal publishers EBRAINS Collaboratory, where they can effect changes and to demand a statement on data availability from the authors afford further explorations. Analysis and simulations can prior to publication (Hofer et al. 2019; Hrynaszkiewicz 2019). therefore be run on the cloud via the EBRAINS infrastruc- This certainly does encourage authors to make underlying data ture. This feature requires the user to have an EBRAINS resources available to the scientific community. But what is account to avail of computational resources. As mentioned lacking is a structured and systematic way of offering these previously, it is quite simple to request for an EBRAINS resources. Very often authors simply resort to stating something account, and would provide the user access to several other similar to “Data are available from the authors upon request”. tools and services as well. Studies have reported that it is uncommon for published articles Users are free to access and use the resources provided to contain the underlying resources, or offer links to access them in the live papers under the license terms specified by the (Nüst et al. 2018; Stagge et al. 2019). As may be imagined, this authors of the live paper. Any additional queries or requests often results in a situation where users interested in a particular regarding the provided resources should be communicated computational work face a brick wall because authors are unre- to the live paper authors. For any issues regarding usability sponsive, have left academia, or because code or data have or accessibility of resources, users are requested to contact been lost or are otherwise no longer available. https:// ebrai ns. eu/ suppo rt for further assistance. With the concept of Live Papers presented here, we intend to establish a platform by means of which authors can easily aggregate the various data components underly- Discussion ing their computational study into a systematic, structured and distributable format. The live paper builder tool has Live Papers are intended at making neuroscience publica- been developed with the primary focus on making the data tions more valuable to the scientific community by offering a sharing process as simple as possible. The curation process holistic view of the various digital components of a publica- ensures that resources are hosted on reliable data storage tion, such as data analysis code, the data underlying model services, either by transferring the specified resources to the development, or simulation results. The availability and EBRAINS archival data repository, or by linking to other accessibility of underlying code and data will enable review- established neuroscience repositories. This helps tackle ers and other scientists to better evaluate a given model or the issue of long term retrievability of publication related analysis. This would, in turn, permit informed extensions/ resources. Other additional resources, such as Jupyter note- enhancements to the model/analysis by virtue of possessing books, allow for enhanced documentation of simulation pro- knowledge of how they were developed from the outset. For tocols or data analysis pipelines by demonstrating how vari- example, it is not uncommon in the modelling community ous simulations were undertaken. With data being managed to pick up published models from any of the various model via the EBRAINS KG, the issue of data provenance can also 1 3 Neuroinformatics be better tackled, along with tighter integration with other groups—“from trained research software engineers to self- tools and services offered under the EBRAINS ecosystem. taught beginners”. The learning curve involved in adopting One apparent limitation of the live papers is the lack of such applications in the publication workflow can often be standardisation of the content of the live papers. In its cur- a huge deterrent to their uptake. Some of the applications rent form, the authors are free to determine what resources are commercial and therefore require authors to have paid they wish to provide in the manuscript. This is in part to accounts to access all features. Also, some applications encourage wider uptake by not imposing rigid requirements, restricted users to open licenses for the shared content and but also borne out of the need for handling the diversity in some of them didn’t have an online version and required computational studies, where the set of employed resources users to host it themselves, thereby adding to the technical can greatly vary. Currently, each live paper submission is overhead. Moreover, regardless of whether the considered verified to ensure that all input resources are actually acces - applications can be self-hosted or not, they require installa- sible, functional, and hosted on a reliable data storage repos- tion, configuration and maintenance operations in addition to itory. Live papers can be password-protected, so they can be quota management and resource monitoring. Finally, those shared with reviewers of an associated manuscript prior to applications usually require a previous registration to the publication. Reviewers would be best placed to identify and hosting platform for accessing the resources. In a review recommend what missing features should be made available process, this could prevent a double-blind procedure in that in the live paper. We have already had instances where live the reviewers would not be able to visualize the material papers have been used to furnish resources and other details anonymously. In concluding, the authors of the study also demanded during the review phase. strongly urged research authors to publish material resources in well-established, reliable repositories that guarantee long- Comparison with Other Reproducibility Efforts term availability of these resources, in addition to an exe- cutable version using any of the reproducibility supporting (Konkol et  al. 2020) undertook a review of several applications. applications and services created with the purpose of The primary objective underlying the concept of live furthering transparent and reproducible research. These papers is twofold: 1) to offer a human-friendly and content- included applications such as Authorea, Binder, eLife rich platform to the scientific community for accessing Reproducible Document Stack (RDS) - extended recently to resources related to neuroscientific publications; 2) to pro- a web-native format with eLife Executable Research Articles vide authors, aiming at (or required to) share data, models (ERAs), and ReproZip (in combination with ReproServer), and methods adopted in their manuscripts, with a flexible amongst many others. A large number of stakeholders and easy-to-use environment able to reduce the overhead are involved in the scientific process - publishers, editors, of the publishing-and-sharing process —in terms of time, authors, readers, reviewers, and librarians. Each group comes effort, and cost. All of these are well-known obstacles that with their own set of requirements and considerations, and authors need to face in the path of open and reproducible the authors therefore found it infeasible and inappropriate research. This in part justifies the choice of a low entry to provide a ranking for these reviewed applications as each threshold for developing live papers, whereby the authors satisfies user-specific requirements to varying extents. decide what resources are made available, and the curation (Konkol et al. 2020) reported that, though many of these process for publication, for the moment, simply ensures that applications were in active usage at the time of reporting, it these resources are accessible, functional and made available might take greater effort to have these accepted and adopted long-term. into publishers’ infrastructures. Also, journals and publish- A key and novel feature of the Live Papers is its capac- ers often differ in the formats of accepted submissions, and ity to complement existing data storage repositories by this transformation is often non-trivial. The authors therefore working as an aggregator of resources from established suggest that it might be simpler to have reproducible docu- platforms (by linking them together), while offering both ments as a supplementary resource to the actual publication, its own data storage capabilities and dedicated tools and especially for the immediate future, before a transition is services for the exploration and use of the resources successfully made by both researchers and publishers to have made available. For example, a neural data visualiser is manuscript embedded reproducible elements. The concept seamlessly integrated in the Live Papers for reading and of live papers, presented here, provides exactly such a sup- displaying all the neural data type supported by the Neo porting document associated with a published article. library and only requires the authors to specify the url of Most of the applications in their study were found to the data source. The same holds true for 3D visualiza- make use of literate programming to support reproducible tion of neural morphologies and model data and metadata research. (Konkol et al. 2020) importantly point out that the access. In addition to the user-friendliness of the Live range of programming expertise varies widely between user Papers webpage, the insertion of online resources is made 1 3 Neuroinformatics even easier thanks to an ad hoc developed search engine Future Developments that is able to query and fetch content data, based on key- words inserted by the users, from several online scien- We intend to implement more features within the live papers tific repositories (e.g., ModelDB, OSB, EBRAINS KG). based on community feedback and requirements. One such Differently from other applications, the Live Papers web feature that we are currently working towards is the ability to interface is extremely lightweight, in that it consists in an launch NEURON (Hines and Carnevale 1997) based models easy-to-maintain JavaScript-based web frontend; hence, using cloud services, through which NEURON parameters and no specific installation and configuration are required to l fi es can be configured and utilized, from within the live papers. authors, publishers and developers. At the same time, Provisions for enabling such advanced or custom functionalities being part of the EBRAINS Research Infrastructure, the already exist and some published live papers already leverage Live Papers take advantage of the rich ensemble of tools a web-socket based service to remotely run NEURON models and services it offers: 1) Jupyter notebooks can be created using the BlueNaaS application; nonetheless this and configured in the EBRAINS Collaboratory environ- requires web development skills. In future we hope to offer a ment and linked to the Live Papers; 2) the Live Papers user-friendly interface and new widgets for this functionality platform is integrated with the BlueNaas simulator engine and extend it to other simulators as well, once the development (https://ebr ains- cls- inter activ e.git hub.io/ online- use- cases. of corresponding cloud-based services will be mature enough. html#/sing le_cell_ insil ico_ e xperiments ) via a web-socket This would eliminate the need to download models and run communication channel; with a little programming effort them in a suitable simulation environment. We also plan to a NEURON model can be run without any specific instal- extend the data analysis capability of the Live Paper documents. lation and the results shown in the user’s browser; 3) if For example, while the neural data visualiser is transparently needed and duly justified, authors can request dedicated integrated in a live paper by simply linking the data source, no HPC resources available in the EBRAINS framework for functionality is currently offered for the analysis of the displayed running demanding operations/simulations; 4) long-term electrophysiological recordings. We plan to add a dedicated data repositories are offered to the Live Papers creators. widget/panel that allows to extract the most signic fi ant measures Thanks to this tight integration, Live Papers delegate the from the neural data (e.g., mean firing rate, number of spikes, burden of software development and maintenance to exter- inter-spike interval values) and, eventually, further extend nal services keeping the interface easily accessible and this functionality to more specific analysis, depending on the maintainable. community requests. In case a simulation panel is integrated in In addition, Live paper documents are built separate from the live papers, as envisioned above, such a tool would also be the article manuscript, thereby allowing authors to follow instrumental for the analysis of the simulated activity. the traditional approach to manuscript preparation, while The Live Paper platform was initially setup in 2018, pri- being able to work in parallel on the supplementary live marily targeting publications arising from the Human Brain paper document. Once the resources are uploaded online, Project. The platform currently hosts over twenty live papers a live paper of moderate complexity, such as most of those associated with scientific publications. This initial release currently available on the platform, can be developed in enabled us to identify shortcomings and incorporate features a matter of a few hours and does not require any specific that were found to be essential for a better user experience programming skill. Also, in a double-blind review scenario, in terms of both utility and ease of use. Having been suc- the document can be password-protected thus guaranteeing cessfully tested, we now intend to roll this out to the wider an exclusive access without any requirement for user scientific community, and hope to see widespread adoption. registration. Finally, the authors of live papers are free to The concept of live papers presented here is readily appli- specify the licensing policy for the resources that they wish cable to scientific studies more broadly, and need not be to share. restricted to neuroscience, although the current implemen- It should also be noted that the issue of reproducibility is tation is mostly oriented to the neuroscientific field. At its a much greater challenge requiring technical interventions core, the live papers are simply a means for effectively dis- that tackle differences in hardware, operating systems, seminating scientific resources, to help further research in versioning, and so on. As briefly stated above, many of a collaborative environment. Most scientific disciplines can the available applications, along with tools such Sumatra, benefit from such a service, and therefore the concept of live CDE, and NoWorkflow, attempt to address these matters papers holds immense promise and potential. through different approaches. With the concept of Live In summary, Live Papers are intended to be a structured Papers presented here, the issue being addressed is tied more and interactive supplementary document, either to comple- closely to data availability, which can be considered as a ment a journal publication or as stand-alone resource, that prerequisite, or a first step, for any kind of reproducibility allows users to readily access, explore and reuse the various effort. kinds of code and data underlying scientific studies. 1 3 Neuroinformatics Author Contributions S.A. designed and implemented the web plat- Reproducibility: Principles, Problems, Practices, and Prospects form, and wrote the first draft of the manuscript. L.L.B. helped with chapter 9. (pp.205–231). John Wiley & Sons, Ltd. the design and testing of the platform, handling the submission of Bakker, R., & Tiesinga, P. H. (2016). Web-based neuron morphology live papers, and helped revise the manuscript. F.S. and M.M. con- viewer as an aid to develop new standards for neuron morphol- ceptualized the idea of interactive live papers, and helped revise the ogy file formats. In Frontiers in Neuroinformatics Conference manuscript. A.P.D. handled the integration of the platform with the Abstract: Neuroinformatics 2016. EBRAINS platform and helped revise the manuscript. All authors have Dillen, M., Groom, Q., Agosti, D., & Nielsen, L. H. (2019). Zenodo, reviewed and approved the manuscript. an archive and publishing repository: A tale of two herbarium specimen pilot projects. Biodiversity Information Science and Standards Funding Information This project was developed in part or in whole Foster, E. D., & Deardorff, A. (2017). Open science framework (osf). in the Human Brain Project, funded from the European Union’s Hori- Journal of the Medical Library Association: JMLA, 105, 203. zon 2020 Framework Programme for Research and Innovation under Garcia, S., Guarino, D., Jaillet, F., Jennings, T. R., Pröpper, R., Rautenberg, Specific Grant Agreements No. 785907 and No. 945539 (Human Brain P. L., Rodgers, C., Sobolev, A., Wachtler, T., Yger, P., & Davison, A. Project SGA2 and SGA3). FS was supported by funding to the Blue P. (2014). Neo: an object model for handling electrophysiology data Brain Project, a research center of École polytechnique fédérale de in multiple formats. Frontiers in Neuroinformatics, 8, 10. Lausanne (EPFL), from the Swiss government’s ETH Board of the Gleeson, P., Piasini, E., Crook, S., Cannon, R., Steuber, V., Jaeger, Swiss Federal Institutes of Technology. D., Solinas, S., D’Angelo, E., & Silver, R. A. (2012). The Open Source Brain Initiative: enabling collaborative modelling in com- Data Availability The EBRAINS Live Papers platform is entirely open- putational neuroscience. BMC Neuroscience, 13, 1–2. source, with the code being available on GitHub (https:// github. com/ Hahnel, M. (2013). Referencing: The reuse factor. Nature News, 502, appuk uttan- shail esh/ ebrai ns- live- papers). This includes both the live paper viewer and the live paper builder tools. All live paper resources Hines, M. L., & Carnevale, N. T. (1997). The (NEURON) simulation are publicly accessible, via the live paper platform, without requiring environment. Neural Computation, 9, 1179–1209. any registration or authentication. The use of resources hosted in each Hines, M. L., Morse, T., Migliore, M., Carnevale, N. T., & Shepherd, live paper is governed by the license specified by the authors of that G. M. (2004). ModelDB: a database to support computational specific live paper. neuroscience. Journal of Computational Neuroscience, 17, 7–11. Hofer, B., Broman, K. W., Granell, C., Graser, A., Hettne, K., Daniel Nüst, Declarations D., & Teperek, M. (2019). Reproducible publications at (AGILE) con- ferences–proposed guidelines for authors and reviewers. In Accepted Short Papers and Posters from the 22nd AGILE Conference on Geo- Ethical Approval Not applicable information Science, Limassol, Chipre, Editorial, Stichting AGILE. Hrynaszkiewicz, I. (2019). Publishers’ responsibilities in promoting Consent to Participate Not applicable data quality and reproducibility. In Good Research Practice in Non-Clinical Pharmacology and Biomedicine (pp. 319–348). Consent for Publication Not applicable Springer. Jones, A. R., Overly, C. C., & Sunkin, S. M. (2009). The allen brain Conflicts of Interest The authors declare that they have no conflict of atlas: 5 years and beyond. Nature Reviews Neuroscience, 10, interest 821–828. Konkol, M., Nüst, D., & Goulier, L. (2020). Publishing computational Open Access This article is licensed under a Creative Commons Attri- research-a review of infrastructures for reproducible and transpar- bution 4.0 International License, which permits use, sharing, adapta- ent scholarly communication. Research integrity and peer review, tion, distribution and reproduction in any medium or format, as long 5, 1–8. as you give appropriate credit to the original author(s) and the source, Lopez, P. (2009). GROBID: Combining automatic bibliographic data provide a link to the Creative Commons licence, and indicate if changes recognition and term extraction for scholarship publications. In were made. The images or other third party material in this article are International conference on theory and practice of digital librar- included in the article's Creative Commons licence, unless indicated ies (pp. 473–474). organization Springer. otherwise in a credit line to the material. If material is not included in Nüst, D., Granell, C., Hofer, B., Konkol, M., Ostermann, F. O., Sileryte, the article's Creative Commons licence and your intended use is not R., & Cerutti, V. (2018). Reproducible research and (GIScience): permitted by statutory regulation or exceeds the permitted use, you will an evaluation using (AGILE) conference papers. Peer J, 6, e5072. need to obtain permission directly from the copyright holder. To view a Perkel, J. M. (2018). Why Jupyter is data scientists’ computational copy of this licence, visit http://cr eativ ecommons. or g/licen ses/ b y/4.0/ . notebook of choice. Nature, 563, 145–147. Perkel, J. M. (2020). Challenge to scientists: does your ten-year-old code still run? Nature, 584, 656–658. Sever, R., Roeder, T., Hindle, S., Sussman, L., Black, K.-J., Argentine, References J., Manos, W., & Inglis, J. R. (2019). bioRxiv: the preprint server for biology. BioRxiv, (p. 833400). Ascoli, G. A., Donohue, D. E., & Halavi, M. (2007). NeuroMorpho. Stagge, J. H., Rosenberg, D. E., Abdallah, A. M., Akbar, H., Attallah, Org: a central resource for neuronal morphologies. Journal of N. A., & James, R. (2019). Assessing data availability and research Neuroscience, 27, 9247–9251. reproducibility in hydrology and water resources. Scientific Data, Bailey, D. H., Borwein, J. M., & Stodden, V. (2016). Facilitating repro- 6, 1–12. ducibility in scientific computing: Principles and practice. In 1 3 http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Neuroinformatics Springer Journals

EBRAINS Live Papers - Interactive Resource Sheets for Computational Studies in Neuroscience

Loading next page...
 
/lp/springer-journals/ebrains-live-papers-interactive-resource-sheets-for-computational-1Oiw7dH8Eu
Publisher
Springer Journals
Copyright
Copyright © The Author(s) 2022
ISSN
1539-2791
eISSN
1559-0089
DOI
10.1007/s12021-022-09598-z
Publisher site
See Article on Publisher Site

Abstract

We present here an online platform for sharing resources underlying publications in neuroscience. It enables authors to easily upload and distribute digital resources, such as data, code, and notebooks, in a structured and systematic way. Interactivity is a prominent feature of the Live Papers, with features to download, visualise or simulate data, models and results presented in the corresponding publications. The resources are hosted on reliable data storage servers to ensure long term availability and easy accessibility. All data are managed via the EBRAINS Knowledge Graph, thereby helping maintain data provenance, and enabling tight integration with tools and services offered under the EBRAINS ecosystem. Keywords Computational neuroscience · Data retrieval · Data provenance · Documentation · Experimental data Introduction users to replicate at least a single figure from the correspond- ing publication. It is not intended to tackle issues regard- A major shortcoming in computational approaches to neu- ing access to experimental data, and therefore does not, in roscience, such as modelling/simulation and data analysis, general, hold information about data used to tune, validate is the absence of a widely used and well-defined system for or simulate the model. Similarly, OSB is a model repository sharing code and data that would enable researchers to easily that focuses on model formats and simulator-independent access the data resources employed in published studies and model representations, but not on prioritising the data under- understand in detail the provenance of published results and lying the models. NeuroMorpho.org is a curated inventory figures. This impedes community-based, collaborative mod- of digitally reconstructed neurons associated with publica- elling efforts, reduces the utility of published models within tions, but doesn’t link with other data such as electrophysi- the neuroscience community, and hinders the reproducibility ological recordings or models of the reconstructed neurons. of data analyses. More general services such as Figshare (Hahnel 2013), Open Existing tools and services, such as ModelDB (Hines et al. Science Framework (Foster and Deardorff  2017), or (Dillen 2004), Open Source Brain (OSB) (Gleeson et al. 2012), and et al. 2019) have much less structured records, with limited NeuroMorpho.org (Ascoli et al. 2007), make models, code domain-specific metadata. Finally, applications/infrastruc - and/or data available to the scientific community, with each tures specifically addressing reproducibility and transpar - attempting to fulfil a specific need. For example, ModelDB ency in scientific publishing (e.g., eLife RDS, Galaxy, Repro is a repository of published models with code that allows Zip) require possibly demanding ad hoc installations, con- figurations and maintenance operations (Konkol et al. 2020). EBRAINS Live Papers are interactive documents bring- * Andrew P. Davison ing together code, models, and data. They can be stand- andrew.davison@cnrs.fr alone publications, but in general complement published Université Paris-Saclay, CNRS, Institut des Neurosciences scientific articles. Interactivity is a prominent feature of the Paris-Saclay, Saclay 91400, France “Live Papers” with several integrated tools and services that Institute of Biophysics, National Research Council, allow users to download, visualise or simulate data, models Palermo 90143, Italy and results presented in the corresponding publications. By Blue Brain Project, École polytechnique fédérale de bringing together the various resources underlying compu- Lausanne (EPFL), Campus Biotech, Geneva 1202, tational approaches in neuroscience publications (whether Switzerland Vol.:(0123456789) 1 3 Neuroinformatics modelling/simulation or data analysis), they provide a more are (i) long-term, archival-quality data storage, provided complete picture of the original researchers’ workspace. By through an agreement with the Swiss National Supercom- virtue of being developed within the EBRAINS infrastruc- puting Centre; (ii) a state-of-the-art metadata store, the ture, Live Papers take advantage of EBRAINS platforms Knowledge Graph; (iii) a wide range of tools for modelling, and services for data and model integration and are able to simulation, and data analysis at different scales and levels of effectively leverage the Knowledge Graph (KG) database abstraction; (iv) facilitated access to high-performance com- for storing all information. The KG, being a graph-based puting systems; (v) collaborative workspaces (the Collabo- database, interlinks all data units thereby readily offering a ratory) with fine-grained access control for the documents, high degree of data provenance. code and data for a given project. It is this infrastructure, Below we describe the various features and function- fulfilling the requirements outlined above, that makes the alities currently made available through the Live Papers Live Paper platform possible. platform, and also how authors can distribute their own The Human Brain Project will be completed in 2023, but resources by creating and publishing live papers. Our goal the tools and services developed under it will continue to is to demonstrate to the scientific community the utility of exist as part of EBRAINS. It will continue to be funded by building live papers to complement their publications, and the participating countries, with national nodes being set to develop them not post facto, but rather as a tool to support up in multiple European countries, and with further fund- their manuscript submissions by making available informa- ing from the EC until at least 2026, to ensure continued tion that could assist reviewers in making a more informed availability of these services. Live Papers is one of several evaluation (Bailey et al. 2016). Computational studies in services in the EBRAINS ecosystem, and their sustainability neuroscience, being potentially fully deterministic, lack- is the primary objective of EBRAINS AISBL. ing the intrinsic variability of experimental studies, should be easily reproducible. In reality, this is often not the case, Live Paper Resources which to a large extent is due to the black box situation aris- ing from missing or incomplete access to data and docu- The live papers allow for diverse types of resources to be mentation. With the concept of Live Papers presented here, presented, with practically no limitations. For computational we hope to take significant strides towards resolving these modelling studies, we have found that the most common problems. resources being distributed comprise the following: Electrophysiological Recordings Overview Data obtained from experimental recordings often lay the In this section we begin with a brief overview of the foundations for modeling studies. These recordings can be EBRAINS infrastructure, and move on to describing the obtained from a variety of different experimental techniques various features and functionalities that are currently avail- and protocols, such as intracellular recordings via sharp able for live papers. microelectrodes or patch clamp setup, calcium or dye imag- ing studies, and so on. The Live Paper platform provides EBRAINS Infrastructure for listing and access of all kinds of experimental data. A typical characteristic of experimental data is that they are An online platform for sharing digital resources and pub- often recorded as time series. The platform integrates an lishing interactive, scholarly documents places several online visualiser for such data, as seen in Fig.  1, without requirements on the underlying infrastructure. Resources requiring users to download them. It handles all data for- and documents must be persistent—reliably available over a mats supported by the Neo library (Garcia et al. 2014). The long time period. Tools for visualising or re-using resources Live Paper platform currently integrates with the EBRAINS (models, code, data) should be available, reliable, give repro- Knowledge Graph and the Allen Brain Atlas (Jones et al.  ducible results, and be underpinned by adequate computa- 2009) to enable authors to link to previously registered elec- tional resources. Accurate and complete metadata must be trophysiological recordings from these databases. available. EBRAINS is a new digital research infrastructure devel- • Neuronal Morphologies oped under the Human Brain Project and now on the Euro- pean Strategy Forum on Research Infrastructures (ESFRI) Computational studies involving modeling of single roadmap. It aims to bring together an extensive range of neurons or networks of neurons typically employ digitally data, alongside tools to help analyse and utilise these data. reconstructed neuronal morphologies for configuring Among the key tools and services provided by EBRAINS the anatomical structure of model neurons. In the case of 1 3 Neuroinformatics Fig. 1 The electrophysiology resource section embeds an interactive visualiser for each electrophysiology data file. This enables users to visual- ise and to interact with the data live from within the document neuronal circuits and networks, a multitude of such neuronal It is almost always non-trivial to reconstruct a model from morphologies might be employed. As cellular biophysics scratch solely based on the model description provided in is greatly affected by the anatomical architecture, it is publications. Without access to the model source code, all essential to have access to such data to ensure reproducibility inferences and conclusions have to be evaluated at face value of results. The live paper tool allows authors to list all without any scientific appraisal. This has encouraged most the morphologies associated with their study, and allow scientific publishers to ensure that the source code underly - interested users to download these. Authors may also link ing modeling studies be made publicly accessible, or at least to established repositories such as NeuroMorpho.Org available on request. and the Allen Brain Atlas. The platform also integrates a Live papers allow authors to distribute the source code 3D morphology viewer that users can use to explore and for their models. As shown in Figs. 2 and 3, they also ena- examine the morphologies (Bakker and Tiesinga 2016). ble importing and linking to corresponding entries in other neuroscience repositories such as ModelDB, OSB and Bio- Model Source Code Models, in addition to the EBRAINS Knowledge Graph. Additionally, live papers are closely integrated with the One of the fundamental requirements to ensure reproducibil- EBRAINS Model Catalog and KG. This permits authors ity of a computational study is access to the original source to link their models to corresponding entries in the model code used by the authors to arrive at the reported results. catalog, which then provides additional information on the 1 3 Neuroinformatics Fig. 2 Widgets allow live paper creators to import resources from established neurosci- ence repositories. The figure illustrates how models can be imported from the EBRAINS Knowledge Graph, ModelDB, OSB or BioModels. Filters are available for shortlisting models based on specified criteria, or alternatively selected based on their identifiers models, including outcomes of any validation tests they enables users to launch these within the EBRAINS Collabora- might have undertaken using the EBRAINS Validation tory by simply clicking on each of them. This allows users to Framework. readily explore and execute the contents of these notebooks, without requiring any further setup, and can help in reproduc- • Jupyter Notebooks ing simulations and analysis from the original study. Jupyter Notebook is a free, open-source interactive web appli- Live papers are highly flexible and can easily accommo- date other types of resources, not covered above. Authors cation that supports several programming languages. Over the past few years, notebooks have become the foremost choice can create listings of any kind of data, thereby enabling access to the scientific community for their perusal and fur - for researchers to demonstrate their computational work. It allows them to aggregate software code, simulated outputs, ther reuse in scientific studies. documentation and other relevant resources into a single document (Perkel 2018). A highlight of Jupyter notebooks is Live Paper Builder Tool their ability to capture output, and thereby act as the equiva- lent of laboratory notebooks for computational researchers. A Initially, the process of creating live papers involved the Jupyter notebook basically creates a snapshot of the actions authors having to download an HTML page template and of the original authors and the outcomes they witnessed, and editing this as required to incorporate all the data resources. in principle should enable any other user to follow the same This was somewhat arduous and required some knowledge of footsteps to arrive at the same results. Well designed Jupy- web development. Such an approach was evidently a barrier ter notebooks can also serve as documentation of a study for to wider adoption of the live paper concept, and so we have the better understanding of reviewers and readers. The Live now developed a live paper builder tool that allows authors Paper platform allows authors to add Jupyter Notebooks, and to construct live papers by simply entering information into 1 3 Neuroinformatics Fig. 3 Based on the specified filters, data from the specified neuroscience repository, here ModelDB, are retrieved and tabulated. Authors are provided with relevant metadata for these entries, and are able to add these items to the live paper a form based interface. Fig. 4 shows a snapshot of the user JSON format, and to save the project to the KG. This allows interface for entering the required information. users to develop the live papers across multiple sessions and/ Where the live paper is associated with a traditional pub- or enhance them over time, by either loading the previously lication or preprint, metadata about the associated publica- downloaded project files, or via selecting from a list of live tion can be automatically extracted from the PDF version paper projects from the KG that the user has access to. Once or entered manually. The tool provides widgets for furnish- the live paper development is completed, users can raise ing information on the various kinds of resources discussed requests, from within the tool, for them to be published on earlier. Additionally, it also provides more advanced users the platform. the ability to add custom functionality by specifying custom HTML or Markdown content. The latter ensures that the Provenance Tracking live paper builder tool does not limit the complexity of live papers that can be developed using it. The EBRAINS KG is a multi-modal metadata store that As seen in the lower toolbar in Fig. 4, the tool allows aggregates information from various fields of brain research authors to preview changes, to download the resultant into a single interconnected network of data. It integrates HTML file as well as the project file, with all data saved in experimental data, models, software and other related 1 3 Neuroinformatics Fig. 4 The browser-based live paper builder tool allows users to fill and is very intuitive. Live papers with any level of complexity can be information in a form-based interface to create live papers. This pro- developed from within this tool cess does not require any programming or web development skills resources into a graph based database, and links these data also allows for connecting datasets to relevant software tools units by their relationships to each other. This enables iden- that can help with analysing and visualising the contained tifying all relevant resources associated with a particular data. research object. For example, for a given electrophysi- The Live Paper platform stores all the information pro- ological recording stored in the KG, it would be possible to vided within the live papers in the KG. While creating the retrieve all metadata associated with that recording, such as live paper, users have the option of selecting existing KG information on species, cell type, data modality, and addi- resources (e.g. models, experimental recordings) or, alter- tionally other linked research objects such as experimen- natively, specifying new resource units not currently avail- tal protocols, software, and models that have employed the able in the KG. We encourage the former approach, and given recording in their development or validation. The KG accordingly guide users to tools/processes for registering 1 3 Neuroinformatics resources on the KG, such as the EBRAINS Model Catalog on the journal’s website, or prior to publication hosted on a for registering new models and the EBRAINS Data Cura- preprint server such as bioRxiv (Sever et al. 2019). The live tion Service. paper itself can be created by any of the original authors on the study, or by a third person, in which case the publication Limiting Visibility: Password‑protection of the live paper will require an approval from one of the original authors. It is understandable that authors might prefer to have their Figure 5 presents the steps involved in the live paper crea- resources kept private prior to publication. At the same tion process in the form of a flowchart. The first step towards time, the resources made available in the live paper are creating a live paper is to apply for an EBRAINS account. Inter- often valuable to potential reviewers for a better assessment ested users with a current affiliation to an academic institution of the manuscript under consideration. To allow authors to can directly create an account at: https:// ebrai ns. eu/ regis ter/. develop and share live papers with reviewers, without risk- Users without such an affiliation may request an account. Once ing reviewer anonymity by requiring reviewers to create an they have an EBRAINS account, they can access the live paper account, live papers offer the possibility of restricting their builder tool at: https://ebr ains. eu/ ser vice/ liv e-papers/ . On open- accessibility via password-protection. Authors can share the ing the tool, users are given an option to either begin creation of live paper URL and the associated password with journal a new live paper, or to continue working on an existing project editors, who can then pass them on to reviewers. This could - by loading a previously downloaded project file or selecting a also help address the data availability requirements posed live paper project saved in the KG. by journal publishers at the time of manuscript submissions. When starting a new live paper project, users are given Password-protected live papers are not listed on the Live the option to upload the PDF file of an associated manu - Paper platform, and can only be accessed via their direct script. The tool then attempts to auto-extract all the nec- URLs and assigned password. To further ensure preservation essary publication related information from the uploaded of reviewer anonymity, the web server logs for the platform file. We employ GROBID (Lopez et al.  2009), a tool for do not store any information allowing the identification of extracting metadata from scholarly publications, for this individual users, and are anyway not accessible to live paper purpose. As another option, users can specify the DOI of authors. the published article, and the tool will retrieve the associ- ated metadata. Note that this second method is typically Issuing DOIs more accurate, but can retrieve limited info using the DOI. Alternatively, authors can manually enter all this information The EBRAINS platform issues Digital Object Identifi- pertaining to the publication. ers (DOIs) for curated research objects that are published Widgets are provided for listing the different types of through its Data and Knowledge Services, including experi- resources discussed earlier. Each widget requests, for each mental datasets and Live Papers. Before publication, live item in the listing, information such as the download URL papers go through a quality control process; described in for the resource, the label to be used in the listing, and the following section. Issuance of a DOI further assists the optionally other resource-type specific information, such as citation of published data and models, thereby incentivising the URL to a Model Catalog entry for model source code authors to publish and share such resources, and ensuring resources. An example of such a widget for listing morphol- that they are duly acknowledged when these resources are ogies is illustrated in Fig.  6. The widgets generally offer reused. multiple input formats, with an eye to support both manual entry of information, as well as to assist copying over com- mon programming constructs, such as lists and dictionaries. Live Paper Life Cycle We have recently incorporated a spreadsheet-based input tool that makes it simpler to enter multiple entries at once Below we outline different phases of the live paper life cycle, via the GUI. The user does not need to carry out any addi- from development to publication and usage. tional tasks to integrate the visualisation tools such as the morphology viewer and neural activity visualiser. Authors For Authors: Development to Publication Phase are also requested to select a licensing policy to apply to the resources that they have listed on the live paper. A live paper can be developed as a stand-alone resource, or Live papers, during their development phase, are not pub- as a supplement to an existing or future publication. In this licly accessible. They have their access restricted to users second case, the requirement is for the availability of an belonging to a group, as defined by membership of a user- associated manuscript. This may be an already published selected workspace (known as a ‘Collab’) in the EBRAINS manuscript, in which case authors could point to the article Collaboratory. The authors, at the time of saving the live 1 3 Neuroinformatics Fig. 5 Flowchart depicting the steps involved in creating a live paper. Live papers can either be initiated from scratch, or have the publication related metadata extracted and auto-populated from the associated publication. Live papers are often devel- oped and updated over multiple sessions, and can therefore be saved to KG at any time, or also downloaded locally paper in the EBRAINS KG, therefore need to specify a Col- should be noted that live papers are not peer-reviewed lab for the live paper being developed; this could be one of from a scientific perspective. For now, the live paper their existing Collabs or they can create a new one. Mem- curation process primarily involves verifying that all bers of the Collab with administrator permissions can add or contained resources are actually accessible and that remove team members, and can therefore control visibility of these are hosted on reliable data storage repositories. the under-development live paper. During the development Resources hosted on authors’ or universities’ own phase, all the data resources are controlled by the authors, websites are copied to the EBRAINS archival data including their storage locations. The live papers can con- repository, to ensure long term accessibility and tinuously be updated in this phase. availability of these resources. The URLs within the live Once completed, the authors can submit the live papers are automatically updated to ref lect these new paper for publication, after which it will undergo a storage locations. Resources hosted on other established curation process for the purposes of quality control. neuroscience data repositories, such as ModelDB, Open Understandably, it is not feasible to verify that a given Source Brain, BioModels, NeuroMorpho.org and the live paper contains every single resource employed in Allen Brain Atlas, are not duplicated, but instead we link an associated publication. This aspect is potentially best directly to the corresponding entries in these repositories. assisted by reviewers during the peer-review process of Fig. 7 illustrates, through a f lowchart, the steps involved that publication, and we discuss this further later. It in publishing a live paper. 1 3 Neuroinformatics Fig. 6 A widget for allowing users to input a collection of neuronal morphologies. The data can be input either by manually filling the fields, or by directly updating the underly- ing JSON content. Each widget additionally allows users to select an icon, specify a title and description, and offers the option of grouping items into categories. Widgets also enable the import of data from well- known neuroscience reposito- ries Fig. 7 Flowchart depicting the workflow for processing a live paper once all the contents have been finalised. Authors can request to make the live paper password-protected, whereby access is restricted, or to have it published publicly 1 3 Neuroinformatics Once the live paper metadata and contents have been repositories, reuse these to arrive at further inferences and verified, we publish the live papers on the platform. This conclusions, without really being aware of the model’s ori- makes it available to everyone. Once published, the process gin, scope or limitations. This is very often attributed to the to make further changes to the live papers by the authors will difficulty in locating or accessing the data underlying model require to again undergo the curation process to verify any development. changes. Hence it is strongly encouraged to request publica- Another common problem is with regards to difficulty tion only once the contents are finalised. It should be noted in reproducing results reported in publications. This can be that password-protected live papers, described earlier, are simply owing to unavailability of the model source code or not considered as “published” and therefore do not undergo lack of simulation/protocol specific details being provided the curation process until submitted for publication. in the publications, that are needed to allow reproducibility. An extreme case of this problem was posed by ReScience For Users: Post‑publication Phase C, a journal that encourages testing the reproducibility/rep- licability of computational methods based solely on the cor- Published live papers are categorized by year and listed on responding published article. In 2020, they launched a “Ten the Live Paper platform. All published live papers can be Years Reproducibility Challenge” where scientists were accessed freely by the scientific community. These can be asked to reproduce their own computational work published freely accessed without an EBRAINS account, although at least 10 years earlier (Perkel 2020). The purpose of this certain integrated tools for re-running analyses or simula- challenge was to highlight the difficulties involved in even tions might only be available for registered users owing to reproducing one’s own work (let alone that of others), the the need for accessing EBRAINS computing resources. For need for reliable storage of all relevant modelling resources, example: certain resources, such as Jupyter notebooks, can and proper documentation. be easily copied over to the user’s own workspace in the It is common practice in current times for journal publishers EBRAINS Collaboratory, where they can effect changes and to demand a statement on data availability from the authors afford further explorations. Analysis and simulations can prior to publication (Hofer et al. 2019; Hrynaszkiewicz 2019). therefore be run on the cloud via the EBRAINS infrastruc- This certainly does encourage authors to make underlying data ture. This feature requires the user to have an EBRAINS resources available to the scientific community. But what is account to avail of computational resources. As mentioned lacking is a structured and systematic way of offering these previously, it is quite simple to request for an EBRAINS resources. Very often authors simply resort to stating something account, and would provide the user access to several other similar to “Data are available from the authors upon request”. tools and services as well. Studies have reported that it is uncommon for published articles Users are free to access and use the resources provided to contain the underlying resources, or offer links to access them in the live papers under the license terms specified by the (Nüst et al. 2018; Stagge et al. 2019). As may be imagined, this authors of the live paper. Any additional queries or requests often results in a situation where users interested in a particular regarding the provided resources should be communicated computational work face a brick wall because authors are unre- to the live paper authors. For any issues regarding usability sponsive, have left academia, or because code or data have or accessibility of resources, users are requested to contact been lost or are otherwise no longer available. https:// ebrai ns. eu/ suppo rt for further assistance. With the concept of Live Papers presented here, we intend to establish a platform by means of which authors can easily aggregate the various data components underly- Discussion ing their computational study into a systematic, structured and distributable format. The live paper builder tool has Live Papers are intended at making neuroscience publica- been developed with the primary focus on making the data tions more valuable to the scientific community by offering a sharing process as simple as possible. The curation process holistic view of the various digital components of a publica- ensures that resources are hosted on reliable data storage tion, such as data analysis code, the data underlying model services, either by transferring the specified resources to the development, or simulation results. The availability and EBRAINS archival data repository, or by linking to other accessibility of underlying code and data will enable review- established neuroscience repositories. This helps tackle ers and other scientists to better evaluate a given model or the issue of long term retrievability of publication related analysis. This would, in turn, permit informed extensions/ resources. Other additional resources, such as Jupyter note- enhancements to the model/analysis by virtue of possessing books, allow for enhanced documentation of simulation pro- knowledge of how they were developed from the outset. For tocols or data analysis pipelines by demonstrating how vari- example, it is not uncommon in the modelling community ous simulations were undertaken. With data being managed to pick up published models from any of the various model via the EBRAINS KG, the issue of data provenance can also 1 3 Neuroinformatics be better tackled, along with tighter integration with other groups—“from trained research software engineers to self- tools and services offered under the EBRAINS ecosystem. taught beginners”. The learning curve involved in adopting One apparent limitation of the live papers is the lack of such applications in the publication workflow can often be standardisation of the content of the live papers. In its cur- a huge deterrent to their uptake. Some of the applications rent form, the authors are free to determine what resources are commercial and therefore require authors to have paid they wish to provide in the manuscript. This is in part to accounts to access all features. Also, some applications encourage wider uptake by not imposing rigid requirements, restricted users to open licenses for the shared content and but also borne out of the need for handling the diversity in some of them didn’t have an online version and required computational studies, where the set of employed resources users to host it themselves, thereby adding to the technical can greatly vary. Currently, each live paper submission is overhead. Moreover, regardless of whether the considered verified to ensure that all input resources are actually acces - applications can be self-hosted or not, they require installa- sible, functional, and hosted on a reliable data storage repos- tion, configuration and maintenance operations in addition to itory. Live papers can be password-protected, so they can be quota management and resource monitoring. Finally, those shared with reviewers of an associated manuscript prior to applications usually require a previous registration to the publication. Reviewers would be best placed to identify and hosting platform for accessing the resources. In a review recommend what missing features should be made available process, this could prevent a double-blind procedure in that in the live paper. We have already had instances where live the reviewers would not be able to visualize the material papers have been used to furnish resources and other details anonymously. In concluding, the authors of the study also demanded during the review phase. strongly urged research authors to publish material resources in well-established, reliable repositories that guarantee long- Comparison with Other Reproducibility Efforts term availability of these resources, in addition to an exe- cutable version using any of the reproducibility supporting (Konkol et  al. 2020) undertook a review of several applications. applications and services created with the purpose of The primary objective underlying the concept of live furthering transparent and reproducible research. These papers is twofold: 1) to offer a human-friendly and content- included applications such as Authorea, Binder, eLife rich platform to the scientific community for accessing Reproducible Document Stack (RDS) - extended recently to resources related to neuroscientific publications; 2) to pro- a web-native format with eLife Executable Research Articles vide authors, aiming at (or required to) share data, models (ERAs), and ReproZip (in combination with ReproServer), and methods adopted in their manuscripts, with a flexible amongst many others. A large number of stakeholders and easy-to-use environment able to reduce the overhead are involved in the scientific process - publishers, editors, of the publishing-and-sharing process —in terms of time, authors, readers, reviewers, and librarians. Each group comes effort, and cost. All of these are well-known obstacles that with their own set of requirements and considerations, and authors need to face in the path of open and reproducible the authors therefore found it infeasible and inappropriate research. This in part justifies the choice of a low entry to provide a ranking for these reviewed applications as each threshold for developing live papers, whereby the authors satisfies user-specific requirements to varying extents. decide what resources are made available, and the curation (Konkol et al. 2020) reported that, though many of these process for publication, for the moment, simply ensures that applications were in active usage at the time of reporting, it these resources are accessible, functional and made available might take greater effort to have these accepted and adopted long-term. into publishers’ infrastructures. Also, journals and publish- A key and novel feature of the Live Papers is its capac- ers often differ in the formats of accepted submissions, and ity to complement existing data storage repositories by this transformation is often non-trivial. The authors therefore working as an aggregator of resources from established suggest that it might be simpler to have reproducible docu- platforms (by linking them together), while offering both ments as a supplementary resource to the actual publication, its own data storage capabilities and dedicated tools and especially for the immediate future, before a transition is services for the exploration and use of the resources successfully made by both researchers and publishers to have made available. For example, a neural data visualiser is manuscript embedded reproducible elements. The concept seamlessly integrated in the Live Papers for reading and of live papers, presented here, provides exactly such a sup- displaying all the neural data type supported by the Neo porting document associated with a published article. library and only requires the authors to specify the url of Most of the applications in their study were found to the data source. The same holds true for 3D visualiza- make use of literate programming to support reproducible tion of neural morphologies and model data and metadata research. (Konkol et al. 2020) importantly point out that the access. In addition to the user-friendliness of the Live range of programming expertise varies widely between user Papers webpage, the insertion of online resources is made 1 3 Neuroinformatics even easier thanks to an ad hoc developed search engine Future Developments that is able to query and fetch content data, based on key- words inserted by the users, from several online scien- We intend to implement more features within the live papers tific repositories (e.g., ModelDB, OSB, EBRAINS KG). based on community feedback and requirements. One such Differently from other applications, the Live Papers web feature that we are currently working towards is the ability to interface is extremely lightweight, in that it consists in an launch NEURON (Hines and Carnevale 1997) based models easy-to-maintain JavaScript-based web frontend; hence, using cloud services, through which NEURON parameters and no specific installation and configuration are required to l fi es can be configured and utilized, from within the live papers. authors, publishers and developers. At the same time, Provisions for enabling such advanced or custom functionalities being part of the EBRAINS Research Infrastructure, the already exist and some published live papers already leverage Live Papers take advantage of the rich ensemble of tools a web-socket based service to remotely run NEURON models and services it offers: 1) Jupyter notebooks can be created using the BlueNaaS application; nonetheless this and configured in the EBRAINS Collaboratory environ- requires web development skills. In future we hope to offer a ment and linked to the Live Papers; 2) the Live Papers user-friendly interface and new widgets for this functionality platform is integrated with the BlueNaas simulator engine and extend it to other simulators as well, once the development (https://ebr ains- cls- inter activ e.git hub.io/ online- use- cases. of corresponding cloud-based services will be mature enough. html#/sing le_cell_ insil ico_ e xperiments ) via a web-socket This would eliminate the need to download models and run communication channel; with a little programming effort them in a suitable simulation environment. We also plan to a NEURON model can be run without any specific instal- extend the data analysis capability of the Live Paper documents. lation and the results shown in the user’s browser; 3) if For example, while the neural data visualiser is transparently needed and duly justified, authors can request dedicated integrated in a live paper by simply linking the data source, no HPC resources available in the EBRAINS framework for functionality is currently offered for the analysis of the displayed running demanding operations/simulations; 4) long-term electrophysiological recordings. We plan to add a dedicated data repositories are offered to the Live Papers creators. widget/panel that allows to extract the most signic fi ant measures Thanks to this tight integration, Live Papers delegate the from the neural data (e.g., mean firing rate, number of spikes, burden of software development and maintenance to exter- inter-spike interval values) and, eventually, further extend nal services keeping the interface easily accessible and this functionality to more specific analysis, depending on the maintainable. community requests. In case a simulation panel is integrated in In addition, Live paper documents are built separate from the live papers, as envisioned above, such a tool would also be the article manuscript, thereby allowing authors to follow instrumental for the analysis of the simulated activity. the traditional approach to manuscript preparation, while The Live Paper platform was initially setup in 2018, pri- being able to work in parallel on the supplementary live marily targeting publications arising from the Human Brain paper document. Once the resources are uploaded online, Project. The platform currently hosts over twenty live papers a live paper of moderate complexity, such as most of those associated with scientific publications. This initial release currently available on the platform, can be developed in enabled us to identify shortcomings and incorporate features a matter of a few hours and does not require any specific that were found to be essential for a better user experience programming skill. Also, in a double-blind review scenario, in terms of both utility and ease of use. Having been suc- the document can be password-protected thus guaranteeing cessfully tested, we now intend to roll this out to the wider an exclusive access without any requirement for user scientific community, and hope to see widespread adoption. registration. Finally, the authors of live papers are free to The concept of live papers presented here is readily appli- specify the licensing policy for the resources that they wish cable to scientific studies more broadly, and need not be to share. restricted to neuroscience, although the current implemen- It should also be noted that the issue of reproducibility is tation is mostly oriented to the neuroscientific field. At its a much greater challenge requiring technical interventions core, the live papers are simply a means for effectively dis- that tackle differences in hardware, operating systems, seminating scientific resources, to help further research in versioning, and so on. As briefly stated above, many of a collaborative environment. Most scientific disciplines can the available applications, along with tools such Sumatra, benefit from such a service, and therefore the concept of live CDE, and NoWorkflow, attempt to address these matters papers holds immense promise and potential. through different approaches. With the concept of Live In summary, Live Papers are intended to be a structured Papers presented here, the issue being addressed is tied more and interactive supplementary document, either to comple- closely to data availability, which can be considered as a ment a journal publication or as stand-alone resource, that prerequisite, or a first step, for any kind of reproducibility allows users to readily access, explore and reuse the various effort. kinds of code and data underlying scientific studies. 1 3 Neuroinformatics Author Contributions S.A. designed and implemented the web plat- Reproducibility: Principles, Problems, Practices, and Prospects form, and wrote the first draft of the manuscript. L.L.B. helped with chapter 9. (pp.205–231). John Wiley & Sons, Ltd. the design and testing of the platform, handling the submission of Bakker, R., & Tiesinga, P. H. (2016). Web-based neuron morphology live papers, and helped revise the manuscript. F.S. and M.M. con- viewer as an aid to develop new standards for neuron morphol- ceptualized the idea of interactive live papers, and helped revise the ogy file formats. In Frontiers in Neuroinformatics Conference manuscript. A.P.D. handled the integration of the platform with the Abstract: Neuroinformatics 2016. EBRAINS platform and helped revise the manuscript. All authors have Dillen, M., Groom, Q., Agosti, D., & Nielsen, L. H. (2019). Zenodo, reviewed and approved the manuscript. an archive and publishing repository: A tale of two herbarium specimen pilot projects. Biodiversity Information Science and Standards Funding Information This project was developed in part or in whole Foster, E. D., & Deardorff, A. (2017). Open science framework (osf). in the Human Brain Project, funded from the European Union’s Hori- Journal of the Medical Library Association: JMLA, 105, 203. zon 2020 Framework Programme for Research and Innovation under Garcia, S., Guarino, D., Jaillet, F., Jennings, T. R., Pröpper, R., Rautenberg, Specific Grant Agreements No. 785907 and No. 945539 (Human Brain P. L., Rodgers, C., Sobolev, A., Wachtler, T., Yger, P., & Davison, A. Project SGA2 and SGA3). FS was supported by funding to the Blue P. (2014). Neo: an object model for handling electrophysiology data Brain Project, a research center of École polytechnique fédérale de in multiple formats. Frontiers in Neuroinformatics, 8, 10. Lausanne (EPFL), from the Swiss government’s ETH Board of the Gleeson, P., Piasini, E., Crook, S., Cannon, R., Steuber, V., Jaeger, Swiss Federal Institutes of Technology. D., Solinas, S., D’Angelo, E., & Silver, R. A. (2012). The Open Source Brain Initiative: enabling collaborative modelling in com- Data Availability The EBRAINS Live Papers platform is entirely open- putational neuroscience. BMC Neuroscience, 13, 1–2. source, with the code being available on GitHub (https:// github. com/ Hahnel, M. (2013). Referencing: The reuse factor. Nature News, 502, appuk uttan- shail esh/ ebrai ns- live- papers). This includes both the live paper viewer and the live paper builder tools. All live paper resources Hines, M. L., & Carnevale, N. T. (1997). The (NEURON) simulation are publicly accessible, via the live paper platform, without requiring environment. Neural Computation, 9, 1179–1209. any registration or authentication. The use of resources hosted in each Hines, M. L., Morse, T., Migliore, M., Carnevale, N. T., & Shepherd, live paper is governed by the license specified by the authors of that G. M. (2004). ModelDB: a database to support computational specific live paper. neuroscience. Journal of Computational Neuroscience, 17, 7–11. Hofer, B., Broman, K. W., Granell, C., Graser, A., Hettne, K., Daniel Nüst, Declarations D., & Teperek, M. (2019). Reproducible publications at (AGILE) con- ferences–proposed guidelines for authors and reviewers. In Accepted Short Papers and Posters from the 22nd AGILE Conference on Geo- Ethical Approval Not applicable information Science, Limassol, Chipre, Editorial, Stichting AGILE. Hrynaszkiewicz, I. (2019). Publishers’ responsibilities in promoting Consent to Participate Not applicable data quality and reproducibility. In Good Research Practice in Non-Clinical Pharmacology and Biomedicine (pp. 319–348). Consent for Publication Not applicable Springer. Jones, A. R., Overly, C. C., & Sunkin, S. M. (2009). The allen brain Conflicts of Interest The authors declare that they have no conflict of atlas: 5 years and beyond. Nature Reviews Neuroscience, 10, interest 821–828. Konkol, M., Nüst, D., & Goulier, L. (2020). Publishing computational Open Access This article is licensed under a Creative Commons Attri- research-a review of infrastructures for reproducible and transpar- bution 4.0 International License, which permits use, sharing, adapta- ent scholarly communication. Research integrity and peer review, tion, distribution and reproduction in any medium or format, as long 5, 1–8. as you give appropriate credit to the original author(s) and the source, Lopez, P. (2009). GROBID: Combining automatic bibliographic data provide a link to the Creative Commons licence, and indicate if changes recognition and term extraction for scholarship publications. In were made. The images or other third party material in this article are International conference on theory and practice of digital librar- included in the article's Creative Commons licence, unless indicated ies (pp. 473–474). organization Springer. otherwise in a credit line to the material. If material is not included in Nüst, D., Granell, C., Hofer, B., Konkol, M., Ostermann, F. O., Sileryte, the article's Creative Commons licence and your intended use is not R., & Cerutti, V. (2018). Reproducible research and (GIScience): permitted by statutory regulation or exceeds the permitted use, you will an evaluation using (AGILE) conference papers. Peer J, 6, e5072. need to obtain permission directly from the copyright holder. To view a Perkel, J. M. (2018). Why Jupyter is data scientists’ computational copy of this licence, visit http://cr eativ ecommons. or g/licen ses/ b y/4.0/ . notebook of choice. Nature, 563, 145–147. Perkel, J. M. (2020). Challenge to scientists: does your ten-year-old code still run? Nature, 584, 656–658. Sever, R., Roeder, T., Hindle, S., Sussman, L., Black, K.-J., Argentine, References J., Manos, W., & Inglis, J. R. (2019). bioRxiv: the preprint server for biology. BioRxiv, (p. 833400). Ascoli, G. A., Donohue, D. E., & Halavi, M. (2007). NeuroMorpho. Stagge, J. H., Rosenberg, D. E., Abdallah, A. M., Akbar, H., Attallah, Org: a central resource for neuronal morphologies. Journal of N. A., & James, R. (2019). Assessing data availability and research Neuroscience, 27, 9247–9251. reproducibility in hydrology and water resources. Scientific Data, Bailey, D. H., Borwein, J. M., & Stodden, V. (2016). Facilitating repro- 6, 1–12. ducibility in scientific computing: Principles and practice. In 1 3

Journal

NeuroinformaticsSpringer Journals

Published: Aug 20, 2022

Keywords: Computational neuroscience; Data retrieval; Data provenance; Documentation; Experimental data

References