Skip to content

artifact purpose and organize recommended readings #169

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
davisjam wants to merge 1 commit into
base: master
Choose a base branch
from
Open

artifact purpose and organize recommended readings #169

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 27 additions & 17 deletions docs/supplements/OpenScience.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,14 @@ Artifacts related to a study and the paper itself should, in principle, be made

provided that there are no ethical, legal, technical, economical, or practical barriers preventing their disclosure.

Artifacts accompanying software engineering submissions serve several purposes:
- **Reproducibility**: by providing the exact code, data, and scripts needed to rerun experiments
- **Validation**: by allowing reviewers and readers to verify empirical claims
- **Reuse and Extension**: by offering well‐documented components that others can build upon
- **Transparency and Trust**: by open and standardized dissemination of research outputs

By adhering to established artifact review and badging processes (*e.g.*, ACM, IEEE) or checklist requirements (*e.g.*, NeurIPS reproducibility checklist), authors demonstrate that their work meets community standards for **usability**, **documentation**, and **accessibility**, ultimately accelerating the advancement of the field.

## Specific Attributes

### Desirable Attributes
Expand Down Expand Up @@ -45,29 +53,31 @@ Researchers should not complain that a study involves artifacts which— for goo

## Suggested Readings

Noemi Betancort Cabrera, Elke C Bongartz, Nora Dörrenbächer, Jan Goebel, Harald Kaluza, & Pascal Siegers. 2020. White Paper on implementing the FAIR principles for data in the Social, Behavioural, and Economic Sciences (No. 274). RatSWD Working Paper. [https://www.econstor.eu/handle/10419/229719](https://www.econstor.eu/handle/10419/229719)
### Open Science Policy Frameworks

Carlos Diego Nascimento Damasceno. 2022. Guidelines for Quality Management of Research Artifacts in Model-Driven Engineering. _MOdeling LAnguages (blog)_. Retrieved July 17, 2022 from [https://modeling-languages.com/guidelines-for-quality-management-of-research-artifacts-in-model-driven-engineering/#](https://modeling-languages.com/guidelines-for-quality-management-of-research-artifacts-in-model-driven-engineering/#)
- Noemi Betancort Cabrera, Elke C Bongartz, Nora Dörrenbächer, Jan Goebel, Harald Kaluza, & Pascal Siegers. 2020. White Paper on implementing the FAIR principles for data in the Social, Behavioural, and Economic Sciences (No. 274). RatSWD Working Paper. [https://www.econstor.eu/handle/10419/229719](https://www.econstor.eu/handle/10419/229719)
- Carlos Diego Nascimento Damasceno. 2022. Guidelines for Quality Management of Research Artifacts in Model-Driven Engineering. _MOdeling LAnguages (blog)_. Retrieved July 17, 2022 from [https://modeling-languages.com/guidelines-for-quality-management-of-research-artifacts-in-model-driven-engineering/#](https://modeling-languages.com/guidelines-for-quality-management-of-research-artifacts-in-model-driven-engineering/#)
- Daniel Graziotin. 2020. SIGSOFT open science policies. Retrieved July 12, 2020 from [https://github.com/acmsigsoft/open-science-policies/blob/master/sigsoft-open-science-policies.md](https://github.com/acmsigsoft/open-science-policies/blob/master/sigsoft-open-science-policies.md)
- Daniel Méndez, Daniel Graziotin, Stefan Wagner, and Heidi Seibold. 2019. Open science in software engineering. _arXiv_. [https://arxiv.org/abs/1904.06499](https://arxiv.org/abs/1904.06499)

Daniel Graziotin. 2020. SIGSOFT open science policies. Retrieved July 12, 2020 from [https://github.com/acmsigsoft/open-science-policies/blob/master/sigsoft-open-science-policies.md](https://github.com/acmsigsoft/open-science-policies/blob/master/sigsoft-open-science-policies.md)
### Platforms for Artifact Sharing

Daniel Graziotin. 2018. How to disclose data for double-blind review and make it archived open data upon acceptance
Retrieved Feb 24, 2024 from [https://github.com/dgraziotin/disclose-data-dbr-first-then-opendata](https://github.com/dgraziotin/disclose-data-dbr-first-then-opendata)
- GitHub. 2016. Making Your Code Citable. Retrieved July 12, 2020 from [https://guides.github.com/activities/citable-code/](https://guides.github.com/activities/citable-code/). (How to automatically archive a GitHub repository to Zenodo)
- Figshare. How to connect Figshare with your GitHub account. Retrieved July 12, 2020 from [https://knowledge.figshare.com/articles/item/how-to-connect-figshare-with-your-github-account](https://knowledge.figshare.com/articles/item/how-to-connect-figshare-with-your-github-account) (How to automatically archive a GitHub repository to Figshare)

Daniel Méndez, Daniel Graziotin, Stefan Wagner, and Heidi Seibold. 2019. Open science in software engineering. _arXiv_. [https://arxiv.org/abs/1904.06499](https://arxiv.org/abs/1904.06499)
### Artifact Evaluation and Badging Policies

GitHub. 2016. Making Your Code Citable. Retrieved July 12, 2020 from [https://guides.github.com/activities/citable-code/](https://guides.github.com/activities/citable-code/). (How to automatically archive a GitHub repository to Zenodo)
- ACM. 2022. Artifact Review and Badging Version 1.1. August 24, 2020 from [https://www.acm.org/publications/policies/artifact-review-and-badging-current](https://www.acm.org/publications/policies/artifact-review-and-badging-current)
- NeurIPS. 2025. Reproducibility Checklist. Retrieved May 13, 2025 from [https://neurips.cc/public/guides/PaperChecklist](https://neurips.cc/public/guides/PaperChecklist)
- USENIX has no organization-wide policy. Individual USENIX conferences may run artifact committees. See for example the call in USENIX Security 2024. [https://www.usenix.org/conference/usenixsecurity24/call-for-artifacts#evaluation-information](https://www.usenix.org/conference/usenixsecurity24/call-for-artifacts#evaluation-information).
- IEEE has no organization-wide policy. Individual IEEE conferences may run artifact committees. See for example the Open Science Policy for IEEE/ACM ICSE 2025. [https://conf.researchr.org/track/icse-2025/icse-2025-research-track?#Call-for-Papers](https://conf.researchr.org/track/icse-2025/icse-2025-research-track?#Call-for-Papers).

Figshare. How to connect Figshare with your GitHub account. Retrieved July 12, 2020 from [https://knowledge.figshare.com/articles/item/how-to-connect-figshare-with-your-github-account](https://knowledge.figshare.com/articles/item/how-to-connect-figshare-with-your-github-account) (How to automatically archive a GitHub repository to Figshare)

### Best Practices

1. Systems Research Artifacts. 2024. Artifact Packaging Guide. Retrieved May 13, 2025 from [https://sysartifacts.github.io/packaging-guide.html](https://sysartifacts.github.io/packaging-guide.html)

2. Artifact Evaluation Committees (AEC). HOWTO for AEC Submitters. Retrieved May 13, 2025 from [https://docs.google.com/document/d/1pqzPtLVIvwLwJsZwCb2r7yzWMaifudHe1Xvn42T4CcA/edit?tab=t.0](https://docs.google.com/document/d/1pqzPtLVIvwLwJsZwCb2r7yzWMaifudHe1Xvn42T4CcA/edit?tab=t.0)

3. Tianyin Xu. 2021. How Are Award‑winning Systems Research Artifacts Prepared – Part 1. Retrieved May 13, 2025 from [https://www.sigops.org/2021/how-are-award-winning-systems-research-artifacts-prepared-part-1/](https://www.sigops.org/2021/how-are-award-winning-systems-research-artifacts-prepared-part-1/)

4. Tianyin Xu. 2021. How Are Award‑winning Systems Research Artifacts Prepared – Part 2. Retrieved May 13, 2025 from [https://www.sigops.org/2021/how-are-award-winning-systems-research-artifacts-prepared-part-2/](https://www.sigops.org/2021/how-are-award-winning-systems-research-artifacts-prepared-part-2/)

5. Rohan Padhye. 2019. Artifact Evaluation: Tips for Authors. Retrieved May 13, 2025 from [https://blog.padhye.org/Artifact-Evaluation-Tips-for-Authors/](https://blog.padhye.org/Artifact-Evaluation-Tips-for-Authors/)
- Daniel Graziotin. 2018. How to disclose data for double-blind review and make it archived open data upon acceptance. Retrieved Feb 24, 2024 from [https://github.com/dgraziotin/disclose-data-dbr-first-then-opendata](https://github.com/dgraziotin/disclose-data-dbr-first-then-opendata)
- Artifact Evaluation Committees (AEC). HOWTO for AEC Submitters. Retrieved May 13, 2025 from [https://docs.google.com/document/d/1pqzPtLVIvwLwJsZwCb2r7yzWMaifudHe1Xvn42T4CcA/edit?tab=t.0](https://docs.google.com/document/d/1pqzPtLVIvwLwJsZwCb2r7yzWMaifudHe1Xvn42T4CcA/edit?tab=t.0)
- Tianyin Xu. 2021. How Are Award‑winning Systems Research Artifacts Prepared – Part 1. Retrieved May 13, 2025 from [https://www.sigops.org/2021/how-are-award-winning-systems-research-artifacts-prepared-part-1/](https://www.sigops.org/2021/how-are-award-winning-systems-research-artifacts-prepared-part-1/)
- Tianyin Xu. 2021. How Are Award‑winning Systems Research Artifacts Prepared – Part 2. Retrieved May 13, 2025 from [https://www.sigops.org/2021/how-are-award-winning-systems-research-artifacts-prepared-part-2/](https://www.sigops.org/2021/how-are-award-winning-systems-research-artifacts-prepared-part-2/)
- Rohan Padhye. 2019. Artifact Evaluation: Tips for Authors. Retrieved May 13, 2025 from [https://blog.padhye.org/Artifact-Evaluation-Tips-for-Authors/](https://blog.padhye.org/Artifact-Evaluation-Tips-for-Authors/)
- Systems Research Artifacts. 2024. Artifact Packaging Guide. Retrieved May 13, 2025 from [https://sysartifacts.github.io/packaging-guide.html](https://sysartifacts.github.io/packaging-guide.html)