Summer digest on collaborative and reproducible R&D

Upcoming events

• 9.Oct.2015: Don't miss our ADAPT'16 paper submission deadline!
  
  For the first time, we will be trying our open submission model with community-driven reviewing, reddit-based discussions and artifact evaluation:
  
  http://adapt-workshop.org
  https://www.reddit.com/r/reviewpapers

• 19.Oct.2015: Software Credit Workshop (organized by Software Sustainability Instutite in London). This workshop will explore what contribution software can and should make for academic reputational credit.

• 1-4.Nov.2015: Dagstuhl perspective workshop on artifact evaluation (page, participant list). Public report will be available afterwards.

• 20.Nov.2015: Numerical Reproducibility at Exascale (NRE2015). Workshop co-located with SC'15

• 20.Nov.2015: Artifact submission deadline for PPoPP'16 / CGO'16 accepted papers:

• 10-13.May.2016: MMSys 2016 Dataset Track

Past events

• Research Data Alliance 6th plenary meeting in Paris 

• Workshop "Reproducibility of Computation Based Research: Languages, Standards, Methodologies and Platforms"

• Elsevier:The Executable Paper Grand Challenge

Articles

• "Science's Reproducibility Problem", Bob Grant, The Scientist,

• "From Repeatability to Reproducibility and Corroboration", Dror G. Feitelson, SIGOPS Operating Syst. Rev. 49(1), pp. 3-11, Jan 2015

• "International vocabulary of metrology - Basic and general concepts and associated terms (VIM)"

• "Special section on Data replication & reproducibility", Science magazine, 2 December 2011

• "PL conference papers to get a journal?" - discussing ACM proposal to formally recognize conference publications as equal in quality as journal publications

 Presentations

• 25.Nov.2015: "From ad hoc computer engineering to collaborative and reproducible data science", Grigori Fursin, Computer Science Seminar, University of Manchester, UK

• "Reproducible Research in Computational Science: Problems and Solutions For Data and Code Sharing", Victoria Stodden, ICML workshop 2010

Notes

• "Reviewing peer review", Jeannette M. Wing, ACM Blog

• "How Should Peer Review Evolve?", Ed H. Chi, ACM Blog

• Cambridge University news about open and reproducible research

• "Interesting Replicable 'Badge' for journal articles", Daniel S. Katz

National requirements

• UK EPSRC expectations on research data management

• University of Manchester, UK: Policy, Planning

• Cornell University, USA: Best Practices

Events with artifact sharing/evaluation

• ACM TRANSACTIONS ON MATHEMATICAL SOFTWARE (TOMS)

• ACM Multimedia Systems Conference, Dataset Track

Related blogs

• http://reproduciblescience.blogspot.com

• http://dividiti.blog.com

Tools and websites

• OCCAM portal - open curation for computer architecture modeling

• http://www.execandshare.org - creates a companion website associated with a submitted paper to implement the methodology presented in the paper.

• Org mode - keeping notes, maintaining TODO lists, planning projects, and authoring documents with a fast and effective plain-text system

• Reprozip - automatically packing experiments (tool, article)

• Collective Knowledge Framework - decomposing Docker, Virtual Machine and complex experimental setups into reusable components shared via GitHub; exposing them to predictive analytics; creating reproducible, reusable and interactive articles.
  • Example of a reproducible, reusable and interactive article: http://cknowledge.org/repo
  • Examples of GitHub repositories:
    • Benchmarks
    • Data sets
    • Software wrappers
    • Predictive analytics
    • Universal SW/HW autotuning
  • Projects using CK:
    • EU FP7 CARP project
    • UK PAMELA project
    • PLDI'15 CLSmith artifact converted to CK
    • CK-based reproducible report
  • Example of obtaining, compiling and running any shared benchmark
    on a Linux machine with GCC:
    
    $ (sudo) pip install ck
    $ ck pull repo:ctuning-programs
    $ ck compile program:cbench-automotive-susan --speed
    $ ck run program:cbench-automotive-susan

You can find all past aggregated notes at http://cTuning.org/reproducibility-wiki