Summer digest on collaborative and reproducible R&D

Upcoming events

• 9.Oct.2015: Don't miss our ADAPT'16 paper submission deadline!
  
  For the first time, we will be trying our open submission model with community-driven reviewing, reddit-based discussions and artifact evaluation:
  
  http://adapt-workshop.org
  https://www.reddit.com/r/reviewpapers

• 19.Oct.2015: Software Credit Workshop (organized by Software Sustainability Instutite in London). This workshop will explore what contribution software can and should make for academic reputational credit.

• 1-4.Nov.2015: Dagstuhl perspective workshop on artifact evaluation (page, participant list). Public report will be available afterwards.

• 20.Nov.2015: Numerical Reproducibility at Exascale (NRE2015). Workshop co-located with SC'15

• 20.Nov.2015: Artifact submission deadline for PPoPP'16 / CGO'16 accepted papers:

• 10-13.May.2016: MMSys 2016 Dataset Track

Past events

• Research Data Alliance 6th plenary meeting in Paris 

• Workshop "Reproducibility of Computation Based Research: Languages, Standards, Methodologies and Platforms"

• Elsevier:The Executable Paper Grand Challenge

Articles

• "Science's Reproducibility Problem", Bob Grant, The Scientist,

• "From Repeatability to Reproducibility and Corroboration", Dror G. Feitelson, SIGOPS Operating Syst. Rev. 49(1), pp. 3-11, Jan 2015

• "International vocabulary of metrology - Basic and general concepts and associated terms (VIM)"

• "Special section on Data replication & reproducibility", Science magazine, 2 December 2011

• "PL conference papers to get a journal?" - discussing ACM proposal to formally recognize conference publications as equal in quality as journal publications

 Presentations

• 25.Nov.2015: "From ad hoc computer engineering to collaborative and reproducible data science", Grigori Fursin, Computer Science Seminar, University of Manchester, UK

• "Reproducible Research in Computational Science: Problems and Solutions For Data and Code Sharing", Victoria Stodden, ICML workshop 2010

Notes

• "Reviewing peer review", Jeannette M. Wing, ACM Blog

• "How Should Peer Review Evolve?", Ed H. Chi, ACM Blog

• Cambridge University news about open and reproducible research

• "Interesting Replicable 'Badge' for journal articles", Daniel S. Katz

National requirements

• UK EPSRC expectations on research data management

• University of Manchester, UK: Policy, Planning

• Cornell University, USA: Best Practices

Events with artifact sharing/evaluation

• ACM TRANSACTIONS ON MATHEMATICAL SOFTWARE (TOMS)

• ACM Multimedia Systems Conference, Dataset Track

Related blogs

• http://reproduciblescience.blogspot.com

• http://dividiti.blog.com

Tools and websites

• OCCAM portal - open curation for computer architecture modeling

• http://www.execandshare.org - creates a companion website associated with a submitted paper to implement the methodology presented in the paper.

• Org mode - keeping notes, maintaining TODO lists, planning projects, and authoring documents with a fast and effective plain-text system

• Reprozip - automatically packing experiments (tool, article)

• Collective Knowledge Framework - decomposing Docker, Virtual Machine and complex experimental setups into reusable components shared via GitHub; exposing them to predictive analytics; creating reproducible, reusable and interactive articles.
  • Example of a reproducible, reusable and interactive article: http://cknowledge.org/repo
  • Examples of GitHub repositories:
    • Benchmarks
    • Data sets
    • Software wrappers
    • Predictive analytics
    • Universal SW/HW autotuning
  • Projects using CK:
    • EU FP7 CARP project
    • UK PAMELA project
    • PLDI'15 CLSmith artifact converted to CK
    • CK-based reproducible report
  • Example of obtaining, compiling and running any shared benchmark
    on a Linux machine with GCC:
    
    $ (sudo) pip install ck
    $ ck pull repo:ctuning-programs
    $ ck compile program:cbench-automotive-susan --speed
    $ ck run program:cbench-automotive-susan

You can find all past aggregated notes at http://cTuning.org/reproducibility-wiki

Collective Knowledge is now available via PyPi

Dear colleagues,

 

Just a note that we finally added Collective Knowledge framework to PyPi archive. It should now be possible to install CK simply via:

 

$ pip install ck

 

The idea is to really minimize set up time for researchers before prototyping their ideas and sharing experimental results. For example, on Linux with GCC, it should take just 3 steps to compile and run a given shared benchmark:

 

$ ck pull repo:ctuning-benchmarks

$ ck compile program:cbench-automotive-susan --speed

$ ck run program:cbench-automotive-susan

 

Similarly, it is possible to run various shared scenarios (including compilation on Windows, cross-compilation forAndroid, multi-objective autotuning, predictive analytics, compiler bug detection) as described here:

• http://github.com/ctuning/ck/wiki
• http://cknowledge.org/repo

Feedback and suggestions are very welcome! Also, do not hesitate to open enhancement or bug requests here:

https://github.com/ctuning/ck/issues

 

Have fun,

dividiti team