Join our experiment on public discussion of ADAPT'16 paper submissions!

Dear colleagues,

As a part of our ongoing initiative towards open, collaborative and reproducible computer systems' research, we cordially invite you to participate in the public pre-reviewing of ADAPT'16 paper submissions co-located with HiPEAC'16(6th international workshop on adaptive, self-tuning

computing systems).

Each submission is now available at ArXiv and has a separate Reddit discussion thread here:

• https://www.reddit.com/r/adaptworkshop
• https://www.reddit.com/r/reviewpapers/comments/3sogux/join_our_exciting_experiment_on_reddit

Note that several papers have shared artifacts (benchmarks, data sets, models) to help you validate presented techniques and even build upon them!

Please, feel free to comment on these papers, exchange ideas, reproduce results, suggest extensions, note missing references and related work, etc. We hope such public pre-reviewing will speed up dissemination of novel ideas while also letting authors actively engage in discussions and eventually improve their open articles before the final reviewing by the ADAPT Program Committee!

You can find more details about this publication model at http://adapt-workshop.org/motivation2016.html

Looking forward to your participation!

Summer digest on collaborative and reproducible R&D

Upcoming events

• 9.Oct.2015: Don't miss our ADAPT'16 paper submission deadline!
  
  For the first time, we will be trying our open submission model with community-driven reviewing, reddit-based discussions and artifact evaluation:
  
  http://adapt-workshop.org
  https://www.reddit.com/r/reviewpapers

• 19.Oct.2015: Software Credit Workshop (organized by Software Sustainability Instutite in London). This workshop will explore what contribution software can and should make for academic reputational credit.

• 1-4.Nov.2015: Dagstuhl perspective workshop on artifact evaluation (page, participant list). Public report will be available afterwards.

• 20.Nov.2015: Numerical Reproducibility at Exascale (NRE2015). Workshop co-located with SC'15

• 20.Nov.2015: Artifact submission deadline for PPoPP'16 / CGO'16 accepted papers:

• 10-13.May.2016: MMSys 2016 Dataset Track

Past events

• Research Data Alliance 6th plenary meeting in Paris 

• Workshop "Reproducibility of Computation Based Research: Languages, Standards, Methodologies and Platforms"

• Elsevier:The Executable Paper Grand Challenge

Articles

• "Science's Reproducibility Problem", Bob Grant, The Scientist,

• "From Repeatability to Reproducibility and Corroboration", Dror G. Feitelson, SIGOPS Operating Syst. Rev. 49(1), pp. 3-11, Jan 2015

• "International vocabulary of metrology - Basic and general concepts and associated terms (VIM)"

• "Special section on Data replication & reproducibility", Science magazine, 2 December 2011

• "PL conference papers to get a journal?" - discussing ACM proposal to formally recognize conference publications as equal in quality as journal publications

 Presentations

• 25.Nov.2015: "From ad hoc computer engineering to collaborative and reproducible data science", Grigori Fursin, Computer Science Seminar, University of Manchester, UK

• "Reproducible Research in Computational Science: Problems and Solutions For Data and Code Sharing", Victoria Stodden, ICML workshop 2010

Notes

• "Reviewing peer review", Jeannette M. Wing, ACM Blog

• "How Should Peer Review Evolve?", Ed H. Chi, ACM Blog

• Cambridge University news about open and reproducible research

• "Interesting Replicable 'Badge' for journal articles", Daniel S. Katz

National requirements

• UK EPSRC expectations on research data management

• University of Manchester, UK: Policy, Planning

• Cornell University, USA: Best Practices

Events with artifact sharing/evaluation

• ACM TRANSACTIONS ON MATHEMATICAL SOFTWARE (TOMS)

• ACM Multimedia Systems Conference, Dataset Track

Related blogs

• http://reproduciblescience.blogspot.com

• http://dividiti.blog.com

Tools and websites

• OCCAM portal - open curation for computer architecture modeling

• http://www.execandshare.org - creates a companion website associated with a submitted paper to implement the methodology presented in the paper.

• Org mode - keeping notes, maintaining TODO lists, planning projects, and authoring documents with a fast and effective plain-text system

• Reprozip - automatically packing experiments (tool, article)

• Collective Knowledge Framework - decomposing Docker, Virtual Machine and complex experimental setups into reusable components shared via GitHub; exposing them to predictive analytics; creating reproducible, reusable and interactive articles.
  • Example of a reproducible, reusable and interactive article: http://cknowledge.org/repo
  • Examples of GitHub repositories:
    • Benchmarks
    • Data sets
    • Software wrappers
    • Predictive analytics
    • Universal SW/HW autotuning
  • Projects using CK:
    • EU FP7 CARP project
    • UK PAMELA project
    • PLDI'15 CLSmith artifact converted to CK
    • CK-based reproducible report
  • Example of obtaining, compiling and running any shared benchmark
    on a Linux machine with GCC:
    
    $ (sudo) pip install ck
    $ ck pull repo:ctuning-programs
    $ ck compile program:cbench-automotive-susan --speed
    $ ck run program:cbench-automotive-susan

You can find all past aggregated notes at http://cTuning.org/reproducibility-wiki

Collective Knowledge is now available via PyPi

Dear colleagues,

 

Just a note that we finally added Collective Knowledge framework to PyPi archive. It should now be possible to install CK simply via:

 

$ pip install ck

 

The idea is to really minimize set up time for researchers before prototyping their ideas and sharing experimental results. For example, on Linux with GCC, it should take just 3 steps to compile and run a given shared benchmark:

 

$ ck pull repo:ctuning-benchmarks

$ ck compile program:cbench-automotive-susan --speed

$ ck run program:cbench-automotive-susan

 

Similarly, it is possible to run various shared scenarios (including compilation on Windows, cross-compilation forAndroid, multi-objective autotuning, predictive analytics, compiler bug detection) as described here:

• http://github.com/ctuning/ck/wiki
• http://cknowledge.org/repo

Feedback and suggestions are very welcome! Also, do not hesitate to open enhancement or bug requests here:

https://github.com/ctuning/ck/issues

 

Have fun,

dividiti team

Our vision – efficient and reliable computing everywhere

The complexity of engineering computer systems has been growing exponentially (along with the other things exponential growth is actually good for e.g. transistor count). "Quality, cost to deliver, time to deliver – pick any two", an old adage goes. But the rising complexity means that even the ability to pick any two is becoming elusive. Both hardware and software take longer and cost more to produce, only to achieve quality that users will view as merely acceptable.

We thus believe the time is ripe for accelerating computer systems' R&D – from hardware design to programming tools to software development. We aim for liberating engineers from complex, time-consuming and error-prone tasks, therefore allowing them to focus on what they enjoy most – creative problem solving and innovation.

We invite you to join us on our journey!