The Project

Wiki New Zealand

Wiki New Zealand

What do you need to connect people with data?

• Access to data

• Domain knowledge

• Data Science skills

• Statistical Graphics skills

• Graphical Design skills

Few individuals possess all of these

Wiki New Zealand

Wiki New Zealand

Some problems:

• Access to data

• Domain knowledge

• Data Science skills

• Statistical Graphics skills

• Graphical Design skills

Some solutions:

• Open Data, Open Government, Open Access

• Education and experience

• Software that does everything for you

The openapi Project

Some problems:

• Domain knowledge

• Data Science skills

• Statistical Graphics skills

• Graphical Design skills

Some solutions:

• Education and experience

• Software that does everything for you

• Software that lets everyone do a small piece

• Allows more ways to contribute

• Allows small contributions

• Allows contributions to be combined

openapi

openapi is NOT visual programming

openapi is NOT visual programming

openapi is NOT visual programming

openapi is Data and Scripts

openapi is Modules

openapi is Pipelines

openapi is a Glue System

openapi is ...

• Data and script "sources"
• Modules that wrap data and scripts
• Pipelines that connect modules
• A glue system that runs pipelines

An openapi example

Andrew Balemi wanted to add an annotation to the Wiki New Zealand plot of NZ birth rate to show the end of World War II (the onset of the baby boomers)

An openapi example

An openapi example

An openapi example

An openapi example

Another openapi example

Scripts can be in any programming language

Another openapi example

Another openapi example

Another openapi example

Another openapi example

Yet another openapi example

Yet another openapi example

What if all of the New Zealand Youth (18-24) who did NOT vote in 2011 all voted for the Internet Party in 2014?

Yet another openapi example

What do I need?

• Access to data

• Domain knowledge

• Data Science skills

• Statistical Graphics skills

• Graphical Design skills

Yet another openapi example

The accessibility of the data is problematic

Yet another openapi example

The accessibility of the data is problematic

Yet another openapi example

What if there was a nice web site that already made pictures of these data?

Yet another openapi example

What if there was a nice web site that already made pictures of these data?

Yet another openapi example

What if those pictures were part of a modular and reusable framework?

Yet another openapi example

Yet another openapi example

Yet another openapi example

What if you could search for an existing script?
Or request a script?
Or request a module wrapper for an existing script?
Or write your own wrapper on an existing script?

Yet another openapi example

The purple wedge shows what proportion of the overall vote the Internet Party would get.

Yet another openapi example

Yet another openapi example

Where did I get 5.51 from?
My statement is informed
We can share and remix the data and the code
Our discussion can be informed

Summary

• We want to connect people with data

  • more informed individuals

  • more informed discussion

• We propose a framework that enables small contributions

  • more ways to contribute

  • more contributors

  • more contributions

• We have made a start

  • modules

  • pipelines

  • glue systems

References and links

• Two (partly compatible) experimental frameworks are under development (both on GitHub): 'oaglue' and 'conduit'

• These are R packages that have functions to read, write, and run modules and pipelines.

• These slides and the resources used to create them, including example data, scripts, modules, and pipelines, are available at: https://www.stat.auckland.ac.nz/~paul/Talks/OpenAPI2014/.

Acknowledgements

• Bradley Drayton helped with some early exploratory work supported by a Faculty of Science Research Development Grant.

• Ashley Noel Hinton was supported by a Faculty of Science Research Development Grant.

• Wiki New Zealand

• The birth rate data, 2011 election data, and population estimates are all from Statistics NZ.

• The 2011 election results pie chart came from the Rock Enrol web site.

openapi wants to be ...

• As simple as possible

  • no looping contructs or conditionals

• As few concepts as possible

  • modules, pipelines, glue system

• As independent as possible

  • script author, module author, pipeline author can all be different people

• As open as possible

  • open source, open data, open access

• As portable as possible

  • cross platform, language agnostic

openapi could be (with funding) ...

• Multiple glue systems

• Repositories of modules and pipelines

• Request sites for modules and pipelines

• Search, ratings, and recommendations for modules and pipelines

• Tools for creating modules and pipelines
  (including GUIs)

• Automated pipeline creation
  (from simple script markup)

openapi challenges

• Debugging modules and pipelines

• Documentation of modules and pipelines

• Reliability of modules and pipelines

  • Availability of required packages/libraries

• Speed of execution, strong typing, security

• Bundling modules and pipelines (and resources)

• Relative/absolute and remote/local resources

• Creating general-purpose modules

• Capturing "non-functional" scripts as modules

• Modules for "annotating" data

• Why should people contribute?