Jupyter Blog - reproducibility

Simplifying and speeding up Binder builds with BuildKit

2025-03-03T14:05:00+00:00

The Binder Project allows users to build reproducible, sharable environments for interactive computing. To accomplish this, Binder uses a tool called repo2docker to generate an executable Docker image using the Reproducible Execution Environment Specification.

The first time a repository is launched on Binder, repo2docker must build the reproducible environment for it. This process can take a long time because of all the dependencies that need to be installed and turned into the image. As a result, Binder launches can feel slow and clunky, which is a poor UX for workflows that are designed around quick interactive sessions.

repo2docker was built several years ago, and followed patterns that were commonplace at the time. However, in the past few years the Docker community has made significant advances in optimizing the image building process. One-such improvement is the creation of BuildKit, a replacement for Docker’s historical build system that is much more sophisticated. However, repo2docker hasn’t leveraged these improvements because it was still using the original Docker Build system.

So, we’ve decided to spend a few cycles modernizing repo2docker’s image building logic by using the more modern BuildKit API (via ‘docker buildx build’). This allows for optimizations like build parallelization, better build caching, and supporting some Dockerfile features that Binder didn’t support earlier (particularly, COPY --chown). It also lays a foundation for significantly simplifying the repo2docker build infrastructure and leveraging more of BuildKit’s parallelization functionality. For example, we’d like to leverage BuildKit’s Kubernetes driver which distributes builds much more efficiently and in parallel.

Authors of Binder repositories won’t need to take any action¹, and they’ll simply notice that mybinder.org (and any other community-run BinderHub instance) will be a bit snappier at building images.

If you’d like to learn more about the changes that enabled this, check out this mybinder.org pull request which has links to the repo2docker pull requests that added this functionality. We’re excited keep improving Binder, and are hopeful that this makes the experience of using mybinder.org and community Binders a little bit better.

Authors and acknowledgements

Yuvi Panda is a co-founder and the Technical Lead at 2i2c. He is passionate about building participatory open infrastructure for scientific & educational use cases. He is a Project Jupyter team member primarily focused on infrastructure related projects (JupyterHub, Binder, etc). He also wrote most of the code for this docker buildx transition, and shepherded it through to deployment on mybinder.org as well. He is ex-Wikimedia and ex-GNOME. Let’s eliminate accidental complexities wherever we find them.

Chris Holdgraf is a co-founder and the Executive Director of 2i2c. He is on the Executive Council of Project Jupyter, and co-leads the JupyterHub and Binder team as well as the Jupyter Book team). He was previously a post-doctoral researcher in the Department of Statistics at UC Berkeley, and a Community Architect with the Division of Data Science at Berkeley. He’s interested in using open infrastructure to support interactive computing workflows in research and education.

Many thanks to @minrk, @manics, and @consideRatio for their help reviewing and shaping this work.

Footnotes

¹ Unless they were relying on undocumented implementation details of the old builder — in particular, the presence of a /.dockerenv file to detect if you are running in repo2docker scikit-learn/scikit-learn#30835 has an example.

Updating defaults on mybinder.org

2023-03-09T08:31:00+00:00

This week, we are deploying some updates to repo2docker, the image-building tool that underpins mybinder.org. This may affect your repos. Please get in touch if you have problems! Hopefully this will make things better, more stable, and less surprising overall, but change always breaks something, however hard we work to keep things working.

To highlight the changes:

Python 3.10 is now the default Python version
Python 3.11 is now supported, and new versions of Python should be supported more quickly
If a repo requests an old version of Python (≤ 3.6), it will be installed in a separate environment, as we’ve been doing with Python 2.7 for a long time.
The server implementation is switched from jupyter-notebook to jupyter-server

More detail on each change, the reasons behind them, and their consequences below.

Python 3.11 and separate envs for legacy Python support

When building an image, repo2docker installs packages in one or two environments — the kernel environment where your code runs, and the server environment, which runs the Jupyter server. If you’re using a reasonably supported version of Python, these are actually the same environment. This is the vast majority of repositories on mybinder.org. In order to function properly, we have certain requirements of what goes in the ‘server’ environment. Eventually, those requirements start making it difficult to setup the server environment with an outdated version of Python, as packages march ahead with their minimum required Python (The widely adopted NEP 29 suggested dropping Python 3.7 support in December, 2021). Our solution for this has been to separate the kernel environment from the server environment, so they don’t create a conflict. Until recently, only Python 2 was handled this way (most packages dropped Python 2 support years ago). As part of adding support for Python 3.11, we have expanded this behavior to include Python ≤3.6, so if you are using a ‘legacy Python’, currently defined to be Python ≤3.6, your requested packages will be installed in the kernel environment instead of the server environment. This should only have a noticeable effect if you are installing something with server extension components , which includes anything modifying the JupyterLab or notebook interfaces— those will not be available in the server environment. To address this, you will have to upgrade Python or handle the server extension installation in postBuild . PR adding Python 3.11 and adding Python ≤3.6 to legacy Python versions.

Default Python is now 3.10

The default Python for repo2docker and thereby mybinder.org is now Python 3.10, upgraded from 3.7. 3.10 will be used when a repository does not specify which version of Python to use. Following Python’s own annual release cadence, we plan to update this more regularly in the future, lagging behind Python itself by a version or two. This has no effect on your repos if they specify a Python version, which is why we encourage you to specify your Python version! If you don’t, then your Python version will change over time, probably about once a year. PR switching the default Python.

Now launching Jupyter Server

The default UI for mybinder.org has been JupyterLab for some time, with the classic notebook still available if you specify a url of /tree/or /notebooks/mynb.ipynb. When we rolled out that change, we kept the default server as the jupyter-notebook server, for improved compatibility with existing server extensions. As the newer Jupyter Server has taken over, the situation is becoming reversed — using the older notebook server is now the cause of compatibility issues with server extensions, so we are switching the server to launch jupyter-server instead of jupyter-notebook. The classic UI is still available at the same URLs, thanks to nbclassic. This should have no visible effects, as jupyter-server and nbclassic make a great deal of effort to keep classic notebook extensions working, but some extensions may need an update for jupyter-server compatibility if they haven’t been updated already. Because of where the server is specified, there is no supported way for images to express a preference for the deprecated notebook server (PR switching the default server command.

A 2019 retrospective from the Binder Project

2020-01-15T17:48:00+00:00

2019 was a busy year for the Binder and JupyterHub projects — each saw growth in both their community and technology. Now that the year has wrapped up, it is a good time to reflect on some of the highlights from the year.

Overall, 2019 was about improving the robustness, stability, and team dynamics around the JupyterHub and Binder projects, as well as connecting these projects with other tools and services in the open source community. Here are a few things that we are most-excited about.

More people are using mybinder.org

The Binder Federation is a collection of BinderHubs accessible from mybinder.org. This deployment is run as a public service and a demonstration of BinderHub, the underlying technology of the Binder project. This deployment is run on a volunteer basis by Binder community members, and is supported through grants and donations in infrastructure from project stakeholders. In 2019, the user base of mybinder.org grew from around 70,000 users per week to around 100,000 users (a growth of nearly 40%). mybinder.org is being used for teaching classes, sharing reproducible analyses, creating interactive documentation and narratives, and much more. We’re astonished at the rapid growth of Binder-ready repositories, and we’re excited to see what the community creates next.

Weekly user sessions at mybinder.org. Here you can see a typical pattern of activity over the course of a year. There are dips in activity over the summer and winter months, reflecting reduced activity from academic institutions.

JupyterHub reaches 1.0

JupyterHub, the underlying technology that provides interactive computing sessions to multiple users, is now at 1.0 status. The JupyterHub Python application was written several years ago, and reaching 1.0 reflects the work of dozens of open source contributors over time. JupyterHub is now a robust and stable application, having been used at smaller scales (think 5–10 people running on a single VM) as well as much larger scales (think 5,000 students running Jupyter sessions for a class).

The JupyterHub logo

BinderHub is out of beta

BinderHub, the kubernetes-based technology that powers mybinder.org, also came out of Beta this year. This reflects the fact that BinderHub is a battle-hardened application that can provide a stable service over time. mybinder.org runs nearly 100,000 sessions a week, and requires minimal maintenance time from the Binder’s projects team of volunteer operators.

The BinderHub federation is launched

The Binder Project envisions a world in which technology can be used in vendor-agnostic and decentralized ways. BinderHub runs on Kubernetes, which can be deployed on a variety of cloud and local infrastructure. While the Binder team runs one BinderHub deployment at mybinder.org, our goal has always been to see other organizations running their own BinderHubs. This year, we went one step beyond this by launching the BinderHub Federation. This is a collection of research and technology organizations that combine their expertise and computational resources to power mybinder.org. When users visit mybinder.org, they are now directed to one of several BinderHub instances. This makes mybinder.org more robust, and grows the number of organizations that utilize the project’s technology for their communities. A BIG THANKS goes out to Google, OVH, GESIS, and the Turing Institute for supporting the Binder Federation.

The (rough) location of each BinderHub deployment in the mybinder.org federation

Binder connects with open science services

Another goal of Binder is to be a part of the solution to more transparent, sharable, reproducible computational work. This means plugging in to other ecosystems and projects in order to leverage the broader open science community. This year we saw a number of new connections with other services. BinderHub now supports links that point directly to Zenodo and Dataverse repositores, and we are working on a few other integrations in the coming months. This means that projects utilizing these resources will be able to share reproducible and interactive links to their work out-of-the-box.

Binder now works with Zenodo repositories!

The Binder community grows

The Binder project’s most important asset is its people — this is a collection of volunteers spread across the world and from a variety of organizations. Binder community members do a variety of things — from working on technology, to teaching others how to make their work more reproducible, to participating in community discussions, to maintaining and debugging Binder tech. There is also a “core team” of Binder members that dedicates a significant part of their time to supporting the project. In 2019, we saw several new members join the core team, as well as a general growth in the Binder community. Welcome to all of our new team members! You can find a list of our current team members here.

We welcome our first Contributor in Residence

One challenge with running large, open projects is that resources tend to be scarce. The Binder Project has no formal project funding, and must find ways to both grow its technology as well as run mybinder.org on resources that are donated from its community. One thing that often suffers as a result is the maintenance and general improvement of our open source technology. This work is often under-appreciated, difficult, and unlikely to happen with purely volunteer labor.

For this reason, the Binder project decided to apply for the CZI Essential Open Source grant series. We proposed the creation of the “Binder Contributor in Residence” position — an annual contractor position that pays a member of the Binder community to do many of the daily things that are crucial for the project’s growth. We are excited to have Georgiana Dolocan as our first contributor in residence, and look forward to where this project will go in 2020.

Many thanks to CZI for their support of the Binder and JupyterHub projects in 2020!

There are more BinderHub deployments

The Binder Project aims to create technology that is deployable anywhere so that other organizations can support their communities with Binder infrastructure. In 2019 we ran a number of training sessions for how other groups can run their own BinderHub. In particular, the Turing Institute ran several workshops that had attendees up-and-running with their own functioning BinderHubs.

Thanks to our community

As you can see, 2019 was a busy and exciting year for the Binder community. As a final note, we want to say thanks to all of you who have supported Binder in one form or another over the years. Binder is a project run by the community, for the community. It wouldn’t be possible without all of your hard work and friendly faces, thanks! We look forward to what’s coming next in 2020!

Binder + Zenodo: A how-to guide

2019-06-21T13:46:00+00:00

Interactive and reproducible repositories powered by Zenodo and Binder.

When the Binder project was first launched, we imagined a world in which scientific scholarship and reproducibility could be carried out by the community using a fully-open stack of technology. We’re happy to say that this is now possible!

Zenodo DOIs now work with BinderHub

The BinderHub team recently added in support for building Binder links that point to Zenodo repositories. Zenodo is a general purpose open-access repository hosted by CERN that allows researchers to archive and apply a DOI to information that they put up on the web.

Zenodo has the ability to archive GitHub repositories, which means that you can archive the code, data, or reports that underly a scientific analysis and assign it a unique, citeable identifier. Now that BinderHub knows how to resolve a Zenodo identifier, you’ll be able to share Binder links that point to Zenodo and provide interactive access to your repository, letting readers reproduce results and interact with your analyses!

Here’s a quick primer for how to do this:

Step 1: Create a Zenodo account

First off you’ll need to create an account on Zenodo. You can
do so using a number of different log-in options.

Step 2: Create your Binder-ready repository on GitHub

Next, you should create your Binder-ready repository on GitHub. Binder uses pre-existing best practices in data science in order to infer and build the environment needed for your repository.

To make a repository Binder-ready, follow the instructions in the Binder docs.
Briefly, what you need to do is add the configuration files that define the environment needed to run your code. Once those files are in place, and you’ve added an analysis script (a Jupyter or R Notebook) that actually runs your code and displays the results, your repository is ready to build with Binder.

Step 3: Make sure your repository is ready to be published!

Once you create a DOI for your repository, it will be frozen in time — you won’t be able to easily update it. So double check that the repository builds properly with Binder and runs the way that you’d expect it to.

Make sure to launch a Binder from your repository and run the analyses you’d like others to run. If they produce the expected result from within a Binder session, then they’ll continue to do so for others (assuming you have pinned your versions and followed other best practices in reproducibility).

Step 4: Create a Zenodo DOI for your repository

Now that your repository is ready, you’ll connect Zenodo with GitHub to create a DOI for your repository. Remember that this will be unique to the current state of the repo — future changes to this repository won’t be reflected in the DOI.

We recommend following the GitHub Citable Code Guide which provides some best-practices for creating your Zenodo DOI for a GitHub repository. Click the image below to be taken to this (excellent) guide.

The GitHub Citable Code guide.

Once you’re done, you should have a Zenodo DOI badge
like the one below:

A Zenodo DOI badge

Step 5: Create a Binder link for your Zenodo DOI

Finally, use your Zenodo DOI to create a Binder link that allows others to interact with and replicate your results. You can create a Binder link for your Zenodo record by heading to https://mybinder.org and filling in the form:

Building a Binder link for your Zenodo archive with the Binder UI.

This will give you a link you can share with others as well as the Markdown and reStructured text snippets for creating a badge.

The link’s structure should look like this:

https://mybinder.org/v2/zenodo/<zenodo-DOI>

For example, if your Zenodo DOI is 10.5281/zenodo.3242074 (corresponding to this zenodo repository), the Binder link for it would be:

https://mybinder.org/v2/zenodo/10.5281/zenodo.3242074/

You can even pair this Binder link with your Zenodo DOI badge that
we showed above!

The “launch binder” button contains a Binder link for your Zenodo archive. Clicking will launch a Binder session.

And that’s it! You now have an archived version of your analysis with a unique identifier. This identifier can be used in conjunction with BinderHub to allow readers to interact with your code and results!

Last minute update 🎉

A few hours before publishing this post we merged a contribution from Tom Morrell who works at CalTech’s library that allows you to launch records from CalTech’s Data Repository as well! For example a notebook to analyze traffic to the archive itself: https://mybinder.org/v2/zenodo/10.22002/d1.1250. You should also check out a quick response to this post from Lorena Barba, who describes her group’s approach to creating reproducible, citable bundles for figures in papers, and how Binder+Zenodo DOIs might fit into that workflow!

What’s next?

We are close to closing the loop of fully reproducible computational environments for scientific publication. We’re excited to see journals begin to integrate these workflows with their own publishing pipelines. For example, the Neurolibre project is deploying their own BinderHub and using it alongside their reviewing and archiving process in order to provide more rich
interaction with submitted material.

Each type of repository needs a small amount of custom work to be integrated with Binder. We started with Zenodo because it is well known,
general purpose and integrated with GitHub already. If there is an archive you’d like to see integrated please do stop by this repository and open a new issue or contribute the code to do so directly.

If you’re working with a publisher and are interested in this
please reach out! The Binder community would love to work with you in deploying these open tools to make your published work more open and accessible. Whether it is big or small, we hope that these workflows can make an impact across the publishing landscape, and we’re looking forward to seeing what people do next!

elife sprint: Integrating Stencila and Binder

2018-12-10T10:15:00+00:00

This article reports on a project, integrating Stencila and Binder, which started at the eLife Innovation Sprint 2018 (#eLifeSprint). A longer version has been cross-posted on multiple blogs (eLife Labs, Stencila, o2r).

eLife, an open science journal published by the non-profit organisation eLife Sciences Publications from the UK, hosted the first eLife Innovation Sprint 2018 as part of their Innovation Initiative in Cambridge, UK: “[…] a two-day gathering of 62 researchers, designers, developers, technologists, science communicators and more, with the goal of developing prototypes of innovations that bring cutting-edge technology to open research communication.” One of the 13 projects at the excellently organised event was an integration of Binder and Stencila.

This article reports on the project’s results and changes made to Binder-related tools. Today, Binder has first class Stencila support. Read the full story at the eLife Labs blog post or try out to open Stencila documents from any online code repository on mybinder.org with the click of a single button:

Click the Binder badge to open a Stencila document on mybinder.org

This project is a collaboration between Min from the Simula Research Laboratory, Norway, a core developer of Binder and related tools, Daniel from the o2r project at the Institute for Geoinformatics, Germany, and Nokome, initiator and developer of Stencila. The final changes were made with the help of Tim. Thanks! The project was also part of the Mozilla Global Sprint 2018, see mozilla/global-sprint#317.

The building blocks and a challenge

Stencila Desktop is an office suite for reproducible research documents. It allows scientists to use languages like R and Python within familiar and intuitive word processor and spreadsheet user interfaces to lower the barriers to reproducible research for those with little or no software development skills. Binder (a part of Project Jupyter) makes it simple to generate reproducible computing environments from code repositories (e.g. GitHub or GitLab, see binder examples) and mybinder.org is the most prominent example. Binder uses repo2docker, for generating Dockerfiles (a human- and machine-readable recipe for setting up a computational environmentt used by the popular Docker container software) and building Docker images from software projects. While containers have become a commodity for developers, researchers still struggle to grasp and control the complexity of computational environments. This is where the two building blocks join: Running Stencila as part of a Binder helps researchers to communicate their work openly, to collaborate effectively with other scientists, and to ensure a high quality and transparency of their workflow and findings. Min and Daniel formulated their goal in the sprint project form: “[…] to connect them so that users can edit reproducible documents (DAR files) as part of a Binder project”.

Connecting Stencila and Jupyter: `nbstencilaproxy`

Stencila has “execution contexts”, an equivalent to Jupyter’s “kernels”. The contexts use code dependency analysis and return execution results as data values to enable a reactive, functional execution model. To open Stencila documents on binder, these execution contexts must be installed and configured in the environment created by repo2docker. This is achieved with a new software project initiated at the sprint: nbstencilaproxy - a Jupyter notebook server extension and proxy for Stencila.

The projects consists of a Python module with the Jupyter notebook server and “non-server” extensions of the same name, and a bundled JavaScript module (of the same name). The Python module allows proper versioned installation, dependency management, and installation from an established software repository. It takes care of the plumbing between the user interface and the services in the background, so that the binder is viewable over one port in the browser, while the many different background components run on their own ports. The “no server” extension adds a “Stencila session” menu entry and conveniently lives in the same directory structure as the server extension. The JavaScript module manages the required JavaScript dependencies and provides an well-defined structure for the code files. It serves the Dar document and provides access to the Stencila host.

repo2docker was extended with automatic detection of Dar documents, including the languages and execution contexts. As with most Binder repositories, a no configuration is needed for most common user cases: users can open a Dar document on Binder and trust the required environment to provide all required software. Daniel created a few example repositories to provide a starting point for users. The binder team generously welcomed the changes to mybinder.org and and the examples to the binder examples organisation on GitHub:

https://github.com/binder-examples/stencila-py contains Python code cells, using both the Jupyter and plain Python execution contexts:

Click the Binder badge to open a Stencila document with Python code on mybinder.org

https://github.com/binder-examples/stencila-r contains R code cells and two plots:

Click the Binder badge to open a Stencila document with R code on mybinder.org

One of the cool features of Stencila are the reactive cells, as demonstrated in a tweet following the feature release:

Thanks to @nordholmen working on @stencila support for https://t.co/Zlj6FrYgBw you now have reactive cells with Python code on @mybinderteam! Give it a go https://t.co/ToIuQPq0Fy

— Tim Head, @betatim on the internet (@betatim), November 12, 2018

Binder + Stencila is a demonstration of the power that the Open Source and Open Science community can foster. Many people are working together on the organisational and technological challenges of science today towards full research transparency and reproducibility, even if we use computers to an unprecedented level. Many small contributions on “side projects” such as these can make a difference, and connecting these two great projects hopefully helps to solve some problem in science down the road.

Join the public Stencila and binder chats to stay in touch or get help. We look forward to see scientists using nbstencilaproxy for communicating their work and new challenges that come with it.

#eLifeSprint-ers @minrk and @nordholmen are working to connect #JupyterNotebooks / #Binder with DAR / #Texture / @Stencila, so that users can edit reproducible documents as part of a Binder project https://t.co/2GoGNydsmX (@mybinderteam @ProjectJupyter @_substance)

— eLife Innovation (@eLifeInnovation), May 10, 2018

JupyterCon 2018, NYC August 21–25

2018-03-15T18:30:00+00:00

Discover how data-driven organizations are using Jupyter to analyze data, share insights, and foster practices for dynamic, reproducible data science.

I’m grateful to join Fernando Pérez and Brian Granger as a program co-chair for JupyterCon 2018. Project Jupyter, NumFOCUS, and O’Reilly Media will present the second annual JupyterCon in New York City on August 21–25.

Timing for this event couldn’t be better. The human side of data science, machine learning/AI, and scientific computing is more important than ever. This is seen in the broad adoption of data-driven decision making in human organizations of all kinds, the increasing importance of human centered design in tools for working with data, the urgency for better data insights in the face of complex socioeconomic conditions worldwide, as well as dialogue about the social issues these technologies bring to the fore: collaboration, security, ethics, data privacy, transparency, propaganda, etc.

To paraphrase our co-chairs, Brian Granger:

“Jupyter is where humans and data science intersect”

and Fernando Perez:

“The better the technology, the more important that human judgement becomes”

Consequently, we’ll explore three main themes at JupyterCon:

Interactive computing with data at scale: the technical best practices and organizational challenges of supporting interactive computing in companies, universities, research collaborations, etc (JupyterHub)
Extensible user interfaces for data science, machine learning/AI, and scientific computing (JupyterLab)
Computational communication: taking the artifacts of interactive computing and communicating them to different audiences

A meta-theme which ties these together is extensible software architecture for interactive computing with data. Jupyter is built on a set of flexible, extensible, and re-usable building blocks which can be combined and assembled to address a wide range of usage cases. These building blocks are expressed through the various open protocols, APIs, and standards of Jupyter.

The Jupyter community has much to discuss and share this year. For example, success stories such as the data science program at UC Berkeley illustrate the power of JupyterHub deployments at scale, in both education, research and industry. As universities and enterprise firms learn to handle the technical challenges of rolling out hands-on, interactive computing at scale, a cohort of organizational challenges come to the fore: practices regarding collaboration, security, compliance, data privacy, ethics, etc. These points are especially poignant in verticals such as healthcare, finance and education, where the handling of sensitive data is rightly constrained by ethical and legal requirements (HIPAA, FERPA, etc.). Overall, this dialogue is extremely relevant — it is happening at the intersection of contemporary political and social issues, industry concerns, new laws (GDPR), the evolution of computation, plus good storytelling and communication in general — as we’ll explore with practitioners throughout the conference.

Recent beta release of JupyterLab embodies the meta-theme of extensible software architecture for interactive computing with data. While many people think of Jupyter as a “notebook,” that’s merely one building block needed for interactive computing with data. Other building blocks include terminals, file browsers, LaTeX, markdown, rich outputs, text editors, and renderers/viewers for different data formats. JupyterLab is the next-generation user interface for Project Jupyter, and provides these different building blocks in a flexible, configurable, customizable environment. This opens the door for Jupyter users to build custom workflows, and also for organizations to extend JupyterLab with their own custom functionality.

Thousands of organizations require data infrastructure for reporting, sharing data insights, reproducing results of analytics, etc. Recent business studies estimate that more than half of all companies globally are precluded from adopting AI technologies due to a lack of digital infrastructure — often because their efforts toward data and reporting infrastructure are buried in technical debt. So much of that infrastructure was built from scratch, even when organizations needed essentially the same building blocks. JupyterLab’s primary goal is to make it routine to build highly customized, interactive computing platforms, while supporting more than 90 different popular programming environments.

Screenshot from the JupyterLab beta release. Image used with permission from Project Jupyter contributors.

A third major theme builds on top of the other two: computational communication. For data and code to be useful for humans, who need to make decisions, it has to be embedded into a narrative — a story — that that can be communicated to others. Examples of this pattern include: data journalism, reproducible research and open science, computational narratives, open data in society and government, citizen science, and really any area of scientific research (physics, zoology, chemistry, astronomy, etc.), plus the range of economics, finance, and econometric forecasting.

Another growing segment of use cases involves Jupyter as a “last-mile” layer for leveraging AI resources in the cloud. This becomes especially important in light of new hardware emerging for AI needs, vying with competing demand from online gaming, virtual reality, cryptocurrency mining, etc.

Please take the following as personal opinion, observations, perspectives: We’ve reached a point where hardware appears to be evolving more rapidly than software, while software appears to be evolving more rapidly than effective process. At O’Reilly Media we work to map the emerging themes in industry, in a process nicknamed “radar”. This perspective about hardware is a theme I’ve been mapping, and meanwhile comparing notes with industry experts. A few data points to consider: Jeff Dean’s talk at NIPS 2017, “Machine Learning for Systems and Systems for Machine Learning” about comparisons of CPUs/GPUs/TPUs, and how AI is transforming the design of computer hardware; The Case for Learned Index Structures, also from Google, about the impact of “branch vs. multiple” costs on decades of database theory; this podcast interview “Scaling machine learning” with Reza Zadeh about the critical importance of hardware/software interfaces in AI apps; the video interview that Wes McKinney and I recorded at JupyterCon 2017 about how Apache Arrow presents a much different take on how to leverage hardware and distributed resources.

The notion that “hardware > software > process” contradicts the past 15–20 years of software engineering practice. It’s an inversion of the general assumptions we make. In response, industry will need to rework approaches for building software within the context of AI — which was articulated succinctly by Lenny Pruss from Amplify Partners in “Infrastructure 3.0: Building blocks for the AI revolution”. In this light, Jupyter provides an abstraction layer — a kind of buffer to help “future proof” — for complex use cases in NLP, machine learning, and related work. We’re seeing this from most of the public cloud vendors, who are also leaders in AI, Google, Amazon, Microsoft, IBM, etc., and who will be represented at the conference in August.

Our program at JupyterCon will feature expert speakers across all of these themes. However, to me, that’s merely the tip of the iceberg. So much of the real value that I get from conferences happens in the proverbial “Hallway Track”, where you run into people who are riffing off news they’ve just learned in a session — perhaps in line with your thinking, perhaps in a completely different direction. Those conversations have space to flourish when people get immersed in the community, the issues, the possibilities.

It’ll be a busy week. We’ll have two days of training courses: intensive, hands-on coding, lots of interaction with expert instructors. Training will overlap with one day of tutorials: led by experts, generally larger than training courses though more detailed than session talks, featuring lots of Q&A.

Then we’ll have two days of keynotes and session talks, expo hall, lunches and sponsored breaks, plus Project Jupyter sponsored events. Events include Jupyter User Testing, author signings, “Meet the Experts” office hours, demos in the vendor expo hall — plus related meetups in the evenings. Last year the Poster Session was one of the biggest surprises to me: it was difficult to move through the room, walkways were packed with people asking presenters questions about their projects.

This year we’ll introduce a Business Summit, similar to the popular summits at Strata Data Conference and The AI Conf. This will include high-level presentations on the most promising and important developments in Jupyter for executives and decision-makers. Brian Granger and I will be hosting the Business Summit, along with Joel Horwitz of IBM. One interesting data point: among the regional events, we’ve seen much more engagement this year from enterprise and government than we’d expected, more emphasis on business use cases and new product launches. The ecosystem is growing, and will be represented well at JupyterCon!

We will also feature an Education Track in the main conference, expanding on the well-attended Education Birds-of-a-Feather and related talks during JupyterCon 2017. Use of Jupyter in education has grown rapidly across many contexts: middle/high-school, universities, corporate training, and online courses. Lorena Barba and Robert Talbert will be organizing this track.

Following our schedule of conference talks, the week wraps up with a community sprint day on Saturday. You can work side-by-side with leaders and contributors in the Jupyter ecosystem to implement that feature you’ve always wanted, fix bugs, work on design, write documentation, test software, or dive deep into the internals of something in the Jupyter ecosystem. Be sure to bring your laptop.

Note that we believe true innovation depends on hearing from, and listening to, people with a variety of perspectives. Please read our Diversity Statement for more details. Also, we’re committed to creating a safe and productive environment for everyone at all of our events. Please read our Code of Conduct. Last year we were able to work with the community plus matching donations to provide several Diversity & Inclusion scholarships, as well as more than dozen student scholarships. Looking forward to building on that this year!

That’s a sample of what’s coming up for JupyterCon in NYC this August. Meanwhile, we’ll be helping present and sponsor regional community events to help build momentum for the conference:

Jupyter Pop-up Boston, March 21
Jupyter Day Atlanta, March 31
Jupyter Pop-up DC (TBD)
plus, talks at related meetups in some North American metro areas

We look forward to many opportunities to showcase new work and ideas, to meet each other, to learn about the architecture of the project itself, and to contribute to the future of Jupyter.

Sign-up for email updates on the JupyterCon web site. See you there!

[kudos to Brian Granger for help developing and editing this article]

Reproducible Data Dependencies for Python [Guest Post]

2018-03-13T17:01:00+00:00

This article is the first in a series of guest blog posts about open source projects in the Jupyter ecosystem and the problems they attempt to solve. If you would like to submit a guest post to highlight a specific tool or project, please get in touch with us.

Jupyter Notebooks go a long way towards making computations reproducible and sharable. Nevertheless, for many Jupyter users, it remains a challenge to manage datasets across machines, over time, and across collaborators — especially when those datasets are large or change often. Quilt Data is a company that supports Quilt, an open source project to version and package data. The Quilt team recently released an extension for JupyterLab.

We are excited to see how the community will extend Jupyter and JupyterLab to manage datasets. Thanks to the Quilt team for submitting this guest blog post.

— The Jupyter Team

The open-source community has developed strong foundations for reproducible source code. Git supports versioning, GitHub supports collaboration. PyPI and Conda deliver code in immutable packages. Docker executes code in uniform, scalable containers.

But what about reproducible data? Data poses unique challenges: it’s larger than code, and resides in a wide variety of formats. Each data format implies different tradeoffs in serialization performance, compression, and file size. As a result, managing data becomes intractable in line-based version control systems like git. This presents a problem for Jupyter users: source code gets shared, but data gets left behind.

One solution to this problem is to port successful abstractions from source code management over to data. Versioning, packaging, and execution are well understood and universally adopted in source code management. In this article we’ll explore a collection of services that version, package, and marshal data — Quilt.

Getting data into notebooks

Notebooks that depend on data from files are fragile. File formats change, file stores move, files are copied, and file copies diverge. As a result, notebooks break as we share them across collaborators, across machines, and over time.

Quilt hides network, files, and storage behind a data package abstraction so that anyone can create durable, reproducible data dependencies for notebooks.

To run the sample code in this article, launch your favorite Python environment and install quilt:

$ pip install quilt

Now we use quilt to pull data dependencies into a Jupyter notebook:

import quilt

# install a small subpackage
quilt.install("uciml/heart_disease/tables/processed/switzerland", force=True)

# import the data package
from quilt.data.uciml import heart_disease

In the above code, uciml/heart_disease is a data package. Packages live in repositories and have handles of the form USER/PACKAGE. The package repository includes a versioned history of the data, which we can access as follows:

In [1]: import quilt
In [2]: quilt.log("uciml/heart_disease")

Hash                  Pushed              Author
eb79ef86f84fdcf61e... 2017-06-13 19:54:44 uciml
780397900f81c3c088... 2017-06-13 19:52:21 uciml

We could install a specific version of the data by providing the hash= keyword argument to quilt.install.

Let’s access the data in our subpackage. Data are loaded into memory by adding parentheses to a package path, as follows:

heart_disease.tables.processed.switzerland()

Heart disease data in a data frame

Command line and Python interfaces

Virtually all Quilt commands are available on both the command line and in Python. For example $ quilt log uciml/heart_disease in Terminal is equivalent to quilt.log("uciml/heart_disease") in Python.

What about unstructured data, like images?

In the preceding example we saw that Quilt reads columnar data into data frames. Semi-structured and unstructured data — such as JSON, images, and text — are also supported. Unstructured data skip serialization and are simply copied into Quilt’s object store. For instance if a user calls pkg.unstructured_txt() they receive a path to the unstructured file on disk, not a data frame. Future versions of Quilt will provide a wider variety of native deserializers (e.g. JSON to dict for Python).

Understanding the object store

All package data are read from Quilt’s object store. The object store provides three performance enhancements:

deduplication — Each unique data fragment is named by its SHA-256 hash so that when Quilt users push and pull data, only the fragments that have changed are sent over the network
fast reads of large files — the package akarve/nyc_taxi contains a 1.7GB CSV table. Quilt produces a data frame from the table in 4.88 seconds, thanks to PyArrow’s efficient handling of Parquet. By comparison, pandas.read_csv() takes 47 seconds to produce the same data frame from its CSV source. And pandas.read_sql() takes more than 5 minutes to acquire the same data from a database.
columnar storage — Quilt converts tabular data into Apache Parquet columns. Parquet columns can be efficiently compressed, deserialized, and searched by tools like PrestoDB and HiveSQL.

Browsing data packages in JupyterLab

As Jupyter users we want to make it easy to find and consume new data packages. Quilt leverages JupyterLab’s extension architecture to create an extension that lets you search Quilt for data packages.

If desired, switch to an environment of your choice. Next, install the Quilt extension for JupyterLab as follows (see the quilt extension repo for further documentation):

$ pip install quilt
$ pip install jupyter -U
$ pip install jupyterlab
$ jupyter labextension install jupyterlab-quilt
$ jupyter lab

Quilt extension for JupyterLab

If you type “uciml” in the search box you’ll see a list of packages from the UCI Machine Learning Repository. If you click on a package in the list, the extension will generate the code to install and import the package. Click on the link icon, far right, to visit the package repository page, where you’ll find further documentation on the data.

quilt.yml is like requirements.txt for data

We can express a notebook’s data dependencies in a YAML file, conventionally called quilt.yml:

packages:
  - asah/mnist
  - uciml/heart_disease/tables/processed/switzerland:hash:eb79ef86
  - akarve/pydata_book/titanic:tag:features

Anyone with the above wishes to reproduce a notebook that depends on the above packages can type the following:

$ quilt install quilt.yml

Jupyter notebooks on the same machine now have access to the quilt.yml-specified versions of MNIST, Titanic, and Heart Disease.

quilt.yml for Binder notebooks

If you’re running Jupyter notebooks on the web with Binder, here’s an example of how to install data dependencies in a Binder container.

Build your own data package

Let’s create a data package that contains Bitcoin prices. You can start by downloading the source data BTC_prices.csv, placing it in a clean directory, and changing to the same directory:

$ mkdir quilt-btc
$ cd quilt-btc
$ curl https://s3.amazonaws.com/quilt-web-public/data/BTC_prices.csv -o BTC_prices.csv
$ ls
BTC_prices.csv

We can use quilt generate to create a build.yml file. build.yml specifies how data are transformed into an in-memory package tree.

$ quilt generate .
Generated build-file ./build.yml.

Here’s what’s in build.yml:

contents:
BTC_prices:
  file: BTC_prices.csv

quilt generate recursively descends a directory to include all descendants folders and files in a generated build.yml file. Quilt packages can contain thousands of files and hundreds of gigabytes of data.

Let’s modify our build.yml to specify how we want our data package to be structured. We’ll rename the node, and use kwargs to parse the Date column as a datetime, specify the quote character, and skip some comment rows.

contents:
  prices:
    kwargs:
      parse_dates: ['Date']
      quotechar: "\""
      skiprows: [2714, 2715, 2716]
    file: BTC_prices.csv

(Behind the scenes, kwargs are handed to pandas.read_csv.)

Finally, we’ll document our data package with a README.md file:

# Bitcoin prices

This data was produced from the [CoinDesk price page](http://www.coindesk.com/price/).

We need to add README.md to build.yml so that the Quilt compiler knows to include it in the package:

contents:
  README:
    file: README.md
  prices:
    kwargs:
      parse_dates: ['Date']
      quotechar: "\""
      skiprows: [2714, 2715, 2716]
    file: BTC_prices.csv

If you wish to upload your package you’ll need a Quilt account. You can log in to Quilt from Terminal:

$ quilt login

Now we rebuild the package to include our README and new kwargs:

$ quilt build USERNAME/PACKAGE build.yml

Finally, we publish our package to the registry so that other users can access the data:

$ quilt push USERNAME/PACKAGE --public

Your package is now live in the web catalog, like this akarve/BTC example.

quilt log USERNAME/PACKAGE shows the time and tophash for our recent push. Here’s what quilt log outputs for my package:

Hash                             Pushed              Author
e81757c6cc6fd5ccfcd589e0b62a7d9… 2017–12–22 13:43:12 akarve

Any notebooks that depend on this version of the akarve/BTC package can declare the dependency in a quilt.yml file or directly in code:

quilt.install("akarve/BTC", hash="e81757c")

Alternatively, we can associate a human-readable tag with any of the hashes from quilt log.

$ quilt tag add akarve/BTC crypto e81757c

Any notebooks that contain the following line of code point to the same immutable data package and are reproducible across machines:

quilt.install("akarve/BTC", tag="crypto")

Where does my data live?

By default Quilt store data in the registry at quiltdata.com. Alternatively, you can host your own registry by running the open source containers, then using quilt config to point clients to your private registry.

Conclusion

We’ve explored data packages as versioned, immutable building blocks that encapsulate data dependencies. The data package lifecycle is driven by four commands: build, push, install, and import.

In the near future we plan to make the JupyterLab extension bidirectional, so that users can not only pull data from Quilt, but easily push cell data into Quilt. Support for R, Spark, and HDFS are also on the Quilt roadmap.

The Quilt compiler, registry, and catalog are open source. We welcome your contributions.