Jupyter Blog - open science

Exploring Petabytes of the Night Sky — Jupyter Notebooks at NOIRLab’s Astro Data Lab Science Platform

2026-04-20T18:04:00+00:00

By Robert Nikutta & Stéphanie Juneau (NSF NOIRLab)

Imagine querying 420+ billion rows of astronomical catalog data — spanning 30 major sky surveys, observed over decades with telescopes on three continents — from a Jupyter notebook in your browser in seconds. No download. No HPC allocation request. No waiting.

That is what 4,800+ astronomers in over 90 countries can do every day at the Astro Data Lab science platform. Data Lab is operated by NSF NOIRLab, the National Optical-Infrared Astronomy Research Laboratory, headquartered in Tucson, Arizona, with observatories in Arizona, Hawai’i, and Chile. Since its public launch in June 2017, Astro Data Lab has quietly become one of the largest deployments of Jupyter notebooks in professional science — and a case study in what happens when you bring the compute to the data instead of the other way around.

World map of 2024 number of data queries at Astro Data Lab by country (log-scale color): 64.2 million queries from 72 countries that year.

The Data Problem Astronomy Had to Solve

Modern sky surveys are data machines. The Dark Energy Survey cataloged 690 million objects. Gaia measured positions and motions for 1.8 billion stars. The DESI Legacy Surveys cover 20,000 square degrees, nearly half of the full sky, in three optical bands. And the upcoming Rubin Observatory’s Legacy Survey of Space and Time (LSST) will generate roughly 10 million transient alerts per night starting later this year.

Traditional astronomy workflows begin with downloading relevant data to a local computer and to use locally installed specialized software tools to process and analyze the data. However, downloading these catalogs to a local machine is now often physically impossible. A single survey’s measurements table can exceed the combined disk space of an entire research group. And even if you could download it, the computing resources needed to query it efficiently at scale requires infrastructure most astronomers don’t have.

The answer the community converged on, like many industries dealing with big data: bring the compute to the data. Host the catalogs in databases, co-locate a computing environment next door, and give scientists a familiar interface to work in. That interface, increasingly, is a Jupyter notebook.

Astro Data Lab: Jupyter at the Observatory

Astro Data Lab was conceived in 2014 and went public in June 2017, originally built to support data releases from the Dark Energy Survey — a few terabytes of catalogs and tens of terabytes of images. We imagined a ceiling of roughly 500 users. We were wrong, in the best way.

Today the platform hosts:

About 420 billion catalog rows across 30+ major astronomical surveys (DES, Legacy Surveys, DESI, NOIRLab Source Catalog, SDSS, Gaia, unWISE, SMASH, S-PLUS, VHS, 2MASS, and dozens more)
31 million spectra via SPARCL, our spectral access service (DESI DR1+EDR, SDSS/BOSS DR17)
Petabytes of images, accessible through a Simple Image Access service and cutout API
Over 4,800 registered users from over 90 countries, who submit tens of millions of data queries each year

Every registered user gets a persistent JupyterHub environment with the full astronomy Python stack pre-loaded — Astropy, NumPy, SciPy, Matplotlib, Pandas, Scikit-learn — and our own astro-datalab client library. The library provides core services, for instance auth and DB queries:

from dl import authClient, queryClient
from getpass import getpass

# Log in
token = authClient.login(input("username: "),getpass("password: "))

# Query 10 objects from the NOIRLab Source Catalog near a sky position
# Right Ascension (RA) = 150.12 degrees
# Declination (Dec) = 2.21 degrees
# Search radius = 0.05 degrees
# q3c (Quad Tree Cube) is a spatial indexing scheme for Postgres
# gmag and rmag are the g-band and r-band magnitudes of objects
# in the NOIRLab Source Catalog Data Release 2, ‘object’ table.
result = queryClient.query(
    """SELECT ra, dec, gmag, rmag FROM nsc_dr2.object
       WHERE q3c_radial_query(ra, dec, 150.12, 2.21, 0.05) LIMIT 10""",
    fmt="pandas"
)

It’s as simple as that. The query runs on the database server next to the data; only the result set crosses the network.

Use Case 1 — Seeing the Sky Inside a Notebook with AladinLite

One of the most immediate joys of working with astronomical data is visualization: not just numbers in a table, but where things are in the sky, what the images look like, and how your query results relate to the underlying survey footprint.

We’ve integrated AladinLite v3 — the interactive sky atlas from Centre de Données Astronomiques de Strasbourg (CDS) — directly into the notebook environment via the ipyaladin widget. With a handful of lines, astronomers can embed a fully interactive sky viewer in a notebook cell or next to their notebook in a “sidecar”, and overlay their own data on top of real survey imagery:

import time
from astropy import units as u
from astropy.table import Table
from astropy.coordinates import SkyCoord
from ipyaladin import Aladin
from sidecar import Sidecar

# Instantiate the Aladin interactive sky viewer 
aladin = Aladin(full_screen=True)
with Sidecar(title="aladin_output",anchor='split-right'):
    display(aladin)

# globular cluster NGC 1851 (RA, Dec)
aladin.target = SkyCoord(78.52809*u.deg, -40.04656*u.deg)
aladin.coo_frame = "ICRSd"  # set coordinate frame to ICRS, angles in deg
time.sleep(1)  # race condition
aladin.fov = 0.4  # set field of view to 0.4 degrees

# Overlay catalog query results as circle markers
t = Table.from_pandas(df)  # e.g., from a previous query around NGC 1851
aladin.add_table(t,shape='circle',source_size=15,color='green')

The result is a pannable, zoomable sky viewer — right in the notebook — with your query results overlaid as green circles on the actual sky image of a globular cluster (see figure below). Users can overlay MOCs (Multi-Order Coverage maps, which encode survey footprints), user-generated catalogs from a prior query, or any Virtual Observatory-standard data source.

This capability turns what was once a static plot into an exploratory tool: zoom into a cluster, click on a source, cross-match on the fly. For students and scientists unfamiliar with a dataset, it is often the fastest path from “I have a list of objects” to “I understand where they are and what I’m looking at.”

AladinLite integration is now active in our notebook library, with full deployment into the new Data Lab Web Portal on the roadmap for later this year.

AladinLite v3 sky viewer inside a Jupyter notebook, showing a globular cluster with catalog query results in the outskirts of the cluster overlaid as green circles.

Use Case 2 — Stacking Galaxy Spectra with SPARCL

Spectroscopy — measuring how much light a star or a galaxy emits at each wavelength — is one of astronomy’s most powerful tools. But individual spectra are often noisy. The signal-to-noise ratio of a single optical spectrum for a faint galaxy can be too low to measure the emission lines that encode star formation rate, gas chemical content (Oxygen, Nitrogen, etc.) or the even more subtle absorption lines that create small wiggles in the shape of the spectrum, yet encapsulate crucial information such as the mass and age of the stars making up a galaxy.

One trick that astronomers have used for decades: combining or “stacking” spectra. Average hundreds of spectra together, and the noise level reduces while the signal builds up. What was invisible in a single spectrum becomes unmistakable in the stack. While the concept is simple, reading and manipulating large numbers of spectra can be time consuming or cumbersome.

SPARCL (SPectra Analysis and Retrievable Catalog Lab) makes this possible at scale directly in a notebook. With sparclclient, users can currently search 31 million spectra by redshift range, target type, and survey, then retrieve flux arrays and wavelength grids ready for stacking:

from sparcl.client import SparclClient

# Instantiate the SPARCL client (connected to production server)
client = SparclClient()

# Find SDSS spectra of galaxies in a redshift slice 0.1<z<0.3
found = client.find(
    outfields=['sparcl_id', 'ra', 'dec', 'redshift', 'spectype'],
    constraints={'spectype': ['GALAXY'],
                 'redshift': [0.1, 0.3],
                 'data_release': ['SDSS-DR17']},
)

# Retrieve flux, wavelength, and inverse-variance arrays
retrieved = client.retrieve(found.ids,
            include=['flux', 'wavelength', 'ivar'])

In our SpectralStacking_SDSS science example notebook, users first stack a small number of galaxy spectra (N=5) in eight bins of astrophysical color g−r (green and red filters), revealing trends from blue spectra with emission lines to red spectra with absorption lines but with noisy spectra. Then users stack hundreds of galaxy spectra for the same bins of color g−r and obtain much cleaner spectra where the small wiggles are now real astrophysical features and no longer buried in the noise.

The spectral rainbows below — N=5 then N=200 stacked galaxy spectra color-coded in bins of astrophysical color g−r — are each a single output cell from this notebook, generated entirely within the Data Lab environment.

SPARCL spectral rainbows: top panel shows N=5 galaxy spectra stacked and color-coded by g−r color, spanning wavelengths 3750–6450 Ångstrom. Bottom panel shows the same exercise but with N=200 galaxy spectra per bin, greatly enhancing the signal-to-noise ratio.

Use Case 3 — Variable Stars and the Coming Flood of Time-Domain Data

Not all astronomical data is a static snapshot of the sky. Many of the most scientifically rich phenomena — pulsating stars, transiting exoplanets, exploding supernovae, gravitational lensing events — reveal themselves through change over time.

Among the most useful calibration tools in astrophysics are RR Lyrae stars: old, low-mass stars that pulsate with periods of 0.2–1 day and a brightness variation that traces their distance. Finding and characterizing them across millions of square degrees of sky requires querying multi-epoch photometry catalogs, computing period statistics, and folding light curves — all tasks that fit naturally in a notebook workflow.

Our TimeSeriesAnalysisRrLyraeStar notebook demonstrates the full pipeline: query the SMASH DR2 catalog for stars with high photometric variability, run a Lomb-Scargle periodogram on the light curve, identify the dominant period, and phase-fold the observations to reveal the characteristic sawtooth pulsation profile:

from astropy.timeseries import LombScargle
import numpy as np

ls = LombScargle(t, y)  # time and magnitude from a previous query
frequency, power = ls.autopower()
period = 1./frequency # period is the inverse of frequency
best_period = period[np.argmax(power)]
phase = (t / best_period) % 1  # folded timeseries = light curve

The resulting phase-folded light curve shown in the figure below is clean, precise, and immediately recognizable to any variable-star astronomer — produced entirely from archival survey data without a single new observation.

This kind of workflow is also a proving ground for the upcoming Vera C. Rubin Observatory’s Legacy Survey of Space and Time (LSST). When Rubin begins operations and delivers 10 million nightly alerts, the only workflows that will scale are ones already designed to run against large databases or specialized file systems, in shared computing environments, with notebook-native tooling. Astro Data Lab users are building those workflows today.

Phase-folded RR Lyrae light curve from the TimeSeriesAnalysisRrLyraeStar notebook, showing characteristic sawtooth pulsation with a 0.65 day period.

The Notebook Ecosystem

The three use cases above are drawn from our library of 80+ open-source Jupyter notebooks at github.com/astro-datalab/notebooks-latest. The library is organized into six sections:

Directory                      Contents
---------------------------------------------------------------------------
01_GettingStartedWithDataLab/  Authentication, dataset discovery, first
                               queries

02_DataAccessOverview/         More advanced queries, image searches, etc.

03_ScienceExamples/            Many complete science cases (stellar
                               streams, dwarf galaxies, large-scale
                               structure, SED fitting, ...)

04_HowTos/                     Service-specific tutorials (SPARCL, SIA
                               image cutouts, cross-matching, file storage)

05_Contrib/                    Community-contributed notebooks (ANTARES
                               alert broker, user science cases, etc.)

06_EPO/                        Education & public outreach (Teen Astronomy
                               Cafe, La Serena School for Data Science)

All notebooks are open-source and community contributions are welcome via pull request. We use them as living teaching materials in workshops at Astronomical Data Analysis Software & Systems (ADASS) and American Astronomical Society (AAS) conferences, summer schools, and university courses around the world. We have also recently translated most of our notebooks to the Spanish language.

In the coming months we will launch a tagged, searchable notebook gallery — filterable by science topic, Unified Astronomy Thesaurus (UAT) keywords, target audience, and difficulty level. The pilot framework was developed by two summer students working with the team.

Footprints of 24 sky survey datasets hosted at Astro Data Lab. This montage shows the wide variety of astronomical surveys, with some that cover the full sky, others focusing on the Milky Way (central plane), and yet others studying the extragalactic regions beyond the Milky Way. We ensure that each survey is represented in at least one of our example notebooks.

Looking Ahead

Nine years in, the Astro Data Lab science platform is evolving on several fronts simultaneously.

GPU computing. We are deploying a GPU node, which will be connected to the Jupyter notebook service. This opens deep learning and large-scale ML workflows in the same notebook environment where the data lives.

An AI assistant. Our first-ever user survey, conducted in September 2025, ranked an in-notebook AI assistant as one of the top requested features. We are actively exploring what responsible, science-aware AI assistance looks like in this context — helping users construct SQL/ADQL queries, navigate datasets, and debug notebook code, without hallucinating catalog column names. jupyter-ai might come in very handy here.

New integrated Web Portal. Our Data Explorer — an integrated web interface combining catalog browsing, query execution, image cutouts, spectral search, and job status monitoring — was rolled out last year. Some of the next milestones include integration of AladinLite into the portal, bringing the sky-visualization capability described above out of the notebook and into the browser-native interface, and a new integrated positional cross-matching service.

Try It

The full notebook library is open-source: github.com/astro-datalab/notebooks-latest. Community notebook contributions are welcome — see CONTRIBUTING.md in the repository. You can also run all notebooks locally, after installing the Data Lab command-line client and Python module: pip install astro-datalab

Astro Data Lab also offers a JupyterLab environment as a service to the broad astronomy community — students, researchers, educators, and citizen scientists. Registration takes just a moment at datalab.noirlab.edu.

Questions and feedback: datalab@noirlab.edu

Robert Nikutta is a scientist at NSF NOIRLab’s Community Science and Data Center, and lead of the Astro Data Lab science platform. Stéphanie Juneau is an associate astronomer at CSDC and lead of the SPARCL spectroscopy initiative. The platform is the work of the full Astro Data Lab team, past and present.

Jupyter recognized as “Champion of Open Science” by U.S. White House

2024-05-07T20:51:00+00:00

Jupyter is recognized as a “Champion of Open Science” by the United States White House Office of Science and Technology Policy, in celebration of 2023 being the Year of Open Science. For more information, see the White House press release.

This recognition honors the large group of contributors, users, and organizations that have made Jupyter what it is today. More specifically, we, the Project Jupyter Executive Council, lists and thanks here the current (as of December 2023) and previous members of the following Jupyter leadership groups: Executive Council, Software Steering Council, Software Subproject Councils, Working Groups, Standing Committees, Distinguished Contributors, and the former Jupyter Steering Council. (If there is someone in these groups that we accidentally missed in the list below, please let us know.)

Project leaders and distinguished contributors (alphabetical by last name)

Safia Abdalla
Damián Avila
Lorena A. Barba
Kevin Bates
Mehmet Bektas
Douglas Blank
Nicholas Bollweg
Alex Bozarth
Maarten Breddels
Nicolas Brichet
David Brochart
Matthias Bussonnier
Eric Charles
S. Chris Colbert
Frédéric Collonval
Sylvain Corlay
Martha Cryan
Carlos Córdoba
Itay Dafna
Afshin Darian
Jacob Diamond-Reivich
Georgiana Dolocan
R Ely
Tony Fast
Vidar T. Fauske
Sharan Foga
Jessica Zosa Forde
Gabriel Fouasnon
Jonathan Frederic
Eric Gentry
Tim George
Sarah Gibson
Kevin Goldsmith
Rahul Goyal
Brian Granger
Jason Grout
Jessica Hamrick
Tim Head
Lindsey Heagy
Carlos Herrero
Chris Holdgraf
Andrii Ieroshenko
Paul Ivanov
Piyush Jain
Lucy Jimenez
Kyle Kelley
Max C Klein
Thomas Kluyver
Michał Krassowski
Simon Li
Ryan Lovett
Joe Lucas
Johan Mabille
Grant Nestor
Cameron Oelsen
M Pacer
Yuvi Panda
Peter Parente
Fernando Pérez
Isabela Presedo-Floyd
David Qiu
Min Ragan-Kelley
Martin Renou
Luciano Resende
Rosio Reyes
Ian Rose
Ana Ruvalcaba
Zach Sailer
Ayaz Salikhov
Matthew Seal
Saul Shanabrook
Steve Silvester
William Stein
Erik Sundell
Nicolas Thiéry
Rollin Thomas
Jeremy Tuloup
Rick Wagner
Mariko Wakabayashi
Jason Weill
Carol Willing
Jessica Xu

Additionally, we’d like to thank the following organizations that were directly involved in the work (other than funding agencies) in terms of staff or other significant resources to directly support the development of Project Jupyter.

Organizations (alphabetical)

Amazon Web Services
Anaconda
Bloomberg
California Polytechnic State University, San Luis Obispo
Databricks
Lawrence Berkeley National Laboratory
MongoDB
QuanSight
QuantStack
Simula Research Lab
Two Sigma
University of California, Berkeley

Curvenote joins the mybinder.org federation

2024-02-22T16:19:00+00:00

Curvenote is now a part of the mybinder.org federation. By hosting a BinderHub instance running on the company’s infrastructure, additional capacity is available on the https://mybinder.org service.

curvenote.com/for/journals

The mybinder.org project provides a critical, free public service for scientists, researchers and educators worldwide. The ability to spin up a Jupyter server to run someone’s research code or run a class without local configuration is something that many academics and educators rely on.

Mybinder.org runs on a set of open source software components and standards, that not only powers the service but that also allows us to make solid steps forward in reproducibility in scientific research, namely REES and open source software components (like repo2docker) .

This is something we’re keen to shout about as Curvenote runs numerous small scale, independent, BinderHubs to power its journal infrastructure for notebook based publishing for reproducible scientific journals with BinderHub, Jupyter and MyST Markdown (https://mystmd.org) being important components of that.

Over the last few months a lot of work has gone into bringing up the Curvenote BinderHub instance on AWS. This is mainly because in the process, the AWS workflow based deployment has been pushed forward by Simon Li (@manics, University of Dundee, JupyterHub Core Team), solving many previous roadblocks around EKS (Elastic Kubernetes Service) usage and paving the way for easier future AWS based deployments (watch out for a future post from Simon with all the technical details).

It’s been great to collaborate with Simon during that time and get exposure to the details of BinderHub deployment on AWS.

The mybinder.org service makes a significant positive impact on public education, science and research. This impact is directly supported by the volunteer time of the BinderHub core team and the federation member organizations. Curvenote’s instance adds capacity to the service and a broader group of organizations contributing to the organization would lead to a more resilient public service, that is less prone to disruption when a single federation member leaves. If your organization has the resources to run and maintain a BinderHub, or makes heavy use of the mybinder.org service, encourage your organizations to run an instance and join the federation.

JupyterCon 2023 is just around the corner, and it promises to be an exciting and action-packed…

2023-02-09T23:43:00+00:00

JupyterCon 2023 is just around the corner, and it promises to be an exciting and action-packed event that you won’t want to miss. With a lineup of world-renowned keynote speakers, including Alyssa Goodman, Paul Romer, Craig Peters, and Cory Gwin, this year’s conference is set to be a true highlight of the Jupyter community.

The three-day conference will be packed with talks, tutorials, and a full schedule of community events. From May 10th to May 12th, attendees will have the chance to hear from some of the brightest minds in the field and engage in discussions with like-minded individuals from around the world.

But that’s not all. The conference will be held at the largest science museum in Europe, providing attendees with a truly unique and immersive experience. And, to add to the excitement, two days of sprints will follow the conference from May 13th to May 14th. This is a fantastic opportunity for attendees to work together on new projects, share their knowledge and skills, and collaborate with other Jupyter enthusiasts.

If you’re thinking of attending JupyterCon 2023, now is the time to act. Early bird passes are still available and provide substantial savings, so make sure to grab your pass before the full price kicks in on March 1st. In addition to individual and corporate passes, there are also economically priced student passes available. Please note, a valid student ID will be required to pick up your student pass at the conference.

https://www.jupytercon.com/tickets

At JupyterCon, diversity is a priority, and NumFOCUS and the JupyterCon team believe that an inclusive community is key to success. To help foster this environment, financial aid and scholarship opportunities are available for those who need it. There are two types of financial aid grants available: a free ticket, which includes access to all scheduled events during the conference, and one for travel and accommodation costs, which will partially cover the costs of getting to and staying at the conference. To apply for these scholarships, visit jupytercon.com and submit your application before the March 3rd deadline.

Stay tuned for the final schedule, which will be released soon, and get ready for a fantastic few days at JupyterCon 2023. With so much happening at this year’s conference, it promises to be an unforgettable experience that you won’t want to miss.

Jupyter meets the Earth: EarthCube Community Meeting

2020-08-17T19:42:00+00:00

By: Lindsey Heagy, Fernando Pérez, Joe Hamman and the Jupyter meets the Earth team (cross-posted on the Pangeo Blog)

As a part of the 2020 EarthCube annual meeting, we held a Jupyter meets the Earth community discussion session on July 27. The Jupyter meets the Earth project is an EarthCube funded effort that combines research use cases in geosciences with technical developments within the Jupyter and Pangeo ecosystems. In this model of equal partners, scientific questions help drive software infrastructure development, and new technologies expand the horizons of viable research. This online workshop was an opportunity to gather members of the community, welcome newcomers, provide updates on the Jupyter and Pangeo ecosystems, and have time for discussion.

The goals for the meeting were to:

Provide an overview of the Jupyter & Pangeo ecosystems for researchers from the EarthCube community.
Outline avenues for getting involved.
Gather input for what advancements would best serve your research needs.

Over 100 participants registered, and we had contributions from 9 speakers. The meeting was a mix of presentations and Q&A from the community. The full recording of the meeting is available on youtube, and we encourage continued discussion on the associated discourse post.

Presentations (google drive folder)

Fernando Pérez (slides) started off the meeting by introducing the Jupyter meets the Earth project — an effort aimed at driving forward technological developments in the Jupyter and Pangeo ecosystems in partnership with researchers in the geosciences. The motivation is to advance research and the software that supports it by combining domain expertise with methods in data science, software & data engineering practices. He provided an overview of Project Jupyter, highlighting the interplay between software and content, services, standards, community and governance that is necessary for broad-impact scientific open source software projects. He presented the extensible JupyterLab platform, that can be adapted to domain-specific needs as illustrated by the FlyBrainLab and Cloud Robotics Command Station efforts. The Jupyter meets the Earth team aims to similarly develop tools and extensions that will support interactive computing workflows in the geosciences.

Overview of Jupyter and Jupyter meets the Earth from Fernando Pérez

Next up, Scott Henderson (slides) provided an overview of Pangeo and associated community events, including Hackweeks. “Pangeo is first and foremost a community promoting open, reproducible, and scalable science.” In terms of technology, this involves developing fully open source tools that can be deployed on shared computational infrastructure, such as HPC centers or the cloud, and hosting several forums to foster communication between scientists and software developers. Software is an important avenue for connection, but the overarching goals are a rallying point for a community. The critical mass of enthusiastic people has been key to the success of the Pangeo model.

Overview of Pangeo and Hackweeks from Scott Henderson

The Pangeo model is intended for use both on the cloud and on High Performance Computing (HPC) infrastructure. Kevin Paul gave the third talk on Pangeo on HPC (slides, notebook). HPC and cloud computing environments present technical differences in terms of usage patterns, file access, and resource allocations, however, the goals of Jupyter and Pangeo are similar in both cases — to enable interactive computing and simplify the user experience on both. Tools such as dask and dask-kubernetes/dask-jobqueue are targeted at enabling parallel computing on both infrastructures.

Kevin Paul giving us a demo of Pangeo on the Cheyenne supercomputer

After a Q&A session that included questions on the computational cost of running Pangeo Infrastructure and efficient use of tools including Zarr, we moved on to a series of lightning talks.

Lightning Talks

Six speakers presented short lightning talks on aspects of the Jupyter and Pangeo ecosystems ranging from technologies to scientific applications to opportunities to engage with the Pangeo community.

Anderson Banihirwe (notebook and details in intake-esm) kicked off the lightning talks by giving a demo and overview of Intake — a project to streamline loading and sharing of data. He showed a demo that included both Optimum Interpolation Sea Surface Temperature (OISST) data, as well as data from the Coupled Model Intercomparison Project (CMIP) running interactively on Cheyenne, the supercomputer at NCAR and using Dask for distributing the workload across nodes, with real-time diagnostics of the distributed computation provided by Dask’s JupyterLab extension.

Demo from Anderson Banihirwe using intake to access OSSIT and CMIP data

Next up, Scott Dale Peckham (notebook) gave a presentation that demonstrated the use of ipywidgets and ipyleaflet to create an interactive interface for fast access to geoscience data on servers that support the OpenDAP protocol. The project he champions is called BALTO, the Brokered Alignment of Long Tail Observations (also a famous Siberian Husky and sled dog). These graphical interface elements can be used in a programmatic workflow such as a Jupyter Notebook, but they conveniently encapsulate many details of accessing the data and resources provided by BALTO. This allows the scientists to focus on their research questions, without having to break their workflow to access data with external tools.

BALTO GUI demo from Scott Peckham

We then had a talk from Edom Moges (slides) who presented work he is conducting with Laurel Larsen’s research group in hydrology as one use case in the Jupyter meets the Earth project. The presentation focused on a data synthesis work that aims to build a Jupyter based interactive platform that transforms raw hydrometeorological data to a gap-filled ready to use data for several intensively monitored watersheds across the US. The platform will be a basis for future community initiatives to benchmark data processing approaches, support comparative hydrological studies and comprehensive data-driven forecasts.

Lightning talk from Edom Moges and Laurel Larsen on the hydrology use-case in the Jupyter meets the Earth project

Georgiana Dolocan (video) impressed us next with an animated video accompanied with her narration to explain JupyterHub, its components (authenticator, spawner, proxy) as well as deployment options. The littlest JupyterHub (TLJH) is designed to make it simple to deploy multi-user Jupyter infrastructure on a single machine, and the more sophisticated Zero 2 JupyterHub Kubernetes (Z2JH) option is meant to scale to many users and large computational needs.

Animations from Georgiana Dolocan on JupyterHub

Presenting from the perspective of an enthusiastic user, Erik Sundell (slides) gave us an overview of Jupyter Book: a tool to quickly create beautiful websites from notebooks and markdown. He walked through how to host them for free online in a time efficient way, and highlighted features including connections to Binder, which enable users to run content interactively.

Overview of JupyterBook from Erik Sundell

Joe Hamman (slides) finished off our lightning talk session by outlining avenues for connecting with the Pangeo community. These include day-to-day communication on GitHub, Gitter, discourse and twitter, as well as more recent coffee-breaks. Depending on your topic of interest, there are also working groups that you can join on topics including data, machine learning, education, cloud computing, or you can suggest your own!

Connecting with the Pangeo community — an overview from Joe Hamman

Follow up and further discussion

To continue the discussion afterwards, we posed (slides) a few questions where we hope to learn from the community’s needs, such as:

What does your interactive computing workflow look like today? What do you envision it will be in 5 years?
How would you like to publish and share your computational research and where can improvements be made?
How do you stay up to date with the evolving open-source ecosystem? How would you like to be keeping up-to-date?

We are looking for your input and ideas! Please add your thoughts to the discourse post.

Thanks

Thank you to the participants, speakers, and especially Lynne Schreiber and Ouida Meier from the EarthCube office for all of their support and work (even with very last-minute requests!).

This work is part of the Jupyter meets the Earth project, supported by the NSF EarthCube program under awards 1928406, 1928374.

Jupyter meets the Earth

2019-09-09T17:48:00+00:00

By Lindsey Heagy and Fernando Pérez

We are thrilled to announce that the NSF is funding our EarthCube proposal “Jupyter meets the Earth: Enabling discovery in geoscience through interactive computing at scale” (pdf). The team working on this project consists of Fernando Pérez [1, 2, 3], Joe Hamman [4], Laurel Larsen [5], Kevin Paul [6], Lindsey Heagy [1], Chris Holdgraf [1, 2] and Yuvi Panda [7]. Our project team includes members from the Jupyter and Pangeo communities, with representation across the geosciences including climate modeling, water resource applications, and geophysics. Three active research projects, one in each domain, will motivate developments in the Jupyter and Pangeo ecosystems. Each of these research applications demonstrates aspects of a research workflow which requires scalable, interactive computational tools.

In this project we intend to follow the patterns that have made Jupyter an effective and successful platform: we will drive the development of computational machinery by concrete use cases from our own experience and research needs, and then find the appropriate points for extension, abstraction, and generalization. We are motivated to advancing research of contemporary importance in geoscience, and are equally committed to producing work that leads to broad impact, general use infrastructure that benefits scientists, educators, industry, and the general community.

The adoption of open languages such as Python and the coalescence of communities of practice around open-source tools, is visible in nearly every domain of science. This is a fundamental shift in how science is conducted and shared. In recent years, there have been several high-profile examples in which open tools from the Python and Jupyter ecosystems played an integral role in the research, from data analysis to the dissemination of results. These include the first image of a black hole from the Event Horizon Telescope team and the detection of gravitational waves by the LIGO collaboration. The utility of open-source software in projects like these and the success of open communities such as Pangeo, provide evidence of the force-multiplying impact of investing in an ecosystem of open, community-driven tools. Through this project, we will advance this open paradigm in geoscience research, while strengthening and improving the infrastructure that supports it. We made this argument when discussing the intended impacts of our proposal, and we are pleased that the NSF is investing in this vision.

Geoscience use cases

Given our project’s aims and approach, participating actively in domain research is crucial to our success. The following descriptions are meant to offer a flavour of the research questions we are tackling, each led by a geoscientist in the team.

CMIP6 climate data analysis (Hamman). The World Climate Research Program’s Coupled Model Intercomparison Project is now in its sixth phase and is expected to provide the most comprehensive and robust projections of future climate predictions. When complete, the archive is expected to exceed 18 PB in size. In the coming years, this collection of climate model experiments will form the basis for fundamental research, climate adaptation studies, and policy initiatives. While the CMIP6 dataset is likely to hold new answers to many pressing climate questions, the sheer volume of data is likely to present significant challenges to researchers. Indeed, new tools for scalable data analysis, machine learning, and inference are required to make the most out of these data.

Large-Scale Hydrologic Modeling (Larsen). Streamflow forecasts are a valuable tool for flood mitigation and water management. Creating these forecasts requires that a variety of data types be brought together including model-generated streamflow estimates, sensor-based observations of water discharge, and hydrometeorological forcing factors, such as precipitation, temperature, relative humidity, and snow-water equivalent. The integration of simulated and observed data over disparate spatial and temporal scales creates new avenues for exploring data science techniques for effective water management.

Geophysical inversions (Heagy). Geophysical inversions construct models of the subsurface by combining simulations of the governing physics with optimization techniques. These models are critical tools for locating and managing natural resources, such as groundwater, or for assessing the risk from natural hazards, such as volcanoes. Today, we need models applicable to increasingly complex scenarios, such as the socially delicate task of developing groundwater management policies in water-limited regions. This will require the development of new techniques for combining multiple geophysical data sets in a joint inversion, as well as the use of statistical and data science methods for including geologic and hydrologic data in the construction of 3D models.

These scientific problems exhibit, each with its own flavour, similar technical challenges with respect to handling large volumes of data, performing expensive computations, and doing both of these as a part of the interactive, exploratory workflow that is necessary for scientific discovery.

Jupyter & Pangeo: empowering scientists

Jupyter and Pangeo are both open communities that share the goal of developing tools and practices for interactive scientific workflows; these tools aim to deliver practical value to scientists who, in the course of everyday research, face a combination of big data and large-scale computing needs.

Project Jupyter creates open-source tools and standards for interactive computing. These span the spectrum from low-level protocols for running code interactively up to the web-based JupyterLab interface that a researcher uses. Jupyter is agnostic of programming language: over 130 different Jupyter kernels exist, and they provide support for most programming languages in widespread use today. Jupyter can be run on a laptop, in an HPC center, or in the cloud. Shared-infrastructure deployments (e.g. HPC / cloud) are enabled by JupyterHub, a component in the Jupyter toolbox that supports the deployment and management of Jupyter sessions for multiple users. The development process of tools in the Jupyter ecosystem is community-oriented and includes a diverse set of stakeholders across research, education, and industry. The project has a strong tradition of building tools that are first designed to solve specific problems, and then generalized to other users and applications.

A Pangeo Platform is a modular composition of open, community-driven projects, tailored to the scientific needs of a specific scientific domain. In its simplest form, it is based on the following generic components: a browser-based user interface (Jupyter), a data model and analytics toolkit (Xarray), a parallel job distribution system (Dask), a resource management system (either Kubernetes or a job queuing system such as PBS), and a storage system (either cloud object store or traditional HPC file system). These are complemented by problem- and domain-specific libraries.

This modular design allows for individual components to be readily exchanged and the system to be applied in new use cases. Pangeo was created by, and for, geoscientists faced with large-scale data and computation challenges, but such problems are now common in science. Researchers in a variety of disciplines including neuroscience and astrophysics are working to adapt the Pangeo design pattern for their communities. Beyond the initial Pangeo deployments supported by the NSF EarthCube grant for Pangeo, the platform has been adopted internationally, including by the UK Met office. It has also supported new research and education efforts from other federal agencies, such as NASA’s HackWeek focused on the analysis of ICESat-2 data.

User-Centered Development

Pushing the boundaries of any toolset unveils areas for improvement and opportunities for developments that streamline and upgrade the user experience. We aim to take a holistic view of the scientific discovery process, from initial data acquisition through computational analysis to the dissemination of findings. Our development efforts will make technological improvements within the Jupyter and Pangeo ecosystems in order to reduce pain-points along the discovery lifecycle and to advance the infrastructure that serves scientists. Following established patterns in Jupyter’s development, we take a user-first, needs-driven approach and then generalize these ideas to work across related fields. Broadly, there are 4 areas along the research lifecycle where we will invest development efforts, which we discuss next.

Data discovery. An early step in the research process is locating and acquiring data of interest. Data catalogs provide a way to expose datasets to the community in a way that is structured. Within the geosciences, there are a number of emerging community standards for data catalogs (e.g. THREDDS, STAC). To streamline access to such data sets, we plan to develop JupyterLab extensions which provide a user-interface that exposes these catalogs to researchers. This work will build upon the JupyterLab Data Registry, which will provide a consistent set of standards for how data can be consumed and displayed by extensions in the Jupyter ecosystem, as well as Intake, a lightweight library for finding, loading, and sharing data which is already serving the Pangeo community. Our communities are already discussing potential avenues for integration between these tools.

Scientific discovery through interactive computing. The Jupyter Notebook has been adopted by many scientists because it supports an iterative, exploratory workflow combining code and narrative. Beyond code, text, and images, Jupyter supports the creation of Graphical User Interfaces (GUIs) with minimal programming effort on the part of the scientist. The Jupyter widgets framework lets scientists create a “Research GUI” that combines scientific code with interactive elements such as sliders, buttons and menus in just a single line of code, while still allowing for extensive customization and more complex interfaces when required. In this project, we will develop custom widgets tailored at the specific scientific needs of each of our driving use cases.

Beyond their utility in the exploratory phase of research, interactive interfaces, or “dashboards” provide a mechanism for delivering custom scientific displays to collaborators, stakeholders, and students for whom the details of the code may not be pertinent. Voilà is a project, led by the QuantStack team, that enables dashboards to be generated from Jupyter notebooks. We plan to develop interactive dashboards using Voilà for our geoscience use-cases, contribute generic improvements to the Voilà codebase, and provide a demonstration of how researchers can deploy dashboards to share their research.

Research GUIs to explore Maxwell’s equations in research and education. Photo credit: SEOGI KANG

Established tools and data visualization. Many widely-used tools, particularly for visualization (e.g. Ncview, Paraview), are desktop-based applications and therefore cannot easily be used in cloud or HPC workflows. In some cases, modern, open-source alternatives are available. But often for specialized tasks, modern tools may not yet have functionality equivalent to the desktop version. JupyterHub can readily serve non-Jupyter web-native software applications such as RStudio, Shiny applications, and Stencila to users; under this project we aim to extend JupyterHub to also be able to serve desktop-native applications.

Using and managing shared computational infrastructure. JupyterHub makes it possible to manage computing resources, user accounts, and provide access to computational environments online. Currently, JupyterHubs in the Pangeo project are deployed and maintained using the Zero to JupyterHub guide alongside the HubPloy library. Together, these libraries have simplified the initial setup and automated upgrades to the Hubs. There are still many improvements that can make JupyterHub more suitable for larger, more complex deployments, both in terms of managing users and efficiently allocating resources. Under this project, we plan to build tools which collect metrics such as CPU and memory usage and expose these to both users and administrators so they can make more efficient use of the Hub. For shared deployments, we will improve user management so that user-groups can be used to manage permissions and allocations, and so that usage can be tracked and appropriately billed to the relevant grants. Within the HubPloy library, we also plan to make improvements to streamline continuous deployments so that installation and upgrade processes are repeatable and reliable.

An opportunity for meaningful impact — join us!

The impacts of climate change and the need for data-driven management of resources are some of the most critical and complex challenges facing society today. In recent years, we have experienced severe droughts in California and are in the midst of water management crises in the Central Valley, while devastating wildfires have destroyed entire communities. Through this partnership between researchers and Jupyter developers, we hope to contribute to the advancement of science-based solutions to these challenges, both by contributing directly to the research and by improving the open ecosystem of tools available to researchers and the stakeholders impacted by these issues.

If developing open-source tech to advance research in geoscience and beyond excites you, then please get in touch! At UC Berkeley, we will be hiring in 2 positions: a dev-ops position focussed on JupyterHub and shared infrastructure deployments, and a JupyterLab-oriented role focussed on extensions, dashboards, and interactivity. There will also be a position opening up at NCAR for a software engineer targeting improvements to improving the user experience of Xarray and Dask workflows.

Even if you aren’t looking for a new job, there are other ways to get involved with both the Jupyter and Pangeo communities. We welcome new participants to the weekly Pangeo meetings (on Wednesdays alternating between 4p GMT and 8p GMT) and there are monthly Jupyter community calls, which are open and meant to be accessible to a wide audience. Outside of calls, general Jupyter conversations happen on the Jupyter discourse and Pangeo conversations are typically on the Pangeo GitHub.

In closing: a step toward sustainable open science

This project provides our team with $2 Million in funding over 3 years as a part of the NSF EarthCube program. It also represents the first time federal funding is being allocated for the development of core Jupyter infrastructure.

The open source ecosystem that Jupyter and Pangeo belong to has become part of the backbone that supports much of today’s computation in science, from astronomy and cosmology to microbiology and subatomic physics. Such broad usage represents a victory for this open and collaborative model of building scientific tools. Much of this success has come through the efforts of scientists and engineers who are committed to an open model of science, but who have had to work with little direct funding, minimal institutional support, and few viable career paths within science.

There is real strategic risk to continuing with the implicit assumption that scientific open-source tools can be developed and maintained “for free.” If open, community-driven tools are to sustainably grow into the computational backbone of science, we need to recognize this role and support those who create them as regular members of the scientific community (we recently talked about this in more detail in a talk at NSF headquarters). Projects like ours, where funding and resources are explicitly allocated toward this goal, are a step in the right direction. We hope that our experiences will contribute to ongoing conversations in the scientific community around these complex issues.

In the past, we have tried to maintain a close relationship between domain problems and software development in Jupyter. However, this has typically been done in an ad-hoc manner, either by “hiding” the software development under the cover of science or by having funding to Jupyter alone. This is the first project where we explicitly partner with a team of domain scientists to simultaneously drive forward domain research and the development of Jupyter infrastructure.

We are excited about this opportunity and hope to be able to demonstrate that investing in open tools can be a force-multiplier of resources. As always, our work will be done openly, transparently, and with constant community engagement. We look forward to your critiques, ideas, and contributions to make this effort as successful as possible.

Acknowledgments

Thanks to Joe Hamman, Chris Holdgraf, and Doug Oldenburg for constructive feedback and edits on this blog post.

Many thanks to Ryan Abernathey (Columbia), Paul Bedrosian (USGS), Sylvain Corlay (QuantStack), Rich Signell (USGS), and Rollin Thomas (NERSC), who provided us with letters of support for this project; we look forward to working with you all! We are also grateful to Shree Mishra, our NSF Program Director on this project, and to Dave Stuart for their support as we move forward with the project. This project is a part of the to the EarthCube program and we look forward to engaging with its working group.

Finally, the sustained growth of Jupyter to the large-scale project that it has become would not have happened without the generous support of the Alfred P. Sloan, the Gordon and Betty Moore, and the Helmsley Foundations, as well as the leadership of Josh Greenberg and Chris Mentzel respectively at Sloan and Moore.

This work is supported by the NSF EarthCube program under awards 1928406, 1928374

Affiliations

[1] UC Berkeley, Statistics Department

[2] UC Berkeley, Berkeley Institute for Data Science

[3] Lawrence Berkeley National Lab, Computational Research Division

[4] National Center for Atmospheric Research, Climate and Global Dynamics Laboratory

[5] UC Berkeley, Department of Geography

[6] National Center for Atmospheric Research, Computational Information Systems Laboratory

[7] UC Berkeley, Division of Data Sciences

Binder + Zenodo: A how-to guide

2019-06-21T13:46:00+00:00

Interactive and reproducible repositories powered by Zenodo and Binder.

When the Binder project was first launched, we imagined a world in which scientific scholarship and reproducibility could be carried out by the community using a fully-open stack of technology. We’re happy to say that this is now possible!

Zenodo DOIs now work with BinderHub

The BinderHub team recently added in support for building Binder links that point to Zenodo repositories. Zenodo is a general purpose open-access repository hosted by CERN that allows researchers to archive and apply a DOI to information that they put up on the web.

Zenodo has the ability to archive GitHub repositories, which means that you can archive the code, data, or reports that underly a scientific analysis and assign it a unique, citeable identifier. Now that BinderHub knows how to resolve a Zenodo identifier, you’ll be able to share Binder links that point to Zenodo and provide interactive access to your repository, letting readers reproduce results and interact with your analyses!

Here’s a quick primer for how to do this:

Step 1: Create a Zenodo account

First off you’ll need to create an account on Zenodo. You can
do so using a number of different log-in options.

Step 2: Create your Binder-ready repository on GitHub

Next, you should create your Binder-ready repository on GitHub. Binder uses pre-existing best practices in data science in order to infer and build the environment needed for your repository.

To make a repository Binder-ready, follow the instructions in the Binder docs.
Briefly, what you need to do is add the configuration files that define the environment needed to run your code. Once those files are in place, and you’ve added an analysis script (a Jupyter or R Notebook) that actually runs your code and displays the results, your repository is ready to build with Binder.

Step 3: Make sure your repository is ready to be published!

Once you create a DOI for your repository, it will be frozen in time — you won’t be able to easily update it. So double check that the repository builds properly with Binder and runs the way that you’d expect it to.

Make sure to launch a Binder from your repository and run the analyses you’d like others to run. If they produce the expected result from within a Binder session, then they’ll continue to do so for others (assuming you have pinned your versions and followed other best practices in reproducibility).

Step 4: Create a Zenodo DOI for your repository

Now that your repository is ready, you’ll connect Zenodo with GitHub to create a DOI for your repository. Remember that this will be unique to the current state of the repo — future changes to this repository won’t be reflected in the DOI.

We recommend following the GitHub Citable Code Guide which provides some best-practices for creating your Zenodo DOI for a GitHub repository. Click the image below to be taken to this (excellent) guide.

The GitHub Citable Code guide.

Once you’re done, you should have a Zenodo DOI badge
like the one below:

A Zenodo DOI badge

Step 5: Create a Binder link for your Zenodo DOI

Finally, use your Zenodo DOI to create a Binder link that allows others to interact with and replicate your results. You can create a Binder link for your Zenodo record by heading to https://mybinder.org and filling in the form:

Building a Binder link for your Zenodo archive with the Binder UI.

This will give you a link you can share with others as well as the Markdown and reStructured text snippets for creating a badge.

The link’s structure should look like this:

https://mybinder.org/v2/zenodo/<zenodo-DOI>

For example, if your Zenodo DOI is 10.5281/zenodo.3242074 (corresponding to this zenodo repository), the Binder link for it would be:

https://mybinder.org/v2/zenodo/10.5281/zenodo.3242074/

You can even pair this Binder link with your Zenodo DOI badge that
we showed above!

The “launch binder” button contains a Binder link for your Zenodo archive. Clicking will launch a Binder session.

And that’s it! You now have an archived version of your analysis with a unique identifier. This identifier can be used in conjunction with BinderHub to allow readers to interact with your code and results!

Last minute update 🎉

A few hours before publishing this post we merged a contribution from Tom Morrell who works at CalTech’s library that allows you to launch records from CalTech’s Data Repository as well! For example a notebook to analyze traffic to the archive itself: https://mybinder.org/v2/zenodo/10.22002/d1.1250. You should also check out a quick response to this post from Lorena Barba, who describes her group’s approach to creating reproducible, citable bundles for figures in papers, and how Binder+Zenodo DOIs might fit into that workflow!

What’s next?

We are close to closing the loop of fully reproducible computational environments for scientific publication. We’re excited to see journals begin to integrate these workflows with their own publishing pipelines. For example, the Neurolibre project is deploying their own BinderHub and using it alongside their reviewing and archiving process in order to provide more rich
interaction with submitted material.

Each type of repository needs a small amount of custom work to be integrated with Binder. We started with Zenodo because it is well known,
general purpose and integrated with GitHub already. If there is an archive you’d like to see integrated please do stop by this repository and open a new issue or contribute the code to do so directly.

If you’re working with a publisher and are interested in this
please reach out! The Binder community would love to work with you in deploying these open tools to make your published work more open and accessible. Whether it is big or small, we hope that these workflows can make an impact across the publishing landscape, and we’re looking forward to seeing what people do next!

elife sprint: Integrating Stencila and Binder

2018-12-10T10:15:00+00:00

This article reports on a project, integrating Stencila and Binder, which started at the eLife Innovation Sprint 2018 (#eLifeSprint). A longer version has been cross-posted on multiple blogs (eLife Labs, Stencila, o2r).

eLife, an open science journal published by the non-profit organisation eLife Sciences Publications from the UK, hosted the first eLife Innovation Sprint 2018 as part of their Innovation Initiative in Cambridge, UK: “[…] a two-day gathering of 62 researchers, designers, developers, technologists, science communicators and more, with the goal of developing prototypes of innovations that bring cutting-edge technology to open research communication.” One of the 13 projects at the excellently organised event was an integration of Binder and Stencila.

This article reports on the project’s results and changes made to Binder-related tools. Today, Binder has first class Stencila support. Read the full story at the eLife Labs blog post or try out to open Stencila documents from any online code repository on mybinder.org with the click of a single button:

Click the Binder badge to open a Stencila document on mybinder.org

This project is a collaboration between Min from the Simula Research Laboratory, Norway, a core developer of Binder and related tools, Daniel from the o2r project at the Institute for Geoinformatics, Germany, and Nokome, initiator and developer of Stencila. The final changes were made with the help of Tim. Thanks! The project was also part of the Mozilla Global Sprint 2018, see mozilla/global-sprint#317.

The building blocks and a challenge

Stencila Desktop is an office suite for reproducible research documents. It allows scientists to use languages like R and Python within familiar and intuitive word processor and spreadsheet user interfaces to lower the barriers to reproducible research for those with little or no software development skills. Binder (a part of Project Jupyter) makes it simple to generate reproducible computing environments from code repositories (e.g. GitHub or GitLab, see binder examples) and mybinder.org is the most prominent example. Binder uses repo2docker, for generating Dockerfiles (a human- and machine-readable recipe for setting up a computational environmentt used by the popular Docker container software) and building Docker images from software projects. While containers have become a commodity for developers, researchers still struggle to grasp and control the complexity of computational environments. This is where the two building blocks join: Running Stencila as part of a Binder helps researchers to communicate their work openly, to collaborate effectively with other scientists, and to ensure a high quality and transparency of their workflow and findings. Min and Daniel formulated their goal in the sprint project form: “[…] to connect them so that users can edit reproducible documents (DAR files) as part of a Binder project”.

Connecting Stencila and Jupyter: `nbstencilaproxy`

Stencila has “execution contexts”, an equivalent to Jupyter’s “kernels”. The contexts use code dependency analysis and return execution results as data values to enable a reactive, functional execution model. To open Stencila documents on binder, these execution contexts must be installed and configured in the environment created by repo2docker. This is achieved with a new software project initiated at the sprint: nbstencilaproxy - a Jupyter notebook server extension and proxy for Stencila.

The projects consists of a Python module with the Jupyter notebook server and “non-server” extensions of the same name, and a bundled JavaScript module (of the same name). The Python module allows proper versioned installation, dependency management, and installation from an established software repository. It takes care of the plumbing between the user interface and the services in the background, so that the binder is viewable over one port in the browser, while the many different background components run on their own ports. The “no server” extension adds a “Stencila session” menu entry and conveniently lives in the same directory structure as the server extension. The JavaScript module manages the required JavaScript dependencies and provides an well-defined structure for the code files. It serves the Dar document and provides access to the Stencila host.

repo2docker was extended with automatic detection of Dar documents, including the languages and execution contexts. As with most Binder repositories, a no configuration is needed for most common user cases: users can open a Dar document on Binder and trust the required environment to provide all required software. Daniel created a few example repositories to provide a starting point for users. The binder team generously welcomed the changes to mybinder.org and and the examples to the binder examples organisation on GitHub:

https://github.com/binder-examples/stencila-py contains Python code cells, using both the Jupyter and plain Python execution contexts:

Click the Binder badge to open a Stencila document with Python code on mybinder.org

https://github.com/binder-examples/stencila-r contains R code cells and two plots:

Click the Binder badge to open a Stencila document with R code on mybinder.org

One of the cool features of Stencila are the reactive cells, as demonstrated in a tweet following the feature release:

Thanks to @nordholmen working on @stencila support for https://t.co/Zlj6FrYgBw you now have reactive cells with Python code on @mybinderteam! Give it a go https://t.co/ToIuQPq0Fy

— Tim Head, @betatim on the internet (@betatim), November 12, 2018

Binder + Stencila is a demonstration of the power that the Open Source and Open Science community can foster. Many people are working together on the organisational and technological challenges of science today towards full research transparency and reproducibility, even if we use computers to an unprecedented level. Many small contributions on “side projects” such as these can make a difference, and connecting these two great projects hopefully helps to solve some problem in science down the road.

Join the public Stencila and binder chats to stay in touch or get help. We look forward to see scientists using nbstencilaproxy for communicating their work and new challenges that come with it.

#eLifeSprint-ers @minrk and @nordholmen are working to connect #JupyterNotebooks / #Binder with DAR / #Texture / @Stencila, so that users can edit reproducible documents as part of a Binder project https://t.co/2GoGNydsmX (@mybinderteam @ProjectJupyter @_substance)

— eLife Innovation (@eLifeInnovation), May 10, 2018

Binder 2.0, a Tech Guide

2017-11-30T16:01:00+00:00

Authors: The Binder project is comprised of many individuals within and outside of the core Jupyter team. A list of members that contributed to this post is at the end of this article.

Note: this post focuses more on technical changes in the Binder 2.0 reboot. For a post about user-facing features and future plans, see this eLife blog post.

We are undergoing a dramatic increase in the complexity of techniques for analyzing data, doing scientific research, and sharing our work with others. In early 2016, the Binder project was announced, attempting to connect these three components. A blogpost in eLife described a vision where scientists could specify dependencies along with a collection of Jupyter notebooks. Binder builds a Docker image from these dependencies, and provides a URL where any user in the world can instantly recreate this environment.

Want to see it in action? Click the button below.

With this post we are proud to announce the next version of Binder. It aims to be more modular, more flexible, more stable, faster, and more extensible than its predecessor. Powering this version of Binder is a collection of tools in the Jupyter ecosystem. Since being released, the Binder project has learned many things about implementing fast, flexible online deployments. In addition, its vision has expanded to include not only Jupyter notebooks, but many other computational workflows. You can access an open beta version of this deployment here:

mybinder.org

You can find a list of sample repositories to learn how to create “Binder”-ready repositories here:

github.com/binder-examples

You can also see what the Binder community has been up to in creating their own repositories by checking the GitHub Binder topic:

https://github.com/topics/binder

Give it a shot, build some repositories, and importantly, tell us what could be improved on our GitHub repo. Below we’ll describe a bit about what’s new.

The new Binder UI. Users input a URL to a git repository (or specify a specific branch/tag/commit). Upon clicking “launch”, you will be directed to a live environment where you can interact with the code.

What’s new?

First off we’ll describe how the experience will change for users. The short answer is: not much. The goal of Binder is still enabling you to intantly create interactive and shareable repositories. We’ve completely rebuilt the backend of Binder, but we’ve made minimal changes to the front-end user experience.

The biggest difference you should notice is that Binder is both faster and more stable. You’ll still be able to generate Binder links from a git URL from a single web-page. However, there are a few key differences:

New Default Environment

Old versions of Binder were based off of a Docker image that contained a fairly heavy computational environment. The new Binder deployment makes minimal assumptions about what environment you want installed, by default the only thing that will be installed is the Jupyter Notebook and Python 3. This means you’ll need to be more expressive in the dependencies you include in your dependency files. For example, if you want numpy or matplotlib, you should specify them in a requirements.txt or environment.yml file. Since you’re explicitly listing your requirements it makes reproducing your work more reliable, and allows the Binder infrastructure to change more freely without breaking your repository code.

New URL structure

The new URL structure for Binder follows the following convention:

https://mybinder.org/v2/gh/<org-name>/<repo-name>/<branch|tag|hash-name>?filepath=<path-to-file>

For example, below is the URL for a basic Binder-ready Python 3 repository, it includes basic information about the repository:

https://mybinder.org/v2/gh/binder-examples/requirements/master

You can also specify parameters that do things like point users to a particular file or initialize a user-interface. For example, the following URL starts JupyterLab once users click the link:

https://mybinder.org/v2/gh/binder-examples/jupyterlab/master?urlpath=lab

In each case, note the gh at the very beginning — this specifies that the git URL exists on github.com. It is possible to build new URL parsers for other online repositories such as BitBucket, osf.io, or any other provider. We are currently focusing on Git and GitHub, but nothing prevents Binder from being compatible with other kinds of content providers. Shortly before this blog post we added support for arbitrary git URLs to binderhub, so watch this space.

Specify a specific branch / tag / commit

Also notice that in the URL above you can specify a branch, tag, or commit hash for the Binder image. This allows you to ensure that a Binder image will always remain the same (if you specify package versions properly). This is a crucial step for reproducibility and maintaining consistency in how users experience the files in your repository.

Notice that in the URL above you can specify a commit hash or git tag for the Binder image. This hash is unique to the state of the code at the moment that the commit was made, ensuring that Binder can rebuild the exact same environment any time. Note that if authors don’t want to guarantee the same reproducible environment, they can specify a branch and BinderHub will resolve it to the latest commit hash before building the environment.

Binder auto-building

When a git repository is launched, Binder will now check whether an image has already been built for that repository at the same commit hash. If it has, then Binder will skip the building process and take you straight to a JupyterHub instance that serves this image.

If the image hasn’t been built, then it will automatically be generated before sending the user to JupyterHub. The only difference will be the amount of time it takes before entering the JupyterHub environment. This means that authors no longer need to explicitly build their Binder images when they update a branch. The next time someone clicks a Binder link, it will happen automatically. If you don’t want this behavior, be sure to point Binder to a specific tag or commit hash, rather than a branch name or tag. For example, here’s a Binder URL that will always point to the same commit hash:

https://mybinder.org/v2/gh/wildtreetech/explore-open-data/binder20-elife

While this URL points to a branch, and will thus be re-built each time a new commit is made to that branch:

https://mybinder.org/v2/gh/wildtreetech/explore-open-data/master

More options for dependency files

Users often want to specify a computational environment that is more complex than a simple list of Python requirements. While this is possible by specifying a Dockerfile, it’s often an overly-complicated solution to this problem. Binder now uses repo2docker to build a Docker image from your repository. This makes it possible to specify a more complex environment with text files. For example, you can use an apt.txt file to install packages with apt-get, or use a file called postBuild to define shell commands that are run before generating the Docker image (e.g. for downloading some data or running scripts). See the repo2docker documentation for a list of files that are supported with Binder.

For a selection of examples that show off how to specify dependencies take a look at the example gallery: https://github.com/binder-examples

More user interfaces

The JupyterLab interface running on Binder. You can access the JupyterLab demo repository at ``mybinder.org/v2/gh/binder-examples/jupyterlab/master?urlpath=lab

The previous iteration of Binder only supported the classic Jupyter Notebook user interface, while the new deployment will additionally support RStudio and JupyterLab. Because of the extra build configuration files specified above, you can also utilize more tools in the Jupyter widgets ecosystem, such as the RISE plugin for interactive presentations or the appmode plugin to generate interactive apps from your repository. We also welcome contributions to add support for other user interfaces.

More online repository providers

While GitHub is a fantastic repository of open-source code, it’s not the only repository. The new Binder deployment makes it straightforward to adding support for new sources of code (for example, GitLab, the OSF, or even non-git codebases). Currently GitHub is the only supported source for code, but we welcome contributions enabling support for new sources.

We’re excited about this next step in Binder’s development, and hopeful that we can build a community around this powerful set of tools. Don’t hesitate to open an issue or pull request on our GitHub repository, or to reach out via our Gitter channel. We look forward to seeing what comes next, and to continue enabling reproducible and open workflows in data science and research.

For developers

The next few sections are meant for developers interested in deploying their own Binder, or for those interested in the technical details behind the new deployment.

Tech components

The three main technical components behind the new Binder backend are:

BinderHub, currently on display at mybinder.org and contained in the binderhub repository.
repo2docker, a tool that converts a code repository into a Docker image with an environment specified via dependency files (e.g., requirements.txt).
JupyterHub, which hosts user instances with a server in the cloud. We use a distribution of JupyterHub that runs on top of Kubernetes.

For more information on the infrastructure behind Binder, see the documentation.

A prototype of RStudio running in a Binder. This is currently support with a Dockerfile, and we are working on supporting R build files natively. You can access this repository at: <https://mybinder.org/v2/gh/binder-examples/dockerfile-rstudio/master>

Kubernetes

Binder now also heavily relies on Kubernetes for scaling our image building service and the JupyterHub. Kubernetes is massively scalable and has a strong community of developers behind it. Moreover, Kubernetes is cloud-agnostic. It can be run on Google Cloud, Microsoft Azure, and AWS among others, as well as on your own bare metal hardware if needed. Because BinderHub is built to run on top of Kubernetes, you can deploy Binder off of any of these resources as well (see below).

Deploying your own Binder server

While mybinder.org will continue to exist as a public service, we hope to see new Binder deployments for many different use cases in the wild. One of our primary goals is to make it easier for users to deploy their own Binder servers. This is relatively straightforward by following the instructions on the BinderHub documentation, which are currently under active development to make ongoing improvements as the Kubernetes technology evolves. We’re continuously updating these steps to make them as clear as possible, so please don’t hesitate to open an issue or a pull request on our github repository and make suggestions.

We would love to see others deploy their own BinderHub servers, either for their own communities, or as part of a federated public service of BinderHubs.

Future development

This is the just the beginning of new features and improvements to Binder. We’re working hard to grow an open-source community around these tools, and we encourage issues, comments, and PRs on the BinderHub, repo2docker, and JupyterHub repositories. We look forward to growing the Binder ecosystem, and we’re excited to see all of the Binders that people design.

Acknowledgements (alphabetical order)

C. Titus Brown (UC Davis), Matthias Bussonnier (UC Berkeley), Jessica Forde (UC Berkeley), Brian Granger (Cal Poly), Tim Head (Wild Tree Tech), Chris Holdgraf (UC Berkeley), Andrew Osheroff (Google), Naomi Penfold (eLife Sciences), M Pacer (UC Berkeley), Yuvi Panda (UC Berkeley), Fernando Perez (UC Berkeley), Min Ragan-Kelley (Simula Research Laboratory), and Carol Willing (Cal Poly). The Binder project is currently being funded from a grant from the Moore Foundation.