Jupyter Blog - science

JupyterGIS 0.16: New visualization capabilities, collaborative Story Maps, and more

2026-08-24T16:09:00+00:00

Read this article in Notebook.link, as a live story-map! https://notebook.link/@martinRenou/jupytergis-announcement

Earlier this year, we introduced STAC browsing and Story Maps in JupyterGIS, making it easier to discover geospatial datasets and communicate results without leaving Jupyter.

JupyterGIS 0.16 continues in the same direction. This release adds support for new geospatial formats, tighter integration with the scientific Python ecosystem, a redesigned Story Map editor, and a more expressive way to style geographic data.

Story Maps are getting a new look!

Story Maps in JupyterGIS let you build a scrollable presentation around your map. A Story Map is made up of a sequence of segments that can combine Markdown content with map views, so you can guide the reader through a geographic story as they scroll.

Each segment can define its own map state, including the current map location, visible layers, and layer styling. This means that the map can change as the reader moves through the story: layers can appear or disappear, the view can move to a new location, and symbology can change to highlight different aspects of the data.

Story Maps have received a significant update in this release.

Read this article in Notebook.link, as a live story-map! https://notebook.link/@martinRenou/jupytergis-announcement

The Story-map associated to this release annoucement. It contains Text, images, and map views with associated layer states.

A better Story Maps editing experience

We’ve also made substantial improvements to the Story Map editing experience.

The editor has been redesigned around Jupyter’s real-time collaboration infrastructure, allowing multiple people to edit the same Story Map simultaneously. Changes appear immediately for everyone, making it much easier to prepare presentations, reports, or educational material as a team.

Collaboratively edit the Story Map markdown.

The editor now gives you a much better sense of what the final story will look like while you are working on it. Markdown sections can be previewed directly in the editor, and a new Story Map preview makes it possible to see the complete presentation without leaving the editing workflow. This makes it easier to write, arrange, and refine a story while keeping an eye on the final result. We’ve also introduced a new layout that is better suited for long-form content. In addition to guided geographic narratives, Story Maps can now be used to create richer articles combining text, maps, images, and other interactive content.

New Story Map editor: Set story segment viewport, preview markdown, set layers properties for the story segment.

OpenEO layers

More and more geospatial workflows rely on remote processing instead of downloading datasets locally. openEO provides a common API to describe these processing pipelines as process graphs that are executed by a backend.

JupyterGIS can now display openEO process graphs directly as map layers. Instead of exporting intermediate results before visualizing them, you can connect an openEO backend and inspect the output of your processing pipeline directly in the map.

The visualization is tile-based and lazy: JupyterGIS only requests the data needed for the current map view and zoom level. This makes it possible to explore large remote sensing workflows interactively, without materializing the full result locally.

JupyterGIS can make use of any openEO server that supports tiling, such as titiler-openeo

import openeo
from jupytergis import GISDocument
from openeo.processes import process

connection = openeo.connect(SERVER_URL)
connection.authenticate_basic(username=BASIC_AUTH, password=BASIC_AUTH)

cube = connection.load_collection(
    "sentinel-2-global-mosaics",
    bands=["B03", "B08"],
    temporal_extent=["2022-04-15", "2022-12-31"],
)

cube = cube.reduce_dimension(
    dimension="t",
    reducer="first",
)

cube = cube / 10000.0

# NDWI = (GREEN - NIR) / (GREEN + NIR)
ndwi = cube.ndvi(nir="0", red="1")

ndwi_vis = (ndwi + 1) / 2

ndwi_png = ndwi_vis.linear_scale_range(
    input_min=0,
    input_max=1,
    output_min=0,
    output_max=255,
)

result = ndwi_png.save_result(format="PNG")

doc = GISDocument(latitude=40.75, longitude=-73.95, zoom=12)
await doc.ready()
doc.add_openeo_tile_layer(result)
doc

Define a process graph to compute a NDWI, using the Python API of OpenEO and JupyterGIS. It is then lazily evaluated on a per-tile basis while the user pans/zooms on the map.

In addition to defining openEO process graphs from the scripting Python API, JupyterGIS provides an advanced openEO process graph editor, allowing you to:

connect to an openEO tile server
define the graph graphically, with boxes and arrows
load data collections and define processes with a drag-and-drop UI
directly edit the JSON content

Editing an openEO process graph from the JupyterGIS front-end

Another interesting aspect of openEO is that process graphs have a well-defined, declarative JSON representation. Because of this structured format, they are a natural target for LLM-assisted workflows. Users can describe the analysis they want in natural language, have an LLM generate or refine the corresponding process graph (e.g. using jupyterlite-ai), and immediately visualize the result in JupyterGIS. Combined with the lazy, tile-based rendering, this makes it possible to quickly iterate on processing pipelines without waiting for complete datasets to be exported or downloaded.

Lazy visualization of Xarray datasets with jupyter-tiler

JupyterGIS now integrates with the new jupyter-tiler package, making it straightforward to visualize Xarray datasets from Python.

Datasets can come from anywhere: they may already exist in your notebook, or they can be loaded on demand from a STAC catalog using stackstac. Once you have an Xarray object, JupyterGIS can display it in the map without requiring an export to another format.

Rendering happens lazily, generating only the tiles needed for the current view. This makes it possible to explore datasets that are much larger than memory while keeping navigation responsive.

The result is a smoother workflow from data loading, to analysis, to visualization, all within the same notebook.

await doc.add_data_array_layer(
    name="NDSI Layer",
    data_array=ndsi,
    colormap_name="viridis",
    colormap_range=(-1, 1),
)

Visualizing an Xarray dataset in JupyterGIS.

This feature requires the optional dependency jupyter-tiler to be installed.

pip install jupyter-tiler

A more expressive symbology model

Styling geographic data often requires combining multiple visual properties to communicate patterns effectively.

JupyterGIS 0.16 introduces a new symbology model inspired by the Grammar of Graphics. Instead of relying on a fixed set of styling options, visual properties such as color, size, and opacity can be defined in a more flexible and composable way.

This makes it easier to build everything from simple thematic maps to more advanced visualizations while keeping styling definitions consistent and reproducible.

A symbology example: apply a Viridis color map to the circle colors, a linear scale to the radius of circles, and a fixed stroke color.

GeoZarr and GeoPackage support

This release also expands the range of formats that JupyterGIS can open directly.

Support for GeoZarr makes it possible to work with cloud-native multidimensional geospatial datasets, while GeoPackage support improves interoperability with existing GIS software and common data exchange workflows.

New Collaborative Editing Capabilities

JupyterGIS 0.16 also brings collaborative editing to vector layers. When working on a shared JupyterGIS document, multiple users can now edit the same vector data at the same time.

Features can be created, moved, and edited collaboratively, with changes synchronized in real time between users. This makes it possible to work together on tasks such as digitizing features, annotating a map, or refining a dataset without having to exchange files or manually merge changes.

Combined with the collaborative Story Map editor, this makes collaboration a more integral part of JupyterGIS: users can work together on the data itself, and then use the same shared document to explore and communicate their results.

A new R API

JupyterGIS 0.16 also introduces an R client, bringing JupyterGIS to R users through the new r-jupytergis package. The R client provides bindings for interacting with JupyterGIS widgets from an R notebook, using the same JavaScript front-end as the Python client.

The main interface is the GISDocument widget, which can be used to create and manipulate JupyterGIS documents directly from R. This makes it possible to build geospatial workflows in R while using the same interactive map interface available to Python users.

The R client also uses the same underlying collaborative infrastructure as the Python client, including the Yrs CRDT library. This means that R users can participate in the same collaborative JupyterGIS workflows rather than working in a separate environment.

doc <- GISDocument$new("france_hiking.jGIS")
layer <- doc$add_raster_layer(
    url = "https://mt1.google.com/vt/lyrs=y&x={x}&y={y}&z={z}",
    name = "Google Satellite",
    attribution = "Google",
    opacity = 0.6
)

Bug fixes and performance improvements

As usual, this release also includes many smaller improvements throughout the project.

We’ve fixed a number of bugs, improved performance in several parts of the application, and continued polishing both the user interface and the Python API.

JupyterGIS continues to evolve as a collaborative GIS environment that fits naturally within the Jupyter ecosystem. Whether your workflow starts from a notebook, a STAC catalog, an openEO backend, or a local dataset, the goal remains the same: make it easier to move between analysis, visualization, and communication without switching tools.

Acknowledgements

This work on JupyterGIS by QuantStack was funded by the European Space Agency (ESA) for the Story-maps development, the R API, openEO layers support and the collaborative labelling. Additionally, QuantStack was funded by the French National Centre for Space Studies (CNES) for the lazy visualization of xarray datasets.

Contributors to this release

By order of number of contributions:

Arjun Verma is a Developer at QuantStack. He worked on the server-side geoprocessing infrastructure and on the openEO editor in the JupyterGIS front-end.

Martin Renou is a Technical Director at QuantStack and a maintainer of JupyterGIS. For this release, Martin coordinated and guided much of the development, and worked on the integration of openEO layers.

Gregory Mooney is a Developer at QuantStack and a long-time contributor to JupyterGIS. He led much of the work on the new Story Map editor and its collaborative editing capabilities.

Matthias Meschede is Chief Operating Officer at QuantStack. He introduced the new Grammar of Graphics-inspired symbology model, bringing a more expressive and composable approach to styling geographic data.

Matt Fisher is the Community Manager of GeoJupyter. He contributed to many of the discussions around the release and helped shape several of the design decisions across the project.

Nakul Verma is an open-source contributor to JupyterGIS. He contributed numerous bug fixes and improvements throughout the release, and introduced support for Vega expressions in the new symbology system.

Antoine Prouvost is a Developer at QuantStack. He led the initial work on the R API for JupyterGIS, building its first skeleton and establishing the foundations for the r-jupytergis package.

Sandrine Pataut is a Developer at QuantStack. She worked on bringing many of JupyterGIS’s features to the R API, helping make the new client more complete and useful for R users.

We are grateful to everyone who contributed code, reviews, ideas, discussions, and feedback to this release. JupyterGIS continues to benefit from an increasingly diverse community of contributors, and we look forward to seeing what comes next!

Exploring Petabytes of the Night Sky — Jupyter Notebooks at NOIRLab’s Astro Data Lab Science Platform

2026-04-20T18:04:00+00:00

By Robert Nikutta & Stéphanie Juneau (NSF NOIRLab)

Imagine querying 420+ billion rows of astronomical catalog data — spanning 30 major sky surveys, observed over decades with telescopes on three continents — from a Jupyter notebook in your browser in seconds. No download. No HPC allocation request. No waiting.

That is what 4,800+ astronomers in over 90 countries can do every day at the Astro Data Lab science platform. Data Lab is operated by NSF NOIRLab, the National Optical-Infrared Astronomy Research Laboratory, headquartered in Tucson, Arizona, with observatories in Arizona, Hawai’i, and Chile. Since its public launch in June 2017, Astro Data Lab has quietly become one of the largest deployments of Jupyter notebooks in professional science — and a case study in what happens when you bring the compute to the data instead of the other way around.

World map of 2024 number of data queries at Astro Data Lab by country (log-scale color): 64.2 million queries from 72 countries that year.

The Data Problem Astronomy Had to Solve

Modern sky surveys are data machines. The Dark Energy Survey cataloged 690 million objects. Gaia measured positions and motions for 1.8 billion stars. The DESI Legacy Surveys cover 20,000 square degrees, nearly half of the full sky, in three optical bands. And the upcoming Rubin Observatory’s Legacy Survey of Space and Time (LSST) will generate roughly 10 million transient alerts per night starting later this year.

Traditional astronomy workflows begin with downloading relevant data to a local computer and to use locally installed specialized software tools to process and analyze the data. However, downloading these catalogs to a local machine is now often physically impossible. A single survey’s measurements table can exceed the combined disk space of an entire research group. And even if you could download it, the computing resources needed to query it efficiently at scale requires infrastructure most astronomers don’t have.

The answer the community converged on, like many industries dealing with big data: bring the compute to the data. Host the catalogs in databases, co-locate a computing environment next door, and give scientists a familiar interface to work in. That interface, increasingly, is a Jupyter notebook.

Astro Data Lab: Jupyter at the Observatory

Astro Data Lab was conceived in 2014 and went public in June 2017, originally built to support data releases from the Dark Energy Survey — a few terabytes of catalogs and tens of terabytes of images. We imagined a ceiling of roughly 500 users. We were wrong, in the best way.

Today the platform hosts:

About 420 billion catalog rows across 30+ major astronomical surveys (DES, Legacy Surveys, DESI, NOIRLab Source Catalog, SDSS, Gaia, unWISE, SMASH, S-PLUS, VHS, 2MASS, and dozens more)
31 million spectra via SPARCL, our spectral access service (DESI DR1+EDR, SDSS/BOSS DR17)
Petabytes of images, accessible through a Simple Image Access service and cutout API
Over 4,800 registered users from over 90 countries, who submit tens of millions of data queries each year

Every registered user gets a persistent JupyterHub environment with the full astronomy Python stack pre-loaded — Astropy, NumPy, SciPy, Matplotlib, Pandas, Scikit-learn — and our own astro-datalab client library. The library provides core services, for instance auth and DB queries:

from dl import authClient, queryClient
from getpass import getpass

# Log in
token = authClient.login(input("username: "),getpass("password: "))

# Query 10 objects from the NOIRLab Source Catalog near a sky position
# Right Ascension (RA) = 150.12 degrees
# Declination (Dec) = 2.21 degrees
# Search radius = 0.05 degrees
# q3c (Quad Tree Cube) is a spatial indexing scheme for Postgres
# gmag and rmag are the g-band and r-band magnitudes of objects
# in the NOIRLab Source Catalog Data Release 2, ‘object’ table.
result = queryClient.query(
    """SELECT ra, dec, gmag, rmag FROM nsc_dr2.object
       WHERE q3c_radial_query(ra, dec, 150.12, 2.21, 0.05) LIMIT 10""",
    fmt="pandas"
)

It’s as simple as that. The query runs on the database server next to the data; only the result set crosses the network.

Use Case 1 — Seeing the Sky Inside a Notebook with AladinLite

One of the most immediate joys of working with astronomical data is visualization: not just numbers in a table, but where things are in the sky, what the images look like, and how your query results relate to the underlying survey footprint.

We’ve integrated AladinLite v3 — the interactive sky atlas from Centre de Données Astronomiques de Strasbourg (CDS) — directly into the notebook environment via the ipyaladin widget. With a handful of lines, astronomers can embed a fully interactive sky viewer in a notebook cell or next to their notebook in a “sidecar”, and overlay their own data on top of real survey imagery:

import time
from astropy import units as u
from astropy.table import Table
from astropy.coordinates import SkyCoord
from ipyaladin import Aladin
from sidecar import Sidecar

# Instantiate the Aladin interactive sky viewer 
aladin = Aladin(full_screen=True)
with Sidecar(title="aladin_output",anchor='split-right'):
    display(aladin)

# globular cluster NGC 1851 (RA, Dec)
aladin.target = SkyCoord(78.52809*u.deg, -40.04656*u.deg)
aladin.coo_frame = "ICRSd"  # set coordinate frame to ICRS, angles in deg
time.sleep(1)  # race condition
aladin.fov = 0.4  # set field of view to 0.4 degrees

# Overlay catalog query results as circle markers
t = Table.from_pandas(df)  # e.g., from a previous query around NGC 1851
aladin.add_table(t,shape='circle',source_size=15,color='green')

The result is a pannable, zoomable sky viewer — right in the notebook — with your query results overlaid as green circles on the actual sky image of a globular cluster (see figure below). Users can overlay MOCs (Multi-Order Coverage maps, which encode survey footprints), user-generated catalogs from a prior query, or any Virtual Observatory-standard data source.

This capability turns what was once a static plot into an exploratory tool: zoom into a cluster, click on a source, cross-match on the fly. For students and scientists unfamiliar with a dataset, it is often the fastest path from “I have a list of objects” to “I understand where they are and what I’m looking at.”

AladinLite integration is now active in our notebook library, with full deployment into the new Data Lab Web Portal on the roadmap for later this year.

AladinLite v3 sky viewer inside a Jupyter notebook, showing a globular cluster with catalog query results in the outskirts of the cluster overlaid as green circles.

Use Case 2 — Stacking Galaxy Spectra with SPARCL

Spectroscopy — measuring how much light a star or a galaxy emits at each wavelength — is one of astronomy’s most powerful tools. But individual spectra are often noisy. The signal-to-noise ratio of a single optical spectrum for a faint galaxy can be too low to measure the emission lines that encode star formation rate, gas chemical content (Oxygen, Nitrogen, etc.) or the even more subtle absorption lines that create small wiggles in the shape of the spectrum, yet encapsulate crucial information such as the mass and age of the stars making up a galaxy.

One trick that astronomers have used for decades: combining or “stacking” spectra. Average hundreds of spectra together, and the noise level reduces while the signal builds up. What was invisible in a single spectrum becomes unmistakable in the stack. While the concept is simple, reading and manipulating large numbers of spectra can be time consuming or cumbersome.

SPARCL (SPectra Analysis and Retrievable Catalog Lab) makes this possible at scale directly in a notebook. With sparclclient, users can currently search 31 million spectra by redshift range, target type, and survey, then retrieve flux arrays and wavelength grids ready for stacking:

from sparcl.client import SparclClient

# Instantiate the SPARCL client (connected to production server)
client = SparclClient()

# Find SDSS spectra of galaxies in a redshift slice 0.1<z<0.3
found = client.find(
    outfields=['sparcl_id', 'ra', 'dec', 'redshift', 'spectype'],
    constraints={'spectype': ['GALAXY'],
                 'redshift': [0.1, 0.3],
                 'data_release': ['SDSS-DR17']},
)

# Retrieve flux, wavelength, and inverse-variance arrays
retrieved = client.retrieve(found.ids,
            include=['flux', 'wavelength', 'ivar'])

In our SpectralStacking_SDSS science example notebook, users first stack a small number of galaxy spectra (N=5) in eight bins of astrophysical color g−r (green and red filters), revealing trends from blue spectra with emission lines to red spectra with absorption lines but with noisy spectra. Then users stack hundreds of galaxy spectra for the same bins of color g−r and obtain much cleaner spectra where the small wiggles are now real astrophysical features and no longer buried in the noise.

The spectral rainbows below — N=5 then N=200 stacked galaxy spectra color-coded in bins of astrophysical color g−r — are each a single output cell from this notebook, generated entirely within the Data Lab environment.

SPARCL spectral rainbows: top panel shows N=5 galaxy spectra stacked and color-coded by g−r color, spanning wavelengths 3750–6450 Ångstrom. Bottom panel shows the same exercise but with N=200 galaxy spectra per bin, greatly enhancing the signal-to-noise ratio.

Use Case 3 — Variable Stars and the Coming Flood of Time-Domain Data

Not all astronomical data is a static snapshot of the sky. Many of the most scientifically rich phenomena — pulsating stars, transiting exoplanets, exploding supernovae, gravitational lensing events — reveal themselves through change over time.

Among the most useful calibration tools in astrophysics are RR Lyrae stars: old, low-mass stars that pulsate with periods of 0.2–1 day and a brightness variation that traces their distance. Finding and characterizing them across millions of square degrees of sky requires querying multi-epoch photometry catalogs, computing period statistics, and folding light curves — all tasks that fit naturally in a notebook workflow.

Our TimeSeriesAnalysisRrLyraeStar notebook demonstrates the full pipeline: query the SMASH DR2 catalog for stars with high photometric variability, run a Lomb-Scargle periodogram on the light curve, identify the dominant period, and phase-fold the observations to reveal the characteristic sawtooth pulsation profile:

from astropy.timeseries import LombScargle
import numpy as np

ls = LombScargle(t, y)  # time and magnitude from a previous query
frequency, power = ls.autopower()
period = 1./frequency # period is the inverse of frequency
best_period = period[np.argmax(power)]
phase = (t / best_period) % 1  # folded timeseries = light curve

The resulting phase-folded light curve shown in the figure below is clean, precise, and immediately recognizable to any variable-star astronomer — produced entirely from archival survey data without a single new observation.

This kind of workflow is also a proving ground for the upcoming Vera C. Rubin Observatory’s Legacy Survey of Space and Time (LSST). When Rubin begins operations and delivers 10 million nightly alerts, the only workflows that will scale are ones already designed to run against large databases or specialized file systems, in shared computing environments, with notebook-native tooling. Astro Data Lab users are building those workflows today.

Phase-folded RR Lyrae light curve from the TimeSeriesAnalysisRrLyraeStar notebook, showing characteristic sawtooth pulsation with a 0.65 day period.

The Notebook Ecosystem

The three use cases above are drawn from our library of 80+ open-source Jupyter notebooks at github.com/astro-datalab/notebooks-latest. The library is organized into six sections:

Directory                      Contents
---------------------------------------------------------------------------
01_GettingStartedWithDataLab/  Authentication, dataset discovery, first
                               queries

02_DataAccessOverview/         More advanced queries, image searches, etc.

03_ScienceExamples/            Many complete science cases (stellar
                               streams, dwarf galaxies, large-scale
                               structure, SED fitting, ...)

04_HowTos/                     Service-specific tutorials (SPARCL, SIA
                               image cutouts, cross-matching, file storage)

05_Contrib/                    Community-contributed notebooks (ANTARES
                               alert broker, user science cases, etc.)

06_EPO/                        Education & public outreach (Teen Astronomy
                               Cafe, La Serena School for Data Science)

All notebooks are open-source and community contributions are welcome via pull request. We use them as living teaching materials in workshops at Astronomical Data Analysis Software & Systems (ADASS) and American Astronomical Society (AAS) conferences, summer schools, and university courses around the world. We have also recently translated most of our notebooks to the Spanish language.

In the coming months we will launch a tagged, searchable notebook gallery — filterable by science topic, Unified Astronomy Thesaurus (UAT) keywords, target audience, and difficulty level. The pilot framework was developed by two summer students working with the team.

Footprints of 24 sky survey datasets hosted at Astro Data Lab. This montage shows the wide variety of astronomical surveys, with some that cover the full sky, others focusing on the Milky Way (central plane), and yet others studying the extragalactic regions beyond the Milky Way. We ensure that each survey is represented in at least one of our example notebooks.

Looking Ahead

Nine years in, the Astro Data Lab science platform is evolving on several fronts simultaneously.

GPU computing. We are deploying a GPU node, which will be connected to the Jupyter notebook service. This opens deep learning and large-scale ML workflows in the same notebook environment where the data lives.

An AI assistant. Our first-ever user survey, conducted in September 2025, ranked an in-notebook AI assistant as one of the top requested features. We are actively exploring what responsible, science-aware AI assistance looks like in this context — helping users construct SQL/ADQL queries, navigate datasets, and debug notebook code, without hallucinating catalog column names. jupyter-ai might come in very handy here.

New integrated Web Portal. Our Data Explorer — an integrated web interface combining catalog browsing, query execution, image cutouts, spectral search, and job status monitoring — was rolled out last year. Some of the next milestones include integration of AladinLite into the portal, bringing the sky-visualization capability described above out of the notebook and into the browser-native interface, and a new integrated positional cross-matching service.

Try It

The full notebook library is open-source: github.com/astro-datalab/notebooks-latest. Community notebook contributions are welcome — see CONTRIBUTING.md in the repository. You can also run all notebooks locally, after installing the Data Lab command-line client and Python module: pip install astro-datalab

Astro Data Lab also offers a JupyterLab environment as a service to the broad astronomy community — students, researchers, educators, and citizen scientists. Registration takes just a moment at datalab.noirlab.edu.

Questions and feedback: datalab@noirlab.edu

Robert Nikutta is a scientist at NSF NOIRLab’s Community Science and Data Center, and lead of the Astro Data Lab science platform. Stéphanie Juneau is an associate astronomer at CSDC and lead of the SPARCL spectroscopy initiative. The platform is the work of the full Astro Data Lab team, past and present.

Expanding Geospatial Workflows in JupyterGIS: STAC Browsing and Story Maps

2026-02-19T17:34:00+00:00

JupyterGIS story maps in action

Since its initial announcement less than two years ago, JupyterGIS has been a growing effort to bring interactive geospatial workflows into the Jupyter ecosystem. Early milestones focused on laying the foundations: a composable GIS interface in Jupyter, a shared document model allowing real-time collaboration, a Python scripting API for exploration in the Jupyter Notebook, and a compatibility layer with the QGIS file format.

With the project maturing, recent work has focused on improving two key aspects of everyday geospatial workflows: discovering data and sharing results.

In this post, we (the JupyterGIS team) introduce two new features that move JupyterGIS further in that direction. First, a STAC browser integrated directly into the JupyterGIS user interface, allowing users to explore SpatioTemporal Asset Catalogs and add selected items to a project as map layers. Second, a new Story Map feature, inspired by existing GIS storytelling tools, which makes it possible to combine maps and narrative content in a single, interactive view.

Together, these additions aim to make JupyterGIS not only a place to analyze geospatial data, but also a place to explore datasets and communicate results, all within Jupyter.

Exploring STAC Catalogs in JupyterGIS

The SpatioTemporal Asset Catalog (STAC) specification provides a common way to describe and access geospatial datasets, particularly large collections of Earth observation data. By standardizing how data and metadata are exposed, STAC enables tools to discover and query datasets across different providers using consistent spatial, temporal, and property-based criteria.

JupyterGIS now includes a STAC browser user interface that allows users to explore STAC catalogs directly from within a project. The browser makes use of the STAC Filter extension, enabling users to define rich queries on catalog items. Based on the catalog metadata, JupyterGIS automatically generates the corresponding UI components for defining these filters, making it possible to refine searches and add selected items to a project as layers without leaving the Jupyter environment.

Exploring Sentinel 3 STAC in JupyterGIS

At the moment, the STAC browser ships with a small set of preconfigured catalogs: CDSE’s Copernicus catalog, CNES’s Geodes catalog, and University of Southampton’s WorldPop. This limited selection reflects the early stage of the feature. Work is already underway by the Eric and Wendy Schmidt Center for Data Science & Environment (DSE) at UC Berkeley to allow users to connect to arbitrary STAC catalogs by providing their own catalog URL, and to introduce a catalog of STAC catalogs to make discovering and configuring additional data sources easier in future releases.

Story Maps in JupyterGIS

Communicating geospatial results often requires more than interactive maps alone. To address this, JupyterGIS now introduces a Story Map feature that makes it possible to combine narrative content with map-based views inside a single interface. Inspired by existing GIS storytelling tools, this feature allows users to structure a sequence of markdown text and map position, where each step can capture a specific state of the project, including visible layers, layers symbology, and map extent.

Story Maps in JupyterGIS are designed to complement exploratory and analytical workflows, providing a lightweight way to present results directly from the same environment in which they were produced. This makes it easier to move from data exploration to communication without exporting projects to external tools.

Try it live in your browser by clicking on the following link, https://notebook.link/@quantstack/earthquakes-story-map, hosted by notebook.link, a free service for sharing Jupyter documents.

JupyterGIS story maps showcased in notebook.link

Acknowledgments

This work on JupyterGIS by QuantStack was funded by the French National Centre for Space Studies (CNES).

About the author

Greg Mooney is a Scientific Computing Developer at QuantStack. He is one of the core developers of JupyterGIS, the author of jupyterlab-gather, and a contributor to several other Jupyter extensions.

Collaborative editing for GIS workflows with Jupyter and QGIS

2025-02-26T18:50:00+00:00

The QGIS open-source project is a cornerstone in the geosciences ecosystem, providing robust tools for spatial data analysis and visualisation. While QGIS excels in empowering researchers and engineers, the evolving landscape of geospatial research demands enhanced collaboration and accessibility, prompting a shift towards web-based technologies.

Collaboration in Jupyter

Collaboration has long been a key requirement for data science and scientific computing workflows, but Jupyter Notebooks historically lacked built-in real-time collaboration features.

To bridge this gap, the jupyter-collaboration project has focused on integrating multi-user editing capabilities directly into JupyterLab and Jupyter Notebook. This effort builds on foundational work in conflict-free replicated data types (CRDTs), leveraging technologies like Yjs, and required integration into the Notebook document model and WebSocket-based architecture.

Making Notebook code suggestions in collaboration mode with https://github.com/jupyterlab-contrib/jupyter-suggestions

This approach did not only enable collaborative editing of notebooks, but also opened the door for collaboration on other editing UIs. This was done e.g. in JupyterCAD, a 3D parametric modeler that allows multiple users to collaboratively design and manipulate CAD models within JupyterLab.

Read more about JupyterCAD here https://blog.jupyter.org/announcing-jupytercad-3-0-d8f4b7b0a719

Collaboration in JupyterCAD

Introducing JupyterGIS

Today, we are excited to announce JupyterGIS, a web-based, collaborative, and extensible interface for GIS, leveraging the JupyterLab application framework and integrating seamlessly with the Jupyter notebook interface.

JupyterGIS provides the following key features:

a JupyterLab extension for collaborative editing of geospatial data files: with COG, Shapefile, GeoJSON data source support,
support for importing, exporting and collaborating with QGIS files,
advanced analysis and editing features with an identify tool and a symbology panel approaching what QGIS provides,
easily share and deploy GIS projects using JupyterLite.

Note: JupyterGIS is not an official Jupyter subproject.

GIS features

Raster and vector layers

JupyterGIS allows you to create multiple types of raster and vector layers, including local or remote GeoTIFFs, GeoJSON, shapefiles, and tile services.

You can select from a set of pre-defined tile services in our gallery, or you can provide your own source.

Identify tool and filtering

Akin to QGIS, we provide an identify tool and filtering capabilities, helping you to inspect your datasource.

Symbology

The symbology of a layer refers to its visual representation on the map, helping you to analyze your data comfortably. JupyterGIS provides a similar feature for its raster and vector layers.

Spatio-temporal data

You can easily animate and analyze your spatio-temporal data.

Python API

Even though you can do most operations from the UI, JupyterGIS also provides a Python API enabling scripting capabilities, which can be used to process data and operate on JupyterGIS documents programmatically from a Jupyter notebook.

Collaborative features

Building upon our previous work to enable collaborative editing in JupyterCAD, we developed a shared document model to support multi-user workflows in JupyterGIS.

Everything is shared between collaborators, from layers data to symbology and filtering rules.

We also provide some collaboration features like the ability to see other collaborators’ cursors, follow collaborators’ viewports, add geolocated Google Docs-like comment conversations, and see what layer features collaborators are identifying.

Try it!

You can directly try JupyterGIS in your browser, without installing anything, thanks to JupyterLite:

https://jupytergis.readthedocs.io/en/latest/lite/lab/index.html

Documentation

Check out the Python API reference and tutorials on how to get started in our documentation:

https://jupytergis.readthedocs.io

What’s next?

We are excited to welcome new contributors and see the JupyterGIS community grow. Here are some exciting developments on our radar for the future:

Support for more data source types.
Processing tools,
Support for suggestions by collaborators, similar to the inline suggestions feature of JupyterCAD,
An improved and more featureful Python API,
A better coverage of QGIS features.

How can you support our work?

Your support is important, if you would like to help us build more exciting tools around JupyterGIS, do not hesitate to contact us on the public geojupyter Zulip channel!

Additionally, you can join us for the bi-weekly GeoJupyter hackathon on Wednesday https://geojupyter.org/calendar.html

Acknowledgments

This work on JupyterGIS by QuantStack and Simula Research Lab (Simula) was funded by the European Space Agency (ESA) for our proposal “Real-time collaboration and collaborative editing for GIS workflows with Jupyter and QGIS.”

We are also grateful to The Eric and Wendy Schmidt Center for Data Science & Environment (DSE) at UC Berkeley for funding the work of Matt Fisher on the project.

Finally, this wouldn’t have been possible without the foundational work on JupyterCAD initiated by Duc Trung Le.

About the JupyterGIS contributors

Meriem Ben Ismail is a Scientific Software Developer at QuantStack. She just completed her Software Engineering degree at INSAT (National Institute of Applied Sciences and Technology in Tunisia) and her six-month internship as an open-source scientific software engineer at QuantStack.

Nicolas Brichet is a Scientific Software Developer at QuantStack. He holds a master’s degree in robotics from the University of Montpellier. Prior to joining Quantstack, Nicolas worked for almost fourteen years in French research institutes (INRAE and CNRS), as software developer for scientific research. He also worked on different parts of the data lifecycle, such as data acquisition, analysis, and storage.

David Brochart is a Technical Director at QuantStack. David used to work in the semiconductor industry as a digital circuit designer, mostly using FPGAs. He studied telecommunications at the French “Telecom Bretagne” engineering school, with major in digital circuit design. David also worked as a researcher in hydrology at Irstea, a French scientific research institute, where he heavily used the Python scientific stack.

Matt Fisher is a Research Software Engineer and Community Manager at Schmidt Center for Data Science & Environment. Matt is passionate about open science, community-owned software, teaching and learning, accessibility, and inclusion. It is his treasured privilege to support researchers and educators in their missions by working on projects like QGreenland and JupyterGIS!

Anne Fouilloux, a Senior Research Engineer, specializes in Open Science, FAIR data, and Big Data analytics. With a PhD in Atmospheric Physics, she has contributed to Pangeo and EOSC (European Open Science Cloud), focusing on data management, Earth system modeling, and scalable computing. A strong advocate for Open Science, she advances data accessibility and interoperability to drive scientific innovation.

Greg Mooney is a Scientific Computing Developer at QuantStack. He received a Bachelors degree in Software Engineering from Arizona State University (ASU). Before joining QuantStack, he worked as an Aegis Computer Network Technician, working on the network infrastructure connecting various sensor systems onboard naval vessels.

Martin Renou is a Technical Director at QuantStack and a maintainer of Project Jupyter. Among other projects Martin is a core team member of the ipywidgets project and maintains many Jupyter widget packages such as ipyleaflet, ipydatagrid, ipygany, ipycanvas, and bqplot. He is a co-creator of the Voilà dashboarding system, and the xeus-python kernel.

Arjun Verma is a Scientific Software Development Intern at QuantStack. He is also a final year undergraduate student at Cluster Innovation Centre, University of Delhi. He loves 3D & GIS and also contributes to JupyterCAD and is one of the maintainers of PyBaMM where he has also been GSoC contributor and mentor.

Interactive Mapping with ipyopenlayers

2024-09-05T15:30:00+00:00

Bringing the Power of OpenLayers to Jupyter.

Interactive maps play a crucial role in presenting complex geospatial information in a clear and engaging manner. While the Python ecosystem already offers an impressive range of libraries for visualization, ipyopenlayers stands out as a powerful new tool that allows for the seamless integration of the advanced features of the OpenLayers JavaScript library into Jupyter. In this article, we will present the main features of ipyopenlayers and demonstrate how this library can transform your geospatial data into dynamic, interactive visualizations.

1. What is ipyopenlayers?

ipyopenlayers is an open-source Python library that integrates OpenLayers, a popular JavaScript library for creating interactive maps, directly into Jupyter Notebooks. By combining the strengths of Jupyter and OpenLayers, ipyopenlayers allows users to create interactive maps with ease. Its versatility and simplicity make it an ideal choice for a wide range of applications, from data exploration to presenting geospatial analysis.

2. Getting Started with ipyopenlayers

To start using ipyopenlayers, you only need to install the library via pip:

pip install ipyopenlayers

Once installed, you’re ready to embark on your interactive map-making adventure!

a) Creating Your First Map

ipyopenlayers has a user-friendly interface that makes creating maps simple and quick. Here’s how to generate a basic map with just a few lines of code:

from ipyopenlayers import Map

# Create a map centered on a specific location
m = Map(center=[0, 0], zoom=2)
# Display the map
m

Screenshot of a simple ipyopenlayers map displayed inline in the Jupyter notebook

b) Adding Layers and Controls

Maps are more than just images; they allow you to mark specific locations and overlay various data. ipyopenlayers makes it easy to add layers and controls to your maps. For example, here’s how to add a raster tile layer from OpenStreetMap and a zoom slider control:

from ipyopenlayers import RasterTileLayer, ZoomSlider
m =Map()
# Add a raster tile layer
raster_layer = RasterTileLayer(url='https://{a-c}.basemaps.cartocdn.com/dark_all/{z}/{x}/{y}.png')
m.add_layer(raster_layer)

# Add a zoom slider control
zoom_slider = ZoomSlider()
m.add_control(zoom_slider)
m

Screenshot of a simple ipyopenlayers map displayed inline in the Jupyter notebook showcasing the basic controls

3. Main Features of ipyopenlayers

ipyopenlayers is packed with a wide array of functionalities that make it ideal for creating advanced interactive maps in Jupyter Notebooks. From adding custom controls to enabling dynamic layer and overlays, this library offers you tools for enhancing your geospatial visualizations. Here’s a sneak peek at some of its key features.

a) Vector Tile Layer Support

Vector Tile Layers enable efficient, high-quality rendering of geospatial data, allowing for smooth zooming and dynamic styling of map features. They offer flexibility and customization for detailed interactive maps in Jupyter Notebooks.

from ipyopenlayers import Map, VectorTileLayer

# Create a map centered at coordinates [0, 0] with zoom level 0
m = Map(center=[0, 0], zoom=0)

# Add a VectorTileLayer to the map
vector_layer =  VectorTileLayer(
url='https://basemaps.arcgis.com/arcgis/rest/services/World_Basemap_v2/VectorTileServer/tile/{z}/{y}/{x}.pbf',
source_format={
    'type': 'MVT',
}
)
m.add_layer(vector_layer)

# Display the map
m

Screenshot of an ipyopenlayers map displaying a vector layout in the Jupyter notebook

b) GeoJSON Support

GeoJSON is a popular format for encoding geographic data structures. Here’s how to display GeoJSON data on your map:

import os
import json
from ipyopenlayers import Map, GeoJSON

m = Map(center=[-77.15641232105159, 34.561348453091625], zoom=9.5)
# Note the coordinate order: [latitude, longitude]

with open("demo.json") as f:
    data = json.load(f)

# Add the GeoJSON layer
geo_json = GeoJSON(
    data=data
)

# Add the GeoJSON layer to the map
m.add_layer(geo_json)
m

Screenshot of an ipyopenlayers map displaying a GeoJSON dataset as a layer on a map

c) GeoTIFF Support

ipyopenlayers also enables the visualization of GeoTIFF files, which are essential for displaying raster data with geographic context:

from ipyopenlayers import GeoTIFFTileLayer

# Create a GeoTIFF layer
geo_tiff_layer = GeoTIFFTileLayer(url='https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/36/Q/WD/2020/7/S2A_36QWD_20200701_0_L2A/TCI.tif')
m.add_layer(geo_tiff_layer)

Screenshot of an ipyopenlayers map displaying a GeoTIFF dataset as a layer on a map

4. Advanced Use Cases: Building an Electricity Map Dashboard

The potential of ipyopenlayers extends far beyond basic mapping. One innovative application is its use in creating dynamic and interactive dashboards that integrate with other powerful Python tools.

A notable example is the Electricity Map Dashboard, a project designed to visualize and analyze electricity data from around the world using ipyopenlayers in conjunction with libraries such as electricityMap, ipywidgets, voila, and bqplot.

Key Features:

Real-time Visualization: Using electricityMap, the dashboard overlays real-time electricity consumption and carbon intensity data onto a global map with ipyopenlayers.
Interactive Controls: ipywidgets enables users to filter data by region, select specific time periods, and toggle between different map layers, making the data exploration highly interactive.
Dynamic Plotting: bqplot allows users to generate customizable plots synchronized with the map, offering deeper insights into electricity trends and carbon emissions.

Screenshot of the Electricity Map dashboard built with bqplot and ipyopenlayers

This project showcases the powerful integration of ipyopenlayers with other Python tools to create a dynamic, user-friendly dashboard for analyzing global electricity trends.

5. Developments

The future of ipyopenlayers is promising, with upcoming features like improved integration with other Python libraries, advanced customization tools, and expanded support for geospatial data formats. We invite the open-source community to contribute to this project, whether through suggestions, bug reports, or code contributions.

6. Conclusion

With ipyopenlayers, the power of interactive maps is at your fingertips within your Jupyter Notebooks. This Python library allows you to present your geospatial data in an engaging and insightful way, making your analyses more dynamic and accessible. Whether you’re a data analyst, researcher, or simply a mapping enthusiast, ipyopenlayers provides the tools you need to turn your data into compelling visualizations.

So, what are you waiting for? Install ipyopenlayers and start creating interactive maps that bring your data to life!

7. Acknowledgments

This project was made possible through the collaboration and support of several individuals. Special thanks to Martin Renou , Technical Director at QuantStack, whose guidance and expertise were instrumental throughout this project. His mentorship greatly contributed to the success of this ipyopenlayers.

8. About the Author

Nour Cheour has just completed her Engineering degree at INSAT and her six-month internship as an open-source scientific software engineer at QuantStack.

JupyterGIS

2024-06-12T17:09:00+00:00

Pioneering Web-based, Collaborative, and Open-source GIS Tools

By Sylvain Corlay [1], Anne Fouilloux [2], and Monika Weissschnur [3]

We are thrilled to announce that the European Space Agency (ESA) is funding our proposal “Real-time collaboration and collaborative editing for GIS workflows with Jupyter and QGIS.”

The consortium spearheading this project comprises QuantStack and Simula Research Lab (Simula), two organizations with a long history of contributions to the Jupyter project and the broader open-source scientific computing ecosystem.

The goal of the project is to build solid foundations for a versatile web-based user interface for Geographic Information Systems (GIS) workflows. It will comprise several components, including a JupyterLab extension for collaboratively editing QGIS project files and the integration of these APIs in the Jupyter Notebook. We will then explore integration with platforms like the Copernicus Data Space Ecosystem (CDSE) and the European Open Science Cloud (EOSC).

Collaborative workflows in geosciences

Collaborative editing of documents has become an integral part of our digital lives and has made us collectively more productive. Gone are the days of cumbersome email exchanges with documents shuttling back and forth.

Looking ahead, the potential of co-editing extends far beyond text documents and will apply to all authoring UIs, from CAD to image processing. We think that the shift to collaborative editing will be even more transformative for larger and more complex projects, which are inherently social and require the concerted effort of large teams. Whether you are designing a stadium, a plane, or an ocean liner, you need to coordinate a diverse expertise to build a unified model. In geosciences, it may range from climate modeling to agriculture, ecology, urban planning, and many more areas of expertise. For such endeavors, we must embrace tools favoring collaboration, and this applies to the future web-based user interfaces for geoscience research.

We have been working on collaborative editing in the core of JupyterLab for the past three years. Our approach is based on the Yjs framework, an implementation of CRDT data structures (Conflict-free Replicated Data Type). We have learned that since it is so tied to the data model, retrofitting these features into an existing application is considerably more arduous than building the initial data model on the appropriate paradigm from inception. This is why JupyterGIS will be built from the ground up with collaborative editing in mind.

JupyterGIS will be the first open-source GIS tool to provide collaborative editing features.

Importantly, JupyterGIS is part of a broader mission to enable collaborative workflows in open-source technical computing. The JupyterCAD project, a collaborative CAD modeler, is another example of this endeavor. By sharing knowledge and resources, both JupyterGIS and JupyterCAD will benefit from and contribute to each other’s progress, ultimately driving innovation and productivity in their respective fields.

From the desktop to the web

Advanced authoring tools, including IDEs, CAD modelers, image processing software, and GIS applications, are essential for professionals who rely on them for extended periods. These users have high expectations and demand key features to optimize their productivity and workflow. Among these features are extensibility with plugins, configurable keyboard shortcuts, themability, internationalization, scriptability, a unified settings system, and the ability to operate across multiple browser windows and devices.

Developing a new application from the ground up that meets all these requirements is a formidable challenge. This is where the JupyterLab application framework proves invaluable. By leveraging this framework, developers can build custom, feature-rich authoring tools that incorporate these essential features from the outset, making it an ideal solution for creating modern, user-centric applications.

Beyond the pure authoring user interface, for which JupyterLab is a great tool, it is also the perfect tool for integration with notebook-based workflows. As part of this project, we will develop an advanced Python API to manage JupyterGIS sessions, leveraging Jupyter’s robust display system to incorporate sophisticated GIS features inline in Jupyter notebooks and consoles. This integration will further enhance the capabilities and versatility of our application. This feature will follow the same architecture as that of JupyterCAD for the integration with the notebook user interface.

Real-world applications

JupyterGIS is meant to become a general-purpose tool. However, we will work with teams of practitioners on specific real-world applications to ensure that it addresses their needs.

One such project concerns the use of digital and GIS solutions for emergency management. While practitioners involved in emergency management already have experience with such tools, we are convinced that web-based applications built with collaboration in mind from the start can foster improved coordination and collaboration, and therefore the effectiveness of the response.

Response teams comprise different organizations, including the affected municipalities, police, emergency response organizations like the Red Cross, and advisers such as the U.S. Army Corps of Engineers (USACE) in the United States or the Norwegian Water Resources and Energy Directorate (NVE) in Norway. Even when using the same incident management system, miscommunications and misunderstandings can occur between these stakeholders, resulting in delays or improper use of resources. An improved integration of collaborative GIS software can improve upon the existing tooling.

Building collaborations in Europe and beyond

We will partner with key practitioners and make sure JupyterGIS addresses their use cases. Integration in the Copernicus Data Space Ecosystem (CDSE) and the European Open Science Cloud (EOSC) will be key to the adoption of JupyterGIS and demonstrate its deployment in such environments.

Beyond Europe, we are currently working on including this project into a broader scope. We are excited to be partnering with the “LIVE-Env” project, an open-source initiative spearheaded by Alyssa Goodman at Harvard University. Furthermore, we are establishing a key collaboration with the Berkeley Institute for Data Sciences (BIDS) and the Schmidt Center for Data Science and Environment (DSE) at UC Berkeley and 2i2c, with a focus on Jupyter-based geosciences.

Affiliations

[1] QuantStack[2] Simula[3] Simula

Desktop GIS software in the cloud with JupyterHub: A QGreenland workshop success story

2023-08-02T16:23:00+00:00

👋 We are Trey Stafford and Matt Fisher, co-authors of the QGreenland data package’s source code. This year, we had the pleasure of running a hands-on geospatial data and open science QGreenland Researcher Workshop. It was important for attendees to participate in the workshop in a hands-on way while minimizing the negative impacts of installing software, requiring expensive personal computers, and troubleshooting unique computer configurations. We felt a JupyterHub was a good fit for our workshop for this reason, if it could accommodate our need for running QGIS — a desktop application.

In this blog post, we will introduce QGreenland, describe our experience using JupyterHub in the cloud for our workshop’s computing environment, and discuss challenges we overcame to enable our attendees to use QGIS in a cloud graphical desktop environment. Finally, we will highlight some workshop outcomes and discuss opportunities for enhancement based on new developments in the Jupyter ecosystem.

In our workshop, 25–30 international learners (including from Germany, India, France, Canada, Poland, and the United States) used QGIS in a JupyterHub’s browser-based Linux desktop environment to collaboratively test, explore, visualize, and process Earth science data simultaneously with the same user experience they expect from using QGIS on their personal computers! Better yet, getting started was as simple as logging in. Our workshop was a success story not just in education, but also in open source and collaborative development, and we want to share what we learned.

Note the browser tabs at the top of this screenshot; this is a full desktop-based GIS environment running QGreenland in the cloud!

The JupyterHub used by the QGreenland 2023 Researcher Workshop was generously provided by the NASA CryoCloud team, whose mission is to help researchers transition to cloud-based collaboration.

About QGreenland

QGreenland is an open-source Greenland-focused geospatial data package for QGIS, a community-owned graphical Geographic Information System (GIS) platform. Researchers and members of the public leverage QGreenland’s ready-to-use interdisciplinary datasets to do field planning, teach about glaciers, and much more.

QGreenland’s MIT-licensed source code uses community-maintained open software like GDAL and PyQGIS to automate data normalization and populate the QGIS project with important information like data provenance and the order of layers in the QGIS Layers Panel. Check out our documentation to learn more! QGreenland also has a YouTube channel with tutorials produced by CIRES Education and Outreach.

Once QGIS is installed, opening QGreenland is as easy as double-clicking the included “.qgs” file. Here, a representative view of QGreenland v3 alpha in QGIS is displayed with newly updated layers: Arctic sea routes (National Geospatial Intelligence Agency), bathymetric depth (General Bathymetric Chart of the Oceans (GEBCO)), and ice thickness (IceBridge BedMachine Greenland v5) layers.

Based on user research, QGreenland has enabled:

the public to more easily access data gathered by researchers visiting Greenland: “In Greenland, people are often asking, ‘how can we find the data the foreign scientists bring back from Greenland?’ Now we can directly utilize much of it.”
researchers to plan field work: “Being able to use QGreenland at our field station was critical to our research process!”
educators to develop interactive lessons about Greenland and climate change: “…using QGreenland for presentations because it is presentation quality already.”

QGreenland’s 2023 researcher workshop

One of the QGreenland team’s most important forms of direct user interaction and support is facilitating workshops. Most recently, we hosted a 3-day (total of 9 hours) virtual workshop for researchers focused on working with geospatial data in an open science framework. All of the materials covered in the workshop were built using open-source tools and are MIT-licensed and published on GitHub.

A “personal computer” in the cloud

We decided early on that we wanted to use JupyterHub to solve the diverse problems that come with “bring your own device” workshops. We experimented with administering our own JupyterHub on Kubernetes, but the setup overhead was too high for our short workshop. CryoCloud’s JupyterHub enabled us to avoid this overhead and focus on serving our participants. Because the software that comprises CryoCloud is open-source and developed in collaboration with the communities CryoCloud serves, we could directly contribute to curating a computing environment ideal for our participants.

JupyterHub is known for providing access to Jupyter Notebooks via JupyterLab, but it turns out it can also be used to host pretty much any interactive web based application! The jupyter-server-proxy project enables this, and there are additional packages that make running specific applications easier. jupyter-rsession-proxy makes it easy to run RStudio inside JupyterHub, jupyter-vscode-proxy allows running code-server (fully open source self-hosted version of Visual Studio Code) inside a JupyterHub, etc. Pertinent to our use case is jupyter-remote-desktop-proxy, which lets you run a complete Linux desktop environment inside your JupyterHub! This was critical for our workshop, as it allowed us to use QGIS — purely desktop software, not adapted for the web — from inside a web browser. Workshop participants did not need to install anything. This enabled participants to focus on the content of our workshop rather than the logistics of setting up and debugging tools on their varied machines.

The CryoCloud JupyterHub enabled each of our workshop participants to provision their own compute environment (JupyterLab + Linux Desktop) with all of our workshop’s dependencies pre-installed. It also set everyone on equitable footing — someone accessing the workshop on a 10 year old laptop would get the same computing resources as someone on a brand new MacBook Pro.

Challenges scaling QGreenland

The CryoCloud JupyterHub already had jupyter-remote-desktop-proxy and QGIS installed, so we could validate this approach to our workshop quickly. However, to use QGreenland at this scale, we needed to solve a couple of usability problems. The first issue was a user experience problem: the operating system did not have appropriate file type associations for QGIS, so files like the QGreenland project file would not open in QGIS when double-clicked in the desktop file browser. We quickly discovered a solution and integrated it with a simple pull request to the Docker image we were using.

The second problem was a performance problem: QGIS would take several minutes to open QGreenland from the hub’s shared storage drive. After some investigation, it turned out this was due to us loading multiple GB of data from an NFS share! While a long term solution might involve getting QGIS to load data directly from cloud object storage (like S3), we instead decided to go a different route — provision each user a small, fast and temporary Elastic Block Store disk. At the start of the workshop, we provided all users a small script that would copy the dataset from NFS to this faster disk once, and this drastically reduced load times from about 5 minutes to under 3 seconds! You can follow our debugging process on this issue, and find the JupyterHub config used to provision these disks here.

By overcoming these challenges, we created a smooth, intuitive, and performant computing experience for all of our participants, most of whom had never been exposed to this sort of collaborative computing environment.

Outcomes

The workshop participants engaged in small group work to complete various exercises, group discussions, and data scenarios. Each group produced Jupyter Notebooks and GitHub Discussions posts as deliverables. We created an outcomes webpage to summarize our participants’ accomplishments. One highlight was participants’ insightful commentary on FAIR & CARE principles.

Based on these outcomes, we consider our workshop a success. While we put in a significant amount of time creating our materials, CryoCloud’s cloud costs and our time investment in preparing computing resources were relatively small. For approximately 25 people, our cloud costs break down to roughly $1/person/day!

Conclusion

The CryoCloud JupyterHub met our workshop needs and provided a delightful experience for administrators and participants alike, and we are excited for what’s next. JupyterLab 4 and jupyter_collaboration v1.0.0, a real-time collaboration extension, were just announced, and the CryoCloud team is currently working to integrate these new releases into their hub. Real-time collaboration will enable exciting cloud use cases, like small groups working together on the same notebook without a screen share, or organizers providing technical support in a live notebook. We anticipate running this workshop again. We are excited to use JupyterHub again and look forward to experimenting with these new features!

Acknowledgements

Reviewers

In alphabetical order, thanks to Twila Moon, Yuvi Panda, Tasha Snow, and Alyse Thurber for their time contributing to this post!

CryoCloud

Snow, Tasha, Millstein, Joanna, Scheick, Jessica, Sauthoff, Wilson, Leong, Wei Ji, Colliander, James, Pérez, Fernando, James Munroe, Felikson, Denis, Sutterley, Tyler, & Siegfried, Matthew. (2023). CryoCloud JupyterBook (2023.01.26). Zenodo.https://doi.org/10.5281/zenodo.7576602

2i2c

2i2c is a non-profit organization that runs open-source infrastructure for collaborative computing, and maintains the CryoCloud JupyterHub used in this workshop. You can see the complete configuration of this JupyterHub in this public repository.

Genomic data visualization in Jupyter

2021-02-08T13:29:00+00:00

If there is one thing that recent events tell us, it is that genomics is a large source of data, and that its manipulation and understanding allow for the quick development of new drugs and… vaccines.

We decided to build upon the Jupyter ecosystem and enrich its capabilities in this space, with a genomics data visualization widget.

In doing so, we chose to leverage the igv.js JavaScript library, developped by the Integrative Genomics Viewer (IGV) team, and used by the web app of the same name. This is all done thanks for the power of ipywidgets , which helps linking the JS genomic representation object to our Jupyter notebook.

Hence, we are proud to announce the release of ipyigv, a Jupyter widget to render genomics data, based on igv.js!

Installing ipyigv

ipyigv is available on PyPI and conda-forge. It can be installed with both pip and mamba / conda:

mamba install -c conda-forge ipyigv

You can find additional installation instruction on the project’s page on GitHub.

Rendering genomic data

igv.js consumes genomic data in two chunks:

the genome itself as documented here
specificities of the genome underlined as Tracks, which are displayed alongside or on top of the genome

ipyigv uses the same logic, with a helper function to use the public genomes available. Data is then displayed in an IgvBrowser widget, a wrapper for the igv.js browser.

To make things easier, there are a few public genomes made directly available via a helper Bunch , PUBLIC_GENOMES .

Here is what it looks like:

Creating a genome browser with data from a public genome

Now that we have created a genome browser, we may add tracks to it. Tracks may be of different sorts, about ten according to the igv.js documentation. Describing all the kinds of tracks is out of the scope of this article, but it should be noted that:

a common Track class is defined with all the common properties. Each track type corresponds to a subclass inheriting Track .
in order to ease things out, some class introspection was introduced in order to identify the type of Track based on the extension of the containing file. As shown below, this allows for instantiating a track by just using the Track constructor, without knowing the name of the actual subclass being instantiated (e.g. AnnotationTrack in the example below). Type inference is made through the type of file, or by specifying a type property.

Now, let’s add a Track to our browser, then remove it:

Adding and removing a track to/from our browser

In this first version, two other functions come in as handy:

browser.search('chr3:1-190,100,300') would position the browser at the requested position in the genome;
browser.dump_json() prepares a JSON representing the configuration of the browser. Handy if you would like to reinstantiate another browser with the same configuration without redoing it manually. Use browser.out to output the JSON content.

We hope you’ll have fun manipulating genome data in Jupyter, and sharing visual knowledge thanks to ipyigv!

Acknowledgments

The development of ipyigv by Jean-David Harrouet at QuantStack was funded as part of the PLASMA project, led by Claire Vandiedonck, Pierre Poulain, and Sandrine Caburet, associate professors at Université de Paris.

Sponsors to the PLASMA initiative include:

Région Île-de-France, via the “Trophées franciliens de l’innovation numérique dans le supérieur” (EdTech 2018) grant program,
Université de Paris, via the Initiative of Excellence (IdEx) Label and its “inovating teaching” grant program,
EUR G.E.N.E., the graduate school on Genetics and Epigenetics,
the university training “Création, analyse et valorisation de données biologiques omiques” (DU Omiques).

About the author

Jean-David Harrouet is an innovator helping companies with their digital transformation.

He believes that the right mix of coding and business acumen is a way to make life better for many people.

Jupyter meets the Earth: EarthCube Community Meeting

2020-08-17T19:42:00+00:00

By: Lindsey Heagy, Fernando Pérez, Joe Hamman and the Jupyter meets the Earth team (cross-posted on the Pangeo Blog)

As a part of the 2020 EarthCube annual meeting, we held a Jupyter meets the Earth community discussion session on July 27. The Jupyter meets the Earth project is an EarthCube funded effort that combines research use cases in geosciences with technical developments within the Jupyter and Pangeo ecosystems. In this model of equal partners, scientific questions help drive software infrastructure development, and new technologies expand the horizons of viable research. This online workshop was an opportunity to gather members of the community, welcome newcomers, provide updates on the Jupyter and Pangeo ecosystems, and have time for discussion.

The goals for the meeting were to:

Provide an overview of the Jupyter & Pangeo ecosystems for researchers from the EarthCube community.
Outline avenues for getting involved.
Gather input for what advancements would best serve your research needs.

Over 100 participants registered, and we had contributions from 9 speakers. The meeting was a mix of presentations and Q&A from the community. The full recording of the meeting is available on youtube, and we encourage continued discussion on the associated discourse post.

Presentations (google drive folder)

Fernando Pérez (slides) started off the meeting by introducing the Jupyter meets the Earth project — an effort aimed at driving forward technological developments in the Jupyter and Pangeo ecosystems in partnership with researchers in the geosciences. The motivation is to advance research and the software that supports it by combining domain expertise with methods in data science, software & data engineering practices. He provided an overview of Project Jupyter, highlighting the interplay between software and content, services, standards, community and governance that is necessary for broad-impact scientific open source software projects. He presented the extensible JupyterLab platform, that can be adapted to domain-specific needs as illustrated by the FlyBrainLab and Cloud Robotics Command Station efforts. The Jupyter meets the Earth team aims to similarly develop tools and extensions that will support interactive computing workflows in the geosciences.

Overview of Jupyter and Jupyter meets the Earth from Fernando Pérez

Next up, Scott Henderson (slides) provided an overview of Pangeo and associated community events, including Hackweeks. “Pangeo is first and foremost a community promoting open, reproducible, and scalable science.” In terms of technology, this involves developing fully open source tools that can be deployed on shared computational infrastructure, such as HPC centers or the cloud, and hosting several forums to foster communication between scientists and software developers. Software is an important avenue for connection, but the overarching goals are a rallying point for a community. The critical mass of enthusiastic people has been key to the success of the Pangeo model.

Overview of Pangeo and Hackweeks from Scott Henderson

The Pangeo model is intended for use both on the cloud and on High Performance Computing (HPC) infrastructure. Kevin Paul gave the third talk on Pangeo on HPC (slides, notebook). HPC and cloud computing environments present technical differences in terms of usage patterns, file access, and resource allocations, however, the goals of Jupyter and Pangeo are similar in both cases — to enable interactive computing and simplify the user experience on both. Tools such as dask and dask-kubernetes/dask-jobqueue are targeted at enabling parallel computing on both infrastructures.

Kevin Paul giving us a demo of Pangeo on the Cheyenne supercomputer

After a Q&A session that included questions on the computational cost of running Pangeo Infrastructure and efficient use of tools including Zarr, we moved on to a series of lightning talks.

Lightning Talks

Six speakers presented short lightning talks on aspects of the Jupyter and Pangeo ecosystems ranging from technologies to scientific applications to opportunities to engage with the Pangeo community.

Anderson Banihirwe (notebook and details in intake-esm) kicked off the lightning talks by giving a demo and overview of Intake — a project to streamline loading and sharing of data. He showed a demo that included both Optimum Interpolation Sea Surface Temperature (OISST) data, as well as data from the Coupled Model Intercomparison Project (CMIP) running interactively on Cheyenne, the supercomputer at NCAR and using Dask for distributing the workload across nodes, with real-time diagnostics of the distributed computation provided by Dask’s JupyterLab extension.

Demo from Anderson Banihirwe using intake to access OSSIT and CMIP data

Next up, Scott Dale Peckham (notebook) gave a presentation that demonstrated the use of ipywidgets and ipyleaflet to create an interactive interface for fast access to geoscience data on servers that support the OpenDAP protocol. The project he champions is called BALTO, the Brokered Alignment of Long Tail Observations (also a famous Siberian Husky and sled dog). These graphical interface elements can be used in a programmatic workflow such as a Jupyter Notebook, but they conveniently encapsulate many details of accessing the data and resources provided by BALTO. This allows the scientists to focus on their research questions, without having to break their workflow to access data with external tools.

BALTO GUI demo from Scott Peckham

We then had a talk from Edom Moges (slides) who presented work he is conducting with Laurel Larsen’s research group in hydrology as one use case in the Jupyter meets the Earth project. The presentation focused on a data synthesis work that aims to build a Jupyter based interactive platform that transforms raw hydrometeorological data to a gap-filled ready to use data for several intensively monitored watersheds across the US. The platform will be a basis for future community initiatives to benchmark data processing approaches, support comparative hydrological studies and comprehensive data-driven forecasts.

Lightning talk from Edom Moges and Laurel Larsen on the hydrology use-case in the Jupyter meets the Earth project

Georgiana Dolocan (video) impressed us next with an animated video accompanied with her narration to explain JupyterHub, its components (authenticator, spawner, proxy) as well as deployment options. The littlest JupyterHub (TLJH) is designed to make it simple to deploy multi-user Jupyter infrastructure on a single machine, and the more sophisticated Zero 2 JupyterHub Kubernetes (Z2JH) option is meant to scale to many users and large computational needs.

Animations from Georgiana Dolocan on JupyterHub

Presenting from the perspective of an enthusiastic user, Erik Sundell (slides) gave us an overview of Jupyter Book: a tool to quickly create beautiful websites from notebooks and markdown. He walked through how to host them for free online in a time efficient way, and highlighted features including connections to Binder, which enable users to run content interactively.

Overview of JupyterBook from Erik Sundell

Joe Hamman (slides) finished off our lightning talk session by outlining avenues for connecting with the Pangeo community. These include day-to-day communication on GitHub, Gitter, discourse and twitter, as well as more recent coffee-breaks. Depending on your topic of interest, there are also working groups that you can join on topics including data, machine learning, education, cloud computing, or you can suggest your own!

Connecting with the Pangeo community — an overview from Joe Hamman

Follow up and further discussion

To continue the discussion afterwards, we posed (slides) a few questions where we hope to learn from the community’s needs, such as:

What does your interactive computing workflow look like today? What do you envision it will be in 5 years?
How would you like to publish and share your computational research and where can improvements be made?
How do you stay up to date with the evolving open-source ecosystem? How would you like to be keeping up-to-date?

We are looking for your input and ideas! Please add your thoughts to the discourse post.

Thanks

Thank you to the participants, speakers, and especially Lynne Schreiber and Ouida Meier from the EarthCube office for all of their support and work (even with very last-minute requests!).

This work is part of the Jupyter meets the Earth project, supported by the NSF EarthCube program under awards 1928406, 1928374.

SlicerJupyter: a 3D Slicer kernel for interactive publications

2020-07-08T16:29:00+00:00

The Jupyter ecosystem is a powerful platform for exploratory computational science, and now it can connect with some of the deep and rich domain-specific desktop applications that have decades of feature development already invested in them. With the integration of 3D Slicer with Jupyter through the xeus-python’s interpreter, we demonstrate how a Qt-based graphical desktop application with 3D visualization provided by Visualization Toolkit (VTK), image processing provided by the Insight Toolkit (ITK), can be used through a Jupyter notebook. This approach is available on the SlicerJupyter GitHub repository and could be extended to other applications that embed Python, such as Blender, FreeCAD, or ParaView.

This xeus-python integration is beneficial both for the Jupyter ecosystem and desktop applications. Features that have been developed for decades for desktop applications become readily available for Jupyter users without learning a new working environment or redeveloping features. For a desktop application, the Jupyter notebook can serve as a way to create reproducible data processing workflows, scientific publications, and maintainable tutorials without requiring local software installation. From Jupyter, you can now quickly create simple medical imaging applications with extremely rich interactivity from 3D Slicer. Read below for a discussion of the features available and the history of the project.

Powerful Medical Imaging Capabilities Available Through Jupyter

3D Slicer (or Slicer for short) is a C++ desktop application that uses Qt, ITK, and VTK libraries for visualization and medical image analysis. Slicer’s embedded Python interpreter makes all its features accessible with the Python programming language. Slicer has a simple built-in console to run Python commands interactively and can run Python scripts from files, but these are not as convenient as cell-base interactive notebooks, which have become popular among data scientists and researchers in recent years.

By integrating the xeus-python kernel, we can use a Slicer process as a Jupyter kernel. xeus-python leverages the xeus C++ implementation of the Jupyter kernel protocol. xeus-python is an alternative to ipykernel, which can be used with a vanilla CPython interpreter, interfacing to standard CPython Jupyter widgets like itkwidgets, but it can also be coupled with custom interpreters and GUI event loops, like the Slicer interpreter and its Qt event loop. This allows you to represent a complete scene in Medical Reality Markup Language, MRML, Slicer’s internal data structure. The kernel exposes the full medical imaging API and representation of your data in a meaningful way for Python developers, allowing access through standard Python ecosystem formats such as pandas dataframes and NumPy arrays in the Notebook.

Interactivity Levels

You can also use Jupyter interactive widgets (sliders, buttons, etc.) to control Slicer, modify data, or adjust processing and visualization parameters. Interactivity can be implemented at different levels.

Level 1: Standard Jupyter widgets display application specific objects by automatic conversion of application-specific data objects to standard Python objects. For example, Slicer markup fiducial lists are displayed as a nicely formatted table and model nodes are rendered as 3D objects.
Level 2: Static image widgets display content that the desktop application renders. These widgets can be made interactive by modifying data and rendering parameters using additional standard widgets. This makes rich visualization capabilities — sophisticated rendering various data types, rendering of very large data sets, etc. directly available in Jupyter.
Level 3: Dynamic viewer widgets display 2D and 3D views rendered by the desktop application. Mouse and keyboard events are forwarded to the desktop application that allows zooming/rotating views, and to utilize all 3D interactions implemented in the desktop such as placing annotations, making measurements, or segmenting images the same way as if it was done on the desktop appication’s screen. This is implemented in the Slicer Jupyter kernel using ipycanvas and ipyevents packages.
Level 4: Full desktop graphical user interface integration. Users can see parts of the application window rendered in notebook cells, including standard desktop widgets (sliders, menus, etc.). It is implemented using noVNC and TigerVNC in Slicer Jupyter. This is particularly useful when the application runs on a remote server.

Video demonstrating how to run 3D Slicer using Binder

These interactive tools allow developers to implement complete data processing workflows in a notebook, even if certain steps require manual user inputs such as segmenting 3D regions or setting seed points.

Try it online!

Since Jupyter notebooks can be used from any web browser, it can essentially turn any desktop application to a web application. By setting up a remote Jupyter server, users do not have to install anything on their computers. We have set up a demonstration of this using Binder (www.mybinder.org) that anybody can try at https://mybinder.org/v2/gh/Slicer/SlicerNotebooks/master.

Click on the binder image to launch the demo

The docker image that installs and configures Slicer and all dependencies (ipycanvas, ipyevents, VNC, etc.) is available at https://github.com/Slicer/SlicerDocker/tree/master/slicer-notebook. For deployment to non-technical users, applications can be deployed using Voilà (https://voila.readthedocs.io/), which only shows relevant content and interactive widgets, so the notebook looks like a simple dynamic web page.

The current implementation is already stable and offers a wide range of features, but there is still room for design and performance improvements. For example, we could not implement fully automatic conversion of application-specific data objects to displayable Python objects (due to complex implementation of display hooks); xeus-python debugger’s threading model needs to be improved to allow using it without locking the application’s main thread; and dynamic viewer widget’s performance (level 3 interaction) could be optimized to achieve higher refresh rates.

History of Slicer and this integration

Built over two decades with support from the NIH and a worldwide open source developer community, 3D Slicer is a unique, multi-platform desktop application for analysis, integration, and visualization of medical images that is heavily used by researchers globally for basic and applied research in a wide range of topics. The 3D Slicer Community includes physician-scientists with disease-specific knowledge of clinical challenges, computer scientists and physicists who develop novel algorithms, imaging informatics researchers, software engineers with the ability to understand clinical problems and create reliable tools, and application engineers with the multidisciplinary skills to deploy these tools in a range of cancer research settings. Slicer is maintained by Kitware, Inc., and the NAMIC consortium, and has an active open source software community. Slicer is used in hospitals and by researchers, with more than 10k academic citations and more than 150k downloads in the last year alone.

Click on the image to download 3D Slicer

It can be used for image processing workflows on 2D, 3D, and 4D images. What makes Slicer so useful is its huge set of community contributed modules that extend its functionality, as well as its ability to read DICOM and a wide array of exotic file formats.

Slicer has been scriptable in Python for well over a decade, but a robust SlicerJupyter extension is born from significant development effort by the community. The idea for this integration dates back to SciPy in 2014, when Jean-Christophe Fillion-Robin of Kitware organized a sprint intended to integrate Slicer and an IPython notebook, motivated by the dream of creating interactive and fun tutorials for Slicer. Mike Sarahan and Jean-Christophe created this proof of concept https://github.com/commontk/QEmbedIPython#qt-embed-ipython, but it was far from usable. In 2015 Matt McCormick of Kitware, created the SlicerDocker repository to support headless builds and rendering in a Docker image. Then in June 2018, while attending the Slicer Project Week, Andras Lasso (Queen’s University) and Jean-Christophe learned about Xeus, a C++ implementation of the Jupyter kernel protocol developed by QuantStack that would help streamline the integration of Slicer with Jupyter. To support this effort, Andras and Jean-Christophe created the Slicer/SlicerJupyter GitHub repository. They also contributed changes to Xeus to support this new integration paradigm where the event loop of the kernel is driven by a Qt-based Desktop application (see here). Building on this foundation, Isaiah Norton, then working at Brigham and Women’s Hospital, contributed additional improvements like a better auto-completion using jedi (see https://github.com/Slicer/SlicerJupyter/pull/12) as well as integration with Binder. More recently, Jean-Christophe and Sylvain Corlay (QuantStack) met after the Slicer project week while attending SciPy in Austin. Following the creation of a new project called xeus-python (started by Martin Renou at QuantStack), we took the integration of Jupyter and Slicer to the next level by adding support for improved interactive use and MRML data node visualization directly in the notebook.

About the authors

Alphabetically ordered

Sylvain Corlay is the founder and CEO of QuantStack, and a core Jupyter developer. He co-authored xeus and xeus-python.

Sylvain Corlay

Jean-Christophe Fillion-Robin is an open-source enthusiast, original author of the SlicerJupyter extension and a principal engineer at Kitware Inc where he leads the development of “3D Slicer” based commercial applications. J-Christophe also maintains scikit-build, an improved build system generator for CPython C/C++/Fortran/Cython extensions.

Jean-Christophe Fillion-Robin

Mike Grauer is a Technical Leader on the data and analytics team at Kitware. He is specialized in building scalable server-side processing frameworks that enable scientific workflows over web platforms.

Mike Grauer

Andras Lasso is an original author of the SlicerJupyter extension, Senior Research Engineer and Associate Director of the Laboratory for Percutaneous Surgery at Queen’s University.

Andras Lasso

Matt McCormick is an open source, medical imaging researcher working at Kitware Inc. Matt is an active, contributing member of scientific open source software efforts such as the Insight Toolkit (ITK) and scientific Python (SciPy) communities, and maintains the Jupyter 3D widget, itkwidgets.

Matt McCormick

Isaiah Norton is a Senior Software Developer at TileDB, Inc. with experience in digital pathology and image-guided surgical navigation.

Isaiah Norton

Steve Pieper is a Chief Architect and active developer of the 3D Slicer application for well over a decade. He is CEO of Isomics, Inc, where he uses a range of software technologies to perform medical imaging research with leading universities and companies.

Steve Pieper

Martin Renou is a scientific software developer at QuantStack. He is the original author of xeus-python, the xeus-based Python kernel, and contributed to the new concurrency model used for the debugger.

Martin Renou

Mike Sarahan is a software engineer at RStudio, PBC working on bringing language ecosystems together. He is passionate about making software work for data scientists in ways that are easy to maintain and improve.

Mike Sarahan

Interactive GIS in Jupyter with ipyleaflet

2019-09-24T22:54:00+00:00

As Jupyter grew in popularity, a broad ecosystem of visualization packages based on Jupyter widgets has been developed, bringing even more interactivity to the Jupyter world.

In this article, we dive into Jupyter Interactive Widgets and the ipyleaflet package, an interactive maps visualization system for Jupyter.

A non-interactive map.

Jupyter Widgets

Jupyter Interactive Widgets are “special objects” that can be instantiated by the user in their code and result in a counterpart component being created in the front-end.

The core ipywidgets package provides a collection of controls that Jupyter users can use to build simple UIs as part of their notebooks (sliders, buttons, dropdowns, layout components).

Basic ipywidgets controls.

More than a collection of controls, it provides a framework upon which a large ecosystem of components has been built, allowing notebook authors to capture user inputs in very diverse ways.

Popular libraries built upon interactive widgets include

bqplot, a 2-D plotting system for Jupyter,
ipyvolume, a 3-D plotting package based on WebGL and ThreeJS,
PythreeJS, a 3-D scene description package exposing a large part of the ThreeJS API to Jupyter,
ipyvuetify, a large collection of VuetifyJS components exposed to Jupyter,
ipywebrtc, a library exposing the features of the WebRTC protocol to Jupyter kernels,
beakerx, a collection of widgets, extensions, and kernels for Jupyter,

and many more… This list is not comprehensive and dozens of other widget packages have been developed.

Key aspects of Jupyter widgets include:

BidirectionalityWidgets are not just meant for display but can also be used to capture user inputs, which can then trigger new computation. Notebook authors can compose sophisticated applications including a variety of components from different packages.
Language AgnosticismBuilt upon the Jupyter ecosystem, the interactive widget protocol used for the synchronization between the kernel and the front-end is well-specified and can be implemented for any kernel.
Back-ends for other languages than Python already exist, such as for C++ (with the xeus-cling Jupyter kernel), and languages of the JVM such as Clojure or Groovy (with the beakerx kernels).
ExtensibilityJupyter widgets are not meant as a monolithic system with one and only one way to achieve a specific task. We strive to provide a foundational layer allowing third-party widget authors to be as inventive as possible.

A common pattern for Jupyter widget packages has been to bring the capabilities of popular JavaScript visualization frameworks to Jupyter with a bridge based on ipywidgets. This is the case for the ipyleaflet package, as well as pythreejs and ipyvuetify.

Another use case is the development of ad hoc controls that are not necessarily relevant for a mainstream visualization package, but may be specific to a scientific field. An example is the ipymesh project by Chris Kees which can be used to draw PSLG (planar straight-line graphs) in the Jupyter notebook.

ipyleaflet

ipyleaflet is a Jupyter - LeafletJS bridge, bringing mapping capabilities to the notebook and JupyterLab.

Built as a bridge between the LeafletJS package and Jupyter, the ipyleaflet API maps to that of LeafletJS, bringing most of the core features of the package to Jupyter, and enabling a few popular LeafletJS extensions. A small difference is that following the Python coding style, ipyleaflet makes use of snake_case instead of CamelCase for attribute names.

ipyleaflet features

The main components to the library are layers and controls, respectively items to be displayed on the map, and interactive widgets overlayed on the map area for greater interactivity.

The first thing for which you may want to change the default value are the zoom level, the position, or the base layer for the map.

Changing the basemap layer in an ipyleaflet map.

Maps can be interactively edited in the Jupyter notebook, by dynamically changing or adding layers. In this screenshot, we add a custom layer including a GeoJSON dataset.

Note: An alternative layer to GeoJSON is GeoData, which lets the user load the data in the form of a GeoPandas dataframe instead of raw GeoJSON.

Adding a GeoJSON dataset to an interactive map.

A number of simple primitives are available such as markers and heatmaps. In the following screencast, we show how primitive properties can be linked with other widgets, and used as means to take user input on a map:

Linking properties of ipyleaflet primitives to other Jupyter widgets.

The splitmap control can be used to compare to different set of ipyleaflet layers at the same location.

The splitmap control.

Another interesting layer is the velocity layer which can be used to display wind velocity data. This control can take data in the form of an xarray dataset.

The velocity layer.

More classical visualization tools are also available, such as choropleths.

A choropleth layer.

JupyterLab Integration with the Sidecar and Theming Support

ipyleaflet is well integrated with JupyterLab

ipyleaflet controls make use of the JupyterLab themes for coloring so that they don’t stand out when using e.g. a dark theme.

The support for JupyterLab themes in ipyleaflet.

ipyleaflet can be used in combination with the JupyterLab sidecar widget.

Using the sidecar widget in JupyterLab to display a map aside of the notebook.

Together with the sidecar, a map can be programmatically added to the right-side toolbar of the JupyterLab application, and interactively edited in the notebook. This prevents the back-and-forth scrolling often required to see how changes are reflected visually.

The workflow enabled by ipyleaflet in combination with the sidecar is similar to that of the geonotebook project by Christopher Kotfila, Jonathan Beezley, and Dan LaManna from Kitware).

The geonotebook (which was based on the classic notebook) provided a similar workflow to the lab sidecar.

A more advanced example

Combined with other widget libraries such as bqplot or the core ipywidget package, ipyleaflet users can easily compose more complex applications and dashboards. In the following screencast, we explore the wealth-of-nations dataset with a leafletmap, a bqplot line chart and a dropdown widget:

Exploring the wealth-of-nations dataset with ipyleaflet, bqplot and core ipywidgets.

Trying ipyleaflet online with mybinder

If you would like to try out ipyleaflet now, it is possible thanks to the binder project. Just click on the image below!

Interactive GIS in C++: xleaflet

As mentioned earlier, interactive widgets back-ends for other programming languages have be implemented. There is a C++ back-end for ipywidgets: xwidgets which is the building block for creating other C++ widgets back-ends.

xleaflet is the C++ - LeafletJS bridge that exposes almost the same API as ipyleaflet, only that you use it from a C++ interpreter! It is based upon xwidgets which brings the bidirectional communication with the front-end.

You can learn more about the xeus-cling Jupyter kernel, xwidgets and xleaflet by reading the following blogpost.

xleaflet is the C++ backend to the jupyter-leaflet integration.

Deploying ipyleaflet-based dashboards with Voilà

Voilà is a tool that turns Jupyter notebooks into standalone dashboards. Built upon the Jupyter stack, it inherits the language agnosticism of the ecosystem, and can be used to produce standalone applications based on ipyleaflet.

A companion project to Voilà is the Voilà gallery project, a public facing set up of JupyterHub serving Voilà dashboard. It is kindly hosted by OVH. You can check out the Voilà gallery at URL https://voila-gallery.org.

For more resources about voilà, check out

the original announcement of the project: And Voilà!
and for the gallery: A Gallery of Voilà Examples.

If you check out the Voilà gallery, don’t miss Jeremy Tuloup’s GPX loader demo!

Jupyter for Geo Sciences

Jupyter’s adoption is exploding in the GeoScience space. Notable projects building upon Jupyter include

JEODPP (JRC Earth Observation Data and Processing Platform) is a EU project providing petabyte scale storage and high-throughput computing capacities to facilitate large scale analysis of Earth Observation data. The main front-end to the platform is base on Jupyter. The mapping capability is based on ipyleaflet. End users can request custom visualization that are returned to them in the form of lazily computed tile layers.
Pangeo is a community platform providing open, reproducible, and scalable sciences. The Pangeo software ecosystem involves open source tools such as xarray, Iris, Dask, Jupyter, and many other packages.
It was recently announced that the NSF would be funding the UC Berkeley + NCAR “EarthCube” proposal “Jupyter meets the Earth: Enabling discovery in geoscience through interactive computing at scale”. The plan involves the development of interactive dashboards with Voilà and contributions to the Voilà codebase.

Jupyter-gmaps

We should mention the jupyter-gmaps project by Pascal Bugnion. Jupyter-gmaps is a bridge between Google maps and Jupyter. Just like ipyleaflet, jupyter-gmaps is built upon the jupyter interactive widgets framework but relies on Google maps for the display instead of LeafletJS library.

This is a high-quality widget by another core developer of ipywidgets. Pascal is also one of the people behind Voilà and the Voilà gallery.

Folium

The Folium project enables maps visualization in the Jupyter notebook. Just like ipyleaflet, it is based on LeafletJS. Folium was created by Rob Story and is now maintained by Frank Conengmo and Filipe Fernandes.

A key difference between Folium and ipyleaflet is that ipyleaflet is built upon ipywidgets and allows bidirectional communication between the front-end and the backend enabling the use of the map to capture user input, while Folium is meant for displaying static data only. Folium enables many LeafletJS extensions, some of which may not be available in ipyleaflet at the moment.

Acknowledgements

The ipyleaflet project was started in 2015 by Brian Granger, and funded by the ERDC. The further development by Sylvain Corlay and Martin Renou was supported by QuantStack.

We are grateful to OVH for kindly hosting the Voilà gallery.

Other key contributors to Voilà and ipyleaflet that should be thanked here are Maarten Breddels, Pascal Bugnion, and Yuvi Panda. We should also mention the recent contributions by Vasavan Thirusittampalam who worked on the full-screen control and better interoperability with geopandas.

About the Authors

Sylvain Corlay, and Martin Renou are Scientific Software Developers at QuantStack.

Sylvain Corlay

Sylvain Corlay is the founder and CEO of QuantStack.

As an Open Source Developer, Sylvain contributes to Project Jupyter in the areas of interactive widgets and language kernels and is a steering committee member of the Project. Beyond QuantStack, Sylvain serves as a member of the board of directors of the NumFOCUS foundation. He also co-organizes the PyData Paris Meetup.

Martin Renou

Martin Renou is a Scientific Software Developer at QuantStack. Prior to joining QuantStack, Martin also worked as a Software developer at Enthought. He studied at the French Aerospace Engineering School ISAE-Supaero, with major in autonomous systems and programming.

As an open source developer, Martin has worked on a variety of projects, such as SciviJS (a JavaScript 3-D mesh visualization library), Xtensor, and Xeus.

Jupyter meets the Earth

2019-09-09T17:48:00+00:00

By Lindsey Heagy and Fernando Pérez

We are thrilled to announce that the NSF is funding our EarthCube proposal “Jupyter meets the Earth: Enabling discovery in geoscience through interactive computing at scale” (pdf). The team working on this project consists of Fernando Pérez [1, 2, 3], Joe Hamman [4], Laurel Larsen [5], Kevin Paul [6], Lindsey Heagy [1], Chris Holdgraf [1, 2] and Yuvi Panda [7]. Our project team includes members from the Jupyter and Pangeo communities, with representation across the geosciences including climate modeling, water resource applications, and geophysics. Three active research projects, one in each domain, will motivate developments in the Jupyter and Pangeo ecosystems. Each of these research applications demonstrates aspects of a research workflow which requires scalable, interactive computational tools.

In this project we intend to follow the patterns that have made Jupyter an effective and successful platform: we will drive the development of computational machinery by concrete use cases from our own experience and research needs, and then find the appropriate points for extension, abstraction, and generalization. We are motivated to advancing research of contemporary importance in geoscience, and are equally committed to producing work that leads to broad impact, general use infrastructure that benefits scientists, educators, industry, and the general community.

The adoption of open languages such as Python and the coalescence of communities of practice around open-source tools, is visible in nearly every domain of science. This is a fundamental shift in how science is conducted and shared. In recent years, there have been several high-profile examples in which open tools from the Python and Jupyter ecosystems played an integral role in the research, from data analysis to the dissemination of results. These include the first image of a black hole from the Event Horizon Telescope team and the detection of gravitational waves by the LIGO collaboration. The utility of open-source software in projects like these and the success of open communities such as Pangeo, provide evidence of the force-multiplying impact of investing in an ecosystem of open, community-driven tools. Through this project, we will advance this open paradigm in geoscience research, while strengthening and improving the infrastructure that supports it. We made this argument when discussing the intended impacts of our proposal, and we are pleased that the NSF is investing in this vision.

Geoscience use cases

Given our project’s aims and approach, participating actively in domain research is crucial to our success. The following descriptions are meant to offer a flavour of the research questions we are tackling, each led by a geoscientist in the team.

CMIP6 climate data analysis (Hamman). The World Climate Research Program’s Coupled Model Intercomparison Project is now in its sixth phase and is expected to provide the most comprehensive and robust projections of future climate predictions. When complete, the archive is expected to exceed 18 PB in size. In the coming years, this collection of climate model experiments will form the basis for fundamental research, climate adaptation studies, and policy initiatives. While the CMIP6 dataset is likely to hold new answers to many pressing climate questions, the sheer volume of data is likely to present significant challenges to researchers. Indeed, new tools for scalable data analysis, machine learning, and inference are required to make the most out of these data.

Large-Scale Hydrologic Modeling (Larsen). Streamflow forecasts are a valuable tool for flood mitigation and water management. Creating these forecasts requires that a variety of data types be brought together including model-generated streamflow estimates, sensor-based observations of water discharge, and hydrometeorological forcing factors, such as precipitation, temperature, relative humidity, and snow-water equivalent. The integration of simulated and observed data over disparate spatial and temporal scales creates new avenues for exploring data science techniques for effective water management.

Geophysical inversions (Heagy). Geophysical inversions construct models of the subsurface by combining simulations of the governing physics with optimization techniques. These models are critical tools for locating and managing natural resources, such as groundwater, or for assessing the risk from natural hazards, such as volcanoes. Today, we need models applicable to increasingly complex scenarios, such as the socially delicate task of developing groundwater management policies in water-limited regions. This will require the development of new techniques for combining multiple geophysical data sets in a joint inversion, as well as the use of statistical and data science methods for including geologic and hydrologic data in the construction of 3D models.

These scientific problems exhibit, each with its own flavour, similar technical challenges with respect to handling large volumes of data, performing expensive computations, and doing both of these as a part of the interactive, exploratory workflow that is necessary for scientific discovery.

Jupyter & Pangeo: empowering scientists

Jupyter and Pangeo are both open communities that share the goal of developing tools and practices for interactive scientific workflows; these tools aim to deliver practical value to scientists who, in the course of everyday research, face a combination of big data and large-scale computing needs.

Project Jupyter creates open-source tools and standards for interactive computing. These span the spectrum from low-level protocols for running code interactively up to the web-based JupyterLab interface that a researcher uses. Jupyter is agnostic of programming language: over 130 different Jupyter kernels exist, and they provide support for most programming languages in widespread use today. Jupyter can be run on a laptop, in an HPC center, or in the cloud. Shared-infrastructure deployments (e.g. HPC / cloud) are enabled by JupyterHub, a component in the Jupyter toolbox that supports the deployment and management of Jupyter sessions for multiple users. The development process of tools in the Jupyter ecosystem is community-oriented and includes a diverse set of stakeholders across research, education, and industry. The project has a strong tradition of building tools that are first designed to solve specific problems, and then generalized to other users and applications.

A Pangeo Platform is a modular composition of open, community-driven projects, tailored to the scientific needs of a specific scientific domain. In its simplest form, it is based on the following generic components: a browser-based user interface (Jupyter), a data model and analytics toolkit (Xarray), a parallel job distribution system (Dask), a resource management system (either Kubernetes or a job queuing system such as PBS), and a storage system (either cloud object store or traditional HPC file system). These are complemented by problem- and domain-specific libraries.

This modular design allows for individual components to be readily exchanged and the system to be applied in new use cases. Pangeo was created by, and for, geoscientists faced with large-scale data and computation challenges, but such problems are now common in science. Researchers in a variety of disciplines including neuroscience and astrophysics are working to adapt the Pangeo design pattern for their communities. Beyond the initial Pangeo deployments supported by the NSF EarthCube grant for Pangeo, the platform has been adopted internationally, including by the UK Met office. It has also supported new research and education efforts from other federal agencies, such as NASA’s HackWeek focused on the analysis of ICESat-2 data.

User-Centered Development

Pushing the boundaries of any toolset unveils areas for improvement and opportunities for developments that streamline and upgrade the user experience. We aim to take a holistic view of the scientific discovery process, from initial data acquisition through computational analysis to the dissemination of findings. Our development efforts will make technological improvements within the Jupyter and Pangeo ecosystems in order to reduce pain-points along the discovery lifecycle and to advance the infrastructure that serves scientists. Following established patterns in Jupyter’s development, we take a user-first, needs-driven approach and then generalize these ideas to work across related fields. Broadly, there are 4 areas along the research lifecycle where we will invest development efforts, which we discuss next.

Data discovery. An early step in the research process is locating and acquiring data of interest. Data catalogs provide a way to expose datasets to the community in a way that is structured. Within the geosciences, there are a number of emerging community standards for data catalogs (e.g. THREDDS, STAC). To streamline access to such data sets, we plan to develop JupyterLab extensions which provide a user-interface that exposes these catalogs to researchers. This work will build upon the JupyterLab Data Registry, which will provide a consistent set of standards for how data can be consumed and displayed by extensions in the Jupyter ecosystem, as well as Intake, a lightweight library for finding, loading, and sharing data which is already serving the Pangeo community. Our communities are already discussing potential avenues for integration between these tools.

Scientific discovery through interactive computing. The Jupyter Notebook has been adopted by many scientists because it supports an iterative, exploratory workflow combining code and narrative. Beyond code, text, and images, Jupyter supports the creation of Graphical User Interfaces (GUIs) with minimal programming effort on the part of the scientist. The Jupyter widgets framework lets scientists create a “Research GUI” that combines scientific code with interactive elements such as sliders, buttons and menus in just a single line of code, while still allowing for extensive customization and more complex interfaces when required. In this project, we will develop custom widgets tailored at the specific scientific needs of each of our driving use cases.

Beyond their utility in the exploratory phase of research, interactive interfaces, or “dashboards” provide a mechanism for delivering custom scientific displays to collaborators, stakeholders, and students for whom the details of the code may not be pertinent. Voilà is a project, led by the QuantStack team, that enables dashboards to be generated from Jupyter notebooks. We plan to develop interactive dashboards using Voilà for our geoscience use-cases, contribute generic improvements to the Voilà codebase, and provide a demonstration of how researchers can deploy dashboards to share their research.

Research GUIs to explore Maxwell’s equations in research and education. Photo credit: SEOGI KANG

Established tools and data visualization. Many widely-used tools, particularly for visualization (e.g. Ncview, Paraview), are desktop-based applications and therefore cannot easily be used in cloud or HPC workflows. In some cases, modern, open-source alternatives are available. But often for specialized tasks, modern tools may not yet have functionality equivalent to the desktop version. JupyterHub can readily serve non-Jupyter web-native software applications such as RStudio, Shiny applications, and Stencila to users; under this project we aim to extend JupyterHub to also be able to serve desktop-native applications.

Using and managing shared computational infrastructure. JupyterHub makes it possible to manage computing resources, user accounts, and provide access to computational environments online. Currently, JupyterHubs in the Pangeo project are deployed and maintained using the Zero to JupyterHub guide alongside the HubPloy library. Together, these libraries have simplified the initial setup and automated upgrades to the Hubs. There are still many improvements that can make JupyterHub more suitable for larger, more complex deployments, both in terms of managing users and efficiently allocating resources. Under this project, we plan to build tools which collect metrics such as CPU and memory usage and expose these to both users and administrators so they can make more efficient use of the Hub. For shared deployments, we will improve user management so that user-groups can be used to manage permissions and allocations, and so that usage can be tracked and appropriately billed to the relevant grants. Within the HubPloy library, we also plan to make improvements to streamline continuous deployments so that installation and upgrade processes are repeatable and reliable.

An opportunity for meaningful impact — join us!

The impacts of climate change and the need for data-driven management of resources are some of the most critical and complex challenges facing society today. In recent years, we have experienced severe droughts in California and are in the midst of water management crises in the Central Valley, while devastating wildfires have destroyed entire communities. Through this partnership between researchers and Jupyter developers, we hope to contribute to the advancement of science-based solutions to these challenges, both by contributing directly to the research and by improving the open ecosystem of tools available to researchers and the stakeholders impacted by these issues.

If developing open-source tech to advance research in geoscience and beyond excites you, then please get in touch! At UC Berkeley, we will be hiring in 2 positions: a dev-ops position focussed on JupyterHub and shared infrastructure deployments, and a JupyterLab-oriented role focussed on extensions, dashboards, and interactivity. There will also be a position opening up at NCAR for a software engineer targeting improvements to improving the user experience of Xarray and Dask workflows.

Even if you aren’t looking for a new job, there are other ways to get involved with both the Jupyter and Pangeo communities. We welcome new participants to the weekly Pangeo meetings (on Wednesdays alternating between 4p GMT and 8p GMT) and there are monthly Jupyter community calls, which are open and meant to be accessible to a wide audience. Outside of calls, general Jupyter conversations happen on the Jupyter discourse and Pangeo conversations are typically on the Pangeo GitHub.

In closing: a step toward sustainable open science

This project provides our team with $2 Million in funding over 3 years as a part of the NSF EarthCube program. It also represents the first time federal funding is being allocated for the development of core Jupyter infrastructure.

The open source ecosystem that Jupyter and Pangeo belong to has become part of the backbone that supports much of today’s computation in science, from astronomy and cosmology to microbiology and subatomic physics. Such broad usage represents a victory for this open and collaborative model of building scientific tools. Much of this success has come through the efforts of scientists and engineers who are committed to an open model of science, but who have had to work with little direct funding, minimal institutional support, and few viable career paths within science.

There is real strategic risk to continuing with the implicit assumption that scientific open-source tools can be developed and maintained “for free.” If open, community-driven tools are to sustainably grow into the computational backbone of science, we need to recognize this role and support those who create them as regular members of the scientific community (we recently talked about this in more detail in a talk at NSF headquarters). Projects like ours, where funding and resources are explicitly allocated toward this goal, are a step in the right direction. We hope that our experiences will contribute to ongoing conversations in the scientific community around these complex issues.

In the past, we have tried to maintain a close relationship between domain problems and software development in Jupyter. However, this has typically been done in an ad-hoc manner, either by “hiding” the software development under the cover of science or by having funding to Jupyter alone. This is the first project where we explicitly partner with a team of domain scientists to simultaneously drive forward domain research and the development of Jupyter infrastructure.

We are excited about this opportunity and hope to be able to demonstrate that investing in open tools can be a force-multiplier of resources. As always, our work will be done openly, transparently, and with constant community engagement. We look forward to your critiques, ideas, and contributions to make this effort as successful as possible.

Acknowledgments

Thanks to Joe Hamman, Chris Holdgraf, and Doug Oldenburg for constructive feedback and edits on this blog post.

Many thanks to Ryan Abernathey (Columbia), Paul Bedrosian (USGS), Sylvain Corlay (QuantStack), Rich Signell (USGS), and Rollin Thomas (NERSC), who provided us with letters of support for this project; we look forward to working with you all! We are also grateful to Shree Mishra, our NSF Program Director on this project, and to Dave Stuart for their support as we move forward with the project. This project is a part of the to the EarthCube program and we look forward to engaging with its working group.

Finally, the sustained growth of Jupyter to the large-scale project that it has become would not have happened without the generous support of the Alfred P. Sloan, the Gordon and Betty Moore, and the Helmsley Foundations, as well as the leadership of Josh Greenberg and Chris Mentzel respectively at Sloan and Moore.

This work is supported by the NSF EarthCube program under awards 1928406, 1928374

Affiliations

[1] UC Berkeley, Statistics Department

[2] UC Berkeley, Berkeley Institute for Data Science

[3] Lawrence Berkeley National Lab, Computational Research Division

[4] National Center for Atmospheric Research, Climate and Global Dynamics Laboratory

[5] UC Berkeley, Department of Geography

[6] National Center for Atmospheric Research, Computational Information Systems Laboratory

[7] UC Berkeley, Division of Data Sciences

National Scale Interactive Computing

2019-08-22T19:06:00+00:00

This is an invited post from Jim Colliander, Professor of Mathematics at UBC and Director of the Pacific Institute for the Mathematical Sciences.¹

In 2017, the Pacific Institute for the Mathematical Sciences (PIMS), in partnership with Compute Canada and Cybera, launched Syzygy, a cloud-hosted interactive computing platform that delivers JupyterHub deployments for universities across Canada.

Syzygy has been used by over 16,000 students at 20 universities. The main results of the Syzygy experiment so far are:

Demand for interactive computing is ubiquitous² and growing strongly at universities.
The Jupyter ecosystem is an effective way to deliver interactive computing.
A scalable, sustainable, and cost-effective interactive computing service for universities is needed as soon as possible.

Demand for interactive computing

Both research and teaching at universities are adapting to major societal changes driven by explosions in data and computational tools. New educational programs that prepare students to think computationally are emerging, while research strategies are changing in ways that are more open, reproducible, collaborative, and interdisciplinary. These transformations are inextricably linked and are accelerating demand for interactive computing. The Syzygy experiment has shown that using Jupyter in educational programs drives interest in using Jupyter for research (and vice versa). For example, students in mathematics, statistics, and computer science collaborated with a researcher from St. Paul’s Hospital in Vancouver using Syzygy to identify new pathways to prevent death from sepsis. Research communities typically need access to deeper computational resources and often span multiple universities, but the common thread is the need to expand access to interactive computing.

A map of JupyterHub deployments deployed by Syzygy.

Technical milestone achieved

The Syzygy project has demonstrated that it’s possible to deploy tools for interactive computation at a national scale rapidly and efficiently using an entirely open source technology stack. Students, faculty and staff across Canada use Syzygy to access Jupyter through their browsers with their university single-sign-on credentials. The JupyterHubs range from a “standard” configuration to bespoke environments with specially curated tools and data integrations. This richness is possible because of the architecture of the Syzygy and Jupyter projects and the flexibility of the underlying cloud resources. As a case-study, Syzygy demonstrates that the Jupyter community has achieved a significant technical milestone: interactive computing can be delivered at national scale using cloud technologies.

Service level requirements

The validation that interactive computing can be technically delivered at national scale prompts universities to ask a variety of questions. Can interactive computing service be delivered robustly? How will users be supported? What are the uptime expectations? What is the data security policy? How is privacy protected? Can the robustness of the service be clarified in a service level agreement? Syzygy, as an experimental service offered to universities at no charge and without a service level agreement, does not properly address these questions. To advance on their education-research-service mission and address growing demand, universities need a reliable interactive computing service with a service level agreement.

What’s next?

How should universities address their needs for interactive computing over the next five years? Right now, universities are following two primary approaches:

🙏 Ad hoc: faculty figure out how to meet their own needs for interactive computing; IT staff deploys JupyterHub on local or commercial cloud servers; this approach gives universities control over their deployments and hardware, though requires time and expertise that many may not have.
🎩 Use a cloud provider’s service: Google Colab, Amazon Sagemaker, Microsoft Azure Notebooks, IBM Watson Studio; this approach allows universities to quickly launch interactive computing services, with a loss of flexibility and some risks by becoming reliant upon a particular vendor’s closed-source and proprietary software.

These approaches are not sustainable over the long term. If universities all deploy their own JupyterHub services, many will need technical expertise they do not currently have and will involve a significant duplication of effort. If universities rely on hosted cloud notebook services, the reliance on proprietary technology will impair their ability to switch between different cloud vendors, change hardware, customize software, etc. Vendor lock-in will limit the ability of universities to respond to changes in price for the service. Universities will lose agility in responding to changes in faculty, staff, and student computing needs.

There is a third option that addresses the issues with these two approaches and generates other benefits for universities:

🤔 Form an interactive computing consortium: universities collaborate to build an interactive computing service provider aligned with their missions to better serve their students, facilitate research, and avoid risks associated with vendor lock-in.

To retain control over their interactive computing stacks, avoid dependence⁴ on cloud providers, and accelerate the emergence of new programs, universities should work together to deploy interactive computing environments in a vendor-agnostic manner. This might take the form of a consortium — an organization dedicated to serving the needs of universities through customized shared infrastructure for interactive computing. The consortium would also ensure that universities will continue to play a leadership role in the development of the interactive computing tools used for education and research.

The Syzygy experiment confirmed that growing demand for interactive computation within universities can be supplied with the available technologies advanced by the Jupyter open source community. In the coming year, we aim to build upon the success of the Syzygy experiment and seed an initial node of a consortium in Canada with the intention of fostering a global network of people invested in advanced interactive computing. If you are interesting in partnering, please get in touch! See this Jupyter Community Forum post to continue the discussion.

The author gratefully acknowledges feedback on this piece from Ian Allison, Lindsey Heagy, Chris Holdgraf, Fernando Perez, and Lindsay Sill.
Interactive computing needs have been identified in agriculture, applied mathematics, astronomy, chemistry, climate science, computer science, data science, digital humanities, ecology, economics, engineering, genomics, geoscience, health sciences, K-12 education, neuroscience, political science, physics, pure mathematics, statistics, and sociology.
Relying on commercial cloud vendors to provide the interactive computing service for universities risks recreating the problems associated with scientific publishing that emerged with the internet.

Jupyter for Science User Facilities and High Performance Computing

2019-07-09T23:53:00+00:00

Jupyter is the “Google Docs” of data science. It provides that same kind of easy-to-use ecosystem, but for interactive data exploration, modeling, and analysis. Just as people have come to expect to be able to use Google Docs everywhere, scientists assume that Jupyter is there for them whenever and wherever they open their laptops.

But what if the data you want to interact with through Jupyter doesn’t fit on your laptop or is excruciating to move? What if the model you want to build and test requires more computing power and storage than you have right in front of you? As a scientist, you want the same interactive experience and all the benefits of Jupyter, but you also need to “reach out” to put something big into your science process: A supercomputer, a telescope data archive, a beam-line at a synchrotron. Can Jupyter help you do that big science? What efforts are in motion already to make this a reality, what work still needs to be done, and who needs to do it?

Doing this right will take a community: New collaborations between core Jupyter developers, engineers from high-performance computing (HPC) centers, staff from large-scale experimental and observational data (EOD) facilities, users and other stakeholders. Many facilities have figured out how to deploy, manage, and customize Jupyter, but have done it while focused on their unique requirements and capabilities. Still others are just taking their first steps and want to avoid reinventing the wheel. With some initial critical mass, we can start contributing what we’ve learned separately into a shared body of knowledge, patterns, tools, and best practices.

40+ participants from universities, national labs, industry, and science user facilities. Credit: Fernando Perez.

In June, a Jupyter Community Workshop held at the National Energy Research Scientific Computing Center (NERSC) and the Berkeley Institute for Data Science (BIDS) brought about 40 members of this community together to start distilling. Over three days in talks and breakout sessions, we addressed pain points and best practices in Jupyter deployment, infrastructure, and user support; securing Jupyter in multi-tenant environments; sharing notebooks; HPC/EOD-focused Jupyter extensions; and strategies for communication with stakeholders.

Here are just a few highlights from the meeting:

Michael Milligan from the Minnesota Supercomputing Center perfectly set the tone for the workshop with his keynote, “Jupyter is a One-Stop Shop for Interactive HPC Services.” Michael is the creator of BatchSpawner and WrapSpawner, JupyterHub Spawners that let HPC users run notebooks on compute nodes supporting a variety of batch queue systems. Contributors to both packages met in an afternoon-long breakout to build consensus around some technical issues, start managing development and support in a collaborative way, and gel as a team.
Securing Jupyter is a huge topic. Thomas Mendoza from Lawrence Livermore National Laboratory talked about his work to enable end-to-end SSL in JupyterHub and best practices for securing Jupyter. Outcomes from two breakouts on security include a plan to more prominently document security best practices, and a future meeting (perhaps another Jupyter Community Workshop?) focused specifically on security in Jupyter.
Speakers from Lawrence Livermore and Oak Ridge National Laboratories, the European Space Agency showed off a variety of beautiful JupyterLab extensions, integrations, and plug-ins for climate science, complex physical simulations, astronomical images and catalogs, and atmospheric monitoring. People at a variety of facilities are finding ways to adapt Jupyter to meet the specific needs of their scientists.

Really, there’s just too much to pack into a blog post so we encourage you to look at the talk slides and notes on Discourse — all the breakout notes have been posted there to this topic. We’re working on getting videos of the slide presentations up on the workshop website as well. Watch for announcements of future meeting opportunities and documentation on Discourse as well.

Finally we want to thank Project Jupyter, NumFOCUS, and Bloomberg for their help making this meeting happen. We all came away with a better sense of who is doing what in our community, and how we can work together on this new area of growth for the Jupyter community. The organizers also want to thank their respective institutions’ administrative staff (Seleste Rodriguez at NERSC, and Stacy Dorton at BIDS) for helping with workshop logistics.

Kellie Ottoboni Receives Award for Post-Election Auditing Notebook

2019-04-02T17:01:00+00:00

On Thursday, March 14th at their annual conference in Washington DC, Kellie Ottoboni received the Innovation Award from the Election Verification Network (EVN) for her work with Professor Philip Stark developing the Stratified Union-Intersection Tests of Elections (SUITE) tool for use in the pilot of risk-limiting audits. The SUITE tool is a Jupyter notebook that encompasses all steps of a post-election risk-limiting audit: risk calculations, an estimate of the number of ballots to examine, and a procedure to sample and locate ballots. An earlier post on the blog describes the tool in greater detail and how it was used in Michigan’s 2018 pilot risk-limiting audits.

The Election Verification Network is a professional society of election officials, researchers, and advocates committed to accurate, accessible, transparent, reliable, and verifiable elections. The Innovation Award is given annually for path-breaking product development that greatly facilitates robust election verification and auditing.

Jupyter notebooks for post-election audits

2019-01-22T16:01:00+00:00

The following is a guest blog post authored by Kellie Ottoboni, describing her recent work where they used Jupyter to support statistical audits of election results.

In December 2018, I facilitated pilot post-election risk-limiting audits in three cities in Michigan. This was the first time that SUITE, a new method for “hybrid” risk-limiting audits that I helped develop, has been used in practice. I wrote a Jupyter notebook tool to do the SUITE risk calculations, to determine the necessary sample size, and to sample the ballots using a cryptographically secure pseudo-random number generator.

Twenty 10-sided dice were rolled to select the seed for the pseudo-random number generator.

What are risk-limiting audits?

A risk-limiting audit (RLA) is a statistical check that the reported outcome (the reported winner(s), as opposed to exact vote totals) of an election is correct. The procedure limits the chance that an incorrect outcome will go uncorrected, if the reported outcome is in fact wrong. RLAs involve sampling and examining paper ballots. 34 RLA pilots have been done in California, Colorado, Indiana, New Jersey, Ohio, Virginia, and Denmark. Colorado began requiring RLAs by law in 2017, and Rhode Island will begin requiring RLAs in 2019.

SUITE is a general method for conducting RLAs of stratified samples, where a population of ballots is divided into non-overlapping strata and samples are drawn independently from each stratum. We began developing SUITE for Colorado, treating absentee votes and in-precinct votes as two strata. Aside from this special case, SUITE may be useful for auditing states where counties work independently.

Michigan used the specific two-stratum version of SUITE, which combines two types of RLAs: ballot polling, which involves “polling” the ballots and recording the proportion of votes for each candidate, and ballot-level comparison, which involves comparing paper ballots to their electronic record and counting the number of discrepancies. Ballot-level comparison audits require looking at fewer ballots than ballot polling, but can only be done when ballots can be matched to their electronic records. In two of the Michigan cities, absentee ballots could be matched to their electronic record. SUITE allows you to use these two strategies side-by-side for a single RLA.

The Jupyter tool

Previous RLA pilots have used HTML pages with Javascript code, written by Philip Stark, to conduct the audits. With a short timeframe to create a similar SUITE tool, Philip and I decided that the best solution was to build out the Python library we had begun and use a Jupyter notebook for the interface.

The final results of the audit in Kalamazoo were shown in the Jupyter notebook.

The SUITE notebook made the audits more transparent. In each city, I projected the notebook running locally on my laptop on a screen for about 30 local election officials. They could observe me entering the reported vote totals for each candidate and the 20-digit random seed to initialize the pseudo-random number generator, then see the sampled ballots appear on the screen in a nicely formatted table. These folks are used to working in Excel spreadsheets, so the interactivity of a Jupyter notebook is more familiar interface than simply running an executable file in the terminal. Each step of the audit was interactive and annotated with Markdown.

Our hope is that the SUITE notebook can serve as a proof of concept for further development. Free & Fair turned the HTML RLA tools into industrial strength software and Democracy Works built upon it to create the program that Colorado uses for statewide RLAs.

Pitfalls

One main difference between the HTML RLA tools and the SUITE notebook is that the webpages hide all the code. While we moved most of the code to modules and limited most code cells to one function, the notebook still contained a lot of code to scroll through. Voila, a tool for hiding code in notebooks with interactive widgets, is one possible solution to display only the crucial input and output pieces of the tool.

Another issue we faced was the sequential nature of notebooks. RLAs are iterative: if the risk of the ballots from a first round of sampling is too high, then the audit proceeds to more rounds, until either the risk is sufficiently small or all ballots have been counted. A more proficient software developer than I might have come up with an elegant way to make this possible in a Jupyter notebook while tracking the data from each round of sampling. My hack was to assume that in Michigan we would need at most two rounds of sampling and to copy the code for the first round, with some modifications. (In fact, none of the three cities used more than one round.)

The RLA tools written by Free & Fair and Democracy Works are written in Java and Clojure. Academics continue to improve the statistics, so it would be helpful to have an API accessible with Python. It would enable the open source community to contribute to the codebase rather than reinvent the wheel to build a tool every time new statistics are developed.

Michigan pilots

Rochester Hills, Lansing, and Kalamazoo participated in the pilots of their November, 2018 election. I traveled to each city with a team of election auditing experts from MIT, the NYU Brennan Center for Justice, the Electoral Assistance Commission, and Democracy Works. We hope that this Jupyter notebook helped demystify the code and math, and illustrate that RLAs are feasible and efficient way to insure election integrity.

Interpreted C++ for GIS with Jupyter

2018-04-24T08:35:00+00:00

A live interactive map in a C++ Jupyter notebook

The recent release of the Jupyter kernel for C++, based on the Cling interpreter enabled a number of new workflows for the users of the C++ programming language.

Features of the xeus-cling C++ kernel for Project Jupyter include:

Showing quick-help pages for functions and classes of the STL and user-defined types, by prefixing them with a question mark:
For example, typing ?std::vector results in a pager displaying the page from cppreference on std::vector.

Quick-help page for classes and functions of the STL

Making use of the rich display features of the Jupyter stack, for user-defined types.
This can be enabled simply by overloading mime_bundle_repr in the namespace of the class for which we wish to have a rich representation in the front-end. The overload is picked up by the display system through argument-dependent lookup (ADL).

Using Jupyter’s rich display mechanism in C++

Another aspect of the newly released C++ kernel is the implementation of the Jupyter widgets protocol, enabling bi-directional communication between the front-end and the kernel.The xwidgets package, built upon xeus provides a complete implementation of the protocol, together with the implementation of most of the controls available in the reference ipywidgets Python package.

Bidirectional communication with the front-end using Jupyter interactive widgets

More than a limited set of base controls, Jupyter widgets are a framework upon which one can build arbitrarily complex interactions. A large number of interactive widget libraries has been built upon ipywidgets. Popular examples include pythreejs (a Jupyter-threejs bridge), bqplot (an interactive plotting library for Jupyter), and ipyleaflet (a Jupyter-leafletjs bridge) allowing rich interactive maps in the Jupyter notebook.

A common trait of most of these packages is that most of the logic is implemented in the front-end, while the back-end only involves synchronization of data attributes.

A fully-specified communication protocol and a thin back-end architecture facilitate the job of kernel authors willing to bring the power of these visualization libraries to their language of choice.

Hence, we have taken on the endeavor of providing a C++ implementation of the most popular Jupyter interactive widget libraries. These packages can be used in the C++ kernel, as well as in compiled application making use of the Jupyter kernel protocol.

Today, we are proud to announce the first release of xleaflet, the C++ counterpart to the popular ipyleaflet package, and which makes use of the same front-end component.

You can get started by simply creating a map inline in the Jupyter notebook.

Specifying a center location and zoom level
Specifying the tile layers to be displayed among the predefined base maps

A simple map with a specified center and zoom level, displaying the default tiles

A number of other attributes can be set in the map widget. To mimic named parameters, all widgets of xwidgets and xleaflet are provided with a generator class which can be used to initialize attributes using method-chaining syntax.

Making use of the generator class to specify any number of attributes of the map upon construction

In addition to the base map feature, a broad number of features of the leaflet JavaScript library are exposed to the C++ backend directly. This includes markers, marker clusters, image overlays, a variety of controls.

Using the marker widget

Whenever an attribute of a widget is modified in the front-end or in the back-end, the other side will properly reflect the data change.

For example, setting marker.location to a new value in the previous example will actually move the marker on the map. Reversely, if the draggable attribute was set to true, whenever the marker position changes in the front-end, the value is reflected in the C++ model.

Observer on the marker position

Another example is the support of the GeoJSON format, which allows one to load a JSON file locally and display its content on the map.

Support for the GeoJSON format

The bidirectional communication between the front-end and the C++ back-end makes it easier for the end user to create interactive web applications without having to write any JavaScript.

Using the rich features of xleaflet, one can start building fully-fledged GIS application in C++.

If you are interested in trying xleaflet right now in your web browser, we provided a binder for you.

Simply click on the following binder link and start playing with interactive GIS in C++ in your web browser:

Click on the image to launch the live demo.

Check out the documentation for more detailed information about xleaflet.

Aknowledgements

The software presented in this post was built upon the work of a large number of people including the Jupyter team, the Cling developers, the developers of xeus and xwidgets, and the developers of leafletjs.

We are especially grateful to Sylvain Corlay, Loic Gouarin, Johan Mabille, and Wolf Vollprecht.

The development of xeus, xwidgets and related packages at QuantStack is sponsored by Bloomberg.

About the Author

Martin Renou is a Scientific Software developer at QuantStack. Prior to joining QuantStack, Martin studied at the French Institute of Aeronautics and Space. As an open source developer, he worked on a variety of projects, notably SciviJS, a JavaScript library for 3-D mesh visualization.

Interactive Workflows for C++ with Jupyter

2017-11-29T16:33:00+00:00

Scientists, educators and engineers not only use programming languages to build software systems, but also in interactive workflows, using the tools available to explore a problem and reason about it.

Running some code, looking at a visualization, loading data, and running more code. Quick iteration is especially important during the exploratory phase of a project.

For this kind of workflow, users of the C++ programming language currently have no choice but to use a heterogeneous set of tools that don’t play well with each other, making the whole process cumbersome, and difficult to reproduce.

We currently lack a good story for interactive computing in C++.

In our opinion, this hurts the productivity of C++ developers:

Most of the progress made in software projects comes from incrementalism. Obstacles to fast iteration hinder progress.
This also makes C++ more difficult to teach. The first hours of a C++ class are rarely rewarding as the students must learn how to set up a small project before writing any code. And then, a lot more time is required before their work can result in any visual outcome.

Project Jupyter and Interactive Computing

The goal of Project Jupyter is to provide a consistent set of tools for scientific computing and data science workflows, from the exploratory phase of the analysis to the presentation and the sharing of the results. The Jupyter stack was designed to be agnostic of the programming language, and also to allow alternative implementations of any component of the layered architecture (back-ends for programming languages, custom renderers for file types associated with Jupyter). The stack consists of

a low-level specification for messaging protocols, standardized file formats,
a reference implementation of these standards,
applications built on top of these libraries: the Notebook, JupyterLab, Binder, JupyterHub
and visualization libraries integrated into the Notebook and JupyterLab.

Adoption of the Jupyter ecosystem has skyrocketed in the past years, with millions of users worldwide, over a million Jupyter notebooks shared on GitHub and large-scale deployments of Jupyter in universities, companies and high-performance computing centers.

Jupyter and C++

One of the main extension points of the Jupyter stack is the kernel, the part of the infrastructure responsible for executing the user’s code. Jupyter kernels exist for numerous programming languages.

Most Jupyter kernels are implemented in the target programming language: the reference implementation ipykernel in Python, IJulia in Julia, leading to a duplication of effort for the implementation of the protocol. A common denominator to a lot of these interpreted languages is that the interpreter generally exposes a C API, allowing the embedding into a native application. In an effort to consolidate these commonalities and save work for future kernel builders, we developed xeus.

Xeus is a C++ implementation of the Jupyter kernel protocol. It is not a kernel itself but a library that facilitates the authoring of kernels, and other applications making use of the Jupyter kernel protocol.

A typical kernel implementation using xeus would in fact make use of the target interpreter as a library.

There are a number of benefits of using xeus over implementing your kernel in the target language:

Xeus provides a complete implementation of the protocol, enabling a lot of features from the start for kernel authors, who only need to deal with the language bindings.
Xeus-based kernels can very easily provide a back-end for Jupyter interactive widgets.
Finally, xeus can be used to implement kernels for domain-specific languages such as SQL flavors. Existing approaches use a Python wrapper. With xeus, the resulting kernel won’t require Python at run-time, leading to large performance benefits.

Interpreted C++ is already a reality at CERN with the Cling C++ interpreter in the context of the ROOT data analysis environment.

As a first example for a kernel based on xeus, we have implemented xeus-cling, a pure C++ kernel.

Redirection of outputs to the Jupyter front-end, with different styling in the front-end.

Complex features of the C++ programming language such as, polymorphism, templates, lambdas, are supported by the cling interpreter, making the C++ Jupyter notebook a great prototyping and learning platform for the C++ users. See the image below for a demonstration:

Features of the C++ programming language supported by the cling interpreter

Finally, xeus-cling supports live quick-help, fetching the content on cppreference in the case of the standard library.

Live help for the C++standard library in the Jupyter notebook

We realized that we started using the C++ kernel ourselves very early in the development of the project. For quick experimentation, or reproducing bugs. No need to set up a project with a cpp file and complicated project settings for finding the dependencies… Just write some code and hit Shift+Enter.

Visual output can also be displayed using the rich display mechanism of the Jupyter protocol.

Using Jupyter’s rich display mechanism to display an image inline in the notebook

Another important feature of the Jupyter ecosystem are the Jupyter Interactive Widgets. They allow the user to build graphical interfaces and interactive data visualization inline in the Jupyter notebook. Moreover it is not just a collection of widgets, but a framework that can be built upon, to create arbitrary visual components. Popular interactive widget libraries include

bqplot (2-D plotting with d3.js)
pythreejs (3-D scene visualization with three.js)
ipyleaflet (maps visualization with leaflet.js)
ipyvolume (3-D plotting and volume rendering with three.js)
nglview (molecular visualization)

Just like the rest of the Jupyter ecosystem, Jupyter interactive widgets were designed as a language-agnostic framework. Other language back-ends can be created reusing the front-end component, which can be installed separately.

xwidgets, which is still at an early stage of development, is a native C++ implementation of the Jupyter widgets protocol. It already provides an implementation for most of the widget types available in the core Jupyter widgets package.

C++ back-end to the Jupyter interactive widgets

Just like with ipywidgets, one can build upon xwidgets and implement C++ back-ends for the Jupyter widget libraries listed earlier, effectively enabling them for the C++ programming language and other xeus-based kernels: xplot, xvolume, xthreejs…

xplot is an experimental C++ back-end for the bqplot 2-D plotting library. It enables an API following the constructs of the Grammar of Graphics in C++.

In xplot, every item in a chart is a separate object that can be modified from the back-end, dynamically.

Changing a property of a plot item, a scale, an axis or the figure canvas itself results in the communication of an update message to the front-end, which reflects the new state of the widget visually.

Changing the data of a scatter plot dynamically to update the chart

Warning: the xplot and xwidgets projects are still at an early stage of development and are changing drastically at each release.

Interactive computing environments like Jupyter are not the only missing tool in the C++ world. Two key ingredients to the success of Python as the lingua franca of data science is the existence of libraries like NumPy and Pandas at the foundation of the ecosystem.

xtensor is a C++ library meant for numerical analysis with multi-dimensional array expressions.

xtensor provides

an extensible expression system enabling lazy NumPy-style broadcasting.
an API following the idioms of the C++ standard library.
tools to manipulate array expressions and build upon xtensor.

xtensor exposes an API similar to that of NumPy covering a growing portion of the functionalities. A cheat sheet can be found in the documentation:

Scrolling the NumPy to xtensor cheat sheet

However, xtensor internals are very different from NumPy. Using modern C++ techniques (template expressions, closure semantics) xtensor is a lazily evaluated library, avoiding the creation of temporary variables and unnecessary memory allocations, even in the case complex expressions involving broadcasting and language bindings.

Still, from a user perspective, the combination of xtensor with the C++ notebook provides an experience very similar to that of NumPy in a Python notebook.

Using the xtensor array expression library in a C++ notebook

In addition to the core library, the xtensor ecosystem has a number of other components

xtensor-blas: the counterpart to the numpy.linalg module.
xtensor-fftw: bindings to the fftw library.
xtensor-io: APIs to read and write various file formats (images, audio, NumPy’s NPZ format).
xtensor-ros: bindings for ROS, the robot operating system.
xtensor-python: bindings for the Python programming language, allowing the use of NumPy arrays in-place, using the NumPy C API and the pybind11 library.
xtensor-julia: bindings for the Julia programming language, allowing the use of Julia arrays in-place, using the C API of the Julia interpreter, and the CxxWrap library.
xtensor-r: bindings for the R programming language, allowing the use of R arrays in-place.

Detailing further the features of the xtensor framework would be beyond the scope of this post.

If you are interested in trying the various notebooks presented in this post, there is no need to install anything. You can just use binder:

The Binder project, which is part of Project Jupyter, enables the deployment of containerized Jupyter notebooks, from a GitHub repository together with a manifest listing the dependencies (as conda packages).

All the notebooks in the screenshots above can be run online, by just clicking on one of the following links:

xeus-cling: the main xeus-cling example notebook,

xtensor: the C++ N-D array expression library in a C++ notebook,

xwidgets: the C++ back-end for Jupyter interactive widgets,

xplot: the C++ back-end to the bqplot 2-D plotting library for Jupyter.

JupyterHub is the multi-user infrastructure underlying open wide deployments of Jupyter like Binder but also smaller deployments for authenticated users.

The modular architecture of JupyterHub enables a great variety of scenarios on how users are authenticated, and what service is made available to them. JupyterHub deployment for several hundreds of users have been done in various universities and institutions, including the Paris-Sud University, where the C++ kernel was also installed for the students to use.

In September 2017, the 350 first-year students at Paris-Sud University who took the “Info 111: Introduction to Computer
Science” class wrote their first lines of C++ in a Jupyter notebook.

The use of Jupyter notebooks in the context of teaching C++ proved especially useful for the first classes, where students can focus on the syntax of the language without distractions such as compiling and linking.

Acknowledgements

The software presented in this post was built upon the work of a large number of people including the Jupyter team and the Cling developers.

We are especially grateful to Patrick Bos (who authored xtensor-fftw), Nicolas Thiéry, Min Ragan Kelley, Thomas Kluyver, Yuvi Panda, Kyle Cranmer, Axel Naumann and Vassil Vassilev.

We thank the DIANA/HEP organization for supporting travel to CERN and encouraging the collaboration between Project Jupyter and the ROOT team.

We are also grateful to the team at Paris-Sud University who worked on the JupyterHub deployment and the class materials, notably Viviane Pons.

The development of xeus, xtensor, xwidgets and related packages at QuantStack is sponsored by Bloomberg.

About the Authors (alphabetical order)

Sylvain Corlay, Scientific Software Developer at QuantStack

Loic Gouarin, Research Engineer at Laboratoire de Mathématiques at Orsay

Johan Mabille, Scientific Software Developer at QuantStack

Wolf Vollprecht, Scientific Software Developer at QuantStack

Congratulations to the LIGO and VIRGO Collaborations from Project Jupyter

2017-11-13T23:03:00+00:00

Aerial view of LIGO Hanford Observatory (image credit to LIGO Lab/Caltech/MIT).

This blog post was co-authored by Jessica Forde.

On behalf of the entire Project Jupyter team, we’d like to say congratulations to Rainer Weiss, Barry C. Barish, Kip S. Thorne and the rest of the LIGO and VIRGO teams for the Nobel Prize in Physics 2017. Since 2015, the LIGO and VIRGO Collaborations have observed multiple instances of gravitational waves due to colliding black holes (and more recently neutron stars). These observations represent decades of work and confirm what Einstein had theorized a hundred years ago.

The LIGO and VIRGO Collaborations are exemplars in utilizing open-source software to promote practices of open science through their Open Science Center. The Open Science Center is a portal for the sharing of data, software, analysis, and discoveries from the LIGO/VIRGO Collaboration. To communicate to the broader community, the LIGO/VIRGO Collaboration has created tutorials with Jupyter Notebooks that describe how to use LIGO/VIRGO data and reproduce analyses related to their academic publications. Most notably, they published a tutorial for working with Binary Black Hole Data.

Posting notebooks and data has enabled the Jupyter community, including our own Min Ragan-Kelley, to make these notebooks interactive by putting them on Binder, a service originally created by Jeremy Freeman and Andrew Osheroff, then at Janelia. Binder allows any GitHub repository with Jupyter Notebooks to be turned into a live, runnable notebook instance that anyone in the world can access instantly. Today, LIGO makes interactive tutorials for each new gravitational wave event by posting links to Binder and Azure Notebooks so users can interact with the data and Jupyter Notebooks in an online environment. They are truly some of our best examples of Binder in action to date!

To visit the LIGO/VIRGO Collaboration’s tutorials, click here.

Watch this blog for some exciting upcoming announcements related to Binder.