Jupyter Blog - kernels

GNU Octave Meets JupyterLite: Compute Anywhere, Anytime!

2025-10-16T15:04:00+00:00

We are thrilled to announce the newest member of our JupyterLite kernel ecosystem: Xeus-Octave. Xeus-Octave allows you to run GNU Octave code directly on your browser. GNU Octave is a free and open-source Scientific Programming Language that can be used to run Matlab scripts. In this article, we present the challenges encountered when targeting WebAssembly, the current state of the Xeus-Octave kernel, and the future plans for expanding the GNU Octave ecosystem.

Earlier this year, we introduced the JupyterLite kernel for R, Xeus-R-Lite. Much like R, cross-compiling GNU Octave to WebAssembly required the same custom toolchain to enable the compilation of Fortran code, combining LLVM Flang and Emscripten.

Similar to many other mathematically oriented language packages, GNU Octave requires a BLAS/LAPACK implementation. Fortunately, OpenBLAS and the Netlib implementations of BLAS/LAPACK had already been added to the emscripten-forge WebAssembly distribution. Initially, OpenBLAS was the preferred implementation, but for the successful compilation of Octave, Netlib LAPACK was selected as it presented fewer hurdles during the build process.

Cross-Compilation of GNU Octave

One of the complications of cross-compiling Octave to WebAssembly, which had not been encountered with the R source code, was the extensive use of Fortran common symbols blocks in the internal libraries of Octave such as odepack.

C Source: liboctave/external/odepack/slsode.f
C-----------------------------------------------------------------------
C The following internal Common block contains
C (a) variables which are local to any subroutine but whose values must
C     be preserved between calls to the routine ("own" variables), and
C (b) variables which are communicated between subroutines.
C The block SLS001 is declared in subroutines SLSODE, SINTDY, SSTODE,
C SPREPJ, and SSOLSY.
C Groups of variables are replaced by dummy arrays in the Common
C declarations in routines where those variables are not used.
C-----------------------------------------------------------------------
     COMMON /SLS001/ CONIT, CRATE, EL(13), ELCO(13,12),
    1   HOLD, RMAX, TESCO(3,12),
    1   CCMAX, EL0, H, HMIN, HMXI, HU, RC, TN, UROUND,
    2   INIT, MXSTEP, MXHNIL, NHNIL, NSLAST, NYH,
    3   IALTH, IPUP, LMAX, MEO, NQNYH, NSLP,
    3   ICF, IERPJ, IERSL, JCUR, JSTART, KFLAG, L,
    4   LYH, LEWT, LACOR, LSAVF, LWM, LIWM, METH, MITER,
    5   MAXORD, MAXCOR, MSBP, MXNCF, N, NQ, NST, NFE, NJE, NQU

Initially, it was not possible to cross-compile these common blocks to WebAssembly because the latest version of LLVM (v20 at the time of testing) did not support common symbol linkage.

// Source: llvm/lib/MC/MCWasmStreamer.cpp
void MCWasmStreamer::emitCommonSymbol(MCSymbol *S, uint64_t Size,
                                      Align ByteAlignment) {
  llvm_unreachable("Common symbols are not yet implemented for Wasm");
}

As a temporary solution, LLVM was patched with the help of Serge Guelton to simulate common symbols as weak symbols.

void MCWasmStreamer::emitCommonSymbol(MCSymbol *S, uint64_t Size,
                                      Align ByteAlignment) {
-  llvm_unreachable("Common symbols are not yet implemented for Wasm");
+  auto *Symbol = cast<mcsymbolwasm>(S);
+  getAssembler().registerSymbol(*Symbol);
+  Symbol->setWeak(true);
+  Symbol->setExternal(true);
}

A proper solution to enable support of common symbols is currently in progress and will likely be included in the next release of LLVM v22 (see llvm-project/pull/151478). For curious readers, the patched version of LLVM can be found here (linux only).

In addition to the patches for LLVM, GNU Octave required a few minor modifications to target WebAssembly; mainly this entailed disabling the GUI functionalities and consolidating the Fortran function signatures and calling conventions. A full list of patches can be found in the recipe directory on emscripten-forge.

Xeus-Octave

Once GNU Octave had been successfully packaged for WebAssembly, bringing Xeus-Octave to JupyterLite was a simple matter of adding a recipe to emscripten-forge!

Try Xeus-Octave in JupyterLite!

Future Work

For our next steps, the team is planning on expanding the Octave ecosystem by adding Octave packages to both conda-forge and emscripten-forge. The packaging work will require defining a process where Octave packages can be installed in predetermined conda environments, perhaps with some minor modifications to the existing pkg utility.

About the Author

Isabel Paredes, who led the charge on bringing GNU Octave to emscripten-forge, is a senior scientific software developer at QuantStack. Prior to working on this project, she focused on porting the R programming language and the Robot Operating System (ROS) framework to WebAssembly.

Acknowledgments

This project synthesizes work from many open-source contributors.

Emscripten-forge, the distribution of conda packages for WebAssembly, was created by Thorsten Beier, who continues to lead the project. Many recipes were contributed by Isabel Paredes, Anutosh Bhat, Martin Renou, Ian Thomas, Wolf Vollprecht, and Johan Mabille.
JupyterLite, the Jupyter distribution that runs entirely in the web browser, was created by Jeremy Tuloup.
Xeus, the C++ library implementing the Jupyter kernel protocol, enabling a custom communication layer, and is foundational to kernels like xeus-r, xeus-python, running in JupyterLite, was created by Johan Mabille and is maintained by a broader team including Martin Renou, Sylvain Corlay, and Thorsten Beier, who worked on the first integration with JupyterLite.
Xeus-Octave, the Xeus-based Jupyter kernel for GNU Octave, was created by Giulio Girardi and Antoine Prouvost.

Exploring Open Access Art Using the Deno Jupyter Kernel

2025-04-17T15:27:00+00:00

Jupyter notebooks are the de facto standard for interactive computing and data analysis, combining code, prose, and visualizations in a single document.

In fact, this post was written in a notebook!

Deno is a simplified and secure runtime for JavaScript and TypeScript, with built-in tooling and modern defaults. Since v1.37, Deno ships with a built-in Jupyter kernel, making it easy to use TypeScript for data science and interactive computing.

Having worked extensively with computational notebooks (mainly in Python), I find this exciting for these reasons:

Simple setup — The kernel is built into the Deno CLI, so there’s no need for additional installation. Just install Deno.
Improved dependency management — Notebooks run like standalone scripts, and Deno’s support for importing dependencies in the code itself improves dependency management and reproducibility.
A unified ecosystem for interactive data analysis — Jupyter supports rich HTML/CSS/JS outputs, and the JavaScript ecosystem is ideal for interactive UIs. Deno connects the kernel and frontend, making notebooks more powerful and flexible.

In this post, we’ll explore the National Gallery of Art’s Open Access dataset in Jupyter with Deno. We’ll clean and analyze the data, looking at public domain works, artists, origins, and patterns. Along the way, we’ll introduce Deno-compatible tools like npm:nodejs-polars and npm:@observablehq/plot, and wrap up by adding widgets and custom displays for a more interactive experience.

The dataset

The National Gallery of Art (NGA) Open Data Program provides access to over 130,000 artworks and their creators, available on GitHub. This dataset includes valuable metadata such as titles, dates, artists, and classifications, all under a Creative Commons 0 (CC0) license, meaning it’s free to use and share.

The collection spans a wide variety of artworks, from sculptures to paintings, by famous artists like Mary Cassatt, M.C. Escher, Vincent van Gogh, Pablo Picasso, and Georgia O’Keeffe.

However, navigating this resource is challenging. The National Gallery’s website is not very user-friendly, making it difficult to make sense of what is actually in the dataset. It’s nearly impossible to get any high-level insights, like the number of paintings, which time periods are most represented, or which artists are most prevalent–let alone make specific queries.

Fortunately, the dataset is available on GitHub as a collection of related tables, exported as CSV files. For our analysis, we’ll focus on three key tables:

objects.csv — Metadata about artworks, including titles, dates, materials, and classifications.
constituents.csv — Artist details such as names, nationalities, and lifespans.
published_images.csv — Links to artwork images via the NGA’s IIIF API.

We’ll clean and join these tables to create a unified dataset.

But wait!

While these NGA tables are free to use, they only include metadata. The actual artwork images have separate licensing, with only about half being CC0-licensed. We’ll gather this info separately to identify freely available images.

Our goal is to unify the data to identify public domain images, explore subsets, and view available artworks. Maybe you’ll even find an upgrade for that default OS wallpaper.

Loading and cleaning the data

Open Access dataset

To get started, we first need to download the data. If you’ve worked with CSVs in JavaScript before, this typically looks like a “fetch” request followed by some parsing. Here, we’re using jsr:@std/csv to stream the data, which gives us an array of objects.

import * as csv from "jsr:@std/csv";

let baseUrl = new URL(
  "https://github.com/NationalGalleryOfArt/opendata/raw/refs/heads/main/data/",
);
let response = await fetch(new URL("objects.csv", baseUrl));

let objects = await Array.fromAsync(
  response.body.pipeThrough(new TextDecoderStream()).pipeThrough(
    new csv.CsvParseStream({ skipFirstRow: true }),
  ),
  (row) => ({
    objectid: +row.objectid,
    title: row.title,
    year: +row.beginyear,
    medium: row.medium,
    type: row.visualbrowserclassification,
  }),
);

objects.slice(0, 3);

[
  {
    "objectid": 0,
    "title": "Saint James Major",
    "year": 1310,
    "medium": "tempera on panel",
    "type": "painting"
  },
  {
    "objectid": 1,
    "title": "Saint Paul and a Group of Worshippers",
    "year": 1333,
    "medium": "tempera on panel",
    "type": "painting"
  },
  {
    "objectid": 4,
    "title": "Saint Bernard and Saint Catherine of Alexandria with the Virgin of the Annunciation [right panel]",
    "year": 1387,
    "medium": "tempera on poplar panel",
    "type": "painting"
  }
]

If we were just plotting the data, this would work fine. But an array of objects isn’t ideal for building our dataset. The NGA data is relational and needs merging, cleaning, and restructuring. A DataFrame is better suited for our use case, offering higher-level, efficient APIs that eliminate the need to write manipulation functions by hand.

Let’s see how we can load the same dataset using a DataFrame from npm:nodejs-polars:

import * as pl from "npm:nodejs-polars";

let response = await fetch(new URL("objects.csv", baseUrl));

let objects: pl.DataFrame = pl.readCSV(await response.text(), {
  quoteChar: '"',
});

objects = objects.select(
  pl.col("objectid"),
  pl.col("title"),
  pl.col("beginyear").as("year"),
  pl.col("medium"),
  pl.col("visualbrowserclassification").as("type"),
);

objects.head()

Here, we fetch the dataset as before, but instead of parsing the CSV ourselves, we read the text response directly with Polars to create a pl.DataFrame.

We then chain a .select expression to choose and rename the columns we want. Notice how, rather than operating on each row in a loop, with Polars we work directly with columns. This API allows us to express complex, efficient operations (in Rust) without materializing JavaScript objects.

Let’s load our other two datasets as pl.DataFrames as well:

The constituents.csv table contains information about any person or entity associated with a piece of art, such as artists, curators, or collectors.

let response = await fetch(new URL("constituents.csv", baseUrl));
let constituents = pl.readCSV(await response.text(), { quoteChar: '"' })
  .select(
    pl.col("constituentid"),
    pl.col("forwarddisplayname").alias("name"),
    pl.col("visualbrowsernationality").alias("nationality"),
  );

constituents.head();

The published_images.csv table contains additional information about the artwork’s image, including a URL for the thumbnail and mapping the objectid to the IIIF (International Image Interoperability Framework) for the image.

let response = await fetch(new URL("published_images.csv", baseUrl));
let publishedImages = pl.readCSV(await response.text(), { quoteChar: '"' })
  .select(
    pl.col("depictstmsobjectid").alias("objectid"),
    pl.col("uuid"),
    pl.col("iiifthumburl").alias("thumburl"),
  );
publishedImages.head();

The final table, objects_constituents.csv, represents a many-to-many relationship between artworks and the people or entities associated with them. Each artwork may have multiple people linked to it.

For our analysis, we’re interested in selecting the “primary” constituent (i.e., the artist). We define the primary artist as the one with the highest displayorder for each object.

let response = await fetch(new URL("objects_constituents.csv", baseUrl));
let objectToArtist = pl.readCSV(await response.text(), { quoteChar: '"' })
  .filter(pl.col("role").eq(pl.lit("artist")))
  .sort({ by: "displayorder" })
  .groupBy("objectid")
  .first()
  .select("objectid", "constituentid");

objectToArtist.head();

This is a much more advanced usage of Polars, so let’s break down what’s going on:

Filter the table to select only rows where the role is “artist”.
Sort the filtered data by displayorder to ensure all entries are at the top.
Group the rows by objectid, ensuring one row per artwork.
Get the first in each group (i.e., artist with highest display order).
Select just the “objectid” and “constituentid” for joining our tables.

Writing this logic by hand in JavaScript would be tedius, but with Polars it’s declarative and reads like a sentence. This type of chaining is common when working with relational data.

Determining whether art is in public domain

We’ve loaded the relevant tables from the Open Access dataset, but they do not indicate whether images associated with an artwork are in the public domain. While CC0 images can be freely downloaded, shared, and repurposed, not all art with the CC0 license are legally in the public domain everywhere due to local copyright laws and edge cases.

Let’s find a way to add public domain information to our data set.

That information is available on the NGA website, but only through the search user interface. Extracting it manually would be impractical, so I reverse-engineered an API call to retrieve the IDs of artworks with public domain images.

We only need the IDs, not the full data, but the query is slow. Since this isn’t an official API, I’ve saved the results separately. There are about 50k IDs identifying public domain images in our dataset.

// This is not an official API, so I’ve cached the results to avoid 
// repeatedly querying the server.
// 
// let response = await fetch("https://www.nga.gov/bin/ngaweb/collection-search-result/search.pageSize__100000.pageNumber__1.lastFacet__artobj_downloadable.json?artobj_downloadable=Image_download_available"); 
// let data = await response.json();
// Deno.writeTextFileSync("public-domain-ids.txt", data.results.map((object) => object.id).join("\n"));

let publicDomainIds = Deno.readTextFileSync("public-domain-ids.txt")
  .split("\n").map((d) => +d);

“Join” to single unified table

Now that we have all our data cleaned and loaded, we’ll perform a large join to combine all these tables into a single unified table.

Again, this challenging to do manually, but with Polars, we can nicely express these complex join operations. Finally, we use a .isIn expression to derive a new column that indicates whether a piece of art is public domain.

let df = objects
  .join(objectToArtist, { on: "objectid" })
  .join(constituents, { on: "constituentid" })
  .join(publishedImages, { on: "objectid" })
  .select(pl.exclude("constituentid"))
  .withColumns(pl.col("objectid").isIn(publicDomainIds).alias("public"))
  .sort({ by: "public" })
  .sort({ by: "year", descending: true, nullsLast: true });

df.head();

With our unified dataset finally loaded, let’s move on to some plotting.

Exploring art, who created it, when, and more with plots

The NGA portal allows some exploration of these data but focuses on specific artworks. We’re more interested in broader insights — where most art comes from, who created it, when it was made, and whether it’s public domain.

As a JavaScript/TypeScript runtime, Deno gives us access to many visualization libraries to ask these kinds of questions. We’ll use npm:@observablehq/plot (Observable Plot) to create plots, spot patterns, and better understand the collection.

Note this post is not a tutorial on Observable Plot, so don’t worry if some of the code below is not immediately clear. The goal is to demonstrate how to use Observable Plot in Jupyter with Deno to extract insights from our data. I’ll highlight any Deno-specific details or useful tips along the way.

Let’s start by examining the distribution of artworks across different “types” in the NGA collection.

import * as Plot from "npm:@observablehq/plot";
import { document } from "jsr:@manzt/jupyter-helper";

// Convert our DataFrame to Array<Object>
let records = df.toRecords();

Plot.plot({
  width: 900,
  marginLeft: 50,
  color: { legend: true },
  marks: [
    Plot.barY(
      records,
      Plot.groupX({ y: "count" }, {
        x: "type",
        sort: { x: "-y" },
        fill: (d) => d.public ? "Public domain" : "Copyrighted",
      }),
    ),
  ],
  // Provide a custom `document`
  document,
});

For the code above, the key things to keep in mind are:

Custom document — Observable Plot relies on the browser DOM, but since we’re in Deno, we provide a custom document from jsr:@manzt/jupyter-helper to enable rendering.
Converting our Polars DataFrame to records — Observable Plot works best with arrays of objects, so we use df.toRecords() to make the data easier to work with.

For specifics on the Plot API, see the documentation and examples available.

Right away, we can see that prints, drawings, and photographs make up most of the collection. Each category varies in how many images are public domain — images of prints, drawings, sculptures, and paintings are mostly public domain, while images of photographs and portfolios are largely copyrighted.

We can get more granular in our plot very easily with Observable Plot, which is composable and expressive. With just a few tweaks, you can completely change how the data is represented. For example, starting from the plot above, we can modify marks and some encoding fields and produce an entirely different chart, allowing for quick exploration and iteration — like this “waffle” bar chart.

Plot.plot({
  width: 900,
  marginLeft: 50,
  color: { legend: true },
  marks: [
    Plot.waffleY(
      /* changed, Plot.barY */
      records,
      Plot.groupZ(
        /* changed, Plot.groupX */
        { y: "count" },
        {
          fx: "type",
          fill: (d) => d.public ? "Public Domain" : "Copyrighted",
          sort: { fx: "-y" }, /* changed, sort: { x: "-y"  } */
          unit: 300, /* new */
        },
      ),
    ),
    Plot.ruleY([0]),
  ],
  document,
});

With both Polars and Plot, we can ask more specific questions about the data, such as “Which artists have the most artwork in the collection?” Here, we first use Polars to group the data by artist (“name”) and whether the artwork is public domain (“public”), counting the number of works for each. Then, we use Plot to visualize the top 25 artists.

const artworkTotals = df
  .groupBy("name", "public")
  .len()
  .sort("name_count", true)
  .head(25)
  .toRecords();

Plot.plot({
  marginLeft: 200,
  color: { legend: true },
  marks: [
    Plot.barX(artworkTotals, {
      x: "name_count",
      y: "name",
      sort: { y: "-x" },
      fill: (d) => d.public ? "Public domain" : "Copyrighted",
    }),
  ],
  document,
});

Again, we can draw some high-level takeaways from this plot: The photographer Robert Frank has nearly twice as many works as the next highest artist. Some entries, like “American 20th Century” and “German 15th Century,” represent groups rather than individuals. Additionally, an artist’s work is typically either entirely public domain or not at all — likely because public domain status is determined at the collection level, with institutions clearing entire groups of works at once rather than evaluating individual pieces separately.

With our tidy and clean dataset, we can quickly iterate in Deno to ask both broad questions (“What even is in this collection?”) and very specific ones (“Who created this artwork in a given year?”).

Deeper exploration with interactivity

So far, we’ve used static plots to explore our data, but Jupyter notebooks support rich, interactive outputs using HTML, CSS, and JavaScript.

With Deno, we can add more interactivity to our data exploration by either using off-the-shelf interactive anywidgets by quickly building custom components for visualizing subsets in ways prebuilt tools don’t support. Deno provides direct access to the full web ecosystem, making custom visualizations seamless.

In this section, we’ll look at:

Prebuilt interactive table anywidgets for exploring data
Create custom component to visualize subsets of the dataset

Exploring data with DataFrame viewer anywidgets

In Jupyter, “widgets” extend notebook outputs with interactive views and controls for objects in the kernel (i.e., backend). Unlike standard outputs, they consist of both kernel-side and front-end code, communicating directly via custom messages. While the Jupyter Widget ecosystem is largely Python-centric, the anywidget project provides a kernel-agnostic way to create and share these components.

With anywidget, widgets can be published to JSR and used in the Deno Jupyter kernel. For example, jsr:@manzt/jupyter-helper provides some off-the-shelf anywidgets for interactively viewing Polars data frames.

import { agGrid } from "jsr:@manzt/jupyter-helper";

agGrid(df.head(200)); // just look at the first 200 items

The agGrid export renders the pl.DataFrame using the popular AG Grid library, allowing for interactive sorting, filtering, and pagination. This default view is minimal but can be easily extended with AG Grid’s many features.

import { quak } from "jsr:@manzt/jupyter-helper";

const paintings = df.filter(pl.col("type").eq(pl.lit("painting")));
quak(paintings);

The quak export renders the pl.DataFrame with the quak data table viewer. Like agGrid, it also includes summary visualizations above each column, showing 1D distributions, and can scale to much more data (e.g., the entire collection). These 1D summary visualizations are interactive, enabling cross-filtering across columns.

Together, the programmatic and interactive layers enable unique hybrid workflows working with data. Some queries are easier to express in code, while others are more intuitive through direct manipulation. In notebooks, the two approaches come together naturally.

For example, Polars can be used to subset the data to just public domain paintings (~2k out of ~115k artworks), and then a date range can be selected interactively by brushing in the UI to explore variation across other columns.

Creating a custom component

Deno gives us direct access to the web ecosystem, making it easy to build small, useful UIs for exploring our data in ways that off-the-shelf tools don’t support.

For example, it would be helpful to actually see a set of artwork in a “gallery view” to provide intuition about what’s in that subset. We can do that since the dataset contains image URLs.

Let’s create a custom JSX component that server-side renders (SSR) a gallery view for any dataset subset using the render export from jsr:@manzt/jupyter-helper. This helper simply converts JSX to an HTML string, which we display using the Deno.jupyter.Displayable interface.

import * as React from "npm:react";
import { render } from "jsr:@manzt/jupyter-helper";

function Gallery({ objects, size = 100 }) {
  return (
    <div
      style={{
        display: "grid",
        gridTemplateColumns: `repeat(auto-fill, minmax(${size}px, 1fr))`,
        gap: "4px",
      }}
    >
      {objects.select("objectid", "thumburl", "title", "public")
        .map(([objectid, thumburl, title, publicDomain]) => (
          <div
            key={objectid}
            style={{ position: "relative", textAlign: "center" }}
          >
            <a
              href={`https://www.nga.gov/collection/art-object-page.${objectid}.html`}
              style={{
                display: "block",
                width: `${size}px`,
                height: `${size}px`,
                position: "relative",
              }}
            >
              <img
                src={thumburl}
                alt={title}
                style={{
                  width: "100%",
                  height: "100%",
                  objectFit: "cover",
                  borderRadius: "5px",
                }}
              />{" "}
              {publicDomain && (
                <img
                  src="https://mirrors.creativecommons.org/presskit/icons/zero.svg"
                  alt="Public Domain"
                  style={{
                    position: "absolute",
                    bottom: "3px",
                    right: "3px",
                    width: "20px",
                    height: "20px",
                    opacity: 0.60,
                  }}
                />
              )}
            </a>
          </div>
        ))}
    </div>
  );
}

Gallery is a custom component that displays a grid of artwork thumbnails from a given dataset. Each image links to its NGA collection page, and public domain works are marked with a CC0 icon.

We can SSR Gallery with our data to better understand a smaller subset, making it easier to explore specific artists, time periods, or categories visually. For example, a random sample from all the collection:

render(<Gallery objects={df.sample(20)} />);

Or sample of just paintings:

render(<Gallery objects={paintings.sample(20)} />);

It’s worth taking a moment to appreciate that in just a few lines of code, we’ve built a domain-specific utility that makes our data more intuitive to explore. This kind of flexibility is powerful and unique to Deno, letting us quickly craft custom utilities that provide deeper insights.

With our toolkit, let’s take a closer look at the NGA dataset:

Drilling down on when the art was created

So far, our data exploration has focused on high-level summaries of the NGA collection by artist and artwork type using static plots. Now, let’s drill down into something potentially more interesting using both our complete toolkit: when these works were created.

We’ll start with a stacked histogram to see the overall distribution of artwork types over time, then facet by public domain status to uncover any patterns.

Plot.plot({
  y: { grid: true },
  color: { legend: true },
  marks: [
    Plot.rectY(
      records.filter((r) => r.year > 1401), // 15th century or later
      Plot.binX({ y: "count" }, {
        x: (d) => new Date(d.year, 0, 1),
        fill: "type",
        fy: (d) => d.public ? "Public Domain" : "Copyrighted",
      }),
    ),
    Plot.ruleY([0]),
  ],
  marginLeft: 100,
  marginRight: 100,
  width: 1000,
  height: 400,
  document,
});

The distribution of artworks with copyrighted and public domain images are noticeably different. Nearly all copyrighted works appear after 1850, while public domain artworks are more evenly spread over time — except for a sharp spike in public domain drawings from the 1940s, with over 15k artworks added.

This is a striking anomaly. Are they all from the same artist? Why only drawings? Let’s take a closer look.

We’ll filter down the public domain data to this span.

let notablePeriod = df.filter("public") // just public domain
  .filter(pl.col("year").gt(pl.lit(1925))) // between 1925 - 1955
  .filter(pl.col("year").lt(pl.lit(1955))).sort({ by: "year" });

Plot.plot({
  y: { grid: true },
  color: { legend: true },
  marks: [Plot.rectY(
    notablePeriod.toRecords(),
    Plot.binX({ y: "count" }, {
      x: (d) => new Date(d.year, 0, 1),
      fill: "type",
    }),
  )],
  document,
});

Zooming into the 1925–1955 range, we see that almost all the data falls within a much narrower window. Interestingly, the spike is highly concentrated between 1935 and 1942, with very little artwork outside this period.

Narrowing in closer, we can see that in this window there are ~18k individual drawings:

const totalNumberOfDrawings = notablePeriod
  .filter(pl.col("year").gt(pl.lit(1934)))
  .filter(pl.col("year").lt(pl.lit(1943)))
  .shape
  .height;

totalNumberOfDrawings; // 18096

made by over 1000 individuals:

const numberOfArtists = notablePeriod
  .filter(pl.col("year").gt(pl.lit(1934)))
  .filter(pl.col("year").lt(pl.lit(1943)))
  .groupBy("name")
  .len()
  .shape
  .height;

numberOfArtists; // 1034

This gets even more interesting! The artwork is not from just one specific source — there was a surge of public works between 1935 and 1942.

Let’s use our custom view to explore a subset of these works — maybe seeing them will give us some insight.

render(<Gallery objects={notablePeriod.sample(100)} />);

So all these artworks, despite being from different individuals, have a very similar style and medium. That’s something worth digging into — what might explain this pattern?

Thanks to our custom component, we can click on artworks and view their metadata. Sampling a few, we see that they all belong to the “Index of American Design” collection.

Conceived as an effort to document and preserve American folk and decorative arts, the Index of American Design consists of 18,257 watercolor renderings created between 1935 and 1942 as part of a Federal Art Project (FAP) work-relief program. Around 400 artists meticulously recreated textiles, woodcarvings, weathervanes, and other objects from across the U.S., aiming to establish a distinct American visual lineage. The project was ultimately housed at the National Gallery of Art, becoming a widely exhibited visual archive.

That’s it! We found a large collection of public works funded by a federal program during the Great Depression — clear in both the data and historical context.

Conclusion

Using Deno’s Jupyter kernel, we explored the NGA Open Access dataset, combining static plots, interactive widgets, and custom JSX components to uncover patterns in the data. This led us to a historical insight — the Index of American Design, a public art project from the 1930s.

With Deno, JSX, and anywidget, we combine data analysis with interactive web-based visualization, making it easy to build domain-specific tools for deeper exploration.

R in the Browser: Announcing Our WebAssembly Distribution

2025-02-28T08:31:00+00:00

In the past few months, QuantStack has received funding from the Bill & Melinda Gates Foundation through a grant to CourseKata to develop support for the R programming language in JupyterLite and the Emscripten-forge distribution for WebAssembly.

Today, we are thrilled to announce the availability of the R programming language in the Emscripten-Forge distribution for WebAssembly, which enables R workflows in JupyterLite through the Xeus-R Jupyter kernel.

JupyterLite with an R Jupyter notebook in use

To experience R in JupyterLite, simply click on the link below:

In this article, we present the details of our efforts to enable R workflows in JupyterLite. We cover the challenges encountered in packaging R and its dependencies for this platform, the solutions we implemented, and the process of handling R mamba packages on the frontend.

Why WebAssembly? Scaling scientific computing workflows in Jupyter to millions of users

The main challenge in scaling Jupyter deployments lies in providing a live computing environment on the server side for each user session.

While Kubernetes-based JupyterHub deployments have been successfully implemented at a university scale (such as the UC Berkeley Data 8 class, or the JupyterHub deployment of Université Paris Saclay), they demand substantial computing resources and a skilled DevOps team to achieve and maintain this level of scalability.

In contrast, JupyterLite represents a fundamentally different paradigm. As a browser-based distribution of Jupyter, it runs language kernels directly in the browser, enabling significantly greater scalability and ease of deployment. Notable JupyterLite-based applications include:

A JupyterLite console integrated into NumPy’s front page, providing immediate access to NumPy functionality without requiring installation.
A similar console application on SymPy’s project page, fully hosted statically.
The official Jupyter website¹.
The examples in Scikit-Learn’s documentation.

These examples illustrate how WebAssembly can be transformative by providing ready-to-use computing environments for scientific workflows, in documentation, blog posts, and scientific communications. The WebAssembly environments are made available to the hundreds of thousands of monthly visitors of these websites without requiring any computing resources on the backend.

JupyterLite deployments displayed on the NumPy, SymPy, and Jupyter websites

[1] Even though it powers the deployment of Jupyter on the main Jupyter website, JupyterLite is not an officially sanctioned Jupyter project.

Beyond Python, enabling R in the browser, and more

While the initial focus of the JupyterLite team has been on the Python programming language, the JupyterLite infrastructure was designed to be language-agnostic from the outset. The same principle applies to the package management stack we have been contributing to over the past few years, namely, mamba and conda-forge.

When we began targeting the WebAssembly platform, we chose to build our efforts on the mamba and conda-forge ecosystem. This decision allowed us to provide customizable environments for WebAssembly and expand beyond Python. This enabled applications such as the JupyterLite terminal, created by Ian Thomas, which comprises a bash-like shell written in TypeScript called Cockle, and WebAssembly builds of native shell commands such as cat, grep, ls, and vim. You can read the earlier announcement introducing the JupyterLite terminal.

Today, we are excited to reveal the availability of R in emscripten-forge, enabling Jupyter kernels such as Xeus-R to be used in JupyterLite.

With the addition of R, emscripten-forge now provides a unified packaging solution for Python, R, and terminal applications for WebAssembly. This covers the scopes of Pyodide (a Python distribution for WebAssembly), WebR (an R distribution for WebAssembly), and Sandbox.bio (a shell emulator with WebAssembly commands).

Building R for Emscripten-forge

Emscripten-forge is a GitHub organization containing recipes to build conda packages for the emscripten-wasm32 platform; it is also the first distribution of conda packages targeting this platform. It is built upon a modernized stack, leveraging rattler-build (the successor to the conda-build package builder) and mamba. The emscripten-forge project, which was created and is still led by Thorsten Beier, is now a team effort covering a broad range of packages.

In order to build R with emscripten-forge, we first required a toolchain capable of converting Fortran and C/C++ code into WebAssembly. Since R relies on BLAS and LAPACK (which are written in Fortran) and many essential R packages wrap native libraries that use Fortran, this conversion capability was crucial.

Fortran

By default, the emscripten-forge toolchain can cross-compile any C/C++ packages into WebAssembly; however, the first challenge in building the R stack was expanding the toolchain to also compile Fortran code. The two compilers that we considered for this task are LFortran and Flang.

LFortran is a compiler built on top of LLVM and designed from the ground up as a cross-compiler. Despite its potential, LFortran is still in its early stages compared to Flang and does not yet fully support all Fortran features.
LLVM Flang is the Fortran compiler of the LLVM project. Unfortunately, it does not support cross-compilation.

Working with Serge Guelton and Kerim Birgi, we initially experimented with the LFortran compiler and contributed to addressing issues encountered when building Netlib LAPACK. The BLAS implementation from this package served as a valuable testing ground since BLAS is one of the core dependencies for R.

After careful consideration, we found that Flang, with its ability to compile a wider variety of Fortran projects, brought us closer to building the Fortran components of R to WebAssembly. George Stagg, the main developer of the WebR project, has written a detailed article on using Flang to produce WebAssembly, which was adapted to the (then) latest version of LLVM (v18). Inspired by George Stagg’s endeavors, Serge posted upstream contributions to the Flang project (#99465, #99822, #101242, #105589), resolving issues with 32-bit platforms that were impacting our effort to support WebAssembly. All of these changes will be included in LLVM/Flang 20.0.

The remaining patches which cannot be merged upstream at this point involve:

(1) enabling code generation that targets WebAssembly (198c0e1),
(2) encoding target-specific size definitions for various data types given that the host and target platforms have different architectures (1da66cd), and
(3) disabling support for 128-bit floating-point types (f20c7c0).

A version of (2) that could be contributed upstream is in the works, which would be one step closer to making WebAssembly an official target for Flang. Additionally, (3) will no longer be needed starting with Flang v20.

These patches were integrated by Axel Obermeier into the Flang build recipe for the conda-forge distribution, making the modified build accessible to everyone. It can be installed on Linux with mamba or micromamba by executing:

micromamba install conda-forge/label/emscripten::flang libllvm19 --no-channel-priority

The final ingredient for our toolchain involved creating a compatible Fortran runtime library. Since Flang implements its runtime library in C/C++, compiling this library with Emscripten proved to be straightforward. We added a recipe for libflang to emscripten-forge, which packages the runtime library and includes all the aforementioned patches.

In the following example, we invoke Flang on a simple Fortran script, and execute the resulting JavaScript and WebAssembly with NodeJS.

> ls
hello.f90  libFortranRuntime.a
> cat hello.f90
File: hello.f90
program hello
 print *, “Hello, Fortran!”
end program hello
> source ~/emsdk/emsdk_env.sh
> flang-new --target=wasm32-unknown-emscripten -c hello.f90 -o hello.o
> emcc hello.o libFortranRuntime.a -o hello.js
> ls
hello.f90  hello.js  hello.o  hello.wasm  libFortranRuntime.a
> node hello.js
Hello, Fortran!

With a working Fortran-to-WebAssembly toolchain, we managed to compile BLAS and LAPACK (See the LAPACK recipe). But before we could get started with building R, we needed to tackle the remaining R dependencies.

Walking up the stack

In R (v4.4), there are a handful of dependencies that are required to provide basic functionality; these are: libiconv, zlib, bzip2, xz, pcre2, and libcurl. Thanks to the packaging efforts led by Thorsten Beier and Wolf Vollprecht, these dependencies are available on emscripten-forge with the exception of libcurl. Because of its socket-based networking architecture and reliance on system-level resource access, libcurl faces fundamental incompatibilities with WebAssembly’s sandboxed environment constraints. Therefore, libcurl had to be disabled in the R source code. As a consequence, downstream R packages which depend on libcurl cannot be ported to WebAssembly until a suitable libcurl replacement is available.

Apart from the core dependencies, R also requires a BLAS implementation. Although external BLAS libraries can be used with R, the R code base contains linear algebra routines from Netlib BLAS and a subset of LAPACK where both libraries are written in C and Fortran. To simplify configuration, we opted for these internal implementations instead of using external BLAS and LAPACK libraries.

Furthermore, several key optional dependencies were cross-compiled to WebAssembly to enhance R’s graphical capabilities; these include libpng, libtiff, cairo, and pango, along with their respective subdependencies. One particularly challenging aspect of packaging these libraries was the cross-compilation of glib, which both cairo and pango depend on. Compiling glib was only possible thanks to the efforts of Kleis Auke Wolthuizen whose patches made glib compatible with Emscripten, and Johan Mabille who disabled multi-threading.

The R dependencies are summarized in the following table:

Table of R dependencies

Cross-compilation of R

Although R typically configures and builds seamlessly on most unix platforms, cross-compiling is not straightforward because R uses a bootstrap approach; first it creates a minimal version of R which is then used to run several R scripts to complete the build process. Bootstrapping is a challenge because the built binaries for the target platform (WebAssembly) are not executable (at least not directly) from the host platform. Therefore, the r-base package necessitates two build phases. First, we compile R for the host platform (Linux) with GCC and Flang; this generates the R and Rscript executables needed for the second phase. And then, we cross-compile to WebAssembly with Emscripten and Flang.

One of the major challenges of this setup was generating the R Data Base (.rdb) and R Data Index (.rdx) files for the internal packages of r-base (base, compiler, grDevices, graphics, grid, methods, parallel, splines, stats, stats4, tools, and utils). To accomplish this, it was necessary to dynamically load some of the internal shared libraries (tools, grDevices, graphics, utils, stats, and methods) which provide essential functionality to generate the data and index files. However, the R executables we created during the first phase of the build are incompatible with the WebAssembly shared libraries. In order to work around this, we temporarily replaced the WebAssembly libraries of the internal packages with the linux shared libraries we built during the first phase (See cross_libraries.sh). This makeshift solution is also used to cross-compile all other R packages; the setup is carried out by the activation script of the cross-r-base package (See activate-cross-r-base.sh).

Additionally, the R source code required a few modifications to enable cross-compilation to WebAssembly.

The Emscripten platform was added to select the correct configuration for the platform.
The dependency on libcurl was removed (as mentioned in the previous section) and as a consequence, the internet package which relies on libcurl was disabled.
System calls which are not compatible with our WebAssembly environment were disabled.
Cairo replaced Xlib as the default bitmap type to enable graphics.

The complete list of patches applied to R is available here: r-base patches.

The R package ecosystem

Building R itself was just the beginning. The next challenge was to provide a wide array of R packages to create a robust development environment. Fortunately, many packages are written in pure R and are available as “noarch” packages on conda-forge, making them directly compatible with emscripten-forge environments. However, for the R packages that require compilation, a package recipe needs to be created and added to emscripten-forge. Thanks to the help of Anutosh Bhat, the following packages have been compiled and are available to use.

r-askpass     r-digest    r-haven      r-magrittr  r-purrr    r-tidyr
r-base64enc   r-dplyr     r-hexbin     r-mass      r-rcpp     r-tzdb
r-bit         r-ellipsis  r-htmltools  r-matrix    r-readr    r-utf8
r-bit64       r-fansi     r-isoband    r-mgcv      r-rlang    r-vctrs
r-cachem      r-farver    r-jsonlite   r-mime      r-sp       r-vroom
r-cli         r-fastmap   r-later      r-nlme      r-stringi  r-xfun
r-colorspace  r-ggrepel   r-lattice    r-plyr      r-sys      r-yaml
r-data.table  r-glue      r-lazyeval   r-promises  r-tibble

Most of the time, cross-compiling an R package is very simple because the bulk of the work is handled by r-base. Building a package typically boils down to a single line:

$R CMD INSTALL $R_ARGS .

Occasionally, minor patches are applied to the package source code to enable the WebAssembly target platform or to simplify cross-compilation. Some R packages, such as r-nlme, contain Fortran code; for such packages, the custom flang compiler needs to be installed before the package can be cross-compiled. Should any package be needed for your use case, feel free to contribute them to emscripten-forge!

Xeus-R

Once a sufficient set of packages is available for the platform, the next step is to provide a development environment for end users. Our chosen tool for this purpose is Jupyter.

The most popular Jupyter kernel for R is IRkernel, which has been developed over several years. It includes complete ZeroMQ bindings for the R language and comprehensive coverage of the Jupyter protocol. However, since IRkernel is tied to ZeroMQ, we opted to work with the Xeus-R project instead, which was announced last year. Xeus-R was developed by Romain François in collaboration with our team and with JupyterLite in mind. It is built upon the Xeus library, which provides a native implementation of the Jupyter protocol and is adapted to work with the JupyterLite frontend.

Fortunately, the components of IRkernel responsible for the rich representation of R objects and rich display in Jupyter frontends were split into two separate R packages: IRdisplay and repr. These packages can be reused in Xeus-R. As a result, Xeus-R and IRkernel offer the same rich display functionality, ensuring that Jupyter notebooks created with one kernel will work with the other. However, Xeus-R provides a different implementation of the communication layer through jupyterlite-xeus, which enables integration with the JupyterLite stack. This integration was previously implemented for Xeus-Python and Xeus-Lua.

One key aspect of this integration is the populating of the in-memory file system used by JupyterLite kernels with the required runtime dependencies, particularly shared libraries. This process is done upon bootstrapping the kernel. For this purpose, Anastiasiia Sliusar and Martin Renou developed mambajs, which, in combination with jupyterlite-xeus, processes conda packages in the frontend and installs them in the kernel filesystem at startup. In WebAssembly, shared libraries need to be handled with special care in order to function as expected. Drawing inspiration from the Python community, we adapted the pioneering work of Pyodide to properly load shared libraries into the filesystem.

How to make a deployment with custom packages

The GitHub repository https://github.com/jupyterlite/xeus-lite-demo is a template for creating a JupyterLite deployment on GitHub pages that includes the packages specified in a conda environment.

The process is as follows:

Create a new repository from the GitHub template.
Enable the deployment on GitHub pages from a GitHub action, as shown in the README.
Edit the environment file to include the desired packages.

For example, to deploy an R kernel with coursekata installed, the environment.yml file would contain the following:

name: xeus-r
channels:
  - https://repo.mamba.pm/emscripten-forge
  - conda-forge
dependencies:
  - xeus-r
  - r-coursekata

The deployment linked at the beginning of this article was created with this template.

How to contribute

Contributors are always welcome! New R packages can be requested by opening an issue on emscripten-forge, or added directly by opening a pull request with the desired package recipe. For instructions on how to contribute, please visit https://emscripten-forge.org/.

What is in the works

Currently, we are developing the infrastructure to dynamically install emscripten-forge and conda-forge noarch packages in JupyterLite kernel environments. This will apply to all Xeus kernels that make use of these packages, such as Xeus-Python, Xeus-R, and Xeus-Lua. In the case of R, this may be directly bound to install.packages.

Additionally, we are working on more advanced features for the R kernel, including support for Jupyter interactive widgets and the split of the pure R parts of xeus-r into a separate package, which would facilitate a possible convergence with IRkernel.

Finally, other language kernels based on the same package management stack are also in the works. Stay tuned for future announcements!

About the Author

Isabel Paredes, who led the charge on bringing R to emscripten-forge, is a senior scientific software developer at QuantStack. Prior to working on this project, she focused on porting the Robot Operating System (ROS) framework to WebAssembly.

Acknowledgments

This effort brought together several endeavors from many open-source developers.

JupyterLite, the Jupyter distribution that runs entirely in the web browser, was created by Jeremy Tuloup.
Xeus, the C++ library implementing the Jupyter kernel protocol, enabling a custom communication layer, and is foundational to kernels like xeus-r, xeus-python, running in JupyterLite, was created by Johan Mabille and is maintained by a broader team including Martin Renou, Sylvain Corlay, and Thorsten Beier, who worked on the first integration with JupyterLite.
Emscripten-forge, the distribution of conda packages for WebAssembly, was created by Thorsten Beier, who continues to lead the project. Many recipes were contributed by Isabel Paredes (for the R ecosystem), but also Kerim Birgi, Anutosh Bhat, Martin Renou, Wolf Vollprecht, and Johan Mabille.
Xeus-R, the Xeus-based Jupyter kernel for R, was created by Romain François.
IRDisplay and repr are pure R packages providing rich display and rich mime type rendering of many core R types, which are used in xeus-r. These two packages stem from the IRkernel project, a Jupyter kernel for R written in R, which was started by Thomas Kluyver, and has been maintained over the years by Philipp Angerer and Jan Katins.
MambaJS, the library enabling the processing of conda packages in the frontend, was developed by Anastasiia Sliusar and Martin Renou.
Conda-forge, the main distribution of conda packages, includes a large number of “noarch” recipes, including for R packages, and is developed by a broad community of contributors worldwide.
WebR, an R distribution for WebAssembly, was created by George Stagg. Although not directly used in this project, it documented numerous patches and changes to the Flang compiler that were adapted for this effort.

Funding

The work by Isabel Paredes and other QuantStack team members on bringing R and Xeus-R to Emscripten-Forge was supported by the Bill & Melinda Gates Foundation through a grant to CourseKata. CourseKata is a nonprofit project dedicated to improving statistics and data science education through interactive, research-backed curricula that integrate professional tools like R and Jupyter notebooks. This grant was part of CourseKata’s broader mission to scale its innovative curriculum, equipping students with the skills and confidence to engage with statistics, data science, and coding in meaningful ways.

Meet Xeus-R: a future-proof Jupyter kernel for R

2024-01-22T17:20:00+00:00

JupyterLab session with a notebook powered by the Xeus-R kernel for R

The Jupyter ecosystem provides a versatile and user-friendly platform for data analysis and scientific computing, it has democratised access to powerful tools for researchers, scientists, educators, and data professionals.

The name Jupyter, a blend of Julia, Python, and R has always hinted at its relevance in multiple scientific programming languages and has since conquered new horizons thanks to its flexible and extensible design.

Underneath the interactive surface of Jupyter lie the Jupyter kernels, a set of language specific engines orchestrating the dynamic execution and exploration of code. Kernels implement the Jupyter protocol, a shared specification of the communication process between notebooks and kernels, defining how messages, commands and outputs are exchanged and processed.

The Xeus framework is a central piece of the puzzle, offering a reference to facilitate Jupyter kernel implementations. The speed, extensibility, and alignment with the official protocol make Xeus a perfect technology to implement new kernels.

Xeus-R

Today, we, a collaborative team led by Romain François and supported by QuantStack, are thrilled to announce the initial release of Xeus-R, a future-proof Jupyter kernel for R. Xeus-R builds upon the components of Xeus to facilitate the use of R in Jupyter notebooks and already supports typical capabilities: R code execution, inspection, completion, and contextual help.

The recent advancements in support for WebAssembly in the R world led by the WebR project, the flexible design of Xeus, and the rise of JupyterLite all contribute to our strong belief that Xeus-R has the potential to deliver Jupyter in-browser experience for data scientists using R.

Xeus-R is currently at the stage of a proof of concept, but we wanted to communicate about it early to broadcast our excitement about its potential, share some of our thoughts, and initiate a conversation.

Relationship with IRkernel

While the IRkernel project paved the way and is the current reference implementation of a Jupyter kernel for R, it does not take advantage of the Xeus infrastructure and includes a complete implementation of the Jupyter protocol, down to the composition of all of the JSON messages and most importantly how they are transported through ZeroMQ sockets.

In contrast, Xeus-R fully embraces and leverages the capabilities of Xeus, while borrowing some higher level concepts from IRkernel and reusing features from its dependent packages IRdisplay and repr.

We believe this path will benefit future development as it gives us the freedom to work on innovations that are specific to the needs of R users, and let Xeus absorb evolutions of the Jupyter protocol.

Key features

Xeus-R implements the basic building blocks of what you would expect from a Jupyter kernel. The results of a code cell, i.e. the value of its last expression are displayed in the output, taking advantage of ANSI escape codes for colours as were made popular by the crayon and later cli R packages.

Traditional and ggplot2-powered graphics are captured and displayed as images. The aspect ratio of plots can be controlled by setting the repr.plot.height and repr.plot.width options either globally, or locally via the cell_options() function.

Thanks to the scaffolding in place in the IRdisplay and repr packages, Xeus-R is also able to emit richer displays through mime bundles.

In contrast with IRkernel, the rich displays need to be explicitly requested with the display() function. There are cases however, such as HTML widgets, where automatic rich display makes the most sense because a text/plain representation is not valuable.

Xeus-R also supports contextual help and includes a basic completion system, based on the internal utils::.getTokenFromLine() system.

Installation and Getting Started

At the time of writing this article, Xeus-R is only available and tested to work on Unix-based systems. We consciously decided to focus initial efforts on features rather than platform comprehensiveness, but we should be able to fix this soon enough.

The easiest way to get up and running with Xeus-R is to install it from conda-forge:

mamba install xeus-r

Instructions to build from source are available in the github repository for Xeus-R: https://github.com/jupyter-xeus/xeus-r

Looking ahead: JupyterLite Integration

Another benefit of a Xeus-based approach is that it has been designed in a way that supports in-browser experience to power JupyterLite. Together with the recent developments around WebR, we can envision R becoming a first-class citizen in the JupyterLite world.

This is an exciting vision and will serve as our North Star ⭐️ for future developments of Xeus-R.

Conclusion

Join us in welcoming Xeus-R, a Jupyter kernel for R that leverages the robust foundation of Xeus architecture. R is a language of paramount importance for Data Science and Scientific Computing and it rightfully deserves its place as a first-class citizen within the Jupyter ecosystem.

We firmly believe that Xeus-R has the potential to become a reference implementation for an R kernel, well positioned to tackle the challenges of the ever-evolving landscape of in-browser Scientific Computing, a promising future led by the JupyterLite and WebR projects.

Acknowledgements

The work by Romain François on Xeus-R was funded by QuantStack.

Author Bio

Romain François is a prominent figure in the R community, recognised for his longstanding contributions to R programming and Data Analysis. With a profound passion for open-source software and commitment to advancing the capabilities of R, he has earned a well-deserved reputation as a respected expert.

Romain’s expertise encompasses a wide spectrum of R-related fields, including package development, data manipulation, and interoperability with other programming languages. He is widely recognized for his work on packages like dplyr, which has become a cornerstone for efficient data manipulation in R, and for his contributions to Rcpp, a library that bridges the power of C++ with R for optimizing R packages and enhancing performance.

Plug your application into the Jupyter world

2023-10-23T06:09:00+00:00

Kernels are a simple but powerful abstraction in the Jupyter architecture. They encapsulate language interpreters and make them accessible through a standardized interface. This is the key to Jupyter’s remarkable versatility, with over 100 supported languages. Embedding a kernel in your custom application can seamlessly expose it to the Jupyter ecosystem. Not only can you attach JupyterLab to your program for e.g. state inspection, as you would do with a debugger, but you can truly extend your application with all the power of the Jupyter ecosystem.

Applications: GUI and API

Several applications such as FreeCAD, Blender, or QGIS provide a Python API to access their functionalities programmatically, and include an embedded Python console which allows using this API from within the application. Users can both use the graphical user interface and operate on the application in a scripted fashion depending on their needs.

Python consoles in FreeCAD, Blender, and QGIS (from left to right).

If you think about it, these are the kinds of workflows that Jupyter has always been about, but Jupyter has evolved in the opposite direction. Instead of adding scripting to GUIs, Jupyter added GUIs to scripting. At first, Jupyter was IPython, a console for interactive computing. Then it went to the Web and became IPython Notebook, renamed to Jupyter Notebook. With the browser it became possible to display richer content and create small GUIs, using Jupyter Widgets. Then JupyterLab was created and allowed to create fully-fledged desktop-like applications in the browser, like 3-D CAD modelers.

JupyterLab showing notebooks using JupyterCAD and ipyleaflet.

Jupyter put a lot of attention on building a good experience interacting with the Python REPL, which is not the main focus of domain-specific desktop applications. The Python console embedded in desktop applications is often quite raw. Allowing users to connect from Jupyter can enable a much better experience.

What if there was a bridge that allowed you to connect both worlds, your application on one side and Jupyter on the other side? What is the common denominator allowing code to interact with an application, while also being shared with Jupyter? In the Jupyter infrastructure, we call it a kernel.

Jupyter kernels

Kernels provide a standardized way to expose language interpreters to applications, in a language-agnostic fashion. You can think of it as a server that accepts requests such as “execute this piece of code” and that replies with a result. The result is actually a data representation, it can be as simple as “3” for an input of “1+2”, or a complex interactive plot. A kernel usually lives in a separate process on your machine, but it can also run in the same process, or even remotely on another machine. Someone wanting to connect to this kernel will need the connection information such as the IP address and the ports of the different communication “channels”. It is stored in what is called a connection file.

The Jupyter kernel protocol allows clients to interact with different languages.

Anyone who has access to this connection file can then execute code in the kernel. This is “shared code”, meaning that every client to this kernel eventually runs code in the same interpreter. If one client sets a variable “a=1”, other clients will see this variable. They are all running the “same program”. If the kernel runs in the same process as the application, then the kernel can interact directly with it. This way of sharing the execution engine serves as a bridge between the application and Jupyter.

Now it is just a matter of exposing the connection file so that e.g. JupyterLab can see it. The application launching the kernel can save the connection file to a specific directory, and give the path to JupyterLab. And that’s it, JupyterLab can run a notebook using this kernel, allowing it to interact with the application.

Getting started

The xeus-qt-python repository is a great place to start and see how to embed a Python kernel inside a Qt application. Under the hood, it uses xeus-qt, an integration of the xeus kernel architecture in the Qt event loop. The Qt application actually runs JupyterLab in a Qt WebEngine window. Users can run code in a notebook and interact with the application, which in turn can interact with the notebook. Here, we create a Qt button from a notebook. When clicking on the button, a callback prints a message in the notebook. This demonstrates a two-way binding between the Qt application and JupyterLab.

Example of a Qt application embedding JupyterLab.

SlicerJupyter from Kitware was one of the first applications to take advantage of this approach. You can read more in this blog post. We hope that this technology will unlock a new brand of use-cases and allow applications to reach a new audience through the Jupyter ecosystem. Please reach out to us if you need help!

About the author

David Brochart is a technical director at QuantStack. He mostly works on the Jupyter backend stack. He contributed to the development of real-time collaboration in JupyterLab, and created the Jupyverse project, a next-generation implementation of the Jupyter server.

Acknowledgments

This work was made possible thanks to several contributors. Mariana Meireles was the initial author of xeus-qt. Thorsten Beier is a scientific software developer at QuantStack and contributes to the xeus kernel ecosystem. He is the initial creator of the xeus-qt-python demonstration. Johan Mabille is a technical director at QuantStack and he is the main author of xeus. Afshin Darian is a technical director at QuantStack and worked on the UI in JupyterLab 4.0 allowing to use an external kernel.

A Jupyter kernel for GNU Octave

2023-01-11T17:33:00+00:00

GNU Octave is a long-standing member of the scientific computing ecosystem, featuring a mathematics-oriented syntax with built-in 2D/3D plotting and visualization tools, and compatible with the Matlab syntax. It is a natural candidate for high-quality integration in the Jupyter ecosystem.

Jupyter has been devised with language agnosticism in mind since it was spun off from the IPython project. The Kernel (the part of the infrastructure responsible for executing the code input by the user) is a key extension point to Jupyter, and dozens of language kernels exist in the ecosystem.

Prior art on Jupyter / GNU Octave integration includes the Calysto/Octave_kernel project by Steven Silvester (who is also a core Jupyter maintainer and the co-creator of JupyterLab). This existing kernel is part of the family of kernels built on top of the ipykernel reference implementation of the Jupyter protocol, in Python.

Today, we are happy to announce the xeus-octave project, a Jupyter kernel for GNU Octave. Xeus-octave was created by Giulio Girardi, recently joined by Antoine Prouvost — and has been incorporated into the Project Jupyter governance.

Xeus-octave is built upon the Xeus library, a C++ implementation of the Jupyter kernel protocol that enables fast development of new kernels using programming languages’ native APIs, unlocking new possibilities without parsing standard outputs or requiring a Python interpreter at runtime. After all, a Jupyter kernel is merely an executable providing a well-defined communication protocol. It is not bound to Python APIs.

A fully-featured kernel

We strove to make the xeus-octave kernel as complete as possible for this first iteration.

Multiple graphical toolkits

Notebook

A notebook Octave graphical toolkit can be used to render plots natively, and present the figure as an image in the notebook.

The classic Octave sombrero plot rendered as an image inside the notebook

Plotly

An experimental plotly Octave graphical toolkit can also use Plotly to render code in the browser, with dynamic views, zooms, and hover tooltips.

Multiple plots rendered in a single plotly figure

Rich display

The kernel binds with the C++ interface of GNU Octave to provide a complete Octave experience. Having access to the exact internal type of the objects, xeus-octave can present data with rich output.

Matrices can be presented as HTML or LaTeX tables.

Symbolic expressions can be presented as LaTeX formulas.

Structured data can be visualized as interactive tree views

Interactive help

Formatted help, extracted from the GNU Octave reference manual, can be queried using the ? syntax.

Trying it online

You can try the new Jupyter kernel for GNU Octave by clicking on the link below:

The Binder linked above includes examples of notebooks in the fields of Electronics, demonstrating the usability of xeus-octave in real-life complex scenarios.

This interactive demo is also featured on the Jupyter website at https://jupyter.org/try#kernels.

Installing xeus-octave

Xeus-octave has been packaged on conda-forge and can be installed with mamba or conda.

$ mamba install xeus-octave -c conda-forge

Future Work

In future iterations, we will work towards consolidating the xeus-octave experience, by adding cell magics and improving the Plotly integration.

A major ongoing development is the implementation of an Octave backend for Jupyter interactive widgets, based on the xwidgets project. Combined with Voilà, it will be possible to use GNU Octave to create standalone web dashboards.

Another ongoing endeavor is to port a growing portion of the scientific computing stack to WebAssembly (using the emscripten-forge project). Producing a Wasm build of GNU Octave will enable its use with Jupyterlite, enabling large-scale deployments on websites, blogs, and arbitrary web pages without any need for scalable cloud infrastructure.

About the authors

Antoine Prouvost is a Scientific Software Engineer at QuantStack and a xeus-octave maintainer. Previously, Antoine was researching Machine Learning and Combinatorial Optimization at the DS4DM research chair and Mila.

Giulio Girardi is a Master’s degree student in Electronic Engineering at the UNIPD, currently working as an Electronic Engineer at Protech Engineering. Giulio is the creator and a maintainer of the xeus-octave project.

Xeus-Lite

2021-10-25T09:40:00+00:00

JupyterLite is a JupyterLab distribution that runs entirely in the web browser, backed by in-browser language kernels. Xeus is C++ library for writing Jupyter kernels. In this blogpost we show how these two can work together.

JupyterLite

With vanilla Jupyter, kernels are running in separate processes. The communication between the kernels and the client is achieved with ZeroMQ. In contrast to this, JupyterLite runs entirely in the browser, including the language kernels. The following kernels are already implemented in the JupyterLite framework:

JavaScript Kernel
P5 Kernel: Since P5 is a JavaScript library, it is easy to provide a P5 kernel running in the browser.
A Python kernel based on pyodide called pyolite: While the two JavaScript based kernels are trivial to implement as in-browser kernels, pyodide is a Python interpreter compiled to WebAssembly via Emscripten. Not only Python itself has been compiled to WebAssembly, but also the whole Scientific Stack including packages like: Numpy, Scipy, Pandas and many more. An overview of packages can be found here. Furthermore any pure Python package can be installed from within pyolite.

Since JupyterLite runs entirely in the browser, it is very easy to try it out!

Xeus

Xeus is C++ library for writing Jupyter kernels. It is not a kernel but a library that simplifies the process of writing kernels. Xeus implements the Jupyter messaging protocol, so that kernel authors only need to implement a handful of methods related to the actual interpreter, like executing pieces of code and auto-completing.
With the help of Xeus, it is very easy to write Jupyter kernels . In fact, Xeus has already been used for making several language kernels:

xeus-cling is a Xeus-based C++ kernel.
xeus-python is a Xeus-based Python kernel.
xeus-lua is a Xeus-based Lua kernel.
xeus-sql / xeus-sqlite are Xeus-based kernel for SQL / SQLite databases.
xeus-robot is a Jupyter kernel for the Robot Framework .
LFortran is a modern interactive Fortran compiler with a Xeus-based Jupyter kernel.

Since it is very easy to write Xeus based kernels for Jupyter, we also want to leverage Xeus for writing JupyterLite kernels. This has been the main motivation for Xeus 2.0.

Xeus Architecture

Xeus is has three extension points as depicted below

The Server is the middleware component responsible for receiving and sending messages to the Jupyter client. The default server it is built upon ZeroMQ and handles the concurrency model of the application.
As we will see later, a custom server is needed to connect Xeus with JupyterLite.
The Interpreter provides the interface that kernel authors must implement.
The Debugger provides the interface that can to be implemented when the kernel shall have debug capabilities, but we do not yet support this for Xeus-JupyterLite-Kernels.

JupyterLite Xeus-Kernels

Traditionally, the Jupyter kernel and the Client to the Jupyter kernel run in separate processes as depicted below:

The communication between the kernels and the client (ie the Jupyter-server) are done with inter-process communication via ZeroMQ. But with JupyterLite the kernel runs directly in the browser. In that case, ZeroMQ is not required anymore to communicate with the frontend. This was the motivation for Xeus 2.0, where it is possible to build Xeus without ZeroMQ. To implement JupyterLite Kernels with Xeus, we need to implement a custom Server which handles the Emscripten/Embind-based communication layer

The overall architecture is best described with the picture below: The C++ Xeus-Kernel is compiled to WebAssembly with Emscripten / Embind. The WebAssembly compiled code runs in a dedicated WebWorker thread. The WebWorker communicates via Message Passing with a thin layer of TypeScript which implements the IKernel API of JupyterLite. The communication between JupyterLite and the JupyterLite k ernel is done entirely in JavaScript.

The C++ code needed to compile an existing xeus-kernel to WebAssembly is very minimal:

Furthermore one needs a thin layer of TypeScript / JavaScript to implement the JupyterLite IKernel API. A good example of this code is given in the JupyterLite Xeus-Lua repository. While most of the repositories code is boilerplate, one needs to provide a simple Dockerfile which compiles the Xeus-Kernel to WebAssembly.

Xeus-Lua

We recently released a Xeus based Lua kernel. Even though its worthwhile to have Xeus-Lua kernel for vanilla Jupyter, the main purpose of Xeus-Lua was to experiment with Xeus-based JupyterLite kernels.
We choose Lua as the first language to try out Xeus-based JupyterLite kernels since Lua is easy to compile to WebAssembly.
A JupyterLite instance with Xeus-Lua included can be found here

Xeus-Lua has built in support for interactive widgets

Xeus-Wren

Wren is relatively new “Classy Little scripting Language” created by the author of the great book “Crafting Interpreters”. Wren can be described with the following:

Think Smalltalk in a Lua-sized package with a dash of Erlang and wrapped up in a familiar, modern syntax.

We choose Wren as a second language to experiment with Xeus + JupyterLite since it is small, simple to compile to WebAssembly and easy to embed in C++. A JupyterLite instance with Xeus-Wren included can be found here.

Xeus-Wren has built in support for displaying data of various MIME-types

Outlook

There are many languages which can be compiled to WebAssembly like:

Julia: https://github.com/Keno/julia-wasm
Octave: https://news.ycombinator.com/item?id=22926120
C++: https://github.com/binji/wasm-clang
Python: https://github.com/pyodide/pyodide

Also databases as sqlite can be compiled to WebAssembly:

Sqlite: https://github.com/sql-js/sql.js/

Our goal is to provide Xeus-based JupyterLite kernels for these projects to make the JupyterLite ecosystem even more powerful.

Acknowledgements

We like to thank

Jeremy Tuloup, Madhur Tandon, Martin Renou, and Nicholas Bollweg for their work on JupyterLite.
Johan Mabille for Xeus and his help implementing the Xeus-Emscripten server.

About the Author

Thorsten Beier is a Scientific Software Engineer at QuantStack. Before joining QuantStack, he graduated in computer science at the University of Heidelberg and worked at the EMBL. As an open source developer, Thorsten worked on a variety of projects, from nifty and vigra in C++ to inferno, kipoi and ilastik in Python.

From Jupyter to the Moon

2021-09-24T12:24:00+00:00

Jupyter architecture in a nutshell

A key principle in Jupyter’s design is language agnosticism, and one of the main extension points of the Jupyter ecosystem is the kernel, the part of the architecture responsible for executing the user’s code.

In the end, a Jupyter kernel is merely an executable implementing a well-defined inter-process communication protocol.

A reference implementation of the kernel is ipykernel, which provides the default Python kernel, and utilities to build other language kernels. Another reference implementation of the protocol is Xeus.

Xeus

Xeus is C++ library for writing Jupyter kernels. It is not a kernel but a library that simplifies the process of writing kernels. Xeus implements the Jupyter messaging protocol, so that kernel authors only need to implement a handful of methods related to the actual interpreter, like executing pieces of code and auto-completing. There is a long list of scripting languages which can be embedded in C/C++. With the help of Xeus, it is very easy to write kernels for such languages. In fact, Xeus has already been used for making several language kernels:

xeus-cling is a Xeus-based C++ kernel.
xeus-python is a Xeus-based Python kernel.
xeus-sql / xeus-sqlite are Xeus-based kernel for SQL / SQLite databases.
xeus-robot is a Jupyter kernel for the Robot Framework .
LFortran is a modern interactive Fortran compiler with a Xeus-based Jupyter kernel.

xwidgets

The xwidget library is a C++ implementation of the Jupyter Interactive Widget protocol. Xwidgets were initially developed for xeus-cling, the Xeus C++ kernel, but any Xeus-based kernel can reuse these components.

A xeus-based Lua kernel

Today, I am pleased to announce the release of xeus-lua, a new Jupyter kernel for Lua based on Xeus.

Lua

Lua, which means “Moon” in Portuguese, is a lightweight embeddable scripting language written in C. While Python is one of the leading scripting languages for scientific application, Lua is the leading scripting language used in games / game engines. The list of video games using Lua is long and includes triple-a games like Crisys and Farcry. Not only games rely on Lua, but also applications like Adobe Photoshop Lightroom.

Sol2

With the Sol2 library, Lua can be embedded into C++ in a very elegant way.

While Lua itself can be embedded in a C++ application quite easy, it is even easier when using Sol2. Sol2 is a C++ library that simplifies the usage of Lua from C++. Furthermore Sol2 can be used to expose C++ classes to Lua, in a similar fashion as pybind11 can be used to expose C++ classes to Python.

Xeus-lua

The compulsory “Hello, World!” from xeus-lua. This blog-post` would be incomplete without it.

I am pleased to announce xeus-lua, a new Jupyter kernel for the Lua language based on Xeus. Using Sol2 and Xeus, it was very easy to implement a feature rich kernel using modern C++. With the help of Sol2, it was trivial to expose the xwidgets classes to Lua such that we can use the widgets within xeus-lua notebooks.

But why a Lua kernel?

While Lua might not be the hippest language in 2021, there are multiple reasons while I choose to implement such a Lua kernel, and why such a kernel can be useful:

Writing a Xeus-based kernel is the best way to learn and familiarize oneself with the xeus ecosystem. And since Lua is so easy to build and embed, Lua and Xeus are perfect match. Implementing this kernel gave me great insights into Xeus and Jupyter itself.
Learning Lua with xeus-lua is fun! Before writing xeus-lua, I had zero Lua experience. While implementing xeus-lua I learned Lua from within Jupyter by playing around with xwidgets from within Lua — this was a very joyful experience.
The xeus-lua implementation might serve as reference for future kernel authors. In particular the integration of xwidgets in xeus-lua might be useful for future implementations.
Since Lua is so easy to build, xeus-lua can be built for WebAssembly and will be integrated in JupyterLite — stay tuned!

Features

Obviously xeus-lua supports the execution of code.

Simple code execution

Streams are redirected to the notebooks output

Redirection of streams

Errors messages are shown in the notebooks output

Error Handling

Input streams are redirected to the notebooks input system

User Input

Statements can be auto-completed with a press of tab

Code Completion

Data of different sources as Latex, JSON and html can be displayed.

Display Data

Interactive widgets can be added to the notebook!

Widgets

We can feed the widgets with data!

Binary buffers for widgets

Try It Out

You can try it right now with binder, without the need of installing anything on your computer, just by clicking on this link.

Other Lua kernels

There are several Lua kernels available:

lupyter: A Python based implementation of a Lua Jupyter kernel. The heavy lifting is done in a C-extension.
ILua: A feature rich implementation. Ilua is implemented in Python and does the communication with Lua via IPC.
IPyLua: IPyLua is a pure Lua implementation.

While these implementations are feature rich and work well, none of the above supports interactive widgets. Since xeus-lua is based on Xeus, we can reuse components from the Xeus ecosystem like xwidgets with minimal effort.

Acknowledgments

We are grateful to Juliette Taka for the illustration.

About the Author

Thorsten Beier is a Scientific Software Engineer at QuantStack. Before joining QuantStack, he graduated in computer science at the University of Heidelberg and worked at the EMBL. As an open source developer, Thorsten worked on a variety of projects, from nifty, and vigra in C++ to inferno, kipoi and ilastik in Python.

Enabling the JupyterLab debugger with ipykernel

2021-05-13T11:45:00+00:00

Support for the Jupyter Debugger Protocol just landed in ipykernel

JupyterLab 3.0 includes a visual debugger that allows to interactively set breakpoints, step into functions, and inspect variables with any Jupyter kernel that implements the Jupyter debugger protocol.

The first two language kernels to implement the new protocol were xeus-python (a Python kernel) and xeus-robot (a kernel for Robot Framework). Unfortunately, the reference Python kernel, ipykernel, did not support debugging yet, until now!

Today, we are pleased to announce that debugging support landed in ipykernel, and will be available in the next major release, ipykernel 6.0. Pre-releases including the ipykernel debugger are available.

Debugging with ipykernel

What will change with ipykernel 6.0?

Enabling support for debugging in ipykernel required important changes in the code base regarding the concurrency model of the kernel. The main change is that the processing of messages on the “control channel” now happens in a different thread, allowing for the processing to happen while user code is running.

Ipykernel 6.0 includes several other updates. Tornado coroutines were dropped in favor of native coroutines. The Matplotlib inline backend was split into a separate package, and ipykernel depends on debugpy, an implementation of the Debug Adapter Protocol for Python.

If you are interested in testing out the new features, check out the beta release!

pip install ipykernel --pre

Try it now!

Thanks to MyBinder, you can also try it out without the need of installing anything on your computer. Just follow this link:

What about the future?

A lot of new features are in the works with the JupyterLab debugger.

JupyterLab 3.1 will include several usability improvements to the debugger.
It will also add the ability to submit code for execution when stopped at a breakpoint.
We are working on a richer variable explorer, using Jupyter’s rich display system to enable the rich-rendering of variables in the explorer, to e.g. render dataframes as tables.

Finally, we also plan on adding debugging support to other language kernels.

Acknowledgements

The work of Johan and Sylvain at QuantStack on the debugger support in ipykernel was funded by Two Sigma. We are grateful to Min Ragan Kelley and Matthias Bussonnier, who reviewed the pull requests on debugger support.

About the authors

Johan Mabille is a scientific software developer at QuantStack.

Johan is very active in the Jupyter ecosystem, as the creator of xeus, a C++ implementation of the Jupyter protocol, and several language kernels, such as xeus-python, xeus-cling, and xeus-robot. Johan also made contributions to the Jupyter widgets ecosystem and to JupyterLab.

Sylvain Corlay is the founder and CEO of QuantStack.

As an open-source developer, Sylvain is very active in the Jupyter project with contributions in several components of the stack, including widgets, kernels, nbconvert, and others. He is also a steering committee member of the project.

Sylvain also does volunteer work for the community, as member of board of directors of NumFOCUS, co-organizer of the PyData Paris Meetup, and vice-chair of JupyterCon 2020.

Abracadabra! Bringing the magics to xeus-python

2021-02-18T13:30:00+00:00

Last year, we set ourselves to implement a visual debugger for JupyterLab. This endeavor required major developments in the JupyterLab front-end, in core-Jupyter protocols, and on the kernel side (the part of the Jupyter infrastructure responsible for executing the code).

For kernel-side development, we decided to start with the xeus-python kernel, a lightweight implementation of a Jupyter kernel for the Python programming language. Based on xeus, xeus-python has a flexible concurrency model that was required to e.g. add breakpoints while executing code.

Unfortunately, xeus-python did not provide all ipykernel features (magics, Matplotlib support, etc.). Furthermore, many notebooks depend on IPython, as they import it explicitely or make use of the IPython configuration system.

Today, we are proud to announce that xeus-python supports 100% of the IPython magics! This was achieved by leveraging the core IPython package. This is getting us closer to feature parity with ipykernel.

Magics

xeus-python now supports all magics that IPython provides and even user-defined magics!

Support for builtin IPython magics and user-defined magics

Matplotlib

Most Matplotlib back-ends are now supported by xeus-python, allowing you to show static plots with the inline back-end or interactive plots with ipympl in your Notebook:

Matplotlib inline back-end and ipympl interactive back-end

Extending

IPython has a rich configuration system that you can tweak in order to get an enhanced Notebook experience. Xeus-python now supports any configuration you setup for IPython:

Automatically import NumPy in the IPython settings

Visual Debugging

The latest JupyterLab version introduced a visual debugger in its interface. xeus-python was the first Jupyter kernel to add support for it!

Stepping into Python code in JupyterLab with the visual debugger

xeus-python is not the only Jupyter kernel that supports debugging anymore: xeus-robot is a xeus-based Jupyter kernel for the RobotFramework language that supports the JupyterLab visual debugger.

Try it online

Thanks to MyBinder, you can try it out without the need of installing anything on your computer. Just follow this link:

Installation

You can install the latest xeus-python version using mamba or conda:

mamba install xeus-python -c conda-forge

conda install xeus-python -c conda-forge

You can also compile it yourself, instructions can be found in the repository:
https://github.com/jupyter-xeus/xeus-python

Acknowledgments

My work on xeus and xeus-python at QuantStack was funded by Bloomberg.

About the Author

My name is Martin Renou, I am a Scientific Software Engineer at QuantStack. Before joining QuantStack, I studied at the aerospace engineering school SUPAERO in Toulouse, France. I also worked at Logilab in Paris, France and Enthought in Cambridge, UK. As an open-source developer at QuantStack, I work on a variety of projects, from xtensor and xeus-python in C++ to ipyleaflet and bqplot in Python and JavaScript/TypeScript.

An SQL solution for Jupyter

2021-02-03T12:03:00+00:00

A few months ago we released xeus-sqlite, a Jupyter kernel that allows users to make SQLite queries directly from the notebook. With the needs of the Jupyter data science community in mind we decided to expand on this project.

RDBMSes supported by xeus-sql

Today, we’re happy to announce the next step of this endeavour, bringing you xeus-sql a general-purpose database access tool for Jupyter, which allows you to make SQL queries on many different databases:

MySQL
PostgreSQL
SQLite3
DB2
Oracle
Firebird
and any database with an ODBC driver.

To offer all these integrations we rely on the SOCI library as the backbone of the project. SOCI abstracts all the different database connections and query details behind a unified C++ API. xeus-sql uses SOCI and xeus to expose SQL functionalities to Jupyter.

Xeus-sql, as well as the MySQL, PostgreSQL and SQLite3 backends of SOCI are available on conda-forge, meaning you can install them easily with conda or mamba:

# Installing xeus-sql and the mysql backend
mamba install xeus-sql soci-mysql -c conda-forge

# Installing xeus-sql and the postgresql backend
mamba install xeus-sql soci-postgresql -c conda-forge

# Installing xeus-sql and the sqlite backend
mamba install xeus-sql soci-sqlite -c conda-forge

There are idiosyncrasies between these different SQL backends which we detail in the xeus-sql documentation and examples.

Visualizing your queries

HTML and text table visualizations for different Jupyter interfaces

You’re probably familiar with visualizing your SQL tables and query results in the form of… tables, of course. Jupyter’s rich display system offers the option of showing them in rich-text display or plain-text display depending on which application you’re using.

Beyond displaying queries with tables, you can now easily create Vega-Lite graphs from your query results directly in the notebook:

Running a series of examples on a notebook

To support this feature xeus-sql relies on xvega, a C++ backend to vega, and custom Jupyter magics for plotting through a mini-language implemented in the xvega-bindings utility library. Besides using the mini-language you can directly provide the JSON specification for your visualization, thanks to contributions by Wang Fenjin.

Vega-Lite is a powerful library and there are many different visualizations you can create from your relational data using xeus-sql. You can see some other examples below:

Examples of different visualizations generated with xeus-sql and xvega-lite by Wang Fenjin

We’re looking forward to exchange with users and learn more about their use cases for xeus-sql and xvega, so if you have any questions please drop them on the public QuantStack chat, or open an issue at the project’s GitHub page. As always, contributions are welcome!

Try it online!

You can try the stable version on Binder without the need of installing anything on your computer just by clicking on the image below:

Acknowledgements

We are grateful to Wang Fenjin for his contributions with respect to the use of vega specs in xeus-sql. The plotting mini-language was built upon the xvega library by Madhur Tandon. We are also grateful to the SOCI contributors for their awesome package that made this work possible.

About the author

My name is Mariana Meireles and I’m a software developer working for QuantStack. I care deeply about the impacts that technology has in the world and try my best to be the change I want to see by contributing to open source projects that stand upon libre and diverse standards.

SlicerJupyter: a 3D Slicer kernel for interactive publications

2020-07-08T16:29:00+00:00

The Jupyter ecosystem is a powerful platform for exploratory computational science, and now it can connect with some of the deep and rich domain-specific desktop applications that have decades of feature development already invested in them. With the integration of 3D Slicer with Jupyter through the xeus-python’s interpreter, we demonstrate how a Qt-based graphical desktop application with 3D visualization provided by Visualization Toolkit (VTK), image processing provided by the Insight Toolkit (ITK), can be used through a Jupyter notebook. This approach is available on the SlicerJupyter GitHub repository and could be extended to other applications that embed Python, such as Blender, FreeCAD, or ParaView.

This xeus-python integration is beneficial both for the Jupyter ecosystem and desktop applications. Features that have been developed for decades for desktop applications become readily available for Jupyter users without learning a new working environment or redeveloping features. For a desktop application, the Jupyter notebook can serve as a way to create reproducible data processing workflows, scientific publications, and maintainable tutorials without requiring local software installation. From Jupyter, you can now quickly create simple medical imaging applications with extremely rich interactivity from 3D Slicer. Read below for a discussion of the features available and the history of the project.

Powerful Medical Imaging Capabilities Available Through Jupyter

3D Slicer (or Slicer for short) is a C++ desktop application that uses Qt, ITK, and VTK libraries for visualization and medical image analysis. Slicer’s embedded Python interpreter makes all its features accessible with the Python programming language. Slicer has a simple built-in console to run Python commands interactively and can run Python scripts from files, but these are not as convenient as cell-base interactive notebooks, which have become popular among data scientists and researchers in recent years.

By integrating the xeus-python kernel, we can use a Slicer process as a Jupyter kernel. xeus-python leverages the xeus C++ implementation of the Jupyter kernel protocol. xeus-python is an alternative to ipykernel, which can be used with a vanilla CPython interpreter, interfacing to standard CPython Jupyter widgets like itkwidgets, but it can also be coupled with custom interpreters and GUI event loops, like the Slicer interpreter and its Qt event loop. This allows you to represent a complete scene in Medical Reality Markup Language, MRML, Slicer’s internal data structure. The kernel exposes the full medical imaging API and representation of your data in a meaningful way for Python developers, allowing access through standard Python ecosystem formats such as pandas dataframes and NumPy arrays in the Notebook.

Interactivity Levels

You can also use Jupyter interactive widgets (sliders, buttons, etc.) to control Slicer, modify data, or adjust processing and visualization parameters. Interactivity can be implemented at different levels.

Level 1: Standard Jupyter widgets display application specific objects by automatic conversion of application-specific data objects to standard Python objects. For example, Slicer markup fiducial lists are displayed as a nicely formatted table and model nodes are rendered as 3D objects.
Level 2: Static image widgets display content that the desktop application renders. These widgets can be made interactive by modifying data and rendering parameters using additional standard widgets. This makes rich visualization capabilities — sophisticated rendering various data types, rendering of very large data sets, etc. directly available in Jupyter.
Level 3: Dynamic viewer widgets display 2D and 3D views rendered by the desktop application. Mouse and keyboard events are forwarded to the desktop application that allows zooming/rotating views, and to utilize all 3D interactions implemented in the desktop such as placing annotations, making measurements, or segmenting images the same way as if it was done on the desktop appication’s screen. This is implemented in the Slicer Jupyter kernel using ipycanvas and ipyevents packages.
Level 4: Full desktop graphical user interface integration. Users can see parts of the application window rendered in notebook cells, including standard desktop widgets (sliders, menus, etc.). It is implemented using noVNC and TigerVNC in Slicer Jupyter. This is particularly useful when the application runs on a remote server.

Video demonstrating how to run 3D Slicer using Binder

These interactive tools allow developers to implement complete data processing workflows in a notebook, even if certain steps require manual user inputs such as segmenting 3D regions or setting seed points.

Try it online!

Since Jupyter notebooks can be used from any web browser, it can essentially turn any desktop application to a web application. By setting up a remote Jupyter server, users do not have to install anything on their computers. We have set up a demonstration of this using Binder (www.mybinder.org) that anybody can try at https://mybinder.org/v2/gh/Slicer/SlicerNotebooks/master.

Click on the binder image to launch the demo

The docker image that installs and configures Slicer and all dependencies (ipycanvas, ipyevents, VNC, etc.) is available at https://github.com/Slicer/SlicerDocker/tree/master/slicer-notebook. For deployment to non-technical users, applications can be deployed using Voilà (https://voila.readthedocs.io/), which only shows relevant content and interactive widgets, so the notebook looks like a simple dynamic web page.

The current implementation is already stable and offers a wide range of features, but there is still room for design and performance improvements. For example, we could not implement fully automatic conversion of application-specific data objects to displayable Python objects (due to complex implementation of display hooks); xeus-python debugger’s threading model needs to be improved to allow using it without locking the application’s main thread; and dynamic viewer widget’s performance (level 3 interaction) could be optimized to achieve higher refresh rates.

History of Slicer and this integration

Built over two decades with support from the NIH and a worldwide open source developer community, 3D Slicer is a unique, multi-platform desktop application for analysis, integration, and visualization of medical images that is heavily used by researchers globally for basic and applied research in a wide range of topics. The 3D Slicer Community includes physician-scientists with disease-specific knowledge of clinical challenges, computer scientists and physicists who develop novel algorithms, imaging informatics researchers, software engineers with the ability to understand clinical problems and create reliable tools, and application engineers with the multidisciplinary skills to deploy these tools in a range of cancer research settings. Slicer is maintained by Kitware, Inc., and the NAMIC consortium, and has an active open source software community. Slicer is used in hospitals and by researchers, with more than 10k academic citations and more than 150k downloads in the last year alone.

Click on the image to download 3D Slicer

It can be used for image processing workflows on 2D, 3D, and 4D images. What makes Slicer so useful is its huge set of community contributed modules that extend its functionality, as well as its ability to read DICOM and a wide array of exotic file formats.

Slicer has been scriptable in Python for well over a decade, but a robust SlicerJupyter extension is born from significant development effort by the community. The idea for this integration dates back to SciPy in 2014, when Jean-Christophe Fillion-Robin of Kitware organized a sprint intended to integrate Slicer and an IPython notebook, motivated by the dream of creating interactive and fun tutorials for Slicer. Mike Sarahan and Jean-Christophe created this proof of concept https://github.com/commontk/QEmbedIPython#qt-embed-ipython, but it was far from usable. In 2015 Matt McCormick of Kitware, created the SlicerDocker repository to support headless builds and rendering in a Docker image. Then in June 2018, while attending the Slicer Project Week, Andras Lasso (Queen’s University) and Jean-Christophe learned about Xeus, a C++ implementation of the Jupyter kernel protocol developed by QuantStack that would help streamline the integration of Slicer with Jupyter. To support this effort, Andras and Jean-Christophe created the Slicer/SlicerJupyter GitHub repository. They also contributed changes to Xeus to support this new integration paradigm where the event loop of the kernel is driven by a Qt-based Desktop application (see here). Building on this foundation, Isaiah Norton, then working at Brigham and Women’s Hospital, contributed additional improvements like a better auto-completion using jedi (see https://github.com/Slicer/SlicerJupyter/pull/12) as well as integration with Binder. More recently, Jean-Christophe and Sylvain Corlay (QuantStack) met after the Slicer project week while attending SciPy in Austin. Following the creation of a new project called xeus-python (started by Martin Renou at QuantStack), we took the integration of Jupyter and Slicer to the next level by adding support for improved interactive use and MRML data node visualization directly in the notebook.

About the authors

Alphabetically ordered

Sylvain Corlay is the founder and CEO of QuantStack, and a core Jupyter developer. He co-authored xeus and xeus-python.

Sylvain Corlay

Jean-Christophe Fillion-Robin is an open-source enthusiast, original author of the SlicerJupyter extension and a principal engineer at Kitware Inc where he leads the development of “3D Slicer” based commercial applications. J-Christophe also maintains scikit-build, an improved build system generator for CPython C/C++/Fortran/Cython extensions.

Jean-Christophe Fillion-Robin

Mike Grauer is a Technical Leader on the data and analytics team at Kitware. He is specialized in building scalable server-side processing frameworks that enable scientific workflows over web platforms.

Mike Grauer

Andras Lasso is an original author of the SlicerJupyter extension, Senior Research Engineer and Associate Director of the Laboratory for Percutaneous Surgery at Queen’s University.

Andras Lasso

Matt McCormick is an open source, medical imaging researcher working at Kitware Inc. Matt is an active, contributing member of scientific open source software efforts such as the Insight Toolkit (ITK) and scientific Python (SciPy) communities, and maintains the Jupyter 3D widget, itkwidgets.

Matt McCormick

Isaiah Norton is a Senior Software Developer at TileDB, Inc. with experience in digital pathology and image-guided surgical navigation.

Isaiah Norton

Steve Pieper is a Chief Architect and active developer of the 3D Slicer application for well over a decade. He is CEO of Isomics, Inc, where he uses a range of software technologies to perform medical imaging research with leading universities and companies.

Steve Pieper

Martin Renou is a scientific software developer at QuantStack. He is the original author of xeus-python, the xeus-based Python kernel, and contributed to the new concurrency model used for the debugger.

Martin Renou

Mike Sarahan is a software engineer at RStudio, PBC working on bringing language ecosystems together. He is passionate about making software work for data scientists in ways that are easy to maintain and improve.

Mike Sarahan

A Jupyter kernel for SQLite

2020-06-11T08:42:00+00:00

While it is well known in the Python scientific computing community, Jupyter is in fact a language-agnostic development environment. High-quality language kernels exist for the main languages of data sciences, such as Python, C++, R or Julia.
But another important tool for data science is the SQL family of programming languages. Today, we announce the release of a Jupyter kernel for SQLite.

This new kernel allows the user to use the complete SQLite syntax as well as some extra operations such as opening or closing a database file, or visualizing the data in different ways using Jupyter magics.

HTML and Text table visualizations for different Jupyter interfaces

To make the task of implementing a new Kernel for Jupyter easier, we make use of Xeus, a library providing a solid implementation of the Jupyter kernel protocol, so that we can focus on implementing the language-specific parts of the kernel.

As a connection between SQLite’s native C implementation and C++ we’re using SRombauts’ library SQLiteCpp.

Current state

xeus-SQLite is still under active development but it offers a fully functional SQLite interface and magics to perform higher-level operations that are outside of the scope of the SQL syntax, such as creating, opening, or closing SQLite databases. Some of the things you can do with xeus-SQLite are creating a new database, loading it, backing it up or deleting it. You can also check if a table exists, set and reset keys of a database and get information about it. You can find more about the use of these methods in SQLite’s documentation.

xeus-SQLite provides rich HTML display of tables in the Jupyter Notebook and Jupyter Lab. For the Jupyter Console we make use of the tabulate library for textual display.

Different commands being executed on xeus-SQLite

The future

For the future of xeus-SQLite we want to create an intuitive form of visualizing data: creating plots, graphs, charts, maps and much more from your SQLite query results, all in the same notebook. Currently we’re exploring the use of Vega in xeus-SQLite, a declarative language for creating interactive visualization designs and can do bar plots using jupyter magics:

SQLite query result with bar and table outputs for same query

This feature is still in very early stages and being developed in this branch.

We also plan on producing a static build of xeus-SQLite bundling xeus and the SQLite library into a single executable that can be easily distributed.

Try it online!

You can try the stable version without the need of installing anything on your computer just by clicking on the image below:

This won’t grant you access to the experimental branch where we’re testing, if you’re interested in trying it please check the installation guide and make sure you’re in the right branch.

Installation

Note that you first need to have Jupyter installed on your computer. You can install xeus-sqlite using mamba:

mamba install xeus-sqlite -c conda-forge

Or conda:

conda install xeus-sqlite -c conda-forge

About the author

Prior to QuantStack I worked as a developer on the PySide team at the Qt Company and as a web performance developer at Mozilla.

A visual debugger for Jupyter

2020-03-25T15:53:00+00:00

Most of the progress made in software projects comes from incrementalism. The ability to quickly see the outcome of an execution and iterate has been one of the main reasons for the success of Jupyter, especially in scientific exploratory workflows.

Jupyter users like to experiment in the notebook, and to use the notebook as an interactive communication tool. However, for more classical software development tasks such as the refactoring of a large codebase, they often switch to general-purpose IDEs.

The JupyterLab environment.

The Jupyter project has made strides in the past few years towards filling that gap, notably with the JupyterLab project, which enables a richer UI including a file browser, text editors, consoles, notebooks, and a rich layout system.

However, a missing piece (which has remained one of the main reasons for users to switch to a different tool) is a visual debugger. This feature has long been requested by users, especially those accustomed to general-purpose development environments.

A debugger for Jupyter

Today, after several months of development, we are glad to announce the first public release of the Jupyter visual debugger!

This is just the first release, but we can already set breakpoints in notebook cells and source files, inspect variables, navigate the call stack and more.

Screencast of the JupyterLab visual debugger in action

Try the debugger on binder

You can also try the debugger online with binder. Just click on the binder link:

Click on the binder link to launch the demo

Installation

The debugger front-end can be installed as a JupyterLab extension.

jupyter labextension install @jupyterlab/debugger

The debugger front-end will be included in JupyterLab by default in a future release.

In the back-end, a kernel implementing the Jupyter Debug Protocol (which will be detailed in the next section) is required. The only kernel implementing this protocol, for now, is xeus-python a new Jupyter kernel for the Python programming language. (Support for the debugger protocol in ipykernel is also on the roadmap).

conda install xeus-python -c conda-forge

Once xeus-python and the debugger extension are installed, you should be all set to use the Jupyter visual debugger!

Note: Depending on the platform, PyPI wheels are available for xeus-python, but they are still experimental.

The Jupyter Debug Protocol

New message types for the Control and IOPub channels

Jupyter kernels (the part of the infrastructure that executes the user’s code) communicate with the rest of the infrastructure with a well-specified inter-process communication protocol.

Several communication channels exist, such as

the Shell channel, which is a request/reply channel for e.g. execution requests
the IOPub channel, which is a one-directional communication channel from the kernel to the client, and is used e.g. to forward the content of the standard output streams (stdout and stderr).

The Control channel is similar to Shell but operates on a separate socket so that messages are not queued behind execution requests, and have a higher priority. Control was already used for Interrupt and Shutdown requests, and we decided to use the same channel for the commands sent to the debugger.

Two message types were added to the protocol:

the debug_[request/reply] to request specific actions to be performed by the debugger such as adding a breakpoint or stepping into a code, which is sent to the Control channel.
the debug_event uni-directional message used by debugging kernels to send debugging events to the front-end. Debug events are sent over the IOPub channel.

Extending the Debug Adapter Protocol

A key principle to the Jupyter design is the agnosticism to the programming language. It is important for the Jupyter debug protocol to be adaptable to other kernel implementations.

A popular standard for debugging is Microsoft’s “Debug Adapter Protocol” (DAP) which is a JSON-based protocol underlying the debugger of Visual Studio Code and for which there already exist multiple language back-ends.

It was therefore natural for us to use the DAP messages over the debug_[request/reply] and debug_event messages that we just added.

However, it was not quite sufficient in the case of Jupyter. Indeed

In order to support page reloading, or a client connecting at a later stage, Jupyter kernels must store the state of the debugger (breakpoints, whether the debugger is currently stopped). The front-end can request that state over with a debug_request message.
In order to support the debugging of notebook cells and of Jupyter consoles, which are not based on source files, we also needed messages to submit code to the debugger to which breakpoints can be added.

Besides these two differences, the content of the debug requests and replies corresponds to the debug adapter protocol.

All these extensions to the Jupyter kernel protocol have been proposed for inclusion in the official specification. The JEP (Jupyter Enhancement Proposal) can be found here.

Xeus-python, the first Jupyter Kernel to support debugging

Xeus is a C++ implementation of the Jupyter kernel protocol. It is not a kernel by itself but a library that helps kernel authoring. Xeus is useful when developing a kernel for a language that has a C or a C++API (like Python, Lua, or SQL). It takes the cumbersome task of implementing the Jupyter messaging protocol for the kernel author to focus on the core interpreter tasks: executing code, inspecting, etc.

Several kernels have been developed with xeus, including the popular xeus-cling kernel for the C++ programming language, based on the cling C++ interpreter from CERN. The xeus-python kernel is an alternative Python kernel to ipykernel, based on xeus. The first release of the xeus-python kernel was announced on this blog earlier this year: https://blog.jupyter.org/a-new-python-kernel-for-jupyter-fcdf211e30a8

Xeus-python was an appropriate choice for this first implementation of the debugging protocol because

it has a pluggable concurrency model, which allowed running the processing of the Control channel in a different thread.
it has a lighter-weight codebase which made it a convenient sandbox to iterate upon. Implementing the first version of the protocol in ipykernel would have required more significant refactoring and consensus building at an early stage.

The roadmap of Xeus-python

The short-term roadmap for xeus-python includes

adding support for IPython magics in xeus-python, which is the main missing feature with respect to ipykernel.
improving the PyPI wheels of xeus-python.

What about other kernels?

The work in the front-end is valid for any kernel implementing the extended kernel protocol.

We will be working in 2020 to enable debugging with as many kernels as possible.

This will soon be the case for other xeus-based kernels which share a large part of the implementation with xeus-python, such as xeus-cling.

Diving into the debugger front-end architecture

The debugger extension for JupyterLab provides what users would typically expect from an IDE:

a sidebar with a variable explorer, a list of breakpoints, a source preview and the possibility to navigate the call stack
the ability to set breakpoints directly next to the code, namely in code cells and code consoles
visual markers to indicate where the current execution has stopped

When working with Jupyter notebooks, the state of the execution is kept in the kernel. But a cell can be executed and then deleted from the notebook. What should happen when a user wants to step in deleted code?

The extension supports that particular use case and enables retrieving a read-only view of the previously executed cell.

Stepping into a deleted cell

Consoles and files also have support for debugging.

Debugging code consoles in JupyterLab

Debugging files in JupyterLab

Debugging can be enabled on a notebook level, which lets users debug a notebook and work on a different one at the same time.

Debugging multiple notebooks simultaneously

Variables can be inspected using a tree viewer and a table viewer:

The variable explorer

The debugger extension for JupyterLab has been designed to work with any kernel that supports debugging.

By relying on the Debug Adapter Protocol, the debugger extension abstracts away language-specific features and provides a consistent debugging interface to the user.

The following diagram shows how the debug messages flow between the user, the JupyterLab extension and the kernel during a debugging session.

Using the Debug Adapter Protocol in the debugger extension (source)

Future developments

In 2020, we plan on making major improvements to the debugger experience:

Support for rich mime type rendering in the variable explorer.
Support for conditional breakpoints in the UI.
General improvements of the debugger user experience.
Enable the debugging of Voilà dashboards, from the JupyterLab Voilà preview extension.

Acknowledgements

The JupyterLab debugger is the result of the collaboration and coordination of developers from several institutions, including QuantStack, Two Sigma, and Bloomberg.

The work on both the front-end and the back-end at QuantStack by Jeremy Tuloup, Johan Mabille, Martin Renou, and Sylvain Corlay, was funded by Bloomberg.
The work on the Jupyter debugger by Borys Palka and Afshin Darian was made possible by Two Sigma.

About the developers

Jeremy Tuloup

Jeremy Tuloup is a Scientific Software developer at QuantStack. He authored a large part of the front-end of the JupyterLab debugger.
.
.
.

Borys Palka

Borys Palka is a software developer at Codete. He authored a large part of the front-end of the JupyterLab debugger.
.
.
.

Johan Mabille

Johan Mabille is a scientific software developer at QuantStack. Johan is a co-author of xeus, and developed the debugger extension to xeus-python. He also authored a large part of the debugger front-end.
.

Martin Renou

Afshin Darian

Afshin Darian is a software developer at Two Sigma, and one of the authors of JupyterLab.
.
.
.
.

Sylvain Corlay

Sylvain Corlay is the founder and CEO of QuantStack, and a core Jupyter developer. He co-authored xeus and xeus-python.

Field Report on the Kernel Community Workshop

2019-10-07T13:24:00+00:00

From May 27th to May 29th 2019, thirty developers from the Jupyter community met in Paris for a three-days workshop on the Jupyter kernel protocol.

Attendees to the Jupyter Community Workshop on Kernels

For three days, attendees worked full time on the Jupyter project, including hacking sessions and discussions on improvements to Jupyter protocols and standards.

We were lucky to count five core developers to the project and several more regular contributors to the larger ecosystem in the group!

Beyond the hacking sessions, each day was concluded with a series of presentations and demos of the progress made during the workshop.

Why a workshop on Jupyter kernels?

The Jupyter Kernel protocol is one of the main extension points of the Jupyter ecosystem. Dozen of language kernels have been developed by the community, and these languages can then leverage the other components of the stack, such as the notebook, the console, and interactive widgets.

One of the objectives of this event was to foster collaboration between kernel developers and core contributors and develop common tools used across all language kernels. Another goal was to work improving the protocol, to support visual debugging, parameterized kernels, etc.

Technical Achievements

A lot of work was done during the workshop. Several people were working on new Jupyter kernels for programming languages.

We were amazed to see how much was achieved in just three days. Today, we are still working on projects that stemmed from this community workshop.

JupyterLab Debugger

One area in which the Jupyter team is actively working is support for visual debugging in JupyterLab. This is a major endeavor spanning from the front-end to the back-end, including changes to the Jupyter protocol.

The kernel workshop was the occasion for several developers (Wolf Vollprecht, Maarten Breddels, Johan Mabille, Loic Gouarin) to get together and hack on the implementation of the frontend. A lot of work had been done ahead of the workshop with a new Python kernel based on the Xeus library. This new kernel includes a backend to the Debug Adapter Protocol over kernel messages.

The JupyterLab Visual Debugger

There are other key contributors to the debugger project who did not attend the community workshop. We should mention Jeremy Tuloup and Afshin Darian who are spearheading the frontend development.

A Calculator Jupyter Kernel based on xeus

Vasavan Thirusittampalam and Thibault Lacharme were in the middle of their summer internships as scientific software developers at QuantStack when they attended the workshop. During the event, they set themselves up to implement a new Jupyter kernel in C++!

At the end of the workshop, they were able to demonstrate a functional calculator kernel based on xeus, a C++ implementation of the Jupyter protocol! This was later polished and resulted in the publication of a blog post on the Jupyter blog: “Building a Calculator Jupyter Kernel”. This example may serve as an example for people interested in creating new language kernels with xeus.

A prototype Julia backend to Jupyter interactive Widgets.

During the workshop Sebastian Gutsche (who is the author of the Singular Jupyter kernel and a co-author of the GAP kernel), set himself to develop a backend to Jupyter interactive widgets for the Julia progamming language. The current state of the code can be found here.

The Julia backend to jupyter widgets is still early-stage but should this project be completed, it would enable other Jupyter widget libraries for the users of the Julia Jupyter kernel, such as ipyvolume, ipyleaflet, bqplot, etc.

The Julia backend to Jupyter widgets

Parameterized Kernelspecs

Paul Ivanov and Kyle Kelley worked on ironing out the proposal for the support of parameterized kernelspecs. The objective is to make it possible to pass user-specified arguments to kernels upon launch.

In the proposal, the parameters expected by the kernel executable and the possible values for these parameters may be specified in the kernelspec file or a companion file to the kernelspec in the form of a JSON schema.

Paul and Kyle produced extensive notes on various aspects on the subjects such as

the storing of previously used values of the parameters (in notebooks)
the web UI design on how to specify parameters
the definition of default values

More details will be shared at a later stage on the subject of parameterized kernelspecs.

Note: the subject of parameterized kernelspecs was also central in the earlier community workshop about the Jupyter server organized at IBM by Luciano Resende.

Forking of Jupyter Kernels

Starting of kernels is an expensive operation, and executing a whole notebook even more. For certain use cases (such as dashboards based on Voilà), it is useful to have a pre-executed notebook ready, so that dashboards can be presented instantly to a user. To enable this, Maarten Breddels came up with the idea of forking kernels and put together a proof-of-concept implementation. The implementation can be found in pull requests to jupyter_client (PR #441) and ipykernel (PR #410).

Forking kernels will also allow interesting features such as an undo/rollback of cell execution in the notebook. While it is still a proof of concept, this subject has already sparked interest in the community.

Acknowledgments

This event would not have been possible without the generous support provided by Bloomberg, who made this workshop series possible

We are grateful to CFM for gracefully hosting the event.

We should also thank Ana Ruvalcaba from Project Jupyter for her incredible work on the logistics and finances of the Jupyter Community Workshop series.

Building a Calculator Jupyter Kernel

2019-07-16T08:36:00+00:00

A step-by-step guide for authoring language kernels with Xeus

An early device for interactive computing

In order to provide a language-agnostic scientific development environment, the Jupyter project is built upon a well-specified protocol to communicate with the Kernel, the part of the infrastructure responsible for executing the code.

For a programming language to leverage the potential of the Jupyter ecosystem, such as JupyterHub, JupyterLab, and interactive widgets, all that is needed is a Kernel to be created for that language that is, an executable implementing the specified inter-process communication. Dozens of kernels have already been implemented bringing Jupyter to many programming languages.

We are completing our engineering degree and interning at QuantStack. We recently attended the Jupyter Community Workshop on the kernel protocol that took place in Paris in late May. In this occasion, we set ourselves to write a new Jupyter kernel.

Today, we are proud to announce the first release of xeus-calc, a calculator kernel for Jupyter! xeus-calc is meant to serve as a minimal, self-containedexample of Jupyter kernel. It is built upon the xeus project, a modern C++ implementation of the protocol. This article is a step-by-step description on how the kernel was implemented.

You may find this post especially useful if you are creating a new programming language and you want it to work in Jupyter from the start.

Xeus

Implementing the Jupyter kernel protocol from scratch may be a tedious and difficult task. One needs to deal with ZMQ sockets and complex concurrency issues, rely on third-party libraries for cryptographically signing messages or parsing JSON efficiently. This is where the xeus project comes into play: it takes all of that burden so that developers can focus on the parts that are specific to their use case.

In the end, the kernel author only needs to implement a small number of virtual functions inherited from the xinterpreter class.

#include "xeus/xinterpreter.hpp"
#include "nlohmann/json.hpp"

using xeus::xinterpreter;
namespace nl = nlohmann;

namespace custom
{
    class custom_interpreter : public xinterpreter
    {
    public:

        custom_interpreter() = default;
        virtual ~custom_interpreter() = default;

     private:

        void configure() override;
        nl::json execute_request_impl(int execution_counter,
                                      const std::string& code, 
                                      bool silent, 
                                      bool store_history,
                      const nl::json::node_type* user_expressions,
                                      bool allow_stdin) override;
    
        nl::json complete_request_impl(const std::string& code,
                                       int cursor_pos) override;

        nl::json inspect_request_impl(const std::string& code,
                                      int cursor_pos,
                                      int detail_level) override;

        nl::json is_complete_request_impl(const std::string& code)
        override;

        nl::json kernel_info_request_impl() override;
    };
}

Typically, a kernel author will make use of the C or C++ API of the target programming language and embed the interpreter into the application.

This differs from the wrapper kernel approach documented in the ipykernel package where kernel authors make use of the kernel protocol implementation of ipykernel, typically spawning a separate process for the interpreter and capturing its standard output.

Jupyter kernels based on xeus include:

xeus-cling: a C++ kernel built upon the cling C++ interpreter from CERN
xeus-python: a new Python kernel for Jupyter.
JuniperKernel: a new R kernel for Jupyter based on xeus.

In this post, instead of calling into the API of an external interpreter, we implement the internal logic of the calculator in the kernel itself.

Exposing the xeus calculator to Jupyter

A calculator project

First, to implement your own Jupyter kernel, you should install Xeus. You can either download it with conda, or install it from sources as detailed in the readme.

Now that the installation is out of the way, let’s focus on the implementation itself.

Recall that the main class for the calculator kernel must inherit from the xinterpreterclass so that Xeus can correctly route the messages received from the front-end.

This class defines the behavior of the kernel for each message type that is received from the front-end.

kernel_info_request_impl: returns the information about the kernel, such as the name, the version or even a “banner”, that is a message that is prompted to console clients upon launch. This is a good place to be creative with ASCII art.
complete_request_impl: checks if the code can be completed, by that we mean semantic completion, and makes a suggestion accordingly. This way the user can receive a proposition for an adequate completion to the code he is currently writing. We did not use it during our implementation as you will see later, it is safe to return a JSON with a status value only, if you do not want to handle completion.
is_complete_request_impl: whether the submitted code is complete and ready for evaluation. For example, if brackets are not all closed, there is probably more to be typed. This message is not used by the notebook front-end but is required for the console, which shows a continuation prompt for further input if it is deemed incomplete. It also checks whether the code is valid or not. Since the calculator expects single-line inputs, it is safe to return an empty JSON object. This may be refined in the future.
inspect_request_impl: concerns documentation. It inspects the code to show useful information to the user. We did not use it in our case and went with the default implementation (that is to return an empty JSON object).
execute_request_impl: the main function. An execute_request message is sent by the front-end to ask the kernel to execute the code on behalf of the user. In the case of the calculator, this means parsing the mathematical expression, evaluating it and returning the result, as described in the next section.

Implementation of the calculator

First things first, we need to find a way to parse mathematical expressions. To do so, we turn the user input into Reverse Polish Notation (or RPN), a name full of meaning for the wisest among our readers (or at least the oldest) who used RPN calculators in high school.

The RPN, also called Postfix notation, presents the mathematical expression in a specific way : the operands go first followed by the operator. The main advantage of this notation is how it implicitly displays the precedence of operators.

Reverse Polish Notation illustration

The main logic of the calculator is provided by two main functions dealing respectively with parsing and evaluating the user expression and a third one for handling spaces in the expression.

First we have the parsing function (parse_rpn) transforming the expression into this representation. For this purpose we implement the Shunting-yard algorithm.

It is based on the use of a stack data structure to change the order of the elements in the expression, depending on their type : operator, operand or parenthesis.

Transforming a user expression into RPN

Now that we have the expression turned into RPN (with spaces delimiting operands and operators) we need to do the computation. For this purpose we have the function compute_rpn. Its implementation is based on a loop through a stringstream (hence the need for space delimiters) which performs operations in the right order.

Note that the result is not returned as an execute_reply message but is sent on a broadcasting channel instead, so that other clients to the kernel can also see it. The function execute_reply_impl actually returns the status of the execution only, as you may see in the code below.

nl::json interpreter::execute_request_impl(int execution_counter,
                                           const std::string& code,
                                           bool /*silent*/,
                                           bool /*store_history*/,
                                           nl::json /*user_exprs*/,
                                           bool /*allow_stdin*/)
{
    nl::json pub_data;
    std::string result = "Result = ";
    auto publish = [this](const std::string& name, 
                          const std::string& text) {
        this->publish_stream(name,text);
    };
    try
    {
        std::string spaced_code = formating_expr(code);
        result += std::to_string(compute_rpn(parse_rpn(spaced_code,
                                                       publish),
                                             publish));
        pub_data["text/plain"] = result;
        publish_execution_result(execution_counter,
                                 std::move(pub_data),
                                 nl::json::object());
        nl::json jresult;
        jresult["status"] = "ok";
        jresult["payload"] = nl::json::array();
        jresult["user_expressions"] = nl::json::object();
        return jresult;
    }
    catch (const std::runtime_error& err)
    {
        nl::json jresult;
        publish_stream("stderr", err.what());
        jresult["status"] = "error";
        return jresult;
    }
}

And that’s it for our calculator! It is as simple as that.

Yet remember that Xeus is a library, not a kernel by itself. We still have to create an executable that gathers the interpreter and the library. This is done in a main function whose implementation looks like:

int main(int argc, char* argv[])                       
{                           
    // Load configuration file                           
    std::string file_name = 
        (argc == 1) ? "connection.json" : argv[2];                           
    xeus::xconfiguration config = 
        xeus::load_configuration(file_name);                                                   
    
    // Create interpreter instance                           
    using interpreter_ptr = std::unique_ptr<xeus_calc::interpreter>;                           
    interpreter_ptr interpreter = 
        std::make_unique<xeus_calc::interpreter>();                              
    
    // Create kernel instance and start it                           
    xeus::xkernel kernel(config, 
                         xeus::get_user_name(), 
                         std::move(interpreter));                           
    kernel.start();                                                   
    return 0;
}

First, we need to load the configuration file. To do so, we check if one was passed as an argument, otherwise, we look for the connection.json file.

Then, we instantiate the interpreter that we previously set up. Finally, we can create the kernel with all that we defined beforehand. The kernel constructor accepts more parameters that allow customizing some predefined behaviors. You can find more details in the Xeus documentation. Start the kernel and we are good to go!

Now that everything is set, we can test out our homemade calculator kernel.

As you can see in the demonstration below, the code displays step-by-step how the computation is done with RPN. This is done with publish_streamstatements, which is equivalent to std::cout for the Jupyter notebook, very useful for debugging purposes.

The final result, a functional calculator!

You should now have all the information you need to implement your own Jupyter kernel. As you noticed, the Xeus library makes this task quite simple. All that you have to do is to inherit from the xinterpreter virtual class and implement the functions related to the messaging protocol. Nothing more is required.

This project can be found on GitHub. Feel free to contribute to the project if you wish to improve it, keeping in mind that xeus-calc should remain lean and simple!

Note that the current implementation only supports arithmetical operators. However it can be easily extended and we may add functional support in the near future.

Acknowledgments

We would like to thank the whole QuantStack team for their help throughout the process of making this blog post.

We are also grateful to the organizers of the Jupyter community workshop on kernels as we actually started to endeavor during the event.

About the authors

Vasavan Thiru is completing a master’s degree at Sorbonne Université Pierre & Marie Curie in applied mathematics for mechanics. He is currently interning as a scientific software developer at QuantStack.

Thibault Lacharme is finishing a master’s degree in Quantitative Finance at Université Paris Dauphine. Thibault is currently on his internship as a scientific software developer at QuantStack.

Jupyter Community Workshop: Jupyter Server Design and Roadmap Workshop

2019-04-30T23:58:00+00:00

We have some exciting news to share regarding the Jupyter Server Design and Roadmap Workshop as part of the Jupyter Community Workshop series funded by Bloomberg. This workshop will take place May 16th and 17th just outside Paris France at the IBM facility in Bois-Colombes.

Jupyter Notebook has proven to be a tremendous tool in the scientific and analytic computing spaces. It is widely used at universities and businesses alike, enabling the ability to interactively analyze and view data in various ways, quickly and easily. However, as computational capabilities improve, the need to move Notebook kernels closer to the compute resources also increases. As a result, new requirements for how a given solution is configured and deployed are introduced.

The Jupyter Server Design and Roadmap Workshop will focus on how we can bring together what has been learned over the years to address the needs of future environments, while clearly defining the separation between client and server. Items that will be discussed include:

What aspects of the current Jupyter Notebook framework should be considered as “the server”?
How will extensions be exposed and consumed?
Backwards compatibility is important. How can we move forward while retaining current capabilities?
How can we introduce the ability for others to provide kernel-deployment frameworks of their own and how those frameworks are discovered?
Basic improvements that bring the server up to date (e.g., async/await — particularly in kernel life-cycle management)
General multi-tenancy capabilities will be explored such that the server can serve more than just a single client.
How to convey kernel-specific parameters from the client, thru the server, to the kernel launch framework?

Should you be interested in joining us for this workshop, please fill out this Google Form. Space is limited.

Acknowledgements

We’d like to acknowledge Bloomberg for their generous support in making this workshop, and the entire series, possible. Thank you!

We’d also like to thank IBM for providing the facility and hosting the Jupyter Server Design and Roadmap Workshop.

Jupyter Community Workshop: Building upon the Jupyter Kernel Protocol

2019-04-04T09:52:00+00:00

We have some exciting news about the Jupyter Community Workshop on kernels!

The workshop will be held in Paris, France, from May 27th to May 29th 2019. The event is being hosted at Capital Fund Management(CFM), in the heart of Paris.

In this three-days event, we will have hands-on discussions, hacking sessions and technical presentations, with core Jupyter developers and custom language kernel authors. We will discuss the potential improvements and additions to the protocol, and work on common tools and infrastructure for testing and evaluating Jupyter kernels.

Should you be interested in joining us for this workshop, please fill this Google Form.

Why a Workshop on Jupyter Kernels?

The Jupyter Kernel protocol is one of the main extension points of the Jupyter ecosystem. Dozen of language kernels have been developed by the community, and these languages can now leverage other components of the stack, such as the notebook, the console, and interactive widgets.

Most Jupyter kernel authors rely on the reference documentation for the Jupyter kernel protocol, and reach out to the core team for questions on the different communication channels. One of the objectives of this event is to foster collaboration between kernel developers, on the development of common tools for testing and supporting all aspect of the protocol.

Beyond implementation of the current protocol, we will discuss the potential improvements and additions to the protocol, such as the support of debugging messages. Another issue to be discussed and worked on is the ability to parameterize kernelspecs, which has been extensively discussed at the last Jupyter developer meeting in Berkeley. Parameterized kernel specs could be used for kernels requiring extra command-line arguments. These arguments could be used to pass database connection credentials, or other metadata and parameters used for e.g. parallel computing frameworks.

Common infrastructure for testing kernels, call into remote kernels with the kernel gateway would also be in scope for this event.

Acknowledgements

This would not have been possible without the generous support provided by Bloomberg, who made this workshop series possible.

We are also grateful to CFM for gracefully hosting the Jupyter kernels community workshop.

A new Python kernel for Jupyter

2019-01-09T09:16:00+00:00

Project Jupyter aims at providing a consistent set of tools for interactive computing workflows across multiple programming languages. Jupyter projects are popular at all stages of a research project from the exploration phase to the communication of results and teaching.

The flagship project of Jupyter, the Notebook, and its modernized version, JupyterLab are web applications allowing the creation of documents including prose, executable code, and interactive visualizations.

The kernel is the part of the backend responsible for executing code written by the user in the web application. For example, in the case of a Python notebook, execution of the code is typically handled by ipykernel, the reference implementation.

History

Initially, the IPython project included everything from the notebook web application to the kernel and server implementation. Later on, the language-agnostic parts (the notebook format, messaging protocol, qtconsole, notebook web application) were split into separate projects with a clearer scope, together forming Project Jupyter.

The Big Split(TM) of IPython

Splitting IPython into multiple packages was a good decision in that it brought a clear separation of concerns between the projects. The ipykernel project was elected as the reference implementation of the Jupyter kernel protocol.

This came at a cost since ipykernel brought all the history and technical debt of IPython with it. Besides, Python is not a natural language for a base implementation. As a result, R and Julia kernels don’t use the reference implementation but use their own implementation of the protocol.

Furthermore, the standard implementation of the Jupyter Interactive Widgets lies in the Python package ipywidgets. That means that once you, as a kernel developer, implemented the Jupyter kernel protocol, you would still need to make your own backend for the interactive widgets if you want widgets support (and you should want it, honestly…).

To prevent this useless duplication of effort, which harms sustainability, we set ourselves to implement a solid reusable implementation of the Jupyter kernel protocol, xeus.

C++ implementation of the kernel protocol

Xeus is a C++ implementation of the Jupyter kernel protocol. It is not a kernel by itself but a library that helps kernel authoring. Xeus is very helpful when developing a kernel for a language that has a C or a C++API (Python, Lua, SQL, etc.). It takes the cumbersome task of implementing the Jupyter messaging protocol for you so you just can focus on the core interpreter tasks: executing code, inspecting, completing, etc.

C++ is a good choice for a standard implementation of the protocol, it is a common denominator of most of the languages out there, it has a massive developer community and is widely adopted in the industry for performance middleware applications.

With xeus, it is also very easy to reimplement core logic (server, kernel, history management etc.), if need be, by simply inheriting from xeus library classes.

Xeus is already known for being used as a base for the C++ kernel xeus-cling.

A xeus-based C++ Kernel

Xeus-cling is a xeus-based C++ kernel for the Jupyter Notebook. It’s very useful for teaching or learning C++. C++ has never been this interactive, thanks to the combined power of Jupyter, xeus, and cling. And even more impressive than executing interpreted C++, it supports interactive widgets, thanks to the xwidgets project.

Just like xeus is a C++ implementation of the Jupyter kernel protocol, xwidgets is a C++ implementation of the Jupyter Interactive Widgets protocol, and it can be used as a base for other implementations.

You can try it right now with binder, without the need of installing anything on your computer, just by clicking on this link.

Interactive C++ in the Jupyter Notebook

A xeus-based Python kernel

Today, I am pleased to announce a new Python kernel based on xeus: xeus-python.

Xeus-python supports error formatting, rich display, interactive widgets, input requests, code completion, code inspection, etc.

Thanks to xeus and pybind11, it was very straightforward to have a first simple version of a Python kernel written in C++. It took a couple of months to have the features listed above, with a relatively small amount of code (< 3000 lines of C++) and a good coverage of ipykernel’s features.

Simple code execution

Error formatting

Code completion

Ipywidgets support

Just like with xeus-cling, you can try it online without the need of installing anything on your computer following this link.

Why another Python kernel?

Obviously, xeus-python does not cover 100% of the features of ipykernel. For examples, IPython magics are not supported yet by xeus-python. However:

xeus-python is a lot lighter than ipykernel and IPython combined, which makes it a lot easier to implement new features on top of it. Our next goal is to augment the protocol to implement a Python debugger in JupyterLab.
xeus-based kernels are more versatile in that one can overload e.g. the concurrency model. This is something that Kitware’s SlicerJupyter project takes advantage of to integrate with the Qt event loop of their Qt-based desktop application.

Acknowledgments

The development of xeus, xwidgets, xeus-python, and related packages are led by QuantStack.

This development is sponsored by Bloomberg.

About the Author

Martin Renou is a Scientific Software Engineer at QuantStack. Before joining QuantStack, he studied at the French Aerospace Engineering School SUPAERO. He also worked at Logilab in Paris and Enthought in Cambridge. As an open source developer at QuantStack, Martin worked on a variety of projects, from xsimd, xtensor, and xframe in C++ to ipyleaflet and ipywebrtc in Python and JavaScript.

Introducing Jupyter Enterprise Gateway

2018-09-17T20:19:00+00:00

by Luciano Resende, Kevin Bates, Alan Chin

Yesterday, the Jupyter Steering Council voted to make Jupyter Enterprise Gateway a top-level Jupyter Project. I want to thank everyone for their contributions so far — code from my teammates at IBM and the community in general; advice from the Jupyter development team and mentors; and questions, issues, and requirements from end users.

As we become an official Jupyter project, I would like to take the opportunity to give an update on the project’s progress during our incubation period.

What is Jupyter Enterprise Gateway?

Jupyter Enterprise Gateway enables Jupyter Notebook to launch remote kernels in a distributed cluster, including Apache Spark managed by YARN, IBM Spectrum Conductor or Kubernetes.

Although Enterprise Gateway is mostly kernel agnostic, it provides out of the box configuration examples for the following kernels:

· Python using IPython kernel

· R using IRkernel

· Scala using Apache Toree kernel

Jupyter Enterprise Gateway does not manage multiple Jupyter Notebook deployments, for that you should look for JupyterHub. Having said that, Enterprise Gateway can enable JupyterHub to launch remote kernels as individual Kubernetes pods, providing better resource allocation and enabling better environment management as each pod can be based on different images (e.g. TensorFlow, Anaconda, etc)

Supported Platforms

Jupyter Enterprise Gateway currently enables remote kernels in the following platforms:

Distributed Kernels in Apache Spark

Jupyter Enterprise Gateway leverages different resource managers to enable distributed kernels in Apache Spark clusters. One example shown below describes kernels being launched in YARN cluster mode across all nodes of a cluster.

Jupyter Enterprise Gateway leverages Apache Spark resource managers to distribute kernels

Note that, Jupyter Enterprise Gateway also provides some other value-added capabilities such as enhanced security and multiuser support with user impersonation.

Jupyter Enterprise Gateway provides Enhanced Security and Multiuser support with user Impersonation

Distributed Kernels in Kubernetes

Jupyter Enterprise Gateway support for Kubernetes enables decoupling the Jupyter Notebook Server and its kernels into multiple pods. This enables running Notebook server pods with minimally necessary resources based on the workload being processed.

Jupyter Enterprise Gateway enable remote kernels on Kubernetes cluster

Jupyter Enterprise Gateway and JupyterHub

JupyterHub is a multi-user server that manages and proxies multiple instances of the single-user Jupyter notebook server. Particularly in a Kubernetes environment, Jupyter Enterprise Gateway can enable JupyterHub to launch remote kernels as individual Kubernetes pods, providing better resource allocation and enabling better environment management as each pod can be based on different images (e.g. TensorFlow, Anaconda, etc). This has proven to be very desired, particularly when working on Deep Learning related Notebooks.

JupyterHub and Jupyter Enterprise Gateway together in a Kubernetes cluster

Some project metrics

The following stats have been collected from the Jupyter Enterprise Gateway GitHub repository during the incubation period:

10 releases

12 individual contributors

90 Stars

34 Forks

Source code, documentation, and other community resources

The Jupyter Enterprise Gateway community provides multiple resources that both users and contributors can use:

Source Code available at GitHub
https://github.com/jupyter/enterprise_gateway

Documentation available at ReadTheDocs
http://jupyter-enterprise-gateway.readthedocs.io/en/latest/

Automated builds available at Travis.CI
https://travis-ci.org/jupyter/enterprise_gateway

Releases available at PyPi.org and Conda Forge
https://pypi.org/project/jupyter_enterprise_gateway/
https://github.com/conda-forge/jupyter_enterprise_gateway-feedstock

Related Docker Images available at Elyra organization at DockerHub
https://hub.docker.com/u/elyra/dashboard/

What’s next?

We are eager to build an even greater community around the project, and tailor the project roadmap based on community advise.

Currently, we are busy working on advancing our Kubernetes support and integration with JupyterHub.

As always, we welcome questions, comments, and suggestions from users and the community in general.