Jupyter Blog - QuantStack

Improving the accessibility of Jupyter

2023-02-24T11:11:00+00:00

Towards a more accessible Jupyter notebook

The adoption of Jupyter has grown significantly in the past few years — especially in education contexts. The project has become a foundational component of our digital experience, from the first lines of code written by high-school students to the most advanced research and engineering use cases.

In this context of global adoption, it is even more important to ensure that as many people as possible can use the project. It is estimated that 15% of the population has a disability that may impair their ability to use online services. If we don’t want them to be excluded from learning sciences, technology, and engineering, we must improve the tools to make them usable by everyone…

To address this issue, a Jupyter accessibility working group formed and has now become an official Jupyter Subproject and received a grant from the Chan Zuckerberg Initiative.

Creating accessible user interfaces is a challenging task as it requires accommodating a broad range of disabilities, including vision, motor, and cognitive impairments. This article presents some of the recent accessibility improvements in the Jupyter Notebook codebase.

CodeMirror 6 and Notebook 7

Jupyter Notebook 7, which is the next major release of the Jupyter Notebook frontend, has been rebuilt with modern JupyterLab components and extensions while maintaining the classic Jupyter Notebook user experience. Notebook 7 will be released shortly after JupyterLab 4 and will take advantage of JupyterLab improvements.

Improving the accessibility of Jupyter had long been impeded by significant obstacles. The primary obstacle was that the text editor underlying the Jupyter Notebook (CodeMirror 5) had major accessibility issues.

Fortunately, this accessibility bottleneck has been unblocked as JupyterLab has been upgraded to use CodeMirror 6, a complete rewrite of the text editor with a strong focus on accessibility. Although this upgrade required extensive codebase modifications, the changes landed a few months ago and will be available with JupyterLab 4. Jupyter Notebook 7 will benefit from the CodeMirror 6 upgrade.

Initial accessibility audits

Shortly after the CodeMirror 6 migration landed, we made an automated audit of the accessibility of Notebook 7 with these changes, and found that the number of warnings and errors reported by Axe Auditor went down from several hundreds to a few dozen, most of which seemed fixable. Encouraged by these results, we decided to work on bringing that count to zero!

With a series of fixes ranging from simple changes to the DOM structure of components to fixing up base Lumino components (Lumino is a JavaScript framework that underlies a lot of the JupyterLab frontend architecture), we were able to make the Notebook 7 codebase pass the Axe Auditor tests with zero error or warning. While we have not reached that yet in JupyterLab, both JupyterLab and Jupyter Notebook benefit from these improvements made for Notebook 7 since they are built from the same components.

Axe Auditor output with the “classic” Notebook UI (v6) with a test notebook.

Axe Auditor output with the “next” Notebook UI (v7 alpha 13) with the same test notebook.

Similar results were achieved with the file explorer and text editor tabs of the application.

Beyond accessibility audits

Improving the accessibility of Jupyter requires more than just addressing issues flagged by automated audit tools and the Jupyter Accessibility Subproject is working on improving accessibility across the board in Project Jupyter. These results with the Notebook 7 codebase are very encouraging, but working with end-users and getting their feedback will also be necessary to make the user interface truly accessible.

To address the diversity of accessibility requirements, we will approach specific use cases separately (for example, users with screen readers, or users who can operate a keyboard but not a pointing device). While the resulting changes may improve usability for everyone, we need to learn from users who have specific needs to make meaningful improvements.

Finally, many of the accessibility challenges in Jupyter stem from the content of the notebooks, as notebook authors may not follow the best practices to make their content usable by everyone (such as adding alt text to images and figures, or properly using headings to communicate the organization of a notebook). To promote these best practices, Jupyter could provide linting tools that produce inline warnings and hints for notebooks that do not follow these guidelines.

Acknowledgements

The work by Johan Mabille at QuantStack on migrating JupyterLab to use CodeMirror 6 was funded by Two Sigma. This upgrade also enabled significant performance improvements in the rendering of Jupyter notebooks, as detailed in this earlier post by Frédéric Collonval.

We are grateful to members of the Jupyter Accessibility Subproject who produced a thorough review of the Notebook 7 UI with a focus on keyboard navigation. This is an example of an issue that cannot be easily detected by auditing tools.

About the Authors

Nicolas Brichet is a scientific software developer at QuantStack active in the Jupyter ecosystem. Among other things, Nicolas made significant contributions to the JupyterLab and Voilà projects, and worked on porting the nbgrader package to JupyterLab. Nicolas worked on accessibility improvements in the JupyterLab and Lumino packages.

Johan Mabille is a technical director at QuantStack. He was honored with the Distinguished Contributors award in 2020 for his contributions to the Jupyter project. Among other things, he is one of the main authors of the JupyterLab visual debugger, and the creator of Xeus, a C++ implementation of the Jupyter kernel protocol at the basis of many Jupyter kernels. Johan was responsible for the migration of JupyterLab to CodeMirror 6.

Jeremy Tuloup is a technical director at QuantStack, and a Jupyter Distinguished Contributor. He is a core maintainer of JupyterLab and Voilà, and the creator of the JupyterLite project. Jeremy is the main initiator of the Notebook 7 project.

Frédéric Collonval is a technical director at QuantStack, and a Jupyter Distinguished Contributor. He is a core maintainer of the JupyterLab project. Frédéric contributed to the CodeMirror 6 migration and helped numerous new contributors to get their enhancements accepted.

Sylvain Corlay is the founder and CEO of QuantStack, and a Jupyter Distinguished Contributor. He has worked on many areas of the Jupyter project, from interactive widgets to language kernels and other core components.

Interactive GIS in Jupyter with ipyleaflet

2019-09-24T22:54:00+00:00

As Jupyter grew in popularity, a broad ecosystem of visualization packages based on Jupyter widgets has been developed, bringing even more interactivity to the Jupyter world.

In this article, we dive into Jupyter Interactive Widgets and the ipyleaflet package, an interactive maps visualization system for Jupyter.

A non-interactive map.

Jupyter Widgets

Jupyter Interactive Widgets are “special objects” that can be instantiated by the user in their code and result in a counterpart component being created in the front-end.

The core ipywidgets package provides a collection of controls that Jupyter users can use to build simple UIs as part of their notebooks (sliders, buttons, dropdowns, layout components).

Basic ipywidgets controls.

More than a collection of controls, it provides a framework upon which a large ecosystem of components has been built, allowing notebook authors to capture user inputs in very diverse ways.

Popular libraries built upon interactive widgets include

bqplot, a 2-D plotting system for Jupyter,
ipyvolume, a 3-D plotting package based on WebGL and ThreeJS,
PythreeJS, a 3-D scene description package exposing a large part of the ThreeJS API to Jupyter,
ipyvuetify, a large collection of VuetifyJS components exposed to Jupyter,
ipywebrtc, a library exposing the features of the WebRTC protocol to Jupyter kernels,
beakerx, a collection of widgets, extensions, and kernels for Jupyter,

and many more… This list is not comprehensive and dozens of other widget packages have been developed.

Key aspects of Jupyter widgets include:

BidirectionalityWidgets are not just meant for display but can also be used to capture user inputs, which can then trigger new computation. Notebook authors can compose sophisticated applications including a variety of components from different packages.
Language AgnosticismBuilt upon the Jupyter ecosystem, the interactive widget protocol used for the synchronization between the kernel and the front-end is well-specified and can be implemented for any kernel.
Back-ends for other languages than Python already exist, such as for C++ (with the xeus-cling Jupyter kernel), and languages of the JVM such as Clojure or Groovy (with the beakerx kernels).
ExtensibilityJupyter widgets are not meant as a monolithic system with one and only one way to achieve a specific task. We strive to provide a foundational layer allowing third-party widget authors to be as inventive as possible.

A common pattern for Jupyter widget packages has been to bring the capabilities of popular JavaScript visualization frameworks to Jupyter with a bridge based on ipywidgets. This is the case for the ipyleaflet package, as well as pythreejs and ipyvuetify.

Another use case is the development of ad hoc controls that are not necessarily relevant for a mainstream visualization package, but may be specific to a scientific field. An example is the ipymesh project by Chris Kees which can be used to draw PSLG (planar straight-line graphs) in the Jupyter notebook.

ipyleaflet

ipyleaflet is a Jupyter - LeafletJS bridge, bringing mapping capabilities to the notebook and JupyterLab.

Built as a bridge between the LeafletJS package and Jupyter, the ipyleaflet API maps to that of LeafletJS, bringing most of the core features of the package to Jupyter, and enabling a few popular LeafletJS extensions. A small difference is that following the Python coding style, ipyleaflet makes use of snake_case instead of CamelCase for attribute names.

ipyleaflet features

The main components to the library are layers and controls, respectively items to be displayed on the map, and interactive widgets overlayed on the map area for greater interactivity.

The first thing for which you may want to change the default value are the zoom level, the position, or the base layer for the map.

Changing the basemap layer in an ipyleaflet map.

Maps can be interactively edited in the Jupyter notebook, by dynamically changing or adding layers. In this screenshot, we add a custom layer including a GeoJSON dataset.

Note: An alternative layer to GeoJSON is GeoData, which lets the user load the data in the form of a GeoPandas dataframe instead of raw GeoJSON.

Adding a GeoJSON dataset to an interactive map.

A number of simple primitives are available such as markers and heatmaps. In the following screencast, we show how primitive properties can be linked with other widgets, and used as means to take user input on a map:

Linking properties of ipyleaflet primitives to other Jupyter widgets.

The splitmap control can be used to compare to different set of ipyleaflet layers at the same location.

The splitmap control.

Another interesting layer is the velocity layer which can be used to display wind velocity data. This control can take data in the form of an xarray dataset.

The velocity layer.

More classical visualization tools are also available, such as choropleths.

A choropleth layer.

JupyterLab Integration with the Sidecar and Theming Support

ipyleaflet is well integrated with JupyterLab

ipyleaflet controls make use of the JupyterLab themes for coloring so that they don’t stand out when using e.g. a dark theme.

The support for JupyterLab themes in ipyleaflet.

ipyleaflet can be used in combination with the JupyterLab sidecar widget.

Using the sidecar widget in JupyterLab to display a map aside of the notebook.

Together with the sidecar, a map can be programmatically added to the right-side toolbar of the JupyterLab application, and interactively edited in the notebook. This prevents the back-and-forth scrolling often required to see how changes are reflected visually.

The workflow enabled by ipyleaflet in combination with the sidecar is similar to that of the geonotebook project by Christopher Kotfila, Jonathan Beezley, and Dan LaManna from Kitware).

The geonotebook (which was based on the classic notebook) provided a similar workflow to the lab sidecar.

A more advanced example

Combined with other widget libraries such as bqplot or the core ipywidget package, ipyleaflet users can easily compose more complex applications and dashboards. In the following screencast, we explore the wealth-of-nations dataset with a leafletmap, a bqplot line chart and a dropdown widget:

Exploring the wealth-of-nations dataset with ipyleaflet, bqplot and core ipywidgets.

Trying ipyleaflet online with mybinder

If you would like to try out ipyleaflet now, it is possible thanks to the binder project. Just click on the image below!

Interactive GIS in C++: xleaflet

As mentioned earlier, interactive widgets back-ends for other programming languages have be implemented. There is a C++ back-end for ipywidgets: xwidgets which is the building block for creating other C++ widgets back-ends.

xleaflet is the C++ - LeafletJS bridge that exposes almost the same API as ipyleaflet, only that you use it from a C++ interpreter! It is based upon xwidgets which brings the bidirectional communication with the front-end.

You can learn more about the xeus-cling Jupyter kernel, xwidgets and xleaflet by reading the following blogpost.

xleaflet is the C++ backend to the jupyter-leaflet integration.

Deploying ipyleaflet-based dashboards with Voilà

Voilà is a tool that turns Jupyter notebooks into standalone dashboards. Built upon the Jupyter stack, it inherits the language agnosticism of the ecosystem, and can be used to produce standalone applications based on ipyleaflet.

A companion project to Voilà is the Voilà gallery project, a public facing set up of JupyterHub serving Voilà dashboard. It is kindly hosted by OVH. You can check out the Voilà gallery at URL https://voila-gallery.org.

For more resources about voilà, check out

the original announcement of the project: And Voilà!
and for the gallery: A Gallery of Voilà Examples.

If you check out the Voilà gallery, don’t miss Jeremy Tuloup’s GPX loader demo!

Jupyter for Geo Sciences

Jupyter’s adoption is exploding in the GeoScience space. Notable projects building upon Jupyter include

JEODPP (JRC Earth Observation Data and Processing Platform) is a EU project providing petabyte scale storage and high-throughput computing capacities to facilitate large scale analysis of Earth Observation data. The main front-end to the platform is base on Jupyter. The mapping capability is based on ipyleaflet. End users can request custom visualization that are returned to them in the form of lazily computed tile layers.
Pangeo is a community platform providing open, reproducible, and scalable sciences. The Pangeo software ecosystem involves open source tools such as xarray, Iris, Dask, Jupyter, and many other packages.
It was recently announced that the NSF would be funding the UC Berkeley + NCAR “EarthCube” proposal “Jupyter meets the Earth: Enabling discovery in geoscience through interactive computing at scale”. The plan involves the development of interactive dashboards with Voilà and contributions to the Voilà codebase.

Jupyter-gmaps

We should mention the jupyter-gmaps project by Pascal Bugnion. Jupyter-gmaps is a bridge between Google maps and Jupyter. Just like ipyleaflet, jupyter-gmaps is built upon the jupyter interactive widgets framework but relies on Google maps for the display instead of LeafletJS library.

This is a high-quality widget by another core developer of ipywidgets. Pascal is also one of the people behind Voilà and the Voilà gallery.

Folium

The Folium project enables maps visualization in the Jupyter notebook. Just like ipyleaflet, it is based on LeafletJS. Folium was created by Rob Story and is now maintained by Frank Conengmo and Filipe Fernandes.

A key difference between Folium and ipyleaflet is that ipyleaflet is built upon ipywidgets and allows bidirectional communication between the front-end and the backend enabling the use of the map to capture user input, while Folium is meant for displaying static data only. Folium enables many LeafletJS extensions, some of which may not be available in ipyleaflet at the moment.

Acknowledgements

The ipyleaflet project was started in 2015 by Brian Granger, and funded by the ERDC. The further development by Sylvain Corlay and Martin Renou was supported by QuantStack.

We are grateful to OVH for kindly hosting the Voilà gallery.

Other key contributors to Voilà and ipyleaflet that should be thanked here are Maarten Breddels, Pascal Bugnion, and Yuvi Panda. We should also mention the recent contributions by Vasavan Thirusittampalam who worked on the full-screen control and better interoperability with geopandas.

About the Authors

Sylvain Corlay, and Martin Renou are Scientific Software Developers at QuantStack.

Sylvain Corlay

Sylvain Corlay is the founder and CEO of QuantStack.

As an Open Source Developer, Sylvain contributes to Project Jupyter in the areas of interactive widgets and language kernels and is a steering committee member of the Project. Beyond QuantStack, Sylvain serves as a member of the board of directors of the NumFOCUS foundation. He also co-organizes the PyData Paris Meetup.

Martin Renou

Martin Renou is a Scientific Software Developer at QuantStack. Prior to joining QuantStack, Martin also worked as a Software developer at Enthought. He studied at the French Aerospace Engineering School ISAE-Supaero, with major in autonomous systems and programming.

As an open source developer, Martin has worked on a variety of projects, such as SciviJS (a JavaScript 3-D mesh visualization library), Xtensor, and Xeus.

Building a Calculator Jupyter Kernel

2019-07-16T08:36:00+00:00

A step-by-step guide for authoring language kernels with Xeus

An early device for interactive computing

In order to provide a language-agnostic scientific development environment, the Jupyter project is built upon a well-specified protocol to communicate with the Kernel, the part of the infrastructure responsible for executing the code.

For a programming language to leverage the potential of the Jupyter ecosystem, such as JupyterHub, JupyterLab, and interactive widgets, all that is needed is a Kernel to be created for that language that is, an executable implementing the specified inter-process communication. Dozens of kernels have already been implemented bringing Jupyter to many programming languages.

We are completing our engineering degree and interning at QuantStack. We recently attended the Jupyter Community Workshop on the kernel protocol that took place in Paris in late May. In this occasion, we set ourselves to write a new Jupyter kernel.

Today, we are proud to announce the first release of xeus-calc, a calculator kernel for Jupyter! xeus-calc is meant to serve as a minimal, self-containedexample of Jupyter kernel. It is built upon the xeus project, a modern C++ implementation of the protocol. This article is a step-by-step description on how the kernel was implemented.

You may find this post especially useful if you are creating a new programming language and you want it to work in Jupyter from the start.

Xeus

Implementing the Jupyter kernel protocol from scratch may be a tedious and difficult task. One needs to deal with ZMQ sockets and complex concurrency issues, rely on third-party libraries for cryptographically signing messages or parsing JSON efficiently. This is where the xeus project comes into play: it takes all of that burden so that developers can focus on the parts that are specific to their use case.

In the end, the kernel author only needs to implement a small number of virtual functions inherited from the xinterpreter class.

#include "xeus/xinterpreter.hpp"
#include "nlohmann/json.hpp"

using xeus::xinterpreter;
namespace nl = nlohmann;

namespace custom
{
    class custom_interpreter : public xinterpreter
    {
    public:

        custom_interpreter() = default;
        virtual ~custom_interpreter() = default;

     private:

        void configure() override;
        nl::json execute_request_impl(int execution_counter,
                                      const std::string& code, 
                                      bool silent, 
                                      bool store_history,
                      const nl::json::node_type* user_expressions,
                                      bool allow_stdin) override;
    
        nl::json complete_request_impl(const std::string& code,
                                       int cursor_pos) override;

        nl::json inspect_request_impl(const std::string& code,
                                      int cursor_pos,
                                      int detail_level) override;

        nl::json is_complete_request_impl(const std::string& code)
        override;

        nl::json kernel_info_request_impl() override;
    };
}

Typically, a kernel author will make use of the C or C++ API of the target programming language and embed the interpreter into the application.

This differs from the wrapper kernel approach documented in the ipykernel package where kernel authors make use of the kernel protocol implementation of ipykernel, typically spawning a separate process for the interpreter and capturing its standard output.

Jupyter kernels based on xeus include:

xeus-cling: a C++ kernel built upon the cling C++ interpreter from CERN
xeus-python: a new Python kernel for Jupyter.
JuniperKernel: a new R kernel for Jupyter based on xeus.

In this post, instead of calling into the API of an external interpreter, we implement the internal logic of the calculator in the kernel itself.

Exposing the xeus calculator to Jupyter

A calculator project

First, to implement your own Jupyter kernel, you should install Xeus. You can either download it with conda, or install it from sources as detailed in the readme.

Now that the installation is out of the way, let’s focus on the implementation itself.

Recall that the main class for the calculator kernel must inherit from the xinterpreterclass so that Xeus can correctly route the messages received from the front-end.

This class defines the behavior of the kernel for each message type that is received from the front-end.

kernel_info_request_impl: returns the information about the kernel, such as the name, the version or even a “banner”, that is a message that is prompted to console clients upon launch. This is a good place to be creative with ASCII art.
complete_request_impl: checks if the code can be completed, by that we mean semantic completion, and makes a suggestion accordingly. This way the user can receive a proposition for an adequate completion to the code he is currently writing. We did not use it during our implementation as you will see later, it is safe to return a JSON with a status value only, if you do not want to handle completion.
is_complete_request_impl: whether the submitted code is complete and ready for evaluation. For example, if brackets are not all closed, there is probably more to be typed. This message is not used by the notebook front-end but is required for the console, which shows a continuation prompt for further input if it is deemed incomplete. It also checks whether the code is valid or not. Since the calculator expects single-line inputs, it is safe to return an empty JSON object. This may be refined in the future.
inspect_request_impl: concerns documentation. It inspects the code to show useful information to the user. We did not use it in our case and went with the default implementation (that is to return an empty JSON object).
execute_request_impl: the main function. An execute_request message is sent by the front-end to ask the kernel to execute the code on behalf of the user. In the case of the calculator, this means parsing the mathematical expression, evaluating it and returning the result, as described in the next section.

Implementation of the calculator

First things first, we need to find a way to parse mathematical expressions. To do so, we turn the user input into Reverse Polish Notation (or RPN), a name full of meaning for the wisest among our readers (or at least the oldest) who used RPN calculators in high school.

The RPN, also called Postfix notation, presents the mathematical expression in a specific way : the operands go first followed by the operator. The main advantage of this notation is how it implicitly displays the precedence of operators.

Reverse Polish Notation illustration

The main logic of the calculator is provided by two main functions dealing respectively with parsing and evaluating the user expression and a third one for handling spaces in the expression.

First we have the parsing function (parse_rpn) transforming the expression into this representation. For this purpose we implement the Shunting-yard algorithm.

It is based on the use of a stack data structure to change the order of the elements in the expression, depending on their type : operator, operand or parenthesis.

Transforming a user expression into RPN

Now that we have the expression turned into RPN (with spaces delimiting operands and operators) we need to do the computation. For this purpose we have the function compute_rpn. Its implementation is based on a loop through a stringstream (hence the need for space delimiters) which performs operations in the right order.

Note that the result is not returned as an execute_reply message but is sent on a broadcasting channel instead, so that other clients to the kernel can also see it. The function execute_reply_impl actually returns the status of the execution only, as you may see in the code below.

nl::json interpreter::execute_request_impl(int execution_counter,
                                           const std::string& code,
                                           bool /*silent*/,
                                           bool /*store_history*/,
                                           nl::json /*user_exprs*/,
                                           bool /*allow_stdin*/)
{
    nl::json pub_data;
    std::string result = "Result = ";
    auto publish = [this](const std::string& name, 
                          const std::string& text) {
        this->publish_stream(name,text);
    };
    try
    {
        std::string spaced_code = formating_expr(code);
        result += std::to_string(compute_rpn(parse_rpn(spaced_code,
                                                       publish),
                                             publish));
        pub_data["text/plain"] = result;
        publish_execution_result(execution_counter,
                                 std::move(pub_data),
                                 nl::json::object());
        nl::json jresult;
        jresult["status"] = "ok";
        jresult["payload"] = nl::json::array();
        jresult["user_expressions"] = nl::json::object();
        return jresult;
    }
    catch (const std::runtime_error& err)
    {
        nl::json jresult;
        publish_stream("stderr", err.what());
        jresult["status"] = "error";
        return jresult;
    }
}

And that’s it for our calculator! It is as simple as that.

Yet remember that Xeus is a library, not a kernel by itself. We still have to create an executable that gathers the interpreter and the library. This is done in a main function whose implementation looks like:

int main(int argc, char* argv[])                       
{                           
    // Load configuration file                           
    std::string file_name = 
        (argc == 1) ? "connection.json" : argv[2];                           
    xeus::xconfiguration config = 
        xeus::load_configuration(file_name);                                                   
    
    // Create interpreter instance                           
    using interpreter_ptr = std::unique_ptr<xeus_calc::interpreter>;                           
    interpreter_ptr interpreter = 
        std::make_unique<xeus_calc::interpreter>();                              
    
    // Create kernel instance and start it                           
    xeus::xkernel kernel(config, 
                         xeus::get_user_name(), 
                         std::move(interpreter));                           
    kernel.start();                                                   
    return 0;
}

First, we need to load the configuration file. To do so, we check if one was passed as an argument, otherwise, we look for the connection.json file.

Then, we instantiate the interpreter that we previously set up. Finally, we can create the kernel with all that we defined beforehand. The kernel constructor accepts more parameters that allow customizing some predefined behaviors. You can find more details in the Xeus documentation. Start the kernel and we are good to go!

Now that everything is set, we can test out our homemade calculator kernel.

As you can see in the demonstration below, the code displays step-by-step how the computation is done with RPN. This is done with publish_streamstatements, which is equivalent to std::cout for the Jupyter notebook, very useful for debugging purposes.

The final result, a functional calculator!

You should now have all the information you need to implement your own Jupyter kernel. As you noticed, the Xeus library makes this task quite simple. All that you have to do is to inherit from the xinterpreter virtual class and implement the functions related to the messaging protocol. Nothing more is required.

This project can be found on GitHub. Feel free to contribute to the project if you wish to improve it, keeping in mind that xeus-calc should remain lean and simple!

Note that the current implementation only supports arithmetical operators. However it can be easily extended and we may add functional support in the near future.

Acknowledgments

We would like to thank the whole QuantStack team for their help throughout the process of making this blog post.

We are also grateful to the organizers of the Jupyter community workshop on kernels as we actually started to endeavor during the event.

About the authors

Vasavan Thiru is completing a master’s degree at Sorbonne Université Pierre & Marie Curie in applied mathematics for mechanics. He is currently interning as a scientific software developer at QuantStack.

Thibault Lacharme is finishing a master’s degree in Quantitative Finance at Université Paris Dauphine. Thibault is currently on his internship as a scientific software developer at QuantStack.

And voilà!

2019-06-11T17:42:00+00:00

… from Jupyter notebooks to standalone applications and dashboards

The goal of Project Jupyter is to improve the workflows of researchers, educators, scientists, and other practitioners of scientific computing, from the exploratory phase of their work to the communication of the results.

But interactive notebooks are not the best communication tool for all audiences. While they have proven invaluable to provide a narrative alongside the source, they are not ideal to address non-technical readers, who may be put off by the presence of code cells, or the need to run the notebook to see the results. Finally, following the order as the code often results in the most interesting content to be at the end of the document.

Another challenge with sharing notebooks is the security model. How can we offer the interactivity of a notebook making use of e.g. Jupyter widgets without allowing arbitrary code execution by the end user?

We set ourselves to solve these challenges, and we are happy to announce the first release of Voilà.

Voilà turns Jupyter notebooks into standalone web applications.

Voilà supports Jupyter interactive widgets, including the roundtrips to the kernel.
Voilà does not permit arbitrary code execution by consumers of dashboards.
Built upon Jupyter standard protocols and file formats, Voilà works with any Jupyter kernel (C++, Python, Julia), making it a language-agnostic dashboarding system.
Voilà is extensible. It includes a flexible template system to produce rich application layouts.

Installation and first-time use

Voilà can be installed from pypi:

pip install voila

or conda-forge:

conda install voila -c conda-forge

Upon installation, several components are installed, one of which is the voila command-line utility. You can try it by typing voila notebook.ipynb. It results in the browser opening to a new tornado application showing markdown cells, rich outputs, and interactive widgets.

From a notebook to a standalone web application

As you can see in the screencast, Jupyter interactive widgets remain fully functional even when they require computation by the kernel.

You can immediately try out some of the command-line options to Voilà

with --strip_sources=False, input cells will be included in the resulting web application (as read-only pygment snippets).
with --theme=dark, Voilà will make use of the dark JupyterLab theme, which will apply to code cells, widgets and all other visible components.

Making use of the dark theme and including input cells

Note that code is only shown, voilà does not allow users to edit or execute arbitrary code.

Voilà’s execution model

The execution model of Voilà is the following: upon connection to a notebook URL, Voilà launches the kernel for that notebook, and runs all the cells as it populates the notebook model with the outputs.

The execution model of Voilà

After the execution, the associated kernel is not shut down. The notebook is converted to HTML and served to the user. The rendered HTML includes JavaScript that establishes a connection to the kernel. Jupyter interactive widgets referred in cell outputs are rendered and connected to their counterpart in the kernel. The kernel is only shut down when the user closes their browser tab.

The current version of Voilà only responds to the initial GET request when all the cells have finished running, which may take a long time, but there is ongoing work on enabling progressive rendering, which should make it into a release soon.

An important aspect of this execution model is that the front-end does not determine what code is run by the backend. In fact, unless specified otherwise (with option --strip-sources=False), the source of the rendered notebook does not even make it to the front-end. The instance of the jupyter_server instantiated by Voilà actually disallows execute requests by default.

Support for custom interactive widgets

Voilà can render custom Jupyter widget libraries, including (but not limited to) bqplot, ipyleafet, ipyvolume, ipympl, ipysheet, plotly, ipywebrtc, etc.

Including bqplot figures with Voilà

Together with ipympl, Voilà is actually a simple means to render interactive matplotlib figures in a standalone web application:

Rendering interactive matplotlib figures in a web application with voilà

Voilà is language-agnostic

Voilà can be used to produce applications with any Jupyter kernel. The following screencast shows how Voilà can be used to produce a simple dashboard in C++ making use of leaflet.js maps, with the xeus-cling C++ kernel and the xleaflet package.

A standalone Voilà page making use of the C++ Jupyter kernel, xeus-cling (input cells display enabled).

We hope that Voilà will be a stimulant to other languages (R, Julia, JVM/Java) to provide stronger widgets support.

Richer layouts with Voilà templates

The main extension point to Voilà is the custom template system. The HTML served to the end-user is produced from the notebook model by applying a Jinja template, which can be defined by the user.

An example template for voilà is the voila-gridstack template, which can be installed from pypi with

pip install voila-gridstack

You can try it by typing voila notebook.ipynb --template=gridstack.

Making use of the Gridstack template to produce a dashboard with bqplot charts

The gridstack Voilà template makes use of the cell metadata to lay out the application.

A roadmap item for the gridstack Voilà template is to support the entire spec for the deprecated jupyter dashboards and to create a WYSIWYG editor for these templates in the form of a JupyterLab extension.

Note that voila-gridstack template is still at an early stage of development.

How to make custom Voilà templates?

A Voilà template is actually a folder placed in the standard directoryPREFIX/share/jupyter/voila/templates and which may include

nbconvert templates (the jinja templates used to transform the notebook into HTML)
static resources
custom tornado templates such as 404.html etc.

All of these are optional. It may also contain a conf.json file to set up which template to use as a base. The directory structure for a Voilà template is the following:

PREFIX/share/jupyter/voila/templates/template_name/
|
├── conf.json                # Template configuration file
├── nbconvert_templates/     # Custom nbconvert templates
├── static/                  # Static directory
└── templates/               # Custom tornado templates

The Voilà template system can be used to completely override the behavior of the front-end. One can make use of modern JavaScript frameworks such as React or Vue.js to produce modern UI including Jupyter widgets and outputs.

Another example template for Voilà is voila-vuetify, which is built upon vue.js:

The voila-vuetify template makes use of Vue.js

The voila-gridstack and voila-vuetify templates are still at an early stage of development, but will be iterated upon quickly in the next weeks as we are exploring templates.

A Jupyter server extension

Beyond the voila command-line utility, the Voilà package also include a Jupyter server extension, so that Voilà dashboards can be served alongside the Jupyter notebook application.

When Voilà is installed, a running Jupyter server will serve the Voilà web application under BASE_URL/voila.

The Jupyter Community Workshop on Dashboarding

From June 3rd to June 6th 2019, a community workshop on dashboarding with Project Jupyter took place in Paris. Over thirty Jupyter contributors and community members gathered to discuss dashboarding technologies and hack together.

The participants to the Paris Jupyter community workshop on dashboarding

Several dashboarding solutions such as Dash and Panel were presented during the workshop and featured at the PyData Paris Meetup which was organized on the same week.

The workshop was also the occasion for several contributors to start working on Voilà. Custom templates, a dashboard gallery, logos and UX mockups for JupyterLab extensions have been developed.

We will soon publish a more detailed post on the workshop, detailing the many tracks of development that have been explored!

What is coming?

There is a lot of planned work around Voilà in the next weeks and months. Current work streams include better integration with JupyterHub for publicly sharing dashboard between users, as well as JupyterLab extensions (a Voilà “preview” extension for notebooks, and a WYSIWYG editor for dashboard layouts). There are also ongoing discussions with the OVH cloud provider (which already supports binder by handling some of its traffic) on hosting a binder-like service dedicated to Voilà dashboards. So stay tuned for more exciting developments!

Last but not least, we are especially excited about what you will be building upon Voilà!

Acknowledgments

The development of Voilà and related packages at QuantStack is sponsored by Bloomberg.

We are also grateful to the attendees of the Jupyter Community Workshop on Dashboarding for their numerous contributions to Voilà!

We would like to thank Chris Holdgraf for his work on improving documentation, and integration with JupyterHub.

We should mention Yuvi Panda and Pascal Bugnion for getting the voila-galleryproject off the ground during the workshop. We are grateful to Zach Sailer for his continued work on improvingjupyter_server. We should finally not forget to mention the prior art by Pascal Bugnion with the Jupyter widgets server which was also an inspiration for Voilà.

About the Authors

Sylvain Corlay is the founder and CEO of QuantStack and a core team member for Project Jupyter.

Maarten Breddels is an independent scientific software developer partnering with QuantStack on numerous projects, and a core developer of Project Jupyter.

Authoring Custom Jupyter Widgets

2018-03-05T12:12:00+00:00

Guest post authored by Olivier Borderies, Olivier Coudray, and Pierre Marion

Jupyter interactive widgets enhance the notebook experience by allowing users to create graphical user interfaces. They enable richer interaction with the data and computing resources.

While the base ipywidgets library comes with a number of controls such as sliders, buttons, and dropdowns, it is in fact much more than a collection of basic controls: it is the foundation of a framework upon which one can build arbitrarily complex interactions.

Examples of custom widget libraries built upon the foundational package are

bqplot, a d3-Jupyter bridge, and a 2-D plotting library following the constructs of the Grammar of Graphics,
ipyleaflet, a leaflet-Jupyter bridge enabling maps visualization in the Jupyter notebook,
pythreejs, a 3-D visualization library bringing the functionalities of Three.js into the Jupyter notebook,
ipyvolume, a 3-D plotting library also based on Three.js enabling volume rendering, quiver plots and much more.

Jupyter Widgets, a Bridge Between Two Continents

Jupyter widgets provide a means to bridge the kernel and the rich ecosystem of JavaScript visualization libraries for the web browser. It is an amazing opportunity for scientific developers to use all these resources in their language of choice.

This article is meant to serve as a guide for developers interested in authoring a custom widget library and bridge the gap between the basic examples of the official documentation and fully-fledged visualization libraries like the ones listed above.

A number of resources are provided alongside the article, including the complete source code of the examples, notebooks, and Binder links.

In this post, we focus on the Python back-end, even though dozens of Jupyter kernels exist. The Python kernel is the reference implementation of the Jupyter protocol and remains the most featureful. We should also mention QuantStack’s Xeus, a native C++ implementation of Jupyter kernel protocol, which supports interactive widgets. Xeus is used as the foundation for the support of Jupyter widgets for the R and C++ kernels.

Xeus: C++ implementation of Jupyter kernel protocol

A Hands-On Guide

While the Jupyter community is thriving, and we start seeing a growth in the number of custom widget libraries as well. Although the learning curve from the examples of the official Jupyter documentation to authoring state-of-the-art libraries like the ones listed above is steep and may be intimidating.

We started on this path a few months ago and benefited from guidance from core Jupyter developers along the way. Now, we would like to share the lessons learned. We hope this will help turn this mountain trail into a new silk road!

Let us start with the ‘Hello World’ example widget from the ipywidgets documentation, before moving on to more advanced use cases.

1 — Improving on the Hello-World Example from the Documentation

This section is derived from the ipywidgets documentation. The code snippets are CC-0 licensed.

The idea behind Jupyter widgets is to enable a bi-directional communication channel between the kernel (back-end) and the JavaScript front-end. In Python, widgets are special objects which are automatically synchronized with a counterpart object in the JavaScript front-end. In the back-end, the change events are handled by the traitlets package, which implements the observer pattern, while on the JavaScript side, this is done with the Backbone.js library.

The front-end implementation follows the MVC (Model View Controller) pattern. This allows the rendering of the same widget in multiple cell outputs, where all views share the same model, analogous to printing out a string variable multiple times.

The official documentation includes a Hello-World example widget, which offers an example of synchronization from Python to JavaScript, but not the other way around. Our jupyter-widget-hello-world-binder example provides a slightly more advanced version of it which demonstrates the bi-directional communication between the Python kernel and the JavaScript front-end. You can experiment with this widget on Binder or simply check out the example notebook with nbviewer.

Let us briefly present the main aspects of the implementation.

The JavaScript front-end defines a custom view by extending the base DOMWidgetView class from the base package:

var HelloView = widgets.DOMWidgetView.extend({
        render: function() {
            this.value_changed();
            this.model.on('change:value', this.value_changed, this);
        },

        value_changed: function() {
            this.el.textContent = this.model.get('value');
        }
    });

The Python back-end extends the corresponding base class from the ipywidgets package:

class HelloWidget(widgets.DOMWidget):
    _view_name = Unicode('HelloView').tag(sync=True)
    _view_module = Unicode('hello').tag(sync=True)
    _view_module_version = Unicode('0.1.0').tag(sync=True)

    value = Unicode('Hello World!').tag(sync=True)

The value attribute get synchronized between the back-end and the front-end.

The value_changed callback, which changes the HTML representation of the widget is attached to changes of the value property with the line:

this.model.on('change:value', this.value_changed, this);

In the Hello-World example from the documentation, there is no way to change the JavaScript value from the notebook front-end. We have added this feature in order to illustrate the bi-directional synchronization between JavaScript and Python. The idea is to trigger an event in the browser which will update the JavaScript model and then automatically - that’s the magic of ipywidgets - the Python back-end. These additional lines trigger the model update:

// update the JavasScript model
this.model.set('value', formElement[0].value);

// sync with Python
this.touch();

Conversely, when changing the value from the Python kernel the corresponding JavaScript value gets updated, as explained above.

Check out the jupyter-widget-hello-world-binder repo for more information. The notebook includes additional details and comments.

2 — First Example Involving Bi-Directional Communication

In the previous section, widgets were entirely defined in the notebook. We now show how to move the implementation outside of the notebook document and produce a proper installable package.

To help you create your own custom widget, the Jupyter team provides a cookiecutter template, producing a custom Jupyter widget library containing all the boilerplate for packaging. The cookiecutter is initialized with the hello-world widget from the documentation.

We used the widget cookiecutter to put the hello-world widget presented in the previous section into a well-organized GitHub repository. It contains

the Python part in the first_widget folder
the JavaScript part in the js folder.

The README provides detailed instructions to install the package, together with information about how to enable the Jupyter extension, and tips for custom widget authors. It delves into the technical details a bit more than this overview blog post.

Now that the basics of two-way synchronization are covered, you can create arbitrarily complex widgets! The key is to identify the data you would like to synchronize between the front-end and the back-end. Then you can set up the events that will update this data when a change is detected using the building blocks above.

Now you can ‘widget-ify’ any JavaScript library!

2.2 — Barebones `setup.py`

The setup.py described in the official documentation tries to automate many of the build steps, at the cost of readability. Fortunately, packaging Jupyter widgets will work with the bare-bones setup.py we provide.

The gain is a clearer, lighter (~100 less lines) setup.py, giving you a better understanding of what is happening.
The drawback is that you need one extra step to install the widget from source.

# Example: d3-slider
$ git clone https://gitlab.com/oscar6echo/jupyter-widget-d3-slider.git
$ cd js
$ npm install
$ cd ..
$ pip install -e .

# Extra line `jupyter nbextension install`
$ jupyter nbextension install --py --symlink --sys-prefix jupyter_widget_d3_slider
$ jupyter nbextension enable --py --sys-prefix jupyter_widget_d3_slider

More importantly we thought that it was clearer to keep the build steps of the Python and JavaScript packages separated. Both build processes are well documented, making the steps easier to follow. Finally, the inclusion of compiled JavaScript bundles in the Python package, is more explicit.

2.3 — Side Note: Naming Conventions

Naming conventions for Python packages are covered by PEP8. They can be a bit tricky in the case of 2-words names like “first-widget”. Where to use underscore (_) or hyphen (-) ? Typically “-” is used in GitHub repositories, and URLs and JavaScript while any folder or file in a Python module can only contain “_”. In the case of a Jupyter widget there is an extra attention point: in the setup.py file, the data_files argument in the setup function is a list of tuples. For each, the first element (representing a path in the filesystem) contains “-” as it relates to JavaScript code while the paths in the second contain “_” as they represent paths in the Python package. For a full example see the first-widget repo and the detailed README.

3 — Increasingly Complex Widgets

We made the following three widgets, gradually adding complexity. The first two examples are meant as educational examples:

d3-slider, a d3.js-based slider,
drawing-pad, a 2-D drawing pad,

We hope that the last example will become more than a demonstration:

ipypivot, a visual Pivot Table UI

In each case, the GitHub repository includes a demo notebook and the required boilerplate for Binder.

3.1 — d3-slider

This custom d3-slider widget wraps a simple custom slider based on the fantastic d3.js library. You can run and try it on the Binder repo or watch it on nbviewer.

d3-slider widget

To install:

$ pip install jupyter_widget_d3_slider

3.2 — drawing-pad

This small drawing pad app, is inspired from this codepen. You can run and try it on the Binder repo or watch it on nbviewer.

drawing pad widget

To install:

$ pip install jupyter-drawing-pad

3.3 — ipypivot

The ipypivot widget, wraps the convenient PivotTable.js library. You can run and try it on the binder repo or watch it on nbviewer.

ipypivot widget

To install:

$ pip install ipypivot

$ conda install -c conda-forge ipypivot

NOTE: The PivotUI widget is a combo of a custom widget and core widgets. This modular approach is more flexible (and looks nicer) but the same features can be made in a single custom widget containing extra buttons and display fields. The alt branch of the repo contains this version.

3.4 — Including JavaScript Callbacks

The ipypivot widget is an example of a widget that transparently wraps a JavaScript library. Transparent in the sense that all parameters of the JavaScript API are exposed, including functions, which are exposed in the form of strings on the Python side. In Python, JavaScript functions can only be strings, so there is an eval() to convert them to actual JavaScript functions.

The benefit is that all the functionalities of the JS libraries are exposed to the Python users with a very thin API. Developers can ‘widget-ify’ a large array of interesting libraries, thereby boosting the productivity of a Jupyter notebook user.

The downside is naturally the security concerns of enabling arbitrary JavaScript code to be injected by the notebook users. It is less a concern in the context of notebooks being shared within a small team of coworkers.

We are currently exploring means to execute the user-provided arbitrary JavaScript function in a sandboxed fashion, for example using the iframe srcdoc field (Cf. this repo for an example, though not in a Jupyter widget context) and messages can be sent back and forth between the main page and an iframe with the Window.postMessage function (see this gist for a bare bones example).

3.5 — Enabling Jupyter Widgets By Default

The jupyter nbextension enable command, arguably cumbersome, has become unnecessary as Jupyter widgets can be enabled by default from notebook version 5.3 (included). See this PR.

In order to be future-proof, all the widgets in this article include the file which triggers this ‘automatic enable’, and require notebook >= 5.3. Thus the pip-installation of our widgets is a one-line command. However, in dev mode, you still need to install and enable the notebook example (see section 3.1 for an example).

If you are working with an older version of notebook (run jupyter notebook --version to check), you will have to run the following command after pip-installing a widget:

$ jupyter nbextension enable --py --sys-prefix name_of_the_widget

4 — Packaging and Publishing

4.1 — PyPI and npm

If you have followed the “best practices” so far, packaging shouldn’t be an issue. Indeed, the cookiecutter provides a template of an easily-packageable widget.

Once your package is ready, you can publish it on:

PyPI for the package to be pip-installable
npmjs for the JavaScript extension, which is necessary to use the widget as a standalone application (outside of the notebook), render it with nbviewer, and also in the JupyterLab context.

To do so, you can follow these instructions in the documentation of first-widget.

4.2 — conda-forge

Conda has several advantages over pip:

It is a general-purpose package manager, which allows for non-python dependencies.
Unlike pip, it also has a real dependency solver, which prevents breaking your environments when updating a single package.
It allows creating virtual environments to isolate your projects.
It allows for one-line installation of Jupyter extension, including the enabling of the extension as a “post-link” script (temporary advantage: see the previous section).

Conda packages are available on different channels. The default channel is administrated by Anaconda Inc. The usually recommended channel to upload open source projects is conda-forge, as this article from Anaconda announces. To add this channel to your conda configuration, run the following command:

conda config --add channels conda-forge

Several steps are needed to publish a package on conda forge, using a so-called ‘recipe’:

writing the recipe, which describes how to build the package along with the dependencies required for building and running it
testing the recipe
publishing the package on the conda-forge GitHub by forking their staged-recipes repo
maintaining the package

The conda doc and the conda-forge doc are very clear and give you much more detailed information about this subject. If you do not want to read the full doc, and jump straight to the necessary information for publishing a new package, you may want to have a look at our first-widget repo. The README contains a section describing the publishing process for conda-forge.

4.3 — Automatic Push Script

The sequence of steps to update the version of a Jupyter widget and do all the pushing to the various repositories is quite long.

run npm prepare to build the js in folder static/
Update version in __meta__.py
push package to pypi
compute package sha256
tag repo with version
push to GitHub / GitLab / Bitbucket including tag
push js to npmjs
push to conda-forge (which includes sha256 hash)

If you want to automate the process we advise to have a look at Maarten Breddels’ releash package (release with relish :-) ), and how it is used in the context of ipyvolume and ipysheet.

4.4 — Binder and nbviewer

nbviewer and Binder are two fantastic tools to share both static and live notebooks.

nbviewer requires a URL to a valid notebook JSON file — typically hosted on GitHub/GitLab. To make widgets render in nbviewer, you need (1) to make the JavaScript package for your widget available on npm, (2) to run the notebook with the corresponding JavaScript extension (with the same version), and (3) to save the notebook widget state (in the ‘Widgets’ tab) before pushing it to GitHub / GitLab.
Binder requires a URL to a GitHub repository containing notebooks and a manifest of the dependencies required to run this notebook, which is used to produce a Docker image including all the resources to run the notebook. Check out the mybinder.org documentation to know how exactly to make use of it. Another resource is the BinderHub documentation if you want to host your own deployment of Binder. Either way we also highly recommend the article Binder 2.0, a Tech Guide.

5 — Conclusion

Hopefully you will have learned something reading this article. We believe in the potential of Jupyter widgets and hope that this intermediate-level article will help getting more people involved in the development of the ecosystem.

Note: This article only covers the case of the classic Jupyter notebook. The integration with JupyterLab will be covered in a future article!

If you find bugs or are interested in improving the example widgets presented here, please do not hesitate contact the authors or open a pull request!

About the Authors

Alphabetical order:

Olivier Borderies, Société Générale
Olivier Coudray, Student at École Polytechnique
Pierre Marion, Student at École Polytechnique

The software presented in this post was built upon the work of a large number of people including the Jupyter team. We are especially grateful to Sylvain Corlay, Jason Grout, Paul Ivanov, and Steven Silvester from the Jupyter Steering Council, as well as Maarten Breddels and Pascal Bugnion.

Interactive Workflows for C++ with Jupyter

2017-11-29T16:33:00+00:00

Scientists, educators and engineers not only use programming languages to build software systems, but also in interactive workflows, using the tools available to explore a problem and reason about it.

Running some code, looking at a visualization, loading data, and running more code. Quick iteration is especially important during the exploratory phase of a project.

For this kind of workflow, users of the C++ programming language currently have no choice but to use a heterogeneous set of tools that don’t play well with each other, making the whole process cumbersome, and difficult to reproduce.

We currently lack a good story for interactive computing in C++.

In our opinion, this hurts the productivity of C++ developers:

Most of the progress made in software projects comes from incrementalism. Obstacles to fast iteration hinder progress.
This also makes C++ more difficult to teach. The first hours of a C++ class are rarely rewarding as the students must learn how to set up a small project before writing any code. And then, a lot more time is required before their work can result in any visual outcome.

Project Jupyter and Interactive Computing

The goal of Project Jupyter is to provide a consistent set of tools for scientific computing and data science workflows, from the exploratory phase of the analysis to the presentation and the sharing of the results. The Jupyter stack was designed to be agnostic of the programming language, and also to allow alternative implementations of any component of the layered architecture (back-ends for programming languages, custom renderers for file types associated with Jupyter). The stack consists of

a low-level specification for messaging protocols, standardized file formats,
a reference implementation of these standards,
applications built on top of these libraries: the Notebook, JupyterLab, Binder, JupyterHub
and visualization libraries integrated into the Notebook and JupyterLab.

Adoption of the Jupyter ecosystem has skyrocketed in the past years, with millions of users worldwide, over a million Jupyter notebooks shared on GitHub and large-scale deployments of Jupyter in universities, companies and high-performance computing centers.

Jupyter and C++

One of the main extension points of the Jupyter stack is the kernel, the part of the infrastructure responsible for executing the user’s code. Jupyter kernels exist for numerous programming languages.

Most Jupyter kernels are implemented in the target programming language: the reference implementation ipykernel in Python, IJulia in Julia, leading to a duplication of effort for the implementation of the protocol. A common denominator to a lot of these interpreted languages is that the interpreter generally exposes a C API, allowing the embedding into a native application. In an effort to consolidate these commonalities and save work for future kernel builders, we developed xeus.

Xeus is a C++ implementation of the Jupyter kernel protocol. It is not a kernel itself but a library that facilitates the authoring of kernels, and other applications making use of the Jupyter kernel protocol.

A typical kernel implementation using xeus would in fact make use of the target interpreter as a library.

There are a number of benefits of using xeus over implementing your kernel in the target language:

Xeus provides a complete implementation of the protocol, enabling a lot of features from the start for kernel authors, who only need to deal with the language bindings.
Xeus-based kernels can very easily provide a back-end for Jupyter interactive widgets.
Finally, xeus can be used to implement kernels for domain-specific languages such as SQL flavors. Existing approaches use a Python wrapper. With xeus, the resulting kernel won’t require Python at run-time, leading to large performance benefits.

Interpreted C++ is already a reality at CERN with the Cling C++ interpreter in the context of the ROOT data analysis environment.

As a first example for a kernel based on xeus, we have implemented xeus-cling, a pure C++ kernel.

Redirection of outputs to the Jupyter front-end, with different styling in the front-end.

Complex features of the C++ programming language such as, polymorphism, templates, lambdas, are supported by the cling interpreter, making the C++ Jupyter notebook a great prototyping and learning platform for the C++ users. See the image below for a demonstration:

Features of the C++ programming language supported by the cling interpreter

Finally, xeus-cling supports live quick-help, fetching the content on cppreference in the case of the standard library.

Live help for the C++standard library in the Jupyter notebook

We realized that we started using the C++ kernel ourselves very early in the development of the project. For quick experimentation, or reproducing bugs. No need to set up a project with a cpp file and complicated project settings for finding the dependencies… Just write some code and hit Shift+Enter.

Visual output can also be displayed using the rich display mechanism of the Jupyter protocol.

Using Jupyter’s rich display mechanism to display an image inline in the notebook

Another important feature of the Jupyter ecosystem are the Jupyter Interactive Widgets. They allow the user to build graphical interfaces and interactive data visualization inline in the Jupyter notebook. Moreover it is not just a collection of widgets, but a framework that can be built upon, to create arbitrary visual components. Popular interactive widget libraries include

bqplot (2-D plotting with d3.js)
pythreejs (3-D scene visualization with three.js)
ipyleaflet (maps visualization with leaflet.js)
ipyvolume (3-D plotting and volume rendering with three.js)
nglview (molecular visualization)

Just like the rest of the Jupyter ecosystem, Jupyter interactive widgets were designed as a language-agnostic framework. Other language back-ends can be created reusing the front-end component, which can be installed separately.

xwidgets, which is still at an early stage of development, is a native C++ implementation of the Jupyter widgets protocol. It already provides an implementation for most of the widget types available in the core Jupyter widgets package.

C++ back-end to the Jupyter interactive widgets

Just like with ipywidgets, one can build upon xwidgets and implement C++ back-ends for the Jupyter widget libraries listed earlier, effectively enabling them for the C++ programming language and other xeus-based kernels: xplot, xvolume, xthreejs…

xplot is an experimental C++ back-end for the bqplot 2-D plotting library. It enables an API following the constructs of the Grammar of Graphics in C++.

In xplot, every item in a chart is a separate object that can be modified from the back-end, dynamically.

Changing a property of a plot item, a scale, an axis or the figure canvas itself results in the communication of an update message to the front-end, which reflects the new state of the widget visually.

Changing the data of a scatter plot dynamically to update the chart

Warning: the xplot and xwidgets projects are still at an early stage of development and are changing drastically at each release.

Interactive computing environments like Jupyter are not the only missing tool in the C++ world. Two key ingredients to the success of Python as the lingua franca of data science is the existence of libraries like NumPy and Pandas at the foundation of the ecosystem.

xtensor is a C++ library meant for numerical analysis with multi-dimensional array expressions.

xtensor provides

an extensible expression system enabling lazy NumPy-style broadcasting.
an API following the idioms of the C++ standard library.
tools to manipulate array expressions and build upon xtensor.

xtensor exposes an API similar to that of NumPy covering a growing portion of the functionalities. A cheat sheet can be found in the documentation:

Scrolling the NumPy to xtensor cheat sheet

However, xtensor internals are very different from NumPy. Using modern C++ techniques (template expressions, closure semantics) xtensor is a lazily evaluated library, avoiding the creation of temporary variables and unnecessary memory allocations, even in the case complex expressions involving broadcasting and language bindings.

Still, from a user perspective, the combination of xtensor with the C++ notebook provides an experience very similar to that of NumPy in a Python notebook.

Using the xtensor array expression library in a C++ notebook

In addition to the core library, the xtensor ecosystem has a number of other components

xtensor-blas: the counterpart to the numpy.linalg module.
xtensor-fftw: bindings to the fftw library.
xtensor-io: APIs to read and write various file formats (images, audio, NumPy’s NPZ format).
xtensor-ros: bindings for ROS, the robot operating system.
xtensor-python: bindings for the Python programming language, allowing the use of NumPy arrays in-place, using the NumPy C API and the pybind11 library.
xtensor-julia: bindings for the Julia programming language, allowing the use of Julia arrays in-place, using the C API of the Julia interpreter, and the CxxWrap library.
xtensor-r: bindings for the R programming language, allowing the use of R arrays in-place.

Detailing further the features of the xtensor framework would be beyond the scope of this post.

If you are interested in trying the various notebooks presented in this post, there is no need to install anything. You can just use binder:

The Binder project, which is part of Project Jupyter, enables the deployment of containerized Jupyter notebooks, from a GitHub repository together with a manifest listing the dependencies (as conda packages).

All the notebooks in the screenshots above can be run online, by just clicking on one of the following links:

xeus-cling: the main xeus-cling example notebook,

xtensor: the C++ N-D array expression library in a C++ notebook,

xwidgets: the C++ back-end for Jupyter interactive widgets,

xplot: the C++ back-end to the bqplot 2-D plotting library for Jupyter.

JupyterHub is the multi-user infrastructure underlying open wide deployments of Jupyter like Binder but also smaller deployments for authenticated users.

The modular architecture of JupyterHub enables a great variety of scenarios on how users are authenticated, and what service is made available to them. JupyterHub deployment for several hundreds of users have been done in various universities and institutions, including the Paris-Sud University, where the C++ kernel was also installed for the students to use.

In September 2017, the 350 first-year students at Paris-Sud University who took the “Info 111: Introduction to Computer
Science” class wrote their first lines of C++ in a Jupyter notebook.

The use of Jupyter notebooks in the context of teaching C++ proved especially useful for the first classes, where students can focus on the syntax of the language without distractions such as compiling and linking.

Acknowledgements

The software presented in this post was built upon the work of a large number of people including the Jupyter team and the Cling developers.

We are especially grateful to Patrick Bos (who authored xtensor-fftw), Nicolas Thiéry, Min Ragan Kelley, Thomas Kluyver, Yuvi Panda, Kyle Cranmer, Axel Naumann and Vassil Vassilev.

We thank the DIANA/HEP organization for supporting travel to CERN and encouraging the collaboration between Project Jupyter and the ROOT team.

We are also grateful to the team at Paris-Sud University who worked on the JupyterHub deployment and the class materials, notably Viviane Pons.

The development of xeus, xtensor, xwidgets and related packages at QuantStack is sponsored by Bloomberg.

About the Authors (alphabetical order)

Sylvain Corlay, Scientific Software Developer at QuantStack

Loic Gouarin, Research Engineer at Laboratoire de Mathématiques at Orsay

Johan Mabille, Scientific Software Developer at QuantStack

Wolf Vollprecht, Scientific Software Developer at QuantStack

Jupyter Blog - QuantStack

Improving the accessibility of Jupyter

CodeMirror 6 and Notebook 7

Initial accessibility audits

Beyond accessibility audits

Acknowledgements

About the Authors

Interactive GIS in Jupyter with ipyleaflet

Jupyter Widgets

Key aspects of Jupyter widgets include:

ipyleaflet

ipyleaflet features

JupyterLab Integration with the Sidecar and Theming Support

A more advanced example

Trying ipyleaflet online with mybinder

Interactive GIS in C++: xleaflet

Deploying ipyleaflet-based dashboards with Voilà

Jupyter for Geo Sciences

Related Projects

Jupyter-gmaps

Folium

Acknowledgements

About the Authors

Sylvain Corlay

Martin Renou

Building a Calculator Jupyter Kernel

Xeus

A calculator project

And voilà!

Installation and first-time use

Voilà’s execution model

Support for custom interactive widgets

Voilà is language-agnostic

Richer layouts with Voilà templates

How to make custom Voilà templates?

A Jupyter server extension

The Jupyter Community Workshop on Dashboarding

What is coming?

Acknowledgments

About the Authors

Authoring Custom Jupyter Widgets

Jupyter Widgets, a Bridge Between Two Continents

A Hands-On Guide

1 — Improving on the Hello-World Example from the Documentation

2 — First Example Involving Bi-Directional Communication

2.1 — First Widget

2.2 — Barebones setup.py

2.3 — Side Note: Naming Conventions

3 — Increasingly Complex Widgets

3.1 — d3-slider

3.2 — drawing-pad

3.3 — ipypivot

3.4 — Including JavaScript Callbacks

3.5 — Enabling Jupyter Widgets By Default

4 — Packaging and Publishing

4.1 — PyPI and npm

4.2 — conda-forge

4.3 — Automatic Push Script

4.4 — Binder and nbviewer

5 — Conclusion

About the Authors

Interactive Workflows for C++ with Jupyter

Project Jupyter and Interactive Computing

Jupyter and C++

Acknowledgements

About the Authors (alphabetical order)

2.2 — Barebones `setup.py`