Jupyter Blog - xeus

GNU Octave Meets JupyterLite: Compute Anywhere, Anytime!

2025-10-16T15:04:00+00:00

We are thrilled to announce the newest member of our JupyterLite kernel ecosystem: Xeus-Octave. Xeus-Octave allows you to run GNU Octave code directly on your browser. GNU Octave is a free and open-source Scientific Programming Language that can be used to run Matlab scripts. In this article, we present the challenges encountered when targeting WebAssembly, the current state of the Xeus-Octave kernel, and the future plans for expanding the GNU Octave ecosystem.

Earlier this year, we introduced the JupyterLite kernel for R, Xeus-R-Lite. Much like R, cross-compiling GNU Octave to WebAssembly required the same custom toolchain to enable the compilation of Fortran code, combining LLVM Flang and Emscripten.

Similar to many other mathematically oriented language packages, GNU Octave requires a BLAS/LAPACK implementation. Fortunately, OpenBLAS and the Netlib implementations of BLAS/LAPACK had already been added to the emscripten-forge WebAssembly distribution. Initially, OpenBLAS was the preferred implementation, but for the successful compilation of Octave, Netlib LAPACK was selected as it presented fewer hurdles during the build process.

Cross-Compilation of GNU Octave

One of the complications of cross-compiling Octave to WebAssembly, which had not been encountered with the R source code, was the extensive use of Fortran common symbols blocks in the internal libraries of Octave such as odepack.

C Source: liboctave/external/odepack/slsode.f
C-----------------------------------------------------------------------
C The following internal Common block contains
C (a) variables which are local to any subroutine but whose values must
C     be preserved between calls to the routine ("own" variables), and
C (b) variables which are communicated between subroutines.
C The block SLS001 is declared in subroutines SLSODE, SINTDY, SSTODE,
C SPREPJ, and SSOLSY.
C Groups of variables are replaced by dummy arrays in the Common
C declarations in routines where those variables are not used.
C-----------------------------------------------------------------------
     COMMON /SLS001/ CONIT, CRATE, EL(13), ELCO(13,12),
    1   HOLD, RMAX, TESCO(3,12),
    1   CCMAX, EL0, H, HMIN, HMXI, HU, RC, TN, UROUND,
    2   INIT, MXSTEP, MXHNIL, NHNIL, NSLAST, NYH,
    3   IALTH, IPUP, LMAX, MEO, NQNYH, NSLP,
    3   ICF, IERPJ, IERSL, JCUR, JSTART, KFLAG, L,
    4   LYH, LEWT, LACOR, LSAVF, LWM, LIWM, METH, MITER,
    5   MAXORD, MAXCOR, MSBP, MXNCF, N, NQ, NST, NFE, NJE, NQU

Initially, it was not possible to cross-compile these common blocks to WebAssembly because the latest version of LLVM (v20 at the time of testing) did not support common symbol linkage.

// Source: llvm/lib/MC/MCWasmStreamer.cpp
void MCWasmStreamer::emitCommonSymbol(MCSymbol *S, uint64_t Size,
                                      Align ByteAlignment) {
  llvm_unreachable("Common symbols are not yet implemented for Wasm");
}

As a temporary solution, LLVM was patched with the help of Serge Guelton to simulate common symbols as weak symbols.

void MCWasmStreamer::emitCommonSymbol(MCSymbol *S, uint64_t Size,
                                      Align ByteAlignment) {
-  llvm_unreachable("Common symbols are not yet implemented for Wasm");
+  auto *Symbol = cast<mcsymbolwasm>(S);
+  getAssembler().registerSymbol(*Symbol);
+  Symbol->setWeak(true);
+  Symbol->setExternal(true);
}

A proper solution to enable support of common symbols is currently in progress and will likely be included in the next release of LLVM v22 (see llvm-project/pull/151478). For curious readers, the patched version of LLVM can be found here (linux only).

In addition to the patches for LLVM, GNU Octave required a few minor modifications to target WebAssembly; mainly this entailed disabling the GUI functionalities and consolidating the Fortran function signatures and calling conventions. A full list of patches can be found in the recipe directory on emscripten-forge.

Xeus-Octave

Once GNU Octave had been successfully packaged for WebAssembly, bringing Xeus-Octave to JupyterLite was a simple matter of adding a recipe to emscripten-forge!

Try Xeus-Octave in JupyterLite!

Future Work

For our next steps, the team is planning on expanding the Octave ecosystem by adding Octave packages to both conda-forge and emscripten-forge. The packaging work will require defining a process where Octave packages can be installed in predetermined conda environments, perhaps with some minor modifications to the existing pkg utility.

About the Author

Isabel Paredes, who led the charge on bringing GNU Octave to emscripten-forge, is a senior scientific software developer at QuantStack. Prior to working on this project, she focused on porting the R programming language and the Robot Operating System (ROS) framework to WebAssembly.

Acknowledgments

This project synthesizes work from many open-source contributors.

Emscripten-forge, the distribution of conda packages for WebAssembly, was created by Thorsten Beier, who continues to lead the project. Many recipes were contributed by Isabel Paredes, Anutosh Bhat, Martin Renou, Ian Thomas, Wolf Vollprecht, and Johan Mabille.
JupyterLite, the Jupyter distribution that runs entirely in the web browser, was created by Jeremy Tuloup.
Xeus, the C++ library implementing the Jupyter kernel protocol, enabling a custom communication layer, and is foundational to kernels like xeus-r, xeus-python, running in JupyterLite, was created by Johan Mabille and is maintained by a broader team including Martin Renou, Sylvain Corlay, and Thorsten Beier, who worked on the first integration with JupyterLite.
Xeus-Octave, the Xeus-based Jupyter kernel for GNU Octave, was created by Giulio Girardi and Antoine Prouvost.

R in the Browser: Announcing Our WebAssembly Distribution

2025-02-28T08:31:00+00:00

In the past few months, QuantStack has received funding from the Bill & Melinda Gates Foundation through a grant to CourseKata to develop support for the R programming language in JupyterLite and the Emscripten-forge distribution for WebAssembly.

Today, we are thrilled to announce the availability of the R programming language in the Emscripten-Forge distribution for WebAssembly, which enables R workflows in JupyterLite through the Xeus-R Jupyter kernel.

JupyterLite with an R Jupyter notebook in use

To experience R in JupyterLite, simply click on the link below:

In this article, we present the details of our efforts to enable R workflows in JupyterLite. We cover the challenges encountered in packaging R and its dependencies for this platform, the solutions we implemented, and the process of handling R mamba packages on the frontend.

Why WebAssembly? Scaling scientific computing workflows in Jupyter to millions of users

The main challenge in scaling Jupyter deployments lies in providing a live computing environment on the server side for each user session.

While Kubernetes-based JupyterHub deployments have been successfully implemented at a university scale (such as the UC Berkeley Data 8 class, or the JupyterHub deployment of Université Paris Saclay), they demand substantial computing resources and a skilled DevOps team to achieve and maintain this level of scalability.

In contrast, JupyterLite represents a fundamentally different paradigm. As a browser-based distribution of Jupyter, it runs language kernels directly in the browser, enabling significantly greater scalability and ease of deployment. Notable JupyterLite-based applications include:

A JupyterLite console integrated into NumPy’s front page, providing immediate access to NumPy functionality without requiring installation.
A similar console application on SymPy’s project page, fully hosted statically.
The official Jupyter website¹.
The examples in Scikit-Learn’s documentation.

These examples illustrate how WebAssembly can be transformative by providing ready-to-use computing environments for scientific workflows, in documentation, blog posts, and scientific communications. The WebAssembly environments are made available to the hundreds of thousands of monthly visitors of these websites without requiring any computing resources on the backend.

JupyterLite deployments displayed on the NumPy, SymPy, and Jupyter websites

[1] Even though it powers the deployment of Jupyter on the main Jupyter website, JupyterLite is not an officially sanctioned Jupyter project.

Beyond Python, enabling R in the browser, and more

While the initial focus of the JupyterLite team has been on the Python programming language, the JupyterLite infrastructure was designed to be language-agnostic from the outset. The same principle applies to the package management stack we have been contributing to over the past few years, namely, mamba and conda-forge.

When we began targeting the WebAssembly platform, we chose to build our efforts on the mamba and conda-forge ecosystem. This decision allowed us to provide customizable environments for WebAssembly and expand beyond Python. This enabled applications such as the JupyterLite terminal, created by Ian Thomas, which comprises a bash-like shell written in TypeScript called Cockle, and WebAssembly builds of native shell commands such as cat, grep, ls, and vim. You can read the earlier announcement introducing the JupyterLite terminal.

Today, we are excited to reveal the availability of R in emscripten-forge, enabling Jupyter kernels such as Xeus-R to be used in JupyterLite.

With the addition of R, emscripten-forge now provides a unified packaging solution for Python, R, and terminal applications for WebAssembly. This covers the scopes of Pyodide (a Python distribution for WebAssembly), WebR (an R distribution for WebAssembly), and Sandbox.bio (a shell emulator with WebAssembly commands).

Building R for Emscripten-forge

Emscripten-forge is a GitHub organization containing recipes to build conda packages for the emscripten-wasm32 platform; it is also the first distribution of conda packages targeting this platform. It is built upon a modernized stack, leveraging rattler-build (the successor to the conda-build package builder) and mamba. The emscripten-forge project, which was created and is still led by Thorsten Beier, is now a team effort covering a broad range of packages.

In order to build R with emscripten-forge, we first required a toolchain capable of converting Fortran and C/C++ code into WebAssembly. Since R relies on BLAS and LAPACK (which are written in Fortran) and many essential R packages wrap native libraries that use Fortran, this conversion capability was crucial.

Fortran

By default, the emscripten-forge toolchain can cross-compile any C/C++ packages into WebAssembly; however, the first challenge in building the R stack was expanding the toolchain to also compile Fortran code. The two compilers that we considered for this task are LFortran and Flang.

LFortran is a compiler built on top of LLVM and designed from the ground up as a cross-compiler. Despite its potential, LFortran is still in its early stages compared to Flang and does not yet fully support all Fortran features.
LLVM Flang is the Fortran compiler of the LLVM project. Unfortunately, it does not support cross-compilation.

Working with Serge Guelton and Kerim Birgi, we initially experimented with the LFortran compiler and contributed to addressing issues encountered when building Netlib LAPACK. The BLAS implementation from this package served as a valuable testing ground since BLAS is one of the core dependencies for R.

After careful consideration, we found that Flang, with its ability to compile a wider variety of Fortran projects, brought us closer to building the Fortran components of R to WebAssembly. George Stagg, the main developer of the WebR project, has written a detailed article on using Flang to produce WebAssembly, which was adapted to the (then) latest version of LLVM (v18). Inspired by George Stagg’s endeavors, Serge posted upstream contributions to the Flang project (#99465, #99822, #101242, #105589), resolving issues with 32-bit platforms that were impacting our effort to support WebAssembly. All of these changes will be included in LLVM/Flang 20.0.

The remaining patches which cannot be merged upstream at this point involve:

(1) enabling code generation that targets WebAssembly (198c0e1),
(2) encoding target-specific size definitions for various data types given that the host and target platforms have different architectures (1da66cd), and
(3) disabling support for 128-bit floating-point types (f20c7c0).

A version of (2) that could be contributed upstream is in the works, which would be one step closer to making WebAssembly an official target for Flang. Additionally, (3) will no longer be needed starting with Flang v20.

These patches were integrated by Axel Obermeier into the Flang build recipe for the conda-forge distribution, making the modified build accessible to everyone. It can be installed on Linux with mamba or micromamba by executing:

micromamba install conda-forge/label/emscripten::flang libllvm19 --no-channel-priority

The final ingredient for our toolchain involved creating a compatible Fortran runtime library. Since Flang implements its runtime library in C/C++, compiling this library with Emscripten proved to be straightforward. We added a recipe for libflang to emscripten-forge, which packages the runtime library and includes all the aforementioned patches.

In the following example, we invoke Flang on a simple Fortran script, and execute the resulting JavaScript and WebAssembly with NodeJS.

> ls
hello.f90  libFortranRuntime.a
> cat hello.f90
File: hello.f90
program hello
 print *, “Hello, Fortran!”
end program hello
> source ~/emsdk/emsdk_env.sh
> flang-new --target=wasm32-unknown-emscripten -c hello.f90 -o hello.o
> emcc hello.o libFortranRuntime.a -o hello.js
> ls
hello.f90  hello.js  hello.o  hello.wasm  libFortranRuntime.a
> node hello.js
Hello, Fortran!

With a working Fortran-to-WebAssembly toolchain, we managed to compile BLAS and LAPACK (See the LAPACK recipe). But before we could get started with building R, we needed to tackle the remaining R dependencies.

Walking up the stack

In R (v4.4), there are a handful of dependencies that are required to provide basic functionality; these are: libiconv, zlib, bzip2, xz, pcre2, and libcurl. Thanks to the packaging efforts led by Thorsten Beier and Wolf Vollprecht, these dependencies are available on emscripten-forge with the exception of libcurl. Because of its socket-based networking architecture and reliance on system-level resource access, libcurl faces fundamental incompatibilities with WebAssembly’s sandboxed environment constraints. Therefore, libcurl had to be disabled in the R source code. As a consequence, downstream R packages which depend on libcurl cannot be ported to WebAssembly until a suitable libcurl replacement is available.

Apart from the core dependencies, R also requires a BLAS implementation. Although external BLAS libraries can be used with R, the R code base contains linear algebra routines from Netlib BLAS and a subset of LAPACK where both libraries are written in C and Fortran. To simplify configuration, we opted for these internal implementations instead of using external BLAS and LAPACK libraries.

Furthermore, several key optional dependencies were cross-compiled to WebAssembly to enhance R’s graphical capabilities; these include libpng, libtiff, cairo, and pango, along with their respective subdependencies. One particularly challenging aspect of packaging these libraries was the cross-compilation of glib, which both cairo and pango depend on. Compiling glib was only possible thanks to the efforts of Kleis Auke Wolthuizen whose patches made glib compatible with Emscripten, and Johan Mabille who disabled multi-threading.

The R dependencies are summarized in the following table:

Table of R dependencies

Cross-compilation of R

Although R typically configures and builds seamlessly on most unix platforms, cross-compiling is not straightforward because R uses a bootstrap approach; first it creates a minimal version of R which is then used to run several R scripts to complete the build process. Bootstrapping is a challenge because the built binaries for the target platform (WebAssembly) are not executable (at least not directly) from the host platform. Therefore, the r-base package necessitates two build phases. First, we compile R for the host platform (Linux) with GCC and Flang; this generates the R and Rscript executables needed for the second phase. And then, we cross-compile to WebAssembly with Emscripten and Flang.

One of the major challenges of this setup was generating the R Data Base (.rdb) and R Data Index (.rdx) files for the internal packages of r-base (base, compiler, grDevices, graphics, grid, methods, parallel, splines, stats, stats4, tools, and utils). To accomplish this, it was necessary to dynamically load some of the internal shared libraries (tools, grDevices, graphics, utils, stats, and methods) which provide essential functionality to generate the data and index files. However, the R executables we created during the first phase of the build are incompatible with the WebAssembly shared libraries. In order to work around this, we temporarily replaced the WebAssembly libraries of the internal packages with the linux shared libraries we built during the first phase (See cross_libraries.sh). This makeshift solution is also used to cross-compile all other R packages; the setup is carried out by the activation script of the cross-r-base package (See activate-cross-r-base.sh).

Additionally, the R source code required a few modifications to enable cross-compilation to WebAssembly.

The Emscripten platform was added to select the correct configuration for the platform.
The dependency on libcurl was removed (as mentioned in the previous section) and as a consequence, the internet package which relies on libcurl was disabled.
System calls which are not compatible with our WebAssembly environment were disabled.
Cairo replaced Xlib as the default bitmap type to enable graphics.

The complete list of patches applied to R is available here: r-base patches.

The R package ecosystem

Building R itself was just the beginning. The next challenge was to provide a wide array of R packages to create a robust development environment. Fortunately, many packages are written in pure R and are available as “noarch” packages on conda-forge, making them directly compatible with emscripten-forge environments. However, for the R packages that require compilation, a package recipe needs to be created and added to emscripten-forge. Thanks to the help of Anutosh Bhat, the following packages have been compiled and are available to use.

r-askpass     r-digest    r-haven      r-magrittr  r-purrr    r-tidyr
r-base64enc   r-dplyr     r-hexbin     r-mass      r-rcpp     r-tzdb
r-bit         r-ellipsis  r-htmltools  r-matrix    r-readr    r-utf8
r-bit64       r-fansi     r-isoband    r-mgcv      r-rlang    r-vctrs
r-cachem      r-farver    r-jsonlite   r-mime      r-sp       r-vroom
r-cli         r-fastmap   r-later      r-nlme      r-stringi  r-xfun
r-colorspace  r-ggrepel   r-lattice    r-plyr      r-sys      r-yaml
r-data.table  r-glue      r-lazyeval   r-promises  r-tibble

Most of the time, cross-compiling an R package is very simple because the bulk of the work is handled by r-base. Building a package typically boils down to a single line:

$R CMD INSTALL $R_ARGS .

Occasionally, minor patches are applied to the package source code to enable the WebAssembly target platform or to simplify cross-compilation. Some R packages, such as r-nlme, contain Fortran code; for such packages, the custom flang compiler needs to be installed before the package can be cross-compiled. Should any package be needed for your use case, feel free to contribute them to emscripten-forge!

Xeus-R

Once a sufficient set of packages is available for the platform, the next step is to provide a development environment for end users. Our chosen tool for this purpose is Jupyter.

The most popular Jupyter kernel for R is IRkernel, which has been developed over several years. It includes complete ZeroMQ bindings for the R language and comprehensive coverage of the Jupyter protocol. However, since IRkernel is tied to ZeroMQ, we opted to work with the Xeus-R project instead, which was announced last year. Xeus-R was developed by Romain François in collaboration with our team and with JupyterLite in mind. It is built upon the Xeus library, which provides a native implementation of the Jupyter protocol and is adapted to work with the JupyterLite frontend.

Fortunately, the components of IRkernel responsible for the rich representation of R objects and rich display in Jupyter frontends were split into two separate R packages: IRdisplay and repr. These packages can be reused in Xeus-R. As a result, Xeus-R and IRkernel offer the same rich display functionality, ensuring that Jupyter notebooks created with one kernel will work with the other. However, Xeus-R provides a different implementation of the communication layer through jupyterlite-xeus, which enables integration with the JupyterLite stack. This integration was previously implemented for Xeus-Python and Xeus-Lua.

One key aspect of this integration is the populating of the in-memory file system used by JupyterLite kernels with the required runtime dependencies, particularly shared libraries. This process is done upon bootstrapping the kernel. For this purpose, Anastiasiia Sliusar and Martin Renou developed mambajs, which, in combination with jupyterlite-xeus, processes conda packages in the frontend and installs them in the kernel filesystem at startup. In WebAssembly, shared libraries need to be handled with special care in order to function as expected. Drawing inspiration from the Python community, we adapted the pioneering work of Pyodide to properly load shared libraries into the filesystem.

How to make a deployment with custom packages

The GitHub repository https://github.com/jupyterlite/xeus-lite-demo is a template for creating a JupyterLite deployment on GitHub pages that includes the packages specified in a conda environment.

The process is as follows:

Create a new repository from the GitHub template.
Enable the deployment on GitHub pages from a GitHub action, as shown in the README.
Edit the environment file to include the desired packages.

For example, to deploy an R kernel with coursekata installed, the environment.yml file would contain the following:

name: xeus-r
channels:
  - https://repo.mamba.pm/emscripten-forge
  - conda-forge
dependencies:
  - xeus-r
  - r-coursekata

The deployment linked at the beginning of this article was created with this template.

How to contribute

Contributors are always welcome! New R packages can be requested by opening an issue on emscripten-forge, or added directly by opening a pull request with the desired package recipe. For instructions on how to contribute, please visit https://emscripten-forge.org/.

What is in the works

Currently, we are developing the infrastructure to dynamically install emscripten-forge and conda-forge noarch packages in JupyterLite kernel environments. This will apply to all Xeus kernels that make use of these packages, such as Xeus-Python, Xeus-R, and Xeus-Lua. In the case of R, this may be directly bound to install.packages.

Additionally, we are working on more advanced features for the R kernel, including support for Jupyter interactive widgets and the split of the pure R parts of xeus-r into a separate package, which would facilitate a possible convergence with IRkernel.

Finally, other language kernels based on the same package management stack are also in the works. Stay tuned for future announcements!

About the Author

Isabel Paredes, who led the charge on bringing R to emscripten-forge, is a senior scientific software developer at QuantStack. Prior to working on this project, she focused on porting the Robot Operating System (ROS) framework to WebAssembly.

Acknowledgments

This effort brought together several endeavors from many open-source developers.

JupyterLite, the Jupyter distribution that runs entirely in the web browser, was created by Jeremy Tuloup.
Xeus, the C++ library implementing the Jupyter kernel protocol, enabling a custom communication layer, and is foundational to kernels like xeus-r, xeus-python, running in JupyterLite, was created by Johan Mabille and is maintained by a broader team including Martin Renou, Sylvain Corlay, and Thorsten Beier, who worked on the first integration with JupyterLite.
Emscripten-forge, the distribution of conda packages for WebAssembly, was created by Thorsten Beier, who continues to lead the project. Many recipes were contributed by Isabel Paredes (for the R ecosystem), but also Kerim Birgi, Anutosh Bhat, Martin Renou, Wolf Vollprecht, and Johan Mabille.
Xeus-R, the Xeus-based Jupyter kernel for R, was created by Romain François.
IRDisplay and repr are pure R packages providing rich display and rich mime type rendering of many core R types, which are used in xeus-r. These two packages stem from the IRkernel project, a Jupyter kernel for R written in R, which was started by Thomas Kluyver, and has been maintained over the years by Philipp Angerer and Jan Katins.
MambaJS, the library enabling the processing of conda packages in the frontend, was developed by Anastasiia Sliusar and Martin Renou.
Conda-forge, the main distribution of conda packages, includes a large number of “noarch” recipes, including for R packages, and is developed by a broad community of contributors worldwide.
WebR, an R distribution for WebAssembly, was created by George Stagg. Although not directly used in this project, it documented numerous patches and changes to the Flang compiler that were adapted for this effort.

Funding

The work by Isabel Paredes and other QuantStack team members on bringing R and Xeus-R to Emscripten-Forge was supported by the Bill & Melinda Gates Foundation through a grant to CourseKata. CourseKata is a nonprofit project dedicated to improving statistics and data science education through interactive, research-backed curricula that integrate professional tools like R and Jupyter notebooks. This grant was part of CourseKata’s broader mission to scale its innovative curriculum, equipping students with the skills and confidence to engage with statistics, data science, and coding in meaningful ways.

Meet Xeus-R: a future-proof Jupyter kernel for R

2024-01-22T17:20:00+00:00

JupyterLab session with a notebook powered by the Xeus-R kernel for R

The Jupyter ecosystem provides a versatile and user-friendly platform for data analysis and scientific computing, it has democratised access to powerful tools for researchers, scientists, educators, and data professionals.

The name Jupyter, a blend of Julia, Python, and R has always hinted at its relevance in multiple scientific programming languages and has since conquered new horizons thanks to its flexible and extensible design.

Underneath the interactive surface of Jupyter lie the Jupyter kernels, a set of language specific engines orchestrating the dynamic execution and exploration of code. Kernels implement the Jupyter protocol, a shared specification of the communication process between notebooks and kernels, defining how messages, commands and outputs are exchanged and processed.

The Xeus framework is a central piece of the puzzle, offering a reference to facilitate Jupyter kernel implementations. The speed, extensibility, and alignment with the official protocol make Xeus a perfect technology to implement new kernels.

Xeus-R

Today, we, a collaborative team led by Romain François and supported by QuantStack, are thrilled to announce the initial release of Xeus-R, a future-proof Jupyter kernel for R. Xeus-R builds upon the components of Xeus to facilitate the use of R in Jupyter notebooks and already supports typical capabilities: R code execution, inspection, completion, and contextual help.

The recent advancements in support for WebAssembly in the R world led by the WebR project, the flexible design of Xeus, and the rise of JupyterLite all contribute to our strong belief that Xeus-R has the potential to deliver Jupyter in-browser experience for data scientists using R.

Xeus-R is currently at the stage of a proof of concept, but we wanted to communicate about it early to broadcast our excitement about its potential, share some of our thoughts, and initiate a conversation.

Relationship with IRkernel

While the IRkernel project paved the way and is the current reference implementation of a Jupyter kernel for R, it does not take advantage of the Xeus infrastructure and includes a complete implementation of the Jupyter protocol, down to the composition of all of the JSON messages and most importantly how they are transported through ZeroMQ sockets.

In contrast, Xeus-R fully embraces and leverages the capabilities of Xeus, while borrowing some higher level concepts from IRkernel and reusing features from its dependent packages IRdisplay and repr.

We believe this path will benefit future development as it gives us the freedom to work on innovations that are specific to the needs of R users, and let Xeus absorb evolutions of the Jupyter protocol.

Key features

Xeus-R implements the basic building blocks of what you would expect from a Jupyter kernel. The results of a code cell, i.e. the value of its last expression are displayed in the output, taking advantage of ANSI escape codes for colours as were made popular by the crayon and later cli R packages.

Traditional and ggplot2-powered graphics are captured and displayed as images. The aspect ratio of plots can be controlled by setting the repr.plot.height and repr.plot.width options either globally, or locally via the cell_options() function.

Thanks to the scaffolding in place in the IRdisplay and repr packages, Xeus-R is also able to emit richer displays through mime bundles.

In contrast with IRkernel, the rich displays need to be explicitly requested with the display() function. There are cases however, such as HTML widgets, where automatic rich display makes the most sense because a text/plain representation is not valuable.

Xeus-R also supports contextual help and includes a basic completion system, based on the internal utils::.getTokenFromLine() system.

Installation and Getting Started

At the time of writing this article, Xeus-R is only available and tested to work on Unix-based systems. We consciously decided to focus initial efforts on features rather than platform comprehensiveness, but we should be able to fix this soon enough.

The easiest way to get up and running with Xeus-R is to install it from conda-forge:

mamba install xeus-r

Instructions to build from source are available in the github repository for Xeus-R: https://github.com/jupyter-xeus/xeus-r

Looking ahead: JupyterLite Integration

Another benefit of a Xeus-based approach is that it has been designed in a way that supports in-browser experience to power JupyterLite. Together with the recent developments around WebR, we can envision R becoming a first-class citizen in the JupyterLite world.

This is an exciting vision and will serve as our North Star ⭐️ for future developments of Xeus-R.

Conclusion

Join us in welcoming Xeus-R, a Jupyter kernel for R that leverages the robust foundation of Xeus architecture. R is a language of paramount importance for Data Science and Scientific Computing and it rightfully deserves its place as a first-class citizen within the Jupyter ecosystem.

We firmly believe that Xeus-R has the potential to become a reference implementation for an R kernel, well positioned to tackle the challenges of the ever-evolving landscape of in-browser Scientific Computing, a promising future led by the JupyterLite and WebR projects.

Acknowledgements

The work by Romain François on Xeus-R was funded by QuantStack.

Author Bio

Romain François is a prominent figure in the R community, recognised for his longstanding contributions to R programming and Data Analysis. With a profound passion for open-source software and commitment to advancing the capabilities of R, he has earned a well-deserved reputation as a respected expert.

Romain’s expertise encompasses a wide spectrum of R-related fields, including package development, data manipulation, and interoperability with other programming languages. He is widely recognized for his work on packages like dplyr, which has become a cornerstone for efficient data manipulation in R, and for his contributions to Rcpp, a library that bridges the power of C++ with R for optimizing R packages and enhancing performance.

Xeus-Lite

2021-10-25T09:40:00+00:00

JupyterLite is a JupyterLab distribution that runs entirely in the web browser, backed by in-browser language kernels. Xeus is C++ library for writing Jupyter kernels. In this blogpost we show how these two can work together.

JupyterLite

With vanilla Jupyter, kernels are running in separate processes. The communication between the kernels and the client is achieved with ZeroMQ. In contrast to this, JupyterLite runs entirely in the browser, including the language kernels. The following kernels are already implemented in the JupyterLite framework:

JavaScript Kernel
P5 Kernel: Since P5 is a JavaScript library, it is easy to provide a P5 kernel running in the browser.
A Python kernel based on pyodide called pyolite: While the two JavaScript based kernels are trivial to implement as in-browser kernels, pyodide is a Python interpreter compiled to WebAssembly via Emscripten. Not only Python itself has been compiled to WebAssembly, but also the whole Scientific Stack including packages like: Numpy, Scipy, Pandas and many more. An overview of packages can be found here. Furthermore any pure Python package can be installed from within pyolite.

Since JupyterLite runs entirely in the browser, it is very easy to try it out!

Xeus

Xeus is C++ library for writing Jupyter kernels. It is not a kernel but a library that simplifies the process of writing kernels. Xeus implements the Jupyter messaging protocol, so that kernel authors only need to implement a handful of methods related to the actual interpreter, like executing pieces of code and auto-completing.
With the help of Xeus, it is very easy to write Jupyter kernels . In fact, Xeus has already been used for making several language kernels:

xeus-cling is a Xeus-based C++ kernel.
xeus-python is a Xeus-based Python kernel.
xeus-lua is a Xeus-based Lua kernel.
xeus-sql / xeus-sqlite are Xeus-based kernel for SQL / SQLite databases.
xeus-robot is a Jupyter kernel for the Robot Framework .
LFortran is a modern interactive Fortran compiler with a Xeus-based Jupyter kernel.

Since it is very easy to write Xeus based kernels for Jupyter, we also want to leverage Xeus for writing JupyterLite kernels. This has been the main motivation for Xeus 2.0.

Xeus Architecture

Xeus is has three extension points as depicted below

The Server is the middleware component responsible for receiving and sending messages to the Jupyter client. The default server it is built upon ZeroMQ and handles the concurrency model of the application.
As we will see later, a custom server is needed to connect Xeus with JupyterLite.
The Interpreter provides the interface that kernel authors must implement.
The Debugger provides the interface that can to be implemented when the kernel shall have debug capabilities, but we do not yet support this for Xeus-JupyterLite-Kernels.

JupyterLite Xeus-Kernels

Traditionally, the Jupyter kernel and the Client to the Jupyter kernel run in separate processes as depicted below:

The communication between the kernels and the client (ie the Jupyter-server) are done with inter-process communication via ZeroMQ. But with JupyterLite the kernel runs directly in the browser. In that case, ZeroMQ is not required anymore to communicate with the frontend. This was the motivation for Xeus 2.0, where it is possible to build Xeus without ZeroMQ. To implement JupyterLite Kernels with Xeus, we need to implement a custom Server which handles the Emscripten/Embind-based communication layer

The overall architecture is best described with the picture below: The C++ Xeus-Kernel is compiled to WebAssembly with Emscripten / Embind. The WebAssembly compiled code runs in a dedicated WebWorker thread. The WebWorker communicates via Message Passing with a thin layer of TypeScript which implements the IKernel API of JupyterLite. The communication between JupyterLite and the JupyterLite k ernel is done entirely in JavaScript.

The C++ code needed to compile an existing xeus-kernel to WebAssembly is very minimal:

Furthermore one needs a thin layer of TypeScript / JavaScript to implement the JupyterLite IKernel API. A good example of this code is given in the JupyterLite Xeus-Lua repository. While most of the repositories code is boilerplate, one needs to provide a simple Dockerfile which compiles the Xeus-Kernel to WebAssembly.

Xeus-Lua

We recently released a Xeus based Lua kernel. Even though its worthwhile to have Xeus-Lua kernel for vanilla Jupyter, the main purpose of Xeus-Lua was to experiment with Xeus-based JupyterLite kernels.
We choose Lua as the first language to try out Xeus-based JupyterLite kernels since Lua is easy to compile to WebAssembly.
A JupyterLite instance with Xeus-Lua included can be found here

Xeus-Lua has built in support for interactive widgets

Xeus-Wren

Wren is relatively new “Classy Little scripting Language” created by the author of the great book “Crafting Interpreters”. Wren can be described with the following:

Think Smalltalk in a Lua-sized package with a dash of Erlang and wrapped up in a familiar, modern syntax.

We choose Wren as a second language to experiment with Xeus + JupyterLite since it is small, simple to compile to WebAssembly and easy to embed in C++. A JupyterLite instance with Xeus-Wren included can be found here.

Xeus-Wren has built in support for displaying data of various MIME-types

Outlook

There are many languages which can be compiled to WebAssembly like:

Julia: https://github.com/Keno/julia-wasm
Octave: https://news.ycombinator.com/item?id=22926120
C++: https://github.com/binji/wasm-clang
Python: https://github.com/pyodide/pyodide

Also databases as sqlite can be compiled to WebAssembly:

Sqlite: https://github.com/sql-js/sql.js/

Our goal is to provide Xeus-based JupyterLite kernels for these projects to make the JupyterLite ecosystem even more powerful.

Acknowledgements

We like to thank

Jeremy Tuloup, Madhur Tandon, Martin Renou, and Nicholas Bollweg for their work on JupyterLite.
Johan Mabille for Xeus and his help implementing the Xeus-Emscripten server.

About the Author

Thorsten Beier is a Scientific Software Engineer at QuantStack. Before joining QuantStack, he graduated in computer science at the University of Heidelberg and worked at the EMBL. As an open source developer, Thorsten worked on a variety of projects, from nifty and vigra in C++ to inferno, kipoi and ilastik in Python.

Xeus 2.0

2021-09-28T09:41:00+00:00

Announcing a major release of the Xeus library

We have just released Xeus 2. This is a major release of the library. While it includes backward-incompatible changes, they are very limited and upgrading your kernels should be relatively easy.

How to upgrade Xeus-based kernels

Let’s illustrate that with the code of a simple kernel before and after upgrading to Version 2. Most typically, only the kernel’s main file will have to be updated:

 25 diff --git a/example/src/main.cpp b/example/src/main.cpp
 26 index 29de437..eb8fdba 100644
 27 --- a/example/src/main.cpp
 28 +++ b/example/src/main.cpp
 29 @@ -21,12 +21,14 @@ int main(int argc, char* argv[])
 30      std::string file_name = (argc == 1) ? "connection.json" : argv[2];
 31      xeus::xconfiguration config = xeus::load_configuration(file_name);
 32 
 33 +    auto context = xeus::make_context<zmq::context_t>();
 34 +
 35      // Create interpreter instance
 36      using interpreter_ptr = std::unique_ptr<custom::custom_interpreter>;
 37      interpreter_ptr interpreter = interpreter_ptr(new custom::custom_interpreter());
 38 
 39      // Create kernel instance and start it
 40 -    xeus::xkernel kernel(config, xeus::get_user_name(), std::move(interpreter), xeus::make_xs    erver_zmq);
 41 +    xeus::xkernel kernel(config, xeus::get_user_name(), std::move(context), std::move(interpr    eter), xeus::make_xserver_zmq);
 42      kernel.start();
 43 
 44      return 0;

The first change is that you now have to instantiate and pass an additional argument to the kernel constructor: the context. Fortunately, Xeus provides a function to build such a context object, making the update straightforward. We will detail the reason for this change in the next section.

The second change is that the kernel constructor now requires a function to build the server while a default value was provided for it in Xeus 1.x. Again, the motivations for this change will be detailed in the next section. Finally the functions for building the servers have moved to different headers.

To summarize, here is the list of the breaking changes:

The kernel’s constructor now requires an additional argument, the context. Xeus provides a function to easily instantiate it.
The order of kernel’s constructor arguments has changed, and the function for building the server is now required.
The functions for building the servers have moved to the headers where the corresponding servers are defined.

Motivation for these changes

When we created Xeus, kernels were expected to run in separate processes. The communication between the kernels and the client was achieved with the ZeroMQ middleware framework, and it was natural to heavily rely on it in Xeus.

Recent developments have pushed in a different direction, where the kernel can run directly in the browser. In that case, a middleware library is not required anymore to communicate with the frontend. The architecture of Xeus was not suitable for this new kind of kernels. An update was required so that it is possible to build Xeus without the dependency on ZeroMQ.

About the developers

Johan Mabille is a scientific software developer at QuantStack. Johan is a co-author of Xeus, and developed the debugger extension to xeus-python.

Sylvain Corlay is the founder and CEO of QuantStack, and a core Jupyter developer. He co-authored Xeus and xeus-python.

From Jupyter to the Moon

2021-09-24T12:24:00+00:00

Jupyter architecture in a nutshell

A key principle in Jupyter’s design is language agnosticism, and one of the main extension points of the Jupyter ecosystem is the kernel, the part of the architecture responsible for executing the user’s code.

In the end, a Jupyter kernel is merely an executable implementing a well-defined inter-process communication protocol.

A reference implementation of the kernel is ipykernel, which provides the default Python kernel, and utilities to build other language kernels. Another reference implementation of the protocol is Xeus.

Xeus

Xeus is C++ library for writing Jupyter kernels. It is not a kernel but a library that simplifies the process of writing kernels. Xeus implements the Jupyter messaging protocol, so that kernel authors only need to implement a handful of methods related to the actual interpreter, like executing pieces of code and auto-completing. There is a long list of scripting languages which can be embedded in C/C++. With the help of Xeus, it is very easy to write kernels for such languages. In fact, Xeus has already been used for making several language kernels:

xeus-cling is a Xeus-based C++ kernel.
xeus-python is a Xeus-based Python kernel.
xeus-sql / xeus-sqlite are Xeus-based kernel for SQL / SQLite databases.
xeus-robot is a Jupyter kernel for the Robot Framework .
LFortran is a modern interactive Fortran compiler with a Xeus-based Jupyter kernel.

xwidgets

The xwidget library is a C++ implementation of the Jupyter Interactive Widget protocol. Xwidgets were initially developed for xeus-cling, the Xeus C++ kernel, but any Xeus-based kernel can reuse these components.

A xeus-based Lua kernel

Today, I am pleased to announce the release of xeus-lua, a new Jupyter kernel for Lua based on Xeus.

Lua

Lua, which means “Moon” in Portuguese, is a lightweight embeddable scripting language written in C. While Python is one of the leading scripting languages for scientific application, Lua is the leading scripting language used in games / game engines. The list of video games using Lua is long and includes triple-a games like Crisys and Farcry. Not only games rely on Lua, but also applications like Adobe Photoshop Lightroom.

Sol2

With the Sol2 library, Lua can be embedded into C++ in a very elegant way.

While Lua itself can be embedded in a C++ application quite easy, it is even easier when using Sol2. Sol2 is a C++ library that simplifies the usage of Lua from C++. Furthermore Sol2 can be used to expose C++ classes to Lua, in a similar fashion as pybind11 can be used to expose C++ classes to Python.

Xeus-lua

The compulsory “Hello, World!” from xeus-lua. This blog-post` would be incomplete without it.

I am pleased to announce xeus-lua, a new Jupyter kernel for the Lua language based on Xeus. Using Sol2 and Xeus, it was very easy to implement a feature rich kernel using modern C++. With the help of Sol2, it was trivial to expose the xwidgets classes to Lua such that we can use the widgets within xeus-lua notebooks.

But why a Lua kernel?

While Lua might not be the hippest language in 2021, there are multiple reasons while I choose to implement such a Lua kernel, and why such a kernel can be useful:

Writing a Xeus-based kernel is the best way to learn and familiarize oneself with the xeus ecosystem. And since Lua is so easy to build and embed, Lua and Xeus are perfect match. Implementing this kernel gave me great insights into Xeus and Jupyter itself.
Learning Lua with xeus-lua is fun! Before writing xeus-lua, I had zero Lua experience. While implementing xeus-lua I learned Lua from within Jupyter by playing around with xwidgets from within Lua — this was a very joyful experience.
The xeus-lua implementation might serve as reference for future kernel authors. In particular the integration of xwidgets in xeus-lua might be useful for future implementations.
Since Lua is so easy to build, xeus-lua can be built for WebAssembly and will be integrated in JupyterLite — stay tuned!

Features

Obviously xeus-lua supports the execution of code.

Simple code execution

Streams are redirected to the notebooks output

Redirection of streams

Errors messages are shown in the notebooks output

Error Handling

Input streams are redirected to the notebooks input system

User Input

Statements can be auto-completed with a press of tab

Code Completion

Data of different sources as Latex, JSON and html can be displayed.

Display Data

Interactive widgets can be added to the notebook!

Widgets

We can feed the widgets with data!

Binary buffers for widgets

Try It Out

You can try it right now with binder, without the need of installing anything on your computer, just by clicking on this link.

Other Lua kernels

There are several Lua kernels available:

lupyter: A Python based implementation of a Lua Jupyter kernel. The heavy lifting is done in a C-extension.
ILua: A feature rich implementation. Ilua is implemented in Python and does the communication with Lua via IPC.
IPyLua: IPyLua is a pure Lua implementation.

While these implementations are feature rich and work well, none of the above supports interactive widgets. Since xeus-lua is based on Xeus, we can reuse components from the Xeus ecosystem like xwidgets with minimal effort.

Acknowledgments

We are grateful to Juliette Taka for the illustration.

About the Author

Thorsten Beier is a Scientific Software Engineer at QuantStack. Before joining QuantStack, he graduated in computer science at the University of Heidelberg and worked at the EMBL. As an open source developer, Thorsten worked on a variety of projects, from nifty, and vigra in C++ to inferno, kipoi and ilastik in Python.

Abracadabra! Bringing the magics to xeus-python

2021-02-18T13:30:00+00:00

Last year, we set ourselves to implement a visual debugger for JupyterLab. This endeavor required major developments in the JupyterLab front-end, in core-Jupyter protocols, and on the kernel side (the part of the Jupyter infrastructure responsible for executing the code).

For kernel-side development, we decided to start with the xeus-python kernel, a lightweight implementation of a Jupyter kernel for the Python programming language. Based on xeus, xeus-python has a flexible concurrency model that was required to e.g. add breakpoints while executing code.

Unfortunately, xeus-python did not provide all ipykernel features (magics, Matplotlib support, etc.). Furthermore, many notebooks depend on IPython, as they import it explicitely or make use of the IPython configuration system.

Today, we are proud to announce that xeus-python supports 100% of the IPython magics! This was achieved by leveraging the core IPython package. This is getting us closer to feature parity with ipykernel.

Magics

xeus-python now supports all magics that IPython provides and even user-defined magics!

Support for builtin IPython magics and user-defined magics

Matplotlib

Most Matplotlib back-ends are now supported by xeus-python, allowing you to show static plots with the inline back-end or interactive plots with ipympl in your Notebook:

Matplotlib inline back-end and ipympl interactive back-end

Extending

IPython has a rich configuration system that you can tweak in order to get an enhanced Notebook experience. Xeus-python now supports any configuration you setup for IPython:

Automatically import NumPy in the IPython settings

Visual Debugging

The latest JupyterLab version introduced a visual debugger in its interface. xeus-python was the first Jupyter kernel to add support for it!

Stepping into Python code in JupyterLab with the visual debugger

xeus-python is not the only Jupyter kernel that supports debugging anymore: xeus-robot is a xeus-based Jupyter kernel for the RobotFramework language that supports the JupyterLab visual debugger.

Try it online

Thanks to MyBinder, you can try it out without the need of installing anything on your computer. Just follow this link:

Installation

You can install the latest xeus-python version using mamba or conda:

mamba install xeus-python -c conda-forge

conda install xeus-python -c conda-forge

You can also compile it yourself, instructions can be found in the repository:
https://github.com/jupyter-xeus/xeus-python

Acknowledgments

My work on xeus and xeus-python at QuantStack was funded by Bloomberg.

About the Author

My name is Martin Renou, I am a Scientific Software Engineer at QuantStack. Before joining QuantStack, I studied at the aerospace engineering school SUPAERO in Toulouse, France. I also worked at Logilab in Paris, France and Enthought in Cambridge, UK. As an open-source developer at QuantStack, I work on a variety of projects, from xtensor and xeus-python in C++ to ipyleaflet and bqplot in Python and JavaScript/TypeScript.

Xeus is now a Jupyter subproject

2020-02-04T12:08:00+00:00

It is a great pleasure to announce that the Xeus project has been incorporated as a Jupyter subproject. Xeus will now be subject to the Jupyter governance and code of conduct.

For reference, the Jupyter Enhancement Proposal (JEP) for the Xeus incorporation is available here.

What is Xeus?

The Xeus project is a C++ implementation of the Jupyter kernel protocol. Xeus is not a kernel, but a library meant to facilitate the authoring of kernels.
Several Jupyter kernels have been created with Xeus:

xeus-cling, a kernel for the C++ programming language, based on the Cling C++ interpreter. The cling project comes from CERN and is at the foundation of the ROOT project.

The xeus-cling Jupyter kernel for the C++ programming language.

xeus-python, a kernel for the Python programming language, embedding the Python interpreter.

The xeus-python Jupyter kernel for the Python programming language

xeus-calc, a calculator kernel, meant as an educational example on how to make Jupyter kernels with Xeus.

Beyond these three kernels built on top of Xeus by the Xeus maintainers, third-parties have developed other Jupyter kernels with Xeus:

JuniperKernel, a kernel for the R programming language by Spencer Aiello.
xeus-fift, a kernel for the fift programming language by Michael Zaikin. The fift programming language was developed by Telegram to create TON blockchain contracts.
SlicerJupyter, a kernel for the Python programming language by Kitware which integrates into the Qt event loop of the Kitware “Slicer” project.

Finally, the xeus-python kernel includes a first implementation of the Jupyter debugger protocol used by the Jupyter debugger project. xeus-python enables the Debug Adapter Protocol over the Control channel through new debug request/reply and debug event messages.

Why moving Xeus under the Jupyter governance?

While Xeus started as a side project for QuantStack engineers, the project now has several stakeholders who depend on it. We think that moving the project to an open governance organization may be a better way to reflect this situation.

Acknowledgements

Xeus was started by the team of open-source developers at QuantStack as a separate project, but with the full intent to incorporate it into Jupyter. The initial project development at QuantStack was funded by Bloomberg.
Now, Xeus contributors work in many institutions, including Université Paris-Saclay and École Polytechnique.

About the Author

Johan Mabille is a Scientific Software Developer at QuantStack, specializing in high-performance computing in C++. He holds master’s degree in computer science from Centrale-Supelec.

As an open source developer, Johan coauthored xtensor, xeus, and xsimd. He also made major contributions to the JupyterLab debugger project and bqplot.

Building a Calculator Jupyter Kernel

2019-07-16T08:36:00+00:00

A step-by-step guide for authoring language kernels with Xeus

An early device for interactive computing

In order to provide a language-agnostic scientific development environment, the Jupyter project is built upon a well-specified protocol to communicate with the Kernel, the part of the infrastructure responsible for executing the code.

For a programming language to leverage the potential of the Jupyter ecosystem, such as JupyterHub, JupyterLab, and interactive widgets, all that is needed is a Kernel to be created for that language that is, an executable implementing the specified inter-process communication. Dozens of kernels have already been implemented bringing Jupyter to many programming languages.

We are completing our engineering degree and interning at QuantStack. We recently attended the Jupyter Community Workshop on the kernel protocol that took place in Paris in late May. In this occasion, we set ourselves to write a new Jupyter kernel.

Today, we are proud to announce the first release of xeus-calc, a calculator kernel for Jupyter! xeus-calc is meant to serve as a minimal, self-containedexample of Jupyter kernel. It is built upon the xeus project, a modern C++ implementation of the protocol. This article is a step-by-step description on how the kernel was implemented.

You may find this post especially useful if you are creating a new programming language and you want it to work in Jupyter from the start.

Xeus

Implementing the Jupyter kernel protocol from scratch may be a tedious and difficult task. One needs to deal with ZMQ sockets and complex concurrency issues, rely on third-party libraries for cryptographically signing messages or parsing JSON efficiently. This is where the xeus project comes into play: it takes all of that burden so that developers can focus on the parts that are specific to their use case.

In the end, the kernel author only needs to implement a small number of virtual functions inherited from the xinterpreter class.

#include "xeus/xinterpreter.hpp"
#include "nlohmann/json.hpp"

using xeus::xinterpreter;
namespace nl = nlohmann;

namespace custom
{
    class custom_interpreter : public xinterpreter
    {
    public:

        custom_interpreter() = default;
        virtual ~custom_interpreter() = default;

     private:

        void configure() override;
        nl::json execute_request_impl(int execution_counter,
                                      const std::string& code, 
                                      bool silent, 
                                      bool store_history,
                      const nl::json::node_type* user_expressions,
                                      bool allow_stdin) override;
    
        nl::json complete_request_impl(const std::string& code,
                                       int cursor_pos) override;

        nl::json inspect_request_impl(const std::string& code,
                                      int cursor_pos,
                                      int detail_level) override;

        nl::json is_complete_request_impl(const std::string& code)
        override;

        nl::json kernel_info_request_impl() override;
    };
}

Typically, a kernel author will make use of the C or C++ API of the target programming language and embed the interpreter into the application.

This differs from the wrapper kernel approach documented in the ipykernel package where kernel authors make use of the kernel protocol implementation of ipykernel, typically spawning a separate process for the interpreter and capturing its standard output.

Jupyter kernels based on xeus include:

xeus-cling: a C++ kernel built upon the cling C++ interpreter from CERN
xeus-python: a new Python kernel for Jupyter.
JuniperKernel: a new R kernel for Jupyter based on xeus.

In this post, instead of calling into the API of an external interpreter, we implement the internal logic of the calculator in the kernel itself.

Exposing the xeus calculator to Jupyter

A calculator project

First, to implement your own Jupyter kernel, you should install Xeus. You can either download it with conda, or install it from sources as detailed in the readme.

Now that the installation is out of the way, let’s focus on the implementation itself.

Recall that the main class for the calculator kernel must inherit from the xinterpreterclass so that Xeus can correctly route the messages received from the front-end.

This class defines the behavior of the kernel for each message type that is received from the front-end.

kernel_info_request_impl: returns the information about the kernel, such as the name, the version or even a “banner”, that is a message that is prompted to console clients upon launch. This is a good place to be creative with ASCII art.
complete_request_impl: checks if the code can be completed, by that we mean semantic completion, and makes a suggestion accordingly. This way the user can receive a proposition for an adequate completion to the code he is currently writing. We did not use it during our implementation as you will see later, it is safe to return a JSON with a status value only, if you do not want to handle completion.
is_complete_request_impl: whether the submitted code is complete and ready for evaluation. For example, if brackets are not all closed, there is probably more to be typed. This message is not used by the notebook front-end but is required for the console, which shows a continuation prompt for further input if it is deemed incomplete. It also checks whether the code is valid or not. Since the calculator expects single-line inputs, it is safe to return an empty JSON object. This may be refined in the future.
inspect_request_impl: concerns documentation. It inspects the code to show useful information to the user. We did not use it in our case and went with the default implementation (that is to return an empty JSON object).
execute_request_impl: the main function. An execute_request message is sent by the front-end to ask the kernel to execute the code on behalf of the user. In the case of the calculator, this means parsing the mathematical expression, evaluating it and returning the result, as described in the next section.

Implementation of the calculator

First things first, we need to find a way to parse mathematical expressions. To do so, we turn the user input into Reverse Polish Notation (or RPN), a name full of meaning for the wisest among our readers (or at least the oldest) who used RPN calculators in high school.

The RPN, also called Postfix notation, presents the mathematical expression in a specific way : the operands go first followed by the operator. The main advantage of this notation is how it implicitly displays the precedence of operators.

Reverse Polish Notation illustration

The main logic of the calculator is provided by two main functions dealing respectively with parsing and evaluating the user expression and a third one for handling spaces in the expression.

First we have the parsing function (parse_rpn) transforming the expression into this representation. For this purpose we implement the Shunting-yard algorithm.

It is based on the use of a stack data structure to change the order of the elements in the expression, depending on their type : operator, operand or parenthesis.

Transforming a user expression into RPN

Now that we have the expression turned into RPN (with spaces delimiting operands and operators) we need to do the computation. For this purpose we have the function compute_rpn. Its implementation is based on a loop through a stringstream (hence the need for space delimiters) which performs operations in the right order.

Note that the result is not returned as an execute_reply message but is sent on a broadcasting channel instead, so that other clients to the kernel can also see it. The function execute_reply_impl actually returns the status of the execution only, as you may see in the code below.

nl::json interpreter::execute_request_impl(int execution_counter,
                                           const std::string& code,
                                           bool /*silent*/,
                                           bool /*store_history*/,
                                           nl::json /*user_exprs*/,
                                           bool /*allow_stdin*/)
{
    nl::json pub_data;
    std::string result = "Result = ";
    auto publish = [this](const std::string& name, 
                          const std::string& text) {
        this->publish_stream(name,text);
    };
    try
    {
        std::string spaced_code = formating_expr(code);
        result += std::to_string(compute_rpn(parse_rpn(spaced_code,
                                                       publish),
                                             publish));
        pub_data["text/plain"] = result;
        publish_execution_result(execution_counter,
                                 std::move(pub_data),
                                 nl::json::object());
        nl::json jresult;
        jresult["status"] = "ok";
        jresult["payload"] = nl::json::array();
        jresult["user_expressions"] = nl::json::object();
        return jresult;
    }
    catch (const std::runtime_error& err)
    {
        nl::json jresult;
        publish_stream("stderr", err.what());
        jresult["status"] = "error";
        return jresult;
    }
}

And that’s it for our calculator! It is as simple as that.

Yet remember that Xeus is a library, not a kernel by itself. We still have to create an executable that gathers the interpreter and the library. This is done in a main function whose implementation looks like:

int main(int argc, char* argv[])                       
{                           
    // Load configuration file                           
    std::string file_name = 
        (argc == 1) ? "connection.json" : argv[2];                           
    xeus::xconfiguration config = 
        xeus::load_configuration(file_name);                                                   
    
    // Create interpreter instance                           
    using interpreter_ptr = std::unique_ptr<xeus_calc::interpreter>;                           
    interpreter_ptr interpreter = 
        std::make_unique<xeus_calc::interpreter>();                              
    
    // Create kernel instance and start it                           
    xeus::xkernel kernel(config, 
                         xeus::get_user_name(), 
                         std::move(interpreter));                           
    kernel.start();                                                   
    return 0;
}

First, we need to load the configuration file. To do so, we check if one was passed as an argument, otherwise, we look for the connection.json file.

Then, we instantiate the interpreter that we previously set up. Finally, we can create the kernel with all that we defined beforehand. The kernel constructor accepts more parameters that allow customizing some predefined behaviors. You can find more details in the Xeus documentation. Start the kernel and we are good to go!

Now that everything is set, we can test out our homemade calculator kernel.

As you can see in the demonstration below, the code displays step-by-step how the computation is done with RPN. This is done with publish_streamstatements, which is equivalent to std::cout for the Jupyter notebook, very useful for debugging purposes.

The final result, a functional calculator!

You should now have all the information you need to implement your own Jupyter kernel. As you noticed, the Xeus library makes this task quite simple. All that you have to do is to inherit from the xinterpreter virtual class and implement the functions related to the messaging protocol. Nothing more is required.

This project can be found on GitHub. Feel free to contribute to the project if you wish to improve it, keeping in mind that xeus-calc should remain lean and simple!

Note that the current implementation only supports arithmetical operators. However it can be easily extended and we may add functional support in the near future.

Acknowledgments

We would like to thank the whole QuantStack team for their help throughout the process of making this blog post.

We are also grateful to the organizers of the Jupyter community workshop on kernels as we actually started to endeavor during the event.

About the authors

Vasavan Thiru is completing a master’s degree at Sorbonne Université Pierre & Marie Curie in applied mathematics for mechanics. He is currently interning as a scientific software developer at QuantStack.

Thibault Lacharme is finishing a master’s degree in Quantitative Finance at Université Paris Dauphine. Thibault is currently on his internship as a scientific software developer at QuantStack.