Jupyter Blog - Thorsten Beier

Mamba meets JupyterLite

2022-07-14T11:19:00+00:00

JupyterLite is a Jupyter distribution that runs entirely in the web browser without any server components. To achieve this, all language kernels must also run in the browser.

JupyterLite running in the browser as a static website

A significant benefit of this approach is the ease of deployment. With JupyterLite, the only requirement to provide a live computing environment is a collection of static assets. It makes it possible to embed a console or a notebook interface on any static page or blog without having to deal with a server architecture deployment. The scalability of this approach allowed several major projects of our ecosystem (NumPy, SymPy, Pandas, PyMC, and many more) to embed interactive examples on their websites, which are visited by millions of users monthly.

JupyterLite is the easiest and most scalable way to embed an interactive console or notebook on a web page without any server component.

The most prominent JupyterLite kernel is the Pyolite Python kernel, which is based on the Pyodide distribution for WebAssembly. Beyond the CPython interpreter, Pyodide includes many popular scientific computing packages such as NumPy, Pandas, and Matplotlib. Pyodide also provides a foreign function interface (FFI) that allows calling Python from JavaScript and vice versa.

The JupyterLite inline console embedded on the SymPy project website

Pyodide — and beyond

While Pyodide provides many scientific computing packages, its monolithic distribution model does not allow to specify package versions, although versions of pure python packages installed on top can be set. Our goal is to enable the composability of computing environments allowed by package managers and to adopt the conda-forge model for large-scale software distribution crowdsourcing.

Being able to pin down package versions in an environment is a strong requirement for software reproducibility. In fact, a locked WebAssembly environment could be seen as a reproducibility time capsule. As WebAssembly is a recognized web standard, it ought to be runnable for much longer than native binary packages: these are bound to a combination of architecture and platform and will eventually require an emulator.

This is why we developed a mamba-based distribution of WebAssembly packages built with Emscripten.

Emscripten-forge

The choice of the Mamba/Conda package manager was natural. Its main strength is the conda-forge community-maintained distribution, which has become the de facto standard source of packages for scientific computing.

Beyond its solid technological foundations and the multi-platform nature of the conda-forge distribution, its main strength is its social model. It allowed for a crowdsourcing approach of the packaging problem, with a balance of separation of concerns between maintainer teams and across-the-board automation, plus an amazing maintainers community.

We plan on contributing this work to the conda-forge project, so that all recipes live in the same space.

The Mamba/Conda package manager has support for many platforms and architectures such as Linux, OS X (for both x86 and arm64), and Windows. However, the WebAssembly family of platforms is not supported yet.

Adding support for WebAssembly to mamba & conda

To create conda packages for the WebAssembly platform, we relied on the Emscripten toolchain.

We defined a new target platform for conda-build and boa, namely wasm32-unknown-emscripten for which we use the emscripten-32 shorthand name. We then associated the Emscripten compiler, wrapped in a conda package as the C/C++ compiler for this new target.
This already allowed us to build many packages, including simple libraries like bzip2 and zlib, but also more complex packages like Python.
For Python extension modules we used crossenv which can create virtual environments for cross-compiling, and cross-python which integrates crossenv into conda. All the code and recipes for cross-compilation are hosted on the emscripten-forge GitHub repository.
We then used GitHub actions to build packages with Emscripten and upload them to a package server.
Packages are hosted on a deployment of the Quetz open-source server.

With this, you can easily create an environment for the emscripten-32 target:

micromamba create -n my-env --platform=emscripten-32 \ 
    -c https://repo.mamba.pm/emscripten-forge \ 
    -c https://repo.mamba.pm/conda-forge \
    python ipython numpy jedi

Note that we not only added emscripten-forge as a channel, but also conda-forge. This means all noarch packages can be used.

Adding new packages to the emscripten-forge channel

Adding new packages is a simple procedure:

fork the repository https://github.com/emscripten-forge/recipes
create a folder for your package in recipes/recipes_emscripten/<my_package>
add a recipe.yaml for your package in recipes/recipes_emscripten/<your_package>
create a pull request containing the recipe. Once the pull request is merged, the package is automatically uploaded to the emscripten-forge channel.

Integration with JupyterLite

Even though this is a general-purpose conda-based distribution for Emscripten packages, we had one particular application in mind for this first iteration: JupyterLite. The existing Pyolite kernel is too tightly coupled with the Pyodide distribution, so we decided to go with xeus-python instead.

The main reason for picking xeus-python (over ipykernel) is that with xeus-based kernels, it is possible to override the communication layer of the kernel (switching e.g. from ZMQ to HTTP/2). In the case of JupyterLite, the implementation simply relies on direct JavaScript function calls.

You can check out our earlier blog post for more details on the JupyterLite xeus-based kernels.

Providing a complete Python development experience

Some remaining intrinsic limitations to the WebAssembly platform need to be worked around to provide a complete experience to end-users. For example, sockets cannot be created in WebAssembly, preventing the use of the default asyncio event loop implementation. Luckily, the Pyodide authors developed a custom asyncio event-loop called WebLoop: it wraps the browser event loop using the Python — JavaScript foreign function interface (FFI) provided with Pyodide.

Pyjs:

Since it is non-trivial to extract Pyodide’s FFI and use it for other projects, we created a modern Python - JavaScript FFI from scratch. This was done with the following tricks:

Pybind11 is used to call Python from C++ and vice versa,
Embind is used to call JavaScript from C++ and vice versa.

When we use Pybind11 and Embind together we can call Python from JavaScript and vice versa, with C++ as a man in the middle. This not only allows us to write a simple FFI from scratch with relatively little code but also avoids calling any low-level CPython APIs and enables using high-level constructs — like pybind11::object and emscripten::val— instead.
The code is available in the pyjs repository. The API is very similar to Pyodide’s so that it can be used as a drop-in replacement in code, like Pyodide’s WebLoop implementation.

Deployment:The xeus-python-kernel allows conda packages to be pre-installed in the Python runtime. This can be done by passing the XeusPythonEnv.packages CLI option to jupyter lite build. The following command will install NumPy, Matplotlib, and ipyleaflet:

jupyter lite build --XeusPythonEnv.packages=\
    numpy,\
    matplotlib,\
    ipyleaflet

Running the xeus-python kernel with the ipyleaflet widget in JupyterLite

More details can be found in the xeus-python-kernel GitHub repository.

What about the future?

This combination of JupyterLite and Mamba has the potential to open Jupyter to millions of additional users.
Given its scalability, ease of deployment, reproducibility, and accessibility, JupyterLite will be everywhere: countries, organizations, and schools that don’t have access to sovereign cloud infrastructure will be able to deploy Jupyter-based education platforms on servers that they truly own, without endangering the data of their students or becoming too reliant on resources that they do not control.

In the short term, we are working on the following “next steps”:

Mambalite: To support the installation of packages at runtime. Similar to Pyodide’s pip-lite, it will allow downloading packages at runtime.
Fortran: Compiling Fortran code with Emscripten is currently not supported, but it is necessary for key packages like SciPy. Pyodide relies on f2c, a Fortran-to-C converter, in conjunction with a set of patches to compile Fortran code with Emscripten. We are working on a more direct approach: compiling SciPy natively with LFortran.
Binderlite: Binder converts a repository of notebooks into an executable JupyterLab environment, making code immediately reproducible by anyone, anywhere. Emscripten-forge is the missing piece to build BinderLite, a version of Binder relying on JupyterLite instances instead of vanilla JupyterLab instances.
Rust/PyO3 support: We are working on integrating the work of the Pyodide team on Rust/PyO3 support in emscripten-forge. This will be important to build Rust extension modules like cryptography.

Credits

This was built upon the work of a much bigger crowd!

The Pyodide team

The Pyodide project was started at the Mozilla foundation by Michael Droettboom and is now maintained by Hood Chatham, Roman Yurchak, and Gyeongjae Choi.

The foundational work of the Pyodide project pioneered the use of Python in the browser and made all of the rest possible, from JupyterLite to this work.

The Emscripten team

Both Pyodide and emscripten-forge are built upon the Emscripten toolchain, which provides the foundational components to be able to meaningfully run WebAssembly programs in the browser.

The JupyterLite team

The JupyterLite project was started by Jeremy Tuloup, with significant contributions from Nick Bollweg and Martin Renou.

The Mamba Org team

The mamba ecosystem has been instrumental in making these developments possible. We use the Quetz open-source server for hosting the packages and the Boa tool to build them. In the mamba development team, we should highlight the work of Wolf Vollprecht, Johan Mabille, Joel Lamotte, and Andreas Trawöger.

The Xeus team

The xeus project was started by Johan Mabille and Sylvain Corlay. It is at the foundation of the JupyterLite integration and helped to get all the pieces together (Xeus, Mamba, Jupyter). We should especially credit the work of Martin Renou and Thorsten Beier on this integration with JupyterLite.

Acknowledgment

The work of Thorsten Beier, Johan Mabille, Martin Renou, Sylvain Corlay, Wolf Vollprecht, Joel Lamotte, and Andreas Trawoger at QuantStack was funded by Bloomberg.

About the Authors

Thorsten Beier

Thorsten Beier is a Scientific Software Engineer at QuantStack. Before joining QuantStack, he graduated in computer science at the University of Heidelberg and worked at the EMBL. As an open-source developer, Thorsten worked on a variety of projects, from xeus and xtensor in C++ to inferno, kipoi, ilastik, and napari-splineit in Python.

Martin Renou

Martin Renou is a Scientific Software Engineer at QuantStack. Before joining QuantStack, he studied at the French Aerospace Engineering School SUPAERO. He also worked at Logilab in Paris and Enthought in Cambridge. As an open-source developer at QuantStack, Martin worked on a variety of projects, from xsimd, xtensor, and xframe in C++ to ipyleaflet and ipywebrtc in Python and JavaScript.

Jupyter Games

2021-12-14T19:38:00+00:00

Motivation

Making their own tiny video games can be a great way for kids to learn programming in a playful matter. While Jupyter is widely used as a scientific and educational tool, Jupyter is seldom used as a platform for game development. In this blog post, we show how Jupyter can be used to develop tiny games based on Box2D. While Jupyter is language agnostic and kernels exist for many programming languages, we will focus on the Python programming language in this blog post.

Mini-Games

When talking about video-games one might think of games such as:

Games as the ones above are usually developed with an engine like:

Not only do these games require complex engines to be developed, but also a whole team developing them and a lot of time and money.
Obviously, these are not the kind of games we can and want to develop in Jupyter. We want tiny games (~ 1000 lines of code) which can be implemented by kids in a matter of a few hours rather than many days, i.e. games like Pong, Pinball, and AngryBirds. In fact, we will focus on a certain class of games: 2D physics-based games, since these can be implemented very easily with the use of a 2D physics simulation engine like Box2D or Chipmunk2D.

Box2D

Box2D is a 2D rigid body simulation library for games. Box2D is best explained by the demo below.

A demo of Box2D compiled to JavaScript with Emscripten. Try it out (only HTTP, no HTTPS 😢 )

Box2D makes it very easy to implement physics-based games like Angry Birds or World of Goo.

Box2D Debug Draw

DebugDraw

Box2D has an abstract class DebugDraw with a handful of methods to draw simple primitives. The API is given by the following:

Box2D DebugDraw API

The ipycanvas based Box2D debug draw in action

Implementing this abstract class allows you to draw the physics for your UI back-end. Box2D is shipped with a glfw based DebugDraw implementation. For Python one can implement a pygame or kivy based DebugDraw implementation. For a finished product, one wants to replace the debug-draw-based renderings with custom shiny rendering routines. But using only debug-draw-based renderings is more than enough to quickly test game ideas and play around with Box2D.

LiquidFun

LiquidFun is a 2D rigid-body and fluid simulation library for games written in C++ based upon Box2D. LiquidFun is best explained by the demo below:

A demo of LiquidFun compiled to JavaScript with Emscripten which runs in the browser. Try it out!

More than just Games

Even though Box2D is advertised as a 2D physics engine for games, Box2D can also be used for educational purposes:
The YouTube channel iforce2d has done some impressive things with Box2D like a Box2D-based combustion engine and a wind tunnel.

Box2D based combustion engine by the youtuber iforce2d

Box2D based wind tunnel by the youtuber iforce2d

Box2D Python

Box2D has bindings for many languages and can be compiled to Wasm. Several Box2D demos exist:

http://www.iforce2d.net/embox2d/testbed.html (only HTTP, no HTTPS 😢 )
https://birchlabs.co.uk/liquidfun-wasm/
https://google.github.io/liquidfun/

We are particularly interested in using Box2D / LiquidFun from Python.
Here we have the following possibilities:

pybox2d

With pybox2d, Box2D has matured and robust Python bindings with extensive documentation. Unfortunately, pybox2d has a few shortcomings:

It only works with an old version of Box2D.
There is no support for LiquidFun.
pybox2d is mostly unmaintained.
The Python bindings are generated with SIWG (not really a shortcoming, but I personally strongly prefer pybind11).

pyb2d

A disclaimer first, I am the author of the pyb2d Box2D Python bindings. The motivation for creating pyb2d was having Box2D / LiquidFun bindings created with pybind11. While pyb2d works with the brand-new 2.4.1 release of Box2D, has support for LiquidFun and pybind11 has been used for generating the Python bindings, there are also some downsides:

pyb2d has fewer examples compared to pybox2d
pyb2d is not as mature and robust as pybox2d
pyb2d is not yet as well documented as pybox2d

A particularly useful feature of pyb2d is its BatchDebugDraw implementation: While we could implement the above mentioned debug draw API directly in Python, this would have a few drawbacks:

There is a certain overhead when calling C++ from Python and vice versa. When our game contains a lot of shapes, for instance circles, we would have to call drawCircle very often and have a lot of calls from C++ to Python and vice versa.
Some back-ends provide batch-drawing capabilities such that we can draw multiple primitives at once. For instance, drawing 100 circles at different locations with a single function call where the function arguments are NumPy nd-arrays with the centers/radii of the circles. Using such a batch API can lead to tremendous speedups.

The python batch debug draw API of pyb2d

To address these issues pyb2d provides a BatchDebugDraw implementation where we first collect all the individual calls like drawCircle, drawSegment, drawPolygon etc. We store the arguments of these calls in NumPy arrays. After collecting all the shapes we call the Python API with functions like draw_circles, draw_segments, draw_polygons, etc. where all the shapes are passed to Python in a single function call. On the Python side, one can pass these batched draw instructions to a batch-drawing API of the UI backed. (Spoiler: We will use the batch drawing API of ipycanvas for the Jupyter pyb2d integration.)

Jupyter Box2D Integration: Requirements

To have a platform to develop tiny games from within Jupyter we need at least the following:

A Canvas: We need a surface/canvas on which we can draw the content of the games.
Input Devices: Games would be boring without any user input. We need access to input devices like the mouse, the keyboard or even game-pads.

Ipycanvas, Ipywidgets, Ipyevents

Jupyter has a huge ecosystem of extensions. For the Jupyter Box2D integration, we just need to pick the appropriate Jupyter extensions: Ipycanvas gives us access to the HTML Canvas from within Python kernels in Jupyter. This serves as the drawable surface for our games. Ipywidets and Ipyevents give us access to input devices like game-pads, the keyboard, and the mouse.

Ipycanvas

Ipycanvas is a lightweight library developed by Martin Renou exposing the browser’s Canvas API to Jupyter. It allows drawing simple primitives directly from Python like text, lines, polygons, arcs, images, etc. With a few tricks, ipycanvas is fast enough to draw smooth animations. This even works when the server and the client are not on the same machines (when we run ipycanvas on MyBinder for example).

Conways game of life in ipycanvas, try it out!

An Ipycanvas-based DebugDraw:

The first step of integrating pyb2d in Jupyter notebooks is implementing an ipycanvas based DebugDraw. We recently released a new version of ipycanvas which provides an extended batch API to draw things very fast. We utilize this batch API when implementing the above mention batch-debug-draw API.

Handling events in Jupyter:

We can use Ipyevents for the event handling in Jupyter. Obviously, we want to handle the events while our game is running. To achieve this within a Jupyter-notebook we need to run our game-loop in a dedicated thread and listen for events in the main thread.

Adding Buttons:

We want to have a few buttons to start, pause and reset the Box2D based games. We can add these with ipywidgets.

Putting it all together

With all the above we can put together a pyb2d based Python Box2D integration in Jupyter. As a proof of concept we implemented a few mini-games:

Billiard:

A very simple billiard game. Try it on binder!

Angry Shapes:

An Angry Birds-like game implemented with pyb2d. Try it on binder!

World of Goo homage:

A World of Goo homage implemented with pyb2d. Try it on binder!

Rocket:

Fly a rocket controlled with the keyboard, but avoid the black hole! Try it on binder!

Compatibility

All the pyb2d examples shown above can also be run in a pygame window or a kivy window since pyb2d also provides back-ends for these. But even when one prefers to experiment with pygame, the Jupyter back-end can still be interesting since one can put the testbed-examples on MyBinder and have them accessible in a convenient fashion.

Caveats

The Jupyter integration of pyb2d gives usable frame rates, even when run through MyBinder. But nevertheless, a pygame based back-end usually leads to a better frame rate and smoother gameplay.

Outlook / Sneak preview

We currently work on:

Adding more examples and better documentation to pyb2d.
Improve the performance of the Jupyter integration.
Add pyb2d to JupyterLite.
Implement remote multiplayer-based gaming via the collaborative mode of JupyterLab.

Sneak-preview: Via the collaborative mode of JupyterLab we can connect to the notebook from two machines and play billiard against each other.

Acknowledgments

I would like to thank Martin Renou for his help with ipycanvas.

About the author

Thorsten Beier is a Scientific Software Engineer at QuantStack. Before joining QuantStack, he graduated in computer science at the University of Heidelberg and worked at the EMBL. As an open-source developer, Thorsten worked on a variety of projects, from nifty and vigra in C++ to inferno, kipoi, and ilastik in Python.

Xeus-Lite

2021-10-25T09:40:00+00:00

JupyterLite is a JupyterLab distribution that runs entirely in the web browser, backed by in-browser language kernels. Xeus is C++ library for writing Jupyter kernels. In this blogpost we show how these two can work together.

JupyterLite

With vanilla Jupyter, kernels are running in separate processes. The communication between the kernels and the client is achieved with ZeroMQ. In contrast to this, JupyterLite runs entirely in the browser, including the language kernels. The following kernels are already implemented in the JupyterLite framework:

JavaScript Kernel
P5 Kernel: Since P5 is a JavaScript library, it is easy to provide a P5 kernel running in the browser.
A Python kernel based on pyodide called pyolite: While the two JavaScript based kernels are trivial to implement as in-browser kernels, pyodide is a Python interpreter compiled to WebAssembly via Emscripten. Not only Python itself has been compiled to WebAssembly, but also the whole Scientific Stack including packages like: Numpy, Scipy, Pandas and many more. An overview of packages can be found here. Furthermore any pure Python package can be installed from within pyolite.

Since JupyterLite runs entirely in the browser, it is very easy to try it out!

Xeus

Xeus is C++ library for writing Jupyter kernels. It is not a kernel but a library that simplifies the process of writing kernels. Xeus implements the Jupyter messaging protocol, so that kernel authors only need to implement a handful of methods related to the actual interpreter, like executing pieces of code and auto-completing.
With the help of Xeus, it is very easy to write Jupyter kernels . In fact, Xeus has already been used for making several language kernels:

xeus-cling is a Xeus-based C++ kernel.
xeus-python is a Xeus-based Python kernel.
xeus-lua is a Xeus-based Lua kernel.
xeus-sql / xeus-sqlite are Xeus-based kernel for SQL / SQLite databases.
xeus-robot is a Jupyter kernel for the Robot Framework .
LFortran is a modern interactive Fortran compiler with a Xeus-based Jupyter kernel.

Since it is very easy to write Xeus based kernels for Jupyter, we also want to leverage Xeus for writing JupyterLite kernels. This has been the main motivation for Xeus 2.0.

Xeus Architecture

Xeus is has three extension points as depicted below

The Server is the middleware component responsible for receiving and sending messages to the Jupyter client. The default server it is built upon ZeroMQ and handles the concurrency model of the application.
As we will see later, a custom server is needed to connect Xeus with JupyterLite.
The Interpreter provides the interface that kernel authors must implement.
The Debugger provides the interface that can to be implemented when the kernel shall have debug capabilities, but we do not yet support this for Xeus-JupyterLite-Kernels.

JupyterLite Xeus-Kernels

Traditionally, the Jupyter kernel and the Client to the Jupyter kernel run in separate processes as depicted below:

The communication between the kernels and the client (ie the Jupyter-server) are done with inter-process communication via ZeroMQ. But with JupyterLite the kernel runs directly in the browser. In that case, ZeroMQ is not required anymore to communicate with the frontend. This was the motivation for Xeus 2.0, where it is possible to build Xeus without ZeroMQ. To implement JupyterLite Kernels with Xeus, we need to implement a custom Server which handles the Emscripten/Embind-based communication layer

The overall architecture is best described with the picture below: The C++ Xeus-Kernel is compiled to WebAssembly with Emscripten / Embind. The WebAssembly compiled code runs in a dedicated WebWorker thread. The WebWorker communicates via Message Passing with a thin layer of TypeScript which implements the IKernel API of JupyterLite. The communication between JupyterLite and the JupyterLite k ernel is done entirely in JavaScript.

The C++ code needed to compile an existing xeus-kernel to WebAssembly is very minimal:

Furthermore one needs a thin layer of TypeScript / JavaScript to implement the JupyterLite IKernel API. A good example of this code is given in the JupyterLite Xeus-Lua repository. While most of the repositories code is boilerplate, one needs to provide a simple Dockerfile which compiles the Xeus-Kernel to WebAssembly.

Xeus-Lua

We recently released a Xeus based Lua kernel. Even though its worthwhile to have Xeus-Lua kernel for vanilla Jupyter, the main purpose of Xeus-Lua was to experiment with Xeus-based JupyterLite kernels.
We choose Lua as the first language to try out Xeus-based JupyterLite kernels since Lua is easy to compile to WebAssembly.
A JupyterLite instance with Xeus-Lua included can be found here

Xeus-Lua has built in support for interactive widgets

Xeus-Wren

Wren is relatively new “Classy Little scripting Language” created by the author of the great book “Crafting Interpreters”. Wren can be described with the following:

Think Smalltalk in a Lua-sized package with a dash of Erlang and wrapped up in a familiar, modern syntax.

We choose Wren as a second language to experiment with Xeus + JupyterLite since it is small, simple to compile to WebAssembly and easy to embed in C++. A JupyterLite instance with Xeus-Wren included can be found here.

Xeus-Wren has built in support for displaying data of various MIME-types

Outlook

There are many languages which can be compiled to WebAssembly like:

Julia: https://github.com/Keno/julia-wasm
Octave: https://news.ycombinator.com/item?id=22926120
C++: https://github.com/binji/wasm-clang
Python: https://github.com/pyodide/pyodide

Also databases as sqlite can be compiled to WebAssembly:

Sqlite: https://github.com/sql-js/sql.js/

Our goal is to provide Xeus-based JupyterLite kernels for these projects to make the JupyterLite ecosystem even more powerful.

Acknowledgements

We like to thank

Jeremy Tuloup, Madhur Tandon, Martin Renou, and Nicholas Bollweg for their work on JupyterLite.
Johan Mabille for Xeus and his help implementing the Xeus-Emscripten server.

About the Author

Thorsten Beier is a Scientific Software Engineer at QuantStack. Before joining QuantStack, he graduated in computer science at the University of Heidelberg and worked at the EMBL. As an open source developer, Thorsten worked on a variety of projects, from nifty and vigra in C++ to inferno, kipoi and ilastik in Python.

From Jupyter to the Moon

2021-09-24T12:24:00+00:00

Jupyter architecture in a nutshell

A key principle in Jupyter’s design is language agnosticism, and one of the main extension points of the Jupyter ecosystem is the kernel, the part of the architecture responsible for executing the user’s code.

In the end, a Jupyter kernel is merely an executable implementing a well-defined inter-process communication protocol.

A reference implementation of the kernel is ipykernel, which provides the default Python kernel, and utilities to build other language kernels. Another reference implementation of the protocol is Xeus.

Xeus

Xeus is C++ library for writing Jupyter kernels. It is not a kernel but a library that simplifies the process of writing kernels. Xeus implements the Jupyter messaging protocol, so that kernel authors only need to implement a handful of methods related to the actual interpreter, like executing pieces of code and auto-completing. There is a long list of scripting languages which can be embedded in C/C++. With the help of Xeus, it is very easy to write kernels for such languages. In fact, Xeus has already been used for making several language kernels:

xeus-cling is a Xeus-based C++ kernel.
xeus-python is a Xeus-based Python kernel.
xeus-sql / xeus-sqlite are Xeus-based kernel for SQL / SQLite databases.
xeus-robot is a Jupyter kernel for the Robot Framework .
LFortran is a modern interactive Fortran compiler with a Xeus-based Jupyter kernel.

xwidgets

The xwidget library is a C++ implementation of the Jupyter Interactive Widget protocol. Xwidgets were initially developed for xeus-cling, the Xeus C++ kernel, but any Xeus-based kernel can reuse these components.

A xeus-based Lua kernel

Today, I am pleased to announce the release of xeus-lua, a new Jupyter kernel for Lua based on Xeus.

Lua

Lua, which means “Moon” in Portuguese, is a lightweight embeddable scripting language written in C. While Python is one of the leading scripting languages for scientific application, Lua is the leading scripting language used in games / game engines. The list of video games using Lua is long and includes triple-a games like Crisys and Farcry. Not only games rely on Lua, but also applications like Adobe Photoshop Lightroom.

Sol2

With the Sol2 library, Lua can be embedded into C++ in a very elegant way.

While Lua itself can be embedded in a C++ application quite easy, it is even easier when using Sol2. Sol2 is a C++ library that simplifies the usage of Lua from C++. Furthermore Sol2 can be used to expose C++ classes to Lua, in a similar fashion as pybind11 can be used to expose C++ classes to Python.

Xeus-lua

The compulsory “Hello, World!” from xeus-lua. This blog-post` would be incomplete without it.

I am pleased to announce xeus-lua, a new Jupyter kernel for the Lua language based on Xeus. Using Sol2 and Xeus, it was very easy to implement a feature rich kernel using modern C++. With the help of Sol2, it was trivial to expose the xwidgets classes to Lua such that we can use the widgets within xeus-lua notebooks.

But why a Lua kernel?

While Lua might not be the hippest language in 2021, there are multiple reasons while I choose to implement such a Lua kernel, and why such a kernel can be useful:

Writing a Xeus-based kernel is the best way to learn and familiarize oneself with the xeus ecosystem. And since Lua is so easy to build and embed, Lua and Xeus are perfect match. Implementing this kernel gave me great insights into Xeus and Jupyter itself.
Learning Lua with xeus-lua is fun! Before writing xeus-lua, I had zero Lua experience. While implementing xeus-lua I learned Lua from within Jupyter by playing around with xwidgets from within Lua — this was a very joyful experience.
The xeus-lua implementation might serve as reference for future kernel authors. In particular the integration of xwidgets in xeus-lua might be useful for future implementations.
Since Lua is so easy to build, xeus-lua can be built for WebAssembly and will be integrated in JupyterLite — stay tuned!

Features

Obviously xeus-lua supports the execution of code.

Simple code execution

Streams are redirected to the notebooks output

Redirection of streams

Errors messages are shown in the notebooks output

Error Handling

Input streams are redirected to the notebooks input system

User Input

Statements can be auto-completed with a press of tab

Code Completion

Data of different sources as Latex, JSON and html can be displayed.

Display Data

Interactive widgets can be added to the notebook!

Widgets

We can feed the widgets with data!

Binary buffers for widgets

Try It Out

You can try it right now with binder, without the need of installing anything on your computer, just by clicking on this link.

Other Lua kernels

There are several Lua kernels available:

lupyter: A Python based implementation of a Lua Jupyter kernel. The heavy lifting is done in a C-extension.
ILua: A feature rich implementation. Ilua is implemented in Python and does the communication with Lua via IPC.
IPyLua: IPyLua is a pure Lua implementation.

While these implementations are feature rich and work well, none of the above supports interactive widgets. Since xeus-lua is based on Xeus, we can reuse components from the Xeus ecosystem like xwidgets with minimal effort.

Acknowledgments

We are grateful to Juliette Taka for the illustration.

About the Author

Thorsten Beier is a Scientific Software Engineer at QuantStack. Before joining QuantStack, he graduated in computer science at the University of Heidelberg and worked at the EMBL. As an open source developer, Thorsten worked on a variety of projects, from nifty, and vigra in C++ to inferno, kipoi and ilastik in Python.