Jupyter Blog - David Brochart

Plug your application into the Jupyter world

2023-10-23T06:09:00+00:00

Kernels are a simple but powerful abstraction in the Jupyter architecture. They encapsulate language interpreters and make them accessible through a standardized interface. This is the key to Jupyter’s remarkable versatility, with over 100 supported languages. Embedding a kernel in your custom application can seamlessly expose it to the Jupyter ecosystem. Not only can you attach JupyterLab to your program for e.g. state inspection, as you would do with a debugger, but you can truly extend your application with all the power of the Jupyter ecosystem.

Applications: GUI and API

Several applications such as FreeCAD, Blender, or QGIS provide a Python API to access their functionalities programmatically, and include an embedded Python console which allows using this API from within the application. Users can both use the graphical user interface and operate on the application in a scripted fashion depending on their needs.

Python consoles in FreeCAD, Blender, and QGIS (from left to right).

If you think about it, these are the kinds of workflows that Jupyter has always been about, but Jupyter has evolved in the opposite direction. Instead of adding scripting to GUIs, Jupyter added GUIs to scripting. At first, Jupyter was IPython, a console for interactive computing. Then it went to the Web and became IPython Notebook, renamed to Jupyter Notebook. With the browser it became possible to display richer content and create small GUIs, using Jupyter Widgets. Then JupyterLab was created and allowed to create fully-fledged desktop-like applications in the browser, like 3-D CAD modelers.

JupyterLab showing notebooks using JupyterCAD and ipyleaflet.

Jupyter put a lot of attention on building a good experience interacting with the Python REPL, which is not the main focus of domain-specific desktop applications. The Python console embedded in desktop applications is often quite raw. Allowing users to connect from Jupyter can enable a much better experience.

What if there was a bridge that allowed you to connect both worlds, your application on one side and Jupyter on the other side? What is the common denominator allowing code to interact with an application, while also being shared with Jupyter? In the Jupyter infrastructure, we call it a kernel.

Jupyter kernels

Kernels provide a standardized way to expose language interpreters to applications, in a language-agnostic fashion. You can think of it as a server that accepts requests such as “execute this piece of code” and that replies with a result. The result is actually a data representation, it can be as simple as “3” for an input of “1+2”, or a complex interactive plot. A kernel usually lives in a separate process on your machine, but it can also run in the same process, or even remotely on another machine. Someone wanting to connect to this kernel will need the connection information such as the IP address and the ports of the different communication “channels”. It is stored in what is called a connection file.

The Jupyter kernel protocol allows clients to interact with different languages.

Anyone who has access to this connection file can then execute code in the kernel. This is “shared code”, meaning that every client to this kernel eventually runs code in the same interpreter. If one client sets a variable “a=1”, other clients will see this variable. They are all running the “same program”. If the kernel runs in the same process as the application, then the kernel can interact directly with it. This way of sharing the execution engine serves as a bridge between the application and Jupyter.

Now it is just a matter of exposing the connection file so that e.g. JupyterLab can see it. The application launching the kernel can save the connection file to a specific directory, and give the path to JupyterLab. And that’s it, JupyterLab can run a notebook using this kernel, allowing it to interact with the application.

Getting started

The xeus-qt-python repository is a great place to start and see how to embed a Python kernel inside a Qt application. Under the hood, it uses xeus-qt, an integration of the xeus kernel architecture in the Qt event loop. The Qt application actually runs JupyterLab in a Qt WebEngine window. Users can run code in a notebook and interact with the application, which in turn can interact with the notebook. Here, we create a Qt button from a notebook. When clicking on the button, a callback prints a message in the notebook. This demonstrates a two-way binding between the Qt application and JupyterLab.

Example of a Qt application embedding JupyterLab.

SlicerJupyter from Kitware was one of the first applications to take advantage of this approach. You can read more in this blog post. We hope that this technology will unlock a new brand of use-cases and allow applications to reach a new audience through the Jupyter ecosystem. Please reach out to us if you need help!

About the author

David Brochart is a technical director at QuantStack. He mostly works on the Jupyter backend stack. He contributed to the development of real-time collaboration in JupyterLab, and created the Jupyverse project, a next-generation implementation of the Jupyter server.

Acknowledgments

This work was made possible thanks to several contributors. Mariana Meireles was the initial author of xeus-qt. Thorsten Beier is a scientific software developer at QuantStack and contributes to the xeus kernel ecosystem. He is the initial creator of the xeus-qt-python demonstration. Johan Mabille is a technical director at QuantStack and he is the main author of xeus. Afshin Darian is a technical director at QuantStack and worked on the UI in JupyterLab 4.0 allowing to use an external kernel.

nbterm: Jupyter Notebooks in the terminal

2021-04-26T16:42:00+00:00

Jupyter notebooks are mostly known for their web-based user interface, such as JupyterLab or the Classic Notebook. They offer a great user experience, allow for rich output rendering, provide interactivity through e.g. widgets, and make possible working with remote kernels.

If just like me you tend to never leave the terminal in your workflow, you may be missing a tool to interact with Jupyter notebooks without switching to a web browser. While the Jupyter Console already allows you to interact with a Jupyter kernel in the terminal, it is more comparable to a shell than a notebook.

The notebook differs from the console in that it’s bound to a document, interleaving rich text and executable code. This makes the code representation more structured, while allowing for interactivity. Well, until recently this wasn’t possible in the terminal. But today, we are proud to announce nbterm!

Side-by-side comparison of the JupyterLab web notebook and nbterm.

Getting started with nbterm

Launching a notebook is as easy as entering:

$ nbterm my_notebook.ipynb

You can also create a new one if you don’t pass a notebook argument. Almost immediately, you will get a full-screen application that looks like this:

nbterm has other command line options:

$ nbterm --help
Usage: nbterm [OPTIONS] [NOTEBOOK_PATH]

Arguments:
  [NOTEBOOK_PATH]  Path to the notebook.  [default: ]

Options:
  --no-kernel                     Don't launch a kernel.
  --run                           Run the notebook.
  --save-path TEXT                Path to save the notebook.
  --version                       Show the version and exit.
  --help                          Show this message and exit.

For example, it is possible to run all cells of a notebook in batch mode, using:

$ nbterm --run my_notebook.ipynb

If you didn’t specify a new name with --save-path, a new notebook named my_notebook_run.ipynb will be created, with all cell outputs populated. This mode of execution is similar to the nbclient library, but accessible in the command line.

Embedding nbterm

It is also possible to use it as a library and embed it in your own application. Besides, all user interactions can be scripted. For instance, you can rearrange the cells and then run all cells like this:

import asyncio
from nbterm import Notebook

nb = Notebook("my_notebook.ipynb")
nb.cut_cell(3)
nb.paste_cell(1)
asyncio.run(nb.run_all())
nb.save()

A lightweight Jupyter stack

While you probably can’t tell the difference, nbterm doesn’t reuse the base Jupyter components such as jupyter-client and nbformat (respectively implementing the Jupyter kernel protocol and the Jupyter Notebook format). The reason is that nbterm was started as a toy project to see how hard it would be to have a minimalist notebook client.

As a Jupyter contributor, I am mostly involved in the maintenance and modernization of these packages. I wanted to experiment with a new project without the backward-compatibility constraints. This resulted in a rather lean code base, and its simplicity makes it suitable to serve as a reference for other language implementations.

The roadmap

While nbterm is obviously more limited than the Notebook, there are several potential improvement directions. Having rich(er) outputs is one of them. The terminal is pretty much limited to showing characters, but people have done great things with ASCII art. An attempt has been made a while ago with an ASCII back-end for matplotlib. And although it’s only on macOS for now, some terminals are capable of showing real pictures, and there exists a Matplotlib back-end for iTerm2.

Another nice feature would be to add some interactivity. In the Notebook, this is typically achieved using widgets, but they are inherently web-based, so this is not an option. Maybe we could imagine something similar to ipywidgets for the terminal, and have simple things like sliders, buttons, menus and such. I am keeping an eye on the awesome Rich library, with which you can already create dashboards, and GUI applications in the future.

Try it!

nbterm really has the look and feel of the Notebook you already know and love, with (almost) the same key bindings, so you won’t be lost. As coders, we love the terminal, and libraries like prompt-toolkit and Rich in Python, or cpp-terminal in C++, make it possible to build beautiful applications. You can install nbterm today using your favorite package manager:

$ pip install nbterm

or:

$ mamba install nbterm -c conda-forge

Be sure to update frequently, as it is improving every day. And if you want to contribute, please get in touch on the GitHub repository.

Acknowledgements

The development of nbterm is sponsored by QuantStack.

About the author

David Brochart is a scientific software developer at QuantStack. He is a contributor to several libraries in the Jupyter ecosystem, such as nbclient, jupyter-client, jupyter-server, ipykernel, and ipywidgets.