Jupyter Blog - Duc Trung Le

And Voici!

2023-12-06T15:21:00+00:00

Voici (meaning “here is” in French) is a novel project reshaping Jupyter-based interactive dashboards by combining Voilà and JupyterLite features. It facilitates the creation of dynamic, in-browser environments for data visualization and exploration.

Built upon the foundations of Voilà, Voici inherits the ability to convert any Jupyter notebook into a standalone user-friendly dashboard. While the simplicity and extensibility of Voilà are kept intact, Voici adopts the in-browser execution model of JupyterLite, replacing the client-server approach.

Dashboards generated by Voici are simply a collection of static files. Deploying and scaling such dashboards is straightforward since there is no need to allocate computational resources in the backend.

Voici dashboards are extraordinarily scalable compared to their “Voilà” counterparts, since they don’t require a container per user session.

Voilà and Voici, as their name suggests, share a lot of commonalities like the CLI, the configuration options, and the templating system. The difference in their execution model results in a distinction: Voilà can hide the source code from the browser, displaying only the rendered dashboard, whereas Voici exposes the entire notebook content to the front end.

From a notebook to a Voici dashboard

Installation and first-time use

You can install Voici using pip:

pip install voici

Or using conda/mamba:

mamba install voici -c conda-forge

Upon installation, the Voici command-line utility becomes available and can be used as a drop-in replacement for the voila command. For example, you can generate static dashboards from a notebook or a directory of notebooks like this:

# Build a single dashboard
voici my-notebook.ipynb

# Build a directory of notebooks
voici notebooks/

# Build a dashboard with a dark theme and voila-gridstack template
voici my-notebook.ipynb --template gridstack --theme dark

Once the dashboards are built in the _output directory, you can simply serve the static web page with your favorite web server. For example, using CPython’s simple HTTP server:

python -m http.server -d _output

A Voici dashboard using the dark theme

For the configurations of the themes, templates, or the tree page, you can refer to the Voilà documentation for available options. The Voici-specific configurations can be found in the Voici documentation.

Deploy Voici dashboard

Besides the static site hosting services, it is also possible to deploy Voici dashboards to GitHub Pages directly from your repository or to embed them into Sphinx documentation.

Publishing Voici dashboards on GitHub Pages

Creating your deployment on GitHub Pages is straightforward with the help of a template repository available on GitHub. You can follow the instructions from the following video:

The dashboard will be built by the CI action and published on the corresponding GitHub page of your repository.

Embedding Voici dashboards in Sphinx documentation

jupyterlite-sphinx is a Python package that allows embedding Jupyterlite in Sphinx documentation using simple directives. It is used in the documentation of major projects like scikit-image or ipywidgets.

In addition to JupyterLite applications, this package also supports embedding Voici dashboards with the voici directive:

.. voici:: my_notebook.ipynb
  :height: 600px
  :prompt: Try Voici!
  :prompt_color: #dc3545

With this directive, the Voici dashboard will be generated automatically with the Sphinx documentation, and the dashboard will be loaded as per user request.

A Voici dashboard embedded in Sphinx documentation

Voici gallery

The Voici gallery is a collection of dashboards built with Voici, it aims to provide a source of inspiration for crafting complex dashboards entirely based on the Jupyter ecosystem.

Voici Gallery home page

If you’ve created an interesting dashboard and would like to share it with a wider audience, don’t hesitate to submit a pull request here to have it included in the list of examples!

Future development

Voici is rapidly evolving, with many improvements already in the works such as:

Better integration with JupyterLab and JupyterLite to preview and generate Voici dashboards from their user interface.
Provide more user-friendly tools to easily deploy Voici dashboards to other platforms like GitLab and HuggingFace,…

User feedback from the community also plays a big role in the project roadmap. Explore Voici and share your thoughts with us using the project’s GitHub issues!

About the authors

Martin Renou is a Technical Director at QuantStack and a maintainer of Project Jupyter. Among other projects Martin is a core team member of the ipywidgets project and maintains many Jupyter widget packages such as ipyleaflet, ipydatagrid, ipygany, ipycanvas, and bqplot. He is a co-creator of the Voilà dashboarding system, and the xeus-python kernel.

Le Duc Trung is a Scientific Software Developer at QuantStack. He works on several projects within the Jupyter ecosystem, from the main projects like JupyterLab, Voilà, and ipywidgets to various JupyterLab extensions and widgets.

Voilà 0.5.0 : Homecoming

2023-09-25T14:54:00+00:00

Voilà 0.5.0: Homecoming

Voilà turns Jupyter notebooks into standalone web applications and dashboards. It provides a simple way for Jupyter users to share their work. For a detailed introduction to Jupyter’s Voilà subproject, you can check out this earlier blog post.

Up until the previous version, Voilà relied on custom JavaScript code to load extension assets and handle kernel connection. However, this approach had drawbacks since it only supported the now outdated Jupyter Notebook 6 extensions.

Following the launch of Jupyter Notebook 7, it is now Voilà’s turn to join the JupyterLab family. In Version 0.5.0, the front-end of Voilà has been rebuilt from scratch using JupyterLab 4.0 components, just like in the case of Notebook 7. A JupyterLab plugin-based application replaces the current custom extension loader approach. This transition enables Voilà to leverage all the latest features and enhancements available upstream.

New features in Voilà 0.5.0

Extension loading system

As of Version 0.5.0, Voilà uses the same prebuilt extension system as JupyterLab. The extension assets are loaded from local paths rather than relying on a CDN, resulting in a significant enhancement to Voilà’s extension compatibility with the rest of the JupyterLab ecosystem. This change also avoids making requests for external services. Additionally, users can now develop extensions for Voilà using the same approach as for JupyterLab extensions.

A Voila dashboard with top bar extension and custom JupyterLab theme.

Mimetype rendering mechanism

In 0.5.0, the logic for rendering cell outputs containing mimetypes has been revised, moving away from relying on nbconvert and instead depending on JupyterLab mimetype renderers. This change adds support for a wide range of custom JupyterLab renderers such as jupyterlab-fasta, vega, jupyterlab-geojson, etc.

Rendering mimetypes data in Voila

New tree page

The default tree page in Voilà has transitioned into a JupyterLab-based application incorporating the file browser widget. While the Jinja-based tree page remains available, users must opt for it using either the --classic-tree CLI option or by appending ?classic-tree=Trueto the query string.

The new tree page in Voilà supports JupyterLab custom themes, whereas the classic tree page is limited to light and dark themes.

New tree page with a custom theme.

Token-based authentication system

Depending on the identity provider of jupyter-server 2, Voilà now supports token authentication but is disabled by default.

To start Voilà with auto-generated token:

voila --token ...

To start Voilà with a personalized token:

voila --token=my-secret-token ...

Installing Voilà

Voila is available on both PyPI and conda-forge:

# Install with pip
pip install voila

# Install with mamba
mamba install -c conda-forge voila

You can try Voilà online using this link.

Acknowledgments

The contributions to Voilà by QuantStack are supported by Bloomberg.

About the Contributors

Voilà is an official Jupyter subproject, it has resulted from the collective efforts of numerous contributors. Below, we highlight the primary contributors for this release:

Jeremy Tuloup is a Technical Director at QuantStack and a Jupyter Distinguished Contributor. Maintainer and contributor of JupyterLab, Jupyter Notebook, JupyterLite, Voilà, and projects within the Jupyter ecosystem.

Martin Renou is a Project Director at QuantStack and a Jupyter Distinguished Contributor. Martin works on a variety of projects, from xtensor and xeus-python in C++ to ipywidgets, ipyleaflet and bqplot in Python and TypeScript.

Le Duc Trung is a Scientific Software Developer at QuantStack. He works on several projects within the Jupyter ecosystem, from the main projects like JupyterLab, Voilà, and ipywidgets to various JupyterLab extensions and widgets.

Collaborative CAD in JupyterLab

2023-06-02T15:15:00+00:00

We are thrilled to introduce JupyterCAD, a tool that integrates Computer-Aided Design (CAD) capabilities into JupyterLab. With its JupyterLab extension, dedicated JupyterCAD application, and Python API for CAD operations, JupyterCAD allows users to effortlessly create, edit and share 3D designs without leaving the Jupyter ecosystem.

This blog post provides an overview of JupyterCAD’s features and highlights its collaborative editing capabilities which enable teamwork in the realm of CAD.

Everything that is shown below can be tested by installing JupyterCAD from PyPI:

# From PyPI
pip install jupytercad

One extension, three interfaces

Built using the latest JupyterLab 4 release, JupyterCAD combines CAD functionalities with the enhanced performance and accessibility improvements introduced in JupyterLab 4. The flexibility of JupyterLab allows us to create the extension once but deploy it in multiple ways to target three groups of audiences: users who prefer working in the JupyterLab environment, users who favor dedicated applications, and those advanced users who do everything programmatically.

JupyterLab Extension

Experience CAD features directly within JupyterLab with the JupyterCAD extension. Open and edit FreeCAD files, perform CAD operations, including creating 3D primitives, applying boolean operators, and exploding the view. JupyterCAD’s extension integrates seamlessly into JupyterLab, providing an intuitive environment for CAD enthusiasts and data scientists alike.

JupyterCAD extension inside JupyterLab

JupyterCAD Application

For a streamlined CAD experience, JupyterCAD offers a dedicated application that eliminates distractions and emphasizes CAD features.

The standalone JupyterCAD Application.

Starting the dedicated JupyterCAD application is as simple as starting JupyterLab or Jupyter Notebook:

jupyter cad

This dedicated application has been built in a similar way to the coming Jupyter Notebook 7. It’s a JupyterLab “remix” built from the ground up using JupyterLab core components and custom extensions. Much like JupyterLab, it offers theming and localization support, and more planned features on the horizon.

Python API for CAD Operations

Advanced users can unlock the full potential of CAD operations with JupyterCAD’s Python API. JupyterCAD allows users to programmatically visualize, create, and manipulate the shapes from a Jupyter Notebook. For example, one can open a FreeCAD file and start modifying it from the notebook with:

from jupytercad import CadDocument
doc = CadDocument('example.FCStd')
# Create a cone, a sphere then cut the cone with the sphere.
doc.add_cone().add_sphere(radius=0.8).cut()
display(doc)

Adding object to FreeCAD file from Jupyter Notebook.

JupyterCAD API also integrates nicely with the OpenCascade Python API, providing expanded capabilities for working with shapes.

from jupytercad import CadDocument
...
# Create a prism shape with OpenCascade
prism = BRepPrimAPI_MakePrism(profile, vec).Shape()
doc = CadDocument()
doc.add_occ_shape(prism)
display(doc)

Visualize OpenCascade shape object with JupyterCAD.

Unlocking teamwork with real-time collaboration

Collaborative Editing

One of the standout features of JupyterCAD is its shared editing functionality, which seamlessly connects users across different interfaces within the JupyterCAD ecosystem. Whether collaborators are using the dedicated JupyterCAD application, the JupyterLab extension, or working with the Python API in a Notebook, any changes made to a shared document are instantly reflected for all users.

Real-time collaboration allows individuals working in the Python API to make modifications to the CAD document, while simultaneously providing a synchronized view for collaborators using the JupyterLab extension or the JupyterCAD application. This ensures that all participants have access to the most up-to-date version of the design, fostering efficient communication and eliminating the need for manual synchronization or file exchange.

Real-time collaborative editing in JupyterCAD.

Follow Mode

With JupyterCAD’s follow-mode feature, collaboration becomes even more fluid. You can follow another user’s camera movements and view adjustments in real-time. This mode allows you to gain insights into the design process, and enhance communication among team members.

Follow Mode in action.

Annotations System

The annotations system in JupyterCAD adds an interactive layer to 3D designs. Now you can add annotations to specific shapes within a CAD file, give context, provide feedback, or write instructions.

Adding annotation at precise positions

What goes on under the hood?

All the above features are made possible thanks to two major open-source components: OpenCascade.js, the WebAssembly build of OpenCascade, and jupyter_collaboration, the real-time collaboration framework of JupyterLab.

In-browser geometric modeling kernel

To execute all geometric operations, JupyterCAD uses a custom build of OpenCascade.js, which is a port of the OpenCascade library to JavaScript and WebAssembly via Emscripten. Running on a separate thread, the CAD Kernel of JupyterCAD allows users to perform complex operations at near-native speed. It also helps to lower the load of the server while serving JupyterCAD to many users.

JupyterLab Real-Time Collaboration framework

In JupyterLab 4, Real-Time Collaboration (RTC) is not only about editing notebooks but it has become a framework for building collaborative applications. Using the components of jupyter_collaboration helps us accelerate the development of the RTC features in JupyterCAD. Collaborative Editing is built upon a shared data model where jupyter_collaboration handles all the transportation and conflict resolution, while Follow Mode makes use of Awareness, a lightweight notification system provided by the library.

Future work

The current state of JupyterCAD illustrates the capabilities of JupyterLab as a foundation to build complex applications, not just for scientific computing but also for more general technical computing applications.

We will continue to improve JupyterCAD on two axes:

Enhancing the 3D viewer with features to help users interact with the objects
Enriching the Python API to improve interoperability with other CAD libraries

We plan on upstreaming some of the features of JupyterCAD, like the annotations system and the Follow Mode, to jupyter_collaboration to enable it in other file contexts in JupyterLab (Notebooks, text files, etc).

User feedback from the community also plays a big role in the project roadmap. Try JupyterCAD and share your feedback with us using the project’s GitHub issues!

About the Authors

Need for Speed: Voilà edition

2021-12-10T14:52:00+00:00

Voilà turns Jupyter notebooks into standalone applications without requiring any modification to the content. You want to share your content with non-technical readers? Just call Voilà with the notebook to turn it into a deployable web application. The simplicity of Voilà comes at a cost: the page load time.

We set ourselves to fix this Achilles’ heel while preserving the ease-of-use of Voilà. Today we are pleased to announce the release of Voilà 0.3.0, with a focus on performance.

Motivation

Page load time has a critical impact on user experience of a website.

According to Kissmetrics, nearly half of web users expect a site to load in 2 seconds or less, and they tend to abandon a site that is not loaded within 3 seconds.

In the case of Voilà, a simple dashboard takes around one second to render and more complicated dashboards (involving libraries import, data fetching, and widgets) can easily push users out of their comfort zone. Several efforts have been made to improve the situation, from adding an execution progress indicator to enhancing the rendering technique. The load time of a medium-to-high complexity notebook has not been cut down to a desirable range yet. That is why in the 0.3.0 version, we aimed at addressing the main bottleneck of Voilà: the execution of notebook.

Preheated kernels

Based upon an excellent idea of the hotpot_km project about the pooling hot-loaded Jupyter kernels, we implemented a new solution called “preheated kernels” to diminish the waiting time for starting a new python kernel and for executing the requested notebook.

Preheated kernels can be activated just by setting thepreheat_kernel option to Truein the Voilà command line or configuration file. For example, this command will activate preheated kernels feature for the voila.ipynb notebook with a pool of five kernels:

voila voila.ipynb --preheat_kernel=True --pool_size=5

Speed comparison between Voilà 0.3.0 with preheated kernel and 0.2.16

Behind the scene, the two following actions are performed:

For each notebook, a queue of kernels is created and the notebook is executed in every kernel. When a new client requests a kernel, the oldest preheated kernel of the pool is used and another kernel is started asynchronously to refill the pool.
The rendered HTML for the notebook is produced in advance for each preheated kernel and directly served to the client upon connection.
If the kernel pool is empty, Voilà will fall back to starting a new kernel and render the notebook as usual. The advanced configurations of preheated kernels are detailed in Voilà’s official documentation.

Partially pre-render notebook

To benefit from the acceleration of preheated kernels mode, the notebooks need to be pre-rendered before users connect to Voilà. But in many real-world applications, the notebook requires user-specific data, which makes pre-rendering impossible. To overcome this limit, Voilà offers a feature to handle the most used method for providing user data: the URL query string.

In preheated kernels mode, Voilà executed the first 4 cells before waiting for the query string from the user.

In normal mode, Voilà users can get the query string at run time through the QUERY_STRING environment variable:

import os
query_string = os.getenv('QUERY_STRING')

In preheated kernels mode, users can simply replace the os.getenv call with the helper get_query_string from voila.utils

from voila.utils import get_query_string
query_string = get_query_string()

get_query_string will pause the execution of the notebook by the preheated kernel at this cell and wait for an actual user to connect to Voilà, then get_query_string will return the URL query string and continue the execution of the remaining cells.

To maximize the benefits of preheated kernels, the user-independent cells should be placed before the get_query_stringcall.

What’s next?

There is more in the works to improve Voilà’s performances. Beyond kernel hot pooling, we will improve the performance of rendering Jupyter widgets in the front-end. Several of these improvements will be enabled with the upcoming 8.0 release of ipywidgets, and more optimizations are in the works.

Acknowledgments

We would like to thank Vidar Tonaas Fauske and Maarten Breddels for the prior art on the hotpot_km project, which was an inspiration for this work in core Voilà.

About the author

Duc Trung Le is a Scientific Software Developer at QuantStack. He works on several projects within the Jupyter ecosystem, from the main projects like JupyterLab, Voilà, ipywidgets to Jupyter extensions and widgets.