Jupyter Blog - visualization

Instantly view Parquet files in JupyterLab with Arbalister

2026-01-29T16:18:00+00:00

A colleague sends you a SQLite file, a Parquet dataset, or an Avro snapshot.
You need to explore the content, but you’re not sure what’s inside or even how to open it. You shouldn’t have to write code, craft SQL queries, or recall the syntax for read_parquetin a Python library. Instead, you should be able to click the file once and instantly see the tables and metadata.

Despite being the de-facto standard for data science and exploration, JupyterLab lacks this basic capability out of the box. Of course, there are extremely well designed libraries to read, process, and transform tabular data — but opening and viewing a file with a table should be simpler. A workflow with libraries like Pandas or Polars require you to install relevant packages, create a notebook, and iterate on queries to get a full picture of the data.

import pandas as pd

pd.set_option("display.max_rows", 50)
pd.set_option("display.max_columns", 20)
pd.set_option("display.width", None)

df = pd.read_parquet("data/2025/PERFORMANCE_SNAPSHOT_2025_01_05.parquet")
df

Whether for a newcomer to data science, or someone that needs to investigate many different files with hard to remember schemas, this is a bit cumbersome. Comma separated value files (CSV) have better support: double click on the file to see it open (see figure below). That is a great way to understand the business logic and answer questions such as What type of information is in this file? What do values typically look like? Does it seem to contain all the data I need? What is the difference between customer_id and user_id?

The JupyterLab CSV viewer

Arbalister, our new JupyterLab extension, changes this. With Arbalister, you can double-click to instantly view a wide range of tabular data files: Parquet, CSV, Avro, ORC, SQLite, and more…

Arbalister viewer can open Parquet files and more

The extension will load data as it is displayed, making it possible to view it almost instantaneously even if the file does not fit in memory. A toolbar lets the user select some reading options, such as the delimiter for a CSV file, or selecting the table to display in the SQLite file!

Arrow over the wire

Fueling it all is the Apache Arrow ecosystem, an in-memory format for tabular data used in most data science libraries. Plenty of files types can already be read into an Arrow data structure so we exploit it heavily in Arbalister (with Apache Datafusion in our case).

The table displayed in the user’s browser reuses JupyterLab Lumino DataGrid with a custom data model. In the latter, we divide the whole table in chunks across both the rows and columns axes (e.g. 512 rows and 24 columns). When we need to display data from a chunk that is not already available, we make an HTTP request to a server-side extension that open the file, read the relevant portion into an Arrow table and return it to the client. The table is returned as Arrow IPC (Inter Process Communication, a binary format close to the original Arrow memory) in the response body. In the client, a lightweight Arrow implementation can read it back efficiently.

Arbalister client-server architecture

When nothing else is happening, the implementation will also pre-fetch some of the next chunks in the background so that scrolling past a chunk boundary is imperceptible.

Because it all works in Arrow, which is standard for tabular data, adding support for new types of files that can be read into the Arrow format is extremely easy. Vortex, Lance, or whichever other format becomes relevant. But that is not all…

The existing SQLite support is made possible through ADBC, an SQL connector that reads databases into Arrow format. SQLite databases fit into single files, which fit the abstraction used in Arbalister, but ADBC could also be used to add general database exploration (Postgres etc.) into Arbalister. With Datafusion, we could also add visualization for data lakehouses: partitioned datasets (e.g. DeltaLake or Apache Iceberg) over remote object storage (S3). With Intelligent predicate and projections pushdowns, we could visualize remote tera-byte table instantly by only downloading the relevant parts.

Looking further

There are multiple things that would make great improvements to Arbalister. We have already mentioned:

Optional support for newer file types: Vortex, Lance;
Object storage and data lakehouses support;
SQL databases support.

Another direction is support for JupyterLite, a WebAssembly distribution of JupyterLab running entirely in the browser. This setting is great for education because it does not require more than a static file server to host.
Being able to work directly with Parquet files in the browser is great for making dashboard and interactive demonstrations. Arrow and Datafusion already have some experimental support for WebAssembly so this is not so far-fetched.

Finally Arbalister could use a few features to investigate deeper questions about the data. For instance, beyond the What does the data look like? question, some column filters could help answer questions such as What does the data look like in November 2024 for John Doe? Perhaps even an SQL console for more advanced queries.

An extension point

The server extension being only a way to serve Arrow data, it can be reused by other libraries to use it in other ways. For instance, if we knew that our table contains time-series data, a second viewer could display it as a graph.
Similarly for geospatial data.

If you are interested to use or contribute to Arbalister, head to our GitHub!

About the Author

Antoine Prouvost is a senior scientific software engineer at QuantStack. His work on Arbalister was funded by Bloomberg.

Build your Jupyter dashboard using Solara

2024-12-17T10:26:00+00:00

Welcome to the first part of a series of articles showing you how to create a dashboard in Jupyter and deploy it as a standalone web app. Importantly, you won’t need to rewrite your app in a different framework for deployment. We will use a pure Python solution with no JavaScript or CSS required.

Jupyter notebooks are an incredible data analysis tool since they blend code, visualization, and narrative into a single document. However, we do not want to show the code if the insights must be presented to a non-technical audience.

Built on top of ipywidgets, the Solara framework integrates into the Jupyter Notebook, Jupyter Lab, and other Jupyter environments and is Open Source. As we will see in a later article, Solara apps can also be deployed efficiently using the Solara server. This, by itself, makes Solara a perfect solution for creating dashboards or data apps.

In this article, we will create a simple dashboard using Solara’s UI components in Jupyter. The final product will allow an end-user to filter, visualize, and explore a dataset on a map.

The end result will be a modern looking deploy app

Pre-requisites

You need to install pandas, matplotlib, folium and solara. Assuming you are using pip, you can execute on your shell:

$ pip install pandas matplotlib folium solara

Or in your notebook

%pip install pandas matplotlib folium solara

The start

We will use a subsample of the San Fransisco crime dataset which contains information on types of crimes and where they were committed.

Download the CSV file to run this locally, or let the code below sort it out.

The first thing we do when we read the data is to print it out to see what the dataset contains.

import pandas as pd
from pathlib import Path
import solara

ROOT = Path(solara.__file__).parent / "website" / "pages" / "docs" / "content" / "04-tutorial"
path = ROOT / Path("SF_crime_sample.csv.gz")
url = "https://raw.githubusercontent.com/widgetti/solara/master/solara/website/pages/docs/content/04-tutorial/SF_crime_sample.csv"

if path.exists():
    df_crime = pd.read_csv(path)
else:
    df_crime = pd.read_csv(url)

df_crime

Pandas printout of the Crime dataset Dataframe

The data looks clean, but since we will work with the Category and PdDistrict column data, let us convert those columns to title case.

df_crime["Category"] = df_crime["Category"].str.title()
df_crime["PdDistrict"] = df_crime["PdDistrict"].str.title()
df_crime

Category and PdDistrict columns nicely formatted with title casing

Using proper software engineering practices, we write a function that filters a dataframe to contain only the rows that match our chosen districts and categories.

def crime_filter(df, district_values, category_values):
    df_dist = df.loc[df["PdDistrict"].isin(district_values)]
    df_category = df_dist.loc[df_dist["Category"].isin(category_values)]
    return df_category

dff_crime = crime_filter(df_crime, ["Bayview", "Northern"], ["Vandalism", "Assault", "Robbery"])

Now, with our filtered dataset, we create two bar charts. We use regular Pandas and Matplotlib, but Seaborn or Plotly are also appropriate choices.

import matplotlib.pyplot as plt

def crime_charts(df):
    cat_unique = df["Category"].value_counts()
    cat_unique = cat_unique.reset_index()

    dist_unique = df["PdDistrict"].value_counts()
    dist_unique = dist_unique.reset_index()

    fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(20, 10))

    ax1.bar(cat_unique["Category"], cat_unique["count"])
    ax1.set_title("Amount of Criminal Case Based on Category")
    ax2.bar(dist_unique["PdDistrict"], dist_unique["count"])
    ax2.set_title("Amount of Criminal Case in Selected District")

    display(fig)
    plt.close(fig)

crime_charts(dff_crime)

Since we do not need bidirectional communication (e.g., we do not need to receive events or data from our map), we use Folium to display the locations of the committed crimes on a map. If we do need bidirectional communication, we can use ipyleaflet.

Since we cannot display all the data on the map without crashing your browser, we limit it to a maximum of 50 points.

import folium
import folium.plugins

def crime_map(df):
    latitude = 37.77
    longitude = -122.42

    sanfran_map = folium.Map(location=[latitude, longitude], zoom_start=12)

    incidents = folium.plugins.MarkerCluster().add_to(sanfran_map)

    # loop through the dataframe and add each data point to the mark cluster
    for (
        lat,
        lng,
        label,
    ) in zip(df.Y, df.X, df.Category):
        folium.Marker(
            location=[lat, lng],
            icon=None,
            popup=label,
        ).add_to(incidents)

    # show map
    display(sanfran_map)

crime_map(dff_crime.iloc[0:50, :])

Making our first reactive visualization

The above code works nicely, but if we want to explore different types of crimes, we need to modify and run all cells that determine our output manually. Would it not be much better to have a UI with controls determining the filtering and a view displaying the filtered data interactively?

Let’s start by importing the solara package and creating three reactive variables.

import solara

districts = solara.reactive(["Bayview", "Northern"])
categories = solara.reactive(["Vandalism", "Assault", "Robbery"])
limit = solara.reactive(100)

A reactive variable is a container around a value (like an int, string, or list) that allows the UI to listen to changes automatically. Any change to your_reactive_variable.value will be picked up by Solara components that use them so that they can automatically redraw or update themselves.

Let us now create our first component (View), which filters the data based on the reactive variables and shows the map and the charts. Solara supports the display mechanism of Jupyter so that we can use our previously defined functions.

@solara.component
def View():
    dff = crime_filter(df_crime, districts.value, categories.value)
    row_count = len(dff)
    if row_count > limit.value:
        solara.Warning(f"Only showing the first {limit.value} of {row_count:,} crimes on map")
    crime_map(dff.iloc[: limit.value])
    if row_count > 0:
        crime_charts(dff)
    else:
        solara.Warning("You filtered out all the data, no charts shown")

View()

Our reactive visualization auto-updates when any reactive variable changes

Note that some UI parts (like the warning and the charts) are conditional. Solara will automatically find out what to add, remove, or update without you having to do this manually. Solara is declarative (similar to ReactJS) but also reactive. If we change the reactive variables, Solara will see those changes and notify the component instances that use its value.

If we run the next lines of code in our notebook, our View will automatically update.

limit.value = 70
districts.value = ["Soutern", "Northern"]

We can now explore our data much faster since we don’t need to re-run the cells that depend on it.

Solara’s reactive and declarative nature makes it scalable to much larger applications than regular ipywidgets, where keeping the UI in sync and adding, removing, and updating widgets is a manual and bug-prone process.

Adding controls

We created a declarative and reactive mini app in our notebook, but we still need to manually modify the values by executing a code cell in our Notebook. Now, let us create a UI to control it. All Solara input components support reactive variables. This means that controlling a reactive variable using a UI element is often a one-liner.

solara.SelectMultiple("District",
    all_values=[str(k) for k in df_crime["PdDistrict"].unique().tolist()],
    values=districts)

In one line of code we made our app filtering interactive by adding a dropdown

Whow, that was simple! We can now easily change the filter and see the results update. Lets do this for all our reactive variables, and put them into a single component.

@solara.component
def Controls():
    solara.SelectMultiple("District", all_values=[str(k) for k in df_crime["PdDistrict"].unique().tolist()], values=districts)
    solara.SelectMultiple("Category", all_values=[str(k) for k in df_crime["Category"].unique().tolist()], values=categories)
    solara.Text("Maximum number of rows to show on map")
    solara.SliderInt("", value=limit, min=1, max=1000)

Controls()

All controls for the filtering of our dataframe and visualization

Note that the reactive variables are bi-directional, meaning that if you change it in the UI elements, it gets reflected on the Python code!

# Note that we can read AND write reactive variables
categories.value = [*categories.value, "Warrants"]

The final dashboard

We now have two parts of our UI in separate cells. This can be an amazing experience when developing in a notebook, as it flows naturally in the data exploration process while writing your notebook.

However, your end user will probably want something more coherent. The components we created are perfectly reusable, so we put them together in a single UI.

@solara.component
def Page():
    with solara.Sidebar():
        Controls()
    View()

Page()

Our end result

Finally, our dashboard is finished!

Conclusions

Using Solara, you created an interactive dashboard within a Jupyter Notebook. Your Solara components are declarative, and when using reactive variables, they are also reactive. Whether you change reactive variables via code or the UI elements, your visualizations and maps update automatically.

Your dashboard prototype now runs in your Jupyter Notebook environment, but we can only deploy it as an app. In our next article, we will focus on deploying our notebook without making any code changes. In our third tutorial, we will expand our dashboard with a few more components and create a more advanced layout.

All documentation for Solara can be found at https://solara.dev, the GitHub repo can be found at https://github.com/widgetti/solara/ and feel free to join our Discord for a chat.

Interactive Mapping with ipyopenlayers

2024-09-05T15:30:00+00:00

Bringing the Power of OpenLayers to Jupyter.

Interactive maps play a crucial role in presenting complex geospatial information in a clear and engaging manner. While the Python ecosystem already offers an impressive range of libraries for visualization, ipyopenlayers stands out as a powerful new tool that allows for the seamless integration of the advanced features of the OpenLayers JavaScript library into Jupyter. In this article, we will present the main features of ipyopenlayers and demonstrate how this library can transform your geospatial data into dynamic, interactive visualizations.

1. What is ipyopenlayers?

ipyopenlayers is an open-source Python library that integrates OpenLayers, a popular JavaScript library for creating interactive maps, directly into Jupyter Notebooks. By combining the strengths of Jupyter and OpenLayers, ipyopenlayers allows users to create interactive maps with ease. Its versatility and simplicity make it an ideal choice for a wide range of applications, from data exploration to presenting geospatial analysis.

2. Getting Started with ipyopenlayers

To start using ipyopenlayers, you only need to install the library via pip:

pip install ipyopenlayers

Once installed, you’re ready to embark on your interactive map-making adventure!

a) Creating Your First Map

ipyopenlayers has a user-friendly interface that makes creating maps simple and quick. Here’s how to generate a basic map with just a few lines of code:

from ipyopenlayers import Map

# Create a map centered on a specific location
m = Map(center=[0, 0], zoom=2)
# Display the map
m

Screenshot of a simple ipyopenlayers map displayed inline in the Jupyter notebook

b) Adding Layers and Controls

Maps are more than just images; they allow you to mark specific locations and overlay various data. ipyopenlayers makes it easy to add layers and controls to your maps. For example, here’s how to add a raster tile layer from OpenStreetMap and a zoom slider control:

from ipyopenlayers import RasterTileLayer, ZoomSlider
m =Map()
# Add a raster tile layer
raster_layer = RasterTileLayer(url='https://{a-c}.basemaps.cartocdn.com/dark_all/{z}/{x}/{y}.png')
m.add_layer(raster_layer)

# Add a zoom slider control
zoom_slider = ZoomSlider()
m.add_control(zoom_slider)
m

Screenshot of a simple ipyopenlayers map displayed inline in the Jupyter notebook showcasing the basic controls

3. Main Features of ipyopenlayers

ipyopenlayers is packed with a wide array of functionalities that make it ideal for creating advanced interactive maps in Jupyter Notebooks. From adding custom controls to enabling dynamic layer and overlays, this library offers you tools for enhancing your geospatial visualizations. Here’s a sneak peek at some of its key features.

a) Vector Tile Layer Support

Vector Tile Layers enable efficient, high-quality rendering of geospatial data, allowing for smooth zooming and dynamic styling of map features. They offer flexibility and customization for detailed interactive maps in Jupyter Notebooks.

from ipyopenlayers import Map, VectorTileLayer

# Create a map centered at coordinates [0, 0] with zoom level 0
m = Map(center=[0, 0], zoom=0)

# Add a VectorTileLayer to the map
vector_layer =  VectorTileLayer(
url='https://basemaps.arcgis.com/arcgis/rest/services/World_Basemap_v2/VectorTileServer/tile/{z}/{y}/{x}.pbf',
source_format={
    'type': 'MVT',
}
)
m.add_layer(vector_layer)

# Display the map
m

Screenshot of an ipyopenlayers map displaying a vector layout in the Jupyter notebook

b) GeoJSON Support

GeoJSON is a popular format for encoding geographic data structures. Here’s how to display GeoJSON data on your map:

import os
import json
from ipyopenlayers import Map, GeoJSON

m = Map(center=[-77.15641232105159, 34.561348453091625], zoom=9.5)
# Note the coordinate order: [latitude, longitude]

with open("demo.json") as f:
    data = json.load(f)

# Add the GeoJSON layer
geo_json = GeoJSON(
    data=data
)

# Add the GeoJSON layer to the map
m.add_layer(geo_json)
m

Screenshot of an ipyopenlayers map displaying a GeoJSON dataset as a layer on a map

c) GeoTIFF Support

ipyopenlayers also enables the visualization of GeoTIFF files, which are essential for displaying raster data with geographic context:

from ipyopenlayers import GeoTIFFTileLayer

# Create a GeoTIFF layer
geo_tiff_layer = GeoTIFFTileLayer(url='https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/36/Q/WD/2020/7/S2A_36QWD_20200701_0_L2A/TCI.tif')
m.add_layer(geo_tiff_layer)

Screenshot of an ipyopenlayers map displaying a GeoTIFF dataset as a layer on a map

4. Advanced Use Cases: Building an Electricity Map Dashboard

The potential of ipyopenlayers extends far beyond basic mapping. One innovative application is its use in creating dynamic and interactive dashboards that integrate with other powerful Python tools.

A notable example is the Electricity Map Dashboard, a project designed to visualize and analyze electricity data from around the world using ipyopenlayers in conjunction with libraries such as electricityMap, ipywidgets, voila, and bqplot.

Key Features:

Real-time Visualization: Using electricityMap, the dashboard overlays real-time electricity consumption and carbon intensity data onto a global map with ipyopenlayers.
Interactive Controls: ipywidgets enables users to filter data by region, select specific time periods, and toggle between different map layers, making the data exploration highly interactive.
Dynamic Plotting: bqplot allows users to generate customizable plots synchronized with the map, offering deeper insights into electricity trends and carbon emissions.

Screenshot of the Electricity Map dashboard built with bqplot and ipyopenlayers

This project showcases the powerful integration of ipyopenlayers with other Python tools to create a dynamic, user-friendly dashboard for analyzing global electricity trends.

5. Developments

The future of ipyopenlayers is promising, with upcoming features like improved integration with other Python libraries, advanced customization tools, and expanded support for geospatial data formats. We invite the open-source community to contribute to this project, whether through suggestions, bug reports, or code contributions.

6. Conclusion

With ipyopenlayers, the power of interactive maps is at your fingertips within your Jupyter Notebooks. This Python library allows you to present your geospatial data in an engaging and insightful way, making your analyses more dynamic and accessible. Whether you’re a data analyst, researcher, or simply a mapping enthusiast, ipyopenlayers provides the tools you need to turn your data into compelling visualizations.

So, what are you waiting for? Install ipyopenlayers and start creating interactive maps that bring your data to life!

7. Acknowledgments

This project was made possible through the collaboration and support of several individuals. Special thanks to Martin Renou , Technical Director at QuantStack, whose guidance and expertise were instrumental throughout this project. His mentorship greatly contributed to the success of this ipyopenlayers.

8. About the Author

Nour Cheour has just completed her Engineering degree at INSAT and her six-month internship as an open-source scientific software engineer at QuantStack.

ipydatagrid is now part of Project Jupyter

2024-08-22T15:01:00+00:00

Today, we are proud to announce that the ipydatagrid open source project has been incorporated into Project Jupyter as part of the Jupyter Widgets subproject.

What is ipydatagrid?

ipydatagrid is a fast data grid widget for Jupyter Notebooks and JupyterLab. Since its inception in 2019, it has been developed as an open source project in Bloomberg’s GitHub organization.

It offers a high-performance fully-featured DataGrid interface, fully integrated with ipywidgets. Built upon the Lumino datagrid, which also powers the JupyterLab CSV viewer, ipydatagrid provides users with a robust and versatile tool for data visualization and manipulation.

Key Features

Users can customize the way data is represented in their grid using a variety of renderers.

ipydatagrid includes a sophisticated selection model with two-way data binding.

It enables conditional formatting powered by Vega Expressions.

The transfer to Project Jupyter

We would like to extend our gratitude to the developers and contributors who have significantly improved the ipydatagrid project over the years: Itay Dafna, Martin Renou, Mehmet Bektas, Kaia Young, Bernát Gábor, Vasilis Themelis, Supriya K., Greg Mooney, Ian Thomas, Olly Hensby, and John Gunstone. Special thanks to Chris Colbert, the creator of the Lumino Datagrid, which forms the foundation of ipydatagrid’s front-end.

Bloomberg’s contribution to Project Jupyter

The transfer of ipydatagrid to Project Jupyter is just one of many contributions that Bloomberg has made to Project Jupyter. As a long-standing sponsor, Bloomberg has been instrumental in the development and success of Jupyter. Home to core maintainers and a major funder of JupyterLab since its inception, Bloomberg has also sponsored all editions of JupyterCon. The sustained support by Bloomberg has been crucial to Jupyter’s growth and impact.

We are excited about the future of ipydatagrid within Project Jupyter, and look forward to continued innovation and collaboration within the broader community.

— on behalf of the Jupyter Widgets Council, Sylvain Corlay

Embed interactive itkwidgets 3D renderings into JupyterLite deployments

2023-03-10T14:04:00+00:00

A tutorial that demonstrates a zero-server, interactive 3D rendering notebook and walks through the quick and easy configuration that can be customized to your needs.

By: Matt McCormick, Brianna Major, Jeremy Tuloup, Wei Ouyang, Stephen Aylward

Zero-install web applications have transformed the way we consume and deliver software. Browser-based interfaces facilitate rapid discovery, exploration, and universal access.

However, for research software engineers (RSEs), developing traditional software stacks for web applications is not only onerous, but those stacks may limit essential future scalability and may be even more onerous to sustain. A RSE must face difficult questions:

Who is going to pay to keep the servers online?
Who is going to pay to scale the servers for many user or datasets?
When are you going to find the time to learn and keep up-to-date with all the devops knowledge and skills required?
Who is going to maintain the system and address security vulnerabilities as they arise?
How is private data on the server managed and kept secure?

As one of my favorite professors used to say, in cases like this we can look to the advice offered by a wise doctor:

Patient: Oh, Doctor, it hurts badly when I move my knee like this.

Doctor: Stop moving your knee like that!

In some cases, components of the traditional web application software stack are necessary, and some of those components are easier, more scalable, and more sustainable than others. However, for many RSE use cases, we now can create useful web applications while avoiding traditional server-related hardships altogether.

In this tutorial, we will demonstrate how to create a zero-server JupyterLite deployment that embeds interactive 3D renderings into advanced scientific applications, such as for deep learning medical image analysis applications using MONAI.

JupyterLite is a JupyterLab distribution that runs entirely in the browser built from the ground-up using JupyterLab components and extensions. JupyterLite uses a WebAssembly-based distribution of scientific Python called Pyodide.

ITKWidgets provides interactive widgets to visualize images, point sets, and 3D geometry on the web. ITKWidgets is powered by the same WebAssembly technology. It is built on ITK-Wasm and ImJoy, a hybrid computing platform that communicates via symmetrical transparent remote procedure calls. ImJoy and ITKWidgets support browser-based Pyodide communication along with a number of additional server-client communication transport mechanisms.

In this tutorial, we will first demonstrate a zero-server, interactive 3D rendering notebook. Then, we walk through the quick and easy configuration that can be customized to your needs. Let’s get started! 🚀

0. Preliminaries

Reproduce the figure below, a rendering of medical imaging volume of an abdominal aortic stent, by running the notebook in your web browser! After the page has loaded, use the standard Shift+Enter keys to execute the Jupyter notebook cells.

a medical imaging volume of an abdominal aortic stent rendered in JupyterLite

Note that unlike other Jupyter deployments, the python code runs on your system instead of a server.

1. Create the JupyterLite environment

To build our sustainable JupyterLite deployment, we will use a Python environment that contains Python packages for:

jupyterlite
imjoy_jupyterlab_extension
Any other JupyterLab federated extensions (JupyterLab 3 extensions) that you want in your JupyterLab deployment.

Create a requirements.txt file with:

jupyterlite[all]==0.1.0b17
imjoy_jupyterlab_extension

And install the packages:

python -m pip install -r ./requirements.txt

See also the related JupyterLite extension addition documentation.

2. Add itkwidgets and other Python packages

Next, we will add itkwidgets, its dependencies, and other Python packages and their dependencies, that we wish to include into the JupyterLite configuration for deployment. These packages, along with the packages available in the Pyodide distribution, will be available in the deployed site.

Create a jupyterlite_config.json file, which specifies the locations of the itkwidgets wheel Python packages. Add other desired packages and their dependencies as follows.

{
    "PipliteAddon": {
        "piplite_urls": [
            "https://files.pythonhosted.org/packages/4c/ee/56f970ca26375176d3e4885f58471a12d5a6794bcefe8ad0ccb8d7158ca3/itkwasm-1.0b82-py3-none-any.whl",
            "https://files.pythonhosted.org/packages/6c/55/c3fc7e2b9671d15f0c0becdcb9fad6c330172988744ad6eaa17b71bace88/imjoy_rpc-0.5.16-py3-none-any.whl",
            "https://files.pythonhosted.org/packages/69/d9/5a6c8af2f4b4f49a809ae316ae4c12937d7dfda4e5b2f9e4167df5f15c0e/imjoy_utils-0.1.2-py3-none-any.whl",
            "https://files.pythonhosted.org/packages/c9/dc/3504845528418aff0b71f4b622bb0e8e12adec2d8f2c1ba21d695b9ac6e6/itkwidgets-1.0a24-py3-none-any.whl",
            "https://files.pythonhosted.org/packages/bb/3e/3667ac685ae83887b874896bcb55584797ba6b52a292df3e4b37736a9610/ngff_zarr-0.1.6-py3-none-any.whl"
        ]
    }
}

You can find links to these URLs by browsing the package on PyPI and copying the link from the Download files page for a package.

For packages that do not have a wheel on PyPI, you can provide one locally by placing them in the pypi/ directory of your JupyterLite configuration. For example, if you want to use a local version of itkwidgets instead of the version on PyPi, you could directly add this dask-image wheel to your pypi/ directory.

$ ls pypi/
pypi/dask_image-2022.9.0-py2.py3-none-any.whl

3. Add notebooks and data

Add notebooks and data you would like available in the deployment in the files/ directory. In this tutorial, we will add a Hello3DWorld.ipynb notebook.

$ ls files/
files/Hello3DWorld.ipynb

In your notebook, install additional packages in the first cell with piplite.

import piplite
await piplite.install(["itkwidgets==1.0a24"])

4. Build and deploy

Build your site with the command:

jupyter lite build

Serve the site locally with python -m http.server --directory ./_output or:

jupyter lite serve

The site can be deployed and shared with free static file hosting services such as GitHub Pages, Netlify, or Fleek.

What’s Next

In this post, we learned how to create a scalable, sustainable, zero-server Jupyter deployment that uses ITKWidgets for 3D rendering. In subsequent posts, we will discuss how to create simple, zero-server, custom web applications written in Python with PyScript.

Enjoy ITK!

Looking at notebooks from a new perspective

2021-10-05T08:33:00+00:00

Illustration by Juliette Taka

Jupyter notebooks are a great tool for practitioners of scientific computing from the research phase of their work to the communication of their results. The interleaving of code and rich text makes notebooks an ideal tool to communicate the complex ideas underlying computational concepts. However, depending on the context, we might want to look at the same source material (the notebook document) with different glasses, to present the content as a slideshow, or to audiences who are not comfortable with code.

One may also want to provide statically-rendered content rather than a fully-fledged executable notebook for security reasons. For each context, we should be able to adapt the notebook’s aesthetics and behavior to our needs.

Fortunately, the Jupyter ecosystem offers a broad range of tools to give a new face to notebooks and adjust them to one’s needs.

Panoramic view of visualization tools on the Jupyter ecosystem

The two main tools in the Jupyter ecosystem for rendering notebooks in a customized fashion are nbconvert and Voilà. In this post, we go over the differences and similarities between these two tools.

nbconvert and Voilà have the same conversion process, the main difference lies in the outcome generated by each tool. While nbconvert outputs static, non-interactive documents, Voilà converts your notebook to a fully interactive application and the way it achieves that is by keeping a Jupyter kernel active at all times.

Simplified scheme of nbconvert and Voilà’s internals

Whichever is the format you chose to generate your notebook, end-users won’t be able to execute arbitrary code, providing a secure way of sharing your results with others.

nbconvert

Nbconvert converts notebooks to various other formats via Jinja templates. It allows you to convert a .ipynb notebook file into various static formats:

HTML
LaTeX
PDF
Markdown document
ReStructured Text
Executable Script

nbconvert is a very versatile tool and can be used not only with the purpose of sharing your code and/or results with peers but also to create tests instances for your Jupyter notebooks and even generate blog posts. You can find more information about its possibilities in the docs.

Voilà

Voilà brings the versatility of nbconvert while allowing you to interact with your code outputs safely both in the Notebook and Lab interface.

As pointed out before, Voilà keeps a Jupyter kernel instance running at all times. It is language-agnostic and it will run any language supported by Jupyter. Furthermore, as it encapsulates nbconvert, Voilà is capable of converting the notebook to any of its output formats as well as make use of all of the templates that come with nbconvert.

Rendering interactive Matplotlib figures in a web application with Voilà. Source.

You will notice that Voilà expands on the default templates: from simple one-page documents to complex grid-based dashboards and slideshows.

Grid template generated by Voilà

Slideshows with Jupyter

A common use case is to share the results as a slideshow. Let’s look at the different ways to do so.

The first step to set up your slideshow is to define the mapping between cells and slides. This mapping is saved in the cell metadata of the notebook, and JupyterLab has the appropriate built-in tool to edit this information.

Different kinds of metadata that can be added to a cell on JupyterLab

The classic notebook offers a similar interface to edit the notebook metadata. Then, to visualize the resulting slideshow, several tools are available:

RISE

RISE is a classic notebook extension that provides a toolbar button to turn the notebook into a slideshow in the main notebook UI.

RISE from the notebook to slides. Source.

Unlike the other solutions presented here, RISE slideshows are still executable in the same way as the notebook, which can be a great feature when teaching a class. However, RISE may not be an appropriate tool to share such a slideshow online because it requires giving access to a notebook server.

Unfortunately, despite the effort of the community RISE still doesn’t offer support for JupyterLab.

nbconvert

Another solution is to use the Reveal template of nbconvert, which will use the same cell metadata as RISE to produce a static slideshow.

nbconvert presentation.ipynb --to html --template reveal

Unlike with RISE, for which cells remain executable, the outcome of nbconvert is a static HTML file, and only the already-computed outputs will be displayed.

Voilà

Just like with nbconvert, the Reveal template can be used with Voilà with the --template reveal command-line argument.

Using Voilà instead of nbconvert will spawn a kernel for each connected user, enabling the use of interactive widgets in the resulting slideshow to interact with the backend.

Besides, the default behavior of Voilà is to hide input cells and only display markdown cells and output.

What’s new: Déjàvu!

Déjàvu is a new utility included in nbconvert 6.2, which simply specifies new default values for several options so as to mimick Voilà’s behavior with respect to hiding input cells and prompt numbers. However, these defaults can still be overwritten by the end-user. You can do it just like you would if you were using nbconvert.

jupyter dejavu notebook.ipynb

Déjàvu output (on the left) and Voilà output (on the right)

Voilà and Déjàvu’s outputs are identical, although, in the case of Déjàvu, interactions with widgets that require a roundtrip to the kernel will not work. In contrast, we can see the result of the calculation on the Voilà side.

To summarize and wrap up:

Déjàvu supports all of the templates available in Voilà, and even though users don’t have access to interactivity they can still benefit from widgets outputs since they’ll be generated and included on the final file;
When building a slideshow you can use JupyterLab’s interface to define specific metadata to your cells;
You can also see your code in Déjàvu’s outputs. Just pass the --show-input option to the command line.

Acknowledgments

Thanks to Juliette Taka for creating the illustrations for this blog post.

About the author

My name is Mariana Meireles, I’m a software engineer working for QuantStack. I care deeply about the impacts that technology has in the world and try my best to be the change I want to see by contributing to open source projects that stand upon libre and diverse standards.
I’m currently focusing on the Jupyter ecosystem and advocating for open science and open knowledge whenever I can.

Genomic data visualization in Jupyter

2021-02-08T13:29:00+00:00

If there is one thing that recent events tell us, it is that genomics is a large source of data, and that its manipulation and understanding allow for the quick development of new drugs and… vaccines.

We decided to build upon the Jupyter ecosystem and enrich its capabilities in this space, with a genomics data visualization widget.

In doing so, we chose to leverage the igv.js JavaScript library, developped by the Integrative Genomics Viewer (IGV) team, and used by the web app of the same name. This is all done thanks for the power of ipywidgets , which helps linking the JS genomic representation object to our Jupyter notebook.

Hence, we are proud to announce the release of ipyigv, a Jupyter widget to render genomics data, based on igv.js!

Installing ipyigv

ipyigv is available on PyPI and conda-forge. It can be installed with both pip and mamba / conda:

mamba install -c conda-forge ipyigv

You can find additional installation instruction on the project’s page on GitHub.

Rendering genomic data

igv.js consumes genomic data in two chunks:

the genome itself as documented here
specificities of the genome underlined as Tracks, which are displayed alongside or on top of the genome

ipyigv uses the same logic, with a helper function to use the public genomes available. Data is then displayed in an IgvBrowser widget, a wrapper for the igv.js browser.

To make things easier, there are a few public genomes made directly available via a helper Bunch , PUBLIC_GENOMES .

Here is what it looks like:

Creating a genome browser with data from a public genome

Now that we have created a genome browser, we may add tracks to it. Tracks may be of different sorts, about ten according to the igv.js documentation. Describing all the kinds of tracks is out of the scope of this article, but it should be noted that:

a common Track class is defined with all the common properties. Each track type corresponds to a subclass inheriting Track .
in order to ease things out, some class introspection was introduced in order to identify the type of Track based on the extension of the containing file. As shown below, this allows for instantiating a track by just using the Track constructor, without knowing the name of the actual subclass being instantiated (e.g. AnnotationTrack in the example below). Type inference is made through the type of file, or by specifying a type property.

Now, let’s add a Track to our browser, then remove it:

Adding and removing a track to/from our browser

In this first version, two other functions come in as handy:

browser.search('chr3:1-190,100,300') would position the browser at the requested position in the genome;
browser.dump_json() prepares a JSON representing the configuration of the browser. Handy if you would like to reinstantiate another browser with the same configuration without redoing it manually. Use browser.out to output the JSON content.

We hope you’ll have fun manipulating genome data in Jupyter, and sharing visual knowledge thanks to ipyigv!

Acknowledgments

The development of ipyigv by Jean-David Harrouet at QuantStack was funded as part of the PLASMA project, led by Claire Vandiedonck, Pierre Poulain, and Sandrine Caburet, associate professors at Université de Paris.

Sponsors to the PLASMA initiative include:

Région Île-de-France, via the “Trophées franciliens de l’innovation numérique dans le supérieur” (EdTech 2018) grant program,
Université de Paris, via the Initiative of Excellence (IdEx) Label and its “inovating teaching” grant program,
EUR G.E.N.E., the graduate school on Genetics and Epigenetics,
the university training “Création, analyse et valorisation de données biologiques omiques” (DU Omiques).

About the author

Jean-David Harrouet is an innovator helping companies with their digital transformation.

He believes that the right mix of coding and business acumen is a way to make life better for many people.

A C++ API for Vega-Lite

2020-12-02T15:27:00+00:00

In this post, we present the first public release of XVega, a C++ library for producing Vega-Lite charts.

Data science workflows differ from traditional software development in that engineers make use of available tools to explore and reason about a problem. In such exploratory work, engineers load data, crunch numbers, produce simple visualizations and iterate… Progress happens in quick incremental iterations, which is possible when tooling does not get in the way.

This kind of interactive computing is generally associated with the Python or R programming languages. However, with the advent of the Cling C++ interpreter from CERN, and the subsequent development of the xeus-cling Jupyter kernel, new possibilities have opened up in this space.

The Jupyter stack — that started in the scientific Python community has evolved into a language-agnostic framework that can now be leveraged by C++ developers. It bridges the gap between the countless scientific computing libraries and tools available in C++ and the Jupyter ecosystem.

The scientific C++ stack now has numerous projects under its belt — such as xtensor, xframe, etc. However, there is little support for visualization — especially for interactive plots. While there exist matplotlib-cpp and matplotplusplus (with their plotting API resembling the original matplotlib library) — they suffer from the same cons as the original library does (such as the imperative API and the confusion between dual object-oriented and state-based interface).

Owing to all these shortcomings, along with the observation that JupyterLab comes with existing support for Vega and Vega-Lite Charts (through the application/vnd.vegalite.v3+json MIME type), one can leverage this support to bridge the gap rather than reinvent the wheel. Apart from standalone use — one could also integrate such a system into other projects such as xeus-SQLite.

The main idea is to programmatically fill in a JSON that conforms to the Vega-Lite specification and respects the notion of grammar of graphics. It is analogous to what Altair did for Python. We will expose different APIs responsible for filling in certain parts of the JSON.

The fundamentals with XVega are still the same, i.e. the three essential elements of a Chart are Data, Marks and Encodings as usual and importing the library is as simple as writing two statements:

#include "xvega/xvega.hpp"

using namespace xv;

The experience is similar to what Altair offers and, hence, the central piece to the library is the Chart() object — which knows how to emit the JSON dictionary representing the data and visualization encodings.

For those unfamiliar with the Vega ecosystem, a quick recap for the above terms is given below:

Marks — What graphic should represent the data?
Encodings — Mapping between Data and Visual Elements of the Chart (such as x-axis, etc.).
Encoding Types: Quantitative (real-valued), Nominal (unordered categorical), Ordinal (ordered categorical), Temporal (time-series).

Basic usage of XVega showcasing the essential elements — Data, Marks and Encodings.

The core strength of using such a system is the separation of specification and execution. The declarative API makes it easy to specify “what” should be done rather than focus on incidental details of the “how”. It means that rather than having a special “hist()” function for plotting a histogram, passing “bin=True” does the job.

Simply stating bin=True bins the x-axis giving us the Histogram directly — without using a dedicated function.

We can of-course customize the binning parameters with a “Bin()” object instead. And while we are doing that, let’s add a colour encoding as well to get a sense of the 3rd dimension.

More control can be achieved using a custom Bin() object — used to set the binning parameters.

Another plus of using Vega-Lite is the possibility of using transformations within the specification rather than doing it before.
(E.g., one can do linear regression as a part of this declarative API).

Usage of layering and transformations in XVega.

Lastly, support for Interactions and Selections is a no-brainer. It’s as simple as defining what to use and adding it to the Chart() object.

Zooming and Panning along with Tooltips using Interval Selection in XVega

Developing such a system for C++ comes with its own challenges and to provide a seamless experience like Altair, several things are needed to be taken care of:

Multiple types for a single entity: the Vega-Lite specification allows variables of different kinds (such as a boolean type and an integer type may be equally valid for a particular property). Variants and Visitors in C++ allow us to achieve this.
Out of order keyword arguments: Method chaining is the classical approach to tackle out-of-order keyword arguments in C++ and is what is used in XVega indeed.
Optional fields: A lot of values in the Vega-Lite specification are optional, and this is made possible by the optionally contained values in C++ (i.e. using std::optional).

Installation

You can install XVega with conda or mamba:

mamba install -c conda-forge xvega

conda install -c conda-forge xvega

What is coming?

XVega is still at an early stage and under active development. We are currently working on integrating it with the xeus-sqlite and other SQL Jupyter kernels to enable the visualization from SQL queries. We are also working on improving the compilation time of XVega with Cling.

Acknowledgements

This work on XVega was funded by QuantStack. Thanks to Sylvain Corlay and Johan Mabille for their continuous support.

It is an exciting time for the interactive C++ ecosystem, as so much innovation is happening in the Cling and Jupyter projects. There is a lot more to come for sure. If you are interested in helping us build that future, come talk to us at Gitter and GitHub.

About the Author

My name is Madhur Tandon, and I currently work with QuantStack as a Scientific Software Engineer. Before joining QuantStack, I have worked with Mozilla, Deepnote, INCF (International Neuroinformatics Coordinating Facility), TCS Research and Elucidata. I have also been a speaker at JupyterCon 2020 and PyData Delhi 2017 and 2018. I graduated from IIIT-Delhi this year with a Bachelor’s degree in Computer Science with Honors. Besides core Data Science and Machine Learning, I am interested in tools that enable and enhance data scientists’ workflow and experience.

ipygany: Jupyter into the third dimension

2020-10-14T09:51:00+00:00

Scientific visualization in the Jupyter notebook

From Paraview to Mayavi, there are multiple solutions for data analysis on 3D meshes on the desktop. Most of these tools provide high-level APIs that can be driven with a scripting language like Python. For example, one could control Paraview from a Jupyter Notebook. But this is not ideal as it relies on a desktop application for the rendering, which prevents using tools like MyBinder.

Nowadays, most scientists run their computation on the cloud, and they need tools for interacting and analyzing with their data.

There are already some solutions that are more integrated into Jupyter:

ipyvolume is a WebGL-based 3D plotting library for Jupyter. It has many features including multi-volume rendering.
K3D-jupyter helps you create 3D plots backed by WebGL with high-level API (surfaces, isosurfaces, voxels, mesh, cloud points, vtk objects, volume renderer, colormaps, etc).
itkwidgets is a tool for visualizing images, point sets, and meshes in 2D and 3D in Jupyter. It works by doing the rendering on the back-end and by streaming the frames to the front-end.
ipyvtk-simple is a Jupyter library for interfacing with any Python vtkRenderWindow. It relies on ipycanvas for streaming the frames to the front-end.

We needed a Paraview-like experience for Jupyter. We could provide this experience by relying on the vtk library in Python, but this means that every time you want to apply a new filter like warp-by-scalar, the Python back-end needs to send the filtered mesh to the front-end for display. This is far from ideal, as it might be hundreds of megabytes to download for every roundtrip to the back-end.

itkwidgets and ipyvtk-simple fix this issue by not sending the mesh to the page, instead, the rendering is done on the back-end and streamed to the user. This solution is, in a way, similar to using Paraview using a vnc solution. It works fine as long as you have a fast and low latency connection to the Jupyter server, and this solution scales properly with the mesh size, as the amount of data sent to the user is the same whether you analyze a fine or a coarse mesh.

Solutions like ipyvolume and K3D-jupyter can be backed by the Python vtk library (or by PyVista), but again, this means doing the mesh filtering in Python and sending the filtered meshes to the page for every change.

Today we are proud to announce ipygany! ipygany is a Jupyter widget that aims at bringing a Paraview-like experience to the webpage. With ipygany, you only send your mesh data to the front-end once, and effects such as mesh warping, contour computation, threshold filtering are directly done in the front-end and most-typically with the GPU.

Introducing ipygany

ipygany is a new interactive widgets library that allows you to visualize and analyze volumetric data in the Jupyter Notebook.

Whether your data comes from a VTK file or NumPy arrays, ipygany allows you to dynamically load your data, display them in the Notebook, and apply different kinds of visual effects on it.

ipygany provides a set of effects inspired by Paraview:

IsoColor: apply color-mapping to your mesh.
Warp: deform your mesh given a 3-D input data (e.g. displacement data on a beam)
WarpByScalar: deform your mesh given a 1-D input data (e.g. terrain elevation)
Threshold: only visualize mesh parts inside a range of data (e.g. 222 K≤ temperature ≤ 240 K)
IsoSurface: only visualize the surface where the mesh respects a data value (e.g. pressure == 3 bar)
Glyph effects like PointCloud
Water visualization
And there’s more to come!

Most of those effects are computed on the GPU only! Which makes them really fast. For example, changing the deformation factor of the Warp effect only sends one floating-point number to the GPU, and the GPU will know how to re-render the mesh (using shaders) according to the new factor value, this update is virtually instantaneous. Changing the warp factor value is technically as fast as moving the camera position, it only requires rendering a new frame.

We strive to use efficient algorithms for better performances, especially for computation that cannot be done with the GPU.

WarpByScalar: completely computed on the GPU

Effects can be easily combined for highlighting important parts in your mesh. For example, you can easily cut your mesh using the Threshold effect then apply a Warp effect for visualizing a deformation on the result.

Combining a Threshold with a Warp

All those effects are interactive widgets, they are stateful objects that you can dynamically update from Python or using widgets like sliders, text inputs, etc.

Your mesh is also an interactive widget: you can dynamically update your data, which is very useful when your data changes through time.

Visualizing an animation from two viewpoints

If you work with fluid dynamics, we even provide refractive/reflective effects and water caustics computation for nice looking visualization:

If you do Water simulation, ipygany also allows you to render you water with reflective/refractive effects and it even computes light caustics!

Installation

You can install ipygany with conda or mamba:

mamba install -c conda-forge ipygany

conda install -c conda-forge ipygany

Or you can install it with pip:

pip install ipygany

If you use JupyterLab you’ll need to install the labextension (not needed with the coming JupyterLab v3):

jupyter labextension install @jupyter-widgets/jupyterlab-manager ipygany

Documentation

You can find the documentation following this link:

https://ipygany.readthedocs.io

Try it online

Thanks to MyBinder, you can try it out without the need of installing anything on your computer. Just follow this link:

Acknowledgement

This work is led at QuantStack and founded by ERDC.

About the Author

My name is Martin Renou, I am a Scientific Software Engineer at QuantStack. Before joining QuantStack, I studied at the aerospace engineering school SUPAERO in Toulouse, France. I also worked at Logilab in Paris, France and Enthought in Cambridge, UK. As an open-source developer at QuantStack, I work on a variety of projects, from xtensor and xeus-python in C++ to ipyleaflet and bqplot in Python and Javascript/TypeScript.

SlicerJupyter: a 3D Slicer kernel for interactive publications

2020-07-08T16:29:00+00:00

The Jupyter ecosystem is a powerful platform for exploratory computational science, and now it can connect with some of the deep and rich domain-specific desktop applications that have decades of feature development already invested in them. With the integration of 3D Slicer with Jupyter through the xeus-python’s interpreter, we demonstrate how a Qt-based graphical desktop application with 3D visualization provided by Visualization Toolkit (VTK), image processing provided by the Insight Toolkit (ITK), can be used through a Jupyter notebook. This approach is available on the SlicerJupyter GitHub repository and could be extended to other applications that embed Python, such as Blender, FreeCAD, or ParaView.

This xeus-python integration is beneficial both for the Jupyter ecosystem and desktop applications. Features that have been developed for decades for desktop applications become readily available for Jupyter users without learning a new working environment or redeveloping features. For a desktop application, the Jupyter notebook can serve as a way to create reproducible data processing workflows, scientific publications, and maintainable tutorials without requiring local software installation. From Jupyter, you can now quickly create simple medical imaging applications with extremely rich interactivity from 3D Slicer. Read below for a discussion of the features available and the history of the project.

Powerful Medical Imaging Capabilities Available Through Jupyter

3D Slicer (or Slicer for short) is a C++ desktop application that uses Qt, ITK, and VTK libraries for visualization and medical image analysis. Slicer’s embedded Python interpreter makes all its features accessible with the Python programming language. Slicer has a simple built-in console to run Python commands interactively and can run Python scripts from files, but these are not as convenient as cell-base interactive notebooks, which have become popular among data scientists and researchers in recent years.

By integrating the xeus-python kernel, we can use a Slicer process as a Jupyter kernel. xeus-python leverages the xeus C++ implementation of the Jupyter kernel protocol. xeus-python is an alternative to ipykernel, which can be used with a vanilla CPython interpreter, interfacing to standard CPython Jupyter widgets like itkwidgets, but it can also be coupled with custom interpreters and GUI event loops, like the Slicer interpreter and its Qt event loop. This allows you to represent a complete scene in Medical Reality Markup Language, MRML, Slicer’s internal data structure. The kernel exposes the full medical imaging API and representation of your data in a meaningful way for Python developers, allowing access through standard Python ecosystem formats such as pandas dataframes and NumPy arrays in the Notebook.

Interactivity Levels

You can also use Jupyter interactive widgets (sliders, buttons, etc.) to control Slicer, modify data, or adjust processing and visualization parameters. Interactivity can be implemented at different levels.

Level 1: Standard Jupyter widgets display application specific objects by automatic conversion of application-specific data objects to standard Python objects. For example, Slicer markup fiducial lists are displayed as a nicely formatted table and model nodes are rendered as 3D objects.
Level 2: Static image widgets display content that the desktop application renders. These widgets can be made interactive by modifying data and rendering parameters using additional standard widgets. This makes rich visualization capabilities — sophisticated rendering various data types, rendering of very large data sets, etc. directly available in Jupyter.
Level 3: Dynamic viewer widgets display 2D and 3D views rendered by the desktop application. Mouse and keyboard events are forwarded to the desktop application that allows zooming/rotating views, and to utilize all 3D interactions implemented in the desktop such as placing annotations, making measurements, or segmenting images the same way as if it was done on the desktop appication’s screen. This is implemented in the Slicer Jupyter kernel using ipycanvas and ipyevents packages.
Level 4: Full desktop graphical user interface integration. Users can see parts of the application window rendered in notebook cells, including standard desktop widgets (sliders, menus, etc.). It is implemented using noVNC and TigerVNC in Slicer Jupyter. This is particularly useful when the application runs on a remote server.

Video demonstrating how to run 3D Slicer using Binder

These interactive tools allow developers to implement complete data processing workflows in a notebook, even if certain steps require manual user inputs such as segmenting 3D regions or setting seed points.

Try it online!

Since Jupyter notebooks can be used from any web browser, it can essentially turn any desktop application to a web application. By setting up a remote Jupyter server, users do not have to install anything on their computers. We have set up a demonstration of this using Binder (www.mybinder.org) that anybody can try at https://mybinder.org/v2/gh/Slicer/SlicerNotebooks/master.

Click on the binder image to launch the demo

The docker image that installs and configures Slicer and all dependencies (ipycanvas, ipyevents, VNC, etc.) is available at https://github.com/Slicer/SlicerDocker/tree/master/slicer-notebook. For deployment to non-technical users, applications can be deployed using Voilà (https://voila.readthedocs.io/), which only shows relevant content and interactive widgets, so the notebook looks like a simple dynamic web page.

The current implementation is already stable and offers a wide range of features, but there is still room for design and performance improvements. For example, we could not implement fully automatic conversion of application-specific data objects to displayable Python objects (due to complex implementation of display hooks); xeus-python debugger’s threading model needs to be improved to allow using it without locking the application’s main thread; and dynamic viewer widget’s performance (level 3 interaction) could be optimized to achieve higher refresh rates.

History of Slicer and this integration

Built over two decades with support from the NIH and a worldwide open source developer community, 3D Slicer is a unique, multi-platform desktop application for analysis, integration, and visualization of medical images that is heavily used by researchers globally for basic and applied research in a wide range of topics. The 3D Slicer Community includes physician-scientists with disease-specific knowledge of clinical challenges, computer scientists and physicists who develop novel algorithms, imaging informatics researchers, software engineers with the ability to understand clinical problems and create reliable tools, and application engineers with the multidisciplinary skills to deploy these tools in a range of cancer research settings. Slicer is maintained by Kitware, Inc., and the NAMIC consortium, and has an active open source software community. Slicer is used in hospitals and by researchers, with more than 10k academic citations and more than 150k downloads in the last year alone.

Click on the image to download 3D Slicer

It can be used for image processing workflows on 2D, 3D, and 4D images. What makes Slicer so useful is its huge set of community contributed modules that extend its functionality, as well as its ability to read DICOM and a wide array of exotic file formats.

Slicer has been scriptable in Python for well over a decade, but a robust SlicerJupyter extension is born from significant development effort by the community. The idea for this integration dates back to SciPy in 2014, when Jean-Christophe Fillion-Robin of Kitware organized a sprint intended to integrate Slicer and an IPython notebook, motivated by the dream of creating interactive and fun tutorials for Slicer. Mike Sarahan and Jean-Christophe created this proof of concept https://github.com/commontk/QEmbedIPython#qt-embed-ipython, but it was far from usable. In 2015 Matt McCormick of Kitware, created the SlicerDocker repository to support headless builds and rendering in a Docker image. Then in June 2018, while attending the Slicer Project Week, Andras Lasso (Queen’s University) and Jean-Christophe learned about Xeus, a C++ implementation of the Jupyter kernel protocol developed by QuantStack that would help streamline the integration of Slicer with Jupyter. To support this effort, Andras and Jean-Christophe created the Slicer/SlicerJupyter GitHub repository. They also contributed changes to Xeus to support this new integration paradigm where the event loop of the kernel is driven by a Qt-based Desktop application (see here). Building on this foundation, Isaiah Norton, then working at Brigham and Women’s Hospital, contributed additional improvements like a better auto-completion using jedi (see https://github.com/Slicer/SlicerJupyter/pull/12) as well as integration with Binder. More recently, Jean-Christophe and Sylvain Corlay (QuantStack) met after the Slicer project week while attending SciPy in Austin. Following the creation of a new project called xeus-python (started by Martin Renou at QuantStack), we took the integration of Jupyter and Slicer to the next level by adding support for improved interactive use and MRML data node visualization directly in the notebook.

About the authors

Alphabetically ordered

Sylvain Corlay is the founder and CEO of QuantStack, and a core Jupyter developer. He co-authored xeus and xeus-python.

Sylvain Corlay

Jean-Christophe Fillion-Robin is an open-source enthusiast, original author of the SlicerJupyter extension and a principal engineer at Kitware Inc where he leads the development of “3D Slicer” based commercial applications. J-Christophe also maintains scikit-build, an improved build system generator for CPython C/C++/Fortran/Cython extensions.

Jean-Christophe Fillion-Robin

Mike Grauer is a Technical Leader on the data and analytics team at Kitware. He is specialized in building scalable server-side processing frameworks that enable scientific workflows over web platforms.

Mike Grauer

Andras Lasso is an original author of the SlicerJupyter extension, Senior Research Engineer and Associate Director of the Laboratory for Percutaneous Surgery at Queen’s University.

Andras Lasso

Matt McCormick is an open source, medical imaging researcher working at Kitware Inc. Matt is an active, contributing member of scientific open source software efforts such as the Insight Toolkit (ITK) and scientific Python (SciPy) communities, and maintains the Jupyter 3D widget, itkwidgets.

Matt McCormick

Isaiah Norton is a Senior Software Developer at TileDB, Inc. with experience in digital pathology and image-guided surgical navigation.

Isaiah Norton

Steve Pieper is a Chief Architect and active developer of the 3D Slicer application for well over a decade. He is CEO of Isomics, Inc, where he uses a range of software technologies to perform medical imaging research with leading universities and companies.

Steve Pieper

Martin Renou is a scientific software developer at QuantStack. He is the original author of xeus-python, the xeus-based Python kernel, and contributed to the new concurrency model used for the debugger.

Martin Renou

Mike Sarahan is a software engineer at RStudio, PBC working on bringing language ecosystems together. He is passionate about making software work for data scientists in ways that are easy to maintain and improve.

Mike Sarahan

Interactive Graph Visualization in Jupyter with ipycytoscape

2020-04-30T16:16:00+00:00

The Jupyter widgets ecosystem offers a broad variety of data visualization tools for exploratory analysis in the notebook. However, we lack a good story for exploratory graph visualization.

Cytoscape is an open-source software platform for visualizing complex networks and integrating these with any type of attribute data. While it comes from the computational biology community, cytoscape is fully-fledged general-purpose tool for graph visualization and analytics. It now includes a modern web front-end (Cytoscape.JS) which is a great candidate for integration with Project Jupyter. This is the raison d’être of ipycytoscape.

Gene visualization in ipycytoscape

The goal of ipycytoscape is to enable users of well-established libraries of the Python ecosystem like Pandas, NetworkX, and NumPy, to visualize their graph data in the Jupyter notebook, and enable them modify the visual outcome programmatically or graphically with a simple API and user interface.

Fortunately, Cytoscape offers a broad enough API that allows ipycytoscape to be a tool that can, in fact, be used to solve any type of problem modeled as a graph. Some examples consist in the development of new chemicals to analyze interactions between substances in the pharmaceutic industry, in security systems to create attack graphs that can be useful to show possible vulnerabilities in systems, modeling human behavior to understand people’s interaction with business or even to understand complex phenomena like the current crisis. Currently, there is an effort to make ipycytoscape an accessible tool for researchers that are trying to find ways to mitigate and understand it, there is more information about this initiative on the COVID OSS Help website and the discussion is happening in this repository if you’re interested in joining it.

Current state

IPycytoscape is part of the PLASMA project (aka in French, Plateforme d’eLearning pour l’Analyse de données Scientifiques MAssives). This project aims at creating an interactive tool to teach computational analysis of massive scientific data. Its first instance, PlasmaBio, is designed for the needs of teachers and students of the European Master of Genetics at Université de Paris. PlasmaBio provides an authentic experience of the actual genomic and bioinformatic analyses performed in research labs. For that purpose, a custom JupyterHub-based system to control many different Jupyter instances is being specially developed by Jeremy Tuloup at QuantStack.

In this first version of ipycytoscape, there are still some limitations to what you may be able to do, but there are also some extents from the Python world that will just work out of the box for you. ipycytoscape offers integration between Pandas DataFrames and NetworkX, meaning that you can have a graph visualization of the data you already have with minimal or none adjustments and just a few lines of code.

Usage with NetworkX and DataFrame

ipycytoscape supports all of the built-in CytoscapeJS layouts. This includes the cola, grid, breadthfirst, circular, concentric and Dagre layout as well as the random, null or preset options to build a graph visualization that fits better to your data .Additionally, ipycytoscape also supports the PopperJS and TippyJS extensions, that allows you to create customizable tips for your nodes and edges .

cola, concentric, dagre and grid layouts

You can also use a variety of labels for a quick visualization of your nodes’ and edges’ contents.

Labels on nodes

Like most Jupyter interactive widgets, ipycytoscape relies on the traitlets library to synchronize data between the back-end and the front-end model.

Unfortunately, traitlets have a limitation when it comes to container objects and other mutable structures, because synchronization is only triggered upon assignment of the container and not when modifying individual elements.

To work around this limitation, we make use of the excellent Spectate library by Ryan Morshead, which triggers observers upon individual element changes in containers.

Interaction between ipywidgets and ipycytoscape

Try it online!

You can try it without the need of installing anything on your computer just by clicking on the image below:

Installation

Note that you first need to have Jupyter installed on your computer. You can install ipycytoscape using pip:

pip install ipycytoscape

Or using conda:

conda install -c conda-forge ipycytoscape

If you use JupyterLab, you would need to install the JupyterLab extension for ipycanvas (this requires nodejs to be installed):

jupyter labextension install @jupyter-widgets/jupyterlab-manager jupyter-cytoscape

About the author

My name is Mariana Meireles and I’m a software developer working for QuantStack. I care deeply about the impacts that technology has in the world and try my best to be the change I want to see by contributing to open source projects that stand upon libre and diverse standards.

Prior to QuantStack I worked as a developer on the PySide team at the Qt Company and as a web performance developer at Mozilla.

Currently I’m working on expanding the Jupyter ecosystem with new libraries and functionalities, like an experimental SQLite kernel.

Acknowledgements

The development of ipycytoscape at QuantStack was funded as part of the PLASMA project, led by Claire Vandiedonck, Pierre Poulain, and Sandrine Caburet, associate professors at Université de Paris.

Sponsors to the PLASMA initiative include:

Région Île-de-France, via the “Trophées franciliens de l’innovation numérique dans le supérieur” (EdTech 2018) grant program,
Université de Paris, via the Initiative of Excellence (IdEx) Label and its “inovating teaching” grant program,
EUR G.E.N.E., the graduate school on Genetics and Epigenetics,
the university training “Création, analyse et valorisation de données biologiques omiques” (DU Omiques).

Report on the Jupyter Community Workshop on Dashboarding

2020-02-14T21:06:00+00:00

From June 3rd to June 6th 2019, thirty-five developers from the Jupyter community met in Paris for a four-day workshop on dashboarding with Project Jupyter.

Attendees to the Jupyter Community Workshop on Kernels (Photo credit to Lindsey Heagy)

For four days, attendees worked full time on the Jupyter project, including hacking sessions and discussions on improvements to Jupyter components and new development. We were lucky to count a large number of core developers to the project in the group.

Beyond the hacking sessions, each day was concluded with a series of presentations and demos of the progress made during the workshop. In partnership with the PyData Paris team, we had a special installment of the PyData Paris Meetup with

an invited presentation by Emmanuelle Gouillart on Plotly Dash,
a series of lightning talks by attendees of the workshop on their achievements, including a talk by Philip Rudiger on the first release of Panel, and an announcement of the first releases of Voilà and the Voilà Gallery.

We ended the week with a social evening at the QuantStack offices in Paris.

Why a workshop on Jupyter Dashboarding with Jupyter?

The Jupyter ecosystem is used extensively in scientific computing both in academia and industry, and a rich ecosystem of data visualization tools has been developed around the Jupyter widgets frameworks, from geographical data visualization to protein folding simulation.

However, the Jupyter ecosystem still did not provide a means for developers to transition from notebooks to stand-alone web applications that can be accessed by multiple users.

This has been a longstanding request from the community: provide better tools built upon the Jupyter stack to share results with students, peers, or the general public.

These are the challenges that we decided to tackle during that week. The workshop was attended by many Jupyter core developers.

Highlights of the week

Many of the developers spent the week working on the Voilà and Panel projects. Both projects had their first public releases during that week (see the first public announcement of Panel and Voilà).

During this week, a team of participants including Yuvi Panda, Pascal Bugnion, and Jeremy Tuloup iterated on the first version of the Voilà gallery. Several first-time contributors to the widget framework authored example dashboards for the gallery, showcasing their existing work. Yuvi also produced the first deployment scenarii for Voilà on Heruku.
Cheryl Quah, from Bloomberg MC-ed a panel on dashboarding in the Jupyter ecosystem, including lots of questions and comparisons with Dash.
Philip Rudigger started working on a Bokeh/ipywidgets integration for better interoperability between the two frameworks.
Other contributors iterated on creating new Voilà templates, such as voila-vuetify, adding the ability to position Jupyter widgets and outputs in arbitrary location in the dashboard template. Grant Nestor created visual mockups for a UI for creating dashboard layouts in JupyterLab. Grant also helped iterating on logos for the project, and gave a presentation on dynamically loading JavaScript modules in the browser.

Acknowledgments

This event would not have been possible without the generous support provided by Bloomberg, who made this workshop series possible

We are grateful to Société Générale for funding the catering for the workshop.

The hosting of the workshop at CRI was paid for by QuantStack.

The public meetup was organized in partnership with the PyData Paris team.

Finally, we especially thank Ana Ruvalcaba from Project Jupyter for her incredible work on the logistics and finances of the Jupyter Community Workshop series.

Voilà is now a Jupyter subproject

2019-12-29T12:11:00+00:00

It is a great pleasure to announce that the Voilà project has been incorporated as a Jupyter subproject. Voilà will now be subject to the Jupyter governance and code of conduct.

For reference, the Jupyter Enhancement Proposal (JEP) for the Voilà incorporation is available here.

What is Voilà?

Voilà helps you communicate insights, by transforming a Jupyter Notebook into a stand-alone web application you can share. It gives you control over what your readers experience in a secure and customizable interactive dashboard.

The easiest way to get started with Voilà is to install it via pip or conda and type voila some_notebook.ipynb to turn the said notebook into a dashboard.

Besides, Voilà includes a templating system that allows to overload the behavior of the front-end. Using this templating system, Voilà can be used to create

slideshows (with voila-reveal)

A Voilà slideshow created with the voila-reveal template.

dashboards (with voila-gridstack)

A Voilà Dashboard based on the voila-gridstack template.

Why moving Voilà under the Jupyter governance?

While the project was initially started by QuantStack, the team now comprises developers from Bloomberg, UC Berkeley, JP Morgan, and Cal Poly San Luis Obispo. OVH has been supportive of the project by kindly providing the free hosting of the gallery on their infrastructure.

We believe that the multi-stakeholder nature of the Voilà project is well-suited for the Jupyter organization.

The Voilà project is largely built upon Jupyter subprojects and standards.

The standard notebook file format is the main entry point to Voilà.
nbconvert is used for the conversion to progressively-rendered HTML.
naturally, we use jupyter_client for handling the execution of notebook cells
jupyter_server is the default back-end.
JupyterHub is at the foundation of the voila-gallery project.
JupyterLab components (mime renderers, input and output areas) are used in the front-end implementation. Voilà also includes a preview JupyterLab extension.
ipywidgets and custom jupyter widget libraries such as bqplot, ipyvolume, ipyleaflets provide the bulk of the interactivity of Voilà applications.

Voilà is more a remix of existing Jupyter components (with changes to enable that use case) than a completely new application.

Resources

Should you be interested in Voilà, feel free to try it on Binder or locally! You can also engage with the developer community during our public team meetings and the various GitHub repositories of the project:

the Voilà GitHub repository is available here: https://github.com/voila-dashboards/voila.
the Documentation is hosted on Read the Docs: https://voila.readthedocs.io.
the Team Compass holds the calendar for the public developer meetings, as well as the meeting minutes: https://voila-dashboards.github.io.
the Announcement of the first Voilà release was published on this blog: https://blog.jupyter.org/and-voilà-f6a2c08a4a93

Acknowledgements

Voilà was started by the team of open-source developers at QuantStack as a separate project, but with the full intent to incorporate it into Jupyter. The initial project development at QuantStack was funded by Bloomberg.
Now, Voilà contributors work in many institutions, including UC Berkeley, Cal Poly San Luis Obispo, JP Morgan, and Faculty (formerly ASI Data Science).
The Voilà Gallery is kindly hosted by OVH.

ipycanvas: A Python Canvas for Jupyter

2019-10-25T12:48:00+00:00

As you may already know, the Jupyter Notebook and JupyterLab are Browser-based applications. Browsers are incredibly powerful, they allow you to swap rich and interactive graphical interfaces containing buttons, sliders, maps, 2D and 3D plots and even video games in your webpages!

All this power is readily made available to the Python ecosystem by Jupyter interactive widgets libraries. Whether you want to create simple controls using ipywidgets, display interactive data on a 2D map with ipyleaflet, plot 2D data using bqplot or plot volumic data with ipyvolume, all of this is made possible thanks to the open-source community.

One powerful tool in the Browser is the HTML5 Canvas element, it allows you to draw 2D or 3D graphics on the webpage. There are two available APIs for the Canvas, the Canvas API which focuses on 2D graphics, and the WebGL API which uses hardware acceleration for 3D graphics.

After some discussions with my work colleague Wolf Vollprecht, we came to the conclusion that it would be a great idea to directly expose the Canvas API to IPython, without making any modification to it. And that’s how we came up with ipycanvas!

ipycanvas: Exposing the Canvas API to IPython

ipycanvas exposes the Canvas API to IPython, making it possible to draw anything you want on a Jupyter Notebook directly in Python! Anything is possible, you can draw custom heatmaps from NumPy arrays, you can implement your own 2D video-game, or you can create yet another IPython plotting library!

ipycanvas provides a low-level API that allows you to draw simple primitives like lines, polygons, arcs, text, images… Once you’re familiar with the API, you’re only limited by your own imagination!

Draw image from NumPy array (left), implementation of the Game Of Life (right)

Draw millions of particles (left), draw custom sprites (right)

Make your own plotting library for Jupyter fully in Python!

Using Matt Craig’s ipyevents library, you can add mouse and key events to the Canvas and react to user interactions.

If you have a GamePad around, you can also use the built-in Controller widget and make your own video-game in a Jupyter Notebook!

Documentation

Check-out the ipycanvas documentation for more information: ipycanvas.readthedocs.io

Github repository

Give it a star on Github if you like it! github.com/martinRenou/ipycanvas

Try it online!

You can try it without the need of installing anything on your computer just by clicking on the image below:

Installation

Note that you first need to have Jupyter installed on your computer. You can install ipycanvas using pip:

pip install ipycanvas

Or using conda:

conda install -c conda-forge ipycanvas

About the author

My name is Martin Renou, I am a Scientific Software Engineer at QuantStack. Before joining QuantStack, I studied at the aerospace engineering school SUPAERO in Toulouse, France. I also worked at Logilab in Paris and Enthought in Cambridge, UK. As an open-source developer at QuantStack, I worked on a variety of projects, from xtensor and xeus-python in C++ to ipyleaflet and ipywebrtc in Python and Javascript.

Interactive GIS in Jupyter with ipyleaflet

2019-09-24T22:54:00+00:00

As Jupyter grew in popularity, a broad ecosystem of visualization packages based on Jupyter widgets has been developed, bringing even more interactivity to the Jupyter world.

In this article, we dive into Jupyter Interactive Widgets and the ipyleaflet package, an interactive maps visualization system for Jupyter.

A non-interactive map.

Jupyter Widgets

Jupyter Interactive Widgets are “special objects” that can be instantiated by the user in their code and result in a counterpart component being created in the front-end.

The core ipywidgets package provides a collection of controls that Jupyter users can use to build simple UIs as part of their notebooks (sliders, buttons, dropdowns, layout components).

Basic ipywidgets controls.

More than a collection of controls, it provides a framework upon which a large ecosystem of components has been built, allowing notebook authors to capture user inputs in very diverse ways.

Popular libraries built upon interactive widgets include

bqplot, a 2-D plotting system for Jupyter,
ipyvolume, a 3-D plotting package based on WebGL and ThreeJS,
PythreeJS, a 3-D scene description package exposing a large part of the ThreeJS API to Jupyter,
ipyvuetify, a large collection of VuetifyJS components exposed to Jupyter,
ipywebrtc, a library exposing the features of the WebRTC protocol to Jupyter kernels,
beakerx, a collection of widgets, extensions, and kernels for Jupyter,

and many more… This list is not comprehensive and dozens of other widget packages have been developed.

Key aspects of Jupyter widgets include:

BidirectionalityWidgets are not just meant for display but can also be used to capture user inputs, which can then trigger new computation. Notebook authors can compose sophisticated applications including a variety of components from different packages.
Language AgnosticismBuilt upon the Jupyter ecosystem, the interactive widget protocol used for the synchronization between the kernel and the front-end is well-specified and can be implemented for any kernel.
Back-ends for other languages than Python already exist, such as for C++ (with the xeus-cling Jupyter kernel), and languages of the JVM such as Clojure or Groovy (with the beakerx kernels).
ExtensibilityJupyter widgets are not meant as a monolithic system with one and only one way to achieve a specific task. We strive to provide a foundational layer allowing third-party widget authors to be as inventive as possible.

A common pattern for Jupyter widget packages has been to bring the capabilities of popular JavaScript visualization frameworks to Jupyter with a bridge based on ipywidgets. This is the case for the ipyleaflet package, as well as pythreejs and ipyvuetify.

Another use case is the development of ad hoc controls that are not necessarily relevant for a mainstream visualization package, but may be specific to a scientific field. An example is the ipymesh project by Chris Kees which can be used to draw PSLG (planar straight-line graphs) in the Jupyter notebook.

ipyleaflet

ipyleaflet is a Jupyter - LeafletJS bridge, bringing mapping capabilities to the notebook and JupyterLab.

Built as a bridge between the LeafletJS package and Jupyter, the ipyleaflet API maps to that of LeafletJS, bringing most of the core features of the package to Jupyter, and enabling a few popular LeafletJS extensions. A small difference is that following the Python coding style, ipyleaflet makes use of snake_case instead of CamelCase for attribute names.

ipyleaflet features

The main components to the library are layers and controls, respectively items to be displayed on the map, and interactive widgets overlayed on the map area for greater interactivity.

The first thing for which you may want to change the default value are the zoom level, the position, or the base layer for the map.

Changing the basemap layer in an ipyleaflet map.

Maps can be interactively edited in the Jupyter notebook, by dynamically changing or adding layers. In this screenshot, we add a custom layer including a GeoJSON dataset.

Note: An alternative layer to GeoJSON is GeoData, which lets the user load the data in the form of a GeoPandas dataframe instead of raw GeoJSON.

Adding a GeoJSON dataset to an interactive map.

A number of simple primitives are available such as markers and heatmaps. In the following screencast, we show how primitive properties can be linked with other widgets, and used as means to take user input on a map:

Linking properties of ipyleaflet primitives to other Jupyter widgets.

The splitmap control can be used to compare to different set of ipyleaflet layers at the same location.

The splitmap control.

Another interesting layer is the velocity layer which can be used to display wind velocity data. This control can take data in the form of an xarray dataset.

The velocity layer.

More classical visualization tools are also available, such as choropleths.

A choropleth layer.

JupyterLab Integration with the Sidecar and Theming Support

ipyleaflet is well integrated with JupyterLab

ipyleaflet controls make use of the JupyterLab themes for coloring so that they don’t stand out when using e.g. a dark theme.

The support for JupyterLab themes in ipyleaflet.

ipyleaflet can be used in combination with the JupyterLab sidecar widget.

Using the sidecar widget in JupyterLab to display a map aside of the notebook.

Together with the sidecar, a map can be programmatically added to the right-side toolbar of the JupyterLab application, and interactively edited in the notebook. This prevents the back-and-forth scrolling often required to see how changes are reflected visually.

The workflow enabled by ipyleaflet in combination with the sidecar is similar to that of the geonotebook project by Christopher Kotfila, Jonathan Beezley, and Dan LaManna from Kitware).

The geonotebook (which was based on the classic notebook) provided a similar workflow to the lab sidecar.

A more advanced example

Combined with other widget libraries such as bqplot or the core ipywidget package, ipyleaflet users can easily compose more complex applications and dashboards. In the following screencast, we explore the wealth-of-nations dataset with a leafletmap, a bqplot line chart and a dropdown widget:

Exploring the wealth-of-nations dataset with ipyleaflet, bqplot and core ipywidgets.

Trying ipyleaflet online with mybinder

If you would like to try out ipyleaflet now, it is possible thanks to the binder project. Just click on the image below!

Interactive GIS in C++: xleaflet

As mentioned earlier, interactive widgets back-ends for other programming languages have be implemented. There is a C++ back-end for ipywidgets: xwidgets which is the building block for creating other C++ widgets back-ends.

xleaflet is the C++ - LeafletJS bridge that exposes almost the same API as ipyleaflet, only that you use it from a C++ interpreter! It is based upon xwidgets which brings the bidirectional communication with the front-end.

You can learn more about the xeus-cling Jupyter kernel, xwidgets and xleaflet by reading the following blogpost.

xleaflet is the C++ backend to the jupyter-leaflet integration.

Deploying ipyleaflet-based dashboards with Voilà

Voilà is a tool that turns Jupyter notebooks into standalone dashboards. Built upon the Jupyter stack, it inherits the language agnosticism of the ecosystem, and can be used to produce standalone applications based on ipyleaflet.

A companion project to Voilà is the Voilà gallery project, a public facing set up of JupyterHub serving Voilà dashboard. It is kindly hosted by OVH. You can check out the Voilà gallery at URL https://voila-gallery.org.

For more resources about voilà, check out

the original announcement of the project: And Voilà!
and for the gallery: A Gallery of Voilà Examples.

If you check out the Voilà gallery, don’t miss Jeremy Tuloup’s GPX loader demo!

Jupyter for Geo Sciences

Jupyter’s adoption is exploding in the GeoScience space. Notable projects building upon Jupyter include

JEODPP (JRC Earth Observation Data and Processing Platform) is a EU project providing petabyte scale storage and high-throughput computing capacities to facilitate large scale analysis of Earth Observation data. The main front-end to the platform is base on Jupyter. The mapping capability is based on ipyleaflet. End users can request custom visualization that are returned to them in the form of lazily computed tile layers.
Pangeo is a community platform providing open, reproducible, and scalable sciences. The Pangeo software ecosystem involves open source tools such as xarray, Iris, Dask, Jupyter, and many other packages.
It was recently announced that the NSF would be funding the UC Berkeley + NCAR “EarthCube” proposal “Jupyter meets the Earth: Enabling discovery in geoscience through interactive computing at scale”. The plan involves the development of interactive dashboards with Voilà and contributions to the Voilà codebase.

Jupyter-gmaps

We should mention the jupyter-gmaps project by Pascal Bugnion. Jupyter-gmaps is a bridge between Google maps and Jupyter. Just like ipyleaflet, jupyter-gmaps is built upon the jupyter interactive widgets framework but relies on Google maps for the display instead of LeafletJS library.

This is a high-quality widget by another core developer of ipywidgets. Pascal is also one of the people behind Voilà and the Voilà gallery.

Folium

The Folium project enables maps visualization in the Jupyter notebook. Just like ipyleaflet, it is based on LeafletJS. Folium was created by Rob Story and is now maintained by Frank Conengmo and Filipe Fernandes.

A key difference between Folium and ipyleaflet is that ipyleaflet is built upon ipywidgets and allows bidirectional communication between the front-end and the backend enabling the use of the map to capture user input, while Folium is meant for displaying static data only. Folium enables many LeafletJS extensions, some of which may not be available in ipyleaflet at the moment.

Acknowledgements

The ipyleaflet project was started in 2015 by Brian Granger, and funded by the ERDC. The further development by Sylvain Corlay and Martin Renou was supported by QuantStack.

We are grateful to OVH for kindly hosting the Voilà gallery.

Other key contributors to Voilà and ipyleaflet that should be thanked here are Maarten Breddels, Pascal Bugnion, and Yuvi Panda. We should also mention the recent contributions by Vasavan Thirusittampalam who worked on the full-screen control and better interoperability with geopandas.

About the Authors

Sylvain Corlay, and Martin Renou are Scientific Software Developers at QuantStack.

Sylvain Corlay

Sylvain Corlay is the founder and CEO of QuantStack.

As an Open Source Developer, Sylvain contributes to Project Jupyter in the areas of interactive widgets and language kernels and is a steering committee member of the Project. Beyond QuantStack, Sylvain serves as a member of the board of directors of the NumFOCUS foundation. He also co-organizes the PyData Paris Meetup.

Martin Renou

Martin Renou is a Scientific Software Developer at QuantStack. Prior to joining QuantStack, Martin also worked as a Software developer at Enthought. He studied at the French Aerospace Engineering School ISAE-Supaero, with major in autonomous systems and programming.

As an open source developer, Martin has worked on a variety of projects, such as SciviJS (a JavaScript 3-D mesh visualization library), Xtensor, and Xeus.

ROS @ Jupyter

2019-04-07T21:15:00+00:00

This may be the right time for Jupyter-based developer tools, as cloud robotics is taking off. A PR2 robot in the browser, and making him dance through the traditional Qt interface

Historically, the ROS (Robot Operating System) community has relied on Qt for building complex user interfaces. Nowadays, the Jupyter notebook and the ipywidgets framework offer a compelling alternative for several reasons:

Code and interface are not separated — both are in the same notebook.
Complex widgets using browser technology are possible: from JavaScript sliders to 3D with WebGL, real time video streaming with WebRTC…
Works with any web browser — not bound to Linux, and no Qt applications need to be compiled.
Doesn’t need to run locally! Applications can run on a server far away, without any manual setup or installation procedure.

However, ROS does not play nicely with Jupyter. For example, rospy is inherently multi-threaded (every new ROS topic subscriber spawns a new thread). Debug printouts in callbacks result in Jupyter printing that content into the output area of the active notebook cell, quickly flooding the user interface — and there is no easy way to stop logging threads once started.

ipywidgets to the rescue: jupyter-ros

That is why jupyter-ros was created. It is a suite of plugins to the Jupyter ecosystem to make working with ROS inside Jupyter a breeze.

Subscribing with jupyter-ros produces an interactive widget

First, when you subscribe using jupyter-ros, the function returns an ipywidget with a start/stop button and a dedicated output area for debug prints. Internally this re-routes all print outs from your subscriber thread to this Jupyter cell, and gives full control over the thread (by being able to stop and restart it at any time).

The jupyros publishing sugar

If you call the publish function in jupyter-ros, a widget is automatically generated from the message specification. For example, a ROS message String field automatically becomes a text input widget, a Float32 becomes a FloatSlider…

Live plotting IMU data from a BBC micro:bit

Another interesting functionality is the live plotting, similar to rqt_plot. For this we use bqplot, a “jupyter-native” solution to plotting. With jupyter-ros, you can select a couple of fields from your message, and they will be automatically plotted to a bqplot.

Going 3-D

Most robots are three dimensional, and so should be the visualizations! The go-to tool in the ROS ecosystem at the moment is RViz, a very powerful 3-D visualization tool.

Some of RViz’s functionality has already been ported over to the web browser as part of the RobotWebTools effort. We’ve been able to piggy-back on their impressive work, and we’re releasing the first iteration of ROS3D Jupyter widgets. Currently, they allow you to programatically plug together RViz like visualizations for different data types, such as laser scans, robot trajectories, and 3D (URDF) models of the robot!

Thanks to the ipywidgetification, you can now bring complex visualizations to the web, without writing any JavaScript, and arrange those visualizations freely in JupyterLab.

Interactive widgets showing ROS data inside JupyterLab

Jupyter, Widgets, and Cloud Robotics

We hope that Jupyter, and the jupyter-ros widgets will play a big role in cloud robotics. In cloud robotics, some of the software powering one or multiple robots runs on powerful computers in data centers. For monitoring purposes, or development, Jupyter and JupyterLab are perfect candidates. Robot customers will be able to login to a single user-friendly interface, without having to install any custom software on their machine, or run a specialized operating system (ROS usually runs on Ubuntu). Exciting possibilities arise: it is already possible to run a Docker container running JupyterLab and jupyter-ros on the leading cloud robotics platforms, Rapyuta Robotics and their freshly launched platform rapyuta.io.

Installation & source

The jupyter-ros widgets can be installed from PyPI using

$ pip install jupyros
$ jupyter nbextension enable --py --sys-prefix jupyros

The examples and the source code can be found on github: https://github.com/robostack/jupyter-ros

This work is still early stages and you might find some rough edges. We look forward to collaborating with the community to polish these widgets to the highest standards!

About QuantStack

QuantStack is located in the center of Europe (Paris). We are core Jupyter developers, and love ROS, and the cloud. If you are interested in working with us on professional user interfaces for developers or clients in the cloud, do not hesitate to send us an email: wolf.vollprecht@quantstack.net

A Diagram Editor for JupyterLab

2018-02-26T11:33:00+00:00

JupyterLab viewing LaTeX source code and a PDF document

The new JupyterLab interface is much more than a replacement for the classic notebook. It aims to bring together all the pieces required for a complete scientific workflow. The extension-based architecture of JupyterLab comes with a number of components already enabled:

a Jupyter notebook,
a text editor,
a file browser in the sidebar,
a number of editors and viewers for various file formats,

and much more. However, some pieces are still missing to complete the picture for a scientific authoring environment. One would be a featureful LaTeX editor. The first LaTeX editor for JupyterLab is a step in the right direction and offers an easy way to live-compile tex documents. Another piece is — of course — a means to produce diagrams, flow charts and draw figures!

Drawing charts and diagrams

On the occasion of the Paris Jupyter Widgets workshop, I started working on a feature to fill that gap and built a JupyterLab extension for the Draw.io diagram editor.

Draw.io is a diagram editor that runs in the web browser and is Apache 2.0 licensed. It’s got a really mature code base, which has been around for many years. However, unlike the other components used by JupyterLab, Draw.io has not yet embraced the new JavaScript packaging tooling such as NPM, which complicated the integration with JupyterLab a little bit, but it all paid off eventually!

Now, I am really pleased to announce the first release of the draw.io extension, a fully fledged integration for JupyterLab of the fully-fledged diagram editor!

The Draw.io JupyterLab extension takes advantages of the JupyterLab architecture: i.e. registering a new mime type (.dio) with the file explorer to open files, and adding a launcher button and menu items. Besides that, multiple synchronized views of the same diagrams can be displayed at the same time, allowing a user to visualize the same content with different zoom levels, or with a bare text editor.

Installation

You can install the jupyterlab-drawio extension with the following command:

jupyter labextension install jupyterlab-drawio

This should set up the extension inside your JupyterLab environment. I hope this will be a useful extension for the larger community. All the code is available on GitHub: https://github.com/QuantStack/jupyterlab-drawio. Don’t hesitate to open issues and come contribute to jupyterlab-drawio.

The future

There are other projects just waiting to be packaged for use inside of JupyterLab: one great webapp for JupyterLab would probably be the ShareLaTeX application, which is Open Source as well and provides a very nicely integrated editing experience for LaTeX documents, with autocomplete of LaTeX commands and reference search. Eventually, we might be able to integrate with the official ShareLaTeX server for a collaborative, hosted, editing experience for LaTeX documents from inside JupyterLab.

Maybe we as a community can come together and start building integrations for these amazing free tools into JupyterLab!

To conclude, thanks to all who’ve organized and participated in the workshop (especially Sylvain for the organization). I’ve used the opportunity to chat with the core developers and get their helpful input: Steven, Afshin, and Jason, thanks for helping me out in getting this off the ground and making JupyterLab! And honestly, the biggest shoutout has to go to the people who’ve worked on improving draw.io and thankfully open sourced this amazing code base: the entire draw.io team.

About the Author

Wolf Vollprecht is a scientific software developer at QuantStack, passionate about High-Performance Computing and Robotics. He is one of the core developers of xtensor.