Jupyter Blog - Mariana Meireles

Looking at notebooks from a new perspective

2021-10-05T08:33:00+00:00

Jupyter notebooks are a great tool for practitioners of scientific computing from the research phase of their work to the communication of their results. The interleaving of code and rich text makes notebooks an ideal tool to communicate the complex ideas underlying computational concepts. However, depending on the context, we might want to look at the same source material (the notebook document) with different glasses, to present the content as a slideshow, or to audiences who are not comfortable with code.

One may also want to provide statically-rendered content rather than a fully-fledged executable notebook for security reasons. For each context, we should be able to adapt the notebook’s aesthetics and behavior to our needs.

Fortunately, the Jupyter ecosystem offers a broad range of tools to give a new face to notebooks and adjust them to one’s needs.

Panoramic view of visualization tools on the Jupyter ecosystem

The two main tools in the Jupyter ecosystem for rendering notebooks in a customized fashion are nbconvert and Voilà. In this post, we go over the differences and similarities between these two tools.

nbconvert and Voilà have the same conversion process, the main difference lies in the outcome generated by each tool. While nbconvert outputs static, non-interactive documents, Voilà converts your notebook to a fully interactive application and the way it achieves that is by keeping a Jupyter kernel active at all times.

Simplified scheme of nbconvert and Voilà’s internals

Whichever is the format you chose to generate your notebook, end-users won’t be able to execute arbitrary code, providing a secure way of sharing your results with others.

nbconvert

Nbconvert converts notebooks to various other formats via Jinja templates. It allows you to convert a .ipynb notebook file into various static formats:

HTML
LaTeX
PDF
Markdown document
ReStructured Text
Executable Script

nbconvert is a very versatile tool and can be used not only with the purpose of sharing your code and/or results with peers but also to create tests instances for your Jupyter notebooks and even generate blog posts. You can find more information about its possibilities in the docs.

Voilà

Voilà brings the versatility of nbconvert while allowing you to interact with your code outputs safely both in the Notebook and Lab interface.

As pointed out before, Voilà keeps a Jupyter kernel instance running at all times. It is language-agnostic and it will run any language supported by Jupyter. Furthermore, as it encapsulates nbconvert, Voilà is capable of converting the notebook to any of its output formats as well as make use of all of the templates that come with nbconvert.

Rendering interactive Matplotlib figures in a web application with Voilà. Source.

You will notice that Voilà expands on the default templates: from simple one-page documents to complex grid-based dashboards and slideshows.

Grid template generated by Voilà

Slideshows with Jupyter

A common use case is to share the results as a slideshow. Let’s look at the different ways to do so.

The first step to set up your slideshow is to define the mapping between cells and slides. This mapping is saved in the cell metadata of the notebook, and JupyterLab has the appropriate built-in tool to edit this information.

Different kinds of metadata that can be added to a cell on JupyterLab

The classic notebook offers a similar interface to edit the notebook metadata. Then, to visualize the resulting slideshow, several tools are available:

RISE

RISE is a classic notebook extension that provides a toolbar button to turn the notebook into a slideshow in the main notebook UI.

RISE from the notebook to slides. Source.

Unlike the other solutions presented here, RISE slideshows are still executable in the same way as the notebook, which can be a great feature when teaching a class. However, RISE may not be an appropriate tool to share such a slideshow online because it requires giving access to a notebook server.

Unfortunately, despite the effort of the community RISE still doesn’t offer support for JupyterLab.

nbconvert

Another solution is to use the Reveal template of nbconvert, which will use the same cell metadata as RISE to produce a static slideshow.

nbconvert presentation.ipynb --to html --template reveal

Unlike with RISE, for which cells remain executable, the outcome of nbconvert is a static HTML file, and only the already-computed outputs will be displayed.

Voilà

Just like with nbconvert, the Reveal template can be used with Voilà with the --template reveal command-line argument.

Using Voilà instead of nbconvert will spawn a kernel for each connected user, enabling the use of interactive widgets in the resulting slideshow to interact with the backend.

Besides, the default behavior of Voilà is to hide input cells and only display markdown cells and output.

What’s new: Déjàvu!

Déjàvu is a new utility included in nbconvert 6.2, which simply specifies new default values for several options so as to mimick Voilà’s behavior with respect to hiding input cells and prompt numbers. However, these defaults can still be overwritten by the end-user. You can do it just like you would if you were using nbconvert.

jupyter dejavu notebook.ipynb

Déjàvu output (on the left) and Voilà output (on the right)

Voilà and Déjàvu’s outputs are identical, although, in the case of Déjàvu, interactions with widgets that require a roundtrip to the kernel will not work. In contrast, we can see the result of the calculation on the Voilà side.

To summarize and wrap up:

Déjàvu supports all of the templates available in Voilà, and even though users don’t have access to interactivity they can still benefit from widgets outputs since they’ll be generated and included on the final file;
When building a slideshow you can use JupyterLab’s interface to define specific metadata to your cells;
You can also see your code in Déjàvu’s outputs. Just pass the --show-input option to the command line.

Acknowledgments

Thanks to Juliette Taka for creating the illustrations for this blog post.

About the author

My name is Mariana Meireles, I’m a software engineer working for QuantStack. I care deeply about the impacts that technology has in the world and try my best to be the change I want to see by contributing to open source projects that stand upon libre and diverse standards.
I’m currently focusing on the Jupyter ecosystem and advocating for open science and open knowledge whenever I can.

An SQL solution for Jupyter

2021-02-03T12:03:00+00:00

A few months ago we released xeus-sqlite, a Jupyter kernel that allows users to make SQLite queries directly from the notebook. With the needs of the Jupyter data science community in mind we decided to expand on this project.

RDBMSes supported by xeus-sql

Today, we’re happy to announce the next step of this endeavour, bringing you xeus-sql a general-purpose database access tool for Jupyter, which allows you to make SQL queries on many different databases:

MySQL
PostgreSQL
SQLite3
DB2
Oracle
Firebird
and any database with an ODBC driver.

To offer all these integrations we rely on the SOCI library as the backbone of the project. SOCI abstracts all the different database connections and query details behind a unified C++ API. xeus-sql uses SOCI and xeus to expose SQL functionalities to Jupyter.

Xeus-sql, as well as the MySQL, PostgreSQL and SQLite3 backends of SOCI are available on conda-forge, meaning you can install them easily with conda or mamba:

# Installing xeus-sql and the mysql backend
mamba install xeus-sql soci-mysql -c conda-forge

# Installing xeus-sql and the postgresql backend
mamba install xeus-sql soci-postgresql -c conda-forge

# Installing xeus-sql and the sqlite backend
mamba install xeus-sql soci-sqlite -c conda-forge

There are idiosyncrasies between these different SQL backends which we detail in the xeus-sql documentation and examples.

Visualizing your queries

HTML and text table visualizations for different Jupyter interfaces

You’re probably familiar with visualizing your SQL tables and query results in the form of… tables, of course. Jupyter’s rich display system offers the option of showing them in rich-text display or plain-text display depending on which application you’re using.

Beyond displaying queries with tables, you can now easily create Vega-Lite graphs from your query results directly in the notebook:

Running a series of examples on a notebook

To support this feature xeus-sql relies on xvega, a C++ backend to vega, and custom Jupyter magics for plotting through a mini-language implemented in the xvega-bindings utility library. Besides using the mini-language you can directly provide the JSON specification for your visualization, thanks to contributions by Wang Fenjin.

Vega-Lite is a powerful library and there are many different visualizations you can create from your relational data using xeus-sql. You can see some other examples below:

Examples of different visualizations generated with xeus-sql and xvega-lite by Wang Fenjin

We’re looking forward to exchange with users and learn more about their use cases for xeus-sql and xvega, so if you have any questions please drop them on the public QuantStack chat, or open an issue at the project’s GitHub page. As always, contributions are welcome!

Try it online!

You can try the stable version on Binder without the need of installing anything on your computer just by clicking on the image below:

Acknowledgements

We are grateful to Wang Fenjin for his contributions with respect to the use of vega specs in xeus-sql. The plotting mini-language was built upon the xvega library by Madhur Tandon. We are also grateful to the SOCI contributors for their awesome package that made this work possible.

About the author

My name is Mariana Meireles and I’m a software developer working for QuantStack. I care deeply about the impacts that technology has in the world and try my best to be the change I want to see by contributing to open source projects that stand upon libre and diverse standards.

A Jupyter kernel for SQLite

2020-06-11T08:42:00+00:00

While it is well known in the Python scientific computing community, Jupyter is in fact a language-agnostic development environment. High-quality language kernels exist for the main languages of data sciences, such as Python, C++, R or Julia.
But another important tool for data science is the SQL family of programming languages. Today, we announce the release of a Jupyter kernel for SQLite.

This new kernel allows the user to use the complete SQLite syntax as well as some extra operations such as opening or closing a database file, or visualizing the data in different ways using Jupyter magics.

HTML and Text table visualizations for different Jupyter interfaces

To make the task of implementing a new Kernel for Jupyter easier, we make use of Xeus, a library providing a solid implementation of the Jupyter kernel protocol, so that we can focus on implementing the language-specific parts of the kernel.

As a connection between SQLite’s native C implementation and C++ we’re using SRombauts’ library SQLiteCpp.

Current state

xeus-SQLite is still under active development but it offers a fully functional SQLite interface and magics to perform higher-level operations that are outside of the scope of the SQL syntax, such as creating, opening, or closing SQLite databases. Some of the things you can do with xeus-SQLite are creating a new database, loading it, backing it up or deleting it. You can also check if a table exists, set and reset keys of a database and get information about it. You can find more about the use of these methods in SQLite’s documentation.

xeus-SQLite provides rich HTML display of tables in the Jupyter Notebook and Jupyter Lab. For the Jupyter Console we make use of the tabulate library for textual display.

Different commands being executed on xeus-SQLite

The future

For the future of xeus-SQLite we want to create an intuitive form of visualizing data: creating plots, graphs, charts, maps and much more from your SQLite query results, all in the same notebook. Currently we’re exploring the use of Vega in xeus-SQLite, a declarative language for creating interactive visualization designs and can do bar plots using jupyter magics:

SQLite query result with bar and table outputs for same query

This feature is still in very early stages and being developed in this branch.

We also plan on producing a static build of xeus-SQLite bundling xeus and the SQLite library into a single executable that can be easily distributed.

Try it online!

You can try the stable version without the need of installing anything on your computer just by clicking on the image below:

This won’t grant you access to the experimental branch where we’re testing, if you’re interested in trying it please check the installation guide and make sure you’re in the right branch.

Installation

Note that you first need to have Jupyter installed on your computer. You can install xeus-sqlite using mamba:

mamba install xeus-sqlite -c conda-forge

Or conda:

conda install xeus-sqlite -c conda-forge

About the author

Prior to QuantStack I worked as a developer on the PySide team at the Qt Company and as a web performance developer at Mozilla.

Interactive Graph Visualization in Jupyter with ipycytoscape

2020-04-30T16:16:00+00:00

The Jupyter widgets ecosystem offers a broad variety of data visualization tools for exploratory analysis in the notebook. However, we lack a good story for exploratory graph visualization.

Cytoscape is an open-source software platform for visualizing complex networks and integrating these with any type of attribute data. While it comes from the computational biology community, cytoscape is fully-fledged general-purpose tool for graph visualization and analytics. It now includes a modern web front-end (Cytoscape.JS) which is a great candidate for integration with Project Jupyter. This is the raison d’être of ipycytoscape.

Gene visualization in ipycytoscape

The goal of ipycytoscape is to enable users of well-established libraries of the Python ecosystem like Pandas, NetworkX, and NumPy, to visualize their graph data in the Jupyter notebook, and enable them modify the visual outcome programmatically or graphically with a simple API and user interface.

Fortunately, Cytoscape offers a broad enough API that allows ipycytoscape to be a tool that can, in fact, be used to solve any type of problem modeled as a graph. Some examples consist in the development of new chemicals to analyze interactions between substances in the pharmaceutic industry, in security systems to create attack graphs that can be useful to show possible vulnerabilities in systems, modeling human behavior to understand people’s interaction with business or even to understand complex phenomena like the current crisis. Currently, there is an effort to make ipycytoscape an accessible tool for researchers that are trying to find ways to mitigate and understand it, there is more information about this initiative on the COVID OSS Help website and the discussion is happening in this repository if you’re interested in joining it.

Current state

IPycytoscape is part of the PLASMA project (aka in French, Plateforme d’eLearning pour l’Analyse de données Scientifiques MAssives). This project aims at creating an interactive tool to teach computational analysis of massive scientific data. Its first instance, PlasmaBio, is designed for the needs of teachers and students of the European Master of Genetics at Université de Paris. PlasmaBio provides an authentic experience of the actual genomic and bioinformatic analyses performed in research labs. For that purpose, a custom JupyterHub-based system to control many different Jupyter instances is being specially developed by Jeremy Tuloup at QuantStack.

In this first version of ipycytoscape, there are still some limitations to what you may be able to do, but there are also some extents from the Python world that will just work out of the box for you. ipycytoscape offers integration between Pandas DataFrames and NetworkX, meaning that you can have a graph visualization of the data you already have with minimal or none adjustments and just a few lines of code.

Usage with NetworkX and DataFrame

ipycytoscape supports all of the built-in CytoscapeJS layouts. This includes the cola, grid, breadthfirst, circular, concentric and Dagre layout as well as the random, null or preset options to build a graph visualization that fits better to your data .Additionally, ipycytoscape also supports the PopperJS and TippyJS extensions, that allows you to create customizable tips for your nodes and edges .

cola, concentric, dagre and grid layouts

You can also use a variety of labels for a quick visualization of your nodes’ and edges’ contents.

Labels on nodes

Like most Jupyter interactive widgets, ipycytoscape relies on the traitlets library to synchronize data between the back-end and the front-end model.

Unfortunately, traitlets have a limitation when it comes to container objects and other mutable structures, because synchronization is only triggered upon assignment of the container and not when modifying individual elements.

To work around this limitation, we make use of the excellent Spectate library by Ryan Morshead, which triggers observers upon individual element changes in containers.

Interaction between ipywidgets and ipycytoscape

Try it online!

You can try it without the need of installing anything on your computer just by clicking on the image below:

Installation

Note that you first need to have Jupyter installed on your computer. You can install ipycytoscape using pip:

pip install ipycytoscape

Or using conda:

conda install -c conda-forge ipycytoscape

If you use JupyterLab, you would need to install the JupyterLab extension for ipycanvas (this requires nodejs to be installed):

jupyter labextension install @jupyter-widgets/jupyterlab-manager jupyter-cytoscape

About the author

Prior to QuantStack I worked as a developer on the PySide team at the Qt Company and as a web performance developer at Mozilla.

Currently I’m working on expanding the Jupyter ecosystem with new libraries and functionalities, like an experimental SQLite kernel.

Acknowledgements

The development of ipycytoscape at QuantStack was funded as part of the PLASMA project, led by Claire Vandiedonck, Pierre Poulain, and Sandrine Caburet, associate professors at Université de Paris.

Sponsors to the PLASMA initiative include:

Région Île-de-France, via the “Trophées franciliens de l’innovation numérique dans le supérieur” (EdTech 2018) grant program,
Université de Paris, via the Initiative of Excellence (IdEx) Label and its “inovating teaching” grant program,
EUR G.E.N.E., the graduate school on Genetics and Epigenetics,
the university training “Création, analyse et valorisation de données biologiques omiques” (DU Omiques).