Jupyter Blog - Marc Wouts

Make your Pandas or Polars DataFrames Interactive with ITables 2.0

2024-03-19T18:39:00+00:00

ITables, or Interactive Tables, is a MIT-licensed Python package that renders Python DataFrames using the DataTables JavaScript library. ITables 2.0, that I have just released, adds support for the DataTables Extensions. In this post we review the functionalities brought by this release.

How to use ITables, and what you get

ITables can be installed using either pip or conda:

pip install itables

conda install itables

ITables is essentially a Python wrapper for DataTables. We have managed to keep its dependencies to a minimum: ITables only requires IPython, pandas and numpy, which you must already have if you use Pandas in Jupyter (add polars and pyarrow if you wish to use ITables with Polars DataFrames).

To use ITables in your notebook, run this code snippet:

from itables import init_notebook_mode

init_notebook_mode(all_interactive=True)

After that, every Pandas or Polars DataFrame will be displayed using the DataTables library. With DataTables, you get an easier and more complete access to your data. You can expand the table, explore the various pages, sort the data or even search through it, without having to go back to the Python prompt.

A Pandas DataFrame rendered with ITables

To render only specific tables as interactive DataTables, or pass arguments to the DataTable constructor, you can use the show function:

from itables import show

A brief history of ITables

I started the ITables project back in 2019. The full changelog is available here, but let me highlight a few important milestones:

v1.0 (June 2022): Offline mode
v1.2 (Aug 2022): In addition to Jupyter Lab, Notebook, Book, HTML export, VS Code, PyCharm, Google Colab, RISE presentations, Voilà applications, ITables works in Shiny applications
v1.3.5 (Nov 2022): Large tables are fast too
v1.5 (March 2023): Polars DataFrames are supported
v1.6 (Sep 2023): Pandas Style objects can be displayed with ITables
v1.7 (Feb 2024): ITables works with Quarto
v2.0 (March 2024): ITables support the DataTables Extensions!

It took me much research (and two years I am afraid) to reach v2 and the DataTables Extensions. But we’re finally there! Let’s review what these extensions mean to your data workflow.

My favorite DataTables Extensions

Download your data!

If you think of a Jupyter Notebook, it might sound strange that we are considering downloading the table data, since we have it already in the Python session. Actually, ITables does not stop at the Notebook. The interactive DataTables continue to be interactive even if you export the notebook to an HTML document with e.g. Jupyter nbconvert (jupyter nbconvert --to html). The are also interactive in a RISE or Quarto presentation. ITables also works in Jupyter Book (which we use for the ITables documentation), and in interactive applications build with e.g. Voilà or Shiny.

Hopefully it’s clearer now why we might want to make the data downloadable! And the good news is that, with the Buttons extension of DataTables, is it as easy as:

show(df, buttons=["copyHtml5", "csvHtml5", "excelHtml5"])

The Copy/CSV/Excel buttons

Cascade search

The SearchPanes extension lets you proceed to a quick and visual search through the columns that have repeated values:

The SearchPanes extension

Search builder

I find the SearchBuilder extension very useful. Also, I love the option to set a pre-defined search and display only the part of the dataset we want to focus on.

The SearchBuilder extension

Downsampling

Last but not least, I need to tell you about ITables’ down-sampling mechanism. By default, only a subset of the table with an estimated size of not more than 64kB (and not more than 200 columns) is displayed. You can change this with e.g.

import itables.options as opt

opt.maxBytes = 131072
opt.maxColumns = 0

You can tell whether your table was down-sampled by looking at the table summary at the bottom right of your table.

Please keep the following in mind: when down-sampling occurs, only a fraction of the data is passed on to DataTables, and hence the search or data export functions will only have access to that partial dataset.

Down-sampling is what makes ITables fast. Think twice before setting opt.maxBytes to a large number or to 0, as this might well freeze your notebook. Displaying a 1G DataFrame will make your notebook at least as big (probably bigger as the data is exported to JSON), and it’s not clear that your browser will support this.

Acknowledgments

ITables would not exist without the beautiful DataTables JavaScript library. The DataTables library (MIT license) is authored and maintained by Allan Jardine. DataTables has a great documentation, many examples, and also a useful forum. Allan also provided precious advice on how to bundle the extensions with ITables.

The point in using DataTables in the context of data exploration and analysis had been demonstrated prior to ITables by the DT package for R.

Last but not least, ITables owes a lot to my brother François Wouts. François patiently helped me find a way to provide DataTables and its extensions that would be compatible with all the editors and rendering contexts that we want to support in ITables.

The Jupytext Menu is back!

2024-01-31T20:29:00+00:00

A few weeks back, jupytext==1.16.0 went out. That release included multiple amazing contributions by Mahendra Paipuri, this post will go over the new features, and thank Mahendra for his great work.

As we will see below, Mahendra restored the Jupytext Menu, made the extension fully compatible with JupyterLab 4 and Jupyter Notebook 7, and added the option to create Text Notebooks directly from the launcher.

What is Jupytext

Jupytext is a Python package that lets you save Jupyter Notebooks as text notebooks. Multiple formats are supported, and the notebooks can be saved either as Markdown documents with a .md extension, or as scripts with e.g. a .py extension (assuming you use Python - otherwise multiple languages are supported).

A .py notebook in the percent format looks like this:

# %% [markdown]
# This is a markdown cell

# %%
def f(x):
  return 3*x+1

# %%
[f(i) for i in range(6)]

To open a text notebook as a notebook in Jupyter, right-click on the document and select “Notebook” (you can also change the default viewer to “Jupytext Notebook” if you wish: see below the section about the settings)

Right click to open a text notebook with the Notebook editor

Text notebooks are conveniently edited and executed in Jupyter as notebooks:

A Text Notebook in JupyterLab

You can also edit them with your favorite text editor. You will get the changes back in Jupyter by re-opening or by reloading the document: click e.g. on reload Python file from disk in the File menu.

Text notebooks only contain the notebook inputs. For that reason they use much less disk space than .ipynb notebooks. They are also better suited for version control as they only contain the content that was actually typed by the user. But even more useful are paired notebooks: text notebooks paired with an .ipynb notebook where the notebook outputs are preserved. When a paired notebook is saved, Jupytext writes the notebook to both files. When the notebook is read or reloaded in Jupyter, the notebook inputs are loaded from the most recent file, meaning that any edits on the text notebook will be reflected in Jupyter (and propagated to the .ipynb file the next time it is saved).

One of the most frequent operation when using Jupytext is to pair an .ipynb notebook with a text notebook in the format of your choice.

In earlier versions of Jupytext, pairing a notebook had to be done through the command palette.

In Jupytext v1.16, thanks to Mahendra’s work on the front-end extension, you can directly use the Jupytext Menu for this. And the menu is available for both JupyterLab 4 and Jupyter Notebook 7!

If you are new to Jupytext, we recommend that you pair your .ipynb notebooks with either a percent script or with a MyST Markdown file. The former format works well if you want to save and edit your notebook as a script, while the latter is well suited for writing documentation.

Once you have paired the notebook with a Jupytext format, save your notebook, and the paired files will be created or updated.

The Jupytext Menu

Creating new text notebooks

For some notebooks you might decide that you don’t need to save the outputs at all. In that case you can work with an (unpaired) text notebook.

To create a text notebook you can use the New Text Notebook sub-menu under File:

Create Text Notebooks directly from the menu

If, later on, you decide that you want to preserve the notebook outputs in a paired .ipynb notebooks, you will just have to pair the document to an .ipynb notebook using the Jupytext Menu documented at the previous paragraph.

Text Notebooks in the launcher

In Jupytext v1.16, thanks to Mahendra’s work on the front end, text notebooks are also available in a Jupytext section in the Jupyter launcher:

Text Notebooks are also available in the launcher

Jupytext Settings in JupyterLab

Since there are many possible formats for text notebooks, we have decided to expose only the most common ones, by default, in the launcher and in the New Text Notebook menu.

You can include more formats in the menu and in the launcher by changing the Jupytext settings in the Settings Editor (in the Settings menu):

The Jupytext Menu/Launcher settings

You might also want to open certain text documents as notebooks with a single click. You can achieve this by setting the default viewer for those documents to “Jupytext Notebook”. For instance, if you want to open .py and .md files as notebooks with a single click, you can configure the default viewers like this:

Changing the default viewers to open Text Notebooks with a single click

Alternatively, you can also list and set the default viewers with the jupytext-config utility (which was developed recently by Thierry Parmentelat), using e.g.

jupytext-config set-default-viewer python markdown

to set “Jupytext Notebook” as the default viewer for both Python and Markdown files.

Pairing notebooks globally

The pairing commands provided through the Jupytext Menu and commands act on individual notebooks, by settings a jupytext.formats metadata in the notebook.

It is also possible to pair all the notebooks within a certain folder using a jupytext.toml configuration file - see Jupytext’s documentation. Please note that:

The local metadata in the notebook takes precedence over the global configuration, and
Jupyter is aware of the global jupytext.toml file and will pair the notebooks accordingly, however the Jupytext Menu is not aware of the global configuration, so the paired formats selected through the global configuration will not be checked in the menu.

Installing Jupytext v1.16

The front end extension for JupyterLab shipped with Jupytext v1.16 requires JupyterLab 4, and/or Jupyter Notebook 7. Please upgrade Jupyter accordingly. Then, install Jupytext with

pip install 'jupytext>=1.16.0'

conda install 'jupytext>=1.16.0' -c conda-forge

and restart your Jupyter server with e.g. jupyter lab.

Acknowledgments

I would like to thank Mahendra Paipuri for his impressive work. I am pretty sure that the Jupytext Menu, and the option to create new text notebooks, will be much appreciated by Jupytext users!

While users will mostly notice Mahendra’s work on the front end extension, it was actually not his only contribution to this release! Mahendra also thoroughly revisited the packaging of Jupytext, and helped us transition the project to the src layout, and our old setup.py to an up-to-date pyproject.toml configuration that uses hatch to build Jupytext.

I also want to thank Cristian Le for his precious advice regarding the layout refactoring, and for helping us to tackle the CI reorganization. Thierry Parmentelat, who had previously ported the front-end extension to JupyterLab 4, contributed much testing and feedback on this new version of the front-end extension.

It is always a pleasure for me to maintain Jupytext, an amazing adventure that started five years ago already. However, I can get busy at times (Jupytext comes in addition to my day job, one cat, two bikes, three kids), so I am really thankful for receiving such contributions, especially when they are of such a great quality!