Jupyter Blog - documentation

The Jupytext Menu is back!

2024-01-31T20:29:00+00:00

A few weeks back, jupytext==1.16.0 went out. That release included multiple amazing contributions by Mahendra Paipuri, this post will go over the new features, and thank Mahendra for his great work.

As we will see below, Mahendra restored the Jupytext Menu, made the extension fully compatible with JupyterLab 4 and Jupyter Notebook 7, and added the option to create Text Notebooks directly from the launcher.

What is Jupytext

Jupytext is a Python package that lets you save Jupyter Notebooks as text notebooks. Multiple formats are supported, and the notebooks can be saved either as Markdown documents with a .md extension, or as scripts with e.g. a .py extension (assuming you use Python - otherwise multiple languages are supported).

A .py notebook in the percent format looks like this:

# %% [markdown]
# This is a markdown cell

# %%
def f(x):
  return 3*x+1

# %%
[f(i) for i in range(6)]

To open a text notebook as a notebook in Jupyter, right-click on the document and select “Notebook” (you can also change the default viewer to “Jupytext Notebook” if you wish: see below the section about the settings)

Right click to open a text notebook with the Notebook editor

Text notebooks are conveniently edited and executed in Jupyter as notebooks:

A Text Notebook in JupyterLab

You can also edit them with your favorite text editor. You will get the changes back in Jupyter by re-opening or by reloading the document: click e.g. on reload Python file from disk in the File menu.

Text notebooks only contain the notebook inputs. For that reason they use much less disk space than .ipynb notebooks. They are also better suited for version control as they only contain the content that was actually typed by the user. But even more useful are paired notebooks: text notebooks paired with an .ipynb notebook where the notebook outputs are preserved. When a paired notebook is saved, Jupytext writes the notebook to both files. When the notebook is read or reloaded in Jupyter, the notebook inputs are loaded from the most recent file, meaning that any edits on the text notebook will be reflected in Jupyter (and propagated to the .ipynb file the next time it is saved).

One of the most frequent operation when using Jupytext is to pair an .ipynb notebook with a text notebook in the format of your choice.

In earlier versions of Jupytext, pairing a notebook had to be done through the command palette.

In Jupytext v1.16, thanks to Mahendra’s work on the front-end extension, you can directly use the Jupytext Menu for this. And the menu is available for both JupyterLab 4 and Jupyter Notebook 7!

If you are new to Jupytext, we recommend that you pair your .ipynb notebooks with either a percent script or with a MyST Markdown file. The former format works well if you want to save and edit your notebook as a script, while the latter is well suited for writing documentation.

Once you have paired the notebook with a Jupytext format, save your notebook, and the paired files will be created or updated.

The Jupytext Menu

Creating new text notebooks

For some notebooks you might decide that you don’t need to save the outputs at all. In that case you can work with an (unpaired) text notebook.

To create a text notebook you can use the New Text Notebook sub-menu under File:

Create Text Notebooks directly from the menu

If, later on, you decide that you want to preserve the notebook outputs in a paired .ipynb notebooks, you will just have to pair the document to an .ipynb notebook using the Jupytext Menu documented at the previous paragraph.

Text Notebooks in the launcher

In Jupytext v1.16, thanks to Mahendra’s work on the front end, text notebooks are also available in a Jupytext section in the Jupyter launcher:

Text Notebooks are also available in the launcher

Jupytext Settings in JupyterLab

Since there are many possible formats for text notebooks, we have decided to expose only the most common ones, by default, in the launcher and in the New Text Notebook menu.

You can include more formats in the menu and in the launcher by changing the Jupytext settings in the Settings Editor (in the Settings menu):

The Jupytext Menu/Launcher settings

You might also want to open certain text documents as notebooks with a single click. You can achieve this by setting the default viewer for those documents to “Jupytext Notebook”. For instance, if you want to open .py and .md files as notebooks with a single click, you can configure the default viewers like this:

Changing the default viewers to open Text Notebooks with a single click

Alternatively, you can also list and set the default viewers with the jupytext-config utility (which was developed recently by Thierry Parmentelat), using e.g.

jupytext-config set-default-viewer python markdown

to set “Jupytext Notebook” as the default viewer for both Python and Markdown files.

Pairing notebooks globally

The pairing commands provided through the Jupytext Menu and commands act on individual notebooks, by settings a jupytext.formats metadata in the notebook.

It is also possible to pair all the notebooks within a certain folder using a jupytext.toml configuration file - see Jupytext’s documentation. Please note that:

The local metadata in the notebook takes precedence over the global configuration, and
Jupyter is aware of the global jupytext.toml file and will pair the notebooks accordingly, however the Jupytext Menu is not aware of the global configuration, so the paired formats selected through the global configuration will not be checked in the menu.

Installing Jupytext v1.16

The front end extension for JupyterLab shipped with Jupytext v1.16 requires JupyterLab 4, and/or Jupyter Notebook 7. Please upgrade Jupyter accordingly. Then, install Jupytext with

pip install 'jupytext>=1.16.0'

conda install 'jupytext>=1.16.0' -c conda-forge

and restart your Jupyter server with e.g. jupyter lab.

Acknowledgments

I would like to thank Mahendra Paipuri for his impressive work. I am pretty sure that the Jupytext Menu, and the option to create new text notebooks, will be much appreciated by Jupytext users!

While users will mostly notice Mahendra’s work on the front end extension, it was actually not his only contribution to this release! Mahendra also thoroughly revisited the packaging of Jupytext, and helped us transition the project to the src layout, and our old setup.py to an up-to-date pyproject.toml configuration that uses hatch to build Jupytext.

I also want to thank Cristian Le for his precious advice regarding the layout refactoring, and for helping us to tackle the CI reorganization. Thierry Parmentelat, who had previously ported the front-end extension to JupyterLab 4, contributed much testing and feedback on this new version of the front-end extension.

It is always a pleasure for me to maintain Jupytext, an amazing adventure that started five years ago already. However, I can get busy at times (Jupytext comes in addition to my day job, one cat, two bikes, three kids), so I am really thankful for receiving such contributions, especially when they are of such a great quality!

Introducing JupyterHub’s Outreachy interns! — December 2022 Cohort

2022-12-07T07:45:00+00:00

As part of the community strategic support project funded by CZI’s EOSS grant series, the JupyterHub sub-project has funding to support Outreachy interns through four cohorts over the next two years. We would like to introduce you to the first cohort and the projects they will be working on!

Ogoh Blessing Onyowoicho — Improve Accessibility in JupyterHub

Accessibility is the ability of tools (in our case web tools) to be used by a variety of communities with different disabilities. There are a variety of standards and tools for evaluating and ensuring that a web page can be used effectively by as many people as possible. Work by the Accessibility team is ongoing to define a set of standard tools to improve accessibility across the Jupyter ecosystem.

The JupyterHub project is working to improve the accessibility of its pages to ensure we are providing tools that are as useful as they can be to as many people as we can. During the internship, we will evaluate JupyterHub’s accessibility, find ways to improve it, and integrate accessibility testing into the development process, in collaboration with the Accessibility team, to ensure we do a better job going forward.

Ogoh Blessing says:

My name is Ogoh Blessing Onyowoicho. I am a self-taught Frontend developer based in Lagos, Nigeria and I am an Outreachy intern working on Improving the accessibility of JupyterHub.
I am excited to work on JupyterHub because it is giving me the opportunity to use skills that I have accrued over the years to proffer solutions to problems that affect people’s lives directly. On hearing accessibility, the first thought one has is that it involves building web products that people with special needs can use seamlessly. Though this is part of it, accessibility goes way beyond this. It involves building products that different users (e.g users at different locations, users with different devices etc) can use easily. The thought of contributing to improving the experience of so many people alone excites me.
In the coming months, I hope to learn and continue to hone my skills as I am guided by my mentors and members of the community I get to interact with.

Allan Wasega — Restructure and improve JupyterHub documentation by implementing the Diátaxis framework

JupyterHub has a range of documentation that covers both developer and user audiences in order to help them deploy, maintain, and use their own instance of a JupyterHub. The success of an open source software project to (i) be adopted by users, and (ii) receive meaningful contributions relies heavily on the quality, navigability and accessibility of documentation so that users and developers have all the information they need to achieve what they want to do.

A framework for organising technical documentation has arisen called diátaxis. It takes a systematic approach to understanding user requirements of documentation throughout the lifecycle of interaction with a product and posits that different user needs require different approaches in creation of the documentation, as well as a layout to navigate these different “modes” of documentation.

This project will focus on a refactoring of the documentation for the JupyterHub package. We will begin by performing a review of the present documentation, categorise these into the diataxis framework, and then restructure the documentation files in the repository. Once we have transformed the documentation into this framework, it will be much easier to identify missing and unclear documentation (those that were difficult to categorise). We can then begin to curate resources that can fill the gaps and improve documentation that is not specific enough.

Allan says:

I am Allan Wasega, from Nairobi, Kenya. Broadly, I like to describe myself as a researcher and a writer. Researcher because looking into things to find patterns or hidden information has always been of interest to me. Writer because I figured early on that words allow me to express myself and to communicate to a larger audience than that inside my head :). As a Computer Science student, I looked for ways to bring these two skills together within the realm of computing and that is how I found myself in the technical writing space.
As an undergraduate student, I used Jupyter Notebooks extensively for most of my programming assignments and projects. As a result, when making my Outreachy application, the JupyterHub project immediately piqued my interest because I did not know that there was a way to serve Jupyter Notebooks to multiple users, such as within a classroom setting. Presently, I serve as a mentor at KamiLimu, which is a mentorship program for students pursuing technology-aligned courses in tertiary institutions in Kenya. One of the objectives of KamiLimu is to introduce students to tech specializations such as Data Science and Machine Learning, which, from experience, uses Jupyter Notebooks! Therefore, I am excited to work on JupyterHub because I will be learning about and helping to improve a product I hold dear and which I can use to advance the skills of the next generation of computer technologists in Kenya.
First, I hope to gain a deep understanding of how JupyterHub works so as to spread the word about it and its functionalities. Second, I hope to apply and advance my technical writing skills. I encountered the Diataxis Framework earlier this year while working as a technical writer at Tingle Software, a Nairobi-based software company. I have also been following Daniele Procida (the author of the framework) on Twitter for a while now and his work has been quite inspiring. Through this project, I will apply the Diataxis Framework to restructure JupyterHub’s documentation and, in doing so, further my understanding of the framework. Finally, I am passionate about mentorship, especially in the tech field. Therefore, besides expanding my professional and personal networks, this project will enable me to learn how to conduct mentorship within a global setting, and I can, in turn, apply this knowledge within my local community.

Sheila Kahwai — Create a reusable JupyterHub pytest plugin

JupyterHub is a modular and extensible project, with parts, like the proxy, authenticator and spawner, that can be easily changed and extended. Testing the functionality of these components against JupyterHub is important and it requires various hub setups that can sometimes become complicated.

Currently, each of these hub components and the hub itself define their own testing infrastructure, building everything from the ground up using the pytest framework. But some of this complex work is either repetitive across JupyterHub sub-projects, or under-specified for some of them.

This project will abstract out these common parts into a separate testing framework. This framework should be a pip-installable pytest plugin that would provide various hub functionalities through pytest fixtures. These fixtures can then be re-used by JupyterHub and its components to bootstrap their own testing suite.

Integrating this plugin will drive some important refactoring work of the current testing architecture of JupyterHub and have a great impact in improving the overall test coverage, maintainability and continuity of the JupyterHub project.

Sheila says:

My name is Sheila Kahwai. I am a self-taught python developer from Nairobi, Kenya, working to specialize in back-end engineering.
I am excited to work on JupyterHub because it has made many products I have used throughout my learning journey. It is a privilege to work with a diverse community that has created products that are very useful to equally diverse users like myself.
While creating a reusable JupyterHub pytest plugin, I hope to gain more insight into creating plugins for massive codebases to improve maintainability and scalability with clean and reusable code. Through this project, I look forward to improving the overall testing infrastructure of the various JupyterHub components.

Welcome to the interns! We’re so excited to start working with you!

Inspector JupyterLab

2022-04-11T11:50:00+00:00

JupyterLab provides multiple ways to improve your coding workflow: code highlighting, code completion, theming, debugger with rich variable rendering and more.

The JupyterLab inspector

The JupyterLab inspector is one of the ways to enhance your coding experience, it is a UI panel that provides contextual help while you are typing:

You can open the inspector using the Ctrl + I shortcut (or ⌘ + I on Mac).

With the Python kernel (ipykernel or xeus-python), this contextual help normally contains a text representation generated using the inspect Python module and the docstrings associated to an object. But there is a way to make it much nicer!

Go-Go-Gadget Docrepr

Docrepr is a Python package which renders docstrings into HTML using Sphinx, maintained by the Spyder team. IPython allows to use docrepr for code inspection, making it the perfect combination for a nice rendering of the contextual help in the inspector!

Thanks to the recent work on the jupyterlab-pygments extension, the docrepr output even respect the current JupyterLab theme for syntax highlighting!

Installation

First you will need to install the package (either with pip or mamba)

pip install docrepr jupyterlab_pygments

Or:

mamba install -c conda-forge docrepr jupyterlab_pygments

In order for IPython to use docrepr, you will need to execute the following in your Notebook (e.g. in the first cell):

ip = get_ipython()
ip.sphinxify_docstring = True
ip.enable_html_pager = True

This also can be set in your IPython configuration file located in ~/.ipython/profile_default/ipython_config.py:

c = get_config()

c.InteractiveShell.sphinxify_docstring = True
c.InteractiveShell.enable_html_pager = True

You are all set! The JupyterLab inspector will now render HTML representations of the contextual help.

Acknowledgments

Docrepr has a long history, it originates from the Sphinxify tool of the Sage project that was created back in 2009 by Tim Dumol. In 2010 Carlos Córdoba used it to power Spyder’s help pane and in 2015 he extracted that code and created the docrepr Python package, so that other projects could benefit from it. It was later integrated in IPython by Sylvain Corlay. In 2021 C.A.M. Gerlach, Ahmed Fasih and myself updated the package to support the latest Sphinx version.

My work on this project at QuantStack was funded by Bloomberg.

About the Author

My name is Martin Renou, I am a Scientific Software Engineer at QuantStack. Before joining QuantStack, I studied at the aerospace engineering school SUPAERO in Toulouse, France. I also worked at Logilab in Paris, France and Enthought in Cambridge, UK. As an open-source developer at QuantStack, I work on a variety of projects, from xtensor and xeus-python in C++ to ipyleaflet and bqplot in Python and JavaScript/TypeScript, I am also the author of several JupyterLab extensions.

Integrating output in documentation with jupyter-sphinx

2019-05-30T18:22:00+00:00

The problem

We love Jupyter notebooks for accommodating a computational narrative — a combination of explanation, code, and the output of this code.
Unfortunately, some tasks cannot be accomplished well by notebooks.
If you are writing documentation for your software project, chances are that you want to provide navigation across many tutorials and explanation pages.
You will also want to automatically document the API, perhaps also maintain a bibliography, and you certainly will want all the classes and functions from your module to automatically link to their documentation pages.
In short, your best bet is Sphinx.

Sphinx does not provide a way to build a computational narrative: by itself, it cannot execute any code, nor does it know how to handle the output of that code. This limitation is well known and there are great tools offering a workaround; I’ll list the ones that I know about:

nbsphinx allows incorporating executed notebooks into a documentation website. Unfortunately, markdown used in notebooks is a much more limited markup language than restructured text.
sphinx-gallery takes a collection of scripts, executes them, shows the code and the output in the documentation, and even automatically links object names occurring in a script to their documentation. It also parses rst-formatted comments and renders those.

Jupyter-sphinx

We have just made an addition to this list, a freshly rewritten jupyter-sphinx extension, that was previously specialized to render Jupyter widgets.
To embed arbitrary output in your documentation using jupyter-sphinx you only need to use the jupyter-execute directive:

.. jupyter-execute::

  print('Hello world!')

Under the hood, all such code chunks are converted to cells in a notebook and executed using nbconvert. We then rely on the Jupyter format and protocol to interpret what to do with the results of executing the code. This means you already know how the output will be shown: we apply exactly the same logic as Jupyter notebook does.

For example, here we make a plot:

And here we are rendering some widgets:

If you want to see jupyter-sphinx used to make package documentation, check out adaptive, the first adopter (full disclosure—I am one of its authors).

An important corollary: building upon the Jupyter kernel protocol makes jupyter-sphinx language-agnostic; jupyter-sphinx works with absolutely any language for which a kernel exists.

Try it

We have freshly published a release candidate, give it a go using pip install jupyter-sphinx==0.2.0rc1 --pre. We would love to hear your feedback, especially if you are using other ways of embedding outputs in the documentation, or if you are using an older version of jupyter-sphinx.