Jupyter Blog - yuvipanda

Berkeley Institute for Data Science (BIDS) joins the mybinder.org

2026-04-08T23:34:00+00:00

Berkeley Institute for Data Science (BIDS) joins the mybinder.org federation in partnership with 2i2c

The Berkeley Institute for Data Science (BIDS) is the birthplace of the current iteration of mybinder.org, all the way back in 2017. In 2026, they are back as a member of the federation, joining 2i2c and GESIS in contributing to the cloud costs of keeping mybinder.org running!

The BIDS node is running on the OVH Cloud Provider, joining the two existing nodes (from 2i2c and GESIS) running on Hetzner. By running on a different cloud provider than our existing nodes, we reduce the risk that a cloud provider outage or policy change will take down the entire mybinder.org service. This immediately came into play, as Hetzner is having a lot of trouble with its object storage service, increasing failure rates on mybinder.org. We were able to shift more of our users to the BIDS node seamlessly, reducing disruptions for our end users.

The BIDS node thus temporarily served 50% of our traffic, taking a higher than usual share of the load as we wait for error rates on Hetzner object store to get better. It has served more than 64,000 users so far in the short time it’s been up, and expected to do more.

Graph of mybinder.org sessions across our federation members over the last 4 months, showing the BIDS member taking on more of the load over time as it is brought online. The graph is jagged due to dips in demand over the weekend

This node was brought up in partnership with 2i2c. 2i2c contributed expertise in extending the mybinder.org deployment infrastructure to run on OVH, as well as working with UC Berkeley procurement to ensure that the cloud costs can be paid for on a continuous basis. Many thanks to the ever awesome MinRK (who is now back to working at BIDS!) for a lot of the technical work in taking this through!

Want your organization to join the mybinder.org federation, materially making a difference in resilience and availability of the service, even at just a few hundred dollars a month? Come talk to us on the Jupyter Zulip!

Scaling “Maintainer Intuition” with Pull Request Triage Boards

2025-11-16T21:22:00+00:00

When I helped start 2i2c.org, one of my goals for the non-profit was to experiment with new and different ways of supporting the Jupyter ecosystem’s long term health. As part of that, we have identified how foundational contributions are very important (and distinct from directed contributions), and have been experimenting with new ways to make foundational contributions in sustainable and structured ways.

One such way was via systematically doing code reviews for Pull Requests from non-maintainers in the JupyterHub ecosystem. Reviewing PRs is a critical way that maintainers keep an open source project moving forward, but identifying PRs that can productively be merged is hard. This is a post describing our system for scaling this in our team. Each 2 week sprint, we:

Looked at all open PRs in the JupyterHub org who were not maintainers
Picked a PR that I deemed was reviewable and ideally mergeable within this time window
Have an engineer on the team pick up that PR, and try to get it to close
Report back on what we have learnt, so we can iterate on our process

We managed to do this for a majority of sprints over the last roughly 12 months! One key bottleneck we identified in the process was Step 2. In particular, I was relying on my maintainer intuition to pick a single PR that I believe can be merged, so others in the team can do review work. I started exploring what this intuition is, and if it can be scaled.

What is this maintainer intuition?

How did I pick a PR from a long list of open PRs? Observing my own behavior a few times, I noticed I was looking for:

PRs that aren’t too big, and are a reasonable size that can be merged within a 2 week window
CI tests passing, so at least our automated checks haven’t caught any issues with it
Features or bug fixes that I believe add value to the project and move us in the right direction towards being able to support our users as they need (this is the hardest!)
If the author of the PR is a newish contributor, as I want to encourage them to stick around by being responsive to their gift. All PRs are gifts that we may or may not choose to accept, but should do so with grace.
How long ago the PR was opened. There is such a big difference between a response to your PR 2 days after you make it vs 2 months vs 2 years. I prioritized newer PRs.
What kind of contribution is it primarily? Different engineers on our team have different skillsets (JS, Python, etc) and I wanted to match the PR to what the engineer preferred code reviewing.

While (3) is hard to scale, everything else seemed like something we could build systems for that let others follow a process, thus removing myself as a bottleneck. I experimented with some GitHub issue filters and project automations, and after finding them lacking, built out a brand new open source project to do this: pr-triage-board-bot!

PR Triage GitHub Boards

pr-triage-board-bot automatically maintains PR Triage Boards for a Github organization. You can check out the current boards to get a sense of how it looks: JupyterHub, JupyterLab, and GeoJupyter.

For each PR Triage Board, the bot will:

Add all open, ready for review PRs
Remove all closed or draft PRs
Annotate each PR with additional deterministic project fields that allow for sorting and filtering in the project board. To start with, it populates the following fields:
1. Author kind (Maintainer, Seasoned Contributor, First Time Contributor, Bot)
2. Size (Number of lines touched)
3. Date it was opened
4. If a maintainer has already interacted with the PR (One, Many, None)
5. Are there merge conflicts?
6. What kind of files were mostly changed? (Python, JS, Docs)
Keep this up to date by automatically running every hour via GitHub actions

Once these project fields are populated automatically, maintainers can create different Project Views for themselves to help with different workflows via filters and sorting. For example, the JupyterHub board contains the following views:

PRs split by author kind, sorted by newness and size so we can try to respond to PRs from new users as early as possible
PRs that have not had a single maintainer interaction on them, so we can acknowledge people’s contributions to us even if we can’t fully review it at the moment
Bot PRs with all tests passing and no merge conflicts, so we can more easily stay on top of automated updates
PRs that have been approved by a maintainer but not merged yet, as sometimes a maintainer wants to give others time to object but needs to go back and hit merge.
PRs that are primarily updating documentation, as the review process for this can be different

For our original purpose of getting more people to do code review, this board has served well — we can roughly say ‘Pick a PR that looks good to you from the top of the “First Time Contributor” or “Seasoned Contributor” list’, and that relieves me from being the bottleneck quite a bit. Other maintainers are also finding a lot of value in this board. For example, if you only have 15 min, you can probably get a clean bot PR reviewed and merged. Or look for PRs that haven’t been acknowledged by a maintainer and engage with them. Our next step is to help run structured social experiments, where we try to establish specific ceremonies in the open source ecosystem to get specific lists of open PRs (such as “PRs with no maintainer engagement, or PRs older than 2y”) down to zero. Stay tuned to hear more :)

We are essentially using GitHub Project Fields as a database, adding additional fields that are very valuable to maintainers but not available in GitHub by default. While ideally these would be contributed by us directly into GitHub, only Microsoft has control over what gets added to the product. So we find interesting workarounds like this to accomplish our goals :)

Successful Adoption

I was just playing with this, and Raniere from the JupyterHub team spotted it and asked about it on Zulip. Jason Grout from the JupyterLab team was also super interested, and with contributions from him, the bot quickly got adopted by the JupyterLab GitHub org as well. We cleaned this up a bit more, and found adoption within the GeoJupyter project too with help from Matt Fisher. As with everything we do at 2i2c, I had tried to design this to be widely useful to many orgs and maintainers rather than just us, and looks like I have wildly succeeded :)

I am happy to announce today that 2i2c is officially donating pr-triage-board-bot to Project Jupyter! I will still continue to contribute to maintaining the project, and welcome contributions from everyone else too!

If this looks useful to your open source project, consider adopting it by following these instructions. The project is still fairly new, so if you run into issues please let us know on the Project Jupyter Zulip chat or by opening an issue.

Accurately counting Daily, Weekly & Monthly active users on JupyterHub

2023-03-27T09:16:00+00:00

Being able to say ‘we served X unique users over the last month’ (the Monthly Active User metric) is very helpful when advocating for resources for a JupyterHub your organization is running. However, until now, figuring out that number accurately has been difficult, requiring keeping and analysing JupyterHub logs.

That changes with JupyterHub 3.1! This Pull Request adds daily, weekly, and monthly active user metrics to JupyterHub, accessible via the prometheus interface by hitting the /metrics URL on your JupyterHub. These metrics are calculated by JupyterHub itself, and are pretty accurate as it already keeps track of when a user was last active. This relies metric on each JupyterHub user matching an actual user, so if you are using your JupyterHub deployment purely as an API with ephemeral users (as binderhub does, for example) or delete inactive users, these will not be useful numbers.

Ideally, you should have a prometheus instance to scrape and store metrics from your JupyterHub over time, so you can track this metric over time. However, in a pinch, you can also go directly to https://<your-hub-url>/hub/api/metrics and see the current value of these metrics. They are in the prometheus exposition format, but you can look for something like:

# HELP jupyterhub_active_users number of users who were active in the given time period
# TYPE jupyterhub_active_users gauge
jupyterhub_active_users{period="24h"} 610.0
jupyterhub_active_users{period="7d"} 2800.0
jupyterhub_active_users{period="30d"} 4526.0

This denotes that this particular hub had 610 daily active users (active over the last 24 hours), 2800 weekly users (over last 7d) and 4526 monthly ones (over the last 30 days). Very helpful if you want to quickly add numbers to a report :)

Note that depending on your hub’s configuration, access to the /metrics endpoint might require authentication.

If you do have a prometheus installation for your JupyterHub, you may benefit from deploying the JupyterHub Grafana Dashboards. These are targeted at installations of zero-to-jupyterhub on kubernetes, and provide a lot of useful usage & diagnostic information.

Grafana Dashboard showing “Hub Usage Stats” for a JupyterHub deployment. Four panels, clockwise: Current Active Users, Daily Active Users, Weekly Active Users, Monthly Active Users.

Happy report writing!

Securely pushing to GitHub from a JupyterHub with gh-scoped-creds

2022-04-21T16:54:00+00:00

Many JupyterHub users want to push and pull their content from GitHub in order to collaborate and share their work. However, working on a JupyterHub means working on shared infrastructure, not your own laptop, and this poses some extra security risks that have made two-way sync with GitHub more difficult. This post describes gh-scoped-creds, a new tool to make it quick and easy to authorize a JupyterHub session with push access to GitHub in a secure and simple manner.

GitHub user credentials are high value targets for cybercriminals in today’s security environment, and any system that stores these credentials long term paints an unwanted target on itself. Current solutions — putting an ssh key on the JupyterHub, using a personal access token or deploy keys — involve storing long term valid GitHub credentials in the filesystem. As users can do this by themselves without admin intervention, admins often are not aware these (often unencrypted) credentials are on their filesystems. If an attacker compromises an ssh key or a personal access token, they have unlimited access to all GitHub repos the compromised user had access to, including repos in high-impact GitHub organizations. In the recent credential theft incident, Travis-CI and Heroku were ‘lucky’ in that the attackers accessed npm infrastructure — and since npm is owned by GitHub, GitHub was able to detect that Travis CI and Heroku had compromised credentials. You and the users of repositories you have rights to might not be so lucky. It’s 2022, and supply chain attacks are everywhere — you aren’t special, you’re just one link in a long chain attackers use to get to someone else.

There is a clear need for a simple solution that lets users push to GitHub from JupyterHub in a secure manner without admins having to worry about securing high-value GitHub credentials long term. It is not acceptable to “Just Say No” to users wanting this functionality either — if you try to ‘sacrifice’ usability for security, you end up getting neither.

gh-scoped-creds attempts to solve this problem by allowing users to grant time-limited push access to specific repositories to specific JupyterHub installations in a user friendly way.

Here’s a quick GIF running through the user workflow.

Push access is scoped both by time (credentials expire after 8 hours) as well as repository (access is granted per-repository, per-hub). While you need to refresh credentials every 8 hours, the list of repositories is remembered until you explicitly revoke access. You can always grant access to your own personal repositories, but repositories belonging to organisations might require admins to approve push access to them.

You can also run the command from the terminal as gh-scoped-creds instead of using the IPython magic %ghscopedcreds as shown in the demo. This way, you can also use this from a HPC system, not just a JupyterHub!

Setting this up for your JupyterHub requires a tiny bit of work from the admin — see the project README for more details. Shouldn’t take long, and it’s a one-time task. Once that’s set up, your users can securely push to GitHub from the comfort of their JupyterHubs!

Thanks to Fernando Perez for using his stat159 class at UC Berkeley to test this project out.

Setting up a “Production Ready” TLJH deployment

2021-06-04T17:05:00+00:00

The Littlest JupyterHub is an extremely capable hub distribution that I’d recommend for situations where you expect, on average, under 100 active users.

The Littlest JupyterHub is a distribution of JupyterHub for single VM instances, best-used with 1–100 users.

Why not Kubernetes?

The primary reason to use Zero to JupyterHub on k8s over TLJH in cases with a smaller number of users is to reduce costs — Kubernetes can spin down nodes when not in use. However, you’ll always have at least one node running (for the hub / proxy pods) and the extra complexity that comes with it — particularly around needing to build your own docker images — may not be worth it. TLJH works perfectly well for these cases!

What is ‘production’?

A JupyterHub that you can run securely without lots of intervention from the person who created it is what I’ll call a production-ready JupyterHub. It’s a pretty arbitrary standard. In this blog post, I’ll lay out what I want in the TLJH hubs I run before I let users on them.

Authentication

Use a real Authenticator, not the default FirstUseAuthenticator. The default authenticator is pretty insecure, and should really not be used in production. If you don’t know what to use, I’ll suggest the Google or GitHub authenticators.

Enable HTTPS

Enable HTTPS. An absolute security requirement now, and TLJH makes it quite easy. You do need to get a domain for this to work, which can be a source of friction. Totally worth it, though.

Resource Limits

In many systems, a single user can often write code that accidentally crashes the whole system. By default, TLJH doesn’t have any memory limits enforced per-user, but it is very easy to configure it to enforce memory limits. Tuning these to match your needs will help prevent a single student from accidentally taking down your whole hub. I’d highly recommend checking how much memory your typical notebook uses, and making sure you have user limits set to above that.

Sizing your VM correctly

If you choose a VM that’s too big, you’ll end up spending a lot of cash for unused resources. If it’s too small, your users will not have the resources they need to do their work. TLJH provides some helpful docs estimating your VM size, and you can always resize your VM afterwards if you get it wrong.

Disk backups

TLJH contains everything on the VM’s disk — your user environment, users’ home directories, current hub configuration, etc. It is very important you back this up, to recover in case of disasters. Automated disk snapshots from your cloud provider are an easy way to do this. Most major cloud providers offer a way to do this — Google Cloud, Digital Ocean, AWS, etc. Some let you automate it as well — Google & AWS certainly do, I’m not sure about other cloud providers. This isn’t the best way to do backup — there’s approximately 1 billion ways to do so. However, this is an absolute minimum, and it might just be enough.

If you want to be more fancy, I’d suggest using a separate disk / volume for your user home directories, possibly on ZFS, and snapshot much more aggressively. Talk to your nearest google search bar for your options.

Pin your public IP

Some cloud providers change your VM’s public IP address if you start / stop them. This can be pretty bad — you’ll have to change your domain’s DNS entry, and re-acquire HTTPS. A hassle! You can tell your cloud provider to hang on to your IP even if your VM goes down / changes. And you should! DigitalOcean doesn’t require this, but Google Cloud does. I think AWS does too, but I’m not sure how you can reserve the public IP for it — since it’s usually a domain name itself.

Base environment setup + snapshot

TLJH has a shared conda environment that is used by all users. Everyone can read from it, but only users who are admin can write to it (via sudo). This is one of TLJH’s core design trade-offs - admins can install packages the way they are used to, without requiring a separate image-build step. But it also means the admin can mess it up - conda environments can be sometimes fickle! So it’s not a bad idea to spend some time in the beginning setting everything up - python packages, JupyterLab extensions, etc. Then make a disk snapshot, so you can revert to it if things go bad. This is where having a separate disk for your user home directories comes in handy, so you can reset your hub environment without losing your user home directories.

SSH admin access

The TLJH documentation strives hard to make sure SSH isn’t required for setup and most common usage. However, if your TLJH breaks in certain ways, you can no longer access the machine — since all access is via TLJH! For this, I recommend making sure someone who is admin has SSH access to the VM. Most cloud providers offer a way to set the root ssh key on creation. If not, you can follow the many guides on the internet to making it happen.

You can also just put your ssh keys in $HOME/.ssh/authorized_keys, and ssh in as jupyter-<username>@<hub-ip>. This works for any / all users!

Others?

I’m sure this isn’t the end — probably need something about firewalls, monitoring and automated system package upgrades. But hey, great start!

Connect to a JupyterHub from Visual Studio Code

2019-12-09T17:23:00+00:00

Visual Studio Code has pretty good support
for running Jupyter Notebooks. But what if your organization has a JupyterHub running remotely, with more compute resources & access to large amounts of data? How can you access that from Visual Studio Code running on your local machine?

It’s pretty easy to do, and this blog post will guide you through it.

Step 1: Get a JupyterHub access token

JupyterHub lets you create tokens for yourself for use by third party applications. These tokens can be used anywhere a Jupyter Notebook access token is needed. Since this is what Visual Studio Code needs, let’s acquire one.

Log-in to your JupyterHub
Access your Control Panel. In classic notebook, there is a ‘Control Panel’ button on the top right. In JupyterLab, you can access it under ‘File -> Hub Control Panel’

Top Right ‘Control Panel’ button in classic Jupyter Notebook

File -> Hub Control Panel in JupyterLab

Go to the ‘Token’ page by clicking ‘Token’ in the top bar

Token link in the top bar to go to the token page

Type in a description for the new token you want, and click ‘Request new API Token’

Type in a description for what this token will be used for

Copy your token and keep it somewhere safe. You should treat this like a password to your JupyterHub. You can (and should!) revoke it (as I have done) from the same page when you are no longer using it.

Copy your token

This is all the information you need from JupyterHub! Now let’s go to vscode.

Step 2: Connect VS Code to your JupyterHub

Visual Studio Code supports connecting to a remote notebook server, and we can use that to connect to our JupyterHub. You must perform these steps before opening your notebook.

Open the command palette in Visual Studio Code (‘Cmd+Shift+P’ on MacOS, ‘Ctrl+Shift+P’ elsewhere)
Select ‘Python: Specify local or remote Jupyter server for connections’

vscode command palette

Construct your notebook server URL with the following template: https://<your-hub-url>/user/<your-hub-user-name>/?token=<your-token>. Note that your hub user name might sometimes be escaped from whatever you used to actually log in, if it has special characters. You can verify this by looking at the URL you get once you log in to your JupyterHub — it should have the right one after ‘user’.

Enter your notebook server URL

Create or open a new notebook. The kernel for this should now live on your JupyterHub! You can verify this by running !hostname, which should return the hostname of your remote JupyterHub server instead of your local hostname. You can also try importing libraries that are in the remote JupyterHub server, but not your local file system.

Tada! That wasn’t so hard, was it?

Limitations

Your JupyterHub notebook server must be running already when you try to open your notebook — Visual Studio Code will not automatically start it. If it has stopped, you need to log in to your JupyterHub & start it again. You do not need a new token though.
Watch out for filesystem access. When you are calling open()(or a helper function that eventually access a file) to read a file, it is going to be read from your remote JupyterHub server’s home directory, not your local system’s current directory. So if you add a new file next to your Jupyter Notebook locally, that is not automatically going to be available to the Jupyter Notebook to read.
Installing pip or conda packages locally will have no effect, since your Python kernel is running on your JupyterHub. Use the %pip or %conda magics to install packages in the correct environment.

99 ways to extend the Jupyter ecosystem

2019-06-18T15:55:00+00:00

Whenever someone says ‘You can do that with an extension’ in the Jupyter ecosystem, it is often not clear what kind of extension they are talking about. The Jupyter ecosystem is very modular and extensible, so there are lots of ways to extend it. This blog post aims to provide a quick summary of the most common ways to extend Jupyter, and links to help you explore the extension ecosystem.

JupyterLab extensions (labextension)

Draw vector graphics in JupyterLab with the jupyterlab-drawio extension

JupyterLab is a popular ‘new’ interface for working with Jupyter Notebooks. It is an interactive development environment for working with notebooks, code and data — and hence extremely extensible. Using JupyterLab extensions, you can add entirely new functionality or change almost any aspect of how the interface behaves. These are written in TypeScript or JavaScript, and run in the browser.

The JupyterLab documentation has information on how to install & use extensions, as well as how to author & distribute them. You can also discover extensions by searching on GitHub or npmjs.com

My favorite JupyterLab extension is jupyterlab-vim — it lets you fully use Vim keybindings inside JupyterLab!

Classic Notebook extensions (nbextension)

Table of Contents nbextension

When people think of ‘the notebook interface’, they are probably thinking of the classic Jupyter Notebook. You can extend any aspect of the notebook user experience with nbextensions. These are little bits of client-side JavaScript that allow you to add / change functionality as you wish. They are the Classic Notebook equivalent to JupyterLab extensions.

The Jupyter Notebook documentation has information on how to install or develop extensions. The Unofficial Jupyter Notebook extensions repository has a lot of popular extensions and a GUI extension manager you can use to install nbextensions.

My favorite nbextension provides a collapsible Table of Contents for your notebooks.

Notebook Server Extensions (serverextension)

Unlike JupyterLab or nbextensions, Jupyter Notebook Server extensions are written in Python to add some serverside functionality. There are two primary use cases for server extensions.

LaTeX previews in JupyterLab

The first use case is to provide a backend for a particular JupyterLab or classic notebook extension. An example is the jupyterlab-latex JupyterLab extension, which provides live previews of LaTeX files in JupyterLab. It has a frontend JupyterLab extension to integrate with the JupyterLab text editor, and a backend serverextension component that actually runs the LaTeX commands to produce the output displayed to you.

The second use case is to provide any user interface backed by any kind of server side processing. Server extensions can function as arbitrary Tornado HTTP handlers — so any web application you can think of, you can write as a Jupyter serverextension. An example is nbgitpuller, which provides UI and mechanisms to distribute notebooks from git repositories to your users in a transparent way.

My favorite here is jupyter-rsession-proxy, which lets you run RStudio in JupyterHub environments!

Jupyter Kernels

You might be most familiar with using Jupyter notebooks with Python, but you can use a ton of other languages when writing your notebook: R, Julia, JavaScript, Octave, Scala/Spark, interactive C++, bash, or even Matlab! These are called kernels, and they speak the language agnostic Jupyter protocol over zeromq. You can write a new kernel for your language by directly implementing the Jupyter protocol, by wrapping it with the metakernel project, or using C++ bindings via Xeus. Once a kernel exists, it seamlessly works with any Jupyter frontend — classic notebook, JupyterLab, nteract, the terminal jupyter console, the graphical Qt Console , etc.

My favorite kernel is the linux kernel.

IPython Magics

If you’ve written %matplotlib inlinein a notebook, you have used an IPython magic. These are almost like macros for Python — you can write custom code that parses the rest of the line (or cell), and do whatever it is that you want.

Line magics start with one % symbol and take some action based on the rest of the line. For example, %cd somedirectory changes the current directory of the python process. Cell magics start with %% and operate on the entire cell contents after it. %%timeit is probably the most famous – it’ll run the code a number of times and report stats on how long it takes to run.

You can also build your own magic command that integrates with IPython. For example, the ipython-sql package provides the%%sql magic command for working seamlessly with databases. However, remember that in contrast to the extensions listed so far, IPython magics only work with the IPython kernel.

My favorite use of IPython magics is this blog post by Matthias Bussonnier, which makes great use of custom magics to seamlessly integrate Python, R, C and Julia in the same notebook.

IPython Widgets (ipywidgets)

Play with plot options with dropdown. Courtesy Towards Data Science by Will Koehrsen

IPython Widgets (ipywidgets) provide interactive GUI widgets for Jupyter notebooks and the IPython kernel. They let you and the people you share your notebooks with explore various options in your code with GUI elements rather than having to modify code. Coupled with something like voila, you can make dashboard-like applications for other people to consume without realizing it was created completely with a Jupyter Notebook!

You can build your own custom widgets to provide domain-specific interactive visualizations. For example, you can interactively visualize maps with ipyleaflet, use itk-jupyter-widget to explore image segmentation/registration problems interactively, or model 3D objects with pythreejs.

Check out the vdom project for a more reactive take on the same problem space, and xwidgets for a C++ implementation

Contents Manager

Whenever you open or save a notebook or file through the web interface, a ContentsManager decides what actually happens. By default it loads and saves files from the local filesystem, but a custom contents manager could do whatever it wants. A popular use case it to load/save contents from somewhere other than the local filesystem — Amazon S3 / Google Cloud Storage, PostgreSQL, HDFS, etc. When using one of these, you can load / save notebooks & files via the web interface as if they are on your local filesystem! This is extremely useful if you are already using any of these to store your data.

My favorite contents manager is Jupytext. It does some magic during save/load to give you a .py equivalent of your .ipynb, and keeps them in sync. You can explore code interactively in your notebook, then open the .py file in an IDE to do some heavy text editing, and automatically get all your changes back in your notebook when you open it again. It’s quite magical.

Jupytext: .ipynb or .py? why not both!

Extending JupyterHub

JupyterHub is a multi-user application for spawning notebooks & other interactive web applications, designed for use in classrooms, research labs and companies. These organizations probably have other systems they are using, and JupyterHub needs to integrate strongly with them. Here is a non-exhaustive list of ways JupyterHub can be extended.

Authenticators

JupyterHub is a multi-user application, so users need to log in somehow — ideally the same way they log in to every other application in their organization. The authenticator is responsible for this. Authenticators already exist for many popular authentication services — LDAP, OAuth (Google, GitHub, CILogon, Globus, Okta, Canvas, etc), most LMS with LTI , SAML, JWT, plain usernames & passwords, linux users, etc. You can write your own or customize one that exists very easily, so whatever your authentication needs — JupyterHub has you covered.

Spawners

Using pluggable spawners, you can start a Jupyter Notebook Server for each user in many different ways. You might want them to spawn on a node with docker containers, scale them out with Kubernetes, use it on your HPC cluster, have them run along your Hadoop / Spark cluster, contain them with systemd, simply run them as different linux users or in many other possible ways. The spawners themselves are usually extremely configurable, and of course you can write your own.

Services

Often you want to provide additional services to your JupyterHub users — cull their servers when idle, or allow them to publish shareable notebooks. You can run a JupyterHub service to provide these — or similar — services. Users can make requests to them with their JupyterHub identities, and the services can make API calls to JupyterHub too. These can be arbitrary processes or web services — BinderHub is implemented as a JupyterHub service, for example.

NBConvert Exporter

nbconvert converts between the notebook format and various other formats — if you’ve exported your notebook to PDF, LaTeX, HTML, or used nbviewer, you have used nbconvert. It has an exporter for each format it exports to, and you can write your own to export to a new format — or to just massively customize an existing export format. If you’re performing complex conversion operations involving notebooks, you might find writing an exporter to be the cleanest way to accomplish your goals.

My happiest moment when researching for this blog post is finding out that a docx exporter exists.

Bundler Extensions

Discoverable way to enable nbconvert exporters

Bundler extensions let you add entries to the Download as item in the menu bar. They are often paired with an nbconvert exporter to make the exporter more discoverable, though you can also write a custom bundler extension to do any kind of custom processing of a notebook before downloading. For example, nbreport provides a bundler extension that cleans up the notebook in a way suitable for viewing as a report & exports it as HTML.

Repo2Docker

repo2docker turns git (and other) repositories into reproducible, data science focused docker images. mybinder.org (and other binderhub installations) rely on it to build and launch interactive Jupyter/RStudio sessions from git repositories. There are currently two ways to extend repo2docker.

BuildPacks

repo2docker looks at the contents of the repository to decide how to build it. For example, if there is a requirements.txt it sets up a miniconda environment to install python packages into, while if there is an install.R file it makes sure R/RStudio is installed. Writing a new BuildPack lets you extend this behavior to add support for your favorite language, or customize how an existing language is built.

ContentProviders

The repo part of repo2docker is a misnomer — you can turn anything into a docker image. Currently, it supports git, local folder and zenodo repositories — but you can add support for your favorite source of reproducible code by making a new ContentProvider!

Is that all?

Of course not? The Jupyter ecosystem is vast, and no one blog post can cover them all. This blog post is already missing a few — enterprise gateway, TLJH Plugins, etc. As time marches on, there will be newer components and newer ways of extending things that have not even been imagined yet. Leave a comment about what else is missing here.

Look forward to seeing what kinda beautiful extensions y’all create!

Outreachy & Jupyter: Supporting diversity in open communities

2018-11-28T17:26:00+00:00

Project Jupyter has accepted 2 interns through the Outreachy program, which supports open community members from under-represented backgrounds. Our Outreachy interns will work on important problems in the JupyterHub community with help from 2 mentors from December 2018 through March 2019. This is a short post describing why Jupyter is committing to this program, as well as what we’ll be working on with our Outreachy participants.

We are grateful to the Berkeley Institute for Data Science & NumFocus for jointly sponsoring our interns. This article is cross-posted with the Berkeley Institute for Data Science blog.

Why focus on diversity and inclusion?

The tech industry isn’t doing great on diversity, and the Open Source community is doing worse . This limits the community’s ability to tackle challenging, diverse projects. The amount of responsibility and power held by open source communities is increasing, so we have a responsibility to be more representative of the diversity present in our world. Many projects grow their community by encouraging people to start making small patches on their own. However, requiring uncompensated work is a big barrier to getting more diverse representation in our open source communities. Paid internships with dedicated mentors are a great way to help people break through this particular barrier and make the open source community more diverse.

What is Outreachy?

Outreachy is an internship program coordinated by the Software Freedom Conservancy. It is run twice a year with a goal to bring people from underrepresented backgrounds in tech into open source projects. 21 open source organizations are participating in this round, and will be working with a total of 46–47 interns. Importantly, these are paid internships, which make them more viable for a much broader slice of the population. Jupyter and BIDS will both contribute funding and mentorship for two Outreachy interns.

Our Interns and Mentors

We have two amazing interns for this Outreachy round, working on important projects in the JupyterHub ecosystem. Below we’ll describe the projects that they’ll work on.

A highly Available Proxy for JupyterHub (link)

Georgiana Dolocan will be mentored by Min RK and Yuvi Panda in building a highly available & scalable proxy for JupyterHub using the Traefik project. This will help JupyterHub deployments scale more easily for thousands of active users with minimal service disruptions. Georgiana will be working from Bucharest, Romania where she likes to paint and explore the city and the villages nearby alongside her dog and camera.

Improvements to user management (link)

Leticia Portella will be mentored by Yuvi Panda and Min RK in building better native user management features into JupyterHub. Small to medium installations of JupyterHub that do not want to depend entirely on an external authentication provider will benefit greatly from this. Leticia will be working from Dublin, Ireland (although she used to live in Florianópolis, Brazil), where she likes to read a lot (especially The Chronicles of Ice and Fire), to swim, and to work on her podcast (the first Brazilian podcast specialized in Data Science topics), Pizza de Dados (Data Pizza, free translation).

About our mentors

Outreachy requires more than just funding, but also mentorship. Two members of the JupyterHub community have offered their time to help mentor our Outreachy contributors, some information about them is below!

Min RK is a research engineer at Simula Research Laboratory in Norway and has been working on the IPython and Jupyter open source projects since joining in 2006 as an undergraduate in Engineering Physics at Santa Clara University with Brian Granger, one of the founders of the Jupyter project. Min currently focuses on JupyterHub, and is excited to welcome new folks to the Jupyter team.

Yuvi Panda is an operations engineer at University of California, Berkeley. He works at the Berkeley Institute for Data Science, where he maintains the JupyterHub infrastructure for the Division of Data Sciences. He has been involved in various Open Source communities in the last ten years, spending time in the GNOME, Wikimedia & Jupyter communities. His life was changed drastically by participating as a student in Google Summer of Code 2010, and he’s excited to give back to the Open Source community.

We would like to thank everyone who applied to JupyterHub for this round of Outreachy. We received a number of robust proposals, out of which we were only able to accept two. JupyterHub mentors spent quite a lot of time mentoring candidates during the application period, in reviewing their pull requests, and giving them feedback on their proposals. We look forward to bringing these new open source contributors into our community, and hope that we can set a path that other open source projects may follow in the future. Open source works best when it is diverse and inclusive, we think this is a small step in that direction.

Thanks to Orianna DeMasi, Sara Stoudt, Chris Holdgraf & others for contributing heavily to this blog post.