Jupyter Blog - Min RK

mybinder.org adding request Origin to events archive

2026-02-02T16:27:00+00:00

Binder logo

In February, the mybinder.org service will start tracking the “origin” of requests that launch a Binder session. The website that created a request to mybinder.org will be visible in the Binder analytics archive. This will help us monitor which websites are embedding Binder sessions for impact or abuse reports. It shouldn’t have any implications for private information but we are sharing here for transparency. The only cases where new information will be recorded is websites which embed mybinder.org kernels via tools like Thebe.

When folks use mybinder.org to run sessions, each launch is added to a public events archive. That means that every time anyone launches a session on mybinder.org, the repo and version used and the time of the launch is public information. There is no such thing as a private launch on mybinder.org (this is part of why we don’t support private repos). No information about the user requesting the build is recorded (ip address, user agent, browser info, etc.), though some of that is sent to Plausible and aggregated and anonymized for analytics. Notably, Plausible only covers regular browser visits, not launches via the API.

There are 3 main ways that people launch sessions on mybinder.org:

(the vast majority) — visiting mybinder.org in a browser
embedding kernels from mybinder.org on another website to make e.g. interactive documentation with a tool like Thebe
‘headless’ launches via scripts

Up to now, these events are indistinguishable in the events archive. We’d like to start distinguishing these cases in the events archive to better classify how mybinder.org is used, but that means recording new information we didn’t track before, so we want to communicate clearly about it ahead of time.

We are planning to add the Origin of the launch request to the public events archive at the end of February. This still doesn’t track anything about the user (no ip address, user agent, etc.), but rather the host of the page where the launch request originated. For the vast majority of cases, this is not new information, as the origin is e.g. gesis.mybinder.org (which we already track confusingly as ‘origin’, though this is not the request Origin, it is the federation member where the launch event takes place). Notably, case 1 still includes any regular browser links to mybinder.org/v1/gh/repo/name/HEAD. We don’t record where those come from (that would be Referer).

Where this changes is cross-origin requests (case 2), that is, pages which embed kernels from mybinder.org, such as documentation websites that use mybinder.org to embed kernels from mybinder.org via tools like Thebe to make executable documentation (which is rad!). These requests will now include the Origin of the request, which would be the hostname (e.g. https://course.spacy.io ), not the full page.

Script launches (case 3) typically lack this information, so will usually have null data in the new field. But of course, scripts can do whatever they want, so this isn’t guaranteed to be accurate for malicious or deliberately spoofing scripts.

If you want to have a private, not-recorded use of Binder, you must run your own BinderHub instance. If you want to suggest that we not do this, or do it differently, please let us know in the proposal discussion. You can also check the privacy policy for more information. We plan to apply the change at the end of February, 2026.

Mybinder.org reducing capacity

2023-04-25T07:28:00+00:00

Funding is running out for the largest member of mybinder.org, which has been sponsored by the Google Open Source Program Office. gke.mybinder.org will shut down and there will be a reduction in capacity at the end of April, 2023 (end of this week).

Key points:

Our funds on gke.mybinder.org, the biggest part of the federation, will run out in April. Google has decided not to provide more credits.
Traffic previously sent to GKE will mostly be delivered to OVHcloud, which has less capacity. We are also working to deploy new federation members to spread the load.

Service reductions to expect:

This may come with an increase in errors and failures, as GKE has been the largest and most stable federation member
We are currently looking at reducing the total capacity of Binder by 60%, so you may get rate limit or capacity errors
We will start more aggressively deleting older cached builds, as storage costs have increased.

We are still working out the details of the shutdown and deploying new federation members, but mybinder.org users should expect disruption to the mybinder.org service and to see an increase in messages that the service is at capacity in the short term.

The plan:

Thursday: gke.mybinder.org will be removed from the federation (user sessions not disrupted, but no new sessions will start there)
Friday: gke.mybinder.org will be shutdown, and mybinder.org migrated to a smaller cluster

You can help us keep mybinder.org running by donating to Binder or encouraging your organization to join the BinderHub Federation.

We are grateful to the Google Open Source Program for their funding thus far, and appreciate the continued support of continuing federation members, OVHCloud and GESIS, which enable us to keep mybinder.org going without shutting down completely when one funding source goes away.

Updating defaults on mybinder.org

2023-03-09T08:31:00+00:00

This week, we are deploying some updates to repo2docker, the image-building tool that underpins mybinder.org. This may affect your repos. Please get in touch if you have problems! Hopefully this will make things better, more stable, and less surprising overall, but change always breaks something, however hard we work to keep things working.

To highlight the changes:

Python 3.10 is now the default Python version
Python 3.11 is now supported, and new versions of Python should be supported more quickly
If a repo requests an old version of Python (≤ 3.6), it will be installed in a separate environment, as we’ve been doing with Python 2.7 for a long time.
The server implementation is switched from jupyter-notebook to jupyter-server

More detail on each change, the reasons behind them, and their consequences below.

Python 3.11 and separate envs for legacy Python support

When building an image, repo2docker installs packages in one or two environments — the kernel environment where your code runs, and the server environment, which runs the Jupyter server. If you’re using a reasonably supported version of Python, these are actually the same environment. This is the vast majority of repositories on mybinder.org. In order to function properly, we have certain requirements of what goes in the ‘server’ environment. Eventually, those requirements start making it difficult to setup the server environment with an outdated version of Python, as packages march ahead with their minimum required Python (The widely adopted NEP 29 suggested dropping Python 3.7 support in December, 2021). Our solution for this has been to separate the kernel environment from the server environment, so they don’t create a conflict. Until recently, only Python 2 was handled this way (most packages dropped Python 2 support years ago). As part of adding support for Python 3.11, we have expanded this behavior to include Python ≤3.6, so if you are using a ‘legacy Python’, currently defined to be Python ≤3.6, your requested packages will be installed in the kernel environment instead of the server environment. This should only have a noticeable effect if you are installing something with server extension components , which includes anything modifying the JupyterLab or notebook interfaces— those will not be available in the server environment. To address this, you will have to upgrade Python or handle the server extension installation in postBuild . PR adding Python 3.11 and adding Python ≤3.6 to legacy Python versions.

Default Python is now 3.10

The default Python for repo2docker and thereby mybinder.org is now Python 3.10, upgraded from 3.7. 3.10 will be used when a repository does not specify which version of Python to use. Following Python’s own annual release cadence, we plan to update this more regularly in the future, lagging behind Python itself by a version or two. This has no effect on your repos if they specify a Python version, which is why we encourage you to specify your Python version! If you don’t, then your Python version will change over time, probably about once a year. PR switching the default Python.

Now launching Jupyter Server

The default UI for mybinder.org has been JupyterLab for some time, with the classic notebook still available if you specify a url of /tree/or /notebooks/mynb.ipynb. When we rolled out that change, we kept the default server as the jupyter-notebook server, for improved compatibility with existing server extensions. As the newer Jupyter Server has taken over, the situation is becoming reversed — using the older notebook server is now the cause of compatibility issues with server extensions, so we are switching the server to launch jupyter-server instead of jupyter-notebook. The classic UI is still available at the same URLs, thanks to nbclassic. This should have no visible effects, as jupyter-server and nbclassic make a great deal of effort to keep classic notebook extensions working, but some extensions may need an update for jupyter-server compatibility if they haven’t been updated already. Because of where the server is specified, there is no supported way for images to express a preference for the deprecated notebook server (PR switching the default server command.

Please don’t disable authentication in Jupyter servers

2022-04-05T09:50:00+00:00

We are aware of ransomware attacks specifically targeting Jupyter servers.

First off: we have no evidence of a vulnerability in Jupyter associated with the attack. The attacks appear to focus on Jupyter servers with authentication disabled. That’s what this post is about: please don’t do that!

Jupyter servers are authenticated with a random token by default, so you should be fine with the defaults. That default random token is regenerated on each server launch. If that random token is inconvenient for you (there are plenty of situations where this is the case), the answer is not to disable auth! More on that below.

We wrote about this before, but it bears repeating:

Jupyter exposes arbitrary code execution and a shell. If folks gain access, they can do anything you can do. You aren’t just protecting your notebooks, you are protecting your whole computer.
People can find your notebook server if it’s accessible on the Internet. If they locate your server and it has auth disabled, that’s game over, and appears to be what happened in the above ransomware event. Jupyter’s popular enough (yay) that folks are always looking (boo).
If the generated token pattern is inconvenient for how you work, you can set a persistent token or password in configuration instead of retrieving the generated token on each server launch. Save this token or password in your password manager for safe keeping, and accessing your server will be just as convenient as auth being disabled.

If you have an interactive prompt, you can run jupyter server password

$ jupyter server password
Enter password:🔐
Verify password:🔐
[JupyterPasswordApp] Wrote hashed password to ~/.jupyter/jupyter_server_config.json

which generates and stores config that looks like:

{
  "ServerApp": {
    "password": "argon2:$argon2id$v=19$m=10240,t=10,p=8$aFE/DuLj//6oGF2PHWy2DQ$eXHn6AbJe8Lryl4z9oGvMtZhX7iEdt41m+mhvyDvw88"
  }
}

This is a hashed, salted form of your password. We can’t get the password out of this, but we can use it to check if you typed it correctly on the login page. If an interactive prompt isn’t available to you (e.g. a cloud vm that starts the notebook server automatically), you can run this command on any machine, and copy the resulting config file to the destination machine as part of setup.

If it works better for you, you can also generate just the hashed password:

from jupyter_server.auth import passwd
hashed_password = passwd("my-great-passphrase") # just the "argon2:..."

and get that into your config files, as best suits your needs. For example, in ~/.jupyter/jupyter_server_config.py:

c.ServerApp.password = "argon2:..."

or the same config in ~/.jupyter/jupyter_server_config.json

{
  "ServerApp": {
    "password": "argon2:..."
  }
}

Read more details in our earlier post, and security documentation.

Please contact security@ipython.org if you have a vulnerability to report.

JupyterHub 2.0

2021-12-01T14:15:00+00:00

Hubs and roles and scopes, oh my!

We’ve just published JupyterHub 2.0, our biggest release in a while.

Roles and scopes (RBAC)

Thanks to the work of Omar Richardson and Ivana Huskova and more contributors along the way, JupyterHub now has fine-grained control of permissions, via scopes (a role is a named collection of scopes).

This makes JupyterHub more secure because you no longer need to give every user or service full admin permissions if they need to take one elevated action (such as a cull-idle service). Now, you can give users and services exactly the permissions they actually need. admin still exists, but nothing should need to be an admin anymore. Fine-grained scopes also allow new configurations that weren’t previously available, such as granting instructors permission to access their students’ servers, which is useful with JupyterLab’s new real-time collaboration features, without needing to grant them full admin permissions, or even access to servers of users not in their specific class.

The significant change in how permissions work means that upgrading to 2.0 requires some downtime. You will need to shut down all services and single-user servers across the upgrade, and make sure you upgrade JupyterHub to 2.0 in both the user environment and the Hub environment at the same time.

All tokens issued prior to the 2.0 upgrade will be revoked.

As always, back up your database before upgrading!

Read more about roles and scopes in the docs.

JupyterLab by default

2.0 also marks JupyterHub’s transition to using JupyterLab as the default UI. You can stick with the classic notebook server with:

c.Spawner.environment = {
    "JUPYTERHUB_SINGLEUSER_APP": "notebook.notebookapp.NotebookApp",
}

or try RetroLab, a classic-style UI built on the newer Jupyter Server:

c.Spawner.default_url = "/retro/"

(assuming you’ve installed it in the user environment, with e.g. pip install retrolab).

And more

There are various other changes and improvements in 2.0, including a new admin UI, API pagination, and no longer relying on any command-line arguments for single-user servers. Check out the changelog for more details.

We encourage you to take 2.0 for a spin, try out the roles and scopes, and let us know how it goes.

Thanks to all the contributors who helped us with 2.0, especially prerelease testers!

IPython Parallel in 2021

2021-11-30T07:55:00+00:00

This post describes work funded by Bodo, Inc.

IPython Parallel’s a bit of an odd duck in the parallel computing space. In a world with dask, ray, bodo, pyspark, and other parallel computing tools, what is IPython Parallel’s role in 2021?

Context

IPython Parallel began (in 2006!) as a natural extension of the Jupyter messaging protocol(though it predates the Jupyter name by a few years): when you have a protocol for REPL-style remote code execution, what can you do with multiple remote execution environments? This question led to the development two basic models for parallel execution in IPython Parallel:

“multiplexed execution” — where you send code explicitly to different workers, and
“load-balanced execution” — where you tell IPython Parallel what to run, and a scheduler takes care of assigning each task to an available worker.

What IPython Parallel doesn’t do

To IPython, your tasks are black boxes (either Python functions or blocks of Python code as text) and you are in complete control of where and when your tasks run, as well as any dependencies or side effects they may have.

IPython Parallel specifically doesn’t and won’t do several things that other tools might:

infer dependencies or relationships between tasks (execution graph)
manage data distribution or locality
construct parallel algorithms or represent ‘natively’ parallel distributed data structures, such as distributed arrays or data frames

Everything is very explicit and implemented at the client-level in IPython Parallel, meaning that while it doesn’t do these things, you can. It also means that many of the things that have overhead costs associated with parallel computing (communication, data movement) have particularly high costs in IPython Parallel.

IPython Parallel doesn’t hide anything from you, for better and worse.

What IPython Parallel does

IPython Parallel makes explicit parallel computations interactive.

IPython Parallel is not just parallel Python, it’s parallel IPython. That means you have all the power of IPython’s inspection, interactivity, magics, and debugging available on all of your distributed workers. This makes it especially well-suited to prototyping and experimentation.

IPython Parallel also presents standard APIs such as Python Executors, compatible with many other implementations, to make it easy to migrate to and from IPython Parallel, enabling developers to write code that uses a single multicore laptop or a thousand cores on an HPC cluster or cloud.

IPython Parallel in 2021

Interactive progress across parallel engines. Those progress bars are interactive widgets running locally, and on each remote engine!

For a lot of today’s workloads, my default recommendation is: use dask or bodo or another modern tool. IPython even makes this easier if you already happen to have an IPython Parallel cluster, you can tell it to “become dask,” and off you go:

dask_client = rc.become_dask(ncores=1)

However, where IPython Parallel can shine, is making traditional SPMD (e.g., MPI) workloads interactive, especially for prototyping and debugging.

If you have an MPI simulation and you wish you could pause it in the middle, poke around and make plots and interact with just one node or all of them with all the interactive tools available to you in Jupyter, IPython Parallel may be the tool for you.

Recent developments

Because I see the main problem IPython Parallel solves well is adding interactivity to direct parallel execution, the focus of recent developments has been on improving that story, based on feedback from users who are often in traditional HPC environments like SLURM or PBS, or using MPI in the cloud.

Much of the feedback on challenges over the years have been around:

Scaling to larger numbers of engines (1,–10,000)
Better feedback and recovery when things go wrong
Security requirements

Thanks to Bodo, I’ve been able to spend a lot of time this year addressing several of those issues, now available as IPython Parallel 8.0.

Last year, Tom-Olav Bøyum developed a broadcast scheduler as part of his Master’s thesis, which vastly improves the efficiency of sending the same task to all engines (the main thing we do in IPP+MPI). This work is now mostly done and landed in IPython Parallel 7. We’ve also addressed longstanding issues with registering large numbers of engines (5–10,000) at once, a common source of failure on HPC clusters.

One of the most frustrating / challenging things when working with MPI or other parallel computations is dealing with hangs and restarting tasks. Through the new Cluster API, IPython Parallel now has one of its most requested features: the ability to send signals to one or all engines, and forcefully restart the cluster if it’s stuck.

We’ve also improved interactive feedback with progress bars, live streaming output, a newJupyterLab plugin based on dask-labextension, and support for the Jupyter widget protocol, so you can instantiate widgets on engines and interact with them directly from a notebook.

Security questions have also been raised, because IPython (and Jupyter’s) use of ZeroMQ assumes a ‘trusted network’, typically localhost or BSD sockets for notebooks. Since IPython Parallel usually runs across a network, security can be more of a concern (whereas notebooks typically only use standard HTTPS connections over a network), and the level of security in the Jupyter protocol alone may not be sufficient. In the past, we’ve resorted to tunneling TCP over SSH for more secure network traffic, but still always implicitly trusting localhost.

IPython Parallel 7.1 enables CurveZMQ for full authentication, encryption, and forward-secrecy at the transport level, solving a longstanding security shortcoming of IPython Parallel.

IPython Parallel in the future

Scaling is the biggest challenge for IPython Parallel due to its communication model. However, that scaling is on the order of IPython engines, which is not the same as the number of cores. You can benefit from this today if your tasks are already multi-threaded, e.g., through OpenMP threads in numpy, or your own Python threads or multiprocess breakdown of tasks.

Better support of multi-level parallelism, where each “engine” may represent a multicore node, as individual nodes get bigger and bigger would mean moving the bar where IPython Parallel communication is the bottleneck back two orders of magnitude on large machines, because one ‘engine’ could represent 128 cores or more. Good support for 1000 128-core nodes would give IPython Parallel some pretty comfortable headroom for our target use cases, scale-wise.

Also related to scaling, bringing the BroadcastView to maturity should allow us to completely replace the DirectView scheduler, as it should be able to match or beat it in almost every scenario with some further development. There are some features lacking, especially when it comes to error handling, but those can certainly be addressed in time.

Finally, I’ll invite you to get involved. Especially if you are interested in prototyping parallel code, or making traditional MPI-style code interactive, check out IPython Parallel 8 and let us know how it goes, or contribute your use case as an example.

pip install --upgrade ipyparallel

nbviewer has a new host: OVHcloud

2020-03-06T19:46:00+00:00

For several years, nbviewer has been generously hosted by Rackspace. That sponsorship program appears to be ending, so nbviewer needed a new home; it has found one in OVHcloud. We are extremely grateful to OVHcloud for their support in keeping nbviewer running, building on their existing participation in the Binder Federation (literally—nbviewer is now running on the same kubernetes cluster as ovh.mybinder.org).

How we moved

Where were we before?

nbviewer was previously deployed using a private repo (because it contained credentials) and various commands using invoke. It was a mixture of custom steps, using the openstack Python API and docker machine to:

allocate two VMs
build docker images
deploy two nbviewer instances per node
deploy memcached via nbcache on each node
deploy statuspage publisher as a separate step
update fastly to point to the running nbviewer instances

The upside was that we had a repo that could immediately deploy nbviewer to anywhere with docker. With this repo, we moved our deployment strategy from a CoreOS cluster to Rackspace’s short-lived Carina service to deploying VMs ourselves with docker-machine. Migrating to a new source of VMs would not have been hard, but it wouldn’t have solved any of our challenges.

Known downsides of this deployment that we’ve experienced over the years:

only a few folks ever knew how to use it and could thus deploy updates to nbviewer
it was private, contributing to above. It’s hard to onboard folks in an open community to a private repo!
independent machines meant cache was not shared (minor, but contributes to our consumption of the GitHub API rate limit)
no automatic recovery based on health monitoring, so when a container had issues, some humans got automated emails but no action was automatically taken. Extra frustrating because the fix was ~always to restart the container.

The credentials used by this repo have been revoked and an archived version of the repo is available (with credentials redacted from history) in the new, public nbviewer.org-deploy repo.

What have we learned?

We’ve learned a lot about open, automatic, and sustainable deployments, and can now comfortably address all of the downsides above. Most of this has been learned from the communities participating in the JupyterHub and Binder projects, as seen in the mybinder.org-deploy repo. mybinder.org-deploy is a public repo that automatically deploys and tests updates to at least four different Kubernetes clusters at the push of a button (the Big Green Merge Button, to be precise). Some things we have learned in the years since we set up our nbviewer deployment:

kubernetes and helm are great :)
git-crypt allows us to have public repos with some secret contents, so deployment repos don’t need to be fully private just to protect a couple api keys.
Continuous Deployment via services like Travis or Circle lowers the bar to adding maintainers on a given deployment since all that’s needed is to press the Big Green Button.

The OVH sponsorship came in the form of a slice of a kubernetes cluster; that meant we had to migrate the nbviewer deployment tools from using docker-machine to kubernetes, which for us means helm.

Step 1: helm chart for nbviewer

The first step was to create a helm chart for nbviewer, which is done here. Before, nbviewer was two docker containers running nbviewer and one running memcached per server. To turn this into a helm chart we need:

dependency on memcached helm chart for easy deployment of the cache. All the nbviewer instances will talk to this memcache, which should improve our GitHub rate limit consumption, since the cache will be shared. This means the deprecation of our own nbcache repo.
Deployment for nbviewer. This is (one of) the kubernetes wrappers around containers. It has a nice ‘replicas’ field to easily scale nbviewer up and down. We are currently running with 3 replicas.
Service to expose nbviewer to the Internet (likely also need an ingress in the future)
Deployment for the statuspage publisher, which updates https://status.jupyter.org with the remaining github rate limit available

We also needed helm charts and docker images for some other components, such as cdn.jupyter.org and the nbviewer statuspage data source.

Step 1b: helm chart for cdn.jupyter.org

nbviewer and some other services interact with cdn.jupyter.org, which is a lightweight nginx configuration to serve the classic notebook’s javascript and css as static files(this is no longer needed for npm-based jupyterlab, which can use unpkg as a CDN). This used to run on a multipurpose server VM, but that is also being retired for the same reason. The result was creating a docker image and helm chart for serving the contents of cdn.jupyter.org. The scripts used to run the existing CDN were already in a repo, so it was a small amount of work to adapt this to run in a container instead of on a server. We may retire cdn.jupyter.org in the future, so please don’t rely on it :)

This is where we are right now — nbviewer.jupyter.org and cdn.jupyter.org are being served by OVH and the Rackspace machines are being retired.

Step 2: automatic deployment

It would have been ideal for this to be step 1, but that’s now how it happened. Sometimes you need to get it done quickly before you get it done right. The task here will be to make a new, public nbviewer-deploy repo following the patterns we have learned in mybinder.org-deploy. That will mean:

create public nbviewer.org-deploy repo with configuration and git-crypt encrypted secrets for deploying nbviewer on the ovh cluster (done)
adopting chartpress to publish our helm charts for nbviewer and version-tagged images from the nbviewer repo
configure Travis-CI or other CI service to automatically deploy updates with helm, so merging a PR is all we need to do to deploy updates to nbviewer

Thanks to OVHCloud for their continued support of Jupyter and Binder, and the JupyterHub and Binder communities for teaching me how to operate services in the open with kubernetes and helm.

Announcing JupyterHub 1.0

2019-05-04T03:58:00+00:00

Today, we are pleased to announce the 1.0 release of JupyterHub. We’ve come a long way since our first release in March, 2015. There are loads of new features and improvements covered in the changelog, but we’ll cover a few of the highlights here.

You can upgrade jupyterhub with conda or pip:

(Before upgrading, always make sure to backup your database!)

conda install -c conda-forge jupyterhub==1.0.*
pip install --upgrade jupyterhub==1.0.*

UI for named servers

Named servers are a concept in JupyterHub that allows each user to have more than one server (e.g. a ‘gpu’ server with access to gpus, or a ‘cs284’ server with the necessary resources for a given class). JupyterHub 1.0 introduces UI for managing these servers, so users can create/start/stop/delete their servers from the hub home page:

UI for managing named servers

Improved pending spawn handling and less implicit spawn behavior

When users launch their server, they will be faced with a progress bar showing the progress. Spawners can emit custom messages to indicate the stages of launch, which are especially useful when it can take a while.

a spawn taking a bit

Support for total internal encryption with SSL

If your JupyterHub is publicly accessible, we hope you are using HTTPS (it’s never been easier, thanks to letsencrypt)! However, JupyterHub exclusively used HTTP for internal communication. This is usually fine, especially for single-machines, but for deployments on distributed or shared infrastructure it’s a good idea to encrypt communication between your components. With 1.0, you can enable SSL encryption and authentication of all internal communication as well. Spawners must support this feature by defining Spawner.move_certs. Currently, local spawners and DockerSpawner support internal ssl.

Support for checking and refreshing authentication

JupyterHub authentication is most often managed by an external authority, e.g. GitHub OAuth. Auth state can be used to persist credentials, such as enabling push access to repositories or access to resources. These credentials can sometimes expire or need refreshing. Until now, expiring or refreshing authentication was not well supported by JupyterHub. 1.0 introduces new configuration to refresh or expire authentication info:

c.Authenticator.auth_refresh_age allows authentication to expire after a number of seconds.

c.Authenticator.refresh_pre_spawn forces a refresh of authentication prior to spawning a server, effectively requiring a user to have up-to-date authentication when they start their server, which can be important when the user environment should have credentials with access to external resources.

Authenticator.refresh_auth defines what it means to refresh authentication (default is nothing, but can check with external authorities to re-load info such as tokens or group membership) and can be customized by Authenticator implementations.

Thanks!

Huge thanks to the many people who have contributed to this release, whether it was through discussion, testing, documentation, or development.

Open Redirect Vulnerability in Jupyter notebook, JupyterHub

2019-03-28T14:03:00+00:00

Update: notebook 5.7.7 and JupyterHub 0.9.5 contained incomplete fixes for this issue. 5.7.8 and 0.9.6 are released with more complete fixes.

JupyterHub ≤ 0.9.5 and Notebook ≤ 5.7.8 are affected. Versions 0.9.6 and 5.7.8 are released with fixes. Upgrade with pip or conda:

pip install --upgrade 'notebook>=5.7.8'
pip install --upgrade 'jupyterhub>=0.9.6'

conda install -c conda-forge 'notebook>=5.7.8'
conda install -c conda-forge 'jupyterhub>=0.9.6'

notebook 5.7.8 also fixes some compatibility issues and regressions introduced in 5.7.5 and 5.7.6, especially on some Windows systems. If you use the jupyterhub helm chart, version 0.8.2 upgrades JupyterHub to 0.9.6.

What is an Open Redirect Vulnerability?

Login pages tend to take a parameter for redirecting back to a page after successful login, e.g. /login?next=/notebooks/mynotebook.ipynb, so that you aren’t disrupted too much if you try to visit a page, but have to authenticate first. An Open Redirect Vulnerability is when a malicious person crafts a link pointing to the login page of a trusted site, but setting the “redirect after successful login” parameter to send the user to their own site, instead of a page on the authenticated site (the notebook or JupyterHub server), e.g. /login?next=http://badwebsite.biz. This doesn’t necessarily compromise anything immediately, but it enables phishing if users don’t notice that the domain has changed, e.g. by showing a fake “re-enter your password” page. Servers generally have to validate the redirect URL to avoid this. Both JupyterHub and Notebook already do this, but the validation didn’t take into account all possible ways to redirect to other sites, so some malicious URLs could still be crafted to redirect away from the server (the above example does not work in any recent version of either package). Only certain browsers (Chrome and Firefox, not Safari) could be redirected from the JupyterHub login page, but all browsers could be redirected away from a standalone notebook server.

This vulnerability has been assigned CVE-2019–10255. This post will be updated when it has been assigned. Thanks to Ronak Odhaviya for reporting the issue. You can report security issues by emailing ipython-security@googlegroups.com.

Jupyter notebook XSSI security fix

2019-03-09T20:46:00+00:00

We have just released Jupyter notebook 5.7.6 with a security fix for a cross-site inclusion (XSSI) vulnerability, where content from a Jupyter server could be included in another page if the visitor is logged in to the Jupyter server and the author of the page knows the URL of the server and the path within the server’s notebook directory that they would like to include. Further, it has been demonstrated with the Internet Explorer browser that some content from the accessed file can be retrieved by the attacking page. This has not yet been demonstrated with other browsers, however.

To upgrade:

pip install --upgrade 'notebook>=5.7.6'

or conda:

conda install 'notebook>=5.7.6'

This vulnerability was reported by the HackerOne hacker Abhishek Bundela, via Jonathan Kamens at Quantopian. The patch was developed by Min Ragan-Kelley with help from Devdatta Akhawe. Jupyter is grateful to Quantopian’s bug bounty program, which has found and reported this and other security flaws in Jupyter.

This vulnerability has been assigned CVE-2019–9644.

Zero to JupyterHub helm chart 0.8

2019-02-21T18:34:00+00:00

We’ve just released version 0.8 of the jupyterhub helm chart.

For those who may not know, Zero to JupyterHub is a guide and helm chart for deploying JupyterHub on Kubernetes. Helm is like a package manager for Kubernetes, which aims to make it easy to install and manage applications on your cluster.

There are loads of bugfixes and improvements to this release, and we encourage you to check out the change log. Here, we’ll discuss two highlights from the new features: profiles and autoscaling.

Profiles

By adopting the profiles pattern established in WrapSpawner, Kubernetes users can now pick from a selection of “profiles” or pre-set configurations, defined by the operator of a cluster. For example, defining small, medium, and large resource requests, or special profiles for GPUs.

The above sample is created from the following snippet in your values.yaml:

singleuser:
  profileList:
    - display_name: "Small: default"
      description: |
        A small job. 1 CPU, no GPU. This is the default.
      default: True
      kubespawner_override:
        cpu_limit: 1
        cpu_guarantee: 1
        mem_limit: "1G"
        mem_guarantee: "512M"
        extra_resource_limits: {}
    - display_name: "Big: 8 CPUs"
      description: |
        A big job. 8 CPUs, no GPU, and 64GB of RAM
      kubespawner_override:
        cpu_limit: 8
        cpu_guarantee: 8
        mem_limit: "64G"
        mem_guaranttee: "64G"
        extra_resource_limits: {}
    - display_name: "GPU job"
      description: |
        This configuration gives you 4 CPUs, 16GB of RAM, and a GPU
      kubespawner_override:
        image: consideratio/singleuser-gpu:v0.3.0
        cpu_limit: 4
        cpu_guarantee: 4
        mem_limit: "16G"
        mem_guarantee: "16G"
        extra_resource_limits:
          nvidia.com/gpu: "1"

Autoscaling

Autoscaling is a big focus of this release, and many of the new features in 0.8 of the chart relate to scaling in some way. At the center is a new ‘user scheduler’ a custom kubernetes scheduler responsible for assigning user pods to nodes. To enable the user scheduler:

scheduling:
  userScheduler:
    enabled: true

There are two major pain-points for autoscaling with JupyterHub. First, is scaling up. Kubernetes scale-up is very basic: If scheduling a pod fails due insufficient resources, a new node is requested. This isn’t a great fit for JupyterHub. If you happen to be the unlucky user who is the first to need a new node, launching your server can take several minutes as the new node is requested and warming up. To prevent this, the chart adds the notion of “placeholder pods,” which are pods that request the same resources as user pods, but have lower priority and will thus be evicted immediately if a user pod needs their resources. This allows a cluster to always have a certain amount of “headroom” of vacant slots available for users so that requesting a new node occurs not when there are 0 slots available for users, but instead when there are 5 or 10 (you choose) slots available. See this exploration of how mybinder.org arrived at 25 placeholders for its deployment. To enable 10 user placeholders:

scheduling:
  podPriority:
    enabled: true
  userPlaceholder:
    enabled: true
    replicas: 10

The second major improvement to autoscaling that comes with the user scheduler is in scaling down. If your cluster is automatically scaling up, you probably want it to scale down as well when you no longer need the added capacity. Unfortunately, because user pods cannot be kicked out, Kubernetes’ default scheduler doesn’t manage to drain nodes that are no longer needed without some manual intervention. The user scheduler adds this feature, ensuring that, over time, your newest nodes eventually drain when they are no longer needed. It accomplishes this by prioritizing the node with the most other user pods when picking a node to start new users. If there is an extra node, over time eventually all the users on that node will stop and the node will become idle and culled by the cluster autoscaler. This is the default behavior . There’s loads more scheduling optimizations available in the updated documentation, including dedicating certain nodes to users so that user pods never run on the same node as others, etc.

And finally, a huge thanks to the numerous contributors who helped make this release, be it via documentation contributions, testing, discussion, or code.

Check it out

Jupyter Notebook security fixes

2018-11-19T22:20:00+00:00

Two security issues have been found and fixed this week, where untrusted javascript could be executed if malicious files could be delivered to the users system and the user takes specific actions with those malicious files.

The first allowed nbconvert endpoints (such as Print Preview) to render untrusted HTML and javascript with access to the notebook server. This is fixed in notebook 5.7.1. All notebook versions prior to 5.7.1 are affected. Thanks to Jonathan Kamens of Quantopian for reporting. This issue has been assigned CVE-2018–19351.

The second issue allowed maliciously crafted directory names to execute javascript when opened in the tree view. This is fixed in notebook 5.7.2. All versions of notebook from 5.3.0 to 5.7.1 are affected. Thanks to Marvin Solano Quesada for reporting. This issue has been assigned CVE-2018–19352.

You can check your version of the notebook package by issuing the following command:

jupyter notebook --version

Whether you are using classic notebook, JupyterLab or any other notebook server extensions, we recommend that you update the notebook package with :

pip install --upgrade notebook

or if you are using conda-forge

conda upgrade notebook

Thanks especially to Jonathan and Marvin for reporting these issues! If you find a security issue in a Jupyter project, please report it to security@ipython.org.

Security fix for Jupyter Notebook

2018-07-18T04:31:00+00:00

We have just released Jupyter Notebook 5.6.0. This release fixes a vulnerability that could allow a maliciously crafted notebook to execute JavaScript when it is opened, bypassing the trusted-notebook mechanism.

We recommend updating the notebook immediately, via pip:

pip install notebook>=5.6.0

or conda:

conda install notebook>=5.6.0

Affected versions: all releases prior to 5.6.0

JupyterLab users are affected, independent of the version of JupyterLab itself. Upgrading the notebook package to 5.6.0 resolves the issue for users of both JupyterLab and the classic notebook.

A CVE has been requested for the vulnerability. Release notes for 5.6.0 and this post will be updated as the CVE is assigned. More details of the vulnerability will be released in 30 days, on August 16, 2018.

Security reports for Jupyter are greatly appreciated. You can report security issues to security@ipython.org.

Thanks to Jonathan Kamens for reporting this issue to the security list.

[Update July 28] The vulnerability have been assigned number CVE-2018–1999024.

[Update August 16] MathJax versions prior to version 2.7.4 contains a Cross Site Scripting (XSS) vulnerability in the \unicode{} macro that can result in potentially untrusted Javascript running within a web browser, for example at notebook load. Notebook 5.6.0 ships with an updated mathjax version 2.7.4 which fixes this vulnerability.

JupyterHub 0.9

2018-06-18T13:18:00+00:00

We are pleased to announce the latest release of JupyterHub. JupyterHub is the multi-user server for Jupyter notebooks, allowing students or researchers to have their own workspace. This release has lots of improvements, especially for stability and performance with large numbers of users.

The biggest change is probably the adoption of asyncio coroutines throughout instead of tornado coroutines and improved support for asyncio coroutines in general, adopting async def coroutine syntax. This means that JupyterHub 0.9 requires Python ≥ 3.5 and tornado ≥ 5.0.

There are lots of improvements to the REST API, including more detailed time information and token management.

There are also several improvements to the ability to customize JupyterHub’s HTML templates.

See the changelog for a more detailed list of changes.

Make sure to backup your jupyterhub database prior to upgrade, and then upgrade with pip:

python3 -m pip install --upgrade 'jupyterhub==0.9.*'

or conda:

conda install -c conda-forge jupyterhub=0.9

For Kubernetes users, we will have an 0.7 release of the helm chart before too long.

Thanks to everyone who has contributed to JupyterHub!

Security fix for JupyterHub GitLab OAuthenticator Group Whitelists

2018-02-16T22:12:00+00:00

This vulnerability has been assigned CVE-2018–7206

If you are using JupyterHub with the GitLab OAuthenticator and its gitlab_group_whitelist support, there is a security issue where the authenticator will allow users outside your intended group whitelist to create accounts. A fix has been released as OAuthenticator 0.6.2 and 0.7.3. No other authentication mechanism, including GitLabOAuthenticator without using the group whitelist feature, is affected. If you are using GitLab authentication with group whitelist support, upgrade oauthenticator immediately:

python3 -m pip install --upgrade oauthenticator

Thanks to Joseph Weston for reporting the issue and providing the fix.

Timeline (all times UTC):

2018–02–16 09:51 Joseph Weston reports security issue to the Jupyter security list

2018–02–16 16:08 Fix is verified and applied to oauthenticator master

2018–02–16 21:52 oauthenticator 0.7.3 and 0.6.2 are released with the fix

2018–02–18 03:02 CVE-2018–7206 assigned

Public Notebooks and Security

2017-11-28T19:33:00+00:00

tl;dr: don’t disable notebook authentication!

Let’s chat a little bit about public Jupyter notebook servers and security. A lot of folks run notebook servers that are publicly accessible, whether they are the access point to a cloud server or remote access to their home or office computer. This is great! It is important to keep in mind that, since arbitrary code execution is the main point of IPython and Jupyter, running a publicly accessible server without authentication or encryption is a very bad idea indeed. Prior to notebook 4.3, this was the default behavior of Jupyter, which has led to a lot of insecure notebook servers running on cloud services. One aspect that may not be apparent is that people can find your public notebook server, especially if it is on one of the popular cloud providers. Systematic port scans will turn up notebook servers, especially for those running on the default port of 8888. And Jupyter has become popular enough (🎉) that we know people are doing this (😠). Running a publicly accessible notebook server without authentication is making your computing resources free for the world to use.

The highlights to keep in mind:

Always enable a password or token for publicly accessible notebook servers (this includes shared computers)
Use HTTPS for authenticated notebooks over the Internet
Unsecured public notebook servers are discoverable
Running on a cloud service should be treated as if it’s publicly accessible

The first two points shouldn’t be new information, and anyone following the documentation should be adding basic authentication when exposing a server to the public. If you are already accessing a notebook server over https with a password or token, you have nothing to worry about. However, if you are running unsecured notebook servers on the Internet, please stop and enable a password. The latter points are important and may not be obvious, but the gist is: don’t disable notebook authentication, even if you think your bases are covered via other means.

In notebook 4.3, a token is enabled by default, making it a little bit more inconvenient to run a totally insecure notebook server. We can’t feasibly require HTTPS by default, but free, trusted HTTPS has never been easier, thanks to LetsEncrypt. If you can’t assign a domain name to your server, self-signed certificates can be used, with some caveats (notably: Safari requires self-signed certificates be trusted at the system level before it will allow websocket connections). For shared deployments, you can also use JupyterHub, which allows you to authenticate notebooks via the system (PAM) or OAuth providers such as GitHub, Google, etc.

Be safe out there.

JupyterHub 0.8

2017-10-03T19:42:00+00:00

We are pleased to announce the release of JupyterHub 0.8. This is a big release with many fixes and improvements, and some major changes.

To upgrade from jupyterhub 0.7:

Stop jupyterhub
pip3 install --upgrade jupyterhub==0.8
Backup your database: e.g.cp -v jupyterhub.sqlite jupyterhub-backup-$(date +%Y-%m-%d).sqlite
Upgrade your database: jupyterhub upgrade-db

See the changelog for more details. The “Notes on Upgrading” below also provides more specifics.

OAuth

Perhaps the biggest change is the switch to using OAuth 2.0 for the internal authentication mechanism in JupyterHub. This shouldn’t have much of a visible effect on your deployments other than hopefully a reduction in redirect-loops when users login.

If you are writing a JupyterHub Service, then you can switch to OAuth by using the HubOAuthenticated or HubOAuth classes if you were previously using HubAuthenticated or HubAuth. All of these classes now support token-based access via the Authorization header, just like the Hub itself.

Stability, Scalability, and Performance

As part of supporting larger deployments, we have done some scalability stress testing of the Hub, finding and fixing several race conditions that could manifest as redirect loops, bugs, and performance improvements when the Hub is under load. This has been led largely by the Berkeley Data Science Education Program team, which has over one thousand students using a JupyterHub instance this Fall. We now know that one JupyterHub 0.8 instance can support a few thousand users logging in and attempting to spawn their servers at the same time. Once servers are active, the Hub is minimally involved, and the proxy becomes the next bottleneck.

Custom Proxy Implementations

Another major new feature targeted at scalability is custom proxy implementations. JupyterHub has always used Configurable-HTTP-Proxy (CHP), a single-process Node.js HTTP proxy. This is a single process bottleneck and potential single point of failure for Hub deployments. CHP can handle a lot of concurrent active users (at least several thousand), but for larger scale applications, Hub deployments may want to use a more scalable and/or robust proxy implementation.

To achieve this, JupyterHub 0.8 introduces a Python API abstracting the proxy needs of JupyterHub and developers can provide their own proxy implementations as alternatives to CHP. The first such implementation uses a Kubernetes Traefik-based Ingress for the proxy, as part of KubeSpawner.

Persisting Authentication State

Another new feature available to Authenticators is the encrypted persistence of authentication state. This is aimed at preserving and passing authenticator-related state, such as client certificates or GitHub API tokens. Authenticators provided by oauthenticator 0.7 support persisting auth state from upstream authentication services.

Other Highlights

Preliminary (API-only, no GUI) support for multiple named servers per user
Token-based access to the single-user server API
A page for users to request new API tokens from the Hub

Notes on upgrading

As with every update to JupyterHub, make sure to back-up your database and run upgrade-db. For example, if using JupyterHub with SQLite:

Backup:cp -v jupyterhub.sqlite jupyterhub-backup-$(date +%Y-%m-%d).sqlite
Upgrade: jupyterhub upgrade-db

Due to the change in internal authentication, it is important that you make sure that single-user servers and the Hub itself are upgraded at the same time. If you are running everything in one env (e.g. the default Spawner, SudoSpawner, or similar), there is nothing to do. However, if you are using a container-based setup to launch single-user server (e.g. DockerSpawner or KubeSpawner), make sure that your user image and your Hub both get the upgrade at the same time. In most cases, this means adding to your Dockerfile:

RUN pip3 install jupyterhub==0.8.0

which ensures the correct version of JupyterHub is installed.

Thanks to everyone who has contributed to this release, especially users who have helped out with testing JupyterHub during the beta process, which helps make this a great release!

Notebook 4.2.2: Security fix

2016-08-03T20:47:00+00:00

TL;DR: upgrade to notebook 4.2.2

We just released notebook 4.2.2 with a few bugfixes and one important security fix.

We (specifically Steve Sylvester) have found a longstanding security issue in the Notebook, regarding output of untrusted notebooks being able to execute code when the notebook is opened.

The vulnerability has been assigned the identifier CVE-2016-6524.

Affected versions:

2.0 ≤ IPython < 4.0 (all IPython notebook versions, starting with the introduction of the concept of untrusted output)
notebook ≤ 4.2.1

Solution:

Upgrade to notebook 4.2.2

Specifically:

pip install --upgrade notebook
or conda install notebook (once conda channels propagate the release)

If you cannot upgrade to 4.2.2, we have prepared a backported patch for the 2.x and 3.x branches of IPython, but have no current plans for making a new release of these branches.

Details

The vulnerability was reported on July 26, and a fix proposed on the same day. After some review and testing of the patch and a check for similar vulnerabilities, on August 3 a CVE was assigned and the 4.2.2 release made immediately thereafter.

The goal of Jupyter notebooks is executing code, and outputs being able to execute code is one of the key features that enables people to do cool stuff. However, we don’t want notebooks written by not-you to be able to execute code as you on your computer as soon as you open them. For this reason, we have the concept of ‘trusted notebooks’ (introduced in IPython 2), which are notebooks where all output has been produced by you. The main result is that if you get a notebook from someone else with fancy dynamic javascript output, you might see things like:

<Javascript uninteresting repr...>

instead of the fancy visualization. Once you either execute the notebook yourself or explicitly trust the notebook from the File menu, you can see the real results.

When loading output from an untrusted notebook, we take one of three actions:

the output is unsafe, and ignored (Javascript)
the output is potentially unsafe, and sanitized (HTML)
the output is safe, and displayed regardless of trust

It’s that third option of “safe” outputs where things went a bit awry. Safe outputs should be outputs that cannot run code in the browser. This includes plain-text output, which we reliably escape, and raster images, such as pngs and jpegs. So far, so good. It also includes latex, which ought to be like plain text, in that it’s properly escaped before being put on the page to be rendered by MathJax. This is where we went wrong. We were putting latex output on the page with jQuery’s $.append, which puts elements on the page verbatim. In this way, if the output that claimed to be latex was actually HTML with script tags and such, it would be able to execute code. The right thing to do (all along) was to use the safe $.text, which escapes HTML entities before putting things on the page.

So that’s the fix.

I’m a bit surprised that we didn’t have bugs caused by inequalities and such in equations being mistakenly interpreted as HTML tags.

See our security docs for further details on the security model in Jupyter Notebooks.

If you find a security issue with the notebook or other Jupyter components, please let us know at [email protected]

Notebook 4.2

2016-04-15T20:31:00+00:00

We have just released version 4.2 of the Jupyter Notebook. The Jupyter Notebook is a web application that allows you to create and share documents that contain live code, equations, visualizations and explanatory text (see the Jupyter project home page for more information).

Version 4.2 of the notebook has many bugfixes and improvements. You can see a summary of changes in the docs, or see all the changes on GitHub.

Altogether, 21 authors contributed to 53 pull requests to make this release.

The notebook extension system was improved greatly in version 4.2. The notebook can be extended in two ways:

nbextensions, which are JavaScript extensions that run in your browser. These can modify the UI or behavior of the notebook, such as adding buttons to your toolbar, new keyboard shortcuts, etc.
serverextensions, which modify the Python webserver. These can add new HTTP endpoints, register extra events to run when you save your notebook, etc.

The improvements to the extension system are focused on making installing and enabling extensions easier, including:

better symmetry between handling of serverextensions and nbextensions
allowing enabling nbextensions system-wide
working better with the Python sys.prefix for installing/enabling extensions using packages and environments, such as conda packages or python wheels
allowing installation via Python packages

In general, installing and enabling extensions is a two-step process:

install the extension:
jupyter nbextension install /path/to/myextension
enable the extension
jupyter nbextension enable myextension/main

Installation is system-wide by default, but you can specify an installation to be just for you with --user. Enabling an extension is per-user by default, but you can enable an extension system-wide with --system. You can install or enable an extension using the Python sys.prefix by using the --sys-prefix option available in either command. For example, to make a conda package that includes an extension, you should install the extension using --sys-prefix when building the package.

Python packages can specify information about what nbextensions and serverextensions they provide. These can be installed with:

jupyter nbextension install --py packagename  
jupyter serverextension enable --py packagename

For more information on installing extensions from Python packages, see the docs.

Edit: updated to add a link to the main Jupyter webpage, based on feedback from HackerNews. Thanks!