nbviewer has a new host: OVHcloud

March 06, 2020 · Min RK · Kubernetes nbviewer

For several years, nbviewer has been generously hosted by Rackspace. That sponsorship program appears to be ending, so nbviewer needed a new home; it has found one in OVHcloud. We are extremely grateful to OVHcloud for their support in keeping nbviewer running, building on their existing participation in the Binder Federation (literally—nbviewer is now running on the same kubernetes cluster as ovh.mybinder.org).

How we moved

Where were we before?

nbviewer was previously deployed using a private repo (because it contained credentials) and various commands using invoke. It was a mixture of custom steps, using the openstack Python API and docker machine to:

• allocate two VMs
• build docker images
• deploy two nbviewer instances per node
• deploy memcached via nbcache on each node
• deploy statuspage publisher as a separate step
• update fastly to point to the running nbviewer instances

The upside was that we had a repo that could immediately deploy nbviewer to anywhere with docker. With this repo, we moved our deployment strategy from a CoreOS cluster to Rackspace’s short-lived Carina service to deploying VMs ourselves with docker-machine. Migrating to a new source of VMs would not have been hard, but it wouldn’t have solved any of our challenges.

Known downsides of this deployment that we’ve experienced over the years:

• only a few folks ever knew how to use it and could thus deploy updates to nbviewer
• it was private, contributing to above. It’s hard to onboard folks in an open community to a private repo!
• independent machines meant cache was not shared (minor, but contributes to our consumption of the GitHub API rate limit)
• no automatic recovery based on health monitoring, so when a container had issues, some humans got automated emails but no action was automatically taken. Extra frustrating because the fix was ~always to restart the container.

The credentials used by this repo have been revoked and an archived version of the repo is available (with credentials redacted from history) in the new, public nbviewer.org-deploy repo.

What have we learned?

We’ve learned a lot about open, automatic, and sustainable deployments, and can now comfortably address all of the downsides above. Most of this has been learned from the communities participating in the JupyterHub and Binder projects, as seen in the mybinder.org-deploy repo. mybinder.org-deploy is a public repo that automatically deploys and tests updates to at least four different Kubernetes clusters at the push of a button (the Big Green Merge Button, to be precise). Some things we have learned in the years since we set up our nbviewer deployment:

• kubernetes and helm are great :)
• git-crypt allows us to have public repos with some secret contents, so deployment repos don’t need to be fully private just to protect a couple api keys.
• Continuous Deployment via services like Travis or Circle lowers the bar to adding maintainers on a given deployment since all that’s needed is to press the Big Green Button.

The OVH sponsorship came in the form of a slice of a kubernetes cluster; that meant we had to migrate the nbviewer deployment tools from using docker-machine to kubernetes, which for us means helm.

Step 1: helm chart for nbviewer

The first step was to create a helm chart for nbviewer, which is done here. Before, nbviewer was two docker containers running nbviewer and one running memcached per server. To turn this into a helm chart we need:

• dependency on memcached helm chart for easy deployment of the cache. All the nbviewer instances will talk to this memcache, which should improve our GitHub rate limit consumption, since the cache will be shared. This means the deprecation of our own nbcache repo.
• Deployment for nbviewer. This is (one of) the kubernetes wrappers around containers. It has a nice ‘replicas’ field to easily scale nbviewer up and down. We are currently running with 3 replicas.
• Service to expose nbviewer to the Internet (likely also need an ingress in the future)
• Deployment for the statuspage publisher, which updates https://status.jupyter.org with the remaining github rate limit available

We also needed helm charts and docker images for some other components, such as cdn.jupyter.org and the nbviewer statuspage data source.

Step 1b: helm chart for cdn.jupyter.org

nbviewer and some other services interact with cdn.jupyter.org, which is a lightweight nginx configuration to serve the classic notebook’s javascript and css as static files(this is no longer needed for npm-based jupyterlab, which can use unpkg as a CDN). This used to run on a multipurpose server VM, but that is also being retired for the same reason. The result was creating a docker image and helm chart for serving the contents of cdn.jupyter.org. The scripts used to run the existing CDN were already in a repo, so it was a small amount of work to adapt this to run in a container instead of on a server. We may retire cdn.jupyter.org in the future, so please don’t rely on it :)

This is where we are right now — nbviewer.jupyter.org and cdn.jupyter.org are being served by OVH and the Rackspace machines are being retired.

Step 2: automatic deployment

It would have been ideal for this to be step 1, but that’s now how it happened. Sometimes you need to get it done quickly before you get it done right. The task here will be to make a new, public nbviewer-deploy repo following the patterns we have learned in mybinder.org-deploy. That will mean:

• create public nbviewer.org-deploy repo with configuration and git-crypt encrypted secrets for deploying nbviewer on the ovh cluster (done)
• adopting chartpress to publish our helm charts for nbviewer and version-tagged images from the nbviewer repo
• configure Travis-CI or other CI service to automatically deploy updates with helm, so merging a PR is all we need to do to deploy updates to nbviewer

Thanks to OVHCloud for their continued support of Jupyter and Binder, and the JupyterHub and Binder communities for teaching me how to operate services in the open with kubernetes and helm.