Jupyter Blog - nbviewer

nbviewer has a new host: OVHcloud

2020-03-06T19:46:00+00:00

For several years, nbviewer has been generously hosted by Rackspace. That sponsorship program appears to be ending, so nbviewer needed a new home; it has found one in OVHcloud. We are extremely grateful to OVHcloud for their support in keeping nbviewer running, building on their existing participation in the Binder Federation (literally—nbviewer is now running on the same kubernetes cluster as ovh.mybinder.org).

How we moved

Where were we before?

nbviewer was previously deployed using a private repo (because it contained credentials) and various commands using invoke. It was a mixture of custom steps, using the openstack Python API and docker machine to:

allocate two VMs
build docker images
deploy two nbviewer instances per node
deploy memcached via nbcache on each node
deploy statuspage publisher as a separate step
update fastly to point to the running nbviewer instances

The upside was that we had a repo that could immediately deploy nbviewer to anywhere with docker. With this repo, we moved our deployment strategy from a CoreOS cluster to Rackspace’s short-lived Carina service to deploying VMs ourselves with docker-machine. Migrating to a new source of VMs would not have been hard, but it wouldn’t have solved any of our challenges.

Known downsides of this deployment that we’ve experienced over the years:

only a few folks ever knew how to use it and could thus deploy updates to nbviewer
it was private, contributing to above. It’s hard to onboard folks in an open community to a private repo!
independent machines meant cache was not shared (minor, but contributes to our consumption of the GitHub API rate limit)
no automatic recovery based on health monitoring, so when a container had issues, some humans got automated emails but no action was automatically taken. Extra frustrating because the fix was ~always to restart the container.

The credentials used by this repo have been revoked and an archived version of the repo is available (with credentials redacted from history) in the new, public nbviewer.org-deploy repo.

What have we learned?

We’ve learned a lot about open, automatic, and sustainable deployments, and can now comfortably address all of the downsides above. Most of this has been learned from the communities participating in the JupyterHub and Binder projects, as seen in the mybinder.org-deploy repo. mybinder.org-deploy is a public repo that automatically deploys and tests updates to at least four different Kubernetes clusters at the push of a button (the Big Green Merge Button, to be precise). Some things we have learned in the years since we set up our nbviewer deployment:

kubernetes and helm are great :)
git-crypt allows us to have public repos with some secret contents, so deployment repos don’t need to be fully private just to protect a couple api keys.
Continuous Deployment via services like Travis or Circle lowers the bar to adding maintainers on a given deployment since all that’s needed is to press the Big Green Button.

The OVH sponsorship came in the form of a slice of a kubernetes cluster; that meant we had to migrate the nbviewer deployment tools from using docker-machine to kubernetes, which for us means helm.

Step 1: helm chart for nbviewer

The first step was to create a helm chart for nbviewer, which is done here. Before, nbviewer was two docker containers running nbviewer and one running memcached per server. To turn this into a helm chart we need:

dependency on memcached helm chart for easy deployment of the cache. All the nbviewer instances will talk to this memcache, which should improve our GitHub rate limit consumption, since the cache will be shared. This means the deprecation of our own nbcache repo.
Deployment for nbviewer. This is (one of) the kubernetes wrappers around containers. It has a nice ‘replicas’ field to easily scale nbviewer up and down. We are currently running with 3 replicas.
Service to expose nbviewer to the Internet (likely also need an ingress in the future)
Deployment for the statuspage publisher, which updates https://status.jupyter.org with the remaining github rate limit available

We also needed helm charts and docker images for some other components, such as cdn.jupyter.org and the nbviewer statuspage data source.

Step 1b: helm chart for cdn.jupyter.org

nbviewer and some other services interact with cdn.jupyter.org, which is a lightweight nginx configuration to serve the classic notebook’s javascript and css as static files(this is no longer needed for npm-based jupyterlab, which can use unpkg as a CDN). This used to run on a multipurpose server VM, but that is also being retired for the same reason. The result was creating a docker image and helm chart for serving the contents of cdn.jupyter.org. The scripts used to run the existing CDN were already in a repo, so it was a small amount of work to adapt this to run in a container instead of on a server. We may retire cdn.jupyter.org in the future, so please don’t rely on it :)

This is where we are right now — nbviewer.jupyter.org and cdn.jupyter.org are being served by OVH and the Rackspace machines are being retired.

Step 2: automatic deployment

It would have been ideal for this to be step 1, but that’s now how it happened. Sometimes you need to get it done quickly before you get it done right. The task here will be to make a new, public nbviewer-deploy repo following the patterns we have learned in mybinder.org-deploy. That will mean:

create public nbviewer.org-deploy repo with configuration and git-crypt encrypted secrets for deploying nbviewer on the ovh cluster (done)
adopting chartpress to publish our helm charts for nbviewer and version-tagged images from the nbviewer repo
configure Travis-CI or other CI service to automatically deploy updates with helm, so merging a PR is all we need to do to deploy updates to nbviewer

Thanks to OVHCloud for their continued support of Jupyter and Binder, and the JupyterHub and Binder communities for teaching me how to operate services in the open with kubernetes and helm.

nbviewer and GitHub’s rate limit

2017-03-01T07:45:00+00:00

If you’ve been using nbviewer recently, you may have seen this message more often than you would like:

It turns out that some overzealous SEO bots were eating up all of our resources. Last week, we discovered this cause of the API rate limit exhaustion and took a few actions to resolve it.

tl;dr; nbviewer shouldn’t hit the rate limit so often anymore. This is what our API usage availability has looked like since we deployed the changes I’ll describe here:

nbviewer’s architecture

I’d like to take this opportunity to give an overview of how we deploy nbviewer, so we can see how each piece comes into play.

To start off, nbviewer is two things:

the web application that finds notebooks on the internet and renders them, and
the specific deployment of nbviewer serving https://nbviewer.jupyter.org

You can even roll your own nbviewer by grabbing the public docker image and running:

docker pull jupyter/nbviewer
docker run -p 5000:5000 -it jupyter/nbviewer

You can then start rendering notebooks left and right. However, there’s a limit to how scalable such a basic deployment can be.

Rendering a notebook

Rendering a notebook involves two major steps:

download the notebook from somewhere on the web
render it to HTML with nbconvert

Each step can take a bit of time. Fetching and rendering a notebook start to finish can take as much few seconds if the notebook is big and/or the upstream server is busy or slow. We use tornado’s async APIs to handle concurrent requests. Fetching notebooks from the web is handled via a subclass of AsyncHTTPClient and rendering is done in the background via concurrent.futures. A single nbviewer instance can handle quite a few concurrent visitors, depending on how many rendering threads are available.

Caching

Most notebooks on the web don’t change very often, so going through the whole process when you just downloaded and rendered the same notebook a minute ago is a bit of a waste. This is where caching comes in.

nbviewer supports caching via a simple in-memory cache or memcache at two levels:

Every time nbviewer completes a render, it caches the result for a period of time (usually 10 minutes to an hour, depending on render time), so that a very popular notebook doesn’t consume too many resources. This is unconditional, and doesn’t take into account whether the original notebook has been updated. If it’s in the cache, it gets reused.
nbviewer separately caches upstream requests. When nbviewer asks an external service for anything, whether it’s a regular web server or GitHub’s API, the response is saved for a time. The next time the same resource is requested, nbviewer checks for a cached response. Unlike the full render cache, this cache doesn’t prevent a new upstream request. Instead, we check the headers of the cached reply (ETag and Last-Modified). nbviewer then takes the content of those headers to populate If-None-Match and If-Modified-Since headers before sending a new request. The server can then check these headers and return an empty reply with status 304, telling nbviewer that it should reuse the cached response instead of running the full process of handling the request again. Critically for nbviewer, when the GitHub API responds with 304, the rate limit is not consumed.

Our nbviewer deployment

To deal with the traffic our public nbviewer instance gets, we have deployed nbviewer with a few extra elements. The moving parts:

we have multiple nbviewer instances on multiple servers deployed. Right now, there are two nbviewer instances (running in docker) on each of two servers running on rackspace, making four total instances.
memcache, also running in docker, with separate instances on each server
fastly sits in front of our nbviewer instances and load-balances requests across them, performing health checks, etc. It also has its own cache layer.
Finally, cloudflare is out front, handles SSL termination, and a small amount of caching

Four instances is quite a bit more than we need for the load nbviewer gets, but having it spread out lets us do things like reboot or upgrade servers without any downtime.

What happened?

So, why were we seeing the GitHub rate limit hit so often? Did nbviewer just get too popular? Sadly, no. How did we figure out what happened? To the logs!

Fun fact: until now, we were using docker’s default json logging, and nbviewer was producing 50GB of logs each month. docker logs --tail does not behave super well with a 30GB log file. I started digging through the logs to see what was going on, but quickly realized that this wasn’t working very well. To try to get a better sense of things, I started by hooking up our logs to Loggly via syslog. Loggly lets me start to extract some metrics, such as status, URL, IP address, and user-agent of failing requests.

What did we learn?

One of the first things I did was group incoming requests by IP address, to see if anything stood out. This is what I saw:

So about two thirds of requests were coming from just two ip addresses! That’s not right. Checking for user-agent showed something else:

Two bots, Ahrefs and DotBot were accounting for almost three quarters of requests. That’s too much.

What we did about it

The very first thing we did was to block the two IP addresses that were driving way too much traffic. This was done in the fastly layer, sending a custom 429 error for any request from these IPs:

This improved availability a bunch, and would probably have been plenty on its own to resolve the current situation. Of course, it wouldn’t be enough to prevent similar situations in the future. Since the problem was caused by robots, the next place to look was robots.txt. Our robots.txt had been the most permissive possible:

User-agent: *
Disallow:

explicitly allowing any bot. Turns out we were being too nice. We’ve now updated our robots.txt to tell all robots to slow way down, and instructed the two overzealous bots that they are not welcome:

User-agent: *
Crawl-delay: 10

User-agent: dotbot
Disallow: /
User-agent: AhrefsBot
Disallow: /

Again, this should be enough to resolve the issue if these and all other bots are honorable and respect the robots.txt.

However, we would prefer to not rely on the kindness of strange robots.

Better caching and better use of the cache

As mentioned earlier, we cache the responses we get from upstream sources and use those to reduce the load on the GitHub API and other sources of notebooks. However, we were not making the best use of that cache. First of all, the responses were only being cached for two hours, which means this caching only help us for relatively short bursts of activity (e.g. a popular notebook being shared on social media). This certainly happens, but does not represent a large fraction of typical traffic. Since the way we check for updates with If-Not-Modified has no risk of stale results, we changed this to cache upstream responses forever, until memcache pushes them out when it gets full. Since most public notebooks don’t change often, we should get a cache hit almost every time.

The next change we made was an improvement to behavior when the rate limit is hit. Previously, if we had a cached upstream response and GitHub told us there was an update that we couldn’t fetch due to the rate limit, we would serve the 503 error page. But we had valid data right there! Now, if we hit the rate limit but happen to have an old version of the data, we work with the stale version rather than serving an error page. Something’s better than nothing, right? This applies to any kind of error fetching from an upstream server, not just failures due to hitting the GitHub rate limit.

So we are now much less likely to hit the rate limit in the first place, and much more likely to be able to serve something if and when we hit it.

Our own rate limit

The last thing we did to help limit the consumption of resources by a small number of actors was to implement our own rate limit in nbviewer. This is a pretty simple rate limit, where we identify a source by the combination of ip address and user-agent. Currently, this limit is set at 60 requests in ten minutes. We only count non-cached requests against the limit, so refreshing the same few pages a bunch won’t consume the limit.

Like the cache, the rate limit is actually implemented with memcache. When we see a request from a given source, we initialize a counter to memcache with an expiry of our rate limit window:

key = cache_key(request)  # combines ip and user-agent
new_visitor = memcache.add(key, 1, rate_limit_window)

If it’s not a new visitor, we increment the counter with memcache’s atomic increment operation:

if not new_visitor:
    count = memcache.incr(key)

The last thing to do is to check if the rate limit is exceeded, and return an HTTP 429 error instead of proceeding with the render:

if count > rate_limit:
    raise HTTPError(429, "Rate limit exceeded...")

If you hit the rate limit you will see:

From looking at our logs, we can see that this rate limit has blocked ~800 requests since deploying it a few days ago. More than 600 of these were from a bot that apparently doesn’t respect robots.txt. A few dozen were made from phantomjs, presumably some manual scraping that made a few too many fetches. That leaves a few probably-real humans who hit the rate limit, but no IP has hit it often or for a sustained period of time.

Let us know if the rate limiting is causing an issue for you. We will keep an eye on the limits, and continue to tune both the limits and limiting conditions to find the best balance.

Monitoring

We run our instances with the New Relic Python agent, which gets us some basic health monitoring. We also have some monitoring and alerts in our Rackspace account to notify us when things go down.

We’ve added the error rate tracked by New Relic and a new metric that tracks how much of our GitHub API limit is available to our public status page, so you can see how healthy nbviewer is at any given time.

One last thing

After confirming that everything was working well, I did a final deploy on a Friday evening, and moved on. Then this came in:

https://twitter.com/williamrthomas/status/835255588116447232

Turns out there had been an update to the New Relic Python agent since the last deploy, which introduced this tiny off-by-one error:

if len(args) > 0:
    args = list(args)
    args[1] = wrapped_callback
    # what if len(args) == 1 ?!

(that > 0 should be > 1)

This little bug just so happens to prevent nbviewer from making any outgoing requests :shrug:. This served as a friendly reminder that it is a good idea to pin your dependencies when deploying applications so that updates only happen when you decide they should.

Rendering Notebooks on GitHub

2015-05-07T19:47:00+00:00

We are pleased to announce that, starting today, and as announced on the GitHub blog, Jupyter/IPython notebook (.ipynb) files will render directly on GitHub. This feature works for notebooks in any of the supported Jupyter programming languages on both public and private repos. This capability will complement nbviewer and make it easier for GitHub users to create, view and share notebooks on GitHub.

Here are some great examples of notebook based content that can now be viewed directly on GitHub:

Cameron Davidson-Pilon’s Probabilistic Programming and Bayesian Methods for Hackers.
Lorena Barba’s Aerodynamics-Hydrodynamics with Python.
Benjamin Laken’s Monsoon Analysis.

We want to thank the GitHub team for bringing this new feature to GitHub. In particular @tclem, @arfon, @sshirokov, and @skalnik have all done an incredible job building this at GitHub. The first discussion of how to bring notebook rendering to GitHub happened almost 18 months ago, and a lot of work made this feature possible today. With over 200,000 notebooks available on GitHub, you already have a lot to read!

We are excited to see how this feature will impact science education and research on GitHub, as well as encourage the adoption of the Jupyter notebook as a standard file format across a wide range of fields. Jupyter notebook files are a JSON-based open document format that supports code and results, narrative text, images, and equations in one file. These documents simplify the creation and communication of computational narratives and make the sharing and replication of scientific results and data analyses simpler.

nbviewer

One key piece of infrastructure in sharing notebooks has been the nbviewer service, which started as a side project of mine in the late summer of 2012. Nbviewer has grown through deep integration with GitHub, with the ability to browse all public users, organizations, repositories, branches, tags, and even Gists. This would not have been possible without @rgbkrk and @bollwyvl, who handle the development and deployment of nbviewer. It would also not have been possible without Rackspace and Fastly, who provide free hosting and fast static asset distribution. Nbviewer has steadily grown to render hundreds of thousands of notebooks every week, and usage is still increasing. Based on this, we believe that rendering on GitHub will be a highly used and appreciated feature.

Although you can now view notebooks directly on GitHub, we are not planning to stop the development of nbviewer. While rendering directly in GitHub repositories is convenient and allows viewing notebooks in private repositories, there are number of reasons that nbviewer will continue to remain an important part of project Jupyter.

First, there are many notebooks not hosted on GitHub. As before, these notebooks can still be viewed on nbviewer. Furthermore, the Jupyter/IPython team are actively working on improving integration with other content providers, building on our experience with GitHub. For example, a recent pull request of @bollwyvl streamlines the process of adding new providers to nbviewer, with proposals for DropBox, GitLab, Google Drive, Stash, and other hosting and collaboration platforms.

Second, because of security concerns, some features available on nbviewer will not be available on GitHub. For example, GitHub will not render any dynamic output display that uses JavaScript, custom CSS, and most custom HTML embedded in Markdown or in outputs. When this is the case, and GitHub cannot fully render your notebooks, you will see an icon that allows you to view the full notebook on nbviewer.

Because nbviewer does not handle authentication, it will continue to support custom CSS, HTML and JavaScript in notebook. Thus, for highly customized or dynamic notebooks, nbviewer will still be the ideal place to share them.

We would love your feedback! If you have comments or find bugs with notebook rendering on GitHub or nbviewer, please open an issue on our nbviewer repo. Also, please send a few tweets to thank all the people without whom this would have not happened.

The Jupyter/IPython Team