Jupyter Blog - DevOps

Automate your releases with the Jupyter Releaser 🚀

2024-10-28T18:09:00+00:00

In the world of open-source software, releases ensure that the latest features, bug fixes, and improvements reach users promptly. However, the process of managing releases, especially for large projects, can be complex and error-prone. This is where Jupyter Releaser comes into play.

Jupyter Releaser is an automation tool developed by the Jupyter team to streamline and standardize the release process across Jupyter projects. Its goal is simple: reduce the manual burden on maintainers, ensure consistency, best practices, improve confidence, and eliminate the potential for human error.

😔 The Problem: Manual Release Processes

Many maintainers of libraries have experienced the pains of a manual release process:

Versioning Issues: Manually updating version numbers across multiple files can lead to inconsistencies and errors.
Changelog Generation: Creating and maintaining accurate changelogs can be time-consuming.
Publishing: Whether it’s for PyPI or npm, the process involves multiple steps that are easy to mess up, and requires maintainers to manage their own credentials and tokens.

These pain points are not just frustrating; they can also delay the release of new versions and introduce issues that affect the end-users.

🤩 The Solution: Jupyter Releaser

Jupyter Releaser automates and standardizes the release process, ensuring that everything from versioning to changelog generation and publishing is handled efficiently. It helps maintainers reduce toil and save time in the release process by enforcing best practices to:

Automate a changelog for every release
Verify the install and import of dist assets, such as Python wheels and npm packages, to ensure packaging is done correctly
Commit a message with hashes of dist files
Annotate the git tag in a standard format
Create a GitHub release with a changelog entry, and automatically populate a changelog file
Trigger a dry run publish on CI, to make sure the project is in a releasable state at any time
Consistently publish final packages to PyPI and npm

➡️ Getting Started with Jupyter Releaser

Step 0: Set up

Setting up a repository to use the Jupyter Releaser involves a few steps which are listed on this page of the documentation.

The first one is to create a GitHub app for the organization or profile that owns the repo, following this guide. Then continue with the steps in the documentation to fill in the values for the app.

To follow best practices it is also recommended to create a release environment on the repo to add the APP_ID variable and APP_PRIVATE_KEY secret.

If you publish a package to PyPI, you will also need to add a trusted publisher, which usually corresponds to the source repository.

For npm, don’t forget to specify a "publishConfig": { "access": "public" } to the package.json.

Finally, copy the prep-release.yml and publish-release.yml workflows from the example-workflows folder to your repository.

Quite a few steps, but you should now be ready to go! Fortunately this setup needs to be done only once per repository. If you release multiple projects hosted under the same organization, you can reuse the same GitHub app.

Again, feel free to refer to the documentation for the complete checklist.

Step 1: Prep Release

To start the release, go to the “Actions” tab of the repo and select “Step 1: Prep Release”. Open the “Run Workflow” dropdown and fill in the appropriate parameters.

Running the “Prep Release” workflow for a JupyterLab extension (speed x5)

Running this workflow will create a draft GitHub release with the generated changelog for the new version. You can edit the changelog by editing the release if needed. Don’t forget to click on “Save Draft” and not “Publish Release” if you make any changes! (the releaser will do the publishing in the next step)

Step 2: Publish Release

If the changelog and draft GitHub release look good, go back to the “Actions” tab and click on “Step 2: Prep Release”.

In most cases, you can leave the fields empty and the releaser will automatically publish the GitHub Release previously created.

Publishing the extension (speed x5)

Congrats, you just made your first release with the Jupyter Releaser! 🎉

Going further

Jupyter Releaser also provides a set of hooks and options, such as running an arbitrary command after a given step, or providing a custom strategy for versioning packages. This allows for great flexibility in more complex repositories.

Check out the documentation for more information.

🌐 How Jupyter Releaser Fits into the Jupyter Ecosystem, and beyond

Releases are moments of celebration in any project, a chance to deliver new features, enhancements, and bug fixes to users. But they shouldn’t be a source of stress. Jupyter Releaser makes releases smoother for maintainers across the Jupyter ecosystem and beyond.

Jupyter Releaser is now used for most of the projects in the Jupyter stack with great success, such a JupyterLab, Jupyter Notebook, Jupyter Server, ipykernel, and many more. It greatly simplified the previously complex JupyterLab release process, which used to take 30 minutes, and now boils down to clicking on just a few buttons on GitHub and waiting for the workflows to complete.

Although its name has “Jupyter” in it, the releaser is not limited to releasing Jupyter related projects only, If you maintain a simple Python or npm package, you can also use the Jupyter Releaser to publish your package(s) to PyPI, npm or both!

In the end, Jupyter Releaser can simplify your workflow and save you valuable time. Give it a try and check out the repo here to see how it can make software releases a breeze!

🙏 Acknowledgments

Many thanks to Steven Silvester for creating the Jupyter Releaser and helping with its adoption across the Jupyter stack.

Jupyter Releaser is an official Jupyter project, and the Jupyter Releaser maintainers would also like to thank the contributors in the Jupyter community who help improve the tool and provide feedback ❤️

Accurately counting Daily, Weekly & Monthly active users on JupyterHub

2023-03-27T09:16:00+00:00

Being able to say ‘we served X unique users over the last month’ (the Monthly Active User metric) is very helpful when advocating for resources for a JupyterHub your organization is running. However, until now, figuring out that number accurately has been difficult, requiring keeping and analysing JupyterHub logs.

That changes with JupyterHub 3.1! This Pull Request adds daily, weekly, and monthly active user metrics to JupyterHub, accessible via the prometheus interface by hitting the /metrics URL on your JupyterHub. These metrics are calculated by JupyterHub itself, and are pretty accurate as it already keeps track of when a user was last active. This relies metric on each JupyterHub user matching an actual user, so if you are using your JupyterHub deployment purely as an API with ephemeral users (as binderhub does, for example) or delete inactive users, these will not be useful numbers.

Ideally, you should have a prometheus instance to scrape and store metrics from your JupyterHub over time, so you can track this metric over time. However, in a pinch, you can also go directly to https://<your-hub-url>/hub/api/metrics and see the current value of these metrics. They are in the prometheus exposition format, but you can look for something like:

# HELP jupyterhub_active_users number of users who were active in the given time period
# TYPE jupyterhub_active_users gauge
jupyterhub_active_users{period="24h"} 610.0
jupyterhub_active_users{period="7d"} 2800.0
jupyterhub_active_users{period="30d"} 4526.0

This denotes that this particular hub had 610 daily active users (active over the last 24 hours), 2800 weekly users (over last 7d) and 4526 monthly ones (over the last 30 days). Very helpful if you want to quickly add numbers to a report :)

Note that depending on your hub’s configuration, access to the /metrics endpoint might require authentication.

If you do have a prometheus installation for your JupyterHub, you may benefit from deploying the JupyterHub Grafana Dashboards. These are targeted at installations of zero-to-jupyterhub on kubernetes, and provide a lot of useful usage & diagnostic information.

Grafana Dashboard showing “Hub Usage Stats” for a JupyterHub deployment. Four panels, clockwise: Current Active Users, Daily Active Users, Weekly Active Users, Monthly Active Users.

Happy report writing!

Introducing Jupyter Scheduler

2022-12-14T15:26:00+00:00

The Open Source Jupyter team at AWS is proud to announce the release of Jupyter Scheduler, a JupyterLab extension that can run your Jupyter notebooks in the background. Jupyter Scheduler was developed from the start as an open-source project under the umbrella of the Jupyter project and governance. You can set up your notebooks to run once or on a schedule. By default, Jupyter Scheduler runs notebooks on your local system where Jupyter Server is installed; you can also extend the scheduler to run your notebooks in the cloud, on an on-premise cluster, on JupyterHub, or even on supercomputers.

We hope you’ll find Jupyter Scheduler useful for running notebooks periodically, such as daily or weekly reports. You can install it in JupyterLab by running pip install jupyter_scheduler, then restarting JupyterLab. You can create jobs (single runs of an individual notebook) and job definitions (scheduled recurring notebook jobs) in two ways. You can right-click on a notebook in the file browser and choose “Create Notebook Job” from the context menu, or you can use the new toolbar button in an open notebook file.

When you right-click on a notebook file in the file browser, a “Create Notebook Job” option appears in the context menu.

In a notebook, the “Create Notebook Job” option appears on the right of the toolbar.

You can give your notebook job or job definition a name, choose an environment to run it in, select different output formats, and provide parameters that are set as local variables when your notebook gets executed. This parameterized execution is similar to Papermill; Jupyter Scheduler provides a UI for managing and accessing the outputs of your notebook jobs inside JupyterLab.

You can use the Create Job form to run a notebook once or to create a job definition that runs on a schedule.

In addition, if you select “Run on a schedule,” you can run your job on a recurring schedule. The scheduler provides a few useful shortcuts to, for example, run your notebook at a specified interval by the minute, hour, day or week. You can also specify a custom schedule in crontab format.

You can schedule a job definition using preset intervals, such as every weekday.

You can also select “custom schedule” and specify a cron expression for the notebook job definition’s schedule.

Once you’ve created a job or job schedule, you can use the “Notebook Jobs” icon on the launcher to view lists of jobs and job definitions. Each item’s name links to a detail view.

With the Jupyter Scheduler extension installed, JupyterLab’s launcher screen shows a “Notebook Jobs” button in the “Other” section. Click on it to view a list of jobs or job definitions.

The list view shows a list of notebook jobs and a list of notebook job definitions. Each one includes links to view details, to download files, and buttons to take quick actions on them.

The Job Detail view shows all available details for one job or one job definition.

When one of your jobs has completed, you can download its files from the list or detail view. When your notebook runs, the scheduler saves a copy of the input file, so that even if you later change the file, it captures the state of the notebook at the time your job ran. Jupyter Scheduler uses unique names for the input and output files so that rerunning the same notebook produces new files every time.

Jupyter Scheduler uses a REST API to create, update, delete and manage jobs and job definitions. You can also customize Jupyter Scheduler by plugging in your own scheduler class, for example, to replace the existing SQLite database or to run jobs in the cloud. AWS SageMaker Studio actually uses Jupyter Scheduler, with custom classes, as part of its automated notebooks feature, announced earlier this year. See the Jupyter Scheduler operator documentation for more information about how to write and use your own scheduler.

We’d love to hear what you think about Jupyter Scheduler! You can check out Jupyter Scheduler’s documentation on ReadTheDocs. Please open issues with ideas for enhancements and about any bugs you find. If you’d like to contribute code, see the Jupyter Scheduler contributors guide.

This article does not represent the official position of amazon.com or any of its subsidiaries.

About the authors

Andrii Ieroshenko is a Software Development Engineer at AWS. Previously, Andrii worked on the Notifications project as a part of the Jupyter Cal Poly student internship program.

Piyush Jain is a Software Development Engineer at AWS working on open source contributions to Project Jupyter. He is also a member of the Jupyter Server team, actively participating in maintenance and development of Jupyter server project.

David Qiu is a Software Development Engineer at AWS working on Project Jupyter. Previously, David was researching physical chemistry at the University of Illinois at Urbana-Champaign.

Jason Weill is a Senior Front-End Engineer at Amazon Web Services; a contributor to JupyterLab, Jupyter Governance, and Jupyter Scheduler; and a member of the JupyterLab Council, the Jupyter Security Working Group, and the Jupyter Diversity, Equity, and Inclusion Standing Committee.

How I automated authorised cloud deployments from Pull Requests with GitHub Actions

2021-11-22T18:30:00+00:00

I recently did some work on the mybinder.org deployment infrastructure to solve a problem with testing Pull Requests before deployment. It had not been possible to test Pull Requests on our staging deployment because our automated workflows don’t have access to secrets. This resulted in my writing the test-this-pr action and this blog is a retrospective of what I learned over that process.

What was the problem we were trying to solve?

It is generally considered best practice to have a staging environment when running complex applications or platforms that have a large userbase, such as mybinder.org. This provides developers a space to test infrastructure changes safely with the knowledge that users won’t be affected should anything go wrong. The deployment configuration for mybinder.org is managed in a GitHub repository and deploys are handled in a CI/CD pipeline run on GitHub Actions.

The mybinder.org operating team’s usual workflow will be very familiar to anyone working as part of a distributed development team: fork the main repository, do some development (fix a bug or implement a feature), and then open a Pull Request (PR) back to the default branch of the main repository. The difficulty arises when we want to test the PR on our staging infrastructure before merging and deploying for real.

Various secrets are required to make a deployment, even to our staging environment. These include a git-crypt secret that decrypts certain sensitive files and credentials to push Docker images to the repositories connected to our clusters, among others. These are stored as encrypted repository secrets so they are available to GitHub Actions when our CD pipeline runs. However by default, repository secrets are not made available to PRs from forks, which introduces a problem for the workflow I described above.

So to summarise our problem:

Contributors often make changes to the code in their fork of the repository
Testing any change requires deploying to an actively-running staging cluster
The secrets needed to make those deploys are only available to the original repository, not to a forked repository
So we needed a way to trigger a test deployment on changes from a forked repository but that ran in the origin repository

What solutions did we already have in place?

We had one team practice, and one technical solution to address this issue, both of which didn’t quite do what we wanted.

One solution was a team deployment practice of “just merge the PR anyway and file a revert PR if things go wrong”. Our CI/CD pipeline is broken into two jobs: deploy to staging infrastructure, and a matrix deployment to each production cluster in the BinderHub federation. The production jobs require the staging job to pass first before they are triggered. This means that if the deploy to staging fails, the changes will never propagate to our production clusters.

Another solution was a “test-staging” label we could apply to open PRs. This triggered the deploy to staging job without a merge. However, this only worked for PRs opened from branches within the origin repository, and as mentioned above, this is not a common workflow for maintainers. Instead they open PRs from forked repositories, which do not have the proper secrets to deploy to staging.

What other solutions were available?

We briefly considered ok-to-test but were put off by the requirement for a GitHub App. However as we will learn, it was probably a mistake to dismiss this so quickly!

What did I end up implementing?

And so I built my first GitHub Action!

The test-this-pr action allows you to trigger a GitHub Action from the origin repository when an authorised user adds a comment to the PR of a forked repository.

test-this-pr is triggered when a user leaves a comment on the PR you’d like to test, but only if the commenter has the appropriate permissions on the repository, e.g. MEMBER. This protects us from malicious code being run automatically on our staging deployment and ensures that project maintainers have vetted the code in some way before triggering the test.

When triggered, test-this-pr uses a small Python script that queries the GitHub API to do the following:

It creates a new branch in the origin repository that uses the merge reference from the PR.
This triggers the test suite in the new branch, because we have configured the origin repository to automatically run deployment tests on branches that match a particular name pattern (in our case, test-this-pr/*, see below)
Adds comments on the original PR with a link to the newly-running CI test logs.

This script is wrapped up into a Dockerfile, which allows GitHub Actions to run the action using its docker runner.

An example of the test-this-pr workflow in action.

Extra configuration needed to use this action

There were a few extra pieces I configured that couldn’t (and shouldn’t!) be implemented in the test-this-pr action, but contribute to the overall workflow.

I also created a CI job that polls the running staging job. Upon completion, it posts a second comment to the original PR stating whether the test passed or failed. This had to be separate from the test-this-pr action since this job needs to run within the same workflow run in order to access the status of the staging job.

Once the pass/fail status of the staging job has been reported, the branch created by test-this-pr is then deleted. We decided this was the best path forward so that the original PR remained the “single source of truth” and maintainers wouldn’t be confused by extra branches in the repository.

For reference if you’d like to understand this implementation

What did I learn along the way?

There is A LOT you can do with the GitHub REST API

I had originally configured the Python script to shell out to git for tasks like creating the new branch and pulling the PR merge ref, and this worked fine when I ran Python locally. However when running the docker container, I struggled to authorise git correctly, even when passing my Personal Access Token. And so I made the decision to switch to the GitHub REST API.

While a lot of GitHub native features can be accessed and manipulated via the API, some git-specific actions can be achieved as well via the git database endpoint, which is the endpoint to use to create branches, commits, and much more. This section of the API might require a deeper dive into the inner-workings of git, but can unlock a lot of automation potential once you have your mental model!

Switching to the GitHub REST API from the git command line meant that I no longer needed to install git into my docker container or locally clone the repository to carry out the actions, making my image much smaller.

Triggering GitHub Actions from other Actions

You can’t trigger another GitHub Actions workflow from an event that was authorised with the GITHUB_TOKEN from within a previous workflow. This is why the comments left by the test-this-pr workflows in the Binder repository come from my account as I had to provide a Personal Access Token for the staging job to be correctly triggered.

The downside of this is that I’m automatically subscribed to notifications for any Binder PR where someone is using test-this-pr. It’s not the worst situation in the world as I have email filters set up, but also not ideal.

This is also why ok-to-test recommends setting up a GitHub App for authorisation, so you don’t have to sacrifice your own account and all the notifications that come with it!

However, this “actions triggering actions” scenario could be smoothed out by using the upcoming Reusable Workflows beta.

Building Actions with Dockerfiles

Providing a Dockerfile with a custom Action will allow you to write your Action in whatever language you like. However, it is one of the slowest ways to implement custom Actions because GitHub’s runner will build the image every time the workflow is triggered.

What would I do differently now?

Setup a GitHub App for authorisation

If only to spare my inbox from the notifications! 😂

Move away from a Python implementation

The test-this-pr action ultimately ended up being a handful of calls to the GitHub REST API. As opposed to maintaining a Python script and Dockerfile, I would probably reimplement this using the github-script action, which is a JavaScript wrapper for the GitHub API that can be directly run from GitHub Actions. This will be less code to maintain and the test-this-pr action itself could then be reworked as a composite action which would build faster than the current Docker-based implementation.

Closing Remarks

Members of the mybinder operating team discussing test-this-pr in gitter

While there is a lot I would change about test-this-pr, including fundamental changes that would make it look radically different, I needed to go on this developmental journey to gain the better understanding of GitHub Actions that I have now. And hopefully by sharing that journey, other folk can learn from it!

Also the mybinder operating team love using this feature and it’s adding value to our development workflow, so I don’t see a need to dive back in and start changing things again right now 🙂

You can read the source code for test-this-pr and mybinder.org’s deployment workflow in full.

How we automated mybinder.org dependency upgrades in 10 steps

2019-07-17T15:34:00+00:00

BinderHub and repo2docker are key components of the service at mybinder.org. In order to give Binder users the best experience, the Binder SRE team must continuously upgrade the version of these tools that mybinder.org uses. To avoid merging in massive updates at irregular intervals, it is desirable to merge updates in frequent intervals of smaller changes in order to more easily identify any breaking changes from the dependency upgrades.

While this process only takes a few minutes following processes outlined in the “Site Reliability Guide,” it is prone to human error (e.g., remembering to use the right SHA in upgrading the packages), and the team must remember to regularly do it in the first place. In the interest of automation, the Binder team decided to use a bot to relieve this burden, and we’ve decided to highlight its functionality in this blog post!

What does the mybinder.org upgrade bot do?

The upgrade bot should automatically update the versions of BinderHub and repo2docker that are deployed on mybinder.org. These are defined in the mybinder.org helm chart. To check whether an upgrade is needed, we want the bot to first “diff” the latest commit hash for both repo2docker and BinderHub repos against the deployed versions in the mybinder.org repo. If either or both are different, the upgrade bot does the following:

Fork the mybinder.org-deploy repo
Clone the fork locally
Checkout a new branch for the bump
Make the appropriate edit to update the commit hash in the mybinder.org fork repo
Add and commit the change
Push to the branch in the forked repo
Create a PR to the main mybinder.org repo
Remove the locally cloned repo

Additionally, it would be ideal if the bot could update an existing PR instead of creating new ones for the version bumps. We’d also like to provide some information in the comments of the PR as to what high level changes were made so we have some idea about what we’re merging in.

Here’s what we’re aiming for. The PR body:

The PR diff:

Now that we’ve broken it down a bit, let’s write up some Python code. Once we have a functioning script, we can worry about how we will run this in the cloud (cron job vs. web app).

Writing the bot

If you don’t care about the step-by-step, you can skip to the final version of the code.

In the interest of linear understanding and simplicity for a bot-writing tutorial, the step-by-step below will not write functions or classes but just list the raw code necessary to carry out the tasks. The final version of the code linked above is one way to refactor it.

Step 1: Retrieve current deployed mybinder.org dependency versions

The first step is to see if any changes are necessary in the first place. Fortunately, @choldgraf had already made a script to do this.

To find the current live commit SHA for BinderHub in mybinder.org, we simply check the requirements.yaml file. We’ll need Python’s yaml and requests modules to make the GET request and parse the yaml in the response. Note that this is also conveniently the file we’d want to change to upgrade the version.

from yaml import safe_load as load
import requests

url_requirements = "https://raw.githubusercontent.com/jupyterhub/mybinder.org-deploy/master/mybinder/requirements.yaml"
requirements = load(requests.get(url_requirements).text)
binderhub_dep = [ii for ii in requirements['dependencies'] if ii['name'] == 'binderhub'][0]
bhub_live = binderhub_dep['version'].split('-')[-1]
print(bhub_live)

Similarly, for repo2docker, we check the mybinder.org values.yaml file:

url_helm_chart = "https://raw.githubusercontent.com/jupyterhub/mybinder.org-deploy/master/mybinder/values.yaml"
helm_chart = requests.get(url_helm_chart)
helm_chart = load(helm_chart.text)
r2d_live = helm_chart['binderhub']['config']['BinderHub']['build_image'].split(':')[-1]
print(r2d_live)

Let’s store these SHAs in a dictionary we can use for later reference:

commit_info = {
               'repo2docker': {},
               'binderhub': {}
              }
commit_info['binderhub']['live'] = bhub_live
commit_info['repo2docker']['live'] = r2d_live
print(commit_info)

Step 2: Retrieve latest commits from the dependency repos

When we get the latest commit SHAs for repo2docker and BinderHub, we need to be careful and make sure we don’t automatically grab the latest one from GitHub. The travis build for mybinder.org looks for the repo2docker Docker image from DockerHub, and the latest BinderHub from the JupyterHub helm chart.

Let’s get the repo2docker version first:

url = "https://hub.docker.com/v2/repositories/jupyter/repo2docker/tags/"
resp = requests.get(url)
r2d_master = resp.json()['results'][0]['name']
print(r2d_master)

Now we can do BinderHub:

url_helm_chart = 'https://raw.githubusercontent.com/jupyterhub/helm-chart/gh-pages/index.yaml'
helm_chart_yaml = load(requests.get(url_helm_chart).text)
# sort by date created
updates_sorted = sorted(helm_chart_yaml['entries']['binderhub'], key=lambda k: k['created'])
bh_master = updates_sorted[-1]['version'].split('-')[-1]
print(bh_master)

Let’s add these to our dictionary too:

# add to commit_info dictionary
commit_info['repo2docker']['latest'] = r2d_master
print('repo2docker', commit_info['repo2docker']['live'], commit_info['repo2docker']['latest'])
commit_info['binderhub']['latest'] = bh_master
print('binderhub', commit_info['binderhub']['live'], commit_info['binderhub']['latest'])
print(commit_info)

Great, now we should have all the information we need to determine whether an update needs to be made or not, and what the new commit SHA should be!

Step 3: Fork mybinder.org repo

If we determine an upgrade for the repo is necessary, we need to fork the mybinder.org repository, make the change, commit, push, and make a PR. Fortunately, the GitHub API has all the functionality we need! Let’s just make a fork first.

If you have permissions to a bunch of repos and organizations on GitHub, you may want to create a new account or organization so that you don’t accidentally start automating git commands through an account that has write access to so much, especially while developing and testing the bot. I created the henchbot account for this.

Once you know which account you want to be making the PRs with, you’ll need to create a personal access token from within that account. I’ve set this as an environment variable so it isn’t hard-coded in the script.

import os
TOKEN = os.environ.get('HENCHBOT_TOKEN')
for repo in ['binderhub', 'repo2docker']:
    if commit_info[repo]['live'] != commit_info[repo]['latest']:
        res = requests.post('https://api.github.com/repos/jupyterhub/mybinder.org-deploy/forks',
			    headers={'Authorization': 'token {}'.format(TOKEN)})

Using the API for a post request to the forks endpoint will fork the repo to your account. That’s it!

Step 4: Clone your fork

You should be quite used to this! We’ll use Python’s subprocess module to run all of our bash commands. We’ll need to run these within the for-loop above.

        subprocess.check_call(['git', 'clone', 'https://github.com/henchbot/mybinder.org-deploy'])

Let’s also cd into it and check out a new branch.

        os.chdir('mybinder.org-deploy')
        subprocess.check_call(['git', 'checkout', '-b', '{}_bump'.format(repo)])

Step 5: Make the file changes

Now we need to edit the file like we would for an upgrade.

For repo2docker, we edit the same values.yaml file we checked above and replace the old SHA (“live”) with the “latest”.

        if repo == 'repo2docker':
            with open('mybinder/values.yaml', 'r') as f:
	        values_yaml = f.read()
            updated_yaml = values_yaml.replace(
			   "jupyter/repo2docker:{}".format(commit_info[upgrade]['live']),
			   "jupyter/repo2docker:{}".format(commit_info[upgrade]['latest']))   
	    fname = 'mybinder/values.yaml'
	    with open(fname, 'w') as f:
	        f.write(updated_yaml)

For BinderHub, we edit the same requirements.yaml file we checked above and replace the old SHA (“live”) with the “latest”.

        elif repo == 'binderhub':
            with open('mybinder/requirements.yaml', 'r') as f:
                requirements_yaml = f.read()
            updated_yaml = requirements_yaml.replace(
                           "version: 0.2.0-{}".format(commit_info[upgrade]['live']),
                           "version: 0.2.0-{}".format(commit_info[upgrade]['latest']))   
            fname = 'mybinder/requirements.yaml'
            with open(fname, 'w') as f:
                f.write(updated_yaml)

Step 6: Stage, commit, push

Now that we’ve edited the correct files, we can stage and commit the changes. We’ll make the commit message the name of the repo and the compare URL for the commit changes so people can see what has changed between versions for the dependency.

        # use var fname from editing step
        subprocess.check_call(['git', 'add', fname])
        
        if repo == 'repo2docker':
            commit_message = 'repo2docker: https://github.com/jupyter/repo2docker/compare/{}...{}'.format(
                              commit_info['repo2docker']['live'],commit_info['repo2docker']['latest'])
        elif repo == 'binderhub':
            commit_message = 'binderhub: https://github.com/jupyterhub/binderhub/compare/{}...{}'.format(
                              commit_info['binderhub']['live'], commit_info['binderhub']['latest'])

        subprocess.check_call(['git', 'config', 'user.name', 'henchbot'])
        subprocess.check_call(['git', 'config', 'user.email', 'henchbot.github@gmail.com'])
        subprocess.check_call(['git', 'commit', '-m', commit_message])
        subprocess.check_call(['git', 'push', 'https://henchbot:{}@github.com/henchbot/mybinder.org-deploy'.format(TOKEN), repo+'_bump'])

Awesome, we now have a fully updated fork ready to make a PR to the main repo!

Step 7: Make the body for the PR

We want the PR to have a nice comment explaining what’s happening and linking any helpful information so that the merger knows what they’re doing. We’ll note that this is a version bump and link the URL diff so it can be clicked to see what has changed.

        if repo == 'repo2docker':
            compare_url = 'https://github.com/jupyter/repo2docker/compare/{}...{}'.format(
                           commit_info['repo2docker']['live'], 
                           commit_info['repo2docker']['latest'])
            body = '\n'.join(['This is a repo2docker version bump. See the link below for a diff of new changes:\n', compare_url + ' \n'])

        elif repo == 'binderhub':
            compare_url = 'https://github.com/jupyterhub/binderhub/compare/{}...{}'.format(
                           commit_info['binderhub']['live'], 
                           commit_info['binderhub']['latest'])
            body = '\n'.join(['This is a binderhub version bump. See the link below for a diff of new changes:\n', compare_url + ' \n'])

Step 8: Make the PR

We can use the GitHub API to make a pull request by calling the pulls endpoint with the title, body, base, and head. We’ll use the nice body we formatted above, call the title the same as the commit message we made with the repo name and the two SHAs, and put the base as master and the head the name of our fork. Then we just make a POST request to the pulls endpoint of the main repo.

        pr = {
              'title': '{}: {}...{}'.format(repo,
                                            commit_info[repo]['live'],
                                            commit_info[repo]['latest']),
              'body': body,
              'base': 'master',
              'head': 'henchbot:{}_bump'.format(repo)
              }
        res = requests.post('https://api.github.com/repos/jupyterhub/mybinder.org-deploy/pulls',
                             headers={'Authorization': 'token {}'.format(TOKEN)}, json=pr)

Step 9: Confirm and merge!

If we check the mybinder.org PRs, we would now see the automated PR from our account!

Step 10: Automating the script (cron)

Now that we have a script we can simply execute to create a PR ($ python henchbot.py), we want to make this as hands-off as possible. Generally we have two options: (1) set this script to be run as a cron job; (2) have a web app listener that gets pinged whenever a change is made and executes your script as a reaction to the ping.

Given that these aren’t super urgent updates that need to be made seconds or minutes after a repository update, we will go for the easier and less computationally-expensive option of cron.

If you aren’t familiar with cron, it’s simply a system program that will run whatever command you want at whatever time or time interval you want. For now, we’ve decided that we want to execute this script every hour.

Cron can be run on your local computer (though it would need to be continuously running) or a remote server. I’ve elected to throw it on my raspberry pi, which is always running. Since I have a few projects going on, I like to keep the cron jobs in a file.

$ vim crontab-jobs

You can define your cron jobs here with the correct syntax (space-separated). Check out this site for help with the crontab syntax. Since we want to run this every hour, we will set it to run on the 0 minutes, for every hour, every day, every month, every year. We also need to make sure it has the correct environment variable with the GitHub personal access token we created, so we’ll add that to the command.

0 * * * * cd /home/pi/projects/mybinder.org-upgrades && HENCHBOT_TOKEN='XXXXX' /home/pi/miniconda3/bin/python henchbot.py

Now we point our cron to the file we’ve created to load the jobs.

$ crontab crontab-jobs

To see our active crontab, we can list it:

$ crontab -l

That’s it! At the top of every hour, our bot will check to see if an update needs to be made, and if so, create a PR. To clean up files and handle existing PRs, in addition to some other details, I’ve written a few other functions. It is also implemented as a class with appropriate methods. You can check out the final code here.

DevOps Intelligence

2019-04-06T14:50:00+00:00

Responding to internal stakeholders via feedback loops relies on readily available knowledge.

A JupyterHub Showcase

DevOps Intelligence turns data from software development and delivery processes into actionable insight, just like BI does for the business side. Jupyter is the ideal instrument for that, with its combination of powerful coding environments and a user interface facilitating experimentation with ultra-short feedback cycles.

A Jupyter-based setup supports risk analysis and decision making within development and operations processes — typical business intelligence / data science procedures can be applied to the ‘business of making and running software’. The idea is to create feedback loops, and facilitate human decision making by automatically providing reliable input in form of up-to-date facts. After all development is our business — so let’s have KPIs for developing, releasing, and operating software.

Typical Use-Cases in DevOps Intelligence

Here are some obvious application areas where data analysis can be helpful on the technical side.

Migration processes of all kinds (current state, progress tracking, achievement of objectives).
Inventory reporting for increased transparency and support of operational decisions.
Automate internal reporting processes to free up scarce assets and human expertise.

Platform Architecture

A simple JupyterHub setup can enable you to do analysis on your already available but under-used and hardly understood data, without any great investment of effort or capital. By adding a single JupyterHub host, you can use the built-in Python3 kernel to access existing internal data sources.

The following diagram shows what role JupyterHub can play in an existing environment.

Integration of JupyterHub into an existing development and operations system landscape.

To make such a deployment easy, the 1and1/debianized-jupyterhub project provides a JupyterHub service including a fully equipped Python3 kernel as a single Debian package — only Python3, NodeJS, and Chromium packages must be installed in addition to the jupyterhub one. If you raised an eyebrow on Chromium being in that list, it’s used by JavaScript-based visualization frameworks to render PNG images.

Including a NginX-powered SSL off-loader, the complete setup can be done in under an hour by an experienced Linux administrator.

Use-Case: Migration Reporting

At the time of this writing (early 2019), a widespread challenge is migration from Oracle Java to other vendors, and also to start migration from Java 8 to newer versions (Java 11). If you do that at scale across many machines and teams, you definitely need some kind of governance, and constant feedback on the current status and the rate of progress.

What follows is an excerpt from a productive notebook, with anonymized data about AdoptOpenJDK deployments. That data was originally retrieved from a system called “Patch Management Reporting”, which collects information about installed packages for all hosts in the data center. We’re in the yellow “Data Sources” box of the above figure here.

First off, we read the data and show the value sets of categorical columns, plus a sample.

import numpy as np
import pandas as pd

raw_data = pd.read_csv("../data/cmdb-aoj.csv", sep=',')
print('♯ of Records: {}\n'.format(len(raw_data)))

for name in raw_data.columns[1:]:
    if not name.startswith('Last '):
        print(name, '=', 
              list(sorted(set(raw_data[name].fillna('')))))

print(); print(raw_data.head(2).transpose())

♯ of Records: 104

Distribution = ['Debian 8.10', 'Debian 8.11', 'Debian 8.6',
                'Debian 8.9', 'Debian 9.6', 'Debian 9.7', 
                'Debian 9.8']
Architecture = ['amd64']
Environment = ['--', 'DEV', 'LIVE', 'QA']
Team = ['Team Blue', 'Team Green', 'Team Red', 'Team Yellow']
Installed version = ['11.0.2.9-83(amd64)', '11.0.2.9-85(amd64)', '8.202.b08-66(amd64)', '8.202.b08-83(amd64)', '8.202.b08-85(amd64)']

                                    0                   1
CMDB_Id                     108380195           298205230
Distribution              Debian 8.11          Debian 9.6
Architecture                    amd64               amd64
Environment                       DEV                  --
Team                         Team Red            Team Red
Last seen            2019-03-18 06:42    2019-03-18 06:42
Last modified        2019-03-18 06:42    2019-03-18 06:42
Installed version  11.0.2.9-83(amd64)  11.0.2.9-83(amd64)

Next comes the usual data cleanup. The Distribution column is a bit diverse, and not everyone has Debian code names and associated major versions memorized. The map_distro function fixes that.

def map_distro(name):
    """Helper to create canonical OS names."""
    return (name.split('.', 1)[0]
        .replace('Debian 7', 'wheezy')
        .replace('Debian 8', 'jessie')
        .replace('Debian 9', 'stretch')
        .replace('Debian 10', 'buster')
        .replace('squeeze', 'Squeeze [6]')
        .replace('wheezy', 'Wheezy [7]')
        .replace('jessie', 'Jessie [8]')
        .replace('stretch', 'Stretch [9]')
        .replace('buster', 'Buster [10]')
    )

Together with other cleanup steps, the mapper function is applied in a dfply pipeline. The result can be controlled by showing a sample of data points with unique version numbers.

from dfply import *

cleaned = (raw_data
    >> mutate(Version=X['Installed version']
                      .str.split('[()]', 1, expand=True)[0])
    >> mutate(Environment=X.Environment.fillna('--')
                           .str.replace('--', 'N/A')
                           .str.upper())
    >> mutate(Distribution=X.Distribution.apply(map_distro))
    >> drop(X.CMDB_Id, X['Last seen'], 
            X['Last modified'], X['Installed version'])
)
print((cleaned >> distinct(X.Version)).transpose())

                       0            13            14            62  \
Distribution   Jessie [8]  Stretch [9]    Jessie [8]   Stretch [9]   
Architecture        amd64        amd64         amd64         amd64   
Environment           DEV          N/A           DEV           DEV   
Team             Team Red    Team Blue      Team Red     Team Blue   
Version       11.0.2.9-83  11.0.2.9-85  8.202.b08-83  8.202.b08-85

                        68  
Distribution    Jessie [8]  
Architecture         amd64  
Environment            DEV  
Team             Team Blue  
Version       8.202.b08-66

Time to present the cleaned up data, starting with a table of teams and their number of installed packages. In the production notebook, an API of the corporate identity management is used to enrich the table with contact data of the team leads. Having the organizational data available also makes it possible to filter or aggregate the data by business units.

counts = cleaned.groupby(['Team']).size()
print(counts.reset_index(name='Count'))

          Team  Count
0    Team Blue     42
1   Team Green     16
2     Team Red     45
3  Team Yellow      1

To create a heatmap of how diverse a team’s version spectrum is, we calculate percentages of versions per team.

percentage = cleaned.groupby(
    ['Team', 'Version']).size().reset_index(name='Count')
percentage = percentage.assign(Percent=
    percentage.apply(lambda x: 100.0 * x[-1] / counts[x[0]], axis=1))
print(percentage.head(3))

        Team       Version  Count    Percent
0  Team Blue   11.0.2.9-85      1   2.380952
1  Team Blue  8.202.b08-66      1   2.380952
2  Team Blue  8.202.b08-85     40  95.238095

HoloViews makes creating the heatmap including a label overlay a breeze.

import holoviews as hv
hv.extension('bokeh')
publish = 1  # publishing or interactive mode?

heatmap = hv.HeatMap(
    percentage[['Version', 'Team', 'Percent', 'Count']]
).opts(
    hv.opts.HeatMap(
        title='Version Distribution by Team',
        width=480, xrotation=25, 
        zlim=(0, 100), cmap='kbc_r',
        clipping_colors=dict(NaN='#ffffe0'),
        colorbar=True, tools=['hover'], 
        toolbar=None if publish else 'right',
    )
).sort()

label_dimension = hv.Dimension(
    'Percent', value_format=lambda x: '%.1f' % x)
labels = hv.Labels(heatmap, vdims=label_dimension).opts(
    hv.opts.Labels(
        text_color='Percent', 
        text_font_size='10pt',
        text_font_style='bold',
    )
)

chart = heatmap * labels

if publish:
    import time
    import phantomjs_bin
    from IPython.display import HTML, clear_output

    %env BOKEH_PHANTOMJS_PATH={phantomjs_bin.executable_path}
    chart_img = 'img/aoj-heatmap.png'
    hv.save(chart, chart_img)
    chart = HTML('<img src="{}?{}"></img>'
                 .format(chart_img, time.time()))
    clear_output()

chart

From the heatmap, you can easily glance whether a team uses predominantly one version, and how recent the used versions are.

Conclusion

Using a platform powered by Project Jupyter and a big chunk of the scientific Python stack lets you easily mold your data into the shape you need, and then choose from a wide range of visualization options to bring your message across.

Incident Report: Jupyter services down

2017-12-14T19:52:00+00:00

update: December 14, 20:45 UTC, all services should be restored and back up.

On December 13, at 22:10 UTC (4:10pm EST), a large number of Jupyter-provided services stopped responding. This included, but was not limited to https://nbviewer.jupyter.org, https://try.jupyter.org (powered by tmpnb) and https://cdn.jupyter.org. We quickly narrowed this down to an issue with our hosting provider and have been working with them to resolve the issue as fast as possible.

When outages happen, the Jupyter Status page should show which services are affected and we publish updates there.

How are Jupyter services hosted?

To understand the cause of the outage, we need to understand how the Jupyter services are hosted and maintained. As Jupyter is an open organization which is mostly maintained by volunteers, we do not have a dev-ops team assigned to maintaining our infrastructure. Even with full-time developers hired through universities or companies, the time spent fixing infrastructure is taken on nights and weekends. These developers are often stretched thin and cannot be available 24/7.

Most of our cloud infrastructure is donated to us by companies like CloudFlare, Rackspace, Fastly, Google, and Microsoft. Donating resources can be challenging, both technically and legally. In this particular case, Rackspace graciously created a special account for Jupyter that handles invoices on our behalf, thereby making resources free to the project. Following a hiccup, this Jupyter account was suspended and all services are unavailable as a result.

Temporary resolution

As nbviewer is one of the most used services provided by Jupyter, we’ve moved it to one of our personal account at another cloud-provider. Fastly was set up to load-balance on the yet-to-come-back-up instances as well as this newly created instance, so all should be fine now.

The other services (tmpnb, mails@jupyter.org, cdn.jupyter.org, …) will still unavailable or highly degraded until a permanent solution is found, or the services are restarted. try.jupyter.org will likely redirect to a repo on https://mybinder.org in the meantime so people can still try out Jupyter.

Low bus factor

The outage of all these services lasted for a significant time (more than 18 hours). Which perturbed many of you relying on these services. We understand that this is hardly acceptable and we hope you’ll indulge us as these services are provided for free and without ads. One of the factors leading to the slow reestablishment of service was a relatively low bus factor, with only one and a half of our developers knowing how to deploy and maintain these services. Documentation and access to credentials was also limited.

This is one of the challenges in a distributed team like Jupyter where contributors self-organize. It is easy to forget that new code is not the only way to contribute and that infrastructure and maintenance are crucial.

We also overly rely on a single vendor (in this case Rackspace), and while we are happy with Rackspace and have no reason to move to another provider, we should have a plan to restore critical services even temporarily in case of failure.

A couple of months ago, the subject was brought to our attention, and we developed a plan to move many of our deployment to Kubernetes (which is provider agnostic). We underestimated the probability to need an emergency plan this early.

How can you help

Jupyter is mainly governed by the community all around the world. Contributing is not limited to writing code! We need members with knowledge in multiple languages, in design, dev-ops, etc. Whether you are an expert, or still learning, we would like you to get involved.

Thanks everyone for your patience and the kind words when you reached to us when discovering the services were down.