Jupyter Blog - GitHub

Requiring 2FA for Jupyter GitHub Organizations

2022-07-05T15:00:00+00:00

This requirement and the outlined plan was discussed and agreed upon at the Jupyter Governance meeting on Friday, July 1, 2022.

By October 1, 2022, Project Jupyter aims to require two-factor authentication (2FA) for all GitHub organizations hosting repositories for official Jupyter Subprojects.

Enabling 2FA is the single most important step Jupyter contributors can take to protect their GitHub accounts from bad actors. This benefits the entire Jupyter Community by reducing the chance for malicious code to be slipped into a repository.

Fortunately, most Jupyter GitHub organization members and external collaborators have 2FA enabled, at this time. This process will get us to 100% so we can enable the requirement as a GitHub org setting.

What’s the process?

GitHub Documentation on 2FA

We recommend that all GitHub users secure their account with 2FA, even if you don’t have an organizational role like admin or owner, or write permissions to a Jupyter repository.

Over the next month (by August 1, 2022) contributors to the Jupyter Security Subproject will reach out to owners of each Jupyter GitHub org with a list of their users without 2FA enabled. The list of users will be an intersection across the GitHub orgs so that users aren’t contacted multiple times, and org maintainers are asked to do double work.

From there, the org owners can decide how they want to work within their area of the Community. They may choose to contact the users or ask the Security Subproject to reach out to the users. Or, the users without 2FA may no longer need the access or role they were granted.

At end of August, 2022, we’ll review the list of the remaining accounts without 2FA. (Hopefully none!) If possible, we’ll begin enabling the requirement on our GitHub orgs. The Security Subproject will work with org owners on plans for contacting any remaining users.

At the end of September, 2022, users without 2FA enabled may lose explicit permissions or roles within Jupyter GitHub orgs. This will only impact access to private repositories, commit privileges, or having a role such as owner or admin. Read access to public repositories will remain the same, along with opening issues or pull requests. And once users enable 2FA on their account, any previous permissions or roles can be restored.

What GitHub orgs does this apply to?

All GitHub orgs hosting repositories for official Jupyter Subprojects.

Securely pushing to GitHub from a JupyterHub with gh-scoped-creds

2022-04-21T16:54:00+00:00

Many JupyterHub users want to push and pull their content from GitHub in order to collaborate and share their work. However, working on a JupyterHub means working on shared infrastructure, not your own laptop, and this poses some extra security risks that have made two-way sync with GitHub more difficult. This post describes gh-scoped-creds, a new tool to make it quick and easy to authorize a JupyterHub session with push access to GitHub in a secure and simple manner.

GitHub user credentials are high value targets for cybercriminals in today’s security environment, and any system that stores these credentials long term paints an unwanted target on itself. Current solutions — putting an ssh key on the JupyterHub, using a personal access token or deploy keys — involve storing long term valid GitHub credentials in the filesystem. As users can do this by themselves without admin intervention, admins often are not aware these (often unencrypted) credentials are on their filesystems. If an attacker compromises an ssh key or a personal access token, they have unlimited access to all GitHub repos the compromised user had access to, including repos in high-impact GitHub organizations. In the recent credential theft incident, Travis-CI and Heroku were ‘lucky’ in that the attackers accessed npm infrastructure — and since npm is owned by GitHub, GitHub was able to detect that Travis CI and Heroku had compromised credentials. You and the users of repositories you have rights to might not be so lucky. It’s 2022, and supply chain attacks are everywhere — you aren’t special, you’re just one link in a long chain attackers use to get to someone else.

There is a clear need for a simple solution that lets users push to GitHub from JupyterHub in a secure manner without admins having to worry about securing high-value GitHub credentials long term. It is not acceptable to “Just Say No” to users wanting this functionality either — if you try to ‘sacrifice’ usability for security, you end up getting neither.

gh-scoped-creds attempts to solve this problem by allowing users to grant time-limited push access to specific repositories to specific JupyterHub installations in a user friendly way.

Here’s a quick GIF running through the user workflow.

Push access is scoped both by time (credentials expire after 8 hours) as well as repository (access is granted per-repository, per-hub). While you need to refresh credentials every 8 hours, the list of repositories is remembered until you explicitly revoke access. You can always grant access to your own personal repositories, but repositories belonging to organisations might require admins to approve push access to them.

You can also run the command from the terminal as gh-scoped-creds instead of using the IPython magic %ghscopedcreds as shown in the demo. This way, you can also use this from a HPC system, not just a JupyterHub!

Setting this up for your JupyterHub requires a tiny bit of work from the admin — see the project README for more details. Shouldn’t take long, and it’s a one-time task. Once that’s set up, your users can securely push to GitHub from the comfort of their JupyterHubs!

Thanks to Fernando Perez for using his stat159 class at UC Berkeley to test this project out.

Binder + Zenodo: A how-to guide

2019-06-21T13:46:00+00:00

Interactive and reproducible repositories powered by Zenodo and Binder.

When the Binder project was first launched, we imagined a world in which scientific scholarship and reproducibility could be carried out by the community using a fully-open stack of technology. We’re happy to say that this is now possible!

Zenodo DOIs now work with BinderHub

The BinderHub team recently added in support for building Binder links that point to Zenodo repositories. Zenodo is a general purpose open-access repository hosted by CERN that allows researchers to archive and apply a DOI to information that they put up on the web.

Zenodo has the ability to archive GitHub repositories, which means that you can archive the code, data, or reports that underly a scientific analysis and assign it a unique, citeable identifier. Now that BinderHub knows how to resolve a Zenodo identifier, you’ll be able to share Binder links that point to Zenodo and provide interactive access to your repository, letting readers reproduce results and interact with your analyses!

Here’s a quick primer for how to do this:

Step 1: Create a Zenodo account

First off you’ll need to create an account on Zenodo. You can
do so using a number of different log-in options.

Step 2: Create your Binder-ready repository on GitHub

Next, you should create your Binder-ready repository on GitHub. Binder uses pre-existing best practices in data science in order to infer and build the environment needed for your repository.

To make a repository Binder-ready, follow the instructions in the Binder docs.
Briefly, what you need to do is add the configuration files that define the environment needed to run your code. Once those files are in place, and you’ve added an analysis script (a Jupyter or R Notebook) that actually runs your code and displays the results, your repository is ready to build with Binder.

Step 3: Make sure your repository is ready to be published!

Once you create a DOI for your repository, it will be frozen in time — you won’t be able to easily update it. So double check that the repository builds properly with Binder and runs the way that you’d expect it to.

Make sure to launch a Binder from your repository and run the analyses you’d like others to run. If they produce the expected result from within a Binder session, then they’ll continue to do so for others (assuming you have pinned your versions and followed other best practices in reproducibility).

Step 4: Create a Zenodo DOI for your repository

Now that your repository is ready, you’ll connect Zenodo with GitHub to create a DOI for your repository. Remember that this will be unique to the current state of the repo — future changes to this repository won’t be reflected in the DOI.

We recommend following the GitHub Citable Code Guide which provides some best-practices for creating your Zenodo DOI for a GitHub repository. Click the image below to be taken to this (excellent) guide.

The GitHub Citable Code guide.

Once you’re done, you should have a Zenodo DOI badge
like the one below:

A Zenodo DOI badge

Step 5: Create a Binder link for your Zenodo DOI

Finally, use your Zenodo DOI to create a Binder link that allows others to interact with and replicate your results. You can create a Binder link for your Zenodo record by heading to https://mybinder.org and filling in the form:

Building a Binder link for your Zenodo archive with the Binder UI.

This will give you a link you can share with others as well as the Markdown and reStructured text snippets for creating a badge.

The link’s structure should look like this:

https://mybinder.org/v2/zenodo/<zenodo-DOI>

For example, if your Zenodo DOI is 10.5281/zenodo.3242074 (corresponding to this zenodo repository), the Binder link for it would be:

https://mybinder.org/v2/zenodo/10.5281/zenodo.3242074/

You can even pair this Binder link with your Zenodo DOI badge that
we showed above!

The “launch binder” button contains a Binder link for your Zenodo archive. Clicking will launch a Binder session.

And that’s it! You now have an archived version of your analysis with a unique identifier. This identifier can be used in conjunction with BinderHub to allow readers to interact with your code and results!

Last minute update 🎉

A few hours before publishing this post we merged a contribution from Tom Morrell who works at CalTech’s library that allows you to launch records from CalTech’s Data Repository as well! For example a notebook to analyze traffic to the archive itself: https://mybinder.org/v2/zenodo/10.22002/d1.1250. You should also check out a quick response to this post from Lorena Barba, who describes her group’s approach to creating reproducible, citable bundles for figures in papers, and how Binder+Zenodo DOIs might fit into that workflow!

What’s next?

We are close to closing the loop of fully reproducible computational environments for scientific publication. We’re excited to see journals begin to integrate these workflows with their own publishing pipelines. For example, the Neurolibre project is deploying their own BinderHub and using it alongside their reviewing and archiving process in order to provide more rich
interaction with submitted material.

Each type of repository needs a small amount of custom work to be integrated with Binder. We started with Zenodo because it is well known,
general purpose and integrated with GitHub already. If there is an archive you’d like to see integrated please do stop by this repository and open a new issue or contribute the code to do so directly.

If you’re working with a publisher and are interested in this
please reach out! The Binder community would love to work with you in deploying these open tools to make your published work more open and accessible. Whether it is big or small, we hope that these workflows can make an impact across the publishing landscape, and we’re looking forward to seeing what people do next!

Introducing repo2docker

2018-12-14T18:24:00+00:00

The Binder Project’s repo2docker tool gives data scientists the benefits of containerization technology without needing to learn Docker itself. To make your repository compatible with repo2docker, you only need to add text files that are already present in many repositories. This means that you get the benefits of containerization, a powerful and complex ecosystem, without having to change your workflow.

repo2docker is a lightweight command-line tool written in Python that takes a path or URL to a git repository and creates a suitable docker image for it. To achieve this it follows the steps that a human would take to do so. The steps are:

Inspect the repository for common “configuration” files (like requirements.txt),
From these well-known files infer the Docker commands to run; and
Build a Docker image.

It has a few more tricks up its sleeve, such as automatically installing RStudio for you when it detects that you are using R. Once the image has been built, a Docker container is created and executed, giving you access to the environment in which the repository author wanted the code to be executed. To achieve this, one needs access to two things: repo2docker and a docker daemon (they do not necessarily have to have docker installed on their local computer).

The JupyterHub team just released v0.7 of repo2docker, so we decided to spend a bit of time explaining what it’s all about.

An example repo2docker workflow. In this case, repo2docker is invoked locally. repo2docker is passed a URL to a git repository (https://github.com/norvig/pytudes). It then clones the repository, discovers configuration files in the repo (in this case, requirements.txt), builds a Docker image with this environment installed, and opens a local Jupyter server to explore and run the contents of the repo.

The guiding principles behind repo2docker

repo2docker is meant to be as lightweight and common-sense as possible. The driving principles behind repo2docker are as follows:

Leverage pre-existing workflows in data science as much as possible. This means using standard configuration files (like requirements.txt) instead of requiring people to learn new configuration patterns.
The shareable unit is a repository or directory containing human-readable files. Not a single file (like a notebook) nor a binary blob (like a built docker image). This means that humans can inspect and extend other repositories meant for repo2docker, and that they can manually do what repo2docker does automatically. No black box.
Be workflow agnostic. repo2docker supports many languages and user interfaces, it can run arbitrary shell scripts that are baked into the image, or it can trigger a script to be run each time a person runs the Docker image.
Be extensible and composable. repo2docker should allow for multiple languages, tools, or workflows to be defined in a single GitHub repository. It should also be relatively easy to extend to support new use-cases.
Enable deterministic outputs. We want repo2docker to make it possible for authors to generate the exact same environment from their repository every time, provided that they follow best-practices in computational methods (like providing specific version numbers for packages). repo2docker can build a specific commit, tag, or branch of a repository, which allows for an image to be deterministically built.

How can repo2docker be used?

Over the last 18 months, we have been using repo2docker in production to automatically generate images that run repositories for mybinder.org.
It is used to build around 1000 unique repositories every week. The core functionality has proven itself and is considered production ready.

Over the last year, we’ve seen a few major use-cases come out of repo2docker:

First, it can be used as a part of production systems like BinderHub. BinderHub automatically uses repo2docker to build images that run a user’s environment, and lets them share links that let others interact with the image.

Second, repo2docker can be used to build an image for use with a JupyterHub. For example, teachers have used repo2docker to convert their GitHub repository with course materials into a runnable Docker image that students access via a shared jupyterhub in the cloud.

Finally, repo2docker has been used by individuals who wish to build reproducible images from their local work. repo2docker can optionally run a Jupyter server from within the built image, which makes it possible to verify the results of analyses in an environment that was built solely from the configuration files present in the repository.

What next?

We think that repo2docker serves as a useful tool for the community and that it is an important part of the large reproducible scientific software stack. It gives data scientists the benefits of containerization technology without needing to learn a new tool like Docker. It achieves this by being a lightweight command-line tool written in Python that automates the creation of the environment in which the authors of a piece of software wanted it to be executed.

We’d love to see the repo2docker community grow, and for more
languages, interfaces, use-cases, and workflows to be supported
with repo2docker’s build pack system. Let us know what you think!

repo2docker is primarily maintained by the JupyterHub and Binder teams. If you’d like to get involved with the community or want to learn
more about the tool, reach out! Check out these links for more information:

Note*: some folks might be wondering why we developed repo2docker instead of contributing to a pre-existing containerization tool such as the excellent* source2image project. We take the decision to create new open-source tech very seriously, and wrote a blog post about our decision to do-so in this case: http://words.yuvi.in/post/why-not-s2i/

We Analyzed 1 Million Jupyter Notebooks — Now You Can Too [Guest Post]

2018-03-23T08:58:00+00:00

One of the 1 million+ notebooks we scraped from GitHub in July 2017. This notebook combines code, visualizations, and text to create an effective computational narrative.

This is a guest post on how members of the Jupyter community publish code, visualizations, and text using Jupyter Notebooks. We’re excited the Design Lab is sharing their research and data on the blog.

If you have a post relevant to the community you’d like to share on the Jupyter Blog, please contact us.

-The Jupyter Team

In July 2017, my team in the Design Lab at UC San Diego scraped and analyzed over 1 million Jupyter Notebooks from GitHub. Today I am excited to announce we are making these data publicly available for you to explore! While only a snapshot of one corner of the Jupyter universe, these data provide unique perspective into how people use and share Jupyter Notebooks.

The collection includes over 1 million notebooks as well as metadata about the nearly 200,000 repositories where they lived. The full dataset is nearly 600GB so we have created a smaller 5GB sampler dataset for you to get started. This includes roughly 6,000 notebooks from 1000 repositories.

We originally collected these data to explore how people use narrative text in Jupyter Notebooks. We found many notebooks, even those accompanying academic publications, had little in the way of descriptive text. This is likely because many analysts view their notebooks as personal and messy works-in-progress. On the other hand, many of the notebooks we collected were masterpieces of computational narrative, elegantly explaining complex analyses (one notebook even had more text than The Great Gatsby). We think this spread reflects a tension between data exploration, which tends to produce messy notebooks, and process explanation, in which analysts clean and organize their notebooks for a particular audience.

Over 25% of the 1 million+ notebooks we collected from GitHub had no descriptive text, yet some rivaled classic novels in length.

Beyond simply counting lines and words, we also looked at how authors organized their code and text. For example, most notebooks had markdown headers and nearly a third linked to other resources. Most notebooks had code comments and over a third defined new functions.

Analyzing this data helped us see how people organize notebook code and text.

We will be presenting the full results of our analysis in April at the 2018 ACM CHI Conference on Human Factors in Computing Systems and you can read more about our work in this preprint copy of our paper. In the meantime, our team has moved on to developing tools that take some of the effort out of cleaning and organizing Jupyter Notebooks.

There is so much left to explore in the data we collected. We are excited to see what you do with them! Thank you to the UC San Diego Library for graciously hosting the data. If you encounter a problem downloading them, please open an issue on this GitHub repo.

Rendering Notebooks on GitHub

2015-05-07T19:47:00+00:00

We are pleased to announce that, starting today, and as announced on the GitHub blog, Jupyter/IPython notebook (.ipynb) files will render directly on GitHub. This feature works for notebooks in any of the supported Jupyter programming languages on both public and private repos. This capability will complement nbviewer and make it easier for GitHub users to create, view and share notebooks on GitHub.

Here are some great examples of notebook based content that can now be viewed directly on GitHub:

Cameron Davidson-Pilon’s Probabilistic Programming and Bayesian Methods for Hackers.
Lorena Barba’s Aerodynamics-Hydrodynamics with Python.
Benjamin Laken’s Monsoon Analysis.

We want to thank the GitHub team for bringing this new feature to GitHub. In particular @tclem, @arfon, @sshirokov, and @skalnik have all done an incredible job building this at GitHub. The first discussion of how to bring notebook rendering to GitHub happened almost 18 months ago, and a lot of work made this feature possible today. With over 200,000 notebooks available on GitHub, you already have a lot to read!

We are excited to see how this feature will impact science education and research on GitHub, as well as encourage the adoption of the Jupyter notebook as a standard file format across a wide range of fields. Jupyter notebook files are a JSON-based open document format that supports code and results, narrative text, images, and equations in one file. These documents simplify the creation and communication of computational narratives and make the sharing and replication of scientific results and data analyses simpler.

nbviewer

One key piece of infrastructure in sharing notebooks has been the nbviewer service, which started as a side project of mine in the late summer of 2012. Nbviewer has grown through deep integration with GitHub, with the ability to browse all public users, organizations, repositories, branches, tags, and even Gists. This would not have been possible without @rgbkrk and @bollwyvl, who handle the development and deployment of nbviewer. It would also not have been possible without Rackspace and Fastly, who provide free hosting and fast static asset distribution. Nbviewer has steadily grown to render hundreds of thousands of notebooks every week, and usage is still increasing. Based on this, we believe that rendering on GitHub will be a highly used and appreciated feature.

Although you can now view notebooks directly on GitHub, we are not planning to stop the development of nbviewer. While rendering directly in GitHub repositories is convenient and allows viewing notebooks in private repositories, there are number of reasons that nbviewer will continue to remain an important part of project Jupyter.

First, there are many notebooks not hosted on GitHub. As before, these notebooks can still be viewed on nbviewer. Furthermore, the Jupyter/IPython team are actively working on improving integration with other content providers, building on our experience with GitHub. For example, a recent pull request of @bollwyvl streamlines the process of adding new providers to nbviewer, with proposals for DropBox, GitLab, Google Drive, Stash, and other hosting and collaboration platforms.

Second, because of security concerns, some features available on nbviewer will not be available on GitHub. For example, GitHub will not render any dynamic output display that uses JavaScript, custom CSS, and most custom HTML embedded in Markdown or in outputs. When this is the case, and GitHub cannot fully render your notebooks, you will see an icon that allows you to view the full notebook on nbviewer.

Because nbviewer does not handle authentication, it will continue to support custom CSS, HTML and JavaScript in notebook. Thus, for highly customized or dynamic notebooks, nbviewer will still be the ideal place to share them.

We would love your feedback! If you have comments or find bugs with notebook rendering on GitHub or nbviewer, please open an issue on our nbviewer repo. Also, please send a few tweets to thank all the people without whom this would have not happened.

The Jupyter/IPython Team