Jupyter Blog - Binder

Berkeley Institute for Data Science (BIDS) joins the mybinder.org

2026-04-08T23:34:00+00:00

Berkeley Institute for Data Science (BIDS) joins the mybinder.org federation in partnership with 2i2c

The Berkeley Institute for Data Science (BIDS) is the birthplace of the current iteration of mybinder.org, all the way back in 2017. In 2026, they are back as a member of the federation, joining 2i2c and GESIS in contributing to the cloud costs of keeping mybinder.org running!

The BIDS node is running on the OVH Cloud Provider, joining the two existing nodes (from 2i2c and GESIS) running on Hetzner. By running on a different cloud provider than our existing nodes, we reduce the risk that a cloud provider outage or policy change will take down the entire mybinder.org service. This immediately came into play, as Hetzner is having a lot of trouble with its object storage service, increasing failure rates on mybinder.org. We were able to shift more of our users to the BIDS node seamlessly, reducing disruptions for our end users.

The BIDS node thus temporarily served 50% of our traffic, taking a higher than usual share of the load as we wait for error rates on Hetzner object store to get better. It has served more than 64,000 users so far in the short time it’s been up, and expected to do more.

Graph of mybinder.org sessions across our federation members over the last 4 months, showing the BIDS member taking on more of the load over time as it is brought online. The graph is jagged due to dips in demand over the weekend

This node was brought up in partnership with 2i2c. 2i2c contributed expertise in extending the mybinder.org deployment infrastructure to run on OVH, as well as working with UC Berkeley procurement to ensure that the cloud costs can be paid for on a continuous basis. Many thanks to the ever awesome MinRK (who is now back to working at BIDS!) for a lot of the technical work in taking this through!

Want your organization to join the mybinder.org federation, materially making a difference in resilience and availability of the service, even at just a few hundred dollars a month? Come talk to us on the Jupyter Zulip!

mybinder.org adding request Origin to events archive

2026-02-02T16:27:00+00:00

Binder logo

In February, the mybinder.org service will start tracking the “origin” of requests that launch a Binder session. The website that created a request to mybinder.org will be visible in the Binder analytics archive. This will help us monitor which websites are embedding Binder sessions for impact or abuse reports. It shouldn’t have any implications for private information but we are sharing here for transparency. The only cases where new information will be recorded is websites which embed mybinder.org kernels via tools like Thebe.

When folks use mybinder.org to run sessions, each launch is added to a public events archive. That means that every time anyone launches a session on mybinder.org, the repo and version used and the time of the launch is public information. There is no such thing as a private launch on mybinder.org (this is part of why we don’t support private repos). No information about the user requesting the build is recorded (ip address, user agent, browser info, etc.), though some of that is sent to Plausible and aggregated and anonymized for analytics. Notably, Plausible only covers regular browser visits, not launches via the API.

There are 3 main ways that people launch sessions on mybinder.org:

(the vast majority) — visiting mybinder.org in a browser
embedding kernels from mybinder.org on another website to make e.g. interactive documentation with a tool like Thebe
‘headless’ launches via scripts

Up to now, these events are indistinguishable in the events archive. We’d like to start distinguishing these cases in the events archive to better classify how mybinder.org is used, but that means recording new information we didn’t track before, so we want to communicate clearly about it ahead of time.

We are planning to add the Origin of the launch request to the public events archive at the end of February. This still doesn’t track anything about the user (no ip address, user agent, etc.), but rather the host of the page where the launch request originated. For the vast majority of cases, this is not new information, as the origin is e.g. gesis.mybinder.org (which we already track confusingly as ‘origin’, though this is not the request Origin, it is the federation member where the launch event takes place). Notably, case 1 still includes any regular browser links to mybinder.org/v1/gh/repo/name/HEAD. We don’t record where those come from (that would be Referer).

Where this changes is cross-origin requests (case 2), that is, pages which embed kernels from mybinder.org, such as documentation websites that use mybinder.org to embed kernels from mybinder.org via tools like Thebe to make executable documentation (which is rad!). These requests will now include the Origin of the request, which would be the hostname (e.g. https://course.spacy.io ), not the full page.

Script launches (case 3) typically lack this information, so will usually have null data in the new field. But of course, scripts can do whatever they want, so this isn’t guaranteed to be accurate for malicious or deliberately spoofing scripts.

If you want to have a private, not-recorded use of Binder, you must run your own BinderHub instance. If you want to suggest that we not do this, or do it differently, please let us know in the proposal discussion. You can also check the privacy policy for more information. We plan to apply the change at the end of February, 2026.

Simplifying and speeding up Binder builds with BuildKit

2025-03-03T14:05:00+00:00

The Binder Project allows users to build reproducible, sharable environments for interactive computing. To accomplish this, Binder uses a tool called repo2docker to generate an executable Docker image using the Reproducible Execution Environment Specification.

The first time a repository is launched on Binder, repo2docker must build the reproducible environment for it. This process can take a long time because of all the dependencies that need to be installed and turned into the image. As a result, Binder launches can feel slow and clunky, which is a poor UX for workflows that are designed around quick interactive sessions.

repo2docker was built several years ago, and followed patterns that were commonplace at the time. However, in the past few years the Docker community has made significant advances in optimizing the image building process. One-such improvement is the creation of BuildKit, a replacement for Docker’s historical build system that is much more sophisticated. However, repo2docker hasn’t leveraged these improvements because it was still using the original Docker Build system.

So, we’ve decided to spend a few cycles modernizing repo2docker’s image building logic by using the more modern BuildKit API (via ‘docker buildx build’). This allows for optimizations like build parallelization, better build caching, and supporting some Dockerfile features that Binder didn’t support earlier (particularly, COPY --chown). It also lays a foundation for significantly simplifying the repo2docker build infrastructure and leveraging more of BuildKit’s parallelization functionality. For example, we’d like to leverage BuildKit’s Kubernetes driver which distributes builds much more efficiently and in parallel.

Authors of Binder repositories won’t need to take any action¹, and they’ll simply notice that mybinder.org (and any other community-run BinderHub instance) will be a bit snappier at building images.

If you’d like to learn more about the changes that enabled this, check out this mybinder.org pull request which has links to the repo2docker pull requests that added this functionality. We’re excited keep improving Binder, and are hopeful that this makes the experience of using mybinder.org and community Binders a little bit better.

Authors and acknowledgements

Yuvi Panda is a co-founder and the Technical Lead at 2i2c. He is passionate about building participatory open infrastructure for scientific & educational use cases. He is a Project Jupyter team member primarily focused on infrastructure related projects (JupyterHub, Binder, etc). He also wrote most of the code for this docker buildx transition, and shepherded it through to deployment on mybinder.org as well. He is ex-Wikimedia and ex-GNOME. Let’s eliminate accidental complexities wherever we find them.

Chris Holdgraf is a co-founder and the Executive Director of 2i2c. He is on the Executive Council of Project Jupyter, and co-leads the JupyterHub and Binder team as well as the Jupyter Book team). He was previously a post-doctoral researcher in the Department of Statistics at UC Berkeley, and a Community Architect with the Division of Data Science at Berkeley. He’s interested in using open infrastructure to support interactive computing workflows in research and education.

Many thanks to @minrk, @manics, and @consideRatio for their help reviewing and shaping this work.

Footnotes

¹ Unless they were relying on undocumented implementation details of the old builder — in particular, the presence of a /.dockerenv file to detect if you are running in repo2docker scikit-learn/scikit-learn#30835 has an example.

Curvenote joins the mybinder.org federation

2024-02-22T16:19:00+00:00

Curvenote is now a part of the mybinder.org federation. By hosting a BinderHub instance running on the company’s infrastructure, additional capacity is available on the https://mybinder.org service.

curvenote.com/for/journals

The mybinder.org project provides a critical, free public service for scientists, researchers and educators worldwide. The ability to spin up a Jupyter server to run someone’s research code or run a class without local configuration is something that many academics and educators rely on.

Mybinder.org runs on a set of open source software components and standards, that not only powers the service but that also allows us to make solid steps forward in reproducibility in scientific research, namely REES and open source software components (like repo2docker) .

This is something we’re keen to shout about as Curvenote runs numerous small scale, independent, BinderHubs to power its journal infrastructure for notebook based publishing for reproducible scientific journals with BinderHub, Jupyter and MyST Markdown (https://mystmd.org) being important components of that.

Over the last few months a lot of work has gone into bringing up the Curvenote BinderHub instance on AWS. This is mainly because in the process, the AWS workflow based deployment has been pushed forward by Simon Li (@manics, University of Dundee, JupyterHub Core Team), solving many previous roadblocks around EKS (Elastic Kubernetes Service) usage and paving the way for easier future AWS based deployments (watch out for a future post from Simon with all the technical details).

It’s been great to collaborate with Simon during that time and get exposure to the details of BinderHub deployment on AWS.

The mybinder.org service makes a significant positive impact on public education, science and research. This impact is directly supported by the volunteer time of the BinderHub core team and the federation member organizations. Curvenote’s instance adds capacity to the service and a broader group of organizations contributing to the organization would lead to a more resilient public service, that is less prone to disruption when a single federation member leaves. If your organization has the resources to run and maintain a BinderHub, or makes heavy use of the mybinder.org service, encourage your organizations to run an instance and join the federation.

Mybinder.org reducing capacity

2023-04-25T07:28:00+00:00

Funding is running out for the largest member of mybinder.org, which has been sponsored by the Google Open Source Program Office. gke.mybinder.org will shut down and there will be a reduction in capacity at the end of April, 2023 (end of this week).

Key points:

Our funds on gke.mybinder.org, the biggest part of the federation, will run out in April. Google has decided not to provide more credits.
Traffic previously sent to GKE will mostly be delivered to OVHcloud, which has less capacity. We are also working to deploy new federation members to spread the load.

Service reductions to expect:

This may come with an increase in errors and failures, as GKE has been the largest and most stable federation member
We are currently looking at reducing the total capacity of Binder by 60%, so you may get rate limit or capacity errors
We will start more aggressively deleting older cached builds, as storage costs have increased.

We are still working out the details of the shutdown and deploying new federation members, but mybinder.org users should expect disruption to the mybinder.org service and to see an increase in messages that the service is at capacity in the short term.

The plan:

Thursday: gke.mybinder.org will be removed from the federation (user sessions not disrupted, but no new sessions will start there)
Friday: gke.mybinder.org will be shutdown, and mybinder.org migrated to a smaller cluster

You can help us keep mybinder.org running by donating to Binder or encouraging your organization to join the BinderHub Federation.

We are grateful to the Google Open Source Program for their funding thus far, and appreciate the continued support of continuing federation members, OVHCloud and GESIS, which enable us to keep mybinder.org going without shutting down completely when one funding source goes away.

Updating defaults on mybinder.org

2023-03-09T08:31:00+00:00

This week, we are deploying some updates to repo2docker, the image-building tool that underpins mybinder.org. This may affect your repos. Please get in touch if you have problems! Hopefully this will make things better, more stable, and less surprising overall, but change always breaks something, however hard we work to keep things working.

To highlight the changes:

Python 3.10 is now the default Python version
Python 3.11 is now supported, and new versions of Python should be supported more quickly
If a repo requests an old version of Python (≤ 3.6), it will be installed in a separate environment, as we’ve been doing with Python 2.7 for a long time.
The server implementation is switched from jupyter-notebook to jupyter-server

More detail on each change, the reasons behind them, and their consequences below.

Python 3.11 and separate envs for legacy Python support

When building an image, repo2docker installs packages in one or two environments — the kernel environment where your code runs, and the server environment, which runs the Jupyter server. If you’re using a reasonably supported version of Python, these are actually the same environment. This is the vast majority of repositories on mybinder.org. In order to function properly, we have certain requirements of what goes in the ‘server’ environment. Eventually, those requirements start making it difficult to setup the server environment with an outdated version of Python, as packages march ahead with their minimum required Python (The widely adopted NEP 29 suggested dropping Python 3.7 support in December, 2021). Our solution for this has been to separate the kernel environment from the server environment, so they don’t create a conflict. Until recently, only Python 2 was handled this way (most packages dropped Python 2 support years ago). As part of adding support for Python 3.11, we have expanded this behavior to include Python ≤3.6, so if you are using a ‘legacy Python’, currently defined to be Python ≤3.6, your requested packages will be installed in the kernel environment instead of the server environment. This should only have a noticeable effect if you are installing something with server extension components , which includes anything modifying the JupyterLab or notebook interfaces— those will not be available in the server environment. To address this, you will have to upgrade Python or handle the server extension installation in postBuild . PR adding Python 3.11 and adding Python ≤3.6 to legacy Python versions.

Default Python is now 3.10

The default Python for repo2docker and thereby mybinder.org is now Python 3.10, upgraded from 3.7. 3.10 will be used when a repository does not specify which version of Python to use. Following Python’s own annual release cadence, we plan to update this more regularly in the future, lagging behind Python itself by a version or two. This has no effect on your repos if they specify a Python version, which is why we encourage you to specify your Python version! If you don’t, then your Python version will change over time, probably about once a year. PR switching the default Python.

Now launching Jupyter Server

The default UI for mybinder.org has been JupyterLab for some time, with the classic notebook still available if you specify a url of /tree/or /notebooks/mynb.ipynb. When we rolled out that change, we kept the default server as the jupyter-notebook server, for improved compatibility with existing server extensions. As the newer Jupyter Server has taken over, the situation is becoming reversed — using the older notebook server is now the cause of compatibility issues with server extensions, so we are switching the server to launch jupyter-server instead of jupyter-notebook. The classic UI is still available at the same URLs, thanks to nbclassic. This should have no visible effects, as jupyter-server and nbclassic make a great deal of effort to keep classic notebook extensions working, but some extensions may need an update for jupyter-server compatibility if they haven’t been updated already. Because of where the server is specified, there is no supported way for images to express a preference for the deprecated notebook server (PR switching the default server command.

MyBinder @ OVHcloud

2022-11-23T11:28:00+00:00

OVHcloud is a long-time supporter of the Jupyter project. In the past few years, they have provided computing resources to several Jupyter-related endeavors to support its sustainability.

In 2019, Project Jupyter and OVHcloud started collaborating on the hosting of the MyBinder service. With tens of thousands of sessions daily, MyBinder was increasingly dependent on a single instance on a single cloud provider. The offer of OVHcloud triggered a community effort to make MyBinder truly multi-cloud. This is how the Binder Federation was created, with OVHcloud as its first member. Since then, the Binder Federation has grown with new members gracefully hosting a portion of the traffic. OVHcloud also hosts the nbviewer.org service.
When the global pandemic hit in 2020, the JupyterCon organizers decided to pivot from an in-person event set to happen in Berlin to a fully online experience. Lorena Barba, the general chair of the committee, envisioned an experience for talks and tutorials in which attendees could interact with the technology. OVHcloud stepped up as the Platinum sponsor, gracefully providing the infrastructure for the various deployments of Jupyter required for the conference, including GPUs for the online tutorials requiring them.

A new partnership with NumFOCUS

Today, we are thrilled to announce that OVHcloud is partnering with NumFOCUS to provide infrastructure to affiliated projects.

Two pilot projects will benefit from this new agreement: Pandas and Jupyter.

On the Jupyter side, these new resources will be used to increase the OVHcloud hosting of MyBinder, thereby helping the project’s sustainability in the long term, continuing to avoid dependency on a single cloud provider. A significant proportion of the traffic of the Binder federation will now be handled by the OVHcloud deployment.

Acknowledgement

We are grateful to Kevin Amil, Marie-Christine Ribeiro, Marie Hering, and Mael Le Gal from OVHcloud for facilitating this new partnership and helping with the migration to the new infrastructure.

We are grateful to OVHcloud for their continued support to the Jupyter project and the NumFOCUS foundation.

A 2019 retrospective from the Binder Project

2020-01-15T17:48:00+00:00

2019 was a busy year for the Binder and JupyterHub projects — each saw growth in both their community and technology. Now that the year has wrapped up, it is a good time to reflect on some of the highlights from the year.

Overall, 2019 was about improving the robustness, stability, and team dynamics around the JupyterHub and Binder projects, as well as connecting these projects with other tools and services in the open source community. Here are a few things that we are most-excited about.

More people are using mybinder.org

The Binder Federation is a collection of BinderHubs accessible from mybinder.org. This deployment is run as a public service and a demonstration of BinderHub, the underlying technology of the Binder project. This deployment is run on a volunteer basis by Binder community members, and is supported through grants and donations in infrastructure from project stakeholders. In 2019, the user base of mybinder.org grew from around 70,000 users per week to around 100,000 users (a growth of nearly 40%). mybinder.org is being used for teaching classes, sharing reproducible analyses, creating interactive documentation and narratives, and much more. We’re astonished at the rapid growth of Binder-ready repositories, and we’re excited to see what the community creates next.

Weekly user sessions at mybinder.org. Here you can see a typical pattern of activity over the course of a year. There are dips in activity over the summer and winter months, reflecting reduced activity from academic institutions.

JupyterHub reaches 1.0

JupyterHub, the underlying technology that provides interactive computing sessions to multiple users, is now at 1.0 status. The JupyterHub Python application was written several years ago, and reaching 1.0 reflects the work of dozens of open source contributors over time. JupyterHub is now a robust and stable application, having been used at smaller scales (think 5–10 people running on a single VM) as well as much larger scales (think 5,000 students running Jupyter sessions for a class).

The JupyterHub logo

BinderHub is out of beta

BinderHub, the kubernetes-based technology that powers mybinder.org, also came out of Beta this year. This reflects the fact that BinderHub is a battle-hardened application that can provide a stable service over time. mybinder.org runs nearly 100,000 sessions a week, and requires minimal maintenance time from the Binder’s projects team of volunteer operators.

The BinderHub federation is launched

The Binder Project envisions a world in which technology can be used in vendor-agnostic and decentralized ways. BinderHub runs on Kubernetes, which can be deployed on a variety of cloud and local infrastructure. While the Binder team runs one BinderHub deployment at mybinder.org, our goal has always been to see other organizations running their own BinderHubs. This year, we went one step beyond this by launching the BinderHub Federation. This is a collection of research and technology organizations that combine their expertise and computational resources to power mybinder.org. When users visit mybinder.org, they are now directed to one of several BinderHub instances. This makes mybinder.org more robust, and grows the number of organizations that utilize the project’s technology for their communities. A BIG THANKS goes out to Google, OVH, GESIS, and the Turing Institute for supporting the Binder Federation.

The (rough) location of each BinderHub deployment in the mybinder.org federation

Binder connects with open science services

Another goal of Binder is to be a part of the solution to more transparent, sharable, reproducible computational work. This means plugging in to other ecosystems and projects in order to leverage the broader open science community. This year we saw a number of new connections with other services. BinderHub now supports links that point directly to Zenodo and Dataverse repositores, and we are working on a few other integrations in the coming months. This means that projects utilizing these resources will be able to share reproducible and interactive links to their work out-of-the-box.

Binder now works with Zenodo repositories!

The Binder community grows

The Binder project’s most important asset is its people — this is a collection of volunteers spread across the world and from a variety of organizations. Binder community members do a variety of things — from working on technology, to teaching others how to make their work more reproducible, to participating in community discussions, to maintaining and debugging Binder tech. There is also a “core team” of Binder members that dedicates a significant part of their time to supporting the project. In 2019, we saw several new members join the core team, as well as a general growth in the Binder community. Welcome to all of our new team members! You can find a list of our current team members here.

We welcome our first Contributor in Residence

One challenge with running large, open projects is that resources tend to be scarce. The Binder Project has no formal project funding, and must find ways to both grow its technology as well as run mybinder.org on resources that are donated from its community. One thing that often suffers as a result is the maintenance and general improvement of our open source technology. This work is often under-appreciated, difficult, and unlikely to happen with purely volunteer labor.

For this reason, the Binder project decided to apply for the CZI Essential Open Source grant series. We proposed the creation of the “Binder Contributor in Residence” position — an annual contractor position that pays a member of the Binder community to do many of the daily things that are crucial for the project’s growth. We are excited to have Georgiana Dolocan as our first contributor in residence, and look forward to where this project will go in 2020.

Many thanks to CZI for their support of the Binder and JupyterHub projects in 2020!

There are more BinderHub deployments

The Binder Project aims to create technology that is deployable anywhere so that other organizations can support their communities with Binder infrastructure. In 2019 we ran a number of training sessions for how other groups can run their own BinderHub. In particular, the Turing Institute ran several workshops that had attendees up-and-running with their own functioning BinderHubs.

Thanks to our community

As you can see, 2019 was a busy and exciting year for the Binder community. As a final note, we want to say thanks to all of you who have supported Binder in one form or another over the years. Binder is a project run by the community, for the community. It wouldn’t be possible without all of your hard work and friendly faces, thanks! We look forward to what’s coming next in 2020!

The JupyterHub and Binder Contributor in Residence!

2019-11-26T19:52:00+00:00

The Chan Zuckerberg Initiative and Project Jupyter are teaming up to pilot a new role in the Jupyter community: the JupyterHub Contributor in Residence

Thanks to funding from the CZI Essential Open Source Software for Science initiative we are welcoming Georgiana Elena as our first Contributor in Residence!

Managing a codebase that spans many repositories and sub-projects as well as operating mybinder.org is a significant effort. Over time we have identified a need for dedicated support to ensure that activity and communication within these repositories is efficient and productive. Driven by this need and inspired by the Django Fellowship model, we shaped the idea of a JupyterHub/Binder Contributor in Residence (CIR).

Thanks to the Chan Zuckerberg Initiative, a Contributor in Residence within the community is now possible. Starting this year, CZI launched the Essential Open Source Software for Science grant program. It is aimed at helping open source communities that make up the core foundation of the scientific stack. We applaud CZI in providing support for core infrastructure across the sciences, and we’re honored to be part of CZI’s vision of funding maintenance, growth, development, and community engagement for open-source projects.

In this post we’ll introduce the community to our first Contributor in Residence (Georgiana) and her mentor (Tim)! The rest of this post is in the form of a conversation between Georgiana and Tim. We hope you enjoy it and that it let’s you get to know the team a bit better.

Tim: Hello Georgiana. Congrats on becoming the inaugural CIR. We’ve been lucky to have you already as a contributor on JupyterHub especially your work adding support for using Traefik as a proxy.
Geo: Hey Tim! Thank you, it’s been a really amazing experience to work on JupyterHub. I’m really excited for this next year’s journey.
Tim: You are the very first Contributor in Residence. Can you explain a little what this means?
Geo: The CIR role is about helping make the little things around the projects be amazing. Little things like answering newly opened issues, guide people towards the right place for discussion, improve bits of documentation or code, helping with releases, give people the right tools to make great contributions. And I have a great team of existing contributors to work with.
Tim: That’s right! My role in all this is to be a mentor and guide. JupyterHub and Binder have a small team that does a good job looking after the main repositories, but the universe of JupyterHub and Binder repositories is huge. So sometimes things get overlooked or missed. For example Pull Requests or Issues in one of the less-popular repositories can go for a long time without getting attention. This may hold up progress somewhere else, and people have a bad experience contributing to the project.
Geo: Hopefully we can improve on this a bit so that more people have a great experience when contributing.

Tim: Let’s find out a bit more about you. What did you do before getting involved with JupyterHub?
Geo: I studied computer science in Bucharest spending the last semester developing my bachelor’s project as an Erasmus exchange student in Madrid. During summers I worked in the industry, doing a few internships abroad. I tried to gain as much practical experience as possible, to get in touch with different people and cultures, and interning seemed like the right move.
Tim: How did you get involved with JupyterHub?
Geo: I got involved with JupyterHub thanks to the amazing Outreachy community. The Outreachy program introduced me to the world of opensource and guided me through my first steps. I now feel very lucky to have applied for the Outreachy internship at the same time that JupyterHub came up with two amazing projects. I chose JupyterHub because I “clicked” with the TraefikProxy project and I was happy to find a very welcoming community that made contributing a very pleasant and not at all a scary experience.
Tim: Have you turned into someone who spends all their time contributing or do you still have other hobbies?
Georgiana: Because I currently live in the busiest/loudest city of Romania, I love to escape it at the end of the week and spend some time in nature. My favorite place to be is the village I grew up in, where I recharge my batteries walking my dogs and cooking with my mother and sister. I like painting and photography and sometimes I do both using the photos I take as an inspiration for my paintings.

Tim: Moving on to the things you will be doing as CIR, have you already picked out something to get started with?
Geo: At first, I will be focusing on trying to get an overview of what issues are feature requests, support questions and which have been solved already. I will also spend some time getting familiar with more of the JH/Binder projects. And what better way to learn about a project than to try and use it, and to be a first time contributor.
Tim: If you had to name the top three goals you hope to achieve while being contributor in residence, what would those be?
Geo: From the perspective of someone with still a lot to learn and figure out, I would be extremely happy if during this year I will grow to be the CIR this community needs. I would love to help improve the contributing experience, increase the overall productivity of the projects and become a better contributor myself.

Tim: How can people interact with you?
Geo: People can use the usual community communication channels, GitHub, Gitter and Discourse to provide feedback or ask questions and I’ll do my best to help out.
Tim: Now that we have a CIR, do people still need to contribute to the projects?
Geo: Definitely! People are always welcome to contribute. The CIR role wasn’t created to replace anything, but to be an extra pair of hands helping the daily activities around the community become more efficient.
Tim: How can people stay engaged with the CIR program or the broader JupyterHub community moving forward?
Geo: As the CIR, I am a team member just like anyone else, and the best way to stay connected is to connect is through our team channels — we have team discussions in the Jupyter Community Forum, and team information in the JupyterHub Team Compass repository (including monthly team meetings that anyone is welcome to join).

How we automated mybinder.org dependency upgrades in 10 steps

2019-07-17T15:34:00+00:00

BinderHub and repo2docker are key components of the service at mybinder.org. In order to give Binder users the best experience, the Binder SRE team must continuously upgrade the version of these tools that mybinder.org uses. To avoid merging in massive updates at irregular intervals, it is desirable to merge updates in frequent intervals of smaller changes in order to more easily identify any breaking changes from the dependency upgrades.

While this process only takes a few minutes following processes outlined in the “Site Reliability Guide,” it is prone to human error (e.g., remembering to use the right SHA in upgrading the packages), and the team must remember to regularly do it in the first place. In the interest of automation, the Binder team decided to use a bot to relieve this burden, and we’ve decided to highlight its functionality in this blog post!

What does the mybinder.org upgrade bot do?

The upgrade bot should automatically update the versions of BinderHub and repo2docker that are deployed on mybinder.org. These are defined in the mybinder.org helm chart. To check whether an upgrade is needed, we want the bot to first “diff” the latest commit hash for both repo2docker and BinderHub repos against the deployed versions in the mybinder.org repo. If either or both are different, the upgrade bot does the following:

Fork the mybinder.org-deploy repo
Clone the fork locally
Checkout a new branch for the bump
Make the appropriate edit to update the commit hash in the mybinder.org fork repo
Add and commit the change
Push to the branch in the forked repo
Create a PR to the main mybinder.org repo
Remove the locally cloned repo

Additionally, it would be ideal if the bot could update an existing PR instead of creating new ones for the version bumps. We’d also like to provide some information in the comments of the PR as to what high level changes were made so we have some idea about what we’re merging in.

Here’s what we’re aiming for. The PR body:

The PR diff:

Now that we’ve broken it down a bit, let’s write up some Python code. Once we have a functioning script, we can worry about how we will run this in the cloud (cron job vs. web app).

Writing the bot

If you don’t care about the step-by-step, you can skip to the final version of the code.

In the interest of linear understanding and simplicity for a bot-writing tutorial, the step-by-step below will not write functions or classes but just list the raw code necessary to carry out the tasks. The final version of the code linked above is one way to refactor it.

Step 1: Retrieve current deployed mybinder.org dependency versions

The first step is to see if any changes are necessary in the first place. Fortunately, @choldgraf had already made a script to do this.

To find the current live commit SHA for BinderHub in mybinder.org, we simply check the requirements.yaml file. We’ll need Python’s yaml and requests modules to make the GET request and parse the yaml in the response. Note that this is also conveniently the file we’d want to change to upgrade the version.

from yaml import safe_load as load
import requests

url_requirements = "https://raw.githubusercontent.com/jupyterhub/mybinder.org-deploy/master/mybinder/requirements.yaml"
requirements = load(requests.get(url_requirements).text)
binderhub_dep = [ii for ii in requirements['dependencies'] if ii['name'] == 'binderhub'][0]
bhub_live = binderhub_dep['version'].split('-')[-1]
print(bhub_live)

Similarly, for repo2docker, we check the mybinder.org values.yaml file:

url_helm_chart = "https://raw.githubusercontent.com/jupyterhub/mybinder.org-deploy/master/mybinder/values.yaml"
helm_chart = requests.get(url_helm_chart)
helm_chart = load(helm_chart.text)
r2d_live = helm_chart['binderhub']['config']['BinderHub']['build_image'].split(':')[-1]
print(r2d_live)

Let’s store these SHAs in a dictionary we can use for later reference:

commit_info = {
               'repo2docker': {},
               'binderhub': {}
              }
commit_info['binderhub']['live'] = bhub_live
commit_info['repo2docker']['live'] = r2d_live
print(commit_info)

Step 2: Retrieve latest commits from the dependency repos

When we get the latest commit SHAs for repo2docker and BinderHub, we need to be careful and make sure we don’t automatically grab the latest one from GitHub. The travis build for mybinder.org looks for the repo2docker Docker image from DockerHub, and the latest BinderHub from the JupyterHub helm chart.

Let’s get the repo2docker version first:

url = "https://hub.docker.com/v2/repositories/jupyter/repo2docker/tags/"
resp = requests.get(url)
r2d_master = resp.json()['results'][0]['name']
print(r2d_master)

Now we can do BinderHub:

url_helm_chart = 'https://raw.githubusercontent.com/jupyterhub/helm-chart/gh-pages/index.yaml'
helm_chart_yaml = load(requests.get(url_helm_chart).text)
# sort by date created
updates_sorted = sorted(helm_chart_yaml['entries']['binderhub'], key=lambda k: k['created'])
bh_master = updates_sorted[-1]['version'].split('-')[-1]
print(bh_master)

Let’s add these to our dictionary too:

# add to commit_info dictionary
commit_info['repo2docker']['latest'] = r2d_master
print('repo2docker', commit_info['repo2docker']['live'], commit_info['repo2docker']['latest'])
commit_info['binderhub']['latest'] = bh_master
print('binderhub', commit_info['binderhub']['live'], commit_info['binderhub']['latest'])
print(commit_info)

Great, now we should have all the information we need to determine whether an update needs to be made or not, and what the new commit SHA should be!

Step 3: Fork mybinder.org repo

If we determine an upgrade for the repo is necessary, we need to fork the mybinder.org repository, make the change, commit, push, and make a PR. Fortunately, the GitHub API has all the functionality we need! Let’s just make a fork first.

If you have permissions to a bunch of repos and organizations on GitHub, you may want to create a new account or organization so that you don’t accidentally start automating git commands through an account that has write access to so much, especially while developing and testing the bot. I created the henchbot account for this.

Once you know which account you want to be making the PRs with, you’ll need to create a personal access token from within that account. I’ve set this as an environment variable so it isn’t hard-coded in the script.

import os
TOKEN = os.environ.get('HENCHBOT_TOKEN')
for repo in ['binderhub', 'repo2docker']:
    if commit_info[repo]['live'] != commit_info[repo]['latest']:
        res = requests.post('https://api.github.com/repos/jupyterhub/mybinder.org-deploy/forks',
			    headers={'Authorization': 'token {}'.format(TOKEN)})

Using the API for a post request to the forks endpoint will fork the repo to your account. That’s it!

Step 4: Clone your fork

You should be quite used to this! We’ll use Python’s subprocess module to run all of our bash commands. We’ll need to run these within the for-loop above.

        subprocess.check_call(['git', 'clone', 'https://github.com/henchbot/mybinder.org-deploy'])

Let’s also cd into it and check out a new branch.

        os.chdir('mybinder.org-deploy')
        subprocess.check_call(['git', 'checkout', '-b', '{}_bump'.format(repo)])

Step 5: Make the file changes

Now we need to edit the file like we would for an upgrade.

For repo2docker, we edit the same values.yaml file we checked above and replace the old SHA (“live”) with the “latest”.

        if repo == 'repo2docker':
            with open('mybinder/values.yaml', 'r') as f:
	        values_yaml = f.read()
            updated_yaml = values_yaml.replace(
			   "jupyter/repo2docker:{}".format(commit_info[upgrade]['live']),
			   "jupyter/repo2docker:{}".format(commit_info[upgrade]['latest']))   
	    fname = 'mybinder/values.yaml'
	    with open(fname, 'w') as f:
	        f.write(updated_yaml)

For BinderHub, we edit the same requirements.yaml file we checked above and replace the old SHA (“live”) with the “latest”.

        elif repo == 'binderhub':
            with open('mybinder/requirements.yaml', 'r') as f:
                requirements_yaml = f.read()
            updated_yaml = requirements_yaml.replace(
                           "version: 0.2.0-{}".format(commit_info[upgrade]['live']),
                           "version: 0.2.0-{}".format(commit_info[upgrade]['latest']))   
            fname = 'mybinder/requirements.yaml'
            with open(fname, 'w') as f:
                f.write(updated_yaml)

Step 6: Stage, commit, push

Now that we’ve edited the correct files, we can stage and commit the changes. We’ll make the commit message the name of the repo and the compare URL for the commit changes so people can see what has changed between versions for the dependency.

        # use var fname from editing step
        subprocess.check_call(['git', 'add', fname])
        
        if repo == 'repo2docker':
            commit_message = 'repo2docker: https://github.com/jupyter/repo2docker/compare/{}...{}'.format(
                              commit_info['repo2docker']['live'],commit_info['repo2docker']['latest'])
        elif repo == 'binderhub':
            commit_message = 'binderhub: https://github.com/jupyterhub/binderhub/compare/{}...{}'.format(
                              commit_info['binderhub']['live'], commit_info['binderhub']['latest'])

        subprocess.check_call(['git', 'config', 'user.name', 'henchbot'])
        subprocess.check_call(['git', 'config', 'user.email', 'henchbot.github@gmail.com'])
        subprocess.check_call(['git', 'commit', '-m', commit_message])
        subprocess.check_call(['git', 'push', 'https://henchbot:{}@github.com/henchbot/mybinder.org-deploy'.format(TOKEN), repo+'_bump'])

Awesome, we now have a fully updated fork ready to make a PR to the main repo!

Step 7: Make the body for the PR

We want the PR to have a nice comment explaining what’s happening and linking any helpful information so that the merger knows what they’re doing. We’ll note that this is a version bump and link the URL diff so it can be clicked to see what has changed.

        if repo == 'repo2docker':
            compare_url = 'https://github.com/jupyter/repo2docker/compare/{}...{}'.format(
                           commit_info['repo2docker']['live'], 
                           commit_info['repo2docker']['latest'])
            body = '\n'.join(['This is a repo2docker version bump. See the link below for a diff of new changes:\n', compare_url + ' \n'])

        elif repo == 'binderhub':
            compare_url = 'https://github.com/jupyterhub/binderhub/compare/{}...{}'.format(
                           commit_info['binderhub']['live'], 
                           commit_info['binderhub']['latest'])
            body = '\n'.join(['This is a binderhub version bump. See the link below for a diff of new changes:\n', compare_url + ' \n'])

Step 8: Make the PR

We can use the GitHub API to make a pull request by calling the pulls endpoint with the title, body, base, and head. We’ll use the nice body we formatted above, call the title the same as the commit message we made with the repo name and the two SHAs, and put the base as master and the head the name of our fork. Then we just make a POST request to the pulls endpoint of the main repo.

        pr = {
              'title': '{}: {}...{}'.format(repo,
                                            commit_info[repo]['live'],
                                            commit_info[repo]['latest']),
              'body': body,
              'base': 'master',
              'head': 'henchbot:{}_bump'.format(repo)
              }
        res = requests.post('https://api.github.com/repos/jupyterhub/mybinder.org-deploy/pulls',
                             headers={'Authorization': 'token {}'.format(TOKEN)}, json=pr)

Step 9: Confirm and merge!

If we check the mybinder.org PRs, we would now see the automated PR from our account!

Step 10: Automating the script (cron)

Now that we have a script we can simply execute to create a PR ($ python henchbot.py), we want to make this as hands-off as possible. Generally we have two options: (1) set this script to be run as a cron job; (2) have a web app listener that gets pinged whenever a change is made and executes your script as a reaction to the ping.

Given that these aren’t super urgent updates that need to be made seconds or minutes after a repository update, we will go for the easier and less computationally-expensive option of cron.

If you aren’t familiar with cron, it’s simply a system program that will run whatever command you want at whatever time or time interval you want. For now, we’ve decided that we want to execute this script every hour.

Cron can be run on your local computer (though it would need to be continuously running) or a remote server. I’ve elected to throw it on my raspberry pi, which is always running. Since I have a few projects going on, I like to keep the cron jobs in a file.

$ vim crontab-jobs

You can define your cron jobs here with the correct syntax (space-separated). Check out this site for help with the crontab syntax. Since we want to run this every hour, we will set it to run on the 0 minutes, for every hour, every day, every month, every year. We also need to make sure it has the correct environment variable with the GitHub personal access token we created, so we’ll add that to the command.

0 * * * * cd /home/pi/projects/mybinder.org-upgrades && HENCHBOT_TOKEN='XXXXX' /home/pi/miniconda3/bin/python henchbot.py

Now we point our cron to the file we’ve created to load the jobs.

$ crontab crontab-jobs

To see our active crontab, we can list it:

$ crontab -l

That’s it! At the top of every hour, our bot will check to see if an update needs to be made, and if so, create a PR. To clean up files and handle existing PRs, in addition to some other details, I’ve written a few other functions. It is also implemented as a class with appropriate methods. You can check out the final code here.

The International Binder Federation

2019-07-01T09:11:00+00:00

About two years ago, the Binder project evolved into the community led project that it is today. The deployment at mybinder.org was upgraded to use
BinderHub, a scalable open-source web application that runs on Kubernetes and provides free, sharable, interactive computing environments to people
all around the world.

In the ensuing years, the Binder community has grown considerably.
Now people use the public deployment at mybinder.org around 100,000 times each week, and there are around 8,000 unique repositories compatible with Binder. Binder links now work with multiple repository providers, such as GitHub, GitLab, BitBucket, and even Zenodo! This means that providing enough compute power for everyone is a challenge.

This is why today we are happy to announce that mybinder.org is now backed by two clusters hosted by two different cloud providers: Google Cloud and OVH. This is the beginning of the International Binder Federation.

How did we get here? Alongside mybinder.org’s growth we’ve seen growth in another part of the Binder ecosystem: people deploying a BinderHub for their own groups. In the last year, we have seen BinderHubs deployed for large-scale earth analytics with the Pangeo project, for communities of best-practices in open science with The Turing Way, and for specific domains such as the social sciences like GESIS.

As more BinderHubs are deployed, we realized that there was an opportunity to leverage the strengths and resources of the community to improve the large, public BinderHub deployment at mybinder.org. Instead of a single BinderHub run by the Binder team, we could build a network of BinderHubs that shares the load and keeps mybinder.org stable and quick.

OVH Joins the mybinder.org Federation

Today, we are thrilled to announce that the Binder Project now has a world-wide federation of BinderHubs powering mybinder.org. We’ve partnered with OVH, a cloud hosting company based in Europe that is supportive of open projects such as Jupyter and Binder.

Through the partnership with OVH, all traffic to mybinder.org will now be split between two BinderHubs - one run by the Binder team, and another run by a team of open-source advocates at OVH. They have generously offered their resources and computing time to allow Binder to serve the scientific and educational communities.

The mybinder.org federation currently has two BinderHubs, but this network can (and will!) grow as more groups offer to connect their own BinderHubs to the network.

What does this mean?

So what has changed for you, the user? Probably not much. The
biggest difference you’ll notice is that landing at mybinder.org
will now redirect you to one of two places:

gke.mybinder.org is the BinderHub hosted on the Google Cloud Platform
ovh.mybinder.org is the BinderHub hosted on the OVH platform in France.

Other than this, your experience should be the same. You might even notice an improvement in speed and load times because the BinderHub you’re using has a little bit less traffic on it 😀.

As always, we are pushing this out as soon as we think it is useful. You can help us to scale this model and make it rock solid by reporting weird things you notice. Expect tweaks over the next few weeks based on feedback from users. Let us know about the things you like, dislike or have questions about at https://discourse.jupyter.org/t/the-binder-federation/1286.

Why is this a big deal?

We think that having a federation of BinderHubs behind mybinder.org is pretty cool, for a few different reasons. First, it demonstrates the dedication that the Binder community has towards building tools that anybody can deploy — we certainly don’t want to be the only ones running a BinderHub, and having other BinderHubs behind mybinder.org is a great example of this. Second, a network of BinderHubs for mybinder.org means that we can eventually do some clever things like use geolocation to distribute users, which should improve the performance that you all experience. Third, having fewer users per BinderHub means that we can optimize the resources available to each hub — resulting in improvements in speed and stability. Finally, having a federation of BinderHubs makes the mybinder.org service significantly more robust, and less-dependent on a single team, deployment, platform, and funding source. We are now more confident than ever that mybinder.org will continue to be available as a stable, free, public service that keeps growing.

What’s next?

Now that we’ve reached N=2 BinderHubs powering mybinder.org,
it is straightforward for us to grow this network to N=3 and beyond. Over the summer we will continue our work on approaching universities, research councils, and cloud hosting companies.

If you are interested in helping out: we’d love to see other community members come forward to offer their time or resources to provide more nodes in the international BinderHub federation. If you’re interested in doing so, please open an issue in the JupyterHub team compass repository
to discuss the possibilities of working together!

We’re excited about the ability for the Binder community continuing
to grow, and to build more tools for distributed, community-run
infrastructure for open science and education. This is all possible
because of the hard work of many people in the community, so a big
thank you to those who spend their time working on open tools in
the Jupyter and Binder ecosystems. We’re excited to see what comes
next!

Binder + Zenodo: A how-to guide

2019-06-21T13:46:00+00:00

Interactive and reproducible repositories powered by Zenodo and Binder.

When the Binder project was first launched, we imagined a world in which scientific scholarship and reproducibility could be carried out by the community using a fully-open stack of technology. We’re happy to say that this is now possible!

Zenodo DOIs now work with BinderHub

The BinderHub team recently added in support for building Binder links that point to Zenodo repositories. Zenodo is a general purpose open-access repository hosted by CERN that allows researchers to archive and apply a DOI to information that they put up on the web.

Zenodo has the ability to archive GitHub repositories, which means that you can archive the code, data, or reports that underly a scientific analysis and assign it a unique, citeable identifier. Now that BinderHub knows how to resolve a Zenodo identifier, you’ll be able to share Binder links that point to Zenodo and provide interactive access to your repository, letting readers reproduce results and interact with your analyses!

Here’s a quick primer for how to do this:

Step 1: Create a Zenodo account

First off you’ll need to create an account on Zenodo. You can
do so using a number of different log-in options.

Step 2: Create your Binder-ready repository on GitHub

Next, you should create your Binder-ready repository on GitHub. Binder uses pre-existing best practices in data science in order to infer and build the environment needed for your repository.

To make a repository Binder-ready, follow the instructions in the Binder docs.
Briefly, what you need to do is add the configuration files that define the environment needed to run your code. Once those files are in place, and you’ve added an analysis script (a Jupyter or R Notebook) that actually runs your code and displays the results, your repository is ready to build with Binder.

Step 3: Make sure your repository is ready to be published!

Once you create a DOI for your repository, it will be frozen in time — you won’t be able to easily update it. So double check that the repository builds properly with Binder and runs the way that you’d expect it to.

Make sure to launch a Binder from your repository and run the analyses you’d like others to run. If they produce the expected result from within a Binder session, then they’ll continue to do so for others (assuming you have pinned your versions and followed other best practices in reproducibility).

Step 4: Create a Zenodo DOI for your repository

Now that your repository is ready, you’ll connect Zenodo with GitHub to create a DOI for your repository. Remember that this will be unique to the current state of the repo — future changes to this repository won’t be reflected in the DOI.

We recommend following the GitHub Citable Code Guide which provides some best-practices for creating your Zenodo DOI for a GitHub repository. Click the image below to be taken to this (excellent) guide.

The GitHub Citable Code guide.

Once you’re done, you should have a Zenodo DOI badge
like the one below:

A Zenodo DOI badge

Step 5: Create a Binder link for your Zenodo DOI

Finally, use your Zenodo DOI to create a Binder link that allows others to interact with and replicate your results. You can create a Binder link for your Zenodo record by heading to https://mybinder.org and filling in the form:

Building a Binder link for your Zenodo archive with the Binder UI.

This will give you a link you can share with others as well as the Markdown and reStructured text snippets for creating a badge.

The link’s structure should look like this:

https://mybinder.org/v2/zenodo/<zenodo-DOI>

For example, if your Zenodo DOI is 10.5281/zenodo.3242074 (corresponding to this zenodo repository), the Binder link for it would be:

https://mybinder.org/v2/zenodo/10.5281/zenodo.3242074/

You can even pair this Binder link with your Zenodo DOI badge that
we showed above!

The “launch binder” button contains a Binder link for your Zenodo archive. Clicking will launch a Binder session.

And that’s it! You now have an archived version of your analysis with a unique identifier. This identifier can be used in conjunction with BinderHub to allow readers to interact with your code and results!

Last minute update 🎉

A few hours before publishing this post we merged a contribution from Tom Morrell who works at CalTech’s library that allows you to launch records from CalTech’s Data Repository as well! For example a notebook to analyze traffic to the archive itself: https://mybinder.org/v2/zenodo/10.22002/d1.1250. You should also check out a quick response to this post from Lorena Barba, who describes her group’s approach to creating reproducible, citable bundles for figures in papers, and how Binder+Zenodo DOIs might fit into that workflow!

What’s next?

We are close to closing the loop of fully reproducible computational environments for scientific publication. We’re excited to see journals begin to integrate these workflows with their own publishing pipelines. For example, the Neurolibre project is deploying their own BinderHub and using it alongside their reviewing and archiving process in order to provide more rich
interaction with submitted material.

Each type of repository needs a small amount of custom work to be integrated with Binder. We started with Zenodo because it is well known,
general purpose and integrated with GitHub already. If there is an archive you’d like to see integrated please do stop by this repository and open a new issue or contribute the code to do so directly.

If you’re working with a publisher and are interested in this
please reach out! The Binder community would love to work with you in deploying these open tools to make your published work more open and accessible. Whether it is big or small, we hope that these workflows can make an impact across the publishing landscape, and we’re looking forward to seeing what people do next!

BinderHub is out of Beta!

2019-04-26T16:49:00+00:00

Nearly two years ago, the Binder Project released the beta version
of BinderHub, the technology behind mybinder.org. Since then, mybinder.org has grown to serve nearly 90,000 launches each week and hit the two million launches in a year milestone last year. Over these two years BinderHub has matured as technology and community. Several new organizations now run their own BinderHubs and have joined the community of maintainers.

A year of Binder sessions served from mybinder.org. Darker colors show users in countries that have launched more Binder sessions. The pins show the approximate locations of public BinderHubs that we know of. Where is mybinder.org? It is a global effort so there is no one location associated with it or the team that runs it.

Today we are proud to announce that we’ve removed the “beta” label from all pages served by BinderHub. BinderHub is now a tool with a track record of working well in production (both at mybinder.org and in other deployments), its general features and API have stabilized, and JupyterHub, the underlying technology that BinderHub uses, is close to its 1.0 release.

A big thank you to all those who have used, commented, advocated, advertised, contributed, maintained and funded this journey. The deployment at mybinder.org is funded with grants from the Moore Foundation and the Google Cloud Platform.

Who else has deployed a BinderHub?

A goal of Project Binder is to build modular, open-source tools that
others can deploy for their own communities. BinderHub, the core technology behind mybinder.org, runs on Kubernetes this means it can be deployed on many cloud providers or even on your own hardware. Over the years, we have seen many new organizations deploy their own BinderHubs. Here are our highlights of other organizations who have joined the BinderHub community.

GESIS were the first to deploy a public BinderHub that is operated independently of mybinder.org. The GESIS BinderHub instance went live
in December 2017 and has been running ever since. They are frequent
contributors to the upstream project. Their BinderHub runs on a bare metal
Kubernetes cluster.

The Pangeo Project instance was the next to come online, launching their
public BinderHub instance in September 2018 (more details in their blog post). They provide additional compute resources and have customized their setup to provide on-demand dask clusters to users. Try out one
of their examples using ocean data. The Pangeo cluster is hosted on Google Kubernetes Engine.

Recently, the Turing Way project has been working on deploying a BinderHub at the Turing Institute in the UK. This BinderHub will serve both internal and external users with the goal of making it easier to share data science projects. They have run several workshops including one that teaches scientists and research software engineers how to deploy their own BinderHub instance. Recently they led a workshop in which ten academics and IT staff deployed their own BinderHub on the Microsoft Azure cloud! Sarah Gibson from their team has recently joined the team that operates mybinder.org.

Finally, the Pacific Institute for the Mathematical Sciences (PIMS) runs a service called Syzygy. It deploys several JupyterHubs and BinderHubs for scientific organizations around Canada. A Binder team recently held a tutorial on deploying Binder and JupyterHub at the PEARC (you can find slides for the talk here). They have also provided a script for deploying JupyterHub (and Binder) on Terraform.

What’s next?

Now that BinderHub is not in beta anymore, what is next? As Project Binder we will focus on adoption, training, and growing the Binder community. This means growing the creation of Binder-ready repositories (repositories that have the necessary structure for Binder to create the environment needed to run the repository’s code). We will increase our outreach and marketing efforts to make sure a diverse audience everywhere around the world knows about Binder.

We will also work on making it easier to setup and operate a public BinderHub no matter what cloud vendor you are using. We are excited to see more organizations deploy their own BinderHubs for their communities. In the coming months, our goal is to create a federation of public BinderHubs
that operate in unison to serve the global user base of mybinder.org.

If this caught your attention consider joining the Binder community, or contributing to Project Binder! A good place to start is the Jupyter Community Forum or dive straight into the code on GitHub.

elife sprint: Integrating Stencila and Binder

2018-12-10T10:15:00+00:00

This article reports on a project, integrating Stencila and Binder, which started at the eLife Innovation Sprint 2018 (#eLifeSprint). A longer version has been cross-posted on multiple blogs (eLife Labs, Stencila, o2r).

eLife, an open science journal published by the non-profit organisation eLife Sciences Publications from the UK, hosted the first eLife Innovation Sprint 2018 as part of their Innovation Initiative in Cambridge, UK: “[…] a two-day gathering of 62 researchers, designers, developers, technologists, science communicators and more, with the goal of developing prototypes of innovations that bring cutting-edge technology to open research communication.” One of the 13 projects at the excellently organised event was an integration of Binder and Stencila.

This article reports on the project’s results and changes made to Binder-related tools. Today, Binder has first class Stencila support. Read the full story at the eLife Labs blog post or try out to open Stencila documents from any online code repository on mybinder.org with the click of a single button:

Click the Binder badge to open a Stencila document on mybinder.org

This project is a collaboration between Min from the Simula Research Laboratory, Norway, a core developer of Binder and related tools, Daniel from the o2r project at the Institute for Geoinformatics, Germany, and Nokome, initiator and developer of Stencila. The final changes were made with the help of Tim. Thanks! The project was also part of the Mozilla Global Sprint 2018, see mozilla/global-sprint#317.

The building blocks and a challenge

Stencila Desktop is an office suite for reproducible research documents. It allows scientists to use languages like R and Python within familiar and intuitive word processor and spreadsheet user interfaces to lower the barriers to reproducible research for those with little or no software development skills. Binder (a part of Project Jupyter) makes it simple to generate reproducible computing environments from code repositories (e.g. GitHub or GitLab, see binder examples) and mybinder.org is the most prominent example. Binder uses repo2docker, for generating Dockerfiles (a human- and machine-readable recipe for setting up a computational environmentt used by the popular Docker container software) and building Docker images from software projects. While containers have become a commodity for developers, researchers still struggle to grasp and control the complexity of computational environments. This is where the two building blocks join: Running Stencila as part of a Binder helps researchers to communicate their work openly, to collaborate effectively with other scientists, and to ensure a high quality and transparency of their workflow and findings. Min and Daniel formulated their goal in the sprint project form: “[…] to connect them so that users can edit reproducible documents (DAR files) as part of a Binder project”.

Connecting Stencila and Jupyter: `nbstencilaproxy`

Stencila has “execution contexts”, an equivalent to Jupyter’s “kernels”. The contexts use code dependency analysis and return execution results as data values to enable a reactive, functional execution model. To open Stencila documents on binder, these execution contexts must be installed and configured in the environment created by repo2docker. This is achieved with a new software project initiated at the sprint: nbstencilaproxy - a Jupyter notebook server extension and proxy for Stencila.

The projects consists of a Python module with the Jupyter notebook server and “non-server” extensions of the same name, and a bundled JavaScript module (of the same name). The Python module allows proper versioned installation, dependency management, and installation from an established software repository. It takes care of the plumbing between the user interface and the services in the background, so that the binder is viewable over one port in the browser, while the many different background components run on their own ports. The “no server” extension adds a “Stencila session” menu entry and conveniently lives in the same directory structure as the server extension. The JavaScript module manages the required JavaScript dependencies and provides an well-defined structure for the code files. It serves the Dar document and provides access to the Stencila host.

repo2docker was extended with automatic detection of Dar documents, including the languages and execution contexts. As with most Binder repositories, a no configuration is needed for most common user cases: users can open a Dar document on Binder and trust the required environment to provide all required software. Daniel created a few example repositories to provide a starting point for users. The binder team generously welcomed the changes to mybinder.org and and the examples to the binder examples organisation on GitHub:

https://github.com/binder-examples/stencila-py contains Python code cells, using both the Jupyter and plain Python execution contexts:

Click the Binder badge to open a Stencila document with Python code on mybinder.org

https://github.com/binder-examples/stencila-r contains R code cells and two plots:

Click the Binder badge to open a Stencila document with R code on mybinder.org

One of the cool features of Stencila are the reactive cells, as demonstrated in a tweet following the feature release:

Thanks to @nordholmen working on @stencila support for https://t.co/Zlj6FrYgBw you now have reactive cells with Python code on @mybinderteam! Give it a go https://t.co/ToIuQPq0Fy

— Tim Head, @betatim on the internet (@betatim), November 12, 2018

Binder + Stencila is a demonstration of the power that the Open Source and Open Science community can foster. Many people are working together on the organisational and technological challenges of science today towards full research transparency and reproducibility, even if we use computers to an unprecedented level. Many small contributions on “side projects” such as these can make a difference, and connecting these two great projects hopefully helps to solve some problem in science down the road.

Join the public Stencila and binder chats to stay in touch or get help. We look forward to see scientists using nbstencilaproxy for communicating their work and new challenges that come with it.

#eLifeSprint-ers @minrk and @nordholmen are working to connect #JupyterNotebooks / #Binder with DAR / #Texture / @Stencila, so that users can edit reproducible documents as part of a Binder project https://t.co/2GoGNydsmX (@mybinderteam @ProjectJupyter @_substance)

— eLife Innovation (@eLifeInnovation), May 10, 2018

MyBinder.org serves two million launches

2018-11-13T12:35:00+00:00

by the Binder Team

Since the beginning of 2018, the Binder community has been hosting a BinderHub at https://mybinder.org as a free public service. Today, we are proud to announce that this hub has served over two million Binders. To mark this milestone we would like to say a huge Thank You! to the large community of people who use, build, and fund the project. Without you this important public infrastructure would not be the user-friendly, reliable, well-supported and documented resource that we enjoy today. mybinder.org has enabled people from almost every country in the world to learn, participate and share countless projects, ideas and stories. Here’s to two million more!

A huge thank you to all those who help with building, using, and operating https://mybinder.org.

What is mybinder.org?

mybinder.org let’s you take a repository full of Jupyter notebooks or RMarkdown and turn it into a collection of interactive notebooks. You can share your work with anyone by sending them a simple link (like this one). All they have to do is open the link in a web browser and they can run those notebooks from anywhere in the world without having to install anything.

Who is using mybinder.org?

Currently about 70–80,000 Binders are launched every week. A lot of those are people who are looking for a quick and easy way to launch a Python or RStudio environment. However in the last week a notebook showing off fifty ways to solve Fizz Buzz has been getting a lot of love. Beyond those heavy hitters and short-lived audience favorites there is a long tail of over 400 unique repositories that get launched every week. It would take the rest of the post to list them all!

One last statistic that we are particularly proud of: over the last 80 days we have had users from almost all around the globe! Binder was started as a way to make computational research easier to share and reuse. We have been amazed at how many people around the world have used Binder for teaching classes, reproducing results, sharing interactive analyses, and making their work more accessible to others. We are particularly proud that this includes people from around the entire world.

Countries from which https://mybinder.org has received visitors between 22 August 2018 and 10 November 2018.

Data set of all launches on mybinder.org

mybinder.org is operated as a public infrastructure that is transparent, open, and inclusive. We chat, discuss, and work in the open. This is why we are now publishing a new data set: a continuously updated log of every launch that happens on mybinder.org!

MyBinder.org Events Archive

We would love to see people explore this data set as a public resource that describes the kinds of repositories being shared and launched on the public mybinder.org deployment.

A new badge!

One more thing … we thought now is a good time to give the trusty “Launch Binder” badge an overhaul. To improve the badge we put together some suggestions, reached out to the community, and within a few days received a lot of feedback and new ideas. After combining all the inputs, our new badge went live earlier this week. We present to you our new badge:

The new “Launch Binder” badge. Binder blue instead of bright red!

We hope you like it as much as we do. If you are in love with the old design or not quite ready to switch yet, do not worry! The old badge is not going anywhere. If you are using the old badge in your README it will continue to look the same as it always has.

If you do want to change a previously generated link to the new badge, edit the name of the SVG in the link from the old:

[![Binder](http://mybinder.org/badge.svg)](http://mybinder.org/v2/gh/binder-examples/r/master)

to the new:

[![Binder](http://mybinder.org/badge_logo.svg)](http://mybinder.org/v2/gh/binder-examples/r/master)Outro.

Open infrastructure in the cloud

The Binder project is a community-driven experiment in radically-open infrastructure. BinderHub, the underlying technology that powers a Binder deployment, is an open project and can be deployed in many other cloud environments. For example, see the Pangeo Binder deployment for geospatial analytics, or the Gesis Binder deployment for social sciences. We are excited to see the project head in new directions as we continue to grow the technology and the community around Binder.

Finally, we could not have done any of this without a ton of support from the Binder community. First, many thanks to the Moore Foundation for funding initial development of Binder’s underlying tech, and for helping us finance running the deployment at mybinder.org. Second, many thanks to the Binder project core team for fostering excellent technology and a great community. Finally, thanks to everybody in the Binder community — whether you’ve launched repositories, shared your Binders, participated in discussions, built features, helped design this post’s banner image (❤), or gave us some critical feedback. Binder’s purpose is to serve the community, and you all have made it so worth it!

If you’d like to get involved in the Binder project, here are a few helpful links:

To learn about Binder, see the Binder documentation: docs.mybinder.org
For information on how to deploy your own BinderHub, see the BinderHub deployment docs: binderhub.readthedocs.io
To participate in conversations with the Binder community, say hello on the Binder gitter channel (https://gitter.im/jupyterhub/binder) or the JupyterHub/Binder Discourse forum (discourse.jupyter.org).
If you’d like to see the code itself, see the three main open projects that make up a Binder deployment: BinderHub (github.com/jupyterhub/binderhub), repo2docker (github.com/jupyter/repo2docker), and JupyterHub (github.com/jupyterhub/jupyterhub)

Binder 2.0, a Tech Guide

2017-11-30T16:01:00+00:00

Authors: The Binder project is comprised of many individuals within and outside of the core Jupyter team. A list of members that contributed to this post is at the end of this article.

Note: this post focuses more on technical changes in the Binder 2.0 reboot. For a post about user-facing features and future plans, see this eLife blog post.

We are undergoing a dramatic increase in the complexity of techniques for analyzing data, doing scientific research, and sharing our work with others. In early 2016, the Binder project was announced, attempting to connect these three components. A blogpost in eLife described a vision where scientists could specify dependencies along with a collection of Jupyter notebooks. Binder builds a Docker image from these dependencies, and provides a URL where any user in the world can instantly recreate this environment.

Want to see it in action? Click the button below.

With this post we are proud to announce the next version of Binder. It aims to be more modular, more flexible, more stable, faster, and more extensible than its predecessor. Powering this version of Binder is a collection of tools in the Jupyter ecosystem. Since being released, the Binder project has learned many things about implementing fast, flexible online deployments. In addition, its vision has expanded to include not only Jupyter notebooks, but many other computational workflows. You can access an open beta version of this deployment here:

mybinder.org

You can find a list of sample repositories to learn how to create “Binder”-ready repositories here:

github.com/binder-examples

You can also see what the Binder community has been up to in creating their own repositories by checking the GitHub Binder topic:

https://github.com/topics/binder

Give it a shot, build some repositories, and importantly, tell us what could be improved on our GitHub repo. Below we’ll describe a bit about what’s new.

The new Binder UI. Users input a URL to a git repository (or specify a specific branch/tag/commit). Upon clicking “launch”, you will be directed to a live environment where you can interact with the code.

What’s new?

First off we’ll describe how the experience will change for users. The short answer is: not much. The goal of Binder is still enabling you to intantly create interactive and shareable repositories. We’ve completely rebuilt the backend of Binder, but we’ve made minimal changes to the front-end user experience.

The biggest difference you should notice is that Binder is both faster and more stable. You’ll still be able to generate Binder links from a git URL from a single web-page. However, there are a few key differences:

New Default Environment

Old versions of Binder were based off of a Docker image that contained a fairly heavy computational environment. The new Binder deployment makes minimal assumptions about what environment you want installed, by default the only thing that will be installed is the Jupyter Notebook and Python 3. This means you’ll need to be more expressive in the dependencies you include in your dependency files. For example, if you want numpy or matplotlib, you should specify them in a requirements.txt or environment.yml file. Since you’re explicitly listing your requirements it makes reproducing your work more reliable, and allows the Binder infrastructure to change more freely without breaking your repository code.

New URL structure

The new URL structure for Binder follows the following convention:

https://mybinder.org/v2/gh/<org-name>/<repo-name>/<branch|tag|hash-name>?filepath=<path-to-file>

For example, below is the URL for a basic Binder-ready Python 3 repository, it includes basic information about the repository:

https://mybinder.org/v2/gh/binder-examples/requirements/master

You can also specify parameters that do things like point users to a particular file or initialize a user-interface. For example, the following URL starts JupyterLab once users click the link:

https://mybinder.org/v2/gh/binder-examples/jupyterlab/master?urlpath=lab

In each case, note the gh at the very beginning — this specifies that the git URL exists on github.com. It is possible to build new URL parsers for other online repositories such as BitBucket, osf.io, or any other provider. We are currently focusing on Git and GitHub, but nothing prevents Binder from being compatible with other kinds of content providers. Shortly before this blog post we added support for arbitrary git URLs to binderhub, so watch this space.

Specify a specific branch / tag / commit

Also notice that in the URL above you can specify a branch, tag, or commit hash for the Binder image. This allows you to ensure that a Binder image will always remain the same (if you specify package versions properly). This is a crucial step for reproducibility and maintaining consistency in how users experience the files in your repository.

Notice that in the URL above you can specify a commit hash or git tag for the Binder image. This hash is unique to the state of the code at the moment that the commit was made, ensuring that Binder can rebuild the exact same environment any time. Note that if authors don’t want to guarantee the same reproducible environment, they can specify a branch and BinderHub will resolve it to the latest commit hash before building the environment.

Binder auto-building

When a git repository is launched, Binder will now check whether an image has already been built for that repository at the same commit hash. If it has, then Binder will skip the building process and take you straight to a JupyterHub instance that serves this image.

If the image hasn’t been built, then it will automatically be generated before sending the user to JupyterHub. The only difference will be the amount of time it takes before entering the JupyterHub environment. This means that authors no longer need to explicitly build their Binder images when they update a branch. The next time someone clicks a Binder link, it will happen automatically. If you don’t want this behavior, be sure to point Binder to a specific tag or commit hash, rather than a branch name or tag. For example, here’s a Binder URL that will always point to the same commit hash:

https://mybinder.org/v2/gh/wildtreetech/explore-open-data/binder20-elife

While this URL points to a branch, and will thus be re-built each time a new commit is made to that branch:

https://mybinder.org/v2/gh/wildtreetech/explore-open-data/master

More options for dependency files

Users often want to specify a computational environment that is more complex than a simple list of Python requirements. While this is possible by specifying a Dockerfile, it’s often an overly-complicated solution to this problem. Binder now uses repo2docker to build a Docker image from your repository. This makes it possible to specify a more complex environment with text files. For example, you can use an apt.txt file to install packages with apt-get, or use a file called postBuild to define shell commands that are run before generating the Docker image (e.g. for downloading some data or running scripts). See the repo2docker documentation for a list of files that are supported with Binder.

For a selection of examples that show off how to specify dependencies take a look at the example gallery: https://github.com/binder-examples

More user interfaces

The JupyterLab interface running on Binder. You can access the JupyterLab demo repository at ``mybinder.org/v2/gh/binder-examples/jupyterlab/master?urlpath=lab

The previous iteration of Binder only supported the classic Jupyter Notebook user interface, while the new deployment will additionally support RStudio and JupyterLab. Because of the extra build configuration files specified above, you can also utilize more tools in the Jupyter widgets ecosystem, such as the RISE plugin for interactive presentations or the appmode plugin to generate interactive apps from your repository. We also welcome contributions to add support for other user interfaces.

More online repository providers

While GitHub is a fantastic repository of open-source code, it’s not the only repository. The new Binder deployment makes it straightforward to adding support for new sources of code (for example, GitLab, the OSF, or even non-git codebases). Currently GitHub is the only supported source for code, but we welcome contributions enabling support for new sources.

We’re excited about this next step in Binder’s development, and hopeful that we can build a community around this powerful set of tools. Don’t hesitate to open an issue or pull request on our GitHub repository, or to reach out via our Gitter channel. We look forward to seeing what comes next, and to continue enabling reproducible and open workflows in data science and research.

For developers

The next few sections are meant for developers interested in deploying their own Binder, or for those interested in the technical details behind the new deployment.

Tech components

The three main technical components behind the new Binder backend are:

BinderHub, currently on display at mybinder.org and contained in the binderhub repository.
repo2docker, a tool that converts a code repository into a Docker image with an environment specified via dependency files (e.g., requirements.txt).
JupyterHub, which hosts user instances with a server in the cloud. We use a distribution of JupyterHub that runs on top of Kubernetes.

For more information on the infrastructure behind Binder, see the documentation.

A prototype of RStudio running in a Binder. This is currently support with a Dockerfile, and we are working on supporting R build files natively. You can access this repository at: <https://mybinder.org/v2/gh/binder-examples/dockerfile-rstudio/master>

Kubernetes

Binder now also heavily relies on Kubernetes for scaling our image building service and the JupyterHub. Kubernetes is massively scalable and has a strong community of developers behind it. Moreover, Kubernetes is cloud-agnostic. It can be run on Google Cloud, Microsoft Azure, and AWS among others, as well as on your own bare metal hardware if needed. Because BinderHub is built to run on top of Kubernetes, you can deploy Binder off of any of these resources as well (see below).

Deploying your own Binder server

While mybinder.org will continue to exist as a public service, we hope to see new Binder deployments for many different use cases in the wild. One of our primary goals is to make it easier for users to deploy their own Binder servers. This is relatively straightforward by following the instructions on the BinderHub documentation, which are currently under active development to make ongoing improvements as the Kubernetes technology evolves. We’re continuously updating these steps to make them as clear as possible, so please don’t hesitate to open an issue or a pull request on our github repository and make suggestions.

We would love to see others deploy their own BinderHub servers, either for their own communities, or as part of a federated public service of BinderHubs.

Future development

This is the just the beginning of new features and improvements to Binder. We’re working hard to grow an open-source community around these tools, and we encourage issues, comments, and PRs on the BinderHub, repo2docker, and JupyterHub repositories. We look forward to growing the Binder ecosystem, and we’re excited to see all of the Binders that people design.

Acknowledgements (alphabetical order)

C. Titus Brown (UC Davis), Matthias Bussonnier (UC Berkeley), Jessica Forde (UC Berkeley), Brian Granger (Cal Poly), Tim Head (Wild Tree Tech), Chris Holdgraf (UC Berkeley), Andrew Osheroff (Google), Naomi Penfold (eLife Sciences), M Pacer (UC Berkeley), Yuvi Panda (UC Berkeley), Fernando Perez (UC Berkeley), Min Ragan-Kelley (Simula Research Laboratory), and Carol Willing (Cal Poly). The Binder project is currently being funded from a grant from the Moore Foundation.

Jupyter Blog - Binder

Berkeley Institute for Data Science (BIDS) joins the mybinder.org

Berkeley Institute for Data Science (BIDS) joins the mybinder.org federation in partnership with 2i2c

mybinder.org adding request Origin to events archive

Simplifying and speeding up Binder builds with BuildKit

Authors and acknowledgements

Footnotes

Curvenote joins the mybinder.org federation

Mybinder.org reducing capacity

Updating defaults on mybinder.org

Python 3.11 and separate envs for legacy Python support

Default Python is now 3.10

Now launching Jupyter Server

MyBinder @ OVHcloud

A new partnership with NumFOCUS

Acknowledgement

A 2019 retrospective from the Binder Project

More people are using mybinder.org

JupyterHub reaches 1.0

BinderHub is out of beta

The BinderHub federation is launched

Binder connects with open science services

The Binder community grows

We welcome our first Contributor in Residence

There are more BinderHub deployments

Thanks to our community

The JupyterHub and Binder Contributor in Residence!

How we automated mybinder.org dependency upgrades in 10 steps

What does the mybinder.org upgrade bot do?

Writing the bot

Step 1: Retrieve current deployed mybinder.org dependency versions

Step 2: Retrieve latest commits from the dependency repos

Step 3: Fork mybinder.org repo

Step 4: Clone your fork

Step 5: Make the file changes

Step 6: Stage, commit, push

Step 7: Make the body for the PR

Step 8: Make the PR

Step 9: Confirm and merge!

Step 10: Automating the script (cron)

The International Binder Federation

OVH Joins the mybinder.org Federation

What does this mean?

Why is this a big deal?

What’s next?

Binder + Zenodo: A how-to guide

Step 1: Create a Zenodo account

Step 2: Create your Binder-ready repository on GitHub

Step 3: Make sure your repository is ready to be published!

Step 4: Create a Zenodo DOI for your repository

Step 5: Create a Binder link for your Zenodo DOI

Last minute update 🎉

What’s next?

BinderHub is out of Beta!

Who else has deployed a BinderHub?

What’s next?

elife sprint: Integrating Stencila and Binder

The building blocks and a challenge

Connecting Stencila and Jupyter: nbstencilaproxy

MyBinder.org serves two million launches

What is mybinder.org?

Who is using mybinder.org?

Data set of all launches on mybinder.org

A new badge!

Open infrastructure in the cloud

Binder 2.0, a Tech Guide

What’s new?

New Default Environment

New URL structure

Specify a specific branch / tag / commit

Binder auto-building

More options for dependency files

More user interfaces

More online repository providers

For developers

Tech components

Kubernetes

Deploying your own Binder server

Future development

Acknowledgements (alphabetical order)

Connecting Stencila and Jupyter: `nbstencilaproxy`