Jupyter Blog - cloud computing

Simplifying and speeding up Binder builds with BuildKit

2025-03-03T14:05:00+00:00

The Binder Project allows users to build reproducible, sharable environments for interactive computing. To accomplish this, Binder uses a tool called repo2docker to generate an executable Docker image using the Reproducible Execution Environment Specification.

The first time a repository is launched on Binder, repo2docker must build the reproducible environment for it. This process can take a long time because of all the dependencies that need to be installed and turned into the image. As a result, Binder launches can feel slow and clunky, which is a poor UX for workflows that are designed around quick interactive sessions.

repo2docker was built several years ago, and followed patterns that were commonplace at the time. However, in the past few years the Docker community has made significant advances in optimizing the image building process. One-such improvement is the creation of BuildKit, a replacement for Docker’s historical build system that is much more sophisticated. However, repo2docker hasn’t leveraged these improvements because it was still using the original Docker Build system.

So, we’ve decided to spend a few cycles modernizing repo2docker’s image building logic by using the more modern BuildKit API (via ‘docker buildx build’). This allows for optimizations like build parallelization, better build caching, and supporting some Dockerfile features that Binder didn’t support earlier (particularly, COPY --chown). It also lays a foundation for significantly simplifying the repo2docker build infrastructure and leveraging more of BuildKit’s parallelization functionality. For example, we’d like to leverage BuildKit’s Kubernetes driver which distributes builds much more efficiently and in parallel.

Authors of Binder repositories won’t need to take any action¹, and they’ll simply notice that mybinder.org (and any other community-run BinderHub instance) will be a bit snappier at building images.

If you’d like to learn more about the changes that enabled this, check out this mybinder.org pull request which has links to the repo2docker pull requests that added this functionality. We’re excited keep improving Binder, and are hopeful that this makes the experience of using mybinder.org and community Binders a little bit better.

Authors and acknowledgements

Yuvi Panda is a co-founder and the Technical Lead at 2i2c. He is passionate about building participatory open infrastructure for scientific & educational use cases. He is a Project Jupyter team member primarily focused on infrastructure related projects (JupyterHub, Binder, etc). He also wrote most of the code for this docker buildx transition, and shepherded it through to deployment on mybinder.org as well. He is ex-Wikimedia and ex-GNOME. Let’s eliminate accidental complexities wherever we find them.

Chris Holdgraf is a co-founder and the Executive Director of 2i2c. He is on the Executive Council of Project Jupyter, and co-leads the JupyterHub and Binder team as well as the Jupyter Book team). He was previously a post-doctoral researcher in the Department of Statistics at UC Berkeley, and a Community Architect with the Division of Data Science at Berkeley. He’s interested in using open infrastructure to support interactive computing workflows in research and education.

Many thanks to @minrk, @manics, and @consideRatio for their help reviewing and shaping this work.

Footnotes

¹ Unless they were relying on undocumented implementation details of the old builder — in particular, the presence of a /.dockerenv file to detect if you are running in repo2docker scikit-learn/scikit-learn#30835 has an example.

Desktop GIS software in the cloud with JupyterHub: A QGreenland workshop success story

2023-08-02T16:23:00+00:00

👋 We are Trey Stafford and Matt Fisher, co-authors of the QGreenland data package’s source code. This year, we had the pleasure of running a hands-on geospatial data and open science QGreenland Researcher Workshop. It was important for attendees to participate in the workshop in a hands-on way while minimizing the negative impacts of installing software, requiring expensive personal computers, and troubleshooting unique computer configurations. We felt a JupyterHub was a good fit for our workshop for this reason, if it could accommodate our need for running QGIS — a desktop application.

In this blog post, we will introduce QGreenland, describe our experience using JupyterHub in the cloud for our workshop’s computing environment, and discuss challenges we overcame to enable our attendees to use QGIS in a cloud graphical desktop environment. Finally, we will highlight some workshop outcomes and discuss opportunities for enhancement based on new developments in the Jupyter ecosystem.

In our workshop, 25–30 international learners (including from Germany, India, France, Canada, Poland, and the United States) used QGIS in a JupyterHub’s browser-based Linux desktop environment to collaboratively test, explore, visualize, and process Earth science data simultaneously with the same user experience they expect from using QGIS on their personal computers! Better yet, getting started was as simple as logging in. Our workshop was a success story not just in education, but also in open source and collaborative development, and we want to share what we learned.

Note the browser tabs at the top of this screenshot; this is a full desktop-based GIS environment running QGreenland in the cloud!

The JupyterHub used by the QGreenland 2023 Researcher Workshop was generously provided by the NASA CryoCloud team, whose mission is to help researchers transition to cloud-based collaboration.

About QGreenland

QGreenland is an open-source Greenland-focused geospatial data package for QGIS, a community-owned graphical Geographic Information System (GIS) platform. Researchers and members of the public leverage QGreenland’s ready-to-use interdisciplinary datasets to do field planning, teach about glaciers, and much more.

QGreenland’s MIT-licensed source code uses community-maintained open software like GDAL and PyQGIS to automate data normalization and populate the QGIS project with important information like data provenance and the order of layers in the QGIS Layers Panel. Check out our documentation to learn more! QGreenland also has a YouTube channel with tutorials produced by CIRES Education and Outreach.

Once QGIS is installed, opening QGreenland is as easy as double-clicking the included “.qgs” file. Here, a representative view of QGreenland v3 alpha in QGIS is displayed with newly updated layers: Arctic sea routes (National Geospatial Intelligence Agency), bathymetric depth (General Bathymetric Chart of the Oceans (GEBCO)), and ice thickness (IceBridge BedMachine Greenland v5) layers.

Based on user research, QGreenland has enabled:

the public to more easily access data gathered by researchers visiting Greenland: “In Greenland, people are often asking, ‘how can we find the data the foreign scientists bring back from Greenland?’ Now we can directly utilize much of it.”
researchers to plan field work: “Being able to use QGreenland at our field station was critical to our research process!”
educators to develop interactive lessons about Greenland and climate change: “…using QGreenland for presentations because it is presentation quality already.”

QGreenland’s 2023 researcher workshop

One of the QGreenland team’s most important forms of direct user interaction and support is facilitating workshops. Most recently, we hosted a 3-day (total of 9 hours) virtual workshop for researchers focused on working with geospatial data in an open science framework. All of the materials covered in the workshop were built using open-source tools and are MIT-licensed and published on GitHub.

A “personal computer” in the cloud

We decided early on that we wanted to use JupyterHub to solve the diverse problems that come with “bring your own device” workshops. We experimented with administering our own JupyterHub on Kubernetes, but the setup overhead was too high for our short workshop. CryoCloud’s JupyterHub enabled us to avoid this overhead and focus on serving our participants. Because the software that comprises CryoCloud is open-source and developed in collaboration with the communities CryoCloud serves, we could directly contribute to curating a computing environment ideal for our participants.

JupyterHub is known for providing access to Jupyter Notebooks via JupyterLab, but it turns out it can also be used to host pretty much any interactive web based application! The jupyter-server-proxy project enables this, and there are additional packages that make running specific applications easier. jupyter-rsession-proxy makes it easy to run RStudio inside JupyterHub, jupyter-vscode-proxy allows running code-server (fully open source self-hosted version of Visual Studio Code) inside a JupyterHub, etc. Pertinent to our use case is jupyter-remote-desktop-proxy, which lets you run a complete Linux desktop environment inside your JupyterHub! This was critical for our workshop, as it allowed us to use QGIS — purely desktop software, not adapted for the web — from inside a web browser. Workshop participants did not need to install anything. This enabled participants to focus on the content of our workshop rather than the logistics of setting up and debugging tools on their varied machines.

The CryoCloud JupyterHub enabled each of our workshop participants to provision their own compute environment (JupyterLab + Linux Desktop) with all of our workshop’s dependencies pre-installed. It also set everyone on equitable footing — someone accessing the workshop on a 10 year old laptop would get the same computing resources as someone on a brand new MacBook Pro.

Challenges scaling QGreenland

The CryoCloud JupyterHub already had jupyter-remote-desktop-proxy and QGIS installed, so we could validate this approach to our workshop quickly. However, to use QGreenland at this scale, we needed to solve a couple of usability problems. The first issue was a user experience problem: the operating system did not have appropriate file type associations for QGIS, so files like the QGreenland project file would not open in QGIS when double-clicked in the desktop file browser. We quickly discovered a solution and integrated it with a simple pull request to the Docker image we were using.

The second problem was a performance problem: QGIS would take several minutes to open QGreenland from the hub’s shared storage drive. After some investigation, it turned out this was due to us loading multiple GB of data from an NFS share! While a long term solution might involve getting QGIS to load data directly from cloud object storage (like S3), we instead decided to go a different route — provision each user a small, fast and temporary Elastic Block Store disk. At the start of the workshop, we provided all users a small script that would copy the dataset from NFS to this faster disk once, and this drastically reduced load times from about 5 minutes to under 3 seconds! You can follow our debugging process on this issue, and find the JupyterHub config used to provision these disks here.

By overcoming these challenges, we created a smooth, intuitive, and performant computing experience for all of our participants, most of whom had never been exposed to this sort of collaborative computing environment.

Outcomes

The workshop participants engaged in small group work to complete various exercises, group discussions, and data scenarios. Each group produced Jupyter Notebooks and GitHub Discussions posts as deliverables. We created an outcomes webpage to summarize our participants’ accomplishments. One highlight was participants’ insightful commentary on FAIR & CARE principles.

Based on these outcomes, we consider our workshop a success. While we put in a significant amount of time creating our materials, CryoCloud’s cloud costs and our time investment in preparing computing resources were relatively small. For approximately 25 people, our cloud costs break down to roughly $1/person/day!

Conclusion

The CryoCloud JupyterHub met our workshop needs and provided a delightful experience for administrators and participants alike, and we are excited for what’s next. JupyterLab 4 and jupyter_collaboration v1.0.0, a real-time collaboration extension, were just announced, and the CryoCloud team is currently working to integrate these new releases into their hub. Real-time collaboration will enable exciting cloud use cases, like small groups working together on the same notebook without a screen share, or organizers providing technical support in a live notebook. We anticipate running this workshop again. We are excited to use JupyterHub again and look forward to experimenting with these new features!

Acknowledgements

Reviewers

In alphabetical order, thanks to Twila Moon, Yuvi Panda, Tasha Snow, and Alyse Thurber for their time contributing to this post!

CryoCloud

Snow, Tasha, Millstein, Joanna, Scheick, Jessica, Sauthoff, Wilson, Leong, Wei Ji, Colliander, James, Pérez, Fernando, James Munroe, Felikson, Denis, Sutterley, Tyler, & Siegfried, Matthew. (2023). CryoCloud JupyterBook (2023.01.26). Zenodo.https://doi.org/10.5281/zenodo.7576602

2i2c

2i2c is a non-profit organization that runs open-source infrastructure for collaborative computing, and maintains the CryoCloud JupyterHub used in this workshop. You can see the complete configuration of this JupyterHub in this public repository.

National Scale Interactive Computing

2019-08-22T19:06:00+00:00

This is an invited post from Jim Colliander, Professor of Mathematics at UBC and Director of the Pacific Institute for the Mathematical Sciences.¹

In 2017, the Pacific Institute for the Mathematical Sciences (PIMS), in partnership with Compute Canada and Cybera, launched Syzygy, a cloud-hosted interactive computing platform that delivers JupyterHub deployments for universities across Canada.

Syzygy has been used by over 16,000 students at 20 universities. The main results of the Syzygy experiment so far are:

Demand for interactive computing is ubiquitous² and growing strongly at universities.
The Jupyter ecosystem is an effective way to deliver interactive computing.
A scalable, sustainable, and cost-effective interactive computing service for universities is needed as soon as possible.

Demand for interactive computing

Both research and teaching at universities are adapting to major societal changes driven by explosions in data and computational tools. New educational programs that prepare students to think computationally are emerging, while research strategies are changing in ways that are more open, reproducible, collaborative, and interdisciplinary. These transformations are inextricably linked and are accelerating demand for interactive computing. The Syzygy experiment has shown that using Jupyter in educational programs drives interest in using Jupyter for research (and vice versa). For example, students in mathematics, statistics, and computer science collaborated with a researcher from St. Paul’s Hospital in Vancouver using Syzygy to identify new pathways to prevent death from sepsis. Research communities typically need access to deeper computational resources and often span multiple universities, but the common thread is the need to expand access to interactive computing.

A map of JupyterHub deployments deployed by Syzygy.

Technical milestone achieved

The Syzygy project has demonstrated that it’s possible to deploy tools for interactive computation at a national scale rapidly and efficiently using an entirely open source technology stack. Students, faculty and staff across Canada use Syzygy to access Jupyter through their browsers with their university single-sign-on credentials. The JupyterHubs range from a “standard” configuration to bespoke environments with specially curated tools and data integrations. This richness is possible because of the architecture of the Syzygy and Jupyter projects and the flexibility of the underlying cloud resources. As a case-study, Syzygy demonstrates that the Jupyter community has achieved a significant technical milestone: interactive computing can be delivered at national scale using cloud technologies.

Service level requirements

The validation that interactive computing can be technically delivered at national scale prompts universities to ask a variety of questions. Can interactive computing service be delivered robustly? How will users be supported? What are the uptime expectations? What is the data security policy? How is privacy protected? Can the robustness of the service be clarified in a service level agreement? Syzygy, as an experimental service offered to universities at no charge and without a service level agreement, does not properly address these questions. To advance on their education-research-service mission and address growing demand, universities need a reliable interactive computing service with a service level agreement.

What’s next?

How should universities address their needs for interactive computing over the next five years? Right now, universities are following two primary approaches:

🙏 Ad hoc: faculty figure out how to meet their own needs for interactive computing; IT staff deploys JupyterHub on local or commercial cloud servers; this approach gives universities control over their deployments and hardware, though requires time and expertise that many may not have.
🎩 Use a cloud provider’s service: Google Colab, Amazon Sagemaker, Microsoft Azure Notebooks, IBM Watson Studio; this approach allows universities to quickly launch interactive computing services, with a loss of flexibility and some risks by becoming reliant upon a particular vendor’s closed-source and proprietary software.

These approaches are not sustainable over the long term. If universities all deploy their own JupyterHub services, many will need technical expertise they do not currently have and will involve a significant duplication of effort. If universities rely on hosted cloud notebook services, the reliance on proprietary technology will impair their ability to switch between different cloud vendors, change hardware, customize software, etc. Vendor lock-in will limit the ability of universities to respond to changes in price for the service. Universities will lose agility in responding to changes in faculty, staff, and student computing needs.

There is a third option that addresses the issues with these two approaches and generates other benefits for universities:

🤔 Form an interactive computing consortium: universities collaborate to build an interactive computing service provider aligned with their missions to better serve their students, facilitate research, and avoid risks associated with vendor lock-in.

To retain control over their interactive computing stacks, avoid dependence⁴ on cloud providers, and accelerate the emergence of new programs, universities should work together to deploy interactive computing environments in a vendor-agnostic manner. This might take the form of a consortium — an organization dedicated to serving the needs of universities through customized shared infrastructure for interactive computing. The consortium would also ensure that universities will continue to play a leadership role in the development of the interactive computing tools used for education and research.

The Syzygy experiment confirmed that growing demand for interactive computation within universities can be supplied with the available technologies advanced by the Jupyter open source community. In the coming year, we aim to build upon the success of the Syzygy experiment and seed an initial node of a consortium in Canada with the intention of fostering a global network of people invested in advanced interactive computing. If you are interesting in partnering, please get in touch! See this Jupyter Community Forum post to continue the discussion.

The author gratefully acknowledges feedback on this piece from Ian Allison, Lindsey Heagy, Chris Holdgraf, Fernando Perez, and Lindsay Sill.
Interactive computing needs have been identified in agriculture, applied mathematics, astronomy, chemistry, climate science, computer science, data science, digital humanities, ecology, economics, engineering, genomics, geoscience, health sciences, K-12 education, neuroscience, political science, physics, pure mathematics, statistics, and sociology.
Relying on commercial cloud vendors to provide the interactive computing service for universities risks recreating the problems associated with scientific publishing that emerged with the internet.

The International Binder Federation

2019-07-01T09:11:00+00:00

About two years ago, the Binder project evolved into the community led project that it is today. The deployment at mybinder.org was upgraded to use
BinderHub, a scalable open-source web application that runs on Kubernetes and provides free, sharable, interactive computing environments to people
all around the world.

In the ensuing years, the Binder community has grown considerably.
Now people use the public deployment at mybinder.org around 100,000 times each week, and there are around 8,000 unique repositories compatible with Binder. Binder links now work with multiple repository providers, such as GitHub, GitLab, BitBucket, and even Zenodo! This means that providing enough compute power for everyone is a challenge.

This is why today we are happy to announce that mybinder.org is now backed by two clusters hosted by two different cloud providers: Google Cloud and OVH. This is the beginning of the International Binder Federation.

How did we get here? Alongside mybinder.org’s growth we’ve seen growth in another part of the Binder ecosystem: people deploying a BinderHub for their own groups. In the last year, we have seen BinderHubs deployed for large-scale earth analytics with the Pangeo project, for communities of best-practices in open science with The Turing Way, and for specific domains such as the social sciences like GESIS.

As more BinderHubs are deployed, we realized that there was an opportunity to leverage the strengths and resources of the community to improve the large, public BinderHub deployment at mybinder.org. Instead of a single BinderHub run by the Binder team, we could build a network of BinderHubs that shares the load and keeps mybinder.org stable and quick.

OVH Joins the mybinder.org Federation

Today, we are thrilled to announce that the Binder Project now has a world-wide federation of BinderHubs powering mybinder.org. We’ve partnered with OVH, a cloud hosting company based in Europe that is supportive of open projects such as Jupyter and Binder.

Through the partnership with OVH, all traffic to mybinder.org will now be split between two BinderHubs - one run by the Binder team, and another run by a team of open-source advocates at OVH. They have generously offered their resources and computing time to allow Binder to serve the scientific and educational communities.

The mybinder.org federation currently has two BinderHubs, but this network can (and will!) grow as more groups offer to connect their own BinderHubs to the network.

What does this mean?

So what has changed for you, the user? Probably not much. The
biggest difference you’ll notice is that landing at mybinder.org
will now redirect you to one of two places:

gke.mybinder.org is the BinderHub hosted on the Google Cloud Platform
ovh.mybinder.org is the BinderHub hosted on the OVH platform in France.

Other than this, your experience should be the same. You might even notice an improvement in speed and load times because the BinderHub you’re using has a little bit less traffic on it 😀.

As always, we are pushing this out as soon as we think it is useful. You can help us to scale this model and make it rock solid by reporting weird things you notice. Expect tweaks over the next few weeks based on feedback from users. Let us know about the things you like, dislike or have questions about at https://discourse.jupyter.org/t/the-binder-federation/1286.

Why is this a big deal?

We think that having a federation of BinderHubs behind mybinder.org is pretty cool, for a few different reasons. First, it demonstrates the dedication that the Binder community has towards building tools that anybody can deploy — we certainly don’t want to be the only ones running a BinderHub, and having other BinderHubs behind mybinder.org is a great example of this. Second, a network of BinderHubs for mybinder.org means that we can eventually do some clever things like use geolocation to distribute users, which should improve the performance that you all experience. Third, having fewer users per BinderHub means that we can optimize the resources available to each hub — resulting in improvements in speed and stability. Finally, having a federation of BinderHubs makes the mybinder.org service significantly more robust, and less-dependent on a single team, deployment, platform, and funding source. We are now more confident than ever that mybinder.org will continue to be available as a stable, free, public service that keeps growing.

What’s next?

Now that we’ve reached N=2 BinderHubs powering mybinder.org,
it is straightforward for us to grow this network to N=3 and beyond. Over the summer we will continue our work on approaching universities, research councils, and cloud hosting companies.

If you are interested in helping out: we’d love to see other community members come forward to offer their time or resources to provide more nodes in the international BinderHub federation. If you’re interested in doing so, please open an issue in the JupyterHub team compass repository
to discuss the possibilities of working together!

We’re excited about the ability for the Binder community continuing
to grow, and to build more tools for distributed, community-run
infrastructure for open science and education. This is all possible
because of the hard work of many people in the community, so a big
thank you to those who spend their time working on open tools in
the Jupyter and Binder ecosystems. We’re excited to see what comes
next!

Incident Report: Jupyter services down

2017-12-14T19:52:00+00:00

update: December 14, 20:45 UTC, all services should be restored and back up.

On December 13, at 22:10 UTC (4:10pm EST), a large number of Jupyter-provided services stopped responding. This included, but was not limited to https://nbviewer.jupyter.org, https://try.jupyter.org (powered by tmpnb) and https://cdn.jupyter.org. We quickly narrowed this down to an issue with our hosting provider and have been working with them to resolve the issue as fast as possible.

When outages happen, the Jupyter Status page should show which services are affected and we publish updates there.

How are Jupyter services hosted?

To understand the cause of the outage, we need to understand how the Jupyter services are hosted and maintained. As Jupyter is an open organization which is mostly maintained by volunteers, we do not have a dev-ops team assigned to maintaining our infrastructure. Even with full-time developers hired through universities or companies, the time spent fixing infrastructure is taken on nights and weekends. These developers are often stretched thin and cannot be available 24/7.

Most of our cloud infrastructure is donated to us by companies like CloudFlare, Rackspace, Fastly, Google, and Microsoft. Donating resources can be challenging, both technically and legally. In this particular case, Rackspace graciously created a special account for Jupyter that handles invoices on our behalf, thereby making resources free to the project. Following a hiccup, this Jupyter account was suspended and all services are unavailable as a result.

Temporary resolution

As nbviewer is one of the most used services provided by Jupyter, we’ve moved it to one of our personal account at another cloud-provider. Fastly was set up to load-balance on the yet-to-come-back-up instances as well as this newly created instance, so all should be fine now.

The other services (tmpnb, mails@jupyter.org, cdn.jupyter.org, …) will still unavailable or highly degraded until a permanent solution is found, or the services are restarted. try.jupyter.org will likely redirect to a repo on https://mybinder.org in the meantime so people can still try out Jupyter.

Low bus factor

The outage of all these services lasted for a significant time (more than 18 hours). Which perturbed many of you relying on these services. We understand that this is hardly acceptable and we hope you’ll indulge us as these services are provided for free and without ads. One of the factors leading to the slow reestablishment of service was a relatively low bus factor, with only one and a half of our developers knowing how to deploy and maintain these services. Documentation and access to credentials was also limited.

This is one of the challenges in a distributed team like Jupyter where contributors self-organize. It is easy to forget that new code is not the only way to contribute and that infrastructure and maintenance are crucial.

We also overly rely on a single vendor (in this case Rackspace), and while we are happy with Rackspace and have no reason to move to another provider, we should have a plan to restore critical services even temporarily in case of failure.

A couple of months ago, the subject was brought to our attention, and we developed a plan to move many of our deployment to Kubernetes (which is provider agnostic). We underestimated the probability to need an emergency plan this early.

How can you help

Jupyter is mainly governed by the community all around the world. Contributing is not limited to writing code! We need members with knowledge in multiple languages, in design, dev-ops, etc. Whether you are an expert, or still learning, we would like you to get involved.

Thanks everyone for your patience and the kind words when you reached to us when discovering the services were down.