Jupyter Blog - Sarah Gibson

Join JupyterHub on Zulip chat!

2025-02-06T16:05:00+00:00

The JupyterHub team have just moved all their chat to a Zulip channel!

The Jupyter Zulip instance is a Jupyter-wide chat space allowing cross-team communication in channels, some of which can be viewed publicly without an account.

For the JupyterHub team, we have found better engagement from team members and across the wider eco-system by using the Zulip instance. To that end, we recently updated all of our Gitter rooms to “invite-only” so no new members could join, and left a pinned message with a link to the JupyterHub channel on Zulip and an invite to join the conversation there. We have also updated any of our repositories that had Gitter links to point to Zulip as well.

If you would like to keep in touch with the JupyterHub, we would love to hear from you over on Zulip!

Online Collaboration Café launch: JupyterHub team meetings to become more collaborative spaces!

2023-02-27T08:49:00+00:00

This illustration is created by Scriberia with The Turing Way community. Used under a CC-BY 4.0 licence. DOI: 10.5281/zenodo.3332807

The JupyterHub team are refactoring our monthly meeting into a collaborative, co-working space that is more accessible and inclusive to those who are just getting started in the community — an Online Collaboration Café! This blog post aims to explain what that means, and what to expect when you attend. We plan to hold our first Online Collaboration Café on 21st March 2023. Please come along and let us know your feedback!

TL;DR

The team meeting will be held in a new format for 2 hours in the original time slot. Instead of a typical meeting style with an agenda, the group will be divided into breakout rooms to work more collaboratively on ideas and perhaps begin actioning them. There will always be breakout rooms available for onboarding newcomers and those who wish some dedicated time to undertake maintenance tasks, as well as a quiet working space in the main room. Participants should feel free to swap rooms and join/drop out as they need.

Why the change?

The JupyterHub community often cite the team meetings as a touchpoint for newcomers to familiarise themselves with the JupyterHub project. However, this is not always the case. The team meetings have an emergent agenda built by the community members — which is great! — but it also means that it is a potluck as to whether the meeting you happen to attend will actually be useful depending on the agenda, and a newcomer may have to attend multiple meetings over a long period before feeling comfortable to ask questions and know where they can begin to help.

By reformatting the meeting into a collaborative co-working space using breakout rooms, we can cater for both the need of newcomers to be oriented to the project, and for the community to discuss in-depth topics in an emergent nature.

What is an Online Collaboration Café?

The Collaboration Café is a concept that was developed by The Turing Way community, and you can read more about it and how they are run in their Community Handbook.

In short, an Online Collaboration Café utilises breakout rooms and pomodoro sprints to allow groups of community members to work together on a topic that best suits them. The space between the pomodoros are used as shareouts to the rest of the group, or as biobreaks.

What are the logistics?

This is an online café. We will meet on a video call.
We will be using the same alternating time slot that the team meetings used to occur in, but the slot will now be two hours. You can view the Team Calendar for details.
A chair will always be present in the main room to manage breakout rooms and greet folk who may arrive mid-sprint.
There will always be some default breakout rooms available:

Onboarding: For new arrivals to the community. A member of the team will be available on the call to support anyone wanting to learn more about collaborating on GitHub, getting a virtual tour of our GitHub organisation, and help you in any way we can.
Maintenance: Folks working in this room will be triaging issues, reviewing pull requests, and other maintenance-related activities across the JupyterHub organisation.
Quiet working in the main room: If you would just like some quiet time dedicated to any JupyterHub-related work you have, you are invited to hang out in the main room (and keep the chair company 😉)

What will happen when I attend?

When you arrive to the Online Collaboration Café, there will first be some housekeeping, such as introductions, Code of Conduct review. There will then be some goal setting around what folks are hoping to achieve during the time. These goals will determine the topics for the breakout rooms. Any attendee is welcome to: suggest their own topic, join a suggested topic, join one of the default rooms, or work quietly in the main room. The breakouts will then run in sprints with breaks to share their progress. The Café is closed with some reflections, for example: how did your work progress, what should the project think about working towards next?

Here is an example schedule of how an Online Collaboration Café could happen inspired by The Turing Way:

Time: Activity
Start: Welcome, CoC review
10 mins: Introductions and goal setting
20 mins: Pomodoro 1
5 mins: Break
20 mins: Pomodoro 2
5 mins: Break
20 mins: Open discussion: celebrations, reflections, future plans
5 mins: Close

2 hours seems like a long time… Why not multiple meetings?

Various individual meetings covering separate topics, such as onboarding and maintenance, were considered. However given that the majority of JupyterHub’s community are volunteers, it didn’t seem practical to fill up the calendar with lots of meetings. Another reason to use parallel breakout rooms is that participants can swap rooms if the current conversation doesn’t appeal to them, as if you were at tables in a real café. This practice is highly encouraged.

The extension to 2 hours is important because this is a pivot towards collaboration and co-working, and we want to provide a dedicated time for folk to achieve that and begin actioning ideas. However it is a large chunk of time to devote when we are all busy, and so it is not a requirement to arrive on time and participate for the full 2 hours. Joining when you can and dropping when you need to is encouraged and doesn’t require apologies.

Will there be cake at this café?

It is an online café so there will be as many virtual cakes as you like! 🍰🍰🍰

This is an informal space, so you are encouraged to bring along any beverages and/or snacks. We hope you will join us!

Introducing JupyterHub’s Outreachy interns! — December 2022 Cohort

2022-12-07T07:45:00+00:00

As part of the community strategic support project funded by CZI’s EOSS grant series, the JupyterHub sub-project has funding to support Outreachy interns through four cohorts over the next two years. We would like to introduce you to the first cohort and the projects they will be working on!

Ogoh Blessing Onyowoicho — Improve Accessibility in JupyterHub

Accessibility is the ability of tools (in our case web tools) to be used by a variety of communities with different disabilities. There are a variety of standards and tools for evaluating and ensuring that a web page can be used effectively by as many people as possible. Work by the Accessibility team is ongoing to define a set of standard tools to improve accessibility across the Jupyter ecosystem.

The JupyterHub project is working to improve the accessibility of its pages to ensure we are providing tools that are as useful as they can be to as many people as we can. During the internship, we will evaluate JupyterHub’s accessibility, find ways to improve it, and integrate accessibility testing into the development process, in collaboration with the Accessibility team, to ensure we do a better job going forward.

Ogoh Blessing says:

My name is Ogoh Blessing Onyowoicho. I am a self-taught Frontend developer based in Lagos, Nigeria and I am an Outreachy intern working on Improving the accessibility of JupyterHub.
I am excited to work on JupyterHub because it is giving me the opportunity to use skills that I have accrued over the years to proffer solutions to problems that affect people’s lives directly. On hearing accessibility, the first thought one has is that it involves building web products that people with special needs can use seamlessly. Though this is part of it, accessibility goes way beyond this. It involves building products that different users (e.g users at different locations, users with different devices etc) can use easily. The thought of contributing to improving the experience of so many people alone excites me.
In the coming months, I hope to learn and continue to hone my skills as I am guided by my mentors and members of the community I get to interact with.

Allan Wasega — Restructure and improve JupyterHub documentation by implementing the Diátaxis framework

JupyterHub has a range of documentation that covers both developer and user audiences in order to help them deploy, maintain, and use their own instance of a JupyterHub. The success of an open source software project to (i) be adopted by users, and (ii) receive meaningful contributions relies heavily on the quality, navigability and accessibility of documentation so that users and developers have all the information they need to achieve what they want to do.

A framework for organising technical documentation has arisen called diátaxis. It takes a systematic approach to understanding user requirements of documentation throughout the lifecycle of interaction with a product and posits that different user needs require different approaches in creation of the documentation, as well as a layout to navigate these different “modes” of documentation.

This project will focus on a refactoring of the documentation for the JupyterHub package. We will begin by performing a review of the present documentation, categorise these into the diataxis framework, and then restructure the documentation files in the repository. Once we have transformed the documentation into this framework, it will be much easier to identify missing and unclear documentation (those that were difficult to categorise). We can then begin to curate resources that can fill the gaps and improve documentation that is not specific enough.

Allan says:

I am Allan Wasega, from Nairobi, Kenya. Broadly, I like to describe myself as a researcher and a writer. Researcher because looking into things to find patterns or hidden information has always been of interest to me. Writer because I figured early on that words allow me to express myself and to communicate to a larger audience than that inside my head :). As a Computer Science student, I looked for ways to bring these two skills together within the realm of computing and that is how I found myself in the technical writing space.
As an undergraduate student, I used Jupyter Notebooks extensively for most of my programming assignments and projects. As a result, when making my Outreachy application, the JupyterHub project immediately piqued my interest because I did not know that there was a way to serve Jupyter Notebooks to multiple users, such as within a classroom setting. Presently, I serve as a mentor at KamiLimu, which is a mentorship program for students pursuing technology-aligned courses in tertiary institutions in Kenya. One of the objectives of KamiLimu is to introduce students to tech specializations such as Data Science and Machine Learning, which, from experience, uses Jupyter Notebooks! Therefore, I am excited to work on JupyterHub because I will be learning about and helping to improve a product I hold dear and which I can use to advance the skills of the next generation of computer technologists in Kenya.
First, I hope to gain a deep understanding of how JupyterHub works so as to spread the word about it and its functionalities. Second, I hope to apply and advance my technical writing skills. I encountered the Diataxis Framework earlier this year while working as a technical writer at Tingle Software, a Nairobi-based software company. I have also been following Daniele Procida (the author of the framework) on Twitter for a while now and his work has been quite inspiring. Through this project, I will apply the Diataxis Framework to restructure JupyterHub’s documentation and, in doing so, further my understanding of the framework. Finally, I am passionate about mentorship, especially in the tech field. Therefore, besides expanding my professional and personal networks, this project will enable me to learn how to conduct mentorship within a global setting, and I can, in turn, apply this knowledge within my local community.

Sheila Kahwai — Create a reusable JupyterHub pytest plugin

JupyterHub is a modular and extensible project, with parts, like the proxy, authenticator and spawner, that can be easily changed and extended. Testing the functionality of these components against JupyterHub is important and it requires various hub setups that can sometimes become complicated.

Currently, each of these hub components and the hub itself define their own testing infrastructure, building everything from the ground up using the pytest framework. But some of this complex work is either repetitive across JupyterHub sub-projects, or under-specified for some of them.

This project will abstract out these common parts into a separate testing framework. This framework should be a pip-installable pytest plugin that would provide various hub functionalities through pytest fixtures. These fixtures can then be re-used by JupyterHub and its components to bootstrap their own testing suite.

Integrating this plugin will drive some important refactoring work of the current testing architecture of JupyterHub and have a great impact in improving the overall test coverage, maintainability and continuity of the JupyterHub project.

Sheila says:

My name is Sheila Kahwai. I am a self-taught python developer from Nairobi, Kenya, working to specialize in back-end engineering.
I am excited to work on JupyterHub because it has made many products I have used throughout my learning journey. It is a privilege to work with a diverse community that has created products that are very useful to equally diverse users like myself.
While creating a reusable JupyterHub pytest plugin, I hope to gain more insight into creating plugins for massive codebases to improve maintainability and scalability with clean and reusable code. Through this project, I look forward to improving the overall testing infrastructure of the various JupyterHub components.

Welcome to the interns! We’re so excited to start working with you!

How I automated authorised cloud deployments from Pull Requests with GitHub Actions

2021-11-22T18:30:00+00:00

I recently did some work on the mybinder.org deployment infrastructure to solve a problem with testing Pull Requests before deployment. It had not been possible to test Pull Requests on our staging deployment because our automated workflows don’t have access to secrets. This resulted in my writing the test-this-pr action and this blog is a retrospective of what I learned over that process.

What was the problem we were trying to solve?

It is generally considered best practice to have a staging environment when running complex applications or platforms that have a large userbase, such as mybinder.org. This provides developers a space to test infrastructure changes safely with the knowledge that users won’t be affected should anything go wrong. The deployment configuration for mybinder.org is managed in a GitHub repository and deploys are handled in a CI/CD pipeline run on GitHub Actions.

The mybinder.org operating team’s usual workflow will be very familiar to anyone working as part of a distributed development team: fork the main repository, do some development (fix a bug or implement a feature), and then open a Pull Request (PR) back to the default branch of the main repository. The difficulty arises when we want to test the PR on our staging infrastructure before merging and deploying for real.

Various secrets are required to make a deployment, even to our staging environment. These include a git-crypt secret that decrypts certain sensitive files and credentials to push Docker images to the repositories connected to our clusters, among others. These are stored as encrypted repository secrets so they are available to GitHub Actions when our CD pipeline runs. However by default, repository secrets are not made available to PRs from forks, which introduces a problem for the workflow I described above.

So to summarise our problem:

Contributors often make changes to the code in their fork of the repository
Testing any change requires deploying to an actively-running staging cluster
The secrets needed to make those deploys are only available to the original repository, not to a forked repository
So we needed a way to trigger a test deployment on changes from a forked repository but that ran in the origin repository

What solutions did we already have in place?

We had one team practice, and one technical solution to address this issue, both of which didn’t quite do what we wanted.

One solution was a team deployment practice of “just merge the PR anyway and file a revert PR if things go wrong”. Our CI/CD pipeline is broken into two jobs: deploy to staging infrastructure, and a matrix deployment to each production cluster in the BinderHub federation. The production jobs require the staging job to pass first before they are triggered. This means that if the deploy to staging fails, the changes will never propagate to our production clusters.

Another solution was a “test-staging” label we could apply to open PRs. This triggered the deploy to staging job without a merge. However, this only worked for PRs opened from branches within the origin repository, and as mentioned above, this is not a common workflow for maintainers. Instead they open PRs from forked repositories, which do not have the proper secrets to deploy to staging.

What other solutions were available?

We briefly considered ok-to-test but were put off by the requirement for a GitHub App. However as we will learn, it was probably a mistake to dismiss this so quickly!

What did I end up implementing?

And so I built my first GitHub Action!

The test-this-pr action allows you to trigger a GitHub Action from the origin repository when an authorised user adds a comment to the PR of a forked repository.

test-this-pr is triggered when a user leaves a comment on the PR you’d like to test, but only if the commenter has the appropriate permissions on the repository, e.g. MEMBER. This protects us from malicious code being run automatically on our staging deployment and ensures that project maintainers have vetted the code in some way before triggering the test.

When triggered, test-this-pr uses a small Python script that queries the GitHub API to do the following:

It creates a new branch in the origin repository that uses the merge reference from the PR.
This triggers the test suite in the new branch, because we have configured the origin repository to automatically run deployment tests on branches that match a particular name pattern (in our case, test-this-pr/*, see below)
Adds comments on the original PR with a link to the newly-running CI test logs.

This script is wrapped up into a Dockerfile, which allows GitHub Actions to run the action using its docker runner.

An example of the test-this-pr workflow in action.

Extra configuration needed to use this action

There were a few extra pieces I configured that couldn’t (and shouldn’t!) be implemented in the test-this-pr action, but contribute to the overall workflow.

I also created a CI job that polls the running staging job. Upon completion, it posts a second comment to the original PR stating whether the test passed or failed. This had to be separate from the test-this-pr action since this job needs to run within the same workflow run in order to access the status of the staging job.

Once the pass/fail status of the staging job has been reported, the branch created by test-this-pr is then deleted. We decided this was the best path forward so that the original PR remained the “single source of truth” and maintainers wouldn’t be confused by extra branches in the repository.

For reference if you’d like to understand this implementation

What did I learn along the way?

There is A LOT you can do with the GitHub REST API

I had originally configured the Python script to shell out to git for tasks like creating the new branch and pulling the PR merge ref, and this worked fine when I ran Python locally. However when running the docker container, I struggled to authorise git correctly, even when passing my Personal Access Token. And so I made the decision to switch to the GitHub REST API.

While a lot of GitHub native features can be accessed and manipulated via the API, some git-specific actions can be achieved as well via the git database endpoint, which is the endpoint to use to create branches, commits, and much more. This section of the API might require a deeper dive into the inner-workings of git, but can unlock a lot of automation potential once you have your mental model!

Switching to the GitHub REST API from the git command line meant that I no longer needed to install git into my docker container or locally clone the repository to carry out the actions, making my image much smaller.

Triggering GitHub Actions from other Actions

You can’t trigger another GitHub Actions workflow from an event that was authorised with the GITHUB_TOKEN from within a previous workflow. This is why the comments left by the test-this-pr workflows in the Binder repository come from my account as I had to provide a Personal Access Token for the staging job to be correctly triggered.

The downside of this is that I’m automatically subscribed to notifications for any Binder PR where someone is using test-this-pr. It’s not the worst situation in the world as I have email filters set up, but also not ideal.

This is also why ok-to-test recommends setting up a GitHub App for authorisation, so you don’t have to sacrifice your own account and all the notifications that come with it!

However, this “actions triggering actions” scenario could be smoothed out by using the upcoming Reusable Workflows beta.

Building Actions with Dockerfiles

Providing a Dockerfile with a custom Action will allow you to write your Action in whatever language you like. However, it is one of the slowest ways to implement custom Actions because GitHub’s runner will build the image every time the workflow is triggered.

What would I do differently now?

Setup a GitHub App for authorisation

If only to spare my inbox from the notifications! 😂

Move away from a Python implementation

The test-this-pr action ultimately ended up being a handful of calls to the GitHub REST API. As opposed to maintaining a Python script and Dockerfile, I would probably reimplement this using the github-script action, which is a JavaScript wrapper for the GitHub API that can be directly run from GitHub Actions. This will be less code to maintain and the test-this-pr action itself could then be reworked as a composite action which would build faster than the current Docker-based implementation.

Closing Remarks

Members of the mybinder operating team discussing test-this-pr in gitter

While there is a lot I would change about test-this-pr, including fundamental changes that would make it look radically different, I needed to go on this developmental journey to gain the better understanding of GitHub Actions that I have now. And hopefully by sharing that journey, other folk can learn from it!

Also the mybinder operating team love using this feature and it’s adding value to our development workflow, so I don’t see a need to dive back in and start changing things again right now 🙂

You can read the source code for test-this-pr and mybinder.org’s deployment workflow in full.

Diving into Leadership to Build Push-Button Code

2019-10-15T18:34:00+00:00

“Hi everyone, I’m Sarah! I’m a Research Data Scientist at the Alan Turing Institute and I’m also an operator of mybinder.org. It’s really cool seeing how many people here are interested in BinderHub!”

And it is cool. Really cool! But also a bit scary as a room full of Research Software Engineers (each of them much further on in their careers than I am) suddenly turn to me, eager for the knowledge I was surely about to impart to them.

But let’s rewind a bit.

Joining the Binder community

Its May 2019 and I’m attending the Research Software Reactor sprint jointly hosted by Microsoft and Imperial College London. This is a 3-day hackathon where researchers from different areas (though all with a computing background) come together to collaboratively build cloud-based resources on Microsoft’s Azure platform.

As for me, I’m only 6 months into my role at the Turing, having graduated from my PhD in Astrophysics at the start of the year. Also, my role as a “Binder Operator” is barely 2 months old… which is why I’m starting to feel nervous about the amount of interest in the room! This also happens to be the second hackathon I’ve attended ever, so the adrenaline is high.

So, what is this “Binder” that we’re all so excited about?

Binder is an amazing resource that can host reproducible and interactive code in a web browser. Great… what does that mean? It means that if I have some scripts or notebooks in a repository (like this one) and I describe the packages in a configuration file (such as requirements.txt), then I can go to mybinder.org, copy the URL of the repository into the form and hit launch. This will begin a series of events culminating in my notebook appearing in a browser window with all of the packages installed, and the code will just run.

Sounds like magic, right? You can even combine Binder with Jupyter Books to create interactive documents! Below is a comic explaining how a scientist may use Binder.

The user experience of mybinder.org. Comic courtesy of Juliette Taka.

BinderHub is the technology powering Binder (or where the magic happens). It is a multi-user server that can create a custom, user-specified computing environment and make it accessible via a URL. It utilizes different tools to make this possible and can be deployed onto either a cloud provider or an on-premise compute cluster. These tools are depicted in the below illustration.

A pictorial representation of the different tools constituting BinderHub. This image was created by Scriberia for The Turing Way community and is used under a CC-BY licence. Zenodo record.

Since BinderHub is a cloud-neutral technology (mybinder.org itself runs on Google Cloud and OVH.com), I previously worked on refining the documentation around deploying BinderHub on Azure and ran workshops teaching people how to use Binder and deploy BinderHub. It was this work that lead to me being invited to join the Binder team. Being a maintainer for an open-source project involves checking if the service is still running smoothly and being an active, friendly member of the community — one who can answer questions when they arise.

Now that we’ve covered a bit of background, let’s get back to my slightly uncomfortable moment at the sprint.

Day 1: An unexpected leadership position

Everybody in the room is now trying to whittle down which projects we should work on — all of which are awesome! The magic of BinderHub is that it covers so many aspects of software engineering tools and practices, such as continuous integration/deployment and Kubernetes. The BinderHub idea is quickly amassing a lot of satellite projects!

“So Sarah, is it OK if I name you leader of ‘Team BinderHub’?” My second uncomfortable moment. Gerard Gorman, Senior Lecturer in the Department of Earth Sciences at Imperial and co-organizer of the sprint, has just elected me leader of my first ever hack project. If I’m honest, I’d intended to spend the sprint ironing out some niggles with a colleague. But I’ve always been a “learn by doing” person so I agree and desperately begin wracking my brain for a project to give my team for the next 3 days.

After a bit of shuffling (where for a while it seemed like most of the attendees would be joining ‘Team BinderHub’!), I finally assemble a team. My teammates are: Tania Allard, a Microsoft Developer Advocate; Tim Greaves and Diego Alonso Álvarez, Research Software Engineers at Imperial; and Gerard himself.

The project I decide to bring to the table is one that I’d started but had stalled. The process of deploying a BinderHub can be quite long (usually an hour) and consists mainly of typing stuff into the command line. So a couple of months prior, I’d begun designing a set of scripts that would install the relevant tools, deploy a BinderHub on Azure and connect it to a Docker Hub account, retrieve various pieces of information (IP addresses, logs, etc.) and then remove the BinderHub from the cloud. However, my bash scripting and operating system knowledge is not especially strong. Within the team, we have a bit of a discussion around the Deploy to Azure feature — a button that can be copy-pasted into the README of a GitHub project and facilitates a one-click deployment of the project to Azure. It seems like the perfect hack is to combine my scripts with the deploy button and make deploying BinderHub as easy as possible.

Tim is a bash scripting, containerizing pro and has a little experience with the Azure “blue button” from the OKPy project. I ask him to check my setup script that installs the required Command Line Interfaces (CLIs) and explain that I’d like it to be generalized for different operating systems.

Gerard is a BinderHub enthusiast and is keen to get one deployed as a teaching resource at Imperial. As a starter task, I ask him to read up on how the blue button works.

Diego has never heard of Binder or BinderHub before and is quite confused as to what the rest of the team are so excited about! I suggest that he works through my Zero to BinderHub workshop in order to bring him up to speed with the concepts. This is also the perfect opportunity to get feedback on my workshop if parts are not clear! I encourage him to open an issue describing any problems he comes across or extra information he’d like to see.

And just like that, I find I’ve delegated myself out of a job!

Now Imposter Syndrome is beginning to creep up on me. Not only is it taking four extra people to pull together my work and make it usable, but I am also unsure how I would assist them in these tasks I’d set them. Especially Tim, as the complexity of what I wanted the bash scripts to achieve was the reason this project stalled in the first place.

Except there’s one very important aspect of any software project I’ve not mentioned yet: Documentation!

Since starting my position at the Turing, I’ve come to fully appreciate how fundamental good documentation is to the success of a project. Clearly-written instructions covering the purpose of the project, how to install/run the code, and what kind of inputs/outputs to expect make it much easier for a new person to quickly get to grips with the project. And as result, it’s far more likely to be referenced and reused. While my team begin familiarizing themselves with the infrastructure and goals of my project, I begin an overhaul of the documentation whilst keeping an eye on the repository to manage incoming issues and pull requests.

Take that, Imposter Syndrome!

Day 2: The team makes progress

It’s the second day of the sprint and ‘Team BinderHub’ gather again, our enthusiasm not diminished yet!

I very quickly set goals for the day. I want the button to work by the end of the day as I’m hoping the final day can be used to work on an idea Tania has to use Azure’s DevOps Pipelines to automatically update the deployed BinderHub as new commits come into the host repository. This would be a very handy feature for those (like me!) maintaining BinderHubs at their own institutions. Anything to automate and reduce the number of commands we have to type!

I ask Tim to begin working on the deploy script itself, to add tests where necessary and tidy up some of the parsing of the variables. The next steps are to build a Dockerfile that runs the deploy script and an ARM (Azure Resource Manager) template that controls the form that the blue button links to.

Again, I’m managing the documentation, keeping up with changes we make to the code-base and functionality. Similarly, I keep watch over the repository to manage incoming pull requests and merge conflicts and also make a start on the amendments to the BinderHub workshop Diego has compiled.

My Imposter Syndrome is definitely subsiding as I begin to feel more like I understand the skills of my team and we are all making the best use of our time.

Day 3: Document, document, document

For the third and final day of the sprint we are in a new venue: the Microsoft Reactor. This is a workspace in the Shoreditch area of London where we are offered free pizza and cookies and a DJ to provide the soundtrack to our code. (OK, less an actual DJ, more of a software engineer with Spotify Premium. 😜) What I will say though, Microsoft’s beverage-making facilities have nothing on the Turing’s iPad coffee maker! 😉

Our goals for the last day? Consolidate and document! Ideally get the button working if possible, but the top priority is to make it as easy as possible for the team (or someone new!) to come along to the repository and finish what we have started.

We decide that there isn’t enough time left or infrastructure in place to implement the Azure DevOps Pipeline for automatic upgrades; instead, Tania begins working on a tutorial so we can implement it later.

Tim and I end up in a bit of GitHub hell as we realize that the code to create the Kubernetes cluster is in the setup script, not the deploy script. We refactor some parts so that all of the resources are deployed from the deploy script, and this also means that the Dockerfile only needs to find one script to execute. However, this refactoring causes a complicated merge conflict and some of the bug fixes Tim implemented disappear in a squash merge. (This sounded difficult enough to resolve that I’ve been put off learning about squash merges since!) As team leader, I try to orchestrate whose pull request should be merged first so that all the right code ends up in master.

What did we learn and do?

By the end of the sprint, we don’t quite have a working button deployment, but we do have:

a set of streamlined scripts for auto-deployment based on the contents of a JSON file,
an ARM template and Dockerfile that will provide the backend to the blue button after some further debugging,
a plan to move the project forwards after the sprint.

So what did I learn from those three days?

Coding with other people is fun!

Paired programming is something we try to achieve at the Turing, but it can be difficult to do depending on the project and the time constraints of those working on it.

This was the first time I’d experienced true collaboration. Where I had an idea that I wasn’t sure how to implement, and someone with the skills had helped me shape it and realize it. It gave me a sense of community and belonging. I hope I’ve forged connections with ‘Team BinderHub’ that will last the duration of our careers and we can work together again.

There’s a role for everyone in a team and these are equally important

Even if it’s reviewing code or writing documentation, these are as important (arguably, more so) than the code itself. Code that does what you think it’s doing and has been explained well will have a much longer lifespan than code that is difficult to follow, regardless of how clever it is, and poorly documented.

I found my strength as a leader

You might remember at the beginning of this blog post I mentioned that I’d never led a hack project before, so I also learned a lot about my leadership capabilities.

I think I did well. I identified the strengths of each of my team members and gave them a task suited to them whilst letting them explore new concepts, offering my own insight and opinions when required.

I also think that it was a good choice for me to not be too involved in the coding aspect of this project. Managing the flow into the repository and updating the documentation as new code came in meant that I managed to maintain an overall perspective of the project and could switch gears as questions came in from different areas. I don’t think I could have maintained such a view if I’d been buried in code, and I can always learn from the scripts that we’ve developed at a later point.

Hackathons are not where projects end

While the first 80% of a project to get the infrastructure in place can be achieved in a short amount of time like a hackathon, the last 20% is hard and often takes longer. But this extra effort is necessary for the software to be taken up by others.

‘Team BinderHub’ continued working on this project over Slack and GitHub to finally make our version 1 release on June 11th — almost 3 weeks after the end of the sprint!

If you’d like to try the button to deploy your own BinderHub (or contribute a new feature!), the repo can be found here 👉 github.com/alan-turing-institute/binderhub-deploy. Look for the button below!

Thank You! 💖

I’d like to thank a few people who made this possible:

Gerard, Imperial College, Tania and Lee Stott from Microsoft for organising such an inspiring event;
‘Team BinderHub’ for coming along on this wild ride with me;
The Binder Team for accepting me into the community and giving me the space to develop such projects;
and The Turing Way team who introduced me to Binder and the value of community (and documentation!).

—

Note: this is cross-posted with the Turing Institute blog.