Jupyter Blog - hackathons

Diving into Leadership to Build Push-Button Code

2019-10-15T18:34:00+00:00

“Hi everyone, I’m Sarah! I’m a Research Data Scientist at the Alan Turing Institute and I’m also an operator of mybinder.org. It’s really cool seeing how many people here are interested in BinderHub!”

And it is cool. Really cool! But also a bit scary as a room full of Research Software Engineers (each of them much further on in their careers than I am) suddenly turn to me, eager for the knowledge I was surely about to impart to them.

But let’s rewind a bit.

Joining the Binder community

Its May 2019 and I’m attending the Research Software Reactor sprint jointly hosted by Microsoft and Imperial College London. This is a 3-day hackathon where researchers from different areas (though all with a computing background) come together to collaboratively build cloud-based resources on Microsoft’s Azure platform.

As for me, I’m only 6 months into my role at the Turing, having graduated from my PhD in Astrophysics at the start of the year. Also, my role as a “Binder Operator” is barely 2 months old… which is why I’m starting to feel nervous about the amount of interest in the room! This also happens to be the second hackathon I’ve attended ever, so the adrenaline is high.

So, what is this “Binder” that we’re all so excited about?

Binder is an amazing resource that can host reproducible and interactive code in a web browser. Great… what does that mean? It means that if I have some scripts or notebooks in a repository (like this one) and I describe the packages in a configuration file (such as requirements.txt), then I can go to mybinder.org, copy the URL of the repository into the form and hit launch. This will begin a series of events culminating in my notebook appearing in a browser window with all of the packages installed, and the code will just run.

Sounds like magic, right? You can even combine Binder with Jupyter Books to create interactive documents! Below is a comic explaining how a scientist may use Binder.

The user experience of mybinder.org. Comic courtesy of Juliette Taka.

BinderHub is the technology powering Binder (or where the magic happens). It is a multi-user server that can create a custom, user-specified computing environment and make it accessible via a URL. It utilizes different tools to make this possible and can be deployed onto either a cloud provider or an on-premise compute cluster. These tools are depicted in the below illustration.

A pictorial representation of the different tools constituting BinderHub. This image was created by Scriberia for The Turing Way community and is used under a CC-BY licence. Zenodo record.

Since BinderHub is a cloud-neutral technology (mybinder.org itself runs on Google Cloud and OVH.com), I previously worked on refining the documentation around deploying BinderHub on Azure and ran workshops teaching people how to use Binder and deploy BinderHub. It was this work that lead to me being invited to join the Binder team. Being a maintainer for an open-source project involves checking if the service is still running smoothly and being an active, friendly member of the community — one who can answer questions when they arise.

Now that we’ve covered a bit of background, let’s get back to my slightly uncomfortable moment at the sprint.

Day 1: An unexpected leadership position

Everybody in the room is now trying to whittle down which projects we should work on — all of which are awesome! The magic of BinderHub is that it covers so many aspects of software engineering tools and practices, such as continuous integration/deployment and Kubernetes. The BinderHub idea is quickly amassing a lot of satellite projects!

“So Sarah, is it OK if I name you leader of ‘Team BinderHub’?” My second uncomfortable moment. Gerard Gorman, Senior Lecturer in the Department of Earth Sciences at Imperial and co-organizer of the sprint, has just elected me leader of my first ever hack project. If I’m honest, I’d intended to spend the sprint ironing out some niggles with a colleague. But I’ve always been a “learn by doing” person so I agree and desperately begin wracking my brain for a project to give my team for the next 3 days.

After a bit of shuffling (where for a while it seemed like most of the attendees would be joining ‘Team BinderHub’!), I finally assemble a team. My teammates are: Tania Allard, a Microsoft Developer Advocate; Tim Greaves and Diego Alonso Álvarez, Research Software Engineers at Imperial; and Gerard himself.

The project I decide to bring to the table is one that I’d started but had stalled. The process of deploying a BinderHub can be quite long (usually an hour) and consists mainly of typing stuff into the command line. So a couple of months prior, I’d begun designing a set of scripts that would install the relevant tools, deploy a BinderHub on Azure and connect it to a Docker Hub account, retrieve various pieces of information (IP addresses, logs, etc.) and then remove the BinderHub from the cloud. However, my bash scripting and operating system knowledge is not especially strong. Within the team, we have a bit of a discussion around the Deploy to Azure feature — a button that can be copy-pasted into the README of a GitHub project and facilitates a one-click deployment of the project to Azure. It seems like the perfect hack is to combine my scripts with the deploy button and make deploying BinderHub as easy as possible.

Tim is a bash scripting, containerizing pro and has a little experience with the Azure “blue button” from the OKPy project. I ask him to check my setup script that installs the required Command Line Interfaces (CLIs) and explain that I’d like it to be generalized for different operating systems.

Gerard is a BinderHub enthusiast and is keen to get one deployed as a teaching resource at Imperial. As a starter task, I ask him to read up on how the blue button works.

Diego has never heard of Binder or BinderHub before and is quite confused as to what the rest of the team are so excited about! I suggest that he works through my Zero to BinderHub workshop in order to bring him up to speed with the concepts. This is also the perfect opportunity to get feedback on my workshop if parts are not clear! I encourage him to open an issue describing any problems he comes across or extra information he’d like to see.

And just like that, I find I’ve delegated myself out of a job!

Now Imposter Syndrome is beginning to creep up on me. Not only is it taking four extra people to pull together my work and make it usable, but I am also unsure how I would assist them in these tasks I’d set them. Especially Tim, as the complexity of what I wanted the bash scripts to achieve was the reason this project stalled in the first place.

Except there’s one very important aspect of any software project I’ve not mentioned yet: Documentation!

Since starting my position at the Turing, I’ve come to fully appreciate how fundamental good documentation is to the success of a project. Clearly-written instructions covering the purpose of the project, how to install/run the code, and what kind of inputs/outputs to expect make it much easier for a new person to quickly get to grips with the project. And as result, it’s far more likely to be referenced and reused. While my team begin familiarizing themselves with the infrastructure and goals of my project, I begin an overhaul of the documentation whilst keeping an eye on the repository to manage incoming issues and pull requests.

Take that, Imposter Syndrome!

Day 2: The team makes progress

It’s the second day of the sprint and ‘Team BinderHub’ gather again, our enthusiasm not diminished yet!

I very quickly set goals for the day. I want the button to work by the end of the day as I’m hoping the final day can be used to work on an idea Tania has to use Azure’s DevOps Pipelines to automatically update the deployed BinderHub as new commits come into the host repository. This would be a very handy feature for those (like me!) maintaining BinderHubs at their own institutions. Anything to automate and reduce the number of commands we have to type!

I ask Tim to begin working on the deploy script itself, to add tests where necessary and tidy up some of the parsing of the variables. The next steps are to build a Dockerfile that runs the deploy script and an ARM (Azure Resource Manager) template that controls the form that the blue button links to.

Again, I’m managing the documentation, keeping up with changes we make to the code-base and functionality. Similarly, I keep watch over the repository to manage incoming pull requests and merge conflicts and also make a start on the amendments to the BinderHub workshop Diego has compiled.

My Imposter Syndrome is definitely subsiding as I begin to feel more like I understand the skills of my team and we are all making the best use of our time.

Day 3: Document, document, document

For the third and final day of the sprint we are in a new venue: the Microsoft Reactor. This is a workspace in the Shoreditch area of London where we are offered free pizza and cookies and a DJ to provide the soundtrack to our code. (OK, less an actual DJ, more of a software engineer with Spotify Premium. 😜) What I will say though, Microsoft’s beverage-making facilities have nothing on the Turing’s iPad coffee maker! 😉

Our goals for the last day? Consolidate and document! Ideally get the button working if possible, but the top priority is to make it as easy as possible for the team (or someone new!) to come along to the repository and finish what we have started.

We decide that there isn’t enough time left or infrastructure in place to implement the Azure DevOps Pipeline for automatic upgrades; instead, Tania begins working on a tutorial so we can implement it later.

Tim and I end up in a bit of GitHub hell as we realize that the code to create the Kubernetes cluster is in the setup script, not the deploy script. We refactor some parts so that all of the resources are deployed from the deploy script, and this also means that the Dockerfile only needs to find one script to execute. However, this refactoring causes a complicated merge conflict and some of the bug fixes Tim implemented disappear in a squash merge. (This sounded difficult enough to resolve that I’ve been put off learning about squash merges since!) As team leader, I try to orchestrate whose pull request should be merged first so that all the right code ends up in master.

What did we learn and do?

By the end of the sprint, we don’t quite have a working button deployment, but we do have:

a set of streamlined scripts for auto-deployment based on the contents of a JSON file,
an ARM template and Dockerfile that will provide the backend to the blue button after some further debugging,
a plan to move the project forwards after the sprint.

So what did I learn from those three days?

Coding with other people is fun!

Paired programming is something we try to achieve at the Turing, but it can be difficult to do depending on the project and the time constraints of those working on it.

This was the first time I’d experienced true collaboration. Where I had an idea that I wasn’t sure how to implement, and someone with the skills had helped me shape it and realize it. It gave me a sense of community and belonging. I hope I’ve forged connections with ‘Team BinderHub’ that will last the duration of our careers and we can work together again.

There’s a role for everyone in a team and these are equally important

Even if it’s reviewing code or writing documentation, these are as important (arguably, more so) than the code itself. Code that does what you think it’s doing and has been explained well will have a much longer lifespan than code that is difficult to follow, regardless of how clever it is, and poorly documented.

I found my strength as a leader

You might remember at the beginning of this blog post I mentioned that I’d never led a hack project before, so I also learned a lot about my leadership capabilities.

I think I did well. I identified the strengths of each of my team members and gave them a task suited to them whilst letting them explore new concepts, offering my own insight and opinions when required.

I also think that it was a good choice for me to not be too involved in the coding aspect of this project. Managing the flow into the repository and updating the documentation as new code came in meant that I managed to maintain an overall perspective of the project and could switch gears as questions came in from different areas. I don’t think I could have maintained such a view if I’d been buried in code, and I can always learn from the scripts that we’ve developed at a later point.

Hackathons are not where projects end

While the first 80% of a project to get the infrastructure in place can be achieved in a short amount of time like a hackathon, the last 20% is hard and often takes longer. But this extra effort is necessary for the software to be taken up by others.

‘Team BinderHub’ continued working on this project over Slack and GitHub to finally make our version 1 release on June 11th — almost 3 weeks after the end of the sprint!

If you’d like to try the button to deploy your own BinderHub (or contribute a new feature!), the repo can be found here 👉 github.com/alan-turing-institute/binderhub-deploy. Look for the button below!

Thank You! 💖

I’d like to thank a few people who made this possible:

Gerard, Imperial College, Tania and Lee Stott from Microsoft for organising such an inspiring event;
‘Team BinderHub’ for coming along on this wild ride with me;
The Binder Team for accepting me into the community and giving me the space to develop such projects;
and The Turing Way team who introduced me to Binder and the value of community (and documentation!).

—

Note: this is cross-posted with the Turing Institute blog.

University of Edinburgh Jupyter Community nbgrader Hackathon

2019-05-22T16:59:00+00:00

The University of Edinburgh is happy to announce our upcoming event as part of the Jupyter Community Workshop series funded by Bloomberg. The University will be hosting a three-day event, the core aspect of this event being a hackathon focused on adding improvements, fixes and extra documentation for the nbgrader extension. Alongside this we will also hold an afternoon of talks highlighting how Jupyter can be used in education at varying levels. The event will take place on 29 to 31 May at the University of Edinburgh, with the afternoon of talks taking place on 30 May.

The first and main part of the event will be the nb grader hackathon. nbgrader is a Jupyter extension that allows for the creation and marking of notebook-based assignments. Here at the University, we have adopted nbgrader and our developers have integrated the extension into our Jupyter service Noteable. The hackathon will focus on improving the core features and extending the abilities of nbgrader; by adding such features, it will be easier for institutions to adopt and embed both Jupyter and nbgrader into their teaching practice.

The second, equally important part of our event is a series of talks aimed at highlighting the uses of Jupyter within education. As part of developing our local Jupyter service, we have uncovered many use cases across the University of how Jupyter can be adopted in a variety of disciplines and scenarios that we are keen to share. We will also be able to showcase an institutional approach to adopting and supporting Jupyter at scale. On top of this, there is also the opportunity to hear from many of our hackathon attendees. This series of talks is aimed at academic colleagues, and teaching and support staff at any level of education and includes an evening networking event to allow attendees to further explore how they may introduce Jupyter to their institution.

What are we working on?

We have worked with our local developers and scoured the nbgrader Github repo to devise a plan of attack for the hackathon in terms of features and improvement. We’re keen to engage with the wider community regarding these goals and have created a post on the Jupyter Discourse to allow further discussion.

Support for Multiple course/Multiple classes. Instructors that teach multiple course using nbgrader or students enrolled on multiple course. Multiple Courses (ref: PR #1040) vs Support for multiple classes via Jupyterhub groups (PR #893)
Considerations for LTI use: Users/Courses not in the database at startup
Support for Multiple markers for one assignment (part of issue #1030, which extends #998)
API Tests have hard-coded file copy methods to pre-load the system to enable testing, and os.file_exists-type tests for release & submit tests
Generation of feedback copies for students within the formgrader UI (to mirror the existing terminal command). Also consider ability to disseminate this feedback back to students within nbgrader.

Want to get involved?

If you would like to be involved in the hackathon, we still have an amount of funding left for travel and accommodation. We are looking for participants who have a good understanding of Jupyter and nbgrader who would be able to attend all three days of the event. If you would be able to attend, please complete the following form: [https://edin.ac/2vP8bFS].

If you can’t attend but want to have a say on what is worked on during the hackathon then join the discussion on the Jupyter Discourse (if you’re new to Jupyter this is an excellent place to head to for community discussions)

If you’d like to come along to our Jupyter community afternoon on 30 May, book a place via Eventbrite.

Jupyter Hackathon Series in Hawaii

2018-12-06T20:17:00+00:00

Hacking on Jupyter Labs and more!

PyHawaii hosted a three-part Jupyter Hackathon as part of a workshop grant, generously offered by Bloomberg. PyHawaii, a user group in Hawaii, led the three-part event series composed of a:

Git/GitHub hands-on workshop (4-hrs — September)
Jupyter Lab, Notebook and more hands-on tutorial (4-hrs — October)
Jupyter Hackathon/open source sprint (2-days — November)

In this post, Project Jupyter had announced the availability of Bloomberg funding to host small-scale community events. PyHawaii wrote up a grant application for three events and was approved. The original grant application was for an event with ~25 people. Since the venue was donated by the University of Hawaii, PyHawaii basically needed funds to cover food and incidentals. PyHawaii also included in the application a request for funding to have a Jupyter Core Developer attend the two-day hackathon in November. Ultimately, Project Jupyter was able to send two Core Developers: Dr. Brian Granger and Dr. Ian Rose to the third event.

Based on the strong responses to the initial Meetup postings, PyHawaii sought additional funding via their relationships with Booz Allen Hamilton to provide food for more attendees. PyHawaii advertised via Twitter, Facebook, meetup.com and relied upon their relationships with professors, college students, technical professionals and members of the community to get the word out. PyHawaii was on Bytemarks Cafe, their local tech broadcast on Hawaii Public Radio.

How it all played out…

Each of the events, described below, played in slightly different ways.

Git/GitHub Hands-on Workshop

PyHawaii had a number of mentors and organizers (Jeff, Joe, Jim, John and Chalmer) who walked 35 attendees through PyHawaii’s “How to contribute to open source” workshop. This workshop is used to prep folks for open source sprints and has been run at conferences around the world (twice at Pycon (2018, 2017), PyOhio (2016), DjangoCon (2018) and more).

The material is designed to maximize hands-on use of git and GitHub and enable attendees to practice the most frequently used commands and features by contributing to a repo full of poems (this allows them to focus on just the mechanics of git/GitHub and not have to simultaneously digest all the nuances of an unknown code base.)

Feedback from the first event

Jupyter Hands-on Tutorial

For PyHawaii’s Jupyter Tutorial, we used a 4 hour lesson created and led by Chalmer Lowe, with support from PyHawaii mentors and organizers to introduce 39 attendees to Jupyter Lab, Notebooks, widgets and more. Like their git/GitHub tutorial, this PyHawaii lesson is also open-source and freely available.

Feedback on the second event and kudos to the folks who make Jupyter great!

Jupyter Hackathon

Over the course of a weekend 25 attendees came together to work on adding features, fixing bugs and improving documentation in Jupyter Lab, Ipywidgets and other repositories. Jupyter core developers (as well as mentors from PyHawaii) spent both days working with attendees to:

understand the codebase for the various projects
troubleshoot the process of setting up their dev environments
identify suitable issues to work on
craft suitable pull requests

In addition to the hackathon, a meet-and-greet event was hosted for core developers, professors and students at the University of Hawaii.

Outcomes and Lessons Learned

These events achieved success on multiple levels. Dozens of attendees had a chance to experience the fun (and frustration), the excitement (and exasperation) of contributing to open source, and they got to do it in a friendly environment with mentors on-hand to help guide them past the initial hurdles. These new skills and experiences have the potential to shape career opportunities and strengthen bonds in the local developer community.

The attendees had multiple opportunities to network and grow their relationships. Hawaii has seen tremendous growth in their development community over the past few years and events like this help to speed that growth. The attendees had the opportunity to network on an international scale through discussions about possible fixes to issues with core developers in person and via the web. Based on feedback, a number of attendees initially had no idea of the breadth of capabilities in Jupyter. As attendees apply this experience to their work, adoption of Jupyter as a data analysis, data science and interactive, exploratory platform will grow.

So far, PyHawaii has at least 13 pull requests, 10 of which have been accepted!

PyHawaii’s experiences yielded the following lessons learned, should other organizations seek to facilitate similar events:

a 1:12–15 person ratio of experienced project developer to attendee allowed folks to get quick, hands-on mentoring without over-burdening the developers. Having other mentors (even if they aren’t experienced in the nuances of the project) is a must to support the project developer in staying focused on answering project-specific questions vs more administrative/low-level questions about git, etc)
host a short event (i.e. the evening OR previous weekend) before the hackathon to help walk folks through existing issues and through setting up their dev environment (build tools, documentation build tools, source code, compiler, etc). This will enable attendees to hit the ground running when the hackathon actually starts
RSVPs are tricky when you host an event at no cost to the attendee. Depending on the event, we had between 28% and 53% no-shows which greatly impacts food costs, which often has to be purchased beforehand. Regular tracking of attendance at other events can help identify no-show trends and facilitate more effective budget estimates.

All in all, this was a great experience that promises to have a lasting impact on the data science and programming communities in Hawaii.