Scaling “Maintainer Intuition” with Pull Request Triage Boards

· yuvipanda · community JupyterHub

When I helped start 2i2c.org, one of my goals for the non-profit was to experiment with new and different ways of supporting the Jupyter ecosystem’s long term health. As part of that, we have identified how foundational contributions are very important (and distinct from directed contributions), and have been experimenting with new ways to make foundational contributions in sustainable and structured ways.

One such way was via systematically doing code reviews for Pull Requests from non-maintainers in the JupyterHub ecosystem. Reviewing PRs is a critical way that maintainers keep an open source project moving forward, but identifying PRs that can productively be merged is hard. This is a post describing our system for scaling this in our team. Each 2 week sprint, we:

  1. Looked at all open PRs in the JupyterHub org who were not maintainers
  2. Picked a PR that I deemed was reviewable and ideally mergeable within this time window
  3. Have an engineer on the team pick up that PR, and try to get it to close
  4. Report back on what we have learnt, so we can iterate on our process

We managed to do this for a majority of sprints over the last roughly 12 months! One key bottleneck we identified in the process was Step 2. In particular, I was relying on my maintainer intuition to pick a single PR that I believe can be merged, so others in the team can do review work. I started exploring what this intuition is, and if it can be scaled.

What is this maintainer intuition?

How did I pick a PR from a long list of open PRs? Observing my own behavior a few times, I noticed I was looking for:

  1. PRs that aren’t too big, and are a reasonable size that can be merged within a 2 week window
  2. CI tests passing, so at least our automated checks haven’t caught any issues with it
  3. Features or bug fixes that I believe add value to the project and move us in the right direction towards being able to support our users as they need (this is the hardest!)
  4. If the author of the PR is a newish contributor, as I want to encourage them to stick around by being responsive to their gift. All PRs are gifts that we may or may not choose to accept, but should do so with grace.
  5. How long ago the PR was opened. There is such a big difference between a response to your PR 2 days after you make it vs 2 months vs 2 years. I prioritized newer PRs.
  6. What kind of contribution is it primarily? Different engineers on our team have different skillsets (JS, Python, etc) and I wanted to match the PR to what the engineer preferred code reviewing.

While (3) is hard to scale, everything else seemed like something we could build systems for that let others follow a process, thus removing myself as a bottleneck. I experimented with some GitHub issue filters and project automations, and after finding them lacking, built out a brand new open source project to do this: pr-triage-board-bot!

PR Triage GitHub Boards

pr-triage-board-bot automatically maintains PR Triage Boards for a Github organization. You can check out the current boards to get a sense of how it looks: JupyterHub, JupyterLab, and GeoJupyter.

For each PR Triage Board, the bot will:

  1. Add all open, ready for review PRs
  2. Remove all closed or draft PRs
  3. Annotate each PR with additional deterministic project fields that allow for sorting and filtering in the project board. To start with, it populates the following fields:
    1. Author kind (Maintainer, Seasoned Contributor, First Time Contributor, Bot)
    2. Size (Number of lines touched)
    3. Date it was opened
    4. If a maintainer has already interacted with the PR (One, Many, None)
    5. Are there merge conflicts?
    6. What kind of files were mostly changed? (Python, JS, Docs)
  4. Keep this up to date by automatically running every hour via GitHub actions

Once these project fields are populated automatically, maintainers can create different Project Views for themselves to help with different workflows via filters and sorting. For example, the JupyterHub board contains the following views:

  1. PRs split by author kind, sorted by newness and size so we can try to respond to PRs from new users as early as possible
  2. PRs that have not had a single maintainer interaction on them, so we can acknowledge people’s contributions to us even if we can’t fully review it at the moment
  3. Bot PRs with all tests passing and no merge conflicts, so we can more easily stay on top of automated updates
  4. PRs that have been approved by a maintainer but not merged yet, as sometimes a maintainer wants to give others time to object but needs to go back and hit merge.
  5. PRs that are primarily updating documentation, as the review process for this can be different

For our original purpose of getting more people to do code review, this board has served well — we can roughly say ‘Pick a PR that looks good to you from the top of the “First Time Contributor” or “Seasoned Contributor” list’, and that relieves me from being the bottleneck quite a bit. Other maintainers are also finding a lot of value in this board. For example, if you only have 15 min, you can probably get a clean bot PR reviewed and merged. Or look for PRs that haven’t been acknowledged by a maintainer and engage with them. Our next step is to help run structured social experiments, where we try to establish specific ceremonies in the open source ecosystem to get specific lists of open PRs (such as “PRs with no maintainer engagement, or PRs older than 2y”) down to zero. Stay tuned to hear more :)

We are essentially using GitHub Project Fields as a database, adding additional fields that are very valuable to maintainers but not available in GitHub by default. While ideally these would be contributed by us directly into GitHub, only Microsoft has control over what gets added to the product. So we find interesting workarounds like this to accomplish our goals :)

Successful Adoption

I was just playing with this, and Raniere from the JupyterHub team spotted it and asked about it on Zulip. Jason Grout from the JupyterLab team was also super interested, and with contributions from him, the bot quickly got adopted by the JupyterLab GitHub org as well. We cleaned this up a bit more, and found adoption within the GeoJupyter project too with help from Matt Fisher. As with everything we do at 2i2c, I had tried to design this to be widely useful to many orgs and maintainers rather than just us, and looks like I have wildly succeeded :)

I am happy to announce today that 2i2c is officially donating pr-triage-board-bot to Project Jupyter! I will still continue to contribute to maintaining the project, and welcome contributions from everyone else too!

If this looks useful to your open source project, consider adopting it by following these instructions. The project is still fairly new, so if you run into issues please let us know on the Project Jupyter Zulip chat or by opening an issue.

More posts