Jupyter Blog - Rollin Thomas

Trusted CI Cybersecurity Engagement with Jupyter

2021-09-15T18:09:00+00:00

Trusted CI is the US National Science Foundation Cybersecurity Center of Excellence, staffed by cybersecurity experts who have spent decades working with science and engineering communities and who have established track records in terms of usable, high-quality solutions suited to the needs of those communities. The team draws from best operational practices and includes leaders in the research and development of new methodologies and high-quality implementations. In addition to providing leadership, education, outreach, and training to raise the state of security practice across the sciences, Trusted CI undertakes one-on-one engagements with projects to address their cybersecurity challenges.

As a part of timely, broader efforts to make Project Jupyter more responsive and proactive to security, the new security sub-project this past summer has undertaken a one-on-one engagement with Trusted CI to run through the end of 2021. This Trusted CI engagement was originally motivated by an upcoming Jupyter Security Best Practices Workshop funded by NumFOCUS as part of the Community Workshop series. The workshop is tentatively scheduled to be held April 2022 at the Ohio Supercomputer Center.

The goals of the engagement between Project Jupyter and Trusted CI include the following tasks:

Review existing Jupyter deployment documentation related to security, identify gaps, and create recommendations for improvements.
Identify Jupyter deployment use-cases as targets for Jupyter Security Best Practices documentation. Example use-cases include supercomputing centers, campus research clusters, workshops, small scientific projects, etc. Prioritize these use-cases based on which audiences would benefit most from new security documentation.
Write Jupyter Security Best Practices documentation for high priority use-cases identified above. Work through other use-cases as time permits.

The Jupyter Security Best Practices documentation produced by this engagement will be shared with Project Jupyter for inclusion in their documentation, and also presented at the workshop.

Here’s how you can get involved and learn more:

The security working group meets every other Friday. You can follow and open issues the repository for public questions and meeting minutes. At this weeks meeting, our Trusted CI colleagues presented an evolving census of all Jupyter-related security documentation: Watch for an upcoming Discourse post about it!
As always, you can send your questions and concerns about security to security@ipython.org to reach out to our security team.

We always welcome feedback, questions, and help regarding security in Jupyter. Here is a link to the original Trusted CI blog post that this blog post was based upon (with permission):

https://blog.trustedci.org/2021/08/engagement-with-jupyter.html

Jupyter for Science User Facilities and High Performance Computing

2019-07-09T23:53:00+00:00

Jupyter is the “Google Docs” of data science. It provides that same kind of easy-to-use ecosystem, but for interactive data exploration, modeling, and analysis. Just as people have come to expect to be able to use Google Docs everywhere, scientists assume that Jupyter is there for them whenever and wherever they open their laptops.

But what if the data you want to interact with through Jupyter doesn’t fit on your laptop or is excruciating to move? What if the model you want to build and test requires more computing power and storage than you have right in front of you? As a scientist, you want the same interactive experience and all the benefits of Jupyter, but you also need to “reach out” to put something big into your science process: A supercomputer, a telescope data archive, a beam-line at a synchrotron. Can Jupyter help you do that big science? What efforts are in motion already to make this a reality, what work still needs to be done, and who needs to do it?

Doing this right will take a community: New collaborations between core Jupyter developers, engineers from high-performance computing (HPC) centers, staff from large-scale experimental and observational data (EOD) facilities, users and other stakeholders. Many facilities have figured out how to deploy, manage, and customize Jupyter, but have done it while focused on their unique requirements and capabilities. Still others are just taking their first steps and want to avoid reinventing the wheel. With some initial critical mass, we can start contributing what we’ve learned separately into a shared body of knowledge, patterns, tools, and best practices.

40+ participants from universities, national labs, industry, and science user facilities. Credit: Fernando Perez.

In June, a Jupyter Community Workshop held at the National Energy Research Scientific Computing Center (NERSC) and the Berkeley Institute for Data Science (BIDS) brought about 40 members of this community together to start distilling. Over three days in talks and breakout sessions, we addressed pain points and best practices in Jupyter deployment, infrastructure, and user support; securing Jupyter in multi-tenant environments; sharing notebooks; HPC/EOD-focused Jupyter extensions; and strategies for communication with stakeholders.

Here are just a few highlights from the meeting:

Michael Milligan from the Minnesota Supercomputing Center perfectly set the tone for the workshop with his keynote, “Jupyter is a One-Stop Shop for Interactive HPC Services.” Michael is the creator of BatchSpawner and WrapSpawner, JupyterHub Spawners that let HPC users run notebooks on compute nodes supporting a variety of batch queue systems. Contributors to both packages met in an afternoon-long breakout to build consensus around some technical issues, start managing development and support in a collaborative way, and gel as a team.
Securing Jupyter is a huge topic. Thomas Mendoza from Lawrence Livermore National Laboratory talked about his work to enable end-to-end SSL in JupyterHub and best practices for securing Jupyter. Outcomes from two breakouts on security include a plan to more prominently document security best practices, and a future meeting (perhaps another Jupyter Community Workshop?) focused specifically on security in Jupyter.
Speakers from Lawrence Livermore and Oak Ridge National Laboratories, the European Space Agency showed off a variety of beautiful JupyterLab extensions, integrations, and plug-ins for climate science, complex physical simulations, astronomical images and catalogs, and atmospheric monitoring. People at a variety of facilities are finding ways to adapt Jupyter to meet the specific needs of their scientists.

Really, there’s just too much to pack into a blog post so we encourage you to look at the talk slides and notes on Discourse — all the breakout notes have been posted there to this topic. We’re working on getting videos of the slide presentations up on the workshop website as well. Watch for announcements of future meeting opportunities and documentation on Discourse as well.

Finally we want to thank Project Jupyter, NumFOCUS, and Bloomberg for their help making this meeting happen. We all came away with a better sense of who is doing what in our community, and how we can work together on this new area of growth for the Jupyter community. The organizers also want to thank their respective institutions’ administrative staff (Seleste Rodriguez at NERSC, and Stacy Dorton at BIDS) for helping with workshop logistics.

Jupyter Community Workshop: Jupyter for Scientific User Facilities and High-Performance Computing

2019-01-29T19:35:00+00:00

We are excited to share more news about the Jupyter Community Workshop for Scientific User Facilities and High-Performance Computing! This is part of a series of Jupyter Community Workshops funded by Bloomberg to “bring together small groups of Jupyter community members and core contributors for high-impact strategic work and community engagement on focused topics.”

This workshop will be held in Berkeley, California, from Tuesday June 11 to Thursday June 13, 2019. The workshop is being jointly hosted at the National Energy Research Scientific Computing Center (NERSC, part of Lawrence Berkeley National Laboratory) and the Berkeley Institute for Data Science (BIDS, at the University of California). The workshop executive committee consists of Rollin Thomas (NERSC), Dan Allan (Brookhaven National Laboratory), and Chris Holdgraf (BIDS — UC Berkeley).

We, the organizers, invite your expression of interest in the effort through this Google form. Let us know whether you’d be interested potentially in attending, just want to be kept in the loop, or just want to express your support. Finding out who is doing what with Jupyter in this space is the first step in building our community.

Why? Advances in technology at experimental and observational science facilities (EOS facilities: telescopes, particle accelerators, light sources, genome sequencers and so on), in robust high-bandwidth global networks, and in high-performance computing (HPC) have resulted in an exponential growth of data for scientists to collect, manage, and understand. Interpreting these data streams requires computational and storage resources greatly exceeding those available on laptops, workstations, or university department clusters. Funding agencies increasingly look to HPC centers to address the growing and changing data needs of their scientists. These institutions are uniquely equipped to provide the resources needed for extreme scale science. At the same time, scientists seek new ways to seamlessly and transparently integrate HPC into their EOS workflows. That’s where we see Jupyter fitting in.

We know that scientists love Jupyter because it combines visualization, data analytics, text, and code into a document they can share, modify, and even publish. What about using Jupyter to control experiments in real-time, or steer complex simulations on a supercomputer, or even connect experiments to HPC for real-time feedback and decision making? How can users reach outside the notebook to corral external data and computational resources in a seamless, Jupyter-friendly manner?

These were the questions on our minds when we proposed a three-day workshop for Jupyter developers, HPC engineers, and staff from EOS facilities. We are looking to foster a new collaborative community that can make Jupyter the pre-eminent interface for managing EOS workflows and data analytics at HPC centers. EOS scientists need Jupyter to work well at their facilities and HPC centers, and this workshop will help us address the technical, sociological, and policy challenges involved.

The workshop itself will include presentations, posters, and a couple half-day hack-a-thon/breakout sessions for collaboration. We will identify best practices, share lessons learned, clarify gaps and challenges in supporting deployments, and work on new tools to make Jupyter easier to use for big science.

During the workshop, participants will be invited to collaborate on a survey white paper that documents the current state of the art in Jupyter deployments at various facilities and HPC centers. The document will include deployment descriptions, maintenance and user support strategies, security discussions, use cases, and lessons learned. A forward-looking summary provided at the end of the white paper will tie together common threads across various facilities and highlight areas for future research, development, and implementation. We will aim to have the paper completed and published to arXiv within three months of the end of the workshop. These ideas in a single document should help developers, maintainers, and researchers make the case they need to management and policymakers to drive the effort forward.

So let us know if you’re interested in the effort by filling out the Google form, even if you think you can’t make it. Part of what we’re doing is finding out who is doing what with Jupyter where in HPC and EOS. That’s the real first step in building our community!