Jupyter Blog - Jupyter Server

Introducing Jupyter Scheduler

2022-12-14T15:26:00+00:00

The Open Source Jupyter team at AWS is proud to announce the release of Jupyter Scheduler, a JupyterLab extension that can run your Jupyter notebooks in the background. Jupyter Scheduler was developed from the start as an open-source project under the umbrella of the Jupyter project and governance. You can set up your notebooks to run once or on a schedule. By default, Jupyter Scheduler runs notebooks on your local system where Jupyter Server is installed; you can also extend the scheduler to run your notebooks in the cloud, on an on-premise cluster, on JupyterHub, or even on supercomputers.

We hope you’ll find Jupyter Scheduler useful for running notebooks periodically, such as daily or weekly reports. You can install it in JupyterLab by running pip install jupyter_scheduler, then restarting JupyterLab. You can create jobs (single runs of an individual notebook) and job definitions (scheduled recurring notebook jobs) in two ways. You can right-click on a notebook in the file browser and choose “Create Notebook Job” from the context menu, or you can use the new toolbar button in an open notebook file.

When you right-click on a notebook file in the file browser, a “Create Notebook Job” option appears in the context menu.

In a notebook, the “Create Notebook Job” option appears on the right of the toolbar.

You can give your notebook job or job definition a name, choose an environment to run it in, select different output formats, and provide parameters that are set as local variables when your notebook gets executed. This parameterized execution is similar to Papermill; Jupyter Scheduler provides a UI for managing and accessing the outputs of your notebook jobs inside JupyterLab.

You can use the Create Job form to run a notebook once or to create a job definition that runs on a schedule.

In addition, if you select “Run on a schedule,” you can run your job on a recurring schedule. The scheduler provides a few useful shortcuts to, for example, run your notebook at a specified interval by the minute, hour, day or week. You can also specify a custom schedule in crontab format.

You can schedule a job definition using preset intervals, such as every weekday.

You can also select “custom schedule” and specify a cron expression for the notebook job definition’s schedule.

Once you’ve created a job or job schedule, you can use the “Notebook Jobs” icon on the launcher to view lists of jobs and job definitions. Each item’s name links to a detail view.

With the Jupyter Scheduler extension installed, JupyterLab’s launcher screen shows a “Notebook Jobs” button in the “Other” section. Click on it to view a list of jobs or job definitions.

The list view shows a list of notebook jobs and a list of notebook job definitions. Each one includes links to view details, to download files, and buttons to take quick actions on them.

The Job Detail view shows all available details for one job or one job definition.

When one of your jobs has completed, you can download its files from the list or detail view. When your notebook runs, the scheduler saves a copy of the input file, so that even if you later change the file, it captures the state of the notebook at the time your job ran. Jupyter Scheduler uses unique names for the input and output files so that rerunning the same notebook produces new files every time.

Jupyter Scheduler uses a REST API to create, update, delete and manage jobs and job definitions. You can also customize Jupyter Scheduler by plugging in your own scheduler class, for example, to replace the existing SQLite database or to run jobs in the cloud. AWS SageMaker Studio actually uses Jupyter Scheduler, with custom classes, as part of its automated notebooks feature, announced earlier this year. See the Jupyter Scheduler operator documentation for more information about how to write and use your own scheduler.

We’d love to hear what you think about Jupyter Scheduler! You can check out Jupyter Scheduler’s documentation on ReadTheDocs. Please open issues with ideas for enhancements and about any bugs you find. If you’d like to contribute code, see the Jupyter Scheduler contributors guide.

This article does not represent the official position of amazon.com or any of its subsidiaries.

About the authors

Andrii Ieroshenko is a Software Development Engineer at AWS. Previously, Andrii worked on the Notifications project as a part of the Jupyter Cal Poly student internship program.

Piyush Jain is a Software Development Engineer at AWS working on open source contributions to Project Jupyter. He is also a member of the Jupyter Server team, actively participating in maintenance and development of Jupyter server project.

David Qiu is a Software Development Engineer at AWS working on Project Jupyter. Previously, David was researching physical chemistry at the University of Illinois at Urbana-Champaign.

Jason Weill is a Senior Front-End Engineer at Amazon Web Services; a contributor to JupyterLab, Jupyter Governance, and Jupyter Scheduler; and a member of the JupyterLab Council, the Jupyter Security Working Group, and the Jupyter Diversity, Equity, and Inclusion Standing Committee.

Jupyter Server 2.0 is released!

2022-12-06T23:25:00+00:00

We are thrilled to announce the release of Jupyter Server 2.0!

Jupyter Server provides the core web server that powers JupyterLab and Jupyter Notebook.

This release brings many new features and lays groundwork for a more multi-user, (real-time) collaborative Jupyter experience.

Major Highlights

Below is a short list of highlights for everyone! 🚀

New Identity API

As Jupyter continues to innovate its real-time collaboration experience, identity is an important component. While we all love that mysterious “Anonymous Jovyan” appearing in our notebooks, we’d prefer to know who is editing/viewing our documents.

The new Identity API in Jupyter Server makes this possible. It is a configurable, pluggable API that provides a clear extension point to customize how Jupyter Server authenticates and identifies its users.

Deployers of Jupyter Server can configure Server’s default IdentityProvider or swap in their own IdentityProvider to authenticate users with any auth provider.

New Authorization API

Enabling collaboration on a notebook shouldn’t mean “allow everyone with access to my Jupyter Server to edit my notebooks”. What if I want to share my notebook with e.g. a subset of my teammates?

The new Jupyter Server Authorization API is a critical step towards making this type of “fine-grained” permission management possible.

It provides a clear, pluggable extension point where we can build authorization layers (e.g. RBAC) backed by any auth provider. Deployers of Jupyter Server can extend Jupyter Server’s new Authorizer API to control user access in a running Jupyter Server.

New Event System API

Looking ahead at Jupyter’s future, it became clear that there is a whole class of untapped features — like notebook commenting or notifications— that need a Jupyter “event system"; these features react to events happening in and around a running Server.

This led us to create jupyter_events—a package that provides a JSON-schema-based event-driven system to Jupyter Server and server extensions. Jupyter Server leverages this new system to emit events from the server. Keep an eye out for future development where we will define more “core” events (similar to the Contents Service Events) to enable better event-driven functionality in the Server.

Other major changes

Below is list of other major changes and features that impact folks who extend + customize the Jupyter Server experience. This section is a little more technical in nature, but offers some exciting updates for folks who customize Jupyter Server.

New Kernel Websocket API

Jupyter Server now provides a configurable, pluggable layer between the kernel’s ZMQ sockets and websocket, called the KernelWebsocketConnection. This enables folks to innovate in the space between kernels and the server’s clients.

Server ships its own Kernel Manager

Jupyter Server now provides its own kernel manager class, ServerKernelManager. This enables us to innovate the kernel management layer to specifically fit the needs of Jupyter Server and add features like kernel-based events, persistence, and other kernel lifecycle-related functionality

This affects folks who have extended the default KernelManager with custom functionality — they will need to derive from this new class in order to take advantage of new server-specific functionality.

Terminals Service is now a Server Extension

Jupyter Server now ships the “Terminals Service” as an extension (installed and enabled by default) rather than a core Jupyter Service.

Most users will not notice this change, but this enables terminals to be optionally disabled. For more sensitive deployment environments, this is a big win!

pytest-jupyter, a pytest plugin for Jupyter

As we worked to increase our test coverage to ~80% (major “thank you” to Steve Silvester!), we built a handy set of pytest fixtures that can benefit Jupyter developers and extension authors.

This led us to release pytest-jupyter, a packaged pytest plugin for testing Jupyter core libraries and extensions.

Changelog

This post was only an overview of what’s new in 2.0! If you are looking for a more complete list of the changes including bug fixes, be sure to have a look at the changelog.

Installation

To install Jupyter Server with pip:

pip install jupyter_server

or with conda:

conda install -c conda-forge jupyter_server=2.0.0

Documentation

Find our full documentation here.

We are still catching up! Please have patience with us as we continue to update our documentation. If this is something you’d be interested in helping us improve, we’d love your help!

Acknowledgements

Massive “thank you” to the whole Jupyter Server Team. The Jupyter Server project is largely shepherded by the amazing folks listed here.

We’re also incredibly grateful to the large list of contributors outside the team. Y’all are the best! Thank you for helping us make Jupyter Server better each day!

If you’re interested in meeting or getting involved with the team, stop by one of our weekly community meetings, or better yet, join our Contributing Hour!

About the author

Zach Sailer is a Senior Software Engineer at Apple, where he is the Jupyter Open Source Champion. He is a Jupyter Distinguished Contributor and the Jupyter Server Steering Council Representative.

Jupyter Community Workshop: Jupyter Server Design and Roadmap Workshop

2019-04-30T23:58:00+00:00

We have some exciting news to share regarding the Jupyter Server Design and Roadmap Workshop as part of the Jupyter Community Workshop series funded by Bloomberg. This workshop will take place May 16th and 17th just outside Paris France at the IBM facility in Bois-Colombes.

Jupyter Notebook has proven to be a tremendous tool in the scientific and analytic computing spaces. It is widely used at universities and businesses alike, enabling the ability to interactively analyze and view data in various ways, quickly and easily. However, as computational capabilities improve, the need to move Notebook kernels closer to the compute resources also increases. As a result, new requirements for how a given solution is configured and deployed are introduced.

The Jupyter Server Design and Roadmap Workshop will focus on how we can bring together what has been learned over the years to address the needs of future environments, while clearly defining the separation between client and server. Items that will be discussed include:

What aspects of the current Jupyter Notebook framework should be considered as “the server”?
How will extensions be exposed and consumed?
Backwards compatibility is important. How can we move forward while retaining current capabilities?
How can we introduce the ability for others to provide kernel-deployment frameworks of their own and how those frameworks are discovered?
Basic improvements that bring the server up to date (e.g., async/await — particularly in kernel life-cycle management)
General multi-tenancy capabilities will be explored such that the server can serve more than just a single client.
How to convey kernel-specific parameters from the client, thru the server, to the kernel launch framework?

Should you be interested in joining us for this workshop, please fill out this Google Form. Space is limited.

Acknowledgements

We’d like to acknowledge Bloomberg for their generous support in making this workshop, and the entire series, possible. Thank you!

We’d also like to thank IBM for providing the facility and hosting the Jupyter Server Design and Roadmap Workshop.