Jupyter Blog - machine learning

CUDA enabled Jupyter Docker Images

2024-04-16T22:08:00+00:00

I am happy to announce that Jupyter Docker Stacks project now provides GPU accelerated Docker images. This allows PyTorch or TensorFlow operations to use compatible NVIDIA GPUs for accelerated computation.

How to find these images

All our images are available on Quay.io registry. We provide CUDA accelerated versions of images are available for tensorflow-notebook and pytorch-notebook.

To use such an image, you have to specify a special prefix tag to the image: versioned CUDA prefix like cuda11- or cuda12- for pytorch-notebook or just cuda- for tensorflow-notebook.

We build pytorch-notebook only for 2 last major versions of CUDA, tensorflow-notebook image supports only the latest CUDA version listed in the officially tested build configurations list. CUDA-enabled images are available on x86_64 platform.

For example, you can use an image quay.io/jupyter/pytorch-notebook:cuda12-python-3.11.8 or quay.io/jupyter/tensorflow-notebook:cuda-latest.

Running the images

To enable CUDA support on Linux host machine, you need to:

have compatible Nvidia GPU
have NVIDIA Linux driver installed
add --gpus all (or --gpus '"device=all"') flag to if you’re using Docker
add --device 'nvidia.com/gpu=all' flag if you’re using Podman

For example, you can run the image like this: docker run -it --rm -p 8888:8888 --gpus all quay.io/jupyter/tensorflow-notebook:cuda-latest.

You can also enable GPU support on Windows using Docker or Podman.

Reaching out

You will find our documentation here.
If you encounter any problems, please, take a look at GitHub issues.
If you would like to improve GPU support in our images, please, reach out here.

Acknowledgments

I would like to thank:

Johanna Reiml, for implementing general variant concept for our images and making it work for pytorch-notebook (PR #2091)
Christof Kaufmann, for making it work for tensorflow-notebook (PR #2100)
Olivier Benz, for helping me out to write this blog post

Ploomber: Maintainable and Collaborative Pipelines in Jupyter

2021-09-01T14:17:00+00:00

Jupyter is a fantastic tool for data exploration. The ability to transform our data interactively and get immediate visual feedback allows us to understand it quickly. However, when working on large projects, collaboration can become difficult. Features such as live collaboration are a gigantic leap forward for teamwork. Still, it has to complement an asynchronous workflow that allows team members to work at different times, especially in a remote-first workplace.

Back in 2020, I introduced Ploomber at JupyterCon to help practitioners build maintainable and reproducible data workflows. Fortunately, the community is growing. We’ve received great feedback from teams that use Ploomber to develop production-ready pipelines using Jupyter, debunking the notion that notebooks are only for prototyping.

As we’ve gathered more feedback from our community, we realized that while users had improved the reproducibility of their workflows, team dynamics didn’t change much. In many cases, collaborators worked in isolation, sharing processed data, which severely hindered reproducibility.

Teams often share processed data, which hinders reproducibility.

Since its first release, Ploomber aimed to promote software development best practices to produce more maintainable data projects. We’re now doubling our efforts to enable a collaborative and asynchronous workflow.

Enabling Code Reviews

We frequently hear from teams using Jupyter notebooks that it’s challenging to manage .ipynb files. For example, we heard from a data scientist that his company considered banning Jupyter notebooks because they couldn’t figure out how to follow software engineering best practices. Our fellow data scientist was highly frustrated by this situation since Jupyter turbocharges his ability to explore and understand data.

Managing .ipynb files is challenging for multiple reasons. Suppose I push a notebook to a git repository. Then, I add a comment and push it to the repository again. The difference between the two versions looks like this:

Diff view from a notebook with a new cell.

.ipynb files contain input and outputs in a single file, making them extremely useful when we want to share code and results with a colleague but complicates code reviews where we want to compare the previous version with the current one. However, we can fix this problem by changing the underlying file format.

Many practitioners don’t know that Jupyter is agnostic to the underlying file representation, allowing us to interact with different file formats as notebooks. Jupytext is a fantastic project that enables us to open various file formats such as .py and .md as notebooks.

Ploomber integrates with jupytext, allowing users to store their source code as .py files and explore data interactively with Jupyter. Since source code exists in .py files, this enables code reviews, file merging, and the flexibility to edit the code either in Jupyter or in any text editor, giving anyone on the team the freedom to use whatever tool they like the most.

The same .py file is displayed as a notebook in JupyterLab and as a script in VS Code.

Enabling Modularization

Often, teams develop significant parts of a project in a single notebook for convenience; however, this makes it hard to maintain the project in the long run. We know from decades of advancement in software engineering practice that modularizing our code produces more maintainable projects. Yet, we tend to work on single notebooks because breaking down analysis in multiple parts involves managing project structure, ensuring we route outputs correctly, and writing code to orchestrate all steps.

Ploomber allows users to concatenate multiple notebooks into a pipeline in two steps: list the notebooks in a YAML file and declare execution dependencies (e.g., download data, then clean it). Furthermore, Ploomber parses our execution dependencies and injects inputs into our notebook when opening it. Thus, there’s no need to hard-code any paths. The following image illustrates how to declare a pipeline in a pipeline.yaml file and the notebook’s code injection process:

Ploomber integrates with JupyterLab to auto-complete outputs from task dependencies.

Modularization has another benefit. It allows teams to collaborate on separate, well-defined streams of work. Data projects have a sequential nature, and by explicitly structuring them into small tasks, it is easy to assign parts to various team collaborators. For example, if a two-person team is working on a project that uses two sources of data, each member can take one dataset:

Modularization allows teams to work efficiently.

Enabling Testing

Modularization facilitates testing. A recommended practice when developing data pipelines is to test the output data from each task to ensure it has some minimum quality. Since we have clear boundaries among tasks, we can embed integration tests that check the output coming out of each step, ensuring that we know when something breaks.

Ploomber allows testing output artifacts from each task to check for data quality.

Examples of integration tests include: checking there are no NULL values in a specific column or verifying values fall into a certain range. In Ploomber, you can execute an arbitrary function after your notebook finishes to assert statements on your data:

Example of an integration test that ensures that a particular column does not have NAs.

Ensuring Reproducibility

Users can adopt a continuous integration workflow to test for reproducibility.

One of Jupyter notebook’s most recurring problems is hidden state; this happens when we execute code cells in an arbitrary order. However, when running cells in sequential order, the results do not match the recorded output.

Hidden state creates significant issues. For example, teams often execute notebooks locally, store them in a git repository, and don’t execute them again. For instance, I heard from a fellow data scientist that her team struggled to update a model in production because of a broken notebook that prevented her team from re-training the model: the notebook had been executed locally once but never tested for reproducibility.

The answer to hidden state is straightforward: test continuously. In software engineering, it’s common to run code and test it on each git push. However, such practice hasn’t found its way into the data world, primarily because running data processing code may take hours, making testing unfeasible.

Fortunately, most errors are detectable with small amounts of data, and Ploomber simplifies managing pipeline configurations. For example, a user can define a sample parameter to run the pipeline with a fraction of the input data:

An example parametrized pipeline that can execute with a sample of the data.

Then, a continuous integration script can run the pipeline with a sample by switching the sample parameter:

ploomber build --sample true

Furthermore, users can easily download remote artifacts to debug broken pipelines.

Enabling Pull Requests

Users can submit Pull Requests containing code (.py files) and results (executed notebooks).

Once a new feature is ready, a team member can open a pull request. Since the code is in .py files, the reviewer can easily compare code versions. However, metrics or charts are essential to evaluate data processing code. For this reason, Ploomber generates a .ipynb file for each .py file; such executed notebooks can be attached to a pull request to review code and output.

Pipeline execution generates executed notebook files that can be attached to a Pull Request.

A CI process can orchestrate pipeline execution with a single command: ploomber build. However, if working with large datasets, Ploomber can export pipelines to execute in AWS Batch, Airflow, or Kubernetes (Argo Workflows).

Enabling Fast Iterations

Ploomber skips execution of tasks whose source code has not changed since the last run.

Data projects are highly iterative and require us to run small experiments to evaluate our final results. For example, we may add a new feature and assess whether that improves model performance. The more experiments we try, the higher our chance of success.

Such experiments are small and often only touch a small portion of the pipeline. If most of our tasks are unaffected, re-running tasks is a waste of time since they’ll generate the same results. Given that a training pipeline may take hours to run, it is essential to speed things up as much as possible. To enable faster iterations, Ploomber builds pipelines incrementally, skipping tasks whose source code hasn’t changed. Incremental builds help in multiple scenarios, for example, when trying out local experiments or running the pipeline in the CI system. Furthermore, they enable crash recovery: if we execute our pipeline and it crashes, we can fix the failing task, submit it again, and execution will take off from the point of failure.

JupyterLab Is a Production-Ready Platform

It is common for teams to develop prototypes in Jupyter, then refactor them into Python modules; such an approach creates a lot of overhead and a tremendous burden for data scientists (who produce the models) and engineers (who have to refactor notebook-based prototypes). Instead, we believe data projects should start with production in mind by following best software development practices that allow teams to iterate quickly.

Providing such an experience is challenging. Nevertheless, we are working hard to achieve that goal; we want data scientists and engineers to collaborate to produce production-ready projects that instantly go from Jupyter to production.

The Future

There is still a long way to capture our vision of the future of data workflows. Furthermore, we want to keep building the project with the help of our users. So if you’re interested in building Ploomber with us, join our community, or follow us on Twitter! And if you believe in our mission, please show your support with a star on GitHub.

Thanks to Ido Michael and Filip Jankovic for providing feedback.

Reusable code snippets in JupyterLab

2020-11-18T18:08:00+00:00

by Patrick Titzler, Luciano Resende

Reusable code snippets in JupyterLab

In this blog post, we will introduce the Elyra code snippet extension, which enables us to reuse arbitrary snippets of code in your notebooks, source code, or markdown files in JupyterLab.

Let’s say you are working on a notebook and want to include a standard header or footer (like licensing text) or some code that you’ve previously created. This typically results in a hectic search. In which notebook did I use the code? Where’s the most current version of the disclaimer I need to add? While maintaining a “scratchpad” might help, native support for reusable code or documentation assets in JupyterLab holds a lot more appeal.

Code snippet ui

Installing the code snippet extension

The code snippet extension can be installed as part of Elyra or individually using pip or conda. In the context of this article, see below how to install the code snippet extension in your existing JupyterLab deployment.

Installing the extension using pip

$ pip install elyra-code-snippet-extension && jupyter lab build

Installing the extension using Anaconda

$ conda install -c conda-forge elyra-code-snippet-extension && jupyter lab build

Managing and using code snippets

Code snippets are accessed by opening the code snippets tab on the JupyterLab sidebar:

The code snippets UI is divided into two parts:

a quick search panel, and
the code snippet list, providing access to code snippet specific actions, such as copy, insert, edit, and delete.

Creating a snippet

To create a snippet click + above the search bar and provide the required information. You can optionally tag snippets to make them more easily discoverable.

Finding a snippet

To locate a code snippet, enter a search term and/or pick one of the listed tags.

Copying a snippet to the clipboard

You copy the snippet content to the clipboard by clicking the snippet’s copy icon.

Inserting a snippet

Click the snippet’s insert icon to paste the snippet content in the desired destination, such as a cell in a notebook or an open editor window. Drag-and-Drop of the snippet has also been incorporated in the latest Elyra release.

Do note that the editor does not keep track of where you’ve inserted a snippet. Also, note that snippets don’t act like macros and changes to a snippet are therefore not reflected in the locations where the snippet was embedded.

Editing a snippet

You edit a snippet by clicking the snippet’s pencil icon.

Deleting a code snippet

You delete a code snippet by clicking the snippet’s trash icon.

The code snippet extension currently doesn’t provide any import or export capabilities. However, you can access the relevant metadata files that contain the definitions.

The definitions are stored in the $JUPYTER_DATA_DIR/metadata/code-snippets directory, where $JUPYTER_DATA_DIR refers to the Jupyter data directory, which you can identify by running the following command in a terminal window:

$ jupyter --data-dir

In my environment, the code snippets are stored in /Users/ptitzler/Library/Jupyter/metadata/code-snippets/. To back up or share my snippets, I typically create an archive from the content of this directory.

Try Code Snippet from Binder

To experiment with the code snippet extension without installing it locally, just click on the binder link below and then look for the code-snippet icon on the JupyterLab side-bar :

Try the code snippet extension online on Binder

Acknowledgments

We’d like to thank Zach Sailer and Timothy George for all his UX contributions in conjunction with JaeWook (Jay) Ahn, Kiran Pinnipati, and Ai-Vy Dang the CalPoly interns that helped to implement these UX enhancements.

Closing thoughts

Elyra is constantly making Jupyter Notebooks better aiming to help data scientists, machine learning engineers, and AI developers through the model development life cycle complexities, making JupyterLab even better for AI practitioners.

Come and join our weekly dev meetings to learn more about the most recent updates to Elyra and to ask your questions to the community, including the project core developers.

We would also appreciate for you to get involved with the Elyra project. Read our contributing guidelines, create new issues if you have questions, suggestions for new features, or to report any bugs. We also welcome contributions via GitHub pull requests. And last but not least, don’t forget to star the project on Github.

Elyra reaches 1.0.0

2020-08-10T15:55:00+00:00

Building on a Jupyter Notebooks foundation, the de facto tool for data scientists, machine learning engineers and AI developers, Elyra is an open-source project that provides a set of AI-centric extensions to JupyterLab aiming to help users through the model development life cycle complexities, making JupyterLab even better for AI practitioners.

Elyra is proud to announce its 1.0.0 Release. This release brings usability enhancements and bug fixes for existing features, such as enhanced inline user documentation and validation capabilities for the Pipeline Editor, improved performance for pipeline submission to Kubeflow Pipelines runtime. It also provides new capabilities such as a new reusable Code Snippets extension and the ability to configure runtimes directly on the JupyterLab user interface.

On the platform side, Elyra 1.0.0 is now based on JupyterLab 2.x and can easily be integrated into a JupyterHub deployment or other containerized environments such as Kubernetes and OpenShift or even be run in standalone mode using local docker deployment.

With Elyra 1.0.0 we also provide some helpful use-case scenarios leveraging Elyra features to build AI pipelines in order to demonstrate in-depth capabilities exposed by the project. Use cases include:

If this is the first time you are hearing about Elyra, check out our announcement blog for more details about the project.

The 1.0.0 release of Elyra includes:

Notebook Pipelines visual editor
Ability to run notebooks as batch jobs
Reusable Code Snippets (new)
Hybrid runtime support (based on Jupyter Enterprise Gateway)
Python script execution capabilities within the editor
Python script navigation using auto-generated Table of Contents
Notebook navigation using auto-generated Table of Contents
Notebook versioning based on Git integration
Reusable configuration and editor for runtimes (new)
Support for JupyterLab 2.x (new)
JupyterHub Support (new)
Ability to try Elyra from Binder (new)
Support for JupyterLab Dark Theme

Elyra 1.0 extensions to JupyterLab

Notebook Pipelines visual editor

Elyra’s Notebook Pipeline Editor simplifies the conversion of multiple notebooks into batch jobs or workflows. By leveraging cloud-based resources to run their experiments faster, the data scientists, machine learning engineers, and AI developers are then more productive, and therefore able to spend more of their time focusing on their technical skills.

Based on the great feedback from the Elyra user base, this release brings lots of bug fixes usability enhancements such as:

Enhanced inline user documentation
Validation capabilities to Pipeline Editor that notifies users of missing or invalid configuration values
Optimized dependency handling providing a much faster submission of pipelines
Easier access to previously submitted experiments from the Pipeline Editor
Support for “bring your own image” to be used as the environment to execute Notebooks on the external runtime

Below you can see some of the Pipeline Editor UI enhancements introduced in Elyra 1.0:

Elyra Pipeline visual editor UI enhancements

Reusable Code Snippets

Code snippets give you the ability to save time and reuse task-oriented blocks of code. Elyra’s new code snippets extension enables easy discovery, creation, and insertion of reusable snippets of code into your Notebooks, Python Scripts, or even Markdown files used for documentation directly from the JupyterLab workspace. This makes the process of writing code more efficient and accessible.

The list of available code snippets is found in the left side pane and includes a preview of each snippet, and an option to either copy a snippet or insert it directly inline.

Elyra code snippets pannel

Code snippets can also be conveniently created and edited from within JupyterLab.

Elyra code snippets editor

Leveraging Table of Contents for Notebooks and Python Scripts

Navigating large files to find specific sections in Notebooks or function definitions in Python Scripts can be difficult tasks. The Table of Contents extension, which was enhanced to support navigating Python Scripts provides an easy outline of your contents, enabling easy navigation.

Below is an example of an auto-generated table of contents for a Notebook based on its markdown sections:

Auto-generated Notebook Table of Contents

To allow for streamlined python development, Elyra’s python editor is now accompanied by an auto-generated Table of Contents, which allows for efficient navigation within large python scripts.

Auto-generated Python Script outline using Table Of Contents

Reusable configuration and editor for runtimes

Elyra introduced a ‘shared configuration service’ that simplifies workspace configuration management, enabling things like information around accessing external runtimes to be configured once and shared across multiple components.

With Elyra 1.0, this service is now used by multiple components and has been enhanced with schema-based validation capabilities and a full set of REST APIs. With this release of Elyra, users can also easily browse, create, and edit these configurations from within the JupyterLab user interface.

Runtime configuration editor

JupyterHub support

With Elyra 1.0.0 we have also created a docker image and provided necessary configuration steps to integrate Elyra with JupyterHub.

Elyra & JupyterHub deployment diagram

Using Elyra in real Analytics and AI scenarios

While building Elyra, we work very closely with data scientists, machine learning engineers, and AI developers, and we have been building a few scenarios to validate the user experience when developing models and other applications using Elyra.

Analyzing COVID-19 time-series data

One of the examples creates a pipeline to analyze COVID-19 time series data sets from both the USA and Europe which is available as open-source in the covid-notebook GitHub repository.

COVID-19 notebook pipeline

Analyzing NOAA weather time series data set and explore forecasting

Another example utilizes DAX — Data Asset Exchange NOAA dataset and produces a pipeline that consumes and applies ETL into the dataset, and then goes about analyzing and experimenting with different forecasting capabilities.

NOAA weather time-series notebook pipeline

Elyra community adoption

The Elyra community is working very hard to promote adoption and create a healthy community around the project. In the past few months, we are starting to see some momentum, and below are some details :

Github Stars: Elyra main repository is reaching close to 500 as of August 2020, please continue to show your support for the project by spreading the word about the project and giving us more stars.

Projects depending on Elyra: Other then the two examples scenarios mentioned above, we are starting to see other communities adopting Elyra into their projects, CalPoly has been using the Elyra code snippets extension in their summer-intern projects and other communities are also experimenting with Elyra.

GitHub projects depending on Elyra

Downloads: We have also seen an increase of downloads of the Elyra npm packages in the past several weeks, and we are floating between 60k — 70k weekly downloads for the past month:

Elyra packages download stats

Elyra in the Enterprise: Components of Elyra are integrated and available in IBM Cloud Pak for Data and Watson Studio offerings.

Try Elyra from Binder

To experiment with Elyra without installing it locally, just click on the binder link below:

Try Elyra online on Binder

What is next for Elyra

Come and join our weekly dev meetings to learn more about the most recent updates to Elyra and to ask your questions to the community, including the project core developers.

Jupyter Blog - machine learning

CUDA enabled Jupyter Docker Images

How to find these images

Running the images

Reaching out

Acknowledgments

Ploomber: Maintainable and Collaborative Pipelines in Jupyter

Enabling Code Reviews

Enabling Modularization

Enabling Testing

Ensuring Reproducibility

Enabling Pull Requests

Enabling Fast Iterations

JupyterLab Is a Production-Ready Platform

The Future

Reusable code snippets in JupyterLab

Installing the code snippet extension

Installing the extension using pip

Installing the extension using Anaconda

Managing and using code snippets

Creating a snippet

Finding a snippet

Copying a snippet to the clipboard

Inserting a snippet

Editing a snippet

Deleting a code snippet

Sharing code snippets

Try Code Snippet from Binder

Acknowledgments

Closing thoughts

Elyra reaches 1.0.0

Notebook Pipelines visual editor

Reusable Code Snippets

Leveraging Table of Contents for Notebooks and Python Scripts

Reusable configuration and editor for runtimes

JupyterHub support

Using Elyra in real Analytics and AI scenarios

Analyzing COVID-19 time-series data

Analyzing NOAA weather time series data set and explore forecasting

Elyra community adoption

Try Elyra from Binder

What is next for Elyra