Jupyter Blog - Sylvain Corlay

ipydatagrid is now part of Project Jupyter

2024-08-22T15:01:00+00:00

Today, we are proud to announce that the ipydatagrid open source project has been incorporated into Project Jupyter as part of the Jupyter Widgets subproject.

What is ipydatagrid?

ipydatagrid is a fast data grid widget for Jupyter Notebooks and JupyterLab. Since its inception in 2019, it has been developed as an open source project in Bloomberg’s GitHub organization.

It offers a high-performance fully-featured DataGrid interface, fully integrated with ipywidgets. Built upon the Lumino datagrid, which also powers the JupyterLab CSV viewer, ipydatagrid provides users with a robust and versatile tool for data visualization and manipulation.

Key Features

Users can customize the way data is represented in their grid using a variety of renderers.

ipydatagrid includes a sophisticated selection model with two-way data binding.

It enables conditional formatting powered by Vega Expressions.

The transfer to Project Jupyter

We would like to extend our gratitude to the developers and contributors who have significantly improved the ipydatagrid project over the years: Itay Dafna, Martin Renou, Mehmet Bektas, Kaia Young, Bernát Gábor, Vasilis Themelis, Supriya K., Greg Mooney, Ian Thomas, Olly Hensby, and John Gunstone. Special thanks to Chris Colbert, the creator of the Lumino Datagrid, which forms the foundation of ipydatagrid’s front-end.

Bloomberg’s contribution to Project Jupyter

The transfer of ipydatagrid to Project Jupyter is just one of many contributions that Bloomberg has made to Project Jupyter. As a long-standing sponsor, Bloomberg has been instrumental in the development and success of Jupyter. Home to core maintainers and a major funder of JupyterLab since its inception, Bloomberg has also sponsored all editions of JupyterCon. The sustained support by Bloomberg has been crucial to Jupyter’s growth and impact.

We are excited about the future of ipydatagrid within Project Jupyter, and look forward to continued innovation and collaboration within the broader community.

— on behalf of the Jupyter Widgets Council, Sylvain Corlay

JupyterGIS

2024-06-12T17:09:00+00:00

Pioneering Web-based, Collaborative, and Open-source GIS Tools

By Sylvain Corlay [1], Anne Fouilloux [2], and Monika Weissschnur [3]

We are thrilled to announce that the European Space Agency (ESA) is funding our proposal “Real-time collaboration and collaborative editing for GIS workflows with Jupyter and QGIS.”

The consortium spearheading this project comprises QuantStack and Simula Research Lab (Simula), two organizations with a long history of contributions to the Jupyter project and the broader open-source scientific computing ecosystem.

The goal of the project is to build solid foundations for a versatile web-based user interface for Geographic Information Systems (GIS) workflows. It will comprise several components, including a JupyterLab extension for collaboratively editing QGIS project files and the integration of these APIs in the Jupyter Notebook. We will then explore integration with platforms like the Copernicus Data Space Ecosystem (CDSE) and the European Open Science Cloud (EOSC).

Collaborative workflows in geosciences

Collaborative editing of documents has become an integral part of our digital lives and has made us collectively more productive. Gone are the days of cumbersome email exchanges with documents shuttling back and forth.

Looking ahead, the potential of co-editing extends far beyond text documents and will apply to all authoring UIs, from CAD to image processing. We think that the shift to collaborative editing will be even more transformative for larger and more complex projects, which are inherently social and require the concerted effort of large teams. Whether you are designing a stadium, a plane, or an ocean liner, you need to coordinate a diverse expertise to build a unified model. In geosciences, it may range from climate modeling to agriculture, ecology, urban planning, and many more areas of expertise. For such endeavors, we must embrace tools favoring collaboration, and this applies to the future web-based user interfaces for geoscience research.

We have been working on collaborative editing in the core of JupyterLab for the past three years. Our approach is based on the Yjs framework, an implementation of CRDT data structures (Conflict-free Replicated Data Type). We have learned that since it is so tied to the data model, retrofitting these features into an existing application is considerably more arduous than building the initial data model on the appropriate paradigm from inception. This is why JupyterGIS will be built from the ground up with collaborative editing in mind.

JupyterGIS will be the first open-source GIS tool to provide collaborative editing features.

Importantly, JupyterGIS is part of a broader mission to enable collaborative workflows in open-source technical computing. The JupyterCAD project, a collaborative CAD modeler, is another example of this endeavor. By sharing knowledge and resources, both JupyterGIS and JupyterCAD will benefit from and contribute to each other’s progress, ultimately driving innovation and productivity in their respective fields.

From the desktop to the web

Advanced authoring tools, including IDEs, CAD modelers, image processing software, and GIS applications, are essential for professionals who rely on them for extended periods. These users have high expectations and demand key features to optimize their productivity and workflow. Among these features are extensibility with plugins, configurable keyboard shortcuts, themability, internationalization, scriptability, a unified settings system, and the ability to operate across multiple browser windows and devices.

Developing a new application from the ground up that meets all these requirements is a formidable challenge. This is where the JupyterLab application framework proves invaluable. By leveraging this framework, developers can build custom, feature-rich authoring tools that incorporate these essential features from the outset, making it an ideal solution for creating modern, user-centric applications.

Beyond the pure authoring user interface, for which JupyterLab is a great tool, it is also the perfect tool for integration with notebook-based workflows. As part of this project, we will develop an advanced Python API to manage JupyterGIS sessions, leveraging Jupyter’s robust display system to incorporate sophisticated GIS features inline in Jupyter notebooks and consoles. This integration will further enhance the capabilities and versatility of our application. This feature will follow the same architecture as that of JupyterCAD for the integration with the notebook user interface.

Real-world applications

JupyterGIS is meant to become a general-purpose tool. However, we will work with teams of practitioners on specific real-world applications to ensure that it addresses their needs.

One such project concerns the use of digital and GIS solutions for emergency management. While practitioners involved in emergency management already have experience with such tools, we are convinced that web-based applications built with collaboration in mind from the start can foster improved coordination and collaboration, and therefore the effectiveness of the response.

Response teams comprise different organizations, including the affected municipalities, police, emergency response organizations like the Red Cross, and advisers such as the U.S. Army Corps of Engineers (USACE) in the United States or the Norwegian Water Resources and Energy Directorate (NVE) in Norway. Even when using the same incident management system, miscommunications and misunderstandings can occur between these stakeholders, resulting in delays or improper use of resources. An improved integration of collaborative GIS software can improve upon the existing tooling.

Building collaborations in Europe and beyond

We will partner with key practitioners and make sure JupyterGIS addresses their use cases. Integration in the Copernicus Data Space Ecosystem (CDSE) and the European Open Science Cloud (EOSC) will be key to the adoption of JupyterGIS and demonstrate its deployment in such environments.

Beyond Europe, we are currently working on including this project into a broader scope. We are excited to be partnering with the “LIVE-Env” project, an open-source initiative spearheaded by Alyssa Goodman at Harvard University. Furthermore, we are establishing a key collaboration with the Berkeley Institute for Data Sciences (BIDS) and the Schmidt Center for Data Science and Environment (DSE) at UC Berkeley and 2i2c, with a focus on Jupyter-based geosciences.

Affiliations

[1] QuantStack[2] Simula[3] Simula

JupyterCon 2023 keynotes

2023-01-18T20:39:00+00:00

Alyssa Goodman, Paul Romer, Cory Gwin, Craig Peters

We are excited to announce the keynote speakers for JupyterCon 2023!

First up, we have Alyssa Goodman from Harvard University. Alyssa is a professor of astronomy at Harvard University, and was the founding director of the Harvard Initiative in Innovative Computing. She will be sharing her insights on how Jupyter, together with the glue project, opens up new opportunities for exploratory data exploration and visualization in the field of astronomy and beyond.

Next, we have Paul Romer from NYU. Paul is a Nobel laureate economist and former Chief Economist of the World Bank. He will describe the potential for Jupyter to revolutionize not just how scholars do their research but also how they communicate their results.

Finally, we have a joint talk from Craig Peters and Cory Gwin from GitHub. They will be discussing the development and capabilities of GitHub Codespaces, and how Jupyter is integrated into this powerful tool for code collaboration and development.

We are thrilled to have such an inspiring group of keynote speakers sharing their expertise at JupyterCon 2023. JupyterCon will be held at the Cité des Sciences in Paris on May 10–12, 2023.

Don’t miss out on this opportunity to hear from these inspiring speakers and get your tickets now at https://www.jupytercon.com/. We look forward to seeing you all there!

— Sylvain Corlay, JupyterCon 2023 General Chair

Jupyter Community Workshop: Jupyter for Education

2022-12-05T23:37:00+00:00

We are excited to announce the next in-person Jupyter Community Workshop! It will focus on the use of Jupyter for Education.

The event will be held at the Conservatoire National des Arts et Métiers (CNAM) headquarters in Paris, France, from January 24th to January 26th, 2023. Funding for travel expenses is available for attendees from academia and those from groups which are not well-represented in the Jupyter and wider tech community!

Jupyter Community Workshop are a series of community-organized events to tackle challenging development and design projects, growing the community of contributors, and strengthening collaborations.

This workshop will focus on the use of Jupyter for education. The goal is to bring together contributors, and community members to further the development of Jupyter-based tools and practices for education. Given the broad variety of topics relevant to this workshop (pedagogical practices, automatic grading, deployment challenges), we will compose the program based on the attendance. We are also planning an installment of the PyData Paris meetup at CNAM on the 26th of January, with a focus on open-source tools for education.

The workshop will last three days, with hands-on discussions, hacking sessions, and technical presentations. The goal of this event is to foster collaboration and the sharing of knowledge between Jupyter maintainers and downstream library authors and power users.

Should you be interested in joining us for this workshop, please fill this form. A limited number of spots are available for this event.

We are grateful to CNAM for hosting us and to INRIA for supporting this event. We are also grateful to the sponsors of the Jupyter Community Workshop series, Bloomberg and Amazon AWS.

Avion III de Clément Ader, at the CNAM — photo by Roi Boshifor Wikimedia Commons

MyBinder @ OVHcloud

2022-11-23T11:28:00+00:00

OVHcloud is a long-time supporter of the Jupyter project. In the past few years, they have provided computing resources to several Jupyter-related endeavors to support its sustainability.

In 2019, Project Jupyter and OVHcloud started collaborating on the hosting of the MyBinder service. With tens of thousands of sessions daily, MyBinder was increasingly dependent on a single instance on a single cloud provider. The offer of OVHcloud triggered a community effort to make MyBinder truly multi-cloud. This is how the Binder Federation was created, with OVHcloud as its first member. Since then, the Binder Federation has grown with new members gracefully hosting a portion of the traffic. OVHcloud also hosts the nbviewer.org service.
When the global pandemic hit in 2020, the JupyterCon organizers decided to pivot from an in-person event set to happen in Berlin to a fully online experience. Lorena Barba, the general chair of the committee, envisioned an experience for talks and tutorials in which attendees could interact with the technology. OVHcloud stepped up as the Platinum sponsor, gracefully providing the infrastructure for the various deployments of Jupyter required for the conference, including GPUs for the online tutorials requiring them.

A new partnership with NumFOCUS

Today, we are thrilled to announce that OVHcloud is partnering with NumFOCUS to provide infrastructure to affiliated projects.

Two pilot projects will benefit from this new agreement: Pandas and Jupyter.

On the Jupyter side, these new resources will be used to increase the OVHcloud hosting of MyBinder, thereby helping the project’s sustainability in the long term, continuing to avoid dependency on a single cloud provider. A significant proportion of the traffic of the Binder federation will now be handled by the OVHcloud deployment.

Acknowledgement

We are grateful to Kevin Amil, Marie-Christine Ribeiro, Marie Hering, and Mael Le Gal from OVHcloud for facilitating this new partnership and helping with the migration to the new infrastructure.

We are grateful to OVHcloud for their continued support to the Jupyter project and the NumFOCUS foundation.

Jupyter Community Workshop: JupyterLite

2022-11-10T15:45:00+00:00

We are thrilled to announce the next in-person Jupyter Community Workshop, which will focus on the JupyterLite project!

The event will be held at the OVHCloud headquarters in Paris, France, from December 7th to December 9th, 2022. Funding for travel expenses is available for attendees from academia and those from groups which are not well-represented in the Jupyter and wider tech community!

Jupyter Community Workshop are a series of community-organized events to tackle challenging development and design projects, growing the community of contributors, and strengthening collaborations.

This specific workshop will focus on the JupyterLite project, a JupyterLab distribution that runs entirely in the browser built from the ground-up using JupyterLab components and extensions. JupyterLite allows for very scalable deployments, and already powers inline consoles and notebooks on the websites of major projects of our ecosystem, such as NumPy, SymPy, Pandas, and many more.

Should you be interested in joining us for this workshop, please fill this form. A limited number of spots are available for this event.

We are grateful to OVHCloud for hosting this event. We are also grateful to the sponsors of the Jupyter Community Workshop series, Bloomberg and Amazon AWS.

Enabling the JupyterLab debugger with ipykernel

2021-05-13T11:45:00+00:00

Support for the Jupyter Debugger Protocol just landed in ipykernel

JupyterLab 3.0 includes a visual debugger that allows to interactively set breakpoints, step into functions, and inspect variables with any Jupyter kernel that implements the Jupyter debugger protocol.

The first two language kernels to implement the new protocol were xeus-python (a Python kernel) and xeus-robot (a kernel for Robot Framework). Unfortunately, the reference Python kernel, ipykernel, did not support debugging yet, until now!

Today, we are pleased to announce that debugging support landed in ipykernel, and will be available in the next major release, ipykernel 6.0. Pre-releases including the ipykernel debugger are available.

Debugging with ipykernel

What will change with ipykernel 6.0?

Enabling support for debugging in ipykernel required important changes in the code base regarding the concurrency model of the kernel. The main change is that the processing of messages on the “control channel” now happens in a different thread, allowing for the processing to happen while user code is running.

Ipykernel 6.0 includes several other updates. Tornado coroutines were dropped in favor of native coroutines. The Matplotlib inline backend was split into a separate package, and ipykernel depends on debugpy, an implementation of the Debug Adapter Protocol for Python.

If you are interested in testing out the new features, check out the beta release!

pip install ipykernel --pre

Try it now!

Thanks to MyBinder, you can also try it out without the need of installing anything on your computer. Just follow this link:

What about the future?

A lot of new features are in the works with the JupyterLab debugger.

JupyterLab 3.1 will include several usability improvements to the debugger.
It will also add the ability to submit code for execution when stopped at a breakpoint.
We are working on a richer variable explorer, using Jupyter’s rich display system to enable the rich-rendering of variables in the explorer, to e.g. render dataframes as tables.

Finally, we also plan on adding debugging support to other language kernels.

Acknowledgements

The work of Johan and Sylvain at QuantStack on the debugger support in ipykernel was funded by Two Sigma. We are grateful to Min Ragan Kelley and Matthias Bussonnier, who reviewed the pull requests on debugger support.

About the authors

Johan Mabille is a scientific software developer at QuantStack.

Johan is very active in the Jupyter ecosystem, as the creator of xeus, a C++ implementation of the Jupyter protocol, and several language kernels, such as xeus-python, xeus-cling, and xeus-robot. Johan also made contributions to the Jupyter widgets ecosystem and to JupyterLab.

Sylvain Corlay is the founder and CEO of QuantStack.

As an open-source developer, Sylvain is very active in the Jupyter project with contributions in several components of the stack, including widgets, kernels, nbconvert, and others. He is also a steering committee member of the project.

Sylvain also does volunteer work for the community, as member of board of directors of NumFOCUS, co-organizer of the PyData Paris Meetup, and vice-chair of JupyterCon 2020.

A Curiously Recurring Widget Library

2021-01-27T14:22:00+00:00

Diving into the implementation of xwidgets

Interactive widgets allow Jupyter users to create user interfaces inline in their notebooks, and to turn them into standalone applications with tools such as Voilà.

Language backends for Jupyter interactive widgets exist in Python (with ipywidgets), and C++ (with xwidgets, and the xeus-cling C++ Jupyter kernel), reusing the same frontend implementation of the widgets in JavaScript.

Simple interactive widgets at play in JupyterLab with the xeus-cling C++ kernel

In this article, we dive into some of the C++ techniques used in the implementation of the xwidgets library, including

discussions on value semantics and RAII (Resource Acquisition Is Initialization),
an original application of CRTP (Curiously Recurring Template Pattern),
an original implementation of the observer pattern, xproperty,
a plea for allowing the overloading of the dot operator in C++, to enable better proxy types.

For readers interested in knowing more about interpreted C++, we recommend the following posts:

Interactive workflows for C++ with Jupyter (Jupyter blog), by Sylvain Corlay, Loic Gouarin, Johan Mabille, and Wolf Vollprecht.
- Interactive C++ for Data Science (LLVM blog), by Vassil Vassilev, David Lange, Simeon Ehrig, and Sylvain Corlay
- Interpreted C++ for GIS with Jupyter (Jupyter blog), by Martin Renou

Jupyter widgets are special objects that trigger the creation of a counterpart JavaScript model object in the Jupyter frontend upon creation. The state of the object in the backend is synchronized with the state of the JavaScript frontend object. Views of that widget model are instantiated upon display.

The MVC (Model View Controller) architecture of Jupyter widgets, and synchronization with the backend

This MVC (Model-View-Controller) architecture for Jupyter interactive widgets allowed us to reuse all of the frontend implementation, by simply providing an alternative backend in C++, implementing the same messaging protocol.

In order to tie the lifetime of the kernel and frontend objects, we decided to use the RAII (Resource Acquisition Is Initialization) pattern. RAII is a common programming idiom that consists of tying the lifetime of an object with the holding of a resource. Most typically, the resource is

acquired in the constructor of the object,
released in the destructor of the object.

The RAII pattern is not common in garbage-collected languages because unlike in C++, the time when objects are destroyed is not deterministic. The Python programming language mitigates that issue by introducing context managers, often used to e.g. open and close files.

A consequence of relying on object lifetime for resource management in C++ is to adopt the “value semantics” for widget instances, instead of “reference semantics” which is more typical for widget frameworks like Qt.

Note: value semantics vs reference semantics

If you are not familiar with the concepts of value and reference semantics in C++, I recommend reading the excellent FAQ of isocpp.org.

To summarize:

- with value semantics, objects hold actual values, and copying an object copies their attributes. With value semantics, one should provide implementations of copy, and move constructors*, as well as* copy and move assignment operators*. Besides, values should not have virtual methods.*

- with reference semantics*, objects are manipulated through references (or pointers) and never by value. Reference semantics is typically used for polymorphic programming with* virtual methods*. With reference semantics, it is recommended to delete copy and move constructors, as well as copy and move assignment operators to avoid accidental copies when passing objects to functions taking arguments by values, causing object slicing. (They can also be made private). Explicit cloning of a reference semantics object is generally allowed via a call to a* clone virtual method.

In a C++ codebase, the existence of public copy or move constructors alongside virtual methods in the same class is generally a sign of a bad design.

The use of value semantics for xwidgets provides clear lifetime management for associated resources. Careful use of the move semantics provides fine-grained control.

Illustration of the move semantics for xwidgets

Another advantage of using value semantics in xwidgets is that C++ beginners who are typical users of the Jupyter notebook (often used by instructors) can easily manipulate “widgets as values” without having to deal with manual memory allocation etc.

With value semantics, addressing the lifetime of objects becomes the responsibility of the framework author, in a carefull implementation of (copy and move) constructors, destructors, and (copy and move) assignment operators.

Static polymorphism and the CRTP pattern

While the use of value semantics provides fine-grained control over the lifetime of widgets, it prevents the use of virtual methods in their implementation. Code reuse is achieved with static polymorphism techniques and specifically the CRTP pattern.

The Curiously Recurring Template Pattern (CRTP) is the practice of making a class X derive from a class template instantiation using X itself as a template argument.

class X : public base<X>

CRTP is commonly used by matrix or tensor algebra libraries making use of expression templates (such as xtensor, eigen, or blitz) to prevent the overhead of virtual function dispatch for operations likely to be performed in a loop, such as element access.

Placing common code in a CRTP base allows code reuse without the overhead of virtual dispatch. However, the template base class is specialized for each final type increasing the resulting binary size. This can be mitigated by factoring as much of the code in a base class that would not be templated by the derived type.

CRTP in xwidgets — closing the recursion

In order to allow for code reuse without virtual inheritance, xwidgets’ class hierarchy is entirely based on CRTP. Our naming scheme is that CRTP bases, which should not be instantiated directly are prefixed with the letter x and final concrete widget types are not.

Upon construction of e.g. the button widget, constructors of base types are called in the order of inheritance, which is why we need to establish the connection with the frontend in the constructor of the most derived type, after all attributes have been initialized, and send a message to the frontend with all the values.

The pattern for creating the most derived type being always the same, we defined the xmaterialize template class closing the CRTP hierarchy with

using button = xmaterialize<xbutton>;

The xmaterialize template class is defined as final, to prevent further inheritance. The constructors and assignment operators forward to those of the CRTP bases and implement the RAII pattern.

template <template <class> class B, class... P>
class xmaterialize final : public B<xmaterialize<B, P...>>
{
public:

    using self_type = xmaterialize<B, P...>;
    using base_type = B<self_type>;

    template <class... A>
    xmaterialize(A&&...)
        : base_type(std::forward<A>(args)...)
    {
        this->open();      //  RAII: create the frontend model.
    }

    template <template <class> class B, class... P>
    inline xmaterialize<B, P...>::~xmaterialize()
    {
        if (!this->moved_from())
        {
            this->close();  // RAII: delete the frontend model.
        }
    }

...

};

The full implementation of xmaterialize (as of xwidgets 0.25) is available here. Beyond the logic described in this section, it also includes the handling of the move semantics and the method chaining API which is the subject of a later section.

Precompilation and binary size optimization

Even though xwidgets is fully based on template types, we decided to precompile all final widget types for faster interactive use with the xeus-cling kernel. This is achieved with

an extern declaration in the header (here in xbutton.hpp)

extern template class xmaterialize<xbutton>;

and in the source file (here in xbutton.cpp), an instruction for the precompilation

template class XWIDGETS_API xmaterialize<xbutton>;

Doing this for all widget types of the library initially resulted in a large compiled binary size. Using the button widget as an example, we see that base types are templated by the final type in the class hierarchy button -> xbutton<button> -> xwidgets<button> -> xobject<button>, and therefore, their binary representation is duplicated for each final type.

A strategy for reducing the binary size has been to factor out as much of the logic of xobject<D> in a non-template base xcommon improving compilation speed and preventing binary code duplication.

Xproperty: an implementation of the observer pattern

In order to update the frontend upon changes of widget properties, xwidgets relies on an implementation of the observer pattern called xproperty. xproperty is to xwidgets what traitlets are to ipywidgets.

In order to trigger observers and validators in the owner object upon assignment of new values, xproperty relies on the overload of the assignment operator =.

template <class T, class O>
template <class V>
inline auto xproperty<T, O>::operator=(V&& value) -> reference
{
    // Before assigning the new value, invoke validators
    // which may also mutate the value.
    m_value = owner()->template invoke_validators<T>(m_name, std::forward<V>(value));

    // Call class-level observer.
    owner()->notify(m_name, m_value);

    // Call registered observers for that attribute
    owner()->invoke_observers(m_name);

    // Return the new value 
    return m_value;
}

Jupyter interactive widgets have many attributes that may be specified at construction time.

In the Python implementation, this is handled with keyword arguments, but the C++ programming language does not support keyword arguments. There exist various approaches to enable this feature with advanced metaprogramming techniques. In the case of xwidgets, this need is limited to the initialization of xproperty attributes, which allowed us to adopt a more scoped approach: a method chaining API for property initialization:

auto slider = slider<double>::initialize()
  .min(-1.0)
  .max(1.0)
  .description("Another slider")
.finalize();

this was enabled by overriding the function call operator () on xproperties to pass an intial value (only when the said property is an rvalue). A static initialize method is used instead of the default constructor to prevent the initialization of the JavaScript counterpart while all attributes may not have been set yet. The frontend counterpart is only acquired with the finalize() call.

Building upon xwidgets

Jupyter interactive widgets are not limited to the controls available in the core package. In fact, there is a rich ecosystem of widget libraries built upon the core framework: ipyvolume (3-D plotting), ipyleaflet (maps visualization), bqplot (2-D plotting), ipygany (3-D mesh visualization), ipycanvas (generic drawing), ipywebrtc (streaming video and audio), and many many more.

For the C++ programming language, we have already provided:

xleaflet (the C++ equivalent to ipyleaflet, reusing the same frontend),
xwebrtc (the C++ equivalent to ipywebrtc, reusing the same frontend).

Screencast of xleaflet in JupyterLab, loading and visualizing a GeoJSON dataset in a C++ notebook.

Potentially, C++ backend to all Jupyter interactive widget packages could be provided, creating a huge opportunity for interactive data visualization in C++.

C++ should allow overloading the dot operator

xwidgets and xproperty makes heavy use of value semantics, and proxy objects.

At the moment, to access an attribute or method of a value held in an xproperty object, we must first call the function call operator () to access the undelying object first, which is cumbersome.
This is also an issue in other places in the xwidgets stack, when making use of e.g. reference proxies.

Being able to automatically map all methods of the underlying type to be accessible in the xproperty would be incredibly powerful, and remove the need for explicitly accessing the underlying.

Interestingly, the C++ standard does have the equivalent operator overload when it comes to pointer semantics, with the arrow operator ->. If overloading the dot operator . was allowed, this could enable this kind of usecase:

struct V
{
    void f();
};

struct X
{
    V& operator.() { return m_value; }
    V m_value;
};

in which case, X::f would call V::f .

Allowing the overloading of the dot operator . would also enable usecases such as smart references (similar to smart pointers, but with value semantics) and fully-fledged reference proxies (such as the return type of operator[] for std::vector<bool>).

Try it online!

Thanks to MyBinder, you can try xwidgets without the need of installing anything on your computer. Just follow this link:

About the author

Sylvain Corlay is the founder and CEO of QuantStack, an open-source software development studio comprising maintainers of key projects of the scientific computing ecosystem.

As an open-source developer, Sylvain is very active in the Jupyter project, contributing to the Xeus stack, Jupyter interactive widgets, Voilà dashboards. He is a member of the Jupyter steering committee and was the vice chair of JupyterCon 2020. Sylvain also contributes to the conda-forge project and he is the co-creator of the Xtensor C++ tensor algebra library.

Benjamin Ragan-Kelley

2020-10-03T21:02:00+00:00

JupyterCon 2020 keynote speaker announcement

Benjamin Ragan Kelley

It all started in April 2005 when freshman Min RK reached out to his physics professor, Brian Granger,

Dr. Granger,
Talking to you this afternoon prompted me to put the rather vague idea I had into a less vague writing, so I thought I would send you a better explanation of the application I had in mind…

laying out his vision for an interactive development environment for computational physics. Reading this today, we can see how prescient that vision was, and included key elements of what Jupyter has become.

Min then joined Brian and Fernando in contributing to IPython and became one of the main driving forces behind a project that has grown to global prominence, with a lasting influence on the entire field of scientific computing.

Some of Min’s early experiments for a web-based notebook.

The rest is history, Min has spent the following fifteen years contributing to IPython and Jupyter. We owe him some of the key components of the project such as JupyterHub. He was honored along with the rest of the Jupyter steering council with the 2017 ACM Software System Award.

Today, Min is a senior research engineer and the head of the Department of Numerical Analysis and Scientific Computing at the Simula Research Laboratory in Oslo, Norway.

Beyond Jupyter, he has contributed widely to open source software, especially in the scientific Python community. He helps maintain numerous scientific packages in the conda-forge package management system.

It is an honor for me to introduce Min’s keynote! We hope you will join us at the conference and enjoy his talk. There are only a few days left and it is still time to sign up.

— Sylvain Corlay, JupyterCon 2020 Vice Chair

The JupyterCon schedule of tutorials and talks is published. Join us!

The templating system of nbconvert 6

2020-09-26T07:31:00+00:00

One of the main changes in nbconvert 6 is the refactor of the template system, which should be easier to extend and build upon.

In this article, we dive into the template system, and provide a tutorial on how to build a custom template for nbconvert or Voilà.

From Classic to Lab

My notebooks look different!

If you are accustomed to convert notebook files to HTML by typing

jupyter nbconvert notebook.ipynb --to html

you may have noticed differences in the generated HTML when switching to the latest release of nbconvert. In fact, nbconvert now produces the same DOM structure as JupyterLab’s notebook implementation, which is styled with JupyterLab’s CSS.

One can even apply the dark theme:

jupyter nbconvert xleaflet.ipynb --to html --HTMLExporter.theme=dark

The HTML output of nbconvert with the lab template and the dark theme

While JupyterLab uses CodeMirror to render code cells, nbconvert makes use of the Pygments library to produce syntax-colored static HTML. To mimick the JupyterLab CodeMirror styling, we created a Pygments theme called jupyterlab-pygments. JupyterLab Pygments uses JupyterLab’s CSS variables for coloring and will therefore reflect the theme that is applied to the notebook.

Note: at the moment, only the default light and dark themes are supported, but we plan on adding support for third-party JupyterLab themes after the release of JupyterLab 3, which introduces a new packaging system for extensions.

But I wanted my notebooks to look the same!

Well, if you want to retain the classic notebook styling that was used in earlier versions of CSS, it is still possible using the classic template.

jupyter nbconvert xleaflet.ipynb --to html --template classic

The HTML output of nbconvert with the classic template

With this template, you retrieve the original style of nbconvert outputs and of the classic notebook.

Another perk of the new nbconvert release is the WebPDF exporter. The WebPDF exporter supports the same templates and themes as the HTML exporter, and produces a PDF output that renders the same rich content as the HTML exporter, such as rich HTML tables, widgets etc.

jupyter nbconvert xleaflet.ipynb --to webpdf --HTMLExporter.theme=dark

The WebPDF output of nbconvert with the lab template and the dark theme

Executing the notebook before rendering

Nbconvert’s two main categories of transformations are preprocessors and exporters. Preprocessors take a notebook as an input, and return a transformed notebook, while exporters return other types of content, such as HTML or PDF. An important preprocessor is the ExecutePreprocessor, which spawns a kernel for the notebook, execute all cells, and populate outputs.

It can be invoked before the export by passing --execute. For example, the xleaflet.ipynb notebook uses the xeus-cling C++ kernel and makes use of the xleaflet interactive widget, which can be displayed when converting to HTML or with the WebPDF exporter.

jupyter nbconvert xleaflet.ipynb --to html --execute

The HTML output of nbconvert with the execute preprocessor

Similarly, the WebPDF exporter will also display interactive widgets!

The PDF output of nbconvert with the execute preprocessor

Note: Future plans for the WebPDF exporter include offering more options to users with respect to page breaks, and providing bookmarks for the main sections of the document.

Nbconvert templates

Unlike with earlier versions of nbconvert, templates are now directories, which may contain a jinja template but also other assets, such as macros, CSS files etc. The nbconvert template system also provides an inherittance mechanism which makes it simple to tweak existing templates in a derived one, by overriding bits of it.

Selecting a template

Most exporters in nbconvert are subclasses of TemplateExporter, and make use of jinja to render notebooks into the destination format. Nbconvert templates can be selected by name with the --template command line option.

For example, the reveal template, shipped with nbconvert, turns Jupyter notebooks into HTML slideshows using the RevealJS library. Which cells should be skipped, or where breaks betweens slides should be, are specified in the notebook cell metadata. The classic notebook and JupyterLab both provide means to set the appropriate values.

To select the reveal template, simply type:

jupyter nbconvert <path-to-notebook> --to html --template reveal

In the case of the xleaflet.ipynb notebook showed earlier, we get:

Creating a reveal slideshow with the dark theme

This demonstrates that the nbconvert templating system can be used to completely override how we look at notebook documents. Using metadata, we can even create arbitrary layouts and rich views of the same content.

Where are nbconvert templates installed?

Nbconvert 6 templates are directories containing resources such as jinja templates and other assets. They are installed in the data directory of nbconvert, namely <installation prefix>/share/jupyter/nbconvert.

Running jupyter --paths shows all Jupyter directories and search paths. For example, on Linux, jupyter --paths returns:

$ jupyter --paths
config:
    /home/<username>/.jupyter
    /<sys-prefix>/etc/jupyter
    /usr/local/etc/jupyter
    /etc/jupyter
data:
    /home/<username>/.local/share/jupyter
    /<sys-prefix>/share/jupyter
    /usr/local/share/jupyter
    /usr/share/jupyter
runtime:
    /home/<username>/.local/share/jupyter/runtime

In our case, only the data section is relevant. Listing the content of <sys-prefix>/share/jupyter/nbconvert/templates in a raw installation of nbconvert will show

$ ls <sys-prefix>/share/jupyter/nbconvert/templates
asciidoc base classic compatibility html lab latex markdown python reveal rst script

The base template should not be used directly, but is typically inherited from. Thecompatibility directory provides some content for backward compatibility with earlier verions of nbconvert. Three templates are available for the HTML exporter: lab, classic, and reveal.

The content of nbconvert templates

The conf.json file

Nbconvert templates all include a conf.json file used to indicate the base template that it is inheriting from, the mimetype corresponding to that template (which determines which exporters are compatible with it, and which file is the entry point), and preprocessors to run when using that template before running the exporter. For example, inspecting the configuration of the reveal template we see that

it inherits from the lab template,
exports text/html, and therefore will only work with the HTML and WebPDF exporters.
and runs two preprocessors called 100-pygments and 500-reveal:

{
  "base_template": "lab",
  "mimetypes": {
    "text/html": true
  },
  "preprocessors": {
    "100-pygments": {
      "type":         "type": "nbconvert.preprocessors.CSSHTMLHeaderPreprocessor",
        "enabled": true
    },
    "500-reveal": {
      "type": "nbconvert.exporters.slides._RevealMetadataPreprocessor",
      "enabled": true
    }
  }
}

The CSSHTMLHeaderPreprocessor inlines the CSS required for the syntax highlighting of input cells.
The RevealMetadataPreprocessor massages the notebook metadata and consumes the information required to set up the layout of the slideshow.

Nbconvert walks up the inheritance structure determined by conf.json and produces an agregated configuration, merging the dictionaries of registered preprocessors. The ordering of the preprocessor names determines the order in which they will be run.

Jinja templates

Besides the conf.json file, nbconvert templates most typically include jinja templates files. They may also override files from the base templates, or provide extra content.

For example, inspecting the content of the classic template located in share/jupyter/nbconvert/templates/classic, we find the following content:

share/jupyter/nbconvert/templates/classic
├── static
│   └── styles.css
├── conf.json
├── index.html.j2
└── base.html.j2

we see that it includes the index.html.j2 jinja template (which is the main entry point for HTML exporters) as well as CSS and a base template file in base.html.j2. The only preprocessor listed in conf.json is the pygments syntax highlighting…

Inheritance in Jinja

In nbconvert, jinja templates can inherit from any other jinja template available in its current directory or base template directory by name. Jinja templates of other directories can be addressed by their path from the Jupyter data directory. Using the path is also useful when using a jinja template that may be overriden locally.

For example, in the reveal template, index.html.j2 extends base.html.j2 which is in the same directory, and base.html.j2 extends lab/base.html.j2. This approach allows using content that is available in other templates or may be overriden in the current template.

Building a custom template

Now, let’s create a custom template! If you work at ACME Corporation, you may want to create a template that follows the graphical charter of ACME Corp, and includes the logo in a banner.

Besides the logo, titles should also use the “ACME Regular” font, which has a cartoon-style look. The other parts of the template are inherited from the regular lab template.

Setting up a logo banner, and the font change, the acme nbconvert template produces the following result with a very simple notebook:

jupyter nbconvert acme.ipynb --to html --template acme

The HTML output of nbconvert with the acme template

Now you can create sophisticated templates making use of sophisticated front-end framework and processing the notebook metadata in creative ways!

The source for the ACME nbconvert template is available here.
Beyond the template files in share/jupyter/nbconvert/acme, the repo provides the logic for packaging this template into a PyPI wheel with data files. You will also find content related to the use of that template with Voilà, which is the subject of the next section.

From nbconvert templates to Voilà templates

Voilà turns Jupyter notebooks into standalone web applications and dashboards.

It differs from nbconvert in that the output web application is connected to a Jupyter kernel, allowing it to respond to user input through widget controls and other UI components, while with nbconvert, any action that requires a roundtrip to the kernel will not work.

However, Voilà leverages the nbconvert template system to benefit from their flexibility in overriding the front-end looks and behavior. From a user standpoint, the system is made so that the same templates will be usable for both systems.

However, template authors interested in advanced features of Voilà may be interested in the following information.

Where are Voilà templates installed?

Voilà templates are installed in the <installation prefix>/share/jupyter/voila directory (while nbconvert templates are in <installation prefix>/share/jupyter/nbconvert).
Just like nbconvert templates, Voilà templates are directories, they use the same conf.json configuration mechanism.
Voilà can use nbconvert HTML templates without modification.
When there exists an nbconvert and a Voilà template of the same name, the conf.json files are recursively merged, as well as the content of the directory, with a higher precedence for the Voilà template.

Overriding nbconvert templates with Voilà

When specifying the acme template that we developed earlier, with command

voila xleaflet.ipynb --template acme

Voilà will look pick up the acme nbconvert template. Since this template inherits from lab and Voilà has an overridden lab template (with e.g. the logic for rendering widgets), it will pick up the Voilà flavor of the lab template. Most typically, the Voilà flavor of a template does not add much on top of nbconvert besides boilerplate such as

a call to the macro that includes the JavaScript assets for the Voilà front-end logic.
calls to macro related to error logging when using the Voilà preview.
calls to macros related to the progressive rendering of notebooks as it is being executed, and the display of an “in-progress” spinner.

These macros are provided in the base Voilà template, and can also be overridden in derived templates.

Future developments

In the coming weeks and months, we plan on polishing the experience of nbconvert users.

JupyterLab 3, which should be released shortly, includes a more dynamic extension system which does not require the main JupyterLab application to be rebuilt. We plan on adding support for a category of lab extensions called “mime renderers“ which are used for rich rendering of data in cell outputs. This should enable the use of complex mime types such as GeoJSON or Vega visualizations in nbconvert and Voilà.
We are working on improving the reveal template, to include a custom reveal theme making use of JupyterLab CSS variables, so that it can be easily combined with JupyterLab themes.
The JupyterLab 3 extension system may also enable us to enable third-party JupyterLab themes in nbconvert.
The WebPDF exporter should expose options on output format and where page breaks should be. At the moment, we prevent page breaks until the maximum dimensions of PDF documents are reached.

Acknowledgements

Many people were involved in the nbconvert 6 release! The full list of contributors is available here.

Among them, we are especially indebted to Maarten Breddels, who was the main architect of the new template system.
We owe the split of the execute preprocessor and the new nbclient package to Matthew Seal and David Brochart. Matthew took on a large amount of maintenance work on the project over the past year.

The work on Voilà and nbconvert by the QuantStack team was funded by Bloomberg.

About the Author

Sylvain Corlay is the CEO of QuantStack, an open-source software development team specialized scientific computing comprising maintainers of major projects of the ecosystem.

As an open-source developer, Sylvain is mostly active in the Jupyter ecosystem, and the general PyData stack. He is currently a steering committee member for Project Jupyter, and a member of the board of directors of NumFOCUS.

Report on the Jupyter Community Workshop on Dashboarding

2020-02-14T21:06:00+00:00

From June 3rd to June 6th 2019, thirty-five developers from the Jupyter community met in Paris for a four-day workshop on dashboarding with Project Jupyter.

Attendees to the Jupyter Community Workshop on Kernels (Photo credit to Lindsey Heagy)

For four days, attendees worked full time on the Jupyter project, including hacking sessions and discussions on improvements to Jupyter components and new development. We were lucky to count a large number of core developers to the project in the group.

Beyond the hacking sessions, each day was concluded with a series of presentations and demos of the progress made during the workshop. In partnership with the PyData Paris team, we had a special installment of the PyData Paris Meetup with

an invited presentation by Emmanuelle Gouillart on Plotly Dash,
a series of lightning talks by attendees of the workshop on their achievements, including a talk by Philip Rudiger on the first release of Panel, and an announcement of the first releases of Voilà and the Voilà Gallery.

We ended the week with a social evening at the QuantStack offices in Paris.

Why a workshop on Jupyter Dashboarding with Jupyter?

The Jupyter ecosystem is used extensively in scientific computing both in academia and industry, and a rich ecosystem of data visualization tools has been developed around the Jupyter widgets frameworks, from geographical data visualization to protein folding simulation.

However, the Jupyter ecosystem still did not provide a means for developers to transition from notebooks to stand-alone web applications that can be accessed by multiple users.

This has been a longstanding request from the community: provide better tools built upon the Jupyter stack to share results with students, peers, or the general public.

These are the challenges that we decided to tackle during that week. The workshop was attended by many Jupyter core developers.

Highlights of the week

Many of the developers spent the week working on the Voilà and Panel projects. Both projects had their first public releases during that week (see the first public announcement of Panel and Voilà).

During this week, a team of participants including Yuvi Panda, Pascal Bugnion, and Jeremy Tuloup iterated on the first version of the Voilà gallery. Several first-time contributors to the widget framework authored example dashboards for the gallery, showcasing their existing work. Yuvi also produced the first deployment scenarii for Voilà on Heruku.
Cheryl Quah, from Bloomberg MC-ed a panel on dashboarding in the Jupyter ecosystem, including lots of questions and comparisons with Dash.
Philip Rudigger started working on a Bokeh/ipywidgets integration for better interoperability between the two frameworks.
Other contributors iterated on creating new Voilà templates, such as voila-vuetify, adding the ability to position Jupyter widgets and outputs in arbitrary location in the dashboard template. Grant Nestor created visual mockups for a UI for creating dashboard layouts in JupyterLab. Grant also helped iterating on logos for the project, and gave a presentation on dynamically loading JavaScript modules in the browser.

Acknowledgments

This event would not have been possible without the generous support provided by Bloomberg, who made this workshop series possible

We are grateful to Société Générale for funding the catering for the workshop.

The hosting of the workshop at CRI was paid for by QuantStack.

The public meetup was organized in partnership with the PyData Paris team.

Finally, we especially thank Ana Ruvalcaba from Project Jupyter for her incredible work on the logistics and finances of the Jupyter Community Workshop series.

Voilà is now a Jupyter subproject

2019-12-29T12:11:00+00:00

It is a great pleasure to announce that the Voilà project has been incorporated as a Jupyter subproject. Voilà will now be subject to the Jupyter governance and code of conduct.

For reference, the Jupyter Enhancement Proposal (JEP) for the Voilà incorporation is available here.

What is Voilà?

Voilà helps you communicate insights, by transforming a Jupyter Notebook into a stand-alone web application you can share. It gives you control over what your readers experience in a secure and customizable interactive dashboard.

The easiest way to get started with Voilà is to install it via pip or conda and type voila some_notebook.ipynb to turn the said notebook into a dashboard.

Besides, Voilà includes a templating system that allows to overload the behavior of the front-end. Using this templating system, Voilà can be used to create

slideshows (with voila-reveal)

A Voilà slideshow created with the voila-reveal template.

dashboards (with voila-gridstack)

A Voilà Dashboard based on the voila-gridstack template.

Why moving Voilà under the Jupyter governance?

While the project was initially started by QuantStack, the team now comprises developers from Bloomberg, UC Berkeley, JP Morgan, and Cal Poly San Luis Obispo. OVH has been supportive of the project by kindly providing the free hosting of the gallery on their infrastructure.

We believe that the multi-stakeholder nature of the Voilà project is well-suited for the Jupyter organization.

The Voilà project is largely built upon Jupyter subprojects and standards.

The standard notebook file format is the main entry point to Voilà.
nbconvert is used for the conversion to progressively-rendered HTML.
naturally, we use jupyter_client for handling the execution of notebook cells
jupyter_server is the default back-end.
JupyterHub is at the foundation of the voila-gallery project.
JupyterLab components (mime renderers, input and output areas) are used in the front-end implementation. Voilà also includes a preview JupyterLab extension.
ipywidgets and custom jupyter widget libraries such as bqplot, ipyvolume, ipyleaflets provide the bulk of the interactivity of Voilà applications.

Voilà is more a remix of existing Jupyter components (with changes to enable that use case) than a completely new application.

Resources

Should you be interested in Voilà, feel free to try it on Binder or locally! You can also engage with the developer community during our public team meetings and the various GitHub repositories of the project:

the Voilà GitHub repository is available here: https://github.com/voila-dashboards/voila.
the Documentation is hosted on Read the Docs: https://voila.readthedocs.io.
the Team Compass holds the calendar for the public developer meetings, as well as the meeting minutes: https://voila-dashboards.github.io.
the Announcement of the first Voilà release was published on this blog: https://blog.jupyter.org/and-voilà-f6a2c08a4a93

Acknowledgements

Voilà was started by the team of open-source developers at QuantStack as a separate project, but with the full intent to incorporate it into Jupyter. The initial project development at QuantStack was funded by Bloomberg.
Now, Voilà contributors work in many institutions, including UC Berkeley, Cal Poly San Luis Obispo, JP Morgan, and Faculty (formerly ASI Data Science).
The Voilà Gallery is kindly hosted by OVH.

Field Report on the Kernel Community Workshop

2019-10-07T13:24:00+00:00

From May 27th to May 29th 2019, thirty developers from the Jupyter community met in Paris for a three-days workshop on the Jupyter kernel protocol.

Attendees to the Jupyter Community Workshop on Kernels

For three days, attendees worked full time on the Jupyter project, including hacking sessions and discussions on improvements to Jupyter protocols and standards.

We were lucky to count five core developers to the project and several more regular contributors to the larger ecosystem in the group!

Beyond the hacking sessions, each day was concluded with a series of presentations and demos of the progress made during the workshop.

Why a workshop on Jupyter kernels?

The Jupyter Kernel protocol is one of the main extension points of the Jupyter ecosystem. Dozen of language kernels have been developed by the community, and these languages can then leverage the other components of the stack, such as the notebook, the console, and interactive widgets.

One of the objectives of this event was to foster collaboration between kernel developers and core contributors and develop common tools used across all language kernels. Another goal was to work improving the protocol, to support visual debugging, parameterized kernels, etc.

Technical Achievements

A lot of work was done during the workshop. Several people were working on new Jupyter kernels for programming languages.

We were amazed to see how much was achieved in just three days. Today, we are still working on projects that stemmed from this community workshop.

JupyterLab Debugger

One area in which the Jupyter team is actively working is support for visual debugging in JupyterLab. This is a major endeavor spanning from the front-end to the back-end, including changes to the Jupyter protocol.

The kernel workshop was the occasion for several developers (Wolf Vollprecht, Maarten Breddels, Johan Mabille, Loic Gouarin) to get together and hack on the implementation of the frontend. A lot of work had been done ahead of the workshop with a new Python kernel based on the Xeus library. This new kernel includes a backend to the Debug Adapter Protocol over kernel messages.

The JupyterLab Visual Debugger

There are other key contributors to the debugger project who did not attend the community workshop. We should mention Jeremy Tuloup and Afshin Darian who are spearheading the frontend development.

A Calculator Jupyter Kernel based on xeus

Vasavan Thirusittampalam and Thibault Lacharme were in the middle of their summer internships as scientific software developers at QuantStack when they attended the workshop. During the event, they set themselves up to implement a new Jupyter kernel in C++!

At the end of the workshop, they were able to demonstrate a functional calculator kernel based on xeus, a C++ implementation of the Jupyter protocol! This was later polished and resulted in the publication of a blog post on the Jupyter blog: “Building a Calculator Jupyter Kernel”. This example may serve as an example for people interested in creating new language kernels with xeus.

A prototype Julia backend to Jupyter interactive Widgets.

During the workshop Sebastian Gutsche (who is the author of the Singular Jupyter kernel and a co-author of the GAP kernel), set himself to develop a backend to Jupyter interactive widgets for the Julia progamming language. The current state of the code can be found here.

The Julia backend to jupyter widgets is still early-stage but should this project be completed, it would enable other Jupyter widget libraries for the users of the Julia Jupyter kernel, such as ipyvolume, ipyleaflet, bqplot, etc.

The Julia backend to Jupyter widgets

Parameterized Kernelspecs

Paul Ivanov and Kyle Kelley worked on ironing out the proposal for the support of parameterized kernelspecs. The objective is to make it possible to pass user-specified arguments to kernels upon launch.

In the proposal, the parameters expected by the kernel executable and the possible values for these parameters may be specified in the kernelspec file or a companion file to the kernelspec in the form of a JSON schema.

Paul and Kyle produced extensive notes on various aspects on the subjects such as

the storing of previously used values of the parameters (in notebooks)
the web UI design on how to specify parameters
the definition of default values

More details will be shared at a later stage on the subject of parameterized kernelspecs.

Note: the subject of parameterized kernelspecs was also central in the earlier community workshop about the Jupyter server organized at IBM by Luciano Resende.

Forking of Jupyter Kernels

Starting of kernels is an expensive operation, and executing a whole notebook even more. For certain use cases (such as dashboards based on Voilà), it is useful to have a pre-executed notebook ready, so that dashboards can be presented instantly to a user. To enable this, Maarten Breddels came up with the idea of forking kernels and put together a proof-of-concept implementation. The implementation can be found in pull requests to jupyter_client (PR #441) and ipykernel (PR #410).

Forking kernels will also allow interesting features such as an undo/rollback of cell execution in the notebook. While it is still a proof of concept, this subject has already sparked interest in the community.

Acknowledgments

This event would not have been possible without the generous support provided by Bloomberg, who made this workshop series possible

We are grateful to CFM for gracefully hosting the event.

We should also thank Ana Ruvalcaba from Project Jupyter for her incredible work on the logistics and finances of the Jupyter Community Workshop series.

Jupyter Community Workshop: Building upon the Jupyter Kernel Protocol

2019-04-04T09:52:00+00:00

We have some exciting news about the Jupyter Community Workshop on kernels!

The workshop will be held in Paris, France, from May 27th to May 29th 2019. The event is being hosted at Capital Fund Management(CFM), in the heart of Paris.

In this three-days event, we will have hands-on discussions, hacking sessions and technical presentations, with core Jupyter developers and custom language kernel authors. We will discuss the potential improvements and additions to the protocol, and work on common tools and infrastructure for testing and evaluating Jupyter kernels.

Should you be interested in joining us for this workshop, please fill this Google Form.

Why a Workshop on Jupyter Kernels?

The Jupyter Kernel protocol is one of the main extension points of the Jupyter ecosystem. Dozen of language kernels have been developed by the community, and these languages can now leverage other components of the stack, such as the notebook, the console, and interactive widgets.

Most Jupyter kernel authors rely on the reference documentation for the Jupyter kernel protocol, and reach out to the core team for questions on the different communication channels. One of the objectives of this event is to foster collaboration between kernel developers, on the development of common tools for testing and supporting all aspect of the protocol.

Beyond implementation of the current protocol, we will discuss the potential improvements and additions to the protocol, such as the support of debugging messages. Another issue to be discussed and worked on is the ability to parameterize kernelspecs, which has been extensively discussed at the last Jupyter developer meeting in Berkeley. Parameterized kernel specs could be used for kernels requiring extra command-line arguments. These arguments could be used to pass database connection credentials, or other metadata and parameters used for e.g. parallel computing frameworks.

Common infrastructure for testing kernels, call into remote kernels with the kernel gateway would also be in scope for this event.

Acknowledgements

This would not have been possible without the generous support provided by Bloomberg, who made this workshop series possible.

We are also grateful to CFM for gracefully hosting the Jupyter kernels community workshop.

Jupyter Community Workshop: Dashboarding with Project Jupyter

2019-02-19T18:02:00+00:00

We have some exciting news about the Jupyter Community Workshop on dashboarding!

The workshop will be held in Paris, France, from June 3rd to June 6th, 2019. The event is being hosted at Center for Interdisciplinary Research (CRI), in the heart of Paris.

The workshop committee consists of Maarten Breddels (Freelance), Pascal Bugnion (Faculty.ai), Sylvain Corlay (QuantStack), Alexandre Gramfort (INRIA), and Vidar Tonaas Fauske (Simula).

The workshop will last four days, with hands-on discussions, hacking sessions, and technical presentations. The goal of the event is to foster collaboration and the sharing of knowledge between downstream library authors and contributors, and favor upstream contributions.

Should you be interested in joining us for this workshop, please fill this Google Form.

In addition to the Community Workshop, we plan on holding a public Meetup on June 5th, in partnership with the PyData Paris Meetup, with a series of lightning talks of Project Jupyter and related projects.

Why a Workshop on Dashboarding?

The Jupyter ecosystem has great tools for teaching, exploration and development. Dashboards allow users to interact with a kernel with interactive controls, plots, maps, etc., and allow researchers and data scientists to share their results with students, with their peers, and with the general public. Currently, users of Jupyter are (mostly) forced towards other Python or R libraries or they make direct use of front-end technologies or develop directly in JavaScript.

There are existing early technologies that allow serving dashboards based on notebooks, most notably voila. The goal of this workshop is to gather core Jupyter widgets developers, members of the community and users with experience in dashboarding to bring dashboarding to a level where it can be used by all members of the Jupyter ecosystem. Ultimately, we envisage users being able to develop and deploy dashboards entirely within the Jupyter ecosystem.

We will lay the foundations for dashboarding as a first-class citizen in the Jupyter ecosystem.

Acknowledgements

This would not have been possible without the generous support provided by Bloomberg, who made this workshop series possible.

We are also grateful to the CRI for gracefully hosting the dashboarding community workshop.

Release of ipywidgets 6.0

2017-03-01T15:44:00+00:00

We are pleased to announce the release of ipywidgets 6.0, the Jupyter interactive widget library. Jupyter interactive widgets enable building simple GUIs in the Jupyter notebook.

ipywidgets 6.0 is a major release of the project. In this release, we closed 197 issues and 309 pull requests with 732 commits.

Installation

Using conda:

conda install ipywidgets -c conda-forge

Using pip:

pip install ipywidgets
jupyter nbextension enable --py --sys-prefix widgetsnbextension

What’s new in ipywidgets 6.0?

Custom widget styling

In addition to the existing Widget.layout attribute, which enables the specification of layout-related css properties for the top-most DOM element of a widget, we added a new Widget.style attribute. This attribute enables custom styling of various widget types (such as Button.style.button_color).

The top-level styling properties that were deprecated in ipywidgets 5.0 have been removed.

Using widgets outside of the notebook

Another change in 6.0 is the ability to render Jupyter interactive widgets outside of the notebook:

Widgets can be rendered in Sphinx documentation, either with the jupyter-sphinx extension or the nbsphinx notebook converter.
Widgets now render on nbviewer.
Jupyter interactive widgets can be embedded into static web sites, such as http://jupyter.org/widgets.

Custom widget libraries built upon ipywidgets 6.0 can also take advantage of these features.

This feature required the formal specification of a mime type for Jupyter interactive widgets, which is now contained in the new jupyter-widgets-schema npm package.

Redesign of the core widgets

The widgets provided with the base Jupyter widget libraries have gone through a major re-design.

We also migrated from the less CSS preprocessor to the use of css variables. In addition to the main benefits of adopting web standards, we take advantage of css variables defined in JupyterLab to style the interactive widgets consistent with the environment.

Migration to Typescript

The jupyter-js-widgets JavaScript package, which is the front-end component of ipywidgets, has been completely refactored and migrated to the Typescript programming language. We also adopted the PhosphorJS JavaScript framework, which is at the foundation of the JupyterLab project, for better layout capability and integration with JupyterLab.

Credits

This release has been a team effort of a large number of contributors. We would like to thank the following 31 people who contributed, and especially the 14 people who contributed for the first time in this release.

Contributors to this release (alphabetical order):

Afshin Darian (first contribution)
Adam Chainz (first contribution)
Benjamin Ragan-Kelley
Brian Granger
Cameron Oelsen (first contribution)
Carol Willing
Dave Willmer
denfromufa
Giles Weaver (first contribution)
Gino Bustelo
Grant Nestor (first contribution)
Jason Grout
Javier Pedemonte
Jeff (first contribution)
Jeroen Demeyer
Jonathan Frederic
Justin McCandless (first contribution)
Ludwig Schmidt-Hackenberg (first contribution)
Maarten Breddels (first contribution)
Martin Renou (first contribution)
Matthew Craig
Matthias Bussonnier
Michael Pacer (first contribution)
Oliver Evans (first contribution)
Paul Ivanov
Philipp Rudiger (first contribution)
Srinivas Kumar Sunkara (first contribution)
Steven Silvester
stonebig (first contribution)
Thomas Kluyver
Sylvain Corlay
Yoshiki Vázquez Baeza

Jupyter Blog - Sylvain Corlay

ipydatagrid is now part of Project Jupyter

What is ipydatagrid?

Key Features

The transfer to Project Jupyter

Bloomberg’s contribution to Project Jupyter

JupyterGIS

Collaborative workflows in geosciences

From the desktop to the web

Real-world applications

Building collaborations in Europe and beyond

Affiliations

JupyterCon 2023 keynotes

Jupyter Community Workshop: Jupyter for Education

MyBinder @ OVHcloud

A new partnership with NumFOCUS

Acknowledgement

Jupyter Community Workshop: JupyterLite

Enabling the JupyterLab debugger with ipykernel

What will change with ipykernel 6.0?

Try it now!

What about the future?

Acknowledgements

About the authors

A Curiously Recurring Widget Library

Using the RAII pattern for a widget library

Static polymorphism and the CRTP pattern

CRTP in xwidgets — closing the recursion

Precompilation and binary size optimization

Xproperty: an implementation of the observer pattern

Method chaining for widget initialization

Building upon xwidgets

C++ should allow overloading the dot operator

Try it online!

About the author

Benjamin Ragan-Kelley

The templating system of nbconvert 6

From Classic to Lab

My notebooks look different!

But I wanted my notebooks to look the same!

Executing the notebook before rendering

Nbconvert templates

Selecting a template

Where are nbconvert templates installed?

The content of nbconvert templates

Inheritance in Jinja

Building a custom template

From nbconvert templates to Voilà templates

Where are Voilà templates installed?

Overriding nbconvert templates with Voilà

Future developments

Acknowledgements

About the Author

Report on the Jupyter Community Workshop on Dashboarding

Why a workshop on Jupyter Dashboarding with Jupyter?

Highlights of the week

Acknowledgments

Voilà is now a Jupyter subproject

What is Voilà?

Why moving Voilà under the Jupyter governance?

Resources

Acknowledgements

Field Report on the Kernel Community Workshop

Why a workshop on Jupyter kernels?

Technical Achievements

JupyterLab Debugger

A Calculator Jupyter Kernel based on xeus

A prototype Julia backend to Jupyter interactive Widgets.

Parameterized Kernelspecs

Forking of Jupyter Kernels

Acknowledgments

Jupyter Community Workshop: Building upon the Jupyter Kernel Protocol

Jupyter Community Workshop: Dashboarding with Project Jupyter

Release of ipywidgets 6.0

Installation

What’s new in ipywidgets 6.0?

Credits