Jupyter Blog - M Bussonnier

Release of IPython 8.0

2022-01-12T13:37:00+00:00

IPython is a powerful Python REPL that gives you tab completion, better tracebacks, multiline editing, and several useful features on top of pure Python Scripts. It is also the library that powers the Jupyter Kernel via the IPykernel.

Today I am pleased to announce the release of IPython 8.0, which has been long in the making and arrived a bit over three years after the 7.0 release.

I also suggest to read the companion blog post on Quansight-Labs site that goes into some technical details on how we removed some old code.

EDIT: You can also ask questions/comments on the jupyter discourse, or read the discussion on HN

Many new features

This major release comes with many improvements to the existing codebase and several new features. These new features are code reformatting with Black in the CLI, ghost suggestions, and better tracebacks which highlight the error node, thus making complex expressions easier to debug (see below).

Tracebacks now highlight in which AST node the error occurs. In complex code, this helps to quickly narrow down what causes an error. Here we also turn on xmode verbose to see values of local variables (xmode verbose is disabled by default for security reasons as it could leak secrets, but is highly recommended)

But really, if you want to read about all the new features in-depth, we recommend you spend some time reading the what’s new and the list of features in 7.x.

Moving forward by shedding weight

While IPython 8.0 has many new features described in the what’s new section, the majority of changes that demanded a bump in major version numbers are removals.

IPython was created more than two decades ago by Fernando Pérez while procrastinating on his graduation. Even if a few lines from this period remain, it was time to remove a large number of deprecated and unused code in the IPython code base and drop old dependencies (like nose) in favor of more recent ones (pytest).

Thanks to NumFOCUS Small Developer Grant, we hired Nikita Kniazev (@Kojoley) to help us with some of the heavy lifting. Nikita did a fantastic job updating our codebase, migrating to pytest, enhancing our coverage, and fixing numerous bugs. I highly recommend contacting them if you need help with C++ and Python projects.

Some of Nikita’s most notable contributions are:

IPython is no longer reliant on Nose, which has been unmaintained for many years.
Significantly increased IPython coverage.
Considerable refactor of areas that were calling into deprecated features.

On top of this, IPython 8.0 also comes with the following capabilities:

Remove most of the deprecated functions and parameters that were marked as such between IPython 1.0 and 5.0
Bumped the minimal required Python version to 3.8 (following NEP 29). This means that we now use the native Python top-level async instead of crazy hacks. And we can start to use def fun(a,/,b=None): syntax for positional only arguments.

Nikita also went well beyond the scope of work we gave them by suggesting more robust and often simpler code patterns.

In total, this allowed us, despite all the added features, added type annotations and added tests, to decrease the size of the codebase from 37 500 LOC across 348 files to 36 100 across 294 files. We hope that this reduction in the codebase size combined with work on speeding the startup of the CLI will make IPython easier to use and contribute to.

Some advice on how to do that same are on the companion blog post and the Quansight-Labs site.

Moving toward pyproject.toml

Another place in IPython that has seen many changes in the build process. We now use pyproject.tomlto have a declarative build. We still require setuptools and have setup.py files right now, but we are working towards removing/simplifying them soon.

IPython’s wheels and sdist can now be built with python -m build instead of invoking setup.py directly.

Our process still needs manual setup of SOURCE_DATE_EPOCH and repacking of the sdist wheel to obtain reproducible builds. We encourage you to get the git repository, build it yourself, and get a byte-for-byte identical artifacts.

Monthly Release and following NEP 29

We strongly believe that predictability is key to adoption and trust in IPython. In particular, in business settings, it is critical to plan ahead while rapidly receiving bug fixes. This is why for IPython 8.0 we’ll continue to do minor monthly releases on the last Friday of each month as long at it’s reasonable and does not impact maintainers life too much.

We’ll still publish a few 7.x releases with critical bug fixes, but starting last Friday of this month, you will start to see stable releases of 8.x.
All releases are announced in this subcategory in discourse, to which you can get subscribed to get notifications.

We also now follow NEP 29, meaning that IPython 8.0 is not only compatible with NumPy 1.19+ and Python 3.8+. But our future Python support schedule is predictably aligned with all other libraries following NEP 29.
We believe the new NEP 29 is critical. It gives the core Scientific Python Ecosystem a clear schedule of available features and the removal of deprecated features.

Looking forward to your contributions

Don’t think IPython is too hard to contribute to or has no missing features. We try to keep a list of beginner-friendly issues, but we really want your feedback and your ideas. In the meantime, have fun with this new release that you can install with :

pip install ipython ipykernel --upgrade

conda update ipython ipykernel

(updating ipykernel may not be necessary for you, but we recommend it).

Thanks

Thanks to Tania Allard and Eric Charles for review , Quansight Labs and Quansight (my employer) for all the open-source work I can do. Of course, thanks to all the contributors to the IPython and Jupyter community, and in particular MrMino who recently joined as an IPython core dev…

Jupyter’s role in #ChaosDB

2021-09-01T21:43:00+00:00

On August 26 it was revealed that a misconfiguration in Microsoft’s internal deployment of CosmosDB using Jupyter would allow attackers to access all customer data. Fortunately, they report no evidence that customer data was compromised.

Nonetheless, articles onlines, like Ars Technica’s and Reuter’s have strong headlines associated Jupyter, for example “Worst cloud vulnerability you can imagine.”

This can be especially alarming for our community as no details on the vulnerability have been released yet, and members of our community wonder about Jupyter’s possible role in this vulnerability.

What the Jupyter team knows

We learned about the CosmoDB vulnerability at the same time as everyone else; we had no prior notice, and received no privileged communication about this issue. We have not seen any evidence suggesting this relates to a vulnerability in Jupyter itself, as opposed to a misconfiguration of Microsoft’s internal services.

We also had no prior interaction with the Microsoft team about their internal Jupyter deployment in CosmoDB.

From the descriptions posted by Wiz and Microsoft, there is no suggestion of any vulnerability in Jupyter itself, and rather expect that Jupyter was used as convenient shell to exploit a vulnerability in the configuration of Microsoft’s internal services, but we have no information beyond what is publicly available to support that claim.

What are we doing internally

Even if Jupyter does not have a vulnerability to fix, it is often possible for us to warn end users when risky configurations options are set. For example, if you try to login to JupyterHub over a non https connections, you will a see a warning.

JupyterHub warning about login over unsecured HTTP

We are preparing for the full information release of #ChaosDB details, to see if there are any relevant safeguards and warnings to implement on the Jupyter side. We are also trying to reach the involved Microsoft Security Team personally to know whether there are steps we can take before public disclosure.

In the meantime you can contribute and get involved:

The security working group meets every other Friday. You can follow and open issues the repository for public questions and meeting minutes.
You can send your questions and concerns about security to security@ipython.org to reach out to our security team.

We always welcome feedback, questions, and help regarding security in Jupyter.

CVE-2021–32797 and CVE-2021–32798 Remote Code execution in JupyterLab and Jupyter Notebook

2021-08-09T20:05:00+00:00

TL:DR; All recent JupyterLab and Notebook versions are susceptible to a attack where a maliciously crafted notebook can trigger arbitrary code execution when a user views these malicious files.

We strongly advise all users to deploy the new version of JupyterLab and Jupyter Notebook.

Jupyter Notebook 6.4.1 or above, 5.7.11 or above.

Jupyter Lab 3.1.4 or above, 3.0.17 or above, 2.3.2 or above, 2.2.10 or above , 1.2.21 or above

This blog post will be updated with links to the various patches, exploit and disclosure later once the final links are available.

Jupyter Security Model

Jupyter Notebook act as a REPL (Read Eval Print Loop) in a browser, our main goal is to expose as many functionalities to our users, with the least restrictions. We also want users to be able to share their results with other, and let everyone be capable of reproducing the result.

When receiving an untrusted notebook from a potentially malicious source we still want users to be able to inspect a notebook without risks, our approach is that an untrusted notebook has restrictive capabilities until all cells have been manually inspected and explicitly run by a user, or the notebook is explicitly marked as trusted. If one finds a way to bypass this trust mechanism, a notebook might be able to execute code in the browser at at a time where a user is not expecting execution to occur.

This is what happen in these particular CVEs, where some content of a notebook were improperly handled.

[This part of the blog post will be updated with links to actual reports, and proof of concept once the patched version have reached enough package repositories]

CVE Timeline

We want to thanks Guillaume Jeanne (Google), and Timo Schmid (Google) for the vulnerability report and helping us through the fixing process.

Thursday, 15 Jul 2021 07:11:13 -0700 (PDT): Initial question about security issue disclosure on security@ipython.org (no specific on the vulnerability)
Tuesday, 20 Jul 2021 05:52:09 -0700 (PDT): Actual Vulnerability Report.
Tuesday, 20 Jul 2021 : Open relevant GitHub security advisory on Notebook and JupyterLab Repositories
Thursday, 5 August 2021 : First releases with patched version on PyPI.
Monday, 9 August : publication of this blog post and publish security advisory on GitHub.
[Further item may be added to list publication by downstream repositories, like conda, conda-forge, debian…, contact us to add an item]

What we learned

Dealing with Security vulnerability in Jupyter and more generally in Open-Source is far from easy. We initially thought about treating the Notebook and Lab CVE separately but they are/were too similar and the cross discussion too revealing to treat both independently.

As Jupyter is mostly a volunteer based organisation, there is often no contributor responsible to replying to security issues, and when someone steps up they might not be a expert, nor available all the time. I want to give special thanks to Steve Silvester (Apple) , Afshin Darian (Two Sigma), and Zach Sailer (Apple), and Matthias Bussonnier (Quansight) for writing the fixes, planning through the release and notifying stakeholders.

Advance notice to stakeholder is complicated, we know of a few large deployment and have personal connections to a couple organisation, and were able to reached out to let them know critical release would be published. There is an inherent tension between publicly warning our user base that security release would be published, which might push malicious actor to closely survey the codebase changes and re-derive the attack vectors, and publishing the release first, with the details later. Especially since we are trying to be transparent in our communication, security fixes are the opposite of our normal communication workflow.

Our communication process is imperfect. We have 2 mailing list for security-related discussion. The first one – security@ipython.org – has only a couple of members all core contributors and can receive email from the outside, it is used for triage. It receive a high number of spam as this is public email. As it has only a few members and we are all busy, mail can slip through. The second mailing list is slightly larger, and used for internal announcement for stakeholder. It has a fairly open membership model, (ask a Jupyter developer if you can be on it, and the reason why and we’ll likely add you), though it’s content seem to be ignored (it even lands on my spam folder, not sure why).

What we’ll do better

In order to attempt to better react and be proactive with respect to security we are attempting to form and new security-focused subproject/workgroup to educate and have procedure for everything security related. We welcome your involvement and feedback in how to improve Jupyter security and how to better involve the community.

IPython 7.0, Async REPL

2018-09-27T17:41:00+00:00

Today we are pleased to announce the release of IPython 7.0, the powerful Python interactive shell that goes above and beyond the default Python REPL with advanced tab completion, syntactic coloration, and more. It’s the jupyter kernel for python used by millions of users, hopefully including you. This is the second major release of IPython since we stopped support for Python 2.

Not having to support Python 2 allowed us to make full use of new Python 3 features and bring never before seen capability in a Python Console. We are still encouraging library authors and users to look at the Python 3 Statement to learn about the end of life of Python 2 and how to stop support for Python 2 without breaking installation for Python 2 end users.

As developers and maintainers of IPython, it was a large gain of time to be able to only develop for a single version of python. Avoiding the use of conditional imports, being able to rely on type annotations, and make use of the newly available Python APIs were some of the advantages that made us more productive. Especially as most of the work on IPython is done by volunteers who work on nights and weekends, with only a couple of minutes here and there, this often made the difference between a patch reaching completion, or the contributor moving on to other pastures.

One of the core features we focused on for this release is the ability to (ab)use the async and await syntax available in Python 3.5+. There are of course many other improvements in this release you can read about in the what’s new.

Demo of awaiting coroutine in IPython 7.0

TL;DR: You can now use async/await at the top level in the IPython terminal and in the notebook, it should — in most of the cases — “just work”. Update IPython to version 7+, IPykernel to version 5+, and you’re off to the races.

See how to install IPython by reading the “what’s new”.

The recipes are currently building on conda-forge and should be available soon. For the time being you can install it via pip:

$ pip install ipython ipykernel --upgrade

A Primer on concurrency

You may have heard about async/await, threads, concurrency, preemptive scheduling and cooperative scheduling without really understanding what all this is about. If you are not familiar will all the above terms, all the hype may be confusing so let’s talk about concurrency in a really high level way.

Typically when your computer needs to execute many tasks, it will switch between them really fast, so from the human point of view it looks like everything is being processed at the same time. There are two main ways of doing so under the hood: Preemptive Scheduling, and Cooperative Scheduling.

With preemptive scheduling changing tasks can happen at any time. For example, while writing this blog post, I could stop in the middle of a word to start writing an email, which will itself be interrupted to check Gitter/Slack, before coming back, writing 5 words and stopping to get dinner.

TL;DR: Concurrency (from Geek And Poke, 2009)

With cooperative scheduling, the task switches can happen only at agreed spots. The term co-operative comes from the fact that tasks need to co-operate for the whole process to function. If a task decides to never take a break to let you to do something else, the illusion of many tasks being completed at once disappears.

Each approach has its own advantages and drawbacks, and we will not focus on these. Let’s just say that with co-operative scheduling async/await let you mark the areas where interruption is allowed to occur.

Moreover, async/await syntax allows cooperative scheduling in Python in a way that lets you write code that looks synchronous (without task switches), while actually being able to be interrupted, from the point of view of the computer. It also keeps the programmers from having to worry about global state changing under their feet, as this can occur only at the proximity of await keywords.

When going to a restaurant, social conventions (and common sense) tell us when and how these interactions can or cannot be interrupted, but programming languages need markers when using cooperative scheduling. These are asyncandawaitkeywords in Python.Asyncmarks a function that *may* be interrupted,awaitis required to call async-functions (akacoroutine`) and marks a point were task can be switched.

If you want to learn more we strongly recommend reading the Trio Tutorial Primer on async programming.

Async in the Python world

In the current Python ecosystem, packages tend to standardize around AsyncIO, provided in the Python standard library. AsyncIO can sometimes be judged as complex even by well known developers; this is in part due to the necessity of supporting other older asynchronous projects like twisted or tornado, but it’s also what makes a lots of its power: One event loop to rule them all.

Running a single async task requires you to learn about AsyncIO, write a non negligible amount of boilerplate code in order to fetch a single result. This can be especially cumbersome when doing interactive exploration, and likely will keep users from experimenting with AsyncIO code.

How to run a single async task in Python repl without async integration.

As Raymond Hettinger would says (slamming hand on podium): “There must be a better way”.

IPython AsyncIO Integration

Thanks to a multiple month effort (actually this work started close to 2 years ago), and the work of many talented people, you can now directly await code in the REPL and IPython will do “the right thing”.

Awaiting AsyncIO code should now automagically work.

With the new integration, you don’t have to import or learn about asyncio, deal with the loop yourself, or wrap your task in its own function. You are now able to just focus on the business logic and move along.

The only thing you need to remember is: If it is an async function you need to await it.

We hope that this will free users to experiment and play with asynchronous programming. Of course this will not magically make your code faster, or run in parallel, simply easier to write and reason about.

Other Async Libraries (aka: curio and trio integration)

The addition of async and await keyword in Python did not only simplify the use of asynchronous programing and the standardization around asyncio; it also allowed experimentation with new paradigms for asynchronous libraries. David Beazley created Curio, and Nathaniel Smith Trio, which both explore new ways to write asynchronous programs and explore how async, await and coroutines could be used when starting from a blank slate. The Trio documentation introduction and which problems it attempt to solve [1, 2, 3, 4, 5, 6] are highly recommended reading with varying level of technicality.

Interactive uses of libraries is key to getting insight and intuition on how a system works, intuition is critical to rapid prototyping, development and creation of higher levels of abstraction. It was natural for us to build support for Curio, Trio, (and potentially new other async libraries) into IPython.

You can setup IPython to run async code via Curio, or Trio and experiment or write production code using these libraries. To do so use the %autoawait magic, and tell it which library to use.

Defining an asynchronous function and spawning multiple concurrent task in IPython using Trio.

As you can see code looks really natural, and it is easy to forget that the above snippet is usually a syntax error in Python or older version of IPython. The astute reader and IPython expert will have suggested to use the %%time cell magic instead of doing it manually, though a couple of magics still need updates to properly handle async code. We look forward to your contribution on this front, and are excited to see what you can come up with.

Async in Notebooks (and other Jupyter Clients)

If you are a Jupyter user, you most likely use a Notebook interface, and interact with IPython via the ipykernel package.

Using AsyncIO in nteract desktop works out of the box with newer IPython and IPykernel

We’ve been working hard on making async code work in a notebook when using ipykernel. While most of the heavy lifting was done in IPython, the work in IPykernel was non-negligible, and required the accommodation of a number of use cases, which are not working. You now have to update both IPython to 7.0+ and ipykernel to version 5.0+ for async to be available. If you are using pip: $ pip install IPython ipykernel --update. As for conda, the packages should be available on conda-forge soon. With these new releases, async will work with all the frontends that support the Jupyter Protocol, including the classic Notebook, JupyterLab, Hydrogen, nteract desktop, and nteract web. The default code will run in the existing asyncio/tornado loop that runs the kernel. Integration with Trio and Curio is still available, but tasks will not be interleaved with the asyncio one — at least not yet. We welcome work on this front.

Submitting background tasks still requires you to access the asyncio event loop, and we are still be looking for contributions on this front as well, to make it even easier to run async code.

There are still some question on how to handle nested asyncio eventloop. It is indeed usually impossible to run nested eventloop, in the case of asyncio, trying to do so raises a RuntimeError the kernel already ran in and asyncio eventloop, calling directly or indirectly loop.run_until_complete and alike is not possible. There are discussions to use libraries like nest_asyncio as pointed out on this comment, but until those are more battle tested we do not want to commit a default solution in the core of IPython and let the ecosystem develop.

Future improvements

As far as we know, this is the first Async-aware Python REPL, and libraries like Trio/Curio are still young, thus there are still a number of use-cases we have not yet even thought about! We are encouraging you to come forward to talk about your use cases, what you tried and what did not work. There is also a number of new features to implement (making magics work with async, tab completion, background tasks) on which we would welcome new contributors.

I Python, You R, We Julia

2018-05-29T16:11:00+00:00

When we decided to rename part of the IPython project to Jupyter in 2014, we had many good reasons. Our goal was to make (Data)Science and Education better, by providing Free and Open-Source tools that can be used by everyone. The name “Jupyter” is a strong reference to Galileo, who detailed his discovery of the Moons of Jupiter in his astronomical notebooks. The name is also a play on the languages Julia, Python, and R, which are pillars of the modern scientific world. While we ❤️🐍(Love Python), and use it for much of the architecture in Jupyter, we believe that all open-source languages have an important role in scientific and data analysis workflows. We have strived to make Jupyter a platform that treats all open-source languages as first-class citizens.

You may know that Jupyter has several dozen kernels in as many languages, and that you can choose any of them to power the code execution in a single notebook. However, the possibilities for cross-language integration go way beyond this, which I’ll attempt to demonstrate here.

What I’ll describe below has been possible for now several years – from even before the name Jupyter was first mentioned. It relies on the work of many Open Source libraries, too many to cite all the authors. It is not the only solution — neither the first, not the last. RStudio recently blogged about reticulate, which allows you to intertwine Python and R code. BeakerX is also another solution that appears to to support many languages.

We hope that showing how multiple languages can be use together will help make you more efficient in your work, and that it promotes cooperation across our communities to use the strengths of each language. This article only scratches the surface, you can read more in depth what you can do and how this works in a notebook I wrote some time ago.

Follow along on Binder

We created Jupyter and Binder to make science more trustworthy and allow results to be replicate. If you doubt what I have written below, or just want to follow along feel free to try on your own using Binder — the docker image is quite big so can take a while to launch. In the linked notebook we show a couple of extra languages.

The Tail of Fibonacci

A famous example of recursion in Computer Science is the Fibonacci series, its ubiquity allows the reader not to focus on the sequence itself but on the environment around it. As a reminder, the Fib sequence is defined with its first two terms being one, then each subsequent term as the sum of the two preceding terms; i.e F(1)= 1, F(2)=1, F(n) = F(n-1)+F(n-2)

We can calculate the first few terms: 1, 1, 2, 3, 5, 8 … note that F(5) is a fixed point F(5) = 5, and trust that asymptotically the sequence behaves like exp(n).

Let’s see how one can use many languages to play with fibonacci.

I, Python

For this exploration we’ll start with Python. It is my language of choice, the one I’m the most familiar with:

def fib(n):
    """
    A simple definition of fibonacci manually unrolled
    """
    if n<2:
        return 1
    x,y = 1,1
    for i in range(n-2):
        x,y = y,x+y
    return y

We can check that the fib function works correctly.

>>> [fib(i) for i in range(1,10)]
[1, 1, 2, 3, 5, 8, 13, 21, 34]

And plot it:

%matplotlib inline
import numpy as np
X = np.arange(1,30)
Y = np.array([fib(x) for x in X])
import matplotlib.pyplot as plt
fig, ax = plt.subplots()
ax.scatter(X, Y)
ax.set(xlabel='n', ylabel='fib(n)',
       title='The Fibonacci sequence grows fast !')

As you can see it grows quite quickly, actually it’s exponential. Now let’s see how we can check this exponential behavior using multi-language integration.

You R

With the fantastic RPy2 package, we can integrate code seamlessly between Python and R, allowing you to send data back and forth between the two languages. RPy2 will translate R data structures to Python and NumPy, and vice versa.

In addition, RPy2 has extra integration with IPython and provides “Magics” to write inline or multiline R code. Loading the RPy2 extension exposes the %R , %%R, %Rpush and %%Rpull commands for writing R.

%load_ext rpy2.ipython

We can use %RPush to send data to a stateful R process.

%Rpush Y X

and use %%R in order to instruct the R process to run an R cell.

%%R
my_summary = summary(lm(log(Y)~X))
val <- my_summary$coefficients

plot(X, log(Y))
abline(my_summary)

Here we make a linear regression model on log(Y) vs X. As Y is (hopefully) exponential, we should get a nice line. RPy2 provides rich display integration which will nicely display outputs and plots inline in a notebook:

We can of course ask for the linear regression summary:

%%R
my_summary

Which outputs:

Call:
lm(formula = log(Y) ~ X)

Residuals:
      Min        1Q    Median        3Q       Max 
-0.183663 -0.013497 -0.004137  0.006046  0.296094 

Coefficients:
             Estimate Std. Error t value Pr(>|t|)    
(Intercept) -0.775851   0.026173  -29.64   <2e-16 ***
X            0.479757   0.001524  314.84   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.06866 on 27 degrees of freedom
Multiple R-squared:  0.9997,	Adjusted R-squared:  0.9997 
F-statistic: 9.912e+04 on 1 and 27 DF,  p-value: < 2.2e-16

We can also lift the results from R to Python using %Rget:

coefs = %Rget val
y0,k = coefs.T[0:2]
y0,k

Which yields

(-0.77585097534858738, 0.4797570904348315)

Here we saw that RPy2 allows us to pass data back and forth between Python and R; This is incredibly useful to leverage the strengths of each language. This is a toy example, but you could imagine using various python libraries to get data from servers, and move to R for the statistical analysis.

However, sometime moving data between languages may be too limiting. Let’s see how we can leverage the same mechanism to gain some performance, by integrating with a lower level language.

Let’s C

Python and R are not the most performant languages for pure numerical speed. When performance improvement is necessary, developers tend to utilize compiled language like C/C++/Fortran.

Unfortunately, compiled languages generally have a poor interactive experience, and where CPU cycles are gained, human developer time may be lost.

Using magics, we can, as we did for R, include snippets of C, Cython, Fortran, Rust … and many other languages.

import cffi_magic
%%cffi int cfib(int);

int cfib(int n)
{
    int res=0;
    if (n <= 2){  
        res = 1;
    } else {
        res = cfib(n-1)+cfib(n-2);
    }
    return res;
}

We can interactively redefine this function, and it will magically appear on the Python namespace. It works identically to the fib we defined earlier, but is much faster. Note that the Python Fib, and C fib time here are difficult to compare as the C one is recursive (behave in O(exp(n)))and the Python one is hand unrolled, so behave in O(n) .

More technical details can be found in a notebook I wrote earlier, but the same can be done with other languages that call one another, and lines like the following work perfectly:

assert py_fib(cython_fib(c_fib(fortran_fib(rust_fib(5)))) == 5

Julia to bind them all

The last example is a technical marvel that was first developed by Steven Johnson and Fernando Pérez, it relies on starting a Julia and Python interpreter together, allowing them to share memory. This allow both languages not only to exchange data and functions, but to manipulate live object references from the other interpreter. Extra integration with IPython via magics allows us to run inline Julia in Python (%julia, %%julia), while Julia Macros (@pyimport) allows python code to be run from within Julia.

Below we’ll show integration with Graphing libraries (matplotlib), so let’s set up our environment.

%matplotlib inline
%load_ext julia.magic

We’ll start from within Julia, and import a few python packages:

%julia @pyimport matplotlib.pyplot as plt
%julia @pyimport numpy as np

We now have access – from within Julia – to matplotlib and numpy. We can now seamlessly integrate Julia native numerical capabilities and functions with our Python kernel.

%%julia                                        
t = linspace(0, 2*pi,1000);             
s = sin(3*t + 4*np.cos(2*t));           
fig = plt.gcf()                         
plt.plot(t, s, color="red", linewidth=2.0, linestyle="--", label="sin(3t+4.cos(2t))")

Note that in above block, t, pi are native julia; s is computed via sin (julia), t (julia), cos (numpy); fig is a Python object. As the Julia Magic provides IPython display integration, the code above displays this nice graph.

We now want to annotate this graph from Python, as the API is more convenient:

import numpy as np
fig = %julia fig
fig.axes[0].plot(X[:6], np.log(Y[:6]), '--', label='fib')
fig.axes[0].set_title('A weird Julia function and Fib')
fig.axes[0].legend()
fig

After passing a reference to fig from Julia to Python, we can annotate it (and plot one of the fib functions we defined earlier in C, Fortran, Rust, etc…)

Here we can see that unlike BeakerX, R-Reticular or RPy2, we are actually sharing live objects, and can manipulate them from both languages. But let’s push things a bit further.

The fib function can be defined recursively; let’s have some fun and define a pyfib function in Python that recurses via the a jlfib function in Julia. Meanwhile, the jlfib function in Julia recurses using the python function. We’ll print (J , or (P when switching language:

jlfib = %julia _fib(n, pyfib) = n <= 2 ? 1 : pyfib(n-1, _fib) + pyfib(n-2, _fib)

def pyfib(n, _fib):
    print('(P', end='')
    if n <= 2:
         r = 1
    else:
        print('(J', end='')
        # here we tell julia (_fib) to recurse using Python
        r =  _fib(n-1, pyfib) + _fib(n-2, pyfib)
        print(')',end='')
    print(')',end='')
    return r

fibonacci = lambda x: pyfib(x, jlfib)
fibonacci(10)

We can now transparently call the function :

(P(J(P(J(P(J(P(J(P)(P)))(P(J))(P(J))(P)))(P(J(P(J))(P)(P)(P)))(P(J(P(J))(P)(P)(P)))(P(J(P)(P)))))(P(J(P(J(P(J))(P)(P)(P)))(P(J(P)(P)))(P(J(P)(P)))(P(J))))(P(J(P(J(P(J))(P)(P)(P)))(P(J(P)(P)))(P(J(P)(P)))(P(J))))(P(J(P(J(P)(P)))(P(J))(P(J))(P)))))
55

If you are interested in diving more into details see this post from a couple of years ago with all the actual code.

I hope that this post has convinced you that Jupyter – via the IPython kernel – has deep cross-language integration (and has had this for many years). I also hope it lifted the misconception in that in Jupyter “1 kernel == 1 language” or even that “1 notebook == 1 language”. Each of the approaches shown here (as well as Reticulate, BeakerX, etc) have their pros and cons. Use the approach that fits your needs and makes your workflow efficient, regardless of the tool, language, or libraries you use.

JupyterCon 2018: Registration Open

2018-04-11T15:01:00+00:00

Dear Jupyter Community,

For the past six months, the Project Jupyter team in collaboration with O’Reilly Media and NumFOCUS have been planning JupyterCon 2018. In January, we opened the Call For Proposal, during which we received numerous high-quality proposals. The total submissions exceed our expectations: more than 3 times the number of available slots! With the help of the Program Committee Reviewers and co-chairs Fernando Pérez, Brian Granger and Paco Nathan, we had the hard job of selecting among the fantastic submissions we received. Today we are happy to announce that most of the JupyterCon 2018 Program is ready and registration is open! We are exited to bring you sessions about Scaling JupyterHub, leveraging GPUs for Jupyter, Running C++, in Jupyter, and many more.

Fernando Pérez and Andrew Odewahn during JupyterCon 2017 Opening Keynote

As Paco Nathan previously announced, this year will have a dedicated Education Track, and Business Summit to supplement the Main Tracks, Trainings, and Tutorials that we already brought to you last year. Many of the highlights of last year such as “Meet the Experts” office hours, a Poster Session for extended discussions with presenters, and the Vendor Expo Hall will return this year.

As with last year, JupyterCon will be held at the New York Hilton Midtown, NYC, August 21-24 and Saturday 25th. You can register today for the main conference. Early Bird pricing ends on May 18th. You can also use the discount code PJ20. We also have a limited amount of financial support for JupyterCon for attendees thanks to our partners. Last year, we provided scholarships to 13 students from diverse backgrounds to attend JupyterCon 2017.

Community Sprint Day, August 25th.

Thanks to Bloomberg, Saturday, August 25th will be reserved for a separate Community Sprint day, free of charge. All community members, whether or not you plan to attend the main conference, are invited. This day will be focused on community, contributing to Jupyter, and Open Source in a “Open Studio” form. Whether you are new to Jupyter or a power user, we invite you to come and mingle with the rest of the attendees to lean about any Jupyter-related project.

Several activities will be available. Whether you have coding, design, or writing skills, we encourage you to contribute, pitch your ideas, and get started on something brand new.

Are you new to open source, Git, and GitHub? We’ll offer a introduction to Open-Source 101 and how to get development versions on your machine.

You’ve never tried Jupyter or you are an advanced user of Jupyter with specific needs? We’ll be hosting a JupyterLab user-testing session where you will have the chance to try upcoming features and give us critical insight on how to improve usability.

Interested in contributing back to Jupyter or related projects? Many experts will be here to help you move your project forward. If you are coming to JupyterCon and would like to help with Community Sprint Day, or have a project you’d like attendees to work on let us know !

The Community Sprint Day is free to attend, but registration is required.

Jupyter Pop-Up, DC, May 15th

You can’t wait to attend JupyterCon 2018 ? You can attend Jupyter Pop-Up, DC, May 15th for a Day long event, and a taste of what is to come !

Do not forget to follow this blog, @projectJupyter or @jupytercon, for further updates on JupyterCon.

Thanks to The O’Reilly Media Team; The Conference Chairs (Brian Granger, Paco Nathan and Fernando Pérez); The Program Committee (Dan Allan, Ian Allison, Paige Bailey, Lorena Barba, Tom Caswell, Afshin Darian, John Detlefs, Chris Erdmann, Jessica Forde, Stuart Geiger, Tim George, Michelle Gill, Tim Head, Jennifer Klay, Cierra Martinez, Emiliy Jane McTavish, Omoju Miller, M Pacer, Peter Parente, Eszti Schoell, Steve Silvester, Robert Talbert, Dwight Townsend, Wolf Vollprecht, Jamie Whitacre, Kevin Zielnicki) and all the people making JupyterCon 2018 possible, and Jupyter a reality.

Jupyter Pop-Up, March 21, Boston

2018-02-09T22:20:00+00:00

A new series of local Jupyter events, starting in Boston.

Jupyter Pop-Up, brought to you by NumFOCUS and O’Reilly Media, March 21, Boston, MA.

Many of you are looking forward to JupyterCon 2018 and have submitted a talk. Alongside these large, multi-day events, we are seeing demand for smaller, local events as well. Since 2015 we have co-organized several Jupyter Days events (Paris, Hawaii, Atlanta, Boston, Philadelphia, NYC) with local community organizers. In 2018, we are preparing Jupyter Day Atlanta (March 31st), and hoping to offer other community organized events as well.

In addition to Jupyter Days, we are pleased to announce the first Jupyter Pop-Up, which is brought to you by the NumFOCUS Foundation and O’Reilly Media.

This single-day conference will take place in the District Hall of Boston, MA on March 21, Early Bird ticket prices end on February 23rd, and the Call For Proposals is still open until the end of the week (Sunday, Feb. 11th).

You can learn more on the Jupyter Pop-Up conference website.

Looking forward to seeing you there !

JupyterCon 2018: Call For Proposal

2018-01-18T22:53:00+00:00

Dear fellow Jovyans,

It is with great pleasure that we are opening the Call For Proposals (CFP) for JupyterCon 2018!

Last August, Project Jupyter, the NumFOCUS Foundation, and O’Reilly Media came together to host our first JupyterCon. We attracted over 700 attendees and 23 scholarship recipients for 4 days of talks and tutorials. There were 5 parallel session tracks featuring 55 talks, 11 keynotes, 55 talks, 8 tutorials, and 2 training courses. In addition, the conference poster session featured 33 posters and fostered great discussions within the community. Our Community Day, held at at the end of the conference featured free registration was open to the general public. Videos of the event have been made available on Safari Online and YouTube.

JupyterCon 2018, CFP Open

JupyterCon 2017 was a huge success and we’ve been working hard since then to make JupyterCon 2018 even better. It will be held in New York City in August from Tuesday the 21st to Friday the 24th. We’ll also host an open Community Day on August 25th, which will be open to everyone.

Today we are happy to open the conference website and open the Call For Proposal with submissions due by early March. A couple of changes have been made to the CFP since last year. In particular if your talk is not accepted, you can ask us to automatically consider the proposal for the poster session.

We encourage you to submit a proposal, and reach out to us if you have any questions. We’ll do our best to help you and and give you feedback on your proposal.

Like last year, we will have diversity and student scholarships available; further information will be provided on the website. We also encourage you to follow the JupyterCon Twitter account for announcements or corrections.

Community Day

The final day of JupyterCon 2017 was a blast with a large number of people making their first contribution to the Jupyter codebase, to the documentation, editing the wiki, or deploying it in the cloud. During the conference days, a separate room was also reserved for user testing of different Jupyter software, which proved to be fantastic source of feedback for User Experience (UX) and driving various Jupyter Tools forward.

We are happy to offer this “Community Day” experience again. At JupyterCon 2017, the Saturday was branded “Sprints” with the connotation of a code-centric experience. While we’re happy to see users coming to “Sprint” on code, we want to let you know that the Community Day will be open to anyone. Whether you are a teacher, coder, researcher, or user of Jupyter, the Community Day will have something for you. The Community Day is not limited to attendees of the main JupyterCon event, and it’s intended to be a “grass-roots” celebration of Jupyter and its community. We hope to see you at JupyterCon 2018.

Thanks

JupyterCon 2018 would not be possible without O’Reilly Media, NumFocus, as well as our sponsors.

Incident Report: Jupyter services down

2017-12-14T19:52:00+00:00

update: December 14, 20:45 UTC, all services should be restored and back up.

On December 13, at 22:10 UTC (4:10pm EST), a large number of Jupyter-provided services stopped responding. This included, but was not limited to https://nbviewer.jupyter.org, https://try.jupyter.org (powered by tmpnb) and https://cdn.jupyter.org. We quickly narrowed this down to an issue with our hosting provider and have been working with them to resolve the issue as fast as possible.

When outages happen, the Jupyter Status page should show which services are affected and we publish updates there.

How are Jupyter services hosted?

To understand the cause of the outage, we need to understand how the Jupyter services are hosted and maintained. As Jupyter is an open organization which is mostly maintained by volunteers, we do not have a dev-ops team assigned to maintaining our infrastructure. Even with full-time developers hired through universities or companies, the time spent fixing infrastructure is taken on nights and weekends. These developers are often stretched thin and cannot be available 24/7.

Most of our cloud infrastructure is donated to us by companies like CloudFlare, Rackspace, Fastly, Google, and Microsoft. Donating resources can be challenging, both technically and legally. In this particular case, Rackspace graciously created a special account for Jupyter that handles invoices on our behalf, thereby making resources free to the project. Following a hiccup, this Jupyter account was suspended and all services are unavailable as a result.

Temporary resolution

As nbviewer is one of the most used services provided by Jupyter, we’ve moved it to one of our personal account at another cloud-provider. Fastly was set up to load-balance on the yet-to-come-back-up instances as well as this newly created instance, so all should be fine now.

The other services (tmpnb, mails@jupyter.org, cdn.jupyter.org, …) will still unavailable or highly degraded until a permanent solution is found, or the services are restarted. try.jupyter.org will likely redirect to a repo on https://mybinder.org in the meantime so people can still try out Jupyter.

Low bus factor

The outage of all these services lasted for a significant time (more than 18 hours). Which perturbed many of you relying on these services. We understand that this is hardly acceptable and we hope you’ll indulge us as these services are provided for free and without ads. One of the factors leading to the slow reestablishment of service was a relatively low bus factor, with only one and a half of our developers knowing how to deploy and maintain these services. Documentation and access to credentials was also limited.

This is one of the challenges in a distributed team like Jupyter where contributors self-organize. It is easy to forget that new code is not the only way to contribute and that infrastructure and maintenance are crucial.

We also overly rely on a single vendor (in this case Rackspace), and while we are happy with Rackspace and have no reason to move to another provider, we should have a plan to restore critical services even temporarily in case of failure.

A couple of months ago, the subject was brought to our attention, and we developed a plan to move many of our deployment to Kubernetes (which is provider agnostic). We underestimated the probability to need an emergency plan this early.

How can you help

Jupyter is mainly governed by the community all around the world. Contributing is not limited to writing code! We need members with knowledge in multiple languages, in design, dev-ops, etc. Whether you are an expert, or still learning, we would like you to get involved.

Thanks everyone for your patience and the kind words when you reached to us when discovering the services were down.

Enjoy JupyterCon 2017 on YouTube

2017-11-24T10:09:00+00:00

About a year ago the Jupyter team started a partnership with O’Reilly Media and NumFOCUS to organize JupyterCon 2017, the first in a series of Jupyter-related conferences. JupyterCon 2017 drew over 700 attendees and offered a fantastic set of tutorials, talks and keynotes from all corners of the Jupyter ecosystem. We could not have offered you such a beautiful conference without the O’Reilly Media and NumFOCUS partnership and teams.

Did you miss JupyterCon 2017?

We have a strong commitment to open-source and we believe that everyone should have access to high-quality tools and content to push science, education and technology forward. O’Reilly Media is one of the top publishers in this space, and they already provide a variety of material and training related to Jupyter and the open-source data ecosystem of which it is a part (see O’Reilly’s Jupyter related content here). It is no surprise then that the full content of JupyterCon 2017, including keynotes, tutorials, talks and exclusive interviews, is available as videos on Safari Online.

We are also pleased to announce that videos of all keynotes and talks (69 videos in total) are now available on YouTube on the O’Reilly Media channel. If you missed JupyterCon, or want to re-watch a talk, you can now add these to your watch-list! As an example here is the opening keynote given by Fernando Pérez:

See you next August for JupyterCon 2018

We are thrilled to be able to provide you with all this content from JupyterCon 2017 to enjoy on YouTube. JupyterCon 2018 will be in NYC next August 22–24. We hope to see you there next year as an attendee, speaker or sponsor. You can sign up for email updates about JupyterCon 2018 on this page.

[Update Jan 2018] Missing Talks

The Talk “Jupyter and the changing rituals around computation” is missing from the channel due to technical issue during the recording.

Release of IPython 5.5 and 6.2

2017-09-15T18:42:00+00:00

Four month after releasing IPython 6.1 and 5.4, and a couple of hours after the release of the notebook 5.1, we are happy to announce the release of IPython 6.2 (Python 3 only), and it’s cousin IPython 5.5 still compatible with Python 2.7.

You can update now by using:

pip install --upgrade ipython

If you have a recent enough version of pip you will get the latest compatible version of IPython regardless of the version of Python you are running.

The conda packages are on their way; once available you will be able to update with:

conda install ipython

New Features

As IPython 6.2 and 5.5 are minor releases you will only find a small number of new features. When API additions were done on IPython 6.2 they were backported on 5.5 to simplify the maintenance of code compatible both with Python 2.7 and 3+. You can find the full list of new features in the changelog.

As a quick teaser, IPython 6.2 can now:

Show function signature in the terminal while completing.
Assignments can trigger the display mechanism
IPdb can be called recursively
Support for system-wide configuration
Built-in support for Progress Bar.

Head there for more complete description.

Slowing down backports to 5.x

As stated on our roadmap, we’ll keep releasing a 5.x for some time, though starting at end of year. However, we will decrease our active involvement in fixing bugs affecting the 5.x branch. We will still accept PRs, and backport if you nicely ask us. Releases will happen occasionally if fixes are available, but we will be sunsetting the Python 2 support slowly.

If you are interested in further maintenance of the 5.x branch, we would love help with that work. Feel free to contact us on GitHub.

What’s next ?

We are going to start thinking about IPython 7, and start to embrace more of the Python 3 only features. Slowing down backports should allow us to be more confident that changes will not affect the automatic application of patches on old branches. Trimming down old legacy code may also help to regain some speed on interpreter startup, and should lead to plenty of opportunities for new contributors to join.

We will also try to simplify our documentation, and make often requested sections easier to find.

If you are looking for a project to contribute to – code, documentation, example, design, helping others, feel free to contact us so we can guide you through the process.

Enjoy this new release, and hope to see you around the mailing list and bug tracker!