Jupyter Blog - IPython

Release of IPython 8.0

2022-01-12T13:37:00+00:00

IPython is a powerful Python REPL that gives you tab completion, better tracebacks, multiline editing, and several useful features on top of pure Python Scripts. It is also the library that powers the Jupyter Kernel via the IPykernel.

Today I am pleased to announce the release of IPython 8.0, which has been long in the making and arrived a bit over three years after the 7.0 release.

I also suggest to read the companion blog post on Quansight-Labs site that goes into some technical details on how we removed some old code.

EDIT: You can also ask questions/comments on the jupyter discourse, or read the discussion on HN

Many new features

This major release comes with many improvements to the existing codebase and several new features. These new features are code reformatting with Black in the CLI, ghost suggestions, and better tracebacks which highlight the error node, thus making complex expressions easier to debug (see below).

Tracebacks now highlight in which AST node the error occurs. In complex code, this helps to quickly narrow down what causes an error. Here we also turn on xmode verbose to see values of local variables (xmode verbose is disabled by default for security reasons as it could leak secrets, but is highly recommended)

But really, if you want to read about all the new features in-depth, we recommend you spend some time reading the what’s new and the list of features in 7.x.

Moving forward by shedding weight

While IPython 8.0 has many new features described in the what’s new section, the majority of changes that demanded a bump in major version numbers are removals.

IPython was created more than two decades ago by Fernando Pérez while procrastinating on his graduation. Even if a few lines from this period remain, it was time to remove a large number of deprecated and unused code in the IPython code base and drop old dependencies (like nose) in favor of more recent ones (pytest).

Thanks to NumFOCUS Small Developer Grant, we hired Nikita Kniazev (@Kojoley) to help us with some of the heavy lifting. Nikita did a fantastic job updating our codebase, migrating to pytest, enhancing our coverage, and fixing numerous bugs. I highly recommend contacting them if you need help with C++ and Python projects.

Some of Nikita’s most notable contributions are:

IPython is no longer reliant on Nose, which has been unmaintained for many years.
Significantly increased IPython coverage.
Considerable refactor of areas that were calling into deprecated features.

On top of this, IPython 8.0 also comes with the following capabilities:

Remove most of the deprecated functions and parameters that were marked as such between IPython 1.0 and 5.0
Bumped the minimal required Python version to 3.8 (following NEP 29). This means that we now use the native Python top-level async instead of crazy hacks. And we can start to use def fun(a,/,b=None): syntax for positional only arguments.

Nikita also went well beyond the scope of work we gave them by suggesting more robust and often simpler code patterns.

In total, this allowed us, despite all the added features, added type annotations and added tests, to decrease the size of the codebase from 37 500 LOC across 348 files to 36 100 across 294 files. We hope that this reduction in the codebase size combined with work on speeding the startup of the CLI will make IPython easier to use and contribute to.

Some advice on how to do that same are on the companion blog post and the Quansight-Labs site.

Moving toward pyproject.toml

Another place in IPython that has seen many changes in the build process. We now use pyproject.tomlto have a declarative build. We still require setuptools and have setup.py files right now, but we are working towards removing/simplifying them soon.

IPython’s wheels and sdist can now be built with python -m build instead of invoking setup.py directly.

Our process still needs manual setup of SOURCE_DATE_EPOCH and repacking of the sdist wheel to obtain reproducible builds. We encourage you to get the git repository, build it yourself, and get a byte-for-byte identical artifacts.

Monthly Release and following NEP 29

We strongly believe that predictability is key to adoption and trust in IPython. In particular, in business settings, it is critical to plan ahead while rapidly receiving bug fixes. This is why for IPython 8.0 we’ll continue to do minor monthly releases on the last Friday of each month as long at it’s reasonable and does not impact maintainers life too much.

We’ll still publish a few 7.x releases with critical bug fixes, but starting last Friday of this month, you will start to see stable releases of 8.x.
All releases are announced in this subcategory in discourse, to which you can get subscribed to get notifications.

We also now follow NEP 29, meaning that IPython 8.0 is not only compatible with NumPy 1.19+ and Python 3.8+. But our future Python support schedule is predictably aligned with all other libraries following NEP 29.
We believe the new NEP 29 is critical. It gives the core Scientific Python Ecosystem a clear schedule of available features and the removal of deprecated features.

Looking forward to your contributions

Don’t think IPython is too hard to contribute to or has no missing features. We try to keep a list of beginner-friendly issues, but we really want your feedback and your ideas. In the meantime, have fun with this new release that you can install with :

pip install ipython ipykernel --upgrade

conda update ipython ipykernel

(updating ipykernel may not be necessary for you, but we recommend it).

Thanks

Thanks to Tania Allard and Eric Charles for review , Quansight Labs and Quansight (my employer) for all the open-source work I can do. Of course, thanks to all the contributors to the IPython and Jupyter community, and in particular MrMino who recently joined as an IPython core dev…

IPython Parallel in 2021

2021-11-30T07:55:00+00:00

This post describes work funded by Bodo, Inc.

IPython Parallel’s a bit of an odd duck in the parallel computing space. In a world with dask, ray, bodo, pyspark, and other parallel computing tools, what is IPython Parallel’s role in 2021?

Context

IPython Parallel began (in 2006!) as a natural extension of the Jupyter messaging protocol(though it predates the Jupyter name by a few years): when you have a protocol for REPL-style remote code execution, what can you do with multiple remote execution environments? This question led to the development two basic models for parallel execution in IPython Parallel:

“multiplexed execution” — where you send code explicitly to different workers, and
“load-balanced execution” — where you tell IPython Parallel what to run, and a scheduler takes care of assigning each task to an available worker.

What IPython Parallel doesn’t do

To IPython, your tasks are black boxes (either Python functions or blocks of Python code as text) and you are in complete control of where and when your tasks run, as well as any dependencies or side effects they may have.

IPython Parallel specifically doesn’t and won’t do several things that other tools might:

infer dependencies or relationships between tasks (execution graph)
manage data distribution or locality
construct parallel algorithms or represent ‘natively’ parallel distributed data structures, such as distributed arrays or data frames

Everything is very explicit and implemented at the client-level in IPython Parallel, meaning that while it doesn’t do these things, you can. It also means that many of the things that have overhead costs associated with parallel computing (communication, data movement) have particularly high costs in IPython Parallel.

IPython Parallel doesn’t hide anything from you, for better and worse.

What IPython Parallel does

IPython Parallel makes explicit parallel computations interactive.

IPython Parallel is not just parallel Python, it’s parallel IPython. That means you have all the power of IPython’s inspection, interactivity, magics, and debugging available on all of your distributed workers. This makes it especially well-suited to prototyping and experimentation.

IPython Parallel also presents standard APIs such as Python Executors, compatible with many other implementations, to make it easy to migrate to and from IPython Parallel, enabling developers to write code that uses a single multicore laptop or a thousand cores on an HPC cluster or cloud.

IPython Parallel in 2021

Interactive progress across parallel engines. Those progress bars are interactive widgets running locally, and on each remote engine!

For a lot of today’s workloads, my default recommendation is: use dask or bodo or another modern tool. IPython even makes this easier if you already happen to have an IPython Parallel cluster, you can tell it to “become dask,” and off you go:

dask_client = rc.become_dask(ncores=1)

However, where IPython Parallel can shine, is making traditional SPMD (e.g., MPI) workloads interactive, especially for prototyping and debugging.

If you have an MPI simulation and you wish you could pause it in the middle, poke around and make plots and interact with just one node or all of them with all the interactive tools available to you in Jupyter, IPython Parallel may be the tool for you.

Recent developments

Because I see the main problem IPython Parallel solves well is adding interactivity to direct parallel execution, the focus of recent developments has been on improving that story, based on feedback from users who are often in traditional HPC environments like SLURM or PBS, or using MPI in the cloud.

Much of the feedback on challenges over the years have been around:

Scaling to larger numbers of engines (1,–10,000)
Better feedback and recovery when things go wrong
Security requirements

Thanks to Bodo, I’ve been able to spend a lot of time this year addressing several of those issues, now available as IPython Parallel 8.0.

Last year, Tom-Olav Bøyum developed a broadcast scheduler as part of his Master’s thesis, which vastly improves the efficiency of sending the same task to all engines (the main thing we do in IPP+MPI). This work is now mostly done and landed in IPython Parallel 7. We’ve also addressed longstanding issues with registering large numbers of engines (5–10,000) at once, a common source of failure on HPC clusters.

One of the most frustrating / challenging things when working with MPI or other parallel computations is dealing with hangs and restarting tasks. Through the new Cluster API, IPython Parallel now has one of its most requested features: the ability to send signals to one or all engines, and forcefully restart the cluster if it’s stuck.

We’ve also improved interactive feedback with progress bars, live streaming output, a newJupyterLab plugin based on dask-labextension, and support for the Jupyter widget protocol, so you can instantiate widgets on engines and interact with them directly from a notebook.

Security questions have also been raised, because IPython (and Jupyter’s) use of ZeroMQ assumes a ‘trusted network’, typically localhost or BSD sockets for notebooks. Since IPython Parallel usually runs across a network, security can be more of a concern (whereas notebooks typically only use standard HTTPS connections over a network), and the level of security in the Jupyter protocol alone may not be sufficient. In the past, we’ve resorted to tunneling TCP over SSH for more secure network traffic, but still always implicitly trusting localhost.

IPython Parallel 7.1 enables CurveZMQ for full authentication, encryption, and forward-secrecy at the transport level, solving a longstanding security shortcoming of IPython Parallel.

IPython Parallel in the future

Scaling is the biggest challenge for IPython Parallel due to its communication model. However, that scaling is on the order of IPython engines, which is not the same as the number of cores. You can benefit from this today if your tasks are already multi-threaded, e.g., through OpenMP threads in numpy, or your own Python threads or multiprocess breakdown of tasks.

Better support of multi-level parallelism, where each “engine” may represent a multicore node, as individual nodes get bigger and bigger would mean moving the bar where IPython Parallel communication is the bottleneck back two orders of magnitude on large machines, because one ‘engine’ could represent 128 cores or more. Good support for 1000 128-core nodes would give IPython Parallel some pretty comfortable headroom for our target use cases, scale-wise.

Also related to scaling, bringing the BroadcastView to maturity should allow us to completely replace the DirectView scheduler, as it should be able to match or beat it in almost every scenario with some further development. There are some features lacking, especially when it comes to error handling, but those can certainly be addressed in time.

Finally, I’ll invite you to get involved. Especially if you are interested in prototyping parallel code, or making traditional MPI-style code interactive, check out IPython Parallel 8 and let us know how it goes, or contribute your use case as an example.

pip install --upgrade ipyparallel

Enabling the JupyterLab debugger with ipykernel

2021-05-13T11:45:00+00:00

Support for the Jupyter Debugger Protocol just landed in ipykernel

JupyterLab 3.0 includes a visual debugger that allows to interactively set breakpoints, step into functions, and inspect variables with any Jupyter kernel that implements the Jupyter debugger protocol.

The first two language kernels to implement the new protocol were xeus-python (a Python kernel) and xeus-robot (a kernel for Robot Framework). Unfortunately, the reference Python kernel, ipykernel, did not support debugging yet, until now!

Today, we are pleased to announce that debugging support landed in ipykernel, and will be available in the next major release, ipykernel 6.0. Pre-releases including the ipykernel debugger are available.

Debugging with ipykernel

What will change with ipykernel 6.0?

Enabling support for debugging in ipykernel required important changes in the code base regarding the concurrency model of the kernel. The main change is that the processing of messages on the “control channel” now happens in a different thread, allowing for the processing to happen while user code is running.

Ipykernel 6.0 includes several other updates. Tornado coroutines were dropped in favor of native coroutines. The Matplotlib inline backend was split into a separate package, and ipykernel depends on debugpy, an implementation of the Debug Adapter Protocol for Python.

If you are interested in testing out the new features, check out the beta release!

pip install ipykernel --pre

Try it now!

Thanks to MyBinder, you can also try it out without the need of installing anything on your computer. Just follow this link:

What about the future?

A lot of new features are in the works with the JupyterLab debugger.

JupyterLab 3.1 will include several usability improvements to the debugger.
It will also add the ability to submit code for execution when stopped at a breakpoint.
We are working on a richer variable explorer, using Jupyter’s rich display system to enable the rich-rendering of variables in the explorer, to e.g. render dataframes as tables.

Finally, we also plan on adding debugging support to other language kernels.

Acknowledgements

The work of Johan and Sylvain at QuantStack on the debugger support in ipykernel was funded by Two Sigma. We are grateful to Min Ragan Kelley and Matthias Bussonnier, who reviewed the pull requests on debugger support.

About the authors

Johan Mabille is a scientific software developer at QuantStack.

Johan is very active in the Jupyter ecosystem, as the creator of xeus, a C++ implementation of the Jupyter protocol, and several language kernels, such as xeus-python, xeus-cling, and xeus-robot. Johan also made contributions to the Jupyter widgets ecosystem and to JupyterLab.

Sylvain Corlay is the founder and CEO of QuantStack.

As an open-source developer, Sylvain is very active in the Jupyter project with contributions in several components of the stack, including widgets, kernels, nbconvert, and others. He is also a steering committee member of the project.

Sylvain also does volunteer work for the community, as member of board of directors of NumFOCUS, co-organizer of the PyData Paris Meetup, and vice-chair of JupyterCon 2020.

Abracadabra! Bringing the magics to xeus-python

2021-02-18T13:30:00+00:00

Last year, we set ourselves to implement a visual debugger for JupyterLab. This endeavor required major developments in the JupyterLab front-end, in core-Jupyter protocols, and on the kernel side (the part of the Jupyter infrastructure responsible for executing the code).

For kernel-side development, we decided to start with the xeus-python kernel, a lightweight implementation of a Jupyter kernel for the Python programming language. Based on xeus, xeus-python has a flexible concurrency model that was required to e.g. add breakpoints while executing code.

Unfortunately, xeus-python did not provide all ipykernel features (magics, Matplotlib support, etc.). Furthermore, many notebooks depend on IPython, as they import it explicitely or make use of the IPython configuration system.

Today, we are proud to announce that xeus-python supports 100% of the IPython magics! This was achieved by leveraging the core IPython package. This is getting us closer to feature parity with ipykernel.

Magics

xeus-python now supports all magics that IPython provides and even user-defined magics!

Support for builtin IPython magics and user-defined magics

Matplotlib

Most Matplotlib back-ends are now supported by xeus-python, allowing you to show static plots with the inline back-end or interactive plots with ipympl in your Notebook:

Matplotlib inline back-end and ipympl interactive back-end

Extending

IPython has a rich configuration system that you can tweak in order to get an enhanced Notebook experience. Xeus-python now supports any configuration you setup for IPython:

Automatically import NumPy in the IPython settings

Visual Debugging

The latest JupyterLab version introduced a visual debugger in its interface. xeus-python was the first Jupyter kernel to add support for it!

Stepping into Python code in JupyterLab with the visual debugger

xeus-python is not the only Jupyter kernel that supports debugging anymore: xeus-robot is a xeus-based Jupyter kernel for the RobotFramework language that supports the JupyterLab visual debugger.

Try it online

Thanks to MyBinder, you can try it out without the need of installing anything on your computer. Just follow this link:

Installation

You can install the latest xeus-python version using mamba or conda:

mamba install xeus-python -c conda-forge

conda install xeus-python -c conda-forge

You can also compile it yourself, instructions can be found in the repository:
https://github.com/jupyter-xeus/xeus-python

Acknowledgments

My work on xeus and xeus-python at QuantStack was funded by Bloomberg.

About the Author

My name is Martin Renou, I am a Scientific Software Engineer at QuantStack. Before joining QuantStack, I studied at the aerospace engineering school SUPAERO in Toulouse, France. I also worked at Logilab in Paris, France and Enthought in Cambridge, UK. As an open-source developer at QuantStack, I work on a variety of projects, from xtensor and xeus-python in C++ to ipyleaflet and bqplot in Python and JavaScript/TypeScript.

IPython 7.0, Async REPL

2018-09-27T17:41:00+00:00

Today we are pleased to announce the release of IPython 7.0, the powerful Python interactive shell that goes above and beyond the default Python REPL with advanced tab completion, syntactic coloration, and more. It’s the jupyter kernel for python used by millions of users, hopefully including you. This is the second major release of IPython since we stopped support for Python 2.

Not having to support Python 2 allowed us to make full use of new Python 3 features and bring never before seen capability in a Python Console. We are still encouraging library authors and users to look at the Python 3 Statement to learn about the end of life of Python 2 and how to stop support for Python 2 without breaking installation for Python 2 end users.

As developers and maintainers of IPython, it was a large gain of time to be able to only develop for a single version of python. Avoiding the use of conditional imports, being able to rely on type annotations, and make use of the newly available Python APIs were some of the advantages that made us more productive. Especially as most of the work on IPython is done by volunteers who work on nights and weekends, with only a couple of minutes here and there, this often made the difference between a patch reaching completion, or the contributor moving on to other pastures.

One of the core features we focused on for this release is the ability to (ab)use the async and await syntax available in Python 3.5+. There are of course many other improvements in this release you can read about in the what’s new.

Demo of awaiting coroutine in IPython 7.0

TL;DR: You can now use async/await at the top level in the IPython terminal and in the notebook, it should — in most of the cases — “just work”. Update IPython to version 7+, IPykernel to version 5+, and you’re off to the races.

See how to install IPython by reading the “what’s new”.

The recipes are currently building on conda-forge and should be available soon. For the time being you can install it via pip:

$ pip install ipython ipykernel --upgrade

A Primer on concurrency

You may have heard about async/await, threads, concurrency, preemptive scheduling and cooperative scheduling without really understanding what all this is about. If you are not familiar will all the above terms, all the hype may be confusing so let’s talk about concurrency in a really high level way.

Typically when your computer needs to execute many tasks, it will switch between them really fast, so from the human point of view it looks like everything is being processed at the same time. There are two main ways of doing so under the hood: Preemptive Scheduling, and Cooperative Scheduling.

With preemptive scheduling changing tasks can happen at any time. For example, while writing this blog post, I could stop in the middle of a word to start writing an email, which will itself be interrupted to check Gitter/Slack, before coming back, writing 5 words and stopping to get dinner.

TL;DR: Concurrency (from Geek And Poke, 2009)

With cooperative scheduling, the task switches can happen only at agreed spots. The term co-operative comes from the fact that tasks need to co-operate for the whole process to function. If a task decides to never take a break to let you to do something else, the illusion of many tasks being completed at once disappears.

Each approach has its own advantages and drawbacks, and we will not focus on these. Let’s just say that with co-operative scheduling async/await let you mark the areas where interruption is allowed to occur.

Moreover, async/await syntax allows cooperative scheduling in Python in a way that lets you write code that looks synchronous (without task switches), while actually being able to be interrupted, from the point of view of the computer. It also keeps the programmers from having to worry about global state changing under their feet, as this can occur only at the proximity of await keywords.

When going to a restaurant, social conventions (and common sense) tell us when and how these interactions can or cannot be interrupted, but programming languages need markers when using cooperative scheduling. These are asyncandawaitkeywords in Python.Asyncmarks a function that *may* be interrupted,awaitis required to call async-functions (akacoroutine`) and marks a point were task can be switched.

If you want to learn more we strongly recommend reading the Trio Tutorial Primer on async programming.

Async in the Python world

In the current Python ecosystem, packages tend to standardize around AsyncIO, provided in the Python standard library. AsyncIO can sometimes be judged as complex even by well known developers; this is in part due to the necessity of supporting other older asynchronous projects like twisted or tornado, but it’s also what makes a lots of its power: One event loop to rule them all.

Running a single async task requires you to learn about AsyncIO, write a non negligible amount of boilerplate code in order to fetch a single result. This can be especially cumbersome when doing interactive exploration, and likely will keep users from experimenting with AsyncIO code.

How to run a single async task in Python repl without async integration.

As Raymond Hettinger would says (slamming hand on podium): “There must be a better way”.

IPython AsyncIO Integration

Thanks to a multiple month effort (actually this work started close to 2 years ago), and the work of many talented people, you can now directly await code in the REPL and IPython will do “the right thing”.

Awaiting AsyncIO code should now automagically work.

With the new integration, you don’t have to import or learn about asyncio, deal with the loop yourself, or wrap your task in its own function. You are now able to just focus on the business logic and move along.

The only thing you need to remember is: If it is an async function you need to await it.

We hope that this will free users to experiment and play with asynchronous programming. Of course this will not magically make your code faster, or run in parallel, simply easier to write and reason about.

Other Async Libraries (aka: curio and trio integration)

The addition of async and await keyword in Python did not only simplify the use of asynchronous programing and the standardization around asyncio; it also allowed experimentation with new paradigms for asynchronous libraries. David Beazley created Curio, and Nathaniel Smith Trio, which both explore new ways to write asynchronous programs and explore how async, await and coroutines could be used when starting from a blank slate. The Trio documentation introduction and which problems it attempt to solve [1, 2, 3, 4, 5, 6] are highly recommended reading with varying level of technicality.

Interactive uses of libraries is key to getting insight and intuition on how a system works, intuition is critical to rapid prototyping, development and creation of higher levels of abstraction. It was natural for us to build support for Curio, Trio, (and potentially new other async libraries) into IPython.

You can setup IPython to run async code via Curio, or Trio and experiment or write production code using these libraries. To do so use the %autoawait magic, and tell it which library to use.

Defining an asynchronous function and spawning multiple concurrent task in IPython using Trio.

As you can see code looks really natural, and it is easy to forget that the above snippet is usually a syntax error in Python or older version of IPython. The astute reader and IPython expert will have suggested to use the %%time cell magic instead of doing it manually, though a couple of magics still need updates to properly handle async code. We look forward to your contribution on this front, and are excited to see what you can come up with.

Async in Notebooks (and other Jupyter Clients)

If you are a Jupyter user, you most likely use a Notebook interface, and interact with IPython via the ipykernel package.

Using AsyncIO in nteract desktop works out of the box with newer IPython and IPykernel

We’ve been working hard on making async code work in a notebook when using ipykernel. While most of the heavy lifting was done in IPython, the work in IPykernel was non-negligible, and required the accommodation of a number of use cases, which are not working. You now have to update both IPython to 7.0+ and ipykernel to version 5.0+ for async to be available. If you are using pip: $ pip install IPython ipykernel --update. As for conda, the packages should be available on conda-forge soon. With these new releases, async will work with all the frontends that support the Jupyter Protocol, including the classic Notebook, JupyterLab, Hydrogen, nteract desktop, and nteract web. The default code will run in the existing asyncio/tornado loop that runs the kernel. Integration with Trio and Curio is still available, but tasks will not be interleaved with the asyncio one — at least not yet. We welcome work on this front.

Submitting background tasks still requires you to access the asyncio event loop, and we are still be looking for contributions on this front as well, to make it even easier to run async code.

There are still some question on how to handle nested asyncio eventloop. It is indeed usually impossible to run nested eventloop, in the case of asyncio, trying to do so raises a RuntimeError the kernel already ran in and asyncio eventloop, calling directly or indirectly loop.run_until_complete and alike is not possible. There are discussions to use libraries like nest_asyncio as pointed out on this comment, but until those are more battle tested we do not want to commit a default solution in the core of IPython and let the ecosystem develop.

Future improvements

As far as we know, this is the first Async-aware Python REPL, and libraries like Trio/Curio are still young, thus there are still a number of use-cases we have not yet even thought about! We are encouraging you to come forward to talk about your use cases, what you tried and what did not work. There is also a number of new features to implement (making magics work with async, tab completion, background tasks) on which we would welcome new contributors.

Release of IPython 5.5 and 6.2

2017-09-15T18:42:00+00:00

Four month after releasing IPython 6.1 and 5.4, and a couple of hours after the release of the notebook 5.1, we are happy to announce the release of IPython 6.2 (Python 3 only), and it’s cousin IPython 5.5 still compatible with Python 2.7.

You can update now by using:

pip install --upgrade ipython

If you have a recent enough version of pip you will get the latest compatible version of IPython regardless of the version of Python you are running.

The conda packages are on their way; once available you will be able to update with:

conda install ipython

New Features

As IPython 6.2 and 5.5 are minor releases you will only find a small number of new features. When API additions were done on IPython 6.2 they were backported on 5.5 to simplify the maintenance of code compatible both with Python 2.7 and 3+. You can find the full list of new features in the changelog.

As a quick teaser, IPython 6.2 can now:

Show function signature in the terminal while completing.
Assignments can trigger the display mechanism
IPdb can be called recursively
Support for system-wide configuration
Built-in support for Progress Bar.

Head there for more complete description.

Slowing down backports to 5.x

As stated on our roadmap, we’ll keep releasing a 5.x for some time, though starting at end of year. However, we will decrease our active involvement in fixing bugs affecting the 5.x branch. We will still accept PRs, and backport if you nicely ask us. Releases will happen occasionally if fixes are available, but we will be sunsetting the Python 2 support slowly.

If you are interested in further maintenance of the 5.x branch, we would love help with that work. Feel free to contact us on GitHub.

What’s next ?

We are going to start thinking about IPython 7, and start to embrace more of the Python 3 only features. Slowing down backports should allow us to be more confident that changes will not affect the automatic application of patches on old branches. Trimming down old legacy code may also help to regain some speed on interpreter startup, and should lead to plenty of opportunities for new contributors to join.

We will also try to simplify our documentation, and make often requested sections easier to find.

If you are looking for a project to contribute to – code, documentation, example, design, helping others, feel free to contact us so we can guide you through the process.

Enjoy this new release, and hope to see you around the mailing list and bug tracker!

Release of IPython 5.4, 6.1, and rlipython.

2017-05-31T22:47:00+00:00

Slightly over a month ago we released IPython 6.0 : the first version of IPython to be compatible only with Python 3. The reception of IPython 6.0 was great, and the codebase way more maintainable. So while 6.0 was mostly focused on internal refactoring and working upstream to make the release of a Python 3 only version possible, the next releases can focus on bug fixes and new features. Today we not only have the new release of IPython 6.1, but IPython 5.4 as well (the LTS branch targeting Python 2.7). Read on for more information! As usual, you can upgrade using pip and the 🔥 Conda Forge 🔥 builder is warming up: IPython 5.4, and 6.1 should be available via these channels soon.

IPython 5.4 (and 6.1) major new features

IPython 5.x is the LTS branch which is still compatible with Python 2. The IPython team ♥️ Python 2 users so we continue to make fixes to our 5.x branch. You can install it today by making sure you have pip 9 or above:

$ pip --version
pip 9.0.1

Then upgrade IPython:

$ pip install ipython --upgrade

If you do not have pip 9 and are using Python 2, pip may download IPython 6, which will not work on your system. If you are on Python 3, see the next section to upgrade to IPython 6.1 but read on in this section for the new features.

IPython 5.4 got a number of new features that went into 6.0, plus a number of API improvements from 6.1 allowing the underlying library to target both 5.4+ (Python 2+3) and 6.1+ (Python 3) without having conditional branches. This should make the transition easier for users and developers

You can see the what’s new for:

`TerminalIPythonApp.interactive_shell_class`

In 6.0 and 5.4 IPython gained a new c.TerminalIPythonApp.interactive_shell_class option, to customize the class used to start the terminal frontend. This should enable you to use custom interfaces, such as reviving the former readline interface which is now a separate package. See the next section for more.

Define `_repr_mimebundle_`

In IPython 6.1+ and 5.4+ object can now define _repr_mimebundle_ in place of multiple _repr_*_ methods and return a dictionary of MIME types and MIME data for rich display. IPython provides _ipython_display_ but this mechanism was complex and difficult to use for frontends with different architectures, such as nteract and hydrogen. This new _repr_mimebundlde_ methods are thus simpler for both backend and frontends developers to implement, and are already handled in projects like nteract. Our own JupyterLab, the next-generation evolution of the Jupyter Notebook, takes full advantage of these capabilities to expose rich and flexible data display capabilities that work not only in notebook documents, but across all of JupyterLab.

This new model will simplify things for client implementations and will facilitate the publication of custom mimetypes (GeoJson, Plotly, DataFrames, etc.). See the Custom Display Logic example notebook for examples and technical details.

Execution Heuristics

In IPython 5.4+ and 6.1 +, the heuristic for execution in the command line interface is now more biased toward executing for single statement. Use Ctrl-O to force the insertion of a new line. The execution semantics of Enter were still debated, so if you like to configure that you can define your own function which decides what to do.

Display IDs

IPython 5.4 and 6.1+ implement display ids, which can be used to update a displayed object. This should provide simple patterns to implement common cases of dynamic display updates (see Updating Displays.ipynb notebook). It will also simplify the implementation of some types of interactive user interface elements, such as progress-bars. Display objects with IDs can be updated from other cells or frontends.

The display function is now always available

The display() function is now available by default in the IPython session, without users having to import anything. display() is the "moral equivalent” of print(), but takes advantage of the rich display capabilities of Jupyter (and defaults to printing a string for objects lacking rich representations). By making display() appear, in practice, as if it was built-in, we hope to encourage library authors to provide their objects with rich and informative representations that users can now more conveniently access, whether at the IPython REPL or in any of the richer clients in the Jupyter ecosystem (JupyterLab, classic Notebook, nteract, etc.). Note that this automatic import of display() is only done in the live IPython kernel; scripts and libraries that rely on display and may be run outside of IPython still need to import it using from IPython.display import display.

Rlipython

Many users were pleased with the transition from readline to prompt_toolkit for IPython 5.0, this gave IPython multiline editing, nice completion and highlighting as you type. Still some users are missing the old readline interface (for its reactiveness an custom keybindings) though are still using IPython 4.x series.

While prompt toolkit is still the default interface, you can now use the new configuration option c.TerminalIPythonApp.interactive_shell_class .With this you can now replace the Prompt_Toolkit interface. For one of your liking.

Rlipython is one of these alternative interface that resurect readline and is lighter, and more responsive on some system than default IPython. Using it still give you most of the new features of IPython 5.x or 6.x ! This should also please many of our users who use vi keybinding in terminals as this should respect .inputrc.

To try rlipython, install it with pip,

$ pip install rlipython

And follow the instructions.

We look forward to your contribution and please give lots of 💖 by to @ivanov (tweet) for making this possible.

IPython 6.1 and Python 3 only features

IPython 6.1 is the first minor release of our Python 3 only branch, it is the branch that get the most work, and start to use great new features such as type annotations, async/await [more to be added]. You can install it today please upgrade pip to pip 9 if you can, then upgrade IPython:

$ pip install ipython --upgrade

You’ll get all the new features in 5.4, some of which are already in IPython 6.0, plus a number of additional enhancements:

You can read more on the what’s new for IPython 6 what’s new.

In this release, we had a number of bug fixes and improvements from developers at TwoSigma, who participated in an internal open-source hackathon for Jupyter/IPython in May. Hackathon attendees tackled some long standing, hard to debug issues, and feature requests. Among these:

Quotes in a filename are always escaped during tab-completion on non-Windows, which should make many of our windows users happy. As none of the core devs use windows this one needed someone to simply use some elbow grease!
Variables now shadow magics in autocompletion which make autocompletion of things like matplotlib simpler and less surprising. - Magic aliases can now have parameters.
?? /pinfo2 machinery will show docstrings if source can’t be retrieved.

The completer also got a number of small improvements, and is smarter in many cases. For example, it understands the arguments to the %config and %colors magics. Long methods and snake_case function names can now be completed from abbreviations. For example, if foo_bar_kitten is an existing object, f_b<tab> will find it as a possible completion.

Look out for a blog post from TwoSigma.

Only a small number of improvements are describe here, and we encourage you to read the full change logs if you want to know more.

We hope that all these releases will make you more productive. We welcome your contributions, and questions. Feel free to open issues on GitHub, we’ll do our best to respond.

Release of IPython 6.0

2017-04-19T22:37:00+00:00

It is with great pleasure that today we released IPython 6.0 — almost a year after the 5.0 version.
Users on Python 3.3 and above can get this latest version with all its new features by asking your package manager to upgrade IPython. If you are using pip:

Ensure you have pip 9+:

pip --version

if not upgrade pip:

pip install pip --upgrade

Then upgrade IPython.

pip install ipython --upgrade

or if you have conda:

conda update ipython

It might take a few hours/days for the various conda channels to be updated.

The What’s New page has an extensive description of the changes, which includes new, modified and deprecated features.

Highlighted feature: Jedi completion

One of the biggest changes is integration with Jedi, which provides completions using static analysis, and can display completion types. This will provide a foundation for a richer tab completion experience (e.g., it allows extracting and displaying function signatures in the completer). As of now, these features are available only for command line users, but work is underway in IPykernel and in various frontends (like nteract) to make use of these features.

Sunsetting Python 2 support

As we announced last year when we released IPython 5.0, IPython 6 and above require Python 3. The 5.x branch is still maintained and will be regularly released, so Python 2 users will still see updates, but new features in IPython 6.x will not be backported.

We have worked to make installing IPython on Python 2 and 3 as seamless as possible, but you may still encounter errors. If you do, please check the [readme](https://github.com/ipython/ipython/blob/master/README.rst and if you still have a problem, open an issue on Github.

If you want to hear more about the process of creating a seamless Python 2 sunset and are planning on attending PyCon 2017, we encourage you to attend our talk “Ending Py2/Py3 compatibility in a user friendly manner” on Saturday, May 20, 2017 at 15:15–16:00 PDT.

Make sure you have Pip 9 or above

For most end users the only thing you should pay attention is to have pip version 9, or above. To check your pip version, run:

pip --version

If you have a version of pip older than 9.0.1, you should upgrade pip to its latest version before upgrading ipython ****:

pip install pip --upgrade

Pip 9+ will install the latest compatible version of IPython – that is to say IPython 5.x if you are on Python 2, and IPython 6.x if you are on Python 3. Make sure this is the case for all your Python environments, and build scripts. Users of Ubuntu Trusty 14.04 LTS should be aware that the default pip is too old and will not install the correct version of IPython. Don’t forget to upgrade pip in your various install script, continuous integration settings and docker images.

The Python 3 statement

We care deeply about the seamless coexistence of Python 2 and Python 3 software. This is one of the reasons we started this transition with a new release that is only available for Python 3, while still maintaining the 5.x release series, which is compatible with Python 2. If you want to read more about these reasons you can find more information at the python 3 statement.

As expected, we discovered many challenges and pitfalls as we moved from a single source to a Python3-only source. In the course of overcoming these obstacles, we learned a great deal. We have documented the the risks and errors you can encounter (and their solutions) at the practicalities section of the python 3 statement.

If you are a library maintainer planning to stop support for an older version of Python (even a minor version), and you want to limit the frustration faced by users who use your library on the older version of Python, we could use your help! If you want to add to the resources available for trouble-shooting this transition, please join the conversation on our GitHub repository.

Migrating to Python3-only source

The migration of the IPython source code to be Python3-only was not an easy task, and was – as far as we can tell – unexplored territory. But we’ve bitten the bullet and hopefully anticipated most of the problems that can arise. With this, we hope to make the transition easier for other projects who want to follow our lead.

First, let’s speak about our personal experience writing Python3-only source code.

The size of the IPython codebase has decreased by about 1500 lines of Python code relative to the last release. Of course, that’s not solely due to the removal of Python 2 support, but a non-negligible amount is. And this reduction is even more remarkable in light of completely new features that required adding hundreds of lines of code. A large number of conditionals are gone, making the code more straightforward, easier to read, and simpler to maintain. Across the codebase we saw improvements in code compactness just from removing utility functions that existed only to provide identical behavior across Python 2 and Python 3. Even then, parts of the codebase remain affected by “Python 2 code”, so the potential gains are even greater. We will continue our quest to remove things as we go, and, as usual, contributions are welcome.

This change eases the burden on contributors to IPython. Contributors can can spend less time thinking “what about Python 2”, or rewriting a pull request because the Python 2 test suite fails. At the same time, our tests now complete more quickly on continuous integration services because they need to run on fewer versions of Python.

A couple of new APIs are using type annotations, Python3-only syntax which, properly used, make the code clearer and easier to refactor, and allow the documentation to focus on usage / reason than to describe the types of the function. We can also make use of nifty Python 3 features like keyword-only arguments, which is definitely appreciated when designing APIs.

From a developer point of view we are extremely pleased with having the possibility to write Python3-only code, and are looking forward to even more improvements like pathlib.

Keeping a Python 2 stable branch

Keeping a Python 2 stable branch, and making the install experience as seamless as possible for Python 2 users was a key point in our decision to migrate to requiring Python 3. One of our first questions was how could we make sure that Python 2 users using pip install would get the latest 5.x version of IPython still compatible with their system (and not the latest >6.x version that would not be compatible)? This lead us to a number of upstream patches (we’re giving a talk at PyCon, come see us) and a number of guides as to how to make the transitions as a seamless as possible. We patched pip, pypi-legacy, warehouse (we had to learn postgres for that one), amended peps and made use of recent improvements in setuptools to allow Python users to peacefully coexist. The only thing the user should need to do is to make sure to use pip 9+ , and to use pip install . and pip install -e . instead of invoking setup.py directly.

Migrating more code to Python3-only

IPython is the first of the Python packages we maintain which has been migrated from Python 2+3 compatibility to requiring Python 3. Some of our more recent projects (like JupyterHub) have required Python 3 from their beginnings. We expect more Jupyter & IPython projects will move to requiring Python 3 in the future.

As the Jupyter protocol is language agnostic, it is perfectly possible to run Python 2 code in a Python 2 kernel using a Notebook Server running on Python 3. Especially since JupyterHub already requires Python 3, we envisage that the single-user notebook server will follow suit soon.

Conclusion

We hope you will enjoy this release. It will be the base for some awesome features, like async/await REPL. We hope our work in the packaging ecosystem will minimise the inevitable teething difficulties from our first release to require Python 3. We look forward for your feedback.

HN and reddit

The Continued Existence of the Emacs IPython Notebook

2017-02-23T22:26:00+00:00

This is a guest post by John, who initially sent it to our mailing list; we believe that it’s an interesting story, so we wanted to share it on here on the blog.

All,

After the recent, exciting announcement of the eminent JupyterCon, I was somewhat saddened to see no mention of the EIN, or the Emacs IPython Notebook, as an available client to the Jupyter notebook server. Drama queen that I am, I quickly wrote a note to Fernando, who patiently and kindly explained that not many were aware that this project still existed and that perhaps an announcement or note to the main Jupyter list might be warranted. Hence the following, brief history. I also promise to try to not be a drama queen.

Some of you may remember, from back when the Jupyter project was still known as the IPython notebook, a talented and prolific coder of the name Takafumi Arakaki, or tkf, who created an alternative client to the notebook server’s default web browser interface.

This client, which he called the Emacs IPython Notebook (you can still find his project on GitHub), or EIN, provided a complete IPython notebook experience in the venerable Emacs editor. Not only was EIN nearly feature complete when compared to the browser interface it also provided some useful features for the Python programmer, like allowing one to connect Python buffers to a notebook and using jedi for autocompletion in the notebook buffer.

Around March/April of 2014, just as IPython was advancing towards 1.0 and making big changes in the notebook/contents API and the kernel communication protocol, tkf mysteriously stopped pushing commits to his github repository.

I did not know tkf other than from a couple brief conversations. I sincerely hope tkf’s story has a happy end (he does appear to still push the occasional commit); he is clearly a talented programmer and without him this impressive piece of software would not exist.

This is the point where yours truly enters the story. I had discovered the IPython notebook the previous year and had found it an exteremly useful for analyzing the performance of catalytic process units in the refining industry and for working with Python in general, but being a long-time Emacs user I had somewhat bounced of the web interface. Discovering EIN was a godsend, and it quickly became a mainstay in my set of analytic tools.

Unfortunately the changes in going to v1.0 of the IPython notebook broke EIN, and with tkf apparently out of the picture there did not seem much hope in EIN staying compatible. Considering that I am a father of two with a full-time job that has absolutely nothing to do with programming (and yet I am a long-time Emacs user — it’s complicated, don’t ask), I can only describe what happened next as an act of complete insanity: I decided to fork tkf’s code, dig in and try to keep up with the changes in ipython.

Truthfully, no one was more surprised than I when I was actually able to keep ein working with versions 1.0 and, soon after, 2.0 of IPython. In fact that compatibility, in theory, is still in the code. One, again in theory, should be able to fire up a 1.x or 2.x version of the IPython notebook and connect to it using my fork of EIN. I say in theory, though, as I haven’t touched that part of the code in some time and it undoubtedly has suffered some bit rot in the intervening years.

The rest of the story is less interesting. Eventually I managed to convince GitHub and MELPA to treat my repository as the official version of ein. There was some short-lived talk of renaming my fork to ‘zwei’, but the consensus was that things were confusing enough with the change in ownership and to keep the name as ein.

Currently one can download ein through either MELPA or el-get, and someone has even been kind enough to create a spacemacs layer with convenient VIM keybindings for the heathens.

At the moment EIN supports the recent incarnations of Jupyter notebook, v4.3.1, token authentication, _xsrf cookies and all. By the time you read this I may even have pushed some commits that allow one to start and automatically log in to a jupyter notebook server all from Emacs without having to drop into the terminal.

In all, EIN continues to be a viable alternative to the web browser client. It is not 100% feature complete, though, as it notably does not support widgets and quite possibly never will.

I haven’t kept close track of who is using EIN, but as of this writing it has 341 stars on github and 28,175 downloads from MELPA. I know EIN is being used in at least a couple businesses and from what I have heard it tends to be more popular among those with a programming background — scientists and engineers tend to prefer the web client which is not surprising since Emacs is not so much a text editor as it is a Way of (Un)Life.

I encourage anyone who is interested in trying out ein to install it via MELPA or from the Spacemacs ipython-notebook layer. For slightly more information see the Quick Try and Usage sections in the documentation. The information there is not perfect but it is hopefully good enough to get you started with the tool. If you run into trouble do not hesitate to open an issue on GitHub; this is a hobby project but I get a lot of joy working with this code and will do my best to support it.

If you have made it this far then my sincere thanks for staying patient through my ramblings. As a parting thought I want to express my sincere thanks to Takafumi Arakaki, wherever he may be, and to the Jupyter team for their fantastic work in creating this amazing piece of software.

John Miller

Release of IPython 5.0

2016-07-08T06:56:00+00:00

IPython 5.0-LTS is out!

We are pleased to announce the release of IPython 5.0 LTS (or Long Term Support). IPython is the Python kernel for Jupyter and the interactive Python shell; it provides a rich set of features for fluid interactive computation in Python at the terminal, in the Jupyter Notebook and across all other clients that support the Jupyter architecture.

This release has some exciting new features and lots of new development (227 commits by 27 contributors over 191 PRs). Most importantly, there have been significant improvements to the classic IPython command line interface.

As usual you can try this new release with:

pip install ipython --upgrade

The package should also be available through conda and other package managers in the next few days.

Note: IPython is now developed under the umbrella of the broader Project Jupyter, but like other components of Jupyter, with its own independent schedule. Therefore, this release does not impact the Jupyter Notebook, Qt Console, nbconvert, or other packages that were formerly part of IPython.

A brand new terminal interface

Decoupling IPython from the Jupyter Notebook package has allowed the core team to focus on improving the command line interface independently of the notebook. The awkward dependencies on pyreadline for Windows and gnureadline for Mac prompted Thomas Kluyver to replace the old machinery with a brand new pure-python readline replacement: prompt_toolkit.

The prompt_toolkit package is an amazing library from Jonathan Slenders and recently reached version 1.0. Going beyond readline, prompt_toolkit provides many advanced features for editing text in the terminal that significantly improve the user experience. Since it is a cross-platform library, all our users on Linux/Unix, macOS and Windows benefit from these improvements. Thanks to prompt_toolkit, IPython now supports:

Syntax highlighting as you type
Real multi-line editing (up and down arrow keys move between lines)
Multi-line paste without breaking indentation or immediately executing code
Better code completion interface (we plan to improve that more)
Optional mouse support

We’re not using all of the features of prompt_toolkit yet, but after working with it for a few weeks, it already feels strange to go back to older versions of IPython without these improvements. We are hopeful that you will enjoy them. We’re extremely grateful to Jonathan Slenders, who has been super responsive with all our questions and feature requests!

You can get a more detailed list of changes to expect by reading the “What’s new in IPython 5.0” document.

Jupyter Console

The Jupyter Console provides the interactive client-side experience of IPython at the terminal, but with the ability to connect to any Jupyter kernel instead of only to IPython. This lets you test any Jupyter Kernel you may have installed at the terminal, without needing to fire up a full-blown Notebook for it. The Jupyter console gained also most of the functionality described above and also makes use of prompt_toolkit.

A few days ago we released Jupyter Console 5.0 as well, which brings compatibility with IPython 5. If you are a Jupyter Console user you will need to upgrade as well.

$ pip install jupyter_console --upgrade

Long Term Support (LTS)

You might have picked this up from the title of this blog post: IPython 5.x will be the first release series to get Long Term Support (hence the LTS name).

With IPython, we usually only offer support for a single major release at a time; once a new major release comes out, previous major releases stop getting bug fixes. For the 5.x series releases we are making an exception to that rule: until the end of 2017 the core team will do its best to provide fixes for critical bugs in the 5.x release series. Beyond that, we will deprioritise this work, but we will continue to accept pull requests from the community to fix bugs through 2018 and 2019, and make releases when necessary.

We hope that this will help organisations that need long term support for IPython version 5.x.

End of support for Python 2

IPython has been compatible with Python 3 for several years, since Thomas Kluyver ported the codebase to be Python 3 compatible using 2to3 in 2011, and we moved to a single-source codebase for Python 2 and 3 in 2013. The day to day development of IPython is now completely done using Python 3, and we’re starting to accidentally break Python 2 compatibility until tests or users flag it. We’re also keen to use many of the new Python 3 features, such as type annotation, yield from, asyncio, async def, await and other improvements the language and its standard library have gained in recent years.

We have therefore decided that IPython 5.x will be the last major version to support Python 2.

This is, of course, why we are planning to support IPython 5.x for much longer than usual. We recognise that many people still use Python 2, and they will be able to continue with a supported version of IPython for several years, and transition at a time that suits them. Beyond the end of 2017 we are willing to provide minor bug fix releases in the 5.x with community contributed patches. Most importantly, no new features will be added to a Python 2 supported IPython beyond the upcoming 5.0 release.

Thus, the next major version of IPython, IPython 6.x will require Python 3. It will start to make use of new syntax, and shed the compatibility layer we have in place.

If you are a Python 2 user, be reassured, we will make sure that upgrading does not unexpectedly install IPython 6.x and break your system. You can decide to stay for a longer period of time on IPython 5.x LTS and decide to leapfrog a few IPython versions once you migrate to Python 3, though we recommend keeping up to date with the latest stable versions as they are released, and of course to migrate to Python 3 when possible.

IPython is the first IPython/Jupyter project to drop support for Python 2, but you can expect other components of IPython/Jupyter to follow. Since its inception, JupyterHub for example as always been Python 3 only.

It is important to note that users will always be able to use a Python 2 kernel with the Jupyter Notebook, even when all of our projects have transitioned to Python 3: as part of our LTS commitment, we will make any necessary updates to the IPython kernel so it can continue to work in a Jupyter Notebook for the duration of our LTS support.

Help us with the Python 3 transition

We understand that migrating to Python 3 can be difficult for various reasons, and that planning ahead is often necessary. For this reason we are helping to gather a non-exhaustive list of projects that have decided to drop support for Python 2 in or before 2020, when support for Python 2.7 itself ends. Projects such as Matplotlib and SymPy plan to drop support in the next few years, while a few projects like Scikit-Bio are already ahead of us, and should be Python 3 only soon.

Thus we decided to sign the Python3 Statement that lists projects that are taking this step, as well as — when possible — provide a planned release schedule for which versions will still be Python 2 compatible, and which versions will be Python 2 only.

If you’d like to add your project to this page, or you know a project that’s thinking about the Python 3 transition, please get in touch there. We believe that giving enough information to Python users as early as possible will help ease the transition.

See you at SciPy!

Some of us will be at SciPy this year in Austin. We’ll be happy to meet with you, and hopefully run sprints on IPython and Jupyter projects. We hope to see you there.

[Update] Friday July 8, 11 Pacific

Hackernews and Reddit threads

Jupyter Ascending

2015-08-12T05:48:00+00:00

Today is an exciting day! We have just finished releasing IPython 4.0, the last piece of the first release of Jupyter and IPython after The Big Split™.

The quick way to get Jupyter upgraded is:

pip install --upgrade jupyter

If you are using conda or canopy, we hope to have packages on those distros soon.

This means that when you do

pip install --upgrade ipython[all]

You will get jupyter instead.

Refresher on The Big Split™

IPython 3 was the last major monolithic release of IPython containing all of the language-agnostic code, such as the notebook, which is moving under the new name Jupyter. As part of that process, we have created several discrete packages for each component — nbformat for the notebook format, nbconvert for the notebook conversion machinery, notebook for the HTML notebook application, etc.

You can get each of these components independently, e.g. with

pip install --upgrade notebook

which will not include the pieces that the notebook doesn’t depend on (e.g. qtconsole). Or you can get the whole thing with the jupyter metapackage, which will pull in everything that was part of IPython 3 (with the exception of IPython parallel, now called ipyparallel).

Since all of these packages were part of IPython 3, they are all starting out at 4.0. They won’t be updated in coordination with each other as their APIs and release schedules will be allowed to advance at their own respective paces.

There are import shims in the IPython repo for backward-compatibility, so any code that imports IPython.nbformat, for instance, should continue to work with a warning about the new home of the package:

In [1]: from IPython.nbformat import read
/.../site-packages/IPython/nbformat.py:13: ShimWarning: The `IPython.nbformat` package has been deprecated. You should import from nbformat instead.

If you have any code that imported from the old subpackage that worked on 3.x but doesn’t on 4.x, please let us know.

This split also means that

pip install ipython

gets you a much smaller install, since it only includes the terminal IPython interface, which some people have been asking for for a long time.

Configuration

Many of the ipython subcommands are now jupyter subcommands, e.g.

ipython notebook

is now

jupyter notebook

The configuration files for these jupyter commands are now in your ~/.jupyter directory instead of ~/.ipython/profile_default. Jupyter should automatically migrate your notebook and qtconsole config to the new location if they exist.

See the Jupyter config docs for more info.

Docs

See the new Jupyter docs for more information about Jupyter, its components, and installation.

Issues

Please let us know on the mailing list if you have issues with the new install, or open an Issue on GitHub for the repo that is misbehaving. There’s a lot of reorganization, and we thank you for your patience during this transition.

Thanks

Thanks to everyone who helped with the Big Split! Here’s to a bright future for IPython and Jupyter.

IPython 3.2 released

2015-06-23T01:11:00+00:00

Hello Jovyans,

We just release a few hours a ago a new minor version of IPython. Please upgrade your IPython notebook installation to 3.2 as soon as possible. There are important security patches in this release. Users are strongly encouraged to upgrade immediately.

Highlights:

A security improvement that sets the secure attribute of the login cookie to prevent them from being sent over http
Revert the face color of matplotlib axes in the inline backend to not be transparent
Enable mathjax safe mode by default
Fix XSS vulnerability in JSON error messages
Various widget-related fixes

You can get the latest via pip using pip install --upgrade ipython[notebook]. Use conda update ipython if you are using Anaconda.

A CVE has been requested for the XSS vulnerability on the OSS Security mailing list http://permalink.gmane.org/gmane.comp.security.oss.general/17131

[edit] CVE are tracked under numbers CVE-2015–4706 and CVE-2015–4707 [/edit]

Thank you to Ahmad Khan of IBM for reporting the XSS vulnerability and Jason Grout for setting MathJax safe mode.

The Big Split™

2015-04-16T02:39:00+00:00

IPython has grown a great deal over the years. As of 3.0, IPython includes:

an interactive shell
a REPL protocol
a notebook document fromat
a notebook document conversion tool
a web-based notebook authoring tool
tools for building interactive UI (widgets)
interactive parallel Python based on the above REPL protocol

While all of these are part of the same story of tools for the lifecycle of a computational idea, they are increasingly becoming distinct projects that happen to live in a single repo. One significant part of the development is that pieces like the notebook and protocol are not even specific to Python, so it doesn’t make sense anymore that they reside in a project called Interactive Python. This is the impetus for Project Jupyter, announced at SciPy 2014, which is the new home of language-agnostic projects that began as part of IPython, such as the notebook.

If anyone has been confused by what Jupyter is[1], it’s the exact same code that lived in IPython, developed by the same people, just in a new home under a new name.

As IPython has matured, the interfaces between these different components have stabilized:

the notebook format
the REPL protocol

There has been growing tension in the project as we work to stabilize some APIs, such as the notebook format and message protocol, while we also start work on highly unstable experimental projects, such as the interactive widgets. The tension comes from the divergent maturity of these components, and associated appropriate release cycle of the different pieces. It simply no longer makes sense to release all of these pieces at the same time.

We think the maturity of these APIs has reached the point that we can reasonably separate development of the different components, and rely on these interfaces to communicate between the projects. That should allow the projects to have their own release cycles, with more frequent bugfix releases, and grow their own developer and user communities.

This means that IPython 3.0 will be the last monolithic major release containing all of these projects. IPython 4.0 will primarily consist of splitting IPython into these subprojects.

Many of these pieces (REPL protocol, notebook-related tools) are language-agnostic, and not appropriately called IPython anymore.

What it will mean

Once this is done and we’ve made the 4.0 releases of everything, each project will have its own package. To get the notebook, you would:

pip install notebook
jupyter notebook

And nbconvert:

pip install nbconvert
jupyter nbconvert

etc.

This is not true yet, and these commands won’t work until we have made stable releases of these packages.

What it means now: moar repoz

Splitting IPython means creating new repos for each of the subprojects, which we did last week. We created eleven new repos (!), which you can find on the ipython and jupyter GitHub organizations.

Since we haven’t made a release of any of these packages, it’s a pain to get up and running with all of these, since you need to clone each one from master. The only way to automatically resolve dependencies to GitHub repos is with the deprecated, discouraged, and disabled-by-default dependency_links. This means you need to explicitly clone and install each of the dependencies in order, which you can do with a requirements.txt:

curl -O https://gist.githubusercontent.com/minrk/2b74fb7465a3702e2ce4/raw/requirements.txt
pip install -r requirements.txt

We’ve put a requirements.txt in each repo with the git URLs of the unreleased dependencies, so for any given repo, you should be able to get working on master with:

git clone https://github.com/jupyter/notebook
cd notebook
pip install -r requirements.txt -e .

As we start releasing the dependencies, we’ll trim down these requirements files as they become unnecessary.

Updated Imports

For the most part, each of the new packages was an IPython subpackage, relocated to a new top-level package. This means your import should only need to replace the IPython.subpackage part with its new jupyter name and you should be set. We’ve also added shims to IPython, so that imports that worked in 3.0 ought to keep working even after the next release, with a warning pointing to the new location.

If you want to live in The Future™, here are most of the updated imports, after the move:

IPython.utils.traitlets ⇒ traitlets
IPython.config ⇒ traitlets.config
IPython.html ⇒ notebook
IPython.nbconvert ⇒ nbconvert
IPython.nbformat -> nbformat
IPython.parallel ⇒ ipyparallel
IPython.qt ⇒ qtconsole
IPython.terminal.console ⇒ jupyter_console

IPython.kernel has split in a slightly more complex way. For the most part, it’s split into these two packages:

IPython.kernel ⇒ jupyter_client, ipykernel

The client code is now in jupyter_client, so if you were importing kernel managers, clients, or specs, they would come from jupyter_client. This includes IPython.kernel.zmq.session, which has moved to jupyter_client.session. IPython’s kernel-side code is in ipykernel.

The kernel.zmq subpackage is also removed, so anything in IPython.kernel.zmq will be top-level in ipykernel or jupyter_client, for example:

IPython.kernel.zmq.session ⇒ jupyter_client.session
IPython.kernel.zmq.kernelapp ⇒ ipykernel.kernelapp

How we did the split

We did most of the split in two stages. The first stage was to split out the subpackages, still within the main ipython/ipython repo (referred to as IPython Prime to avoid ambiguity). We did these mostly one at a time. For example, we moved IPython/html to notebook. We also added shims, so that anything that tried to import from IPython.html woud see a warning, but still work.

This allowed us to keep IPython master working (for pip install -e development installs, but not regular installs), and the tests running on Travis, to verify that we weren’t breaking the universe.

Phase II

Once all of the new packages were separated within the IPython Prime repo, the time to perform the actual split had arrived. IPython’s git repo isn’t huge, but it is over 50MB. Since simply copying the repo and removing unused files would preserve the history of all of IPython in every repo, that would mean that each repo would also start at over 50MB. We didn’t want to increase the size of a clone of IPython/Jupyter past 500MB, but we also didn’t want to lose all of the history on the migrated files. To accomplish that, we used a combination of git filter-branch and bfg to prune the history of the relocated files. We put together a few scripts that run filter-branch to exclude the history of any files not in a specified whitelist, following history across renames as best we can, using git log --follow. Since we are not running filter-branch on the original IPython repo, we felt comfortable being relatively aggressive with the cleaning, as all the true history is preserved in IPython Prime.

IPython’s build, release, and docs machinery is old and has grown many gnarly bits over time, much of which is irrelevant to the new packages, so these were not included in most of the new packages.

We split the packages in dependency order, so each new package only depended on other packages already split. After making each new repo, we would get it back to working order by adding basic setup.py, readme, license files, and .travis.yml. We would then update the contents of the new package to stop using the shim imports, which would be measured by the shim warnings. Once the tests were passing in the new repo, the package would be removed from the IPython repo. This process was repeated until the IPython repo contained only the shims.

Don’t look at me, I’m genutils

Part of what made the split possible to do in such a short time was ackowledging that there were a few common utilities that we use everywhere that we couldn’t decouple in a reasonable amount of time. During Phase I, we found a few imports of utilities that made sense while everything was in one repo, but were really unnecessary. These were handled by moving utilities into new packages, or duplicating small amounts of utility code to avoid the dependency.

The rest that we couldn’t split has been dumped into an ipython_genutils package, which should not be used by anyone for anything. This is a package that shouldn’t exist, has zero public APIs, and should be monotonically decreasing in functionality. The plan is to never add code to this repo, and to slowly move the functionality to downstream repos, or new standalone packages. Hopefully we can accomplish this relatively soon, and the genutils package can go away.

Still to do

There is actually one split left to do: we are going to split the html widgets out of notebook into ipywidgets as a standalone package, but we need to figure out some detangling first, specifically what the installation process for projects with kernel-side and client-side components. In the long term, we want to split ipython_widgets (Python-side) and jupyter_widgets (client-side js), but the messaging interface between them is still an active research project, and they cannot feasibly be decoupled at this point.

We still have some work to do migrating issues, and getting docs put together on the new repos. We have some scripts for the issue migration, and some PyCon sprinters have been helping get the docs back on their feet (thanks!). We are also going to need a lot of new docs on installation and usage. We also need to write some basic configuration migration for the new Jupyter locations of files, no longer in the .ipython directory.

Conclusion

We’re going to do our best to make this a smooth transition. One thing you can help with that is to report issues on the new repos. If you have a problem in the notebook, please report it at notebook instead of on IPython Prime. The same goes for the other repos.

I saw “Jupyter is like IPython, but language agnostic” immediately after the announcement, which is a great illustration of why the project needs to not have Python in the name anymore, since it was already language agnostic at the time. ↩︎

Jupyter Blog - IPython

Release of IPython 8.0

Moving forward by shedding weight

Moving toward pyproject.toml

Monthly Release and following NEP 29

Looking forward to your contributions

Thanks

IPython Parallel in 2021

Context

What IPython Parallel doesn’t do

What IPython Parallel does

IPython Parallel in 2021

Recent developments

IPython Parallel in the future

Enabling the JupyterLab debugger with ipykernel

What will change with ipykernel 6.0?

Try it now!

What about the future?

Acknowledgements

About the authors

Abracadabra! Bringing the magics to xeus-python

Magics

Matplotlib

Extending

Visual Debugging

Try it online

Installation

Acknowledgments

About the Author

IPython 7.0, Async REPL

A Primer on concurrency

Async in the Python world

IPython AsyncIO Integration

Other Async Libraries (aka: curio and trio integration)

Async in Notebooks (and other Jupyter Clients)

Future improvements

Release of IPython 5.5 and 6.2

New Features

Slowing down backports to 5.x

What’s next ?

Release of IPython 5.4, 6.1, and rlipython.

IPython 5.4 (and 6.1) major new features

TerminalIPythonApp.interactive_shell_class

Define _repr_mimebundle_

Execution Heuristics

Display IDs

The display function is now always available

Rlipython

IPython 6.1 and Python 3 only features

Release of IPython 6.0

Highlighted feature: Jedi completion

Sunsetting Python 2 support

Make sure you have Pip 9 or above

The Python 3 statement

Migrating to Python3-only source

Keeping a Python 2 stable branch

Migrating more code to Python3-only

Conclusion

The Continued Existence of the Emacs IPython Notebook

Release of IPython 5.0

IPython 5.0-LTS is out!

A brand new terminal interface

Jupyter Console

Long Term Support (LTS)

End of support for Python 2

Help us with the Python 3 transition

See you at SciPy!

Jupyter Ascending

Refresher on The Big Split™

Configuration

Docs

Issues

Thanks

IPython 3.2 released

The Big Split™

What it will mean

What it means now: moar repoz

Updated Imports

How we did the split

Phase II

`TerminalIPythonApp.interactive_shell_class`

Define `_repr_mimebundle_`