Make kernel and shell cleanup easier.

[mdickinson]

[Related: #219]

This is a feature request, aimed at benefiting applications that:

1. embed an IPython kernel, and
2. have good unit tests.

We have such an application, and we're finding it painful to reset the main Python process to a sensible state after running a unit test that creates, uses, and then destroys an IPython kernel (specifically, an InternalIPKernel). The main issues we're experiencing are thread leaks and file descriptor leaks; currently our test suite leaks 22 file descriptors (related to the ZMQ sockets being used) per IPython-using unit test; which leads to the OS running out of file handles early on in the test suite. More details below.

It would be very useful to have a reasonably simple, documented way to release OS resources that have been acquired by a kernel / shell / frontend client.

I guess this is rather a niche use-case (I'd hope that anyone doing (1) above is also doing (2), but that's probably a bit naïve :-), so I understand if it's not something that the IPython devs want to pursue, but it seemed worth at least registering that some of us have encountered this need, and documenting some of the specific issues encountered. (Also, I can't help wondering how IPython's own test suites avoid running into these sort of issues.)

For reference, here are some of the specific issues that we're running into:

1. After tests that create a kernel, shell and qtconsole client, we end up with a singleton ZMQInteractiveShell object. That object has a configurables attribute (inherited from IPython.core.InteractiveShell) that's often appended to, but AFAICT never has items removed from it. In particular, that list includes methods related to each front end created over the course of the test suite, and so keeps all those front end objects and all their associated resources alive.

2. The ZMQ sockets rely on reference-counting (via a __del__ method) for freeing the underlying Unix sockets; most don't seem to be closed explicitly at any point. So when objects are being kept alive, those sockets are kept alive, too.

3. PySide plays particularly badly with reference-count based cleanup, since PySide habitually creates objects with artificially inflated reference counts, which then become uncollectable. This is biting us particularly with the qtconsole. This is very much a PySide bug (and we'll probably eventually migrate to PyQt precisely to avoid this sort of problem), but having a way to do explicit cleanup within IPython would at least allow us to work around that bug.

4. Under Python 2 (which unfortunately is what we're stuck on for the forseeable future for this app), some objects are truly uncollectable, ending up in gc.garbage. The ScriptMagics object seems to be one such: we end up with a cycle containing those objects, and the fact that they have a __del__ method means that they never get collected (and also that the cleanup that the __del__ method is supposed to perform never happens). And again, that uncollectable garbage is keeping the ZMQInteractiveShell object alive.

5. So the upshot of the above is that the ZMQInteractiveShell object keeps many Python objects and OS resources alive. If we want to fix that, our options at this point are: (1) try to clean up that shell, or (2) try to remove all references to the shell. (1) seems very involved, and I guess is the point of this issue: it would be great to have some kind of one-line method call that cleans everything up. I tried to go with (2), but that's also painful: there are direct and indirect references to the shell in many places: various singletons, module-level variables, atexit handlers, sys-module attributes. To give some idea, here's a snippet from our current tearDown code:

        # Reset IPython global instances
        IPKernelApp.clear_instance()
        IPythonKernel.clear_instance()
        ZMQInteractiveShell.clear_instance()
        InlineBackend.clear_instance()
        IOLoop.clear_instance()

        del IPython.utils.io.stdout.session
        del IPython.utils.io.stderr.session
        del IPython.utils.io.stdout.stream
        del IPython.utils.io.stderr.stream

        del IPython.utils.io.stdout
        del IPython.utils.io.stderr

        # Reset global zmq context (calling destroy does not do it !
        zmq.Context._instance = None

        # Reset exception hook. (Is there something in the IPython machinery
        # that does this?)
        sys.displayhook = sys.__displayhook__
        sys.excepthook = sys.__excepthook__

        import matplotlib.backends.qt_editor.formlayout

        del matplotlib.backends.qt_editor.formlayout.STDERR.session

        # Remove atexit handlers.
        import atexit
        while atexit._exithandlers:
            handler = atexit._exithandlers.pop()
            del handler

Any help with doing this sort of cleanup (even if it's just documentation on how to do it properly, or pointers to other places where it's done) would be very valuable.

[mdickinson]

so I understand if it's not something that the IPython devs want to pursue

OTOH, if it is something that the IPython folks think is worth pursuing, I'm happy to contribute tests, code, PR reviews, etc.

[minrk]

Thanks, @mdickinson! I think this is worthwhile, and we can start with a teardown method (or similar) on InteractiveShell, and probably a similar one on Kernel in this repo.

This hasn't been a priority for how we've done things thus far, since shells typically live for the duration of a process for us, but I think we should do it anyway, for the reasons you give above.

[chrisjbillington]

+1 for this. I use IPython.embed() for debugging often, and IPKernelApp (connecting to it with a qtconsole) for debugging something that doesn't have a stdin. In both cases there are side effects (that generally for the IPKernelApp prevent me from using it again at a later time in the same interpreter) and sometimes interact with my own code in undesirable ways. It would great to have a proper cleanup, as well as use of zmq objects that isn't interpreter-wide. It seems like .instance() is used a lot in the IPython code such that if I call context.term() on a zmq context I'll also take down other parts of a running app, which is undesirable.