Embed IPython in qtconsole

[llchan]

I have a script that loads/processes some data and gives the user an IPython.embed() session to explore that dataset. I'd like to convert this to a qtconsole instead so they can get inline plots. What's involved in setting this up? I saw the inprocess_qtconsole.py example, but it's not immediately obvious how to hook it up to the existing process so that variables are exposed.

[takluyver]

I wouldn't recommend using the inprocess machinery, because that's usually problematic. You probably want to use the embed_kernel() function, and then start up the Qt console in a separate process to talk to the embedded kernel. The tricky bit is figuring out how to start the Qt console after the kernel has started. I'm not exactly sure how best to add a callback to the kernel's event loop before it starts.

[llchan]

Thanks for the pointers. Here's what I have so far, and it seems to at least get me into a working kernel in qtconsole:

import IPython
import errno
import multiprocessing
import os
import sys
import tempfile
import time

from qtconsole.client import QtKernelClient
from qtconsole.qt import QtGui
from qtconsole.rich_jupyter_widget import RichJupyterWidget


def run_embedded_qtconsole(connection_file):
    for i in xrange(100):
        try:
            st = os.stat(connection_file)
        except OSError as exc:
            if exc.errno != errno.ENOENT:
                raise
        else:
            if st.st_size > 0:
                break
        time.sleep(0.1)

    app = QtGui.QApplication([])

    kernel_client = QtKernelClient(connection_file=connection_file)
    kernel_client.load_connection_file()
    kernel_client.start_channels()

    def exit():
        # FIXME: tell the kernel to shutdown
        kernel_client.shutdown()
        kernel_client.stop_channels()
        app.exit()

    ipython_widget = RichJupyterWidget()
    # ipython_widget.kernel_manager = kernel_manager
    ipython_widget.kernel_client = kernel_client
    ipython_widget.exit_requested.connect(exit)
    ipython_widget.show()

    app.exec_()


def embed_qtconsole():
    connection_file = os.path.join(
        tempfile.gettempdir(),
        'connection-{:d}.json'.format(os.getpid()))

    try:
        p = multiprocessing.Process(
            target=run_embedded_qtconsole,
            args=(connection_file,),
        )
        p.start()

        IPython.embed_kernel(
            local_ns=sys._getframe(1).f_locals,
            connection_file=connection_file,
            gui='qt4',
        )

        p.join()

    finally:
        try:
            os.unlink(connection_file)
        except OSError as exc:
            if exc.errno != errno.ENOENT:
                raise


def main():
    x = 123
    embed_qtconsole()


if __name__ == "__main__":
    main()

What do I need to do to tell the embedded kernel to shutdown?

[takluyver]

Nice.

Shutdown is a bit complex, and I forget the exact details of how it works. There are two routes to shutdown: the user can try to close the Qt console window, in which case it should ask the kernel to shutdown (shutdown_request), wait to see that it does, and then exit itself. Or you can type exit into the kernel, in which case the kernel sends the frontend a message, and then the procedure above follows.

[llchan]

I'm working on getting the exit in kernel route working first (should be similar in both cases). After some trial and error, looks like if I remove the gui='qt4' flag from embed_kernel, then it sort of works. Instead of returning from the embed_kernel function though, it exits the process directly. So that confirms that the kernel is at least getting the shutdown message, but in 'gui=qt4' mode it's not shutting down properly. Any ideas why embed_kernel would be ignoring the shutdown message in that mode? Is this more appropriate for the ipykernel repo now that I have the qtconsole attached to the embedded kernel?

[takluyver]

It's probably the same people seeing it whichever repo it's on, so let's continue here unless/until we identify a bug in something else.

First off, do you need gui='qt4'? Your initial post described it as a 'script' you wanted to embed it in. That option is not necessary to use the Qt console, because the Qt code runs in a separate process. If your script has a Qt gui itself, though, you probably want that to keep running the Qt event loop while the kernel runs.

When the kernel gets a shutdown_request, it stops the tornado event loop which runs the kernel. It's possible if that's integrated with a Qt event loop, the Qt event loop carries on running. But I'm not quite sure what's going on there.

[llchan]

I only had the gui=qt4 because I saw it elsewhere in example code, but if that's only needed for a Qt main loop, you're right I don't need it. However, when the secondary qtconsole process comes up, the console is empty and it requires a manual return to show the prompt. Is there a way to get that happen automatically?

If the shutdown request is simply stopping an event loop, I'd expect it to return to the embed_qtconsole function afterwards, but it seems that it exits the process directly, which might be a bug with embed_kernel shutdown handling.

[llchan]

Oh, I think I figured out what it is: embed_kernel does indeed shut down and return to the call site, but stdout is left intercepted, so my debug prints were actually happening, they were just being sent nowhere. Probably just need to have embed_kernel reset the sys streams.

[llchan]

I think this just requires two changes:

1. Add a IPKernelApp reset or teardown func that resets the stuff that happens during initialize.
2. In embed_kernel, if IPKernelApp was newly initialized within the func, call the reset func after start returns (probably in a finally clause).

[takluyver]

Can you have a look at ipython/ipykernel#221? I just remembered it, and I think it's talking about something similar.

[takluyver]

Needing a return to show the prompt is a bug that I've seen sometimes just running the Qt console normally. I forget if there's already an issue about it - @wmvanvliet might know more.

[wmvanvliet]

There might still be some bugs lurking around concerning the proper display of the prompt. If we have a reproducible example, I'm happy to investigate.

[ccordoba12]

There is one (on Linux at least): running the %qtconsole magic in the notebook always open a blank console. You have to press Enter on it to get a prompt.

[llchan]

@takluyver Yes, ipython/ipykernel#221 looks like it's very similar in requesting proper teardown. That would be a superset of what I mentioned here (just sys streams).

As for the qtconsole not showing a prompt, I believe that's an orthogonal issue. You can reproduce very easily:

$ ipython kernel

and then in another shell

$ juypter qtconsole --existing kernel-XXXXX.json

[wmvanvliet]

@ccordoba12 @takluyver both scenario's work just fine for me on the latest master. Banner and prompt are both shown properly.

[wmvanvliet]

ah! but it breaks on the latest 4.2.1 release. Ok then, looks like this issue was fixed somewhere.

[llchan]

@wmvanvliet thanks for verifying that it's fixed.

Since the original issue here is more or less resolved by embed_kernel() + a qtconsole child process, I'll just close this issue and follow ipython/ipykernel#221. Thanks everyone!