Chapel output does not propagate to Jupyter notebook cells

[ben-albrecht]

Summary of Problem

When compiling a Chapel library as a python module and calling it from a Jupyter notebook, any writes to stdout or stderr do not propagate to the Jupyter cell notebook.

IPython (which Jupyter uses) redirects the sys.stdout and sys.stderr python objects (source) to their custom objects. However, Chapel libraries write to their own stdout/stderr file descriptors, therefore IPython (and Jupyter) is unable to propagate that output.

There is a known user-end work-around where users can capture the output/stderr with the wurlitzer package. However, this is a bit cumbersome to use, and buffers all the output until the external library call is completed.

Steps to Reproduce

Source Code:

export proc greet() {
  stdout.writeln('hello');
  stderr.writeln('world');
}

Compile command:

chpl --library-python hello.chpl

Execution command:

Using Anaconda 3.6.5:

cd lib/
jupyter notebook .

Inside a Jupyter notebook:

import hello
hello.chpl_setup()
hello.greet()
hello.chpl_cleanup()

The output shows up in the console output among the Jupyter logging output rather than the notebook output, e.g.

[I 09:01:43.721 LabApp] Creating new notebook in /
[I 09:01:43.867 LabApp] Kernel started: fd1145a8-5d01-4273-b312-bd1c5bbf9381
[I 09:01:44.359 LabApp] Adapting to protocol v5.1 for kernel fd1145a8-5d01-4273-b312-bd1c5bbf9381
[I 09:01:44.359 LabApp] Adapting to protocol v5.1 for kernel fd1145a8-5d01-4273-b312-bd1c5bbf9381
[W 09:01:46.506 LabApp] 404 GET /api/kernels/b3ecf8bd-0f20-4669-9f1d-f1efa5f828d5/channels?session_id=6e9b3e24dfebb1bce3e1de744612b489&token=69873e10409e80674fcb933a0940ce929457cca32d85c49e (::1): Kernel does not exist: b3ecf8bd-0f20-4669-9f1d-f1efa5f828d5
[W 09:01:46.533 LabApp] 404 GET /api/kernels/b3ecf8bd-0f20-4669-9f1d-f1efa5f828d5/channels?session_id=6e9b3e24dfebb1bce3e1de744612b489&token=69873e10409e80674fcb933a0940ce929457cca32d85c49e (::1) 29.12ms referer=None
hello
world
[W 09:02:02.569 LabApp] Replacing stale connection: b3ecf8bd-0f20-4669-9f1d-f1efa5f828d5:6e9b3e24dfebb1bce3e1de744612b489

Configuration Information

• Output of chpl --version: 1.19.0 pre-release (9bfff1f9fa)

[ben-albrecht]

Based on ipython/ipykernel#110, it seems like this remains an open issue for Jupyter in general.

For example, TensorFlow is currently wrestling with how to address this issue: tensorflow/community#14 (comment).

[ben-albrecht]

Marked as [user issue] due to a number of users reporting this through separate channels.