Scripts in python/project have to run __init__.py for --help, support including them outside directory

[d-laub]

As far as I've tried, including scripts from inside the python/package directory slows down their help messages since they always import the package first (running everything in __init__.py). Before I had a need for Rust in my library, I could avoid this via poetry with the following layout:

project
├── src
│   ├── __init__.py
│   └── ...
├── cli.py <-- app() lives in here
├── pyproject.toml
└── README.md

The cli.py script looks like this to avoid slow imports for --help:

from typer import Typer

app = Typer()

@app.command
def main(...):
    import project
    # do stuff

if __name__ == "__main__":
    app()

And including cli:app as an entrypoint:

[tool.poetry.scripts]
project = "cli:app"

Moving to a maturin mixed project now, this breaks with the follow error:

project
├── Cargo.toml
├── python/project
│   ├── __init__.py
│   └── ...
├── cli.py <-- app() lives in here
├── pyproject.toml
├── README.md
└── src
    ├── lib.rs
    └── ...

$ project --help
Traceback...
    from cli import app
ModuleNotFoundError: No module named 'cli'

Thus, I've had to go back to including the CLI in the python/project directory and this slows down the help message responsiveness by several seconds. This is not a deal breaker, but it would be great to be able to do this. Otherwise, I'm curious if there are other workarounds to improve help message responsiveness that are compatible with maturin. Thanks!

[messense]

I'd move cli.py to project/__main__.py and use it as the entrypoint, and avoid importing too many stuffs in __init__.py.

[d-laub]

Moving cli.py to project/__main__.py fixes the error, but still runs __init__.py before the script and causes slow --help; this is fine for now. Does it make sense to move this back to as an issue though, since it's a potential enhancement? Otherwise, I don't see an easier way to get fast --help. Lazy imports are a rejected Python PEP and reducing the import time of __init__.py is a relatively high cost effort for a nice but unnecessary feature.

[messense]

It's unlikely we are going to support more alternative project layout in maturin since it's not a general purpose package builder.

Also I'm curious about the file layout in your site-packages after installing the poetry generated wheel, where did it place your cli.py? If it's in site-packages/cli.py, you're going to have trouble when another package does the same thing.

[d-laub]

After testing this in a dummy project, I found some unexpected behavior. To answer your question, the script is placed as an executable bin/project (for me this was in my conda environment ~/miniconda3/envs/test/bin/project). But, it only runs without errors when I use poetry install. Using a local pip install . fails with an ImportError, the same as with maturin. Thus, I would expect that installing from PyPI would have the same error and fail to run the script. I'll close this as I no longer know of a simple way to package scripts that defer running __init__.py for --help. Thanks for your help clarifying things along the way!