build-system.requires ignored for tool.poetry.build.script

[cavedon]

• Poetry version: 1.7.1

• Python version: 3.10.12

• OS version and name: Ubuntu 22.04

• pyproject.toml: pyproject.toml

• build.py: build.py

• I am on the latest stable Poetry version, installed using a recommended method.

• I have searched the issues of this repo and believe that this is not a duplicate.

• I have consulted the FAQ and blog for any relevant entries or release notes.

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

In my pyproject.toml I have

[tool.poetry.build]
script = "build.py"
generate-setup-file = false

[build-system]
requires = ["poetry-core", "protoc-exe"]
build-backend = "poetry.core.masonry.api"

and my build.py invokes the protoc command, which is provided by the protoc-exe module, listed in [build-system].

This works fine with poetry install --sync:

 % poetry install --sync
Installing dependencies from lock file

Preparing build environment with build-system requirements poetry-core, protoc-exe
Installing the current project: mypoetrytest (0.1.0)

But it fails with poetry build:

 % poetry build     
Preparing build environment with build-system requirements poetry-core, protoc-exe
Building mypoetrytest (0.1.0)
Traceback (most recent call last):
  File "/home/cavedon/mypoetrytest/build.py", line 27, in <module>
    build()
  File "/home/cavedon/mypoetrytest/build.py", line 17, in build
    subprocess.check_call(
  File "/usr/lib/python3.10/subprocess.py", line 364, in check_call
    retcode = call(*popenargs, **kwargs)
  File "/usr/lib/python3.10/subprocess.py", line 345, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/usr/lib/python3.10/subprocess.py", line 971, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.10/subprocess.py", line 1863, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'protoc'

Command '['/tmp/tmp0g_4j74h/.venv/bin/python', 'build.py']' returned non-zero exit status 1.

(full output with -vvv)

In the case of poetry build, the command protoc is in the bin directory of the virtualenv, but the bin directory is not added to the PATH environment variable.

Setting generate-setup-file = true does not help.

[dimbleby]

you should not use poetry run in your build script; or anyway definitely not without declaring poetry as a build requirement.

just call protoc directly - that is what is being installed in the isolated build environment.

please close

[cavedon]

you should not use poetry run in your build script; or anyway definitely not without declaring poetry as a build requirement.

just call protoc directly - that is what is being installed in the isolated build environment.

Ah, I am sorry, I did not realize I still had poetry run in my build.py. That was a left-over from my debugging. I have removed it, re-run the tests, and update the description of this issue.

duplicate #8749 please close

Thanks for pointing for that issue. Indeed it seems to be caused by the fact that the bin directory of the virtualenv is not in the PATH when build.py is executed. Let me do some tests and confirm it is indeed exactly the same issue and if so I will close as duplicate.

[cavedon]

I do not think this is a duplicate of #8749:

• I tried to patch as suggested in Poetry fails when path depedency (PEP 517 build) depends on binary #8749 (comment), but it does not help.

index 3edb225c..47ecdc51 100644
--- a/src/poetry/inspection/info.py
+++ b/src/poetry/inspection/info.py
@@ -44,9 +44,8 @@ source = '{source}'
 dest = '{dest}'
 
 with build.env.DefaultIsolatedEnv() as env:
-    builder = build.ProjectBuilder(
-        source_dir=source,
-        python_executable=env.python_executable,
+    builder = build.ProjectBuilder.from_isolated_env(
+        env, source,
         runner=pyproject_hooks.quiet_subprocess_runner,
     )
     env.install(builder.build_system_requires)

• PEP517_META_BUILD is not in the path of my failure. The issue seems to be in poetry-core, in my case.

[dimbleby]

I - almost immediately - deleted the comment referencing #8749, I also don't think this is a duplicate. Sorry for confusion.

You should set generate-setup-file = true if you want to use the isolated environment during build.

[cavedon]

Setting generate-setup-file = true fails in a similar way:

 % poetry build
Preparing build environment with build-system requirements poetry-core, protoc-exe, setuptools
Building mypoetrytest (0.1.0)
A setup.py file already exists. Using it.
Traceback (most recent call last):
  File "/home/cavedon/mypoetrytest/setup.py", line 25, in <module>
    build(setup_kwargs)
  File "/home/cavedon/mypoetrytest/build.py", line 17, in build
    subprocess.check_call(
  File "/usr/lib/python3.10/subprocess.py", line 364, in check_call
    retcode = call(*popenargs, **kwargs)
  File "/usr/lib/python3.10/subprocess.py", line 345, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/usr/lib/python3.10/subprocess.py", line 971, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.10/subprocess.py", line 1863, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'protoc'

Command '['/tmp/tmpokfeqd7d/.venv/bin/python', '/home/cavedon/mypoetrytest/setup.py', 'build', '-b', '/home/cavedon/mypoetrytest/build', '--build-purelib', '/home/cavedon/mypoetrytest/build/lib', '--build-platlib', '/home/cavedon/mypoetrytest/build/lib.linux-x86_64-cpython-310']' returned non-zero exit status 1.

However this patch solves my issue:

diff --git a/src/poetry/utils/env/__init__.py b/src/poetry/utils/env/__init__.py
index ed22bb37..a4211591 100644
--- a/src/poetry/utils/env/__init__.py
+++ b/src/poetry/utils/env/__init__.py
@@ -1,6 +1,7 @@
 from __future__ import annotations
 
 from contextlib import contextmanager
+import os
 from pathlib import Path
 from typing import TYPE_CHECKING
 
@@ -97,7 +98,12 @@ def build_environment(
                 assert io is not None
                 io.write_error_line("")
 
-            yield venv
+            orig_environ = os.environ
+            os.environ = venv.get_temp_environ(environ=os.environ)
+            try:
+                yield venv
+            finally:
+                os.environ = orig_environ
     else:
         yield env

I am happy to create a pull request if you agree with the approach.

[dimbleby]

I do not reproduce:

$ poetry build
Preparing build environment with build-system requirements poetry-core, protoc-exe
Building mypoetrytest (0.1.0)
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
Traceback (most recent call last):
  File "/home/dch/foo/setup.py", line 2, in <module>
    from setuptools import setup
ModuleNotFoundError: No module named 'setuptools'

Command '['/tmp/tmp6ijhbpq3/.venv/bin/python', '/home/dch/foo/setup.py', 'build', '-b', '/home/dch/foo/build', '--build-purelib', '/home/dch/foo/build/lib', '--build-platlib', '/home/dch/foo/build/lib.linux-x86_64-cpython-310']' returned non-zero exit status 1.

which I guess is expected while setuptools is not declared as a build requirement

when I add it the error becomes Could not make proto path relative: protobuf/myproto.proto: No such file or directory , which is to say everything is working as expected

[dimbleby]

take it back, I have a (different) protoc available in my wider environment

[dimbleby]

I don't know what the right fix is but I'm pretty sure that it's not going to be what you suggest - updating os.environ looks horrid

[cavedon]

I don't know what the right fix is but I'm pretty sure that it's not going to be what you suggest - updating os.environ looks horrid

I agree, I am a little hesitant about that. Alternatives can be:

1. passing the additional PATH all the way the subprocess calls in poetry-core builders
2. have poetry-core builders automatically add the directory containing the python executable to the PATH (is that a valid assumption that it will be in the virtualenv bin's directory?
3. have poetry-core builders source activate if such script is present in the directory containing the python executable before executing the build script

[dimbleby]

another approach would just not to try and do this via build requirements - there's at least a case that installing executables isn't really what build requirements are for

eg if you were trying to use the compiler from your build script, you wouldn't expect to be able to list gcc in the python project's build requirements.

you could just tell people that having protoc in their path is a requirement to install your package.

[cavedon]

another approach would just not to try and do this via build requirements - there's at least a case that installing executables isn't really what build requirements are for

I think it would be a reasonable expectation that if I need package X to build my module, I can add it to the build-requirements.
And indeed that's the case when building via poetry install.
The difference in behavior is also confusing.

eg if you were trying to use the compiler from your build script, you wouldn't expect to be able to list gcc in the python project's build requirements.

It would not be a horrible idea, I think

you could just tell people that having protoc in their path is a requirement to install your package.

This adds friction. I am building a pure python module that needs some processing to be built. The tool to do the processing is available as pypi package. Why not allowing users to specify that in the build-requirements instead of having to write it somewhere the user needs read and execute manually?

[abn]

Relates to python-poetry/poetry-core#319

[RobertoRoos]

I am also having problems using a binary that's installed for a build. In my example it's pyside6-uic from PySide6_Essentials. See #9743 for the original thread.

I have a pyproject.toml like this:

[build-system]
requires = [
    "poetry-core",
    "PySide6-Essentials ~= 6.7.2",
]
build-backend = "poetry.core.masonry.api"

[tool.poetry.build]
script = "build_package.py"

With build_package.py like this:

import subprocess


def main():
    print("Running `pyside6-uic --help`...")
    result = subprocess.run(
        ["pyside6-uic", "--help"],
        check=True,  # Raise exception on failure
        capture_output=True,
    )


if __name__ == "__main__":
    main()

This works during poetry install, but not during poetry build.

Note that the package itself can be imported just fine, it runs if I include import PySide6.

To debug this further, I added the following to my build script:

    subprocess.run(["cmd.exe", "/c", "echo", "%PATH%"])
    print(sys.path)
    sleep(30.0)

---

Running poetry install -v now shows:

C:\Temp\tmptc2h5ep9.venv\Scripts;C:\Program Files\PowerShell\7; ...
['C:\Users\ROQ\PycharmProjects\pythonProject1', 'C:\Program Files\Python311\python311.zip', 'C:\Program Files\Python311\DLLs', 'C:\Program Files\Python311\Lib', 'C:\Program Files\Python311', 'C:\Temp\tmptc2h5ep9\.venv', 'C:\Temp\tmptc2h5ep9\.venv\Lib\site-packages']

And (while the install command is hanging) I see the file C:\Temp\tmptc2h5ep9\.venv\Scripts\pyside6-uic.exe does exist.

---

Running poetry build -f wheel -v shows:

C:\Program Files\PowerShell\7; ...
['C:\Users\ROQ\PycharmProjects\pythonProject1', 'C:\Program Files\Python311\python311.zip', 'C:\Program Files\Python311\DLLs', 'C:\Program Files\Python311\Lib', 'C:\Program Files\Python311', 'C:\Temp\tmpuvxz5_s0\.venv', 'C:\Temp\tmpuvxz5_s0\.venv\Lib\site-packages']

There is no Scripts/ directory, but the file C:\temp\tmpuvxz5_s0\.venv\Scripts\pyside6-uic.exe exists.

---

Finally, I noticed something interesting. Installing pyside6-essentials to the venv for this project and running poetry build ... with the venv activated does make the build succeed.

PATH printed from build_package.py when running from the venv looks like:

C:\temp\tmpd7pgbi6j.venv\Lib\site-packages\PySide6;C:\Users\ROQ\PycharmProjects\pythonProject1.venv/Scripts;C:\Program Files\PowerShell\7;...

So the binary from the project venv ends up being used.

But activating the venv is essential here. Doing poetry add ... ; poetry build ... without activating the venv still gives the same errors as above.

[wtdcode]

Having the same issue. What's the current status?