pip-compile expands environment variables

[bo5o]

Environment variables get expanded by pip-compile. This is especially unwanted when the variable describes a password or local path.

Environment Versions

1. Ubuntu 18.04
2. Python version: 3.7.4
3. pip version: 19.3.1
4. pip-tools version: 4.2.0

Steps to replicate

requirements.in

-e ${MY_PROJECT_DIR}/myapp
-e git+https://${USER}:${PASSWD}@gitserver/myapp.git

Expected result

requirements.txt

-e ${MY_PROJECT_DIR}/myapp
-e git+https://${USER}:${PASSWD}@gitserver/myapp.git#egg=myapp

Actual result

The requirements.txt contains the expanded variables.

[atugushev]

Hello @cbows,

Thanks for the issue! How it works on pip side:

• pip expands the variables during parse a requirements file
• substitutes variables by values on the fly

Thus, there is no way to distinguish whether the credentials come from env vars or not using pip API. Possible solutions could be:

• match lines in requirements.in with parsed URL, and inject there variables
• add some heuristics: guess whether the credentials come from env by searching env variables by values

Don't like either of them, though, and honestly, I'd be intuitively against implementing such things outside of pip API.

Consider the alternatives:

• ssh keys
• netrc
• a kung-fu with GIT_ASKPASS, see gitcredentials

[bo5o]

Hello @atugushev,

I agree there is no clean solution outside of pip. I guess for credentials there are, as you mentioned, ways around it, and they are probably better.

This still leaves us with local paths (or any path really) though;
Let's say you want to specify a base directory for some app -e ${MY_PROJECT_DIR}/myapp that varies from machine to machine (could be $HOME for development). This gives you the full path in requirements.txt, which is not distributable.

Maybe there is a way to parse only file:// urls and see if they come from env var?
Heuristics could be to take a diff of the expanded and original line in requirements and see if the diff matches exactly a value in environment?

[atugushev]

@cbows

Maybe there is a way to parse only file:// urls and see if they come from env var?
Heuristics could be to take a diff of the expanded and original line in requirements and see if the diff matches exactly a value in environment?

Maybe... It could be quite tricky, I imagine, and I think we need to fix #204 first 😊

[AndydeCleyre]

@cbows Once #204 is fixed (there is currently a PR pending review), you could avoid using environment variables for local paths by using symlinks and relative paths, like

-e ./myapp

Where myapp is a link to wherever.