Python3 TemporaryFile().truncate() fails in WSL2 (breaking pytest)

[DerekParks]

Environment

Windows build number: Microsoft Windows [Version 10.0.19041.630]
Your Distribution version: Ubuntu  18.04
Whether the issue is on WSL 2 and/or WSL 1: WSL 2 - Linux version 4.19.128-microsoft-standard (oe-user@oe-host) (gcc version 8.2.0 (GCC)) #1 SMP Tue Jun 23 12:58:10 UTC 2020

Steps to reproduce

Creating a temp file in python and calling truncate on it doesn't work in WSL2. This came up when running a set of pytests that work in a linux vm but don't work on WSL2.

$ python3 -c "import tempfile; tempfile.TemporaryFile().truncate()"

I can provide an strace if needed but I'm hoping it will be easy enough to reproduce.

Expected behavior

In a ubuntu 18.04 vm, the above command succeeds without error.

Actual behavior

On WSL2 I get the following error.

$ python3 -c "import tempfile; tempfile.TemporaryFile().truncate()"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
FileNotFoundError: [Errno 2] No such file or directory

[therealkenc]

That error is from userspace; you are on a VM in WSL2. A gist to a threaded strace(1) log would show what file is missing.

[DerekParks]

@therealkenc What version of python are you using? I was using 3.6. Maybe I need to upgrade.

Either way, it looks like my issue not WSL's. Feel free to close. Sorry to trouble.

[therealkenc]

No trouble. I'll move over to discussion. Python version is probably not it.

[DerekParks]

Here is my gist: https://gist.github.com/DerekParks/3261d61e96372d81616346e8bd532b16

Any ideas on what is wrong with my environment would be appricated.

[therealkenc]

Thanks for the gist.

26184 openat(AT_FDCWD, "/home/dparks/opt/anaconda3/envs/allocation-env/lib/python36.zip/<string>", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
26184 openat(AT_FDCWD, "/home/dparks/opt/anaconda3/envs/allocation-env/lib/python3.6/<string>", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
26184 openat(AT_FDCWD, "/home/dparks/opt/anaconda3/envs/allocation-env/lib/python3.6/lib-dynload/<string>", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
26184 openat(AT_FDCWD, "/home/dparks/.local/lib/python3.6/site-packages/<string>", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
26184 openat(AT_FDCWD, "/home/dparks/opt/anaconda3/envs/allocation-env/lib/python3.6/site-packages/<string>", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

I don't really 'do' Python, but you've got a local installation looking for <string> that is not there. Walk back whatever is in your local configuration, likely lives in .bashrc. Big hammer fix would be to start over with a new distribution install, say Ubuntu-20.04, and then walk forward instead of back.

[awanwar]

Sorry to revive this old thread, but I'm facing the same issue running python3 -c "import tempfile; tempfile.TemporaryFile().truncate()" with a fresh install of Ubuntu-22.04 - no other packages installed or environment changes to my knowledge. Here's the gist file: https://gist.github.com/awanwar/e2e3771ac23c51d613a52f3c1cb99a30

It seems like a WSL2 issue or an issue with Python interacting with WSL2 to me. I tried running the same command in a regular Windows 10 environment (although an unclean one) and there were no issues.

[elsaco]

Tested with python3 on Ubuntu 23.04 (preview) and couldn't repro:

elsaco@dragon:~$ python3
Python 3.11.1 (main, Dec 31 2022, 10:23:59) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tempfile
>>> tempfile.TemporaryFile().truncate()
0
>>> quit()
elsaco@dragon:~$ python3 -c "import tempfile; tempfile.TemporaryFile().truncate()"
elsaco@dragon:~$ echo $?
0

---

Looking at your gist this might be of interest:

newfstatat(AT_FDCWD, "/home/test/-c", 0x7ffdda3bb8b0, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
access("/home/test/-c", X_OK)           = -1 ENOENT (No such file or directory)

Try running the command inside a python session. You'll get more feedback if you enable the python debugger (import pdb) also.

[zamar-roura]

Hi! I have digged deep into this problem and found a workaround solution.

Maybe someone with higher expertise in core python modules will change it one day. The short story is that in WSL (The kernel that the linux images run on) files work differently, and files cannot be unlinked (deleted) while they are open by the process, but right now the tempfile.py thinks we are in pure LINUX os and platform, while our files in docker should work WINDOWS like because we are in WSL.

This is the bit that created the use case problem inside tempfile.py

if _os.name != 'posix' or _sys.platform == 'cygwin'
    # On non-POSIX and Cygwin systems, assume that we cannot unlink a file
    # while it is open.
    TemporaryFile = NamedTemporaryFile

else:
    .....

The workaround I have found is to include this:

def is_wsl():
    try:
        with open("/proc/version", "r") as f:
            version_info = f.read().lower()
            return "microsoft" in version_info
    except FileNotFoundError:
        return False

if _os.name != 'posix' or _sys.platform == 'cygwin' or is_wsl():
    # On non-POSIX and Cygwin systems, assume that we cannot unlink a file
    # while it is open.
    TemporaryFile = NamedTemporaryFile

else:
   ....

And running a little script it for sure works:

import tempfile
import os

def demonstrate_temp_file_ops():
    with tempfile.TemporaryFile() as f:
        # Initial state
        print(f"1. Initial position: {f.tell()}")

        # Write operation
        f.write(b'test')
        print(f"2. After write position: {f.tell()}")
        print(f"2a. Content length: {os.fstat(f.fileno()).st_size}")

        # Seek and truncate
        f.seek(0)
        print(f"3. After seek position: {f.tell()}")

        f.truncate()  # Truncates at current position (0)
        print(f"4. After truncate size: {os.fstat(f.fileno()).st_size}")

        # Verify empty file
        f.seek(0)
        print(f"5. Final content: {f.read()}")

if __name__ == "__main__":
    demonstrate_temp_file_ops()

Now the tough part is to get this running smoothly.... I would either create a tempfile.py with this logic and replace it, but if there is any new version on tempfile.py you will miss it...

If you just want a quick solution to your problem add this code to your Dockerfile (pick the part you need and make sure you are creating the tempfile in your python.X.X)

FROM python:3.8

# Append the `is_wsl` function definition to tempfile.py
RUN echo "def is_wsl():\n    try:\n        with open('/proc/version', 'r') as f:\n            version_info = f.read().lower()\n            return 'microsoft' in version_info\n    except FileNotFoundError:\n        return False\n" >> /usr/local/lib/python3.8/tempfile.py

# Replace the condition using sed
RUN sed -i "s/if _os.name != 'posix' or _sys.platform == 'cygwin'/if _os.name != 'posix' or _sys.platform == 'cygwin' or is_wsl()/" /usr/local/lib/python3.8/tempfile.py

# Verify the change
RUN grep "is_wsl" /usr/local/lib/python3.8/tempfile.py


CMD ["tail", "-f", "/dev/null"]

Hope this helps anyone, or it helps someone mantaining tempfile.py to create a solution for WSL