https://pantsbuild.org/ logo
Join Slack
Powered by
Getting permission denied errors when trying to us...
# general
n
Getting permission denied errors when trying to use black, pylint, etc. (img 1). I was able to do this earlier on in the experimentation phase, so not sure what is happening here now. Reinstall did not fix the issue. The file clearly has executable permissions, so not sure (img 2). Tried unzipping it too because it looked like a pex archive, but unzip complains it isn't (img 3).
Curiously, I can 
vi
the file and see the contents of 
__main__.py
if that helps. When I use 
python_shebang path/to/cached/pex/file/pex
it executes fine (so I'd expect the lint goal to run just fine except for the now even more confusing execution permission error...).
e
Could you share text instead of screen shots? Those are unreadablish at full zoom.
n
Sorry, mb.
>./pants test tests/streams:streams_tests
22:24:18.35 [INFO] Completed: Building pytest.pex from pytest_default_lockfile.txt
Exception message: 1 Exception encountered:
 
ProcessExecutionFailure: Process 'Building pytest.pex from pytest_default_lockfile.txt' failed with exit code 1.
stdout:
stderr:
Failed to spawn a job for DistributionTarget(interpreter=PythonInterpreter('/ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/bin/python3.7', PythonIdentity('/ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/bin/python3.7', 'cp37', 'cp37m', 'manylinux_2_17_x86_64', (3, 7, 5)))): [Errno 13] Permission denied: 'casper/.cache/pants/named_caches/pex_root/venvs/d5ac5dc9c20a3a9ae674ee7c4b9b4a3a2b069632/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e/pex'
>ls -la casper/.cache/pants/named_caches/pex_root/venvs/d5ac5dc9c20a3a9ae674ee7c4b9b4a3a2b069632/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e/pex
lrwxrwxrwx 1 taymarti cc4827 11 Nov 13 21:22 casper/.cache/pants/named_caches/pex_root/venvs/d5ac5dc9c20a3a9ae674ee7c4b9b4a3a2b069632/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e/pex -> __main__.py
>unzip casper/.cache/pants/named_caches/pex_root/venvs/d5ac5dc9c20a3a9ae674ee7c4b9b4a3a2b069632/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e/pex
Archive: casper/.cache/pants/named_caches/pex_root/venvs/d5ac5dc9c20a3a9ae674ee7c4b9b4a3a2b069632/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e/pex
 
End-of-central-directory signature not found. Either this file is not
 
a zipfile, or it constitutes one disk of a multi-part archive. In the
 
latter case the central directory and zipfile comment will be found on
 
the last disk(s) of this archive.
note: casper/.cache/pants/named_caches/pex_root/venvs/d5ac5dc9c20a3a9ae674ee7c4b9b4a3a2b069632/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e/pex may be a plain executable, not an archive
unzip: cannot find zipfile directory in one of casper/.cache/pants/named_caches/pex_root/venvs/d5ac5dc9c20a3a9ae674ee7c4b9b4a3a2b069632/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e/pex or
       
casper/.cache/pants/named_caches/pex_root/venvs/d5ac5dc9c20a3a9ae674ee7c4b9b4a3a2b069632/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e/pex.zip, and cannot find casper/.cache/pants/named_caches/pex_root/venvs/d5ac5dc9c20a3a9ae674ee7c4b9b4a3a2b069632/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e/pex.ZIP, period.
>
Can confirm so far dependencies, dependees, count-loc and package goals work; test and lint for any backend result in this error.
e
Yeah, that's a venv and 
pex
/ 
__main__.py
are the same file - just a plain python source file. That file should have a shebang and you want to check two things: 1. The file perms. 2. The perms of the shebang binary.
It's also worth checking the perms of the 
/ms/dist/...
Interpreter listed above.
n
Yah, noticed it's a symlink, not an archive. Indeed, the python shebang has full permissions, but is a symlink to 
/ms/dist/...
which has permissions 
-rwxr-xr-x 2
. When I execute 
/ms/dist/... pex
or 
pex_python_shebang pex
, no issues. When I just do 
pex
I get the permission error. Very strange.
e
Bear with me, what do you mean by "When I just do `pex`"? You attempt to run 
./pex
directly in your shell?
n
Correct, 
pex
the symlink to 
__main__.py
in the venv. I did notice two more things that might be hints: the shebang has a -sE I didn't notice, but doesn't seem to change anything. The permissions of 
__main__.py
are 
-rwxr-xr-x
, but again this should be sufficient to execute without having to do 
python -sE __main__.py
(I continue to get permission errors just invoking as 
__main__.py
).
e
So, this is all in the context of WSL or WSL2? You're in an Ubuntu VM or similar but somehow a Microsoft Python is involved?
n
On REHL7. I experimented with the hashbangs in 
__main__.py
. When I substitute 
/ms/dist/...
directly, it works! Seems bash really hates the symlink to the interpter in the hashbang...
e
Ok, keep in mind bash is not involved in the actual execution Pants does. We're just experimenting in bash.
So, some of your venv PEXes work, some don't. Perhaps the most expedient thing to do is move aside .../named_caches/pex_root and re-run. If that all works, we can backtrack to try and divine how you got in a split broken state.
n
Yup, t's the same result if I use ksh, bash, sh, etc. I went back to one of my older installs and may have at least found a solution. In that one I let Pants figure out a suitable Python executable (it found /usr/bin/python3.6), so the hashbang was a symlink pointing to there. In this installation I set it explicitly to a Python 3.7.5 install on our AFS dist. I'll try to reinstall and just let Pants do its thing as before and see what happens. Frankly I am surprised this could be the issue though, not only because I use the AFS dist interpreter all of the time, it works w/ other symlinks, has the same permissions as the /usr/bin local isntallations, etc. I would also suspect Python 3.6 to not work w/ our codebase since it runs on 3.7, uses features there, etc. and we installed Pants wheel for 3.7 as well. I'll try to unravel everything tomorrow morning.
e
Ok, is AFS some standard term I can educate myself on? Is this the distributed file system?
If so and this is accurate, then there are probably env vars that would need to be exposed. A key thing to know is Pants absolutely locks down process environments. You must add env vars you want Pants to leak to subprocesses to an allow list:
Copy code
AFS[3] has several benefits over traditional networked file systems, particularly in the areas of security and scalability. One enterprise AFS deployment at Morgan Stanley exceeds 25,000 clients.[4] AFS uses Kerberos for authentication, and implements access control lists on directories for users and groups. Each client caches files on the local filesystem for increased speed on subsequent requests for the same file. This also allows limited filesystem access in the event of a server crash or a network outage.
n
Yes, AFS as Andrew File System. It's a different architecture to NFS, but basically the same from a client perspective (network access to files). Can you point me to the Python code where the env is setup for the subprocess? Since I have no idea, I think the easiest thing is just to copy the env of the parent and see if that works or not. So if 
forked_env
is a dict you want to pass to 
Popen
then just do 
forked_env.update(os.environ)
or something similar. To be honest, I am not sure this is really the issue though, since I can't even execute the file when invoked directly, even though I have execute/read permissions, and it works perfectly fine if I use the hashbang explicitly. Ultimately seems to be some weirdness with our filesystem that I have never encountered before, and Pants just happened to set up the right conditions to expose it.
I traced the issue by reinstalling and linting with all combinations of: 
_$PANTS_SETUP_CACHE_ in {*LOCAL*: /var/tmp/.cache, *NFS*: ~/.cache, *NFS*: $(rootdir)/../.cache}
X
 
_$PYTHON_BIN_NAME_ in *{LOCAL*: /usr/bin/python3.6, *AFS*: /ms/dist/python/PROJ/core/3.7.5/exec/bin/python, *NFS*: ~/venv/bin/python (AFS dist symlink), *NFS*: $(rootdir)/.cache/pants/setup/temp_venv/bin/python}
X
 
_named_caches_di_r in {*DEFAULT=*None -> *NFS:* ~/.cache/pants/named_caches, *CONFIGURED=NFS:* $(rootdir)/../.cache/pants/named_caches}
When named_caches_dir is fixed to the deafult (not specifying anything in pants.toml), then all other combinations work. So there appears to be no issues mixing local host, NFS and AFS storage. When I configure named_caches_dir to be one level above the project root in ../.cache/pants (where the setup cache is placed, and where I place all the other caches like mypy for the project), the problem arises. Really have no clue why changing named_caches from NFS: ~/.cache to NFS: ../.cache causes this.
Just to be clear, the problem is caused by setting 
named_caches_dir
in 
pants.toml
explicitly. It doesn't seem to matter the path or filesystem type, it always results in permission denied when attempting to execute (but with no obvious reason why, as using the generated hashbang directly to execute the file works fine). Further, does not seem to be a subprocess/environment issue as the same error occurs when trying to execute the pex directly in user mode. All symlinks and sources have at least 
r-x
permissions. The only case things work is when you do not specify it and it defaults to 
~/.cache/pants/named_caches
. Also, this is with 
local_store_dir
set as well, but that does not seem to be an issue as everything works with that set but not named_caches_dir. Interestingly, when I set named_caches_dir and but keep local_store defaulted, I get a different traceback in a lower part of the code (instead of Failed to spawn a job exception, I get the exact place the exception occurred: Line 520 in 
.../bootstrapper/pex/pex_bootstrapper.py:
`os.execv(venv_pex, [venv_pex] + sys.argv[1:])`o
Another observation: if I 
export HOME="$(echo ~/test)"
and keep defaults, Pants puts named_caches in 
~/test/pants/named_caches
as you would expect, but this time you get 
No such file or directory
error. That is because the generated shebang points to 
../venvs/short/somehash/bin/python
and only 
../venvs/short/somehash
exists (1kb blank file that "doesn't exist" when I try to open it; certainly not a python venv). So similar behavior, slightly different cause. But the pattern seems to be setting 
named_caches_dir
causes issues with the sandboxed venv in 
.../venvs/short
either A) non-executable via hashbang but still executable when invoked directly (when directly setting in config) or B) not existing at all (when using default but exporting 
$HOME
).
e
Alright - thanks for narrowing that all down. Let me see if I can repro; if so I'll file a bug and see what's going on. I'm suspicious of the relative paths though. We probably aren't handling that right I'd guess.
Hrm, I don't repro. This is Pants main:
Copy code
$ git diff
diff --git a/pants.toml b/pants.toml
index 659f5656e..c08a5bdd3 100644
--- a/pants.toml
+++ b/pants.toml
@@ -1,4 +1,5 @@
 [GLOBAL]
+named_caches_dir = "../bob"
 print_stacktrace = true
 
 # Enable our custom loose-source plugins.
Force re-doing all work:
Copy code
$ ./pants --no-process-execution-local-cache fmt lint check test src/python/pants/util::
That completes green. Then I find all venv pexes work when run directly:
Copy code
$ find -L ../bob/pex_root/venvs/s -name pex -type f | while read pex; do echo "Testing: $pex" && $pex --version && echo; done
Testing: ../bob/pex_root/venvs/s/1bfb4fab/venv/pex
pytest 6.2.5

Testing: ../bob/pex_root/venvs/s/82d69b63/venv/pex
pex, version 21.9b0

Testing: ../bob/pex_root/venvs/s/5a31133d/venv/pex
pytest 6.2.5

Testing: ../bob/pex_root/venvs/s/f09c59fd/venv/pex
pytest 6.2.5

Testing: ../bob/pex_root/venvs/s/cb6f0b2b/venv/pex
3.9.2 (flake8_2020: 1.6.0, flake8_pantsbuild: 2.0.0, mccabe: 0.6.1,
pycodestyle: 2.7.0, pyflakes: 2.3.1) CPython 3.7.12 on Linux

Testing: ../bob/pex_root/venvs/s/cd3599d7/venv/pex
Traceback (most recent call last):
  File "../bob/pex_root/venvs/s/cd3599d7/venv/pex", line 143, in <module>
    with open(arg) as fp:
FileNotFoundError: [Errno 2] No such file or directory: '--version'
Testing: ../bob/pex_root/venvs/s/8b88955e/venv/pex
mypy 0.910

Testing: ../bob/pex_root/venvs/s/a60f1695/venv/pex

                 _                 _
                (_) ___  ___  _ __| |_
                | |/ _/ / _ \/ '__  _/
                | |\__ \/\_\/| |  | |_
                |_|\___/\___/\_/   \_/

      isort your imports, so you don't have to.

                    VERSION 5.9.3


Testing: ../bob/pex_root/venvs/s/c65c1a06/venv/pex
pip 20.3.4 from /home/jsirois/dev/pantsbuild/bob/pex_root/venvs/s/c65c1a06/venv/lib/python3.6/site-packages/pip (python 3.6)

Testing: ../bob/pex_root/venvs/s/bcea3bb9/venv/pex
Traceback (most recent call last):
  File "../bob/pex_root/venvs/s/bcea3bb9/venv/pex", line 143, in <module>
    with open(arg) as fp:
FileNotFoundError: [Errno 2] No such file or directory: '--version'
Testing: ../bob/pex_root/venvs/s/3c8e08ad/venv/pex
pip 20.3.4 from /home/jsirois/dev/pantsbuild/bob/pex_root/venvs/s/3c8e08ad/venv/lib/python3.7/site-packages/pip (python 3.7)

Testing: ../bob/pex_root/venvs/s/ed9e3f5b/venv/pex
autoflake 1.4

Testing: ../bob/pex_root/venvs/s/8b0d38eb/venv/pex
docformatter 1.4

Testing: ../bob/pex_root/venvs/s/23b8d641/venv/pex
pytest 6.2.5

Testing: ../bob/pex_root/venvs/s/1806ef91/venv/pex
pip 20.3.4 from /home/jsirois/dev/pantsbuild/bob/pex_root/venvs/s/1806ef91/venv/lib/python3.7/site-packages/pip (python 3.7)
@nice-florist-55958 what version of Pants are you using again?
n
2.7.1
I just upgraded to 2.8rc5, so let me see if I can reproduce there.
8 Views
Open in Slack
PreviousNext
Powered by