https://pantsbuild.org/ logo
Join Slack
Powered by
Seeing some weird behavior with `pants generate-lo...
# general
q
Seeing some weird behavior with 
pants generate-lockfiles
where it works for some members on the team but not for others. When it fails the stack trace looks like this:
Copy code
Traceback (most recent call last):
  File "/Users/plarotta/.cache/pants/named_caches/pex_root/installed_wheels/0/1bcaae0ad8c91ea651d4a15351ebecaafcec98cb4e9aa42a63c19d29c7963188/pex-2.33.2-py2.py3-none-any.whl/pex/result.py", line 105, in catch
    return func(*args, **kwargs)
  File "/Users/plarotta/.cache/pants/named_caches/pex_root/installed_wheels/0/1bcaae0ad8c91ea651d4a15351ebecaafcec98cb4e9aa42a63c19d29c7963188/pex-2.33.2-py2.py3-none-any.whl/pex/cli/command.py", line 110, in run
    return subcommand_func(self)
  File "/Users/plarotta/.cache/pants/named_caches/pex_root/installed_wheels/0/1bcaae0ad8c91ea651d4a15351ebecaafcec98cb4e9aa42a63c19d29c7963188/pex-2.33.2-py2.py3-none-any.whl/pex/cli/command.py", line 102, in _check_no_passthrough_args_and_run
    return subcommand_func(self)
  File "/Users/plarotta/.cache/pants/named_caches/pex_root/installed_wheels/0/1bcaae0ad8c91ea651d4a15351ebecaafcec98cb4e9aa42a63c19d29c7963188/pex-2.33.2-py2.py3-none-any.whl/pex/cli/commands/lock.py", line 1057, in _create
    create(
  File "/Users/plarotta/.cache/pants/named_caches/pex_root/installed_wheels/0/1bcaae0ad8c91ea651d4a15351ebecaafcec98cb4e9aa42a63c19d29c7963188/pex-2.33.2-py2.py3-none-any.whl/pex/resolve/lockfile/create.py", line 443, in create
    create_lock_download_manager = CreateLockDownloadManager.create(
  File "/Users/plarotta/.cache/pants/named_caches/pex_root/installed_wheels/0/1bcaae0ad8c91ea651d4a15351ebecaafcec98cb4e9aa42a63c19d29c7963188/pex-2.33.2-py2.py3-none-any.whl/pex/resolve/lockfile/create.py", line 101, in create
    artifact_and_project_name = source_artifacts_by_pin[pin]
KeyError: Pin(project_name=ProjectName(raw='dm_allennlp_models', validated=False, normalized='dm-allennlp-models'), version=Version(raw='2.10.3', normalized='2.10.3'))
Pin(project_name=ProjectName(raw='dm_allennlp_models', validated=False, normalized='dm-allennlp-models'), version=Version(raw='2.10.3', normalized='2.10.3'))
More details in the thread.
1. The 
dm-allennlp-models
package is a package available in our internal nexus repository. The nexus repo is accessed with credentials. The package can be installed directly if they do 
pip install ...
. So its not something with their access or with the connection. 2. We've tried a bunch of different versions of pants (2.22 - 2.24), versions of pex (default and latest 2.33) and pip (v23.x to v25.0.1) but we have the same issue in all cases.
h
Hmm, what can you discover about the environments where it does work vs those where it doesn’t?
Even if this is a “legitimate” failure, we might want a better error message than that stack trace…
q
At first I thought it was limited to the person with an Intel Mac. But it happened to others with an M1 and M4 Mac as well. Is there something I should ask them about their environment?
I can share their debug logs. I'll need to clean them up a bit because it contains passwords and stuff in our env variables also printed in there.
h
First thing to do is isolate the Pex command that Pants runs. Run Pants with 
--keep-sandboxes=on_failure
and it’ll spit out the location of the sandbox that ran the failing pex command
Then you should be able to reproduce the error with 
./__run.sh
in that sandbox
And you can add 
-vvvvvvvv
to that pex command to get very verbose output
I think that will go up to 9 `v`s
for maximum verbosity
I’m eyeballing the relevant pex code, to see if anything stands out
q
Copy code
pants --pex-verbosity=9 --print-stacktrace -ldebug --keep-sandboxes=on_failure generate-lockfiles
is this good^?
h
Ah yes, we already plumb that through, excellent
q
I've asked my colleagues for whom this command this fail to run it and share that 
__run.sh
file. I am guessing further investigation will require them to run commands from this file in isolation to understand the issue?
h
Something like that, but hopefully the verbose logs will be useful
That said: after eyeballing the pex code, this is specifically failing because Pex expects that artifact to be a “source artifact”, i.e., to be built from sources obtained from a local directory or from git. But you’re saying that this is a prebuilt wheel hosted on your internal nexus repo?
What are the exact filenames of the artifacts of that 
dm-allennlp-models
distribution on nexus? Who builds those?
Is this your code from some other repo that you build and host internally? Or are you mirroring some third-party code?
q
Yes, they are pre-built wheels in our internal nexus repo. The file name is 
dm_allennlp_models-2.10.3-py3-none-any.whl
. Because these packages don't change much they are built and published manually from a separate repo by someone on the team.
This separate repo is a fork of a public open source project but with minor modifications for our use case as this project has been archived and no longer supported.
Attached is a sanitized (mostly removed env variables for user/passwords) version of the logs when running the generate-lockfiles command. I don't have a 
__run.sh
yet but will share as soon as my colleagues respond.
pants.log
The 
__run.sh
with some env variables removed.
__run.sh
h
I notice that the filename isn’t canonical (the underscores get canonicalized to dashes). That shouldn’t matter (there are projects on PyPI that do the same thing, e.g., 
Django
) so if it does it’s a bug. Nonetheless, could you change the filename on the nexus server to the canonical 
dm-allennlp-models-2.10.3-py3-none-any.whl
to see if that changes anything?
q
Hmm, I doubt that would be the cause of the issue because the file names have always been one with underscores. Also, a bit weird that it works for some and not for others. Any way, I can try and get back to you.
h
Yeah, I’m skeptical, but it’s worth a try
22 Views
Open in Slack
PreviousNext
Powered by