https://pantsbuild.org/ logo
#general
Title
# general
b

bored-glass-58755

08/31/2022, 1:29 PM
Is there a way to check what
pants.toml
options are in effect during a particular run of Pants e.g. if I am running
./pants package
, I would like to see what
python-repos
options are active.
e

enough-analyst-54434

08/31/2022, 1:35 PM
The most sure footed way I can think of is adding
-ldebug
to the Pants command line. This prints out the full set of environment variables and argv for every process Pants invokes.
f

fast-nail-55400

08/31/2022, 1:35 PM
You can also run
./pants help python-repos
and the configured value should be listed next to every option.
b

bored-glass-58755

08/31/2022, 1:37 PM
Thanks @enough-analyst-54434 and @fast-nail-55400. I am going to try both of these.
e

enough-analyst-54434

08/31/2022, 1:41 PM
Tom's method is what you want to normally do. If it turns out that method doesn't give an answer that corresponds to behavior you're observing, the -ldebug will help sort that out. We've had some churn recently in options affecting repos and I'm not confident in the status of the plumbing to various resolves. The thrash may have all been in ideas though with pen never actually hitting paper.
b

bored-glass-58755

08/31/2022, 1:46 PM
Ok, so here's what
./pants help python-repos
outputs:
Copy code
`python-repos` subsystem options
--------------------------------

External Python code repositories, such as PyPI.

These options may be used to point to custom cheeseshops when resolving requirements.
 
Activated by pants.core
Config section: [python-repos]
 
None available.
Advanced options available. You can list them by running ./pants help-advanced python-repos.
It doesn't show me anything even though I have the env variable
PANTS_PYTHON_REPOS_INDEXES
set in my CI along with the option in
pants.toml
.
e

enough-analyst-54434

08/31/2022, 1:49 PM
It's probably an "advanced" option, although that classification is odd. Although its probably advanced in a global sense, the minute you're in the scope of the
python-repos
subsystem, those options are bread and butter. Try
./pants help-advanced python-repos
b

bored-glass-58755

08/31/2022, 2:00 PM
Ok, so here's the output:
Copy code
`python-repos` subsystem advanced options
-----------------------------------------
 
  --python-repos-repos="['<str>', '<str>', ...]"
  PANTS_PYTHON_REPOS_REPOS
  repos
      default: []
      current value: []
      URLs of code repositories to look for requirements. In Pip and Pex, this option
      corresponds to the `--find-links` option.

  --python-repos-indexes="['<str>', '<str>', ...]"
  PANTS_PYTHON_REPOS_INDEXES
  indexes
      default: [
          "<https://pypi.org/simple/>"
      ]
      current value: [
          "<http://private.repo.url/simple>",
          "<http://private.repo.url/simple>"
      ] (from pants.toml, from env var PANTS_PYTHON_REPOS_INDEXES)
      URLs of code repository indexes to look for requirements. If set to an empty list, then
      Pex will use no indices (meaning it will not use PyPI). The values should be compliant
      with PEP 503.


`python-repos` subsystem deprecated options
-------------------------------------------
 
None available.
The default is still set to pypi, could that be a problem?
e

enough-analyst-54434

08/31/2022, 2:01 PM
No, the default will always be that.
Rather, yes, it could be a problem, but only if Pants has bugs. The -ldebug info will help get to the bottom of that if
current value
looks good to you but behavior of Pants indicates that's not what is being used.
b

bored-glass-58755

08/31/2022, 2:04 PM
The behavior suggests:
Copy code
pex.result.ResultError: There were 2 errors downloading required artifacts:
1. setuptools 57.5 from <https://files.pythonhosted.org/packages/4b/b9/71687c5d2034c863db1db2e0704f5e27581ff3cb44d7f293968c5e08ceb3/setuptools-57.5.0-py3-none-any.whl>
    <urlopen error [Errno 97] Address family not supported by protocol>
2. wheel 0.37.1 from <https://files.pythonhosted.org/packages/27/d6/003e593296a85fd6ed616ed962795b2f87709c3eee2bca4f6d0fe55c6d00/wheel-0.37.1-py2.py3-none-any.whl>
    <urlopen error [Errno 97] Address family not supported by protocol>
Here's the stack trace:
Copy code
Engine traceback:
  in select
  in pants.core.goals.package.package_asset
  in pants.backend.python.goals.setup_py.package_python_dist (client:ts-client)
  in pants.backend.python.util_rules.dists.run_pep517_build
  in pants.backend.python.util_rules.pex.create_venv_pex (build_backend.pex)
  in pants.backend.python.util_rules.pex.build_pex (build_backend.pex)
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/$XDG_CACHE_HOME/pants/setup/bootstrap-Linux-x86_64/2.12.0_py37/lib/python3.7/site-packages/pants/engine/process.py", line 277, in fallible_to_exec_result_or_raise
    process_cleanup=process_cleanup.val,
pants.engine.process.ProcessExecutionFailure: Process 'Building build_backend.pex from setuptools_default.lock' failed with exit code 1.
And a part snapshot (redacted) of the debug:
Copy code
09:40:31.51 [DEBUG] Running Building build_backend.pex from setuptools_default.lock under semaphore with concurrency id: 2, and concurrency: 8
09:40:31.51 [INFO] Starting: Building build_backend.pex from setuptools_default.lock
09:40:31.51 [DEBUG] Starting: setup_sandbox
09:40:31.53 [DEBUG] Completed: setup_sandbox
09:40:31.57 [DEBUG] spawned local process as Some(122689) for Process { argv: ["/usr/local/python/3.7.6/bin/python3.7", "./pex", "--tmpdir", ".tmp", "--jobs", "8", "--python-path", "/protoc/3.20.1/bin:/redis/6.2.5/bin:/bazel/4.0.0/bin:/golang/1.15/bin:/venv/bin:/nodejs/v12/bin:/usr/local/java/openjdk_8u302_x64/bin:/root/usr/bin:/usr/local/ext/cmake/3.17.1/bin:/maven/3.6.3/bin:/usr/local/ext/git/2.24.0/bin:/usr/local/ext/git/2.24.0/libexec/git-core:/usr/local/perl/5.26.1/bin:/sonar/wrapper/6.32/bin:/sonar/scanner/4.7.0/bin:/helm/3.6.1:/dotnet/3.1:/usr/local/sybase/bin:/etc:/bin:/etc:bin:/usr/bin:/bin", "--output-file", "build_backend.pex", "--no-emit-warnings", "--manylinux", "manylinux2014", "--venv", "--seed", "verbose", "--venv-site-packages-copies", "--python", "/usr/local/python/3.7.6/bin/python3.7", "--sources-directory=source_files", "--lock", "setuptools_default.lock", "--no-pypi", "--index=<http://private.repo.url/simple>", "--index=<http://private.repo.url/simple>", "--layout", "packed"], env: {"CPPFLAGS": "", "LANG": "en_US.UTF-8", "LC_ALL": "en_US.UTF-8", "LDFLAGS": "", "PATH": "/protoc/3.20.1/bin:/redis/6.2.5/bin:/bazel/4.0.0/bin:/golang/1.15/bin:/venv/bin:/nodejs/v12/bin:/usr/local/java/openjdk_8u302_x64/bin:/root/usr/bin:/usr/local/ext/cmake/3.17.1/bin:/maven/3.6.3/bin:/usr/local/ext/git/2.24.0/bin:/usr/local/ext/git/2.24.0/libexec/git-core:/usr/local/perl/5.26.1/bin:/sonar/wrapper/6.32/bin:/sonar/scanner/4.7.0/bin:/helm/3.6.1:/dotnet/3.1:/usr/local/sybase/bin:/etc:/bin:/etc:/bin:/usr/bin:/bin", "PEX_IGNORE_RCFILES": "true", "PEX_ROOT": ".cache/pex_root"}, working_directory: None, input_digests: InputDigests { complete: DirectoryDigest { digest: Digest { hash: Fingerprint<0fe2393a7ac8d77f0c987b7edd9962b1c7c9e23264ca982596b13a0c7f446830>, size_bytes: 340 }, tree: "Some(..)" }, nailgun: DirectoryDigest { digest: Digest { hash: Fingerprint<e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855>, size_bytes: 0 }, tree: "Some(..)" }, input_files: DirectoryDigest { digest: Digest { hash: Fingerprint<0fe2393a7ac8d77f0c987b7edd9962b1c7c9e23264ca982596b13a0c7f446830>, size_bytes: 340 }, tree: "Some(..)" }, immutable_inputs: {}, use_nailgun: [] }, output_files: {}, output_directories: {RelativePath("build_backend.pex")}, timeout: None, execution_slot_variable: None, concurrency_available: 10, description: "Building build_backend.pex from setuptools_default.lock", level: Info, append_only_caches: {CacheName("pex_root"): RelativePath(".cache/pex_root")}, jdk_home: None, platform_constraint: Some(Linux_x86_64), cache_scope: Successful }
09:40:33.88 [INFO] Completed: Building build_backend.pex from setuptools_default.lock
09:40:33.88 [DEBUG] Completed: Scheduling: Building build_backend.pex from setuptools_default.lock
09:40:33.90 [DEBUG] Completed: pants.backend.python.util_rules.pex.build_pex
09:40:33.90 [DEBUG] Completed: `package` goal
09:40:33.90 [DEBUG] computed 1 nodes in 25.237837 seconds. there are 945 total nodes.
09:40:33.90 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Building build_backend.pex from setuptools_default.lock' failed with exit code 1.
One thing that concerns me in debug output is why
PEX_ROOT
is set as
./cache/pex_root
where as I set it as
$XDG_CACHE_HOME/pants/named_caches/pex_root
at the beginning of the CI. Also, the 4th line from last in debug output suggests that package goal is complete whereas according to the stack trace its not.
e

enough-analyst-54434

08/31/2022, 2:48 PM
That
PEX_ROOT
is a relative path in the sandbox to a symlink that points to
$XDG_CACHE_HOME/pants/named_caches/pex_root
. No matter how you configure `PEX_ROOT`externally, Pants will always use that local sandbox path with a symlink to where you set it to.
Ok, so 1st line:
Copy code
Running Building build_backend.pex from setuptools_default.lock ...
That means you ran into this: https://github.com/pantsbuild/pants/issues/16328 In short, you'll need to customize the default lock file location and re-generate the lock for every tool Pants uses. Pants ships with open-source lock files for all the tools it uses. When you don't have access to open-source repos, you need to regenerate all those locks.
This is currently painful and non-obvious to be sure!
So, follow along with that issue which has detailed steps I took to solve this issue in the Pants example-python repo. Your steps should be similar.
b

bored-glass-58755

08/31/2022, 2:58 PM
Thanks a lot for the lead, I'll let you know of the progress I make.
The
tools
directory lockfile location in your example
tools/lockfile
needs to be in repo-root?
e

enough-analyst-54434

08/31/2022, 3:22 PM
Yes. It can be anywhere under the repo root however you see fit.
👍 1
b

bored-glass-58755

08/31/2022, 3:55 PM
Ok, so I executed the command
PIP_INDEX_URL=<http://private.repo.url/simple> ./pants --version
and it ended as following:
Copy code
Installing pantsbuild.pants==2.12.0 into a virtual environment at /$XDG_CACHE_HOME/pants/setup/bootstrap-Linux-x86_64/2.12.0_py37
WARNING: The repository located at private.repo.url is not a trusted or secure host and is being ignored. If this repository is available via HTTPS we recommend you use HTTPS instead, otherwise you may silence this warning and allow it anyway with '--trusted-host private.repo.url'.
WARNING: The repository located at private.repo.url is not a trusted or secure host and is being ignored. If this repository is available via HTTPS we recommend you use HTTPS instead, otherwise you may silence this warning and allow it anyway with '--trusted-host private.repo.url'.
ERROR: Could not find a version that satisfies the requirement setuptools<58 (from versions: none)
ERROR: No matching distribution found for setuptools<58
WARNING: The repository located at private.repo.url is not a trusted or secure host and is being ignored. If this repository is available via HTTPS we recommend you use HTTPS instead, otherwise you may silence this warning and allow it anyway with '--trusted-host private.repo.url'.
##[error]Bash exited with code '1'.
If lines #2 & #3 are just warnings and if it did not use
private.repo.url
, why could not it download setuptools just how it was doing it before passing in the
PIP_INDEX_URL
? Also, the only change I have made is in
pants.toml
by introducing the
[setuptools]
subsystem with
lockfile = tools/setuptools.lock
. I haven't added any other subsystems and also haven't made any change to pants script as it shows in the GitHub link you provided (in function
bootstrap_virtualenv()
).
Retried by making following change to the script, waiting for it to complete:
Copy code
"${staging_dir}/install/bin/pip" install --trusted-host private.repo.url --quiet -U pip "setuptools<58" && \
"${staging_dir}/install/bin/pip" install ${maybe_find_links} --trusted-host private.repo.url --quiet --progress-bar off "${pants_requirement}"
Now it failed with:
Copy code
Installing pantsbuild.pants==2.12.0 into a virtual environment at /$XDG_CACHE_HOME/pants/setup/bootstrap-Linux-x86_64/2.12.0_py37
ERROR: Could not find a version that satisfies the requirement pantsbuild.pants==2.12.0 (from versions: none)
ERROR: No matching distribution found for pantsbuild.pants==2.12.0
##[error]Bash exited with code '1'.
e

enough-analyst-54434

08/31/2022, 4:13 PM
Perhaps add PIP_VERBOSE=3 to see more.
b

bored-glass-58755

08/31/2022, 4:14 PM
Ok, but the changes I described you above look fine?
e

enough-analyst-54434

08/31/2022, 4:27 PM
Probably. A diff would be easier to validate. That said - you need not have edited the script for the experiment you could have stuck with
PIP_<option>
b

bored-glass-58755

08/31/2022, 5:17 PM
Yeah, I wish I could share the diff. Regarding the script, so I need not even make this change (which I haven't done anyway, but just wanted to confirm):
Copy code
@@ -270,7 +270,9 @@ function bootstrap_virtualenv {
       staging_dir=$(tempdir "${PANTS_BOOTSTRAP}")
       cd "${staging_dir}"
       echo "${VIRTUALENV_REQUIREMENTS}" > requirements.txt
-      "${python}" "${pex_path}" -r requirements.txt -c virtualenv -o virtualenv.pex
+      "${python}" "${pex_path}" -r requirements.txt -c virtualenv -o virtualenv.pex \
+        --no-pypi \
+        --index <https://pextest.jfrog.io/artifactory/api/pypi/default-pypi/simple/>
e

enough-analyst-54434

08/31/2022, 5:59 PM
Correct. If you're just experimenting you can use the `PIP_*`env vars. If you want to write it all down, you might then want to edit the script unless you have a mechanism like direnv to write down env vars to activate whenever using Pants.
In my example on the issue I was writing it all down; so I showed final modification of the script.
b

bored-glass-58755

08/31/2022, 6:03 PM
Ok
I executed
PIP_INDEX_URL=<http://private.repo.url/simple> PIP_TRUSTED_HOST=private.repo.url PIP_VERBOSE=3 ./pants --version
and it couldn't
find pantsbuild.pants==2.12.0
in there which is expected. Is there a way that I bootstrap first and then use the
PIP_INDEX_URL
to
generate-lockfiles
? Or do I have to bootstrap using this private repo URL as
PIP_INDEX_URL
for it to work? If yes, then am I expected to onboard pantsbuild.pants onto the private repo? The logs are quite long but I can provide them if you don't mind going thru the long logs.
e

enough-analyst-54434

08/31/2022, 6:25 PM
it couldn't
find pantsbuild.pants==2.12.0
in there which is expected.
Does this mean http://private.repo.url/simple is not a mirror of PyPI, aka it is missing some distributions that are otherwise on PyPI?
b

bored-glass-58755

08/31/2022, 6:25 PM
Yes, it's not a mirror.
e

enough-analyst-54434

08/31/2022, 6:25 PM
Oh wow, this is going to be painful.
Um, let me think...
Yeah, so the initial Pants bootstrap which is just ~
pip install pantsbuild.pants==<pants version>
is the only part of the process that does not have a lock file right now. In order to figure out what distributions you need to mirror to make that work you'll need to run do the following on every platform you target (dev machines and CI):
Copy code
/python/we/use/python -mvenv figure-out-what-pants-needs
figure-out-what-pants-needs/bin/pip wheel --wheel-dir=wheels-pants-needs pantsbuild.pants==<pants version>
You'd then need to add the contents of the
wheels-pants-needs/
directory from each platform you target to your private index. You'll need to do this for every Pants upgrade; so it will behoove you to script it up after getting it all working. Pants will need to address all this awkwardness at some point, but until then!
This just handles the bootstrap though. Once you have that handled and you're migrating Pants tool lockfiles, you'll undoubtedly run into more missing wheels. For those cases though you can review the default tool lock files to see what you're missing. For example, for pytest: https://www.pantsbuild.org/docs/reference-pytest#section-lockfile it's here as indicated in those docs: https://github.com/pantsbuild/pants/blob/release_2.12.1rc4/src/python/pants/backend/python/subsystems/pytest.lock
b

bored-glass-58755

08/31/2022, 6:42 PM
Ok, I completely understood the first part related to bootstrapping pants with only using the private repo. I am trying to look at the second part regarding lockfiles. Regarding lockfiles, do we need to generate them every time CI runs or we can generate them once and then have them checked in thru git? If its later, then maybe I can generate them once and check it in and then bootstrap pants normally.
e

enough-analyst-54434

08/31/2022, 6:46 PM
The latter. But to even re-generate them, the repo at http://private.repo.url/simple will need to have all the required distributions already available! If not, the lock re-gen will fail. As such, I think you proetty much need to read through the default lock file checked into the Pants project and grab the full list of wheels and use that to pre-populate your private index.
b

bored-glass-58755

08/31/2022, 6:48 PM
Ok, I'll check the lock file in Pants project. But if its the latter. then why do I need to re-generate them as long as I am using the same pants version?
e

enough-analyst-54434

08/31/2022, 6:48 PM
Because the lock files have URLs in them and they are the URLs of Open source artifacts.
Presumably, the whole point here is your dev / CI machines can't use those URLs
IOW the Pants lock files lock URL + fingerprint and not just project name, version + fingerprint.
b

bored-glass-58755

08/31/2022, 7:00 PM
Ok, what I am thinking is: 1. Getting a list of wheels for pantsbuild.pants on my local dev machine and check which ones need to be onboarded to the private repo. 2. Once I have them onboarded, I can bootstrap pants on my local using the PIP_INDEX_URL. 3. The third step would be to generate-lockfiles and check them to the code repo. And then when CI runs, it would use the lockfiles I generated and checked in (would it after I bootstrap pants again in CI?) Of course, I need to repeat this exercise for every platform I am expected to work on. The best would be to do this exercise only once on an environment that resembles the CI's environment and have it run.
e

enough-analyst-54434

08/31/2022, 7:16 PM
Step 3 is a one time thing. The generated lock files are universal and will have listed all wheels and sdists needed for any platform. As such, you cannot generate a proper lockfile until your private index has all needed wheels and sdists! Concretely, say your CI is locked down to 1 Linux distribution - CentOS 7, and 1 Python interpreter, Python 3.8.10 and then you have devs working purely on Mac, but some intel and some arm. Lets further say all devs also use Python 3.8.10. That means you have 3 platforms to account for. If so then your steps are, in order: 1. For each platform:
pip wheel pantsbuild.pants
and gather wheels and put them in your private index. 2. For each platform and for each (Python) tool Pants uses: Find the default tool lockfile checked into the Pants repo and gather all the wheels and sdists listed that apply to your platforms and put them in your private index. 3. Now bootstrap Pants using the private index and run
./pants generate-lockfiles
then check in all the lock files. At this point Panst can run on dev machines and in CI.
This is all definitely a huge pain and totally ad hoc right now. Ideally there would be a Pants tool that could look at your pants.toml and allow you to say something like `pants-shrinkwrap download --dir here/ pants.toml`And that would gather all the artifacts Pants needed under `here/`and maybe also generate some form of manifest that could then be used by another tool or command to re-configure Pants once you moved all the downloaded files where they needed to go (namely re-gen lockfiles)
pants-shinkwrap migrate --manifest manifest.json pants.toml
.
Of course you'd need to run
pants-shrinkwrap download
on a machine with public internet access, but then the rest could be done in a locked down environment.
b

bored-glass-58755

08/31/2022, 7:33 PM
Ok, so to check if I understand it well - let’s assume that the only wheels that it currently needs to
package
and tries to download from pythonhosted.org are
setuptools
and
wheel
(as demonstrated in the starting message of this thread). I assume those wheels are present in the default
setuptools.lock
file in the Pants repo and are also present in my private repo. If I download that file and replace the pythonhosted.org URL with my private.repo.url along with versions and hashes that it needs and check this modified
setuptools.lock
file in to my code repo under
tools/
directory. Thereafter, I have my
pants.toml
modified to have
[setuptools]
subsystem with line
lockfile = "tools/setuptools.lock"
. After all this, I assume - if I understand it all correctly - that my
package
goal would run successfully.
e

enough-analyst-54434

08/31/2022, 7:36 PM
That is correct, but hand-editing the lock json is not a good idea. Instead, you download all the wheels by hand, edit pants.toml to have a custom
lockfile = ...
and then run
./pants generate-lockfiles
As long as pants.toml also has the `[python-repos]`config that points at your private repo, it will do the right thing and use urls to your private repo in the locks it generates.
The only issue is all the right wheels must be uploaded to your private repo 1st!
b

bored-glass-58755

08/31/2022, 7:40 PM
Ok, I am going to hand edit json for now for just those two packages.
As long as I have the wheels required by pantsbuild.pants in my private repo, I can run generate-lockfiles and it will re-generate files based on the subsystems present in
pants.toml
or it will re-generate all lockfiles? The config in
pants.toml
is only to provide a lockfile location in the absence of which it will generate files in the default location?
e

enough-analyst-54434

08/31/2022, 7:51 PM
./pants generate-lockfiles
will generate lockfiles for every subsystem in
pants.toml
that has an explicit non-default `lockfile`defined. So, by default it will do nothing. It only does things once you have one or more non-default lockfile defined. This is additional reason why this is all so paintful right now. There is no obvious list of which subsystems you use and thus which ones you'll need to define a custom `lockfile`location for.
If you don't have a custom
lockfile
for a subsystem that needs one, you currently find out with the style of error you led with in the original post.
b

bored-glass-58755

08/31/2022, 7:54 PM
Yeah, I got that. The first thing to do now is to ensure that the private repo has
wheels-that-pants-needs
and then lockfiles can be generated and then private repo must have those wheels too
🤯
e

enough-analyst-54434

08/31/2022, 8:03 PM
One thing that might help at a high level is to ignore Pants here. For any tool that relies on downloading things from public places as its default, you'll need to 1st gather all the things it needs to download, add them to private places, configure the tool to look at the private places, then run the tool using the private places configuration.
b

bored-glass-58755

08/31/2022, 8:04 PM
Yeah
e

enough-analyst-54434

08/31/2022, 8:05 PM
The unique problems Pants presents in this scenario are: 1. It provides no listing of the things it needs to download. 2. It is configured in 3 places: The bootstrap script, pants.toml and lockfiles.
b

bored-glass-58755

08/31/2022, 8:06 PM
Bootstrap script for Pex download?
e

enough-analyst-54434

08/31/2022, 8:06 PM
pants
script. That bootstraps pants and has configuration for the tools it needs to do the job. One of which is download a Pex PEX.
b

bored-glass-58755

08/31/2022, 8:08 PM
Yeah, but I thought bootstrap is only to download Pex. Oh wait, there are those VIRTUALENV_REQUIREMENTS also and not only Pex.
e

enough-analyst-54434

08/31/2022, 8:09 PM
The `pants`script has two roles, 1: install Pants if not already, 2: run the installed Pants. To do 1 it needs the Pex PEX which then downloads virtualenv and its depenedncies and then it creates a virtualenv and then installs Pants in that virtualenv using Pip.
To do 1 it needs the Pex PEX which then downloads virtualenv and its depenedncies and then it creates a virtualenv and then installs Pants in that virtualenv using Pip.
That is "the bootstrap"
b

bored-glass-58755

08/31/2022, 8:10 PM
Yeah, ok!
I have one more question: My CI is bootstrapping pants just fine without the use of
PIP_INDEX_URL
(not sure how). The only place where I am stuck at is because it is unable to download anything from pythonhosted.org and that's where my
package
goal is failing in CI which is a linux platform. What if I just do following on a linux dev box: 1. Bootstrap pants (which it is able to do) 2. Modify
pants.toml
for setuptools to generate a lockfile with PIP_INDEX_URL. 3. Verify the generated lockfile against the private repo to ensure that I have the required wheels. 4. Check this in to code repo and run CI
e

enough-analyst-54434

08/31/2022, 8:27 PM
My CI is bootstrapping pants just fine without the use of
PIP_INDEX_URL
(not sure how).
I think you need to fully understand this. If you don't the rest will be a muddled jumbled chain of hacks around this central key missing fact.
b

bored-glass-58755

08/31/2022, 8:27 PM
Ok
e

enough-analyst-54434

08/31/2022, 8:30 PM
I'm definitely confused at any rate. Are you saying dev machines can access the internet but CI machines can't - or at least partially can't? I'm familiar with locked down environments but not familiar with partially locked down ones - dev can see internet, CI cannot.
b

bored-glass-58755

08/31/2022, 8:46 PM
I am new at my company and trying to figure out this too. I'll come back with answers.
e

enough-analyst-54434

08/31/2022, 8:52 PM
Aha! Ok, great. Yeah, knowing the ground rules would be super useful here.
👍 1
b

bored-glass-58755

08/31/2022, 10:54 PM
Ok, so first of all I am really sorry for causing a lot of confusion. It seems like CI has access to PyPI (and files.pythonhosted.org thru that). I could see it was bootstrapping pants using that source itself and was failing in
package
goal because the subprocess doesn't have access to env proxy variables: https://github.com/pantsbuild/pants/issues/15675. I was able to go past that error by setting proxy variables
env.add
under
[subprocess-environment]
subsystem in
pants.toml
but now
package
is throwing below:
Copy code
File "/$XDG_CACHE_HOME/pants/named_caches/pex_root/installed_wheels/6242b902db69f59e1092b406655c0fb1634486c47ce563f5fd27277cf4561822/pex-2.1.90-py2.py3-none-any.whl/pex/jobs.py", line 396, in handle_spawn_error
    raise self._raise_type(self.spawn_error_message(item, exception))
pex.resolve.resolvers.Untranslatable: Failed to spawn a job for InstallRequest(target=LocalInterpreter(id='usr.local.python.3.7.6.bin.python3.7', platform=Platform(platform='manylinux_2_17_x86_64', impl='cp', version='3.7.6', version_info=(3, 7, 6), abi='cp37m'), marker_environment=MarkerEnvironment(implementation_name='cpython', implementation_version='3.7.6', os_name='posix', platform_machine='x86_64', platform_python_implementation='CPython', platform_release='3.10.0-1160.71.1.el7.x86_64', platform_system='Linux', platform_version='#1 SMP Wed Jun 15 08:55:08 UTC 2022', python_full_version='3.7.6', python_version='3.7', sys_platform='linux'), interpreter=PythonInterpreter('/usr/local/python/3.7.6/bin/python3.7', PythonIdentity('/usr/local/python/3.7.6/bin/python3.7', 'cp37', 'cp37m', 'manylinux_2_17_x86_64', (3, 7, 6)))), wheel_path='/$XDG_CACHE_HOME/pants/named_caches/pex_root/downloads/4bdcd7d840138086126cd09254dc6195fb4fc6f01c050a1d7236f2630db1d22a/wheel-0.37.1-py2.py3-none-any.whl', fingerprint='4bdcd7d840138086126cd09254dc6195fb4fc6f01c050a1d7236f2630db1d22a'): [Errno 2] No such file or directory: '/$XDG_CACHE_HOME/pants/named_caches/pex_root/venvs/5632a685cd3903cb1ab183bd3e3fb9767d9d2e42/15d3ad9788dc0da4a814dd5bb1a36eed666cd279/pex': '/$XDG_CACHE_HOME/pants/named_caches/pex_root/venvs/5632a685cd3903cb1ab183bd3e3fb9767d9d2e42/15d3ad9788dc0da4a814dd5bb1a36eed666cd279/pex'
Could you please help me with that?
e

enough-analyst-54434

08/31/2022, 11:01 PM
No worries, but it looks like you have more to work out still:
Copy code
wheel_path='/$XDG_CACHE_HOME/pants/named_caches/pex_root/downloads/4bdcd7d840138086126cd09254dc6195fb4fc6f01c050a1d7236f2630db1d22a/wheel-0.37.1-py2.py3-none-any.whl'
Unless you're hand editing out the real value of
$XDG_CACHE_HOME
in your posts, it appears the env var is not getting set up correctly.
b

bored-glass-58755

08/31/2022, 11:04 PM
I am just redacting the prefix - essentially the path before
/pants/named_caches/...
e

enough-analyst-54434

08/31/2022, 11:09 PM
Hrm, are you positive? That style error is not making sense.
b

bored-glass-58755

08/31/2022, 11:18 PM
Yes, I double checked. Probably following might help?
Copy code
$XDG_CACHE_HOME is /some/prefix/_work/36334/.xdg-cache


File "/some/prefix/_work/36334/.xdg-cache/pants/named_caches/pex_root/installed_wheels/6242b902db69f59e1092b406655c0fb1634486c47ce563f5fd27277cf4561822/pex-2.1.90-py2.py3-none-any.whl/pex/jobs.py", line 396, in handle_spawn_error
    raise self._raise_type(self.spawn_error_message(item, exception))
pex.resolve.resolvers.Untranslatable: Failed to spawn a job for InstallRequest(target=LocalInterpreter(id='usr.local.python.3.7.6.bin.python3.7', platform=Platform(platform='manylinux_2_17_x86_64', impl='cp', version='3.7.6', version_info=(3, 7, 6), abi='cp37m'), marker_environment=MarkerEnvironment(implementation_name='cpython', implementation_version='3.7.6', os_name='posix', platform_machine='x86_64', platform_python_implementation='CPython', platform_release='3.10.0-1160.71.1.el7.x86_64', platform_system='Linux', platform_version='#1 SMP Wed Jun 15 08:55:08 UTC 2022', python_full_version='3.7.6', python_version='3.7', sys_platform='linux'), interpreter=PythonInterpreter('/usr/local/python/3.7.6/bin/python3.7', PythonIdentity('/usr/local/python/3.7.6/bin/python3.7', 'cp37', 'cp37m', 'manylinux_2_17_x86_64', (3, 7, 6)))), wheel_path='/some/prefix/_work/36334/.xdg-cache/pants/named_caches/pex_root/downloads/4bdcd7d840138086126cd09254dc6195fb4fc6f01c050a1d7236f2630db1d22a/wheel-0.37.1-py2.py3-none-any.whl', fingerprint='4bdcd7d840138086126cd09254dc6195fb4fc6f01c050a1d7236f2630db1d22a'): [Errno 2] No such file or directory: '/some/prefix/_work/36334/.xdg-cache/pants/named_caches/pex_root/venvs/5632a685cd3903cb1ab183bd3e3fb9767d9d2e42/15d3ad9788dc0da4a814dd5bb1a36eed666cd279/pex': '/some/prefix/_work/36334/.xdg-cache/pants/named_caches/pex_root/venvs/5632a685cd3903cb1ab183bd3e3fb9767d9d2e42/15d3ad9788dc0da4a814dd5bb1a36eed666cd279/pex'
Also, I get this warning at the time of bootstrap where if the PEX_ROOT is more than 128 characters. Could that be an issue?
e

enough-analyst-54434

08/31/2022, 11:31 PM
I doubt it. Here is what would be most fruitful at this point since its hard to track where we are (for me) with the revelation that custom indexes aren't needed apparently. Can you please do 3 things: 1. Change 36334 to 36335 or anything else to effectively wipe the PEX_ROOT cache in CI. 2. Re-run. 3. Describe your complete setup and failing CI command with as full a log as you can provide. This should help reset my internal debugger. Right now I feel like I have no firm footing.
b

bored-glass-58755

09/01/2022, 1:20 AM
I can't change that 36334 to 36335 because that's my CI's workspace directory that gets generated/allocated randomly. What I can do to wipe PEX_ROOT cache is to not set PEX_ROOT at all (only set XDG_CACHE_HOME).
In fact, the entire path until
.xdg_cache
directory is generated by CI. I just set XDG_CACHE_HOME variable to that path and then make additional directories
pants/setup
and
pants/named_caches/pex_root
to make paths
$XDG_CACHE_HOME/pants/setup
and
$XDG_CACHE_HOME/pants/named_caches/pex_root
.
e

enough-analyst-54434

09/01/2022, 1:57 AM
Ok, well use your creativity but ensure the next run has a new final XDG_CACHE_HOME so we have a clean slate.
b

bored-glass-58755

09/01/2022, 2:15 AM
Ok, I can do that. But do you want me to not set PEX_ROOT at all?
Hi, please find attached the redacted logs for pants bootstrap and
package
goal. I have also echoed the values for
$XDG_CACHE_HOME
&
$PEX_ROOT
. I only set the former in this run and so, the later is empty.
e

enough-analyst-54434

09/01/2022, 5:46 PM
Ok, thanks. I'm staring at this now. Remind me of the current state of your world? It looks to me like public internet access works. For example see:
Copy code
09:19:02.39 [DEBUG] Completed: Downloading: DownloadFile(url='<https://github.com/pantsbuild/pex/releases/download/v2.1.90/pex>', expected_digest=FileDigest('2781255baf77c2a8fdc85c5e830f7191a6048fd91d2e20b5c7a20e5a0b7beb66', 3755345)... (1 characters truncated)
That's public and it worked. Elsewhere there is evidence this may work because a proxy is configured:
Copy code
"http_proxy": "<http://corp.proxy.com:8080>", "https_proxy": "<http://corp.proxy.com:8080>", "no_proxy": "internal.url1,internal.url2,internal.url3,tools-that-we-use.url"
However, elsewhere still there is still use of a private PyPI:
Copy code
"--no-pypi", "--index=<http://private.repo.url/simple>"
That private PyPI will be used only to create new lockfiles - say for your own code's 3rdparty deps - but it will not be used for existing lock files that ship with Pants for all its tools.
b

bored-glass-58755

09/01/2022, 7:00 PM
Yes, I have kept this private repo as part of my config but I don't think that is being used until the point I am at currently.
I "think" I figured what's causing the issue. However, I am unable to resolve it. This is going to be a long message but I will try to be as elaborate as possible. Following is what I did in my CI: 1. Set
$XDG_CACHE_HOME
to
/path/to/CI/workspace/directory/_work/33042/.xdg-cache
(basically
_work/33042/
is also a part of CI workspace path but I thought I need not redact it). The directory where CI checks out the code and which becomes the repo root is
/path/to/CI/workspace/directory/_work/33042/s
. 2. Create dirs
$XDG_CACHE_HOME/pants/setup
and
$XDG_CACHE_HOME/pants/named_caches/pex_root
. 3. Set
$PANTS_SETUP_CACHE
to
$XDG_CACHE_HOME/pants/setup
and
$PEX_ROOT
to
$XDG_CACHE_HOME/pants/named_caches/pex_root
. 4. Bootstrap Pants. 5. Do a
ls -ltra /path/to/CI/workspace/directory/_work/33042/.xdg-cache/pants/named_caches/pex_root/downloads/
because this is what the wheel path is in the failing step's output. Also do a
ls -ltra /path/to/CI/workspace/directory/_work/33042/.xdg-cache/pants/named_caches/pex_root/venvs/
because this is the path to directory or file that the step is unable to find. 6. Do pants tailor --check. 7. Repeat step #5 again (Output of this step is shared below) 8. Run pants -ldebug package
Copy code
Starting: LS step BEFORE Package
Generating script.
========================== Starting Command Output ===========================
Today's date
Thu Sep  1 13:30:36 EDT 2022
> ls -ltra /path/to/CI/workspace/directory/_work/23715/.xdg-cache/pants/named_caches/pex_root/downloads/
total 0
drwxrwsr-x. 12 owner group 229 Sep  1 13:29 ..
drwxrwsr-x.  2 owner group  10 Sep  1 13:30 .
> ls -ltra /path/to/CI/workspace/directory/_work/23715/.xdg-cache/pants/named_caches/pex_root/venvs/
total 0
drwxrwsr-x.  4 owner group  75 Sep  1 13:29 .
drwxrwsr-x.  3 owner group  72 Sep  1 13:29 s
drwxrwsr-x.  3 owner group 136 Sep  1 13:29 15158fe84406f2733a98f687f98d59238ad4210d
drwxrwsr-x. 12 owner group 229 Sep  1 13:29 ..
> ls -ltra /path/to/CI/workspace/directory/_work/23715/.xdg-cache/pants/named_caches/pex_root/venvs/15158fe84406f2733a98f687f98d59238ad4210d/
total 0
-rwxrwxr-x. 1 owner group   0 Sep  1 13:29 .15d3ad9788dc0da4a814dd5bb1a36eed666cd279.atomic_directory.lck
drwxrwsr-x. 4 owner group  75 Sep  1 13:29 ..
drwxrwsr-x. 5 owner group 156 Sep  1 13:29 15d3ad9788dc0da4a814dd5bb1a36eed666cd279
drwxrwsr-x. 3 owner group 136 Sep  1 13:29 .
Finishing: LS step BEFORE Package
The
package
is failing with following missing file or directory:
Copy code
[Errno 2] No such file or directory: '/path/to/CI/workspace/directory/_work/4052/.xdg-cache/pants/named_caches/pex_root/venvs/5632a685cd3903cb1ab183bd3e3fb9767d9d2e42/15d3ad9788dc0da4a814dd5bb1a36eed666cd279/pex': '/path/to/CI/workspace/directory/_work/4052/.xdg-cache/pants/named_caches/pex_root/venvs/5632a685cd3903cb1ab183bd3e3fb9767d9d2e42/15d3ad9788dc0da4a814dd5bb1a36eed666cd279/pex'
The path getting generated in my CI is
/path/to/CI/workspace/directory/_work/4052/.xdg-cache/pants/named_caches/pex_root/venvs/_*15158fe84406f2733a98f687f98d59238ad4210d*_/15d3ad9788dc0da4a814dd5bb1a36eed666cd279/pex
Basically the directory inside the
pex_root/venvs/
has different name (
5632a685cd3903cb1ab183bd3e3fb9767d9d2e42
) that the
package
goal expects and the directory name actually present is (
15158fe84406f2733a98f687f98d59238ad4210d
). As a fix, I tried adding following steps before running package in the above sequence of steps: 7a. A step
mv actual-dir-name-present-in-the-path expected-dir-name
7b. Again
ls -ltra
over all the directories that I was doing before and included the directories
actual-dir-name-present-in-the-path
and
expected-dir-name
. This 7b step showed that there is no
actual-dir-name-present-in-the-path
anymore which means that mv was successful. But the package command failed even after that with the same error that directory or file is not found.
Oh, so apart from the wrong path (whether in CI or the failing module), the entire path is also missing the last bit of the entire path which is
/pex
. Not sure if this is a directory or a file. Does it make sense?
e

enough-analyst-54434

09/01/2022, 9:13 PM
Is
/path/to/CI/workspace/directory/_work/4052/.xdg-cache/
known to be empty or non-empty before the Pants run in CI? I'd like it to be empty for the purposes of debugging from a clean slate.
b

bored-glass-58755

09/01/2022, 9:18 PM
I just checked following paths before running bootstrap and these are empty:
/path/to/CI/workspace/directory/_work/33042/.xdg-cache/pants/named_caches/pex_root/downloads/
/path/to/CI/workspace/directory/_work/33042/.xdg-cache/pants/named_caches/pex_root/venvs/
/path/to/CI/workspace/directory/_work/33042/.xdg-cache/pants/named_caches/pex_root/venvs/s
/path/to/CI/workspace/directory/_work/4052/.xdg-cache/pants/named_caches/pex_root/venvs/15158fe84406f2733a98f687f98d59238ad4210d
/path/to/CI/workspace/directory/_work/4052/.xdg-cache/pants/named_caches/pex_root/venvs/5632a685cd3903cb1ab183bd3e3fb9767d9d2e42
I will just check
/path/to/CI/workspace/directory/_work/33042/.xdg-cache/
now
Just ignore the number after
work/
in above. I copied those paths from my previous long chat which had outputs from two different pipeline runs - one before fix and one after fix
e

enough-analyst-54434

09/01/2022, 9:23 PM
Create dirs
$XDG_CACHE_HOME/pants/setup
and
$XDG_CACHE_HOME/pants/named_caches/pex_root
.
When you say this, you mean you're manually making these directories? You should not need to do this. Pants automatically makes the directories it needs if they do not exist. Avoiding manually creating these directories avoids potential permission issues (which it does not look like you're hitting) and reduces the surface area of debugging suspects.
b

bored-glass-58755

09/01/2022, 9:26 PM
Ok, I started creating these directories because Pants was creating them but not in the structure:
$XDG_CACHE_HOME/pants/setup
$XDG_CACHE_HOME/pants/named_caches/pex_root
I learnt this structure from my local Mac's bootstrapped Pants.
e

enough-analyst-54434

09/01/2022, 9:27 PM
Yeah, you don't want to create the directories, you want to configure Pants correctly such that Pants creates those directories.
@bored-glass-58755 I'm almost completely lost at this point. This is a huge thread. If I had my hands on your infra I would be debugging by starting over. Eliminating all customizations to scripts and pants.toml and seeing what fails from a clean slate 1st with no custom env vars, etc.
b

bored-glass-58755

09/01/2022, 9:33 PM
Ok, so how would you suggest I start over? Just setting $XDG_CACHE_HOME and $PEX_ROOT and nothing else? I understood that I should not create the directories. Also, following is the output of directory contents of $XDG_CACHE_HOME before bootstrap:
Copy code
Starting: LS $XDG_CACHE_HOME before bootstrap
Generating script.
========================== Starting Command Output ===========================
> ls -ltra /path/to/CI/workspace/directory/_work/19378/.xdg-cache
total 12
drwxrwxr-x. 7 owner group 4096 Sep  1 17:26 ..
drwxrwxr-x. 3 owner group 4096 Sep  1 17:26 .
drwxrwxr-x. 4 owner group 4096 Sep  1 17:26 pants
Finishing: LS $XDG_CACHE_HOME before bootstrap
e

enough-analyst-54434

09/01/2022, 11:02 PM
I would start from a completely clean slate. Have CI try to run
./pants
using what's checked in to the repo. No edits to the
pants
script and no edits
pants.toml
and no custom environment variables. My problem is I have lost track of what you may have modified to try to get CI working. So, for me, starting with a completely blank slate is the only way to build back up a confident picture of the variables in play. And having a confident picture of those variables is critical to debugging something, 5x so when remotely debugging blind.
b

bored-glass-58755

09/01/2022, 11:44 PM
Ok, I tried doing that and following is the progress. Let me know if you have any questions and I would try to answer them: 1. Ran pants without any env variables and
./pants --version
failed because the CI doesn't have perms to create
$HOME/.cache/pants
. 2. Set
$XDG_CACHE_HOME
to
/path/to/CI/workspace/directory/work/<num>/_._xdg_cache
and re-ran
./pants --version
. It gave some PEX_ROOT related warnings that
$HOME/.pex
is un-writable and that it fell back to a temp directory and warned me that it will hurt performance. But apart from this warning, it failed this time for env vars
$LC_ALL
and
$LANG
saying that they were set to
ISO-8859-1
but Pants wants
UTF-8
. 3. Set
$LC_ALL
and
$LANG
to
en_US.UTF-8
along with
$XDG_CACHE_HOME
as in step #2 and reran
./pants --version
. It bootstrapped fine but with the PEX_ROOT warnings that
$HOME/.pex
is un-writable and that it fell back to a temp directory and warned me that it will hurt performance. After that the CI continued to run
./pants tailor --check
which completed successfully. The third step in CI was to
package
a
python_distribution()
and it failed because it could not download
setuptools
and
wheel
from
<http://files.pythonhosted.org|files.pythonhosted.org>
. It needs proxies here because the subprocess that runs
package
doesn't have access to the default proxies that CI uses from the environment (because of which it was able to bootstrap Pants in the first place). 4. Set proxies under
[subprocess-environment]
in
pants.toml
and re-ran CI. This time it failed to download
setuptools
and
wheel
because of
SSL certificate
error. 5. I found out the certificate path and set the variable
$PANTS_CA_CERTS_PATH
with that path in CI and re-ran. This time the
package
step failed with the same error that it can't find the file or directory. The directory names under the directory
/pex_root/venvs/
are exactly the same as before and so, pasting the error here from earlier chats (so please ignore the
<num>
under
_work/
) so that its easier to have a look at it at a single place:
Copy code
[Errno 2] No such file or directory: '/path/to/CI/workspace/directory/_work/4052/.xdg-cache/pants/named_caches/pex_root/venvs/5632a685cd3903cb1ab183bd3e3fb9767d9d2e42/15d3ad9788dc0da4a814dd5bb1a36eed666cd279/pex': '/path/to/CI/workspace/directory/_work/4052/.xdg-cache/pants/named_caches/pex_root/venvs/5632a685cd3903cb1ab183bd3e3fb9767d9d2e42/15d3ad9788dc0da4a814dd5bb1a36eed666cd279/pex'
Apart from this, following is `pants.toml`:
Copy code
[GLOBAL]
pants_version = "2.12.0"

backend_packages.add = [
  "pants.backend.python",
  "pants.backend.python.typecheck.mypy",
  "pants.backend.python.lint.flake8",
  "pants.backend.experimental.python.lint.autoflake",
  "pants.backend.plugin_development",
  "pants.backend.codegen.protobuf.python",
  "pants.backend.experimental.python"
]

[source]
root_patterns = [
  "/",
  "/test",
  "/module1",
  "/module2",
  "/module3",
  "/module4",
  "/module5"
]

[subprocess-environment]
env_vars.add = ["http_proxy", "https_proxy", "no_proxy"]
Please let me know if anything is not clear and I can try to answer that.
e

enough-analyst-54434

09/02/2022, 12:08 AM
Ok, great. That is very helpful. Each modification has a known reason explained. Thinking through what could be going on. As I do that though, can you do? 6.
rm -rf /path/to/CI/workspace/directory/_work/4052/.xdg-cache/pants/named_caches/pex_root
and then re-run
./pants package ...
(Modifying that path to reflect whatever it really is. There should be a bunch of deletes. Perhaps use `rm -rfv ...`to get a flood of messages from
rm
proving it is removing things.
b

bored-glass-58755

09/02/2022, 3:02 AM
I introduced following step just before running the
package
goal:
rm -rfv ${XDG_CACHE_HOME}/pants/named_caches/pex_root
and there are two cases that happened: Case 1: It seemed like that in this case, the CI ran in a workspace where it had already ran (sometime) earlier and so, it did not have to "bootstrap" Pants. Pants was already there. See the output for the
./pants --version
(ignore the warning):
Copy code
Starting: Pants bootstrap step
Generating script.
Script contents:
./pants --version
========================== Starting Command Output ===========================
22:26:21.55 [INFO] Initializing scheduler...
22:26:21.86 [INFO] Scheduler initialized.
22:26:21.91 [WARN] Please either set `enabled = true` in the [anonymous-telemetry] section of pants.toml to enable sending anonymous stats to the Pants project to aid development, or set `enabled = false` to disable it. No telemetry sent for this run. An explicit setting will get rid of this message. See <https://www.pantsbuild.org/v2.12/docs/anonymous-telemetry> for details.
2.12.0
Finishing: Pants bootstrap step
And please find attached the output of the delete step. Case 2: This was more of a normal run where Pants bootstrapped (I could see the green lines that said Bootstrapping pants, Downloading PEX, SHA256 Verified, Installing pantsbuild.pants, etc. I could also see the PEX_ROOT un-writable warning) but there were no directories to delete in this run in the delete step that was introduced:
Copy code
Starting: Deleting ${XDG_CACHE_HOME}/pants/named_caches/pex_root
Generating script.
========================== Starting Command Output ===========================
> rm -rfv /path/to/CI/workspace/directory/_work/34538/.xdg-cache/pants/named_caches/pex_root
Finishing: Deleting ${XDG_CACHE_HOME}/pants/named_caches/pex_root
The
package
goal ended with the same error as before in both the cases.
Hi, I think I misunderstood your last message. I think what you meant was run package, then delete, and then run package again. If that was the case, please ignore my last message. Here's the output of package, delete and then package again in the same CI: First package failed:
Copy code
File "/path/to/CI/workspace/directory/_work/13694/.xdg-cache/pants/named_caches/pex_root/installed_wheels/6242b902db69f59e1092b406655c0fb1634486c47ce563f5fd27277cf4561822/pex-2.1.90-py2.py3-none-any.whl/pex/jobs.py", line 396, in handle_spawn_error
    raise self._raise_type(self.spawn_error_message(item, exception))
pex.resolve.resolvers.Untranslatable: Failed to spawn a job for InstallRequest(target=LocalInterpreter(id='usr.local.python.3.7.6.bin.python3.7', platform=Platform(platform='manylinux_2_17_x86_64', impl='cp', version='3.7.6', version_info=(3, 7, 6), abi='cp37m'), marker_environment=MarkerEnvironment(implementation_name='cpython', implementation_version='3.7.6', os_name='posix', platform_machine='x86_64', platform_python_implementation='CPython', platform_release='3.10.0-1160.71.1.el7.x86_64', platform_system='Linux', platform_version='#1 SMP Wed Jun 15 08:55:08 UTC 2022', python_full_version='3.7.6', python_version='3.7', sys_platform='linux'), interpreter=PythonInterpreter('/usr/local/python/3.7.6/bin/python3.7', PythonIdentity('/usr/local/python/3.7.6/bin/python3.7', 'cp37', 'cp37m', 'manylinux_2_17_x86_64', (3, 7, 6)))), wheel_path='/path/to/CI/workspace/directory/_work/13694/.xdg-cache/pants/named_caches/pex_root/downloads/4bdcd7d840138086126cd09254dc6195fb4fc6f01c050a1d7236f2630db1d22a/wheel-0.37.1-py2.py3-none-any.whl', fingerprint='4bdcd7d840138086126cd09254dc6195fb4fc6f01c050a1d7236f2630db1d22a'): [Errno 2] No such file or directory: '/path/to/CI/workspace/directory/_work/13694/.xdg-cache/pants/named_caches/pex_root/venvs/7590ee51593acd29997ca5259a9f2ec093c0559a/15d3ad9788dc0da4a814dd5bb1a36eed666cd279/pex': '/path/to/CI/workspace/directory/_work/13694/.xdg-cache/pants/named_caches/pex_root/venvs/7590ee51593acd29997ca5259a9f2ec093c0559a/15d3ad9788dc0da4a814dd5bb1a36eed666cd279/pex'
The delete step's output is attached in a
.txt
file. And lastly, here's the output of the re-run of the
package
goal:
Copy code
File "/path/to/CI/workspace/directory/_work/13694/.xdg-cache/pants/named_caches/pex_root/installed_wheels/6242b902db69f59e1092b406655c0fb1634486c47ce563f5fd27277cf4561822/pex-2.1.90-py2.py3-none-any.whl/pex/jobs.py", line 396, in handle_spawn_error
    raise self._raise_type(self.spawn_error_message(item, exception))
pex.resolve.resolvers.Untranslatable: Failed to spawn a job for InstallRequest(target=LocalInterpreter(id='usr.local.python.3.7.6.bin.python3.7', platform=Platform(platform='manylinux_2_17_x86_64', impl='cp', version='3.7.6', version_info=(3, 7, 6), abi='cp37m'), marker_environment=MarkerEnvironment(implementation_name='cpython', implementation_version='3.7.6', os_name='posix', platform_machine='x86_64', platform_python_implementation='CPython', platform_release='3.10.0-1160.71.1.el7.x86_64', platform_system='Linux', platform_version='#1 SMP Wed Jun 15 08:55:08 UTC 2022', python_full_version='3.7.6', python_version='3.7', sys_platform='linux'), interpreter=PythonInterpreter('/usr/local/python/3.7.6/bin/python3.7', PythonIdentity('/usr/local/python/3.7.6/bin/python3.7', 'cp37', 'cp37m', 'manylinux_2_17_x86_64', (3, 7, 6)))), wheel_path='/path/to/CI/workspace/directory/_work/13694/.xdg-cache/pants/named_caches/pex_root/downloads/4bdcd7d840138086126cd09254dc6195fb4fc6f01c050a1d7236f2630db1d22a/wheel-0.37.1-py2.py3-none-any.whl', fingerprint='4bdcd7d840138086126cd09254dc6195fb4fc6f01c050a1d7236f2630db1d22a'): [Errno 2] No such file or directory: '/path/to/CI/workspace/directory/_work/13694/.xdg-cache/pants/named_caches/pex_root/venvs/7590ee51593acd29997ca5259a9f2ec093c0559a/15d3ad9788dc0da4a814dd5bb1a36eed666cd279/pex': '/path/to/CI/workspace/directory/_work/13694/.xdg-cache/pants/named_caches/pex_root/venvs/7590ee51593acd29997ca5259a9f2ec093c0559a/15d3ad9788dc0da4a814dd5bb1a36eed666cd279/pex'
It's weird that it did not find the directory while running the
package
goal first but then deleted the same directory in the very next step and then again failed for the same directory not found.
Hi @enough-analyst-54434, trust you are doing well. May I please request your attention to these findings and help to proceed?
e

enough-analyst-54434

09/06/2022, 7:04 PM
Thanks for the data. At this point you'll need to roll up your sleeves - I'm stumped. If I understand correctly, this error only happens on CI. On your workstation with the same configuration (minus the bits you had to change above for the reasons we isolated) things work. This tells me, as a debugger, to look for what is different between the two settings. We know the list of things you had to change are different, but none of those seem material to this sort of issue. Is there anything further different? Are any of the paths in question in CI on a network mount for example?
Another thing you might try, as a debugger, is varying one thing. Maybe try different releases. 2.12.1 was just released and 2.13 is close to stable, you might try 2.13.0rc3.
b

bored-glass-58755

09/06/2022, 7:22 PM
Ok, let me try to find that out with the help of infra team. Yes,
package
and
publish
work on my local with the exception that
publish
only works for docker targets as I don't have access to push a wheel to private repo using my user ID. It has to be done via a CI which is a separate thing and not a Pants issue. Let me also try to run the existing pipeline with 2.3.0rc3.
I'd like to thank you once again for all the help you provided. I will try to understand how things are working here and will get back, if needed.
Do you think if this could be a bug in reporting the error message? As in the real error could be something else? Just like this bug here: https://pantsbuild.slack.com/archives/C046T6T9U/p1660918888184319?
Is it possible that Pants downloads a PEX version other than what is mentioned in the pants script? Look at this error line where it is referring to
pex-2.1.90
but I am using the latest script that has `2.1.103`:
Copy code
File "/path/to/CI/workspace/directory/_work/13694/.xdg-cache/pants/named_caches/pex_root/installed_wheels/6242b902db69f59e1092b406655c0fb1634486c47ce563f5fd27277cf4561822/pex-2.1.90-py2.py3-none-any.whl/pex/jobs.py", line 396, in handle_spawn_error
    raise self._raise_type(self.spawn_error_message(item, exception))
e

enough-analyst-54434

09/07/2022, 4:26 AM
No, the Pex version used in the bootstrap script (the pants script) to install Pants bears no relation to the Pex version used by Pants once it is installed.
These are simply two separate pieces of software. The bootstrap (pants) script just installs Pants one time; nothing more
b

bored-glass-58755

09/07/2022, 4:31 AM
Ok
Are these operations async? As in could this be a possibility that the job is trying to refer the directory before it gets fully created or is in the process of creation?
e

enough-analyst-54434

09/07/2022, 2:03 PM
They are sync, but they are racy - in other words there can be 8 jobs all trying to synchronously build the same venv in parallel. That, though, has been the case for >3 years and Pex has long since had in its internals an atomic directory abstraction that uses a POSIX file lock to ensure just one racing job can create a given venv directory at a time, basically all the jobs view for
/path/to/venvs/.<hash>.lck
, the lock winner creates
/path/to/venvs/<hash>.<uuid>
and populates it, and then as its last 2 steps, does an atomic
os.rename("/path/to/venvs/<hash>.<uuid>", "/path/to/venvs/<hash>")
and then releases the file lock. Keep in mind as you debug this - the issue only happens for you in CI. Presumably both your machine and CI have multiple cores. It may be the case that your machine has more cores than the CI job, in which case your machine is more likely to race than CI! So the things to focus on are CI differences from your machine. One is the POSIX file lock and that's why I asked if any CI paths were on a network mount for example. Its "well known" that older NFS (v3 or older) does not support POSIX file locks.
b

bored-glass-58755

09/07/2022, 3:22 PM
Ok, I can see the lock file inside the venv but whether or not that is working is based on my CI infra.
9 Views