bored-glass-58755
08/31/2022, 1:29 PMpants.toml
options are in effect during a particular run of Pants e.g. if I am running ./pants package
, I would like to see what python-repos
options are active.enough-analyst-54434
08/31/2022, 1:35 PM-ldebug
to the Pants command line. This prints out the full set of environment variables and argv for every process Pants invokes.fast-nail-55400
08/31/2022, 1:35 PM./pants help python-repos
and the configured value should be listed next to every option.bored-glass-58755
08/31/2022, 1:37 PMenough-analyst-54434
08/31/2022, 1:41 PMbored-glass-58755
08/31/2022, 1:46 PM./pants help python-repos
outputs:
`python-repos` subsystem options
--------------------------------
External Python code repositories, such as PyPI.
These options may be used to point to custom cheeseshops when resolving requirements.
Activated by pants.core
Config section: [python-repos]
None available.
Advanced options available. You can list them by running ./pants help-advanced python-repos.
It doesn't show me anything even though I have the env variable PANTS_PYTHON_REPOS_INDEXES
set in my CI along with the option in pants.toml
.enough-analyst-54434
08/31/2022, 1:49 PMpython-repos
subsystem, those options are bread and butter. Try ./pants help-advanced python-repos
bored-glass-58755
08/31/2022, 2:00 PM`python-repos` subsystem advanced options
-----------------------------------------
--python-repos-repos="['<str>', '<str>', ...]"
PANTS_PYTHON_REPOS_REPOS
repos
default: []
current value: []
URLs of code repositories to look for requirements. In Pip and Pex, this option
corresponds to the `--find-links` option.
--python-repos-indexes="['<str>', '<str>', ...]"
PANTS_PYTHON_REPOS_INDEXES
indexes
default: [
"<https://pypi.org/simple/>"
]
current value: [
"<http://private.repo.url/simple>",
"<http://private.repo.url/simple>"
] (from pants.toml, from env var PANTS_PYTHON_REPOS_INDEXES)
URLs of code repository indexes to look for requirements. If set to an empty list, then
Pex will use no indices (meaning it will not use PyPI). The values should be compliant
with PEP 503.
`python-repos` subsystem deprecated options
-------------------------------------------
None available.
The default is still set to pypi, could that be a problem?enough-analyst-54434
08/31/2022, 2:01 PMcurrent value
looks good to you but behavior of Pants indicates that's not what is being used.bored-glass-58755
08/31/2022, 2:04 PMpex.result.ResultError: There were 2 errors downloading required artifacts:
1. setuptools 57.5 from <https://files.pythonhosted.org/packages/4b/b9/71687c5d2034c863db1db2e0704f5e27581ff3cb44d7f293968c5e08ceb3/setuptools-57.5.0-py3-none-any.whl>
<urlopen error [Errno 97] Address family not supported by protocol>
2. wheel 0.37.1 from <https://files.pythonhosted.org/packages/27/d6/003e593296a85fd6ed616ed962795b2f87709c3eee2bca4f6d0fe55c6d00/wheel-0.37.1-py2.py3-none-any.whl>
<urlopen error [Errno 97] Address family not supported by protocol>
Engine traceback:
in select
in pants.core.goals.package.package_asset
in pants.backend.python.goals.setup_py.package_python_dist (client:ts-client)
in pants.backend.python.util_rules.dists.run_pep517_build
in pants.backend.python.util_rules.pex.create_venv_pex (build_backend.pex)
in pants.backend.python.util_rules.pex.build_pex (build_backend.pex)
in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
File "/$XDG_CACHE_HOME/pants/setup/bootstrap-Linux-x86_64/2.12.0_py37/lib/python3.7/site-packages/pants/engine/process.py", line 277, in fallible_to_exec_result_or_raise
process_cleanup=process_cleanup.val,
pants.engine.process.ProcessExecutionFailure: Process 'Building build_backend.pex from setuptools_default.lock' failed with exit code 1.
And a part snapshot (redacted) of the debug:
09:40:31.51 [DEBUG] Running Building build_backend.pex from setuptools_default.lock under semaphore with concurrency id: 2, and concurrency: 8
09:40:31.51 [INFO] Starting: Building build_backend.pex from setuptools_default.lock
09:40:31.51 [DEBUG] Starting: setup_sandbox
09:40:31.53 [DEBUG] Completed: setup_sandbox
09:40:31.57 [DEBUG] spawned local process as Some(122689) for Process { argv: ["/usr/local/python/3.7.6/bin/python3.7", "./pex", "--tmpdir", ".tmp", "--jobs", "8", "--python-path", "/protoc/3.20.1/bin:/redis/6.2.5/bin:/bazel/4.0.0/bin:/golang/1.15/bin:/venv/bin:/nodejs/v12/bin:/usr/local/java/openjdk_8u302_x64/bin:/root/usr/bin:/usr/local/ext/cmake/3.17.1/bin:/maven/3.6.3/bin:/usr/local/ext/git/2.24.0/bin:/usr/local/ext/git/2.24.0/libexec/git-core:/usr/local/perl/5.26.1/bin:/sonar/wrapper/6.32/bin:/sonar/scanner/4.7.0/bin:/helm/3.6.1:/dotnet/3.1:/usr/local/sybase/bin:/etc:/bin:/etc:bin:/usr/bin:/bin", "--output-file", "build_backend.pex", "--no-emit-warnings", "--manylinux", "manylinux2014", "--venv", "--seed", "verbose", "--venv-site-packages-copies", "--python", "/usr/local/python/3.7.6/bin/python3.7", "--sources-directory=source_files", "--lock", "setuptools_default.lock", "--no-pypi", "--index=<http://private.repo.url/simple>", "--index=<http://private.repo.url/simple>", "--layout", "packed"], env: {"CPPFLAGS": "", "LANG": "en_US.UTF-8", "LC_ALL": "en_US.UTF-8", "LDFLAGS": "", "PATH": "/protoc/3.20.1/bin:/redis/6.2.5/bin:/bazel/4.0.0/bin:/golang/1.15/bin:/venv/bin:/nodejs/v12/bin:/usr/local/java/openjdk_8u302_x64/bin:/root/usr/bin:/usr/local/ext/cmake/3.17.1/bin:/maven/3.6.3/bin:/usr/local/ext/git/2.24.0/bin:/usr/local/ext/git/2.24.0/libexec/git-core:/usr/local/perl/5.26.1/bin:/sonar/wrapper/6.32/bin:/sonar/scanner/4.7.0/bin:/helm/3.6.1:/dotnet/3.1:/usr/local/sybase/bin:/etc:/bin:/etc:/bin:/usr/bin:/bin", "PEX_IGNORE_RCFILES": "true", "PEX_ROOT": ".cache/pex_root"}, working_directory: None, input_digests: InputDigests { complete: DirectoryDigest { digest: Digest { hash: Fingerprint<0fe2393a7ac8d77f0c987b7edd9962b1c7c9e23264ca982596b13a0c7f446830>, size_bytes: 340 }, tree: "Some(..)" }, nailgun: DirectoryDigest { digest: Digest { hash: Fingerprint<e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855>, size_bytes: 0 }, tree: "Some(..)" }, input_files: DirectoryDigest { digest: Digest { hash: Fingerprint<0fe2393a7ac8d77f0c987b7edd9962b1c7c9e23264ca982596b13a0c7f446830>, size_bytes: 340 }, tree: "Some(..)" }, immutable_inputs: {}, use_nailgun: [] }, output_files: {}, output_directories: {RelativePath("build_backend.pex")}, timeout: None, execution_slot_variable: None, concurrency_available: 10, description: "Building build_backend.pex from setuptools_default.lock", level: Info, append_only_caches: {CacheName("pex_root"): RelativePath(".cache/pex_root")}, jdk_home: None, platform_constraint: Some(Linux_x86_64), cache_scope: Successful }
09:40:33.88 [INFO] Completed: Building build_backend.pex from setuptools_default.lock
09:40:33.88 [DEBUG] Completed: Scheduling: Building build_backend.pex from setuptools_default.lock
09:40:33.90 [DEBUG] Completed: pants.backend.python.util_rules.pex.build_pex
09:40:33.90 [DEBUG] Completed: `package` goal
09:40:33.90 [DEBUG] computed 1 nodes in 25.237837 seconds. there are 945 total nodes.
09:40:33.90 [ERROR] 1 Exception encountered:
ProcessExecutionFailure: Process 'Building build_backend.pex from setuptools_default.lock' failed with exit code 1.
One thing that concerns me in debug output is why PEX_ROOT
is set as ./cache/pex_root
where as I set it as $XDG_CACHE_HOME/pants/named_caches/pex_root
at the beginning of the CI. Also, the 4th line from last in debug output suggests that package goal is complete whereas according to the stack trace its not.enough-analyst-54434
08/31/2022, 2:48 PMPEX_ROOT
is a relative path in the sandbox to a symlink that points to $XDG_CACHE_HOME/pants/named_caches/pex_root
. No matter how you configure `PEX_ROOT`externally, Pants will always use that local sandbox path with a symlink to where you set it to.Running Building build_backend.pex from setuptools_default.lock ...
That means you ran into this: https://github.com/pantsbuild/pants/issues/16328
In short, you'll need to customize the default lock file location and re-generate the lock for every tool Pants uses. Pants ships with open-source lock files for all the tools it uses. When you don't have access to open-source repos, you need to regenerate all those locks.bored-glass-58755
08/31/2022, 2:58 PMtools
directory lockfile location in your example tools/lockfile
needs to be in repo-root?enough-analyst-54434
08/31/2022, 3:22 PMbored-glass-58755
08/31/2022, 3:55 PMPIP_INDEX_URL=<http://private.repo.url/simple> ./pants --version
and it ended as following:
Installing pantsbuild.pants==2.12.0 into a virtual environment at /$XDG_CACHE_HOME/pants/setup/bootstrap-Linux-x86_64/2.12.0_py37
WARNING: The repository located at private.repo.url is not a trusted or secure host and is being ignored. If this repository is available via HTTPS we recommend you use HTTPS instead, otherwise you may silence this warning and allow it anyway with '--trusted-host private.repo.url'.
WARNING: The repository located at private.repo.url is not a trusted or secure host and is being ignored. If this repository is available via HTTPS we recommend you use HTTPS instead, otherwise you may silence this warning and allow it anyway with '--trusted-host private.repo.url'.
ERROR: Could not find a version that satisfies the requirement setuptools<58 (from versions: none)
ERROR: No matching distribution found for setuptools<58
WARNING: The repository located at private.repo.url is not a trusted or secure host and is being ignored. If this repository is available via HTTPS we recommend you use HTTPS instead, otherwise you may silence this warning and allow it anyway with '--trusted-host private.repo.url'.
##[error]Bash exited with code '1'.
If lines #2 & #3 are just warnings and if it did not use private.repo.url
, why could not it download setuptools just how it was doing it before passing in the PIP_INDEX_URL
?
Also, the only change I have made is in pants.toml
by introducing the [setuptools]
subsystem with lockfile = tools/setuptools.lock
. I haven't added any other subsystems and also haven't made any change to pants script as it shows in the GitHub link you provided (in function bootstrap_virtualenv()
)."${staging_dir}/install/bin/pip" install --trusted-host private.repo.url --quiet -U pip "setuptools<58" && \
"${staging_dir}/install/bin/pip" install ${maybe_find_links} --trusted-host private.repo.url --quiet --progress-bar off "${pants_requirement}"
Installing pantsbuild.pants==2.12.0 into a virtual environment at /$XDG_CACHE_HOME/pants/setup/bootstrap-Linux-x86_64/2.12.0_py37
ERROR: Could not find a version that satisfies the requirement pantsbuild.pants==2.12.0 (from versions: none)
ERROR: No matching distribution found for pantsbuild.pants==2.12.0
##[error]Bash exited with code '1'.
enough-analyst-54434
08/31/2022, 4:13 PMbored-glass-58755
08/31/2022, 4:14 PMenough-analyst-54434
08/31/2022, 4:27 PMPIP_<option>
bored-glass-58755
08/31/2022, 5:17 PM@@ -270,7 +270,9 @@ function bootstrap_virtualenv {
staging_dir=$(tempdir "${PANTS_BOOTSTRAP}")
cd "${staging_dir}"
echo "${VIRTUALENV_REQUIREMENTS}" > requirements.txt
- "${python}" "${pex_path}" -r requirements.txt -c virtualenv -o virtualenv.pex
+ "${python}" "${pex_path}" -r requirements.txt -c virtualenv -o virtualenv.pex \
+ --no-pypi \
+ --index <https://pextest.jfrog.io/artifactory/api/pypi/default-pypi/simple/>
enough-analyst-54434
08/31/2022, 5:59 PMbored-glass-58755
08/31/2022, 6:03 PMPIP_INDEX_URL=<http://private.repo.url/simple> PIP_TRUSTED_HOST=private.repo.url PIP_VERBOSE=3 ./pants --version
and it couldn't find pantsbuild.pants==2.12.0
in there which is expected. Is there a way that I bootstrap first and then use the PIP_INDEX_URL
to generate-lockfiles
? Or do I have to bootstrap using this private repo URL as PIP_INDEX_URL
for it to work? If yes, then am I expected to onboard pantsbuild.pants onto the private repo? The logs are quite long but I can provide them if you don't mind going thru the long logs.enough-analyst-54434
08/31/2022, 6:25 PMit couldn'tDoes this mean http://private.repo.url/simple is not a mirror of PyPI, aka it is missing some distributions that are otherwise on PyPI?in there which is expected.find pantsbuild.pants==2.12.0
bored-glass-58755
08/31/2022, 6:25 PMenough-analyst-54434
08/31/2022, 6:25 PMpip install pantsbuild.pants==<pants version>
is the only part of the process that does not have a lock file right now. In order to figure out what distributions you need to mirror to make that work you'll need to run do the following on every platform you target (dev machines and CI):
/python/we/use/python -mvenv figure-out-what-pants-needs
figure-out-what-pants-needs/bin/pip wheel --wheel-dir=wheels-pants-needs pantsbuild.pants==<pants version>
You'd then need to add the contents of the wheels-pants-needs/
directory from each platform you target to your private index.
You'll need to do this for every Pants upgrade; so it will behoove you to script it up after getting it all working. Pants will need to address all this awkwardness at some point, but until then!bored-glass-58755
08/31/2022, 6:42 PMenough-analyst-54434
08/31/2022, 6:46 PMbored-glass-58755
08/31/2022, 6:48 PMenough-analyst-54434
08/31/2022, 6:48 PMbored-glass-58755
08/31/2022, 7:00 PMenough-analyst-54434
08/31/2022, 7:16 PMpip wheel pantsbuild.pants
and gather wheels and put them in your private index.
2. For each platform and for each (Python) tool Pants uses: Find the default tool lockfile checked into the Pants repo and gather all the wheels and sdists listed that apply to your platforms and put them in your private index.
3. Now bootstrap Pants using the private index and run ./pants generate-lockfiles
then check in all the lock files.
At this point Panst can run on dev machines and in CI.pants-shinkwrap migrate --manifest manifest.json pants.toml
.pants-shrinkwrap download
on a machine with public internet access, but then the rest could be done in a locked down environment.bored-glass-58755
08/31/2022, 7:33 PMpackage
and tries to download from pythonhosted.org are setuptools
and wheel
(as demonstrated in the starting message of this thread). I assume those wheels are present in the default setuptools.lock
file in the Pants repo and are also present in my private repo. If I download that file and replace the pythonhosted.org URL with my private.repo.url along with versions and hashes that it needs and check this modified setuptools.lock
file in to my code repo under tools/
directory. Thereafter, I have my pants.toml
modified to have [setuptools]
subsystem with line lockfile = "tools/setuptools.lock"
. After all this, I assume - if I understand it all correctly - that my package
goal would run successfully.enough-analyst-54434
08/31/2022, 7:36 PMlockfile = ...
and then run ./pants generate-lockfiles
bored-glass-58755
08/31/2022, 7:40 PMpants.toml
or it will re-generate all lockfiles? The config in pants.toml
is only to provide a lockfile location in the absence of which it will generate files in the default location?enough-analyst-54434
08/31/2022, 7:51 PM./pants generate-lockfiles
will generate lockfiles for every subsystem in pants.toml
that has an explicit non-default `lockfile`defined. So, by default it will do nothing. It only does things once you have one or more non-default lockfile defined. This is additional reason why this is all so paintful right now. There is no obvious list of which subsystems you use and thus which ones you'll need to define a custom `lockfile`location for.lockfile
for a subsystem that needs one, you currently find out with the style of error you led with in the original post.bored-glass-58755
08/31/2022, 7:54 PMwheels-that-pants-needs
enough-analyst-54434
08/31/2022, 8:03 PMbored-glass-58755
08/31/2022, 8:04 PMenough-analyst-54434
08/31/2022, 8:05 PMbored-glass-58755
08/31/2022, 8:06 PMenough-analyst-54434
08/31/2022, 8:06 PMpants
script. That bootstraps pants and has configuration for the tools it needs to do the job. One of which is download a Pex PEX.bored-glass-58755
08/31/2022, 8:08 PMenough-analyst-54434
08/31/2022, 8:09 PMTo do 1 it needs the Pex PEX which then downloads virtualenv and its depenedncies and then it creates a virtualenv and then installs Pants in that virtualenv using Pip.That is "the bootstrap"
bored-glass-58755
08/31/2022, 8:10 PMPIP_INDEX_URL
(not sure how). The only place where I am stuck at is because it is unable to download anything from pythonhosted.org and that's where my package
goal is failing in CI which is a linux platform. What if I just do following on a linux dev box:
1. Bootstrap pants (which it is able to do)
2. Modify pants.toml
for setuptools to generate a lockfile with PIP_INDEX_URL.
3. Verify the generated lockfile against the private repo to ensure that I have the required wheels.
4. Check this in to code repo and run CIenough-analyst-54434
08/31/2022, 8:27 PMMy CI is bootstrapping pants just fine without the use ofI think you need to fully understand this. If you don't the rest will be a muddled jumbled chain of hacks around this central key missing fact.(not sure how).PIP_INDEX_URL
bored-glass-58755
08/31/2022, 8:27 PMenough-analyst-54434
08/31/2022, 8:30 PMbored-glass-58755
08/31/2022, 8:46 PMenough-analyst-54434
08/31/2022, 8:52 PMbored-glass-58755
08/31/2022, 10:54 PMpackage
goal because the subprocess doesn't have access to env proxy variables: https://github.com/pantsbuild/pants/issues/15675. I was able to go past that error by setting proxy variables env.add
under [subprocess-environment]
subsystem in pants.toml
but now package
is throwing below:
File "/$XDG_CACHE_HOME/pants/named_caches/pex_root/installed_wheels/6242b902db69f59e1092b406655c0fb1634486c47ce563f5fd27277cf4561822/pex-2.1.90-py2.py3-none-any.whl/pex/jobs.py", line 396, in handle_spawn_error
raise self._raise_type(self.spawn_error_message(item, exception))
pex.resolve.resolvers.Untranslatable: Failed to spawn a job for InstallRequest(target=LocalInterpreter(id='usr.local.python.3.7.6.bin.python3.7', platform=Platform(platform='manylinux_2_17_x86_64', impl='cp', version='3.7.6', version_info=(3, 7, 6), abi='cp37m'), marker_environment=MarkerEnvironment(implementation_name='cpython', implementation_version='3.7.6', os_name='posix', platform_machine='x86_64', platform_python_implementation='CPython', platform_release='3.10.0-1160.71.1.el7.x86_64', platform_system='Linux', platform_version='#1 SMP Wed Jun 15 08:55:08 UTC 2022', python_full_version='3.7.6', python_version='3.7', sys_platform='linux'), interpreter=PythonInterpreter('/usr/local/python/3.7.6/bin/python3.7', PythonIdentity('/usr/local/python/3.7.6/bin/python3.7', 'cp37', 'cp37m', 'manylinux_2_17_x86_64', (3, 7, 6)))), wheel_path='/$XDG_CACHE_HOME/pants/named_caches/pex_root/downloads/4bdcd7d840138086126cd09254dc6195fb4fc6f01c050a1d7236f2630db1d22a/wheel-0.37.1-py2.py3-none-any.whl', fingerprint='4bdcd7d840138086126cd09254dc6195fb4fc6f01c050a1d7236f2630db1d22a'): [Errno 2] No such file or directory: '/$XDG_CACHE_HOME/pants/named_caches/pex_root/venvs/5632a685cd3903cb1ab183bd3e3fb9767d9d2e42/15d3ad9788dc0da4a814dd5bb1a36eed666cd279/pex': '/$XDG_CACHE_HOME/pants/named_caches/pex_root/venvs/5632a685cd3903cb1ab183bd3e3fb9767d9d2e42/15d3ad9788dc0da4a814dd5bb1a36eed666cd279/pex'
Could you please help me with that?enough-analyst-54434
08/31/2022, 11:01 PMwheel_path='/$XDG_CACHE_HOME/pants/named_caches/pex_root/downloads/4bdcd7d840138086126cd09254dc6195fb4fc6f01c050a1d7236f2630db1d22a/wheel-0.37.1-py2.py3-none-any.whl'
$XDG_CACHE_HOME
in your posts, it appears the env var is not getting set up correctly.bored-glass-58755
08/31/2022, 11:04 PM/pants/named_caches/...
enough-analyst-54434
08/31/2022, 11:09 PMbored-glass-58755
08/31/2022, 11:18 PM$XDG_CACHE_HOME is /some/prefix/_work/36334/.xdg-cache
File "/some/prefix/_work/36334/.xdg-cache/pants/named_caches/pex_root/installed_wheels/6242b902db69f59e1092b406655c0fb1634486c47ce563f5fd27277cf4561822/pex-2.1.90-py2.py3-none-any.whl/pex/jobs.py", line 396, in handle_spawn_error
raise self._raise_type(self.spawn_error_message(item, exception))
pex.resolve.resolvers.Untranslatable: Failed to spawn a job for InstallRequest(target=LocalInterpreter(id='usr.local.python.3.7.6.bin.python3.7', platform=Platform(platform='manylinux_2_17_x86_64', impl='cp', version='3.7.6', version_info=(3, 7, 6), abi='cp37m'), marker_environment=MarkerEnvironment(implementation_name='cpython', implementation_version='3.7.6', os_name='posix', platform_machine='x86_64', platform_python_implementation='CPython', platform_release='3.10.0-1160.71.1.el7.x86_64', platform_system='Linux', platform_version='#1 SMP Wed Jun 15 08:55:08 UTC 2022', python_full_version='3.7.6', python_version='3.7', sys_platform='linux'), interpreter=PythonInterpreter('/usr/local/python/3.7.6/bin/python3.7', PythonIdentity('/usr/local/python/3.7.6/bin/python3.7', 'cp37', 'cp37m', 'manylinux_2_17_x86_64', (3, 7, 6)))), wheel_path='/some/prefix/_work/36334/.xdg-cache/pants/named_caches/pex_root/downloads/4bdcd7d840138086126cd09254dc6195fb4fc6f01c050a1d7236f2630db1d22a/wheel-0.37.1-py2.py3-none-any.whl', fingerprint='4bdcd7d840138086126cd09254dc6195fb4fc6f01c050a1d7236f2630db1d22a'): [Errno 2] No such file or directory: '/some/prefix/_work/36334/.xdg-cache/pants/named_caches/pex_root/venvs/5632a685cd3903cb1ab183bd3e3fb9767d9d2e42/15d3ad9788dc0da4a814dd5bb1a36eed666cd279/pex': '/some/prefix/_work/36334/.xdg-cache/pants/named_caches/pex_root/venvs/5632a685cd3903cb1ab183bd3e3fb9767d9d2e42/15d3ad9788dc0da4a814dd5bb1a36eed666cd279/pex'
enough-analyst-54434
08/31/2022, 11:31 PMbored-glass-58755
09/01/2022, 1:20 AM.xdg_cache
directory is generated by CI. I just set XDG_CACHE_HOME variable to that path and then make additional directories pants/setup
and pants/named_caches/pex_root
to make paths $XDG_CACHE_HOME/pants/setup
and $XDG_CACHE_HOME/pants/named_caches/pex_root
.enough-analyst-54434
09/01/2022, 1:57 AMbored-glass-58755
09/01/2022, 2:15 AMpackage
goal. I have also echoed the values for $XDG_CACHE_HOME
& $PEX_ROOT
. I only set the former in this run and so, the later is empty.enough-analyst-54434
09/01/2022, 5:46 PM09:19:02.39 [DEBUG] Completed: Downloading: DownloadFile(url='<https://github.com/pantsbuild/pex/releases/download/v2.1.90/pex>', expected_digest=FileDigest('2781255baf77c2a8fdc85c5e830f7191a6048fd91d2e20b5c7a20e5a0b7beb66', 3755345)... (1 characters truncated)
That's public and it worked. Elsewhere there is evidence this may work because a proxy is configured:
"http_proxy": "<http://corp.proxy.com:8080>", "https_proxy": "<http://corp.proxy.com:8080>", "no_proxy": "internal.url1,internal.url2,internal.url3,tools-that-we-use.url"
However, elsewhere still there is still use of a private PyPI:
"--no-pypi", "--index=<http://private.repo.url/simple>"
That private PyPI will be used only to create new lockfiles - say for your own code's 3rdparty deps - but it will not be used for existing lock files that ship with Pants for all its tools.bored-glass-58755
09/01/2022, 7:00 PM$XDG_CACHE_HOME
to /path/to/CI/workspace/directory/_work/33042/.xdg-cache
(basically _work/33042/
is also a part of CI workspace path but I thought I need not redact it). The directory where CI checks out the code and which becomes the repo root is /path/to/CI/workspace/directory/_work/33042/s
.
2. Create dirs $XDG_CACHE_HOME/pants/setup
and $XDG_CACHE_HOME/pants/named_caches/pex_root
.
3. Set $PANTS_SETUP_CACHE
to $XDG_CACHE_HOME/pants/setup
and $PEX_ROOT
to $XDG_CACHE_HOME/pants/named_caches/pex_root
.
4. Bootstrap Pants.
5. Do a ls -ltra /path/to/CI/workspace/directory/_work/33042/.xdg-cache/pants/named_caches/pex_root/downloads/
because this is what the wheel path is in the failing step's output. Also do a ls -ltra /path/to/CI/workspace/directory/_work/33042/.xdg-cache/pants/named_caches/pex_root/venvs/
because this is the path to directory or file that the step is unable to find.
6. Do pants tailor --check.
7. Repeat step #5 again (Output of this step is shared below)
8. Run pants -ldebug package
Starting: LS step BEFORE Package
Generating script.
========================== Starting Command Output ===========================
Today's date
Thu Sep 1 13:30:36 EDT 2022
> ls -ltra /path/to/CI/workspace/directory/_work/23715/.xdg-cache/pants/named_caches/pex_root/downloads/
total 0
drwxrwsr-x. 12 owner group 229 Sep 1 13:29 ..
drwxrwsr-x. 2 owner group 10 Sep 1 13:30 .
> ls -ltra /path/to/CI/workspace/directory/_work/23715/.xdg-cache/pants/named_caches/pex_root/venvs/
total 0
drwxrwsr-x. 4 owner group 75 Sep 1 13:29 .
drwxrwsr-x. 3 owner group 72 Sep 1 13:29 s
drwxrwsr-x. 3 owner group 136 Sep 1 13:29 15158fe84406f2733a98f687f98d59238ad4210d
drwxrwsr-x. 12 owner group 229 Sep 1 13:29 ..
> ls -ltra /path/to/CI/workspace/directory/_work/23715/.xdg-cache/pants/named_caches/pex_root/venvs/15158fe84406f2733a98f687f98d59238ad4210d/
total 0
-rwxrwxr-x. 1 owner group 0 Sep 1 13:29 .15d3ad9788dc0da4a814dd5bb1a36eed666cd279.atomic_directory.lck
drwxrwsr-x. 4 owner group 75 Sep 1 13:29 ..
drwxrwsr-x. 5 owner group 156 Sep 1 13:29 15d3ad9788dc0da4a814dd5bb1a36eed666cd279
drwxrwsr-x. 3 owner group 136 Sep 1 13:29 .
Finishing: LS step BEFORE Package
The package
is failing with following missing file or directory:
[Errno 2] No such file or directory: '/path/to/CI/workspace/directory/_work/4052/.xdg-cache/pants/named_caches/pex_root/venvs/5632a685cd3903cb1ab183bd3e3fb9767d9d2e42/15d3ad9788dc0da4a814dd5bb1a36eed666cd279/pex': '/path/to/CI/workspace/directory/_work/4052/.xdg-cache/pants/named_caches/pex_root/venvs/5632a685cd3903cb1ab183bd3e3fb9767d9d2e42/15d3ad9788dc0da4a814dd5bb1a36eed666cd279/pex'
The path getting generated in my CI is /path/to/CI/workspace/directory/_work/4052/.xdg-cache/pants/named_caches/pex_root/venvs/_*15158fe84406f2733a98f687f98d59238ad4210d*_/15d3ad9788dc0da4a814dd5bb1a36eed666cd279/pex
Basically the directory inside the pex_root/venvs/
has different name (5632a685cd3903cb1ab183bd3e3fb9767d9d2e42
) that the package
goal expects and the directory name actually present is (15158fe84406f2733a98f687f98d59238ad4210d
).
As a fix, I tried adding following steps before running package in the above sequence of steps:
7a. A step mv actual-dir-name-present-in-the-path expected-dir-name
7b. Again ls -ltra
over all the directories that I was doing before and included the directories actual-dir-name-present-in-the-path
and expected-dir-name
. This 7b step showed that there is no actual-dir-name-present-in-the-path
anymore which means that mv was successful.
But the package command failed even after that with the same error that directory or file is not found./pex
. Not sure if this is a directory or a file. Does it make sense?enough-analyst-54434
09/01/2022, 9:13 PM/path/to/CI/workspace/directory/_work/4052/.xdg-cache/
known to be empty or non-empty before the Pants run in CI? I'd like it to be empty for the purposes of debugging from a clean slate.bored-glass-58755
09/01/2022, 9:18 PM/path/to/CI/workspace/directory/_work/33042/.xdg-cache/pants/named_caches/pex_root/downloads/
/path/to/CI/workspace/directory/_work/33042/.xdg-cache/pants/named_caches/pex_root/venvs/
/path/to/CI/workspace/directory/_work/33042/.xdg-cache/pants/named_caches/pex_root/venvs/s
/path/to/CI/workspace/directory/_work/4052/.xdg-cache/pants/named_caches/pex_root/venvs/15158fe84406f2733a98f687f98d59238ad4210d
/path/to/CI/workspace/directory/_work/4052/.xdg-cache/pants/named_caches/pex_root/venvs/5632a685cd3903cb1ab183bd3e3fb9767d9d2e42
I will just check /path/to/CI/workspace/directory/_work/33042/.xdg-cache/
nowwork/
in above. I copied those paths from my previous long chat which had outputs from two different pipeline runs - one before fix and one after fixenough-analyst-54434
09/01/2022, 9:23 PMCreate dirsWhen you say this, you mean you're manually making these directories? You should not need to do this. Pants automatically makes the directories it needs if they do not exist. Avoiding manually creating these directories avoids potential permission issues (which it does not look like you're hitting) and reduces the surface area of debugging suspects.and$XDG_CACHE_HOME/pants/setup
.$XDG_CACHE_HOME/pants/named_caches/pex_root
bored-glass-58755
09/01/2022, 9:26 PM$XDG_CACHE_HOME/pants/setup
$XDG_CACHE_HOME/pants/named_caches/pex_root
I learnt this structure from my local Mac's bootstrapped Pants.enough-analyst-54434
09/01/2022, 9:27 PMbored-glass-58755
09/01/2022, 9:33 PMStarting: LS $XDG_CACHE_HOME before bootstrap
Generating script.
========================== Starting Command Output ===========================
> ls -ltra /path/to/CI/workspace/directory/_work/19378/.xdg-cache
total 12
drwxrwxr-x. 7 owner group 4096 Sep 1 17:26 ..
drwxrwxr-x. 3 owner group 4096 Sep 1 17:26 .
drwxrwxr-x. 4 owner group 4096 Sep 1 17:26 pants
Finishing: LS $XDG_CACHE_HOME before bootstrap
enough-analyst-54434
09/01/2022, 11:02 PM./pants
using what's checked in to the repo. No edits to the pants
script and no edits pants.toml
and no custom environment variables. My problem is I have lost track of what you may have modified to try to get CI working. So, for me, starting with a completely blank slate is the only way to build back up a confident picture of the variables in play. And having a confident picture of those variables is critical to debugging something, 5x so when remotely debugging blind.bored-glass-58755
09/01/2022, 11:44 PM./pants --version
failed because the CI doesn't have perms to create $HOME/.cache/pants
.
2. Set $XDG_CACHE_HOME
to /path/to/CI/workspace/directory/work/<num>/_._xdg_cache
and re-ran ./pants --version
. It gave some PEX_ROOT related warnings that $HOME/.pex
is un-writable and that it fell back to a temp directory and warned me that it will hurt performance. But apart from this warning, it failed this time for env vars $LC_ALL
and $LANG
saying that they were set to ISO-8859-1
but Pants wants UTF-8
.
3. Set $LC_ALL
and $LANG
to en_US.UTF-8
along with $XDG_CACHE_HOME
as in step #2 and reran ./pants --version
. It bootstrapped fine but with the PEX_ROOT warnings that $HOME/.pex
is un-writable and that it fell back to a temp directory and warned me that it will hurt performance. After that the CI continued to run ./pants tailor --check
which completed successfully. The third step in CI was to package
a python_distribution()
and it failed because it could not download setuptools
and wheel
from <http://files.pythonhosted.org|files.pythonhosted.org>
. It needs proxies here because the subprocess that runs package
doesn't have access to the default proxies that CI uses from the environment (because of which it was able to bootstrap Pants in the first place).
4. Set proxies under [subprocess-environment]
in pants.toml
and re-ran CI. This time it failed to download setuptools
and wheel
because of SSL certificate
error.
5. I found out the certificate path and set the variable $PANTS_CA_CERTS_PATH
with that path in CI and re-ran. This time the package
step failed with the same error that it can't find the file or directory. The directory names under the directory /pex_root/venvs/
are exactly the same as before and so, pasting the error here from earlier chats (so please ignore the <num>
under _work/
) so that its easier to have a look at it at a single place:
[Errno 2] No such file or directory: '/path/to/CI/workspace/directory/_work/4052/.xdg-cache/pants/named_caches/pex_root/venvs/5632a685cd3903cb1ab183bd3e3fb9767d9d2e42/15d3ad9788dc0da4a814dd5bb1a36eed666cd279/pex': '/path/to/CI/workspace/directory/_work/4052/.xdg-cache/pants/named_caches/pex_root/venvs/5632a685cd3903cb1ab183bd3e3fb9767d9d2e42/15d3ad9788dc0da4a814dd5bb1a36eed666cd279/pex'
Apart from this, following is `pants.toml`:
[GLOBAL]
pants_version = "2.12.0"
backend_packages.add = [
"pants.backend.python",
"pants.backend.python.typecheck.mypy",
"pants.backend.python.lint.flake8",
"pants.backend.experimental.python.lint.autoflake",
"pants.backend.plugin_development",
"pants.backend.codegen.protobuf.python",
"pants.backend.experimental.python"
]
[source]
root_patterns = [
"/",
"/test",
"/module1",
"/module2",
"/module3",
"/module4",
"/module5"
]
[subprocess-environment]
env_vars.add = ["http_proxy", "https_proxy", "no_proxy"]
Please let me know if anything is not clear and I can try to answer that.enough-analyst-54434
09/02/2022, 12:08 AMrm -rf /path/to/CI/workspace/directory/_work/4052/.xdg-cache/pants/named_caches/pex_root
and then re-run ./pants package ...
(Modifying that path to reflect whatever it really is. There should be a bunch of deletes. Perhaps use `rm -rfv ...`to get a flood of messages from rm
proving it is removing things.bored-glass-58755
09/02/2022, 3:02 AMpackage
goal:
rm -rfv ${XDG_CACHE_HOME}/pants/named_caches/pex_root
and there are two cases that happened:
Case 1:
It seemed like that in this case, the CI ran in a workspace where it had already ran (sometime) earlier and so, it did not have to "bootstrap" Pants. Pants was already there. See the output for the ./pants --version
(ignore the warning):
Starting: Pants bootstrap step
Generating script.
Script contents:
./pants --version
========================== Starting Command Output ===========================
22:26:21.55 [INFO] Initializing scheduler...
22:26:21.86 [INFO] Scheduler initialized.
22:26:21.91 [WARN] Please either set `enabled = true` in the [anonymous-telemetry] section of pants.toml to enable sending anonymous stats to the Pants project to aid development, or set `enabled = false` to disable it. No telemetry sent for this run. An explicit setting will get rid of this message. See <https://www.pantsbuild.org/v2.12/docs/anonymous-telemetry> for details.
2.12.0
Finishing: Pants bootstrap step
And please find attached the output of the delete step.
Case 2:
This was more of a normal run where Pants bootstrapped (I could see the green lines that said Bootstrapping pants, Downloading PEX, SHA256 Verified, Installing pantsbuild.pants, etc. I could also see the PEX_ROOT un-writable warning) but there were no directories to delete in this run in the delete step that was introduced:
Starting: Deleting ${XDG_CACHE_HOME}/pants/named_caches/pex_root
Generating script.
========================== Starting Command Output ===========================
> rm -rfv /path/to/CI/workspace/directory/_work/34538/.xdg-cache/pants/named_caches/pex_root
Finishing: Deleting ${XDG_CACHE_HOME}/pants/named_caches/pex_root
The package
goal ended with the same error as before in both the cases.File "/path/to/CI/workspace/directory/_work/13694/.xdg-cache/pants/named_caches/pex_root/installed_wheels/6242b902db69f59e1092b406655c0fb1634486c47ce563f5fd27277cf4561822/pex-2.1.90-py2.py3-none-any.whl/pex/jobs.py", line 396, in handle_spawn_error
raise self._raise_type(self.spawn_error_message(item, exception))
pex.resolve.resolvers.Untranslatable: Failed to spawn a job for InstallRequest(target=LocalInterpreter(id='usr.local.python.3.7.6.bin.python3.7', platform=Platform(platform='manylinux_2_17_x86_64', impl='cp', version='3.7.6', version_info=(3, 7, 6), abi='cp37m'), marker_environment=MarkerEnvironment(implementation_name='cpython', implementation_version='3.7.6', os_name='posix', platform_machine='x86_64', platform_python_implementation='CPython', platform_release='3.10.0-1160.71.1.el7.x86_64', platform_system='Linux', platform_version='#1 SMP Wed Jun 15 08:55:08 UTC 2022', python_full_version='3.7.6', python_version='3.7', sys_platform='linux'), interpreter=PythonInterpreter('/usr/local/python/3.7.6/bin/python3.7', PythonIdentity('/usr/local/python/3.7.6/bin/python3.7', 'cp37', 'cp37m', 'manylinux_2_17_x86_64', (3, 7, 6)))), wheel_path='/path/to/CI/workspace/directory/_work/13694/.xdg-cache/pants/named_caches/pex_root/downloads/4bdcd7d840138086126cd09254dc6195fb4fc6f01c050a1d7236f2630db1d22a/wheel-0.37.1-py2.py3-none-any.whl', fingerprint='4bdcd7d840138086126cd09254dc6195fb4fc6f01c050a1d7236f2630db1d22a'): [Errno 2] No such file or directory: '/path/to/CI/workspace/directory/_work/13694/.xdg-cache/pants/named_caches/pex_root/venvs/7590ee51593acd29997ca5259a9f2ec093c0559a/15d3ad9788dc0da4a814dd5bb1a36eed666cd279/pex': '/path/to/CI/workspace/directory/_work/13694/.xdg-cache/pants/named_caches/pex_root/venvs/7590ee51593acd29997ca5259a9f2ec093c0559a/15d3ad9788dc0da4a814dd5bb1a36eed666cd279/pex'
The delete step's output is attached in a .txt
file.
And lastly, here's the output of the re-run of the package
goal:
File "/path/to/CI/workspace/directory/_work/13694/.xdg-cache/pants/named_caches/pex_root/installed_wheels/6242b902db69f59e1092b406655c0fb1634486c47ce563f5fd27277cf4561822/pex-2.1.90-py2.py3-none-any.whl/pex/jobs.py", line 396, in handle_spawn_error
raise self._raise_type(self.spawn_error_message(item, exception))
pex.resolve.resolvers.Untranslatable: Failed to spawn a job for InstallRequest(target=LocalInterpreter(id='usr.local.python.3.7.6.bin.python3.7', platform=Platform(platform='manylinux_2_17_x86_64', impl='cp', version='3.7.6', version_info=(3, 7, 6), abi='cp37m'), marker_environment=MarkerEnvironment(implementation_name='cpython', implementation_version='3.7.6', os_name='posix', platform_machine='x86_64', platform_python_implementation='CPython', platform_release='3.10.0-1160.71.1.el7.x86_64', platform_system='Linux', platform_version='#1 SMP Wed Jun 15 08:55:08 UTC 2022', python_full_version='3.7.6', python_version='3.7', sys_platform='linux'), interpreter=PythonInterpreter('/usr/local/python/3.7.6/bin/python3.7', PythonIdentity('/usr/local/python/3.7.6/bin/python3.7', 'cp37', 'cp37m', 'manylinux_2_17_x86_64', (3, 7, 6)))), wheel_path='/path/to/CI/workspace/directory/_work/13694/.xdg-cache/pants/named_caches/pex_root/downloads/4bdcd7d840138086126cd09254dc6195fb4fc6f01c050a1d7236f2630db1d22a/wheel-0.37.1-py2.py3-none-any.whl', fingerprint='4bdcd7d840138086126cd09254dc6195fb4fc6f01c050a1d7236f2630db1d22a'): [Errno 2] No such file or directory: '/path/to/CI/workspace/directory/_work/13694/.xdg-cache/pants/named_caches/pex_root/venvs/7590ee51593acd29997ca5259a9f2ec093c0559a/15d3ad9788dc0da4a814dd5bb1a36eed666cd279/pex': '/path/to/CI/workspace/directory/_work/13694/.xdg-cache/pants/named_caches/pex_root/venvs/7590ee51593acd29997ca5259a9f2ec093c0559a/15d3ad9788dc0da4a814dd5bb1a36eed666cd279/pex'
It's weird that it did not find the directory while running the package
goal first but then deleted the same directory in the very next step and then again failed for the same directory not found.enough-analyst-54434
09/06/2022, 7:04 PMbored-glass-58755
09/06/2022, 7:22 PMpackage
and publish
work on my local with the exception that publish
only works for docker targets as I don't have access to push a wheel to private repo using my user ID. It has to be done via a CI which is a separate thing and not a Pants issue. Let me also try to run the existing pipeline with 2.3.0rc3.pex-2.1.90
but I am using the latest script that has `2.1.103`:
File "/path/to/CI/workspace/directory/_work/13694/.xdg-cache/pants/named_caches/pex_root/installed_wheels/6242b902db69f59e1092b406655c0fb1634486c47ce563f5fd27277cf4561822/pex-2.1.90-py2.py3-none-any.whl/pex/jobs.py", line 396, in handle_spawn_error
raise self._raise_type(self.spawn_error_message(item, exception))
enough-analyst-54434
09/07/2022, 4:26 AMbored-glass-58755
09/07/2022, 4:31 AMenough-analyst-54434
09/07/2022, 2:03 PM/path/to/venvs/.<hash>.lck
, the lock winner creates /path/to/venvs/<hash>.<uuid>
and populates it, and then as its last 2 steps, does an atomic os.rename("/path/to/venvs/<hash>.<uuid>", "/path/to/venvs/<hash>")
and then releases the file lock.
Keep in mind as you debug this - the issue only happens for you in CI. Presumably both your machine and CI have multiple cores. It may be the case that your machine has more cores than the CI job, in which case your machine is more likely to race than CI!
So the things to focus on are CI differences from your machine. One is the POSIX file lock and that's why I asked if any CI paths were on a network mount for example. Its "well known" that older NFS (v3 or older) does not support POSIX file locks.bored-glass-58755
09/07/2022, 3:22 PM