bored-glass-58755
08/30/2022, 8:39 PM./pants package
goal in my CI:
pex.result.ResultError: There were 2 errors downloading required artifacts:
1. setuptools 57.5 from <https://files.pythonhosted.org/packages/4b/b9/71687c5d2034c863db1db2e0704f5e27581ff3cb44d7f293968c5e08ceb3/setuptools-57.5.0-py3-none-any.whl>
<urlopen error [Errno 97] Address family not supported by protocol>
2. wheel 0.37.1 from <https://files.pythonhosted.org/packages/27/d6/003e593296a85fd6ed616ed962795b2f87709c3eee2bca4f6d0fe55c6d00/wheel-0.37.1-py2.py3-none-any.whl>
<urlopen error [Errno 97] Address family not supported by protocol>
This is most likely due to restricted access but I have indexes
(not indexes.add
) configured under [python-repos]
subsystem in my pants.toml
. Is there anything else that I can do?fresh-cat-90827
08/31/2022, 9:00 AM[python-repos]
indexes = ["<https://username:password@my.company.com/pypi/simple>", "<https://pypi.python.org/simple>"]
in CI, we extend an env var:
PANTS_PYTHON_REPOS_INDEXES="+['<https://username>:${secret}@my.company.com/pypi/simple']"
How does the authentication work? You could perhaps verify first that you can access the private PyPI by attempting to install an arbitrary package using pip
?bored-glass-58755
08/31/2022, 11:46 AMindexes
are mentioned in pants.toml
?
Thanks for the pip install
suggestion, I shall try that.PANTS_PYTHON_REPOS_INDEXES
and it is still failing. Also, I tried pip installing a package and it was downloaded successfully only that it did not install because of permissions in the CI's venv. The original error I reported was it was not able to even download the packages because it was trying to download it from <https://files.pythonhosted.org>
whereas I have mentioned the indexes
under [python-repo]
subsystem of pants.toml
.fresh-cat-90827
08/31/2022, 1:19 PMDo we need to extend the environment variable you mentioned even ifno, you don’t have to. I mentioned an env var because you wouldn’t normally want to store the username/password as part of your Pants configuration file hence the secret collectionare mentioned inindexes
?pants.toml
Also, I tried pip installing a package and it was downloaded successfullyI assume you’ve provided the additional index to the
pip install
command that’s identical to what you have in your pants.toml
?bored-glass-58755
08/31/2022, 1:20 PMfresh-cat-90827
08/31/2022, 1:21 PMbored-glass-58755
08/31/2022, 1:21 PMI assume you’ve provided the additional index to theI didn't, but I assume CI is configured to check the internal repo only by default.command that’s identical to what you have in yourpip install
?pants.toml
fresh-cat-90827
08/31/2022, 1:22 PMsetuptools
and wheel
bored-glass-58755
08/31/2022, 1:22 PMokay, so only need to provide the extra index URL, gotchaYes, and I have done that in
pants.toml
. Is there a way to see what options are in effect from pants.toml
for a particular run of pants?if your CI doesn’t have access the public PyPI, then I assume you have all the wheels required hosted in a private one, right? e.g.That's right!andsetuptools
wheel
<https://files.pythonhosted.org>
to download those packages?fresh-cat-90827
08/31/2022, 1:25 PMbored-glass-58755
08/31/2022, 1:26 PMindexes
option and not indexes.add
.fresh-cat-90827
08/31/2022, 1:30 PM-ldebug
would give you more information.bored-glass-58755
08/31/2022, 1:31 PM--print-stacktrace
to find out which module is it fetching the public repo from. I will try with -ldebug
.fresh-cat-90827
08/31/2022, 1:33 PMAlso, I tried pip installing a package and it was downloaded successfullydid pip download the package from your private PyPI?
bored-glass-58755
08/31/2022, 1:35 PMfresh-cat-90827
08/31/2022, 2:11 PMbored-glass-58755
08/31/2022, 2:11 PMfresh-cat-90827
08/31/2022, 2:11 PMIt didn’t give such infowell
pip
itself needs to be configured to go to other PyPI directory — e.g. using pip.conf
file or by passing extra command line arguments-ldebug
?bored-glass-58755
08/31/2022, 2:24 PMEngine traceback:
in select
in pants.core.goals.package.package_asset
in pants.backend.python.goals.setup_py.package_python_dist (client:ts-client)
in pants.backend.python.util_rules.dists.run_pep517_build
in pants.backend.python.util_rules.pex.create_venv_pex (build_backend.pex)
in pants.backend.python.util_rules.pex.build_pex (build_backend.pex)
in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
File "/$XDG_CACHE_HOME/pants/setup/bootstrap-Linux-x86_64/2.12.0_py37/lib/python3.7/site-packages/pants/engine/process.py", line 277, in fallible_to_exec_result_or_raise
process_cleanup=process_cleanup.val,
pants.engine.process.ProcessExecutionFailure: Process 'Building build_backend.pex from setuptools_default.lock' failed with exit code 1.
And a part snapshot (redacted) of the debug:
09:40:31.51 [DEBUG] Running Building build_backend.pex from setuptools_default.lock under semaphore with concurrency id: 2, and concurrency: 8
09:40:31.51 [INFO] Starting: Building build_backend.pex from setuptools_default.lock
09:40:31.51 [DEBUG] Starting: setup_sandbox
09:40:31.53 [DEBUG] Completed: setup_sandbox
09:40:31.57 [DEBUG] spawned local process as Some(122689) for Process { argv: ["/usr/local/python/3.7.6/bin/python3.7", "./pex", "--tmpdir", ".tmp", "--jobs", "8", "--python-path", "/protoc/3.20.1/bin:/redis/6.2.5/bin:/bazel/4.0.0/bin:/golang/1.15/bin:/venv/bin:/nodejs/v12/bin:/usr/local/java/openjdk_8u302_x64/bin:/root/usr/bin:/usr/local/ext/cmake/3.17.1/bin:/maven/3.6.3/bin:/usr/local/ext/git/2.24.0/bin:/usr/local/ext/git/2.24.0/libexec/git-core:/usr/local/perl/5.26.1/bin:/sonar/wrapper/6.32/bin:/sonar/scanner/4.7.0/bin:/helm/3.6.1:/dotnet/3.1:/usr/local/sybase/bin:/etc:/bin:/etc:bin:/usr/bin:/bin", "--output-file", "build_backend.pex", "--no-emit-warnings", "--manylinux", "manylinux2014", "--venv", "--seed", "verbose", "--venv-site-packages-copies", "--python", "/usr/local/python/3.7.6/bin/python3.7", "--sources-directory=source_files", "--lock", "setuptools_default.lock", "--no-pypi", "--index=<http://private.repo.url/simple>", "--index=<http://private.repo.url/simple>", "--layout", "packed"], env: {"CPPFLAGS": "", "LANG": "en_US.UTF-8", "LC_ALL": "en_US.UTF-8", "LDFLAGS": "", "PATH": "/protoc/3.20.1/bin:/redis/6.2.5/bin:/bazel/4.0.0/bin:/golang/1.15/bin:/venv/bin:/nodejs/v12/bin:/usr/local/java/openjdk_8u302_x64/bin:/root/usr/bin:/usr/local/ext/cmake/3.17.1/bin:/maven/3.6.3/bin:/usr/local/ext/git/2.24.0/bin:/usr/local/ext/git/2.24.0/libexec/git-core:/usr/local/perl/5.26.1/bin:/sonar/wrapper/6.32/bin:/sonar/scanner/4.7.0/bin:/helm/3.6.1:/dotnet/3.1:/usr/local/sybase/bin:/etc:/bin:/etc:/bin:/usr/bin:/bin", "PEX_IGNORE_RCFILES": "true", "PEX_ROOT": ".cache/pex_root"}, working_directory: None, input_digests: InputDigests { complete: DirectoryDigest { digest: Digest { hash: Fingerprint<0fe2393a7ac8d77f0c987b7edd9962b1c7c9e23264ca982596b13a0c7f446830>, size_bytes: 340 }, tree: "Some(..)" }, nailgun: DirectoryDigest { digest: Digest { hash: Fingerprint<e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855>, size_bytes: 0 }, tree: "Some(..)" }, input_files: DirectoryDigest { digest: Digest { hash: Fingerprint<0fe2393a7ac8d77f0c987b7edd9962b1c7c9e23264ca982596b13a0c7f446830>, size_bytes: 340 }, tree: "Some(..)" }, immutable_inputs: {}, use_nailgun: [] }, output_files: {}, output_directories: {RelativePath("build_backend.pex")}, timeout: None, execution_slot_variable: None, concurrency_available: 10, description: "Building build_backend.pex from setuptools_default.lock", level: Info, append_only_caches: {CacheName("pex_root"): RelativePath(".cache/pex_root")}, jdk_home: None, platform_constraint: Some(Linux_x86_64), cache_scope: Successful }
09:40:33.88 [INFO] Completed: Building build_backend.pex from setuptools_default.lock
09:40:33.88 [DEBUG] Completed: Scheduling: Building build_backend.pex from setuptools_default.lock
09:40:33.90 [DEBUG] Completed: pants.backend.python.util_rules.pex.build_pex
09:40:33.90 [DEBUG] Completed: `package` goal
09:40:33.90 [DEBUG] computed 1 nodes in 25.237837 seconds. there are 945 total nodes.
09:40:33.90 [ERROR] 1 Exception encountered:
ProcessExecutionFailure: Process 'Building build_backend.pex from setuptools_default.lock' failed with exit code 1.
One thing that concerns me in debug output is why PEX_ROOT
is set as ./cache/pex_root
where as I set it as $XDG_CACHE_HOME/pants/named_caches/pex_root
at the beginning of the CI. Also, the 4th line from last in debug output suggests that package goal is complete whereas according to the stack trace its not.fresh-cat-90827
08/31/2022, 3:26 PMsoCopy code"--index=<http://private.repo.url/simple>", "--index=<http://private.repo.url/simple>"
pex
got these arguments, that’s goodbored-glass-58755
08/31/2022, 3:28 PMfresh-cat-90827
08/31/2022, 3:29 PMbored-glass-58755
08/31/2022, 3:36 PMfresh-cat-90827
08/31/2022, 3:39 PM