https://pantsbuild.org/ logo
Join Slack
Powered by
Hi, I am getting following error while running `./...
# general
b
Hi, I am getting following error while running 
./pants package
goal in my CI:
Copy code
pex.result.ResultError: There were 2 errors downloading required artifacts:
1. setuptools 57.5 from <https://files.pythonhosted.org/packages/4b/b9/71687c5d2034c863db1db2e0704f5e27581ff3cb44d7f293968c5e08ceb3/setuptools-57.5.0-py3-none-any.whl>
    <urlopen error [Errno 97] Address family not supported by protocol>
2. wheel 0.37.1 from <https://files.pythonhosted.org/packages/27/d6/003e593296a85fd6ed616ed962795b2f87709c3eee2bca4f6d0fe55c6d00/wheel-0.37.1-py2.py3-none-any.whl>
    <urlopen error [Errno 97] Address family not supported by protocol>
This is most likely due to restricted access but I have 
indexes
(not 
indexes.add
) configured under 
[python-repos]
subsystem in my 
pants.toml
. Is there anything else that I can do?
f
I’d check the contents of this section. This is the format I use locally:
Copy code
[python-repos]
indexes = ["<https://username:password@my.company.com/pypi/simple>", "<https://pypi.python.org/simple>"]
in CI, we extend an env var:
Copy code
PANTS_PYTHON_REPOS_INDEXES="+['<https://username>:${secret}@my.company.com/pypi/simple']"
How does the authentication work? You could perhaps verify first that you can access the private PyPI by attempting to install an arbitrary package using 
pip
?
b
Thank you for your response. Do we need to extend the environment variable you mentioned even if 
indexes
are mentioned in 
pants.toml
? Thanks for the 
pip install
suggestion, I shall try that.
Ok, so I extended the env variable 
PANTS_PYTHON_REPOS_INDEXES
and it is still failing. Also, I tried pip installing a package and it was downloaded successfully only that it did not install because of permissions in the CI's venv. The original error I reported was it was not able to even download the packages because it was trying to download it from 
<https://files.pythonhosted.org>
whereas I have mentioned the 
indexes
under 
[python-repo]
subsystem of 
pants.toml
.
f
Do we need to extend the environment variable you mentioned even if 
indexes
are mentioned in 
pants.toml
?
no, you don’t have to. I mentioned an env var because you wouldn’t normally want to store the username/password as part of your Pants configuration file hence the secret collection
Also, I tried pip installing a package and it was downloaded successfully
I assume you’ve provided the additional index to the 
pip install
command that’s identical to what you have in your 
pants.toml
?
b
Ok, I don't need any username/password to access the repository when I am running it thru CI.
👍 1
f
okay, so only need to provide the extra index URL, gotcha
b
I assume you’ve provided the additional index to the 
pip install
command that’s identical to what you have in your 
pants.toml
?
I didn't, but I assume CI is configured to check the internal repo only by default.
f
if your CI doesn’t have access the public PyPI, then I assume you have all the wheels required hosted in a private one, right? e.g. 
setuptools
and 
wheel
b
okay, so only need to provide the extra index URL, gotcha
Yes, and I have done that in 
pants.toml
. Is there a way to see what options are in effect from 
pants.toml
for a particular run of pants?
if your CI doesn’t have access the public PyPI, then I assume you have all the wheels required hosted in a private one, right? e.g. 
setuptools
and 
wheel
That's right!
I am just surprised why is it going to 
<https://files.pythonhosted.org>
to download those packages?
f
indeed, it shouldn’t go there. Do you have only a single index configured, your private one?
b
Yes, using the 
indexes
option and not 
indexes.add
.
👍 1
f
well, running with 
-ldebug
would give you more information.
b
Ok, I just ran with 
--print-stacktrace
to find out which module is it fetching the public repo from. I will try with 
-ldebug
.
f
Also, I tried pip installing a package and it was downloaded successfully
did pip download the package from your private PyPI?
b
It didn't give such info but I'd assume it will download from there only. The basis of my assumption is I have another CI which uses conda setup and there it provides such information as to where the package is being downloaded from. I do not want to setup conda in my Pants CI.
f
you don’t need conda 🙂
b
Yeah
f
It didn’t give such info
well 
pip
itself needs to be configured to go to other PyPI directory — e.g. using 
pip.conf
file or by passing extra command line arguments
any useful information found when you run with 
-ldebug
?
b
Here's the stack trace:
Copy code
Engine traceback:
  in select
  in pants.core.goals.package.package_asset
  in pants.backend.python.goals.setup_py.package_python_dist (client:ts-client)
  in pants.backend.python.util_rules.dists.run_pep517_build
  in pants.backend.python.util_rules.pex.create_venv_pex (build_backend.pex)
  in pants.backend.python.util_rules.pex.build_pex (build_backend.pex)
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/$XDG_CACHE_HOME/pants/setup/bootstrap-Linux-x86_64/2.12.0_py37/lib/python3.7/site-packages/pants/engine/process.py", line 277, in fallible_to_exec_result_or_raise
    process_cleanup=process_cleanup.val,
pants.engine.process.ProcessExecutionFailure: Process 'Building build_backend.pex from setuptools_default.lock' failed with exit code 1.
And a part snapshot (redacted) of the debug:
Copy code
09:40:31.51 [DEBUG] Running Building build_backend.pex from setuptools_default.lock under semaphore with concurrency id: 2, and concurrency: 8
09:40:31.51 [INFO] Starting: Building build_backend.pex from setuptools_default.lock
09:40:31.51 [DEBUG] Starting: setup_sandbox
09:40:31.53 [DEBUG] Completed: setup_sandbox
09:40:31.57 [DEBUG] spawned local process as Some(122689) for Process { argv: ["/usr/local/python/3.7.6/bin/python3.7", "./pex", "--tmpdir", ".tmp", "--jobs", "8", "--python-path", "/protoc/3.20.1/bin:/redis/6.2.5/bin:/bazel/4.0.0/bin:/golang/1.15/bin:/venv/bin:/nodejs/v12/bin:/usr/local/java/openjdk_8u302_x64/bin:/root/usr/bin:/usr/local/ext/cmake/3.17.1/bin:/maven/3.6.3/bin:/usr/local/ext/git/2.24.0/bin:/usr/local/ext/git/2.24.0/libexec/git-core:/usr/local/perl/5.26.1/bin:/sonar/wrapper/6.32/bin:/sonar/scanner/4.7.0/bin:/helm/3.6.1:/dotnet/3.1:/usr/local/sybase/bin:/etc:/bin:/etc:bin:/usr/bin:/bin", "--output-file", "build_backend.pex", "--no-emit-warnings", "--manylinux", "manylinux2014", "--venv", "--seed", "verbose", "--venv-site-packages-copies", "--python", "/usr/local/python/3.7.6/bin/python3.7", "--sources-directory=source_files", "--lock", "setuptools_default.lock", "--no-pypi", "--index=<http://private.repo.url/simple>", "--index=<http://private.repo.url/simple>", "--layout", "packed"], env: {"CPPFLAGS": "", "LANG": "en_US.UTF-8", "LC_ALL": "en_US.UTF-8", "LDFLAGS": "", "PATH": "/protoc/3.20.1/bin:/redis/6.2.5/bin:/bazel/4.0.0/bin:/golang/1.15/bin:/venv/bin:/nodejs/v12/bin:/usr/local/java/openjdk_8u302_x64/bin:/root/usr/bin:/usr/local/ext/cmake/3.17.1/bin:/maven/3.6.3/bin:/usr/local/ext/git/2.24.0/bin:/usr/local/ext/git/2.24.0/libexec/git-core:/usr/local/perl/5.26.1/bin:/sonar/wrapper/6.32/bin:/sonar/scanner/4.7.0/bin:/helm/3.6.1:/dotnet/3.1:/usr/local/sybase/bin:/etc:/bin:/etc:/bin:/usr/bin:/bin", "PEX_IGNORE_RCFILES": "true", "PEX_ROOT": ".cache/pex_root"}, working_directory: None, input_digests: InputDigests { complete: DirectoryDigest { digest: Digest { hash: Fingerprint<0fe2393a7ac8d77f0c987b7edd9962b1c7c9e23264ca982596b13a0c7f446830>, size_bytes: 340 }, tree: "Some(..)" }, nailgun: DirectoryDigest { digest: Digest { hash: Fingerprint<e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855>, size_bytes: 0 }, tree: "Some(..)" }, input_files: DirectoryDigest { digest: Digest { hash: Fingerprint<0fe2393a7ac8d77f0c987b7edd9962b1c7c9e23264ca982596b13a0c7f446830>, size_bytes: 340 }, tree: "Some(..)" }, immutable_inputs: {}, use_nailgun: [] }, output_files: {}, output_directories: {RelativePath("build_backend.pex")}, timeout: None, execution_slot_variable: None, concurrency_available: 10, description: "Building build_backend.pex from setuptools_default.lock", level: Info, append_only_caches: {CacheName("pex_root"): RelativePath(".cache/pex_root")}, jdk_home: None, platform_constraint: Some(Linux_x86_64), cache_scope: Successful }
09:40:33.88 [INFO] Completed: Building build_backend.pex from setuptools_default.lock
09:40:33.88 [DEBUG] Completed: Scheduling: Building build_backend.pex from setuptools_default.lock
09:40:33.90 [DEBUG] Completed: pants.backend.python.util_rules.pex.build_pex
09:40:33.90 [DEBUG] Completed: `package` goal
09:40:33.90 [DEBUG] computed 1 nodes in 25.237837 seconds. there are 945 total nodes.
09:40:33.90 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Building build_backend.pex from setuptools_default.lock' failed with exit code 1.
One thing that concerns me in debug output is why 
PEX_ROOT
is set as 
./cache/pex_root
where as I set it as 
$XDG_CACHE_HOME/pants/named_caches/pex_root
at the beginning of the CI. Also, the 4th line from last in debug output suggests that package goal is complete whereas according to the stack trace its not.
f
Copy code
"--index=<http://private.repo.url/simple>", "--index=<http://private.repo.url/simple>"
so 
pex
got these arguments, that’s good
also forgot to ask, are you able to run the goal locally at all?
b
Yes, I was able to run the goal locally. Also, I have got a way to proceed from this thread here: https://pantsbuild.slack.com/archives/C046T6T9U/p1661957446116169?thread_ts=1661952554.253599&amp;cid=C046T6T9U
f
oh yes, the lock files!
👍 1
b
Thank you for your help @fresh-cat-90827 🙂
f
well, couldn’t help much, but learned a few things along the way! 😄 good luck with the further work, I’ll keep an eye on your future posts!
👍 1
5 Views
Open in Slack
PreviousNext
Powered by