https://pantsbuild.org/ logo
b

bored-glass-58755

11/10/2022, 7:32 PM
Hi, I am trying to create a wheel from a
python_distribution
target but the
package
goal is failing. Following is the trail of the debug output:
14:26:04.48 [DEBUG] Completed: Resolve transitive targets
14:26:14.43 [DEBUG] Completed: Downloading: <https://github.com/protocolbuffers/protobuf/releases/download/v3.20.1/protoc-3.20.1-osx-aarch_64.zip> (2.58 MB)
14:26:14.43 [DEBUG] Completed: pants.core.util_rules.external_tool.download_external_tool
14:26:14.43 [DEBUG] Completed: Generate Python from Protobuf
14:26:14.43 [DEBUG] Completed: pants.backend.python.util_rules.python_sources.prepare_python_sources
14:26:14.43 [DEBUG] Completed: pants.backend.python.util_rules.python_sources.strip_python_sources
14:26:14.43 [DEBUG] Completed: pants.backend.python.util_rules.python_sources.prepare_python_sources
14:26:14.43 [DEBUG] Canceled: pants.backend.python.util_rules.python_sources.strip_python_sources
`142614.43 [DEBUG] Completed:
package
goal`
14:26:14.43 [DEBUG] computed 1 nodes in 10.502206 seconds. there are 262 total nodes.
14:26:14.43 [ERROR] 1 Exception encountered:
Exception: Error downloading file: error sending request for url (<https://github.com/protocolbuffers/protobuf/releases/download/v3.20.1/protoc-3.20.1-osx-aarch_64.zip>): error trying to connect: proxy authentication required
I have tried adding the subsystem
subprocess-environment
with proxy variables in
pants.toml
but the error remains the same. I am using python 3.9 on an M1 Mac. My question is if the debug output says that package goal completed, why and what is it failing for?
h

happy-kitchen-89482

11/10/2022, 8:57 PM
Sounds like you’re running behind a firewall that restricts access to the internet? and you’ve followed the instructions here https://www.pantsbuild.org/docs/restricted-internet-access?
b

bored-glass-58755

11/10/2022, 9:00 PM
Yes, I have added proxies under
subprocess-environment
in
pants.toml
. But more importantly, what do the debug logs trying to convey?
h

happy-kitchen-89482

11/10/2022, 10:42 PM
Completed:
is probably a misleading word, it just means that the attempt completed, not that it was successful…
The error seems to be that your firewall is blocking access, and setting the proxy env vars didn’t work?
b

bored-glass-58755

11/10/2022, 10:44 PM
Ok, any leads on how to get around with it?
h

happy-kitchen-89482

11/10/2022, 10:46 PM
Setting HTTP_PROXY/HTTPS_PROXY, that is?
b

bored-glass-58755

11/10/2022, 10:48 PM
Yes.
I left their values so they pick whatever is there in the env.
h

happy-kitchen-89482

11/10/2022, 10:59 PM
And you used the exact same name (including case) in
env_vars.add =
as the names in your env?
1
b

bored-glass-58755

11/10/2022, 11:03 PM
curl -O <https://github.com/protocolbuffers/protobuf/releases/download/v3.20.1/protoc-3.20.1-osx-aarch_64.zip>
is downloading the file successfully from the same terminal tab I am running package goal from.
I am now getting this error:
18:06:51.94 [INFO] Initializing scheduler...
18:06:52.12 [INFO] Scheduler initialized.
18:06:54.47 [ERROR] 1 Exception encountered:
Exception: Error downloading file: error sending request for url (<https://github.com/protocolbuffers/protobuf/releases/download/v3.20.1/protoc-3.20.1-osx-aarch_64.zip>): error trying to connect: invalid peer certificate contents: invalid peer certificate: UnknownIssuer
I am using the variable
ca_certs_path
under
GLOBAL
scope. The value of this variable is taken from the command:
curl -vvv -O <https://github.com/protocolbuffers/protobuf/releases/download/v3.20.1/protoc-3.20.1-osx-aarch_64.zip> 2>&1 | grep CAfile
h

happy-kitchen-89482

11/10/2022, 11:30 PM
BTW if the env vars you were using were upper case, try lowercase, or vice versa?
b

bored-glass-58755

11/10/2022, 11:33 PM
I used them in both cases finally which made that error go and now I am facing this certification error.
h

happy-kitchen-89482

11/10/2022, 11:33 PM
That’s hard to debug without seeing the contents of that file, which obviously you should not send or post
b

bored-glass-58755

11/10/2022, 11:34 PM
You mean contents of pants.toml?
h

happy-kitchen-89482

11/10/2022, 11:34 PM
No, of the CA certs file
What happens if you set
SSL_CERT_FILE
to that file?
b

bored-glass-58755

11/10/2022, 11:34 PM
Same error.
But how would contents of CA cert file matter if curl is working fine in the same terminal?
h

happy-kitchen-89482

11/10/2022, 11:39 PM
Unclear, this is being passed through to the Rust TLS client, which tends to be really strict, see https://github.com/briansmith/webpki/issues/232, but I don’t think that’s the issue here
What happens if you don’t set
ca_certs_path
? Same error? different error?
b

bored-glass-58755

11/10/2022, 11:40 PM
Same error with or without setting that.
Did I mention I am using an M1 Mac, could that be a problem?
h

happy-kitchen-89482

11/10/2022, 11:47 PM
Hmm, I doubt it. Probably that var isn’t being plumbed through for some reason
Let’s see
It does look like it’s plumbed through, hmm….
What happens if you set
ca_certs_path
to a nonexistent file? I just want to check that it’s being read at all…
b

bored-glass-58755

11/11/2022, 12:27 AM
Yes, its being read. Changed it to:
ca_certs_path = "/path/to/certs/file"
and it threw an error:
File "$HOME/.cache/pants/setup/bootstrap-Darwin-arm64/2.14.0_py39/lib/python3.9/site-packages/pants/engine/internals/scheduler.py", line 215, in __init__
self._py_scheduler = native_engine.scheduler_create(
ValueError: Error reading root CA certs file /path/to/certs/file: No such file or directory (os error 2)
19:25:17.46 [INFO] waiting for pantsd to start...
19:25:22.45 [INFO] waiting for pantsd to start...
19:25:27.43 [INFO] waiting for pantsd to start...
19:25:32.41 [INFO] waiting for pantsd to start...
19:25:37.48 [INFO] waiting for pantsd to start...
^CInterrupted by user:
h

happy-kitchen-89482

11/11/2022, 12:28 AM
And you’re pointing it to a .pem file?
b

bored-glass-58755

11/11/2022, 12:51 AM
No
I used
ca_certs_path = "/path/to/certs/file"
at first when I got that error I sent. But I tried again changing it to
ca_certs_path = "/path/to/certs/file.pem"
and got the same
ValueError
I sent you before.
h

happy-kitchen-89482

11/11/2022, 1:56 AM
./pants --ca-certs-path=/etc/ssl/cert.pem
works for me with a valid pem file
(and fails on an invalid one)
so I suspect an issue with your certs file somehow
maybe curl accepts a wider set of formats or something
b

bored-glass-58755

11/11/2022, 1:59 AM
Ok, so
package
goal doesn't use curl in the background?
h

happy-kitchen-89482

11/11/2022, 2:34 AM
To clarify, this isn’t specific to the package goal, your code depends on some protobufs, so Pants downloads the proto compiler in order to compile them into python code.
b

bored-glass-58755

11/11/2022, 2:34 AM
This is the stacktrace if this would help:
Engine traceback:
in select
in pants.core.goals.package.package_asset
in pants.backend.python.goals.setup_py.package_python_dist (src/proto:proto-lib)
in pants.backend.python.goals.setup_py.generate_chroot
in pants.backend.python.goals.setup_py.get_sources
in pants.backend.python.util_rules.python_sources.strip_python_sources
in pants.backend.python.util_rules.python_sources.prepare_python_sources
in pants.core.util_rules.source_files.determine_source_files
in pants.engine.internals.graph.hydrate_sources (src/proto/helloworld.proto:protos)
in pants.backend.codegen.protobuf.python.rules.generate_python_from_protobuf
in pants.core.util_rules.external_tool.download_external_tool
in downloaded_file
Traceback (no traceback):
<pants native internals>
Exception: Error downloading file: error sending request for url (<https://github.com/protocolbuffers/protobuf/releases/download/v3.20.1/protoc-3.20.1-osx-aarch_64.zip>): error trying to connect: invalid peer certificate contents: invalid peer certificate: UnknownIssuer
h

happy-kitchen-89482

11/11/2022, 2:34 AM
But that downloading code doesn’t use curl, no, it uses Rust’s HTTP client library
🆗 1
b

bored-glass-58755

11/11/2022, 2:35 AM
Yes, I understand this is not specific to
package
goal.
If I understand correctly from this issue that you shared, I can overcome the error by creating a fork of Pants with no
rusttls-tls
feature on this line and use this fork on my machine to build the repository I am working on?
h

happy-kitchen-89482

11/11/2022, 3:36 AM
I’m really not sure it’s related to that issue, TBH
And I don’t know that this would overcome the issue
That was a very specific issue with a cert that was invalid by the spec, but accepted by other tls implementations in practice
Very very nuanced
Here I think it’s more likely that your certs file is not a valid pem file somehow. Esp since you’re getting the same error when you point it to a dummy file, righT?
b

bored-glass-58755

11/11/2022, 4:18 AM
When I point it to a dummy file (with or without .pem extension), it gives me a ValueError saying no file or directory found. This is not the same error when I use the correct cert file.
Moreover, the same cert file is working with curl.
Maybe rust is too strict?
I have an observation though - the issuer certificate that gets listed when I use curl with the verbose option doesn’t exist in the cert.pem file. It (the cert), however, is present in the Keychain app.
h

happy-kitchen-89482

11/11/2022, 3:46 PM
Interesting
so the next thing to try might be exporting the keychain certs to a pem file
I think this does that:
security find-certificate -a -p > keychain_certs.pem
b

bored-glass-58755

11/11/2022, 4:21 PM
That worked, great. Thanks! This command did not help but then I exported the certs from keychain manually and
package
worked. I exported all root certificates and then only the one which I know curl was using based on issuing authority, bundled them all in one file and it worked. And now in the subsequent runs it doesn't matter if I provide a cert file or not and which file do I provide because it seems it has cached the things required to build that one target. Is there a way to clear that cache because I want to try a couple of things before I proceed from here.
h

happy-kitchen-89482

11/11/2022, 5:00 PM
Glad that’s working. To test without the cache you can run with
--no-pantsd --no-local-cache
b

bored-glass-58755

11/11/2022, 7:20 PM
Cool, let me try that!
These options don't help. It is still building the wheel even after removing the
ca_certs_path
from
pants.toml
.
h

happy-kitchen-89482

11/12/2022, 1:32 AM
Hmm, try nuking ~/.cache/pants/ and .pants.d/ in your repo, and running with those flags
b

bored-glass-58755

11/12/2022, 1:38 AM
Wouldn’t nuking
~/.cache/pants
mean bootstrapping Pants again?
h

happy-kitchen-89482

11/15/2022, 6:04 AM
Yes, but we’re trying to eliminate whatever caches are preventing that download from rerunning
👍 1
5 Views