https://pantsbuild.org/ logo
Join Slack
Powered by
Hi guys, my company is using self-signed certifica...
# general
b
Hi guys, my company is using self-signed certificates so when i try to initialize a "pants project" (just running 
pants
) i get this:
Copy code
Failed to determine release URL for Pants: 2.30.2: pants.2.30.2-cp311-linux_x86_64.pex: URL check failed: <https://github.com/pantsbuild/pants/releases/download/release_2.30.2/pants.2.30.2-cp311-linux_x86_64.pex>: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Missing Authority Key Identifier (_ssl.c:1081)>
Has anyone encountered this before?
Does pants use my system python? I realized pants is written in rust
w
Pants is a complicated beast. It uses Rust + Python, but the python used for Pants itself is served from the scie-pants runner. System (or other) Python is used for your application
https://github.com/pantsbuild/scie-pants/issues/403 This seems to be the same problem?
b
Ok. Yeah it seems to be the same issue
I also am using red hat like him
Is there a way to specify pants to use my python installation instead of using its own?
I think that's causing the issue since my company has some custom ssl python glue
w
I believe so, but you also don’t “need” scie-pants - it’s just a convenience to grab specific versions of python + pants.
b
Okay, i'm not using scie-pants if i just downloaded the pants binary and put it in ~/.local/bin?
w
https://github.com/pantsbuild/scie-pants#firewall-support This describes how you can point at other pythons
If you downloaded that pants binary, then you downloaded scie-pants. Sec, let me grab some examples
❤️ 1
https://www.pantsbuild.org/stable/docs/getting-started/installing-pants#running-pants-from-sources I was trying to find the old ./pants script - but I’ll need to dig through some git history. Essentially that would let you run a system python that you had to manage. Then there are the bootstrap URLs in scie-pants, as mentioned above. Then, one that will hopefully make it into an upcoming Pants release is the idea of using “fat-pants” which is that you download a launcher, python interpreter, and specific version of pants at once. I use a variant of this in CI, and we have some loose PRs around this. I'd recommend you look in the pants and scie-pants documentations for firewall/no-network support, as there are several things downloaded at first-run, and then as you change linters and stuff. So, you’ll get hit by SSL issues for a lot of those. The last point, which I have the least familiarity with as I don’t need to do it, is to point scie-pants/pants to your SSL/CA certs
This last one is discussed in the docs, and in various issues around the repos, and I don’t recall the latest state of the art or suggestions, but I know other pants users are able to get this working
You might be able to try cloning and running the example-python repo, then revert to that commit, and try again - see what happens
b
Okay cool thanks for the help i'll dig some more
👍 1
I could "solve" my issue temporarily just by using those firewall rules you linked 👍
w
Great!
b
well kinda, now that i actually added some dependencies it quickly got out of hand with all the stuff i need to add to the firewall
what i need is to use a version of cpython i build from source i think
w
Why?
Feels like it's coming off the rails
b
because then it uses my openssl-devel and it would fix my ssl issues lol
ive tried setting ca_certs_path in my pants.toml but no difference
Does scie-pants use that?
w
I think it is supposed to have a mechanism to do this, but I believe there were some issues around it. I've not tested it out These problems are not really in my experience, since I never have issues like this. I did make a note that we should have a CI workflow that is dedicated to custom firewall/cert - which might be interesting to do. Out of curiosity, would this work if you directly used one of the downloaded Python Build Standalones? Or are they also using system certs
you're saying something like this doesn't work?
Copy code
SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt ~/.local/bin/pants
b
How would i use 
one of the downloaded Python Build Standalones
?
w
Oh, I just mean, if you went to https://github.com/astral-sh/python-build-standalone and tried using one of those instead of your system python. Just trying to figure out the scope of the problem here (in lieu of compiling from scratch, I meant)
👍 1
b
No, 
SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt ~/.local/bin/pants
doesn't work because it's probably not loaded into the truststore
w
The more correct thing to do is to just setup a container where I can emulate this, and test out options, document it, etc. which I will do I'm just trying to get you passed step 1 here 😆
b
yeah that would probably be better
or maybe i should be using some sort of proxy
instead of modding every tool i use
😄
w
I suffer from the same problem. I re-write tooling that annoys me in the slightest 🤦‍♂️
✅ 1
b
@wide-midnight-78598 it's bit insane scie-pants uses urllib here when it already had ptex (curl) and uses that for other downloads.
@broad-optician-21465 SJ's last question is a good one to answer directly before moving on.
b
Creating a container to reproduce the issue?
b
No
b
SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt ~/.local/bin/pants
?
b
Yes, that is the last!
b
I've done that sorry
Copy code
[user@host]$ SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt  ~/.local/bin/pants --level=debug
Failed to determine release URL for Pants: 2.30.2: pants.2.30.2-cp311-linux_x86_64.pex: URL check failed: <https://github.com/pantsbuild/pants/releases/download/release_2.30.2/pants.2.30.2-cp311-linux_x86_64.pex>: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Missing Authority Key Identifier (_ssl.c:1081)>

If this is unexpected (you are using a known good Pants version), try upgrading scie-pants first.
It may also be that the platform linux_x86_64 isn't supported for this version of Pants, or some other intermittent network/service issue.
To get help, please visit: <https://www.pantsbuild.org/community/getting-help>


Error: Failed to establish atomic directory /home/user/.cache/nce/4f07959fafe3296d25d3e3d2c92165cc6933455f0caa80a335a8aaa14c295fb3/locks/configure-d3637b9a6f6225652051dac8b6387673b1586bd80741a8a52b082e8241f0a621. Population of work directory failed: Boot binding command failed: exit status: 1

Isolates your Pants from the elements.

Please select from the following boot commands:

<default> (when SCIE_BOOT is not set in the environment)  Detects the current Pants installation and launches it.
bootstrap-tools                                           Introspection tools for the Pants bootstrap process.
update                                                    Update scie-pants.

You can select a boot command by setting the SCIE_BOOT environment variable.
b
And is the SSL_CERT_FILE path correct? Does that file exist?
b
Yep!
Copy code
[user@host]$ ll /etc/ssl/certs/ca-bundle.crt
lrwxrwxrwx. 1 root root 49 Oct 29 01:00 /etc/ssl/certs/ca-bundle.crt -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
[user@host]$ ll /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
-r--r--r--. 1 root root 240K Mar  4 13:13 /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
It should be a *.pem no?
b
Yup
✅ 1
b
We've had this issue before with python, so we have created a library to inject some stuff into the truststore
But i have no control over the python version in pants right? or can i make it depend on my library?
b
No, not at this boot phase. This is not pants, its scie pants installer code. It shouldn't even be using Python for this afaict. It has an embedded curl that respects SSL_CERT_FILE.
b
aha
👺
b
So, newer PBS releases should work, not sure which your scie-pants embeds.
You can check with 
SCIE=inspect pants
b
Copy code
[user@host]$ SCIE=inspect pants | jq .ptex
{
  "cpython-3.10.20+20260414-x86_64-unknown-linux-gnu-install_only.tar.gz": "<https://github.com/astral-sh/python-build-standalone/releases/download/20260414/cpython-3.10.20%2B20260414-x86_64-unknown-linux-gnu-install_only.tar.gz>",
  "cpython-3.11.15+20260414-x86_64-unknown-linux-gnu-install_only.tar.gz": "<https://github.com/astral-sh/python-build-standalone/releases/download/20260414/cpython-3.11.15%2B20260414-x86_64-unknown-linux-gnu-install_only.tar.gz>",
  "cpython-3.12.13+20260414-x86_64-unknown-linux-gnu-install_only.tar.gz": "<https://github.com/astral-sh/python-build-standalone/releases/download/20260414/cpython-3.12.13%2B20260414-x86_64-unknown-linux-gnu-install_only.tar.gz>",
  "cpython-3.13.13+20260414-x86_64-unknown-linux-gnu-install_only.tar.gz": "<https://github.com/astral-sh/python-build-standalone/releases/download/20260414/cpython-3.13.13%2B20260414-x86_64-unknown-linux-gnu-install_only.tar.gz>",
  "cpython-3.14.4+20260414-x86_64-unknown-linux-gnu-install_only.tar.gz": "<https://github.com/astral-sh/python-build-standalone/releases/download/20260414/cpython-3.14.4%2B20260414-x86_64-unknown-linux-gnu-install_only.tar.gz>",
  "cpython-3.8.20+20241002-x86_64-unknown-linux-gnu-install_only.tar.gz": "<https://github.com/astral-sh/python-build-standalone/releases/download/20241002/cpython-3.8.20%2B20241002-x86_64-unknown-linux-gnu-install_only.tar.gz>",
  "cpython-3.9.25+20251031-x86_64-unknown-linux-gnu-install_only.tar.gz": "<https://github.com/astral-sh/python-build-standalone/releases/download/20251031/cpython-3.9.25%2B20251031-x86_64-unknown-linux-gnu-install_only.tar.gz>"
}
idk why it points at more than one
b
And which is the default? You need to read more json
The install / download widget runs under default no matter final pick.
b
it seems to use /python/bin/python3.14 which would mean cpython 3.14.4 ig
b
But, underscoring, ptex already successfully downloaded the Python :/ so scie pants is foisting itself on its own petard here.
Hrm, ok re default. If so, it is new enough to have the fix I linked.
b
I mean, if i don't use my firewall.json, it's complaining about pants not cpython
Failed to determine release URL for Pants: 2.30.2: pants.2.30.2-cp311-linux_x86_64.pex: URL check failed: <https://github.com/pantsbuild/pants/releases/download/release_2.30.2/pants.2.30.2-cp311-linux_x86_64.pex>: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Missing Authority Key Identifier (_ssl.c:1081)>
And then later it complains about java dependencies
(if i use my firewall.json)
i just downloaded the file and put it on an tls server which re-encrypts it without the self-signed certificates
but i don't want to do this for every dependency 😧
b
This should be the only "file" for scie pants, I can't speak to pants after that. To be clear, you're fighting scie pants currently (the pants installer), this is not actually pants yet. Pants uses reqwests (rust library) and that's when pants.toml plumbing comes in.
b
aha okok
b
But this installer phase SSL error is unfortunate and totally un-needed since embedded ptex (curl) could be used. But that's a code problem
b
if i check the system calls from pants, i can see its reading from the specified cert file: If i run: 
SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt strace -f -e openat pants
It does open the cert: 
[pid 2848714] openat(AT_FDCWD, "/etc/ssl/certs/ca-bundle.crt", O_RDONLY) = 5
(it doesn't do that without the env variable)
b
I think Python may still have janky SSL code. Last I dove deep many years ago, it has a static Singleton type issue with initialization order where even with env var set, if you tickle wrong order, you get SSL with defaults and not env.
b
yeah i mean im not a python developer but everytime i use it i get a new problem
b
b
i love comments with footnotes in them lol
🫠
b
When you sometimes literally spend months debugging 1 thing, that's what you get. I do that often enough:/ This was only days though.
🥰 1
b
imma go outside in the sun instead
thanks for the help guys
i think this is an issue specific to my usecase probably
so i need to fix some stuff in my end
b
> i think this is an issue specific to my usecase probably That may be but I want to triple underline that scie-pants already had to download a PBS python (over SSL) in order to run the failing code (which FFS is just doing a HEAD check ahead of later using ptex to do the download). It did this PBS download using ptex successfully. Ptex was built exactly for this sort of no-fuss, always works on any Linux / Mac / Windows so that bootstrap would not hit weird SSL errors. Proofs: 1. All PBS are lazy: https://github.com/pantsbuild/scie-pants/blob/14c9998fc188a9965a6d13923dad40983f061954/package/scie-pants.toml#L17-L64 2. Failing HEAD check code: https://github.com/pantsbuild/scie-pants/blob/14c9998fc188a9965a6d13923dad40983f061954/tools/src/scie_pants/pants_version.py#L237-L308 3. Later code that would use ptex to download the Pants PEX if HEAD check didn't fail: https://github.com/pantsbuild/scie-pants/blob/14c9998fc188a9965a6d13923dad40983f061954/tools/src/scie_pants/install_pants.py#L86-L93
👀 1
@wide-midnight-78598 I'm not sure if you're aware of this tragi-comedy of the commons, but its bad.
w
Once you flagged the urllib part, I created an issue in the repo to review (and fix it). That was introduced a couple of years ago when python 3.11 was being used for Pants
💫 1
It's wild...
7 Views
Open in Slack
PreviousNext
Powered by