So I'm running into a problem which is slightly different than most proxy issues. Our environment must be run through our artifactory server. As such when I try to bootstrap pants it tries to download the required PEX installation from the internet. I read quite a few of the slack posts, but have not come up with a way to get around this. Based on the evaluation of the pants shell script it appears that the PEX download and version are hard coded. Any ideas on how to resolve this?

What version of Pants are you using?

using the latest version and yes, the pants binary launcher

So to confirm, you're using the latest latest (as in "release_2.18.0.dev7") or latest stable (as in 2.16.0)?

lemme check. one sec

(Sorry, our distribution model has been changed, so that determines whats going on)

2.16.0

OK, so not the new download-a-PEX-from-GitHub process 👍

the installer does seem to download from github unfortunately

Can you provide more info? Maybe the error output?

right now there isn't an error as I'm using my local machine. But when I run pants for the first time I get

SHA256 fingerprint of <https://github.com/pantsbuild/pex/releases/download/v2.1.103/pex> verified.

But on our build system it won't have access to the github download location

OK, so several pieces of "Pants" reach out to GitHub for assets. Most (but not all, we can circle back later) are configurable. From the top: •

scie-pants

(the name of the

pants

launcher) asks for PythonBuildStandalone for Python 3.9 from the relevant GitHub Release. ◦ I'm surprised this isn't the error you see • Then it finds

pants

from PyPI (in the future this will come from a GitHub release as well, and currently isn't configurable, I'll DM you about this) • Then Pants needs Pex to run. It does so by using the values form this subsystem So to answer your original question. Poke at the docs in that last bullet point

great! will do, and thanks for the fast response

👍 Pass it on 🙂

So looking at the last part, we can certainly do what is needed to honor the config values for the pex download. Are there similar flags/configuration for the download of the PythobnBuildStandalone? I thought I saw docs that you can use your own installation of python, but I'm not sure if this allows you to use the it instead of the 3.9 PythonBuildStandalone.

I think the part about firewalls here is your answer: https://github.com/pantsbuild/scie-pants

We should probably open a tracking issue to ensure the new distribution model works behind firewalls

@bitter-ability-32190 You mentioned

Then it finds pants from PyPI (in the future this will come from a GitHub release as well, and currently isn't configurable, I'll DM you about this)

Is this download from pypi a curl/request download, or is it using pip/poetry?

pip

: https://github.com/pantsbuild/scie-pants/blob/741555cae988265ecd60283cf0dfe21fd6f10869/tools/src/scie_pants/install_pants.py#L26

Also running into an issue with the ptex downloads. I've set

PANTS_BOOTSTRAP_URLS=/build/ptex-mappings.json

with the ptext-mappings.json containing:

{
  "ptex": {
    "cpython-3.8.16+20230507-x86_64-unknown-linux-gnu-install_only.tar.gz": "file:///home/build/ptex/cpython-3.8.16+20230507-x86_64-unknown-linux-gnu-install_only.tar.gz",
    "cpython-3.9.16+20230507-x86_64-unknown-linux-gnu-install_only.tar.gz": "file:///home/build/ptex/cpython-3.9.16+20230507-x86_64-unknown-linux-gnu-install_only.tar.gz"
  }
}

But when I run

SCIE=inpect pants | jq .ptex

I still get the original URLs

{
  "cpython-3.8.16+20230507-x86_64-unknown-linux-gnu-install_only.tar.gz": "<https://github.com/indygreg/python-build-standalone/releases/download/20230507/cpython-3.8.16%2B20230507-x86_64-unknown-linux-gnu-install_only.tar.gz>",
  "cpython-3.9.16+20230507-x86_64-unknown-linux-gnu-install_only.tar.gz": "<https://github.com/indygreg/python-build-standalone/releases/download/20230507/cpython-3.9.16%2B20230507-x86_64-unknown-linux-gnu-install_only.tar.gz>"
}

Did you

export

that env var? Try

PANTS_BOOTSTRAP_URLS=/build/ptex-mappings.json SCIE=inspect pants | jq .ptex

?

yup. Will try the command you posted right now

Ah, the readme is stale: https://github.com/pantsbuild/scie-pants/blob/741555cae988265ecd60283cf0dfe21fd6f10869/tools/lift.json#L62C1-L62C227

Am I supposed to be on 2.17.x?

The binary launcher (

scie-pants

) comes before Pants-proper, so what version of Pants you're using shouldn't come into play (until we want to associate a newer Python version for a release of Pants, but I digress...)

still didn't work. when I unset the env vars and let it run naked I get the following:

SCIE=inspect pants | jq .ptex
{
  "cpython-3.8.16+20230507-x86_64-unknown-linux-gnu-install_only.tar.gz": "<https://github.com/indygreg/python-build-standalone/releases/download/20230507/cpython-3.8.16%2B20230507-x86_64-unknown-linux-gnu-install_only.tar.gz>",
  "cpython-3.9.16+20230507-x86_64-unknown-linux-gnu-install_only.tar.gz": "<https://github.com/indygreg/python-build-standalone/releases/download/20230507/cpython-3.9.16%2B20230507-x86_64-unknown-linux-gnu-install_only.tar.gz>"
}

Ugh, yeah I see the same thing locally. I'll light the bat signal. @enough-analyst-54434?

looking the binary using

strings

I see:

PANTS_BOOTSTRAP_VERSION0.9.1

Perhaps we take this to an issue in https://github.com/pantsbuild/scie-pants for posterity

you want me to report this?

Haha yes please 🙏

gotcha

@blue-city-97042

SCIE=inspect pants

shows you the lift manifest of the scie, it makes no attempt to merge things. If you actually run

PANTS_BOOTSTRAP_URLS=... pants

that works fine. I'll add my steps to prove this to the issue you filed.

interesting. Operationally it'll work for me, but from an informational side of things, it is confusing.

Well, the

SCIE=inspect

just doesn't do what you thought it did. I'm not sure what to do about that. Your issue though implied functionally things didn't work. Not just informationally.

🙂

One thing Pants could be doing is shipping fat binaries to make this a bit easier. As an example, a-scie/lift (science) ships both thin (lazily fetch Python Build Standalone Python distributions) and fat (embed the PBS Python distribution in the scie binary) scies: + https://github.com/a-scie/lift/releases/tag/v0.1.3 + https://github.com/a-scie/lift/blob/main/lift.toml (lazy default) + https://github.com/a-scie/lift/blob/b3cb0ea3e1a515097047e0d70e823bbdac5a720a/noxfile.py#L328-L356 (build thin and fat scies) That would eliminate the need to download the PBS releases when Pants upgrades (it should be currently to pick up the July fix for libcrypt on Linux) as well as the need to concoct a ptex json, use the env var, etc.