https://pantsbuild.org/ logo
Join Slack
Powered by
What should be the value of `PEX_ROOT` at the time...
# general
b
What should be the value of 
PEX_ROOT
at the time of bootstrapping?
w
it shouldn’t matter. Pants filters ~all environment variables as it starts up
the PEX_ROOT is set for individual processes automatically as they run
b
I asked because I am getting a warning 
Scrubbing PEX_ROOT
in my CI. This is not the case when I bootstrap on my local machine.
w
yea, that’s correct. you should/do not need to set PEX_ROOT
the warning is to point out that it will not be used.
Pants will locate the pex root under the named_caches_dir
b
Ok, but it doesn’t seem to be trying to locate under named_caches_dir because I have set it in my 
pants.toml
. Also, would you be able to explain the reason why I don’t get this warning on my local machine but in CI?
I get this warning regardless of whether or not I set 
PEX_ROOT
in my bootstrapping CI. Furthermore, I have set 
named_caches_dir
in 
pants.toml
but it doesn’t seem to be using that location as 
PEX_ROOT
.
e
@ancient-rose-27306 the best help here would be remote eyes. Is there any way you can share the portion of the CI logs frm start until this warning point with redactions / path edits as needed for any sensitive info?
b
Here are the redacted logs:
Copy code
Bootstrapping Pants using /venv/bin/python3
Creating the virtualenv PEX.
Downloading the Pex PEX.
SHA256 fingerprint of <https://github.com/pantsbuild/pex/releases/download/v2.1.62/pex> verified.
Scrubbing PEX_ROOT
$BUILD_ROOT/$BUILD_DIR/tmp/tmpakewj1fy/unzipped_pexes/70e3732fb536dad57e716bdab53bcec29ebaa5e4/.bootstrap/pex/pex_info.py:384: PEXWarning: PEX_ROOT is configured as $HOME/.pex but that path is un-writeable, falling back to a temporary PEX_ROOT of $BUILD_DIR/tmp/tmprg6j0sk7 which will hurt performance.
  "performance.".format(pex_root=pex_root, tmp_root=tmp_root)
$BUILD_ROOT/$BUILD_DIR/tmp/tmpakewj1fy/installed_wheels/affd08bdf38c0eac7b9fd0ac2f464442e4147274/pex-2.1.62-py2.py3-none-any.whl/pex/variables.py:577: PEXWarning: PEX_ROOT is configured as $HOME/.pex but that path is un-writeable, falling back to a temporary PEX_ROOT of $BUILD_ROOT/$BUILD_DIR/tmp/tmpg37ji46o which will hurt performance.
  "performance.".format(pex_root=pex_root, tmp_root=tmp_root)
$BUILD_ROOT/$BUILD_DIR/tmp/tmpakewj1fy/installed_wheels/affd08bdf38c0eac7b9fd0ac2f464442e4147274/pex-2.1.62-py2.py3-none-any.whl/pex/pex_info.py:384: PEXWarning: PEX_ROOT is configured as $HOME/.pex but that path is un-writeable, falling back to a temporary PEX_ROOT of $BUILD_ROOT/$BUILD_DIR/tmp/tmp15i5mqlb which will hurt performance.
  "performance.".format(pex_root=pex_root, tmp_root=tmp_root)
Installing pantsbuild.pants==2.12.0 into a virtual environment at $WORKDIR/bootstrap-Linux-x86_64/2.12.0_py37
Scrubbing PEX_ROOT
$BUILD_ROOT/$BUILD_DIR/tmp/tmphphtbuvw/unzipped_pexes/01a7d467141015c7b3a0aee086ae57d13816e770/.bootstrap/pex/pex_info.py:384: PEXWarning: PEX_ROOT is configured as $HOME/.pex but that path is un-writeable, falling back to a temporary PEX_ROOT of $BUILD_DIR/tmp/tmpyrjkfwyb which will hurt performance.
  "performance.".format(pex_root=pex_root, tmp_root=tmp_root)
New virtual environment successfully created at $WORKDIR/bootstrap-Linux-x86_64/2.12.0_py37.
17:44:18.81 [INFO] Initializing scheduler...
17:44:19.07 [INFO] Scheduler initialized.
2.12.0
And after this bootstrap that seems to have completed successfully, I am getting following error when running the command 
./pants tailor --check update-build-files --check
in CI after bootstrap:
Copy code
17:44:21.83 [INFO] Initializing scheduler...
17:44:22.10 [INFO] Scheduler initialized.
17:44:22.15 [ERROR] 1 Exception encountered:

  Exception: Failed to read link "$WORKDIR/bootstrap-Linux-x86_64/2.12.0_py37": Absolute symlink: "$WORKDIR/bootstrap-Linux-x86_64/pants.faI76R/install"

##[error]Bash exited with code '1'.
e
Ok, as to the 1st warning, that's on the tin: `$HOME/.pex`is unwriteable. So to get that warning to go away you'll need to run something like `PEX_ROOT=/writeable/location ./pants --version`as the 1st step. That said, the warning is ignorable, so it's up to you.
As to the second issue, what is $WORKDIR? Is that the root of the code repo you're running Pants in?
If so, it has to be any other directory not in the repo tree.
b
Thanks for the help with 
PEX_ROOT
. And yes, the 
$WORKDIR
is the root of the code repo.
e
Yeah - that is a no go. Pants needs to have its caches all not be in any repo that uses Pants (this is a white lie, but you really don't want to fight through this). The idea is you setup Pants once globally and then various repos use it. The 
pants
script checked into each repo is just a thin shim.
So, in CI, you don't really get the benefit of a global Pants install - you re-install it every time, but the do not house inside the repo itself still applies.
b
Ok, thanks! That means I need to set 
PANTS_SETUP_CACHE
to another directory which is not the root. And thanks for clarifying the CI install thing as that would have been my follow up question 🙂
e
The easiest way to set all this is just export 
XDG_CACHE_HOME
Every other global Pants cache default based off that.
b
PEX_ROOT
also shouldn’t be the repo root but can be same as 
PANTS_SETUP_CACHE
?
e
It should be 
$XDG_CACHE_HOME/.pex
probably for sanity sake.
b
Ok, will look into 
XDG_CACHE_HOME
.
Just in case if I am unable to export 
XDG_CACHE_HOME
, its ok if I use the same location for both 
PEX_ROOT
and 
PANTS_SETUP_CACHE
?
e
For sanity sake you'd want something like 
PEX_ROOT=$PANTS_SETUP_CACHE/pex_root
. That said, if your CI step can run 
./pants
its almost inconceivable it can't run 
PEX_ROOT=foo XDG_CACHE_HOME=bar ANY_ENV_VAR_I_WANT=baz ./pants ...
b
Ok, so if I run 
XDG_CACHE_HOME=some/dir/ ./pants --version
, would it automatically take 
some/dir/
as 
PANTS_SETUP_CACHE
and 
some/dir/.pex
as 
PEX_ROOT
?
e
No. 
PEX_ROOT
does not resect the XDG / freedesktop standard; so you have to set it separately. That said, that only will squelch a warning that is immaterial in the 1-time bootstrap process. Pants itself sets a 
PEX_ROOT
internally elsewhere under its own cache structure that is all covered by you exporting 
XDG_CACHE_HOME
So, only 
XDG_CACHE_HOME
is required. `PEX_ROOT`is cosmetic and just used to build a venv 1 time to house the Pants binary itself.
This of course needs to be cleaned up! Very confusing.
👍 1
b
So, I tried setting up 
XDG_CACHE_HOME
but it did not prevent the 
Scrubbing PEX_ROOT
warning:
Copy code
echo $XDG_CACHE_HOME && echo $PANTS_SETUP_CACHE && echo $PEX_ROOT && ./pants --version
========================== Starting Command Output ===========================
$WORKDIR/.. # One directory up the repo root, Also $XDG_CACHE_HOME
$WORKDIR/.. 
$WORKDIR/../.pex # One directory up the repo root/.pex
Bootstrapping Pants using /venv/bin/python3
Creating the virtualenv PEX.
Downloading the Pex PEX.
SHA256 fingerprint of <https://github.com/pantsbuild/pex/releases/download/v2.1.62/pex> verified.
Scrubbing PEX_ROOT
/$BUILD_ROOT/$BUILD_DIR/tmp/tmpwfmzxkpt/unzipped_pexes/70e3732fb536dad57e716bdab53bcec29ebaa5e4/.bootstrap/pex/pex_info.py:384: PEXWarning: PEX_ROOT is configured as $HOME/.pex but that path is un-writeable, falling back to a temporary PEX_ROOT of /$BUILD_DIR/tmp/tmpiuaz7z_5 which will hurt performance.
  "performance.".format(pex_root=pex_root, tmp_root=tmp_root)
/$BUILD_ROOT/$BUILD_DIR/tmp/tmpwfmzxkpt/installed_wheels/affd08bdf38c0eac7b9fd0ac2f464442e4147274/pex-2.1.62-py2.py3-none-any.whl/pex/variables.py:577: PEXWarning: PEX_ROOT is configured as $HOME/.pex but that path is un-writeable, falling back to a temporary PEX_ROOT of /$BUILD_ROOT/$BUILD_DIR/tmp/tmpxcr8mdzi which will hurt performance.
  "performance.".format(pex_root=pex_root, tmp_root=tmp_root)
/$BUILD_ROOT/$BUILD_DIR/tmp/tmpwfmzxkpt/installed_wheels/affd08bdf38c0eac7b9fd0ac2f464442e4147274/pex-2.1.62-py2.py3-none-any.whl/pex/pex_info.py:384: PEXWarning: PEX_ROOT is configured as $HOME/.pex but that path is un-writeable, falling back to a temporary PEX_ROOT of /$BUILD_ROOT/$BUILD_DIR/tmp/tmpr1dktv19 which will hurt performance.
  "performance.".format(pex_root=pex_root, tmp_root=tmp_root)
Installing pantsbuild.pants==2.12.0 into a virtual environment at $WORKDIR/bootstrap-Linux-x86_64/2.12.0_py37
Scrubbing PEX_ROOT
/$BUILD_ROOT/$BUILD_DIR/tmp/tmpevybnbwf/unzipped_pexes/01a7d467141015c7b3a0aee086ae57d13816e770/.bootstrap/pex/pex_info.py:384: PEXWarning: PEX_ROOT is configured as /u1/svcazagt/.pex but that path is un-writeable, falling back to a temporary PEX_ROOT of /$BUILD_DIR/tmp/tmpv0nkm9mb which will hurt performance.
  "performance.".format(pex_root=pex_root, tmp_root=tmp_root)
New virtual environment successfully created at $WORKDIR/bootstrap-Linux-x86_64/2.12.0_py37.
22:13:32.67 [INFO] Initializing scheduler...
22:13:32.98 [INFO] Scheduler initialized.
2.12.0
e
Oh, looked at the output more. You say you tried setting up 
XDG_CACHE_HOME
but you actually did set both 
XDG_CACHE_HOME
and 
PEX_ROOT
as evidenced by the echos. Let me check Pex changelogs. The Pex PEX used to bootstrap Pants is fairly old (~2.1.60 I think) and Pex is now ~40 releases down the road and this seems to ring a bell...
Oh, right - see this line above the warning?: 
Scrubbing PEX_ROOT
- that's done here: https://github.com/pantsbuild/setup/blob/eb887ad5fd80766214e8c38102a5bc92d1342beb/pants#L282-L295
So, I think the path forward to get rid of the warning is probably to have Pex support XDG. That would mean you only need to set 
XDG_CACHE_HOME
and that env var would not get scrubbed cleaning up both messes and making the Pants / Pex system well behaved as a free desktop app-system. Shorter term, the 
pants
script could be special-cased to do something like:
Copy code
$ cat test.sh 
function scrub_env_vars {
  # Ensure the virtualenv PEX runs as shrink-wrapped.
  # See: <https://github.com/pantsbuild/setup/issues/105>
  local -r pex_env_vars=(${!PEX_@})
  local -r scrub_pex_env_vars="${pex_env_vars[@]/PEX_ROOT}"
  if [[ -n "${scrub_pex_env_vars[@]}" ]]; then
    echo >&2 "Scrubbing ${scrub_pex_env_vars[@]}"
    unset ${scrub_pex_env_vars[@]}
  fi
}

(
  echo >&2 "Raw env:"
  env | grep PEX_

  scrub_env_vars
  echo >&2 "Scrubbed env:"
  env | grep PEX_
)
$ PEX_FOO=foo PEX_ROOT=root PEX_BAR=bar bash test.sh 
Raw env:
PEX_ROOT=root
PEX_FOO=foo
PEX_BAR=bar
Scrubbing PEX_BAR PEX_FOO 
Scrubbed env:
PEX_ROOT=root
@bored-glass-58755 if that warning bothers you enough and you'd like to contribute something like that fix, I'd be happy to review.
b
Hi @enough-analyst-54434, thanks a lot for your responses. Yes, I had seen pants script before reaching out and thought maybe there’s a way to avoid that warning. Lastly, yes I’d be happy to contribute to remove that warning. I will make the change and test it in my CI first.
e
Great - thank you.
b
Ok, so the 
Scrubbing PEX_ROOT
warning has disappeared after making changes to the script but it is still giving below warnings as before. Moreover, the Pants fails to bootstrap:
Copy code
$XDG_CACHE_HOME/pants/named_caches/pex_root/installed_wheels/affd08bdf38c0eac7b9fd0ac2f464442e4147274/pex-2.1.62-py2.py3-none-any.whl/pex/pex_info.py:384: PEXWarning: PEX_ROOT is configured as $HOME/.pex but that path is un-writeable, falling back to a temporary PEX_ROOT of /$BUILD_ROOT/$BUILD_DIR/tmp/tmp9f796zn1 which will hurt performance.
  "performance.".format(pex_root=pex_root, tmp_root=tmp_root)
$XDG_CACHE_HOME/pants/named_caches/pex_root/installed_wheels/affd08bdf38c0eac7b9fd0ac2f464442e4147274/pex-2.1.62-py2.py3-none-any.whl/pex/pex_bootstrapper.py:493: PEXWarning: The venv for $XDG_CACHE_HOME/pants/named_caches/pex_root/pip.pex/c15b6e999f7b2b553c5ee23e542ae1e620d5dcf2 at $XDG_CACHE_HOME/pants/named_caches/pex_root/venvs/e61f97e90815934b3b69dc4a58ecf7e05555084b/15d3ad9788dc0da4a814dd5bb1a36eed666cd279 has script shebangs of '#!$XDG_CACHE_HOME/pants/named_caches/pex_root/venvs/s/01aec573/venv/bin/python3.7 -sE' with 139 characters. On some systems this may be too long and cause problems running the venv scripts. You may be able adjust PEX_ROOT from $XDG_CACHE_HOME/pants/named_caches/pex_root to a shorter path as a work-around.
  pex_root=pex_info.pex_root,
Failed to spawn a job for DistributionTarget(interpreter=PythonInterpreter('/usr/local/python/3.7.6/bin/python3.7', PythonIdentity('/usr/local/python/3.7.6/bin/python3.7', 'cp37', 'cp37m', 'manylinux_2_17_x86_64', (3, 7, 6)))): [Errno 2] No such file or directory: '$XDG_CACHE_HOME/pants/named_caches/pex_root/venvs/e61f97e90815934b3b69dc4a58ecf7e05555084b/15d3ad9788dc0da4a814dd5bb1a36eed666cd279/pex': '$XDG_CACHE_HOME/pants/named_caches/pex_root/venvs/e61f97e90815934b3b69dc4a58ecf7e05555084b/15d3ad9788dc0da4a814dd5bb1a36eed666cd279/pex'
mv: cannot stat '$XDG_CACHE_HOME/pants/setup/bootstrap-Linux-x86_64/pants.R0iluW/virtualenv.pex': No such file or directory
Installing pantsbuild.pants==2.12.0 into a virtual environment at $XDG_CACHE_HOME/pants/setup/bootstrap-Linux-x86_64/2.12.0_py37
/usr/local/bfm/ext/ado/admin/venv/bin/python3: can't open file '$XDG_CACHE_HOME/pants/setup/bootstrap-Linux-x86_64/virtualenv-20.4.7/virtualenv.pex': [Errno 2] No such file or directory
##[error]Bash exited with code '1'.
Finishing: Pants bootstrap step
I fail to understand why in the first line of logs above it says - 
PEXWarning: PEX_ROOT is configured as $HOME/.pex
. I am looking at entire script now to try and remove these errors.
e
Yeah - 1st stop is to print out the env vars post scrub but pre running pex to confirm Pex is seeing the PEX_ROOT you expect.
b
Yes, will get back to you with my findings.
It seems following line in function 
bootstrap_virtualenv
is resulting in error and therefore, the subsequent move command is failing and therefore, the bootstrap is failing: 
"${python}" "${pex_path}" -r requirements.txt -c virtualenv -o virtualenv.pex
I am now looking at why preventing the 
PEX_ROOT
from being scrubbed is resulting in above command’s failure. During the execution of above command, we get following warning even though 
PEX_ROOT
is not set to what the warning suggests: 
$XDG_CACHE_HOME/pants/named_caches/pex_root/installed_wheels/affd08bdf38c0eac7b9fd0ac2f464442e4147274/pex-2.1.62-py2.py3-none-any.whl/pex/pex_info.py:384: PEXWarning: PEX_ROOT is configured as $HOME/.pex but that path is un-writeable, falling back to a temporary PEX_ROOT of /local/build/tmp/tmpktux9t7v which will hurt performance.
“performance.“.format(pex_root=pex_root, tmp_root=tmp_root)
e
So you have added an echo post scrub but pre-run and that echo proves PEX_ROOT=/what/you/want?
b
It seems to me that 
pex-2.1.62
is set to use 
PEX_ROOT
as 
$HOME/.pex
e
That is the default for all Pex for all time, yes.
b
So you have added an echo post scrub but pre-run and that echo proves PEX_ROOT=/what/you/want?
Yes
e
Ok. Have you tried bumping the Pex version then to latest + edit the sha?
b
But then there must be a way to override the default.
e
There is! PEX_ROOT, also has existed for all time.
b
Ok. Have you tried bumping the Pex version then to latest + edit the sha?
Let me try this now.
e
So this is a more subtle issue somewhere.
The version is 2.1.103 and the hash is 4d45336511484100ae4e2bab24542a8b86b12c8cb89230463593c60d08c4b8d3
👍 1
b
We’ll be in touch.
Ok. Have you tried bumping the Pex version then to latest + edit the sha?
So, I bumped Pex version to 
2.1.103
and the PEXWarning is no more (the one which said that 
PEX_ROOT
is set as 
$HOME/.pex
which was un-writable). The only warning that I get now is regarding the long 
PEX_ROOT
path (which I don’t think can be avoided unless I choose to use a shorter path for 
XDG_CACHE_HOME
):
Copy code
Bootstrapping Pants using /venv/bin/python3
Creating the virtualenv PEX.
Downloading the Pex PEX.
SHA256 fingerprint of <https://github.com/pantsbuild/pex/releases/download/v2.1.103/pex> verified.
/$XDG_CACHE_HOME/pants/named_caches/pex_root/installed_wheels/6a3c9545f9bc666f039874243789bf4ba5cc4fbde5a945eca40a651de360568b/pex-2.1.103-py2.py3-none-any.whl/pex/pex_bootstrapper.py:557: PEXWarning: The venv for /XDG_CACHE_HOME/pants/named_caches/pex_root/pip.pex/d1893e3bec4d47905ad270be40669a781dc00206 at /XDG_CACHE_HOME/pants/named_caches/pex_root/venvs/15158fe84406f2733a98f687f98d59238ad4210d/15d3ad9788dc0da4a814dd5bb1a36eed666cd279 has script shebangs of '#!/XDG_CACHE_HOME/pants/named_caches/pex_root/venvs/s/883b36a3/venv/bin/python3.7 -sE' with 139 characters. On some systems this may be too long and cause problems running the venv scripts. You may be able adjust PEX_ROOT from /XDG_CACHE_HOME/pants/named_caches/pex_root to a shorter path as a work-around.
  pex_root=pex_info.pex_root,
Installing pantsbuild.pants==2.12.0 into a virtual environment at /$XDG_CACHE_HOME/pants/setup/bootstrap-Linux-x86_64/2.12.0_py37
New virtual environment successfully created at /$XDG_CACHE_HOME/pants/setup/bootstrap-Linux-x86_64/2.12.0_py37.
01:34:33.57 [INFO] Initializing scheduler...
01:34:33.85 [INFO] Scheduler initialized.
2.12.0
Finishing: Pants bootstrap step
I tried looking into sources of Pex versions 
2.1.62
, 
2.1.79
, 
2.1.90
, and 
2.1.103
but couldn’t find any difference between them that could point towards the reason for why the warning was generated. Another question that I have is if it is possible at the time of Pants bootstrapping that Pants downloads a version other than what is mentioned in 
pants
script? I ask this because I have two bootstrapped Pants in my local machine’s 
$HOME/.cache/pants
and neither of them has 
2.1.62
version of Pex which was mentioned in the script. I have 
2.1.79
and 
2.1.90
in both of those Pants setups. Could it be because of pex downloads that happen at the time of using Pants (as in running various goals like 
package
, 
publish
, etc.)?
Apart from above, I have below questions regarding 
pants
script: 1. In the function 
bootstrap_pex()
, why do we create 
${PANTS_BOOTSTRAP}
directory just after the message “Downloading the Pex PEX.”? I ask this because we anyways create 
${bootstrapped}
in the same function just after verifying the SHA256. 2. Why do we create 
${PANTS_BOOTSTRAP}
directory again in the function 
bootstrap_virtualenv()
just after after returning from the function 
bootstrap_pex()
which anyways created this directory?
Lastly, could you please guide me if there’s a developer/contributor setup that needs to be done or I can just create a branch and submit a PR with modified script? Shall I also bump the version in the same PR?
Hi, I followed this link to setup Pants for local development, however, the 
pants
script in the root folder isn’t the same as that gets downloaded with the command: `/bin/bash -c "$(curl -fsSL https://static.pantsbuild.org/setup/one_step_setup.sh)"`https://static.pantsbuild.org/setup/one_step_setup.sh)" . Rather it seems to be the one that bootstrap Pants (executing Rust compilation and all) at the time of contributor setup. Could you please advise me on how to proceed with fixing the script?
e
It was buried up here: https://pantsbuild.slack.com/archives/C046T6T9U/p1661519216209729?thread_ts=1661373158.171129&amp;cid=C046T6T9U But the 
pants
setup script used by all repos (except the Pants repo itself), is housed in https://github.com/pantsbuild/setup on the 
gh-pages
branch (a special branch that Github will serve up over plain https). The best way to get setup for contributing to that repo is - well this is the trick I use for any repo - read the CI setup. Whatever CI does you'll need to do. The nice thing is this strategy works for any sane (tested) open source project. So in that repo's case (https://github.com/pantsbuild/setup/blob/gh-pages/.github/workflows/ci.yml) you need to be able to run 
tox -e format-check,lint,typecheck
and 
tox -e test
(You may or may not need the extra `-- --basetemp ...`args CI needs). So, 1st step - install `tox`: https://pypi.org/project/tox/ and next step, ensure 
tox -e format-check,lint,typecheck,test
passes on a fresh clean 
gh-pages
branch.
b
Thanks @enough-analyst-54434, let me start with that.
Ok, so following commands ran successfully: 
tox -e format-run
tox -e format-check
tox -e lint
tox -e typecheck
However, 
tox -e test
finished with 6 passed and 12 failed.
Got following for 10 out of 12 failed test cases:
Copy code
E        subprocess.CalledProcessError: Command '['./pants', '--version']' returned non-zero exit status 1.

../miniconda3/envs/pants-setup-dev/lib/python3.9/subprocess.py:528: CalledProcessError
I am using a conda environment with python 3.9 on a linux machine. Is there anything that I am missing?
e
Ok, then you probably need to work through that. I'm sorry to be terse, but you gave me little to go on!
Ah, yes. So the tests test old Pants IIRC and some of the old Pants probably require old Pythons.
Just a sec.
b
Pants 2.12.0 is bootstrapped for both python 3.7 and 3.9
I used python 3.9 because that’s what ci.yml suggested.
e
Yes it is, but if you read the unit tests code you should see strings like 1.30.1
And that plays out like this: https://github.com/pantsbuild/setup/runs/6712806317?check_suite_focus=true#step:4:47 Expand the node and you see:
Copy code
Exposing /opt/hostedtoolcache/Python/2.7.18/x64/bin: Python 2.7.18
  Exposing /opt/hostedtoolcache/Python/3.10.4/x64/bin: Python 3.10.4
  Exposing /opt/hostedtoolcache/Python/3.6.15/x64/bin: Python 3.6.15
  Exposing /opt/hostedtoolcache/Python/3.7.13/x64/bin: Python 3.7.13
  Exposing /opt/hostedtoolcache/Python/3.8.12/x64/bin: Python 3.8.12
  Exposing /opt/hostedtoolcache/Python/3.9.13/x64/bin: Python 3.9.13
So, bunch of Pythons
You may be able to just ignore the errors and use that set of errors as your baseline assuming they are all because you don't have needed Pythons on the PATH for old Pants.
I personally use pyenv to install a bunch of Pythons on my machine, but I think you need not do that to fix the issue you're trying to fix + add a test for that. That test can probably just run against ~modernish Pants.
b
Ok, I thought to have the required python but digging thru https://github.com/pantsbuild/setup/blob/gh-pages/.github/workflows/ci.yml#L45 I figured it probably expects python 2.7. I’d just rather ignore the errors
e
SGTM - in the end green CI is what matters and some way for you to write + run a new test.
b
And that plays out like this: https://github.com/pantsbuild/setup/runs/6712806317?check_suite_focus=true#step:4:47
I just saw this and found out that it is exposing python 3.9.13 but I have 3.9.12. I can upgrade my minor version and try it once more before moving on to the real stuff.
e
The minor versions do not matter at all.
That's just the set of Pythons that happen to be available on Github runners. That CI step just exposes all the pre-installed Pythons that happen to be there.
b
Ok, gotcha! I have never used GitHub CI so please pardon my ignorance there.
e
No problem.
b
Hi @enough-analyst-54434, I am getting following error in my CI on the line where it tries to extract all but 
PEX_ROOT
from the array `pex_env_vars`: 
./pants: line 336: pex_env_vars[@]: unbound variable
This is happening when I am supplying no 
PEX_
env variables at all in the CI and so it seems it is unable to extract anything out of an empty array. This, however, doesn’t throw any error when I try to run that small logic standalone in bash where I do not supply any 
PEX_
variables. This also doesn’t fail the test case I wrote if I remove the 
PEX_
environment variables from the test case. Nonetheless, I think we should introduce this small check just to make it more robust for all platforms/environments: 
if [ ! ${#pex_env_vars[@]} -eq 0 ]; then
What do you think?
e
Should I not read too much into the fact line 336 does not line up at all with the PR?: https://github.com/pantsbuild/setup/pull/127/files#diff-2c50624bea38c570f59fb28e3fffd11330856e2cd512760acaff9eb82d3df8caR286
I do not repro in a dedicated test file:
Copy code
jsirois@Gill-Windows:~$ cat test.sh
#!/usr/bin/env bash

set -euo pipefail

function scrub_env_vars {
  # Ensure the virtualenv PEX runs as shrink-wrapped.
  # See: <https://github.com/pantsbuild/setup/issues/105>
  local -r pex_env_vars=(${!PEX_@})
  local -r pex_env_vars_to_scrub="${pex_env_vars[@]/PEX_ROOT}"
  if [[ -n "${pex_env_vars_to_scrub[@]}" ]]; then
    echo >&2 "Scrubbing ${pex_env_vars_to_scrub[@]}"
    unset ${pex_env_vars_to_scrub[@]}
  fi
  # Also ensure pip doesn't think packages on PYTHONPATH
  # are already installed.
  if [ -n "${PYTHONPATH:-}" ]; then
    echo >&2 "Scrubbing PYTHONPATH"
    unset PYTHONPATH
  fi
}

scrub_env_vars
jsirois@Gill-Windows:~$ ./test.sh
jsirois@Gill-Windows:~$ echo $?
0
jsirois@Gill-Windows:~$ PEX_FOO=bar PEX_ROOT=baz ./test.sh
Scrubbing PEX_FOO
jsirois@Gill-Windows:~$ echo $?
Maybe this is a bash behavior difference based on version?:
Copy code
jsirois@Gill-Windows:~$ bash --version
GNU bash, version 5.0.17(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <<http://gnu.org/licenses/gpl.html>>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
b
The error line was copied pasted from my CI output which would not match the script from PR because I have some extra 
echo
lines in my CI. Sorry for the confusion. It is giving error on line 286: 
local -r pex_env_vars_to_scrub="${pex_env_vars[@]/PEX_ROOT}"
e
And what bash version in CI then?
Since I do not repro.
Maybe 
local -ar
(read only array) instead of `local -r`(read-only). The `=(...)`form on the RHS is supposed to make the variable a Bash array implicitly, but maybe this needs to be explicit in older versions of Bash? Adding the 
-a
works fine / doesn't change anything in my version of Bash.
b
This is the bash version in pipeline:
Copy code
GNU bash, version 4.2.46(2)-release (x86_64-redhat-linux-gnu)
I was just thinking from the perspective of making the fix robust, and so in my opinion adding a check wouldn't harm?
e
I just confirmed it is a Bash version difference:
Copy code
jsirois@Gill-Windows:~$ /mnt/c/Users/John\ Sirois/Downloads/bash-4.2/bash test.sh
test.sh: line 9: pex_env_vars[@]: unbound variable
@bored-glass-58755 I agree, but I'd also like to understand things. Give me a sec longer to see if there is not a more kosher way to do this in old bash.
👍 1
Ok, this works with 4.2.0 and 5.0.17:
Copy code
local -r pex_env_vars=(
  ${!PEX_@}
  # The explicit empty entry below avoids pex_env_vars being considered an unbound variable by
  # older version of Bash when there are no PEX_ variables in the environment.
  ""
)
b
Sounds good, I will make the change and push it.
7 Views
Open in Slack
PreviousNext
Powered by