I'm using Pants 2.12 in a self-hosted github runner and running in to an issue. I set up python versions for 3.9 and 3.10 on the runner but I get the following error when it runs

./pants generate-lockfiles fmt ::

ProcessExecutionFailure: Process 'Find interpreter for constraints: CPython<4.0,>=3.9' failed with exit code 102.

It seems to think the interpreter is broken:

Skipped the following broken interpreters:
1.) /mnt/github_actions_runner/_work/_tool/Python/3.10.6/x64/bin/python3.10:
/mnt/github_actions_runner/_work/_tool/Python/3.10.6/x64/bin/python3.10: error while loading shared libraries: libpython3.10.so.1.0: cannot open shared object file: No such file or directory
2.) /mnt/github_actions_runner/_work/_tool/Python/3.9.13/x64/bin/python3.9:
/mnt/github_actions_runner/_work/_tool/Python/3.9.13/x64/bin/python3.9: error while loading shared libraries: libpython3.9.so.1.0: cannot open shared object file: No such file or directory

Not sure how to fix this, any help would be appreciated

Are you using a canned GH action to setup Pythons? Those paths make it seem so.

Yeah, I'm using Github's setup python action.

Here is what I do to expose the non-default Pythons: https://github.com/pantsbuild/actions/blob/main/expose-pythons/action.yml You can re-use if you wish: https://github.com/pantsbuild/pex/blob/54fb072e5258215c3a0d5ec4e2fb406c17093644/.github/workflows/ci.yml#L100-L103

I'll try this out and see if I can get it working and if so may look at adding the param to the action

Um, just a sec.

I'll take a look at that 🙂

A good thing to fully take on board. Pants tries to be fully hermetic; so most env vars are masked off, files are snapshotted, etc.

So I just need to add that to the list in my envvars.add in pants.toml, right?

Yes. You can either use PANTS_* env vars in GH workflow toml: https://www.pantsbuild.org/docs/options#environment-variables Or, better than modifying the main

pants.toml

, a CI-specific toml for this: https://www.pantsbuild.org/docs/using-pants-in-ci#configuring-pants-for-ci-pantscitoml-optionalhttps://www.pantsbuild.org/docs/using-pants-in-ci#configuring-pants-for-ci-pantscitoml-optional

Wouldn't using a pants.ci.toml file increase the risk that the CI file and the non-CI file would get out of sync somehow?

The CI toml is additive. Just add or remove config options, don't re-define them all.

The typical idiom is for the ci file to be used as well as the non-ci file, so it’s small and only contains the ci-specific tweaks

Okay, cool, I'll look in to that then

The issue here is LD_LIBRARY_PATH will get mixed into all process cache keys; that means, if you use remote caching, each developer with a different LD_LIBRARY_PATH carves out a different cache space - meaning your devs don't all share the same cache, meaning when Mary gets in at 6am and runs a build, lazy Fred who gets in at 10am pays a penalty and doesn't get a cache hit from Mary's work at 6am. Its not that we set out to reward lazy people, but.

Makes sense, I assume I can give the toml file whatever name I like (this is part of our AutoTransform stuff to keep the repo formated and stuff and I'd like the toml file named appropriately for that, since it's not CI in general).

Exactly.

So I did what was described here and I'm still getting an error:

/mnt/github_actions_runner/_work/_tool/Python/3.9.13/x64/bin/python3.9: error while loading shared libraries: libpython3.9.so.1.0: cannot open shared object file: No such file or directory

I even tried adding the exposing code (which seemed to work correctly for exposing it). I will say it's coming now at a different place:

Process 'Determine Python dependencies for tests/python/<stuff>/test_<something>.py' failed with exit code 127.

Since the runner is self hosted, do you have access to it to play around directly? You're using GH canned actions to download a pre-built Python that clearly works on GH runner images, but maybe not on your GH runner image?

Python works on the runner, I'm invoking pants through a python script

A Python works. Do you know for sure its one of the 2 you get errors for?

The original 2 I got errors for were 3.10 and 3.9, the script I'm running that runs pants uses 3.10

An it definitely uses the same 3.10?

/mnt/github_actions_runner/_work/_tool/Python/3.10.6/x64/bin/autotransform

That's in the stack trace for the AutoTransform script, so it definitely appears to be the same 3.10

Ok, back to your access. Do you have access to the box? If so I can give you some debug steps. If not it gets more painful.

I don't immediately have access, I could probably get it in some time

Ok. Short term you can throw in

-ldebug

or (export

PANTS_LEVEL=debug

or

[GLOBAL] level = "debug"

) and

--pex-verbosity=9

(or

PANTS_PEX_VERBOSITY=9

or

[pex] verbosity = 9

) and see if that turns up more useful info in the CI output log.

I mean, I'm still getting

/mnt/github_actions_runner/_work/_tool/Python/3.9.13/x64/bin/python3.9: error while loading shared libraries: libpython3.9.so.1.0: cannot open shared object file: No such file or directory

So it seems like it just can't find this thing?

You'd run

./pants --no-process-cleanup X

(or

./pants --keep-sandboxes=on_failure X

for bleeding edge Pants versions). That will cause lines like this to print out:

$ ./pants --no-process-cleanup test --force src/python/pants/util/strutil_test.py 
16:00:33.83 [INFO] Preserving local process execution dir /tmp/pants-sandbox-g0DNAk for Run Pytest for src/python/pants/util/strutil_test.py:tests
16:00:34.17 [INFO] Completed: Run Pytest - src/python/pants/util/strutil_test.py:tests succeeded.

✓ src/python/pants/util/strutil_test.py:tests succeeded in 0.29s.

You can then run like so to emulate what is happening ~exactly:

$ /tmp/pants-sandbox-g0DNAk/__run.sh 
============================================================================================================================== test session starts ==============================================================================================================================
collected 17 items                                                                                                                                                                                                                                                              

src/python/pants/util/strutil_test.py .................                                                                                                                                                                                                                   [100%]

----------------------------------------------------------------------------------------- generated xml file: /tmp/pants-sandbox-g0DNAk/src.python.pants.util.strutil_test.py.tests.xml -----------------------------------------------------------------------------------------
============================================================================================================================== 17 passed in 0.08s ===============================================================================================================================

And then you can start hacking on that

__run.sh

script to debug further.

Here's my best guess as to what's happening: Pants is setting up a subprocess for determining the dependencies of tests and in that subprocess it's masking the environment variable. This is the same error we had before we unmasked the environment variable, it's now just happening in a subprocess instead of in the original thing, so I'm suspicious that's what's going on.

Aha, you are right: https://github.com/pantsbuild/pants/blob/37bf6071b9dea6074176cbc67e97cdb70fccb71d/[…]ackend/python/dependency_inference/parse_python_dependencies.py It looks like that has been broken since inception in November of 2020. All? other Python processes Pants launch use a

PexProcess

or `VenvPexProcess`and those automatically support

subprocess_environment

env var propagation.

Ooof, sorry for the trouble!

I'm facing this problem today, and for irritating unrelated reasons I am not easily able to use pyenv as a replacement for setup-python in my CI runners without some additional pain which I haven't yet got to the bottom of. I'm planning to try to see if @bitter-ability-32190's work to have pants manage its own python runtime might help here, but I'd be interested to know if anybody has other thoughts / if any progress was made on the underlying issue of how these subprocesses are spawned?

For those who are suffering from the same issue (using setup-python action with Pants on self hosted Github runner): create a folder at

/opt/hostedtoolcache

, append line

AGENT_TOOLSDIRECTORY=/opt/hostedtoolcache

to the

.env

file where actions runner package is installed and then restart the actions service. I’ve managed to run CI jobs with Pants successfully on our self hosted runners by applying this workaround.

Good tip! Would you be able to send a PR that adds this to docs/markdown/Using Pants/troubleshooting.md ?

Sure! I’ll file Pull Request and post the link at this thread.