pantsbuild #general

Channels

welcome

github-notifications

announce

happy-kitchen-89482

03/03/2023, 2:17 AM

but I'm not sure that works for python_requirements, it was designed to solve a problem with first party code inference, so I'll need to check

happy-kitchen-89482

03/03/2023, 5:00 AM

OK, it should work for 3rdparty deps, but you'll need to set

Copy code

[python-infer]
ambiguity_resolution = "by_source_root"

happy-kitchen-89482

03/03/2023, 5:00 AM

In pants.toml

worried-painter-31382

03/03/2023, 11:14 AM

Trying to upgrade to 2.15 +

scie-pants

, two questions: 1. Is there extra steps to make scie-pants not try to download things each run? Previously

ENV PANTS_SETUP_CACHE="/root/.cache/pants/setup"

./pants version

in a docker image sufficed to then have it be able to run "offline". I tried with

RUN ./get-pants.sh

followed by the same thing (./ omitted), but no dice. Next invocation of pants still checks the fetch URL 2. Running the github action pants-init seemingly does not expose the scie-pants interpreter, as we get

Copy code

$ pants list ::
Engine traceback:
  in Map all targets to their dependents
  in Resolve direct dependencies of target - pants-plugins/extensible_markup_language/register.py
  in Inferring Python dependencies by analyzing source
  in Find Python interpreter for constraints - CPython<3.10,>=3.9
ProcessExecutionFailure: Process 'Find interpreter for constraints: CPython<3.10,>=3.9' failed with exit code 102.
stdout:

stderr:
Could not find a compatible interpreter.

Examined the following interpreters:
1.) /opt/hostedtoolcache/Python/3.10.2/x64/bin/python3.10 CPython==3.10.2
2.)                                    /usr/bin/python3.8 CPython==3.8.10
3.)                                    /usr/bin/python2.7 CPython==2.7.18

for a plugin. The gha does blurt

Copy code

Bootstrapping Pants 2.15.0 using cpython 3.9.15
Installing pantsbuild.pants==2.15.0 into a virtual environment at /home/runner/.cache/nce/d036c571b74670fb7cccdb2ee973b51625c6419b29dd5e2e5390d6267b873904/bindings/venvs/2.15.0

Do I need to configure

pants.toml

some way for it to discover the scie-pants interpreter for plugins?

worried-painter-31382

03/03/2023, 11:17 AM

Copy code

[python-bootstrap]
search_path = [
    "<PATH>",
    "<PYENV>",
    "/usr/bin/",
     # Brew usually installs Python here.
    "/usr/local/bin/",
]

[python]
interpreter_constraints = ["CPython==3.10.2"]
tailor_pex_binary_targets = false
enable_resolves = true

[python.resolves]
python-default = "pants-lockfile.txt"
pants-plugins = "pants-plugins/lockfile.txt"

[python.resolves_to_interpreter_constraints]
pants-plugins = ["CPython>=3.9,<3.10"]

narrow-vegetable-37489

03/03/2023, 11:23 AM

I think

SCIE_BASE

is the equivalent of

PANTS_SETUP_CACHE

for scie-pants.

🎉 1

famous-xylophone-36532

03/03/2023, 12:19 PM

Hey! I am having this really weird intermittent issue when building a Docker image containing a pex. The weird part is that it does not re-occur if I just run the same target / goal. I am on an M1 Mac, and I am running Pants from the latest commit on main. Update: Now Pants just won't work. It's cycling through four different error messages each time I try to run

pants package

pants run

works fine!

worried-painter-31382

03/03/2023, 1:59 PM

Seems like CodeBuild is the real villain here, even with SCIE_BASE pointed out running

pants

still triggers download attempts in AWS CodeBuild, while docker locally works with

--network none

...

worried-painter-31382

03/03/2023, 2:06 PM

I think it is losing the symlinks...

worried-painter-31382

03/03/2023, 2:23 PM

nvm, seems to be intact

enough-analyst-54434

03/03/2023, 2:36 PM

For 2. it is very much intended that the scie-pants interpreter is only for Pants use. You must supply your own Python(s) if you use Pants for Python.

worried-painter-31382

03/03/2023, 2:37 PM

So if I author in-repo plugins I still have to install 3.9 as well?

enough-analyst-54434

03/03/2023, 2:37 PM

Yes, plugins are Python code in your repo.

enough-analyst-54434

03/03/2023, 2:37 PM

The rule is that simple if it helps.

enough-analyst-54434

03/03/2023, 2:38 PM

I mean, you don't need one if you don't run lint etc on your plugin code of course.

worried-painter-31382

03/03/2023, 2:39 PM

Right, and

list

doesn't actually use the IC:d interpreter anyway

worried-painter-31382

03/03/2023, 2:39 PM

But I understand what's causing it then, thanks.

enough-analyst-54434

03/03/2023, 2:40 PM

The reason for the restriction is actually a bit arcane.

enough-analyst-54434

03/03/2023, 2:42 PM

On 1, you're the second person to mention get-pants.sh not caching lookups. @polite-garden-50641 hacked around this in Toolchain CI with a bespoke solution. It would be great if someone improved either get-pants.sh or https://github.com/pantsbuild/actions/tree/v5-scie-pants

worried-painter-31382

03/03/2023, 3:00 PM

To clarify, it isn't

get-pants.sh

that is doing lookup, it is the scie itself (via ptex?) I can see a populated cache

Copy code

[Container] 2023/03/03 14:26:50 Running command /root/.cache/scie/d036c571b74670fb7cccdb2ee973b51625c6419b29dd5e2e5390d6267b873904/bindings/pex_root/venvs/0ff9b45b281248cdbee3cc90962390828e692ba8/e7e0ac7bf7e1e828442be44b19c2da43e06ede6c/bin/python --version
Python 3.9.15
[Container] 2023/03/03 14:26:50 Running command pants version
Failed to fetch <https://binaries.pantsbuild.org/tags/pantsbuild.pants/release_2.15.0>: [28] Timeout was reached (Failed to connect to <http://binaries.pantsbuild.org|binaries.pantsbuild.org> port 443 after 257615 ms: Couldn't connect to server)
Failed to fetch <https://api.github.com/repos/pantsbuild/pants/git/refs/tags/release_2.15.0>: [28] Timeout was reached (Failed to connect to <http://api.github.com|api.github.com> port 443 after 130887 ms: Couldn't connect to server)
Install failed: Command '['/root/.cache/scie/fd8ceac7b3026655b34b2c80997c18caeb7e8d64a9dd08c0340a5ced92f947d3/ptex', '<https://api.github.com/repos/pantsbuild/pants/git/refs/tags/release_2.15.0']'> returned non-zero exit status 1.
More information can be found in the log at: /root/.cache/scie/d036c571b74670fb7cccdb2ee973b51625c6419b29dd5e2e5390d6267b873904/bindings/logs/configure.log

but the

pants

cmd wont use it, seemingly. I'm fairly sure it is a AWS CodeBuild issue, because I can launch the bare container with --network none and still use pants therein, so the cache is discoverable in a "regular" setting

worried-painter-31382

03/03/2023, 3:03 PM

I swapped from

get-pants.sh

to bare

curl

just to be sure, and yes; no difference

enough-analyst-54434

03/03/2023, 3:05 PM

I'm a bit lost at this point. Do you or do you not expect a network connection, and if you do, is it just the fact the connection is failing that is the problem here?

worried-painter-31382

03/03/2023, 3:06 PM

I do not expect a network connection

enough-analyst-54434

03/03/2023, 3:07 PM

Because you ran

pants -V

(or something) once already in the container?

worried-painter-31382

03/03/2023, 3:07 PM

Yes, precisely

enough-analyst-54434

03/03/2023, 3:08 PM

If you add

RUST_LOG=trace pants ...

we'll get more meat about what nce cache entries are missing leading to this.

👍 1

enough-analyst-54434

03/03/2023, 3:10 PM

But, on the face of it, the nce cache directory is missing or unreadable in some way or only has entries for a different Pants version. Each new version requires downloads / network 1 time.

enough-analyst-54434

03/03/2023, 3:10 PM

... just like the

./pants

script.

worried-painter-31382

03/03/2023, 3:25 PM

Copy code

[TRACE scie_pants] Launching: Process {
        exe: "/usr/local/bin/pants",
        args: [],
        env: [
            (
                "SCIE_BOOT",
                "pants",
            ),
            (
                "PANTS_BIN_NAME",
                "pants",
            ),
            (
                "PANTS_DEBUG",
                "",
            ),
            (
                "PANTS_BUILDROOT_OVERRIDE",
                "/codebuild/output/src246925278/src",
            ),
            (
                "PANTS_TOML",
                "/codebuild/output/src246925278/src/pants.toml",
            ),
            (
                "PANTS_VERSION",
                "2.15.0",
            ),
        ],
    }

is the scie thinking it's a "new repo" because of the BUILDROOT_OVERRIDE?

worried-painter-31382

03/03/2023, 3:26 PM

Copy code

[DEBUG jump::atomic] The atomic file at /root/.cache/scie/fd8ceac7b3026655b34b2c80997c18caeb7e8d64a9dd08c0340a5ced92f947d3/ptex has already been established.
[DEBUG TimerFinished] installer::unpack_blob(), Elapsed=8.882µs
[DEBUG jump::atomic] The atomic file at /root/.cache/scie/04c117900eb9bf6005a70e25aa2dfe83ac94d648920c65b356c79cf8efdf48dc/tools.pex has already been established.
[DEBUG TimerFinished] installer::unpack_blob(), Elapsed=11.613µs
[DEBUG jump::atomic] The atomic directory at /root/.cache/scie/cdc3a4cfddcd63b6cebdd75b14970e02d8ef0ac5be4d350e57ab5df56c19e85e/cpython-3.9.15+20221106-x86_64-unknown-linux-gnu-install_only.tar.gz has already been established.

So the cache works for most, it's just the pants installation it wont reuse?