Trying to upgrade to 2.15 +

scie-pants

, two questions: 1. Is there extra steps to make scie-pants not try to download things each run? Previously

ENV PANTS_SETUP_CACHE="/root/.cache/pants/setup"

./pants version

in a docker image sufficed to then have it be able to run "offline". I tried with

RUN ./get-pants.sh

followed by the same thing (./ omitted), but no dice. Next invocation of pants still checks the fetch URL 2. Running the github action pants-init seemingly does not expose the scie-pants interpreter, as we get

$ pants list ::
Engine traceback:
  in Map all targets to their dependents
  in Resolve direct dependencies of target - pants-plugins/extensible_markup_language/register.py
  in Inferring Python dependencies by analyzing source
  in Find Python interpreter for constraints - CPython<3.10,>=3.9
ProcessExecutionFailure: Process 'Find interpreter for constraints: CPython<3.10,>=3.9' failed with exit code 102.
stdout:

stderr:
Could not find a compatible interpreter.

Examined the following interpreters:
1.) /opt/hostedtoolcache/Python/3.10.2/x64/bin/python3.10 CPython==3.10.2
2.)                                    /usr/bin/python3.8 CPython==3.8.10
3.)                                    /usr/bin/python2.7 CPython==2.7.18

for a plugin. The gha does blurt

Bootstrapping Pants 2.15.0 using cpython 3.9.15
Installing pantsbuild.pants==2.15.0 into a virtual environment at /home/runner/.cache/nce/d036c571b74670fb7cccdb2ee973b51625c6419b29dd5e2e5390d6267b873904/bindings/venvs/2.15.0

Do I need to configure

pants.toml

some way for it to discover the scie-pants interpreter for plugins?

I think

SCIE_BASE

is the equivalent of

PANTS_SETUP_CACHE

for scie-pants.

Seems like CodeBuild is the real villain here, even with SCIE_BASE pointed out running

pants

still triggers download attempts in AWS CodeBuild, while docker locally works with

--network none

...

For 2. it is very much intended that the scie-pants interpreter is only for Pants use. You must supply your own Python(s) if you use Pants for Python.

So if I author in-repo plugins I still have to install 3.9 as well?

Yes, plugins are Python code in your repo.

Right, and

list

doesn't actually use the IC:d interpreter anyway

The reason for the restriction is actually a bit arcane.

To clarify, it isn't

get-pants.sh

that is doing lookup, it is the scie itself (via ptex?) I can see a populated cache

[Container] 2023/03/03 14:26:50 Running command /root/.cache/scie/d036c571b74670fb7cccdb2ee973b51625c6419b29dd5e2e5390d6267b873904/bindings/pex_root/venvs/0ff9b45b281248cdbee3cc90962390828e692ba8/e7e0ac7bf7e1e828442be44b19c2da43e06ede6c/bin/python --version
Python 3.9.15
[Container] 2023/03/03 14:26:50 Running command pants version
Failed to fetch <https://binaries.pantsbuild.org/tags/pantsbuild.pants/release_2.15.0>: [28] Timeout was reached (Failed to connect to <http://binaries.pantsbuild.org|binaries.pantsbuild.org> port 443 after 257615 ms: Couldn't connect to server)
Failed to fetch <https://api.github.com/repos/pantsbuild/pants/git/refs/tags/release_2.15.0>: [28] Timeout was reached (Failed to connect to <http://api.github.com|api.github.com> port 443 after 130887 ms: Couldn't connect to server)
Install failed: Command '['/root/.cache/scie/fd8ceac7b3026655b34b2c80997c18caeb7e8d64a9dd08c0340a5ced92f947d3/ptex', '<https://api.github.com/repos/pantsbuild/pants/git/refs/tags/release_2.15.0']'> returned non-zero exit status 1.
More information can be found in the log at: /root/.cache/scie/d036c571b74670fb7cccdb2ee973b51625c6419b29dd5e2e5390d6267b873904/bindings/logs/configure.log

but the

pants

cmd wont use it, seemingly. I'm fairly sure it is a AWS CodeBuild issue, because I can launch the bare container with --network none and still use pants therein, so the cache is discoverable in a "regular" setting

I'm a bit lost at this point. Do you or do you not expect a network connection, and if you do, is it just the fact the connection is failing that is the problem here?

I do not expect a network connection

Because you ran

pants -V

(or something) once already in the container?

Yes, precisely

If you add

RUST_LOG=trace pants ...

we'll get more meat about what nce cache entries are missing leading to this.

[TRACE scie_pants] Launching: Process {
        exe: "/usr/local/bin/pants",
        args: [],
        env: [
            (
                "SCIE_BOOT",
                "pants",
            ),
            (
                "PANTS_BIN_NAME",
                "pants",
            ),
            (
                "PANTS_DEBUG",
                "",
            ),
            (
                "PANTS_BUILDROOT_OVERRIDE",
                "/codebuild/output/src246925278/src",
            ),
            (
                "PANTS_TOML",
                "/codebuild/output/src246925278/src/pants.toml",
            ),
            (
                "PANTS_VERSION",
                "2.15.0",
            ),
        ],
    }

is the scie thinking it's a "new repo" because of the BUILDROOT_OVERRIDE?

It definitely re-uses everything. The cache key does include the environment though for bindings (the Pants version X.Y.Z is done by a binding command). Can you please include the full trace output instead of snippets?

rust_log.txt

And, for clarity, what is the exact GHA action yaml you use - basically what is your complete

uses: pantsbuild/actions/init-pants...

block snippet?

This is not in GHA

Ok, then what is the relevant thing for me to look at?

I have a minimal dockerfile:

# Install pants
COPY pants.toml /opt/build/pants.toml
ENV PANTS_PEX_CLI_VERSION="v2.1.111"
RUN curl -L -O <https://github.com/pantsbuild/pex/releases/download/$PANTS_PEX_CLI_VERSION/pex> && chmod +x ./pex

WORKDIR /opt/build
ENV SCIE_BASE="/root/.cache/nce"
RUN curl -fLO \
  <https://github.com/pantsbuild/scie-pants/releases/download/v0.5.2/scie-pants-linux-x86_64>
RUN curl -fL \
  <https://github.com/pantsbuild/scie-pants/releases/download/v0.5.2/scie-pants-linux-x86_64.sha256> \
  | sha256sum -c -
RUN chmod +x scie-pants-linux-x86_64 && mv scie-pants-linux-x86_64 /usr/local/bin/pants
ENV PANTS_SETUP_CACHE="/root/.cache/pants/setup"
ENV PANTS_PEX_CLI_URL_TEMPLATE="file:/pex"
RUN pants version

running in aws codebuild, it's in a private isolated network. Really I'd just like to understand what causes the

scie-pants

to re-download pants

Ok, the relevant binding that misses the cache is the Pants version install:

[TRACE jump::context] Installing boot binding Binding {
        target: "/root/.cache/scie/d036c571b74670fb7cccdb2ee973b51625c6419b29dd5e2e5390d6267b873904/locks/configure-b7aeeb8b50858aff30ce47276ef8773d8eb6d5815b7942c97912685abcd733e7",
        process: Process {
            env: EnvVars {
                vars: [
                    Default(
                        (
                            "PANTS_VERSION_PROMPT_SALT",
                            "",
                        ),
                    ),
                    RemoveMatching(
                        ComparableRegex(
                            PEX_.*,
                        ),
                    ),
                    Replace(
                        (
                            "PEX_PYTHON_PATH",
                            "/root/.cache/scie/cdc3a4cfddcd63b6cebdd75b14970e02d8ef0ac5be4d350e57ab5df56c19e85e/cpython-3.9.15+20221106-x86_64-unknown-linux-gnu-install_only.tar.gz/python/bin/python3.9",
                        ),
                    ),
                    Replace(
                        (
                            "PEX_ROOT",
                            "/root/.cache/scie/d036c571b74670fb7cccdb2ee973b51625c6419b29dd5e2e5390d6267b873904/bindings/pex_root",
                        ),
                    ),
                ],
            },
            exe: "/root/.cache/scie/cdc3a4cfddcd63b6cebdd75b14970e02d8ef0ac5be4d350e57ab5df56c19e85e/cpython-3.9.15+20221106-x86_64-unknown-linux-gnu-install_only.tar.gz/python/bin/python3.9",
            args: [
                "/root/.cache/scie/04c117900eb9bf6005a70e25aa2dfe83ac94d648920c65b356c79cf8efdf48dc/tools.pex",
                "configure-pants",
                "--ptex-path",
                "/root/.cache/scie/fd8ceac7b3026655b34b2c80997c18caeb7e8d64a9dd08c0340a5ced92f947d3/ptex",
                "--pants-version",
                "2.15.0",
                "--pants-sha",
                "",
                "--pants-config",
                "/codebuild/output/src246925278/src/pants.toml",
                "--github-api-bearer-token",
                "",
                "/root/.cache/scie/d036c571b74670fb7cccdb2ee973b51625c6419b29dd5e2e5390d6267b873904/bindings",
            ],
        },
    }
[DEBUG jump::atomic] The atomic file at /root/.cache/scie/fd8ceac7b3026655b34b2c80997c18caeb7e8d64a9dd08c0340a5ced92f947d3/ptex has already been established.
[DEBUG TimerFinished] installer::unpack_blob(), Elapsed=8.882µs
[DEBUG jump::atomic] The atomic file at /root/.cache/scie/04c117900eb9bf6005a70e25aa2dfe83ac94d648920c65b356c79cf8efdf48dc/tools.pex has already been established.
[DEBUG TimerFinished] installer::unpack_blob(), Elapsed=11.613µs
[DEBUG jump::atomic] The atomic directory at /root/.cache/scie/cdc3a4cfddcd63b6cebdd75b14970e02d8ef0ac5be4d350e57ab5df56c19e85e/cpython-3.9.15+20221106-x86_64-unknown-linux-gnu-install_only.tar.gz has already been established.
[DEBUG TimerFinished] installer::unpack_archive(), Elapsed=4.846µs
[DEBUG TimerFinished] Installer::install(), Elapsed=32.059µs
Failed to fetch <https://binaries.pantsbuild.org/tags/pantsbuild.pants/release_2.15.0>: [28] Timeout was reached (Failed to connect to <http://binaries.pantsbuild.org|binaries.pantsbuild.org> port 443 after 257769 ms: Couldn't connect to server)

So this is not actually a codebuild problem

That's a first 😅

And, as I usually get to point out, the root problem here is me. That's my bug. But the trigger is Mac M1. The only reason the configure step is in there is to support M1 😕

Thanks a lot! My team is very excited for scie-pants

Alright, I was able to work around in scie-pants: https://github.com/pantsbuild/scie-pants/pull/130 I should have a release out later today. I'll take a look now at scie-jump and see if fancier binding caching / invalidation logic can solve this problem for all scie-jump users.