https://pantsbuild.org/ logo
Join Slack
Powered by
good morning. i'm running into an issue that i'm h...
# general
i
good morning. i'm running into an issue that i'm hoping someone can help me with. i encountered this issue when trying to upgrade a particular python package and encountered the following error: 
ERROR: Package 'instructor' requires a different Python: 3.9.18 not in '<4.0,>=3.10'
. this is running inside a docker container based on the 
python:3.10.12
image which only contains python 3.10.12. i dug a little and found that when pants bootstraps itself, it installs a different python version in its own virtualenv. in this case, it's installed python 3.9.18. when pants runs, it's using this version and causing issues with upgrading this package. among other settings i have the following in my `pants.toml`:
Copy code
[python]
enable_resolves = true
interpreter_constraints = ['CPython==3.10.12']
can someone help me understand why pants isn't respecting the 
interpreter_constraints
parameter or using the system python? this is using pants 2.17.0
b
Sorry for the trouble. Can you provide a little more context for the error message? For example, what goal are you running on what target definition?
Potentially https://github.com/pantsbuild/pants/issues/19514 is related. In which case, overrriding the 
shebang
is a suggested workaround for that version of pants.
i
sure thing. the goal is 
docker_image
which is defined like this:
Copy code
docker_image(
    name = "email_service_image",
    dependencies = [":binary"],
    repository = "<http://registry.fly.io/email_service|registry.fly.io/email_service>",
    tags = [
        "latest",
        "<http://registry.fly.io/email_service:latest|registry.fly.io/email_service:latest>",
    ],
)
the 
binary
dependency is defined as follows:
Copy code
pex_binary(
    name = "binary",
    dependencies = [
        ":lib",
        ":celery_with_monitor",
    ],
    execution_mode = "venv",
    include_tools = True,
    # Optimal settings for Docker builds
    layout = "packed",
    platforms = [
        "manylinux_2_17_x86_64-cp-310-cp310",
        "manylinux_2_17_aarch64-cp-310-cp310",
        "macosx_13_0_arm64-cp-310-cp310",
    ],
    shebang = "#!/usr/bin/env python",
)
i've tried with and without the 
shebang
definition and it doesn't appear to have any impact.
i'm hitting the error when running: 
pants package email_service:email_service_image
b
Ah okay. Could you share the full error output?
i
Copy code
19:07:52.45 [INFO] Completed: Building local_dists.pex
19:08:07.66 [INFO] Completed: Building 37 requirements for email_service/binary.pex from the 3rdparty/python/default.lock resolve: beautifulsoup4==4.12.2, celery==5.3.1, fastapi==0.103.0, flower==2.0.1, google-api-python-client... (652 characters truncated)
19:08:07.67 [ERROR] 1 Exception encountered:

Engine traceback:
  in `package` goal

ProcessExecutionFailure: Process 'Building 37 requirements for email_service/binary.pex from the 3rdparty/python/default.lock resolve: beautifulsoup4==4.12.2, celery==5.3.1, fastapi==0.103.0, flower==2.0.1, google-api-python-client==2.91.0, google-auth-httplib2==0.1.0, google-auth-oauthlib==1.0.0, google-cloud-vision==3.5.0, instructor==0.5.2, langsmith==0.0.69, loguru==0.7.1, nltk==3.6.5, openai==1.3.7, psycopg2-binary==2.9.6, pydantic==2.5.3, pypdf==3.12.0, python-dateutil==2.8.2, python-dotenv==1.0.0, python-jose[cryptography]==3.3.0, pytz==2023.3, redis==4.5.4, requests==2.31.0, resend==0.7.2, sentry-sdk==1.34.*, sentry-sdk[celery]==1.34.*, sentry-sdk[fastapi]==1.34.*, sqlalchemy==2.0.23, tiktoken==0.4.0, types-beautifulsoup4, types-python-dateutil, types-pytz, types-requests, types-urllib3, typing-extensions==4.8.*, unidecode==1.3.7, urllib3==1.26.*, uvicorn==0.21.1' failed with exit code 1.
stdout:

stderr:
There was 1 error downloading required artifacts:
1. instructor 0.5.2 from <https://files.pythonhosted.org/packages/3d/57/35e22db00463ac2a2f57e7b8823d0084c5e83f69d518313bdfbf61764463/instructor-0.5.2-py3-none-any.whl>
    ERROR: Package 'instructor' requires a different Python: 3.9.18 not in '<4.0,>=3.10'



Use `--keep-sandboxes=on_failure` to preserve the process chroot for inspection.
this is the output when pants is bootstrapping:
Copy code
# pants
Downloading <https://github.com/indygreg/python-build-standalone/releases/download/20240107/cpython-3.9.18%2B20240107-aarch64-unknown-linux-gnu-install_only.tDownloading> <https://github.com/indygreg/python-build-standalone/releases/download/20240107/cpython-3.9.18%2B20240107-aarch64-unknown-linux-gnu-install_only.tDownloading> <https://github.com/indygreg/python-build-standalone/releases/download/20240107/cpython-3.9.18%2B20240107-aarch64-unknown-linux-gnu-install_only.tDownloading> <https://github.com/indygreg/python-build-standalone/releases/download/20240107/cpython-3.9.18%2B20240107-aarch64-unknown-linux-gnu-install_only.tDownloading> <https://github.com/indygreg/python-build-standalone/releases/download/20240107/cpython-3.9.18%2B20240107-aarch64-unknown-linux-gnu-install_only.tDownloading> <https://github.com/indygreg/python-build-standalone/releases/download/20240107/cpython-3.9.18%2B20240107-aarch64-unknown-linux-gnu-install_only.tDownloading> <https://github.com/indygreg/python-build-standalone/releases/download/20240107/cpython-3.9.18%2B20240107-aarch64-unknown-linux-gnu-install_only.tDownloading> <https://github.com/indygreg/python-build-standalone/releases/download/20240107/cpython-3.9.18%2B20240107-aarch64-unknown-linux-gnu-install_only.tDownloading> <https://github.com/indygreg/python-build-standalone/releases/download/20240107/cpython-3.9.18%2B20240107-aarch64-unknown-linux-gnu-install_only.tDownloading> <https://github.com/indygreg/python-build-standalone/releases/download/20240107/cpython-3.9.18%2B20240107-aarch64-unknown-linux-gnu-install_only.tDownloading> <https://github.com/indygreg/python-build-standalone/releases/download/20240107/cpython-3.9.18%2B20240107-aarch64-unknown-linux-gnu-install_only.tDownloading> <https://github.com/indygreg/python-build-standalone/releases/download/20240107/cpython-3.9.18%2B20240107-aarch64-unknown-linux-gnu-install_only.tDownloading> <https://github.com/indygreg/python-build-standalone/releases/download/20240107/cpython-3.9.18%2B20240107-aarch64-unknown-linux-gnu-install_only.tDownloading> <https://github.com/indygreg/python-build-standalone/releases/download/20240107/cpython-3.9.18%2B20240107-aarch64-unknown-linux-gnu-install_only.tDownloading> <https://github.com/indygreg/python-build-standalone/releases/download/20240107/cpython-3.9.18%2B20240107-aarch64-unknown-linux-gnu-install_only.tDownloading> <https://github.com/indygreg/python-build-standalone/releases/download/20240107/cpython-3.9.18%2B20240107-aarch64-unknown-linux-gnu-install_only.tDownloading> <https://github.com/indygreg/python-build-standalone/releases/download/20240107/cpython-3.9.18%2B20240107-aarch64-unknown-linux-gnu-install_only.tDownloading> <https://github.com/indygreg/python-build-standalone/releases/download/20240107/cpython-3.9.18%2B20240107-aarch64-unknown-linux-gnu-install_only.tDownloading> <https://github.com/indygreg/python-build-standalone/releases/download/20240107/cpython-3.9.18%2B20240107-aarch64-unknown-linux-gnu-install_only.tDownloading> <https://github.com/indygreg/python-build-standalone/releases/download/20240107/cpython-3.9.18%2B20240107-aarch64-unknown-linux-gnu-install_only.tDownloading> <https://github.com/indygreg/python-build-standalone/releases/download/20240107/cpython-3.9.18%2B20240107-aarch64-unknown-linux-gnu-install_only.tDownloading> <https://github.com/indygreg/python-build-standalone/releases/download/20240107/cpython-3.9.18%2B20240107-aarch64-unknown-linux-gnu-install_only.tDownloading> <https://github.com/indygreg/python-build-standalone/releases/download/20240107/cpython-3.9.18%2B20240107-aarch64-unknown-linux-gnu-install_only.tDownloading> <https://github.com/indygreg/python-build-standalone/releases/download/20240107/cpython-3.9.18%2B20240107-aarch64-unknown-linux-gnu-install_only.tDownloading> <https://github.com/indygreg/python-build-standalone/releases/download/20240107/cpython-3.9.18%2B20240107-aarch64-unknown-linux-gnu-install_only.tDownloading> <https://github.com/indygreg/python-build-standalone/releases/download/20240107/cpython-3.9.18%2B20240107-aarch64-unknown-linux-gnu-install_only.tDownloading> <https://github.com/indygreg/python-build-standalone/releases/download/20240107/cpython-3.9.18%2B20240107-aarch64-unknown-linux-gnu-install_only.tDownloading> <https://github.com/indygreg/python-build-standalone/releases/download/20240107/cpython-3.9.18%2B20240107-aarch64-unknown-linux-gnu-install_only.tDownloading> <https://github.com/indygreg/python-build-standalone/releases/download/20240107/cpython-3.9.18%2B20240107-aarch64-unknown-linux-gnu-install_only.tDownloading> <https://github.com/indygreg/python-build-standalone/releases/download/20240107/cpython-3.9.18%2B20240107-aarch64-unknown-linux-gnu-install_only.tDownloading> <https://github.com/indygreg/python-build-standalone/releases/download/20240107/cpython-3.9.18%2B20240107-aarch64-unknown-linux-gnu-install_only.tDownloading> <https://github.com/indygreg/python-build-standalone/releases/download/20240107/cpython-3.9.18%2B20240107-aarch64-unknown-linux-gnu-install_only.tDownloading> <https://github.com/indygreg/python-build-standalone/releases/download/20240107/cpython-3.9.18%2B20240107-aarch64-unknown-linux-gnu-install_only.tDownloading> <https://github.com/indygreg/python-build-standalone/releases/download/20240107/cpython-3.9.18%2B20240107-aarch64-unknown-linux-gnu-install_only.tDownloading> <https://github.com/indygreg/python-build-standalone/releases/download/20240107/cpython-3.9.18%2B20240107-aarch64-unknown-linux-gnu-install_only.tDownloading> <https://github.com/indygreg/python-build-standalone/releases/download/20240107/cpython-3.9.18%2B20240107-aarch64-unknown-linux-gnu-install_only.tDownloading> <https://github.com/indygreg/python-build-standalone/releases/download/20240107/cpython-3.9.18%2B20240107-aarch64-unknown-linux-gnu-install_only.tBootstrapping> Pants 2.17.0
Installing pantsbuild.pants==2.17.0 into a virtual environment at /root/.cache/nce/d4d446890167dccc8fe0f286a7030d591d4ac22718c0ee941000c26ab7c2f7ec/bindings/venvs/2.17.0
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 26.3 MB/s eta 0:00:00
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 819.3/819.3 kB 48.0 MB/s eta 0:00:00
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 65.4/65.4 kB 15.4 MB/s eta 0:00:00
New virtual environment successfully created at /root/.cache/nce/d4d446890167dccc8fe0f286a7030d591d4ac22718c0ee941000c26ab7c2f7ec/bindings/venvs/2.17.0.
19:07:02.73 [INFO] Initializing scheduler...
19:07:07.63 [INFO] Scheduler initialized.
No goals specified.
Use `pants help` to get help.
Use `pants help goals` to list goals.
b
Hmm, I’m not sure what’s going on. Can you create a minimal example that reproduces the problem and file an issue?
i
sure i'll see what i can come up with. thanks for looking at this!
r
I can't say for sure if it is because of using 
platforms
which is discouraged in 
pex_binary
. Try with 
--complete-platforms
Huon has provided a great write-up here. It talks about lambda but check the Docker image artefacts from PEX
i
for my own understanding, should i expect pants to be downloading python version 3.9.18 at any point? i don't understand how pants determines what python version to use
the error goes away when i remove 
platforms
from the 
pex_binary
goal, but then i can't run the docker container locally (using osx). it returns if i add 
complete_platforms
, but i can once again run the docker container locally:
Copy code
# generated by running: 
# docker run -it --platform linux/arm64 python:3.10.12 bash -c 'pip install pex; pex3 interpreter inspect --markers --tags'
# docker run -it --platform linux/amd64 python:3.10.12 bash -c 'pip install pex; pex3 interpreter inspect --markers --tags'
# 
# learned from here: <https://github.com/pantsbuild/pants/discussions/18756>
files( 
    name="docker_python_3_10", 
    sources=["docker_python_3_10_*.json"]
)

pex_binary(
    name = "binary",
    dependencies = [
        ":lib",
        ":celery_with_monitor",
    ],
    execution_mode = "venv",
    include_tools = True,
    # Optimal settings for Docker builds
    layout = "packed",
    complete_platforms = [":docker_python_3_10"]
)
c
should i expect pants to be downloading python version 3.9.18 at any point?
yes, when bootstrapping pants, it installs a Python that it is compatible with, that has nothing to do with the Python you use for your own project.
i
gotcha. thanks for that clarification!
👍 1
a
Was anyone able to solve this seeing the same issue when running pants in github actions 😕
i
i haven't been able to figure out where the problem lies - is it 
instructor
or something w/ pants/pex? i've looked at the instructor config and it requires python 3.10 or higher, but i can't tell if poetry is building their wheel correctly. i've also tried upgrading the pex cli version but this doesn't change anything. my 'solution' has been to have 2 builds - one for local and one in GHA. for GHA, i don't have the 
complete_platforms
spec. for local, i do.
something like:
Copy code
pex_binary(
    name = "binary",
    dependencies = [
        ":lib",
        ":celery_with_monitor",
    ],
    execution_mode = "venv",
    include_tools = True,
    # Optimal settings for Docker builds
    layout = "packed",
)

pex_binary(
    name = "binary_osx",
    dependencies = [
        ":lib",
        ":celery_with_monitor",
    ],
    execution_mode = "venv",
    include_tools = True,
    # Optimal settings for Docker builds
    layout = "packed",
    complete_platforms = [":docker_python_3_10"]
)
then, 2 
docker_image
targets and 2 docker files
a
Ok, that's painful 😓 Same error over here. I got it when I added a different package with a python requirement: 
ERROR: Package 'pbipy' requires a different Python: 3.9.18 not in '>=3.10'
i
agreed that it's painful. i don't have enough understanding of how pants installs deps to dig into this more. i'm leaning towards it being a pants/pex issue due to pants using its own version of python - and that version is 3.9.18 right now.
a
I got it working in Github Actions in a slightly less painful way. For some reason building the docker images succeed if I first run 
pants test ::
followed by 
pants publish ::
. Looks like the 
test
step manages to build all the pex-files, which the 
package
step is not able to do.
i
funky. i'll see if that works on my end. thanks for the tip
this works for me. thanks a lot @able-advantage-61346!
b
Huh, that's peculiar; thanks for finding a work-around. What sort of caching do you use on GHA? Can you share your workflow?
i
here are the relevant parts:
Copy code
jobs:
  search_api_deploy:
    name: Deploy app
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repo
        uses: actions/checkout@v3
      - name: Setup fly
        uses: superfly/flyctl-actions/setup-flyctl@master
      - name: Setup pants
        uses: pantsbuild/actions/init-pants@v5-scie-pants
        # This action bootstraps pants and manages 2-3 GHA caches.
        # See: <http://github.com/pantsbuild/actions/tree/main/init-pants/|github.com/pantsbuild/actions/tree/main/init-pants/>
        with:
          # v0 makes it easy to bust the cache if needed
          # just increase the integer to start with a fresh cache
          gha-cache-key: v1
          # The Python backend uses named_caches for Pip/PEX state,
          # so it is appropriate to invalidate on lockfile changes.
          named-caches-hash: ${{ hashFiles('python-default.lock') }}
          # If you're not using a fine-grained remote caching service (see <https://www.pantsbuild.org/docs/remote-caching>),
          # then you may also want to preserve the local Pants cache (lmdb_store). However this must invalidate for
          # changes to any file that can affect the build, so may not be practical in larger repos.
          # A remote cache service integrates with Pants's fine-grained invalidation and avoids these problems.
          cache-lmdb-store: "true" # defaults to 'false'
          # Note that named_caches and lmdb_store falls back to partial restore keys which
          # may give a useful partial result that will save time over completely clean state,
          # but will cause the cache entry to grow without bound over time.
          # See <https://pants.readme.io/docs/using-pants-in-ci> for tips on how to periodically clean it up.
          # Alternatively you change gha-cache-key to ignore old caches.
      - name: Lint
        run: |
          pants lint ::
      - name: Typecheck
        run: |
          pants check ::
      - name: Publish image
        run: |
          flyctl auth docker
          pants --tag="-no_ci" test :: || true
          IMAGE_TAG=latest pants publish api_search:api_search_image
after the line 
pants --tag="-no_ci" test :: || true
was added, the publish step works
as an update, this does not work all the time for me. it worked a few times but has started failing w/ the same error message as before
Copy code
ERROR: Package 'instructor' requires a different Python: 3.9.18 not in '<4.0,>=3.10'
b
Ah okay. I wonder if this is a caching issue, where if some process runs before packaging it seeds the cache appropriately (specifically a "named cache"), but if it doesn't run, the package step operates incorrectly. This would align with 
pants test ::
running first sometimes fixing the problem (if the 'magic' process is required for a test and the test has changed, it'll be run and seed the named cache, but if those tests don't need to run, the named cache will be left unseeded). If you run locally with 
IMAGE_TAG=latest pants --named-caches-dir=$(mktemp -d) package api_search:api_search_image
does it reproduce the problem? That sets https://www.pantsbuild.org/2.19/reference/global-options#named_caches_dir to a fresh temporary directory, i.e. simulating running without whatever magic seeds the cache. If that doesn't reproduce, could you try in CI with an invocation like:
Copy code
pants --force --tag="-no_ci" test :: || true
IMAGE_TAG=latest pants package api_search:api_search_image
IMAGE_TAG=latest pants --named-caches-dir=$(mktemp -d) package api_search:api_search_image
(Changes being: add 
--force
to test to ensure they all run, switching 
publish
to 
package
so we're not mutating your infrastructure for these tests 😄 , and adding the tempdir-named-cache line second) If the first 
package
work but the second fails, that aligns with my theory that it may be a named-caches issue.
✅ 1
i
IMAGE_TAG=test pants --named-caches-dir=$(mktemp -d) package api_search:api_search_image
locally (osx) reproduces the problem. no issues running 
IMAGE_TAG=test pants package api_search:api_search_image
b
Ah cool, that's a good start 🎉 It'd be good to cut this down to a minimal example, independent of your code. One approach might be narrow down to a particular test, where running that test to seed the named caches directory first works, e.g., knowing nothing, something like running this command with increasingly more specific 
some/subdirectory::
paths (NB. definitely needs a fresh tempdir each time)
Copy code
dir=$(mktemp -d)
pants --force --named-caches-dir=$dir test some/subdirectory::
IMAGE_TAG=test pants --named-caches-dir=$(mktemp -d) package api_search:api_search_image
You might be able to guess at a better way to do that knowing your code (e.g. this is likely to involve a test that requires the 
instructor
package). Once it's narrowed down like that, it'd be good to cut out all the irrelevant stuff, e.g. if the test can become just a 
python_tests()
target with one file containing only 
import instructor
, and the packaged artifact can be cut down to a single 
pex_binary
target also with just 
import instructor
... that'd be amazing (but there might be more going on), and then hopefully we can make a standalone repository reproducing the problem.
a
Got it down to something fairly small here: https://github.com/calleo/pants-package-error On the Actions tab you'll see the error in the recent builds.
🔥 2
b
Thank you! that looks great, I've cloned it but haven't had time to fully investigate
I can reproduce the problem 👍 dno what the cause is yet, though
Okay, that reproducer was crucial. Thank you. I suspect this is a bug in the underlying PEX tool: https://github.com/pex-tool/pex/issues/2395 I believe a reliable work-around might be squishing condition 3 there, by reliably ensuring that the pex_root named cache is "seeded" by a local interpreter: 1. create a 
pex_binary
target that does nothing but depend on the problematic packages, e.g. 
pex_binary(name="__workaround_pex_issue_2395", dependencies=["path/to#instructor", "path/to#pbipy"], ...)
2. run 
pants --no-local-cache  package path/to:__workaround_pex_issue_2395
in CI before all other 
package
The 
--no-local-cache
will force this to always execute the underlying 
pex ...
command that will pre-seed the internal named cache on this machine, even though that can typically be served from cache. (the "local cache" there is referring to caching of exact process inputs and outputs, which is a different cache to the more flexible "named caches" here.) The 
pants test ::
workaround is something similar, and I suspect the occasional failures of that workaround are because all the PEX invocations are sometimes fully cached, and so don't run and thus don't seed the named cache. Having a targeted 
pex_binary
will be much faster than forcing all tests to run without any caching at all. Demonstration in https://github.com/calleo/pants-package-error/pull/1 (NB. if you're using remote caching, you'll probably need 
--no-remote-cache-read
too. If you're using remote execution, this probably won't work.)
🔥 1
i
the workaround works. thank you so much for your help! also, this looks like a pretty hairy thing to find the root cause of, so major kudos for that.
are there any docs on the various types of pants caches? i'm curious to learn more
c
there's some discussion about caches on this page: https://www.pantsbuild.org/2.19/docs/using-pants/using-pants-in-ci not sure if there are any better resources available beyond that.
👍 1
a
Thank you @broad-processor-92400 for taking a thorough look at this problem and providing a better work around 🙇
b
It looks like the PEX bug is fixed in 2.3.1 that was recently released. You can update the version Pants is using internally using https://www.pantsbuild.org/2.19/reference/subsystems/pex-cli, so something like:
Copy code
[pex-cli]
version = "v2.3.1"
known_version = [
   "v2.3.1|macos_arm64|71690e672871b55323f5d6ef9a3fe9705f1668662652c4081080e7ab27d44de3|4124530",
    ...
]
The hash and length are the sha256 hash and byte length of the 
pex
artifact from https://github.com/pex-tool/pex/releases/tag/v2.3.1
🔥 2
i
can confirm that the above works. thanks again for all your help!
👍 1
12 Views
Open in Slack
PreviousNext
Powered by