https://pantsbuild.org/ logo
Join Slack
Powered by
edit: I think I figured it out, there's another bu...
# general
f
edit: I think I figured it out, there's another build target in play which wasn't fully constrained. I thought 
interpreter_constraint
would propagate between dependencies but I guess it's not that clever (it's still quite clever) okay, interesting, I also had to update the lockfile for 
python-default
even though that isn't being touched in this build step I currently have a single-interpreter-version monorepo and I'm trying to move into one where I use resolves to target different dependency trees and different interpreter versions. Here's the relevant root pants.toml:
Copy code
[GLOBAL]
pants_version = "2.21.0"
pants_ignore_use_gitignore = false
pants_ignore.add = ["cdk.out"]
pants_distdir = ""
build_file_prelude_globs = ["pants-plugins/macros.py"]

backend_packages = [
  "pants.backend.python",
  "pants.backend.awslambda.python",

]

[python-infer]
unowned_dependency_behavior="ignore"

[python]
interpreter_constraints = ["CPython==3.9.*", "CPython==3.11.*" ]
enable_resolves = true

[python.resolves]
python-default = "3rdparty/python/default.lock"
csvkit = "3rdparty/python/csvkit.lock"

[generate-lockfiles]
diff = true
this is my current 
convert_to_csv/BUILD
file
Copy code
python_requirements(
    name="reqs_convert_to_csv",
    resolve="csvkit",
    source="pyproject.toml",
)

python_sources(
    name="convert_to_csv",
    resolve="csvkit",
    interpreter_constraints=["CPython==3.11.*"],
)

python_aws_lambda_function(
    name='lambda',
    dependencies=[':reqs_convert_to_csv', ':convert_to_csv'],
    output_path='app/utc/actions/convert_to_csv/dist/lambda.zip',
    tags=['zip'],
    resolve="csvkit",
    runtime="python3.11",
    handler="runtime/lambda_function.py:handler",
)
But it doesn't seem to be respecting either 
runtime
or 
interpreter_constraints
and instead throwing this error on 
pants package mypackage::
Copy code
InvalidTargetException: The 'python_aws_lambda_layer' target app/utc/actions/convert_to_csv/csvkit_layer:csvkit_layer cannot have its runtime platform inferred, because inference requires simple interpreter constraints covering exactly one minor release of Python, and all its patch version. The constraints for this target (CPython==3.11.* OR CPython==3.9.*) aren't understood.

To fix, provide one of the following:

- a value for the `runtime` field, or

- a value for the `complete_platforms` field, or

- simple and narrow interpreter constraints (for example, `==3.10.*` or `>=3.10,<3.11` are simple enough to imply Python 3.10)
here's debug output
Copy code
$> ENV=qa VERSION=v0.0.0 pants -ldebug package app/utc::                                                                                     

[2024-07-16T21:17:05Z DEBUG pantsd] Looking for pantsd metadata in /Users/michaelmcdermott/bp/micro_stacks/.pants.d/pids as specified by [GLOBAL] pants_subprocessdir = .pants.d/pids via Default.
[2024-07-16T21:17:05Z DEBUG pantsd] Parsed port 51448 from /Users/michaelmcdermott/bp/micro_stacks/.pants.d/pids/7852f5a4722c/pantsd/socket.
17:17:07.06 [DEBUG] acquiring lock: <pants.pantsd.lock.OwnerPrintingInterProcessFileLock object at 0x106353c10>
17:17:07.07 [DEBUG] terminating pantsd
17:17:07.07 [DEBUG] sending signal 15 to pid 89445
17:17:07.38 [DEBUG] successfully terminated pid 89445
17:17:07.38 [DEBUG] purging metadata directory: /Users/michaelmcdermott/bp/micro_stacks/.pants.d/pids/7852f5a4722c/pantsd
17:17:07.40 [DEBUG] Launching pantsd
17:17:07.40 [DEBUG] purging metadata directory: /Users/michaelmcdermott/bp/micro_stacks/.pants.d/pids/7852f5a4722c/pantsd
17:17:07.40 [DEBUG] pantsd command is: PANTS_DAEMON_ENTRYPOINT=pants.pantsd.pants_daemon:launch_new_pantsd_instance PYTHONPATH=/Users/michaelmcdermott/Library/Caches/nce/6faa4322d1df41d032e4938795c6f2c262ab92bb642a9bac1101cb7d1631f9c1/bindings/venvs/2.21.0/bin:/Users/michaelmcdermott/Library/Caches/nce/bf0cd90204a2cc6da48cae1e4b32f48c9f7031fbe1238c5972104ccb0155d368/cpython-3.9.18+20240107-aarch64-apple-darwin-install_only.tar.gz/python/lib/python39.zip:/Users/michaelmcdermott/Library/Caches/nce/bf0cd90204a2cc6da48cae1e4b32f48c9f7031fbe1238c5972104ccb0155d368/cpython-3.9.18+20240107-aarch64-apple-darwin-install_only.tar.gz/python/lib/python3.9:/Users/michaelmcdermott/Library/Caches/nce/bf0cd90204a2cc6da48cae1e4b32f48c9f7031fbe1238c5972104ccb0155d368/cpython-3.9.18+20240107-aarch64-apple-darwin-install_only.tar.gz/python/lib/python3.9/lib-dynload:/Users/michaelmcdermott/Library/Caches/nce/6faa4322d1df41d032e4938795c6f2c262ab92bb642a9bac1101cb7d1631f9c1/bindings/venvs/2.21.0/lib/python3.9/site-packages /Users/michaelmcdermott/Library/Caches/nce/6faa4322d1df41d032e4938795c6f2c262ab92bb642a9bac1101cb7d1631f9c1/bindings/venvs/2.21.0/bin/python3.9 /Users/michaelmcdermott/Library/Caches/nce/6faa4322d1df41d032e4938795c6f2c262ab92bb642a9bac1101cb7d1631f9c1/bindings/venvs/2.21.0/bin/pants -ldebug package app/utc::
17:17:10.84 [DEBUG] pantsd is running at pid 90683, pailgun port is 52165
17:17:10.84 [DEBUG] releasing lock: <pants.pantsd.lock.OwnerPrintingInterProcessFileLock object at 0x106353c10>
17:17:10.84 [DEBUG] Connecting to pantsd on port 52165
17:17:10.84 [DEBUG] Connecting to pantsd on port 52165 attempt 1/3
17:17:10.84 [DEBUG] Connected to pantsd
17:17:10.84 [DEBUG] work dir: /Users/michaelmcdermott/bp/micro_stacks
17:17:10.86 [DEBUG] Launching 1 roots (poll=false).
17:17:10.86 [DEBUG] computed 1 nodes in 0.006645 seconds. there are 9 total nodes.
17:17:10.86 [DEBUG] Launching 1 roots (poll=false).
17:17:10.86 [DEBUG] computed 1 nodes in 0.000142 seconds. there are 13 total nodes.
17:17:11.15 [INFO] Initializing scheduler...
17:17:11.16 [DEBUG] File handle limit is: 65535
17:17:11.19 [DEBUG] Using [cache::CommandRunner { inner: bounded::CommandRunner { inner: SwitchedCommandRunner { .. }, .. }, .. }, cache::CommandRunner { inner: bounded::CommandRunner { inner: SwitchedCommandRunner { .. }, .. }, .. }] for process execution.
17:17:13.53 [DEBUG] Changes to /Users/michaelmcdermott/Library/Caches/nce/6faa4322d1df41d032e4938795c6f2c262ab92bb642a9bac1101cb7d1631f9c1/bindings/venvs/2.21.0/bin, outside of the buildroot, will not be invalidated.
17:17:13.53 [DEBUG] Changes to /Users/michaelmcdermott/Library/Caches/nce/6faa4322d1df41d032e4938795c6f2c262ab92bb642a9bac1101cb7d1631f9c1/bindings/venvs/2.21.0/bin, outside of the buildroot, will not be invalidated.
17:17:13.53 [DEBUG] Changes to /Users/michaelmcdermott/Library/Caches/nce/bf0cd90204a2cc6da48cae1e4b32f48c9f7031fbe1238c5972104ccb0155d368/cpython-3.9.18+20240107-aarch64-apple-darwin-install_only.tar.gz/python/lib/python39.zip, outside of the buildroot, will not be invalidated.
17:17:13.53 [DEBUG] Changes to /Users/michaelmcdermott/Library/Caches/nce/bf0cd90204a2cc6da48cae1e4b32f48c9f7031fbe1238c5972104ccb0155d368/cpython-3.9.18+20240107-aarch64-apple-darwin-install_only.tar.gz/python/lib/python3.9, outside of the buildroot, will not be invalidated.
17:17:13.53 [DEBUG] Changes to /Users/michaelmcdermott/Library/Caches/nce/bf0cd90204a2cc6da48cae1e4b32f48c9f7031fbe1238c5972104ccb0155d368/cpython-3.9.18+20240107-aarch64-apple-darwin-install_only.tar.gz/python/lib/python3.9/lib-dynload, outside of the buildroot, will not be invalidated.
17:17:13.53 [DEBUG] Changes to /Users/michaelmcdermott/Library/Caches/nce/6faa4322d1df41d032e4938795c6f2c262ab92bb642a9bac1101cb7d1631f9c1/bindings/venvs/2.21.0/lib/python3.9/site-packages, outside of the buildroot, will not be invalidated.
17:17:13.53 [DEBUG] setting up service <pants.pantsd.service.scheduler_service.SchedulerService object at 0x1073ad670>
17:17:13.53 [DEBUG] setting up service <pants.pantsd.service.store_gc_service.StoreGCService object at 0x1078a4520>
17:17:13.53 [DEBUG] starting service <pants.pantsd.service.scheduler_service.SchedulerService object at 0x1073ad670>
17:17:13.53 [DEBUG] starting service <pants.pantsd.service.store_gc_service.StoreGCService object at 0x1078a4520>
17:17:13.53 [INFO] Scheduler initialized.
17:17:13.53 [DEBUG] Launching 1 roots (poll=false).
17:17:13.53 [DEBUG] computed 1 nodes in 0.000164 seconds. there are 13 total nodes.
17:17:13.53 [DEBUG] Launching 1 roots (poll=false).
17:17:13.53 [DEBUG] computed 1 nodes in 0.000041 seconds. there are 13 total nodes.
17:17:13.56 [DEBUG] specs are: Specs(includes=RawSpecs(description_of_origin='CLI arguments', address_literals=(), file_literals=(), file_globs=(), dir_literals=(), dir_globs=(), recursive_globs=(RecursiveGlobSpec(directory='app/utc'),), ancestor_globs=(), unmatched_glob_behavior=<GlobMatchErrorBehavior.error: 'error'>, filter_by_global_options=True, from_change_detection=False), ignores=RawSpecs(description_of_origin='CLI arguments', address_literals=(), file_literals=(), file_globs=(), dir_literals=(), dir_globs=(), recursive_globs=(), ancestor_globs=(), unmatched_glob_behavior=<GlobMatchErrorBehavior.error: 'error'>, filter_by_global_options=False, from_change_detection=False))
17:17:13.56 [DEBUG] changed_options are: ChangedOptions(since=None, diffspec=None, dependents=<DependentsOption.NONE: 'none'>)
17:17:13.56 [DEBUG] Launching 1 roots (poll=false).
17:17:13.57 [DEBUG] computed 1 nodes in 0.005957 seconds. there are 14 total nodes.
17:17:13.57 [DEBUG] Launching 1 roots (poll=false).
17:17:13.57 [DEBUG] computed 1 nodes in 0.000247 seconds. there are 19 total nodes.
17:17:13.57 [DEBUG] Launching 1 roots (poll=false).
17:17:13.57 [DEBUG] computed 1 nodes in 0.000085 seconds. there are 19 total nodes.
17:17:13.57 [DEBUG] requesting <class 'pants.core.goals.package.Package'> to satisfy execution of `package` goal
17:17:13.57 [DEBUG] Launching 1 roots (poll=false).
17:17:13.60 [WARN] Unmatched glob from //:deploy_assets's `sources` field: "deploy_assets/**/*"

Do the file(s) exist? If so, check if the file(s) are in your `.gitignore` or the global `pants_ignore` option, which may result in Pants not being able to see the file(s) even though they exist on disk. Refer to <https://www.pantsbuild.org/troubleshooting#pants-cannot-find-a-file-in-your-project>.
17:17:13.60 [DEBUG] Completed: Generate `python_requirement` targets from requirements.txt or PEP 621 compliant pyproject.toml
17:17:13.65 [DEBUG] Completed: Generate `python_requirement` targets from requirements.txt or PEP 621 compliant pyproject.toml
17:17:13.65 [DEBUG] Completed: Find targets from input specs
17:17:13.65 [DEBUG] Completed: acquire_command_runner_slot
17:17:13.65 [DEBUG] Running Searching for `bash` on PATH=/usr/bin:/bin:/usr/local/bin:/opt/homebrew/bin under semaphore with concurrency id: 1, and concurrency: 1
17:17:13.65 [DEBUG] Completed: pants.backend.python.goals.package_pex_binary.package_pex_binary
17:17:13.65 [DEBUG] Completed: Resolve transitive targets
17:17:13.66 [DEBUG] /Users/michaelmcdermott/.cache/pants/lmdb_store/immutable/files -> /private/var/folders/d5/7pkp2dhs7nx3pxfxdnfh6_4c0000gn/T hardlinkable: true
17:17:13.66 [DEBUG] Completed: Create Python AWS Lambda Layer - app/utc/actions/convert_to_csv/csvkit_layer:csvkit_layer
17:17:13.66 [DEBUG] Canceled: Create Python AWS Lambda Function - app/utc/actions/convert_to_csv:lambda
17:17:13.66 [DEBUG] Canceled: Resolve transitive targets
17:17:13.66 [DEBUG] Canceled: pants.backend.python.util_rules.pex_from_targets.create_pex_from_targets
17:17:13.66 [DEBUG] Canceled: pants.backend.python.dependency_inference.module_mapper.merge_first_party_module_mappings
17:17:13.66 [DEBUG] Canceled: Creating map of first party Python targets to Python modules
17:17:13.66 [DEBUG] Canceled: Creating map of third party targets to Python modules
17:17:13.66 [DEBUG] Canceled: Find all Python targets in project
17:17:13.66 [DEBUG] Canceled: Find all targets in the project
17:17:13.66 [DEBUG] Completed: `package` goal
17:17:13.66 [DEBUG] Canceled: Finding the `docker` binary and related tooling
17:17:13.66 [DEBUG] computed 1 nodes in 0.088987 seconds. there are 809 total nodes.
17:17:13.66 [ERROR] 1 Exception encountered:

Engine traceback:
  in `package` goal

InvalidTargetException: The 'python_aws_lambda_layer' target app/utc/actions/convert_to_csv/csvkit_layer:csvkit_layer cannot have its runtime platform inferred, because inference requires simple interpreter constraints covering exactly one minor release of Python, and all its patch version. The constraints for this target (CPython==3.11.* OR CPython==3.9.*) aren't understood.

To fix, provide one of the following:

- a value for the `runtime` field, or

- a value for the `complete_platforms` field, or

- simple and narrow interpreter constraints (for example, `==3.10.*` or `>=3.10,<3.11` are simple enough to imply Python 3.10)
I did also run with 
--no-local-cache
to ensure I'm not getting tripped up by stale config
b
Given your edit, it sounds like you’ve solved this… but just in case you haven’t, it looks like that error is for a different target.
f
Yup, I did track down a different target.
👍 1
I did notice something odd in the lockfile though, it says
Copy code
//   "valid_for_interpreter_constraints": [
//     "CPython==3.11.*",
//     "CPython==3.9.*"
//   ],
even though all the targets for the resolve point to one specific interpreter. is it just that the lockfile tries to solve for the broadest case?
b
Creating a lockfile for a resolve only looks at the 
pants.toml
configuration and the 
python_requirement
targets, it doesn't check the sources that are part of it. https://www.pantsbuild.org/2.21/docs/python/overview/lockfiles#interpreter-constraints has a little bit more details about this, and also reminds me about the https://www.pantsbuild.org/2.21/reference/subsystems/python#resolves_to_interpreter_constraints option for customising the ICs for a particular resolve
👀 1
f
oh that actually looks fantastic and exactly what I need
Now upon running 
pants generate-lockfiles
then 
pants package app/utc/actions/convert_to_csv::
I'm getting
Copy code
InvalidLockfileError: You are consuming `aws-lambda-powertools`, `csvkit`, and 2 other requirements from the `csvkit` lockfile at 3rdparty/python/csvkit.lock with incompatible inputs.



- The inputs use interpreter constraints (`CPython==3.11.* OR CPython==3.12.* OR CPython==3.9.*`) that are not a subset of those used to generate the lockfile (`CPython==3.12.*`).

- The input interpreter constraints are specified by your code, using the `[python].interpreter_constraints` option and the `interpreter_constraints` target field.

- To create a lockfile with new interpreter constraints, update the option `[python].resolves_to_interpreter_constraints`, and then generate the lockfile (see below).

See <https://www.pantsbuild.org/2.21/docs/python/overview/interpreter-compatibility> for details.

To regenerate your lockfile, run `pants generate-lockfiles --resolve=csvkit`.

See <https://www.pantsbuild.org/2.21/docs/python/overview/third-party-dependencies> for details.
I have in my 
pants.toml
Copy code
[python]
interpreter_constraints = ["CPython==3.9.*", "CPython==3.11.*", "CPython==3.12.*"]
enable_resolves = true

[python.resolves_to_interpreter_constraints]
python-default = ["CPython==3.9.*"]
csvkit = ["CPython==3.12.*"]

[python.resolves]
python-default = "3rdparty/python/default.lock"
csvkit = "3rdparty/python/csvkit.lock"
in my recently generated csvkit.lock
Copy code
// --- BEGIN PANTS LOCKFILE METADATA: DO NOT EDIT OR REMOVE ---
// {
//   "version": 3,
//   "valid_for_interpreter_constraints": [
//     "CPython==3.12.*"
//   ],
ok I got it to build by doing
Copy code
[python]
interpreter_constraints = ["CPython==3.9.*", "CPython==3.11.*", "CPython==3.12.*"]
enable_resolves = true

[python.resolves_to_interpreter_constraints]
python-default = ["CPython==3.9.*"]
csvkit = ["CPython==3.9.*", "CPython==3.11.*", "CPython==3.12.*"]
and then setting per-target interpreter_constraints. But a) I don't understand why I can't have narrower constraints on resolves_to_interpreter_constraints b) this still breaks for my python-default for the rest of the stack, which right now I pin to specifically python3.9
seems theoretically possible to have narrower resolves_to than python.interpreter_constraints, e.g. here https://github.com/StackStorm/st2/blob/3e8426ee9febcd4c6c7775f6a2658c7df90f66fb/pants.toml#L140
ah HA! I did some debugging with 
pants py-constraints
which has been super helpful. I think I am leaking constraints in certain build files, which is screwing it up (the error messages currently aren't very good at indicating which specific target is mucking it up)
YES! I got it. I needed to make sure I had all my dependencies pointed to the right place
Copy code
python_sources(
    name="csvkit_layer0",
    resolve="csvkit",
    interpreter_constraints=["CPython==3.12.*"]
)


python_aws_lambda_layer(
    name="csvkit_layer",
    resolve="csvkit",
    dependencies=[
        "app/utc/actions/convert_to_csv:reqs_convert_to_csv",
        ":csvkit_layer0", # this was crucial 
    ],
    tags=['layer'],
    include_sources=False,
    runtime="python3.12",
    output_path="app/utc/actions/convert_to_csv/csvkit_layer/dist/csvkit_layer.zip",
)
unfortunately py-constraints wasn't telling the whole story
b
Glad you got it working!
4 Views
Open in Slack
PreviousNext
Powered by