hi all I m having trouble with multiple versions of a depend Pants #general

hi all - I'm having trouble with multiple versions...

quiet-evening-25363

04/22/2022, 8:27 PM

hi all - I'm having trouble with multiple versions of a dependency that I could use some help with. I would like to generate 2 lockfiles will all of the same dependencies except for a single package (one to include psycopg2, one to include psycopg2-binary). Following the docs I'm able to generate 2 lockfiles, but the problem is that the non-default generated 3rdparty/python/beam.lock only includes pyscopg2-binary, but I would like it to include all requirements in requirements.txt and add psycopg2-binary. The default 3rdparty/python/default.lock file is correct. Is that possible or do I need to duplicate my requirements.txt file with the 2 versions of pyscopg2? I'll include my BUILD files in the thread. Thanks!

✅ 1

quiet-evening-25363

04/22/2022, 8:28 PM

I have a single 3rdparty/python/requirements.txt file that does not include psycopg2/psycopg2-binary. pants.toml

Copy code

[python]
interpreter_constraints = ["CPython==3.9.*"]
enable_resolves = true
default_resolve = "default"

[python.resolves]
default = "3rdparty/python/default.lock"
beam = "3rdparty/python/beam.lock"

3rdparty/python/BUILD:

Copy code

python_requirement(
    name="psycopg2",
    requirements=["psycopg2==2.9.3"],
    resolve="default",
)

python_requirement(
    name="psycopg2-binary",
    requirements=["psycopg2-binary==2.9.3"],
    resolve="beam",
)

src/where/I'm/using/the/beam/lockfile/BUILD

Copy code

python_sources(
    dependencies=[
        "3rdparty/python#google-cloud-storage",
    ],
    resolve="beam",
)

pex_binary(
    name="search_index_main",
    entry_point="main.py",
    dependencies=[
        "3rdparty/python#google-cloud-storage",
    ],
    resolve="beam",
)

hundreds-father-404

04/22/2022, 9:36 PM

Hello! Which pants version are you using? The answer changes a little bit

quiet-evening-25363

04/22/2022, 9:49 PM

just upgraded to 2.10.0 to try this out

👍 1

hundreds-father-404

04/22/2022, 10:11 PM

In 2.10, you have to create a distinct target for each resolve

Copy code

# normal one you had
python_requirements()

python_requirements(
  name="beam",
  resolve="beam"
)

In 2.11, you can still do that, or you can use the new parametrize mechanism https://www.pantsbuild.org/v2.11/docs/targets#parametrizing-targets

Copy code

python_requirements(resolve=parametrize("beam", "default-resolve"))

hundreds-father-404

04/22/2022, 10:14 PM

Warning that you will need to not only update the Python requirement targets, but also any transitive dependency of

src/where/I'm/using/the/beam/lockfile:search_index_main

, e.g.

python_sources

targets. If that code should work with both of your resolves, then you either need to use

parametrize

or manually create 2 targets

hundreds-father-404

04/22/2022, 10:16 PM

2.11

Latest release is 2.11.0rc3. We are hoping to stabilize next week If you are upgrading, you will see a deprecation message about the lock file generator. I recommend sticking w/ poetry in the short term while you play w/ multiple lockfiles to minimize changes

quiet-evening-25363

04/22/2022, 10:22 PM

ok great! adding the 2nd python_requirements target was the missing piece. I'll take a look at upgrading all the way to 2.11 and the lock file generator deprecation. thanks for the help and info!

❤️ 1

hundreds-father-404

04/22/2022, 10:25 PM

Feedback is definitely appreciated on this new feature! For example, we are aware that the error messages need to be better when a dependency is not compatible: https://github.com/pantsbuild/pants/issues/14864. I suspect that there are other things we need to make more clear too Maybe we need more quality examples? This is a fairly unique approach to multiple lockfiles

👍 1

5 Views

Open in Slack

Previous Next