Hello, I have been trying to have multiple lockfiles to have one with and one without Pytorch. The context is, let's say I have 2 projects and 3 libs like this:

|lib-
|    |
|    |-libA # no pytorch dep
|    |-libB # has pytorch dep
|    |-libC # no pytorch dep
|
|projects
     |-project1 # uses libA and libB
     |-project2 # uses libA and libC

I have created 2 resolves on the requirements:

resolve-pytorch

and

resolve-default

• Since project1 uses libB that uses Pytorch, I put resolve=resolve-pytorch on the

python_source

and

pex_binary

in project1's BUILD file • Since project2 doesn't use libB that uses Pytorch, I put resolve=resolve-default on the

python_source

and

pex_binary

in project2's BUILD file The problem is that libA which is dependent by Project1 and project2 will have 2 different resolves so I put

resolve=parametrize(resolve-pytorch, resolve-default)

on libA's BUILD file. It works when I build the PEX files for both projects with

pants package

, but it fails on the Mypy check.

>pants check projects/project1
---omitted----
17:53:13.38 [ERROR] Completed: Typecheck using MyPy - mypy - mypy failed (exit code 1).
Partition #1 - resolve-default, ['CPython==3.10.*']:
Found 6 errors

Partition #2 - resolve-torch, ['CPython==3.10.*']:
Success: no issues found in 2 source files

mypy failed

Am I approaching this wrong? Is there other ways to achieve the pex separation? I've been looking at this Optimizing Python Docker page as well but probably I will need to see in depth about PEX_TOOLS The team that uses project2 complains that the docker image is large because it contains pytorch even though they don't use it.

This seems like a bug somewhere in your setup; if you build a pex-file Pants only includes the direct + transitive dependencies actually used. So if a pex ends up with torch it means that it uses torch, or imports another package that depends on torch. Can you reduce and share an example?

@gorgeous-winter-99296 Ah it seems like I stepped too far. Initially, I put everything into one requirement file. I thought that I needed to have multiple lockfiles to solve it. Then based on what you said, for my problem, I just need to separate anything Pytorch-related into different files and

python_requirements

Thank you very much for your answer!

@limited-balloon-79828 you shouldn’t need to split requirements. Pex is smart enough to only pull in deps that are actually referenced.

@gentle-flower-25372 I'm not sure. Without separating it, when I run

pants package projects/project2::

, it builds everything from that requirement file and the pex file for everything is huge.

If you split out sources from deps, check this out: https://www.pantsbuild.org/blog/2022/08/02/optimizing-python-docker-deploys-using-pants

I tried to run

pants paths --from=projects/project2/entrypoint.py --to=//third-party:requirements#torch

Without separation, it shows something like this

[
  [
    "projects/project2/entrypoint.py:../project2",
    "third_party:requirements#torch"
  ]
]

While with project1 it shows:

[
  [
    "projects/project1/entrypoint.py:../project1",
    "third_party:requirements#torch"
  ],
  [
    "projects/project1/entrypoint.py:../project1",
    "lib/libB/torch_endpoint.py:../libB",
    "third_party:requirements#torch"
  ],
]

With separation, project 2 only prints empty list when running the similar command

pants paths --from=projects/project2/entrypoint.py --to=//third-party:torch-requirements#torch

While project 1 shows nice dependency paths

[
  [
    "projects/project1/entrypoint.py:../project1",
    "lib/libB/torch_endpoint.py:../libB",
    "third_party:torch-requirements#torch"
  ],
]

Notice that the first list from project 1 has disappeared. I'm not sure if it's a bug or anything. I'm on Pants 2.17

Hi @limited-balloon-79828 I'm facing a similar issue. I wasn't able to build the PEX file. Could you please let me know how you solved it?https://pantsbuild.slack.com/archives/C046T6T9U/p1724908692110019