Hi I've added

celery[sqs]

and

celery[redis]

package in poetry (

pyproject.toml

) config. I can able to generate

python_default.lock

file. However I'm not sure how to add this extras as dependancy in BUILD file. Can someone please guide me how to configure this?

# pyproject.toml
[tool.poetry.group.pkg-celery_worker.dependencies]
celery = {extras = ["redis", "sqs"], version = "^5.3.6"}

# BUILD file
python_sources(
    name="pkg",
    sources=["celery/**/*.py"],
    dependencies=[
        "3rdparty/python:poetry#boto3",
        "3rdparty/python:poetry#celery",
        "3rdparty/python:poetry#redis",
    ],
)

I think all references to

3rdparty/python:poetry#celery

will activate those extras (i.e. include the dependencies implied); are you observing something different?

Yes. It seems it's not added by default. I'm getting this error

raise ImportError('The curl client requires the pycurl library.')
ImportError: The curl client requires the pycurl library.

Ah I know nothing about celery, but just confirming that error is related to the Sqs or redis extras, because it’s not obviously? Either way, maybe the poetry requirements target doesn’t import extras? I don’t know… If you can create a reduced example (eg one dependency, one extra and one source file) you can share, that’ll help narrow it down.

Sure. I have one more question. If I add a dependency with markers like this in

pyproject.toml

, will it be installed only if the platform is Linux and skipped for macOS?

triton = [
    { version = "2.1.0", markers = 'sys_platform == "linux"' },
]

If I add a dependency with markers like this in

pyproject.toml

, will it be installed only if the platform is Linux and skipped for macOS?

that’s supposed to work, yes.

Thanks for your reply. The Triton package is only available on the Linux platform. My development environment is MacOS, and I'm using an Ubuntu machine to deploy the service. The

pants generate-lockfiles --resolve=python-default

command adds Triton to the

python_default.lock

file when executed on the Ubuntu machine but not when executed on MacOS. I attempted to use the same lock file on both operating systems. However, when I run the

pants check lint

command from the CI (Ubuntu machine), it throws an error because the package was not added, given that the lock file was generated on MacOS. Should I keep two different lock files based on the operating system? How can I resolve this issue?

I don’t reproduce this. I get the exact same lockfile on both Mac OS and ubuntu (on WSL):

❯ shasum 3rdparty/python/default.lock
01f6f30a3da04983372e2e41f13126422c3eca29  3rdparty/python/default.lock

Using this setup:

# pants.toml
[GLOBAL]
pants_version = "2.18.1"
backend_packages.add = [
  "pants.backend.python",
]

[python]
enable_resolves=true
interpreter_constraints = ["==3.11.*"]

# pyproject.toml
[tool.poetry.dependencies]
triton = [
    { version = "2.1.0", markers = 'sys_platform == "linux"' },
]

# BUILD
poetry_requirements(name="lib")

And that is with the triton package in the list of known artifacts, and the platform marker preserved in the requirements..

Are you running an Ubuntu container or a VM? If you're using an Ubuntu container on MacOS, it will still generate the same lock file .But if you are running on Ubuntu EC2 machine it will generate a different lock file. I'm not sure how it will identify the system architecture. But it is generating different lock files on different system architecture.

the ubuntu one was from a different machine (windows box, using the subsystem for linux on it)

Oh okay. Could you please find this line in your lock file

"triton<3,>=2.0.0"

yea, was going to ask which project does that belong to?

this package is required by

openai-whisper

. I'm importing

openai-whisper

package but internally this package is depending on triton package.

so since that depends on triton, it too would only work on linux...

my guess is that you need that platform marker for that project as well, perhaps

Oh is it possible. I'm new to pants ecosystem. Could you please explain how I can configure that?

the same way you did for triton already..

so since that depends on triton, it too would only work on linux...

On MacOS, It's working without this triton package. I don't know how. When I try to install it on Ubuntu then it's throwing

triton

package not available.

I don't have the full picture here... but what my reduced example shows, is that it does not matter that a lib is linux only, it still produces a good lockfile for linux on Mac. So your issue is something else.

Can you please run

pants check

command on both the systems? If you don't mind can you please share the sample lock file?

@white-twilight-61019 running

pants check

won’t do anything for me, as I don’t have any sources. It’ll be you that need to try things out, either start with my example and see how it behaves and go from there, or work with your repo trying to get a small enough reproduction you can share will make it easier to help you.

Sure, I'll try your method. If I still get same error, I will share reproducible code with you