pantsbuild #general

Channels

welcome

github-notifications

announce

flat-zoo-31952

11/26/2022, 10:17 PM

Is there a way to get files into a test execution directory without the test target depending on those files targets? Is there a way to say something like "just include these targets in with this target for this test run?" Idk if what I'm asking makes senes

rich-kite-32423

11/27/2022, 1:30 AM

I've spent the last few days figuring out how to get Pants running on Windows Subsystem for Linux (WSL). There were actually plenty of hurdles. Here are my instructions: https://github.com/BruceEckel/RethinkingObjects#readme

🙏 1

👍 1

rich-kite-32423

11/27/2022, 11:48 PM

Is this the right place to report documentation problems? For example, at https://v1.pantsbuild.org/python_readme.html

Copy code

[python-setup]
interpreter_constraints = ["CPython>=3.5"]

[python-setup]

should be

[python]

broad-processor-92400

11/28/2022, 1:47 AM

We've got quite a few

package

-able Python targets: mostly

python_awslambda

but some

pex_binary

too. We run

./pants package ::

in CI to validate that they're all packageable, but this ends up spending a lot of time bundling up requirements (~10s per target, as a rule of thumb), and thus it'd be nice to make this faster. I've made two observations: 1. changing our code without changing the dependency structure seems to require repackaging the dependencies from scratch (I ran

./pants package ::

before modification locally, made the change, and then

./pants package ::

after, with the same pantsd process) 2. multiple targets that have the same set of requirements seem to build their requirements separately (more details in thread) Is this expected behaviour? Any tips for improving it?

ripe-scooter-10665

11/28/2022, 3:09 PM

I’m trying to set up debugging (for the test and run goals) in VSCode but I’m hitting a problem that google isn’t helping with and I can’t find an issue related to. I’m running pants 2.14 on an M1 mac with Python 3.9. When I try to run

./pants test --debug-adapter <path to test file>

I get the following error:

Copy code

15:06:23.64 [INFO] Completed: Building pytest_runner.pex
15:06:23.64 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Building pytest_runner.pex' failed with exit code 1.
stdout:

stderr:
A distribution for debugpy could not be resolved for ~/.pyenv/versions/3.9.15/bin/python3.9.
Found 1 distribution for debugpy that do not apply:
1.) The wheel tags for debugpy 1.6.0 are cp38-cp38-macosx_10_15_x86_64 which do not match the supported tags of ~/.pyenv/versions/3.9.15/bin/python3.9:
cp39-cp39-macosx_13_0_arm64
... 409 more ...

It seems that debugpy is a requirement of pants itself, pinned at 1.6.0 but there does seem to be py3.9 wheels available for it. I tried adding debugpy to my own 3rd party requirements but that didn’t seem to help. Any ideas on how to get around this. Not being able to debug in VSCode is a bit of a pain.

👀 1

white-pager-32496

11/28/2022, 3:46 PM

Hey all, I wanted to check my understanding after doing some initial due diligence. Given a monorepo, is there a way to model Pants such that the source code in each respective module is only visible to the others via explicit declaration, e.g via BUILD files? Dependency inference seems to suggest that Pants’ philosophy is “see-all” by default, rather than “see-nothing”. Where the monorepo isn’t so much a set of distinct encapsulated modules, but rather a monolithic codebase, which can be carved up into smaller units as desired. My experience is from the JVM/Bazel ecosystem, so I come with baggage :) And what I’m suggesting above isn’t necessarily a better way to do things, it’s just what I’m comfortable with. My main motivation behind wanting this behaviour is that I’m looking to set up rather strict organisational rules around code ownership, dependencies, and modularity. Happy to elaborate, but this is already getting wordy. Thanks!

rich-kite-32423

11/29/2022, 5:39 AM

In a little test setup, I have:

Copy code

[python]
interpreter_constraints = ["==3.9.*"]

Which works successfully. But when I bump it to 3.10.*, I get the following:

Copy code

bruce@groot:~/tmp/semaphore-demo-python-pants$ ./pants test ::
22:34:39.42 [INFO] Initializing scheduler...
22:34:40.15 [INFO] Scheduler initialized.
22:34:43.19 [INFO] Canceled: Building pytest.pex from pytest_default.lock
22:34:44.01 [INFO] Completed: Building requirements.pex
22:34:44.08 [INFO] Completed: Building requirements.pex with 1 requirement: ansicolors>=1.1.8
22:34:44.08 [ERROR] 1 Exception encountered:

Engine traceback:
  in `test` goal
  in Run Pytest - hello_world/test_main.py:tests

ProcessExecutionFailure: Process 'Building requirements.pex with 1 requirement: ansicolors>=1.1.8' failed with exit code 1.
stdout:

stderr:
[Errno 2] No such file or directory: '/tmp/pants-sandbox-zOYk5D/.tmp/pex-pip-log.lhpmsm3l/pip.log'

It's not obvious to me why this is happening and what the error messages mean.

witty-house-77262

11/29/2022, 8:21 AM

Hi, it’s probably a FAQ but why does pants need to be

./pants

and not just

pants

? Why can’t it be installed globally (like with

pipx install pants

)` ?

average-flag-94768

11/29/2022, 10:16 AM

I have a project that looks like: |--> Common/Common/foo.py |--> Specific/Specific/bar.py "from common.foo import ..." |--> Specific/Tests/conftest.py "from specific.bar import ..." Source roots are set to common and specific If I package up bar.py as a pex and run it all the imports work fine, however when I try and run the tests the specific "from common.foo import ..." import fails with no module named common.foo. I could use any pointers on what I might need to configure to get it to work as I have run out of ideas.

✅ 1

refined-addition-53644

11/29/2022, 12:23 PM

Any idea why PEX fetches everything including extra for a given package even when we aren’t asking for it? This is inside a docker image where I am trying to run a lambda.

Copy code

[ERROR] ResolveError: Failed to resolve requirements from PEX environment @ /tmp/tmpoa40lyqx/unzipped_pexes/4185e0a7af669dc757e04bdb6559435de125a6db.
Needed cp39-cp39-manylinux_2_26_x86_64 compatible dependencies for:
1: orjson<4.0.0,>=3.2.1; extra == "all"
Required by: fastapi 0.86.0
But this pex had no ProjectName(raw='orjson', normalized='orjson') distributions.

✅ 1

refined-addition-53644

11/29/2022, 3:48 PM

Anyone knows how to use PEX with custom lambda docker image? https://pypi.org/project/awslambdaric/

purple-umbrella-89891

11/29/2022, 4:23 PM

Hi all, we have some optional imports (

with suppress(...): import x

) in our codebase that are local/internal packages, which are not available in any repository. During pants test, we get warnings in the form

Pants cannot infer owners for the following imports in the target a/b/c.py: x

. We have disabled the warnings using

# pants: no-infer-dep

in the code but would like to do it centrally and are trying the

!{bad_address}

syntax as shown here. However, since

is not in the known dependencies, it does not work. Any ideas how to make it work globally instead of annotating every import?

gentle-painting-24549

11/29/2022, 5:17 PM

I’ve got an issue with an undeclared dependency inside Python tests. I’ve got a test with a dependency that uses an underlying C++ Package: GDCM. Before pants we would perform a

conda install gdcm

inside of tox to run our tests. Without having access to Conda an alternative I’ve found is to use

python-gdcm

instead which is a Python wrapper for GDCM. I’ve tried to attach this dependency to the tests using a `python_requirement`:

Copy code

python_requirement(
    name="gdcm",
    requirements=["python-gdcm"],
)

python_tests(
    name="tests",
    sources=[
        "tests/**/test_*.py",
        "tests/test_*.py",
    ],
    dependencies=[
        ":conftest",
        ":test-resources",
        ":gdcm",
    ],
)

However this doesn’t work and

python-gdcm

doesn’t get installed into the underlying testing PEXs unless I add this snippet to one of my test files:

Copy code

try:
    import gdcm # noqa
except ImportError:
    pass

How can I have the tests depend on

python-gdcm

without the attempted import? Or alternatively, how can I give Pants access to the underlying gdcm binary (for example, one that was installed by brew or conda on the host system)?

polite-garden-50641

11/29/2022, 5:21 PM

think what u r probably missing is a

module_mapping

that will let pants know that

python-gdcm

exposes the

gdcm

module. See: https://www.pantsbuild.org/docs/python-third-party-dependencies#use-modules-and-module_mapping-when-the-module-name-is-not-standard

gentle-painting-24549

11/29/2022, 5:22 PM

Oh, I already have the module mapping. I just didn’t include it in the example above.

refined-addition-53644

11/29/2022, 5:25 PM

You will have to add explicit dependency if you aren’t importing it explicitly anywhere inside your code

refined-addition-53644

11/29/2022, 5:25 PM

Copy code

pex_binary(
    name="service",
    entry_point="awslambdaric",
    dependencies=["3rdparty/python:resolve#awslambdaric", ":infra_sources"],
    layout="packed",
    execution_mode="venv",
)

refined-addition-53644

11/29/2022, 5:26 PM

Here I need to tell pex to depend on

awslambdaric

explictly

gentle-painting-24549

11/29/2022, 5:29 PM

I understand the concept of explicitly adding undeclared dependencies - but on my system this doesn’t work without adding the Try/Except ImportError block: Here is what’s returned when the Try/Except block is included:

Copy code

❯ ./pants dependencies packages/example/tests/test_example.py | grep requirements
requirements:reqs#numpy
requirements:reqs#pytest
requirements:reqs#python-gdcm

And now here’s what happens when I comment it out:

Copy code

❯ ./pants dependencies packages/example/tests/test_example.py | grep requirements
requirements:reqs#numpy
requirements:reqs#pytest

Everything else is the same on the BUILD file for those two examples:

Copy code

python_tests(
    name="tests",
    sources=[
        "tests/**/test_*.py",
        "tests/test_*.py",
    ],
    dependencies=[
        ":conftest",
        ":test-resources",
        "requirements:reqs#python-gdcm",
    ],
)

refined-addition-53644

11/29/2022, 5:33 PM

So I think what’s happening is dependencies are attached to the “:tests” target. If you run your test like, it should work.

Copy code

./pants test directory-to-tests:tests

The file’s dependencies are only being shown for the imports you have.

refined-addition-53644

11/29/2022, 5:34 PM

I would imagine then running test just for this file might fail.

refined-addition-53644

11/29/2022, 5:36 PM

If you want explicit at the file level, then you most probably need to define a single

python_test

Copy code

python_test(name="test_example", source="test_example.py", dependencies=["insert_explicit"]

Please verify the correct syntax Then you can attach python_tests to have explicit dependency on this test target but you can’t use the glob then.

gentle-painting-24549

11/29/2022, 5:38 PM

Unfortunately not 😞, the test still fails when calling the file directly, or by calling the

tests

target directly. But they succeed when I uncomment the try/except block.

refined-addition-53644

11/29/2022, 5:43 PM

By the way you should use

dependencies --transitive

👀 1

gentle-painting-24549

11/29/2022, 5:48 PM

Gotcha, I end up getting my additional dependencies from the lib with

--transitive

but the behavior is still the same: even though the

requirements:reqs#python-gdcm

target is explicitly declared and attached to the

tests

target via dependencies - it doesn’t actually get attached as a dependency unless the try/except ImportError block exists.

refined-addition-53644

11/29/2022, 5:50 PM

where do you put your module mapping?

gentle-painting-24549

11/29/2022, 5:54 PM

We’ve got a bit of a different directory structure than most pants repos, but my build file @

packages/example/BUILD

looks like this:

Copy code

python_sources(
    name="lib",
    sources=[
        "example/**/*.py",
        "example/*.py",
    ],
)

python_tests(
    name="tests",
    sources=[
        "tests/**/test_*.py",
        "tests/test_*.py",
    ],
    dependencies=[
        "requirements:reqs#python-gdcm",
    ],
)


python_distribution(
    name="dist",
    dependencies=[
        ":lib",
    ],
    provides=setup_py(
        name="example",
        version="0.0.0",
    ),
    generate_setup=True,
)

And my BUILD file @

requirements/BUILD

looks like this:

Copy code

python_requirements(
    name="reqs",
    source="requirements.txt",
    module_mapping={
        "python-gdcm": ["gdcm"],
    },
)

👀 1

refined-addition-53644

11/29/2022, 6:19 PM

I am not able to reproduce. Although I use lockfiles. https://github.com/ShantanuKumar/pants-multi-poetry/commit/4492b44330c5723dcee5e2fa31605de375e18e1a

gentle-painting-24549

11/29/2022, 6:37 PM

💀 I’m having trouble reproducing on a new, minimal repo myself. Let me keep plugging on this and see if I can get it to reproduce

gentle-painting-24549

11/29/2022, 7:09 PM

🤷 this is really difficult - I haven’t been able to reproduce. For now I’ll call this user-error until I’m able to identify the issue. I’m sorry if that’s the case. Alternatively, is it possible to give access to GDCM from outside of the sandbox? (i.e. from Conda or Brew or some other installer)?