Hello, we’re integrating black with pants in a django project and we want all the migrations file to be skipped from black… we already have in our

pyproject.toml

the following:

exclude = '''
/(
  | \.git
  | \.mypy_cache
  | \.venv
  | migrations
  | protos
)/
'''

but running

./pants lint ::

it does not skip them (while running

black .

it does). We also tried writing the following in the root BUILD file:

python_sources(
    name="migrations",
    skip_black=True,
    sources=["**/migrations/*.py"]
)

but it didn’t work. We know we can put the

skip_black=True

in each of the

BUILD

files inside all

migrations

folders but this would require us to add it every time we have a new

migrations

folder, which is a bit of a pain. Can you help us?

Hi, do you know of a way to have one lockfile for the whole repository, including the tools? It so happens that we import some of the tools from our code, which means we have to add them to our requirements anyway, so I wanted to ensure that their versions (and if possible also those of their transitive dependencies) match between the code test and execution environment. I set up

pants.toml

so that e.g.

pytest

uses the

python-default.lock

file and I get an error because the lockfile includes more packages, although there are no conflicts. Setting

--python-invalid-lockfile-behavior=ignore

does work, but it would miss any conflicts or missing packages. Any good/elegant ideas getting around this? PS. Here is the error:

16:21:43.92 [ERROR] 1 Exception encountered:

  InvalidLockfileError: You are using the lockfile at pants.lock to install the tool `pytest`, but it is not compatible with your configuration:

- You have set different requirements than those used to generate the lockfile. You can fix this by updating `[pytest].version` and/or `[pytest].extra_requirements`, or by using a new custom lockfile.
In the lockfile, but not in the input requirements: [...lots of unrelated dependencies...]

To regenerate your lockfile based on your current configuration, run `./pants generate-lockfiles --resolve=pytest`.

Hi, what would be the way to debug this failing stage in multi stage docker build with PEX when running?

ERROR [deps 4/4] RUN PEX_TOOLS=1 /usr/local/bin/python3.9 ./infra_plan_batch_job.pex venv     --scope=deps --compile ./infra_plan_batch_job
------
 > [deps 4/4] RUN PEX_TOOLS=1 /usr/local/bin/python3.9 ./infra_plan_batch_job.pex venv     --scope=deps --compile ./infra_plan_batch_job:
#19 33.66 received exit code 1 during execution of `['/usr/local/bin/python3.9', '-s', '-E', '-m', 'compileall', './infra_plan_batch_job']` while trying to execute `['/usr/local/bin/python3.9', '-s', '-E', '-m', 'compileall', './infra_plan_batch_job']`
------
executor failed running [/bin/sh -c PEX_TOOLS=1 /usr/local/bin/python3.9 ./infra_plan_batch_job.pex venv     --scope=deps --compile ./infra_plan_batch_job]: exit code: 1

It’s inspired from this https://pex.readthedocs.io/en/latest/recipes.html#pex-app-in-a-container

Hi folks, pants users at my company are often hitting problems with installing 3rd party libraries (example errors below). However, they can actually run

pip install

to install the problematic libraries with no issue. Any suggestions?

ERROR: Could not find a version that satisfies the requirement omniduct==1.1.17
ERROR: No matching distribution found for omniduct==1.1.17

pex.environment.ResolveError: Failed to resolve requirements from PEX environment @ /Users/user.name/.pex/unzipped_pexes/402fc598cf404ed654017be0d7c53443d79fa859.
Needed macosx_13_0_x86_64-cp-37-cp37m compatible dependencies for:
 1: importlib-metadata>=0.12; python_version < "3.8"
    Required by:
      FingerprintedDistribution(distribution=virtualenv 20.4.7 (/Users/kennethdai/.pex/installed_wheels/bb8d0ead30325881e415ac3e7cbc9080574690a5/virtualenv-20.4.7-py2.py3-none-any.whl), fingerprint='eeaf01de2c4bc1e22cee1a14fd3358ca9251c0c2')
    But this pex had no 'importlib-metadata' distributions.

Hi, sorry for the trouble. Which version of Pants is this, and are you using lockfiles?

pants_version = “2.13.0” requirement_constraints = “constraints.txt” resolve_all_constraints = false

I’m not sure whether means we are using lockfile or not

@curved-manchester-66006 Hello, yes I use direnv.

I would guess this is rooted in python interpreter constraints.. What constraints are you using?

interpreter_constraints = [“>=3.7,<3.8”]

The problem is most users have no problem, but there are five or six of them blocked at the above error, and cannot use pants at all.

so the setting should work for most people, but some snowflakes have issues

huh.. ok. guess that rules out my hunch 🙂

Check out my conversation on the Software at Scale podcast about Pants and Toolchain and stuff: https://www.softwareatscale.dev/p/software-at-scale-52-building-build

thanks anyway @curved-television-6568

@happy-kitchen-89482 the two issues may not stem from the same root cause, but

No matching distribution found for omniduct==1.1.17

is a more common problem and blocking multiple users. Could you please help?

Big tip of the hat to @curved-television-6568 for finally closing out our most requested open issue: https://github.com/pantsbuild/pants/issues/13393 !!

That will be hard to debug without knowing what is different about those users’ system

any thing we can try on our side? If

pip install omniduct==1.1.17

works but pants complain

No matching distribution found for omniduct==1.1.17

, is there a way for us to find out the difference between the two? I guess pants invokes pip under the hood, so understanding how pants exactly uses pip could be a starting point?

Hello, is there a way to graph the dependencies? Google seems to say yes for v1 but I'm not able to find that on v2.

I notice that omniduct is only distributed as an sdist, so maybe there is some issue with its build, that is sensitive to the local environment (pants runs everything in a sandbox, so maybe that is preventing things from working somehow)

Try running with

--no-process-cleanup

, that will keep the sandbox directories, so you can cd into them and see more details

Pants is running pex, which runs pip 20.3.4 under the hood, so try with that version of pip, to be sure it’s apples-to-apples

Does it take a long time before it fails?

Hi! I've seen some references to an upcoming ansible plugin, and I found @wide-midnight-78598's experimental implementation which has not seen commits for a while now. Is there a plan to finish the job, or is it basically done?

It appears that

./pants dependencies --filter-target-type=<target_type> <some_target>

applies the filter to

some_target

, not the list of dependencies, which I find very unintuitive. For example, in directory

db/sample_project_1

I have a

BUILD

that looks like this:

python_sources(
    name="lib",
    sources=["*.py"],
    dependencies=["3rdparty:dbr-9.1-reqs", "db/shared:lib"],
    resolve="databricks",
    tags=["model"]
)

When I call

./pants dependencies db/sample_project_1/__init__.py:lib

I get the following:

3rdparty/requirements-dbr-9.1.txt:dbr-9.1-reqs
3rdparty:dbr-9.1-reqs#boto3
3rdparty:dbr-9.1-reqs#pandas
3rdparty:dbr-9.1-reqs#scikit-learn
3rdparty:dbr-9.1-reqs#scipy
db/shared/__init__.py:lib
db/shared/config.py:lib
db/shared/runner.py:lib
db/shared:reqs#request
db/notebook_utils/__init__.py:lib

which is what I expect. If I do

./pants dependencies db/sample_project_1/__init__.py:lib --filter-target-type="python_source"

, then I get the exact same output. If I do

./pants dependencies db/sample_project_1/__init__.py:lib --filter-target-type="python_requirement"

, then I get nothing. It appears that this is because

db/sample_project_1/__init__.py:lib

itself is a

python_source

I can see how this behavior might be expected if the goal is to filter targets from a glob (e.g. only get dependencies for

python_source

in

::

), but it seems rather unintuitive to me. The question I want to answer here is: “which of this files dependencies are 3rd party requirements vs other sources” and it looks like the best way to do that is something like:

./pants list --filter-target-type=python_requirement $(./pants dependencies db/sample_project_1/__init__.py:lib | tr '\n' ' ')

On a related note, it would be really nice if it was possible to use

--output-file

in append-mode.

Yes, it takes a while to build a bunch of stuff until fails. But another user just hit similar error that immediately fail:

no matching distribution found for pantsbuild.pants=2.13.0

Pants is running pex, which runs pip 20.3.4 under the hood

is that codified somewhere?

We really need a lot more information here

run with

-ldebug

and post as much relevant debug info as possible