I’ve been trying to set up type checking in my monorepo. Found some strange mismatched behaviour between two similar roots of code. Both are FastAPIs with their own unique

requirements.txt

files each included in the

BUILD

file with the

python_requirements()

code. One is called

api

and the other is called

qa

. Here are their structures:

src/api/
├── BUILD
├── Dockerfile
├── app
│   ├── __init__.py
│   └── api.py
├── main.py
└── requirements.txt

and:

src/processors/qa/
├── BUILD
├── Dockerfile
├── app
│   ├── __init__.py
│   ├── api.py
│   └── qa.py
├── main.py
└── requirements.txt

The

qa

one loads all of its requirements from pip great in both mypy runs and repl, but the

api

one completely misses installing them. Both have the same

BUILD

file with just the name swapped out:

python_requirements()

python_library(
    name="<name here>",
    sources=["**/*.py"],
    interpreter_constraints=[">=3.6"],
)

Anyone have any ideas why one loads packages and the other doesn’t? Or a handy command I can use to help debug?

is there a way to turn off the v2 fingerprinting and dependency resolver for all the BUILD files in a repo? We have a rather large monorepo that has 500+ BUILD files. Everytime we run a pants 2.0 goal it takes 15-20 seconds of processing before it begins executing the actual goal. We are looking for speed optimizations at this point. We have just migrated to v2 and things seem much slower than v1 at this point. My deps and constraints are all cached at this point so we are spending 0 time resolving them…its all pre-processing all the build files it seems.

hi there, sorry to keep posting problems! We really appreciate all the work that you’ve put into this project but here goes: We just ran into a

ERROR: Double requirement given:

message when trying to run a recursive test of a top-level project dir: For example: pants test foo:: Imagine the structure of foo as:

foo/
foo//tests
foo//tests/python
foo//tests/python/baz
foo//tests/python/baz/test_baz.py
foo//tests/python/baz/BUILD
foo//tests/python/bar
foo//tests/python/bar/BUILD
foo//tests/python/bar/test_bar.py
foo//src
foo//src/python
foo//src/python/baz
foo//src/python/baz/__init__.py
foo//src/python/baz/BUILD
foo//src/python/bar
foo//src/python/bar/__init__.py
foo//src/python/bar/BUILD

Now in this case, the bar module imports one dependency, say boto3==1.0 and the baz module imports a newer boto, say boto3==2.0. In our example, imagine that the baz module is a complete refactor-in-progress that doesn’t share any imports with bar. It seems that the recursive test tries to resolve all dependencies at once, and thus sees those duplicate boto3 deps and bails. In our real case it’s a py3 upgrade of a project that is using a newer numpy but I feel like the specifics aren’t as important. I realize we could just simply move baz out of the foo top-level. But it seems like pants should be resolving these dependencies separately? Is there a way to force that? I know in past versions of pants it was possible to specify

--no-test-pytest-fast

which would resolve each target’s dependencies separately. Is there something similar for pants v2?

Error message idea: thoughts on Pants starting to analyze subprocess failures and give some suggestions? For example: * If Pytest gives “ModuleNotFoundError”, append a suggestion to run

./pants dependencies path/to/file.py

to ensure everything shows up. * If Pex interpreter constraints are not satisifiable, suggest running

py-constraints

. * “DoubleRequirementError”: give the advice from https://www.pantsbuild.org/docs/troubleshooting#double-requirement-given-error-when-resolving-requirements We’d do something like

if "ModuleNotFoundError" in result.stderr

, and append the message below the process’s output Thoughts? Posted in #documentation

Has anybody been able to get this coursier-s3 plugin to work on any 1.x release? The docs arent very helpful and nobody had an answer in the official coursier gitter. Im upgrading from 1.16 -> 1.27 and was previously doing resolve via s3-ivy-resolver

When running

./pants test ::

is there anyway to ensure that a given test gets run first? I have a very slow tests that I would like to ensure is in the first batch. I think it could speed up my test sweet since it runs long after every other test.

I just tried to upgrade from 2.1.0rc1 to 2.2.0.dev1 and got:

ERROR: Could not find a version that satisfies the requirement pantsbuild.pants==2.2.0dev1
ERROR: No matching distribution found for pantsbuild.pants==2.2.0dev1
./pants: line 261: /home/zdenal/.cache/pants/setup/bootstrap-Linux-x86_64/2.2.0dev1_py36/bin/python: No such file or directory

Any idea what should I do in order to upgrade? 🙂

Just migrated from Bitbucket+Jenkins to Azure DevOps, when running any ./pants command (lint, typecheck, test) I get lots of "Filesystem changed during run: retrying ....". This took my build time from around 3 minutes to around 1 hour. Is there a way to display what files changed? 

-ldebug

 would do that?

I used to have

[test]
use_coverage = true

in my pants.toml but after update to Pants 2.2.0.dev1 my integration tests (using runtime_package_dependencies) started failing with

pytest-cov: Failed to setup subprocess coverage. Environ: {'COV_CORE_SOURCE': '.', 'COV_CORE_CONFIG': '/tmp/process-executionN7mCSU/.coveragerc', 'COV_CORE_DATAFILE': '/tmp/process-executionN7mCSU/.coverage'} Exception: CoverageException("Unrecognized option '[run] relative_files=' in config file /tmp/process-executionN7mCSU/.coveragerc")
pytest-cov: Failed to setup subprocess coverage. Environ: {'COV_CORE_SOURCE': '.', 'COV_CORE_CONFIG': '/tmp/process-executionN7mCSU/.coveragerc', 'COV_CORE_DATAFILE': '/tmp/process-executionN7mCSU/.coverage'} Exception: CoverageException("Unrecognized option '[run] relative_files=' in config file /tmp/process-executionN7mCSU/.coveragerc")
Traceback (most recent call last):
  File "/tmp/process-executionN7mCSU/src.python.arcor2_mocks.scripts/mock_scene.pex/.bootstrap/pex/pex.py", line 444, in execute
  File "/tmp/process-executionN7mCSU/src.python.arcor2_mocks.scripts/mock_scene.pex/.bootstrap/pex/pex.py", line 103, in _activate
  File "/tmp/process-executionN7mCSU/src.python.arcor2_mocks.scripts/mock_scene.pex/.bootstrap/pex/environment.py", line 260, in activate
  File "/tmp/process-executionN7mCSU/src.python.arcor2_mocks.scripts/mock_scene.pex/.bootstrap/pex/environment.py", line 425, in _activate
  File "/tmp/process-executionN7mCSU/src.python.arcor2_mocks.scripts/mock_scene.pex/.bootstrap/pex/environment.py", line 160, in _update_module_paths
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named '__mp_main__'

When I remove

use_coverage

from pants.toml, it works ok again. Any idea?

Does Pants make any guarantees about runtime directory relative to the source tree? Particularly in regards to relative path between some

python_tests

and

resources

?

Im trying to upgrade to pants 1.27, and i am having some weird jackson-annotations conflict for my scala code. Something is pulling in an extra (newer) version of jackson. when i do a depmap on my code, i don’t see it. but when i do a resolve, i do see it. could that be coming from pants tooling?

That is weird!

I'd be surprised if any basic tools (zinc/rsc/ivy/coursier/scalafix etc) depended on jackson, but maybe?

But also, IIRC tools are resolved separately from your code

so they shouldn't collide anyway

Wait - in 1.X depmap is only a listing of dependencies Pants knows about from local information - so it stops at all your direct 3rdparty dependencies. If any of your direct 3rdparty dependencies have transitive requirements on jackson - voila.

depmap doesn't run a resolve?

that makes sense

I have only vague recollection at this point

So there it is.

There is a command to get this though. I'm building 1.27.0 to remember what that command is...

@polite-vase-75369 are you using ivy or coursier for your resolves? I believe they each have a way of dumping the full transitive resolve into HTML so you can see what is pulled in by what. I think we surfaced a way to get this directly in Pants from ivy at least, not sure about Coursier.

For ivy it popped up an ivy report html page that you could use to see who requires what transitively.

@polite-vase-75369 - like so:

./pants resolve.coursier --report 3rdparty:shapeless
...
10:28:07 00:04   [resolve]
10:28:07 00:04     [coursier]
10:28:07 00:04       [coursier]
                     /home/jsirois/.cache/pants/coursier/https/maven-central.storage-download.googleapis.com/maven2/org/scala-lang/scala-library/2.12.0/scala-library-2.12.0.jar
                     /home/jsirois/.cache/pants/coursier/https/maven-central.storage-download.googleapis.com/maven2/com/chuusai/shapeless_2.12/2.3.2/shapeless_2.12-2.3.2.jar
                     /home/jsirois/.cache/pants/coursier/https/maven-central.storage-download.googleapis.com/maven2/org/typelevel/macro-compat_2.12/1.1.1/macro-compat_2.12-1.1.1.jar
                       Result:
                     └─ com.chuusai:shapeless_2.12:2.3.2
                        ├─ org.scala-lang:scala-library:2.12.0
                        └─ org.typelevel:macro-compat_2.12:1.1.1
                           └─ org.scala-lang:scala-library:2.12.0
...

And the target (3rdparty:shapeless in the example above) need not be directly 3rdparty.

let me give it a try. im using coursier

hmmm when i run report, i dont get that nice hierarchy, i just get all the outputs of /home/…etc

i see

/Users/lawrencefinn/.ivy2/pants/https/maven-central.storage-download.googleapis.com/maven2/com/fasterxml/jackson/core/jackson-annotations/2.6.7/jackson-annotations-2.6.7.jar
                     /Users/lawrencefinn/.ivy2/pants/https/repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.10.0/jackson-annotations-2.10.0.jar

im also using managed deps where i pin 2.6.7

unless somehow i broke that