https://pantsbuild.org/ logo
Join Slack
Powered by
Hiya, I have a (test) dependency question.. how/w...
# general
c
Hiya, I have a (test) dependency question.. how/when is recalculation of dependencies triggered? Case:
Copy code
$ ./pants dependencies src/xx/core/data:tests
src/xx/core/data/factory_test.py:tests
src/xx/core/data/manifest.py
src/xx/core/data/manifest_test.py:tests
src/xx/core/data/metadata.py
src/xx/core/data/spec.py
src/xx:deal
expected a few more there, as I’ve added 
import
statements in the factory_test.py file.. and, right enough, after killing 
pantsd
and running again, they show up:
Copy code
$ ./pants dependencies src/xx/core/data:tests
14:45:07.09 [INFO] initializing scheduler...
14:45:07.69 [INFO] scheduler initialized.
src/xx/core/data/factory_test.py:tests
src/xx/core/data/manifest.py
src/xx/core/data/manifest_test.py:tests
src/xx/core/data/metadata.py
src/xx/core/data/spec.py
src/xx:deal
src/xx:hypothesis-jsonschema
src/xx:marshmallow-jsonschema
This is on pants:
Copy code
$ ./pants --version
2.4.0.dev3
w
yikes: that’s definitely not expected!
are you able to capture the 
.pants.d/pants.log
from the time period before/around those runs?
pantsd gets file notification events which invalidate its graph: they usually arrive a few milliseconds after the files have changed
@curved-television-6568: issues like this are a very high priority for us, so eager to debug this if you have time!
c
OK! I’ll be on the look-out for that log file, if I see it again. I think I may have tripped it up, while messing around with imports etc, so I may have ended up with a test that didn’t import any real code (it’s a new project, so very slim and in flux, but also minimal, so shouldn’t be too hard to trim down if I get it to reproduce..)
w
thank you… very sorry for the trouble.
by any chance: do you happen to have any oddly named 
requirements.txt
files?
c
Great, thanks. Wasn’t to big of a trouble. I did place it in a non-root location, but the naming is standard.
👍 1
I’ll play around a bit tomorrow, see if I can get it to repro the issue.
Ok, not exactly the same, but dependency related.. now I got this, from a 3rd party lib when running a test case
Copy code
/Users/x/.cache/pants/named_caches/pex_root/venvs/7226ac3b42a360653de16c64df0654e1f49252be/b0f1efb7774678bcd862110dbf4ecc25d6b24755/lib/python3.8/site-packages/marshmallow_jsonschema/__init__.py:1: in <module>
    from pkg_resources import get_distribution
E   ModuleNotFoundError: No module named 'pkg_resources'
this did not go away by killing pantsd. I’ve added setuptools to the module_mapping of the python_requirements target, but it didn’t seem to help (I guess it should help, but it doesn’t feel like my change to the BUILD file has been picked up..) Also, as my requirements.txt file, and the accompanying BUILD file was not in the root, I moved it up a notch, so it is at least in a source root now… maybe it needs to move all the way up to the root?
Adding setuptools directly as a dependency to the test target worked, however.
h
Please try out https://pantsbuild.slack.com/archives/CK79E5J2Y/p1616484095006300, written yesterday, and do share feedback on those instructions. This is a confusing topic and we're trying to make it better, but those instructions probably need tweaking
w
I’ve added setuptools to the module_mapping of the python_requirements target, but it didn’t seem to help (I guess it should help, but it doesn’t feel like my change to the BUILD file has been picked up..)
there is no caching of the parsing of BUILD files independent of 
pantsd
, so if restarting 
pantsd
didn’t affect it, then it’s unlikely to be a caching issue.
are you able to share the syntax you used to add it to the module mapping? that should definitely work in this case, and 
pkg_resources
is unlikely to have a collision elsewhere in the repository.
c
Like so:
Copy code
python_requirements(
    module_mapping={
        "setuptools":["pkg_resources"]
    }
)
h
From the linked doc, please run 
./pants dependencies --type=3rdparty ::
- do you see 
setuptools
> 1 time?
c
no, it doesn’t show up at all..
I tried to get back to a failing state, by killing pantsd and wiping the .pants.d folder.. not sure if that’s where everything is cached? or if it’s based in my home folder.. because it seems to still work (removed the explicit dependency I added to get it to work earlier..)
👀 1
h
Oh wait yeah that's a bogus recommendation...that command requires that the dependencies were picked up in the first place. The new warning I'm working on will tell you if this is indeed failing because >1 target refers to the same module, but that won't land until the upcoming 2.4 dev release
w
@curved-television-6568: there is no on-disk cache of BUILD file parsing, so nothing you wipe on disk will affect things there. needing to restart 
pantsd
should be pretty rare… and if restarting 
pantsd
does nothing, then it is very likely to just be a configuration/BUILD file content/layout problem
c
yeah, I’m not sure what it was still, though… but I’ll keep poking at it, I always learn something new. 🙂 timezone’s a bitch though (I’m off for today…)
👋 2
h
Thanks for reporting on things like this - as Stu mentioned, if there is an invalidation bug, that is extremely high priority for us to fix
c
Happy to help. I’ll keep you updated with what I find. @witty-crayon-22786 if not caching, but is there a clean way to recreate the python environments that the tests run in?
h
There is! Documented as of a couple hours ago 🙂 https://www.pantsbuild.org/docs/troubleshooting#debug-tip-inspect-the-chroot-with---no-process-execution-cleanup-local-dirs Note the 
__run.sh
script. Although, to be clear, I don't think that would be super relevant to dependency inference not working. That doesn't involve running a process in a chroot
😆 1
c
OK, I think I found an interesting thing on this topic. As I added another library, started really simple with a few lines of code, and a pex binary target, it built fine, but missed a dependency (I didn’t add the required library to my requirements file on purpose, to test what happened) And, running the pex file complained about missing the library (click in this case) I added it to my requirements file, but couldn’t manage to get pants to pick the change up, until I killed pantsd. Then it worked right off. So, I think a warning is in order if you import stuff that isn’t listed in your requirments. Also, there is a cache bug not invalidating stuff when I change my requirements (perhaps most notably, when I add new things that wasn’t there before).
Copy code
$ ./pants package src/xx/cli:
16:05:58.20 [INFO] Completed: Building src.xx.cli/xx.pex
16:05:58.21 [INFO] Wrote dist/src.xx.cli/xx.pex

$ ./dist/src.xx.cli/xx.pex --help
Traceback (most recent call last):
  ...
  File "/Users/zz/src/svs/python/xx/dist/src.xx.cli/xx.pex/xx/cli/main.py", line 2, in <module>
ModuleNotFoundError: No module named 'click'

$ ./pants dependencies src/xx/cli/main.py
## no output

# added click to requirements.txt and retried above with same outcome. then..

KILLED pantsd

$ ./pants dependencies src/xx/cli/main.py
src:click

$ ./pants package src/xx/cli:xx
16:13:53.34 [INFO] Completed: Resolving 1 requirement: click
16:13:54.49 [INFO] Completed: Building src.xx.cli/xx.pex
16:13:54.49 [INFO] Wrote dist/src.xx.cli/xx.pex

$ ./dist/src.xx.cli/xx.pex --help
Traceback (most recent call last):
    ...
AttributeError: module 'xx.cli.main' has no attribute '__path__'

# click is found, unrelated issues remain
w
What is the name/path of this requirements file?
c
Copy code
$ tree src/
src/
├── BUILD
├── xx
│   ├── __init__.py
│   ├── cli
│   │   ├── BUILD
│   │   ├── __init__.py
│   │   ├── __main__.py
│   │   ├── main.py
├── requirements.txt
Copy code
$ cat src/BUILD 
# -*- mode: python -*-

python_requirements(
    module_mapping={
        "setuptools":["pkg_resources"]
    }
)
Copy code
$ cat src/xx/cli/BUILD
# -*- mode: python -*-

python_library()

pex_binary(
    name="xx",
    entry_point="xx.cli.main:main"
)
Copy code
$ cat src/xx/cli/main.py
"""xx console entry point"""
import click

@click.command('xx')
def main() -> int:
    print('hello from xx')
    return 3
w
c
I think this is from after I restarted pantsd, right?
Copy code
$ cat .pants.d/pants.log 
16:10:04.34 [INFO] handling request: `--pants-bin-name=./pants --pants-version=2.4.0.dev3 dependencies --transitive src/xx/cli/main.py`
16:10:08.02 [INFO] request completed: `--pants-bin-name=./pants --pants-version=2.4.0.dev3 dependencies --transitive src/xx/cli/main.py`
16:10:12.92 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {".emacs.desktop", ""}
16:10:12.92 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {".emacs.desktop", ""}
16:10:12.92 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {".#.emacs.desktop", ""}
16:10:12.92 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"", ".#.emacs.desktop"}
16:10:12.93 [INFO] notify invalidation: cleared 1 and dirtied 234 nodes for: {"", ".emacs.desktop"}
16:10:12.93 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {".emacs.desktop", ""}
16:10:12.93 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {".#.emacs.desktop", ""}
16:10:12.93 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"", ".#.emacs.desktop"}
16:13:46.61 [INFO] handling request: `--pants-bin-name=./pants --pants-version=2.4.0.dev3 package src/xx/cli:xx`
16:13:54.49 [INFO] request completed: `--pants-bin-name=./pants --pants-version=2.4.0.dev3 package src/xx/cli:xx`
16:18:41.90 [INFO] handling request: `--pants-bin-name=./pants --pants-version=2.4.0.dev3 dependencies src/xx/cli/main.py`
16:18:42.02 [INFO] request completed: `--pants-bin-name=./pants --pants-version=2.4.0.dev3 dependencies src/xx/cli/main.py`
16:21:02.50 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {".emacs.desktop", ""}
16:21:02.50 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {".emacs.desktop", ""}
16:21:02.50 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"", ".#.emacs.desktop"}
16:21:02.50 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"", ".#.emacs.desktop"}
16:21:02.50 [INFO] notify invalidation: cleared 1 and dirtied 272 nodes for: {".emacs.desktop", ""}
16:21:02.50 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {".emacs.desktop", ""}
16:21:02.50 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"", ".#.emacs.desktop"}
16:21:02.50 [INFO] notify invalidation: cleared 1 and dirtied 272 nodes for: {"", ".#.emacs.desktop"}
16:21:02.50 [INFO] Filesystem changed during run: retrying `PySnapshot` in 500ms...
16:24:46.65 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"src/xx/cli", "src/xx/cli/.#BUILD"}
16:24:46.65 [INFO] notify invalidation: cleared 1 and dirtied 87 nodes for: {"src/xx/cli/.#BUILD", "src/xx/cli"}
16:24:47.86 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"src/xx/cli", "src/xx/cli/BUILD"}
16:24:47.86 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"src/xx/cli", "src/xx/cli/BUILD"}
16:24:47.86 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"src/xx/cli/.#BUILD", "src/xx/cli"}
16:24:47.86 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"src/xx/cli", "src/xx/cli/.#BUILD"}
16:24:47.86 [INFO] notify invalidation: cleared 1 and dirtied 61 nodes for: {"src/xx/cli", "src/xx/cli/BUILD"}
16:24:47.86 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"src/xx/cli", "src/xx/cli/BUILD"}
16:24:47.86 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"src/xx/cli", "src/xx/cli/.#BUILD"}
16:24:47.86 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"src/xx/cli/.#BUILD", "src/xx/cli"}
16:24:56.11 [INFO] handling request: `--pants-bin-name=./pants --pants-version=2.4.0.dev3 package src/xx/cli:xx`
16:24:57.79 [INFO] request completed: `--pants-bin-name=./pants --pants-version=2.4.0.dev3 package src/xx/cli:xx`
16:25:05.11 [INFO] Extending leases
16:25:05.12 [INFO] Done extending leases
16:25:18.18 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"", ".emacs.desktop"}
16:25:18.18 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"", ".emacs.desktop"}
16:25:18.18 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"", ".#.emacs.desktop"}
16:25:18.18 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"", ".#.emacs.desktop"}
16:25:18.18 [INFO] notify invalidation: cleared 1 and dirtied 282 nodes for: {"", ".emacs.desktop"}
16:25:18.18 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"", ".emacs.desktop"}
16:25:18.18 [INFO] notify invalidation: cleared 1 and dirtied 282 nodes for: {".#.emacs.desktop", ""}
16:25:18.18 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {".#.emacs.desktop", ""}
16:25:18.18 [INFO] Filesystem changed during run: retrying `PySnapshot` in 500ms...
16:35:56.56 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {".emacs.desktop", ""}
16:35:56.57 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"", ".emacs.desktop"}
16:35:56.57 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {".#.emacs.desktop", ""}
16:35:56.57 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"", ".#.emacs.desktop"}
16:35:56.57 [INFO] notify invalidation: cleared 1 and dirtied 282 nodes for: {"", ".emacs.desktop"}
16:35:56.57 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {".emacs.desktop", ""}
16:35:56.57 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {".#.emacs.desktop", ""}
16:35:56.57 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {".#.emacs.desktop", ""}
16:40:05.37 [INFO] Extending leases
16:40:05.38 [INFO] Done extending leases
16:55:05.82 [INFO] Extending leases
16:55:05.83 [INFO] Done extending leases
I’ll see what happens if I add another unlisted dep…
yup, built without complaints, but the dummy lib dep is missing:
Copy code
$ ./pants package src/xx/cli:xx
17:03:01.07 [INFO] Completed: Building src.xx.cli/xx.pex
17:03:01.07 [INFO] Wrote dist/src.xx.cli/xx.pex
zz@zz-mac-...:~/src/svs/python/xx (refactoring-setup *)
$ ./dist/src.xx.cli/xx.pex
Traceback (most recent call last):
  ...
  File "/Users/zz/src/svs/python/xx/dist/src.xx.cli/xx.pex/xx/cli/main.py", line 3, in <module>
ModuleNotFoundError: No module named 'kazoo'
zz@zz-mac-...:~/src/svs/python/xx (refactoring-setup *)
$ cat .pants.d/pants.log 
16:10:04.34 [INFO] handling request: `--pants-bin-name=./pants --pants-version=2.4.0.dev3 dependencies --transitive src/xx/cli/main.py`
[...]
16:55:05.83 [INFO] Done extending leases
17:01:49.82 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"src/xx/cli/.#main.py", "src/xx/cli"}
17:01:49.82 [INFO] notify invalidation: cleared 1 and dirtied 97 nodes for: {"src/xx/cli", "src/xx/cli/.#main.py"}
17:02:21.88 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"src/xx/cli/main.py", "src/xx/cli"}
17:02:21.88 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"src/xx/cli", "src/xx/cli/main.py"}
17:02:21.88 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"src/xx/cli", "src/xx/cli/.#main.py"}
17:02:21.88 [INFO] notify invalidation: cleared 1 and dirtied 36 nodes for: {"src/xx/cli", "src/xx/cli/main.py"}
17:02:21.88 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"src/xx/cli", "src/xx/cli/main.py"}
17:02:21.88 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"src/xx/cli", "src/xx/cli/.#main.py"}
17:02:51.94 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"", ".#.emacs.desktop"}
17:02:51.94 [INFO] notify invalidation: cleared 1 and dirtied 282 nodes for: {"", ".#.emacs.desktop"}
17:02:51.94 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"", ".emacs.desktop"}
17:02:51.94 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"", ".emacs.desktop"}
17:02:51.94 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"", ".#.emacs.desktop"}
17:02:51.94 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {".#.emacs.desktop", ""}
17:02:51.94 [INFO] notify invalidation: cleared 1 and dirtied 282 nodes for: {".emacs.desktop", ""}
17:02:51.94 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {".emacs.desktop", ""}
17:02:51.94 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"", ".#.emacs.desktop"}
17:02:51.94 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {".#.emacs.desktop", ""}
17:02:57.59 [INFO] handling request: `--pants-bin-name=./pants --pants-version=2.4.0.dev3 package src/xx/cli:xx`
17:03:01.07 [INFO] request completed: `--pants-bin-name=./pants --pants-version=2.4.0.dev3 package src/xx/cli:xx`
zz@zz-mac-...:~/src/svs/python/xx (refactoring-setup *)
$
tail of pantsd log after adding kazoo to requirements and running 
./pants package
again..
Copy code
17:03:01.07 [INFO] request completed: `--pants-bin-name=./pants --pants-version=2.4.0.dev3 package src/xx/cli:xx`
17:05:44.94 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"", ".emacs.desktop"}
17:05:44.94 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {".emacs.desktop", ""}
17:05:44.94 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {".#.emacs.desktop", ""}
17:05:44.94 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"", ".#.emacs.desktop"}
17:05:44.95 [INFO] notify invalidation: cleared 1 and dirtied 286 nodes for: {".emacs.desktop", ""}
17:05:44.95 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"", ".emacs.desktop"}
17:05:44.95 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {".#.emacs.desktop", ""}
17:05:44.95 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {".#.emacs.desktop", ""}
17:05:58.04 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"src/.#requirements.txt", "src"}
17:05:58.04 [INFO] notify invalidation: cleared 1 and dirtied 282 nodes for: {"src", "src/.#requirements.txt"}
17:06:00.44 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"src", "src/requirements.txt"}
17:06:00.45 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"src", "src/requirements.txt"}
17:06:00.45 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"src", "src/.#requirements.txt"}
17:06:00.45 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"src/.#requirements.txt", "src"}
17:06:00.45 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"src/requirements.txt", "src"}
17:06:00.45 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"src", "src/requirements.txt"}
17:06:00.45 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"src/.#requirements.txt", "src"}
17:06:00.45 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"src", "src/.#requirements.txt"}
17:06:06.68 [INFO] handling request: `--pants-bin-name=./pants --pants-version=2.4.0.dev3 package src/xx/cli:xx`
17:06:06.78 [INFO] request completed: `--pants-bin-name=./pants --pants-version=2.4.0.dev3 package src/xx/cli:xx`
(kazoo was still missing in the pex file, but I expect it would show up if I killed pantsd)
w
oh, argh. it looks like our special case for the requirements file is too lax =( …
so 
src/requirements.txt
is not being matched.
you can work around this by adding the following to the 
[GLOBAL]
section of 
pants.toml
:
Copy code
pantsd_invalidation_globs.add = [
  "src/requirements.txt",
]
or if you have multiple requirements files:
Copy code
pantsd_invalidation_globs.add = [
  "**/requirements.txt",
]
c
Ah, cool, thanks 🙂
w
and this is definitely a workaround: sorry. if you want to leave a TODO there referring to https://github.com/pantsbuild/pants/issues/7022, that would be reasonable. it should definitely not be necessary to restart pantsd like this
👍 1
c
I’m pretty sure that this is what I ran into from my initial post in this thread as well, so I feel we can close it with “requirements.txt” not in root or under 3rdparty tree.
Yeah, but I feel you have it under control 😉
5 Views
Open in Slack
PreviousNext
Powered by