https://pantsbuild.org/ logo
#general
Title
# general
n

nice-florist-55958

03/12/2022, 6:42 PM
Started experimenting with upgrading Pants and resolve files. One thing that I immediately noticed was Pants was able to detect a file system change, cancel the current job, and re-schedule it. This was observed for packaging a
pex_binary
and anything else involving lockfile resolution. I wondered if that meant
pantsd
might work now on NFS (were there any changes made to the file-watcher? e.g., using poll mechanism instead of NFS-incompatible
inotify
?). Unfortunately, I am getting the following errors (
v2.10.0rc2+git40e4c4d
), both on NFS and local file system:
Copy code
<http://ivapp1341241.howard.ms.com|ivapp1341241.howard.ms.com> /v/global/user/t/ta/taymarti/casper/codetree/codetree/src 89$ pants -ldebug --print-stacktrace package proj/libs/tools
09:53:27.19 [DEBUG] acquiring lock: <pants.pantsd.lock.OwnerPrintingInterProcessFileLock object at 0x7f0c09f76fd0>
09:53:27.21 [DEBUG] purging metadata directory: /a/nyn183f2/vol/nyn183f2v18/u_t1445795813/taymarti/casper/codetree/codetree/src/.pids/c2bc7abd26c8/pantsd
09:53:27.21 [DEBUG] Launching pantsd
09:53:27.21 [DEBUG] purging metadata directory: /a/nyn183f2/vol/nyn183f2v18/u_t1445795813/taymarti/casper/codetree/codetree/src/.pids/c2bc7abd26c8/pantsd
09:53:27.21 [DEBUG] pantsd command is: PANTS_DAEMON_ENTRYPOINT=pants.pantsd.pants_daemon:launch_new_pantsd_instance PYTHONPATH=/a/nyn183f2/vol/nyn183f2v18/u_t1445795813/taymarti/.pex/venvs/5dbaf96a5f6c943ed10da656e547daf24522d0aa/779eb2cc0ca9e2fdd204774cbc41848e4e7c5055:/ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/lib/python37.zip:/ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/lib/python3.7:/ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/lib/python3.7/lib-dynload:/a/nyn183f2/vol/nyn183f2v18/u_t1445795813/taymarti/.pex/venvs/s/3e71c54a/venv/lib/python3.7/site-packages: /a/nyn183f2/vol/nyn183f2v18/u_t1445795813/taymarti/.pex/venvs/s/3e71c54a/venv/bin/python3.7 /v/global/user/t/ta/taymarti/.pex/venvs/5dbaf96a5f6c943ed10da656e547daf24522d0aa/779eb2cc0ca9e2fdd204774cbc41848e4e7c5055/pex -ldebug --print-stacktrace package proj/libs/tools
09:53:29.72 [DEBUG] pantsd is running at pid 6723, pailgun port is 32817
09:53:29.72 [DEBUG] releasing lock: <pants.pantsd.lock.OwnerPrintingInterProcessFileLock object at 0x7f0c09f76fd0>
09:53:29.72 [DEBUG] Connecting to pantsd on port 32817
09:53:29.72 [DEBUG] Connecting to pantsd on port 32817 attempt 1/3
09:53:29.72 [DEBUG] Connected to pantsd
09:53:29.75 [DEBUG] Launching 1 roots (poll=false).
09:53:29.77 [DEBUG] computed 1 nodes in 0.018837 seconds. there are 7 total nodes.
09:53:29.90 [INFO] Initializing scheduler...
09:53:29.92 [DEBUG] File handle limit is: 10000
09:53:30.51 [DEBUG] Changes to /a/nyn183f2/vol/nyn183f2v18/u_t1445795813/taymarti/.pex/venvs/5dbaf96a5f6c943ed10da656e547daf24522d0aa/779eb2cc0ca9e2fdd204774cbc41848e4e7c5055, outside of the buildroot, will not be invalidated.
09:53:30.51 [DEBUG] Changes to /a/nyn183f2/vol/nyn183f2v18/u_t1445795813/taymarti/.pex/venvs/5dbaf96a5f6c943ed10da656e547daf24522d0aa/779eb2cc0ca9e2fdd204774cbc41848e4e7c5055, outside of the buildroot, will not be invalidated.
09:53:30.51 [DEBUG] Changes to /ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/lib/python37.zip, outside of the buildroot, will not be invalidated.
09:53:30.51 [DEBUG] Changes to /ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/lib/python3.7, outside of the buildroot, will not be invalidated.
09:53:30.51 [DEBUG] Changes to /ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/lib/python3.7/lib-dynload, outside of the buildroot, will not be invalidated.
09:53:30.51 [DEBUG] Changes to /a/nyn183f2/vol/nyn183f2v18/u_t1445795813/taymarti/.pex/venvs/s/3e71c54a/venv/lib/python3.7/site-packages, outside of the buildroot, will not be invalidated.
09:53:30.51 [DEBUG] Changes to /a/nyn183f2/vol/nyn183f2v18/u_t1445795813/taymarti/casper/codetree/codetree/src, outside of the buildroot, will not be invalidated.
09:53:30.52 [ERROR] no path specified
Traceback (most recent call last):
 File "/a/nyn183f2/vol/nyn183f2v18/u_t1445795813/taymarti/.pex/venvs/s/3e71c54a/venv/lib/python3.7/site-packages/pants/bin/daemon_pants_runner.py", line 130, in single_daemonized_run
   scheduler, options_initializer = self._core.prepare(options_bootstrapper, complete_env)
 File "/a/nyn183f2/vol/nyn183f2v18/u_t1445795813/taymarti/.pex/venvs/s/3e71c54a/venv/lib/python3.7/site-packages/pants/pantsd/pants_daemon_core.py", line 180, in prepare
   scheduler_restart_explanation,
 File "/a/nyn183f2/vol/nyn183f2v18/u_t1445795813/taymarti/.pex/venvs/s/3e71c54a/venv/lib/python3.7/site-packages/pants/pantsd/pants_daemon_core.py", line 122, in _initialize
   raise e
 File "/a/nyn183f2/vol/nyn183f2v18/u_t1445795813/taymarti/.pex/venvs/s/3e71c54a/venv/lib/python3.7/site-packages/pants/pantsd/pants_daemon_core.py", line 116, in _initialize
   self._services = self._services_constructor(bootstrap_options, self._scheduler)
 File "/a/nyn183f2/vol/nyn183f2v18/u_t1445795813/taymarti/.pex/venvs/s/3e71c54a/venv/lib/python3.7/site-packages/pants/pantsd/pants_daemon.py", line 85, in _setup_services
   bootstrap_options,
 File "/a/nyn183f2/vol/nyn183f2v18/u_t1445795813/taymarti/.pex/venvs/s/3e71c54a/venv/lib/python3.7/site-packages/pants/option/global_options.py", line 1572, in compute_pantsd_invalidation_globs
   glob_relpath = os.path.relpath(glob, buildroot)
 File "/ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/lib/python3.7/posixpath.py", line 457, in relpath
   raise ValueError("no path specified")
ValueError: no path specified
 
 
See <https://www.pantsbuild.org/v2.10/docs/troubleshooting> for common issues.
Consider reaching out for help: <https://www.pantsbuild.org/v2.10/docs/getting-help>
Copy code
<http://ivapp1341241.howard.ms.com|ivapp1341241.howard.ms.com> /v/global/user/t/ta/taymarti/casper/codetree/codetree/src 90$ export HOME=/var/tmp/taymarti
<http://ivapp1341241.howard.ms.com|ivapp1341241.howard.ms.com> /v/global/user/t/ta/taymarti/casper/codetree/codetree/src 91$ mkdir -p /var/tmp/taymarti
<http://ivapp1341241.howard.ms.com|ivapp1341241.howard.ms.com> /v/global/user/t/ta/taymarti/casper/codetree/codetree/src 92$ pants -ldebug --print-stacktrace package proj/libs/tools
09:54:02.76 [DEBUG] acquiring lock: <pants.pantsd.lock.OwnerPrintingInterProcessFileLock object at 0x7fafb5b4fe50>
09:54:02.79 [DEBUG] purging metadata directory: /a/nyn183f2/vol/nyn183f2v18/u_t1445795813/taymarti/casper/codetree/codetree/src/.pids/c2bc7abd26c8/pantsd
09:54:02.79 [DEBUG] Launching pantsd
09:54:02.79 [DEBUG] purging metadata directory: /a/nyn183f2/vol/nyn183f2v18/u_t1445795813/taymarti/casper/codetree/codetree/src/.pids/c2bc7abd26c8/pantsd
09:54:02.79 [DEBUG] pantsd command is: PANTS_DAEMON_ENTRYPOINT=pants.pantsd.pants_daemon:launch_new_pantsd_instance PYTHONPATH=/var/tmp/taymarti/.pex/venvs/5dbaf96a5f6c943ed10da656e547daf24522d0aa/779eb2cc0ca9e2fdd204774cbc41848e4e7c5055:/ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/lib/python37.zip:/ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/lib/python3.7:/ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/lib/python3.7/lib-dynload:/var/tmp/taymarti/.pex/venvs/s/d4b17305/venv/lib/python3.7/site-packages: /var/tmp/taymarti/.pex/venvs/s/d4b17305/venv/bin/python3.7 /var/tmp/taymarti/.pex/venvs/5dbaf96a5f6c943ed10da656e547daf24522d0aa/779eb2cc0ca9e2fdd204774cbc41848e4e7c5055/pex -ldebug --print-stacktrace package proj/libs/tools
09:54:04.70 [DEBUG] pantsd is running at pid 6917, pailgun port is 45659
09:54:04.70 [DEBUG] releasing lock: <pants.pantsd.lock.OwnerPrintingInterProcessFileLock object at 0x7fafb5b4fe50>
09:54:04.70 [DEBUG] Connecting to pantsd on port 45659
09:54:04.71 [DEBUG] Connecting to pantsd on port 45659 attempt 1/3
09:54:04.71 [DEBUG] Connected to pantsd
09:54:04.73 [DEBUG] Launching 1 roots (poll=false).
09:54:04.75 [DEBUG] computed 1 nodes in 0.018034 seconds. there are 7 total nodes.
09:54:04.86 [INFO] Initializing scheduler...
09:54:04.88 [DEBUG] File handle limit is: 10000
09:54:05.00 [DEBUG] Changes to /var/tmp/taymarti/.pex/venvs/5dbaf96a5f6c943ed10da656e547daf24522d0aa/779eb2cc0ca9e2fdd204774cbc41848e4e7c5055, outside of the buildroot, will not be invalidated.
09:54:05.00 [DEBUG] Changes to /var/tmp/taymarti/.pex/venvs/5dbaf96a5f6c943ed10da656e547daf24522d0aa/779eb2cc0ca9e2fdd204774cbc41848e4e7c5055, outside of the buildroot, will not be invalidated.
09:54:05.00 [DEBUG] Changes to /ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/lib/python37.zip, outside of the buildroot, will not be invalidated.
09:54:05.00 [DEBUG] Changes to /ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/lib/python3.7, outside of the buildroot, will not be invalidated.
09:54:05.00 [DEBUG] Changes to /ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/lib/python3.7/lib-dynload, outside of the buildroot, will not be invalidated.
09:54:05.00 [DEBUG] Changes to /var/tmp/taymarti/.pex/venvs/s/d4b17305/venv/lib/python3.7/site-packages, outside of the buildroot, will not be invalidated.
09:54:05.00 [DEBUG] Changes to /a/nyn183f2/vol/nyn183f2v18/u_t1445795813/taymarti/casper/codetree/codetree/src, outside of the buildroot, will not be invalidated.
09:54:05.00 [ERROR] no path specified
Traceback (most recent call last):
 File "/var/tmp/taymarti/.pex/venvs/s/d4b17305/venv/lib/python3.7/site-packages/pants/bin/daemon_pants_runner.py", line 130, in single_daemonized_run
   scheduler, options_initializer = self._core.prepare(options_bootstrapper, complete_env)
 File "/var/tmp/taymarti/.pex/venvs/s/d4b17305/venv/lib/python3.7/site-packages/pants/pantsd/pants_daemon_core.py", line 180, in prepare
   scheduler_restart_explanation,
 File "/var/tmp/taymarti/.pex/venvs/s/d4b17305/venv/lib/python3.7/site-packages/pants/pantsd/pants_daemon_core.py", line 122, in _initialize
   raise e
 File "/var/tmp/taymarti/.pex/venvs/s/d4b17305/venv/lib/python3.7/site-packages/pants/pantsd/pants_daemon_core.py", line 116, in _initialize
   self._services = self._services_constructor(bootstrap_options, self._scheduler)
 File "/var/tmp/taymarti/.pex/venvs/s/d4b17305/venv/lib/python3.7/site-packages/pants/pantsd/pants_daemon.py", line 85, in _setup_services
   bootstrap_options,
 File "/var/tmp/taymarti/.pex/venvs/s/d4b17305/venv/lib/python3.7/site-packages/pants/option/global_options.py", line 1572, in compute_pantsd_invalidation_globs
   glob_relpath = os.path.relpath(glob, buildroot)
 File "/ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/lib/python3.7/posixpath.py", line 457, in relpath
   raise ValueError("no path specified")
ValueError: no path specified
e

enough-analyst-54434

03/12/2022, 8:43 PM
I wondered if that meant
pantsd
might work now on NFS (were there any changes made to the file-watcher? e.g., using poll mechanism instead of NFS-incompatible
inotify
?).
There have been no changes to that code in a while:
Copy code
$ git blame src/rust/engine/Cargo.lock | grep -A1 notify
6db9ab4bec (Henry Fuller       2020-03-26 12:01:39 -0700 1389) name = "inotify"
97886dc325 (Stu Hood           2020-12-22 12:08:58 -0800 1390) version = "0.8.3"
--
b4b0d043c4 (Tom Dyas           2020-10-10 17:20:43 -0700 1395)  "inotify-sys",
b4b0d043c4 (Tom Dyas           2020-10-10 17:20:43 -0700 1396)  "libc",
--
6db9ab4bec (Henry Fuller       2020-03-26 12:01:39 -0700 1400) name = "inotify-sys"
97886dc325 (Stu Hood           2020-12-22 12:08:58 -0800 1401) version = "0.1.4"
--
6db9ab4bec (Henry Fuller       2020-03-26 12:01:39 -0700 1805) name = "notify"
506a8568ad (Stu Hood           2020-07-14 15:23:14 -0700 1806) version = "5.0.0-pre.3"
8302092046 (Stu Hood           2020-07-15 11:28:24 -0700 1807) source = "git+<https://github.com/pantsbuild/notify?rev=64880f0662db2b5ecbf25f1cccdca64bb8fac1bc#64880f0662db2b5ecbf25f1cccdca64bb8fac1bc>"
5fcb492457 (Greg Shuflin       2019-12-04 15:54:40 -0800 1808) dependencies = [
--
b4b0d043c4 (Tom Dyas           2020-10-10 17:20:43 -0700 1815)  "inotify",
b4b0d043c4 (Tom Dyas           2020-10-10 17:20:43 -0700 1816)  "libc",
--
b4b0d043c4 (Tom Dyas           2020-10-10 17:20:43 -0700 3840)  "notify",
de70c5b918 (dependabot[bot]    2022-02-02 22:18:04 -0800 3841)  "parking_lot 0.12.0",
We configure the notify crate using the recommend watcher, which is unconditionally inotify on Linux fwict: https://github.com/pantsbuild/pants/blob/9d6a09589c69f76aff88115155a7a0fc54bd482b/src/rust/engine/watch/src/lib.rs#L84-L91 I think we'd need to plumb an option that allowed you to select the poll watcher (which the notify crate has). All that said, I have no experience here and @witty-crayon-22786 is expert. It may be that polling has fatal issues and we can't use it, or it may be we can and can provide this option.
w

witty-crayon-22786

03/12/2022, 9:05 PM
Yea, polling is an option. Although how much overhead it would add (particularly on NFS) would be interesting.
As to the exception that was raised: it sounds like you either have an explicit entry in
[GLOBAL].pants_ignore
, or a pythonpath entry which is ... Empty? Or malformed? Is the root of the repository a pythonpath entry? It's a bug on our end (since we should ignore or properly handle the path), but will need a bit more info.
n

nice-florist-55958

03/12/2022, 9:19 PM
Not a pythonpath entry, but aha, we do have the root ignored explicitly, probably from back when we were less experienced and iirc to avoid getting errors from Pants interrogating the root for build files or something. Will revisit, but you're saying that should fix it? Any ideas on how Pants is detecting file changes in the code tree during lock file resolution when using NFS? Is this distinct from pantsd’s watcher?
w

witty-crayon-22786

03/12/2022, 9:39 PM
No idea... Is only a portion of the working copy on NFS?
Currently,
pantsd
only uses inotify on Linux (no polling).
n

nice-florist-55958

03/12/2022, 10:51 PM
Still no luck after cleaning up
pants.toml
; even removed
[GLOBAL].pants_ignore
entirely. So far traced the problem to
sys.path
containing
''
in the list at the time
global_options.compute_pantsd_invalidation_globs
is called. I can filter that out for now to test locally, but wondering how it's there in the first place. Confirmed the
pythonpath
list is empty there.
Patched for now by adding
if not glob: continue
in the for loop. Shocked to see that
pantsd
is working now--reacting to changes to source files when packaging--both with
$HOME
referring to local storage and NFS. In both cases the repo is checked out on NFS. I'm going to go back and check all this with v2.8 and v2.9 and v2.10 with the script instead of PEX. Still very surprised, considering my IDE's file watcher at this very moment is complaining and asking me if my files are mounted, lol.