Started experimenting with upgrading Pants and resolve files. One thing that I immediately noticed was Pants was able to detect a file system change, cancel the current job, and re-schedule it. This was observed for packaging a

pex_binary

and anything else involving lockfile resolution. I wondered if that meant

pantsd

might work now on NFS (were there any changes made to the file-watcher? e.g., using poll mechanism instead of NFS-incompatible

inotify

?). Unfortunately, I am getting the following errors (

v2.10.0rc2+git40e4c4d

), both on NFS and local file system:

<http://ivapp1341241.howard.ms.com|ivapp1341241.howard.ms.com> /v/global/user/t/ta/taymarti/casper/codetree/codetree/src 89$ pants -ldebug --print-stacktrace package proj/libs/tools
09:53:27.19 [DEBUG] acquiring lock: <pants.pantsd.lock.OwnerPrintingInterProcessFileLock object at 0x7f0c09f76fd0>
09:53:27.21 [DEBUG] purging metadata directory: /a/nyn183f2/vol/nyn183f2v18/u_t1445795813/taymarti/casper/codetree/codetree/src/.pids/c2bc7abd26c8/pantsd
09:53:27.21 [DEBUG] Launching pantsd
09:53:27.21 [DEBUG] purging metadata directory: /a/nyn183f2/vol/nyn183f2v18/u_t1445795813/taymarti/casper/codetree/codetree/src/.pids/c2bc7abd26c8/pantsd
09:53:27.21 [DEBUG] pantsd command is: PANTS_DAEMON_ENTRYPOINT=pants.pantsd.pants_daemon:launch_new_pantsd_instance PYTHONPATH=/a/nyn183f2/vol/nyn183f2v18/u_t1445795813/taymarti/.pex/venvs/5dbaf96a5f6c943ed10da656e547daf24522d0aa/779eb2cc0ca9e2fdd204774cbc41848e4e7c5055:/ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/lib/python37.zip:/ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/lib/python3.7:/ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/lib/python3.7/lib-dynload:/a/nyn183f2/vol/nyn183f2v18/u_t1445795813/taymarti/.pex/venvs/s/3e71c54a/venv/lib/python3.7/site-packages: /a/nyn183f2/vol/nyn183f2v18/u_t1445795813/taymarti/.pex/venvs/s/3e71c54a/venv/bin/python3.7 /v/global/user/t/ta/taymarti/.pex/venvs/5dbaf96a5f6c943ed10da656e547daf24522d0aa/779eb2cc0ca9e2fdd204774cbc41848e4e7c5055/pex -ldebug --print-stacktrace package proj/libs/tools
09:53:29.72 [DEBUG] pantsd is running at pid 6723, pailgun port is 32817
09:53:29.72 [DEBUG] releasing lock: <pants.pantsd.lock.OwnerPrintingInterProcessFileLock object at 0x7f0c09f76fd0>
09:53:29.72 [DEBUG] Connecting to pantsd on port 32817
09:53:29.72 [DEBUG] Connecting to pantsd on port 32817 attempt 1/3
09:53:29.72 [DEBUG] Connected to pantsd
09:53:29.75 [DEBUG] Launching 1 roots (poll=false).
09:53:29.77 [DEBUG] computed 1 nodes in 0.018837 seconds. there are 7 total nodes.
09:53:29.90 [INFO] Initializing scheduler...
09:53:29.92 [DEBUG] File handle limit is: 10000
09:53:30.51 [DEBUG] Changes to /a/nyn183f2/vol/nyn183f2v18/u_t1445795813/taymarti/.pex/venvs/5dbaf96a5f6c943ed10da656e547daf24522d0aa/779eb2cc0ca9e2fdd204774cbc41848e4e7c5055, outside of the buildroot, will not be invalidated.
09:53:30.51 [DEBUG] Changes to /a/nyn183f2/vol/nyn183f2v18/u_t1445795813/taymarti/.pex/venvs/5dbaf96a5f6c943ed10da656e547daf24522d0aa/779eb2cc0ca9e2fdd204774cbc41848e4e7c5055, outside of the buildroot, will not be invalidated.
09:53:30.51 [DEBUG] Changes to /ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/lib/python37.zip, outside of the buildroot, will not be invalidated.
09:53:30.51 [DEBUG] Changes to /ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/lib/python3.7, outside of the buildroot, will not be invalidated.
09:53:30.51 [DEBUG] Changes to /ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/lib/python3.7/lib-dynload, outside of the buildroot, will not be invalidated.
09:53:30.51 [DEBUG] Changes to /a/nyn183f2/vol/nyn183f2v18/u_t1445795813/taymarti/.pex/venvs/s/3e71c54a/venv/lib/python3.7/site-packages, outside of the buildroot, will not be invalidated.
09:53:30.51 [DEBUG] Changes to /a/nyn183f2/vol/nyn183f2v18/u_t1445795813/taymarti/casper/codetree/codetree/src, outside of the buildroot, will not be invalidated.
09:53:30.52 [ERROR] no path specified
Traceback (most recent call last):
 File "/a/nyn183f2/vol/nyn183f2v18/u_t1445795813/taymarti/.pex/venvs/s/3e71c54a/venv/lib/python3.7/site-packages/pants/bin/daemon_pants_runner.py", line 130, in single_daemonized_run
   scheduler, options_initializer = self._core.prepare(options_bootstrapper, complete_env)
 File "/a/nyn183f2/vol/nyn183f2v18/u_t1445795813/taymarti/.pex/venvs/s/3e71c54a/venv/lib/python3.7/site-packages/pants/pantsd/pants_daemon_core.py", line 180, in prepare
   scheduler_restart_explanation,
 File "/a/nyn183f2/vol/nyn183f2v18/u_t1445795813/taymarti/.pex/venvs/s/3e71c54a/venv/lib/python3.7/site-packages/pants/pantsd/pants_daemon_core.py", line 122, in _initialize
   raise e
 File "/a/nyn183f2/vol/nyn183f2v18/u_t1445795813/taymarti/.pex/venvs/s/3e71c54a/venv/lib/python3.7/site-packages/pants/pantsd/pants_daemon_core.py", line 116, in _initialize
   self._services = self._services_constructor(bootstrap_options, self._scheduler)
 File "/a/nyn183f2/vol/nyn183f2v18/u_t1445795813/taymarti/.pex/venvs/s/3e71c54a/venv/lib/python3.7/site-packages/pants/pantsd/pants_daemon.py", line 85, in _setup_services
   bootstrap_options,
 File "/a/nyn183f2/vol/nyn183f2v18/u_t1445795813/taymarti/.pex/venvs/s/3e71c54a/venv/lib/python3.7/site-packages/pants/option/global_options.py", line 1572, in compute_pantsd_invalidation_globs
   glob_relpath = os.path.relpath(glob, buildroot)
 File "/ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/lib/python3.7/posixpath.py", line 457, in relpath
   raise ValueError("no path specified")
ValueError: no path specified
 
 
See <https://www.pantsbuild.org/v2.10/docs/troubleshooting> for common issues.
Consider reaching out for help: <https://www.pantsbuild.org/v2.10/docs/getting-help>

<http://ivapp1341241.howard.ms.com|ivapp1341241.howard.ms.com> /v/global/user/t/ta/taymarti/casper/codetree/codetree/src 90$ export HOME=/var/tmp/taymarti
<http://ivapp1341241.howard.ms.com|ivapp1341241.howard.ms.com> /v/global/user/t/ta/taymarti/casper/codetree/codetree/src 91$ mkdir -p /var/tmp/taymarti
<http://ivapp1341241.howard.ms.com|ivapp1341241.howard.ms.com> /v/global/user/t/ta/taymarti/casper/codetree/codetree/src 92$ pants -ldebug --print-stacktrace package proj/libs/tools
09:54:02.76 [DEBUG] acquiring lock: <pants.pantsd.lock.OwnerPrintingInterProcessFileLock object at 0x7fafb5b4fe50>
09:54:02.79 [DEBUG] purging metadata directory: /a/nyn183f2/vol/nyn183f2v18/u_t1445795813/taymarti/casper/codetree/codetree/src/.pids/c2bc7abd26c8/pantsd
09:54:02.79 [DEBUG] Launching pantsd
09:54:02.79 [DEBUG] purging metadata directory: /a/nyn183f2/vol/nyn183f2v18/u_t1445795813/taymarti/casper/codetree/codetree/src/.pids/c2bc7abd26c8/pantsd
09:54:02.79 [DEBUG] pantsd command is: PANTS_DAEMON_ENTRYPOINT=pants.pantsd.pants_daemon:launch_new_pantsd_instance PYTHONPATH=/var/tmp/taymarti/.pex/venvs/5dbaf96a5f6c943ed10da656e547daf24522d0aa/779eb2cc0ca9e2fdd204774cbc41848e4e7c5055:/ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/lib/python37.zip:/ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/lib/python3.7:/ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/lib/python3.7/lib-dynload:/var/tmp/taymarti/.pex/venvs/s/d4b17305/venv/lib/python3.7/site-packages: /var/tmp/taymarti/.pex/venvs/s/d4b17305/venv/bin/python3.7 /var/tmp/taymarti/.pex/venvs/5dbaf96a5f6c943ed10da656e547daf24522d0aa/779eb2cc0ca9e2fdd204774cbc41848e4e7c5055/pex -ldebug --print-stacktrace package proj/libs/tools
09:54:04.70 [DEBUG] pantsd is running at pid 6917, pailgun port is 45659
09:54:04.70 [DEBUG] releasing lock: <pants.pantsd.lock.OwnerPrintingInterProcessFileLock object at 0x7fafb5b4fe50>
09:54:04.70 [DEBUG] Connecting to pantsd on port 45659
09:54:04.71 [DEBUG] Connecting to pantsd on port 45659 attempt 1/3
09:54:04.71 [DEBUG] Connected to pantsd
09:54:04.73 [DEBUG] Launching 1 roots (poll=false).
09:54:04.75 [DEBUG] computed 1 nodes in 0.018034 seconds. there are 7 total nodes.
09:54:04.86 [INFO] Initializing scheduler...
09:54:04.88 [DEBUG] File handle limit is: 10000
09:54:05.00 [DEBUG] Changes to /var/tmp/taymarti/.pex/venvs/5dbaf96a5f6c943ed10da656e547daf24522d0aa/779eb2cc0ca9e2fdd204774cbc41848e4e7c5055, outside of the buildroot, will not be invalidated.
09:54:05.00 [DEBUG] Changes to /var/tmp/taymarti/.pex/venvs/5dbaf96a5f6c943ed10da656e547daf24522d0aa/779eb2cc0ca9e2fdd204774cbc41848e4e7c5055, outside of the buildroot, will not be invalidated.
09:54:05.00 [DEBUG] Changes to /ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/lib/python37.zip, outside of the buildroot, will not be invalidated.
09:54:05.00 [DEBUG] Changes to /ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/lib/python3.7, outside of the buildroot, will not be invalidated.
09:54:05.00 [DEBUG] Changes to /ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/lib/python3.7/lib-dynload, outside of the buildroot, will not be invalidated.
09:54:05.00 [DEBUG] Changes to /var/tmp/taymarti/.pex/venvs/s/d4b17305/venv/lib/python3.7/site-packages, outside of the buildroot, will not be invalidated.
09:54:05.00 [DEBUG] Changes to /a/nyn183f2/vol/nyn183f2v18/u_t1445795813/taymarti/casper/codetree/codetree/src, outside of the buildroot, will not be invalidated.
09:54:05.00 [ERROR] no path specified
Traceback (most recent call last):
 File "/var/tmp/taymarti/.pex/venvs/s/d4b17305/venv/lib/python3.7/site-packages/pants/bin/daemon_pants_runner.py", line 130, in single_daemonized_run
   scheduler, options_initializer = self._core.prepare(options_bootstrapper, complete_env)
 File "/var/tmp/taymarti/.pex/venvs/s/d4b17305/venv/lib/python3.7/site-packages/pants/pantsd/pants_daemon_core.py", line 180, in prepare
   scheduler_restart_explanation,
 File "/var/tmp/taymarti/.pex/venvs/s/d4b17305/venv/lib/python3.7/site-packages/pants/pantsd/pants_daemon_core.py", line 122, in _initialize
   raise e
 File "/var/tmp/taymarti/.pex/venvs/s/d4b17305/venv/lib/python3.7/site-packages/pants/pantsd/pants_daemon_core.py", line 116, in _initialize
   self._services = self._services_constructor(bootstrap_options, self._scheduler)
 File "/var/tmp/taymarti/.pex/venvs/s/d4b17305/venv/lib/python3.7/site-packages/pants/pantsd/pants_daemon.py", line 85, in _setup_services
   bootstrap_options,
 File "/var/tmp/taymarti/.pex/venvs/s/d4b17305/venv/lib/python3.7/site-packages/pants/option/global_options.py", line 1572, in compute_pantsd_invalidation_globs
   glob_relpath = os.path.relpath(glob, buildroot)
 File "/ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/lib/python3.7/posixpath.py", line 457, in relpath
   raise ValueError("no path specified")
ValueError: no path specified

I wondered if that meant

pantsd

might work now on NFS (were there any changes made to the file-watcher? e.g., using poll mechanism instead of NFS-incompatible

inotify

?).

There have been no changes to that code in a while:

$ git blame src/rust/engine/Cargo.lock | grep -A1 notify
6db9ab4bec (Henry Fuller       2020-03-26 12:01:39 -0700 1389) name = "inotify"
97886dc325 (Stu Hood           2020-12-22 12:08:58 -0800 1390) version = "0.8.3"
--
b4b0d043c4 (Tom Dyas           2020-10-10 17:20:43 -0700 1395)  "inotify-sys",
b4b0d043c4 (Tom Dyas           2020-10-10 17:20:43 -0700 1396)  "libc",
--
6db9ab4bec (Henry Fuller       2020-03-26 12:01:39 -0700 1400) name = "inotify-sys"
97886dc325 (Stu Hood           2020-12-22 12:08:58 -0800 1401) version = "0.1.4"
--
6db9ab4bec (Henry Fuller       2020-03-26 12:01:39 -0700 1805) name = "notify"
506a8568ad (Stu Hood           2020-07-14 15:23:14 -0700 1806) version = "5.0.0-pre.3"
8302092046 (Stu Hood           2020-07-15 11:28:24 -0700 1807) source = "git+<https://github.com/pantsbuild/notify?rev=64880f0662db2b5ecbf25f1cccdca64bb8fac1bc#64880f0662db2b5ecbf25f1cccdca64bb8fac1bc>"
5fcb492457 (Greg Shuflin       2019-12-04 15:54:40 -0800 1808) dependencies = [
--
b4b0d043c4 (Tom Dyas           2020-10-10 17:20:43 -0700 1815)  "inotify",
b4b0d043c4 (Tom Dyas           2020-10-10 17:20:43 -0700 1816)  "libc",
--
b4b0d043c4 (Tom Dyas           2020-10-10 17:20:43 -0700 3840)  "notify",
de70c5b918 (dependabot[bot]    2022-02-02 22:18:04 -0800 3841)  "parking_lot 0.12.0",

We configure the notify crate using the recommend watcher, which is unconditionally inotify on Linux fwict: https://github.com/pantsbuild/pants/blob/9d6a09589c69f76aff88115155a7a0fc54bd482b/src/rust/engine/watch/src/lib.rs#L84-L91 I think we'd need to plumb an option that allowed you to select the poll watcher (which the notify crate has). All that said, I have no experience here and @witty-crayon-22786 is expert. It may be that polling has fatal issues and we can't use it, or it may be we can and can provide this option.

Yea, polling is an option. Although how much overhead it would add (particularly on NFS) would be interesting.

Not a pythonpath entry, but aha, we do have the root ignored explicitly, probably from back when we were less experienced and iirc to avoid getting errors from Pants interrogating the root for build files or something. Will revisit, but you're saying that should fix it? Any ideas on how Pants is detecting file changes in the code tree during lock file resolution when using NFS? Is this distinct from pantsd’s watcher?

No idea... Is only a portion of the working copy on NFS?

Still no luck after cleaning up

pants.toml

; even removed

[GLOBAL].pants_ignore

entirely. So far traced the problem to

sys.path

containing

''

in the list at the time

global_options.compute_pantsd_invalidation_globs

is called. I can filter that out for now to test locally, but wondering how it's there in the first place. Confirmed the

pythonpath

list is empty there.