https://pantsbuild.org/ logo
Join Slack
Powered by
This might be a dumb question, but are changes to ...
# general
s
This might be a dumb question, but are changes to source code supposed to cause pantsd to "restart"?
c
Not dumb! No they are not.
s
That's good to know
w
Are you running 
—loop
? Or are you getting ‘watch’ changes? https://www.pantsbuild.org/2.18/reference/global-options#watch_filesystem
The logs in .pants.d will usually indicate the reason for the restart, btw
🙏 1
s
I'm not running 
loop
w
So, is it like… you’re formatting, and during that process, it tells you the file system has changed and the format restart>
s
For example if I run 
pants run workflows/src/workflows/scratch.py
and add a new line to the file I get and run the same command again*
Copy code
18:59:55.87 [INFO] waiting for pantsd to start...
18:59:56.47 [INFO] pantsd started
18:59:56.65 [INFO] Initializing scheduler...
19:00:00.96 [INFO] Scheduler initialized.
...
fwiw If I don't make a modification pantsd doesn't need to start
The term "restart" might have been confusing. I mean I'm seeing 
waiting for pantsd to start...
a lot more
From 
pants.log
Copy code
19:58:11.15 [ERROR] saw filesystem changes covered by invalidation globs: SnapshotDiff(our_unique_files=(), our_unique_dirs=(), their_unique_files=(), their_unique_dirs=(), changed_files=('workflows/src/workflows/scratch.py',)). terminating the daemon.
19:58:11.34 [ERROR] service failure for <pants.pantsd.service.scheduler_service.SchedulerService object at 0x7f9274528fd0>.
19:58:11.34 [INFO] Waiting for ongoing runs to complete before exiting...
19:58:11.34 [INFO] Server exiting with Ok(())
19:58:11.34 [INFO] Waiting for Sessions to complete before exiting...
19:58:11.34 [INFO] Waiting for shutdown of: ["scheduler_service_session", "store_gc_service_session", "pants_run_2024_01_25_19_56_44_326_8e9fc3c653014b43bf40a975e4ad5beb", "pants_run_2024_01_25_19_57_47_372_6333bcb072a34666bc8a66e3041da35d"]
19:58:11.34 [INFO] Shutdown completed: "scheduler_service_session"
19:58:11.34 [INFO] Shutdown completed: "store_gc_service_session"
19:58:11.34 [INFO] Shutdown completed: "pants_run_2024_01_25_19_56_44_326_8e9fc3c653014b43bf40a975e4ad5beb"
19:58:11.34 [INFO] Shutdown completed: "pants_run_2024_01_25_19_57_47_372_6333bcb072a34666bc8a66e3041da35d"
19:58:11.34 [INFO] Exiting pantsd
w
Oh, that doesnt' sound/look/feel right. The daemon is re-initializing on each change What's your BUILD file look like?
s
python_sources()
and
Copy code
local_environment(
    name="linux_x86",
    compatible_platforms=["linux_x86_64"],
    fallback_environment="docker",
)

docker_environment(
    name="docker",
    platform="linux_x86_64",
    image="python:3.10.12-slim-bullseye",
)
at the root
Currently there are some warnings that get printed about not being able to infer a couple imports but those have been there for a while
b
Can you share your pants.toml too?
s
Copy code
[GLOBAL]
pants_version = "2.18.1"
backend_packages = [
  "pants.backend.python",
  "pants.backend.docker",
  "pants.backend.python.lint.flake8",
  "pants.backend.python.lint.black",
]

[anonymous-telemetry]
enabled = false

[python]
enable_resolves = true
interpreter_constraints = ["CPython==3.10.*"]

[environments-preview.names]
linux_x86 = "//:linux_x86"
docker = "//:docker"

[python.resolves]
python-default = "3rdparty/python/default.lock"

[python-infer]
string_imports = true

[python-repos]
indexes.add = ["https://%(env.FURY_AUTH)s:@pypi.fury.io/amlg/"]

[debugpy]
interpreter_constraints = ["CPython==3.10.*"]
install_from_resolve = "python-default"

[pytest]
install_from_resolve = "python-default"
requirements = [
  "//3rdparty/python:reqs-dev#pytest",
  "//3rdparty/python:reqs-dev#freezegun",
  "//3rdparty/python:reqs-dev#pytest-mock",
  "//3rdparty/python:reqs-dev#responses",
  "//3rdparty/python:reqs-dev#pytest-cov",
  "//3rdparty/python:reqs-dev#pytest-django",
]
execution_slot_var = "PANTS_EXECUTION_SLOT"

[export]
py_editable_in_resolve = ["python-default"]

[black]
interpreter_constraints = ["CPython==3.10.*"]
install_from_resolve = "python-default"
requirements = ["//3rdparty/python:reqs-dev#black"]

[flake8]
install_from_resolve = "python-default"
requirements = ["//3rdparty/python:reqs-dev#flake8"]

[docker]
# build_args = ['FURY_AUTH=%(env.FURY_AUTH)s', 'INSTALL_DEV=1']
tailor = false
build_args = ["PROJECT=i-amlg-dev", "DEFAULT_TAG=dummy"]
# <https://www.pantsbuild.org/docs/docker#docker-authentication>
env_vars = ["DOCKER_CONFIG=%(homedir)s/.docker","DOCKER_DEFAULT_PLATFORM=linux/amd64"]
tools = [
  "docker-credential-gcloud",
  "dirname",
  "readlink",
  "python3",
  # These may be necessary if using Pyenv-installed Python.
  "cut",
  "sed",
  "bash",
]

[repl]
shell = "ipython"

[subprocess-environment]
env_vars = [
  'GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1',
  'GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1',
  'GRPC_BUILD_WITH_BORING_SSL_ASM=',
]

[pex-cli]
version = "v2.1.159"
known_versions =[
    'v2.1.159|macos_arm64|83c3090938b4d276703864c34ba50bcb3616db0663c54b56dd0521a668d9555f|3671772',
    'v2.1.159|macos_x86_64|83c3090938b4d276703864c34ba50bcb3616db0663c54b56dd0521a668d9555f|3671772',
    'v2.1.159|linux_x86_64|83c3090938b4d276703864c34ba50bcb3616db0663c54b56dd0521a668d9555f|3671772',
    'v2.1.159|linux_arm64|83c3090938b4d276703864c34ba50bcb3616db0663c54b56dd0521a668d9555f|3671772',
]
IDK why it's saying "saw filesystem changes covered by invalidation globs". pantsd_invalidation_globs isn't set and it seems like the default is 
[]
?
w
Couple more followups: • Is it any file, or that one specifically? • Is it any goal? Or just 
run
(e.g. fmt/test/lint/fix?) • Is it a problem on either 2.18.0 or the new 2.19's?
s
This happens with other goals, I can reproduce with 
test
. I'm running 2.18 but I tried downgrading to 2.17 and experienced the same problem. It also seems to be triggered by changing any file. I notice logs like 
17:52:59.28 [DEBUG] Dependency Snapshot(!*.pyc, !__pycache__/, .gitignore, pants.toml, pants.toml/**, project_1/src/, project_2/src/) of Some("Snapshot") changed.
Is this snapshot supposed to be all of my source code? (FYI I changed some directory names)
This line always precedes
Copy code
17:52:59.28 [DEBUG] computed 1 nodes in 0.016150 seconds. there are 385 total nodes.
17:52:59.28 [ERROR] saw filesystem changes covered by invalidation globs: SnapshotDiff(our_unique_files=(), our_unique_dirs=(), their_unique_files=(), their_unique_dirs=(), changed_files=('recommended-for-you/src/recommended_for_you/listener.py',)). terminating the daemon.
17:52:59.90 [ERROR] service failure for <pants.pantsd.service.scheduler_service.SchedulerService object at 0x7f12f25b2e20>.
h
Do you have in-repo Pants plugins?
s
I don't
w
Just as a sanity, are you able to re-clone your project to another directory and run it from scratch again? Just want to make sure there isn't some weird caching issue. Might be worth wiping out the pants folder, cache folders, etc
s
I just double checked - I don't actually run into this issue on my Mac. It's only one my (remote) linux machine
If I re-clone on linux the issue persists
Running 
rm -fr .pids/ .pants.d/ ~/.cache/pants/ ~/.cache/nce/
doesn't help
On my mac I see
Copy code
15:43:56.46 [DEBUG] notify invalidating {"archipelago/src/archipelago/__main__.py"} because of Modify(Data(Content))
15:43:56.46 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"archipelago/src/archipelago/__main__.py"}
15:43:56.46 [DEBUG] notify invalidating {"archipelago/src/archipelago/__main__.py"} because of Modify(Data(Content))
15:43:56.46 [INFO] notify invalidation: cleared 1 and dirtied 24 nodes for: {"archipelago/src/archipelago/__main__.py"}
15:43:56.91 [DEBUG] Launching 1 roots (poll=true).
15:43:57.36 [INFO] Extending leases
15:43:57.38 [INFO] Done extending leases
but on linux
Copy code
23:47:34.02 [DEBUG] notify invalidating {"archipelago/src/archipelago/__main__.py"} because of Modify(Data(Any))
23:47:34.02 [INFO] notify invalidation: cleared 1 and dirtied 26 nodes for: {"archipelago/src/archipelago/__main__.py"}
23:47:34.02 [DEBUG] notify invalidating {"archipelago/src/archipelago/__main__.py"} because of Modify(Data(Any))
23:47:34.02 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"archipelago/src/archipelago/__main__.py"}
23:47:34.02 [DEBUG] notify invalidating {"archipelago/src/archipelago/__main__.py"} because of Modify(Data(Any))
23:47:34.02 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"archipelago/src/archipelago/__main__.py"}
23:47:34.02 [DEBUG] notify invalidating {"archipelago/src/archipelago/__main__.py"} because of Modify(Data(Any))
23:47:34.02 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"archipelago/src/archipelago/__main__.py"}
23:47:34.02 [DEBUG] Dependency DigestFile(archipelago/src/archipelago/__main__.py) of Some("Snapshot(!*.pyc, !__pycache__/, .gitignore, archipelago/src, archipelago/src/**, pants.toml, pants.toml/**, [[[[ I deleted a bunch of directories in here. Nothing confidential, I just didn't mean to share them ]]]]])") changed.
23:47:34.03 [DEBUG] notify invalidating {"archipelago/src/archipelago", "archipelago/src/archipelago/__main__.py"} because of Access(Close(Write))
23:47:34.03 [INFO] notify invalidation: cleared 2 and dirtied 425 nodes for: {"archipelago/src/archipelago", "archipelago/src/archipelago/__main__.py"}
23:47:34.03 [DEBUG] notify invalidating {"archipelago/src/archipelago", "archipelago/src/archipelago/__main__.py"} because of Access(Close(Write))
23:47:34.03 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"archipelago/src/archipelago", "archipelago/src/archipelago/__main__.py"}
23:47:34.03 [DEBUG] Dependency Snapshot(!*.pyc, !__pycache__/, .gitignore, archipelago/src, archipelago/src/**, pants.toml, pants.toml/**, [[[[ I deleted a bunch of directories in here. Nothing confidential, I just didn't mean to share them ]]]]] ) of Some("Snapshot") changed.
23:47:34.03 [DEBUG] computed 1 nodes in 0.357482 seconds. there are 2884 total nodes.
23:47:34.03 [ERROR] saw filesystem changes covered by invalidation globs: SnapshotDiff(our_unique_files=(), our_unique_dirs=(), their_unique_files=(), their_unique_dirs=(), changed_files=('archipelago/src/archipelago/__main__.py',)). terminating the daemon.
23:47:34.53 [ERROR] service failure for <pants.pantsd.service.scheduler_service.SchedulerService object at 0x7fca983d9be0>.
My mac doesn't have the debug line
Copy code
23:47:34.02 [DEBUG] Dependency DigestFile(archipelago/src/archipelago/__main__.py) of Some("Snapshot(!*.pyc, !__pycache__/, .gitignore, archipelago/src, archipelago/src/**, pants.toml, pants.toml/**, [[[[ I deleted a bunch of directories in here. Nothing confidential, I just didn't mean to share them ]]]]])") changed.
also "notify invalidation: cleared 2 and dirtied 425 nodes" is strange. Seems like its doing a lot
If i try deleting some directories I can sometimes get a pants on linux that doesn't break but honestly it feels non deterministic. It works maybe 1 time out of 20 attempts
w
Deleting which directories?
s
source code
w
Hmm, and predominantly on linux. Its weird that it happens on python files - we've seen something in the way back past about temporary files (locks and stuff) causing grief, and I think it might have been related to pants_ignore or gitignore
Just as a quick check, is there any difference in your global gitignores on each machine that might cause something to reload?
Feels like no, since you're just saving a python file
s
I tried setting
Copy code
pants_ignore_use_gitignore = false
as deleting my .gitignore seemed to make it work sometimes kinda
but no luck
w
If there is any way to make a repro repo, I think it'd be a lot easier to test on. I've never seen this in the wild, myself - either on my mac or linux devices. I was wondering if there was something in your pants.toml or BUILD files which could cause it, but unless it's something iwth docker or the environments
?
s
It's a repo at work so I can't share as is.
w
Yep, understood - have you tried running with debug mode enabled to see if anything comes out?
``pants -ldebug whatever``
s
Yep
The logs above have a bunch of debug info
I'll look at the output of the commands more carefully though. debug is pretty verbose
In fact, if I just look at 
pants.d/pants.log
I can see it crash when I save a file On mac I don't see anything like
Copy code
01:01:13.49 [DEBUG] Dependency Snapshot(!*.pyc, !__pycache__/, .gitignore, archipelago/src, archipelago/src/**, capstan
w
Have you made a GH ticket for this? I wouldn't want to lose this - in case
s
I haven't
will do
Given 
[ERROR] service failure for <pants.pantsd.service.scheduler_service.SchedulerService object at 0x7fca983d9be0>
I was thinking I could try
Copy code
PANTS_SOURCE=../pants
and setting a break point somehow
Is there a sanity test I can run, like
Copy code
PANTS_FROM_SOURCE=../pants pants --version
to make sure its working?
Maybe I can run 
../pants/pants ...
. Looks like you have to set 
--no-pantsd
if you want to use 
PANTS_DEBUG=1
though. So no idea how i'd debug this
lol I can't reproduce the issues with 
../pants/pants -ldebug run archipelago/src/archipelago/__main__.py
With 
../pants/pants -ldebug run archipelago/src/archipelago/__main__.py
when I change a file I get
Copy code
01:47:27.89 [DEBUG] notify invalidating {"archipelago/src/archipelago/__main__.py"} because of Modify(Data(Any))
01:47:27.89 [INFO] notify invalidation: cleared 1 and dirtied 24 nodes for: {"archipelago/src/archipelago/__main__.py"}
01:47:27.89 [DEBUG] notify invalidating {"archipelago/src/archipelago/__main__.py"} because of Modify(Data(Any))
01:47:27.89 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"archipelago/src/archipelago/__main__.py"}
01:47:27.89 [DEBUG] notify invalidating {"archipelago/src/archipelago/__main__.py"} because of Modify(Data(Any))
01:47:27.89 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"archipelago/src/archipelago/__main__.py"}
01:47:27.89 [DEBUG] notify invalidating {"archipelago/src/archipelago/__main__.py"} because of Modify(Data(Any))
01:47:27.89 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"archipelago/src/archipelago/__main__.py"}
01:47:27.89 [DEBUG] notify invalidating {"archipelago/src/archipelago/__main__.py", "archipelago/src/archipelago"} because of Access(Close(Write))
01:47:27.89 [INFO] notify invalidation: cleared 1 and dirtied 423 nodes for: {"archipelago/src/archipelago/__main__.py", "archipelago/src/archipelago"}
01:47:27.89 [DEBUG] notify invalidating {"archipelago/src/archipelago", "archipelago/src/archipelago/__main__.py"} because of Access(Close(Write))
01:47:27.89 [INFO] notify invalidation: cleared 0 and dirtied 0 nodes for: {"archipelago/src/archipelago", "archipelago/src/archipelago/__main__.py"}
and 0 errors
I can simplify the setup to basically a directory containing a 
pants.toml
file with nothing set except the version and a 
.env
file. For whatever reason 
pants
modifies my 
sys.path
to include the 
PYTHONPATH
in the 
.env
file only one this machine (and other "remote" machines we spin up)
Unfortunately if I try to create a debian 10 docker image I can't reproduce the problem. The only reliable way to reproduce this is to spin up a google compute engine instance with 
projects/debian-cloud/global/images/family/debian-10
I suspect this is another shebang bug and somehow the -sE is getting cut off
4 Views
Open in Slack
PreviousNext
Powered by