pantsd dies Hi, I am doing some performance tunin...
# general
f
pantsd dies Hi, I am doing some performance tuning of pants. First off the list is that pantsd does not hang around. It dies straight away.
Copy code
~/pro/git/project 
❯ rm ../../.pants.d/workdir/pants.log   

~/pro/git/project 
❯ pants list ::                         
09:41:52.68 [INFO] (pe_nailgun::nailgun_pool) Initializing Nailgun pool for 28 processes...
09:41:55.12 [INFO] (pe_nailgun::nailgun_pool) Initializing Nailgun pool for 28 processes...
.......>8....... snipped output

~/pro/git/project 
❯ cat   ../../.pants.d/workdir/pants.log
09:41:59.04 [WARN] (watch) File watcher exiting with: The watcher was shut down.

~/pro/git/project 
❯
This happens to all developers - so is not a local workstation problem ? How do we go about debugging this
This is the DEBUG output
Copy code
❯ pants list ::                      
10:04:01.86 [DEBUG] (engine::externs::interface) File handle limit is: 1048575
10:04:01.87 [INFO] (pe_nailgun::nailgun_pool) Initializing Nailgun pool for 28 processes...
10:04:01.87 [DEBUG] (engine::context) Using [cache::CommandRunner { inner: bounded::CommandRunner { inner: SwitchedCommandRunner { .. }, .. }, .. }, cache::CommandRunner { inner: bounded::CommandRunner { inner: SwitchedCommandRunner { .. }, .. }, .. }] for process execution.
10:04:03.64 [DEBUG] (engine::scheduler) Launching 1 roots (poll=false).
10:04:03.64 [DEBUG] (pants.engine.internals.scheduler) computed 1 nodes in 0.007201 seconds. there are 10 total nodes.
10:04:03.64 [DEBUG] (engine::scheduler) Launching 1 roots (poll=false).
10:04:03.64 [DEBUG] (workunit_store) Starting: Finding the `bash` binary
10:04:03.64 [DEBUG] (workunit_store) Starting: Prepare environment for running PEXes
10:04:03.65 [DEBUG] (workunit_store) Starting: pants.core.util_rules.search_paths.validate_search_paths
10:04:03.65 [DEBUG] (workunit_store) Completed: pants.core.util_rules.search_paths.validate_search_paths
10:04:03.65 [DEBUG] (workunit_store) Completed: Prepare environment for running PEXes
10:04:03.65 [DEBUG] (workunit_store) Starting: Scheduling: Searching for `bash` on PATH=/usr/bin:/bin:/usr/local/bin:/opt/homebrew/bin
10:04:03.65 [DEBUG] (workunit_store) Starting: acquire_command_runner_slot
10:04:03.65 [DEBUG] (workunit_store) Completed: acquire_command_runner_slot
10:04:03.65 [DEBUG] (process_execution::bounded) Running Searching for `bash` on PATH=/usr/bin:/bin:/usr/local/bin:/opt/homebrew/bin under semaphore with concurrency id: 1, and concurrency: 1
10:04:03.65 [DEBUG] (workunit_store) Starting: Searching for `bash` on PATH=/usr/bin:/bin:/usr/local/bin:/opt/homebrew/bin
10:04:03.65 [DEBUG] (workunit_store) Starting: setup_sandbox
10:04:03.65 [DEBUG] (store::local) /Users/user1/.cache/pants/lmdb_store/immutable/files -> /private/var/folders/jl/1sbhkb1s543098lp7zl1gq9r0000gp/T hardlinkable: true
10:04:03.65 [DEBUG] (workunit_store) Completed: setup_sandbox
10:04:03.66 [DEBUG] (process_execution::local) spawned local process as Some(93246) for Process { argv: ["./find_binary.sh", "bash"], env: {"PATH": "/usr/bin:/bin:/usr/local/bin:/opt/homebrew/bin"}, working_directory: None, input_digests: InputDigests { complete: DirectoryDigest { digest: Digest { hash: Fingerprint<fc33a6ab65993fa57d52edbff44d8a8a89faa2ec0669950d238b481da5caf15c>, size_bytes: 91 }, tree: "Some(..)" }, nailgun: DirectoryDigest { digest: Digest { hash: Fingerprint<e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855>, size_bytes: 0 }, tree: "Some(..)" }, inputs: DirectoryDigest { digest: Digest { hash: Fingerprint<fc33a6ab65993fa57d52edbff44d8a8a89faa2ec0669950d238b481da5caf15c>, size_bytes: 91 }, tree: "Some(..)" }, immutable_inputs: {}, use_nailgun: {} }, output_files: {}, output_directories: {}, timeout: None, execution_slot_variable: None, concurrency_available: 0, description: "Searching for `bash` on PATH=/usr/bin:/bin:/usr/local/bin:/opt/homebrew/bin", level: Debug, append_only_caches: {}, jdk_home: None, cache_scope: PerRestartSuccessful, execution_environment: ProcessExecutionEnvironment { name: None, platform: Macos_arm64, strategy: Local }, remote_cache_speculation_delay: 0ns, attempt: 0 }
10:04:03.78 [DEBUG] (workunit_store) Completed: Searching for `bash` on PATH=/usr/bin:/bin:/usr/local/bin:/opt/homebrew/bin
10:04:03.78 [DEBUG] (workunit_store) Completed: Scheduling: Searching for `bash` on PATH=/usr/bin:/bin:/usr/local/bin:/opt/homebrew/bin
10:04:03.78 [DEBUG] (workunit_store) Starting: Scheduling: Test binary /opt/homebrew/bin/bash.
10:04:03.78 [DEBUG] (workunit_store) Starting: Scheduling: Test binary /bin/bash.
10:04:03.78 [DEBUG] (workunit_store) Starting: acquire_command_runner_slot
10:04:03.78 [DEBUG] (workunit_store) Completed: acquire_command_runner_slot
10:04:03.78 [DEBUG] (process_execution::bounded) Running Test binary /bin/bash. under semaphore with concurrency id: 2, and concurrency: 1
10:04:03.78 [DEBUG] (workunit_store) Starting: Test binary /bin/bash.
10:04:03.78 [DEBUG] (workunit_store) Starting: setup_sandbox
10:04:03.78 [DEBUG] (workunit_store) Completed: setup_sandbox
10:04:03.79 [DEBUG] (workunit_store) Starting: acquire_command_runner_slot
10:04:03.79 [DEBUG] (workunit_store) Completed: acquire_command_runner_slot
10:04:03.79 [DEBUG] (process_execution::bounded) Running Test binary /opt/homebrew/bin/bash. under semaphore with concurrency id: 3, and concurrency: 1
10:04:03.79 [DEBUG] (workunit_store) Starting: Test binary /opt/homebrew/bin/bash.
10:04:03.79 [DEBUG] (workunit_store) Starting: setup_sandbox
10:04:03.79 [DEBUG] (workunit_store) Completed: setup_sandbox
10:04:03.79 [DEBUG] (process_execution::local) spawned local process as Some(93247) for Process { argv: ["/bin/bash", "--version"], env: {}, working_directory: None, input_digests: InputDigests { complete: DirectoryDigest { digest: Digest { hash: Fingerprint<e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855>, size_bytes: 0 }, tree: "Some(..)" }, nailgun: DirectoryDigest { digest: Digest { hash: Fingerprint<e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855>, size_bytes: 0 }, tree: "Some(..)" }, inputs: DirectoryDigest { digest: Digest { hash: Fingerprint<e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855>, size_bytes: 0 }, tree: "Some(..)" }, immutable_inputs: {}, use_nailgun: {} }, output_files: {}, output_directories: {}, timeout: None, execution_slot_variable: None, concurrency_available: 0, description: "Test binary /bin/bash.", level: Debug, append_only_caches: {}, jdk_home: None, cache_scope: PerRestartAlways, execution_environment: ProcessExecutionEnvironment { name: None, platform: Macos_arm64, strategy: Local }, remote_cache_speculation_delay: 0ns, attempt: 0 }
....
Copy code
10:04:03.79 [DEBUG] (process_execution::local) spawned local process as Some(93248) for Process { argv: ["/opt/homebrew/bin/bash", "--version"], env: {}, working_directory: None, input_digests: InputDigests { complete: DirectoryDigest { digest: Digest { hash: Fingerprint<e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855>, size_bytes: 0 }, tree: "Some(..)" }, nailgun: DirectoryDigest { digest: Digest { hash: Fingerprint<e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855>, size_bytes: 0 }, tree: "Some(..)" }, inputs: DirectoryDigest { digest: Digest { hash: Fingerprint<e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855>, size_bytes: 0 }, tree: "Some(..)" }, immutable_inputs: {}, use_nailgun: {} }, output_files: {}, output_directories: {}, timeout: None, execution_slot_variable: None, concurrency_available: 0, description: "Test binary /opt/homebrew/bin/bash.", level: Debug, append_only_caches: {}, jdk_home: None, cache_scope: PerRestartAlways, execution_environment: ProcessExecutionEnvironment { name: None, platform: Macos_arm64, strategy: Local }, remote_cache_speculation_delay: 0ns, attempt: 0 }
10:04:03.79 [DEBUG] (workunit_store) Completed: Test binary /bin/bash.
10:04:03.79 [DEBUG] (workunit_store) Completed: Scheduling: Test binary /bin/bash.
10:04:03.80 [DEBUG] (workunit_store) Completed: Test binary /opt/homebrew/bin/bash.
10:04:03.80 [DEBUG] (workunit_store) Completed: Scheduling: Test binary /opt/homebrew/bin/bash.
10:04:03.80 [DEBUG] (workunit_store) Completed: Finding the `bash` binary
10:04:03.80 [DEBUG] (workunit_store) Starting: pants.backend.python.util_rules.pex.build_pex
10:04:03.81 [DEBUG] (workunit_store) Starting: pants.core.util_rules.external_tool.download_external_tool
10:04:03.81 [DEBUG] (workunit_store) Starting: Extracting an archive file
10:04:03.81 [DEBUG] (workunit_store) Completed: Extracting an archive file
10:04:03.81 [DEBUG] (workunit_store) Completed: pants.core.util_rules.external_tool.download_external_tool
10:04:03.81 [DEBUG] (workunit_store) Starting: Scheduling: Resolving plugins: python-gitlab, tomlkit
10:04:03.81 [DEBUG] (workunit_store) Completed: Scheduling: Resolving plugins: python-gitlab, tomlkit
10:04:03.81 [DEBUG] (workunit_store) Completed: pants.backend.python.util_rules.pex.build_pex
10:04:03.81 [DEBUG] (workunit_store) Starting: Scheduling: Extracting plugin locations
10:04:03.81 [DEBUG] (workunit_store) Starting: acquire_command_runner_slot
10:04:03.81 [DEBUG] (workunit_store) Completed: acquire_command_runner_slot
10:04:03.81 [DEBUG] (process_execution::bounded) Running Extracting plugin locations under semaphore with concurrency id: 4, and concurrency: 1
10:04:03.81 [DEBUG] (workunit_store) Starting: Extracting plugin locations
10:04:03.81 [DEBUG] (workunit_store) Starting: setup_sandbox
10:04:03.81 [DEBUG] (workunit_store) Completed: setup_sandbox
10:04:03.82 [DEBUG] (process_execution::local) spawned local process as Some(93249) for Process { argv: ["./pants_plugins.pex_pex_shim.sh", "-c", "import os, site; print(os.linesep.join(site.getsitepackages()))"], env: {}, working_directory: None, input_digests: InputDigests { complete: DirectoryDigest { digest: Digest { hash: Fingerprint<2a573cd524795bd84f121edcd4e2d3037ea55be4e6b2e25bf12fd111fbc2c0ef>, size_bytes: 311 }, tree: "Some(..)" }, nailgun: DirectoryDigest { digest: Digest { hash: Fingerprint<e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855>, size_bytes: 0 }, tree: "Some(..)" }, inputs: DirectoryDigest { digest: Digest { hash: Fingerprint<2a573cd524795bd84f121edcd4e2d3037ea55be4e6b2e25bf12fd111fbc2c0ef>, size_bytes: 311 }, tree: "Some(..)" }, immutable_inputs: {}, use_nailgun: {} }, output_files: {}, output_directories: {}, timeout: None, execution_slot_variable: None, concurrency_available: 0, description: "Extracting plugin locations", level: Debug, append_only_caches: {CacheName("pex_root"): RelativePath(".cache/pex_root"), CacheName("python_build_standalone"): RelativePath(".python-build-standalone")}, jdk_home: None, cache_scope: PerRestartSuccessful, execution_environment: ProcessExecutionEnvironment { name: None, platform: Macos_arm64, strategy: Local }, remote_cache_speculation_delay: 0ns, attempt: 0 }
10:04:03.94 [DEBUG] (workunit_store) Completed: Extracting plugin locations
10:04:03.94 [DEBUG] (workunit_store) Completed: Scheduling: Extracting plugin locations
10:04:03.94 [DEBUG] (pants.engine.internals.scheduler) computed 1 nodes in 0.296573 seconds. there are 96 total nodes.
10:04:04.29 [DEBUG] (engine::externs::interface) File handle limit is: 1048575
10:04:04.30 [INFO] (pe_nailgun::nailgun_pool) Initializing Nailgun pool for 28 processes...
10:04:04.30 [DEBUG] (engine::context) Using [cache::CommandRunner { inner: bounded::CommandRunner { inner: SwitchedCommandRunner { .. }, .. }, .. }, cache::CommandRunner { inner: bounded::CommandRunner { inner: SwitchedCommandRunner { .. }, .. }, .. }] for process execution.
10:04:08.19 [DEBUG] (pants.init.specs_calculator) specs are: Specs(includes=RawSpecs(description_of_origin='CLI arguments', address_literals=(), file_literals=(), file_globs=(), dir_literals=(), dir_globs=(), recursive_globs=(RecursiveGlobSpec(directory='lib/int_comp'),), ancestor_globs=(), unmatched_glob_behavior=<GlobMatchErrorBehavior.error: 'error'>, filter_by_global_options=True, from_change_detection=False), ignores=RawSpecs(description_of_origin='CLI arguments', address_literals=(), file_literals=(), file_globs=(), dir_literals=(), dir_globs=(), recursive_globs=(), ancestor_globs=(), unmatched_glob_behavior=<GlobMatchErrorBehavior.error: 'error'>, filter_by_global_options=False, from_change_detection=False))
10:04:08.19 [DEBUG] (pants.init.specs_calculator) changed_options are: ChangedOptions(since=None, diffspec=None, dependents=<DependentsOption.NONE: 'none'>)
10:04:08.20 [DEBUG] (engine::scheduler) Launching 1 roots (poll=false).
10:04:08.21 [DEBUG] (pants.engine.internals.scheduler) computed 1 nodes in 0.008745 seconds. there are 10 total nodes.
10:04:08.21 [DEBUG] (engine::scheduler) Launching 1 roots (poll=false).
10:04:08.21 [DEBUG] (pants.engine.internals.scheduler) computed 1 nodes in 0.000434 seconds. there are 15 total nodes.
10:04:08.21 [DEBUG] (engine::scheduler) Launching 1 roots (poll=false).
10:04:08.21 [DEBUG] (pants.engine.internals.scheduler) computed 1 nodes in 0.000135 seconds. there are 15 total nodes.
10:04:08.21 [DEBUG] (pants.init.engine_initializer) requesting <class 'pants.backend.project_info.list_targets.List'> to satisfy execution of `list` goal
10:04:08.21 [DEBUG] (engine::scheduler) Launching 1 roots (poll=false).
10:04:08.21 [DEBUG] (workunit_store) Completed: acme.corelib_color_range_tests.synthetic_targets.embed_corelib_color_range_synthetic_targets
10:04:08.22 [DEBUG] (workunit_store) Completed: Find targets from input specs

.....>8..........>8..........>8.....

10:04:08.22 [DEBUG] (workunit_store) Completed: `list` goal
10:04:08.22 [DEBUG] (pants.engine.internals.scheduler) computed 1 nodes in 0.010700 seconds. there are 148 total nodes.
the pants.log does not have any more detail beyond just
Copy code
10:04:08.20 [WARN] (watch) File watcher exiting with: The watcher was shut down.
h
I assume pantsd's memory consumption is not excessive?
And this restart happens in two consecutive runs of the same command with no edits in between?
f
Yes - every time - memory is neglibile (all developers have ~ 32 and 64G - build servers are 128 - 156G )
mac & linux has same behaviour
@happy-kitchen-89482 - did you have any debug suggestions ?
h
Are you setting this option anywhere? You can see its current value with
pants help-advanced global | grep -A 20 pantsd-invalidation-globs
If those globs match anything that changes during a run, pantsd will restart
f
Copy code
pantsd_invalidation_globs = [
   "lib/3rdparty/python/requirements-lock.json"
]
Copy code
[GLOBAL]
pants_version = "2.22.0"
level = "debug"


pythonpath = ["%(buildroot)s/tools/python/pants-plugins"]

backend_packages = [
  'pants.backend.python',
  "pants.backend.python.mixed_interpreter_constraints",
  "pants.backend.experimental.python", # provides vcs_version
  'pants.backend.python.lint.black',
  'pants.backend.python.lint.isort',
  'pants.backend.python.lint.bandit', # Bandit: security linter
  'pants.backend.python.lint.black', # Black: code formatter
  'pants.backend.python.lint.docformatter', # Docformatter: docstring formatter
  'pants.backend.python.lint.flake8', # Flake8: style and bug linter
  'pants.backend.python.lint.isort', # isort: import statement formatter
  'pants.backend.python.lint.pylint', # Pylint: style and bug linter
  'pants.backend.python.lint.yapf', # Yapf: code formatter
  'pants.backend.python.lint.autoflake', # Autoflake: remove unused imports
  'pants.backend.docker.lint.hadolint', # docker linting
  'pants.backend.python.lint.pyupgrade',
  'pants.backend.experimental.helm',
  'pants.backend.experimental.adhoc',
  'pants.backend.python.typecheck.mypy',
  'pants.backend.docker',
  'pants.backend.shell',
  'pants.backend.experimental.adhoc',
  'pants.backend.plugin_development',
  'acme.publishing',
  'acme.simulation_tests'
]

plugins = [
  "python-gitlab",
  "tomlkit",
]

pants_ignore = [ ".*/", "/dist/", "__pycache__", "!.semgrep/", "!.github/", "!.config/", "!.git/", "!.cargo/"]

pants_ignore_use_gitignore = false

# default - disable the pants python matrix
tag = ["-test_py3_matrix"]

pantsd_invalidation_globs = [
   "lib/3rdparty/python/requirements-lock.json"
]

build_file_prelude_globs = ["tools/python/pants-plugins/acme/publishing/macro_versioning.py"]
concurrent = true
show_log_target = true

[python]
enable_resolves = true
default_resolve = "python-no-default-is-available"
# default to using 3.10 for sources and tests
# run pants test --tag='test_all_py3' to test against all version
interpreter_constraints = ['CPython>=3.10,<3.11']

[python.resolves]
python-knohh = "lib/3rdparty/python/requirements-knohh-lock.json"
python-alaf = "lib/3rdparty/python/requirements-alaf-lock.json"
python-koppa = "lib/3rdparty/python/requirements-koppa-lock.json"
python-koppa-poc = "lib/3rdparty/python/requirements-koppa-poc-lock.json"
python-j2 = "lib/3rdparty/python/requirements-j2-lock.json"
python-inventory = "lib/3rdparty/python/requirements-inventory-lock.json"
python-iac = "lib/3rdparty/python/requirements-iac-lock.json"
python-cicd = "lib/3rdparty/python/requirements-cicd-lock.json"
python-pants-plugins = "lib/3rdparty/python/requirements-pants-plugins-lock.json"
python-full-basis = "lib/3rdparty/python/requirements-full-basis-lock.json"

[python.resolves_to_interpreter_constraints]
python-pants-plugins = [">=3.9,<3.10"]

[python-bootstrap]
# Pants searches for interpreters both on the $PATH and in the `$(pyenv root)/versions` folder.
# If you're using macOS, you may want to leave off the <PATH> entry to avoid using the
# problematic system Pythons. See
# <https://www.pantsbuild.org/docs/python-interpreter-compatibility#changing-the-interpreter-search-path>.
search_path = ["<PYENV>", "<PATH>"]

[python-repos]
# go to issinternal before pypi to pick up acme-only packages
indexes = [
  "<https://acme.issinternal.local/api/pypi/eng/simple>",
  "<https://pypi.org/simple>"
]

[pytest]
install_from_resolve = "python-knohh"
xdist_enabled = true
requirements = [
  "pytest==7.3.1",
  "pytest-custom-exit-code==0.3.0",
  "pytest-cov>=4.1.0",
  "pytest-xdist==3.3.1",
  "pytest-explicit==1.0.1"
]

[generate-lockfiles]
diff = true

[coverage-py]
report = [ "html", "console", "xml" ]

[anonymous-telemetry]
enabled = false

[subprocess-environment]
env_vars.add = ["HDF5_DIR"]

[repl]
shell = "ipython"

[twine]
install_from_resolve = "python-knohh"
requirements = ["twine", "colorama"]
args = ["--verbose",
        "--skip-existing",]

[docker]
# environment variables are bootstraped in the .pants.bootstrap
build_args = [
  "DEFAULT_TAG",
]

build_verbose = true
run_args = [
  "--rm"
]

# full_directory, build_args, default_repository, directory, name, pants, parent_directory, tags, target_repository
# it would be nice to use the fullpath of the image, but alas, it is not available
# you can set the "target_repository" in the BUILD file
# <https://www.pantsbuild.org/2.20/reference/subsystems/docker#default_repository>
default_repository = "{directory}/{name}"


[docker.registries.acme-registry]
address = "registry.corp.acme"
default = true
# build_args, default_repository, directory, name, pants, parent_directory, tags, target_repository
# it would be nice to use the fullpath of the image, but alas, it is not available
# you can set the "target_repository" in the BUILD file
# <https://www.pantsbuild.org/2.20/reference/subsystems/docker#default_repository>

[docker.registries.acme-library-registry]
address = "registry.corp.acme/internal/library"

[cli.alias]
--all-changed = "--changed-since=HEAD --changed-dependents=transitive"
publishable = "--level=error --filter-target-type=archive,docker_image,pex_binary,python_distribution list //::"
h
And I'm assuming that
lib/3rdparty/python/requirements-lock.json
isn't changing when you run
list
Just for sanity, can you check that its
mtime
hasn't changed either?
f
yeah - no it doesn't change, thats a very manual
pants lock ...
.. checking mtime now
Turns out we don't use / have - requirements-lock.json
h
OK, so that should be a red herring
But just in case, try removing that
pantsd_invalidation_globs
entry?