Hello Getting back to some issues we had trying to onboard o Pants #general

Hello. Getting back to some issues we had trying t...

lively-garden-66504

05/18/2023, 8:23 PM

Hello. Getting back to some issues we had trying to onboard our Go project on pants. We have about 100000 lines of Go code and tailoring the whole project and running

./pants check ::

causes the daemon to take up around 20 GiB of memory and stay around 17 GiB afterwards. Is this an expected level of resource usage? I'm using 2.16.0rc0 and am reserving 64 GiB for the daemon. Otherwise it's default settings.

witty-crayon-22786

05/18/2023, 8:25 PM

that is unexpectedly high. can you run

check

with --stats-memory-summary, and report the result on a ticket?

lively-garden-66504

05/18/2023, 8:38 PM

Sure. tl;dr on some of the bigger ones

Copy code

436080		9085		pants.engine.target.WrappedTarget
  436080		9085		pants.engine.target.WrappedTargetRequest
  456454		1894		(native) pants.backend.go.util_rules.build_pkg_target.setup_build_go_package_target_request
  478350		2126		(native) pants.engine.internals.build_files.maybe_resolve_address
  478350		2126		(native) pants.engine.internals.build_files.resolve_address
  520560		2160		(native) pants.backend.go.util_rules.build_pkg.build_go_package
  520560		2160		(native) pants.backend.go.util_rules.build_pkg.compute_compile_action_id
  613800		2728		(native) pants.engine.internals.graph.resolve_unexpanded_targets
  693357		2877		(native) pants.backend.go.util_rules.sdk.setup_go_sdk_process
  1060380		8121		(native) digest_file
  1451227		1814		(native) snapshot
  2001149		1343		(native) scandir
  2021400		8984		(native) pants.backend.go.util_rules.go_mod.find_owning_go_mod
  2027774		8414		(native) pants.backend.go.util_rules.third_party_pkg.analyze_go_third_party_package
  2044125		9085		(native) pants.engine.internals.graph.resolve_target
  9095065		1669		(native) paths
  54631680		2728		pants.engine.target.UnexpandedTargets
  109056848		3566		pants.engine.addresses.Addresses
  274986319		4405		(native) process

witty-crayon-22786

05/18/2023, 8:40 PM

hm! the

process

one is very interesting. i think that the go backend must be creating some very large … argument lists, perhaps? the Addresses item should be fixed in Pants

2.17.x

👀 1

lively-garden-66504

05/18/2023, 8:43 PM

Argument lists...?

lively-garden-66504

05/18/2023, 8:43 PM

Oh you mean Pants go backend, mb I misunderstood

witty-crayon-22786

05/18/2023, 8:43 PM

yea.

witty-crayon-22786

05/18/2023, 8:44 PM

can you file an issue with that info, and mention that this is related to Go? it seems like it should be straightforward to repro. cc @fast-nail-55400

narrow-vegetable-37489

05/19/2023, 10:49 AM

Seeing the same high memory problem with Go, so I took the liberty of filing an issue: https://github.com/pantsbuild/pants/issues/19053

narrow-vegetable-37489

05/19/2023, 10:49 AM

@lively-garden-66504 feel free to post your

--stats-memory-summary

there as well.

witty-crayon-22786

05/19/2023, 4:26 PM

hm… yours is fairly different. @lively-garden-66504: if you’re able to post yours as well, that would be appreciated

fast-nail-55400

05/19/2023, 5:11 PM

Debug logging would be very useful as well. Run with

-ldebug

fast-nail-55400

05/19/2023, 5:12 PM

Regarding the "large argument list" theory, Pants already tries to avoid such argument lists for source file names when compiling. See https://github.com/pantsbuild/pants/blob/377fd1581ef3be6b921b875ae4ca43d1d5308a91/src/python/pants/backend/go/util_rules/build_pkg.py#L807-L816. So if this is an issue, there is at least a way to maybe avoid.

fast-nail-55400

05/19/2023, 5:13 PM

Seeing the

Process

instances being constructed will be useful.

witty-crayon-22786

05/19/2023, 5:21 PM

Seeing the
Process
instances being constructed will be useful.

yea, definitely for Naveneeth’s case. it sounds like Jonas’ might be something else that i should probably look at a separately.

narrow-vegetable-37489

05/19/2023, 5:55 PM

I'll try to get some debug logs posted in the GH issue either later today or at some point during the weekend.

witty-crayon-22786

05/19/2023, 7:02 PM

looks like Nav filed: https://github.com/pantsbuild/pants/issues/19060: thanks!

witty-crayon-22786

05/19/2023, 7:03 PM

going to resolve as a dupe of #19053 and reference it.

narrow-vegetable-37489

05/19/2023, 9:58 PM

Debug logs added.

fast-nail-55400

05/20/2023, 4:47 AM

Looking at the log, debug level didn't add the information I needed which is weird because the

Process

level is debug. https://github.com/pantsbuild/pants/blob/5b070726eb8228645f523c2d2c6e68a9f85088a3/src/python/pants/backend/go/util_rules/sdk.py#L148 @witty-crayon-22786: Any ideas? Would

-ltrace

print out the

Process

invocations?

witty-crayon-22786

05/20/2023, 4:48 AM

Probably hit the cache for many of them, which won't render the process

fast-nail-55400

05/20/2023, 4:48 AM

ah so they may need to disable cache then for the run?

👍 1

fast-nail-55400

05/20/2023, 4:53 AM

unrelated point: I noticed a lot of messages like

Unable to infer dependency for import path 'fmt' in go_third_party_package at address 'golang#<http://gopkg.in/ini.v1|gopkg.in/ini.v1>'.

in the log. Will file that as a separate minor bug.

witty-crayon-22786

05/20/2023, 5:29 AM

i’ll tackle the

paths

and

scandir

entries… i already know why those are large =/

witty-crayon-22786

05/20/2023, 5:29 AM

have a good weekend!

🖖 1

narrow-vegetable-37489

05/20/2023, 11:47 AM

Ah right! Updated with another gist using

--no-local-cache

and

--no-pantsd

gentle-painting-32087

07/03/2023, 9:50 PM

Hey team, I have a related question but from a different perspective, is it possible to use more memory and speed up the overall processes. I tried to use

pantsd_max_memory_usage

and boost to 64Gb and did not see any significant performance gain. Here are my mem stats(missed the lightweight part):

Copy code

138584                728             pants.engine.internals.mapper.AddressFamilies
  208658                722             (native) pants.engine.internals.synthetic_targets.get_synthetic_targets_spec_paths
  209814                726             (native) pants.engine.internals.graph.find_owners
  210392                728             (native) pants.engine.internals.specs_rules.address_families_from_raw_specs_without_file_owners
  210392                728             (native) pants.engine.internals.specs_rules.addresses_from_raw_specs_without_file_owners
  260400                5425            pants.build_graph.address.MaybeAddress
  273600                199             (native) process
  302976                6312            pants.engine.internals.mapper.AddressFamily
  309696                6452            pants.engine.internals.build_files.OptionalAddressFamily
  309696                6452            pants.engine.internals.synthetic_targets.SyntheticAddressMapsRequest
  309792                12764           pants.engine.internals.build_files.AddressFamilyDir
  309840                6453            pants.engine.internals.synthetic_targets.SyntheticAddressMaps
  326928                12592           pants.source.source_root.SourceRootRequest
  333296                1               pants.engine.target.AllTargets
  368186                1274            (native) pants.engine.internals.specs_rules.addresses_from_owners
  368186                1274            (native) pants.engine.internals.specs_rules.addresses_from_raw_specs_with_only_file_owners
  368764                1276            (native) pants.engine.internals.specs_rules.resolve_addresses_from_raw_specs
  372144                7753            pants.backend.python.dependency_inference.rules.InferConftestDependencies
  378432                7884            pants.backend.python.target_types.PythonFilesGeneratorSettingsRequest
  378432                7884            pants.engine.target.TargetFilesGeneratorSettings
  392304                8173            pants.engine.internals.graph.GenerateFileTargets
  392496                8177            pants.engine.target.SourcesPaths
  392496                8177            pants.engine.target.SourcesPathsRequest
  392592                8179            pants.engine.internals.graph.ResolveTargetGeneratorRequests
  392592                8179            pants.engine.internals.graph.ResolvedTargetGeneratorRequests
  392592                8179            pants.engine.target.GeneratedTargets
  395376                8237            pants.engine.internals.graph._AdaptorAndType
  395376                8237            pants.engine.internals.graph._RequestAdaptorAndType
  395376                8237            pants.engine.internals.parametrize._TargetParametrizationsRequest
  395376                8237            pants.engine.internals.target_adaptor.TargetAdaptorRequest
  398408                1               pants.engine.target.AllUnexpandedTargets
  461272                8237            pants.engine.internals.parametrize._TargetParametrizations
  527168                8237            pants.engine.internals.target_adaptor.TargetAdaptor
  741168                15441           pants.build_graph.address.BuildFileAddress
  741168                15441           pants.build_graph.address.BuildFileAddressRequest
  781200                5425            builtins.Address
  782323                2707            (native) pants.engine.internals.graph.resolve_targets
  782323                2707            (native) pants.engine.internals.graph.resolve_unexpanded_targets
  911400                10850           builtins.AddressInput
  1567825               5425            (native) pants.engine.internals.build_files.maybe_resolve_address
  1567825               5425            (native) pants.engine.internals.build_files.resolve_address
  1586160               33045           pants.core.util_rules.stripped_source_files.StrippedFileName
  1586160               33045           pants.core.util_rules.stripped_source_files.StrippedFileNameRequest
  1586160               33045           pants.source.source_root.OptionalSourceRootsResult
  1586160               33045           pants.source.source_root.SourceRootsResult
  1586160               66090           pants.source.source_root.SourceRootsRequest
  1587408               33071           custom_infer.rules.InferPythonGlobDependencies
  1587408               33071           pants.backend.python.dependency_inference.parse_python_dependencies.ParsePythonDependenciesRequest
  1587408               33071           pants.backend.python.dependency_inference.parse_python_dependencies.ParsedPythonDependencies
  1587408               33071           pants.backend.python.dependency_inference.rules.InferInitDependencies
  1587408               33071           pants.backend.python.dependency_inference.rules.InferPythonImportDependencies
  1587408               33071           pants.backend.python.dependency_inference.rules.ResolvedParsedPythonDependencies
  1587408               33071           pants.backend.python.dependency_inference.rules.ResolvedParsedPythonDependenciesRequest
  1587408               33071           pants.core.util_rules.source_files.SourceFiles
  1587408               33071           pants.core.util_rules.stripped_source_files.StrippedSourceFiles
  1587408               66142           pants.core.util_rules.source_files.SourceFilesRequest
  1587648               33076           pants.backend.python.target_types_rules.PythonValidateDependenciesRequest
  1587648               33076           pants.engine.target.ValidatedDependencies
  1596624               33263           pants.engine.target.HydrateSourcesRequest
  1596624               33263           pants.engine.target.HydratedSources
  1798447               6223            (native) pants.source.source_root.get_source_root
  1824168               6312            (native) pants.engine.internals.build_files.ensure_address_family
  1840641               6369            (native) pants.source.source_root.get_optional_source_root
  1864628               6452            (native) pants.engine.internals.build_files.parse_address_family
  1864628               6452            (native) pants.engine.internals.synthetic_targets.get_synthetic_address_maps
  1957344               40778           pants.backend.python.util_rules.ancestor_files.AncestorFiles
  1957344               40778           pants.backend.python.util_rules.ancestor_files.AncestorFilesRequest
  2240617               7753            (native) pants.backend.python.dependency_inference.rules.infer_python_conftest_dependencies
  2278476               7884            (native) pants.backend.python.target_types_rules.python_files_generator_settings
  2363153               8177            (native) pants.engine.internals.graph.resolve_source_paths
  2363731               8179            (native) pants.engine.internals.graph.resolve_generator_target_requests
  2380493               8237            (native) pants.engine.internals.build_files.find_target_adaptor
  2380493               8237            (native) pants.engine.internals.graph._determine_target_adaptor_and_type
  2390160               49795           pants.engine.target.WrappedTarget
  2390160               49795           pants.engine.target.WrappedTargetRequest
  2492765               8173            (native) pants.engine.internals.graph.generate_file_targets
  2512285               8237            (native) pants.engine.internals.graph.resolve_target_parametrizations
  3452016               71917           pants.backend.python.dependency_inference.module_mapper.PythonModuleOwners
  3452016               71917           pants.backend.python.dependency_inference.module_mapper.PythonModuleOwnersRequest
  3986736               83057           pants.engine.target.ExplicitlyProvidedDependencies
  3986736               132852          pants.engine.target.DependenciesRequest
  4462449               15441           (native) pants.engine.internals.build_files.find_build_file
  5143824               107163          pants.engine.target.InferredDependencies
  7666087               40183           (native) digest_file
  9550005               33045           (native) pants.core.util_rules.stripped_source_files.strip_file_name
  9550005               33045           (native) pants.source.source_root.get_optional_source_roots
  9550005               33045           (native) pants.source.source_root.get_source_roots
  9557519               33071           (native) custom_infer.rules.infer_python_glob_dependencies
  9557519               33071           (native) pants.backend.python.dependency_inference.rules.infer_python_init_dependencies
  9558964               33076           (native) pants.backend.python.target_types_rules.validate_python_dependencies
  10086655              33071           (native) pants.backend.python.dependency_inference.parse_python_dependencies.parse_python_dependencies
  10086655              33071           (native) pants.backend.python.dependency_inference.rules.infer_python_dependencies_via_source
  10086655              33071           (native) pants.backend.python.dependency_inference.rules.resolve_parsed_dependencies
  10086655              33071           (native) pants.core.util_rules.source_files.determine_source_files
  10086655              33071           (native) pants.core.util_rules.stripped_source_files.strip_source_roots
  10145215              33263           (native) pants.engine.internals.graph.hydrate_sources
  11784842              40778           (native) pants.backend.python.util_rules.ancestor_files.find_ancestor_files
  14390755              49795           (native) pants.engine.internals.graph.resolve_dependencies
  14390755              49795           (native) pants.engine.internals.graph.resolve_target
  14402578              11952           (native) scandir
  20031544              2707            pants.engine.target.UnexpandedTargets
  20151744              2708            pants.engine.target.Targets
  21934685              71917           (native) pants.backend.python.dependency_inference.module_mapper.map_module_to_address
  24003473              83057           (native) pants.engine.internals.graph.determine_explicitly_provided_dependencies
  40392772              54061           (native) snapshot
  47780292              10085           (native) paths
  58632664              54502           pants.engine.addresses.Addresses

2 Views

Open in Slack

Previous Next