#16778 Nondeterministic `ImportError: No module named __pants_df_parser` while computing changed targets Issue created by danxmoran (NOTE: There's been a lot of discussion on this in Slack here, but I realized I never filed an issue for it) Describe the bug A few times per day (out of ~hundreds of jobs), our CI jobs will fail when trying to build

dockerfile_parser.pex

. The error always hits at the beginning of goal execution, and affects all our goals (

test

,

lint

,

check

). The error looks like:

13:28:44.08 [INFO] Starting: Resolving plugins: toolchain.pants.plugin==0.20.0
13:28:56.61 [INFO] Completed: Resolving plugins: toolchain.pants.plugin==0.20.0
13:28:57.16 [INFO] [auth-plugin] Successfully loaded Toolchain token from env var 'TOOLCHAIN_AUTH_TOKEN', expiration: 2022-12-20T20:52:09+00:00.
13:29:17.25 [INFO] Starting: Building dockerfile_parser.pex from dockerfile-parser_default.lock
13:29:18.32 [INFO] Canceled: Building dockerfile_parser.pex from dockerfile-parser_default.lock
Exception caught: (pants.engine.internals.scheduler.ExecutionError)
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+gitb75932fb_py38/bin/pants", line 8, in <module>
    sys.exit(main())
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+gitb75932fb_py38/lib/python3.8/site-packages/pants/bin/pants_loader.py", line 115, in main
    PantsLoader.main()
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+gitb75932fb_py38/lib/python3.8/site-packages/pants/bin/pants_loader.py", line 111, in main
    cls.run_default_entrypoint()
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+gitb75932fb_py38/lib/python3.8/site-packages/pants/bin/pants_loader.py", line 93, in run_default_entrypoint
    exit_code = runner.run(start_time)
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+gitb75932fb_py38/lib/python3.8/site-packages/pants/bin/pants_runner.py", line 99, in run
    runner = LocalPantsRunner.create(
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+gitb75932fb_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 159, in create
    specs = calculate_specs(
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+gitb75932fb_py38/lib/python3.8/site-packages/pants/init/specs_calculator.py", line 94, in calculate_specs
    (changed_addresses,) = session.product_request(
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+gitb75932fb_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 577, in product_request
    return self.execute(request)
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+gitb75932fb_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 521, in execute
    self._raise_on_error([t for _, t in throws])
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+gitb75932fb_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 498, in _raise_on_error
    raise ExecutionError(
Exception message: 1 Exception encountered:
Engine traceback:
  in select
  in pants.vcs.changed.find_changed_owners
  in pants.backend.project_info.dependees.find_dependees
  in pants.backend.project_info.dependees.map_addresses_to_dependees
  in pants.engine.internals.graph.resolve_dependencies (src/projects/rhp/operations:rhp-api)
  in pants.backend.docker.util_rules.dependencies.inject_docker_dependencies (src/projects/rhp/operations:rhp-api)
  in pants.backend.docker.subsystems.dockerfile_parser.parse_dockerfile
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/tmp/pants_setup/bootstrap-Linux-x86_64/2.13.0a1+gitb75932fb_py38/lib/python3.8/site-packages/pants/engine/process.py", line 272, in fallible_to_exec_result_or_raise
    raise ProcessExecutionFailure(
pants.engine.process.ProcessExecutionFailure: Process 'Parse Dockerfile.' failed with exit code 1.
stdout:
stderr:
Traceback (most recent call last):
  File "/tmp/pants-sandbox-kVePcL/.cache/pex_root/venvs/e2b00a6620ed72ee63184ca7a64555f5551da8c5/f679563b742633f5edaba61ca7d405616ffc49b3/pex", line 234, in <module>
    runpy.run_module(module_name, run_name="__main__", alter_sys=True)
  File "/opt/python/3.8.12/lib/python3.8/runpy.py", line 203, in run_module
    mod_name, mod_spec, code = _get_module_details(mod_name)
  File "/opt/python/3.8.12/lib/python3.8/runpy.py", line 138, in _get_module_details
    raise error("No module named %s" % mod_name)
ImportError: No module named __pants_df_parser

Pants version • The error started appearing on

PANTS_SHA=5d8a328d72209863986c8959b20305505bc068ba

, and has continued appearing as we've upgraded along the

2.13.x

branch. • As far as I can tell, the error was not appearing on

v2.13.0a0

OS Linux Additional info Through debugging on Slack, we've found: 1.

__pants_df_parser.py

truly isn't in the venv when this error occurs 2. The error doesn't occur if we use

--no-process-cleanup

3. The sandbox PEX created for the dockerfile parser contains the file, even when the pex_root venv does not. Running the PEX creates a venv that contains the file. pantsbuild/pants