I'm trying to strip the source roots of files that have source roots, however cause I'm using python_requirements() my requirements.txt file is added as a PythonRequirementsFileSources, and when I try to get

StrippedFileSources

it complains:

09:12:58.88 [ERROR] Exception caught: (pants.engine.internals.scheduler.ExecutionError)
  File "/Users/nate/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0rc1_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 246, in run
    engine_result = self._perform_run(goals)
  File "/Users/nate/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0rc1_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 167, in _perform_run
    return self._perform_run_body(goals, poll=False)
  File "/Users/nate/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0rc1_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 184, in _perform_run_body
    return self.graph_session.run_goal_rules(
  File "/Users/nate/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0rc1_py38/lib/python3.8/site-packages/pants/init/engine_initializer.py", line 130, in run_goal_rules
    exit_code = self.scheduler_session.run_goal_rule(
  File "/Users/nate/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0rc1_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 568, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "/Users/nate/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0rc1_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 536, in _raise_on_error
    raise ExecutionError(

Exception message: 1 Exception encountered:

  NoSourceRootError: No source root found for `requirements.txt`. See <https://www.pantsbuild.org/v2.3/docs/source-roots> for how to define source roots.

When I get the regular merged SourceFiles however, there is nothing in the

unrooted

tuple? which seems to be how I would want to detect if a particular file is in a source root or not?

Sorry for delayed response, thanks for following up

Thanks @happy-kitchen-89482 here's the source code:

rooted_source_files = []
    unrooted_files = []
    all_deps = await Get(
        TransitiveTargets, TransitiveTargetsRequest([field_set.address])
    )
    
    for i, tgt in enumerate(all_deps.closure):
        if tgt.has_field(Sources):
            if tgt[Sources].address.spec_path:
                rooted_source_files.append(tgt[Sources])
            else:
                unrooted = unrooted_files.append(tgt[Sources])
    stripped, unstripped = await MultiGet(
        Get(StrippedSourceFiles, SourceFilesRequest(rooted_source_files)),
        Get(SourceFiles, SourceFilesRequest(unrooted_files)),
    )
    <http://logger.info|logger.info>("Source Root Files: %s", stripped.snapshot.files)
    <http://logger.info|logger.info>("Unrooted Files: %s", unstripped.snapshot.files)
    all_files = await Get(SourceFiles, SourceFilesRequest(tuple(tgt[Sources] for tgt in  all_deps.closure if tgt.has_field(Sources))))
    <http://logger.info|logger.info>("All Files: %s", all_files.snapshot.files)
    all_stripped_files = await Get(StrippedSourceFiles, SourceFilesRequest(tuple(tgt[Sources] for tgt in  all_deps.closure if tgt.has_field(Sources))))
    <http://logger.info|logger.info>("All Stripped Files: %s", all_stripped_files.snapshot.files)

and here's the logged info:

11:25:12.88 [INFO] Source Root Files: ('__init__.py', 'app.py')
11:25:12.88 [INFO] Unrooted Files: ('example_requirements.txt',)
11:25:12.88 [INFO] All Files: ('example_requirements.txt', 'test_docker/__init__.py', 'test_docker/app.py')
11:25:12.88 [ERROR] Exception caught: (pants.engine.internals.scheduler.ExecutionError)
  File "/Users/nate/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0rc1_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 246, in run
    engine_result = self._perform_run(goals)
  File "/Users/nate/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0rc1_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 167, in _perform_run
    return self._perform_run_body(goals, poll=False)
  File "/Users/nate/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0rc1_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 184, in _perform_run_body
    return self.graph_session.run_goal_rules(
  File "/Users/nate/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0rc1_py38/lib/python3.8/site-packages/pants/init/engine_initializer.py", line 130, in run_goal_rules
    exit_code = self.scheduler_session.run_goal_rule(
  File "/Users/nate/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0rc1_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 568, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "/Users/nate/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0rc1_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 536, in _raise_on_error
    raise ExecutionError(

Exception message: 1 Exception encountered:

  NoSourceRootError: No source root found for `example_requirements.txt`. See <https://www.pantsbuild.org/v2.3/docs/source-roots> for how to define source roots.

Hey Nate! Hm, the culprit is https://github.com/pantsbuild/pants/pull/9640 Could you please share more about the plugin, specifically what source files you want to be used here? E.g. Python files + Docker files + resources, vs. only Python files, and so on. In the builtin Python backend, we use the helper type

PythonSourceFiles

from

pants.backend.python.util_rules.python_sources

, which under-the-hood uses

for_sources_types=(PythonSources, FilesSources, ...etc)

to allow-list what you want, resulting in leaving off

PythonRequirementsFileSources

Sorry for the rough edge here, and pardon the delay in replying!

Hi @average-australia-85137 just checking in to see if you've found a workaround or we should continue to look into this. It sounds like you might need to filter out that

PythonRequirementsFileSources

target

but I think the source root stripping code is supposed to pass through files that don't have a source root

For that, that source root stripping code needs to gain an entry point for an arbitrary

Sources

subclass to indicate it does not have source roots. Right now, the sole way to do that is to subclass

FilesSources

, but as found in https://github.com/pantsbuild/pants/pull/9640, that creates its own can of worms which was deemed more offensive The fix could be simple: add

has_source_roots: ClassVar[bool]

to

Sources

. Stop special-casing

FilesSources

in the

strip_source_roots

rule. Does that make sense? I agree with your intuition that it's an unsatisfying answer to say "Oh, yeah, you have to pre-filter out the sources before hydrating them". That's a gotcha

Yeah, that sounds like a good fix

K, will put up that fix for 2.4.x. @average-australia-85137 won't be cherry-picked to 2.3.x, but will go into the release this week for 2.4.x. Would that work?

Hi @happy-kitchen-89482 @hundreds-father-404 sorry! Busy week! My use-case here is that I'm making a plugin to build a docker image, which means that we want to copy all transitively dependent source files into the docker context, w/out source roots. It sounds like the changes to 2.4 should fix that? thanks

Gotcha, makes sense. There is a workaround if you don't want to wait for 2.4, which is to filter out that target by type. Right @hundreds-father-404?

It sounds like the changes to 2.4 should fix that? thanks

Yeah, so in 2.4, both

PythonRequirementsFileSources

and

FilesSources

will not have their source roots stripped. And you can set your custom

Sources

to never attempt to be stripped also by setting

uses_source_roots = False

class property on it. This has landed in 2.4.0.dev3 In the meantime, you can set

SourceFilesRequest(for_sources_types=(DockerSources, PythonSources)

, etc to allow-list the sources you do want in the docker image. Or you use

await Get(StrippedPythonSourceFiles, PythonSourceFilesRequest)

from

pants.backend.python.util_rules.python_sources

it also seems like you could do:

if tgt[Sources].address.spec_path:
                rooted_source_files.append(tgt[Sources])

(like above) if you want to find files with roots to strip

That would break if you had something like

build-support/config.json

- it only works with files in your build root without a prefix dir