https://pantsbuild.org/ logo
#general
Title
# general
w

wonderful-iron-54019

05/18/2020, 7:33 PM
Not sure if a known issue but using a spec-file with the below content in 1.26.0 results in a
Exception message: Both address specs and filesystem specs given. Please use only one type of spec.
happy to file an issue
the rest of the. erro output gives:
Copy code
Exception message: Both address specs and filesystem specs given. Please use only one type of spec.

Address specs: src/subscription_new/BUILD:BUILD
Filesystem specs: 3rdparty/python/requirements.txt, Pipfile, Pipfile.lock, deploy.sh, pants-docker.ini, src/common/dataframe/test/validate_test.py, src/common/dataframe/validate.py, src/conviva_v2/BUILD, src/conviva_v2/config.yml, src/conviva_v2/conviva_daily_backfill_dag.py, src/conviva_v2/conviva_daily_ingestion.py, src/conviva_v2/helpers.py, src/gdpr_requests/dag.py, src/sfmc_import/BUILD, src/sfmc_import/db_migrations/7a4efabb1dda_20200513_add_sendlog_fields.py, src/sfmc_import/schemas/sendlog.py, src/subscription/check_data.py, src/subscription/dynamo_point_in_time.py, src/subscription/sublime_transform.py, src/subscription/subscription_ingest.py, src/subscription/subscription_state_backfill_dag.py, src/subscription/tests/config_test.py, src/subscription/tests/dynamo_test.py, src/subscription/tests/subscription_ingest_test.py, src/subscription/tests/test_check_data.py, src/subscription/tests/test_sublime_tranform.py, src/subscription/tests/test_utils_funcs.py, src/subscription/tests/utils_for_tests.py, src/subscription/utils.py, src/subscription_new/README.md, src/subscription_new/__init__.py, src/subscription_new/config/__init__.py, src/subscription_new/config/config.py, src/subscription_new/config/config.yml, src/subscription_new/dynamo.py, src/subscription_new/dynamo_to_json.py, src/subscription_new/dynamo_to_parquet.py, src/subscription_new/fixtures/reprocess_hive_data.json, src/subscription_new/fixtures/reprocess_hive_data_expected.json, src/subscription_new/fixtures/unittest_state_engine.json, src/subscription_new/job_config_prod.yml, src/subscription_new/job_config_test.yml, src/subscription_new/schema/__init__.py, src/subscription_new/schema/dynamo_subscription_schema.py, src/subscription_new/schema/state_schema.py, src/subscription_new/scripts/migrate_dynamo_est.py, src/subscription_new/scripts/migrate_dynamo_v2.py, src/subscription_new/scripts/seed_start_date.py, src/subscription_new/scripts/subscription_load_table.sql, src/subscription_new/squirlconf/common.yaml, src/subscription_new/squirlconf/int.yaml, src/subscription_new/squirlconf/prod.yaml, src/subscription_new/squirlconf/qa.yaml, src/subscription_new/sublime_transform.py, src/subscription_new/subscription_backfill_dag.py, src/subscription_new/subscription_dag.py, src/subscription_new/subscription_data_check.py, src/subscription_new/subscription_transform.py, src/subscription_new/tests/__init__.py, src/subscription_new/tests/test_check_data.py, src/subscription_new/tests/test_config.py, src/subscription_new/tests/test_dynamo.py, src/subscription_new/tests/test_sublime_tranform.py, src/subscription_new/tests/test_subscription_transform.py, src/subscription_new/tests/test_utils.py, src/subscription_new/tests/utils.py, src/subscription_new/utils.py, src/watch/data_reconciliation/BUILD
not sure why it thinks theres a
build:build
address
h

hundreds-father-404

05/18/2020, 7:40 PM
An issue would be great! Thank you. I’ve seen this once or twice. I suspected you had a directory like
src/subscriptions
that was getting interpreted as an address spec, because this is shorthand for
src/subscriptions:subscriptions
. But it doesn’t look like that’s what is happening here.
w

wonderful-iron-54019

05/18/2020, 7:41 PM
is it possible its interpreting
src/sfmc_import/BUILD
as
src/sfmc_import/BUILD:BUILD
?
b/c that would. be an issue
h

hundreds-father-404

05/18/2020, 7:41 PM
let me load it up with
./pants repl
to test
w

wonderful-iron-54019

05/18/2020, 7:41 PM
and it looks closer to what the error message is
h

hundreds-father-404

05/18/2020, 7:44 PM
Indeed:
Copy code
▶ ./v2 repl --shell=ipython src/python/pants/base/cmd_line_spec_parser.py

In [1]: from pants.base.cmd_line_spec_parser import CmdLineSpecParser

In [2]: from pants.base.build_environment import get_buildroot

In [3]: spec_parser = CmdLineSpecParser(get_buildroot())

In [4]: spec_parser.parse_spec("src/sfmc_import/BUILD")
Out[4]: SingleAddress(directory='src/sfmc_import/BUILD', name='BUILD')
w

wonderful-iron-54019

05/18/2020, 7:46 PM
funny this hasn't cropped up before, seems like it would happen quickly
h

hundreds-father-404

05/18/2020, 7:50 PM
So, the heuristic
if PurePath(spec).suffix
won’t apply, but I would expect the next heuristic of
if Path(self._root_dir, spec_path).is_file():
to apply. Could you please try running this from your build root:
Copy code
$ python  # open a repl
from pathlib import Path
(Path.cwd() / "src/sfmc_import/BUILD").is_file()
w

wonderful-iron-54019

05/18/2020, 7:51 PM
True
🤔 1
ah
i think its b/c, during local testing i may. have been behind master
and the script does a
git diff <version>
that probably detected changed files that didn't. exist on my filesystem
if i replalcethe path with the file it. was failing for
i get False
h

hundreds-father-404

05/18/2020, 7:53 PM
Ohh, yes, that is it
w

wonderful-iron-54019

05/18/2020, 7:53 PM
i mean still seems fragile, i.e. we delete targets
we cant use
git diff
h

hundreds-father-404

05/18/2020, 7:54 PM
Hm, I’ll update the ticket - that’s an interesting problem of how to disambiguate between files vs. directories when you can’t call
Path.exists()
on it because of deletion.
Possibly, we special case
BUILD
, but that isn’t great because: 1) I believe you could have a directory with that name 2) If you were to delete
Pipfile
, for example, it wouldn’t work properly.
A more general solution is to remove the shorthand of
folder/
expanding to
folder:folder
. That’s been proposed but not explored fully because it has huge implications - for example, that shorthand is really useful in BUILD files for the
dependencies
field.
w

wonderful-iron-54019

05/18/2020, 8:02 PM
sure makes sense
not sure how difficult it would be to allow shorthannd in BUILD file definitions but not via the CLI, but that sounds like a reasonable possibility
h

hundreds-father-404

05/18/2020, 8:06 PM
Definitely feasible implementation wise - the hesitation is more conceptual, that we try to avoid a value meaning one thing in one place, but another thing somewhere else. Bad UX to have to remember special cases
w

wonderful-iron-54019

05/18/2020, 8:06 PM
sure of course
h

hundreds-father-404

05/18/2020, 8:08 PM
hm I don’t think some kind of special syntax to disambiguate, like ending the arg with a
!
, would help. You’re giving Pants a machine generated value, and it would be very clunky to get Git to prepend a special symbol
w

wonderful-iron-54019

05/18/2020, 8:10 PM
right
why is mixing targets and file specs not allowed?
h

hundreds-father-404

05/18/2020, 8:15 PM
Ambiguity in Pants’ rule graph. At a high level, a
@goal_rule
like
test
will ask for
Targets
. There are two paths for the rule graph to fulfill that: 1) FilesystemSpecs -> OwnersOf -> Targets 2) AddressSpecs -> Targets If both were provided at the same time, then we would have two ways to get the
Targets
needed by the goal, and Pants wouldn’t know what to do. (We could possibly rework things to work around, this, though.) -- But even if we allowed address and file specs at the same time, you would still hit a similar issue. Pants would still interpret
src/sfmc_import/BUILD
as
src/sfmc_import/BUILD:BUILD
, and then it would complain that it can’t find any folder called
src/sfmc_import/BUILD
with the target
:BUILD
Hm, maybe the answer is outside of Pants. See https://stackoverflow.com/questions/6894322/how-to-make-git-diff-and-git-log-ignore-new-and-deleted-files for a flag that will allow you to filter out all deleted files
Alternatively, you can use
./pants --changed-since
. Pants knows what to do with deleted files if you use that approach.
w

wonderful-iron-54019

05/18/2020, 8:18 PM
i'll spend some. time with this thanks
❤️ 1
4 Views