https://pantsbuild.org/ logo
Join Slack
Powered by
is there a way to specify a coverage file config ....
# general
f
is there a way to specify a coverage file config .coveragerc for the test ? Found something that this is available in 2.0.0.dev3 . is anything available in the stable version 1.30?
h
Hello! Yes. See https://www.pantsbuild.org/v2.0/docs/python-test-goal#coverage Indeed, 2.0 only. We recommend 2.0.0a1. We’re also likely to release 2.0.0a2 today so that you can use the new setup-py plugin hook, which we merged yesterday and documented at https://www.pantsbuild.org/v2.0/docs/plugins-setup-py-goal See https://www.pantsbuild.org/v2.0/docs/how-to-upgrade-pants-2-0 for an upgrade guide. Afaict, the main thing for you will be using the new 
python_distribution
target for 
provides=setup_py()
, rather than a 
python_library
target
f
Great! Will wait for a2. Thx
❤️ 1
h
Hey Adrian! 2.0.0a2 was released last night. We haven’t announced yet because we’re finishing up some of the 2.0 Upgrade guide (link in above message), but it’s finished enough that you could get started on upgrading. And then see the above links about how to try out the new 
setup-py
plugin hook. We’re happy to help make sense of any of it 🙂
f
I am currently in vacation, but will try it next week. Thx
❤️ 1
I have started to upgrade to pants2. I have a question on migrating from 
python_library
to 
python_distribution
. Did not find any information on what is the difference between them (I understood that 
python_distribution
is used for setup_py). Is this BUILD file equivalent in pants v1 and v2? (I am most worrying about dependencies in python_distribution) # pants v1 BUILD file
Copy code
python_library(
    name = "nci-core.lib",
    sources = ['**/*.py'],
    dependencies = [
        "3rdparty/python:requests",
        "3rdparty/python:kubernetes",
        "3rdparty/python:semver",
        "3rdparty/python:ansible",
        "3rdparty/python:urllib3",
		"3rdparty/python:colorama",
		"3rdparty/python:dotty-dict",
		"3rdparty/python:distro",
		"3rdparty/python:typing-extensions",
        "3rdparty/python:filetype",
    ],
    provides=setup_py(
        name="nci-core",
        description="NCI Core",
    )
)
# pants v2 BUILD file
Copy code
python_library(
    name = "nci-core.lib",
    sources = ['**/*.py'],
    dependencies = [
        "3rdparty/python:requests",
        "3rdparty/python:kubernetes",
        "3rdparty/python:semver",
        "3rdparty/python:ansible",
        "3rdparty/python:urllib3",
		"3rdparty/python:colorama",
		"3rdparty/python:dotty-dict",
		"3rdparty/python:distro",
		"3rdparty/python:typing-extensions",
        "3rdparty/python:filetype",
    ]
)

python_distribution(
    name="nci-core.dist",
    provides=setup_py(
        name="nci-core",
        description="NCI Core",
    )
)
# or pants v2 BUILD file (with full 3rdparty deps in python_distribution definition)
Copy code
python_library(
    name = "nci-core.lib",
    sources = ['**/*.py'],
    dependencies = [
        "3rdparty/python:requests",
        "3rdparty/python:kubernetes",
        "3rdparty/python:semver",
        "3rdparty/python:ansible",
        "3rdparty/python:urllib3",
		"3rdparty/python:colorama",
		"3rdparty/python:dotty-dict",
		"3rdparty/python:distro",
		"3rdparty/python:typing-extensions",
        "3rdparty/python:filetype",
    ]
)

python_distribution(
    name="nci-core.dist",
    dependencies = [
        "3rdparty/python:requests",
        "3rdparty/python:kubernetes",
        "3rdparty/python:semver",
        "3rdparty/python:ansible",
        "3rdparty/python:urllib3",
		"3rdparty/python:colorama",
		"3rdparty/python:dotty-dict",
		"3rdparty/python:distro",
		"3rdparty/python:typing-extensions",
        "3rdparty/python:filetype",
    ]
    provides=setup_py(
        name="nci-core",
        description="NCI Core",
    )
)
# or pants v2 BUILD file (with library dependency in python_distribution definition)
Copy code
python_library(
    name = "nci-core.lib",
    sources = ['**/*.py'],
    dependencies = [
        "3rdparty/python:requests",
        "3rdparty/python:kubernetes",
        "3rdparty/python:semver",
        "3rdparty/python:ansible",
        "3rdparty/python:urllib3",
		"3rdparty/python:colorama",
		"3rdparty/python:dotty-dict",
		"3rdparty/python:distro",
		"3rdparty/python:typing-extensions",
        "3rdparty/python:filetype",
    ]
)

python_distribution(
    name="nci-core.dist",
    dependencies = [
        "src/python/nci_core:nci-core.lib",
    ]
    provides=setup_py(
        name="nci-core",
        description="NCI Core",
    )
)
from testing I think last solution is the correct one, but want to make sure.
h
Hey Adrian! Sorry for the the lack of documentation there. Thanks for sharing that feedback, I’ll update the migration guide today.Yes, the last one is correct. This sentence applies (from the Targets and BUILD Files page):
You only need to declare direct dependencies; there is no need to include the dependencies of your dependencies. Pants will pull in those dependencies for you.
So long as 
:nci-core.lib
has everything you want included, this will work. (Which, btw, you can reference it as 
":nci-core.lib"
rather than 
"src/python/nci_core:nci-core.lib"
because it lives in the same directory; Pants will expand it to the full path for you. I realized I don’t think we document this shorthand yet)
Updated: https://www.pantsbuild.org/docs/how-to-upgrade-pants-2-0#use--python_distribution-target-type-for-providessetup_py-130-vs-20 Lmk if you have any feedback on how to improve this. Thanks again for reporting it
f
👍
Copy code
18:22:28.54 [WARN] Setting LANG=en_US.UTF-8 on subprocesses due to use of deprecated option [subprocess-environment].lang. If you wish to continue to set this option, do so by adding 'LANG=value' (or 'LANG' to take the value from Pants's environment) to the [subprocess-environment].env_var option.
18:22:28.56 [ERROR] 1 Exception encountered:

Engine traceback:
  in select
  in `setup-py` goal
  in pants.backend.python.goals.setup_py.run_setup_py
  in Set up setuptools
  in pants.backend.python.util_rules.pex.create_pex
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b0_py36/lib/python3.6/site-packages/pants/engine/process.py", line 239, in fallible_to_exec_result_or_raise
    description.value,
pants.engine.process.ProcessExecutionFailure: Process 'Building setuptools.pex with 2 requirements: setuptools>=49.6.0,<49.7, wheel==0.31.1' failed with exit code 1.
stdout:

stderr:
ERROR: Could not find a version that satisfies the requirement setuptools<49.7,>=49.6.0 (from versions: 0.6b1, 0.6b2, 0.6b3, 0.6b4, 0.6rc1, 0.6rc2, 0.6rc3, 0.6rc4, 0.6rc5, 0.6rc6, 0.6rc7, 0.6rc8, 0.6rc9, 0.6rc10, 0.6rc11, 0.7.2, 0.7.3, 0.7.4, 0.7.5, 0.7.6, 0.7.7, 0.7.8, 0.8, 0.9, 0.9.1, 0.9.2, 0.9.3, 0.9.4, 0.9.5, 0.9.6, 0.9.7, 0.9.8, 1.0, 1.1, 1.1.1, 1.1.2, 1.1.3, 1.1.4, 1.1.5, 1.1.6, 1.1.7, 1.2, 1.3, 1.3.1, 1.3.2, 1.4, 1.4.1, 1.4.2, 2.0, 2.0.1, 2.0.2, 2.1, 2.1.1, 2.1.2, 2.2, 3.0, 3.0.1, 3.0.2, 3.1, 3.2, 3.3, 3.4, 3.4.1, 3.4.2, 3.4.3, 3.4.4, 3.5, 3.5.1, 3.5.2, 3.6, 3.7, 3.7.1, 3.8, 3.8.1, 4.0, 4.0.1, 5.0, 5.0.1, 5.0.2, 5.1, 5.2, 5.3, 5.4, 5.4.1, 5.4.2, 5.5, 5.5.1, 5.6, 5.7, 5.8, 6.0.1, 6.0.2, 6.1, 7.0, 8.0, 8.0.1, 8.0.2, 8.0.3, 8.0.4, 8.1, 8.2, 8.2.1, 8.3, 9.0, 9.0.1, 9.1, 10.0, 10.0.1, 10.1, 10.2, 10.2.1, 11.0, 11.1, 11.2, 11.3, 11.3.1, 12.0, 12.0.1, 12.0.2, 12.0.3, 12.0.4, 12.0.5, 12.1, 12.2, 12.3, 12.4, 13.0.1, 13.0.2, 14.0, 14.1, 14.1.1, 14.2, 14.3, 14.3.1, 15.0, 15.1, 15.2, 16.0, 17.0, 17.1, 17.1.1, 18.0, 18.0.1, 18.1, 18.2, 18.3, 18.3.1, 18.3.2, 18.4, 18.5, 18.6, 18.6.1, 18.7, 18.7.1, 18.8, 18.8.1, 19.0, 19.1, 19.1.1, 19.2, 19.3, 19.4, 19.4.1, 19.5, 19.6, 19.6.1, 19.6.2, 19.7, 20.0, 20.1, 20.1.1, 20.2.2, 20.3, 20.3.1, 20.4, 20.6.6, 20.6.7, 20.6.8, 20.7.0, 20.8.0, 20.8.1, 20.9.0, 20.10.1, 21.0.0, 21.1.0, 21.2.0, 21.2.1, 21.2.2, 22.0.0, 22.0.1, 22.0.2, 22.0.4, 22.0.5, 23.0.0, 23.1.0, 23.2.0, 23.2.1, 24.0.0, 24.0.1, 24.0.2, 24.0.3, 24.1.0, 24.1.1, 24.2.0, 24.2.1, 24.3.0, 24.3.1, 25.0.0, 25.0.1, 25.0.2, 25.1.0, 25.1.1, 25.1.2, 25.1.3, 25.1.4, 25.1.5, 25.1.6, 25.2.0, 25.3.0, 25.4.0, 26.0.0, 26.1.0, 26.1.1, 27.0.0, 27.1.0, 27.1.2, 27.2.0, 27.3.0, 27.3.1, 28.0.0, 28.1.0, 28.2.0, 28.3.0, 28.4.0, 28.5.0, 28.6.0, 28.6.1, 28.7.0, 28.7.1, 28.8.0, 28.8.1, 29.0.0, 29.0.1, 30.0.0, 30.1.0, 30.2.0, 30.2.1, 30.3.0, 30.4.0, 31.0.0, 31.0.1, 32.0.0, 32.1.0, 32.1.1, 32.1.2, 32.1.3, 32.2.0, 32.3.0, 32.3.1, 33.1.0, 33.1.1, 34.0.0, 34.0.1, 34.0.2, 34.0.3, 34.1.0, 34.1.1, 34.2.0, 34.3.0, 34.3.1, 34.3.2, 34.3.3, 34.4.0, 34.4.1, 35.0.0, 35.0.1, 35.0.2, 36.0.1, 36.1.0, 36.1.1, 36.2.0, 36.2.1, 36.2.2, 36.2.3, 36.2.4, 36.2.5, 36.2.6, 36.2.7, 36.3.0, 36.4.0, 36.5.0, 36.6.0, 36.6.1, 36.7.0, 36.7.1, 36.7.2, 36.8.0, 37.0.0, 38.0.0, 38.1.0, 38.2.0, 38.2.1, 38.2.3, 38.2.4, 38.2.5, 38.3.0, 38.4.0, 38.4.1, 38.5.0, 38.5.1, 38.5.2, 38.6.0, 38.6.1, 38.7.0, 39.0.0, 39.0.1, 39.1.0, 39.2.0, 40.0.0, 40.1.0, 40.1.1, 40.2.0, 40.3.0, 40.4.0, 40.4.1, 40.4.2, 40.4.3, 40.5.0, 40.6.0, 40.6.1, 40.6.2, 40.6.3, 40.7.0, 40.7.1, 40.7.2, 40.7.3, 40.8.0, 40.9.0, 41.0.0, 41.0.1, 41.1.0, 41.2.0, 41.3.0, 41.4.0, 41.5.0, 41.5.1, 41.6.0, 42.0.0, 42.0.1, 42.0.2, 43.0.0, 44.0.0, 44.1.0, 44.1.1)
ERROR: No matching distribution found for setuptools<49.7,>=49.6.0
pid 29681 -> /usr/bin/python2.7 /home/centos/.cache/pants/named_caches/pex_root/pip.pex/aef609891d42d65c887d1aeee58c46f6713a7e49 --disable-pip-version-check --isolated --no-python-version-warning --exists-action a -q --cache-dir /home/centos/.cache/pants/named_caches/pex_root download --dest /tmp/process-executionfj57UO/.tmp/tmpy6_aAI/usr.bin.python2.7 --index-url <https://artifactory3.ds.jdsu.net:443/artifactory/api/pypi/nci-pypi/simple> --header Cache-Control:max-age=3600 --retries 5 --timeout 15 --constraint 3rdparty/python/constraints.txt setuptools>=49.6.0,<49.7 wheel==0.31.1 exited with 1 and STDERR:
None

Traceback (most recent call last):
  File "/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b0_py36/lib/python3.6/site-packages/pants/bin/local_pants_runner.py", line 255, in run
    engine_result = self._run_v2()
  File "/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b0_py36/lib/python3.6/site-packages/pants/bin/local_pants_runner.py", line 166, in _run_v2
    return self._maybe_run_v2_body(goals, poll=False)
  File "/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b0_py36/lib/python3.6/site-packages/pants/bin/local_pants_runner.py", line 189, in _maybe_run_v2_body
    poll_delay=(0.1 if poll else None),
  File "/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b0_py36/lib/python3.6/site-packages/pants/init/engine_initializer.py", line 139, in run_goal_rules
    goal_product, params, poll=poll, poll_delay=poll_delay
  File "/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b0_py36/lib/python3.6/site-packages/pants/engine/internals/scheduler.py", line 561, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b0_py36/lib/python3.6/site-packages/pants/engine/internals/scheduler.py", line 525, in _raise_on_error
    wrapped_exceptions=tuple(t.exc for t in throws),
pants.engine.internals.scheduler.ExecutionError: 1 Exception encountered:

Engine traceback:
  in select
  in `setup-py` goal
  in pants.backend.python.goals.setup_py.run_setup_py
  in Set up setuptools
  in pants.backend.python.util_rules.pex.create_pex
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/home/centos/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b0_py36/lib/python3.6/site-packages/pants/engine/process.py", line 239, in fallible_to_exec_result_or_raise
    description.value,
pants.engine.process.ProcessExecutionFailure: Process 'Building setuptools.pex with 2 requirements: setuptools>=49.6.0,<49.7, wheel==0.31.1' failed with exit code 1.
stdout:

stderr:
ERROR: Could not find a version that satisfies the requirement setuptools<49.7,>=49.6.0 (from versions: 0.6b1, 0.6b2, 0.6b3, 0.6b4, 0.6rc1, 0.6rc2, 0.6rc3, 0.6rc4, 0.6rc5, 0.6rc6, 0.6rc7, 0.6rc8, 0.6rc9, 0.6rc10, 0.6rc11, 0.7.2, 0.7.3, 0.7.4, 0.7.5, 0.7.6, 0.7.7, 0.7.8, 0.8, 0.9, 0.9.1, 0.9.2, 0.9.3, 0.9.4, 0.9.5, 0.9.6, 0.9.7, 0.9.8, 1.0, 1.1, 1.1.1, 1.1.2, 1.1.3, 1.1.4, 1.1.5, 1.1.6, 1.1.7, 1.2, 1.3, 1.3.1, 1.3.2, 1.4, 1.4.1, 1.4.2, 2.0, 2.0.1, 2.0.2, 2.1, 2.1.1, 2.1.2, 2.2, 3.0, 3.0.1, 3.0.2, 3.1, 3.2, 3.3, 3.4, 3.4.1, 3.4.2, 3.4.3, 3.4.4, 3.5, 3.5.1, 3.5.2, 3.6, 3.7, 3.7.1, 3.8, 3.8.1, 4.0, 4.0.1, 5.0, 5.0.1, 5.0.2, 5.1, 5.2, 5.3, 5.4, 5.4.1, 5.4.2, 5.5, 5.5.1, 5.6, 5.7, 5.8, 6.0.1, 6.0.2, 6.1, 7.0, 8.0, 8.0.1, 8.0.2, 8.0.3, 8.0.4, 8.1, 8.2, 8.2.1, 8.3, 9.0, 9.0.1, 9.1, 10.0, 10.0.1, 10.1, 10.2, 10.2.1, 11.0, 11.1, 11.2, 11.3, 11.3.1, 12.0, 12.0.1, 12.0.2, 12.0.3, 12.0.4, 12.0.5, 12.1, 12.2, 12.3, 12.4, 13.0.1, 13.0.2, 14.0, 14.1, 14.1.1, 14.2, 14.3, 14.3.1, 15.0, 15.1, 15.2, 16.0, 17.0, 17.1, 17.1.1, 18.0, 18.0.1, 18.1, 18.2, 18.3, 18.3.1, 18.3.2, 18.4, 18.5, 18.6, 18.6.1, 18.7, 18.7.1, 18.8, 18.8.1, 19.0, 19.1, 19.1.1, 19.2, 19.3, 19.4, 19.4.1, 19.5, 19.6, 19.6.1, 19.6.2, 19.7, 20.0, 20.1, 20.1.1, 20.2.2, 20.3, 20.3.1, 20.4, 20.6.6, 20.6.7, 20.6.8, 20.7.0, 20.8.0, 20.8.1, 20.9.0, 20.10.1, 21.0.0, 21.1.0, 21.2.0, 21.2.1, 21.2.2, 22.0.0, 22.0.1, 22.0.2, 22.0.4, 22.0.5, 23.0.0, 23.1.0, 23.2.0, 23.2.1, 24.0.0, 24.0.1, 24.0.2, 24.0.3, 24.1.0, 24.1.1, 24.2.0, 24.2.1, 24.3.0, 24.3.1, 25.0.0, 25.0.1, 25.0.2, 25.1.0, 25.1.1, 25.1.2, 25.1.3, 25.1.4, 25.1.5, 25.1.6, 25.2.0, 25.3.0, 25.4.0, 26.0.0, 26.1.0, 26.1.1, 27.0.0, 27.1.0, 27.1.2, 27.2.0, 27.3.0, 27.3.1, 28.0.0, 28.1.0, 28.2.0, 28.3.0, 28.4.0, 28.5.0, 28.6.0, 28.6.1, 28.7.0, 28.7.1, 28.8.0, 28.8.1, 29.0.0, 29.0.1, 30.0.0, 30.1.0, 30.2.0, 30.2.1, 30.3.0, 30.4.0, 31.0.0, 31.0.1, 32.0.0, 32.1.0, 32.1.1, 32.1.2, 32.1.3, 32.2.0, 32.3.0, 32.3.1, 33.1.0, 33.1.1, 34.0.0, 34.0.1, 34.0.2, 34.0.3, 34.1.0, 34.1.1, 34.2.0, 34.3.0, 34.3.1, 34.3.2, 34.3.3, 34.4.0, 34.4.1, 35.0.0, 35.0.1, 35.0.2, 36.0.1, 36.1.0, 36.1.1, 36.2.0, 36.2.1, 36.2.2, 36.2.3, 36.2.4, 36.2.5, 36.2.6, 36.2.7, 36.3.0, 36.4.0, 36.5.0, 36.6.0, 36.6.1, 36.7.0, 36.7.1, 36.7.2, 36.8.0, 37.0.0, 38.0.0, 38.1.0, 38.2.0, 38.2.1, 38.2.3, 38.2.4, 38.2.5, 38.3.0, 38.4.0, 38.4.1, 38.5.0, 38.5.1, 38.5.2, 38.6.0, 38.6.1, 38.7.0, 39.0.0, 39.0.1, 39.1.0, 39.2.0, 40.0.0, 40.1.0, 40.1.1, 40.2.0, 40.3.0, 40.4.0, 40.4.1, 40.4.2, 40.4.3, 40.5.0, 40.6.0, 40.6.1, 40.6.2, 40.6.3, 40.7.0, 40.7.1, 40.7.2, 40.7.3, 40.8.0, 40.9.0, 41.0.0, 41.0.1, 41.1.0, 41.2.0, 41.3.0, 41.4.0, 41.5.0, 41.5.1, 41.6.0, 42.0.0, 42.0.1, 42.0.2, 43.0.0, 44.0.0, 44.1.0, 44.1.1)
ERROR: No matching distribution found for setuptools<49.7,>=49.6.0
pid 29681 -> /usr/bin/python2.7 /home/centos/.cache/pants/named_caches/pex_root/pip.pex/aef609891d42d65c887d1aeee58c46f6713a7e49 --disable-pip-version-check --isolated --no-python-version-warning --exists-action a -q --cache-dir /home/centos/.cache/pants/named_caches/pex_root download --dest /tmp/process-executionfj57UO/.tmp/tmpy6_aAI/usr.bin.python2.7 --index-url <https://artifactory3.ds.jdsu.net:443/artifactory/api/pypi/nci-pypi/simple> --header Cache-Control:max-age=3600 --retries 5 --timeout 15 --constraint 3rdparty/python/constraints.txt setuptools>=49.6.0,<49.7 wheel==0.31.1 exited with 1 and STDERR:
None
I am trying to run the command `./pants setup-py src/python/nci_core:nci-core.dist -- sdist`and I get this error
h
Is this b0? Try b1
f
I am not defining anywhere a setuptools version. I am using latest stable version "2.0.0b0"
ok
worked with b1!
h
Yay! Glad to hear.
Writing the b1 release email as we speak 🙂
f
trying to implement my new versioning function . got these two libs # nci-core BUILD file
Copy code
python_library(
    name = "nci-core.lib",
    sources = ['**/*.py'],
    dependencies = [
        "3rdparty/python:requests",
        "3rdparty/python:kubernetes",
        "3rdparty/python:semver",
        "3rdparty/python:ansible",
        "3rdparty/python:urllib3",
		"3rdparty/python:colorama",
		"3rdparty/python:dotty-dict",
		"3rdparty/python:distro",
		"3rdparty/python:typing-extensions",
        "3rdparty/python:filetype",
    ]
)

python_distribution(
    name="nci-core.dist",
    dependencies = [
        "src/python/nci_core:nci-core.lib"
    ],
    provides=setup_py(
        name="nci-core",
        description="NCI Core",
    )
)
# nci-env-validation BUILD file
Copy code
python_library(
    name = "nci-env-validation.lib",
    sources = ['**/*.py'],
    dependencies = [
        "src/python/nci_core:nci-core.lib",
        "src/python/nci_env_validation/additional_data:nci-env-validation-additional-data.rsc",
    ],
    
)

python_distribution(
    name="nci-env-validation.dist",
    dependencies = [
        "src/python/nci_env_validation:nci-env-validation.lib"
    ],
    provides=setup_py(
        name="nci-env-validation",
        description="NCI Env Validation",
    )
)
this is my register.py plugin
Copy code
# this file was built from 
# doc: <https://www.pantsbuild.org/docs/plugins-setup-py-goal>
# example: <https://github.com/pantsbuild/pants/blob/master/pants-plugins/internal_plugins/releases/register.py>

import logging

from pants.base.build_environment import get_buildroot
from pants.backend.python.goals.setup_py import SetupKwargs, SetupKwargsRequest
from pants.engine.addresses import Addresses
from pants.engine.rules import Get, collect_rules, rule
from pants.engine.target import Target, TransitiveTargets
from pants.engine.unions import UnionRule

from utilities.versioning import git


class CustomSetupKwargsRequest(SetupKwargsRequest):
    @classmethod
    def is_applicable(cls, _: Target) -> bool:
        return True


@rule
async def setup_kwargs_plugin(request: CustomSetupKwargsRequest) -> SetupKwargs:
    kwargs = request.explicit_kwargs.copy()
    transitive_targets = await Get(TransitiveTargets, Addresses([request.target.address]))
    for dependency in transitive_targets.dependencies:
        logging.error(f"dependency {dependency.address}")
    # kwargs["version"] = git.get_version(
    #     get_buildroot(), request.target.address.spec_path, transitive_targets.dependencies)
    # <http://logging.info|logging.info>(f'lib name: {kwargs["name"]}, lib version: {kwargs["version"]}')

    return SetupKwargs(kwargs, address=request.target.address)


def rules():
    return (*collect_rules(), UnionRule(SetupKwargsRequest, CustomSetupKwargsRequest))
and this is my outputfile
Copy code
18:45:56.15 [ERROR] dependency src/python/nci_core/__init__.py:nci-core.lib
18:45:56.15 [ERROR] dependency src/python/nci_core/ansible/__init__.py:../nci-core.lib
...
18:45:56.23 [ERROR] dependency src/python/nci_core/yaml/helpers.py:../nci-core.lib
18:45:56.23 [ERROR] dependency 3rdparty/python:ansible
18:45:56.23 [ERROR] dependency 3rdparty/python:colorama
18:45:56.23 [ERROR] dependency 3rdparty/python:distro
18:45:56.23 [ERROR] dependency 3rdparty/python:dotty-dict
18:45:56.23 [ERROR] dependency 3rdparty/python:filetype
18:45:56.23 [ERROR] dependency 3rdparty/python:kubernetes
18:45:56.23 [ERROR] dependency 3rdparty/python:requests
18:45:56.23 [ERROR] dependency 3rdparty/python:semver
18:45:56.23 [ERROR] dependency 3rdparty/python:typing-extensions
18:45:56.23 [ERROR] dependency 3rdparty/python:urllib3
18:45:56.23 [ERROR] dependency 3rdparty/python:requirements.txt
I was expecting transitive dependencies only the libs, not every file.
also this shows the same
Copy code
[centos@dabu-dev1 nci-installer]$ ./pants dependencies --transitive src/python/nci_env_validation:nci-env-validation.dist
3rdparty/python:ansible
3rdparty/python:colorama
3rdparty/python:distro
3rdparty/python:dotty-dict
3rdparty/python:filetype
3rdparty/python:kubernetes
3rdparty/python:requests
3rdparty/python:requirements.txt
3rdparty/python:semver
3rdparty/python:typing-extensions
3rdparty/python:urllib3
src/python/nci_core/__init__.py:nci-core.lib
src/python/nci_core/ansible/__init__.py:../nci-core.lib
src/python/nci_core/ansible/inventory.py:../nci-core.lib
src/python/nci_core/ansible/playbook.py:../nci-core.lib
src/python/nci_core/bundles/__init__.py:../nci-core.lib
src/python/nci_core/bundles/valid_bundle.py:../nci-core.lib
src/python/nci_core/collections/__init__.py:../nci-core.lib
src/python/nci_core/collections/safe_immutable_dict.py:../nci-core.lib
src/python/nci_core/collections/safe_mutable_dict.py:../nci-core.lib
src/python/nci_core/common/__init__.py:../nci-core.lib
....
if I remember correctly in pants v1 it only showed the library name, not all the files
h
Yeah, that’s correct. This is one of the bigger changes in 2.0: files are now the “atomic unit” for Pants, rather than targets. For example, you can now depend on a specific file, rather than having to depend on everything in the target, even if you don’t use it. When you depend on a target like 
project:foo
, that is ~shorthand for depending on each file in the 
sources
field for that target. https://www.pantsbuild.org/docs/how-to-upgrade-pants-2-0#files-are-now-the-atomic-unit-rather-than-targets-130-vs-20 explains a bit more Taking a step back, what are you trying to pass to 
git.get_version
? Does it expect a list of file names?
f
this is my git versioning function
Copy code
import logging
import os

from git import Repo
from git.exc import InvalidGitRepositoryError
from semver import VersionInfo


def get_version(repo_path: str, subtree_path: str, dependencies: list) -> str:
    """
    Function for returning the version based on the latest Git Tag on a subtree
    :type repo_path: str
    :type subtree_path: str
    :return: str
    """
    logging.debug(f"subtree path: {subtree_path}")
    repo = Repo(repo_path)
    subtree_commit_last = repo.git.log("--pretty=%H", "-n", "1", "--", subtree_path)
    logging.debug(f"subtree latest commit id: {subtree_commit_last}")
    subtree_tag_last = repo.git.describe("--abbrev=0", subtree_commit_last)
    logging.debug(f"subtree latest tag: {subtree_tag_last}")

    is_dirty: bool = repo.is_dirty(path=subtree_path)
    logging.debug(f"subtree dirty: {is_dirty}")

    commit_deltas = sum(1 for _ in repo.iter_commits(f"{subtree_tag_last}..{subtree_commit_last}"))
    logging.debug(f"subtree commit deltas: {commit_deltas}")

    commit_sha: str = repo.commit(subtree_commit_last).hexsha[0:7]
    logging.debug(f"subtree sha: {commit_sha}")

    if not is_dirty:
        if commit_deltas == 0:
            # Case 1: User commits the changes and adds a git tag
            version = str(subtree_tag_last)

        # Case 2: User commits the changes without creating a new tag
        verson = f"{str(VersionInfo.parse(subtree_tag_last).bump_patch())}-{commit_deltas}-g{commit_sha}"

    # Case 3: User doens't commit the changes
    version = f"{str(VersionInfo.parse(subtree_tag_last).bump_patch())}-{commit_deltas}-g{commit_sha}-dirty"

    # versioning depends on dependencies: when a child version increases, we want the parent version to increase
    for dependency in dependencies:
        logging.error(f"dependency {dependency.address}")
        dependency_version = get_version(repo_path, dependency.address.spec_path, [])
        if VersionInfo.compare(VersionInfo.parse(dependency_version), VersionInfo.parse(version)) > 0:
            version = dependency_version
            logging.error(f"new version {version} from dependency {dependency.address}")

    return version
so I want to go through all the transitive dependencies and see if there is a dependency with a higher version, and "get"that version
👍 1
it works, but now with a lot of transitive dependencies my versioning algorithm is not very efficient
I can do some optimizations, but wanted to makes sure if there is another way of getting transitive dependencies (in v2 context, more like "transitive dependency targets like `project:foo`")
h
So you almost likely to filter the targets to only include targets that have the 
PythonProvidesField
registered on them, which essentially ends up being any 
python_distribution
target. Is that correct, that only 
python_distribution
targets are relevant when considering dependencies? See the last part of https://www.pantsbuild.org/docs/rules-api-and-target-api#how-to-read-values-from-a-target for how to use 
target.has_field()
f
I don't seem to have python_distribution as the a dependency type
Copy code
$ ./pants dependencies --transitive src/python/nci_env_validation:nci-env-validation.lib | xargs ./pants filter --filter-target-type=python_distribution
19:38:26.12 [INFO] initialization options changed: reinitializing pantsd...
19:38:26.22 [WARN] File handle limit is capped to: 4096. To avoid 'too many open file handle' errors, we recommend a limit of at least 10000: please see <https://www.pantsbuild.org/docs/troubleshooting#too-many-open-files-error> for more information.
19:38:26.36 [INFO] pantsd initialized.
19:38:29.24 [INFO] initialization options changed: reinitializing pantsd...
19:38:29.43 [WARN] File handle limit is capped to: 4096. To avoid 'too many open file handle' errors, we recommend a limit of at least 10000: please see <https://www.pantsbuild.org/docs/troubleshooting#too-many-open-files-error> for more information.
19:38:29.56 [INFO] pantsd initialized.
I only have 3rdparty dependencies and python_libraries. these are my BUILD files # nci-core BUILD file
Copy code
python_library(
    name = "nci-core.lib",
    sources = ['**/*.py'],
    dependencies = [
        "3rdparty/python:requests",
        "3rdparty/python:kubernetes",
        "3rdparty/python:semver",
        "3rdparty/python:ansible",
        "3rdparty/python:urllib3",
		"3rdparty/python:colorama",
		"3rdparty/python:dotty-dict",
		"3rdparty/python:distro",
		"3rdparty/python:typing-extensions",
        "3rdparty/python:filetype",
    ]
)

python_distribution(
    name="nci-core.dist",
    dependencies = [
        "src/python/nci_core:nci-core.lib"
    ],
    provides=setup_py(
        name="nci-core",
        description="NCI Core",
    )
)
# nci-env-validation BUILD file
Copy code
python_library(
    name = "nci-env-validation.lib",
    sources = ['**/*.py'],
    dependencies = [
        "src/python/nci_core:nci-core.lib",
        "src/python/nci_env_validation/additional_data:nci-env-validation-additional-data.rsc",
    ],

)

python_distribution(
    name="nci-env-validation.dist",
    dependencies = [
        "src/python/nci_env_validation:nci-env-validation.lib"
    ],
    provides=setup_py(
        name="nci-env-validation",
        description="NCI Env Validation",
    )
)
h
Ah, yeah, that makes a lot of sense; generally depending directly on a 
python_distribution
isn’t something you’d do. In your code, it looks like you’re using the path to the directory, right? You don’t necessarily care about a Pants Target, you only want to know “which directories are depended upon by this `python_distribution`“? If that’s the case, you could use a set comprehension to convert 
transitive_targets.dependencies
into a set of the unique dirs
Copy code
{tgt.address.spec_path for tgt in transitive_targets.dependencies}
f
shall I change the nci-env-validation.lib to include the dist instead of the lib ?
h
shall I change the nci-env-validation.lib to include the dist instead of the lib ?
No, you have it the right way.
f
yeah. that will work. I was thinking of something similar in terms of optimization. but wanted to make sure if there is something in pants I was missing. thanks for your help
h
Yeah, because you’re making calls to 
git
, which only understands directories, it’s fine to no longer be working with Targets and instead be working with directories. Git wouldn’t understand it otherwise. Generally, with plugins, to ensure that caching and invalidation work correctly, this restriction applies:
Each rule should be pure; you should not use side-effects like subprocess.run(), print(), or the requests library. Instead, the Rules API has its own alternatives that are understood by the Pants engine and which work properly with its caching and parallelism.
It looks like you already have a lot of abstractions around a Git wrapper? I’m trying to think of ways to make it safer to use those side-effecty abstractions in a way that’s still safe, to avoid rewriting lots of things The main issue right now is that when you run your 
@rule
, if the inputs to the rule do not change, the output will be read from cache and the rule won’t be evaluated, until you restart the pants deamon. This is a private API, but you can use 
@_uncachable_rule
instead of 
@rule
(from 
pants.engine.rules
). This will force your rule to rerun every time. The API might change a little (probably just a rename), which we’ll be happy to help walk through when the time comes that we make it public. -- You’re also missing out on some possible parallelization made easy by the engine, like running multiple Git commands in parallel. But that’s an optimization - totally okay to stick with this for now, if you make the caching change
f
Copy code
The main issue right now is that when you run your @rule, if the inputs to the rule do not change, the output will be read from cache and the rule won't be evaluated, until you restart the pants deamon.
you are right. I have already saw this behavior.
how can I import _uncachable_rule
Copy code
from pants.engine.rules import Get, collect_rules, rule, _uncachable_rule
this give me the error
Copy code
pants.base.exceptions.BackendConfigurationError: Failed to load the utilities.register backend: ImportError("cannot import name '_uncachable_rule'",)
h
Normally, that’s a big feature of writing Pants plugins, that you automatically get memoization/caching. But yeah, running side-effects like Git has different semantics If one day you do rewrite your Git abstraction to use Pants abstractions like 
Process
to run Git, Pants would be able to handle the caching properly. And give the other benefits like concurrency
Oh sorry, typo: 
_uncacheable_rule
(with an e)
f
cool. it does not do anymore caching.
💯 1
at the moment I am trying to adopt pants in our tooling and we will do the finishing touches like concurrency afterwards. but it is good to know that pants has this kind of support
❤️ 1
going to sleep. it is almost midnight for me. thanks for your help
h
That makes a lot of sense. And thank you for sharing your use case with us. That’s an important use case for us to robustly support, when you already have your own plugins and want to adopt Pants without rewriting the world. We should document things like 
@_uncacheable_rule
once it’s public
You’re welcome!
f
one more question. it looks like there are tests now that are failing in v2. I am guessing because files are now atomic units, somehow ___init___.py files are taken into consideration. Do you know of a way to skip them? or is just related to the pytest version increase, that probably came with pants v2
Copy code
𐄂 tests/python/nci_core_tests/decorators/__init__.py:../nci-core.tests failed.
𐄂 tests/python/nci_core_tests/filesystem/__init__.py:../nci-core.tests failed.
𐄂 tests/python/nci_core_tests/http_requests/__init__.py:../nci-core.tests failed.
𐄂 tests/python/nci_core_tests/k8s/__init__.py:../nci-core.tests failed.
𐄂 tests/python/nci_core_tests/service/__init__.py:../nci-core.tests failed.
𐄂 tests/python/nci_core_tests/stdlib/__init__.py:../nci-core.tests failed.
𐄂 tests/python/nci_env_validation_tests/__init__.py:nci-env-validation.tests failed.
𐄂 tests/python/nci_env_validation_tests/checks/__init__.py:../nci-env-validation.tests failed.
𐄂 tests/python/nci_env_validation_tests/resources/__init__.py:../nci-env-validation.tests failed.
𐄂 tests/python/nci_installer_tests/__init__.py:nci-installer.tests failed.
𐄂 tests/python/nci_installer_tests/checks/__init__.py:../nci-installer.tests failed.
𐄂 tests/python/nci_installer_tests/cli/__init__.py:../nci-installer.tests failed.
𐄂 tests/python/nci_installer_tests/config/__init__.py:../nci-installer.tests failed.
𐄂 tests/python/nci_installer_tests/debugging/__init__.py:../nci-installer.tests failed.
𐄂 tests/python/nci_installer_tests/fixtures/Dict2Obj.py:../nci-installer.tests failed.
𐄂 tests/python/nci_installer_tests/fixtures/KubeAuthMock.py:../nci-installer.tests failed.
𐄂 tests/python/nci_installer_tests/fixtures/KubeClientMock.py:../nci-installer.tests failed.
𐄂 tests/python/nci_installer_tests/fixtures/__init__.py:../nci-installer.tests failed.
𐄂 tests/python/nci_installer_tests/helpers/__init__.py:../nci-installer.tests failed.
𐄂 tests/python/nci_installer_tests/installer/__init__.py:../nci-installer.tests failed.
𐄂 tests/python/nci_installer_tests/run_script/__init__.py:../nci-installer.tests failed.
𐄂 tests/python/nci_installer_tests/workflow/__init__.py:../nci-installer.tests failed.
h
Are you using an explicit 
sources
field for your 
python_tests
targets like this?
Copy code
python_tests(sources=["*.py"])
If so, that will include 
__init__.py
, which you don’t want, because it’s of course not a test. Instead, you can either use the default 
sources
field, or if you want to use recursive globs to have fewer test targets, you can use:
Copy code
python_tests(sources=["**/*_test.py", "**/conftest.py"])
(Adjust as relevant to your test names)
f
this is my BUILD files for tests
Copy code
python_tests(
    name='nci-core.tests',
    sources = ['**/*.py'],
    dependencies = [
        "3rdparty/python:responses",
        "src/python/nci_core:nci-core.lib",
    ],
    timeout=90,
)
h
Cool, yeah tweak the 
sources
field to be more precise like 
["**/*_test.py", "**/conftest.py"]
Do you have any content in your 
__init__.py
files, or they’re empty? If there’s content, you’ll want to set up some things, but otherwise are good to go
f
I have some content
something like
Copy code
tests/python/nci_installer_tests/conftest.py:13: in <module>
    from .fixtures.KubeAuthMock import KubeAuthMock
E   ModuleNotFoundError: No module named 'nci_installer_tests.fixtures'
can I do a
Copy code
sources = ['**/*.py', '!**/__init__.py'],
or do I obtain the same result?
h
Yep! You can do that too. But it might result in non test files still being included. Generally, 
python_tests
should only be test files and/or 
conftest.py
. Otherwise, use 
python_library
I have some content
Okay. So if you have content, we recommend that you use 
python_library
to capture all of your 
__init__.py
files. Something like this:
Copy code
python_library(name="inits", sources=["**/__init__.py"])
Then, turn on 
inits = true
in the 
[python-infer]
scope. This will automatically infer a dependency on the 
__init__.py
files in the current directory and any ancestor directories, to mirror Python semantics. You can verify by running 
./pants dependencies path/to/f_test.py
. (Tool tip at https://www.pantsbuild.org/docs/python-backend about this all)
f
will try that tomorrow. thx
❤️ 1
4 Views
Open in Slack
PreviousNext
Powered by