hello, I have a python test that basically runs pants itself to enforce an invariant about the build tree (something along the lines of every module has a corresponding test). However, I expect that this test would fail once I turn on

chroot

. What is the canonical way to test the build tree itself? Or maybe there is a short-hand way to declare a dependency on pants + its build files?

Right now, we print the summary for

./pants test

like this at the end of every test run.

src/python/pants/util/socket_test.py:tests                                      .....   SUCCESS
src/python/pants/util/filtering_test.py:tests                                   .....   FAILURE
src/python/pants/util/osutil_test.py:tests                                      .....   SUCCESS
src/python/pants/util/dirutil_test.py:tests                                     .....   FAILURE
src/python/pants/util/eval_test.py:tests                                        .....   SUCCESS

Instead, what do y’all think of aligning with the style that

fmt

and

lint

use, like this?

✓ src/python/pants/util/socket_test.py:tests succeeded.
𐄂 src/python/pants/util/filtering_test.py:tests failed.
✓ src/python/pants/util/osutil_test.py:tests succeeded.
𐄂 src/python/pants/util/dirutil_test.py:tests failed.
✓ src/python/pants/util/eval_test.py:tests succeeded.

(The actual test content will be printed above - this is only the summary at the end)

hello all. I just started with pants and want to use it for version management in python libraries. Is there any functionality in pants that handles version bumps of libraries during build? For more details on this please see the ticket I've opened a couple of hours ago https://github.com/pantsbuild/pants/issues/10633

Hey folks, qq here: I'm working on

pipenv_requirements

build alias to source python requirements from pipenv using the

PythonRequirements

class as a model. its working like a charm. However, we've traditionally kept our

Pipfile.lock

file at the project root but, for ease of migration, i'd still like the build addresses for the requirements to go to

3rdparty/python:<req>

is there a way i can use the

_parse_context

to get this to happen?

another question about moving from 1.29 -> 1.30 seems like my env vars sett like

ENV1=val1 ENV2=val2 ./pants test src::

is no longer being respected, as im getting a lot of

ValueError: 'ENV1' is a required config value

in my tests now

any ideas on this? its as if docopt in my code is conflicting with pytest-cov.

pants_version: 1.23.0

d248ba4f9860a2fdc5/.deps/boto3-1.11.3-py2.py3-none-any.whl
                     pex:     /home/ruthgracewong/python-commons/.pants.d/pyprep/requirements/CPython-3.6.8/285ef0aeec651f37c5dfe3d248ba4f9860a2fdc5/.deps/chardet-3.0.4-py2.py3-none-any.whl
                     pex:     /home/ruthgracewong/python-commons/.pants.d/pyprep/requirements/CPython-3.6.8/285ef0aeec651f37c5dfe3d248ba4f9860a2fdc5/.deps/idna-2.8-py2.py3-none-any.whl
                     pex:     /home/ruthgracewong/python-commons/.pants.d/pyprep/sources/046be5bc3b0575f4d1acc33b48dbdd2eca5d778d/.deps/setuptools-40.6.3-py2.py3-none-any.whl
                     pex:     /home/ruthgracewong/python-commons/.pants.d/test/pytest-prep/CPython-3.6.8/6b63dbeb8f04af2fc6ba79516c3440a6735643d3/.bootstrap
                     pex:   * - paths that do not exist or will be imported via zipimport
                     collected 0 items / 1 errors

                     ===================== ERRORS =====================
                     _ ERROR collecting .pants.d/pyprep/sources/046be5bc3b0575f4d1acc33b48dbdd2eca5d778d/pinterest/container_report/test_container_report.py _
                     .pants.d/pyprep/sources/046be5bc3b0575f4d1acc33b48dbdd2eca5d778d/pinterest/container_report/test_container_report.py:3: in <module>
                         import pinterest.container_report.container_report
                     .pants.d/pyprep/sources/046be5bc3b0575f4d1acc33b48dbdd2eca5d778d/pinterest/container_report/container_report.py:38: in <module>
                         arguments = docopt(__doc__, version=VERSION)
                     .pants.d/pyprep/requirements/CPython-3.6.8/285ef0aeec651f37c5dfe3d248ba4f9860a2fdc5/.deps/docopt-0.6.2-py2.py3-none-any.whl/docopt.py:567: in docopt
                         options_first)
                     .pants.d/pyprep/requirements/CPython-3.6.8/285ef0aeec651f37c5dfe3d248ba4f9860a2fdc5/.deps/docopt-0.6.2-py2.py3-none-any.whl/docopt.py:444: in parse_argv
                         parsed += parse_long(tokens, options)
                     .pants.d/pyprep/requirements/CPython-3.6.8/285ef0aeec651f37c5dfe3d248ba4f9860a2fdc5/.deps/docopt-0.6.2-py2.py3-none-any.whl/docopt.py:312: in parse_long
                         (long, ', '.join(o.long for o in similar)))
                     E   docopt.DocoptExit: --cov is not a unique prefix: --cov-report, --cov-config?
                     E   Usage:
                     E     container_report.py [-c] FILENAME [-o OUTPUT_FILENAME]

from pants.ini (pytest is set to this version to support some code in python 2.7)

[pytest]
version: pytest>=4.6.6,<4.7
pytest_plugins: +["setuptools<45; python_version < '3'","configparser==4.0.2; python_version < '3'","zipp<2; python_version < '3'","importlib-metadata<1.6; python_version < '3'"]

[test.pytest]
coverage=auto
coverage_output_dir=dist/coverage

oh sorry, i should say that the command i'm running is

PEX_VERBOSE=9 ./pants test src/test/python/pinterest/container_report/

; i did it with verbose that time

Any ideas on what might cause this error?

+ ./pants test :: --tag=-integration --tag=aiohttp3
/root/.cache/pants/setup/bootstrap-Linux-x86_64/1.8.0rc0/local/lib/python2.7/site-packages/cryptography/__init__.py:39: CryptographyDeprecationWarning: Python 2 is no longer supported by the Python core team. Support for it is now deprecated in cryptography, and will be removed in a future release.
  CryptographyDeprecationWarning,
Exception caught: (<class 'pants.build_graph.address_lookup_error.AddressLookupError'>)

Exception message: Build graph construction failed: MappingError Failed to parse documentation/api/BUILD:
name 'node_module' is not defined

@Grouchy describe redstone

Hey I’ve been trying to read about creating custom targets in pants. What I’m trying to achieve is create a

python_binary

target that can remove dependencies or transitive dependencies and not packaged them. I’ve got some util targets that depend on

tensorflow

, but don’t use it the code paths in certain binaries. I know we could be disciplined and not have it in the build graph but that could add a lot of targets that aren’t really necessary. I’ve used bazel’s

depset

in before achieving something like this: Is there a good way to get this in pants?

Hey Charlie. With the upcoming Pants 2.0, we added

!

to the

dependencies

field as a way to ignore dependencies from a target. The simplest thing to do would be to write a macro (https://www.pantsbuild.org/v2.0/docs/macros) like

custom_python_binary

and have it always add

'!//:tensorflow'

to the

dependencies

field. —

or transitive dependencies

This is a little trickier. In Pants, creating a custom target type and/or using a macro would only influence the direct dependencies for that target. That is, it would only help to avoid this case:

python_binary(
  dependencies=["bad_dep"]
)

Not this case:

python_library(
  name="util",
  dependencies=["bad_dep"],
)

python_binary(
  dependencies=[":util"],
)

-- But, you could use the Rules API to essentially fork the builtin

binary

implementation. You could do something like check “If the target is a

custom_python_binary

, then no matter what exclude tensorflow; otherwise, use dependencies like normal.” You’d use the builtin rules for everything else like

test

, and swap in your rules that you forked. It sounds like correctly handling transitive deps is important too, right?

Yes It’s important for transitive too. How are transitive deps resolved? I was hoping I could extend the

python_binary

implementation with another field

deps_to_remove

or something.

I was hoping I could extend the implementation and just remove everything in

deps_to_remove

.

Another approach is to have two targets referring to the same util code. One with tensorflow, and one without:

python_library(
  name="utils_without_tf",
  dependencies=["foo", "bar"]
)

python_library(
  name="utils",
  dependencies=[":utils_without_tf", "//:tensorflow"]
)

But this means that the BUILD file authors must now go out of their way to depend on the correct target. And generally Pants is less ergonomic when there’s >1 target owning a file. For example, dependency inference wouldn’t be able to infer a dependency anymore because it doesn’t know which of the two targets you want to use.

How are transitive deps resolved?

Transitive deps are resolved through the Rules API. In call sites that need transitive deps, they say

await Get(Targets, TransitiveTargets([input_address1, input_address2])

(https://www.pantsbuild.org/v2.0/docs/rules-api-and-target-api#transitive-dependencies-with-transitivetargets) In your plugin / fork of the

binary

goal, you could filter that out to exclude TF when relevant.

remove everything in deps_to_remove

Yeah, at the moment

deps_to_remove

(via

!

ignores) only applies to your direct deps, not your transitive deps. Possibly, we could add that mechanism - I’m curious if other would find it useful too.

I’d be happy to help you write a plugin if you’re interested / send you the snippet to use. There wouldn’t be much code involved. The main annoyance would be keeping up to date with new Pants releases.

That would be great.

I haven’t gone through enough of the v2 docs. When do you think 2.0.0 will be marked stable?

We’re hoping to have alpha0 tomorrow or Friday. There’s a pretty bad performance regression in the time to create the rule graph during the past 3 releases. We’re very close to landing a fix for.

Possibly, we could add that mechanism - I’m curious if other would find it useful too.

If you’re willing, might be helpful to open an issue with your feature request to be able to exclude transitive deps. https://github.com/pantsbuild/pants/issues/new That way other people could add if they’d find it useful too.

I think it could definitely be useful for other people. I’ll open an issue

Thanks! One question that would be helpful for us to scope it: Would you want this to be expressed in the BUILD file, through the

dependencies

field? Or it’s okay to have more ceremony like having to define a new target type? If we want it to live in BUILD files, one possible syntax is to use

!!ignore_me_transitively

..? vs.

!ignore_me_directly

.

If the precedent is set

!

can ignore things it could be in the

BUILD

files. My concern is that could be too magical and a user wouldn’t understand what is going on. If there is a easy code snip-it you could put on the website it could be helpful to teach people about more advanced api. I don’t have a strong opinion.

What is the preferred way to have

./pants repl

open either a python2 or a python3 repl? Our old way of doing it (v1.18) does not seem to work anymore (we used

. --pants-config-files="${BUILDROOT}/pants.py3.ini"

where

pants.py3.ini

defined

interpreter_constraints: ["CPython>=3.6.8,<4"]

).

If we add an entry point in the Target API, that would be pretty easy for a codebase admin to do. The code would essentially be:

class CustomPythonBinaryDependencies(Dependencies):
   exclude_transitively = ["bad_target"]

class CustomPythonBinary(Target):
  alias = "custom_python_binary"
  core_fields = (*(FrozenOrderedSet(PythonBinary.core_fields) - {Dependencies}), CustomPythonBinaryDependencies)

Then, anytime the BUILD file uses

custom_python_binary

, those hardcoded excluded targets would be ignored. Other than that, it behaves identically to a normal

python_binary

target. But that’s really really special-cased, and not at all obvious where that would fit into the docs. It also might not be generic enough. It requires you to put a list of hardcoded addresses in a target definition, rather than being able to tweak in a decentralized way in a BUILD file.

I’m think that we will either want to a) not support this mechanism officially because it’s too much of a special case / not wanted broadly enough, or b) add the

!!

syntax and very lightly document it

When I run

./pants repl

against a python library target which has

compatibility="CPython>=3.6.8,<4",

I get a py3 repl. When I run it againt a library that does not have any

compatibility

defined, I get a py2 repl. This is with

pants.ini

having a config

interpreter_constraints: ["CPython>=2.7.15,<3","CPython>=3.6.8,<4"]

When I run

./pants repl 3rdparty/python3:Flask

I get a repl dependent on what is in

.pants.d

. If I delete

.pants.d

and run with an ini file that has ``interpreter_constraints: ["CPython>=2.7.15,<3","CPython>=3.6.8,<4"]` I get a py2 repl. If I edit

pants.ini

to have

interpreter_constraints: ["CPython>=3.6.8,<4"]

I still get a py2 repl. If I delete

.pants.d

I then get a py3 repl.

Setting

interpreter_constraints: ["CPython>=2.7.15,<3","CPython>=3.6.8,<4"]

, I still get py3 repo when I run

./pants repl 3rdparty/python3:Flask

even though I had previously got a py2 repl.

Should I be able to update

interpreter_constraints

without having to change the

pants_workdir?