https://pantsbuild.org/ logo
#general
Title
# general
r

rhythmic-morning-87313

04/29/2022, 12:32 PM
So here is my fist batch of questions. Q1: I have
packages/common/pyproject.toml
. I'd like to refer this in`src/ai/backend/common/BUILD` 's
resource
command, either in relative paths or absolute paths (based on the root), but it says it's not allowed. Is copying
pyproject.toml
to
src/ai/backend/common
the only way? Q2: Is it possible to use existing
setup.cfg
and their
install_requires
configs in each project to specify 3rd-party dependencies per
BUILD
target? Q3: In
pants.toml
I could specify per-tool options such as
pytest.args
and
pytest.extra_requirements.add
. How do I specify per-tool per-project options as well? (e.g., each project requires different sets of
xxx-types
packages for type check) Q4: How to single-source the version across multiple distribution targets in a (Python-based) mono-repository?
1
c

curved-television-6568

04/29/2022, 12:42 PM
A1: The rule is that a BUILD file may only have targets for files in its own directory or its subtree. Commonly, you have the target defintion, such as the
resource
in the BUILD file in the same directory as it’s sources. Rather than copying, you may add a dependency between your
resources
, so there may be a
resource
in your ai backend with a dependency to the resource from your common tree. (dependencies may go any where, so are not restricted the same way as sources are)
👌 1
A2: You may provide your own
setup.py
and
setup.cfg
per
python_distribution
. See https://www.pantsbuild.org/docs/reference-python_distribution#codegenerate_setupcode
A3: I’m a little bit hazy on the details here. With Pants 2.11 there’s a new resolves feature to support multiple disjoint requirements. However I’m not sure if this is usable for the tools, as of now. Also, it does not allow for per resolve tool args, either. So I guess you can’t provide different options on a per target basis here. One possibility could be to have different config files, and pair the config file with the various targets you want to use with those settings. CLI aliases may be used to make this work easier. https://www.pantsbuild.org/docs/reference-global#section-pants-config-files https://www.pantsbuild.org/docs/reference-cli#section-alias
r

rhythmic-morning-87313

04/29/2022, 12:49 PM
About A2, is it sufficient to specify
setup.cfg
as a dependency of
python_distribution()
?
c

curved-television-6568

04/29/2022, 12:50 PM
Not sure. Perhaps @happy-kitchen-89482 has the answer to that one.
r

rhythmic-morning-87313

04/29/2022, 12:51 PM
About A3, hmm... since
setup.cfg
has
extra_requires
for
test
,
typecheck
,
lint
, etc., it would be best to reuse them if possible.
Anyway, thanks for super quick replies!
👍 1
c

curved-television-6568

04/29/2022, 12:52 PM
I’ll give hints at what I have, and leave the rest for the other maintainers to fill in the gaps 🙂
🙌 1
Regarding Q4, do you refer to the version field of the
python_artifact
, i.e. the distribution version you’re building?
If so, the answer is likely to use a plugin that provides custom args to the generated setup.py file. https://www.pantsbuild.org/docs/plugins-setup-py
r

rhythmic-morning-87313

04/29/2022, 1:47 PM
Hm... About A1, I've tried this, but it still says
Copy code
NoOwnerError: No python_distribution target found to own packages/common:pyproject. Note that the owner must be in or above the owned target's directory, and must depend on it (directly or indirectly). See <https://www.pantsbuild.org/v2.12/docs/python-distributions> for how python_sources targets are mapped to distributions. See <https://www.pantsbuild.org/v2.12/docs/python-distributions>.
c

curved-television-6568

04/29/2022, 1:54 PM
Ah, yes, as the message is hinting at, the python_distribution must live as an umbrella for all sources it includes. It’s a little different for distributions than resources. For more info, see: https://www.pantsbuild.org/v2.11/docs/python-distributions#mapping-source-files-to-distributions
r

rhythmic-morning-87313

04/29/2022, 1:59 PM
hmmm so it still requires the pyproject.toml to be placed under the same directory because it infers the "closest ancestry" and in my case it becomes the root directory
the common ancestry of
packages/common:pyproject
and
src/ai/backend/common:lib
is
/
i'm ok with this if I could go with Q5
c

curved-television-6568

04/29/2022, 2:01 PM
Uhm, yes, it seems so. Another question, do you intend to ship the pyproject.toml with multiple distributions? (I think it may complain if you have the same files be part of more than one distribution)
r

rhythmic-morning-87313

04/29/2022, 2:04 PM
I'd like to have separate pyproject.toml (or setup.cfg) per distribution
just wondering how to combine with the unified requirements.txt
c

curved-television-6568

04/29/2022, 2:09 PM
If you have your dependencies listed in your pyproject.toml (as done by Poetry for instance), you don’t need requirements.txt files. (if that helps) https://www.pantsbuild.org/v2.11/docs/python-third-party-dependencies#poetry
r

rhythmic-morning-87313

04/29/2022, 2:10 PM
yeah, but I want to unify the requirements and let individual distributions have the requirements that they really depend on only, if possible.
because I'd like to make it possible to install the whole monorepo in a single virtualenv, while CI/CD packages the distributions separately
c

curved-television-6568

04/29/2022, 2:12 PM
That’s a good goal. What are doing with the pyproject.toml in the distribution?
r

rhythmic-morning-87313

04/29/2022, 2:12 PM
the reason is to keep consistency of 3rd party dependencies across different distributions because they also relies on each other
it's a just test conversion from setup.cfg because the doc mentions PEP-517 and so
c

curved-television-6568

04/29/2022, 2:13 PM
If those are in the same monorepo, there’ll be dependencies for them automatically. Ah, when generating setup.py, I have no experience with when you provide setup.py yourself.
r

rhythmic-morning-87313

04/29/2022, 2:13 PM
i'm not sticking with pyproject.toml until PEP-621 finally arrives
c

curved-television-6568

04/29/2022, 2:14 PM
I think the easiest would be to have the pyproject.toml file closer to the python_distribution target
h

happy-kitchen-89482

04/29/2022, 9:41 PM
Yeah IIRC the
setup.cfg
would need to be in a
resources()
target that the
python_distribution()
target depends on
6 Views