https://pantsbuild.org/ logo
Join Slack
Powered by
I'm trying to make `pants` work with a dummy proje...
# general
e
I'm trying to make 
pants
work with a dummy project with two apps with different requirements. The apps use different version of a 3rd party package (pandas), which is a transitive dependency. (continuing the description and question in comments)
The project file structure is
Copy code
📁 pants-tests/
├─📁 apps/
│ ├─📁 app-A/
│ │ ├─📁 app_a/
│ │ │ ├─📄 __init__.py
│ │ │ ├─📄 app.py
│ │ │ └─📄 BUILD
│ │ ├─📄 pyproject.toml
│ │ └─📄 BUILD
│ └─📁 app-B/
│   ├─📁 app_b/
│   │ ├─📄 __init__.py
│   │ ├─📄 app.py
│   │ └─📄 BUILD
│   ├─📄 pyproject.toml
│   └─📄 BUILD
├─📄 .env
├─📄 .gitignore
├─📁 lib/
│ ├─📁 lib-E/
│ │ ├─📁 lib_e/
│ │ │ ├─📄 __init__.py
│ │ │ ├─📄 something.py
│ │ │ └─📄 BUILD
│ │ └─📄 pyproject.toml
│ ├─📁 lib-C/
│ │ ├─📁 lib_c/
│ │ │ ├─📄 __init__.py
│ │ │ ├─📄 something.py
│ │ │ └─📄 BUILD
│ │ ├─📄 pyproject.toml
│ │ └─📄 BUILD
│ ├─📁 lib-D/
│ │ ├─📁 lib_d/
│ │ │ ├─📄 __init__.py
│ │ │ ├─📄 something.py
│ │ │ └─📄 BUILD
│ │ └─📄 pyproject.toml
│ └─📁 lib-F/
│   ├─📁 lib_f/
│   │ ├─📄 __init__.py
│   │ ├─📄 something.py
│   │ └─📄 BUILD
│   └─📄 pyproject.toml
└─📄 pants.toml
f
This is supported via having multiple resolves that are parametrized across your targets: https://www.pantsbuild.org/docs/python-lockfiles#multiple-lockfiles
I'd read the whole section about lockfiles (and also about third-party dependencies that comes before it) to load the concepts into your head, it can be a bit different unless you've used a build tool like this before.
In your case, lib-f, lib-c, app-a, pandas 1.5, and numba could all be part of the default resolve
app-b, lib-e, and pandas 2.0 could be part of a named resolve
e
The first problem I am facing is that my IDE (VS Code) is complaining about a missing package, 
numba
in `lib/lib-C/lib_c/something.py`: Reading in Setting up an IDE under "Python third-party dependencies and tools" I get the impression I should use multiple "resolves" and lockfiles. ... and now I'm pausing to read what @flat-zoo-31952 said 😄
f
and lib-d would use 
resolve=parametrize(...)
to appear in both
✅ 1
Please give those a read, give it a try, and let us know if it's still troublesome. I had some difficulty setting it up at first myself, but that was months ago and I haven't thought about it since. It works well once you work out the details.
e
Okay, I read the blog post and the related documentation, but can't say I've yet deeply understood how 
pants
works.
I have the following `pants.toml`:
Copy code
[GLOBAL]
pants_version = "2.16.0"
backend_packages = ["pants.backend.python"]

[source]
root_patterns = ['/lib/*', '/apps/*']

[python]
interpreter_constraints = ['==3.10.*']
enable_resolves = true
default_resolve = "app_a_resolve"

[python.resolves]
app_a_resolve = ".lockfiles/app-A"
app_b_resolve = ".lockfiles/app-B"
So I'm hoping to get two sets of requirements locked
But when I'm running 
pants generate-lockfiles
I'm, getting the following error:
Copy code
niko@niko-ubuntu:~/tmp/pants-tests$ pants generate-lockfiles
18:35:41.02 [INFO] waiting for pantsd to start...
18:35:42.22 [INFO] pantsd started
18:35:42.30 [INFO] Initializing scheduler...
18:35:47.29 [INFO] Scheduler initialized.
18:35:48.46 [INFO] Completed: Generate lockfile for app_b_resolve
18:35:49.19 [INFO] Completed: Generate lockfile for app_a_resolve
18:35:49.19 [ERROR] 1 Exception encountered:

Engine traceback:
  in `generate-lockfiles` goal

ProcessExecutionFailure: Process 'Generate lockfile for app_a_resolve' failed with exit code 1.
stdout:

stderr:
pid 187848 -> /home/niko/.cache/pants/named_caches/pex_root/venvs/edc3cdf8d6ebf14b5ebbdb422cf41b6e4a653c5b/5985ed09b49a653d6596b0e14d134c5456cf1a9f/bin/python -sE /home/niko/.cache/pants/named_caches/pex_root/venvs/edc3cdf8d6ebf14b5ebbdb422cf41b6e4a653c5b/5985ed09b49a653d6596b0e14d134c5456cf1a9f/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -q --cache-dir /home/niko/.cache/pants/named_caches/pex_root/pip_cache --log /tmp/pants-sandbox-oeEs8b/.tmp/pex-pip-log.aq4q8ck5/pip.log download --dest /tmp/pants-sandbox-oeEs8b/.tmp/tmpv7iswc9x/usr.bin.python3.10 lib-D lib-E lib_c --index-url <https://pypi.org/simple/> --retries 5 --timeout 15 exited with 1 and STDERR:
ERROR: Could not find a version that satisfies the requirement lib-D
ERROR: No matching distribution found for lib-D




Use `--keep-sandboxes=on_failure` to preserve the process chroot for inspection.
what could be the reason?
pants
is at least finding the lib-D with `pants roots`:
Copy code
niko@niko-ubuntu:~/tmp/pants-tests$ pants roots
apps/app-A
apps/app-B
lib/lib-C
lib/lib-D
lib/lib-E
lib/lib-F
why it says "No matching distribution found for lib-D" ?
I have a feeling it is coming from how I defined the 
pyproject.toml
for lib-C ( 
lib/lib-C/pyproject.toml
):
Copy code
[project]
name = "lib-C"
requires-python = ">=3.10"
version = "0.1.0"
dependencies = ["lib-D"]

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"


[tool.setuptools.packages.find]
where = ["."]
include = ["lib_c*"]
That's probably not a valid 
pyproject.toml
at all. I wonder how should the 
lib-D
be defined in the dependencies of 
lib-C
?
f
Oh you're really using pyproject.toml? I thought that was just an example I don't actually know how Pants parses those
what does lib/lib-D/BUILD look like?
e
It seems the `pyproject.toml`s are perfectly fine. At least when installing with plain pip in a fresh venv. I first installed lib-D and then lib-C. Could import and call functions defined in the 
lib_d.something
and 
lib_c.something
.
Actually seems that I don't have a 
lib/lib-D/BUILD
. Maybe that's the issue.
f
what does 
lib/lib-D/BUILD
look like? Pants doesn't really care about pip packages for first -party code
yeah... you need to run 
pants tailor ::
to have pants populate targets for things that aren't created
Pants only really uses pyproject.toml to learn about thrid-party requirements. https://www.pantsbuild.org/docs/python-third-party-dependencies#pep-621-compliant-pyprojecttoml It will use dependency inference to to figure out what bits of your first-party code depends on the other ones
e
Seems that 
lib/lib-D/BUILD
is not created automatically with 
pants tailor ::
. It creates 
lib/lib-D/lib_d/BUILD
if I remove it, for example.
f
i guess there's no targets it finds there
e
I have created manually 
lib/lib-C/BUILD
which has
Copy code
python_requirements(source="pyproject.toml")
so probably I should also make similar 
lib/lib-D/BUILD
f
I think you might actually get away from the multiple pyproject.toml files, it's probably causing more harm than good
You don't really need them for Pants unless you also want to use setuptools or poetry or something directly on those subtrees as little mini projects
What I would do for your case is just create two requirements.txt files, one for each resolve, and dump all your 3rd-party requirements in those
Copy code
ERROR: Could not find a version that satisfies the requirement lib-D
ERROR: No matching distribution found for lib-D
what that is saying is that you told pants there's a pip package called "lib-D" that it should be able to download
e
really? I'm only using one 
pyproject.toml
per library/app as in the real monorepo (similar to this but many times larger) I have one 
setup.py
per library/app, which are used to define dependencies and possibly some CLI entry points. Since 
pants
does not support 
setup.py
, I thought I would convert all the setup.py to pyproject.toml.
f
but lib-D is first party source that you have in repo, so it really should be picking it up via tagets/dependency inference
Ah, you have a... workspace-based monorepo
e
I have to google out the word "workspace-based"
Maybe I do 😅
f
It's my term, I wouldn't google it
😄 1
I mean you have a collection of well-structured projects living in the same repo, but they are all defined as projects, and you probably have some glue code now that works out how they relate?
e
I thought 
pants
would find 
lib-D
since I've said that the source files are located in these folders in pants.toml:
Copy code
[source]
root_patterns = ['/lib/*', '/apps/*']
f
your python is in 
lib/*/*
and 
apps/*/*
though
oh wait scratch that
if you want to use the pyproject.toml files, you'll need to do this in every directory where they are 
python_requirements(source="pyproject.toml")
Pants doesn't tailor them because I think tailor looks for requirements.txt
✅ 1
but you'll need to make sure each requirement is only mentioned once, and you don't want to include your first party requirements as dependencies there
Pants will treat the union of all 
python_requirements
targets as the universe of 3rd party dependencies, and then it will map each import statement to those dependencies
e
yeah. I actually did add 
lib/lib-D/BUILD
with
Copy code
python_requirements(source="pyproject.toml")
and got
Copy code
niko@niko-ubuntu:~/tmp/pants-tests$ pants generate-lockfiles
18:58:35.20 [ERROR] 1 Exception encountered:

Engine traceback:
  in `generate-lockfiles` goal

KeyError: 'No section `project.dependencies` or `project.optional-dependencies` found in lib/lib-D/pyproject.toml'
The problem being supposedly that I don't have any dependencies in the pyproject.toml. The 
lib/lib-D/pyproject.toml
looks like this:
Copy code
[project]
name = "lib-D"
requires-python = ">=3.10"
version = "0.1.0"
dependencies = []

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"


[tool.setuptools.packages.find]
where = ["."]
include = ["lib_d*"]
and you can see the 
dependencies
is an empty list.
f
if you specify the same requirement twice in the repo, it will get confused and complain
so... lib-D has no external dependencies?
then it doesn't need a 
python_requirements
target
e
No. This is dummy project, so lib-D has no dependencies.
With empty 
lib/lib-D/BUILD
I get again
Copy code
niko@niko-ubuntu:~/tmp/pants-tests$ pants generate-lockfiles
19:10:38.71 [INFO] Completed: Generate lockfile for app_b_resolve
19:10:39.31 [INFO] Completed: Generate lockfile for app_a_resolve
19:10:39.32 [ERROR] 1 Exception encountered:

Engine traceback:
  in `generate-lockfiles` goal

ProcessExecutionFailure: Process 'Generate lockfile for app_a_resolve' failed with exit code 1.
stdout:

stderr:
pid 191531 -> /home/niko/.cache/pants/named_caches/pex_root/venvs/edc3cdf8d6ebf14b5ebbdb422cf41b6e4a653c5b/5985ed09b49a653d6596b0e14d134c5456cf1a9f/bin/python -sE /home/niko/.cache/pants/named_caches/pex_root/venvs/edc3cdf8d6ebf14b5ebbdb422cf41b6e4a653c5b/5985ed09b49a653d6596b0e14d134c5456cf1a9f/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -q --cache-dir /home/niko/.cache/pants/named_caches/pex_root/pip_cache --log /tmp/pants-sandbox-YM7QeM/.tmp/pex-pip-log.5m_08x13/pip.log download --dest /tmp/pants-sandbox-YM7QeM/.tmp/tmpooq34rsw/usr.bin.python3.10 lib-D lib-E lib_c numba --index-url <https://pypi.org/simple/> --retries 5 --timeout 15 exited with 1 and STDERR:
ERROR: Could not find a version that satisfies the requirement lib-D
ERROR: No matching distribution found for lib-D




Use `--keep-sandboxes=on_failure` to preserve the process chroot for inspection.
f
the issue you got before was that 
lib-C
had 
lib-D
as a dependency, but Pants is using 
python_requirements(source="pyproject.toml")
as a source of third party deps
👍 1
exactly, it can't find 
lib-D
on PyPI
this is why I'm telling you it's probably best not to use pyproject.toml for this, I think it's confusing you, because you're bringing in concepts that don't really apply in pants
e
Oh 😄 for 
pip
+ 
venv
workflow it has not mattered if you list 1st party or 3rd party dependencies in setup.py
f
well it does actually, because pip has to know how to find a dependency, that must be configured somewhere in your current setup
e
Yeah you're correct. I think in the end it is just a list of folders in some 
requirements.txt
.
f
just a blank 
pip install lib-d
doesn't work, you have to tell pip the path the file somehow, or the lib has to be published on an index you have configurerd
e
I think it works if you happen to have a folder 
lib-D
at current working directory, though
f
yes, but that's "lucky"
👍 1
because pip works on paths
pip install lib/lib-D
would work from the root dir of your example repo
but if you 
cd lib
it wouldn't anymore, and 
pip install lib-D
would
👍 1
e
Ok, so 
python_requirements
is used to define 3rd party dependencies. Does this mean that BUILD files are used to define only 3rd party requirements?
f
anyways... the typical "Pants way" for setting up shared 3rd party dependnecies would be to have some folder or files that define the requirements for each resolve. 
python_requirments
will pick these up. You can have more than one 
python_requirements
but they need to be disjoint, i.e., you can't have the same dependency in two different 
python_requirements
sources in the same resolve.
Your first party code also goes in BUILD files, but it will show up as 
python_sources
I'm guessing 
cat lib/lib-D/lib_d/BUILD
will yield something like 
python_sources()
as the content
✅ 1
e
that's correct, 
lib/lib-D/lib_d/BUILD
has 
python_sources()
in it
it was created by 
pants tailor ::
f
All these different targets are what allow Pants to construct a graph of dependencies, and it will figure out what depends on what
the target definitions in BUILD files are just about teaching Pants what its even operating on
e
"..same dependency in two different 
python_requirements
sources in the same resolve " <-- what is a "resolve" here?
f
I've gotta run, but I suggest skipping the resolve stuff for a second while you get a basic 
pants test
workflow working with resolves turned off (and just a single pandas version). Resolves make things complicated, so i'd add that after you get the hang of the workflow, it's very different
✅ 1
a "resolve" is one resolution of python requirements to a concrete set of dependencies, expressed via a lockfile
e
Okay, so I will simply make a requirements.txt for each app?
Or, that you would recommend to do that instead of the resolves?
f
I would make a requirements.txt only for the third party deps you need (pandas and numba, right?)
just use one version of pandas for now
e
pandas and numba for this dummy project. Of course many more for the real one.
f
yeah but the dummy is good practice for loading the concepts into your head
✅ 1
start with just one version of pandas, make a requirements.txt file with that dep in it, teach pants about it (either via 
pants tailor
or manually creating the 
python_requirement
target)
and then wherever your python code uses 
import pandas
pants will know to include pandas
e
Do you mean that with the requirements.txt approach it is not possible to use different version of pandas for different app?
f
and wherever your python code uses 
import lib_d
it will know it needs to include 
lib_d
No, but you need to deal with multiple resolves for that, and that complicates things. I'm just suggesting an order to your experimentation process.
Gotta run
e
thanks a lot @flat-zoo-31952! 🙏
2 Views
Open in Slack
PreviousNext
Powered by