https://pantsbuild.org/ logo
Join Slack
Powered by
Hi everyone, I'm a DevOps Engineer working at Glov...
# welcome
b
Hi everyone, I'm a DevOps Engineer working at Glovo ( a food delivery company ). My team has been considering moving to a monorepo and I am looking into build tools that would help. I'm currently comparing Pants with Bazel, Buck and other similar tools ...
👋 8
h
Welcome! Let us know if we can help with the eval
h
Welcome! In case you haven't seen it, some of our thoughts on Bazel vs Pants https://blog.pantsbuild.org/pants-vs-bazel/
👍 1
b
Hi @happy-kitchen-89482 @hundreds-father-404, what is the best way to manage test dependencies with Pants? Let me explain, I currently have 2 files 
requirements.txt
and 
requirements_test.txt
, how do I tell Pants to use 
requirements_test.txt
only for launching tests?
h
Hey! So typically with Pants you don't need to do this at all
The idiomatic way to use Pants is that your 
requirements.txt
files represent a "universe" of possible requirements, but Pants picks the subset actually needed in any given scenario, based on dependency inference
Are there any overlaps or conflicts between 
requirements.txt
and 
requirements_test.txt
?
h
https://www.pantsbuild.org/docs/python-third-party-dependencies explains the conceptual difference, including the first few paragraphs
h
I understand why you have both for historical reasons and why you might want to have both during a migration, but in the long run, if you're only using Pants, it's common to just have one 
requirements.txt
for all possible requirements. Again, Pants selects the necessary subset that a test or binary needs. This is different than non-Pants workflows where you have to install an entire requirements file in a venv
We should probably write up a blog post explaining this
b
Are there any overlaps or conflicts between 
requirements.txt
and 
requirements_test.txt
?
no there aren’t.
I understand why you have both for historical reasons and why you might want to have both during a migration
exactly! So I have to put everything in 
requirements.txt
but if in the meantime I do something like that:
Copy code
python_requirements(
    name="requirements",
    resolve="default",
)

python_requirements(
    name="test-requirements",
    source="requirements_test.txt",
)

python_tests(
    name="tests",
    sources=["test/**/test_*.py" ],
    dependencies=[":test-requirements"]
)
What do you think about that ?
h
Ah you don't have to put everything in requirements.txt, Pants will happily work with multiple requirements files with exactly the two 
python_requirements
targets you have there. But in the future, if you're fully migrated to Pants, that distinction won't be meaningful, so you can unify them.
Note that in your 
python_tests
file you shouldn't need 
dependencies=[":test-requirements"]
➕ 1
Dependency inference should add the right deps for you
And if it doesn't for some reason (e.g., there is no corresponding import statement to infer from) then you add just that dep
dependencies=["#foobar"]
in your example
(a 
python_requirements
target generates a 
foobar
target for the requirement foobar in the requirements file)
More generally 
dependencies=["path/to:target#foobar"]
- you don't need to have the requirements targets in the same dir as the tests of course
b
Ok, understood! Thanks a lot for the support 👍. Just another question, since it is a monorepo, can I avoid repeating it in every 
BUILD
file of every sub-project?
Copy code
python_requirements(
    name="requirements",
    resolve="default",
)

python_requirements(
    name="test-requirements",
    source="requirements_test.txt",
)
h
Definitely! You only need that once, somewhere in your repo
The root for example, or a 
3rdparty
directory, or wherever
Dependency inference will knit it all together
b
Hello! I take advantage of the thread 🙃. In my 
pant.toml
I add 
pytest-mock
as extra requirements:
Copy code
[pytest]
extra_requirements = [
  "pytest-mock~=3.7",
]
but when I throw the goal test
Copy code
InvalidLockfileError: You are using the `<default>` lockfile provided by Pants to install the tool `pytest`, but it is not compatible with your configuration:

- You have set different requirements than those used to generate the lockfile. You can fix this by updating `[pytest].version` and/or `[pytest].extra_requirements`, or by using a new custom lockfile.
In the input requirements, but not in the lockfile: ['pytest-mock~=3.7']
In the lockfile, but not in the input requirements: ['pytest-cov!=2.12.1,<3.1,>=2.12']
I can't understand why he has this conflict
Ok, I see ...
Copy code
If you change either version or extra_requirements, Pants's default lockfile for Pytest will not work. Either set the lockfile option to a custom path or "<none>" to opt out. See Third-party dependencies.
Another issue on test marked as to be skipped ( 
@pytest.mark.skip("")
) . if I use pytest it skips it immediately, with pants instead it remains in stuck 
Building requirements ...
Can I set debug output as "default" for the test goal ?
h
Hey @bright-book-6256 to be clear, it's totally okay to add 
[pytest].extra_requirements
. Only, you should set 
[pytest].lockfile = "3rdparty/pytest.lock
for example (whatever file path you want), then run 
./pants generate-lockfiles --resolve=pytest
Can I set debug output as "default" for the test goal ?
Like, 
./pants test --debug
? Yes. Every option in Pants can be specified on the CLI, env vars, or config file. See https://www.pantsbuild.org/docs/options Here, you would set in `pants.toml`:
Copy code
[test]
debug = true
Although, there are some downsides to setting that. You won't get concurrency, and also tests won't be cached. So 
./pants test ::
will rerun everything every time - a big reason Pants exissts is for its caching
👍 1
b
Hi @hundreds-father-404, thanks for the support. I'm having trouble generating the lockfile, I have this error:
Copy code
$ ./pants generate-lockfiles --resolve=data-creation
09:05:07.70 [INFO] Initializing scheduler...
09:05:08.00 [INFO] Scheduler initialized.
09:08:51.47 [INFO] Completed: Generate lockfile for data-creation
09:08:51.47 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Generate lockfile for data-creation' failed with exit code 1.
stdout:

stderr:
ERROR: Could not find a version that satisfies the requirement pantsbuild.pants.testutil<2.14,>=2.13.0a0
ERROR: No matching distribution found for pantsbuild.pants.testutil<2.14,>=2.13.0a0
Do you have any suggestions? PS.: I have 2 repos set up, the default one and our internal Artifactory.
h
Do you have Pants plugins in your repo? Is that why you have a dependency on pantsbuild.pants.testutil ?
b
Yes, I have a Pants plugin
I don't know, but it seems that something is not working as it should ... pex takes forever to build the requirements 😑
Mmm... it seems that without the extra repos go much faster 🤨
but it is still too slow :
Copy code
⠈ 917.84s Building requirements.pex with 6 requirements: apache-airflow-providers-slack~=3.0.0, apache-airflow[amazon,docker]~=2.0.2, boto3<2.0,>=1.17, pydantic<2.0,>=1.9, pyyaml<6.0.0,>=5.4.1, ruamel.yaml~=0.17
Any suggestions ? @hundreds-father-404 @happy-kitchen-89482
h
Which version of Pants are you on, and are you using a lockfile?
That definitely doesn't seem like it should take that long...
If you're able to create a dummy github repo that reproduces the issue but doesn't expose your proprietary code (and since the problem is with third-party deps that seems possible) then that would be the best way for us to solve it
b
Hello @happy-kitchen-89482, I am working full-time on the migration of a monorepo to introduce Pants. Pants seems to me teorically the best tool but practically I am having a bit of difficulty. As soon as I can I create a dummy repo hoping for your help. 😐
Which version of Pants are you on, and are you using a lockfile?
Now I'm using 
2.13.0a1
For example, now I have a weird 
ModuleNotFoundError
running the 
./pants test
. My test class has something like that in the head :
Copy code
...
from my_module.cli.docker.utils import ImageAttrs
...
where my_package is a common folder inside the project that contains the sources.
Copy code
E   ModuleNotFoundError: No module named 'my_module.cli'
What am I doing wrong? 🤯 PS: Running 
pytest
I don’t have any error BUILD file:
Copy code
python_sources(name="lib", sources=["my_module/**/*.py"])

python_tests(
    name="tests",
    sources=["tests/**/test_*.py"]
)

python_distribution(
    name="dist",
    dependencies=[":lib"],
    provides=python_artifact(
        name="my-app",
        description="My Application.",
        long_description_file="README.md"
    ),
    generate_setup = True,
    wheel_config_settings={"--global-option": ["--python-tag", "py37.py38.py39"]},
)
My dir structure:
Copy code
├── BUILD
├── my_module
│   └── cli
│       └── docker 
│            ├── __init__.py
│            └── main.py
└── tests
    └── unit_tests
        └── test_main.py
@happy-kitchen-89482 @hundreds-father-404 the best way to write a dynamic version file inside a distribution? Use a plugin?
h
See not-yet-published documentation here: https://github.com/pantsbuild/pants/pull/16102
This feature is available in Pants 2.13+
b
but I would like it to be copied from a version file on the root of my monorepo, not from the vcs
@happy-kitchen-89482 @hundreds-father-404 here is a dummy repo (https://github.com/devcrops-official/pants-project-example) which reflects the structure of the original one. At the moment I have a problem with the lockfile generation:
Copy code
./pants generate-lockfiles --resolve=project-b
I would also like to introduce the creation of the _version.py file within each distribution. Can you help me ?
Also with :
Copy code
./pants generate-lockfiles --resolve=default
I have problem
@happy-kitchen-89482 @hundreds-father-404 I saw that I have this constraint using pants plugin:
Copy code
pantsbuild-pants 2.13.0a0 depends on PyYAML<7.0 and >=6.0
for this reason, I have an error during the generation of lockfile. Because my project has this requirement:
Copy code
awscli 1.25.26 depends on PyYAML<5.5 and >=3.10
😕 I didn't think the pants plugin would affect my build in terms of dependencies... furthermore, the error message should be more complete at the end of the generate-lockfile goal
h
You may need a separate lockfile for your pants plugin, so it doesn't interfere with the requirements of the rest of your repo
👍 1
And that should be fine, since your pants plugin is not needed at your code's runtime
b
Thanks @happy-kitchen-89482, with a separate lockfile it works. But I have another error with lockfile generation, but I don’t know why. Now I don’t have dependencies conflict, 
pip install
works. Requirements file:
Copy code
click==8.0.4
pydantic==1.9.0
pyyaml==5.3.1
requests==2.26.0
sparkmeasure==0.14.0
pyspark==3.1.2
arcgis==1.9.1
fiona==1.8.21
geopandas==0.10.2
h3==3.7.4
numpy==1.21.6
pandas==1.1.5
pyproj==3.2.1
shapely==1.8.0
google-api-python-client==2.49.0
gspread==5.1.1
oauth2client==4.1.3
datadog==0.42.0
docker==5.0.3
luigi==3.0.3
mysql-connector-python==8.0.29
python-json-logger==2.0.2
python-logstash==0.4.6
slackclient==2.9.4
wcmatch==8.1.2
google.api_core==2.8.2
The goal end with:
Copy code
09:19:25.79 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Generate lockfile for data-ingestion' failed with exit code 1.
stdout:

stderr:
ERROR: Could not find a version that satisfies the requirement numpy==1.21.6
ERROR: No matching distribution found for numpy==1.21.6
h
Hmm. Which platform and Python version are you running this on?
And what is the command you're running?
And what is your lockfile config?
b
Hi Benjy, I'm on my Mac OS X (platform darwin) using Python 3.8.10 with Pants 2.13.0a1. The command is :
Copy code
./pants generate-lockfiles --resolve=project-b
About the lockfile configuration I had enabled the resolves and I'm using it. You can find more details on the configuration in my example repo: https://github.com/devcrops-official/pants-project-example/blob/main/3rdparty/python/project-b/requirements.txt
Hello @happy-kitchen-89482, any news on this issue? I have practically finished the "Pants-ification" of my monorepo, only this last thing is blocking me 🙃
h
Hey, sorry, was out for a couple of days
Remind me of the issue? I.e., if I clone that example repo what command should I run to expose it?
b
Hey @happy-kitchen-89482 the command is
Copy code
./pants generate-lockfiles --resolve=project-b
h
OK, finally getting around to looking at this, sorry for the delay.
I can reproduce the problem, so far so good.
👍 1
OK, so this works if you add an upper bound on the interpreter constraint, e.g., 
CPython>=3.7,<3.11
😮 1
Pants (via Pex) tries to resolve a universal lockfile for all possible target interpreters
and that version of numpy requires Python 3.7, 3.8, 3.9 or 3.10
b
Hey @happy-kitchen-89482, thanks a lot for your support! I tried it and the generation-lock file goal now is working! but when I run the pants test goal with this new lockfile generated I have this weird pex error during requirements building ... 😕
Copy code
File "/Users/myhome/.cache/pants/named_caches/pex_root/installed_wheels/6242b902db69f59e1092b406655c0fb1634486c47ce563f5fd27277cf4561822/pex-2.1.90-py2.py3-none-any.whl/pex/pep_376.py", line 446, in from_prefix_install
    raise RecordNotFoundError(pex.pep_376.RecordNotFoundError: Could not find the installation RECORD for python-certifi-win32 1.6.1 under /Users/myhome/.cache/pants/named_caches/pex_root/installed_wheels/508fd4fb1730cad2d9dada061df737650c8cfaa205d64657faa4cc6a55384402/python_certifi_win32-1.6.1-py2.py3-none-any.whl.d7cadc0956bf44ceb0ed26ade60f6281
Any idea?
Copy code
ProcessExecutionFailure: Process 'Building 15 requirements for requirements.pex from the 3rdparty/python/data-ingestion/dependencies_lock.txt resolve: arcgis==1.9.1, click==8.0.4, geopandas==0.10.2, google.api_core==2.8.2, gspread==5.1.1, h3==3.7.4, oauth2client==4.1.3, pandas==1.1.5, pydantic==1.9.0, pyproj==3.2.1, pyspark==3.1.2, pyyaml==5.3.1, requests==2.26.0, shapely==1.8.0, sparkmeasure==0.14.0' failed with exit code 1.
Could be something between these reqs that bothers it?
I'm using a Mac OS X (platform darwin), Python 3.8.10 with Pants 2.13.0a1.
It seems that arcgis has python-certifi-win32 as deps. But I didn’t understand why it breakes!
h
Ugh, unfortunately, unlike its other windows-specific requirements, this one is not gated on the 
platform_system == "Windows"
environment marker.
b
is there any workarounds ? 😕
h
Well, that wheel should be installable on other platforms, that error indicates that Pex thinks the wheel itself is invalid. But I do see a RECORD file in it if I download it from https://pypi.org/project/python-certifi-win32/1.6.1/#files
cc @enough-analyst-54434 any ideas?
I don't reproduce directly in PEX with
Copy code
python -m pex.cli lock create  $'--style=universal' --resolver-version pip-2020-resolver  --interpreter-constraint $'CPython>=3.8,<3.9' $'python-certifi-win32'
That works fine
@bright-book-6256 What do you see under 
/Users/myhome/.cache/pants/named_caches/pex_root/installed_wheels/508fd4fb1730cad2d9dada061df737650c8cfaa205d64657faa4cc6a55384402/python_certifi_win32-1.6.1-py2.py3-none-any.whl.d7cadc0956bf44ceb0ed26ade60f6281
on your system?
b
A folder 
lib
and inside
e
I'm catching up, but the problem artifact is Windows only ... Pex / Pants just recently added support for not locking windows artifacts,
b
inside 
site-packages
e
@bright-book-6256 what version of Pants is this? Apologies if I missed it.
b
@enough-analyst-54434 2.13.0a1 but I tried also with 2.12.0
e
Ok, thanks. Well, easy to repro:
Copy code
$ pex python-certifi-win32==1.6.1
Traceback (most recent call last):
  File "/home/jsirois/.venv/pex/bin/pex", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/jsirois/.venv/pex/lib/python3.11/site-packages/pex/bin/pex.py", line 775, in main
    catch(
    ^^^^^^
  File "/home/jsirois/.venv/pex/lib/python3.11/site-packages/pex/result.py", line 103, in catch
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/jsirois/.venv/pex/lib/python3.11/site-packages/pex/bin/pex.py", line 798, in do_main
    pex_builder = build_pex(
                  ^^^^^^^^^^
  File "/home/jsirois/.venv/pex/lib/python3.11/site-packages/pex/bin/pex.py", line 681, in build_pex
    result = resolve(
             ^^^^^^^^
  File "/home/jsirois/.venv/pex/lib/python3.11/site-packages/pex/resolver.py", line 948, in resolve
    build_and_install_request.install_distributions(
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jsirois/.venv/pex/lib/python3.11/site-packages/pex/resolver.py", line 714, in install_distributions
    for install_result in execute_parallel(
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jsirois/.venv/pex/lib/python3.11/site-packages/pex/jobs.py", line 541, in execute_parallel
    yield spawn_result.spawned_job.await_result()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jsirois/.venv/pex/lib/python3.11/site-packages/pex/jobs.py", line 220, in await_result
    job.wait()
    ^^^^^^^^^^
  File "/home/jsirois/.venv/pex/lib/python3.11/site-packages/pex/jobs.py", line 81, in wait
    self._finalize_job()
    ^^^^^^^^^^^^^^^^^^^^
  File "/home/jsirois/.venv/pex/lib/python3.11/site-packages/pex/jobs.py", line 139, in _finalize_job
    self._finalizer(self._process.returncode)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jsirois/.venv/pex/lib/python3.11/site-packages/pex/pip/tool.py", line 711, in fixup_install
    record = Record.from_prefix_install(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jsirois/.venv/pex/lib/python3.11/site-packages/pex/pep_376.py", line 445, in from_prefix_install
    raise RecordNotFoundError(
    ^^^^^^^^^^^^^^^^^^^^^^^^^^
pex.pep_376.RecordNotFoundError: Could not find the installation RECORD for python-certifi-win32 1.6.1 under /home/jsirois/.pex/installed_wheels/508fd4fb1730cad2d9dada061df737650c8cfaa205d64657faa4cc6a55384402/python_certifi_win32-1.6.1-py2.py3-none-any.whl.353db331daf74436a74d93cf324aa9a9
I'm looking at this over here: https://github.com/pantsbuild/pex/issues/1861
b
Thanks a lot @enough-analyst-54434!
e
Interesting. It looks like that distribution is broken:
Copy code
$ pip install --prefix chroot --no-deps python_certifi_win32-1.6.1-py2.py3-none-any.whl
$ tree -L 4 chroot/
chroot/
└── lib
    ├── python3.10
    │   └── site-packages
    │       ├── certifi_win32
    │       └── python_certifi_win32-1.6.1.dist-info
    └── site-packages
        └── python-certifi-win32-init.pth

6 directories, 1 file
It's trying to install a 
.pth
file and misses. Those must belong to the 
site-packages
directory of a venv which is 
chroot/lib/python3.10/site-packages
in this case and not 
chroot/lib/site-packages
. The issue is in how they package the 
.pth
file:
Copy code
$ zipinfo -1 python_certifi_win32-1.6.1-py2.py3-none-any.whl 
certifi_win32/__init__.py
certifi_win32/bootstrap.py
certifi_win32/wincerts.py
certifi_win32/wrapt_certifi.py
certifi_win32/wrapt_pip.py
python_certifi_win32-1.6.1.data/data/lib/site-packages/python-certifi-win32-init.pth
python_certifi_win32-1.6.1.dist-info/LICENSE
python_certifi_win32-1.6.1.dist-info/METADATA
python_certifi_win32-1.6.1.dist-info/WHEEL
python_certifi_win32-1.6.1.dist-info/top_level.txt
python_certifi_win32-1.6.1.dist-info/RECORD
Instead of storing it as 
python-certifi-win32-init.pth
at the top-level, they attempt to use the data files mechanism. All that said, this should not trip Pex up. I'll get up a fix and Pex release today. Once done I can give you some 
pants.toml
edits here to get you using the fixed Pex.
👍 1
Ok, the fix is released in Pex 2.1.101. If you want to try it out, add this to your `pants.toml`:
Copy code
[pex-cli]
version = "v2.1.101"
known_versions = [
    "v2.1.101|macos_arm64|d1e29e060b79307719be2a40d16234d1cadd5fc1abab039e091ce3cba8a47cd9|3813284",
    "v2.1.101|macos_x86_64|d1e29e060b79307719be2a40d16234d1cadd5fc1abab039e091ce3cba8a47cd9|3813284",
    "v2.1.101|linux_arm64|d1e29e060b79307719be2a40d16234d1cadd5fc1abab039e091ce3cba8a47cd9|3813284",
    "v2.1.101|linux_x86_64|d1e29e060b79307719be2a40d16234d1cadd5fc1abab039e091ce3cba8a47cd9|3813284",
]
# When trying a new pex version, you can find out the hash and size-in-bytes as follows:
# $ curl -s -L <https://github.com/pantsbuild/pex/releases/download/v2.1.101/pex> | tee >(wc -c) >(shasum -a 256) >/dev/null
Alternatively, Pants 2.14.x and 2.13.x are being patched and the next releases of those will have this fix.
😮 1
🙏 1
b
I have already tested it and it works! Thanks a lot @enough-analyst-54434!! 🚀 I'll wait for Pants patched version.
❤️ 1
Hello @hundreds-father-404, @happy-kitchen-89482 @enough-analyst-54434 when will we have a new Pants version with patched PEX ? Also, I'm having some problems with the static code analysis step (SonarScan) in my GitHubAction. It appears that sonar does not recognize the new lines of code when examining the Pants coverage reports... in this way, the coverage of new features is lower because it considers the whole code 😕
Forget the problem with SonarScan, was my misconfiguration.
@happy-kitchen-89482 I am having a weird problem with GitHub actions, if I run 
pants test
some test fails to download java 3rdparty dependencies (not always the same test and not always the same deps) but if I run 
pants test --debug
it works, probably because it is sequential. It seems like I have some bandwidth limit on GitHub (locally works without trouble). Is there a way to configure pants to avoid extreme parallelism?
2 Views
Open in Slack
PreviousNext
Powered by