pantsbuild #general

Channels

welcome

github-notifications

announce

flat-byte-50827

05/23/2022, 8:46 PM

Hi everyone, I have a bit of a specific question so apologies for the long message. Any help would be very appreciated! I am posting to ask what the best way is to fix the versions of sub-dependencies in a project that uses the pants build tool. Here is an outline of the issue we faced and how we want to prevent it in the future. Here is our repo structure: our_awesome_repository | pants.toml | pants |--src | |--all_modules | |--api | server.py | BUILD | |--cool_stuff | methods.py | BUILD | |--awesome_stuff | methods.py | BUILD |--3rdparty | |--python | requirements.txt | BUILD Our server.py file uses Flask, so in our 3rdparty/python/requirements.txt file we have: Flask==1.1.2 And in our src/all_modules/api/BUILD file, we have a python_sources looking like this: python_sources( name="lib", dependencies = [ "3rdparty/python:Flask", "src/all_modules/cool_stuff:dist ] ) Recently, we tried deploying the api, but encountered the following issue. Our version of Flask depends on jinja2, and a new version of jinja2 was released which broke our version of Flask. Since we hadn't specified a version for jinja2, our build process was automatically installing the latest version of jinja2, which broke our api. Here is a stackoverflow post where others encountered the same issue: https://stackoverflow.com/questions/71718167/importerror-cannot-import-name-escape-from-jinja2 In order to resolve this, we specified the version jinja2 in our 3rdparty/python/requirements.txt to a previous non-breaking version. So currently, our 3rdparty/python/requirements.txt contains: Flask==1.1.2 jinja2==3.0.3 However, when we tried redeploying, we still saw that the most recent version of jinja2 was being installed. It wasn't until we updated the src/all_modules/api/BUILD file that we were able to fix the version of jinja2 being installed. Currently, our src/all_modules/api/BUILD file contains: python_sources( name="lib", dependencies = [ "3rdparty/python:Flask", "3rdparty/python:jinja2", "src/all_modules/cool_stuff:dist ] ) We would like to specify the versions of all of the subdependencies of our 3rd party imports in order to prevent this issue in the future. To do this, so far we have just updated the 3rdparty/python/requirements.txt file fixing the versions of all of the subdependencies (using virtualenv and pip freeze). However, based on the fix with jinja2, it would seem that we would need to manually update all of the BUILD files with the subdependencies. This seems like a very ugly and difficult solution to maintain, so we are hoping that there is a more elegant option. Essentially I am posting to ask, why was it the case that we needed to specify jinja2 in our BUILD file in order for pants to install the correct version from the requirements.txt? Is there a way to get around manually adding the subdependencies in all of the BUILD files? If there is any additional information needed in order to understand the question or the structure of our repo, please let me know and I will be happy to provide what I can. I really appreciate any help with this! Thanks

witty-crayon-22786

05/23/2022, 9:07 PM

another thread was just started about this with additional context on performance: https://pantsbuild.slack.com/archives/C046T6T9U/p1653336125258069 it seems like the primary thing that influenced this no longer being a symlink to an immutable venv was the desire to be able to

pip install

in the venv. but if that is going to come at this much of a performance cost, it doesn’t seem like it should be the default. “symlink to immutable venv” is pretty safe otherwise.

👍 1

witty-crayon-22786

05/23/2022, 9:20 PM

(they’ve now opened https://github.com/pantsbuild/pants/issues/15607 for it)

cold-soccer-63228

05/23/2022, 9:57 PM

What is the suggested resolution when the name of an in-repo plugin conflicts with a package name? I've written an

ariadne

plugin and

jinja2

plugin to infer some dependencies when using these libraries. I'm following Building in-repo plugins with Pants so that I can write tests for my plugins. But if I add the path to my plugins to the

[source]

part of my

pants.toml

, I start seeing errors like the following.

Copy code

The target path/to/first_party.py imports `ariadne.make_executable_schema`, but Pants cannot safely infer a dependency because more than one target owns this module, so it is ambiguous which to use: ['//:requirements#ariadne', 'pants-plugins/ariadne/__init__.py'].

Please explicitly include the dependency you want in the `dependencies` field of path/to/first_party.py, or ignore the ones you do not want by prefixing with `!` or `!!` so that one or no targets are left.

Alternatively, you can remove the ambiguity by deleting/changing some of the targets so that only 1 target owns this module. Refer to <https://www.pantsbuild.org/v2.11/docs/troubleshooting#import-errors-and-missing-dependencies>.

It would be a huge headache to manually do what the error message suggests for every

BUILD

file. One alternative I'm thinking is publishing these in-repo plugins to PyPI, and then referencing them in our codebase, but that has the downside of slowing down the iteration cycle 😞

green-match-66337

05/23/2022, 10:33 PM

Is there a way to hook in to a build target and run some functionality before/afterwards?

mysterious-motherboard-97366

05/23/2022, 10:55 PM

I have a lockfile generated by pex which contains

bleurt@ git+<https://github.com/google-research/bleurt.git>; extra == "heavy"

under

requires_dists

, but this string causes a parse error later when attempting to build a pex from the lockfile (e.g., in

./pants run

). Is there a potential workaround for this issue? I reported the issue at https://github.com/pantsbuild/pants/issues/15614 which has more details

wide-midnight-78598

05/23/2022, 11:15 PM

This may have been me that asked, but is/was there a plan to add optional encryption to archives?

✅ 1

high-energy-55500

05/24/2022, 1:58 AM

this might be a dumb question, but what’s the “right” way to write a file into the same directory as your repo when running a script through a

pex_binary

? using

__file__

as a starting point doesn’t work since all the files are running in pants’ temp directory

hundreds-father-404

05/24/2022, 4:37 AM

I got ignore CLI args working, so you can do

./pants test project:: '!project/integration_test::'

🎉 The issue is that you have to quote the

or your shell won't be happy. So @curved-television-6568 suggested instead using

, like

-project/integration_test::

, which mirrors how

--tag

works. However, that causes ambiguity with short-flags like

-dmy_custom_dir

, which equals

--logdir=my_custom_dir

. How do people feel about getting rid of *the ability for arbitrary short flags, to make room for ignore CLI args?

✅ 1

green-match-66337

05/24/2022, 7:05 AM

Question - if the name doesnt match then dependency inference removes the package... eg

pybars3

in requirements.txt is used as

pybars

how do we get DI to ignore?

✅ 2

adorable-engine-71736

05/24/2022, 9:36 AM

hi, im trying to add a python package that is a

pytest

plugin - to test it i think i need to have it added in to my

.pytest.lock

but as the package is not yet published im struggling to get

generate-lock-files

to recognize it i have tried adding

"pytest-iters @ file:///src/workspace/pytooling/pytest-iters",

pytest::extra_requirements

pants.toml

but it doesnt find my dep if i try to generate the lockfile i have an existing pytest-plugin so i must have done this before somehow - but seem to be in a chicken and egg situation where i can add the plugin until its published, or publish the plugin till its added

✅ 1

high-energy-55500

05/24/2022, 10:11 AM

is there a way to disable these warnings for specific missing file targets?

Copy code

[WARN] Unmatched globs from //:git-files's `sources` field: [".git/HEAD", ".git/refs/heads/**/*"]

i have a few

files

targets that I don’t care if they’re missing or not, so I don’t mind hiding the warning for them. I’d probably even say that I usually don’t care about missing

files

targets as opposed to missing

resources

targets.

adorable-engine-71736

05/24/2022, 1:48 PM

is there a way to exclude specific files from (

mypy

) typechecking ? if add

--exclude

to my mypy args, or add in my

mypy.ini

they seem to be ignored - im guessing because the filepaths are passed explicitly - im wondering if there is a pants way to prevent some files from being included

✅ 1

rhythmic-morning-87313

05/24/2022, 3:24 PM

Q. When there are resource files for different archtiectures (e.g., prebuilt binaries for x86-64 and aarch64), is there any way to include them selectively when building packages for a specific platform?

brash-baker-91190

05/24/2022, 3:36 PM

I have a question about Python interpreter constraints. I'm moving us from 3.7 to 3.8. In

pants.toml

, I have:

Copy code

[python]
interpreter_constraints = [">=3.8,<4"]

[python-bootstrap]
search_path = ["<PYENV_LOCAL>", "<PYENV>", "<PATH>"]

.python-version

file has

3.8.13

. My

black

configuration is:

Copy code

[black]
lockfile = "3rdparty/python/black.lockfile"
version = "black>=22.3.0"
config = "pyproject.toml"

and

black.lockfile

contains the following metadata:

Copy code

// --- BEGIN PANTS LOCKFILE METADATA: DO NOT EDIT OR REMOVE ---
// {
//   "version": 2,
//   "valid_for_interpreter_constraints": [
//     "CPython<4,>=3.8"
//   ],
//   "generated_with_requirements": [
//     "black>=22.3.0",
//     "typing-extensions>=3.10.0.0; python_version < \"3.10\""
//   ]
// }
// --- END PANTS LOCKFILE METADATA ---

When I run

./pants fmt ::

black

runs fine. However, when I run

./pants update-build-files

, I get the following error:

Copy code

11:33:26.58 [ERROR] 1 Exception encountered:

  InvalidLockfileError: You are using the lockfile at 3rdparty/python/black.lockfile to install the tool `black`, but it is not compatible with your configuration: 

- You have set interpreter constraints (`CPython<4,>=3.7`) that are not compatible with those used to generate the lockfile (`CPython<4,>=3.8`). You can fix this by not setting `[black].interpreter_constraints`, or by using a new custom lockfile. 

To regenerate your lockfile based on your current configuration, run `./pants generate-lockfiles --resolve=black`.

Why would

black

work in one instance but not the other, and where might that

>=3.7

constraint be coming from? Thanks in advance!

brash-baker-91190

05/24/2022, 3:36 PM

(Pants 2.11.0, btw)

calm-ambulance-65371

05/24/2022, 3:42 PM

I'm trying out remote execution using buildgrid and it seems to hang on this operation: `Remote execution: Searching for

bash

on PATH=/usr/bin/bin/usr/local/bin` Any ideas what's up? Is my config bad? Bash is definitely in

/usr/bin/bash

on the remote system

sparse-lifeguard-95737

05/24/2022, 4:58 PM

we’re trying to switch to the new pex-based lockfiles in 2.11, and we’re seeing errors like this in CI runs:

Copy code

Traceback (most recent call last):
  File "/github/home/.cache/pants/named_caches/pex_root/venvs/s/037c1085/venv/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 223, in _main
    status = self.run(options, args)
  File "/github/home/.cache/pants/named_caches/pex_root/venvs/s/037c1085/venv/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 180, in wrapper
    return func(self, options, args)
  File "/github/home/.cache/pants/named_caches/pex_root/venvs/s/037c1085/venv/lib/python3.8/site-packages/pip/_internal/commands/wheel.py", line 174, in run
    build_successes, build_failures = build(
  File "/github/home/.cache/pants/named_caches/pex_root/venvs/s/037c1085/venv/lib/python3.8/site-packages/pip/_internal/wheel_builder.py", line 338, in build
    wheel_file = _build_one(
  File "/github/home/.cache/pants/named_caches/pex_root/venvs/s/037c1085/venv/lib/python3.8/site-packages/pip/_internal/wheel_builder.py", line 238, in _build_one
    _verify_one(req, wheel_path)
  File "/github/home/.cache/pants/named_caches/pex_root/venvs/s/037c1085/venv/lib/python3.8/site-packages/pip/_internal/wheel_builder.py", line 193, in _verify_one
    with zipfile.ZipFile(wheel_path, allowZip64=True) as zf:
  File "/opt/python/3.8.12/lib/python3.8/zipfile.py", line 1269, in __init__
    self._RealGetContents()
  File "/opt/python/3.8.12/lib/python3.8/zipfile.py", line 1336, in _RealGetContents
    raise BadZipFile("File is not a zip file")
zipfile.BadZipFile: File is not a zip file
Traceback (most recent call last):
  File "/github/home/.cache/pants/named_caches/pex_root/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/jobs.py", line 512, in execute_parallel
    yield spawn_result.spawned_job.await_result()
  File "/github/home/.cache/pants/named_caches/pex_root/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/jobs.py", line 216, in await_result
    job.wait()
  File "/github/home/.cache/pants/named_caches/pex_root/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/jobs.py", line 75, in wait
    self._check_returncode(stderr)
  File "/github/home/.cache/pants/named_caches/pex_root/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/jobs.py", line 144, in _check_returncode
    raise self.create_error(msg, stderr=stderr)
  File "/github/home/.cache/pants/named_caches/pex_root/installed_wheels/6a5463833452712ecf58a573b1b489c81d14aaed599e955c2af85cf793217a42/pex-2.1.84-py2.py3-none-any.whl/pex/jobs.py", line 125, in create_error
    raise self.Error(
pex.jobs.Error: Executing /github/home/.cache/pants/named_caches/pex_root/venvs/7bbcd6f261771e5523bda29675060150642cf629/f679563b742633f5edaba61ca7d405616ffc49b3/pex --disable-pip-version-check --no-python-version-warning --exists-action a --use-deprecated legacy-resolver --isolated -q --cache-dir /github/home/.cache/pants/named_caches/pex_root wheel --no-deps --wheel-dir /github/home/.cache/pants/named_caches/pex_root/built_wheels/sdists/CensusData-1.3.tar.gz/127dc284eb8aec79109954d93cc45f450f3a42c683f8beed215cc23610b005ba/cp38-cp38.d045f6a0ac354ac0a72574c89b2710a4 /github/home/.cache/pants/named_caches/pex_root/downloads/127dc284eb8aec79109954d93cc45f450f3a42c683f8beed215cc23610b005ba/CensusData-1.3.tar.gz --index-url <https://pypi.org/simple> --retries 5 --timeout 15 failed with 2

that error specifically mentions

CensusData

but I’ve seen it on other packages too so I don’t think it’s related to the package itself. has anyone else seen this error and know what’s going on?

brainy-solstice-27042

05/24/2022, 5:57 PM

I have a couple more questions on my way to upgrading pants to 2.11: 1. Is there a good way to get pytest to ignore

__init.py

common.py

or any other files that don't have tests in them? 2. I have some tests that import

.json

files, which traditionally have been provided using a resources target and then having

:test-files

in the dependencies for the

python_tests

. After changing the version to 2.11, I find that my tests are no longer able to load the test files like they had in the past. I tried switching to the

files

target from

resources

but that didn't change anything.

high-yak-85899

05/24/2022, 11:41 PM

Got an interesting question: When

pip install ansible

, I get a binary in my venv like

venv/bin/ansible

. When I package

ansible

into a

pex_binary

, is there a place I could still access that binary?

salmon-summer-40505

05/25/2022, 3:56 AM

Copy code

Engine traceback:
  in select
  in pants.core.goals.check.check
  in pants.backend.scala.goals.check.scalac_check (scalac)
  in pants.engine.internals.graph.coarsened_targets
  in pants.engine.internals.graph.transitive_dependency_mapping
  in pants.engine.internals.graph.resolve_unexpanded_targets (/some/file/path.scala)
  in pants.engine.internals.graph.resolve_dependencies (/some/file/path.scala)
  in pants.backend.scala.dependency_inference.rules.infer_scala_dependencies_via_source_analysis (/some/file/path.scala)
  in pants.jvm.dependency_inference.symbol_mapper.merge_first_party_module_mappings
  in pants.backend.scala.dependency_inference.symbol_mapper.map_first_party_scala_targets_to_symbols
  in pants.backend.scala.dependency_inference.scala_parser.resolve_fallible_result_to_analysis
  in pants.backend.scala.dependency_inference.scala_parser.analyze_scala_source_dependencies
  in process
Traceback (no traceback):
  <pants native internals>
Exception: Error communicating with server: invalid digit found in string

salmon-summer-40505

05/25/2022, 3:56 AM

Encounter this, any idea ?

rapid-exabyte-76685

05/25/2022, 4:31 AM

I’m trying to setup a new environment (Windows WSL running Ubuntu) - is there a requirement for there to be at least a system Python 3.7/.8/.9 instead of relying on pyenv? I have pyenv installed and 3.10.2 (required by the project that Pants is running) and also installed 3.9.12 and set as global via pyenv but pants can’t find a python interpreter. I have…

Copy code

[python-bootstrap]
search_path = ["<PYENV>"]

and I tried adding

"<PATH>"

but that didn’t help

adorable-engine-71736

05/25/2022, 12:10 PM

re pytest-plugins - im hitting another ~related issue with the lockfiles - basically if i try to test the pytest-plugin package it always tests against the one that it has in the lockfile, not the one in my repo i guess the workaround is to use a separate lockfile when testing the plugin package, but not sure how to achieve that with pytest. Perhaps this can also be resolved with new PEX way - wondering if i can test this already

hallowed-church-39499

05/25/2022, 12:34 PM

Recently I posted issue #15543 on GitHub (the slack link was not functioning at that time) where I indicated difficulties we have with running

pants test

on a monorepo with several independent Python subprojects. In a nutshell, some of the tests for a subproject will only succeed if the subproject is pip installed (or something equivalent) -- we use e.g.

importlib.metadata.version()

to verify the installed version. The reply I got on the issue helped me somewhat further, but I am still stuck. The layout of the subprojects is as follows

Copy code

src/
  python/
    my-project/
      my_project/
        BUILD
        __init__.py
    	main.py
    	other.py
      tests/
    	BUILD
        __init__.py
    	test_my_project.py
      pyproject.toml
      BUILD

We use poetry-style dependencies, documented in

pyproject.toml

. When running tests manually, we simply activate the subproject's virtual envirnment, poetry install the subproject, and run pytest. I understand that pants uses a different approach, and does not by default install the subproject in a virtual environment, only the dependencies that that subproject has. I was told that to install the subproject, I would need to create a

python_distribution

target, and have the tests depend on that. This is where I get stuck. As far as I can see from experimenting, I need to define the distribution target in the top-level BUILD file (the one under

my-project

. But I seem to be unable to declare a dependency on that target from within the BUILD file under

tests

. When I specify

runtime_package_dependencies

with that target name I specified in the top-level BUILD file, I get a ResolveError that the target was not found in the my-project/test namespace. And when I prefix the target with the full

src/python/my-project

prefix, I get a ResolveError like

file 'src/python/my-project/src/python/my-project'

does not exist. Any help in setting up the dependency between the tests and the distribution target would be greatly appreciated.

sparse-lifeguard-95737

05/25/2022, 2:48 PM

is there a goal / command we could run in CI to enforce that all of our 3rd-party

python_requirement

constraints are actually resolvable? we’re still using old-style

constraints.lock

for now while we migrate onto the new lockfiles. a PR updated a package version in

constraints.lock

, but not the matching version spec in a

python_requirement

target. our CI is set up to use

--changed-since

heavily, so the error wasn’t caught during PR (because the

python_requirement

hadn’t changed, and none of the downstream files using it had changed). I’d like to add a CI job that does a global sanity-check on our dependencies to avoid errors like this in the future. my first thought was to run

./pants export ::

, but that can take awhile to execute in our codebase… is there any other option that might be quicker to run?

cold-soccer-63228

05/25/2022, 5:46 PM

I'm encountering a "No space left on device" error while watching files error on my Gitlab CI Runners. The docs suggest to adjust the limit on file watches, but that feels suboptimal for CI machines. Is there a known alternative?

refined-addition-53644

05/25/2022, 6:21 PM

Hi, I am trying to use multiple resolves for each sub-project. I have also defined a

default_resolve

I have a local

pants-plugin

folder which has

pants_requirements()

and this complains that it can't find a valid resolve name. I thought if I don't provide it an explicit resolve, it would use the

default_resolve

✅ 2

sparse-lifeguard-95737

05/25/2022, 6:43 PM

the docs on “pants in CI” recommend caching

~/.cache/pants/named_caches

, but in our repo I find that the size of that dir can grow pretty quickly & dramatically. is there a “safe” subset of that dir we could cache to keep our cache sizes small while still reaping some benefits? I feel like when I’ve poked around in that dir before I’ve seen symlinks between different subdirectories, which makes me worry about the feasibility of only caching a subset

sparse-pharmacist-76603

05/26/2022, 4:58 AM

Hi guys, How to debug hanging "generate lockfile" operation?