pantsbuild #general

Channels

welcome

github-notifications

announce

happy-kitchen-89482

05/10/2022, 5:23 AM

I'm noticing that setting

[python-infer].unowned_dependency_behavior = "error"

has unfortunate consequences in cases such as

Copy code

if TYPE_CHECKING:
    import something

✅ 1

ambitious-actor-36781

05/10/2022, 11:14 AM

it might not be a pants thing, but I've noticed I can't push docker images using

./pants publish

to a google artifact registry, but I can with

docker push

✅ 1

dazzling-petabyte-91122

05/10/2022, 1:28 PM

Trying to find info but coming up short. Is there a way to pass ldflags to the Go backend like with pants v1?

✅ 1

adorable-thailand-45835

05/10/2022, 1:35 PM

Morning everyone! I've set up a very simple Gitlab pipeline with a single job that simply runs

./pants --version

. The job keeps failing with the error below. Wondering if there is some additional bootstrapping needed for Gitlab or if maybe this issue could be due to network restrictions within the runner? Error:

Copy code

$ $CI_PROJECT_DIR/pants --version
29Bootstrapping Pants using /usr/bin/python3.8
30Creating the virtualenv PEX.
31Downloading the Pex PEX.
32Traceback (most recent call last):
33  File "<stdin>", line 4, in <module>
34FileNotFoundError: [Errno 2] No such file or directory: 'pex'
35SHA256 of <https://github.com/pantsbuild/pex/releases/download/v2.1.62/pex> is not as expected. Aborting.

37
Cleaning up file based variables

.gitlab-ci.yml:

Copy code

image:  path_to_enterprise_image
default:
    tags: ['edopplatform-v3-dev-runner']

stages:
    - setup-pants

setup-pants-job:
    stage: setup-pants
    script:
        - $CI_PROJECT_DIR/pants --version

bitter-ability-32190

05/10/2022, 4:30 PM

Our Bazel build has a

genrule

which generates a file which dumps some git info to be included in packaged things. Trying to use

experimental_shell_command

to do the same thing, I unfortunately noticed the command runs in the sandbox and not in repo root. Alternatively I could use

experimental_run_shell_command

but then I can't capture outputs 😞

✅ 1

busy-vase-39202

05/10/2022, 4:42 PM

Looking for an exciting new job with a company that values developer experience and clean build pipelines? Check out https://www.pantsbuild.org/page/jobs Or are you looking to hire engineers who care about code quality, ergonomic code, and collaborating across teams? Add your job opening to https://www.pantsbuild.org/page/jobs (just use the "Suggest Edits" button) and/or reply in-thread below with job opening details. Cheers!

incalculable-yacht-75851

05/10/2022, 5:11 PM

I'm trying to move from 2.9 to 2.11. One issue that comes up every time we introduce a new Pants version is that it always sets the latest version of black as of the time of release. Is there a way to bump the version of pants but not the version of black for existing projects while allowing future projects to use the version of black that ships with pants?

✅ 1

flat-summer-8204

05/10/2022, 5:11 PM

Hi all. Currently I am exploring Pants and I am enjoying it so far 🙂 I am facing something that I am still not sure if it is an misconfiguration on my side or an issue. Description I have a repo where the

requirements.txt

only has one dependency listed. This dependency lives in a private repository. After configuring the

.pants.rc

to set the index URL with the creds, when I run

./pants generate-lockfiles

it returns a list of the dependency versions available (since I haven’t constraint that in the

requirements.txt

). However, I noticed it doesn’t list the most recent ones. So I constraint the version to the most recent listed in the output of pants (let’s call it

version A

) and was able to generate the lockfile. However, when tried to constrain to the version that was released after that one (let’s call it

version B

), it couldn’t find a match distribution. I was inspecting the

setup.py

files in both distributions and found this change:

Copy code

```
# version A
    'python_requires': '>=3.7.0,<3.8.0',

# version B
    'python_requires': '>=3.7.1,<3.8.0',
```

which aligns to the diff between the `pyproject.toml`s.

Copy code

```
# version A
python = "~3.7.0"

# version B
python = "~3.7.1"
```

In the

pants.toml

I found that I was able to generate the lockfile for

version A

[python].interpreter_constraints = ["==3.7.*"]

. But if changes to

[python].interpreter_constraints = [">=3.7,<4"]

for example it can’t find matching distributions for the dependency. Has anyone faced something similar? Thanks in advance 🙂 EDIT: I am using

pex

as the

[python].lockfile_generator

✅ 1

hundreds-father-404

05/10/2022, 5:15 PM

Hi there! Welcome! What Pants version are you using? If 2.11, what is

[python].lockfile_generator

set to?

flat-summer-8204

05/10/2022, 5:15 PM

Hi Eric. Yes, 2.11 with

[python].lockfile_generator = "pex"

👍 1

hundreds-father-404

05/10/2022, 5:18 PM

Interesting. It is plausible this is an edge case we have not encountered with Pex lockfile generation yet - thank you for reporting! It does sound like interpreter constraints are at play here. Good debugging to find that. I imagine you couldn't share that distribution for us to try to reproduce?

hundreds-father-404

05/10/2022, 5:22 PM

The next step I'd probably try to do is take Pants out of the equation and run directly via Pex. I personally use

pipx install pex

, then it would be something like

Copy code

pex3 lock create --style=universal --resolver-version=pip-2020-resolver --interpreter-constraint='CPython==3.7.*' --repo=$foo -o debug.lock my-requirement

Not sure on the

--repo

part, depends on how you're using

[python-repos]

whether you use

--repo

--index

Copy code

-f PATH/URL, --find-links PATH/URL, --repo PATH/URL
                        Additional repository path (directory or URL) to look for requirements.
  -i URL, --index URL, --index-url URL
                        Additional cheeseshop indices to use to satisfy requirements.

calm-ambulance-65371

05/10/2022, 5:27 PM

It seems relying on a

python_distribution

causes the 3rd-party dependencies to not be pulled into the pex when running

pants test

on Pants

2.11.0

incalculable-yacht-75851

05/10/2022, 5:54 PM

I'm attempting to pin black to

black==21.12b0

(as was configured by default in pants 2.9) in my pants.toml I have the following

Copy code

[GLOBAL]
pants_version = "2.11.0"

backend_packages.add = [
  "pants.backend.python",
  "pants.backend.python.lint.black",
]

[python]
interpreter_constraints = [">=3.9"]
tailor_requirements_targets = false
enable_resolves = true # for use with lock file
resolves = { python-default = "3rdparty/python/python-default.lock" }
default_resolve = "python-default"
lockfile_generator = "pex"

[coverage-py]
interpreter_constraints = [">=3.9"]

[test]
use_coverage = true

[pytest]
version = "pytest>=6.2.4,<6.3"
extra_requirements.add = [
  "pytest-django>=4,<5",
]
lockfile="3rdparty/python/pytest.lock"

[black]
version = "black==21.12b0"
interpreter_constraints = [">=3.9"]
lockfile="3rdparty/python/black.lock"

✅ 1

enough-analyst-54434

05/10/2022, 7:38 PM

The answer is contained in the OP. Some dependency in the resolve says `'python_requires': '>=3.7.0,<3.8.0',`in its setup.py; so you can never generate a lock for any python outside that range, since that one dep, at least, does not support it.

enough-analyst-54434

05/10/2022, 7:41 PM

@flat-summer-8204 the fact you posted the question means that answer must not be making sense however. So, I'm not sure how to better word things. You might have to ask more pointed questions.

enough-analyst-54434

05/10/2022, 7:44 PM

How's this?: If your Pants configuration says:

[python].interpreter_constraints = [">=3.7,<4"]

then that says, you need lockfiles to work with all Pythons convered by that configuration, for example, Python 3.7.4, Python 3.9.10 and Python 3.11b1 (and all the rest <4). SInce the resolve of your requirements runs into a distribution A that itself says it only works with Python in the 3.7 range, that's a conflict. Pants cannot satisfy your request for a lockfile that works with all Pythons covered by

>=3.7,<4

because of that distribution A that has a narrower constraint.

➕ 1

enough-analyst-54434

05/10/2022, 7:46 PM

@hundreds-father-404 were you seeing something I wasn't there? I can't see how we'd ever support this.

enough-analyst-54434

05/10/2022, 7:47 PM

Basically, I'm not seeing the edge case, This seems like it should be a straight up failure and everything is working as it should.

happy-kitchen-89482

05/10/2022, 11:57 PM

What would be a good way to parametrize a python_distribution target so that we can generate wheels for each of multiple interpreter_constraints, when python_distribution itself doesn't have an interpreter_constraints field (its deps do)?

✅ 1

happy-kitchen-89482

05/11/2022, 12:29 AM

Fun - I don't think I've ever used Pants to run tests on pypy, but it worked (almost) seamlessly! Just required setting the repo-wide default constraints to empty, so that we don't try to merge CPython and PyPy constraints...

❤️ 3

icy-hair-30586

05/11/2022, 8:40 AM

Posted in the wrong channel originally 😞

bland-father-19717

05/11/2022, 9:43 AM

I got the following error when trying to import pyqt5. Is there a good solution?

Copy code

from PyQt5.QtWidgets import QApplication
ImportError: dlopen(/Users/xxx/.cache/pants/named_caches/pex_root/installed_wheels/33ced1c876f6a26e7899615a5a4efef2167c263488837c7beed023a64cebd051/PyQt5-5.15.6-cp36-abi3-macosx_10_13_x86_64.whl/PyQt5/QtWidgets.abi3.so, 2): Library not loaded: @rpath/QtWidgets.framework/Versions/5/QtWidgets
  Referenced from: /Users/xxx/.cache/pants/named_caches/pex_root/installed_wheels/33ced1c876f6a26e7899615a5a4efef2167c263488837c7beed023a64cebd051/PyQt5-5.15.6-cp36-abi3-macosx_10_13_x86_64.whl/PyQt5/QtWidgets.abi3.so
  Reason: image not found

flat-summer-8204

05/11/2022, 9:50 AM

thank you all for the quick relies. Adding some more details on my findings: Constraining the interpreter to a version I know I don’t have locally (

==3.7.1

), from the output I could see the possible interpreters:

Copy code

python3.6 CPython==3.6.12
python3.7 CPython==3.7.11
python3.7 CPython==3.7.7
python3.9 CPython==3.9.7
python3.8 CPython==3.8.9

Changed the constraint to

3.7.11

and it was possible to generate the lockfile either for

version A

and

version B

So this seems to suggest there is something going on indeed in the constraints relaxations on

pex

side. @hundreds-father-404 will follow your suggestion and run directly via

pex

, thanks. @enough-analyst-54434 my apologies, I am not sure if I followed. Both versions

and

have ranges on

python_requires

that should match interpreter constraint`["==3.7.*"]]` for any

>=3.7.1

, they overlap. Unless I am missing something.

nice-florist-55958

05/11/2022, 10:29 AM

2 QQs: 1) does 2.11 support PyPI proxies for building lockfiles now? 2) When trying out 2.11, Pants was trying to get things like setuptools from pythonhosted.org, which I have never seen before (will post the exact log message later today), so had to revert back to 2.10 -- is that a new thing/expected? This happened w/o enabling lockfile creation by Pants (i.e., I didn't change anything in pants.toml except the version).

enough-analyst-54434

05/11/2022, 2:08 PM

@flat-summer-8204 I think I still have done a poor job of making this clear. The issue will have nothing to do with local interpreters. The lock process Pants uses only considers 2 things: 1.) What are the interpreter constraints you specify, via Pants configuration. - The lock process will follow the rule that all dependencies locked must work with every interpreter on any machine implied by these constraints. 2.) What is the compatibility of the distribution currently being looked at when attempting to lock (say "requests" or "A"). It must be compatible with each and every interpreter implied by 1. So, If your Pants specified repo interpreter constraints are

==3.7.*

*(That says all locked dependencies must work for all possible Python 3.7 interpreters), and the lock process comes across a needed dependency "A" that only works for

>=3.7.1

, then that dependency conflicts with criteria #1. You said the repo should work for

==3.7.*

and

3.7.0

is not a subset of

>=3.7.1

since

==3.7.*

contains

3.7.0

and

>=3.7.1

does not. In other words, the lock fails since Pants (Pex really), sees it cannot meet the demand you placed in 1 of creating a lock that works for every possible `==3.7.` interpreter. It fails for the

3.7.0

case in particular for dependency "A".

enough-analyst-54434

05/11/2022, 2:08 PM

Does that make sense?

flat-summer-8204

05/11/2022, 4:01 PM

Hi @enough-analyst-54434 . Thanks for the effort on making it clearer. I think I was not clear on the cases I ran as well. Please find the summary below: Use Case 1

requirements.txt

used to generate the lockfile

Copy code

dep-a ==4.2.31 #

dep-a

v 4.2.31 , in the

setup.py

'python_requires': '>=3.7.0,<3.8.0'

Results: I was playing just changing the

interpreter_constraints

✅ lockfile generated, ❌ fails to generate the lockfile with error:

ERROR: No matching distribution found for dep-a==4.2.31

1. If

pants.toml.[python].interpreter_constraints = ["==3.7.11"]

, ✅ 2. If

pants.toml.[python].interpreter_constraints = ["==3.7.*"]

, ✅ 3. If

pants.toml.[python].interpreter_constraints = [">=3.7.11"]

, ❌ Use Case 2 requirements.txt
used to generate the lockfile

Copy code

dep-a ==4.2.32

dep-a

v 4.2.32 , in the

setup.py

'python_requires': '>=3.7.*1*,<3.8.0'

Results: I was playing just changing the

interpreter_constraints

✅ lockfile generated, ❌ fails to generate the lockfile with error:

ERROR: No matching distribution found for dep-a==4.2.32

1. If

pants.toml.[python].interpreter_constraints = ["==3.7.11"]

, ✅ 2. If

pants.toml.[python].interpreter_constraints = ["==3.7.*"]

, ❌ 3. If

pants.toml.[python].interpreter_constraints = [">=3.7.11"]

, ❌ So based on the that, why does it have different results when setting

interpreter_constraints

"==3.7.11"

and

">=3.7.11"

? I agree, seems to be a
pex
issue indeed. I will create an issue there to iterate over this.

enough-analyst-54434

05/11/2022, 4:12 PM

There is no Pex issue here, this is operating exactly as should be expected. Thanks for the details, I'll respond with details as well a bit later, but you may be missing the fact that

>=3.7.11

includes all Python 3.8 interpreters, all Python 3.9 interpreters, all Python 3.10 interpreters, etc ... and none of those work with

dep-a

flat-summer-8204

05/11/2022, 4:39 PM

Thanks a lot @enough-analyst-54434 for your reply, I understand now my problem was on NOT set the upper boundary simple smile Adding

<3.8

to the use cases above, Use Case 1 3. If

pants.toml.[python].interpreter_constraints = [">=3.7.11,<3.8"]

, ✅ Use Case 2 2. If
pants.toml.[python].interpreter_constraints = ["==3.7.*"]
, ❌ , as expected because it considers 3.7.0 as you pointed out. 3. If
pants.toml.[python].interpreter_constraints = [">=3.7.11,<3.8"]
, ✅

❤️ 1