Regarding <https www pantsbuild org docs python third party Pants #general

Regarding <https://www.pantsbuild.org/docs/python-...

cold-soccer-63228

05/02/2022, 5:25 PM

Regarding https://www.pantsbuild.org/docs/python-third-party-dependencies#lockfiles, what is the reason that lockfiles like

Pipfile.lock

cannot be used as the lockfile? Generating lockfiles via Pants seems to be very slow...

Copy code

➜  ./pants generate-lockfiles
⠠ 461.47s Determine all Python interpreter versions used by Pytest in your proje
⠠ 461.47s Determine if MyPy should use Python 3.8+ (for lockfile usage)
⠠

😮 1

cold-soccer-63228

05/02/2022, 5:28 PM

For reference, here's my OS information.

witty-crayon-22786

05/02/2022, 5:32 PM

that is very surprising… how large is the repository? that step should complete in a few seconds for ~1k files

witty-crayon-22786

05/02/2022, 5:33 PM

and does running py-spy like:

sudo py-spy dump -p $pants_pid

show anything interesting?

cold-soccer-63228

05/02/2022, 5:35 PM

Our repository is quite large. Does performance scale non-linearly with the size of the repository?

Copy code

➜  tree .
...
3913 directories, 30054 files

witty-crayon-22786

05/02/2022, 5:40 PM

usually, i would say: “of course we scale linearly! how dare you!“, but we … just fixed a bug in this area. https://github.com/pantsbuild/pants/pull/15141 improved a non-linear case for

mypy

, but this report just made me realize that it hasn’t been fixed for lockfile generation for

mypy

👀 1

😂 3

witty-crayon-22786

05/02/2022, 5:42 PM

i’ll open an issue for this. but there is an underlying non-linear problem that we’ll need to fix as well, already described in https://github.com/pantsbuild/pants/issues/15241.

witty-crayon-22786

05/02/2022, 5:46 PM

oh, but: @cold-soccer-63228: do you have/need a custom

mypy

lockfile? even if you do, being more specific with

./pants generate-lockfiles

would skip regenerating it: i.e., running only:

Copy code

./pants generate-lockfiles --resolve=python-default

…for example

👍 1

cold-soccer-63228

05/02/2022, 5:49 PM

I don't have/need a custom

mypy

lockfile, but

mypy

is one of our

dev-packages

for our

Pipfile

hundreds-father-404

05/02/2022, 5:50 PM

Interesting. We probably want a way to let you "delete" /ignore certain targets from the target geerators like

pipenv_requirements

. Super easy to implement

witty-crayon-22786

05/02/2022, 5:51 PM

that shouldn’t do it in this case…?

➕ 1

coke 1

hundreds-father-404

05/02/2022, 5:51 PM

can you share your

pants.toml

witty-crayon-22786

05/02/2022, 5:51 PM

filed https://github.com/pantsbuild/pants/issues/15301 … noted there that one workaround would be to set interpreter constraints for mypy

cold-soccer-63228

05/02/2022, 5:52 PM

Here's my

pants.toml

file.

Copy code

[GLOBAL]
pants_version = "2.10.0"

use_deprecated_python_macros = false

backend_packages = [
  "pants.backend.python",
  "pants.backend.experimental.python.lint.autoflake",
  "pants.backend.python.lint.black",
  "pants.backend.python.typecheck.mypy",
  "pants.core",
]

[anonymous-telemetry]
enabled = false

[autoflake]
args = [
  "--in-place",
  "--remove-all-unused-imports",
  "--exclude='front_porch/modules/__init__.py'",
]

[python]
enable_resolves = true
interpreter_constraints = ["CPython==3.9.*"]
run_against_entire_lockfile = true

hundreds-father-404

05/02/2022, 5:53 PM

Thanks! Huh

Determine if MyPy should use Python 3.8+ (for lockfile usage)

would ideally not trigger. I'll open an issue

cold-soccer-63228

05/02/2022, 5:54 PM

Regarding my original question, is there any reason a Pants-specific lockfile needs to be generated instead of being able to use something like

Pipfile.lock

hundreds-father-404

05/02/2022, 5:54 PM

https://github.com/pantsbuild/pants/issues/15302

hundreds-father-404

05/02/2022, 5:55 PM

Yes. The reason is that Pants uses Pex (which uses pip) to install dependencies. So we need a format understood by both tools

cold-soccer-63228

05/02/2022, 5:57 PM

Is it not possible to just generate the

3rdparty/python/default.lock

file by parsing the

Pipfile.lock

hundreds-father-404

05/02/2022, 9:18 PM

Yay, fixed: https://github.com/pantsbuild/pants/pull/15304 so that those code paths don't run anymore. Thank you for the report! I will cherry-pick to 2.10

hundreds-father-404

05/02/2022, 9:42 PM

Is it not possible to just generate the 3rdparty/python/default.lock file by parsing the Pipfile.lock?

That is theoretically possible, but Pants would need technology to do that. Instead, you could use the

export

command from pipenv to get it in a format understood by Pants

cold-soccer-63228

05/03/2022, 4:37 PM

Hmm, so I upgraded to 2.11, but am still seeing this for when I run

./pants generate-lockfiles

. Was there something specific I had to do?

Copy code

⠄ 235.18s Determine if MyPy should use Python 3.8+ (for lockfile usage)

hundreds-father-404

05/03/2022, 4:43 PM

Yes, we have not released that fix I made yesterday yet. Will probably do at end of this week

cold-soccer-63228

05/03/2022, 4:47 PM

Ahh, got it. Thanks! No rush 🙂

witty-crayon-22786

05/12/2022, 8:17 PM

closing the loop here: a holistic fix for this issue is in `2.12.x`: https://github.com/pantsbuild/pants/pull/15373…

2.11.x

will continue to have slow lockfile generation though unfortunately.

5 Views

Open in Slack

Previous Next