Hi, a question on MyPy. It seems it takes a long time to build mypy.pex

[2022-04-12T14:27:05.810Z] 10:27:05.59 [INFO] Completed: Building mypy.pex from build-support/mypy_lockfile.txt
[2022-04-12T14:28:27.628Z] 10:28:22.84 [INFO] Long running tasks:
[2022-04-12T14:28:27.628Z]   86.52s	Installing build-support/lockfile.txt for the resolve `python-default`
[2022-04-12T14:28:54.377Z] 10:28:52.92 [INFO] Long running tasks:
[2022-04-12T14:28:54.377Z]   116.60s	Installing build-support/lockfile.txt for the resolve `python-default`
[2022-04-12T14:29:26.494Z] 10:29:22.95 [INFO] Long running tasks:
[2022-04-12T14:29:26.494Z]   146.63s	Installing build-support/lockfile.txt for the resolve `python-default`
[2022-04-12T14:29:53.033Z] 10:29:52.98 [INFO] Long running tasks:
[2022-04-12T14:29:53.033Z]   176.66s	Installing build-support/lockfile.txt for the resolve `python-default`
[2022-04-12T14:30:25.092Z] 10:30:23.03 [INFO] Long running tasks:
[2022-04-12T14:30:25.092Z]   206.71s	Installing build-support/lockfile.txt for the resolve `python-default`
[2022-04-12T14:30:57.197Z] 10:30:53.08 [INFO] Long running tasks:
[2022-04-12T14:30:57.197Z]   236.76s	Installing build-support/lockfile.txt for the resolve `python-default`
[2022-04-12T14:31:23.761Z] 10:31:23.12 [INFO] Long running tasks:
[2022-04-12T14:31:23.761Z]   266.80s	Installing build-support/lockfile.txt for the resolve `python-default`
[2022-04-12T14:31:56.020Z] 10:31:53.18 [INFO] Long running tasks:
[2022-04-12T14:31:56.020Z]   296.86s	Installing build-support/lockfile.txt for the resolve `python-default`
[2022-04-12T14:32:28.129Z] 10:32:23.22 [INFO] Long running tasks:
[2022-04-12T14:32:28.129Z]   326.90s	Installing build-support/lockfile.txt for the resolve `python-default`
[2022-04-12T14:32:54.681Z] 10:32:53.28 [INFO] Long running tasks:
[2022-04-12T14:32:54.681Z]   356.96s	Installing build-support/lockfile.txt for the resolve `python-default`
[2022-04-12T14:33:27.350Z] 10:33:23.34 [INFO] Long running tasks:
[2022-04-12T14:33:27.350Z]   387.02s	Installing build-support/lockfile.txt for the resolve `python-default`

I am wondering if it is related to the cache issue in MyPy mentioned here https://www.pantsbuild.org/docs/python-check-goal#performance-is-often-slower-than-normal?

So there's a few things here: • Building

mypy.pex

finished (probably quickly)

Completed: Building mypy.pex from build-support/mypy_lockfile.txt

• What you're seeing is Pants installing of the dependencies in your resolve. Depending on the size of your resolve, or the size of those deps it can take a while. ◦ This also depends on your settings, which Pants version you're using, and if you're using the new PEX lockfiles • The linked MyPy issue has to do with the performance of running mypy, which you haven't gotten to that step yet 😅

@bitter-ability-32190 thanks for info. I am using a lockfile for mypy since I need a different version of MpPy in Pants 2.10, which is the version of Pants I am using.

Ah, so Pants 2.11.x introduces a new kind of lockfile: a PEX (JSON) lockfile. When 2.11.x goes non-rc I recommend trying ti out 😃 In the meantime your options are

resolve_all_constraints

or not.

I am just wondering why it only happens for MyPy? I dont see the same for isort/black/etc.

Good question! Formatters like

isort

and

black

don't require any of your dependencies to run. linters/checkers like

pylint

or

mypy

do require dependencies, so you just happen to be executing your first command which requires deps.

Thanks. It is clear to me. Btw, is there any docs already available on the new PEX lockfile in 2.11?

(Don't forget the

mypy.pex

just contains mypy and friends (and not your deps)._

I am updating our docs for pants 2.11 this week, so we will have docs soon about the new pex lock files

Hi @hundreds-father-404 I am wondering if you have the relevant docs for pants 2.11. The problem I am facing now is that, in our CI, the unittests are used to complete very fast assuming the conda env is there. But with Pants, given it needs to resolve those dependencies, running unittest become very slow. I am wondering if the PEX lockfile can solve/help this issue? or do you have any other suggestion?

Are you sure the cache is persisting between each step?

what do you mean by cache each step?

Pants stores cached results in

~/.cache/pants

by default. Is that dir persisting between the call to unit tests and integration tests?

em the unittest and integration tests are different stages in a Jenkins CI pipeline. I believe they are shared. No?

🤷‍♂️ I know nothing of Jenkins implmentation details. That's where I'd start investigating

thanks! I will take a look

which version is this? if you can confirm that you are definitely preserving the cache, filing an issue with some more details about which commands you’re running in which sequence would be good. there are some contexts in which you might not be able to reuse a lockfile resolve: across

test

and

package

you might not be able to in some cases, for example. but between tests you always should be able to.

there are some contexts in which you might not be able to reuse a lockfile resolve: across

test

and

package

do you mind sharing some more info or example?

do you mind sharing some more info or example?

there isn’t a very concise explanation, except to say that

test

runs on exactly one interpreter chosen by pants per-run (“internal”), and that interpreter is part of the cache key. whereas

package

is building a portable

PEX

file, and so it will attempt to maximize compatibility with your interpreter constraints, rather than matching exactly one.

interpreter_constraints = ["==3.9.*"]

This is what I have. i can pin it to a particular version and see.

see https://www.pantsbuild.org/docs/using-pants-in-ci for more info

I think i find the answer to my question. It seem the env vars in our jenkins pipeline for unit and int tests are different. I believe when env vars are differernt, the pants will resolve the dependencies again.

Ah yeah that makes sense. Pants today takes a very conservative approach to hermeticity, and invalidates eagerly. Who is to say an env car doesn't change the behavior of a Python install? (After all, setup.py is just code, it could do anything)