https://pantsbuild.org/ logo
Powered by
m

magnificent-toothbrush-17254

03/02/2023, 6:30 PM
Are there known reasons why generating a lockfile would be really slow? I have one that went for 1000s, never returned.. ended up restarting because i accidentally saved a file. Currently it's back up to 400s again
using 2.15, it also got way slower when we moved from requirements.txt in each library with them all resolving to the same lockfile to putting them all in one place in 3rdparty/<>/requirements.txt
e

enough-analyst-54434

03/02/2023, 6:33 PM
Yes, 3: + interpreter constraints are overly broad: Pants ships with 
">=3.7,<4"
by default. + old pip is slow: Pants 2.16.x allows setting `[python] pip_version = "23.0.1"`which makes many resolves faster + Pex can still do 1 more optimization: outlined here: https://github.com/pantsbuild/pex/issues/2044
You have control over the 1st in any version of Pants, obviously need Pants 2.16.x for the 2nd and the 3rd is a feature that is not being worked on yet.
@magnificent-toothbrush-17254 are you potentially a victim of the 1st - no explicit ICs or overly broad ones?
m

magnificent-toothbrush-17254

03/02/2023, 6:36 PM
we do have multiple ICs... also any reason why it would be slow for me and not someone else? My coworker was able to generate it in ~150s on a linux machine
e

enough-analyst-54434

03/02/2023, 6:38 PM
Aha - you are paying a price for using a Mac. Especially if M1. Since M1s are expensive and not CI friendly ~no-one pre-builds wheels for them and puts them on PyPI. As such, a resolve on an M1 Mac in particular will be very slow since it has to build many sdists to find out resolve metadata.
m

magnificent-toothbrush-17254

03/02/2023, 6:38 PM
i have an old intel mac
and it was fast yesterday ahha that's why i'm confused
e

enough-analyst-54434

03/02/2023, 6:39 PM
Ok, that's a new twist.
So what is different between today and yesterday that seems relevant. Anything?
m

magnificent-toothbrush-17254

03/02/2023, 6:41 PM
moving to the 3rdparty/<>/requirements.txt is one.. maybe i missed something there
e

enough-analyst-54434

03/02/2023, 6:41 PM
Do you have this turned on in 
pants.toml
as an org?: https://www.pantsbuild.org/docs/reference-python#enable_resolves
I'm guessing you do since you speak of locks.
m

magnificent-toothbrush-17254

03/02/2023, 6:41 PM
yep!
e

enough-analyst-54434

03/02/2023, 6:42 PM
What is your IC setup exactly? Can you share 
pants.toml
?
m

magnificent-toothbrush-17254

03/02/2023, 6:43 PM
Copy code
[GLOBAL]
pants_version = "2.15.0"

backend_packages = [
  # BUILD files
  "pants.backend.build_files.fmt.black",
  # Docker images
  "pants.backend.docker",
  "pants.backend.docker.lint.hadolint",
  # Python
  "pants.backend.python",
  "pants.backend.experimental.python",
  # Note that we want Autoflake to run before Black and isort, 
  # so it must appear first.
  "pants.backend.python.lint.autoflake",
  "pants.backend.python.lint.black",
  "pants.backend.python.lint.isort",
  "pants.backend.python.lint.docformatter",
  "pants.backend.python.lint.flake8",
  "pants.backend.python.typecheck.mypy",
  "pants.backend.python.mixed_interpreter_constraints",
]

[source]
root_patterns = ['/src/*', '/bin/*', '3rdparty/*']

[python]
tailor_pex_binary_targets = false
enable_resolves = true
interpreter_constraints = ["==3.9.*"]

[python.resolves]
python-default = "resolves/python-default.lock"
py2-py3 = "resolves/py2-py3.lock"
py3 = "resolves/py3.lock"

[python.resolves_to_interpreter_constraints]
python-default = [">=3.7,<3.10"]
py2-py3 = [
  ">=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,!=3.6.*,<3.10",
]
py3 = [">=3.11"]

[pytest]
lockfile = "resolves/pytest.lock"
version = "pytest==4.6.11"
xdist_enabled = false
extra_requirements = [
  "pytest-cov>=2.12,!=2.12.1,<3.1",
  "pytest-xdist>=1"
]

[mypy]
lockfile = "resolves/mypy.lock"
extra_requirements = ["typed_ast"]

[setuptools]
lockfile = "resolves/setuptools.lock"
version = "setuptools==44.1.1"

[python-bootstrap]
search_path = [
  # This will use all interpreters in `$(pyenv root)/versions`.
  "<PYENV>",
  "<PATH>",
  # Brew usually installs Python here.
  "/usr/local/bin",
]

[anonymous-telemetry]
enabled = false
it is a lot of ICs, but again we had this many before... this is for generating py2-py3
e

enough-analyst-54434

03/02/2023, 6:46 PM
Ok, yeah that's huge. So option 2 is the next try. You may not be able to upgrade to an unstable 2.16, but if you re-run the lock generation eith 
-ldebug
thrown in that will show the Pex command line used which will allow trying it out with Pex latest + 
--pip-version 23.0.1
to see if that would help.
But, circling back to reality - this would only make sense if you are in fact having to build many more sdists than your colleagues. Pretty much a guaranty with an M1, but even with your intel Mac, it can happen depending on the particular distributions your resolve depends on.
So, on that angle, do you know if you are or are not the only intel Mac user having this issue?
m

magnificent-toothbrush-17254

03/02/2023, 7:00 PM
It's also slow for other intel mac users, just not on the level I'm experiencing right now runs real fast in a docker container though, wow
e

enough-analyst-54434

03/02/2023, 7:09 PM
That strongly implies sdist building for Mac only is killing you.
m

magnificent-toothbrush-17254

03/02/2023, 7:09 PM
yep, agreed
e

enough-analyst-54434

03/02/2023, 7:09 PM
If you have the luxury of ditching your Mac or wiping it and installing Linux on it - do it!
🙃 1
m

magnificent-toothbrush-17254

03/02/2023, 7:21 PM
hahaha yep, I do
e

enough-analyst-54434

03/02/2023, 7:22 PM
Those ICs / resolves looked familiar!
m

magnificent-toothbrush-17254

03/02/2023, 10:00 PM
hahah yeah thank you for all your help!
I have one more question - I see in the docs that you are supposed to be able to use the environment markers: https://www.pantsbuild.org/docs/reference-python_requirement#coderequirementscode in requirements files or in a specific python_requirement spec. I have not been able to get that to work. A 
pip install -r requirements.txt
will work fine, but for something like this (just an example)
Copy code
MarkupSafe==1.1.1; python_version < '3.9'
MarkupSafe==2.1.1; python_version >= '3.8'
it will fail when trying to generate the lockfile
i've tried it both in a requirements.txt file and directly as 
python_requirement
, no matter what it'll error out saying there is a conflict:
Copy code
The conflict is caused by:
     The user requested MarkupSafe==1.1.1
     The user requested MarkupSafe==2.1.1

 To fix this you could try to:
 1. loosen the range of package versions you've specified
 2. remove package versions to allow pip attempt to solve the dependency conflict
allowing pip to resolve the dependency conflict means anything that imports this could be real slow, hence my desire to pin for two environments
e

enough-analyst-54434

03/02/2023, 10:05 PM
So, you asked for the impossible there. 3.8.1 is in both sets for example.
@magnificent-toothbrush-17254 does that make sense? The 
python_version
constraints look like a typo, they are not disjoint. They should be disjoint.
m

magnificent-toothbrush-17254

03/02/2023, 10:08 PM
jeez
long day
lol
e

enough-analyst-54434

03/02/2023, 10:08 PM
Heh. Pex will keep you honest.
m

magnificent-toothbrush-17254

03/02/2023, 10:09 PM
meant this:
Copy code
MarkupSafe==2.1.1; python_version >= '3.7'
MarkupSafe==1.1.1; python_version == '2.7'
will fail, which it still does. those have no overlap
e

enough-analyst-54434

03/02/2023, 10:12 PM
Hmm, yeah. That's right. This is a single --style universal resolve. It does work for a multi-resolve but Pants does not use those.
So, in Pants you need to split a resolve at that point.
m

magnificent-toothbrush-17254

03/02/2023, 10:14 PM
does that mean the docs aren't quite right for this?
You can specify multiple requirements for the same project in order to use environment markers, such as 
["foo>=1.2,<1.3 ; python_version>'3.6'", "foo==0.9 ; python_version<'3'"]
.
or just that I didn't understand them haha either are possible
e

enough-analyst-54434

03/02/2023, 10:17 PM
That is true in isolation but false with resolves / lockfiles turned on.
A lock file will have 1 version of every artifact when it's a 
--style universal
lock like Pants uses. Pex will allow you to create a bifurcated lock file for 
--style strict
or 
--style sources
but Pants doesn't use either of those styles. In Pants you must instead split the resolve in two and get 2 lock files.
So it looks like you cannot have a py2-py3 resolve basically unless you're willing to settle for older, 2.7 compatible MarkupSafe. IOW, this locks fine: 
pex3 lock create --style universal --interpreter-constraint ">=2.7,<3.10,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*" --resolver-version pip-2020-resolver MarkupSafe==1.1.1
m

magnificent-toothbrush-17254

03/02/2023, 10:22 PM
Got it, thank you for all this context!
e

enough-analyst-54434

03/02/2023, 10:22 PM
You're welcome. I wrote the Pex side of this, creating a lock and resolving from it, but I have much less context on the Pants side. Still a bit mysterious to me on that side.
Powered by