https://pantsbuild.org/ logo
Join Slack
Powered by
Are there known reasons why generating a lockfile ...
# general
m
Are there known reasons why generating a lockfile would be really slow? I have one that went for 1000s, never returned.. ended up restarting because i accidentally saved a file. Currently it's back up to 400s again
using 2.15, it also got way slower when we moved from requirements.txt in each library with them all resolving to the same lockfile to putting them all in one place in 3rdparty/<>/requirements.txt
e
Yes, 3: + interpreter constraints are overly broad: Pants ships with 
">=3.7,<4"
by default. + old pip is slow: Pants 2.16.x allows setting `[python] pip_version = "23.0.1"`which makes many resolves faster + Pex can still do 1 more optimization: outlined here: https://github.com/pantsbuild/pex/issues/2044
You have control over the 1st in any version of Pants, obviously need Pants 2.16.x for the 2nd and the 3rd is a feature that is not being worked on yet.
@magnificent-toothbrush-17254 are you potentially a victim of the 1st - no explicit ICs or overly broad ones?
m
we do have multiple ICs... also any reason why it would be slow for me and not someone else? My coworker was able to generate it in ~150s on a linux machine
e
Aha - you are paying a price for using a Mac. Especially if M1. Since M1s are expensive and not CI friendly ~no-one pre-builds wheels for them and puts them on PyPI. As such, a resolve on an M1 Mac in particular will be very slow since it has to build many sdists to find out resolve metadata.
m
i have an old intel mac
and it was fast yesterday ahha that's why i'm confused
e
Ok, that's a new twist.
So what is different between today and yesterday that seems relevant. Anything?
m
moving to the 3rdparty/<>/requirements.txt is one.. maybe i missed something there
e
Do you have this turned on in 
pants.toml
as an org?: https://www.pantsbuild.org/docs/reference-python#enable_resolves
I'm guessing you do since you speak of locks.
m
yep!
e
What is your IC setup exactly? Can you share 
pants.toml
?
m
Copy code
[GLOBAL]
pants_version = "2.15.0"

backend_packages = [
  # BUILD files
  "pants.backend.build_files.fmt.black",
  # Docker images
  "pants.backend.docker",
  "pants.backend.docker.lint.hadolint",
  # Python
  "pants.backend.python",
  "pants.backend.experimental.python",
  # Note that we want Autoflake to run before Black and isort, 
  # so it must appear first.
  "pants.backend.python.lint.autoflake",
  "pants.backend.python.lint.black",
  "pants.backend.python.lint.isort",
  "pants.backend.python.lint.docformatter",
  "pants.backend.python.lint.flake8",
  "pants.backend.python.typecheck.mypy",
  "pants.backend.python.mixed_interpreter_constraints",
]

[source]
root_patterns = ['/src/*', '/bin/*', '3rdparty/*']

[python]
tailor_pex_binary_targets = false
enable_resolves = true
interpreter_constraints = ["==3.9.*"]

[python.resolves]
python-default = "resolves/python-default.lock"
py2-py3 = "resolves/py2-py3.lock"
py3 = "resolves/py3.lock"

[python.resolves_to_interpreter_constraints]
python-default = [">=3.7,<3.10"]
py2-py3 = [
  ">=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,!=3.6.*,<3.10",
]
py3 = [">=3.11"]

[pytest]
lockfile = "resolves/pytest.lock"
version = "pytest==4.6.11"
xdist_enabled = false
extra_requirements = [
  "pytest-cov>=2.12,!=2.12.1,<3.1",
  "pytest-xdist>=1"
]

[mypy]
lockfile = "resolves/mypy.lock"
extra_requirements = ["typed_ast"]

[setuptools]
lockfile = "resolves/setuptools.lock"
version = "setuptools==44.1.1"

[python-bootstrap]
search_path = [
  # This will use all interpreters in `$(pyenv root)/versions`.
  "<PYENV>",
  "<PATH>",
  # Brew usually installs Python here.
  "/usr/local/bin",
]

[anonymous-telemetry]
enabled = false
it is a lot of ICs, but again we had this many before... this is for generating py2-py3
e
Ok, yeah that's huge. So option 2 is the next try. You may not be able to upgrade to an unstable 2.16, but if you re-run the lock generation eith 
-ldebug
thrown in that will show the Pex command line used which will allow trying it out with Pex latest + 
--pip-version 23.0.1
to see if that would help.
But, circling back to reality - this would only make sense if you are in fact having to build many more sdists than your colleagues. Pretty much a guaranty with an M1, but even with your intel Mac, it can happen depending on the particular distributions your resolve depends on.
So, on that angle, do you know if you are or are not the only intel Mac user having this issue?
m
It's also slow for other intel mac users, just not on the level I'm experiencing right now runs real fast in a docker container though, wow
e
That strongly implies sdist building for Mac only is killing you.
m
yep, agreed
e
If you have the luxury of ditching your Mac or wiping it and installing Linux on it - do it!
🙃 1
m
hahaha yep, I do
e
Those ICs / resolves looked familiar!
m
hahah yeah thank you for all your help!
I have one more question - I see in the docs that you are supposed to be able to use the environment markers: https://www.pantsbuild.org/docs/reference-python_requirement#coderequirementscode in requirements files or in a specific python_requirement spec. I have not been able to get that to work. A 
pip install -r requirements.txt
will work fine, but for something like this (just an example)
Copy code
MarkupSafe==1.1.1; python_version < '3.9'
MarkupSafe==2.1.1; python_version >= '3.8'
it will fail when trying to generate the lockfile
i've tried it both in a requirements.txt file and directly as 
python_requirement
, no matter what it'll error out saying there is a conflict:
Copy code
The conflict is caused by:
     The user requested MarkupSafe==1.1.1
     The user requested MarkupSafe==2.1.1

 To fix this you could try to:
 1. loosen the range of package versions you've specified
 2. remove package versions to allow pip attempt to solve the dependency conflict
allowing pip to resolve the dependency conflict means anything that imports this could be real slow, hence my desire to pin for two environments
e
So, you asked for the impossible there. 3.8.1 is in both sets for example.
@magnificent-toothbrush-17254 does that make sense? The 
python_version
constraints look like a typo, they are not disjoint. They should be disjoint.
m
jeez
long day
lol
e
Heh. Pex will keep you honest.
m
meant this:
Copy code
MarkupSafe==2.1.1; python_version >= '3.7'
MarkupSafe==1.1.1; python_version == '2.7'
will fail, which it still does. those have no overlap
e
Hmm, yeah. That's right. This is a single --style universal resolve. It does work for a multi-resolve but Pants does not use those.
So, in Pants you need to split a resolve at that point.
m
does that mean the docs aren't quite right for this?
You can specify multiple requirements for the same project in order to use environment markers, such as 
["foo>=1.2,<1.3 ; python_version>'3.6'", "foo==0.9 ; python_version<'3'"]
.
or just that I didn't understand them haha either are possible
e
That is true in isolation but false with resolves / lockfiles turned on.
A lock file will have 1 version of every artifact when it's a 
--style universal
lock like Pants uses. Pex will allow you to create a bifurcated lock file for 
--style strict
or 
--style sources
but Pants doesn't use either of those styles. In Pants you must instead split the resolve in two and get 2 lock files.
So it looks like you cannot have a py2-py3 resolve basically unless you're willing to settle for older, 2.7 compatible MarkupSafe. IOW, this locks fine: 
pex3 lock create --style universal --interpreter-constraint ">=2.7,<3.10,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*" --resolver-version pip-2020-resolver MarkupSafe==1.1.1
m
Got it, thank you for all this context!
e
You're welcome. I wrote the Pex side of this, creating a lock and resolving from it, but I have much less context on the Pants side. Still a bit mysterious to me on that side.
Open in Slack
PreviousNext
Powered by