https://pantsbuild.org/ logo
Join Slack
Powered by
Hmm. Yesterday with 2.11.0.rc0 I couldn't generat...
# general
e
Hmm. Yesterday with 2.11.0.rc0 I couldn't generate lockfiles because pants was picking up my python interpreter incorrectly. That's fixed in 2.11.0rc1, but now I seem to be having some problems with my 2.11.0rc1 pex lockfile generation
Copy code
1976.7s Generate lockfile for python-default
I eventually gave up after those 30 minutes and took out 
lockfile_generator="pex"
and poetry spit out a 212k lockfile in 25 sec. Admittedly I have a pretty unclean setup (I'm trying to generate requirements from what was my old constraints.txt file, which represents a 
pip freeze
of the project, so it's sort of generating a lockfile from a lockfile). But that difference was pretty striking and if the new pex-based JSON-style lockfiles are faster to use after generation, I'd love to make the switch. Any suggestions as to what may be going on?
h
are you willing to run w/ ldebug w/ pex and share the argv? we'd love to investigate
e
Untitled.txt
Untitled.txt
if this shines any light, sure. I've (of obvious necessity) pulled out some env vars (in 
<<some_env_vars>>
) to do with authentication and the rest should be reasonably straightforward I think
Otherwise (with old poetry lock) it seems to be working very well.
h
@witty-crayon-22786 @enough-analyst-54434 would either of you have any time this week to try to reproduce this slowness? FYI one other user reported slowness like this too
w
maybe… there are a few other issues near the head of the queue. but i’ll get it into a 2.11.x issue
h
I'll see if I can take a look
e
This will be ~un-actionable without resolve specifics. If you could provide that @echoing-farmer-15630 (basically, if we could have your input requirements) that would be great. Pants use of Pex uses the pip-2020-resolver which is known to have pathologic backtracking cases. An apples to apples with poetery caches nuked and ditto Pants/Pex would be a good 1st step though. I assume you did not nuke the poetry cache in your infinity vs 25sec timing above?
h
John look at the code snippet they sent, it should include all the requirements
e
Opps, missed collapsed text. You already privided great data @echoing-farmer-15630 - thanks.
💯 1
w
e
Ah, thanks Stu. Much better to work with.
@echoing-farmer-15630 can you possibly share the input requirements to the venv that you `pip freeze'`d to generate the 
constraints.txt
that is feeding into the Pex lock? If so, please plop them on the issue Stu created: https://github.com/pantsbuild/pants/issues/14998
e
Hi, John -- sorry for the delay. Unfortunately, it's hard to do so at this point just because of the process of how we got here, which went: 1. Each subproject in our monorepo (before pants) had its own `requirements.txt`; we had our own tool which went through them all and tried to keep version numbers the same. 2. This caused trouble with pants, of course, because you then had multiple targets providing the same libraries; we may have 3-4 entries of "numpy==1.21.0" or whatever, and then each BUILD either had to manually clarify "oh, this is the algorithms version of numpy==1.21.0" or you just watched pages of error messages scroll past. 3. Once it was clear that we could put all the versioning into a single constraints.txt file, we did so, so that all the BUILD files (and a few legacy requirements.txt files) could just have the "pin-free" names of the packages 4. Unfortunately that didn't pin enough, so we `pip freeze`'d the whole thing for 
constraints.txt
, so now 
constraints.txt
was a superset of our direct requirements. 5. ...and now here we are getting a "proper lockfile" for what is already an overly-specified "improper lockfile" listing all the packages. So now I'm actually trying to "unpick" our constraints.txt into just the top-level requirements (rather than the transitive dependencies), which may entail walking through 
pants dependencies...
for every available target that isn't a 
python_requirements
(ugh). But may still be worth my time. Disadvantages of migrating a legacy repo over several versions of pants, I'm afraid. I can try to put in that time in the next few days if it would be useful.
💜 1
h
Disadvantages of our lock file support being very very subpar for far too long. Really sorry about that 😞
e
Hey, I'm here for the dependency analysis; the lockfiles are a pleasant bonus 🙂 I feel like there ought to be a handy snippet involving 
jq
for "get me the immediate third-party requirements of all my pex or python_sources targets (or some subset) in requirements.txt format", like the
Copy code
./pants dependencies :: |
    xargs ./pants filter --target-type=python_requirement |
    xargs ./pants peek |
    jq -r '.[]["requirements"][]'
... in "set up a virtual environment" -- but that as itself probably lists everything if I already have it listed in constraints.txt. But I know there's something close.
❤️ 1
h
This is not yet documented, but the current recommendation is to use pants export and then in the virtual environment run pit freeze. Assuming that you have your lock file set up how you like
Oh I think that's not what you're asking for though. So I think the issue is your dependencies is using ::. You want it to instead first only look at your relevant targets like python source, and then run dependencies on that, and then do the Peak
Start with filter --target-type=python source etc
e
Yep (I did have a brief laugh at the highly-documented 
export
function 🙂 although it looks handy soon). Working on it.
Hmm, more or less
Copy code
./pants list :: 
  | xargs ./pants filter --target-type=python_sources 
  | xargs ./pants dependencies --transitive 
  | xargs ./pants peek | jq -r '.[]["requirements"][]'
...I think. Wow, when you look at it that way the output looks pretty slim. Let me try that.
h
That looks generally right. Note that there's no need for the original list. List and filter are identical if you don't use any options for filter I suspect that you want to use different target types. Right now you are using python sources with an S. Instead use the singular form. You might also want to use the Python test target, and pex binary target etc.
e
Hmm, will compare for certain and add the other target types. At any rate already the pex version is taking much much longer than the poetry version even with the greatly-slimmed requirements, so this could be a reproduction although I forgot the ldebug, let me do that (180 sec and counting)
Yeah, looks like the pex resolver is taking at least 10x as long (still going) as the poetry resolver with a more modest
Copy code
{ argv: ["/home/vputz/.pyenv/versions/3.10.2/bin/python", "./pex", "--tmpdir", ".tmp", "lock", "create", "--python-path", "/home/vputz/.pyenv/versions/2.7.18/bin:/home/vputz/.pyenv/versions/3.10.2/bin:/home/vputz/.pyenv/versions/3.6.15/bin:/home/vputz/.pyenv/versions/3.7.11/bin:/home/vputz/.pyenv/versions/3.8.12/bin:/home/vputz/.pyenv/versions/3.9.9/bin", "--output=lock.json", "--no-emit-warnings", "--style=universal", "--resolver-version", "pip-2020-resolver", "--indent=2", "--no-pypi", "--index=<https://pypi.org/simple/>", "--manylinux", "manylinux2014", "--interpreter-constraint", "CPython==3.9.*", "SQLAlchemy-Paginator==0.2", "SQLAlchemy==1.3.24", "aiohttp==3.7.4.post0", "amazon-braket-ocean-plugin==1.0.8", "amazon-braket-sdk==1.15.0", "arrow==1.2.1", "azure-quantum[qiskit]==0.24.201332", "backoff==1.11.1", "boto3==1.21.1", "botocore==1.24.1", "celery==5.1.2", "colorama==0.4.4", "deepmerge==1.0.1", "dimod==0.10.6", "dogpile.cache==1.1.4", "dwave-qbsolv==0.3.3.post0", "dwave-system==1.9.0", "fenestrate==0.3.0", "hypothesis==6.21.6", "icontract==2.6.0", "intervaltree==3.1.0", "jsonschema==3.2.0", "lz4==3.1.3", "networkx==2.6.3", "numba==0.53.1", "numpy==1.21.2", "opentelemetry-api==1.8.0", "opentelemetry-exporter-zipkin==1.8.0", "packaging==21.0", "psycopg2-binary==2.9.2", "pydantic==1.8.2", "pyrsistent==0.18.0", "python-decouple==3.4", "qcware-quasar==1.0.6", "qcware-transpile==0.1.1a20", "qiskit-aer==0.10.3", "qiskit-ibmq-provider==0.18.3", "qubovert==1.2.3", "redis==3.5.3", "requests==2.26.0", "scikit-learn==0.24.2", "scipy==1.7.1", "setuptools==58.0.4", "sortedcontainers==2.4.0", "structlog==21.5.0", "tabulate==0.8.9", "toolz==0.11.2", "torch==1.10.2", "types-requests==2.25.8", "unitair==0.1.6"], env: {"CPPFLAGS": "", "LDFLAGS": "", "PATH": "/home/vputz/.pyenv/shims:/home/vputz/.cargo/bin:/home/vputz/.local/bin:/home/vputz/.pyenv/bin:/opt/cuda/bin:/home/vputz/.dotnet/tools:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl", "PEX_IGNORE_RCFILES": "true", "PEX_PYTHON_PATH": "/home/vputz/.pyenv/versions/2.7.18/bin:/home/vputz/.pyenv/versions/3.10.2/bin:/home/vputz/.pyenv/versions/3.6.15/bin:/home/vputz/.pyenv/versions/3.7.11/bin:/home/vputz/.pyenv/versions/3.8.12/bin:/home/vputz/.pyenv/versions/3.9.9/bin", "PEX_ROOT": ".cache/pex_root", "PEX_SCRIPT": "pex3", "SSH_AUTH_SOCK": "/tmp/ssh-XXXXXXq0Ja51/agent.1205"}, working_directory: None, input_digests: InputDigests { complete: DirectoryDigest { digest: Digest { hash: Fingerprint<d81aa8a79bc3999a9e01b5df226e7649edb5bc4e16f187a17cdcc480e3f3196c>, size_bytes: 158 }, tree: "Some(..)" }, nailgun: DirectoryDigest { digest: Digest { hash: Fingerprint<e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855>, size_bytes: 0 }, tree: "Some(..)" }, input_files: DirectoryDigest { digest: Digest { hash: Fingerprint<d81aa8a79bc3999a9e01b5df226e7649edb5bc4e16f187a17cdcc480e3f3196c>, size_bytes: 158 }, tree: "Some(..)" }, immutable_inputs: {}, use_nailgun: [] }, output_files: {RelativePath("lock.json")}, output_directories: {}, timeout: None, execution_slot_variable: None, concurrency_available: 0, description: "Generate lockfile for python-default", level: Info, append_only_caches: {CacheName("pex_root"): RelativePath(".cache/pex_root")}, jdk_home: None, platform_constraint: None, cache_scope: PerSession }
Not sure which libraries in there are the tricky bit.
(the 10x was no joke; poetry resolution was about 27 sec, I stopped the pex resolution after 300 sec)
e
Ok. I think I have an angle on this without your original data @echoing-farmer-15630. The more you can constructively participate in the issue instead of here on slack, the better (for me). There is quite a bit of detail there and it would be great to keep context local. Stu was friendly filing the issue, but a disadvantage of that is you have not been connected yet. That's here btw: https://github.com/pantsbuild/pants/issues/14998
🙌 1
@echoing-farmer-15630 you may have missed this, but to get an apples to apples poetry timing, you'll need to 1st wipe the poetry cache / poetry venvs. If you follow the ticket I did that and Poetry takes (on my machine / network) ~11 minutes vs ~5 minutes for Pex with the pip-legacy-resolver. Pants uses the pip-2020-resolver though when locking with Pex and that takes ~infinite time, which I hope to have fixed here soon.
Ok, this is sorted now Vic: https://github.com/pantsbuild/pex/pull/1707#issuecomment-1089708875 I'm not sure if you knew this about your current lock?:
Copy code
WARNING: The candidate selected for download or install is a yanked version: 'ipython' candidate (version 7.30.0 at <https://files.pythonhosted.org/packages/56/32/bf656aed2b2e708ee4aca8730e9be78a5e732c767a67c6068a777af5987d/ipython-7.30.0-py3-none-any.whl#sha256=c8f3e07aefb9cf9e067f39686f035ce09b27a1ee602116a3030b91b6fc138ee4|https://files.pythonhosted.org/packages/56/32/bf656aed2b2e708ee4aca8730e9be78a5e732c767a67c6068a777af5987d/ipython-7.30.0-py3-none-any.whl#sha256=c8f3e07aefb9cf9e067f39686f035ce09b27a1ee602116a3030b91b6fc138ee4> (from <https://pypi.org/simple/ipython/|https://pypi.org/simple/ipython/>) (requires-python:>=3.7))
Reason for being yanked: Autoawait feature is broken in 7.30 See <https://github.com/ipython/ipython/pulls?q=milestone%3A7.30.1|https://github.com/ipython/ipython/pulls?q=milestone%3A7.30.1>
e
I didn't, and apologies for not participating in the issue--I'm in "partial work mode" covering daycare for our kids until a nanny shows up in May so my attention is somewhat patchy. Thanks very much for pursuing this, and really looking forward to using it. Ah... the yanked ipython candidate comes from this being an "overconstrained" constraints.txt (ie "pinned version numbers just from the last time we did a pip freeze, not from a reasonable lock generated from top-level requirements"). The new smaller set of constraints above doesn't directly include ipython, so it should be free to update (I think). Ugh, python packaging 🙂 Thanks so much for your work on pants; it helps a lot of pain points.
❤️ 1
6 Views
Open in Slack
PreviousNext
Powered by