flat-zoo-31952
10/18/2020, 4:04 PMhundreds-father-404
10/18/2020, 5:07 PMflat-zoo-31952
10/18/2020, 5:50 PM-ldebug
shows that it's clearly the pex resolution process that takes longhundreds-father-404
10/18/2020, 5:54 PMflat-zoo-31952
10/18/2020, 6:44 PM14:43:34 $ ./pants dependencies --type=3rdparty --transitive src/python/jumio/services/fawrensian/apps:mesos.devel
cytoolz>=0.11.0
dataclasses>=0.7; python_version < "3.7"
flask-apispec>=0.10.0
flask>=1.0.2
marshmallow-dataclass[enum,union]>=0.1.1
marshmallow>=3.5.1
hundreds-father-404
10/18/2020, 7:53 PMflat-zoo-31952
10/18/2020, 7:54 PMhundreds-father-404
10/18/2020, 7:57 PMflat-zoo-31952
10/19/2020, 1:10 AMhundreds-father-404
10/19/2020, 1:13 AM~/.cache/pants/named_caches/pex
ah...i figured it out, it’s my extra indexwhat do you mean?
flat-zoo-31952
10/19/2020, 1:14 AMhundreds-father-404
10/19/2020, 1:20 AMflat-zoo-31952
10/19/2020, 1:21 AMhundreds-father-404
10/19/2020, 1:26 AM~/.cache/pants/named_caches/pex
if they didn’t have any changes
btw, you can experiment with [python-setup].resolver_jobs
to improve concurrency: https://www.pantsbuild.org/docs/reference-python-setup#advanced-options
The default is 2, but that might be lower than optimal. We set that before the constraints file optimization to only use a single resolve, because it was common to resolve requirements as multiple distinct processes so you could get exhaustion. Generally, now, you only have a single process happening at a timepip install pex; pex --help
. (I think it depends on your number of cores)
So Pants will be performing slower with the default set to 2 than using Pex normallyflat-zoo-31952
10/19/2020, 6:15 PM$(nproc)
hundreds-father-404
10/19/2020, 6:38 PM0
for now. This is the code we use:
if python_setup.resolver_jobs:
argv.extend(["--jobs", str(python_setup.resolver_jobs)])
That’s something we should make less wonky though + revisit the default.flat-zoo-31952
10/19/2020, 7:00 PMsource pex-cmd 2>&1 | ts > long-pex-run.log
because i want to figure out what's taking it so damn long...i can see built wheels in the cache, it really shouldn't be that unless there's some kinda weird invalidation going onhundreds-father-404
10/19/2020, 7:02 PM[pex].verbosity
may also give some insights. I believe we log how log it takes to resolve each req iirc? It takes a value from 0 - 9. Translates to the -v
option when using Pex directlyflat-zoo-31952
10/19/2020, 7:02 PMhundreds-father-404
10/19/2020, 7:03 PM