happy-kitchen-89482
12/05/2021, 5:14 PMPEX_ROOT=${HOME}/.cache/pants/named_caches/pex_root time ${HOME}/.pyenv/versions/3.7.3/bin/python3.7 -m pex -vvvvvvvv --output-file repository.pex --resolver-version pip-2020-resolver --python ${HOME}/.pyenv/versions/3.7.3/bin/python3.7 --manylinux manylinux2014 -r requirements.txt --layout packed
ansicolors==1.1.8
pip==21.3.1
setuptools==56.2.0
types-setuptools==57.4.2
azure-common==1.1.27
azure-core==1.20.1
azure-graphrbac==0.61.1
azure-identity==1.7.1
azure-keyvault-certificates==4.3.0
azure-keyvault-keys==4.4.0
azure-keyvault-secrets==4.3.0
azure-keyvault==4.1.0
azure-mgmt-apimanagement==2.1.0
azure-mgmt-authorization==1.0.0
azure-mgmt-automation==1.0.0
azure-mgmt-batch==16.0.0
azure-mgmt-compute==22.1.0
azure-mgmt-containerinstance==8.0.0
azure-mgmt-containerregistry==8.2.0
azure-mgmt-containerservice==16.4.0
azure-mgmt-core==1.3.0
azure-mgmt-cosmosdb==6.4.0
azure-mgmt-frontdoor==1.0.0
azure-mgmt-hybridkubernetes==1.1.0
azure-mgmt-keyvault==9.3.0
azure-mgmt-logic==9.0.0
azure-mgmt-managementgroups==1.0.0
azure-mgmt-monitor==2.0.0
azure-mgmt-msi==1.0.0
azure-mgmt-network==19.3.0
azure-mgmt-rdbms==8.1.0
azure-mgmt-resource==19.0.0
azure-mgmt-security==1.0.0
azure-mgmt-servicefabric==1.0.0
azure-mgmt-sql==3.0.1
azure-mgmt-storage==18.0.0
azure-mgmt-subscription==1.0.0
azure-mgmt-web==3.0.0
azure-storage-blob==12.8.1
azure-storage-queue==12.1.6
enough-analyst-54434
12/05/2021, 5:27 PMhappy-kitchen-89482
12/05/2021, 5:34 PMpex/resolver.py
-vvvvvv
and nothingenough-analyst-54434
12/05/2021, 5:34 PMhappy-kitchen-89482
12/05/2021, 5:34 PMenough-analyst-54434
12/05/2021, 5:34 PMhappy-kitchen-89482
12/05/2021, 5:35 PMpex: Resolving interpreters: 0.3ms
pex: Building pex :: Resolving distributions (requirements.txt) :: Resolving requirements. :: Resolving for:
DistributionTarget(interpreter=PythonInterpreter('/Users/benjyw/.pyenv/versions/3.7.3/bin/python3.7', PythonIdentity('/Users/benjyw/.pyenv/versions/3.7.3/bin/python3.7', 'cp37', 'cp37m', 'macosx_10_pex: Hashing pex pex: Hashing pex: 179.1ms
pex: Isolating pex: 0.1ms
^CERROR: Operation cancelled by user
Traceback (most recent call last):
File "/Users/benjyw/.pyenv/versions/3.7.3/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/Users/benjyw/.pyenv/versions/3.7.3/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/Users/benjyw/src/pex/pex/__main__.py", line 8, in <module>
__name__ == "__main__" and pex.main()
File "/Users/benjyw/src/pex/pex/bin/pex.py", line 672, in main
env=env,
File "/Users/benjyw/src/pex/pex/bin/pex.py", line 692, in do_main
cache=ENV.PEX_ROOT,
File "/Users/benjyw/src/pex/pex/bin/pex.py", line 596, in build_pex
ignore_errors=options.ignore_errors,
File "/Users/benjyw/src/pex/pex/resolver.py", line 920, in resolve
lock_configuration=lock_configuration,
File "/Users/benjyw/src/pex/pex/resolver.py", line 992, in _download_internal
dest=dest, max_parallel_jobs=max_parallel_jobs
File "/Users/benjyw/src/pex/pex/resolver.py", line 117, in download_distributions
max_jobs=max_parallel_jobs,
File "/Users/benjyw/src/pex/pex/jobs.py", line 510, in execute_parallel
yield spawn_result.spawned_job.await_result()
File "/Users/benjyw/src/pex/pex/jobs.py", line 214, in await_result
job.wait()
File "/Users/benjyw/src/pex/pex/jobs.py", line 72, in wait
_, stderr = self._process.communicate()
File "/Users/benjyw/.pyenv/versions/3.7.3/lib/python3.7/subprocess.py", line 931, in communicate
self.wait()
File "/Users/benjyw/.pyenv/versions/3.7.3/lib/python3.7/subprocess.py", line 990, in wait
return self._wait(timeout=timeout)
File "/Users/benjyw/.pyenv/versions/3.7.3/lib/python3.7/subprocess.py", line 1624, in _wait
(pid, sts) = self._try_wait(0)
File "/Users/benjyw/.pyenv/versions/3.7.3/lib/python3.7/subprocess.py", line 1582, in _try_wait
(pid, sts) = os.waitpid(self.pid, wait_flags)
enough-analyst-54434
12/05/2021, 5:35 PMhappy-kitchen-89482
12/05/2021, 5:36 PMenough-analyst-54434
12/05/2021, 5:36 PMhappy-kitchen-89482
12/05/2021, 5:36 PMenough-analyst-54434
12/05/2021, 5:37 PMhappy-kitchen-89482
12/05/2021, 5:37 PMenough-analyst-54434
12/05/2021, 5:41 PMhappy-kitchen-89482
12/05/2021, 5:43 PMenough-analyst-54434
12/05/2021, 5:44 PMhappy-kitchen-89482
12/05/2021, 6:40 PM--intransitive
does speed things up, to about 12 seconds, but that is still slower than I would have expected. There is still about an 8 second logging pause right after
pex: Resolving interpreters: 0.2ms
pex: Building pex :: Resolving distributions (requirements.txt) :: Resolving requirements. :: Resolving for:
DistributionTarget(interpreter=PythonInterpreter('/Users/benjyw/.pyenv/versions/3.7.3/bin/python3.7', PythonIdentity('/Users/benjyw/.pyenv/versions/3.7.3/bin/python3.7', 'cp37', 'cp37m', 'macosx_10_pex: Spawning a maximum of 8 parallel jobs to process:
DistributionTarget(interpreter=PythonInterpreter('/Users/benjyw/.pyenv/versions/3.7.3/bin/python3.7', PythonIdentity('/Users/benjyw/.pyenv/versions/3.7.3/bin/python3.7', 'cp37', 'cp37m', 'macosx_10_16_x86_64', (3, 7, 3))))
pex: Hashing pex: 135.8ms
pex: Isolating pex: 0.1ms
enough-analyst-54434
12/05/2021, 7:01 PMhappy-kitchen-89482
12/05/2021, 7:01 PMpip download -r requirements.txt
takes ~25 seconds the first time, then ~12 seconds the second time, and it's definitely still sending network requests that second time, I guess either to check for new versions that match the requirement string (i.e., it doesn't optimize for ==
) or to check if a specific version has been yankedwitty-crayon-22786
12/06/2021, 3:27 AMpex3 lock update
impact this? Would we spend less time in pip
, such that its cache mattered less?enough-analyst-54434
12/06/2021, 3:32 AMhappy-kitchen-89482
12/06/2021, 5:18 AMpip install -r requirements,txt
in a venv and then modify requirements.txt
and rerun pip install -r requirements.txt
then it is incrementally additive - new versions will be installed or updated in the existing venv, but anything that hasn't changed doesn't require work (things that were removed from requirements.txt won't be uninstalled, but you can pip uninstall
manually and it's similarly fast). In any case, this is significantly faster than rebuilding a venv from scratch, which is what we do every time. I wonder if there are optimizations to be had along these lines.enough-analyst-54434
12/08/2021, 5:29 PMpip install
3. Translate the mutated venv back into a packed layout repository PEX for storage in LMDB
Alternatively, if Pants weren't to use Pex, it would need to do something similar to steps 1 and 3 on its own, coming up with some sort of cache-efficient packed layout that straddled being cache friendly and being sandbox creation latency friendly.witty-crayon-22786
12/08/2021, 5:35 PMnamed_caches/pex_root
is immutable, it’s only immutable after an entry is created… until then file locking is used, right?enough-analyst-54434
12/08/2021, 5:39 PMwitty-crayon-22786
12/08/2021, 5:45 PMenough-analyst-54434
12/08/2021, 5:47 PMhappy-kitchen-89482
12/08/2021, 10:37 PMwitty-crayon-22786
12/08/2021, 10:39 PMfor one thing, which previous version would we use? and how would we know how to find one?the inputs to the
pex3 lock update
feature answer this question to a degree: the assumption is that you would feed PEX an old lockfile, and the added/removed requirements since that lockfile.happy-kitchen-89482
12/08/2021, 10:48 PMwitty-crayon-22786
12/08/2021, 10:49 PMYou need the venv for an old lockfile, not just the lockfilethere is a point at which we have that during
./pants update-lockfile
, because we have the old file, and new requirements that aren’t included yet.happy-kitchen-89482
12/08/2021, 10:54 PMenough-analyst-54434
12/08/2021, 11:56 PMhappy-kitchen-89482
12/08/2021, 11:56 PMenough-analyst-54434
12/08/2021, 11:58 PMhappy-kitchen-89482
12/09/2021, 12:16 AM$ python -m venv venv/ && source venv/bin/activate && pip install --upgrade pip setuptools wheel
...
$ time pip install -r pants-requirements.txt
<snip>
real 3m47.303s
user 2m20.837s
sys 0m33.380s
$ echo "# whitespace change" >> pants-requirements.txt
$ time pip install -r pants-requirements.txt
Requirement already satisfied: adal==1.2.7 in ./venv/lib/python3.8/site-packages (from -r pants-requirements.txt (line 1)) (1.2.7)
Requirement already satisfied: aenum==2.2.6 in ./venv/lib/python3.8/site-packages (from -r pants-requirements.txt (line 2)) (2.2.6)
<snip>
Requirement already satisfied: zope.interface==5.4.0 in ./venv/lib/python3.8/site-packages (from -r pants-requirements.txt (line 412)) (5.4.0)
Requirement already satisfied: setuptools==58.4.0 in ./venv/lib/python3.8/site-packages (from -r pants-requirements.txt (line 413)) (58.4.0)
Requirement already satisfied: wheel>=0.29.0 in ./venv/lib/python3.8/site-packages (from scikit-build==0.12.0->-r pants-requirements.txt (line 346)) (0.37.0)
real 0m14.104s
user 0m12.643s
sys 0m0.546s
enough-analyst-54434
12/09/2021, 12:30 AMResolving pants-requirements.txt
that corresponds with a spike in network activity.happy-kitchen-89482
12/09/2021, 12:33 AMenough-analyst-54434
12/09/2021, 12:34 AMhappy-kitchen-89482
12/09/2021, 4:16 AM--python-run-against-entire-lockfile
and saw a bunch of "Extracting X requirements..." work, as you'd expect. Then ran with --python-run-against-entire-lockfile
, and still saw those (which you wouldn't expect! - that flag means no subsetting should be necessary). Ctrl-C'd out of that in the middle of all that work, then re-ran with --python-run-against-entire-lockfile
and this time those superfluous `Extracting X requirements..." work units didn't happen (as expected).witty-crayon-22786
12/09/2021, 5:09 AMenough-analyst-54434
12/09/2021, 6:08 PMhappy-kitchen-89482
12/09/2021, 6:16 PMwitty-crayon-22786
12/09/2021, 6:17 PMhappy-kitchen-89482
12/09/2021, 6:22 PMwitty-crayon-22786
12/09/2021, 6:23 PMenough-analyst-54434
12/09/2021, 7:03 PMwitty-crayon-22786
12/09/2021, 7:09 PMhappy-kitchen-89482
12/09/2021, 8:02 PMenough-analyst-54434
12/09/2021, 8:05 PMhappy-kitchen-89482
12/09/2021, 8:05 PMenough-analyst-54434
12/09/2021, 8:06 PMhappy-kitchen-89482
12/09/2021, 8:06 PMenough-analyst-54434
12/09/2021, 8:07 PMwitty-crayon-22786
12/09/2021, 8:08 PMWhen pairing with them I have observed cases where there were no changes at all to the constraints file, yet a re-resolve happened.when you observe this, you should look at the recorded traces to see what changed between runs 1, 2, and 3
enough-analyst-54434
12/09/2021, 8:09 PMwitty-crayon-22786
12/09/2021, 8:17 PMwhen you observe this, you should look at the recorded traces to see what changed between runs 1, 2, and 3even just looking at the log output in
-ldebug
will tell you, because it will render the digest of the inputs to the process.melodic-thailand-99227
12/10/2021, 7:19 AM-ldebug
causes the daemon to restarthappy-kitchen-89482
12/10/2021, 3:00 PMmelodic-thailand-99227
12/11/2021, 7:00 AMhappy-kitchen-89482
12/11/2021, 4:53 PMhundreds-father-404
12/13/2021, 11:10 PMhappy-kitchen-89482
12/14/2021, 1:00 AM