https://pantsbuild.org/ logo
Join Slack
Powered by
Having some problems experimenting with multiple s...
# general
n
Having some problems experimenting with multiple sets of requirements and corresponding resolves. In particular, what happens when Pants generates useless targets from its cartesian product algorithm when parametrization is being used for both dependencies and resolves.. Suppose we have 
python_requirements(name="A")
and 
python_requirements(name="B")
both consuming top-level 
<http://requirements.in|requirements.in>
and, then making a few adjustments to a couple requirements, say 
A
needs 
Flask~=2.0
and 
B
needs 
Flask~=1.0
(that is, there is really 
requirements_A.ini
and 
requirements_B.ini
, but they're both derivatives of a common 
requirements.ini
). Let them both be placed in the same 
BUILD
file 
./3rdparty/python
with 
A
belonging to resolve 
python-A
and 
B
belonging to 
python-B
. Now an application 
pex_binary(name="Z")
only works with resolve 
python-B
because it has a requirement that restricts 
Flask~=1.0
, say on 
apache-airflow>2.0
, which is reflected in the changes to 
requirements.ini
in 
B
. It also imports some first-party module associated with 
python_sources(name="M")
that previously only worked with 
python-A
. No big deal, set that dependency's resolve to 
resolve=parametrize("python-A", "python-B").
Now running 
Z
puts 
python-B
resolve in play, and everyone is happy. In particular, imports in both the app and the dependency get mapped back to requirements in 
B
unambiguously. But now suppose 
M
has an explicit dependency on a common requirement in 
A
and 
B
that Pants cannot infer for some reason, say 
3rdparty/python:A#Jinja2
(for example, it depends on 
pandas
, but needs 
Jinja2
for HTML styling that Pants did not infer). Then when 
python-B
is in play, Pants rightfully complains about the dependency. So then we parametrize it: 
dependencies=parametrize(python_A=["3rdparty/python:A#Jinja2@resolve=python-A"], python_B=["3rdparty/python:B#Jinja2@resolve=python-B"])
(the kwargs in 
parametrize
are needed when the args are not simple strings). Ok, this gets us past the error, but now Pants cannot do implicit inference because now two targets own the module represented by 
M
when 
python-B
resolve is in play: The one we care about and want Pants to infer (
./path/to/M/.__init__.py@resolve=python-B,dependencies=python_B
and the other that is impossible but nonetheless has a target generated because of the cartesian product methodology Pants uses with parametrizations (
./pants/to/M./__init__.py@resolve=python-B,dependencies=python_A
). But now apparently everyone has to disambiguate their dependence on 
M
, because on the flip side when 
python-A
is in play, it is a similar issue as ^. In our repo 
python-default
is 
python-A
in this toy example and 
python-B
is analogous to another resolve that was made for a service that uses a lot of first party code (and all of their thirdparty requirements), but has some conflicts with a few packages that are not consequential to the library/utility codebase, so it's a real use case, but maybe our approach is wrong. Wondering if there has been any thought about heirarchal resolves, where you could factor our the common requirements in 
A
and 
B
, build a common resolve, and then refer to targets in that common set when either 
python_A
or 
python_B
are in play (Pants would know its compatible with both). In the above example, 
Jinja2
could be referred to, say by 
3rdparty/python:A+B#Jinja2
, and this would work when either resolve 
python_B
or 
python_A
are in play and no disambiguation would be necessary (and no useless target permutations would be created). The idea is to imagine a base 
python-default
which generalizes as many of the repo's requirements as possible, and then small deltas to this resolve/requirements, but which are all compatible with each other on their common set of requirements.
w
https://github.com/pantsbuild/pants/issues/14519 is intended to resolve part of this… unfortunately, uptake was slow on actually consuming it, and a few bugs were discovered very recently (… as in, yesterday).
👍 1
#14519 should fill in explicit dependencies by matching the dependee’s resolve to the dependency’s resolve. that should mean that you do not need to 
parametrize
the 
dependencies
field
a fix for #14519 is in `2.12.1rc0`… unfortunately, there is a Python-specific issue that is preventing it from kicking in some critical cases: https://github.com/pantsbuild/pants/issues/16190
👍 1
but this is an important issue: i’m optimistic that we can get it resolved and cherry-picked in the next few (business) days
🙌 1
and regarding the cartesian product aspect of `parametrize`: that is a restriction that we hope to be able to remove in a coming version (possibly via allowing for something like 
python_sources(.., **parametrize(..))
). but at least in this case, parametrizing 
dependencies
should not be necessary.
n
Ah, it's good to know I'm not going crazy! Surprisingly it has been a struggle wrapping my head around the relationship between input requirements <> resolves <> targets that depend on them. Some of my recent questions here are basically from looking into this topic more closely (want to future proof our repo a bit as we start adding more users and more chances for dependency sharing and conflicts). So I gather after the patch we should drop the dependency parametrization and the inference will work as expected then.
w
yea, that’s correct. there is one other caveat that is less likely to be cherry-picked: https://github.com/pantsbuild/pants/issues/16175 … but the workaround isn’t too onerous
n
Not sure I quite follow the bug being described in that issue. Do you just mean we would need to put 
resolve=("python-default", ...)
in most targets now that we have multiple resolves w/ common reqs in play?
w
yes. and when you put it that way, it does seem more onerous.
which Pants version are you targeting?
n
2.13
👍 1
w
ok, thanks. i’ve marked the latter issue for 2.13.
n
In the meantime, we'll keep everything on a single 
python-default
resolve. The first service in needing this can wait (or just have its incompatible requirements hard copy/pasted for now). We're running the beta version of 2.13 now btw.
👍 1
w
some confirmed fixes for this cluster of issues have all landed in 
2.12.x
and 
2.13.x
: we’ll cut some more release candidates this afternoon
n
Thanks will update and let you know if any issues!
Give another should when a 2.13 release is made (just saw the 12). Thanks!
w
ah, yea. more changes bound for 2.13: hopefully today/tomorrow
M1 wheels might not be available until tomorrow
n
Trying now, but did something change with pex's lock creation? 
pants generate-lockfiles --resolve=python-default
takes ~150s to complete from a nuked cache (pants v.2.13.0.dev5 / pex v.2.1.90) whereas this version w/ pex v2.1.99 has been running for over 2,000s now (after multiple retries). The resolve contains only 50 pinned requirements.
@enough-analyst-54434 Just bringing my last question to your attention in case anything immediately comes to mind. I plan to look at all the release notes between those two versions and see if I can figure it out when I have time.
e
Thanks, I had missed that. Yeah, there was a change to lock file artifact downloading from using a bespoke downloader based on the Python stdlib to using Pip to do the downloads. This was triggered by the bespoke / urllib-based downloader not handling `Bearer`authentication in custom indexes; just `Basic`and `Digest`authentication (which are all urllib supports out of the box). That change came in https://github.com/pantsbuild/pex/pull/1811 and fixed https://github.com/pantsbuild/pex/issues/1803 (originally reported by a Pants user) and was released in Pex 2.1.93.
@nice-florist-55958 I'm very interested in a repro of the slowdown. Before I try whipping up an example, is there any chance you can share your lockfile for an apples to apples case I can use? Or just the input requirements used to attempt to generate the lockfile - that is plenty.
I'd really like to get the dust settled on the fallout from https://github.com/pantsbuild/pex/pull/1811 for inclusion in Pants 2.13.0 final. If 
pip download
is in fact the culprit I'll need to find a way to vendor in requests I think to both satisfy 
Bearer
auth support and speed.
n
Here are the inputs (excluding internal libs):
Copy code
##### DO NOT EDIT DIRECTLY - UPDATE CONTENTS OF ./resolves/default/*.in INSTEAD #####

##### APPS #####
gunicorn~=20.1
jupyterhub~=1.5
jupyterlab~=3.4

##### TOOLS #####

black~=22.6
ipython~=7.34
isort~=5.10
mypy~=0.961
pytest~=7.1

##### VENV_TOOLS #####
pex~=2.1
pip-tools~=6.8

##### LIBRARIES #####
cachetools~=5.2
colorama~=0.4
deprecation~=2.1
dill~=0.3
graphviz~=0.20
kerberos~=1.3
matplotlib~=3.5
networkx~=2.6
numpy~=1.21
pampy~=0.3
pandas~=1.3
plotly~=5.8
qPython~=2.0
scipy~=1.6
seaborn~=0.11
statsmodels~=0.13
toolz~=0.11
tornado~=6.1
types-setuptools~=57.4
typing-extensions~=4.2
xmltodict~=0.13

##### FRAMEWORKS #####
# Flask~=2.1
aiohttp~=3.8

##### OTHER #####
Jinja2~=3.1  # pandas html styling
XlsxWriter~=3.0  # pandas to_excel feature
setuptools~=62.4

##### DOCS #####
# sphinx-book-theme~=0.3  # requires pydata-sphinx-theme~=0.9
Sphinx~=4.5
nbconvert~=6.5
nbsphinx~=0.8
pydata-sphinx-theme~=0.9
sphinx-autoapi~=1.8
sphinx-panels~=0.6

##### DOCS_IPY_EXTRAS #####
ipython~=7.33
matplotlib~=3.5
e
Thanks - I'll try with that, I just ran on a handy other requirements.txt from @bitter-ability-32190 with 56 root reqs and observed ~no difference:
Copy code
$ rm -rf ~/.pex && time pex pex==2.1.90 -cpex3 -- lock create --style universal --resolver pip-2020-resolver -r requirements.txt --interpreter-constraint "==3.8.*" -o lock.json --indent 2

real	5m55.465s
user	1m38.191s
sys	0m7.800s
$ rm -rf ~/.pex && time pex pex==2.1.99 -cpex3 -- lock create --style universal --resolver pip-2020-resolver -r requirements.txt --interpreter-constraint "==3.8.*" --target-system linux --target-system mac -o lock2.json --indent 2

real	6m4.098s
user	1m42.499s
sys	0m8.183s
I'll try your and report back ...
Ah @nice-florist-55958 one more thing - what interpreter constraints do you use?
n
=>3.7,<4
e
Thanks! I'll change that to 
>=3.7,<3.11
since I have 3.11 betas.
n
"CPython>="3.7.5,<4"
e
Ok.
n
One thing to note, we don't use public PyPI, we have an internal Artifactory. I know that was an issue in the past when resolves were first released.
e
An auth issue IIRC and not a slowness issue though.
So, I found ~no difference again:
Copy code
$ rm -rf ~/.pex && time pex pex==2.1.90 -cpex3 -- lock create --style universal --resolver pip-2020-resolver -r requirements.txt --interpreter-constraint "CPython>=3.7.5,<3.11" -o lock.json --indent 2

real	5m9.145s
user	2m23.140s
sys	0m2.373s
$ rm -rf ~/.pex && time pex pex==2.1.99 -cpex3 -- lock create --style universal --resolver pip-2020-resolver -r requirements.txt --interpreter-constraint "CPython>=3.7.5,<3.11" --target-system linux --target-system mac -o lock2.json --indent 2

real	5m11.923s
user	2m48.548s
sys	0m5.636s
@nice-florist-55958 what does your 
[python_repos]
setup look like? Do you use list replacement, i.e. `indexes = ["artifactory"]`or list addition like 
indexes.add = ["artifactory"]
? Or... implicit add - which is sneaky 
indexes = "bob"
?!
And one more question @nice-florist-55958 - does your Artifactory serve up a `#<hashname>=<hashvalue>`fragment in its URLs (See SHOULDs here: https://peps.python.org/pep-0503/#specification). If not, that means Pex has to download each artifact to hash it when creating the lock. That, possibly combined with your answer to your custom indexes configuration, might be the crease where the slowness creeps in, but I think I need to set up a local Artifactory to find out.
n
[python-repos
is in 
~/.pants.rc
and is an explicit list declaration with on entry (replacement). I'm not sure about the URL. I think it does under the hood, but the pip log doesn't show it when downloading (but you can see it in the SciPy requirement inspection there is a URL of that form). Here's the full pex command invoked by generate-lockfiles goal:
Copy code
13:35:54.58 [DEBUG] spawned local process as Some(19114) for Process { argv: ["/ms/dist/python/PROJ/core/3.7.5/exec/bin/python", "./pex_v2.1.99", "lock", "create", "--tmpdir", ".tmp", "-vvvvvvvvv", "--cert", "ms-ca-chain.pem", "--python-path", "/ms/dist/python/PROJ/core/3.7.5/exec/bin", "--output=lock.json", "--no-emit-warnings", "--style=universal", "--resolver-version", "pip-2020-resolver", "--target-system", "linux", "--target-system", "mac", "--indent=2", "-r", "__pip_args.txt", "--no-pypi", "--index=<https://taymarti>:***@msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/simple", "--manylinux", "manylinux2014", "--interpreter-constraint", "CPython<4,>=3.7.5", "Jinja2~=3.1", "Sphinx~=4.5", "XlsxWriter~=3.0", "aiohttp~=3.8", "black~=22.6", "cachetools~=5.2", "colorama~=0.4", "deprecation~=2.1", "dill~=0.3", "graphviz~=0.20", "gunicorn~=20.1", "ipython~=7.33", "isort~=5.10", "jupyterhub~=1.5", "jupyterlab~=3.4", "kerberos~=1.3", "matplotlib~=3.5", "mypy~=0.961", "nbconvert~=6.5", "nbsphinx~=0.8", "networkx~=2.6", "numpy~=1.21", "pampy~=0.3", "pandas~=1.3", "pex~=2.1", "pip-tools~=6.8", "plotly~=5.8", "pydata-sphinx-theme~=0.9", "pytest~=7.1", "qPython~=2.0", "scipy~=1.6", "seaborn~=0.11", "setuptools~=62.4", "sphinx-autoapi~=1.8", "sphinx-panels~=0.6", "statsmodels~=0.13", "toolz~=0.11", "tornado~=6.1", "types-setuptools~=57.4", "typing-extensions~=4.2", "xmltodict~=0.13"], env: {"CPPFLAGS": "", "LANG": "C", "LC_ALL": "C", "LDFLAGS": "", "PATH": "/ms/dist/fsf/PROJ/graphviz/2.26.3/bin:/ms/dist/fsf/PROJ/libtool/1.5.18/bin:/ms/dist/syb/PROJ/oc/16.0.03.06_KERB_1.16/python/python34_64r/lib:/ms/dist/fsf/PROJ/git/2.33.1/<bin://ms/dist/releng/PROJ/git-manage/prod/bin://ms/dist/python/PROJ/core/3.7.5-0/exec/bin:/ms/dist/fsf/PROJ/gdb/4.17/bin://ms/dist/ossjava/PROJ/scala/2.9.0.1/bin:/ms/dist/msjava/PROJ/sunjdk/1.6.0_23/.exec/ia32.linux.2.4.glibc.2.3/bin://ms/dist/3rd/PROJ/perforce/2013.1/exec://ms/dist/releng/PROJ/p4manage/prod/bin:/ms/dist/syb/dba_env/oc/12.5.0.4/bin:/v/global/user/t/ta/taymarti/bin:/ms/dist/aurora/bin:/usr/local/bin:/usr/bin:/bin:/ms/dist/perl5/bin:/ms/dist/fsf/bin:/ms/dist/afs/bin:.:/usr/sbin:/v/global/user/t/ta/taymarti/aplus:/ms/dist/go/PROJ/go/1.17.1/bin>", "PEX_IGNORE_RCFILES": "true", "PEX_PYTHON_PATH": "/ms/dist/python/PROJ/core/3.7.5/exec/bin", "PEX_ROOT": ".cache/pex_root", "PEX_SCRIPT": "pex3"}, working_directory: None, input_digests: InputDigests { complete: DirectoryDigest { digest: Digest { hash: Fingerprint<22ef7392efe841e26d11209c84b94b00cba8575350ad2d7856cb64febd603b9d>, size_bytes: 343 }, tree: "Some(..)" }, nailgun: DirectoryDigest { digest: Digest { hash: Fingerprint<e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855>, size_bytes: 0 }, tree: "Some(..)" }, input_files: DirectoryDigest { digest: Digest { hash: Fingerprint<22ef7392efe841e26d11209c84b94b00cba8575350ad2d7856cb64febd603b9d>, size_bytes: 343 }, tree: "Some(..)" }, immutable_inputs: {}, use_nailgun: {} }, output_files: {RelativePath("lock.json")}, output_directories: {}, timeout: None, execution_slot_variable: None, concurrency_available: 0, description: "Generate lockfile for python-default", level: Info, append_only_caches: {CacheName("pex_root"): RelativePath(".cache/pex_root")}, jdk_home: None, platform_constraint: None, cache_scope: PerSession
And the truncated output of installing a package to give you an idea of the URLs:
Copy code
(3.7.5) [taymarti.ivapp1366932]> pip install -v pyfolio
Using pip 22.1.2 from /d/d1/user/taymarti/casper/codetree/codetree/src/admin/build-support/venv/default/lib/python3.7/site-packages/pip (python 3.7)
Looking in indexes: <https://taymarti>:****@msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/simple, <https://taymarti>:****@msde-docker-prod.ms.com:4443/artifactory/pypi-generic
Collecting pyfolio
  Downloading <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/28/b4/99799b743c4619752f88b70354924132a2e9b82f4656fe7c55eaa9101392/pyfolio-0.9.2.tar.gz> (91 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 91.1/91.1 kB 2.1 MB/s eta 0:00:00
  Running command python setup.py egg_info
  running egg_info
  creating /tmp/pip-pip-egg-info-ng69_d09/pyfolio.egg-info
  writing /tmp/pip-pip-egg-info-ng69_d09/pyfolio.egg-info/PKG-INFO
  writing dependency_links to /tmp/pip-pip-egg-info-ng69_d09/pyfolio.egg-info/dependency_links.txt
  writing requirements to /tmp/pip-pip-egg-info-ng69_d09/pyfolio.egg-info/requires.txt
  writing top-level names to /tmp/pip-pip-egg-info-ng69_d09/pyfolio.egg-info/top_level.txt
  writing manifest file '/tmp/pip-pip-egg-info-ng69_d09/pyfolio.egg-info/SOURCES.txt'
  reading manifest file '/tmp/pip-pip-egg-info-ng69_d09/pyfolio.egg-info/SOURCES.txt'
  reading manifest template '<http://MANIFEST.in|MANIFEST.in>'
  adding license file 'LICENSE'
  writing manifest file '/tmp/pip-pip-egg-info-ng69_d09/pyfolio.egg-info/SOURCES.txt'
  /d/d1/user/taymarti/casper/codetree/codetree/src/admin/build-support/venv/default/lib/python3.7/site-packages/setuptools/dist.py:775: UserWarning: Usage of dash-separated 'index-url' will not be supported in future versions. Please use the underscore name 'index_url' instead
    % (opt, underscore_opt)
  /d/d1/user/taymarti/casper/codetree/codetree/src/admin/build-support/venv/default/lib/python3.7/site-packages/setuptools/config/setupcfg.py:459: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
    warnings.warn(msg, warning_class)
  /d/d1/user/taymarti/casper/codetree/codetree/src/admin/build-support/venv/default/lib/python3.7/site-packages/setuptools/command/egg_info.py:637: SetuptoolsDeprecationWarning: Custom 'build_py' does not implement 'get_data_files_without_manifest'.
  Please extend command classes from setuptools instead of distutils.
    SetuptoolsDeprecationWarning
  Preparing metadata (setup.py) ... done
Requirement already satisfied: ipython>=3.2.3 in ./admin/build-support/venv/default/lib/python3.7/site-packages (from pyfolio) (7.33.0)
Requirement already satisfied: matplotlib>=1.4.0 in ./admin/build-support/venv/default/lib/python3.7/site-packages (from pyfolio) (3.5.2)
Requirement already satisfied: numpy>=1.11.1 in ./admin/build-support/venv/default/lib/python3.7/site-packages (from pyfolio) (1.21.1)
Requirement already satisfied: pandas>=0.18.1 in ./admin/build-support/venv/default/lib/python3.7/site-packages (from pyfolio) (1.3.5)
Requirement already satisfied: pytz>=2014.10 in ./admin/build-support/venv/default/lib/python3.7/site-packages (from pyfolio) (2022.1)
Requirement already satisfied: scipy>=0.14.0 in ./admin/build-support/venv/default/lib/python3.7/site-packages (from pyfolio) (1.6.1)
  Link requires a different Python (3.7.5 not in: '>=3.8'): <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/8b/99/b1ec652f2d60a13871a3053f312f9d78977be57e420f2a49d52ba503f1f4/scikit-learn-1.1.0.tar.gz#sha256=80f9904f5b1356adfc32406725dd94c8cc9c8d265047d98390033a6c238cbb29> (from <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/simple/scikit-learn/>) (requires-python:>=3.8)
  Link requires a different Python (3.7.5 not in: '>=3.8'): <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/41/11/e931951f048908ceaf2423db48ca6ad10e0b818c2960a3bc2dacb4fa4c1d/scikit-learn-1.1.1.tar.gz#sha256=3e77b71e8e644f86c8b5be7f1c285ef597de4c384961389ee3e9ca36c445b256> (from <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/simple/scikit-learn/>) (requires-python:>=3.8)
Collecting scikit-learn>=0.16.1
  Downloading <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/bd/05/e561bc99a615b5c099c7a9355409e5e57c525a108f1c2e156abb005b90a6/scikit_learn-1.0.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl> (24.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 24.8/24.8 MB 22.4 MB/s eta 0:00:00
Requirement already satisfied: seaborn>=0.7.1 in ./admin/build-support/venv/default/lib/python3.7/site-packages (from pyfolio) (0.11.2)
Collecting empyrical>=0.5.0
  Downloading <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/74/43/1b997c21411c6ab7c96dc034e160198272c7a785aeea7654c9bcf98bec83/empyrical-0.5.5.tar.gz> (52 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 52.8/52.8 kB 696.9 kB/s eta 0:00:00
FYI, tried the simpler tool resolves and those finished quickly. Have the python-default resolve going now and it's at 1,000s now (stripped off the internal packages too, so we're using the same requirements.in right now).
e
Ok. Thanks for the list replacement verification. So just 1 index and it's Artifactory. I'll set up an Artifactory to see if I repro a Pex 2.1.90 -> 2.1.99 difference.
n
Yep, just 
[python-repos].indexes=["what you see ^ w/o the ***"]
Is there some way to get pex to show any progress output? 
pex-verbosity=9
doesn't do anything/
e
There is not. The whole Pex trick for creating lock files is to have pip redirect a verbose log to a file which Pex post-processes for lock information. So, until the pip resolve is done, there is no data. In Pex 2.1.93 I added https://github.com/pantsbuild/pex/pull/1808 which tails the pip log in realtime for these 2 messages: https://github.com/pantsbuild/pex/pull/1808/files#diff-443e54ea41451726af072f7976469d8bdbb37d285fd49fc1a4bc6447588591c3R554-R571
SO you might see those if pip thinks it's taking a long time. If Pip does not think so or the time is being taken in the download-artifacts-to hash-them-theory, then you'd see no progress.
I should have my Artifactory experiment complete here within the hour. I'm hoping I repro. I'm using their free hosted trial on AWS. I suppose you folks self-host.
n
Hope so too, I’d feel bad if it was something I'm doing on my end. If you can't repro my next plan since the tool locks resolve fine is start small with python-default and see what gradually increasing the number of requirements does.
e
Ok, the default Artifactory PyPI mirroring does serve up urls with hashes:
Copy code
$ curl -nsSL <https://pextest.jfrog.io/artifactory/api/pypi/default-pypi/simple/pex/> | tail -2
<a href="../../packages/packages/f8/76/7e17f82344dd14a5a9cf76391cdf6edeede0e409432a319db98512d11375/pex-2.1.99.tar.gz#sha256=3feaf424e495c0855691044ebec6e9528730078c741eb572cbda18f4274d8fab" data-requires-python=">=2.7,<3.12,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*" rel="internal">pex-2.1.99.tar.gz</a><br />
</body></html>
n
Do you gather that's what ours is configured for from the pip msgs ^? I see such URL when it interrogates scikit for compatible versions but the actual download URL excludes the hash
Can you give me the cmd pex passes to pip and waits on? If I just install the requirements file pip finishes pretty quick. So then I gather from what you said the next step pex takes is to download all the pkgs again to hash them if the repo doesn't include them in the url, but even that to me doesn't seem like it'd take too long for only so many requirements?
e
Those pip logs aren't verbose enough. You'd want to add 
--log pip.log
and then look in there when it's done. For apples-to-apples you'd want to nuke or move aside your pip cache too (
~/.cache/pip/
on Linux).
And you'd want to do this in a new venv with nothing installed. ... AND pip download, not pip install. Heh.
As far as exact command lines go, the usual recipe. Run 
./pants --no-process-cleanup ...
and look in the sandbox dir printed out of the relevant process (usually the last) for the 
__run.sh
script.
n
Ok will do that when I get back to my desk. So no issues resolving w/ Artifactory then? I'll see if I can find out more details about our configuration.
e
Still running that test. I completed my 2.1.90 test in ~8 minutes. Now trying 2.1.99.
👍 1
Ok, I think the fact 2.1.99 shows as faster is just down to network variability, but:
Copy code
$ rm -rf ~/.pex && time pex pex==2.1.90 -cpex3 -- lock create --no-pypi --index <https://pextest.jfrog.io/artifactory/api/pypi/default-pypi/simple> --style universal --resolver pip-2020-resolver -r requirements.txt --interpreter-constraint "CPython>=3.7.5,<3.11" -o lock.json --indent 2

real	7m55.306s
user	2m33.507s
sys	0m2.097s
$ rm -rf ~/.pex && time pex pex==2.1.99 -cpex3 -- lock create --no-pypi --index <https://pextest.jfrog.io/artifactory/api/pypi/default-pypi/simple> --style universal --resolver pip-2020-resolver -r requirements.txt --interpreter-constraint "CPython>=3.7.5,<3.11" --target-system linux --target-system mac -o lock2.json --indent 2

real	7m11.080s
user	3m15.935s
sys	0m6.381s
So I'm still not reproing 😕
n
Damn alright, let me see if I can break it down into a simpler reproducing scenario later then. As an aside, I forget, can the pex version be changed in pants.toml?
e
Yes. Just a sec...
You'll need the sha256 hash of the Pex PEX as well as its size.
n
Copy code
2022-07-22T16:27:02,725  Found link <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/e0/da/55f51ea951e1b7c63a579c09dd7db825bb730ec1fe9c0180fc77bfb31448/urllib3-1.25.6-py2.py3-none-any.whl#sha256=3de946ffbed6e6746608990594d08faac602528ac7015ac28d33cee6a45b7398> (from <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/simple/urllib3/>) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, <4), version: 1.25.6
^ Partial output from the 
pip.log
you suggested. Looks like it does have the 
#hashtype=hash
format.
So I identified the culprit in my resolve....it's 
jupyterlab~=3.6
Copy code
(3.7.5) [taymarti.ivapp1366932]> pants generate-lockfiles --resolve=python-default > jupyterlab.log
16:32:46.37 [INFO] Completed: Generate lockfile for python-default
16:32:46.37 [ERROR] 1 Exception encountered:

 ProcessExecutionFailure: Process 'Generate lockfile for python-default' failed with exit code 1.
stdout:

stderr:
pid 6418 -> /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/venvs/cdfeb520dfa6124682c6d0bbd8d72592968ffe8c/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e/bin/python -sE /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/venvs/cdfeb520dfa6124682c6d0bbd8d72592968ffe8c/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -q --cache-dir /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/pip_cache --log /tmp/pants-sandbox-8vXgH3/.tmp/pex-pip-log.087u044l/pip.log download --dest /tmp/pants-sandbox-8vXgH3/.tmp/tmp1okhvtga/ms.dist.python.PROJ.core.3.7.5-0..exec.@sys.bin.python3.7 --requirement __pip_args.txt jupyterlab~=3.4 --index-url <https://taymarti:AKCp5e2WiL6DMJCGEpzsEq7LJU2dU6RooAWJkgGaN8sPMtkbkUfT1u16eZKHerNcEAyoiaF5H@msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/simple> --retries 5 --timeout 15 exited with 1 and STDERR:
Ignoring the following environment variables in Pex venv mode:
_PEX_TARGET_SYSTEMS_FILE=/tmp/pants-sandbox-8vXgH3/.tmp/tmp8cwburbn/target_systems.json

WARNING: Discarding <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/02/29/27d6ab0b1566db75cce1fde817cb2cf327066fc545c08ceb4e61a77a07de/pywinpty-2.0.5.tar.gz#sha256=e125d3f1804d8804952b13e33604ad2ca8b9b2cac92b27b521c005d1604794f8> (from <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/simple/pywinpty/>) (requires-python:>=3.7). Command errored out with exit status 1: /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/venvs/cdfeb520dfa6124682c6d0bbd8d72592968ffe8c/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e/bin/python /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/venvs/cdfeb520dfa6124682c6d0bbd8d72592968ffe8c/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/pip_cache/.tmp/tmp96vu51hh Check the logs for full command output.
WARNING: Discarding <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/3d/1d/b884c586cb4ff53da97f9c9e177dd73e74a5d848e2954492341f118413fc/pywinpty-1.1.6.tar.gz#sha256=8808f07350c709119cc4464144d6e749637f98e15acc1e5d3c37db1953d2eebc> (from <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/simple/pywinpty/>) (requires-python:>=3.6). Command errored out with exit status 1: /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/venvs/cdfeb520dfa6124682c6d0bbd8d72592968ffe8c/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e/bin/python /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/venvs/cdfeb520dfa6124682c6d0bbd8d72592968ffe8c/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/pip_cache/.tmp/tmpaqki805j Check the logs for full command output.
WARNING: Discarding <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/1f/86/8c320359c04a85084f416bc01e265d09384655c5643953c0eaea2c5c7636/pywinpty-1.1.4.tar.gz#sha256=cc700c9d5a9fcebf677ac93a4943ca9a24db6e2f11a5f0e7e8e226184c5036f7> (from <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/simple/pywinpty/>) (requires-python:>=3.6). Command errored out with exit status 1: /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/venvs/cdfeb520dfa6124682c6d0bbd8d72592968ffe8c/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e/bin/python /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/venvs/cdfeb520dfa6124682c6d0bbd8d72592968ffe8c/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/pip_cache/.tmp/tmpms2qcnm1 Check the logs for full command output.
WARNING: Discarding <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/07/e0/a26c9ac4e16aefc0b66a203f0140373245b6298443fa35978a3c680c8726/pywinpty-1.1.3.tar.gz#sha256=3a1d57b338390333812a5eed31c93c7d8ba82b131078063703e731946d90c9f2> (from <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/simple/pywinpty/>) (requires-python:>=3.6). Command errored out with exit status 1: /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/venvs/cdfeb520dfa6124682c6d0bbd8d72592968ffe8c/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e/bin/python /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/venvs/cdfeb520dfa6124682c6d0bbd8d72592968ffe8c/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/pip_cache/.tmp/tmpkql02wif Check the logs for full command output.
WARNING: Discarding <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/fb/dc/06ed1e684dfba3929a985f1845eb86f8480f0418293fc9402526e4b8bf10/pywinpty-1.1.2.tar.gz#sha256=f1718838e1c7c700e5f0b79d5d5e05243ff583313ff88e47bb94318ba303e565> (from <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/simple/pywinpty/>) (requires-python:>=3.6). Command errored out with exit status 1: /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/venvs/cdfeb520dfa6124682c6d0bbd8d72592968ffe8c/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e/bin/python /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/venvs/cdfeb520dfa6124682c6d0bbd8d72592968ffe8c/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/pip_cache/.tmp/tmpv48751qt Check the logs for full command output.
WARNING: Discarding <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/40/ed/694e4a642e3b8afac6d6e70538a3684e1049f2020a33002b58e1ac5cd4b3/pywinpty-1.1.1.tar.gz#sha256=4a3ffa2444daf15c5f65a76b5b2864447cc915564e41e2876816b9e4fe849070> (from <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/simple/pywinpty/>) (requires-python:>=3.6). Command errored out with exit status 1: /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/venvs/cdfeb520dfa6124682c6d0bbd8d72592968ffe8c/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e/bin/python /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/venvs/cdfeb520dfa6124682c6d0bbd8d72592968ffe8c/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/pip_cache/.tmp/tmpjxhmplla Check the logs for full command output.
WARNING: Discarding <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/46/02/aeb98dc7ac48e3fc6ba13159be5c6fbd0d19e2d11edf996e96d031fa86ca/pywinpty-1.1.0.tar.gz#sha256=f60ed4947fe0841e7d7cafae4885a0040d4994a4647dd21be4ed4266789885c9> (from <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/simple/pywinpty/>). Command errored out with exit status 1: /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/venvs/cdfeb520dfa6124682c6d0bbd8d72592968ffe8c/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e/bin/python /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/venvs/cdfeb520dfa6124682c6d0bbd8d72592968ffe8c/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/pip_cache/.tmp/tmpcnelkw2o Check the logs for full command output.
WARNING: Discarding <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/5d/97/8e43c2152a638cdb83d45644eb125c752abe67249f94bb3e3e29b0709685/pywinpty-0.5.7.tar.gz#sha256=2d7e9c881638a72ffdca3f5417dd1563b60f603e1b43e5895674c2a1b01f95a0> (from <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/simple/pywinpty/>). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
WARNING: Discarding <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/21/ae/acbedcee475d049647e45ee949e23ca492764fcd8027a073fcf07646d47c/pywinpty-0.5.5.tar.gz#sha256=cec9894ecb34de3d7b1ca121dd98433035b9f8949b5095e84b103b349231509c> (from <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/simple/pywinpty/>). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Cannot install jupyterlab because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit <https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies>

The conflict is caused by:
    jupyter-server 1.18.1 depends on pywinpty; os_name == "nt"
    jupyter-server 1.18.0 depends on pywinpty; os_name == "nt"
    jupyter-server 1.17.1 depends on pywinpty; os_name == "nt"
    jupyter-server 1.17.0 depends on pywinpty; os_name == "nt"
    jupyter-server 1.16.0 depends on pywinpty; os_name == "nt"

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict




Use `--no-process-cleanup` to preserve process chroots for inspection.
^ That is from having only that requirement in the input file. Any clues why new pex would not like this or if old pex handled this differently?
If I replace the constraint with 
jupyterlab
(unconstrained), it just hangs forever. When I remove it from the original ~50 requirements, the resolve completes in the expected amount of time.
Here are the relevant parts of the resolved lockfile using 2.1.90 that 2.1.99 reports is impossible:
Copy code
{
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "022759b09c96a4e2feb95de59ce4283e04e17782efe197b91d23a47521609b77",
              "url": "<https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/36/6c/fe8c416a9f1a64b9296918e9096b68da81fc50e5fefba8077841c22d6691/jupyter_server-1.18.1-py3-none-any.whl>"
            }
          ],
          "project_name": "jupyter-server",
          "requires_dists": [
            "Send2Trash",
            "anyio<4,>=3.1.0",
            "argon2-cffi",
            "coverage; extra == \"test\"",
            "ipykernel; extra == \"test\"",
            "jinja2",
            "jupyter-client>=6.1.12",
            "jupyter-core>=4.7.0",
            "nbconvert>=6.4.4",
            "nbformat>=5.2.0",
            "packaging",
            "pre-commit; extra == \"test\"",
            "prometheus-client",
            "pytest-console-scripts; extra == \"test\"",
            "pytest-cov; extra == \"test\"",
            "pytest-mock; extra == \"test\"",
            "pytest-timeout; extra == \"test\"",
            "pytest-tornasync; extra == \"test\"",
            "pytest>=6.0; extra == \"test\"",
            "pywinpty; os_name == \"nt\"",
            "pyzmq>=17",
            "requests; extra == \"test\"",
            "terminado>=0.8.3",
            "tornado>=6.1.0",
            "traitlets>=5.1",
            "websocket-client"
          ],
          "requires_python": ">=3.7",
          "version": "1.18.1"
        },    
      "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "bb2ea2aa81e96eee6a6b79d87e1d1648d3f8b87f9a64499e0b92b30d141e76df",
              "url": "<https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/a4/d9/e80172360b380ea41f423f3ddf2d0d986dd88c7b69c102494f52fd84012f/pywin32-304-cp37-cp37m-win_amd64.whl>"
            },
            {
              "algorithm": "sha256",
              "hash": "3c7bacf5e24298c86314f03fa20e16558a4e4138fc34615d7de4070c23e65af3",
              "url": "<https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/05/6b/9f8421a9a2ab5f33cbb9fd2f282ac971e584f6a83d44f6672bd17f1d68b2/pywin32-304-cp310-cp310-win32.whl>"
            }
          ],
          "project_name": "pywin32",
          "requires_dists": [],
          "requires_python": null,
          "version": "304"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "906a3048ecfec6ece1b141594ebbbcd5c4751960714c50524e8e907bb77c9207",
              "url": "<https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/a8/43/d73c866d6b50ac86ec16c4e70465d4fe1f7aa405f59361cdc6970809444b/pywinpty-2.0.6-cp37-none-win_amd64.whl>"
            }
          ],
         "project_name": "pywinpty",
          "requires_dists": [],
          "requires_python": ">=3.7",
          "version": "2.0.6"
        },
e
Thanks for the details! It may be the `; os_name == "nt"`environment markers. In the 2.1.99 side of things Pants uses a new Pex 
--target-system
feature to avoid locking Windows only deps (since Pex does not support Windows yet) by adding `--target-system linux --target-system mac`to all lock creation commands. I'll see if I can repro this more narrow case here presently.
@nice-florist-55958 do you have hand-rolled jupyterlab in your Artifactory? On PyPI stable maxes out at 3.4.4; so the jupyterlab~=3.6 requirement looks strange / doesn't work since none exist. The failure is very fast: https://pypi.org/project/jupyterlab/#history
Your requirements.txt has jupyterlab~=3.4 so I'll try that.
n
Was a typo, it's ~=3.4 So we can narrow it even further in our case; it's 
pywinpty
Just trying to build a resolve with that requirement, no matter the version constraint, yields: 
ERROR: No matching distribution found for pywinpty.
Here's the full log -- looks like however pex is invoking pip, it causes it to discard each candidate until there are none left and then reports the above no match error:
Copy code
stderr:
pid 3381 -> /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/venvs/cdfeb520dfa6124682c6d0bbd8d72592968ffe8c/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e/bin/python -sE /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/venvs/cdfeb520dfa6124682c6d0bbd8d72592968ffe8c/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -q --cache-dir /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/pip_cache --log /tmp/pants-sandbox-8izcQD/.tmp/pex-pip-log.gly3n_0p/pip.log download --dest /tmp/pants-sandbox-8izcQD/.tmp/tmptqe71f5_/ms.dist.python.PROJ.core.3.7.5-0..exec.@sys.bin.python3.7 --requirement __pip_args.txt pywinpty --index-url <https://taymarti:AKCp5e2WiL6DMJCGEpzsEq7LJU2dU6RooAWJkgGaN8sPMtkbkUfT1u16eZKHerNcEAyoiaF5H@msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/simple> --retries 5 --timeout 15 exited with 1 and STDERR:
Ignoring the following environment variables in Pex venv mode:
_PEX_TARGET_SYSTEMS_FILE=/tmp/pants-sandbox-8izcQD/.tmp/tmp4rc0kn1z/target_systems.json

WARNING: Discarding <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/02/29/27d6ab0b1566db75cce1fde817cb2cf327066fc545c08ceb4e61a77a07de/pywinpty-2.0.5.tar.gz#sha256=e125d3f1804d8804952b13e33604ad2ca8b9b2cac92b27b521c005d1604794f8> (from <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/simple/pywinpty/>) (requires-python:>=3.7). Command errored out with exit status 1: /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/venvs/cdfeb520dfa6124682c6d0bbd8d72592968ffe8c/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e/bin/python /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/venvs/cdfeb520dfa6124682c6d0bbd8d72592968ffe8c/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/pip_cache/.tmp/tmpchbtwc4c Check the logs for full command output.
WARNING: Discarding <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/3d/1d/b884c586cb4ff53da97f9c9e177dd73e74a5d848e2954492341f118413fc/pywinpty-1.1.6.tar.gz#sha256=8808f07350c709119cc4464144d6e749637f98e15acc1e5d3c37db1953d2eebc> (from <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/simple/pywinpty/>) (requires-python:>=3.6). Command errored out with exit status 1: /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/venvs/cdfeb520dfa6124682c6d0bbd8d72592968ffe8c/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e/bin/python /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/venvs/cdfeb520dfa6124682c6d0bbd8d72592968ffe8c/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/pip_cache/.tmp/tmp7ibi7f_a Check the logs for full command output.
WARNING: Discarding <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/1f/86/8c320359c04a85084f416bc01e265d09384655c5643953c0eaea2c5c7636/pywinpty-1.1.4.tar.gz#sha256=cc700c9d5a9fcebf677ac93a4943ca9a24db6e2f11a5f0e7e8e226184c5036f7> (from <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/simple/pywinpty/>) (requires-python:>=3.6). Command errored out with exit status 1: /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/venvs/cdfeb520dfa6124682c6d0bbd8d72592968ffe8c/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e/bin/python /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/venvs/cdfeb520dfa6124682c6d0bbd8d72592968ffe8c/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/pip_cache/.tmp/tmp502b9au1 Check the logs for full command output.
WARNING: Discarding <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/07/e0/a26c9ac4e16aefc0b66a203f0140373245b6298443fa35978a3c680c8726/pywinpty-1.1.3.tar.gz#sha256=3a1d57b338390333812a5eed31c93c7d8ba82b131078063703e731946d90c9f2> (from <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/simple/pywinpty/>) (requires-python:>=3.6). Command errored out with exit status 1: /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/venvs/cdfeb520dfa6124682c6d0bbd8d72592968ffe8c/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e/bin/python /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/venvs/cdfeb520dfa6124682c6d0bbd8d72592968ffe8c/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/pip_cache/.tmp/tmp6dn7chg1 Check the logs for full command output.
WARNING: Discarding <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/fb/dc/06ed1e684dfba3929a985f1845eb86f8480f0418293fc9402526e4b8bf10/pywinpty-1.1.2.tar.gz#sha256=f1718838e1c7c700e5f0b79d5d5e05243ff583313ff88e47bb94318ba303e565> (from <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/simple/pywinpty/>) (requires-python:>=3.6). Command errored out with exit status 1: /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/venvs/cdfeb520dfa6124682c6d0bbd8d72592968ffe8c/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e/bin/python /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/venvs/cdfeb520dfa6124682c6d0bbd8d72592968ffe8c/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/pip_cache/.tmp/tmpdhl34ot0 Check the logs for full command output.
WARNING: Discarding <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/40/ed/694e4a642e3b8afac6d6e70538a3684e1049f2020a33002b58e1ac5cd4b3/pywinpty-1.1.1.tar.gz#sha256=4a3ffa2444daf15c5f65a76b5b2864447cc915564e41e2876816b9e4fe849070> (from <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/simple/pywinpty/>) (requires-python:>=3.6). Command errored out with exit status 1: /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/venvs/cdfeb520dfa6124682c6d0bbd8d72592968ffe8c/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e/bin/python /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/venvs/cdfeb520dfa6124682c6d0bbd8d72592968ffe8c/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/pip_cache/.tmp/tmpnygpxmps Check the logs for full command output.
WARNING: Discarding <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/46/02/aeb98dc7ac48e3fc6ba13159be5c6fbd0d19e2d11edf996e96d031fa86ca/pywinpty-1.1.0.tar.gz#sha256=f60ed4947fe0841e7d7cafae4885a0040d4994a4647dd21be4ed4266789885c9> (from <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/simple/pywinpty/>). Command errored out with exit status 1: /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/venvs/cdfeb520dfa6124682c6d0bbd8d72592968ffe8c/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e/bin/python /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/venvs/cdfeb520dfa6124682c6d0bbd8d72592968ffe8c/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /d/d1/user/taymarti/.cache/pants/named_caches/pex_root/pip_cache/.tmp/tmpt0dt4tzw Check the logs for full command output.
WARNING: Discarding <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/5d/97/8e43c2152a638cdb83d45644eb125c752abe67249f94bb3e3e29b0709685/pywinpty-0.5.7.tar.gz#sha256=2d7e9c881638a72ffdca3f5417dd1563b60f603e1b43e5895674c2a1b01f95a0> (from <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/simple/pywinpty/>). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
WARNING: Discarding <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/21/ae/acbedcee475d049647e45ee949e23ca492764fcd8027a073fcf07646d47c/pywinpty-0.5.5.tar.gz#sha256=cec9894ecb34de3d7b1ca121dd98433035b9f8949b5095e84b103b349231509c> (from <https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/simple/pywinpty/>). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement pywinpty
ERROR: No matching distribution found for pywinpty
e
Ok, I don't repro yet, but this gives me something narrow to work with anyhow. Thank you!
Copy code
$ rm -rf ~/.pex && time pex pex==2.1.90 -cpex3 -- lock create --style universal --resolver-version pip-2020-resolver jupyterlab~=3.4 --interpreter-constraint ">=3.7.5,<4" --indent 2 -o lock-2.1.90.json

real	1m4.829s
user	0m49.383s
sys	0m0.726s
$ rm -rf ~/.pex && time pex pex==2.1.99 -cpex3 -- lock create --style universal --resolver-version pip-2020-resolver jupyterlab~=3.4 --interpreter-constraint ">=3.7.5,<4" --indent 2 -o lock-2.1.99-nots.json

real	1m2.174s
user	0m47.035s
sys	0m0.615s
$ rm -rf ~/.pex && time pex pex==2.1.99 -cpex3 -- lock create --style universal --resolver-version pip-2020-resolver jupyterlab~=3.4 --interpreter-constraint ">=3.7.5,<4" --indent 2 -o lock-2.1.99-ts.json --target-system linux --target-system mac

real	1m11.881s
user	1m41.466s
sys	0m4.847s
Wait, @nice-florist-55958 can you run 
/d/d1/user/taymarti/.cache/pants/named_caches/pex_root/venvs/cdfeb520dfa6124682c6d0bbd8d72592968ffe8c/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e/bin/python -c 'import os; print(os.name)'
and report the result?
I'm guessing 
nt
?
Are you Windows WSL?
I forget.
n
Sure, 1sec. And I don't know much about this, but I can see the only pre-built wheel is for 
win_amd64
and trying to 
pip download ...whl --no-deps --platform=nt
results in a platform not compatible error, but using 
--platform=win_amd64
is works. On the other hand, looks like pex 2.1.99 is trying to download the sdist...
posix
e
Ok. And as to Windows - I do forget that. Are you in facts using Windows + WSL?
Or no?
n
No, SSH into a RHEL7 host.
e
Hrm, then why I can't repro is beyond me. I'm on a Linux laptop.
Ok, I'll drill in more.
n
So a requirements file with just 
pywinpty
and 
python_requirements(source="requirements.txt", resolve="python-default")
works for you?
Is Pex attempting to build the sdist or something? I know that will definitely error since it requires cargo and that's not installed on this host.
e
Well, that's a bogus example. It needs to be 
pywinpty; os_name == "nt"
to mirror the original issue. That environment marker is critical. It should evaluate False unless you're on Windows and short circuit Pip from trying to download / resolve it.
n
(The possible reasonf or all the exit 1 errors iterating through each version?)
Oh, good point...let me add that.
e
The reason for the exit errors is you're not on NT but you're asking for an NT dep.
n
It's actually the same error if I add that marker.
But let me ask a dumb question - does the marker in this context mean do what I say and I should expect an error if I'm on the wrong platform, or only do this if the platform I'm on matches?
e
The latter.
Yeah - I do not repro:
Copy code
$ rm -rf ~/.pex && time pex pex==2.1.99 -cpex3 -- lock create --style universal --resolver-version pip-2020-resolver 'pywinpty; os_name == "nt"' --interpreter-constraint ">=3.7.5,<4" --indent 2 -o lock-2.1.99-ts.json --target-system linux --target-system mac

real	0m14.318s
user	0m48.131s
sys	0m4.079s
To re-group. You're using Pants X (and Pex Y) - what are the X and Y?
✅ 1
n
And for whatever reason, Pex isn't respecting the marker in .99 (at least in my env). If I just do pip install -r requirements.txt pip informs me that it is skipping for precisely that reason.
e
Yeah, again pip download - Pex doesn;t use pip install there. But also, the download code is patched for locking.
n
Same result w/ download (pip skips it).
e
Ok. Back to which Pants & Pex - I want to make sure since I cannot repro using Pex directly.
Maybe you can try the 
pex
command line I'm using above?
The difference would be to add 
--no-pypi --index <your artifactory user:pass@ URL>
n
Hmm, I'll try. For what it's worth and completeness, here is the successfully generated lockfile w/ v.2.1.90 (or pants 2.13.0.dev5). The error from trying to build this lock w/ 2.1.99/2.13.0rc0 is above. Two things strike me as notable: 1. The lockfile here shows a bdist, but all I see are attempts to download the sdist in v2.1.99 2. The version in the lockfile is 2.0.6, but 
pip index versions pywinpty
reveals the latest version available is 2.0.5 (which is where pex v2.1.99 starts searching from) a. It's true 2.0.6 is the latest, but this repo is a cached repo that might not have the latest available on public PyPI because it only has approved packages/releases b. 
pip download
shows that 2.0.6 bdist actually is there, so possible there is some synchronization that hasn't happened yet on the repo -- but this is probably a minor point -- still curious how pex 2.1.90 discovered it though in the first place
Copy code
// This lockfile was autogenerated by Pants. To regenerate, run:
//
//    ./pants generate-lockfiles --resolve=python-default
//
// --- BEGIN PANTS LOCKFILE METADATA: DO NOT EDIT OR REMOVE ---
// {
//   "version": 2,
//   "valid_for_interpreter_constraints": [
//     "CPython<4,>=3.7.5"
//   ],
//   "generated_with_requirements": [
//     "pywinpty>=1.1.0; os_name == \"nt\""
//   ]
// }
// --- END PANTS LOCKFILE METADATA ---

{
  "allow_builds": true,
  "allow_prereleases": false,
  "allow_wheels": true,
  "build_isolation": true,
  "constraints": [],
  "locked_resolves": [
    {
      "locked_requirements": [
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "906a3048ecfec6ece1b141594ebbbcd5c4751960714c50524e8e907bb77c9207",
              "url": "<https://msde-docker-prod.ms.com:4443/artifactory/api/pypi/mspypi-cache/a8/43/d73c866d6b50ac86ec16c4e70465d4fe1f7aa405f59361cdc6970809444b/pywinpty-2.0.6-cp37-none-win_amd64.whl>"
            }
          ],
          "project_name": "pywinpty",
          "requires_dists": [],
          "requires_python": ">=3.7",
          "version": "2.0.6"
        }
      ],
      "platform_tag": [
        "cp37",
        "cp37m",
        "manylinux_2_17_x86_64"
      ]
    }
  ],
  "path_mappings": {},
  "pex_version": "2.1.90",
  "prefer_older_binary": false,
  "requirements": [
    "pywinpty>=1.1.0; os_name == \"nt\""
  ],
  "requires_python": [
    "<4,>=3.7.5"
  ],
  "resolver_version": "pip-2020-resolver",
  "style": "universal",
  "transitive": true,
  "use_pep517": null
}
e
pip index
is apparently not good: https://pypi.org/project/pywinpty/#history
Or you're internal Artifactory is behind.
Does your internal Artifactory have 
maturin
? The newer pywinpty use this as their build backend.
h, sorry - read your point a more thoroughly. Ok.
n
Let me check. The first error pip reports is 
cargo
is missing. Btw if not clear from above on versions: 
pex v2.1.99 / pants v2.13.0rc0 / bad
pex v2.1.90 / pants v.2.13.0dev5 / good
e
Yeah, and I have cargo; so maybe that's the difference.
I can try getting that off my PATH.
Ack - Thanks for the Pants / Pex version confirmations!
If you get cargo missing errors that means your Artifactory has maturin. That's a project to integrate Rust native Python extension builds which uses cargo to build Rust bits.
n
Using your pex cmd above, it's so far just hanging, so not sure if it's doing anything. As a sanity check I tried 
pex pex==2.1.99 -cpex --version
just to see if the tool itself is working and it printed the version.
e
OK! Thank you!:
Copy code
$ rm -rf ~/.cache/pip ~/.pex && time PATH="$(echo $PATH | tr : '\n' | grep -v cargo | xargs | tr ' ' :)" pex --python python3.7 pex==2.1.99 -cpex3 -- lock create --style universal --resolver-version pip-2020-resolver 'pywinpty>=1.1.0; os_name == "nt"' --interpreter-constraint "CPython<4,>=3.7.5" --indent 2 -o lock-2.1.99-ts.json --target-system linux --target-system mac
pid 93480 -> /home/jsirois/.pex/venvs/1a2d739a57fbf583e32383354f8b55a71f546f2a/fd300d5feeace385d1195e058d7c9082241e354a/bin/python -sE /home/jsirois/.pex/venvs/1a2d739a57fbf583e32383354f8b55a71f546f2a/fd300d5feeace385d1195e058d7c9082241e354a/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -q --cache-dir /home/jsirois/.pex/pip_cache --log /tmp/pex-pip-log.x6a_l2zy/pip.log download --dest /tmp/tmp58vvt01j/home.jsirois..pyenv.versions.3.7.13.bin.python3.7 pywinpty>=1.1.0; os_name == "nt" --index-url <https://pypi.org/simple> --retries 5 --timeout 15 exited with 1 and STDERR:
Ignoring the following environment variables in Pex venv mode:
_PEX_TARGET_SYSTEMS_FILE=/tmp/tmpkardc9d5/target_systems.json

WARNING: Discarding <https://files.pythonhosted.org/packages/2e/a6/50815a4aeff3946c5723bcf3363980f98461fa01b8fa125c41fbe852540a/pywinpty-2.0.6.tar.gz#sha256=a91a77d23f29a58b44f62a9474a31ed67df1277cddb69665275f8d22429046ac> (from <https://pypi.org/simple/pywinpty/>) (requires-python:>=3.7). Command errored out with exit status 1: /home/jsirois/.pex/venvs/1a2d739a57fbf583e32383354f8b55a71f546f2a/fd300d5feeace385d1195e058d7c9082241e354a/bin/python /home/jsirois/.pex/venvs/1a2d739a57fbf583e32383354f8b55a71f546f2a/fd300d5feeace385d1195e058d7c9082241e354a/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /home/jsirois/.pex/pip_cache/.tmp/tmp5w7r1775 Check the logs for full command output.
WARNING: Discarding <https://files.pythonhosted.org/packages/02/29/27d6ab0b1566db75cce1fde817cb2cf327066fc545c08ceb4e61a77a07de/pywinpty-2.0.5.tar.gz#sha256=e125d3f1804d8804952b13e33604ad2ca8b9b2cac92b27b521c005d1604794f8> (from <https://pypi.org/simple/pywinpty/>) (requires-python:>=3.7). Command errored out with exit status 1: /home/jsirois/.pex/venvs/1a2d739a57fbf583e32383354f8b55a71f546f2a/fd300d5feeace385d1195e058d7c9082241e354a/bin/python /home/jsirois/.pex/venvs/1a2d739a57fbf583e32383354f8b55a71f546f2a/fd300d5feeace385d1195e058d7c9082241e354a/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /home/jsirois/.pex/pip_cache/.tmp/tmpv_16nsmg Check the logs for full command output.
WARNING: Discarding <https://files.pythonhosted.org/packages/b2/44/c4b593af3c5e54bfb191f5fcb7232ec66d945a1ef632bbef4befacd50047/pywinpty-2.0.4.tar.gz#sha256=a5002c74afc1ddcc85fca25de58ce9cef1a22957582981418c81aaee218675ad> (from <https://pypi.org/simple/pywinpty/>) (requires-python:>=3.7). Command errored out with exit status 1: /home/jsirois/.pex/venvs/1a2d739a57fbf583e32383354f8b55a71f546f2a/fd300d5feeace385d1195e058d7c9082241e354a/bin/python /home/jsirois/.pex/venvs/1a2d739a57fbf583e32383354f8b55a71f546f2a/fd300d5feeace385d1195e058d7c9082241e354a/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /home/jsirois/.pex/pip_cache/.tmp/tmpq6mib484 Check the logs for full command output.
WARNING: Discarding <https://files.pythonhosted.org/packages/c3/61/bda90dba80bc6cb905acebd0bf0710777ab04feb29d0f438202da0e82d72/pywinpty-2.0.3.tar.gz#sha256=6b29a826e896105370c38d53904c3aaac6c36146a50448fc0ed5082cf9d092bc> (from <https://pypi.org/simple/pywinpty/>) (requires-python:>=3.6). Command errored out with exit status 1: /home/jsirois/.pex/venvs/1a2d739a57fbf583e32383354f8b55a71f546f2a/fd300d5feeace385d1195e058d7c9082241e354a/bin/python /home/jsirois/.pex/venvs/1a2d739a57fbf583e32383354f8b55a71f546f2a/fd300d5feeace385d1195e058d7c9082241e354a/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /home/jsirois/.pex/pip_cache/.tmp/tmpkouh_spo Check the logs for full command output.
WARNING: Discarding <https://files.pythonhosted.org/packages/43/88/f7d9b3b45a8b6109f60702cfa3b011edf7cfd8d2680f738bc4d1cb120897/pywinpty-2.0.2.tar.gz#sha256=20ec117183f79642eff555ce0dd1823f942618d65813fb6122d14b6e34b5d05a> (from <https://pypi.org/simple/pywinpty/>) (requires-python:>=3.6). Command errored out with exit status 1: /home/jsirois/.pex/venvs/1a2d739a57fbf583e32383354f8b55a71f546f2a/fd300d5feeace385d1195e058d7c9082241e354a/bin/python /home/jsirois/.pex/venvs/1a2d739a57fbf583e32383354f8b55a71f546f2a/fd300d5feeace385d1195e058d7c9082241e354a/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /home/jsirois/.pex/pip_cache/.tmp/tmp9p8r9xwj Check the logs for full command output.
WARNING: Discarding <https://files.pythonhosted.org/packages/b0/85/4a563e195fd5de30e4e18c51a4c398adbcbf656ca9166182ee44e2454719/pywinpty-2.0.1.tar.gz#sha256=14e7321c6d43743af0de175fca9f111c5cc8d0a9f7c608c9e1cc69ec0d6ac146> (from <https://pypi.org/simple/pywinpty/>) (requires-python:>=3.6). Command errored out with exit status 1: /home/jsirois/.pex/venvs/1a2d739a57fbf583e32383354f8b55a71f546f2a/fd300d5feeace385d1195e058d7c9082241e354a/bin/python /home/jsirois/.pex/venvs/1a2d739a57fbf583e32383354f8b55a71f546f2a/fd300d5feeace385d1195e058d7c9082241e354a/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /home/jsirois/.pex/pip_cache/.tmp/tmpdnyz9e5x Check the logs for full command output.
WARNING: Discarding <https://files.pythonhosted.org/packages/3d/1d/b884c586cb4ff53da97f9c9e177dd73e74a5d848e2954492341f118413fc/pywinpty-1.1.6.tar.gz#sha256=8808f07350c709119cc4464144d6e749637f98e15acc1e5d3c37db1953d2eebc> (from <https://pypi.org/simple/pywinpty/>) (requires-python:>=3.6). Command errored out with exit status 1: /home/jsirois/.pex/venvs/1a2d739a57fbf583e32383354f8b55a71f546f2a/fd300d5feeace385d1195e058d7c9082241e354a/bin/python /home/jsirois/.pex/venvs/1a2d739a57fbf583e32383354f8b55a71f546f2a/fd300d5feeace385d1195e058d7c9082241e354a/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /home/jsirois/.pex/pip_cache/.tmp/tmp_6qvel8w Check the logs for full command output.
WARNING: Discarding <https://files.pythonhosted.org/packages/01/c1/c6c7bf4851d78d71c1859e124626cf53d48638f3907ca80731579382a1b3/pywinpty-1.1.5.tar.gz#sha256=92125f0f8e4e64bb5f3bf270a182c9206dc1765542c59bc07441908a9db17504> (from <https://pypi.org/simple/pywinpty/>) (requires-python:>=3.6). Command errored out with exit status 1: /home/jsirois/.pex/venvs/1a2d739a57fbf583e32383354f8b55a71f546f2a/fd300d5feeace385d1195e058d7c9082241e354a/bin/python /home/jsirois/.pex/venvs/1a2d739a57fbf583e32383354f8b55a71f546f2a/fd300d5feeace385d1195e058d7c9082241e354a/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /home/jsirois/.pex/pip_cache/.tmp/tmph2g9pln7 Check the logs for full command output.
WARNING: Discarding <https://files.pythonhosted.org/packages/1f/86/8c320359c04a85084f416bc01e265d09384655c5643953c0eaea2c5c7636/pywinpty-1.1.4.tar.gz#sha256=cc700c9d5a9fcebf677ac93a4943ca9a24db6e2f11a5f0e7e8e226184c5036f7> (from <https://pypi.org/simple/pywinpty/>) (requires-python:>=3.6). Command errored out with exit status 1: /home/jsirois/.pex/venvs/1a2d739a57fbf583e32383354f8b55a71f546f2a/fd300d5feeace385d1195e058d7c9082241e354a/bin/python /home/jsirois/.pex/venvs/1a2d739a57fbf583e32383354f8b55a71f546f2a/fd300d5feeace385d1195e058d7c9082241e354a/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /home/jsirois/.pex/pip_cache/.tmp/tmpwnoicfjn Check the logs for full command output.
WARNING: Discarding <https://files.pythonhosted.org/packages/07/e0/a26c9ac4e16aefc0b66a203f0140373245b6298443fa35978a3c680c8726/pywinpty-1.1.3.tar.gz#sha256=3a1d57b338390333812a5eed31c93c7d8ba82b131078063703e731946d90c9f2> (from <https://pypi.org/simple/pywinpty/>) (requires-python:>=3.6). Command errored out with exit status 1: /home/jsirois/.pex/venvs/1a2d739a57fbf583e32383354f8b55a71f546f2a/fd300d5feeace385d1195e058d7c9082241e354a/bin/python /home/jsirois/.pex/venvs/1a2d739a57fbf583e32383354f8b55a71f546f2a/fd300d5feeace385d1195e058d7c9082241e354a/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /home/jsirois/.pex/pip_cache/.tmp/tmpegfcae97 Check the logs for full command output.
WARNING: Discarding <https://files.pythonhosted.org/packages/fb/dc/06ed1e684dfba3929a985f1845eb86f8480f0418293fc9402526e4b8bf10/pywinpty-1.1.2.tar.gz#sha256=f1718838e1c7c700e5f0b79d5d5e05243ff583313ff88e47bb94318ba303e565> (from <https://pypi.org/simple/pywinpty/>) (requires-python:>=3.6). Command errored out with exit status 1: /home/jsirois/.pex/venvs/1a2d739a57fbf583e32383354f8b55a71f546f2a/fd300d5feeace385d1195e058d7c9082241e354a/bin/python /home/jsirois/.pex/venvs/1a2d739a57fbf583e32383354f8b55a71f546f2a/fd300d5feeace385d1195e058d7c9082241e354a/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /home/jsirois/.pex/pip_cache/.tmp/tmpfr4mhpay Check the logs for full command output.
WARNING: Discarding <https://files.pythonhosted.org/packages/40/ed/694e4a642e3b8afac6d6e70538a3684e1049f2020a33002b58e1ac5cd4b3/pywinpty-1.1.1.tar.gz#sha256=4a3ffa2444daf15c5f65a76b5b2864447cc915564e41e2876816b9e4fe849070> (from <https://pypi.org/simple/pywinpty/>) (requires-python:>=3.6). Command errored out with exit status 1: /home/jsirois/.pex/venvs/1a2d739a57fbf583e32383354f8b55a71f546f2a/fd300d5feeace385d1195e058d7c9082241e354a/bin/python /home/jsirois/.pex/venvs/1a2d739a57fbf583e32383354f8b55a71f546f2a/fd300d5feeace385d1195e058d7c9082241e354a/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /home/jsirois/.pex/pip_cache/.tmp/tmp7hfk3ir6 Check the logs for full command output.
WARNING: Discarding <https://files.pythonhosted.org/packages/46/02/aeb98dc7ac48e3fc6ba13159be5c6fbd0d19e2d11edf996e96d031fa86ca/pywinpty-1.1.0.tar.gz#sha256=f60ed4947fe0841e7d7cafae4885a0040d4994a4647dd21be4ed4266789885c9> (from <https://pypi.org/simple/pywinpty/>). Command errored out with exit status 1: /home/jsirois/.pex/venvs/1a2d739a57fbf583e32383354f8b55a71f546f2a/fd300d5feeace385d1195e058d7c9082241e354a/bin/python /home/jsirois/.pex/venvs/1a2d739a57fbf583e32383354f8b55a71f546f2a/fd300d5feeace385d1195e058d7c9082241e354a/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /home/jsirois/.pex/pip_cache/.tmp/tmpvg2yc1t_ Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement pywinpty>=1.1.0
ERROR: No matching distribution found for pywinpty>=1.1.0


real	1m0.534s
user	0m10.041s
sys	0m1.089s
So, it is all about the 
cargo
build.
Digging a bit more ...
Works:
Copy code
$ rm -rf ~/.cache/pip ~/.pex && time PATH="$(echo $PATH | tr : '\n' | grep -v cargo | xargs | tr ' ' :)" pex --python python3.7 pex==2.1.99 -cpex3 -- lock create --style universal --resolver-version pip-2020-resolver 'pywinpty>=1.1.0; os_name == "nt"' --interpreter-constraint "CPython<4,>=3.7.5" --indent 2 -o lock-2.1.99-ts.json

real	0m4.234s
user	0m2.857s
sys	0m0.231s
So it is this Windows-specific cargo-based build + 
--target-system linux --target-system mac
n
And confirmed downloaded manchurin 0.12.17
e
Thanks so much Taylor. This is a full repro now that I can work with.
I'll file an issue and link here.
n
🙂
I don't quite understand the issue, but can I safely assume that this would still be an issue even if I had 
cargo
installed because it really should have been skipped? Or at least if so, then Pex is only extracting data / verifying requirements in the sdist, not actually trying to build it (because it needs to build a generalized lockfile for all platforms)?
p.s. - once I realized the locks being generated included Windows-only deps, I tried to see if platform constraints were an option like interpreter constraints to avoid them in the first place, but seems not. Is that something to consider adding or not really since the lock should be resolved for all platforms and platform resolution is meant to happen at build time for a given target (I believe you can express platform constraints for 
pex_binary
)?
e
So, my 
cargo
claims look like they were wrong / irrelevant. Bug here: https://github.com/pantsbuild/pex/issues/1856
As far as Windows-only deps, the new Pex `--target-system {linux,mac,windows}`list option is mean exactly for this, restricting locks to non-Windows (for now). In the case you found though, this does not work!
The current implementation of `--target-system`fakes out values for 
sys_platform
and `platform_system`environment markers but not 
os_name
(See: https://peps.python.org/pep-0508/#environment-markers). I think missing `os_name`is the issue here but I'll find out soon.
Yeah, that was it.
Copy code
$ git diff
diff --git a/pex/resolve/locker.py b/pex/resolve/locker.py
index 186cd34..b240bb2 100644
--- a/pex/resolve/locker.py
+++ b/pex/resolve/locker.py
@@ -309,6 +309,11 @@ class Locker(LogAnalyzer):
 
 
 # See <https://www.python.org/dev/peps/pep-0508/#environment-markers> for more about these values.
+_OS_NAME = {
+    TargetSystem.LINUX: "posix",
+    TargetSystem.MAC: "posix",
+    TargetSystem.WINDOWS: "nt",
+}
 _PLATFORM_SYSTEM = {
     TargetSystem.LINUX: "Linux",
     TargetSystem.MAC: "Darwin",
@@ -364,6 +369,10 @@ def patch(
             TargetSystem.values()
         ):
             target_systems = {
+                "os_names": [
+                    _OS_NAME[target_system]
+                    for target_system in lock_configuration.target_systems
+                ],
                 "platform_systems": [
                     _PLATFORM_SYSTEM[target_system]
                     for target_system in lock_configuration.target_systems
diff --git a/pex/resolve/locker_patches.py b/pex/resolve/locker_patches.py
index 5c55ad5..e2fa82b 100644
--- a/pex/resolve/locker_patches.py
+++ b/pex/resolve/locker_patches.py
@@ -31,6 +31,7 @@ if target_systems_file:
 
     with open(target_systems_file) as fp:
         target_systems = json.load(fp)
+    os_names = target_systems["os_names"]
     platform_systems = target_systems["platform_systems"]
     sys_platforms = target_systems["sys_platforms"]
     platform_tag_regexps = target_systems["platform_tag_regexps"]
@@ -56,6 +57,7 @@ def patch_marker_evaluate():
 
     python_versions_strings = versions_to_string(python_versions) or skip
     python_full_versions_strings = versions_to_string(python_full_versions) or skip
+    os_names_strings = os_names or skip
     platform_systems_strings = platform_systems or skip
     sys_platforms_strings = sys_platforms or skip
 
@@ -66,6 +68,8 @@ def patch_marker_evaluate():
             return python_versions_strings
         if name == "python_full_version":
             return python_full_versions_strings
+        if name == "os_name":
+            return os_names_strings
         if name == "platform_system":
             return platform_systems_strings
         if name == "sys_platform":
Copy code
$ rm -rf ~/.pex && time python3.7 -mpex.cli lock create --style universal --resolver-version pip-2020-resolver 'pywinpty>=1.1.0; os_name == "nt"' --interpreter-constraint "CPython<4,>=3.7.5" --indent 2 -o lock-2.1.99-ts.json --target-system linux --target-system mac

real	0m1.066s
user	0m0.930s
sys	0m0.119s

$ cat lock-2.1.99-ts.json 
{
  "allow_builds": true,
  "allow_prereleases": false,
  "allow_wheels": true,
  "build_isolation": true,
  "constraints": [],
  "locked_resolves": [
    {
      "locked_requirements": [],
      "platform_tag": null
    }
  ],
  "path_mappings": {},
  "pex_version": "2.1.98",
  "prefer_older_binary": false,
  "requirements": [
    "pywinpty>=1.1.0; os_name == \"nt\""
  ],
  "requires_python": [
    "<4,>=3.7.5"
  ],
  "resolver_version": "pip-2020-resolver",
  "style": "universal",
  "target_systems": [
    "linux",
    "mac"
  ],
  "transitive": true,
  "use_pep517": null
}
n
Hmm…so If I understand the conclusion now, this would have failed for everyone because of the slightly different way pants invokes Pex in the upgraded version?
e
No, Pants invokes Pex consistently.
It fails for the environment marker 
os_name
- that is a very uncommon env marker,.
Folks generally use 
sys_platform == "win32"
.
n
I meant when you tried to install my requirements, jupyteab, etc. using v2.1.99 and couldn't repro. The only differentiating factor was cargo missing? But if that's not relevant, than I can only assume it's the slightly different way you were invoking pex compared to pants? And sry I'm on my phone now so might have missed a detail ^ I'll check back later.
e
Because old PEX just included Windows deps.
And that was also usually not a problem, but it can be.
See https://github.com/pantsbuild/pex/issues/1821 for that corner case that someone else was unlucky enough to hit.
So, in summary, old Pex did the wrong thing and locked Windows deps even though Pex doesn't work on Windows. That - in rare cases - could lead to a spurious failure to lock as outlined in https://github.com/pantsbuild/pex/issues/1821. New Pex fixes this by restricting locks to Mac & Linux, but it had a bug in so doing, leaving out 1 of 3 possible env markers used to track OS. You just helped fix that.
n
Cool cool, the conclusion makes rough sense to me now. I still wonder why you didn't reproduce the error right away when using 2.1.99? Because I thought all your invocations has the target system Mac/Linux option but manages to succeed.
Happy to have helped find another edge case! Lmk when it makes it into another .13 :)
e
I still wonder why you didn't reproduce the error right away when using 2.1.99?
Agreed. I'd like to understand what I missed in my rushing around debugging and being sloppy today before I go to bed. It's never good to let details like that slip!
You always pay for it in the end.
OK, it does appear to be contingent on 
cargo
availability. I need to dig more to see how that variable plays in to be able to write a solid unit test repro anyhow.
n
I do not envy you and the rest of the pants/pex developers having to imagine tests to capture stuff like this!
@witty-crayon-22786 I realized I could just use the existing lockfile and Pants would build from that one. So on testing these changes: 1. [The good]: Not getting any errors packaging 
A@resolve=Y
when it depends on 
B@{resolve=X,resolve=Y}
-- the old error being 
A@resolve=Y
depends on 
B@resolve=X
and the resolves are not compatible. 2. Not noticing any errors for 
pex_binary
that don't explicitly declare their resolve (or the 
python_sources
they directly depend on); I remember you said that might still be an issue. 3. [The bad]: If 
A@resolve=Y
depends on 
B@{resolve=X,resovle=Y}
and 
B
explicitly lists the 
python_requirement
3rdparty/python:RequirementsOfX#SomeRequirement
in its dependencies, then Pants still complains that it depends on said requirement and has incompatible resolves in play. I thought one of the fixes was to auto-infer that when 
Y
is in play, that an explicit dependency on a 
python_requirement
in 
X
would be checked for membership in 
Y
so it doesn't have to itself be parametrized? 4. Trying to get out of [3], resorted to parametrizing, but then the ambiguous ownership warning kicks in and the dependency inference fails and 
B
doesn't get included in the dependencies unless itself is explicitly listed in 
A
's dependencies as 
B@resolve=Y,dependencies=YDepsAlias
-- and even then, it still complains about having a dependency on 
3rdparty/python:RequirementsOfX#SomeRequirement
. 5. Tried something just to see - removed the parametrization of 
B
's 3rdparty deps and changed the listed dep of 
A
to merely 
B@resolve=Y
(hoping this is what you meant by still needing to explicitly list resolves), but no luck, same result as [4] [4] definitely seems like an error to me as there should be zero ambiguity about the requirements and resolve in play. Could definitely be a mistake on my side, so take w/ grain of salt. Note: We are taking an approach of keeping 
python_requirements
generators in one-to-one sync with a resolve to avoid dealing with factorizations / requirement hierarchies While probably sub-optimal, I'm not aware of a way to override a requirement when you give the generator a source file, or to feed multiple files with the duplicate requirements masking priors similar to a dictionary update, etc. which would make such an organization scheme easier to create/manage. Note 2: It's late and I'm going out of town for a few days, it's possible I might have made a mistake somewhere, just reporting initial feedback after testing.
w
we should probably vacate this thread since the 
parametrize
content won’t be interesting for John: i’ll ping you from a new one.
9 Views
Open in Slack
PreviousNext
Powered by