https://pantsbuild.org/ logo
Join Slack
Powered by
Using `pex3 lock create .` fails in a folder with ...
# general
a
Using 
pex3 lock create .
fails in a folder with only 
pyproject.toml
when it includes editable dependencies, eg:
Copy code
[tool.poetry.dependencies]
python = "^3.9"
requests = "^2.28.2"
project2 = {path = "../project2", develop = true}
Without the 
project2
line the lock works. However with that line this fails with 
FileNotFoundError: [Errno 2] No such file or directory: '/Users/shalabh/.pex/pip_cache/.tmp/project2'
. is there an easy way to exclude these path based dependencies? Alternatively, what's the best way to create a dependencies pex from such a project. I'm thinking if we export this into a 
requirements.txt
and then feed that to pex - that might be a better option?
ideally pex3 lock would handle local dependencies as well.
r
So it looks like pex lock does support local dependencies. It’s explained here https://www.pantsbuild.org/docs/python-third-party-dependencies#local-requirements
e
@abundant-autumn-67998 works fine with modern pip. Use 
--pip-version 23.0.1
.
Given:
Copy code
jsirois@Gill-Windows:~/support/pex/Shalabh-multi-poetry $ tree project1 project2
project1
├── project1.py
├── pyproject.toml
└── README.md
project2
├── project2.py
├── pyproject.toml
└── README.md

0 directories, 3 files
jsirois@Gill-Windows:~/support/pex/Shalabh-multi-poetry $ cat project1/pyproject.toml
[tool.poetry]
name = "project1"
version = "0.1.0"
description = ""
authors = ["John Sirois <john.sirois@gmail.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.9"
requests = "^2.28.2"
project2 = {path = "../project2", develop = true}


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
jsirois@Gill-Windows:~/support/pex/Shalabh-multi-poetry $ cat project2/pyproject.toml
[tool.poetry]
name = "project2"
version = "0.1.0"
description = ""
authors = ["John Sirois <john.sirois@gmail.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.9"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
I find, as you say:
Copy code
jsirois@Gill-Windows:~/support/pex/Shalabh-multi-poetry $ pex3 lock create project1/
pid 1949 -> /home/jsirois/.pex/venvs/a6f267be531b3188bc0c77f171ec077cc764ed0d/5985ed09b49a653d6596b0e14d134c5456cf1a9f/bin/python -sE /home/jsirois/.pex/venvs/a6f267be531b3188bc0c77f171ec077cc764ed0d/5985ed09b49a653d6596b0e14d134c5456cf1a9f/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --use-deprecated legacy-resolver --isolated -q --cache-dir /home/jsirois/.pex/pip_cache --log /tmp/pex-pip-log.tv17i8qe/pip.log download --dest /tmp/tmpcjko2ckl/home.jsirois.bin.pex.venv.bin.python project1/ --index-url <https://pypi.org/simple> --retries 5 --timeout 15 exited with 2 and STDERR:
ERROR: Exception:
Traceback (most recent call last):
  File "/home/jsirois/.pex/venvs/a6f267be531b3188bc0c77f171ec077cc764ed0d/5985ed09b49a653d6596b0e14d134c5456cf1a9f/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 223, in _main
    status = self.run(options, args)
  File "/home/jsirois/.pex/venvs/a6f267be531b3188bc0c77f171ec077cc764ed0d/5985ed09b49a653d6596b0e14d134c5456cf1a9f/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 180, in wrapper
    return func(self, options, args)
  File "/home/jsirois/.pex/venvs/a6f267be531b3188bc0c77f171ec077cc764ed0d/5985ed09b49a653d6596b0e14d134c5456cf1a9f/lib/python3.10/site-packages/pip/_internal/commands/download.py", line 130, in run
    requirement_set = resolver.resolve(
  File "/home/jsirois/.pex/venvs/a6f267be531b3188bc0c77f171ec077cc764ed0d/5985ed09b49a653d6596b0e14d134c5456cf1a9f/lib/python3.10/site-packages/pip/_internal/resolution/legacy/resolver.py", line 180, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
  File "/home/jsirois/.pex/venvs/a6f267be531b3188bc0c77f171ec077cc764ed0d/5985ed09b49a653d6596b0e14d134c5456cf1a9f/lib/python3.10/site-packages/pip/_internal/resolution/legacy/resolver.py", line 385, in _resolve_one
    dist = self._get_dist_for(req_to_install)
  File "/home/jsirois/.pex/venvs/a6f267be531b3188bc0c77f171ec077cc764ed0d/5985ed09b49a653d6596b0e14d134c5456cf1a9f/lib/python3.10/site-packages/pip/_internal/resolution/legacy/resolver.py", line 337, in _get_dist_for
    dist = self.preparer.prepare_linked_requirement(req)
  File "/home/jsirois/.pex/venvs/a6f267be531b3188bc0c77f171ec077cc764ed0d/5985ed09b49a653d6596b0e14d134c5456cf1a9f/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 480, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
  File "/home/jsirois/.pex/venvs/a6f267be531b3188bc0c77f171ec077cc764ed0d/5985ed09b49a653d6596b0e14d134c5456cf1a9f/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 503, in _prepare_linked_requirement
    local_file = unpack_url(
  File "/home/jsirois/.pex/venvs/a6f267be531b3188bc0c77f171ec077cc764ed0d/5985ed09b49a653d6596b0e14d134c5456cf1a9f/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 263, in unpack_url
    unpack_file(file.path, location, file.content_type)
  File "/home/jsirois/.pex/venvs/a6f267be531b3188bc0c77f171ec077cc764ed0d/5985ed09b49a653d6596b0e14d134c5456cf1a9f/lib/python3.10/site-packages/pip/_internal/utils/unpacking.py", line 265, in unpack_file
    tarfile.is_tarfile(filename) or
  File "/usr/lib/python3.10/tarfile.py", line 2517, in is_tarfile
    t = open(name)
  File "/usr/lib/python3.10/tarfile.py", line 1632, in open
    return func(name, "r", fileobj, **kwargs)
  File "/usr/lib/python3.10/tarfile.py", line 1698, in gzopen
    fileobj = GzipFile(name, mode + "b", compresslevel, fileobj)
  File "/usr/lib/python3.10/gzip.py", line 174, in __init__
    fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
FileNotFoundError: [Errno 2] No such file or directory: '/home/jsirois/.pex/pip_cache/.tmp/project2'
But, with newer Pip, all is ok:
Copy code
jsirois@Gill-Windows:~/support/pex/Shalabh-multi-poetry $ pex3 lock create project1/ --pip-version 23.0.1 --indent 2
{
  "allow_builds": true,
  "allow_prereleases": false,
  "allow_wheels": true,
  "build_isolation": true,
  "constraints": [],
  "locked_resolves": [
    {
      "locked_requirements": [
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "4ad3232f5e926d6718ec31cfc1fcadfde020920e278684144551c91769c7bc18",
              "url": "<https://files.pythonhosted.org/packages/71/4c/3db2b8021bd6f2f0ceb0e088d6b2d49147671f25832fb17970e9b583d742/certifi-2022.12.7-py3-none-any.whl>"
            }
          ],
          "project_name": "certifi",
          "requires_dists": [],
          "requires_python": ">=3.6",
          "version": "2022.12.7"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "75f2568b4189dda1c567339b48cba4ac7384accb9c2a7ed655cd86b04055c795",
              "url": "<https://files.pythonhosted.org/packages/cc/f6/21a66e524658bd1dd7b89ac9d1ee8f7823f2d9701a2fbc458ab9ede53c63/charset_normalizer-3.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl>"
            }
          ],
          "project_name": "charset-normalizer",
          "requires_dists": [],
          "requires_python": ">=3.7.0",
          "version": "3.1.0"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "90b77e79eaa3eba6de819a0c442c0b4ceefc341a7a2ab77d7562bf49f425c5c2",
              "url": "<https://files.pythonhosted.org/packages/fc/34/3030de6f1370931b9dbb4dad48f6ab1015ab1d32447850b9fc94e60097be/idna-3.4-py3-none-any.whl>"
            }
          ],
          "project_name": "idna",
          "requires_dists": [],
          "requires_python": ">=3.5",
          "version": "3.4"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "c0defc642b12234f9964bae3f980f2bcd015d4fa3a758d8eb3cbbd7a18674bda",
              "url": "file:///home/jsirois/support/pex/Shalabh-multi-poetry/project1"
            }
          ],
          "project_name": "project1",
          "requires_dists": [
            "project2@ file:///home/jsirois/support/pex/Shalabh-multi-poetry/project2",
            "requests<3.0.0,>=2.28.2"
          ],
          "requires_python": "<4.0,>=3.9",
          "version": "0.1.0"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "5e2324fe9f20d69fb63b2d57bc29ac4baef9a936cdde24cebc272d807ff22598",
              "url": "file:///home/jsirois/support/pex/Shalabh-multi-poetry/project2"
            }
          ],
          "project_name": "project2",
          "requires_dists": [],
          "requires_python": "<4.0,>=3.9",
          "version": "0.1.0"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "64299f4909223da747622c030b781c0d7811e359c37124b4bd368fb8c6518baa",
              "url": "<https://files.pythonhosted.org/packages/d2/f4/274d1dbe96b41cf4e0efb70cbced278ffd61b5c7bb70338b62af94ccb25b/requests-2.28.2-py3-none-any.whl>"
            }
          ],
          "project_name": "requests",
          "requires_dists": [
            "PySocks!=1.5.7,>=1.5.6; extra == \"socks\"",
            "certifi>=2017.4.17",
            "chardet<6,>=3.0.2; extra == \"use_chardet_on_py3\"",
            "charset-normalizer<4,>=2",
            "idna<4,>=2.5",
            "urllib3<1.27,>=1.21.1"
          ],
          "requires_python": "<4,>=3.7",
          "version": "2.28.2"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "aa751d169e23c7479ce47a0cb0da579e3ede798f994f5816a74e4f4500dcea42",
              "url": "<https://files.pythonhosted.org/packages/7b/f5/890a0baca17a61c1f92f72b81d3c31523c99bec609e60c292ea55b387ae8/urllib3-1.26.15-py2.py3-none-any.whl>"
            }
          ],
          "project_name": "urllib3",
          "requires_dists": [
            "PySocks!=1.5.7,<2.0,>=1.5.6; extra == \"socks\"",
            "brotli>=1.0.9; ((os_name != \"nt\" or python_version >= \"3\") and platform_python_implementation == \"CPython\") and extra == \"brotli\"",
            "brotlicffi>=0.8.0; ((os_name != \"nt\" or python_version >= \"3\") and platform_python_implementation != \"CPython\") and extra == \"brotli\"",
            "brotlipy>=0.6.0; (os_name == \"nt\" and python_version < \"3\") and extra == \"brotli\"",
            "certifi; extra == \"secure\"",
            "cryptography>=1.3.4; extra == \"secure\"",
            "idna>=2.0.0; extra == \"secure\"",
            "ipaddress; python_version == \"2.7\" and extra == \"secure\"",
            "pyOpenSSL>=0.14; extra == \"secure\"",
            "urllib3-secure-extra; extra == \"secure\""
          ],
          "requires_python": "!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7",
          "version": "1.26.15"
        }
      ],
      "platform_tag": [
        "cp310",
        "cp310",
        "manylinux_2_35_x86_64"
      ]
    }
  ],
  "path_mappings": {},
  "pex_version": "2.1.131",
  "pip_version": "23.0.1",
  "prefer_older_binary": false,
  "requirements": [
    "project1==0.1"
  ],
  "requires_python": [],
  "resolver_version": "pip-legacy-resolver",
  "style": "strict",
  "target_systems": [],
  "transitive": true,
  "use_pep517": null
}
The moral is using both 
--pip-version
and 
--resolver-version
to move forward the defaults is probably a good idea. I leave the defaults what they are only to not break people. Pex expects you to explore the CLI help - for now - and use new features, which will all be off by default per the no breaking rule I abide by.
a
ah indeed the latest pip fixes it. would be nice to have 
--pip-version=latest
so we don't need to pin it using the cmdline unless we want to. if we pin pex, it will already pin the pip version based on whatever it supports.
e
You live more dangerously than I!
That sounds reasonable though if you want to file a feature request for it.
Just so its clear though, latest != latest. It's latest supported by Pex.
a
lol. the way i look at it, we already pin pex and carefully upgrade so we're not going to just end up on a newer pip version. however if we do upgrade pex using our pin, i don't also want to go to our code and change the 
--pip-version
string just to match.
e
I don't float high, I test each release of Pip and add it explicitly.
Yeah - ok, gotcha. Sounds reasonable.
Its the pair to "vendored", which is "oldest".
a
could be just one flag that sets both - the latest version of pip and the resolver
but at least the resolver flag value doesn't change that frequently so i'm less concerned about that.
e
The 
--pip-version latest
is fine. The resolver version will go away at some point on Pip's behalf - pretty sure pip-legacy-resolver is deprecated.
a
btw how safe is it to hand modify the lock json? specifically i'm thinking about partitioning it in two groups - local dependencies and published (pypi) dependencies
e
Yer gonna die.
😆 1
Probably fine - you know what you're doing, but obviously totally unsupported.
a
just remembered there was a way to specify a subset of the deps. will try if that works for me first: https://pantsbuild.slack.com/archives/C046T6T9U/p1670955226150879?thread_ts=1670874188.108299&amp;cid=C046T6T9U
e
You probably need to throw in --intransitive, etc.
Or --ignore-errors? When you muck with a resolve Pex is looking for transitive closure.
a
got it. so pex will still try to ensure it is a 'consistent' subset. I assume it is just reading the graph from the lock json and not trying to compute the graph from the wheels or sources.
e
That's right (at buildtime). At runtime, Pex computes the transitive closure to load from the wheel METADATA files in the PEX. At some point I'll probably embed the lock file as a standard part of a built PEX whether or not you used a 
--lock
, but the speed of computing the closure via lock file or METADATA files in a zip is pretty close as it stands. It would just be a whole code path eliminated, so really a maintenance win for me, no real win for the user.
@refined-addition-53644 was nice enough to upgrade Pex to Pip latest (23.1) this past weekend; so I added support for 
--pip-version latest
here: https://github.com/pantsbuild/pex/pull/2116 Those will both go out in Pex 2.1.132 release today.
🚀 2
r
How would it work inside pants? Is it going to use latest?
e
You control the future! Perhaps you can do the Pex upgrade PR in Pants and assert an opinion there and see how reviewers react.
I mean, you don't control the future - physics is a thing - but you know what I mean.
r
I did speed it up because I wanted to use it as early as possible inside pants too.
e
Right, gotcha. So there are 2 things here. One is Pants own use - which is only to generate default lockfiles it ships (see: https://github.com/pantsbuild/pants/pull/18751) and then the default version for the user-facing option in Pants. The former is probably uncontroversial to change to latest, the latter will potentially break users so would at least require a deprecation cycle.
r
I think this should be possible by passing custom 
pip_version
, support for which has been added by Benjy in 2.16. https://github.com/pantsbuild/pants/pull/18751
e
Yes, it definitely is.
I thought you wanted to change the default version of that value. If not, super simple.
✅ 1
r
No, no! Just wanted to use 23.1 as the 
pip_version
.
2 Views
Open in Slack
PreviousNext
Powered by