https://pantsbuild.org/ logo
#general
Title
# general
a

abundant-autumn-67998

04/13/2023, 6:00 PM
Using
pex3 lock create .
fails in a folder with only
pyproject.toml
when it includes editable dependencies, eg:
Copy code
[tool.poetry.dependencies]
python = "^3.9"
requests = "^2.28.2"
project2 = {path = "../project2", develop = true}
Without the
project2
line the lock works. However with that line this fails with
FileNotFoundError: [Errno 2] No such file or directory: '/Users/shalabh/.pex/pip_cache/.tmp/project2'
. is there an easy way to exclude these path based dependencies? Alternatively, what's the best way to create a dependencies pex from such a project. I'm thinking if we export this into a
requirements.txt
and then feed that to pex - that might be a better option?
ideally pex3 lock would handle local dependencies as well.
r

refined-addition-53644

04/13/2023, 6:46 PM
So it looks like pex lock does support local dependencies. It’s explained here https://www.pantsbuild.org/docs/python-third-party-dependencies#local-requirements
e

enough-analyst-54434

04/13/2023, 7:22 PM
@abundant-autumn-67998 works fine with modern pip. Use
--pip-version 23.0.1
.
Given:
Copy code
jsirois@Gill-Windows:~/support/pex/Shalabh-multi-poetry $ tree project1 project2
project1
├── project1.py
├── pyproject.toml
└── README.md
project2
├── project2.py
├── pyproject.toml
└── README.md

0 directories, 3 files
jsirois@Gill-Windows:~/support/pex/Shalabh-multi-poetry $ cat project1/pyproject.toml
[tool.poetry]
name = "project1"
version = "0.1.0"
description = ""
authors = ["John Sirois <john.sirois@gmail.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.9"
requests = "^2.28.2"
project2 = {path = "../project2", develop = true}


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
jsirois@Gill-Windows:~/support/pex/Shalabh-multi-poetry $ cat project2/pyproject.toml
[tool.poetry]
name = "project2"
version = "0.1.0"
description = ""
authors = ["John Sirois <john.sirois@gmail.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.9"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
I find, as you say:
Copy code
jsirois@Gill-Windows:~/support/pex/Shalabh-multi-poetry $ pex3 lock create project1/
pid 1949 -> /home/jsirois/.pex/venvs/a6f267be531b3188bc0c77f171ec077cc764ed0d/5985ed09b49a653d6596b0e14d134c5456cf1a9f/bin/python -sE /home/jsirois/.pex/venvs/a6f267be531b3188bc0c77f171ec077cc764ed0d/5985ed09b49a653d6596b0e14d134c5456cf1a9f/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --use-deprecated legacy-resolver --isolated -q --cache-dir /home/jsirois/.pex/pip_cache --log /tmp/pex-pip-log.tv17i8qe/pip.log download --dest /tmp/tmpcjko2ckl/home.jsirois.bin.pex.venv.bin.python project1/ --index-url <https://pypi.org/simple> --retries 5 --timeout 15 exited with 2 and STDERR:
ERROR: Exception:
Traceback (most recent call last):
  File "/home/jsirois/.pex/venvs/a6f267be531b3188bc0c77f171ec077cc764ed0d/5985ed09b49a653d6596b0e14d134c5456cf1a9f/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 223, in _main
    status = self.run(options, args)
  File "/home/jsirois/.pex/venvs/a6f267be531b3188bc0c77f171ec077cc764ed0d/5985ed09b49a653d6596b0e14d134c5456cf1a9f/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 180, in wrapper
    return func(self, options, args)
  File "/home/jsirois/.pex/venvs/a6f267be531b3188bc0c77f171ec077cc764ed0d/5985ed09b49a653d6596b0e14d134c5456cf1a9f/lib/python3.10/site-packages/pip/_internal/commands/download.py", line 130, in run
    requirement_set = resolver.resolve(
  File "/home/jsirois/.pex/venvs/a6f267be531b3188bc0c77f171ec077cc764ed0d/5985ed09b49a653d6596b0e14d134c5456cf1a9f/lib/python3.10/site-packages/pip/_internal/resolution/legacy/resolver.py", line 180, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
  File "/home/jsirois/.pex/venvs/a6f267be531b3188bc0c77f171ec077cc764ed0d/5985ed09b49a653d6596b0e14d134c5456cf1a9f/lib/python3.10/site-packages/pip/_internal/resolution/legacy/resolver.py", line 385, in _resolve_one
    dist = self._get_dist_for(req_to_install)
  File "/home/jsirois/.pex/venvs/a6f267be531b3188bc0c77f171ec077cc764ed0d/5985ed09b49a653d6596b0e14d134c5456cf1a9f/lib/python3.10/site-packages/pip/_internal/resolution/legacy/resolver.py", line 337, in _get_dist_for
    dist = self.preparer.prepare_linked_requirement(req)
  File "/home/jsirois/.pex/venvs/a6f267be531b3188bc0c77f171ec077cc764ed0d/5985ed09b49a653d6596b0e14d134c5456cf1a9f/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 480, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
  File "/home/jsirois/.pex/venvs/a6f267be531b3188bc0c77f171ec077cc764ed0d/5985ed09b49a653d6596b0e14d134c5456cf1a9f/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 503, in _prepare_linked_requirement
    local_file = unpack_url(
  File "/home/jsirois/.pex/venvs/a6f267be531b3188bc0c77f171ec077cc764ed0d/5985ed09b49a653d6596b0e14d134c5456cf1a9f/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 263, in unpack_url
    unpack_file(file.path, location, file.content_type)
  File "/home/jsirois/.pex/venvs/a6f267be531b3188bc0c77f171ec077cc764ed0d/5985ed09b49a653d6596b0e14d134c5456cf1a9f/lib/python3.10/site-packages/pip/_internal/utils/unpacking.py", line 265, in unpack_file
    tarfile.is_tarfile(filename) or
  File "/usr/lib/python3.10/tarfile.py", line 2517, in is_tarfile
    t = open(name)
  File "/usr/lib/python3.10/tarfile.py", line 1632, in open
    return func(name, "r", fileobj, **kwargs)
  File "/usr/lib/python3.10/tarfile.py", line 1698, in gzopen
    fileobj = GzipFile(name, mode + "b", compresslevel, fileobj)
  File "/usr/lib/python3.10/gzip.py", line 174, in __init__
    fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
FileNotFoundError: [Errno 2] No such file or directory: '/home/jsirois/.pex/pip_cache/.tmp/project2'
But, with newer Pip, all is ok:
Copy code
jsirois@Gill-Windows:~/support/pex/Shalabh-multi-poetry $ pex3 lock create project1/ --pip-version 23.0.1 --indent 2
{
  "allow_builds": true,
  "allow_prereleases": false,
  "allow_wheels": true,
  "build_isolation": true,
  "constraints": [],
  "locked_resolves": [
    {
      "locked_requirements": [
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "4ad3232f5e926d6718ec31cfc1fcadfde020920e278684144551c91769c7bc18",
              "url": "<https://files.pythonhosted.org/packages/71/4c/3db2b8021bd6f2f0ceb0e088d6b2d49147671f25832fb17970e9b583d742/certifi-2022.12.7-py3-none-any.whl>"
            }
          ],
          "project_name": "certifi",
          "requires_dists": [],
          "requires_python": ">=3.6",
          "version": "2022.12.7"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "75f2568b4189dda1c567339b48cba4ac7384accb9c2a7ed655cd86b04055c795",
              "url": "<https://files.pythonhosted.org/packages/cc/f6/21a66e524658bd1dd7b89ac9d1ee8f7823f2d9701a2fbc458ab9ede53c63/charset_normalizer-3.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl>"
            }
          ],
          "project_name": "charset-normalizer",
          "requires_dists": [],
          "requires_python": ">=3.7.0",
          "version": "3.1.0"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "90b77e79eaa3eba6de819a0c442c0b4ceefc341a7a2ab77d7562bf49f425c5c2",
              "url": "<https://files.pythonhosted.org/packages/fc/34/3030de6f1370931b9dbb4dad48f6ab1015ab1d32447850b9fc94e60097be/idna-3.4-py3-none-any.whl>"
            }
          ],
          "project_name": "idna",
          "requires_dists": [],
          "requires_python": ">=3.5",
          "version": "3.4"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "c0defc642b12234f9964bae3f980f2bcd015d4fa3a758d8eb3cbbd7a18674bda",
              "url": "file:///home/jsirois/support/pex/Shalabh-multi-poetry/project1"
            }
          ],
          "project_name": "project1",
          "requires_dists": [
            "project2@ file:///home/jsirois/support/pex/Shalabh-multi-poetry/project2",
            "requests<3.0.0,>=2.28.2"
          ],
          "requires_python": "<4.0,>=3.9",
          "version": "0.1.0"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "5e2324fe9f20d69fb63b2d57bc29ac4baef9a936cdde24cebc272d807ff22598",
              "url": "file:///home/jsirois/support/pex/Shalabh-multi-poetry/project2"
            }
          ],
          "project_name": "project2",
          "requires_dists": [],
          "requires_python": "<4.0,>=3.9",
          "version": "0.1.0"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "64299f4909223da747622c030b781c0d7811e359c37124b4bd368fb8c6518baa",
              "url": "<https://files.pythonhosted.org/packages/d2/f4/274d1dbe96b41cf4e0efb70cbced278ffd61b5c7bb70338b62af94ccb25b/requests-2.28.2-py3-none-any.whl>"
            }
          ],
          "project_name": "requests",
          "requires_dists": [
            "PySocks!=1.5.7,>=1.5.6; extra == \"socks\"",
            "certifi>=2017.4.17",
            "chardet<6,>=3.0.2; extra == \"use_chardet_on_py3\"",
            "charset-normalizer<4,>=2",
            "idna<4,>=2.5",
            "urllib3<1.27,>=1.21.1"
          ],
          "requires_python": "<4,>=3.7",
          "version": "2.28.2"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "aa751d169e23c7479ce47a0cb0da579e3ede798f994f5816a74e4f4500dcea42",
              "url": "<https://files.pythonhosted.org/packages/7b/f5/890a0baca17a61c1f92f72b81d3c31523c99bec609e60c292ea55b387ae8/urllib3-1.26.15-py2.py3-none-any.whl>"
            }
          ],
          "project_name": "urllib3",
          "requires_dists": [
            "PySocks!=1.5.7,<2.0,>=1.5.6; extra == \"socks\"",
            "brotli>=1.0.9; ((os_name != \"nt\" or python_version >= \"3\") and platform_python_implementation == \"CPython\") and extra == \"brotli\"",
            "brotlicffi>=0.8.0; ((os_name != \"nt\" or python_version >= \"3\") and platform_python_implementation != \"CPython\") and extra == \"brotli\"",
            "brotlipy>=0.6.0; (os_name == \"nt\" and python_version < \"3\") and extra == \"brotli\"",
            "certifi; extra == \"secure\"",
            "cryptography>=1.3.4; extra == \"secure\"",
            "idna>=2.0.0; extra == \"secure\"",
            "ipaddress; python_version == \"2.7\" and extra == \"secure\"",
            "pyOpenSSL>=0.14; extra == \"secure\"",
            "urllib3-secure-extra; extra == \"secure\""
          ],
          "requires_python": "!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7",
          "version": "1.26.15"
        }
      ],
      "platform_tag": [
        "cp310",
        "cp310",
        "manylinux_2_35_x86_64"
      ]
    }
  ],
  "path_mappings": {},
  "pex_version": "2.1.131",
  "pip_version": "23.0.1",
  "prefer_older_binary": false,
  "requirements": [
    "project1==0.1"
  ],
  "requires_python": [],
  "resolver_version": "pip-legacy-resolver",
  "style": "strict",
  "target_systems": [],
  "transitive": true,
  "use_pep517": null
}
The moral is using both
--pip-version
and
--resolver-version
to move forward the defaults is probably a good idea. I leave the defaults what they are only to not break people. Pex expects you to explore the CLI help - for now - and use new features, which will all be off by default per the no breaking rule I abide by.
a

abundant-autumn-67998

04/13/2023, 8:23 PM
ah indeed the latest pip fixes it. would be nice to have
--pip-version=latest
so we don't need to pin it using the cmdline unless we want to. if we pin pex, it will already pin the pip version based on whatever it supports.
e

enough-analyst-54434

04/13/2023, 8:28 PM
You live more dangerously than I!
That sounds reasonable though if you want to file a feature request for it.
Just so its clear though, latest != latest. It's latest supported by Pex.
a

abundant-autumn-67998

04/13/2023, 8:29 PM
lol. the way i look at it, we already pin pex and carefully upgrade so we're not going to just end up on a newer pip version. however if we do upgrade pex using our pin, i don't also want to go to our code and change the
--pip-version
string just to match.
e

enough-analyst-54434

04/13/2023, 8:29 PM
I don't float high, I test each release of Pip and add it explicitly.
Yeah - ok, gotcha. Sounds reasonable.
Its the pair to "vendored", which is "oldest".
a

abundant-autumn-67998

04/13/2023, 8:30 PM
could be just one flag that sets both - the latest version of pip and the resolver
but at least the resolver flag value doesn't change that frequently so i'm less concerned about that.
e

enough-analyst-54434

04/13/2023, 8:32 PM
The
--pip-version latest
is fine. The resolver version will go away at some point on Pip's behalf - pretty sure pip-legacy-resolver is deprecated.
a

abundant-autumn-67998

04/13/2023, 8:33 PM
btw how safe is it to hand modify the lock json? specifically i'm thinking about partitioning it in two groups - local dependencies and published (pypi) dependencies
e

enough-analyst-54434

04/13/2023, 8:34 PM
Yer gonna die.
😆 1
Probably fine - you know what you're doing, but obviously totally unsupported.
a

abundant-autumn-67998

04/13/2023, 8:38 PM
just remembered there was a way to specify a subset of the deps. will try if that works for me first: https://pantsbuild.slack.com/archives/C046T6T9U/p1670955226150879?thread_ts=1670874188.108299&amp;cid=C046T6T9U
e

enough-analyst-54434

04/13/2023, 8:40 PM
You probably need to throw in --intransitive, etc.
Or --ignore-errors? When you muck with a resolve Pex is looking for transitive closure.
a

abundant-autumn-67998

04/13/2023, 8:44 PM
got it. so pex will still try to ensure it is a 'consistent' subset. I assume it is just reading the graph from the lock json and not trying to compute the graph from the wheels or sources.
e

enough-analyst-54434

04/13/2023, 8:44 PM
That's right (at buildtime). At runtime, Pex computes the transitive closure to load from the wheel METADATA files in the PEX. At some point I'll probably embed the lock file as a standard part of a built PEX whether or not you used a
--lock
, but the speed of computing the closure via lock file or METADATA files in a zip is pretty close as it stands. It would just be a whole code path eliminated, so really a maintenance win for me, no real win for the user.
@refined-addition-53644 was nice enough to upgrade Pex to Pip latest (23.1) this past weekend; so I added support for
--pip-version latest
here: https://github.com/pantsbuild/pex/pull/2116 Those will both go out in Pex 2.1.132 release today.
🚀 2
r

refined-addition-53644

04/17/2023, 4:05 PM
How would it work inside pants? Is it going to use latest?
e

enough-analyst-54434

04/17/2023, 4:06 PM
You control the future! Perhaps you can do the Pex upgrade PR in Pants and assert an opinion there and see how reviewers react.
I mean, you don't control the future - physics is a thing - but you know what I mean.
r

refined-addition-53644

04/17/2023, 4:07 PM
I did speed it up because I wanted to use it as early as possible inside pants too.
e

enough-analyst-54434

04/17/2023, 4:11 PM
Right, gotcha. So there are 2 things here. One is Pants own use - which is only to generate default lockfiles it ships (see: https://github.com/pantsbuild/pants/pull/18751) and then the default version for the user-facing option in Pants. The former is probably uncontroversial to change to latest, the latter will potentially break users so would at least require a deprecation cycle.
r

refined-addition-53644

04/17/2023, 4:27 PM
I think this should be possible by passing custom
pip_version
, support for which has been added by Benjy in 2.16. https://github.com/pantsbuild/pants/pull/18751
e

enough-analyst-54434

04/17/2023, 4:28 PM
Yes, it definitely is.
I thought you wanted to change the default version of that value. If not, super simple.
1
r

refined-addition-53644

04/17/2023, 4:29 PM
No, no! Just wanted to use 23.1 as the
pip_version
.
2 Views