https://pantsbuild.org/ logo
Join Slack
Powered by
We have a dependency (`seqeval` ) that seems to ha...
# general
a
We have a dependency (
seqeval
) that seems to have broken metadata. I've been looking into it, it seems it has new lines in the long description and that breaks sometimes. To test it, I forked the repo, removed that from 
setup.py
and now 
importlib.metadata.metdata
doesn't complain anymore, so I thought, great success. After finding out you cannot put a git repo in 
constraints.txt
, I just added it to the 
requirements.txt
(even though none of our stuff depends directly on it), and it seems that made it work, but I still get this when I try to generate lockfiles:
Copy code
No distribution metadata found for seqeval==0.1.dev178+gedef455.
How can I get the original exception/error, or at lesat the command that caused that?
Okay, that seems to come from 
pex lock create
, now let's see if --level=DEBUG gives more info.
It does, hm.
pex
is very badly documented 😞
👆 2
Either that, or this isn't actually running pex.
Copy code
22:07:02.03 [DEBUG] spawned local process as Some(70601) for Process { argv: ["/Users/cbirzan/Library/Caches/nce/6faa4322d1df41d032e4938795c6f2c262ab92bb642a9bac1101cb7d1631f9c1/bindings/venvs/2.20.0/bin/python3.9", "./pex", "lock", "create", "--tmpdir", ".tmp", "--no-emit-warnings", "--python-path", "/Users/cbirzan/.pyenv/versions/crpython310/bin:/Users/cbirzan/.pyenv/versions/3.10.14/bin:/Users/cbirzan/.pyenv/versions/crpython310all/bin:/Users/cbirzan/.pyenv/versions/test_seqval/bin", "--output=lock.json", "--style=universal", "--pip-version", "23.1.2", "--resolver-version", "pip-2020-resolver", "--target-system", "linux", "--target-system", "mac", "--indent=2", "--no-pypi", "--index=<https://pypi.org/simple/>", "--manylinux", "manylinux2014", "--interpreter-constraint", "CPython~=3.10.9", "connexion[flask,uvicorn]==3.0.6"], env: {"CPPFLAGS": "", "LC_CTYPE": "UTF-8", "LDFLAGS": "", "PATH": "/opt/homebrew/Cellar/pyenv-virtualenv/1.2.3/shims:/Users/cbirzan/.pyenv/shims:/Users/cbirzan/.pyenv/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Users/cbirzan/.local/bin", "PEX_IGNORE_RCFILES": "true", "PEX_PYTHON": "/Users/cbirzan/Library/Caches/nce/6faa4322d1df41d032e4938795c6f2c262ab92bb642a9bac1101cb7d1631f9c1/bindings/venvs/2.20.0/bin/python3.9", "PEX_ROOT": ".cache/pex_root", "PEX_SCRIPT": "pex3"}, working_directory: None, input_digests: InputDigests { complete: DirectoryDigest { digest: Digest { hash: Fingerprint<11dc55a39ac097c64f6ca402aaa09c1948ccf3822c0d81768cfe72473f6b0fc0>, size_bytes: 158 }, tree: "Some(..)" }, nailgun: DirectoryDigest { digest: Digest { hash: Fingerprint<e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855>, size_bytes: 0 }, tree: "Some(..)" }, inputs: DirectoryDigest { digest: Digest { hash: Fingerprint<11dc55a39ac097c64f6ca402aaa09c1948ccf3822c0d81768cfe72473f6b0fc0>, size_bytes: 158 }, tree: "Some(..)" }, immutable_inputs: {}, use_nailgun: {} }, output_files: {RelativePath("lock.json")}, output_directories: {}, timeout: None, execution_slot_variable: None, concurrency_available: 0, description: "Generate lockfile for connexion_3", level: Info, append_only_caches: {CacheName("pex_root"): RelativePath(".cache/pex_root"), CacheName("python_build_standalone"): RelativePath(".python-build-standalone")}, jdk_home: None, cache_scope: PerSession, execution_environment: ProcessExecutionEnvironment { name: None, platform: Macos_arm64, strategy: Local }, remote_cache_speculation_delay: 0ns, attempt: 0 }
As far as I can tell, that command is run in a sandbox, but the pex there doesn't take 
lock create
as arguments...
c
From the sandbox, you can add PEX_VERBOSE=9 to the __run.sh script to get a lot more output from pex..
a
Oh, that's nice. Thanks. I cheated and used a smaller lockfile, guess I'll have to wait for the monster one to fail 🙂
👍 1
We constantly struggle with these issues from pex (not because of a bug in pex, but just because of crazy number of deps and how terrible we are at keeping them up to date), maybe that should be exposed as a pants option
c
not a bad idea, as it's usually the first knob to try when troubleshooting lockfile issues..
a
Okay, it turns out that it was breaking because of 
use_scm_version
.
🫠 2
s
@ancient-france-42909 Did you have any luck?
a
With what, in particular?
s
With a workaround for Seqeval, or debugging pex. I'm interested in opening a PR against one of those two to try and fix this
a
Ah, so, I have a fork that 'fixes' this issue, let me find it.
👀 1
s
Is it just hardcoding the 
version
in setup.py?
a
The version hardcoding was only needed when installing from a git repo
The part that fixed the first issue was the long_description.
I didn't look into it too much, I don't hate myself enough to touch anything setuptools related.
💯 1
s
Fair enough. Thank you!
a
though, we used this only because it was a dependency of another lib
GLiNER
, and if that happens to be the case for you too, a colleague did this: https://github.com/urchade/GLiNER/pull/141
s
Its coming via a different tool for us, but I'll see if the same could be be applied. Great shout
a
I actually didn't take the time to report it in the original repo, I just complained to the data science person that added that dep and he said "oh, f it, they're using one function"
👍 1
s
Directly invoking 
pex
(with all the same args printed by pants during 
generate-lockfiles
), prints a warning for seqeval, but succeeds:
Copy code
/Users/jasondamour/.cache/pants/named_caches/pex_root/venvs/0deb89b23adbbad7a5b9aac06d2362acf7572cfa/4c982503433997e579e4274163218f2997dc553e/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -v --cache-dir /Users/jasondamour/.cache/pants/named_caches/pex_root/pip/24.0/pip_cache --log /private/var/folders/_9/qvrc0kmj4ssfvbybnz4_wb7c0000gn/T/pants-sandbox-VRP8g7/.tmp/pex-pip-log.nlujgual/pip.log download --dest /private/var/folders/_9/qvrc0kmj4ssfvbybnz4_wb7c0000gn/T/pants-sandbox-VRP8g7/.tmp/tmp1ak1of34/Users.jasondamour..pyenv.versions.3.10.6.bin.python3.10 --index-url <https://pypi.org/simple/> --retries 5 --timeout 15 seqeval==1.2.2
Copy code
Looking in indexes: <https://pypi.org/simple/>
Collecting seqeval==1.2.2
  File was already downloaded /private/var/folders/_9/qvrc0kmj4ssfvbybnz4_wb7c0000gn/T/pants-sandbox-VRP8g7/.tmp/tmp1ak1of34/Users.jasondamour..pyenv.versions.3.10.6.bin.python3.10/seqeval-1.2.2.tar.gz
  Running command python setup.py egg_info
  /Users/jasondamour/.cache/pants/named_caches/pex_root/venvs/s/9983c135/venv/lib/python3.10/site-packages/setuptools/__init__.py:84: _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated.
  !!

          ********************************************************************************
          Requirements should be satisfied by a PEP 517 installer.
          If you are using pip, you can try `pip install --use-pep517`.
          ********************************************************************************

  !!
    dist.fetch_build_eggs(dist.setup_requires)
  WARNING setuptools_scm.pyproject_reading toml section missing 'pyproject.toml does not contain a tool.setuptools_scm section'
  Traceback (most recent call last):
    File "/private/var/folders/_9/qvrc0kmj4ssfvbybnz4_wb7c0000gn/T/pip-download-_ylel21t/seqeval_a3c72d2156ca4fee8ca09410a5d0461b/.eggs/setuptools_scm-8.1.0-py3.10.egg/setuptools_scm/_integration/pyproject_reading.py", line 36, in read_pyproject
      section = defn.get("tool", {})[tool_name]
  KeyError: 'setuptools_scm'
  running egg_info
  ERROR setuptools_scm._file_finders.git listing git files failed - pretending there aren't any
Successfully downloaded seqeval numpy scikit-learn joblib scipy threadpoolctl
a
That's not the command that generates the lockfiles
I can't easily check, but hm, you're either missing some arguments, or that's not it
Nah, that's definitely not it.
s
Aw... I ran 
pants generate-lockfiles --pex-verbosity=9
and it logs 
Executing: ....
and then the command I directly invoked
a
What version of pants?
s
Pants 2.21, which uses pex 2.3.3
a
If you scroll up, you'll see what I got with 2.21 as well.
If nothing else, it should have 
"./pex", "lock", "create",
somewhere in there
I think this doesn't use 
__run.sh
, but it's been more than 5 minutes since I did this, so I can't really be sure
Also, sorry, I need to go have a long cold shower, I just spent over an hour outside, I think it melted my brain, heh
s
Huh, doesn't appear at all. Maybe mine is failing before it even gets there? Attached the raw logs (minimally redacted)
pants-gen-lockfiles.log
Ah no worries. Thanks for even bouncing ideas
a
I knew there was a reason I didn't go out the past few days, except being sick, 4 days in a row of 38C, the sidewalk is hot even in the shade, heh.
Anyway, will take a look in an hour or so
Only 30 minutes, but I remember now that I was wrong.
Do this, run the same command with 
--keep-sandboxes=always
and then go and edit the 
__run.sh
like Andreas suggested, adding 
PEX_VERBOSE=9
to the env vars, run that and paste the output
18 Views
Open in Slack
PreviousNext
Powered by