https://pantsbuild.org/ logo
Join Slack
Powered by
Having some trouble today using a combination of m...
# general
d
Having some trouble today using a combination of manually generating lockfiles and 
install_from_resolves
with 
pytest
Following https://www.pantsbuild.org/stable/docs/python/overview/lockfiles#manually-generating-lockfiles I’ve added the following to my 
pants.toml
Copy code
[python]
interpreter_constraints = ['==3.12.*']
enable_resolves = true
resolves_generate_lockfiles = false
run_against_entire_lockfile = true

[python.resolves]
python-default = "requirements.txt"

[pytest]
install_from_resolve = "python-default"
requirements = [
  "//:reqs0#pytest-mock",
]
Now running 
pants test ::
throws an exception
Copy code
➜  pants-test pants test ::
14:30:05.52 [ERROR] 1 Exception encountered:

Engine traceback:
  in `test` goal

AssertionError:
If 
install_from_resolve
isn’t specified things work as before but I’m forced to use pytest 
7.0.1
instead of the one specified in my generated lockfile (not what I want). I’ve attached the debug logs below:
logs.txt
e
You'll also need to add 
pytest
itself to the 
pytest.requirements
in your pants.toml.
Copy code
[pytest]
install_from_Resolve = "python-default"
requirements = [
  "//:reqs0#pytest",
  "//:reqs0#pytest-mock",
]
d
have tried this, I still get the same error
d
python-default = "requirements.txt"
- this should be pointing to a lock file, not to the requirements.txt file.
e
Try changing your 
python-default = "requirements.txt"
Just guessing, but it be that there's some confusion where pants is trying to interpret your requirements.txt file as alock file
d
According to https://www.pantsbuild.org/stable/docs/python/overview/lockfiles#manually-generating-lockfiles > Manually generated lockfiles must either use Pex’s JSON format or use pip’s 
requirements.txt
-style format (ideally with 
--hash
entries for better supply chain security). For example: My 
requirements.txt
has entries like:
Copy code
aenum==3.1.15 ; sys_platform == 'darwin' or sys_platform == 'linux' \
    --hash=sha256:8cbd76cd18c4f870ff39b24284d3ea028fbe8731a58df3aa581e434c575b9559 \
    --hash=sha256:e0dfaeea4c2bd362144b87377e2c61d91958c5ed0b4daf89cb6f45ae23af6288
Which is what they have in the docs So I believe I’m doing the right thing here…
e
resolves_generate_lockfiles = false, run_against_entire_lockfile = true Are these incompatible? Seems odd to run against lockfile if the lockfile is not generated
d
The way I’d interpreted this combination of flags is to tell pants, “hey I’m providing my own lockfile”, so I there’s no need to run 
pants generate-lockfiles ::
d
are 
pytest
and 
pytest-mock
in the output if you do 
pants list //:reqs0
?
d
Copy code
➜  pants-test pants dependencies //:reqs0 | grep pytest
//:reqs0#pytest
//:reqs0#pytest-cov
//:reqs0#pytest-freezer
//:reqs0#pytest-mock
//:reqs0#pytest-postgresql
the list command just returns itself
Copy code
➜  pants-test pants list //:reqs0
//:reqs0
e
what does your build file containing //:reqs0 look like?
d
Copy code
python_sources(
    name="root",
)

python_requirements(
    name="reqs0",
)

python_tests(
    name="tests0",
)
d
ah, you probably need 
default_resolve = "python-default"
under 
[python]
in pants.toml
unless you're specifying the resolve some other way
d
that’s the default (if not specified), so also hasn’t fixed (just tried). Have just pushed a minimal reproducible example here https://github.com/Jackevansevo/pants-example
e
It might be a little work to try this, but you could try changing the 
requirements.txt
file to just a list of package names.
Copy code
aenum
aiofiles
etc.
If that works, it would suggest the issue is with the file format/parsing.
d
from the output you pasted, it looks like the pex was built and the test ran. So is a dependency issue a red herring?
👍 1
d
I just did a little
Copy code
uv venv
uv pip install -r requirements.txt
uv pip list | awk '{print $1}' > requirements.txt
So now it’s just a list of
Copy code
aenum
aiofiles
aiohappyeyeballs
aiohttp
And I still get the same
Copy code
➜  pants-test git:(main) ✗ pants test ::
14:56:00.57 [ERROR] 1 Exception encountered:

Engine traceback:
  in `test` goal

AssertionError:
So not a parsing issue
d
You could try 
--keep-sandboxes=on_failure
and see what's in there? Or 
--test-output=all
to see if you can get more details
d
Still getting identical output, doesn’t appear to keep around the sandbox
Copy code
pants --no-pantsd -ldebug --keep-sandboxes=on_failure test --test-output=all ::
Copy code
14:59:03.97 [DEBUG] Lockfile for resolve python-default at requirements.txt has no metadata block, so was not generated by Pants. Lockfile will not be validated.
14:59:04.97 [DEBUG] Completed: Find interpreter for constraints: CPython==3.12.*
14:59:04.97 [DEBUG] Completed: Scheduling: Find interpreter for constraints: CPython==3.12.*
14:59:04.97 [DEBUG] Completed: Find Python interpreter for constraints - CPython==3.12.* - Selected /opt/homebrew/Cellar/python@3.12/3.12.5/Frameworks/Python.framework/Versions/3.12/bin/python3.12 to run PEXes with.
14:59:04.98 [DEBUG] Completed: Resolve transitive targets
14:59:04.98 [DEBUG] Completed: Scheduling: Installing requirements.txt for the resolve `python-default`
14:59:04.99 [DEBUG] Completed: pants.backend.python.util_rules.pex.build_pex
14:59:05.03 [DEBUG] Completed: pants.backend.python.util_rules.python_sources.strip_python_sources
14:59:05.07 [DEBUG] Completed: pants.backend.python.util_rules.pex_from_targets.create_pex_from_targets
14:59:05.09 [DEBUG] Lockfile for resolve python-default at requirements.txt has no metadata block, so was not generated by Pants. Lockfile will not be validated.
14:59:05.09 [DEBUG] Completed: pants.backend.python.util_rules.pex.build_pex
14:59:05.09 [DEBUG] Completed: pants.backend.python.goals.pytest_runner.setup_pytest_for_target
14:59:05.09 [DEBUG] Canceled: pants.backend.python.util_rules.pex.build_pex
14:59:05.09 [DEBUG] Completed: Run Pytest - //test_thing.py:tests0
14:59:05.09 [DEBUG] Completed: `test` goal
14:59:05.09 [DEBUG] computed 1 nodes in 1.400057 seconds. there are 596 total nodes.
14:59:05.09 [ERROR] 1 Exception encountered:

Engine traceback:
  in `test` goal

AssertionError:
e
do 
--keep-sandboxes=always
d
it’s preserved a sandbox for me, but it doesn’t have anything to do with 
pytest
inside 
__run.sh
and succeeds when run
Copy code
15:06:17.03 [INFO] Preserving local process execution dir /private/var/folders/yn/4q7r_wwd16v279lryh84bgtc0000gq/T/pants-sandbox-wlNlZn for Find interpreter for constraints: CPython==3.12.*
15:06:17.03 [DEBUG] /Users/jack/.cache/pants/lmdb_store/immutable/files -> /private/var/folders/yn/4q7r_wwd16v279lryh84bgtc0000gq/T hardlinkable: true
15:06:17.03 [DEBUG] Completed: setup_sandbox
Copy code
➜  pants-test git:(main) bat /private/var/folders/yn/4q7r_wwd16v279lryh84bgtc0000gq/T/pants-sandbox-wlNlZn/__run.sh
#!/usr/bin/env bash
# This command line should execute the same process as pants did internally.
cd /private/var/folders/yn/4q7r_wwd16v279lryh84bgtc0000gq/T/pants-sandbox-wlNlZn
env -i CPPFLAGS= LANG=en_GB.UTF-8 LDFLAGS= PATH=$'/Users/jack/.rye/shims:/Users/jack/pnpm:/Users/jack/n/bin:/Users/jack/google-cloud-sdk/bin:/opt/homebrew/bin:/Users/jack/nvim-macos-arm64/bin:/Users/jack/.cargo/bin:/Users/jack/bin:/Users/jack/.bun/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Library/Apple/usr/bin:/Applications/iTerm.app/Contents/Resources/utilities:/Users/jack/.local/bin' PEX_IGNORE_RCFILES=true PEX_PYTHON=/Users/jack/Library/Caches/nce/6faa4322d1df41d032e4938795c6f2c262ab92bb642a9bac1101cb7d1631f9c1/bindings/venvs/2.22.0/bin/python3.9 PEX_ROOT=.cache/pex_root /Users/jack/Library/Caches/nce/6faa4322d1df41d032e4938795c6f2c262ab92bb642a9bac1101cb7d1631f9c1/bindings/venvs/2.22.0/bin/python3.9 ./pex --tmpdir .tmp --no-emit-warnings --pip-version 24.0 --python-path $'/Users/jack/.pyenv/versions/3.11.9/bin:/Users/jack/.pyenv/versions/3.12.2/bin:/Users/jack/.rye/shims:/Users/jack/pnpm:/Users/jack/n/bin:/Users/jack/google-cloud-sdk/bin:/opt/homebrew/bin:/Users/jack/nvim-macos-arm64/bin:/Users/jack/.cargo/bin:/Users/jack/bin:/Users/jack/.bun/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Library/Apple/usr/bin:/Applications/iTerm.app/Contents/Resources/utilities:/Users/jack/.local/bin' --interpreter-constraint $'CPython==3.12.*' -- -c $'import hashlib, os, sys\n\npython = os.path.realpath(sys.executable)\nprint(python)\n\nhasher = hashlib.sha256()\nwith open(python, "rb") as fp:\n  for chunk in iter(lambda: fp.read(8192), b""):\n      hasher.update(chunk)\nprint(hasher.hexdigest())\n'
d
there should be an output like 
Preserving local process execution dir... for Run Pytest for ...
- that's the dir to look at
d
the only match I can find is
Copy code
15:18:18.83 [INFO] Preserving local process execution dir /private/var/folders/yn/4q7r_wwd16v279lryh84bgtc0000gq/T/pants-sandbox-oPTa2w for Find interpreter for constraints: CPython==3.12.*
There isn’t one for 
pytest
I’m still running:
Copy code
pants --no-pantsd -ldebug --keep-sandboxes=always test --test-output=all ::
d
that's odd, I'd expect to see one for each of the items that says 
Completed:
in the debug output you pasted
4 Views
Open in Slack
PreviousNext
Powered by