Are the generated lockfiles (3rdparty/python/defau...
# general
e
Are the generated lockfiles (3rdparty/python/default.lock) meant to be under version control in the same way that poetry lockfiles are, and manually regenerated when stated requirements change? (sorry, trying to understand the, er, dependencies of dependency lists. We were using a constraints.txt style "lockfile" which was a pip freeze of a venv with everything installed (including transitive 3rd-party dependencies we don't directly mention anywhere). Looks like Pants can now generate that full lockfile (and we could only generate
python_requirements
for our direct dependencies) but I wasn't sure of that yet)
b
• In Version control? Yes! (I think most ecosystem's lockfiles are in version control) • manually regenerated when stated requirements change? Yes-ish. If you use Pants to generate them, Pants will put a metadata header at the top, and if your inputs change Pants will warn/error.
Oh, but to clarify, do you mean 2.11.x JSON lockfiles, or the pre-2.11 txt lockfiles?
e
I'm on 2.10 now but could change to 2.11. Good to hear about the warning system.
If I'm making these updates, may as well try 2.11 I guess šŸ™‚
b
The next RC should hopefully be šŸš€ soon. The TL;DR is set
lockfile_generator
to
pex
and when you run
generate-lockfile
you'll get a PEX JSON. the PEX JSON contains transitive package metadata, so when Pants asks PEX to make pexs, it can do so fully in parallel, instead of semi-serially.
But wait for the next RC, it has a huge speedup in it šŸ˜‰
(Well you can generate the lockfiles today, but consuming them is the thing that got fixed to be speedy)
e
Awesome; really looking forward to it!
šŸ™Œ 1
Just checking; setting lockfile_generator in pants.toml in 2.11.0.dev2 and it's an error ("Invalid option 'lockfile_generator' under [python]"). Do I need to wait until the next RC? Was thrown since the documentation mentions it...
h
use 2.11.0rc0 or later today rc1
e
Okay, hmm... I'm not sure what interaction is doing this or what I'm missing, but 2.11.0rc0, python interpreter_constraints = ["CPython==3.9.*"], system python is 3.10, pyenv enabled "local" python is 3.9.9, I'm in a virtualenv is 3.9.9, and ... ...lockfile generation consistently fails because it's using 3.10 (system python)...?
Copy code
stderr:
ERROR: Package 'aiofiles' requires a different Python: 3.10.2 not in '<4.0,>=3.6'
...neither of which makes sense to me (it shouldn't be resolving with 3.10, and 3.10.2 is in fact between 3.6 and 4.0...)
h
which lockfile is being generated?
e
python-default
hmm, it's clearly using the pyenv version (in the generated __run.sh I see "/home/vputz/.pyenv/versions/3.10.2/bin/python") but I've set pyenv local and global to 3.9.9 to try and force the issue. That's... odd
huh; yeah,
Copy code
Process 'Generate lockfile for python-default' failed with exit code 1.
and
Copy code
/tmp/process-executionKTQOsz for "Generate lockfile for python-default"
Copy code
/tmp/process-executionKTQOsz
āÆ grep python __run.sh
/home/vputz/.pyenv/versions/3.10.2/bin/python ./pex --tmpdir .tmp lock ...
h
e
Sigh... yeah, I don't get this at all.
interpreter_constraints
is
["CPython==3.9.*"]
,
resolves_to_interpreter_constraints="{'python-default': ['==3.9.*']}"
, and yet trying
Copy code
./pants --no-process-cleanup generate-lockfiles --generate-lockfiles-resolve="['python-default']"
...gets me that
ERROR: Package 'aiofiles' requires a different Python: 3.10.2 not in '<4.0,>=3.6'
error. The resulting execution still referenced something in my ~/.cache/pants directory so I nuked that,
./.pants.d
and
.pids
and started from scratch.... same thing. Weird. I'll try it with rc1 tomorrow (late over here). Very strange.
ā— 1
h
hey any update on this?
e
Yes! Sorry; works fine now with 2.11.rc1 on the poetry resolver (my issues with the pex resolver are now in another thread). Not sure what the issue is but we've made the switch to the poetry resolver and merged that into our master branch; it's working well.
šŸ™Œ 1