https://pantsbuild.org/ logo
#general
Title
# general
c

cool-yacht-37128

08/24/2022, 7:38 AM
Is it possible to create a single python (pex based) lock file with multiple versions of the same package? I have a small number of components that need to support a broad range of interpreter versions (going as low as 3.6), and it’s impossible to get all our 3rd party deps to support this far. I’ve had success isolating those components to their own resolve/lock-file, but I feel like I’m losing one of the core benefits of the monorepo since I now need to carefully self manage which resolve is used and which components share code. Is there a better way?
b

bitter-ability-32190

08/24/2022, 11:14 AM
I'm very interested in this as well, specifically for
pytorch
.
e

enough-analyst-54434

08/24/2022, 12:25 PM
Yes, but Pants doesn't use this style of lock file. The pex3 lock create command supports
--style {strict, sources, universal}
and Pants always uses
universal
today. The universal mode creates 1 lock that must work for the complete range of interpreters and machines implied by
--interpreter-constraint
and any
--target-system
specified (Pants always passes
--target-system mac --target-system linux
). The strict and sources modes, on the other hand, use
--interpreter-constraint
on the local machine to find all interpreters meeting those constraints and then, for each interpreter, creates a lock and all of these locks are stored in the one lock file. When the lock is later consumed to produce a PEX for a specific interpreter, the best-fit lock is selected from the lock file. The only difference between strict and sources is that strict locks exactly what each selected interpreter needs and no more whereas sources also locks the sdist associated with each wheel locked if there is an sdist. A challenge using these modes is to ensure the machine you create the lock on has all the interpreters needed installed. Otherwise, if you have
--interpreter-constraint ">3.7,<3.10"
and you create the strict or sources lock on a machine with only Python 3.8, the lock file will only contain a single lock for the Python 3.8 interpreter but not 3.7, 3.9 or 3.10 (and even that is simplistic since real artifact in the wild have constraints like `Requires-Python >=3.7.2`; so to cover all bases you really need every minor release of Python too unless you know your resolve real well and know this is not a factor. With universal mode, the lock create command can safely be run on any machine.
@cool-yacht-37128 and @bitter-ability-32190 is there no way to use multiple requirements for the same project but different versions via environment markers?
b

bitter-ability-32190

08/24/2022, 12:47 PM
I'm not sure I follow what you mean. Also just to make sure we're all on the same page, we're talking about different versions of the same package. E.g pytorch 1.2.2 AND pytorch 1.2.2+cu102
I know it's hairbrained from a requirements perspective, but from a users perspective I wanna toggle the GPU lib on (through a pants plugin)
e

enough-analyst-54434

08/24/2022, 12:51 PM
We're on the same page. In plain old non Pants you'd have 2 requirements:
Copy code
pytorch==1.2.2; <marker1>
pytorch==1.2.2+_cu102; <marker2>
You'd arrange marker1 and marker2 to be mutually exclusive.
👍 1
This requires the restriction is based on a property of the Python being used, i.e. expressible in the marker ~language.
If the restriction is based on other things, the technique won;t work.
I lose track of what people know about, so forgive me if this is all known, but I mean these markers: https://peps.python.org/pep-0508/#environment-markers
b

bitter-ability-32190

08/24/2022, 1:02 PM
Yeah, same execution environment in either case. Just wanting to toggle between reqs when told
e

enough-analyst-54434

08/24/2022, 1:03 PM
Yeah, Pex / Pip / PyPA can't help you there. That would have to come from Pants or higher.
✝️ 1
O god
b

bitter-ability-32190

08/24/2022, 1:05 PM
Mxnet got this right. They use a different package name but same module name
e

enough-analyst-54434

08/24/2022, 1:15 PM
That's bad too! Now you can have both packages on sys.path and you'll get the import from a ~random one!
You wouldn't believe the hell this led to over in Twitter when netty did so for the JVM.
Basically, if you know what you're doing, you can get it to work for you and profit but there are just too many ways for things to go wrong for the rest of the people.
b

bitter-ability-32190

08/24/2022, 1:19 PM
Oh I agree it's nasty. But at least it's workable. I'm kneecapped with pytorch
2 Views