https://pantsbuild.org/ logo
#general
Title
# general
e

echoing-farmer-15630

03/08/2022, 1:37 PM
...is this something that could be solved by a locally-running buildfarm instance/container? That would be potentially interesting although I'm not certain how to do so easily.
l

lively-exabyte-12840

03/08/2022, 1:57 PM
Very interested in this as well. I went down a similar path, enamored with the idea of a pex-based Docker image, but most of my team is on MacOS (as am I) and I could not make any meaningful progress on finding a valid platform string. IIRC both PyYAML and MarkupSafe were blockers, and I figure many more blockers are awaiting if I get past those
e

echoing-farmer-15630

03/08/2022, 2:02 PM
I'm sure they are. I'm currently intrigued with the idea of running a local buildfarm (or remote) which may solve multiple problems at once, but unfortunately that's not working out terribly well yet.
h

hundreds-father-404

03/08/2022, 4:55 PM
I could not make any meaningful progress on finding a valid platform string. IIRC both PyYAML and MarkupSafe were blockers, and I figure many more blockers are awaiting if I get past those
Do you remember the issues? Was it that those requirements don't have wheels available for Linux? Fwit, Pants uses
--platforms
and PyYAML https://github.com/pantsbuild/pants/blob/740c7c77cf8bb075e56645caf7555fbe99a0ed5a/build-support/bin/_release_helper.py#L752-L754
e

echoing-farmer-15630

03/08/2022, 4:59 PM
John Sirois gave me some great help (using complete-platforms) and I did get past PyYAML (and MarkupSafe), but then got stuck on source distributions. Not quite wanting to build the infrastructure to build wheels for all source distributions we use, I'm doubling down on trying to learn REAPI tools and set up local (and remote) buildfarms.
👍 1
h

hundreds-father-404

03/08/2022, 4:59 PM
Sg. @lively-exabyte-12840 do you remember the issues you had?
w

witty-crayon-22786

03/08/2022, 5:08 PM
fwiw, cross-building via docker is what is proposed in https://github.com/pantsbuild/pants/issues/13682
was one of the top voted items in the survey
l

lively-exabyte-12840

03/08/2022, 5:19 PM
@hundreds-father-404 I've got a branch where I left off on trying to use platforms, I'll revisit that soon and put together either an issue or respond back here
👍 1
@hundreds-father-404 Running
pex --platform=linux_x86_64-cp-3.9.9-cp39 Flask==1.1.4
gives me an error of: Needed cp39-cp39-macosx_11_0_x86_64 compatible dependencies for: 1: MarkupSafe>=0.23 I can't seem to get any version of Flask to work there. If I include e.g. "-o example.pex", the error doesn't show up until I try running that file, at which point I get the same error
I'm wondering if this matters though because based on the previous thread, @enough-analyst-54434 suggested that the best approach is likely to invest in wheel building infrastructure to pre-build wheels. I don't think that's feasible in the short term for us.
I also looked into a multi-stage Docker build where stage 1 runs "package" to build a pex, and then stage 2 copies the pex over and runs it. The pex runs fine, but I couldn't find a way for changes to source files to be picked up without having to run "docker-compose up --build", which takes quite awhile because "pants package" is effectively building from scratch
e

enough-analyst-54434

03/08/2022, 7:51 PM
@lively-exabyte-12840 wheel building is needed if you will ever have deps that include pure sdist deps, full stop. Iff you don't think is the case or will be the case, you probably want to use --complete-platform instead of --platform to remove the guess work: https://pex.readthedocs.io/en/v2.1.70/buildingpex.html#complete-platform
l

lively-exabyte-12840

03/08/2022, 7:53 PM
Was just looking at that - is there a good example to follow for that? Have been having some difficulty determining the appropriate values for platform/complete-platform
e

enough-analyst-54434

03/08/2022, 7:55 PM
You don;t determine the values for --complete-platform, which is part of the point! It's buried in those docs at https://pex.readthedocs.io/en/v2.1.70/buildingpex.html#complete-platform but you must run
pex3 interpreter inspect --markers --tags > platform.data
on the machine you're targeting to generate a complete platform data file. Then, back on your other machine
--complete-platform platform.data
To be completely explicit, you really want to run
pex3 interpreter inspect --python /this/interpreter/here --markers --tags --output platform.data
l

lively-exabyte-12840

03/08/2022, 8:00 PM
Thanks, I'll give that a spin
e

enough-analyst-54434

03/08/2022, 8:01 PM
Thanks. Like I said though, if the resolve happens to hit any distribution only available as an sdist, things will still blow up and there is nothing to do about that except pre-build a wheel for that distribution and point to it with
--find-links
or else go down your Docker route.
l

lively-exabyte-12840

03/08/2022, 8:12 PM
Good news is that with a minimal requirements.txt file only containing Flask==1.1.4, a simple test worked. Trying all dependencies next.
@enough-analyst-54434 I assume this is a good example of a project where we'd need to build our own wheel for our target platform? - https://pypi.org/project/annoy/#files
e

enough-analyst-54434

03/08/2022, 8:44 PM
Yeah, assuming you're not targeting macosx.
h

hundreds-father-404

03/08/2022, 8:44 PM
If you are trying to use it on any platforms other than cp37-cp37m-macosx_10_14_x86_64, such as linux, then yes. (re the 10_14 part, macOS is future-compatible so newer macOS versions should at least work)
e

enough-analyst-54434

03/08/2022, 8:46 PM
But re the cp37 part - no abi3 so dead in the water for all other interpreters.
1
Afaict this is a developer that published from their laptop.
e

echoing-farmer-15630

03/09/2022, 9:42 PM
The build farm idea has legs, but it's certainly been painful (I will say the experience has reminded me how friendly and helpful the pants community is). I got a build working with buildbarn and buildgrid (buildfarm is being fussy), and locally you can even do ssh Auth passthrough for private repos, although it's failing in one case and the experience is more frustrating than imagined; with multiple targets a single failure seems to result in immediate failure of all running jobs, and things seem to take a lot longer, and I'm worried in some cases the generated blobs will be so large that cloud based buildfarms may not be as useful as I'd like. So not a panacea, but still worth pursuing.
💜 1
5 Views