https://pantsbuild.org/ logo
#general
Title
# general
p

plain-carpet-73994

10/13/2021, 7:01 PM
Hey - I'm trying to get a build working on ARM-64. I've successfully compiled pants and the build starts but I have a step that runs
protoc
via the standard pants
protobuf_library
goal. If I run the build with
--no-process-execution-local-cleanup
I can cd into the directory and if I try to run the command I get
bash: ./bin/protoc: cannot execute binary file: Exec format error
. Indeed, if I just run
./bin/protoc
I get that error. However, I have
protoc
installed so if I change the command to run just
protoc
instead of the
./bin/protoc
in the sandbox it works. I think the issue is that pants seems have bundled their own protobuf compiler and it's an intel build, not an ARM build. Is that correct? Any way I can fix that?
h

hundreds-father-404

10/13/2021, 7:04 PM
Hi! What Pants version? macOS or Linux?
p

plain-carpet-73994

10/13/2021, 7:28 PM
2.6.0 Linux.
On Debian Bullseye (actually in a Docker container that's running Bullseye)
h

happy-kitchen-89482

10/13/2021, 7:34 PM
Hmm, I'm surprised it doesn't error on not finding an appropriate entry in this option: https://www.pantsbuild.org/docs/reference-protoc#section-known-versions
h

hundreds-father-404

10/13/2021, 7:34 PM
Hmm @witty-crayon-22786 I had thought you added to the docs about hosting binaries from local filesystem path, but I don't see it on https://www.pantsbuild.org/docs/proxies#hosting-binaries? Even though Pants doesn't yet support ARM, I think you can "trick it" by pointing to your local binary. Also see https://github.com/pantsbuild/pants/pull/12879, a user started to add ARM64 support but didn't finish
@happy-kitchen-89482 it's because architecture-awareness wasn't added by @ancient-vegetable-10556 until Pants 2.7. On Pants 2.6, Pants only knows "Linux vs macOS"
p

plain-carpet-73994

10/13/2021, 7:35 PM
@hundreds-father-404 so if I upgrade to 2.7 you think I'll be OK?
Or I have to upgrade to 2.7 and then also do something to point it at the local protoc?
w

witty-crayon-22786

10/13/2021, 7:36 PM
Hmm @witty-crayon-22786 I had thought you added to the docs about hosting binaries from local filesystem path
mm… no: it’s only mentioned in the subsystem help.
p

plain-carpet-73994

10/13/2021, 7:36 PM
(about to head to a meeting; back online in 30)
h

happy-kitchen-89482

10/13/2021, 7:36 PM
Ah! So I guess you need to upgrade to 2.7, and then set something like this in pants.toml:
Copy code
[protoc]
known_versions.add=["3.11.4|linux_arm64 |<sha256 of relevant protoc binary>"]
We should add this to the default too
p

plain-carpet-73994

10/13/2021, 7:37 PM
And that'd work on both intel and ARM or would my build then only work on ARM (I need both)
h

hundreds-father-404

10/13/2021, 7:39 PM
2.7 wouldn't work because you're on Linux, not macOS. In fact, Benjy, I think using 2.7 might cause an error because
Platform
does not include Linux ARM 64 in it? I think we may need to cherry-pick to Pants 2.7 that you can attempt to use ARM Linux, only you will have to host the binaries yourself 2.6 won't work if you need to support both Intel and ARM. It would make you choose one. (Which you could override in a
.pantsrc
config file fwit)
h

happy-kitchen-89482

10/13/2021, 8:01 PM
In principle that adds the value to the existing default, which includes linux_x86_64, so should work on both
but apparently even 2.7 won't work for reasons Eric is mentioning
Ah yes, we will need to fix that up
Not hard to do though
p

plain-carpet-73994

10/13/2021, 9:11 PM
OK. It'd be nice to have full ARM support. For now the targets that we need to build for ARM don't actually need protobufs (I think) so I think I have a super-hacky workaround.
@happy-kitchen-89482 related question: is there a way to disable a goal for certain architectures. Essentially something like:
Copy code
if arch == 'x86_64':
   python_library()
h

happy-kitchen-89482

10/13/2021, 9:27 PM
Disable a goal, or a target?
p

plain-carpet-73994

10/13/2021, 9:28 PM
Target
h

happy-kitchen-89482

10/13/2021, 9:30 PM
You can tag targets (
tags=["arm64_only"]
for example) and then include or exclude tags at runtime, e.g.,
./pants --tag=arm64_only ...
or
./pants --tag=-arm64_onlu ...
)
And with the upcoming CLI alias feature (h/t @curved-television-6568) you can create aliases for cumbersome CLI args
so you could do
./pants test_arm64 ...
and an alias could turn that into
./pants --tag=arm64_only test ...
But that feature isn't in any release yet
Tags are though
p

plain-carpet-73994

10/13/2021, 9:33 PM
@happy-kitchen-89482 would
platforms
work? e.g.
Copy code
python_library(
   platforms = [
      'linux-x86_64-cp-37-m',
      'macosx-11.0-x86_64-cp-37-m'
   ]
)
tested it and the answer is "no" 🙂
h

happy-kitchen-89482

10/13/2021, 9:36 PM
Ah no, that is not a thing
p

plain-carpet-73994

10/13/2021, 9:47 PM
I see. Tags aren't working for me either. The issue is that I have a 3rd party library that doesn't exist for ARM. But I don't need the library and pex that uses it. So I tagged those targets with
noarm
and then ran
pants --tag='-noarm'
but pants still tries to install everything in
constraints.txt
even if no target to be built actually needs that dependency. So it fails installing a library I don't need. But I do need/want those targets on Intel and I want them pinned so I don't want to remove them from my constraints. What to do?
h

hundreds-father-404

10/13/2021, 9:51 PM
For that, we need multiple user lockfiles I fear: https://github.com/pantsbuild/pants/issues/12314 Which we made a lot of progress towards doing last month. But blocked by some work John has been doing for Pex to generate lockfiles using pip, which has been his top priority @witty-crayon-22786 do you know of any workarounds here?
p

plain-carpet-73994

10/13/2021, 9:57 PM
FWIW, I've found
pip-compile
generates a much nicer lockfile that pip. Mostly I like the comments it puts under each dependency saying where it came from and the fact that it only locks what's in the file you pass it (e.g.
<http://requirements.in|requirements.in>
) rather than locking everything in the venv.
In general it seems like pants installs everything in
constraints.txt
regardless of what target you're building. That can make things really slow if I want to build a target with few 3rd party deps. Would it be possible to have
constraints.txt
just be ... constraints but only install the dependencies required for a target?
w

witty-crayon-22786

10/13/2021, 10:18 PM
yea: see https://www.pantsbuild.org/docs/reference-python-setup#section-resolve-all-constraints … the default is an optimization that assumes that your cache will be reused
the effect of turning it off is that each target resolves independently, so depending on what fraction of the deps each target uses and whether your cache is surviving, the break even point can be pretty low on just resolving the whole thing.
p

plain-carpet-73994

10/13/2021, 11:04 PM
thanks @witty-crayon-22786. That makes sense. Per the above the issue I'm having is that it's trying to resolve constraints that don't exist for ARM64 and we're never going to build those targets on ARM. Short term it seems like pass the
--no-python-setup-resolve-all-constraints
might make things work though at the cost of some speed. Sound right?
w

witty-crayon-22786

10/13/2021, 11:05 PM
yes, that’s correct: BUT, i think that that is also something that is addressed in 2.7, if I remember correctly… when platforms are in use, the optimization is disabled
p

plain-carpet-73994

10/13/2021, 11:06 PM
From my point of view the ideal would be the ability to disable targets for various reasons. CPU arch is the current constraint but, IMHO, a general
if
statement that could use other constraints (e.g. OS, an env variable, etc.) seems pretty valuable. I do understand that you'd have to limit what could be there to make builds reproducible (e.g. env vars are harder but you do already allow that for tests).
w

witty-crayon-22786

10/13/2021, 11:08 PM
Eric is also right that disabling this optimization is temporary until we’ve implemented multiple user lockfiles, as that would support both resolving the constraints file, and multiple platforms. trying to find the relevant ticket there
👍 1
p

plain-carpet-73994

10/13/2021, 11:08 PM
that is also something that is addressed in 2.7, if I remember correctly… when platforms are in use, the optimization is disabled
@witty-crayon-22786 I'm up for upgrading to 2.7 but if I followed the conversation above it sounds like that won't yet work on ARM+Linux at all (see https://pantsbuild.slack.com/archives/C046T6T9U/p1634153982080700?thread_ts=1634151679.077600&amp;cid=C046T6T9U)
👍 1
trying to find the relevant ticket there
Ticket was already shared above. Thanks.
w

witty-crayon-22786

10/13/2021, 11:09 PM
there is another ticket that would confirm that we disable this optimization automatically in 2.7. but yea, moot until you’re able to upgrade there.
h

hundreds-father-404

10/13/2021, 11:14 PM
p

plain-carpet-73994

10/14/2021, 2:21 AM
Still having issues. I think the issue that remains is that if you use
--tag
with a
::
target it first builds the .pex files for all targets, even those excluded by the
--tag
and then filters by tag so there isn't an easy way to say "run all tests except those with tag X and don't bother resolving dependencies for the tests you won't run". For example, with
-ldebug
I see:
Copy code
02:16:11.58 [DEBUG] Completed: Hit local cache: Scheduling: Determine Python imports for realsense/lib/colorize_depth.py
and then the old error:
Copy code
ERROR: Could not find a version that satisfies the requirement pyrealsense2==2.49.0+companion3
ERROR: No matching distribution found for pyrealsense2==2.49.0+companion3
But that target looks like this:
Copy code
cpy_library(
   tags=['noarm']
)

cpy_tests(
   tags=['noarm']
)
Where the definitions of the
cpy_
macros is:
Copy code
def cpy_library(**kwargs):
    python_library(**kwargs)

def cpy_tests(**kwargs):
    if 'name' not in kwargs:
        kwargs['name'] = 'tests'
    python_tests(**kwargs)
and the command I ran is:
Copy code
pants --tag='-firebase_emulator,-noarm' --no-python-setup-resolve-all-constraints -ldebug test ::
I do seem to be able to run individual targets but, sadly, not "all targets except". For example, there's no dependencies on
pyrealsense2
in anything under
device
so
pants ... test device/::
works fine.
It also seems there isn't an
opencv-python
module on PyPi for ARM but you can
apt get install python3-opencv
. however, I don't know how to make Pants use that version. I didn't see any mention of using system-installed libs here: https://www.pantsbuild.org/docs/python-third-party-dependencies. I could custom-build the wheel myself and host it somewhere...
w

witty-crayon-22786

10/14/2021, 2:47 AM
You would need to set LDFLAGS appropriately I think... away from my computer, but there should be some hits on the docsite
h

happy-kitchen-89482

10/14/2021, 5:36 AM
Still having issues. I think the issue that remains is that if you use 
--tag
 with a 
::
 target it first builds the .pex files for all targets, even those excluded by the 
--tag
 and then filters by tag so there isn't an easy way to say "run all tests except those with tag X and don't bother resolving dependencies for the tests you won't run".
That seems odd.
Try
pants --tag='-firebase_emulator,noarm' --no-python-setup-resolve-all-constraints -ldebug test ::
?
I think the
-
applies to all the tags in the comma-separated string, so
-noarm
is seen as a tag, not as a negation of a tag
or to be even more explicit you can
--tag=-firebase_emulator --tag=-noarm
c

curved-television-6568

10/14/2021, 7:15 AM
Well that’s a gotcha.. I too would’ve expected the
-
to be on a per tag basis..
So if you have both inclusive and exclusive tags, they have to be on separate
--tags
flags, then
Perhaps have
--exclude-tags
instead of
-
prefix on tagnames? That would make it more clear.
(Or support
-
per tag name… 😉 )
h

hundreds-father-404

10/14/2021, 7:18 AM
it's the same code as `filter`: https://www.pantsbuild.org/docs/project-introspection#filter---find-targets-that-match-a-predicate we probably need more examples there
👍 1
c

curved-television-6568

10/14/2021, 7:23 AM
Ah, right. It is this:
You can prefix the filter with 
-
 to negate the filter, meaning that the target must not be true for the filter.
That causes the confusion for me.
(not when reading the docs, just from what was expected)
For me it would’ve been more intuitive to say that “You can prefix the value with
-
to negate the match, meaning that the member must not be matched”. However, this doesn’t work well with “at least one member must be matched”, as that would suddenly include ALL other tags except the one you’re filtering out, so negated ones should be added with AND rather than OR in that case.
So I see why this negates the entire filter, as the multiple flags are joined with AND, it works.
I was thinking if adjusting the syntax/semantics was feasible, but I think not, better to learn how this works (to keep it simple, as anything I could come up with was adding complexity). But there could be a
--explain
option, that could simply dump in a readable form what it would do, and make it very clear what filters it would use inclusively/exclusively on what targets and types etc..
Ok, enough ranting on my part here now 😛
p

plain-carpet-73994

10/14/2021, 6:46 PM
Yup it was the double
-
in the filter that confused me. Sorry for the false alarm.
re: opencv on ARM: I'll move to another thread.
6 Views