https://pantsbuild.org/ logo
#general
Title
# general
n

nice-florist-55958

11/14/2021, 8:50 PM
When placing a package with a compiled binary into
3rdpary/pkg/bin.so
, Pants only copies the
*.py
files when running the test goal. So
import pkg
results in
__init__.py
being invoked and then an import error from
from .bin import *
because
bin.so
is not in the sandboxed environment. Confirmed by dropping into pdb and looking at the folder structure in the execution env. Any ideas? Is that expected behavior or?
e

enough-analyst-54434

11/14/2021, 9:02 PM
Where and what is the associated owning target?
I assume you have a BUILD file somewhere in that path?
But to skip to the point, if the target is
python_sources
, that only globs (small lie)
*.py
by default; so you'll manually need to specify
sources
globs: https://www.pantsbuild.org/v2.8/docs/reference-python_sources
n

nice-florist-55958

11/14/2021, 9:17 PM
Right, added a BUILD file after one of the dependents was not reporting it as a dependency. Then it was, but now this. So everything internal that we want Pants to consider as a source that other targets might depend on, definitely needs a BUILD with at least a python_sources target? And if that target has source files other than .py, we should put explicitly
sources=['*.py', '*.so', etc.]
?
e

enough-analyst-54434

11/14/2021, 9:18 PM
Yes, exactly. Have you stumbled on the
tailor
goal for automating basic BUILD target seeding?
n

nice-florist-55958

11/14/2021, 9:33 PM
Yah, started playing with that. But still need to do a comprehensive read of all the goals/targets/subsystems; think it will clarify things later, but for now we are just trying to get a proof of concept!
Still not having any luck. This is my BUILD:
python_sources(description="fidlib", sources=["*.py", "*.so"])
; when I drop into the debugger I only see
__init__.py
and
__pycache__
; the binary
fidbin.so
is missing. I tried explicitly listing it as well, no luck.
Found a clue in the warnings: `[WARN] Unmatched glob from 3rdparty/fidlib:fidlib's
sources
field: "3rdparty/fidlib/*.so"` But if I grep the exact directory it's complaining with .so I see the file...
Ahha! Just realized
*.so
is a pattern in our
.gitignore
e

enough-analyst-54434

11/14/2021, 9:43 PM
Ok. Does that fix things?
n

nice-florist-55958

11/14/2021, 9:43 PM
Ok so now it matches and tries, but getting this error:
[ERROR] The 'sources' field in target 3rdparty/fidlib:fidlib can only contain files that end in one of ['', '.py', .pyi'] but it had these files: ['3rdparty/fidlib/fidbin.so']
e

enough-analyst-54434

11/14/2021, 9:43 PM
Ok, I personally find this stuff fiddly and confusing too. I think you need 2 targets. Like so: python_sources( name="fidlib", dependencies=[":so"], ) resources( name="so", sources=["*.so"], )
n

nice-florist-55958

11/14/2021, 9:46 PM
Ok, let me try that. I see the logic of that given this is more of a resource than a source file, but would have never thought to! So this just creates a new resources target w/ "./*.so" as sources on which other targets can depend (and by transitivity include their sources as their own)?
e

enough-analyst-54434

11/14/2021, 9:46 PM
Yes.
n

nice-florist-55958

11/14/2021, 9:51 PM
Alright that seems to work now; fingers crossed our firm let's us upload binaries to source control..
🤞 1
h

hundreds-father-404

11/15/2021, 4:39 PM
I see the logic of that given this is more of a resource than a source file
(Indeed, Idea being that you cannot do things like run Black or Flake8 on the
.so
file. It's not a Python file, only a resource understood by Python)
e

enough-analyst-54434

11/15/2021, 5:26 PM
Although that's true since a generic so is not python, a so side-by-side with Python code is almost undoubtedly an extension; so this boilerplate is a bit unfortunate. We'd need an ELF / mach-O parser though to divine all this properly.
2 Views