https://pantsbuild.org/ logo
Join Slack
Powered by
I’ve got a monorepo of Python Libraries and I’m st...
# general
g
I’ve got a monorepo of Python Libraries and I’m struggling with some compiled modules. For example, 
package-1
has a Pyrex file (
foo.pyx
) that gets compiled to a C Extension (
foo.c
) at build time. The issue that I’m having is with plain Python projects in the monorepo that depend on that module. So in 
package-2
the following import raises an error, 
from package_1.foo import bar
. My current workaround has been to create multiple resolves so that 
package-1
is actually a third party dependency for 
package-2
and the 
from package_1.foo import bar
statement passes as expected. But I don’t really like that solution and now I’m stuck with an issue where in order to make a change to both 
package-1
and 
package-2
I have to go through the burdensome process of changing 
package-1
and republishing it and then changing 
package-2
. Is there any pants magic that will allow me to keep both of these projects using the same resolve (i.e. 
package-2
depends on the compiled 
package-1
instead of the uncompiled code)?
b
There was a thread/discussion recently about use of cython: https://pantsbuild.slack.com/archives/C046T6T9U/p1670269859559719 https://github.com/pantsbuild/pants/discussions/17718 As referenced in that thread, sounds a bit like a codegen task (generating the C extension from the .pyx input 'spec', for use by other code). (I don't know the answer at all, or if that thread provides useful insight, just pattern matching on the pyrex mention)
👀 1
g
Thanks @broad-processor-92400. I’ll follow along on that thread and related GH Issue. You’re right though - this is a good use case for a CodeGen Task. Similar to the protobuf plugin the files I need to depend on aren’t actually persisted as Python Sources - but they do need to exist at Testing/Packaging/Dependency Inference Time. Unfortunately, being compiled C files (instead of compiled Python files like 
protobuf_sources
generates) they still won’t be recognized as valid sources during dependency inference without some custom work.
h
I think this may be simpler than all this
sounds like you already have a setup.py that builds the native code into a distribution?
So if you add an explicit dependency from package-2's 
python_sources
target to the 
python_distribution
target for 
package-1
, then Pants should do the right thing
This would also work in the case of that GH issue linked above, it’s just that then Pants won’t apply caching in the most effective way, because the inner workings of the setup.py are opaque to it
But setting that to one side, Pants can build local dependencies on 
python_distribution
into dists, and put them on the syspath in the right way
So unless I’m misunderstanding, this should be straightforward
g
That does sound straightforward - let me give that a shot.
Okay - I think I’ve gotten a sufficient example repo where I’m running into the same kinds of problems I am on the private repo: https://github.com/juftin/pants-compiled-example/tree/main Adding a python distribution to the test dependencies still raises a dependency inference warning. The actual blocking error I get is a Pex issue complaining that the 
"pex had no ProjectName(raw='<package>', normalized='<package>') distributions."
for the third party dependencies.
Copy code
❯ ./pants test packages/package-1:: --no-local-cache --no-pantsd
...
17:55:08.16 [WARN] Pants cannot infer owners for the following imports in the target packages/package-1/tests/test_foo.py:../tests:

  * package_1.foo.bar (line: 1)
...
17:55:19.11 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Building pytest_runner.pex' failed with exit code 1.
stdout:

stderr:
Failed to resolve requirements from PEX environment @ /Users/justinflannery/.cache/pants/named_caches/pex_root/unzipped_pexes/bc2695fa37a2506cb61b51a40f9ca033f533b9f3.
Needed cp37-cp37m-macosx_12_0_x86_64 compatible dependencies for:
 1: numpy<2,>=1
    Required by:
      package-1 0.1.0
    But this pex had no ProjectName(raw='numpy', normalized='numpy') distributions.
 2: scipy<2,>=1
    Required by:
      package-1 0.1.0
    But this pex had no ProjectName(raw='scipy', normalized='scipy') distributions.
Copy code
❯ ./pants test packages/package-2:: --no-local-cache --no-pantsd
...
17:56:56.93 [WARN] Pants cannot infer owners for the following imports in the target packages/package-2/package_2/baz.py:../lib:
  * package_1.foo.bar (line: 4)
...
17:57:07.98 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Building pytest_runner.pex' failed with exit code 1.
stdout:

stderr:
Failed to resolve requirements from PEX environment @ /Users/justinflannery/.cache/pants/named_caches/pex_root/unzipped_pexes/43795547cf6eb6dc875955ce70c7c472da2f57be.
Needed cp37-cp37m-macosx_12_0_x86_64 compatible dependencies for:
 1: numpy<2,>=1
    Required by:
      package-1 0.1.0
      package-2 0.1.0
    But this pex had no ProjectName(raw='numpy', normalized='numpy') distributions.
 2: scipy<2,>=1
    Required by:
      package-1 0.1.0
      package-2 0.1.0
    But this pex had no ProjectName(raw='scipy', normalized='scipy') distributions.
🤦‍♂️ this does work work when I forcibly omit the numpy/scipy dependencies though. I think this is a wheel availability issue for third party dependencies 😢
h
The dep inference warning might be due to source root misconfiguration? What does 
./pants roots
show?
And re the other issue, are you using a lockfile?
g
On that example 
./pants roots
shows:
Copy code
3rdparty
packages/package-1
packages/package-2
I think the dependency inference issue is that 
package_1.foo.bar
is defined in a 
.pyx
file and not a 
.py
file - so its target is a resource and not a 
python_source
. It doesn’t become an importable 
C
file until the dist is built and the file is compiled. I am using a LockFile. The issues with the third party resources don’t show up until I add the 
dist
as a dependency on elements like tests.
2 Views
Open in Slack
PreviousNext
Powered by