https://pantsbuild.org/ logo
Powered by
p

plain-fireman-49959

03/01/2022, 9:48 AM
Hello, I'm trying to move an existing project to a monorepo using pants, I am already using pex's and docker builds, this project however uses cython and I can't find any information regarding the plugin you mention in the documentation, could anyone point a link to the plugin or some documentation? thanks!
h

happy-kitchen-89482

03/01/2022, 11:22 PM
Hi! I think that @calm-ambulance-65371 was working on a cython plugin, if I remember correctly?
But that might have been a proprietary thing
How are you building cython today?
Pants now supports any PEP-517 build backend, so if there is one of those that builds cython, then that might also be a way to do it.
🙌 1
c

calm-ambulance-65371

03/02/2022, 4:21 AM
It's proprietary for now, but I can try to push to open source it, though it's kind of hacky and not a generic solution. Otherwise I'm happy to help out in implementing something similar.
We're also stuck on Pants 2.2 for now
h

happy-kitchen-89482

03/02/2022, 4:48 AM
Hey @calm-ambulance-65371, thanks for the update
So, @plain-fireman-49959, I guess the answer depends on how you build cython code today.
p

plain-fireman-49959

03/02/2022, 9:04 AM
Hello, the library I'm trying to integrate in my monorepo is developed by a coworker so I'm not 100% sure how we build it today. We use poetry and the existing version has a 
[tool.poetry.build]
section with a 
build.py
file, the 
def build():
function passes `setuptools.Extension`s to the 
setup_kwargs
. Again I'm not very familiar with the current build system, but I'm guessing we let poetry handle everything. Of course it would be great to have a generic plugin solution for Cython, but I'm more than happy to develop a specific plugin for my use case, I just need some pointer to the right direction (like this blog post)!
Also, I'm trying to understand the steps pants has to take to package everything as I need. I have a pex (call it 
pex_a
) that today depends on this project. I'd like to move it as use it as a source dependency, this has been straightforward until cython came along. My understanding is that the 
python_sources
should have dependency on a something that runs the cython build process (that would be implemented in a plugin), then pants should package 
pex_a
with the sources and the cython compiled files. Does this make sense? What I'm not 100% sure about is what kind of plugin I'd need: is this a new target? would a macro be enough (if I can run arbitrary python code I should be able to compile it maybe?)?
h

happy-kitchen-89482

03/02/2022, 6:51 PM
Sounds like poetry is delegating to setuptools, so you might be able to get it working with a custom setup.py
I don't know much about cython, does it build wheels? how does regular python code interoperate with cython code normally?
I suspect what you want is for 
python_sources
to depend on a 
python_distribution
target (assuming that setuptools can be configured to build cython)
python_distribution
wraps a 
setup.py
and runs setuptools on it
I'd have to do like 10 minutes of reading about cython to figure out more
But that is my guess so far...
p

plain-fireman-49959

03/03/2022, 9:53 AM
Thank you for taking the time to write this down, this definitely makes sense to me so I'll try to use 
setuptools
to build a distribution which my sources will depend upon. I'll update you when I have news!
h

happy-kitchen-89482

03/03/2022, 2:51 PM
To clarify - if you're able to build a cython distribution using setup.py then Pants can invoke that setup.py for you as needed, you don't need to publish that distribution and then consume it as a built package
So it's still a source dependency, we're just trying to figure out how to get Pants to build it, but it would still be building it on the fly, automatically, if its sources change, just like any other direct in-repo source dependency
And to make all this work, you just need a setup.py that will do the building, and a 
python_distribution
target to point to it
(er, and a dependency on that 
python_distribution
)
p

plain-fireman-49959

03/08/2022, 3:53 PM
Well I got sidetracked... I was able to play around with this again today, I am running my own 
setup.py
file with this 
pyproject.toml
Copy code
[build-system]
requires = ["setuptools", "wheel", "Cython"]
build-backend = "setuptools.build_meta"
I think I am almost there, but I am getting this error (even with the 
__legacy__
build backend:
Copy code
error in REDACTED setup command: <lambda>() missing 1 required positional argument: 'value'
Has anyone seen this before?
3 Views
Powered by