https://pantsbuild.org/ logo
Join Slack
Powered by
We use "--no-binary" with pip today to adhere to t...
# general
l
We use "--no-binary" with pip today to adhere to the psycopg2 guidance of not using psycopg2-binary in production but rather building it - https://www.psycopg.org/docs/install.html#psycopg-vs-psycopg-binary . Is there a way to achieve this with Pants?
✅ 1
h
Hi! There is not, but Pex has that option so it's only a question of wiring it up. Before wiring it up, I think we'd want to figure out a little more how to design the feature. For example, would you want an option on 
[pex]
that causes you to never use wheels, for both user and tool lockfiles? Only user code? For user code, only when doing 
./pants package
or also when installing for 
./pants test
etc?
FYI from `pex help`:
Copy code
--wheel, --binary, --no-wheel, --no-use-wheel, --no-binary, --no-use-binary
                        Whether to allow binary distributions. (default: True)
  --build, --no-build   Whether to allow building of distributions from source. (default: True)
  --prefer-wheel, --prefer-binary, --no-prefer-wheel, --no-prefer-binary
                        Whether to prefer older binary distributions to newer source distributions
                        (prefer not building wheels). (default: False)
l
psycopg2 is the only dependency where we've run into this, and it's actually via dbt-postgres. In order to ensure that psycopg2 is built from source, we have to run the following today: RUN DBT_PSYCOPG2_NAME=psycopg2 pip install -r requirements.txt --no-binary dbt-postgres
Individual developers typically don't do that locally and don't care about it, but we ensure we do that in our CI process and when we build for prod
(that's a command in our Dockerfile today, pre-Pants)
h
What consumes that DBT_PSYCOPG2_NAME env var? the dbt-postgres setup.py, presumably?
So that's something we'd have to set as well, it's not just a matter of plumbing 
--no-binary
One workaround, for now, might be to build that dbt-postgres wheel and host it on an internal repo, and point Pants to it ahead of PyPI
Oh wait, that may not make sense
so psycopg2 is a dependency of dbt-postgres, and that DBT_PSYCOPG2_NAME env var tells it to depend on psycopg2 instead of psycopg2-binary, I'm guessing?
And so the 
--no-binary
there tells pip to not use a wheel for 
dbt-postgres
, thus forcing it to run its setup.py, which in turn processes that env var and thus causes a non-binary dep on psycopg2?
OK, after some reading of the dbt docs, they claim that this should work : 
DBT_PSYCOPG2_NAME=psycopg2 pip install dbt-postgres
Without 
--no-binary
But I don't see how that can be true
Since dbt-postgres has a wheel, and it was built to depend on psycopg2-binary: 
Requires-Dist: psycopg2-binary (~=2.8)
So that 
--no-binary
is presumably required (I was hoping we could work around this with just env var plumbing...)
So, hmmm
l
Thanks @happy-kitchen-89482 - unfortunately the "--no-binary" is required, it took us awhile to figure this all out a couple months ago; both the DBT-specific env var and --no-binary are required
h
That feature can definitely be added imo, only a question of figuring out the optimal design. From there, the implementation should be pretty trivial and only a few lines What are your thoughts on these questions? https://pantsbuild.slack.com/archives/C046T6T9U/p1646256562038929?thread_ts=1646256397.117089&cid=C046T6T9U
l
I don't have the expertise to know if never using wheels would be an option. We've only done this no-binary approach for psycopg2 and that was simply based on the recommendation in the psycopg2 docs. It seems to be that I need the same granularity as pip, which is to be able to specify no-binary on a per-package level. And I think ideally, this would be doable for all goals - we'd certainly want it for run/package/test, as those would all be done in a clean environment either as part of a CI cycle for running tests or when building a release.
Re: dbt-postgres's need for an additional env var to be specified - I think Pants would only need to ensure that env vars can be easily injected for the above goals. I have wondered if a global set of env vars for all Pants goals would be useful.
h
So that part is taken care of, it's just the plumbing of 
--no-binary
we'd have to handle, which should be straightforward
l
Thanks @happy-kitchen-89482, I missed that option
h
Well, it isn't much use yet in this case 🙂
l
Want me to write up a ticket for the --no-binary support? Happy to do so
Just FYI, writing one up now
h
Thanks!
e
So that part is taken care of, it's just the plumbing of 
--no-binary
we'd have to handle, which should be straightforward
I responded on the ticket, but I don't think so. That Pex option is global. All requirements will be forced to be built from sdists and actually fail if an sdist is not provided. Pip has a more targeted form of 
--no-binary
that Pex supports via its support for all Pip options that can legally be passed via a requirements file. Pants gets in the way with that though today; so there will be more serious Pants surgery to support this.
I mean, one option certainly is to add yet another feature to Pex, but there would still be all those other Pip options you should be able to pass via requirements file that that feature would not support. It seems time to get Pants to get out of the way here. If that's too hard though, Pex could sprout a feature.
h
This was addressed in https://github.com/pantsbuild/pants/pull/14985 , thanks for the feature request @lively-exabyte-12840!
🙌 2
Open in Slack
PreviousNext
Powered by