https://pantsbuild.org/ logo
#general
Title
# general
a

acceptable-guitar-79854

09/08/2020, 8:40 PM
I have http_proxy and https_proxy set in my .bashrc and I am setting them again in the pants script
h

hundreds-father-404

09/08/2020, 8:54 PM
Ah, these are env vars, correct? I suspect this would be the issue. Pants runs in a hermetic environment, meaning that it strips most env vars, for better reproducibility and more cache hits. We realized that sometimes you need to be able to set arbitrary env vars, though, like here. We recently added a mechanism to do this when running a test, but that wouldn’t work here as the value needs to be set when building the PEXes. We can add a mechanism to do that for the new alpha release at the end of the week. In general, it sounds like network requests don’t work without both those env vars set, right?
h

hundreds-breakfast-49010

09/08/2020, 8:57 PM
it would be good to confirm that this is the issue by running pants with more verbose debugging
a

acceptable-guitar-79854

09/08/2020, 9:02 PM
I can run in verbose mode
h

hundreds-breakfast-49010

09/08/2020, 9:02 PM
on second thought, it's likely that pants debugging wouldn't currently reveal anything useful about what's likely going on here
h

hundreds-father-404

09/08/2020, 9:03 PM
The more helpful thing would be to confirm this:
In general, it sounds like network requests don’t work without both those env vars set, right?
For example, try doing
pip install flake8
without the env vars set
a

acceptable-guitar-79854

09/08/2020, 9:03 PM
the same timeout happens
👍 1
(i'm not glorifying bazel by any chance - this is why I'm here in the first place, bazel left me with PTSD)
i just tried "example-python", at home all goes well as expected, at work I hit the proxy issue
h

hundreds-father-404

09/08/2020, 9:06 PM
Okay, thanks for confirming. It sounds like the issue is indeed us not propagating those two env vars. We will try to get this fixed in time for the next alpha release - we’d love to unblock you from trying out Pants. Thanks for linking to that! We’ve gone back and forth on how much we want some global “set for every single subprocess” option vs. keeping things more scoped. Here, for example, I think we want to automatically look for
http_proxy
and
https_proxy
, with the ability to override if you want. We do that for
CPPFLAGS
,
LDFLAGS
,
LC_ALL
, and
LANG
, as those are very frequently needed. Generally, we’ve been erring on the more conservative side, as it’s harder for us to take away a feature than to add it.
a

acceptable-guitar-79854

09/08/2020, 9:07 PM
maybe no_proxy as well
💯 1
makes sense, since hermeticity is one of the core principles
h

hundreds-father-404

09/08/2020, 9:08 PM
I’ve honestly never heard of those 3 env vars, and Google isn’t being super helpful at a quick glance. Are those standard, or specific to your work environment? Nvm, issue was DuckDuckGo instead of Google, hehe.
a

acceptable-guitar-79854

09/08/2020, 9:09 PM
yes, they are very common
--action-env is pretty helpful in bazel, their syntax for marking sources is horrible so action-env saved me from that pain by just setting PYTHONPATH (23423422342 pains to go)
thank you a lot @hundreds-father-404 @hundreds-breakfast-49010
❤️ 1
h

hundreds-breakfast-49010

09/08/2020, 9:14 PM
no problem. thanks for giving pants a try and reporting this issue
h

hundreds-father-404

09/08/2020, 9:16 PM
their syntax for marking sources is horrible so action-env saved me from that pain by just setting PYTHONPATH
Er, this is the type of thing that is hard for us to reason about and what gives us pause with arbitrary env vars. I have a feeling we are going to end up needing this mechanism, but I think for now, stay conservative by only handling those 3 env vars. Opened https://github.com/pantsbuild/pants/issues/10746. Thank you for reporting this! Let us know if we can help with anything else or any feedback 🙂 We’re trying to make 2.0 be really polished and helpful for users.
a

acceptable-guitar-79854

09/08/2020, 9:17 PM
thank YOU
💯 1
if i go unresponsive here feel free to ping me at cristianmatache@hotmail.com or linkedin
h

hundreds-father-404

09/08/2020, 9:17 PM
Sg, thanks!
a

acceptable-guitar-79854

09/08/2020, 9:32 PM
just remembered something else that used to cause problems in Bazel
when installing a Python package, setuptools creates some files called "build" and it checks if that file path is not already taken
but - last time i checked that - it is case insesitive
so there is a clash with the BUILD files
Bazel supports renaming to BUILD.bazel
maybe Pants could allow for BUILD.pants
or something like that
That happens when people pip install from my git repo
h

hundreds-father-404

09/08/2020, 9:36 PM
We do 🙂 only not documented beyond the reference page (https://www.pantsbuild.org/v2.0/docs/reference-global), and I’m not sure where we would document it. Probably a tool tip on the “BUILD files and targets” page This is the default:
Copy code
[GLOBAL]
build_patterns = ["BUILD", "BUILD.*"]
This will work with
BUILD.pants
as is. If you want to enforce only
BUILD.pants
, change to
build_patterns = ["BUILD.pants"]
a

acceptable-guitar-79854

09/08/2020, 9:37 PM
Awesome 👍
thank you guys
i will think of other things that caused problems with bazel and put them here. mypy stubs and mypy plugins were pretty tricky too (using regular bazel with a plugin not the dropbox bazel flavour - of which i can't speak)
h

hundreds-father-404

09/08/2020, 9:43 PM
mypy stubs and mypy plugins were pretty tricky too
Oof. I’ve literally been working on improving our MyPy implementation all day. It’s so confusing. Pants does not currently support MyPy plugins, but we did with the old v1 engine, so we know it is possible of course.
a

acceptable-guitar-79854

09/08/2020, 9:45 PM
let me know if i can help, i can share the way it is done in the bazel plugin https://github.com/thundergolfer/bazel-mypy-integration
👀 1
i'm currently running mypy on and off bazel because of this, we have internal stubs, an internal plugin and external stubs and plugins
👍 1
h

hundreds-father-404

09/08/2020, 9:49 PM
Thanks for linking to that! Are y’all using
.pyi
files? That’s one of the simple fixes I think we can make. Right now, Python targets like
python_library
validate that the file ends in
.py
. I think we should loosen it to allow
.pyi
, but I wanted to talk through with someone using stubs if they think it makes sense to have in
python_library
rather than a dedicated
python_type_stubs
target. I don’t think we would handle the stubs any differently; it’s more about the BUILD file modeling things better afaict. But that also is clunky - we made a huge effort in 2.0 to cut down on boilerplate by adding dependency inference, for example
a

acceptable-guitar-79854

09/08/2020, 9:51 PM
dependency inference is very very appealing
❤️ 1
thanks for the effort
h

hundreds-father-404

09/08/2020, 9:53 PM
It removed the biggest pain point I think a lot of Pants users had. Maintaining BUILD files is horrible, and it’s really easy for them to become stale because no one wants to update them. So, your invalidation gets thrown off. We still need to document it in some more places, but if you want to try it out, it’s enabled by default in 2.0.0a2. You may need to teach Pants about some of your third party dependencies. See https://www.pantsbuild.org/v2.0/docs/python-third-party-dependencies#basic-setup
a

acceptable-guitar-79854

09/08/2020, 9:54 PM
i will try it out
i don't have a lot of spare time, since this is the sort of things that i do in my spare time but maybe i can help out documenting or something like that
in terms of stubs, i agree that a separate rule is clunky but it is appealing in a way. I will give you an example: at work we are still stuck with py 3.6 so we must use @cachedproperty as a third part requirement (rather than the built in python 3.8+) so i wrote the stubs for it - so I'm not sure how python_library would work
h

hundreds-father-404

09/09/2020, 6:10 AM
Thanks for the offer, Christian! Help with MyPy would be the most impactful. https://github.com/pantsbuild/pants/issues/10131 tracks some of the improvements we’re working on as part of 2.0. If you’re curious, this is the first fix I was working on today: https://github.com/pantsbuild/pants/pull/10750 Tomorrow is getting 3rd party requirements to work, (which is embarrassing they weren’t). After that, I want to look into type stubs, including first party. Something that would help a lot is if you could please send a gist with an example source plugin? Ideally with the complexity of the setup you describe above, but minified as much as feasible. Here’s an example first-party plugin that we have for Pylint to audit that our support works: https://github.com/pantsbuild/pants/blob/fdac7c3e26a99fc57707e6193b4f27a1c19cee17/src/python/pants/backend/python/lint/pylint/rules_integration_test.py#L333-L424. It’s helpful for the test to be involved, like living in multiple source files and having dependencies on 3rd party reqs, etc
a

acceptable-guitar-79854

09/09/2020, 9:21 AM
i love mypy more than python. Yes, I can put together a gist. If the setup is not very complicated I can try forking the repo and do it in a branch.
w

wooden-thailand-8386

09/09/2020, 12:45 PM
hey @acceptable-guitar-79854 not sure if this will help you or not but I also need to run everything under proxy (have http_proxy/https_proxy/ca_requests_bundle and another infinite amount of envs that I have to set) and what I had to do to have pants work was comment this line:
Copy code
# export no_proxy='*'
a

acceptable-guitar-79854

09/09/2020, 5:45 PM
hey @wooden-thailand-8386 i have that commented out, but the problem persists. Actually I tried with and without it. Also, I set http_proxy, https_proxy and no_proxy in my .bashrc and at the top of the "pants" script
thank you
h

hundreds-father-404

09/09/2020, 5:46 PM
Thanks for the suggestion, Thales! Yeah, this works for Thales because they’re using the v1 engine which is not hermetic, but Cristian is using the v2 engine. @happy-kitchen-89482 is working on fixing this today
a

acceptable-guitar-79854

09/09/2020, 5:46 PM
thanks for clarifying @hundreds-father-404
and @happy-kitchen-89482
h

happy-kitchen-89482

09/09/2020, 5:48 PM
@wooden-thailand-8386 what is in your
ca_requests_bundle
? Is it a .pem file with several concatenated certificates?
w

wooden-thailand-8386

09/09/2020, 5:53 PM
@happy-kitchen-89482 it is a .crt file with a lot of concatenated certs
h

happy-kitchen-89482

09/09/2020, 5:54 PM
cool, so each cert is a Base64-encoded PEM-format I assume
w

wooden-thailand-8386

09/09/2020, 5:55 PM
yes, just checked on an online decoder tool haha
h

hundreds-father-404

09/09/2020, 9:15 PM
Update @acceptable-guitar-79854 that MyPy will now work with 3rdparty libraries and plugins (PEP 561). If you have a moment, it’d be helpful if you could please look over the test cases and think if you know of any other arrangement of 3rd party requirements that Pants should support. https://github.com/pantsbuild/pants/pull/10741
a

acceptable-guitar-79854

09/09/2020, 9:30 PM
looking now
7 Views