https://pantsbuild.org/ logo
#general
Title
# general
b

brash-baker-91190

02/12/2021, 12:44 AM
I'm adding typechecking with mypy to our project, and have hit a bit of a snag. I suspect it's due to Pants' implementation, but I'm not an mypy expert by any stretch. Basically, we've got a monorepo with several Python apps and libraries. Everything but one library was typed by mypy in our pre-Pants repository; that one library was instead typed by Pytype because it's got some peculiarities that are better served by being analyzed by Pytype. Ideally, I'd like to just run mypy via Pants over everything but that one library. However, it looks like Pants supplies the files of the targeted apps and libraries, as well as their local dependencies, directly to mypy. This results in our pytype-typed library being fed into mypy for analysis, and then failing the run as a result. This seems to happen no matter what kind of filtering I provide. Is there any way of performing this kind of flitering? Thanks in advance!
h

hundreds-father-404

02/12/2021, 1:06 AM
Indeed. We could change it so that we don't say in the argv all the deps too. When I added MyPy, iirc, it behaved the same way either way, so we went with specifying every file under the thinking that it would improve the rate of cache hits But if you can show that that is not the case, that leaving off a dep from the argv does change things, then we can very easily fix this! So, my ask: could you please experiment with this and report back? No need to use Pants in your experiment, only a simple example repro
b

brash-baker-91190

02/12/2021, 1:08 AM
yeah, I'll see what I can cook up
h

hundreds-father-404

02/12/2021, 1:11 AM
Thanks! Also note that transitive deps will still always be in the chroot and I don't think we can change that
b

brash-baker-91190

02/12/2021, 1:16 AM
I think them being present should be OK. I think the key bit is whether they're passed directly to mypy for analysis
👍 1
So, I think my suspicion was correct. When I run the same mypy run manually (using all the input files that Pants gives, as determined by passing
--verbose
to mypy) I get the same failures as under Pants. When I remove the files that are part of my pytype-only package, it works (and I can configure the run to ignore imports for that package). If I even add one file of my pytype-only package, that's enough to blow things up in a way that I can't seem to filter out. So, I think passing just the files from the targeted packages seems like the right thing to do. It seems that this will work better with Pants' filtering (e.g., tags, target globbing, etc), and will also allow you to adopt mypy incrementally. This is all caveated by the fact that I've personally only started using mypy this week, so I'm certainly open to the idea that I've missed something important 😅
h

hundreds-father-404

02/12/2021, 2:01 AM
To confirm, you are saying that the change in behavior is purely based on the arguments that you pass to MyPy?
mypy project/app.py project/dep.py
vs.
mypy project/app.py
? not based on whether the file is present in the chroot?
b

brash-baker-91190

02/12/2021, 2:20 AM
I believe so, yes. I may need to do some additional poking around in the morning, but I was not manipulating any files on disk; it was purely through manipulating arguments to mypy.
💯 1
Do you have any pointers on how to use Pants to set up a chroot that it would use for executing a command, but then run arbitrary commands in it?
h

hundreds-father-404

02/12/2021, 2:26 AM
Yes! use
--no-process-execution-cleanup-local-dirs
. There will be something like
mypy.pex
in it, and the chroot of all the relevant files. You can then do
./mypy.pex arg1 arg2
b

brash-baker-91190

02/12/2021, 3:11 AM
Cool! So, I did that and discovered that I'd need to modify the
__run.sh
file and/or the
__files.txt
file. When I ran
__run.sh
pointed to a version of
__files.txt
that I had removed my pytest-only library files from, I get the behavior I'm after. I didn't change any of the files that were in the chroot. So, I think that is fairly strong support for this theory 😄
❤️ 1
💯 1
h

hundreds-father-404

02/12/2021, 3:12 AM
Excellent! Would you be interested in putting up a PR to fix this? I can type up some instructions if so
b

brash-baker-91190

02/12/2021, 3:12 AM
(trying to run
mypy.pex
directly failed because I didn't have the environment present in __run.sh)
👍 1
@hundreds-father-404 Yeah, if I had some pointers, I could take a stab at a PR for this.
💯 1
It'll likely be good practice for the handful of plugins I want to add for our repo anyway 🙃
h

hundreds-father-404

02/12/2021, 3:49 AM
Hope this helps! Let me know if you have questions! https://github.com/pantsbuild/pants/issues/11553 Check out https://www.pantsbuild.org/docs/contributor-overview for a guide on making contributions, including how to get set up. It would also be fruitful to read over https://www.pantsbuild.org/docs/plugins-overview and the Target API and Rules API overviews. It's okay if things don't make perfect sense, you should be able to monkey see, monkey do for most of this
b

brash-baker-91190

02/12/2021, 1:36 PM
This is great @hundreds-father-404... thanks very much!
❤️ 1
h

hundreds-father-404

03/12/2021, 7:49 PM
Hey @brash-baker-91190, checking if you're still interested in working on #11553? I can try to fit it in today or next week if not
b

brash-baker-91190

03/12/2021, 8:02 PM
I am still interested... I got the code working, but not the tests, then got sidetracked with some life stuff, then some code on the master branch changed and I have yet to get the resulting conflicts with my code resolved 😕 I hope to be able to get back to it soon, but I don't want to hold up anything.
❤️ 1
h

hundreds-father-404

03/12/2021, 8:04 PM
Okay sounds good, thanks for the update! I hear you there with that all, same happens to me 🤷‍♂️ And no worries, I don't think anyone is being held up
b

brash-baker-91190

03/12/2021, 8:05 PM
cool, thanks... hopefully I'll have some time this weekend to get back to it 🤞
❤️ 1
4 Views