https://pantsbuild.org/ logo
Join Slack
Powered by
It looks like pants is correctly inferring all our...
# general
d
It looks like pants is correctly inferring all our deps except one (pymongo), for reasons I do not understand. my requirements.txt has 
pymongo==3.7.1
and it's being imported as 
import pymongo
in a bunch of places, but for some reason when I run tests I'm getting
Copy code
ImportError while importing test module '/private/var/folders/yv/1kp6r8h57p91lxsszfdppwsw0000gn/T/process-executionGbSusG/test/python/foursquare_test/mongo/lib/handles_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
test/python/foursquare_test/mongo/lib/handles_test.py:10: in <module>
    from foursquare.mongo import mongo_ops
src/python/foursquare/mongo/mongo_ops.py:13: in <module>
    import pymongo
E   ImportError: No module named pymongo
It seems to be specific to that 
mongo_ops.py
file. Everything that depends on mongo_ops.py fails with the import error. But every other file that has 
import pymongo
is working fine. I can fix it by editing the BUILD file:
Copy code
python_sources(
    dependencies = [
        "3rdparty/python:pymongo",
    ],
)
But it seems extremely odd that I would need to.
h
I had this happen with 
requests
Still unknown why but at least it's unblocking.
d
That's bizarre. I wonder what's going on.
➕ 1
h
a really naive guess is that there are hidden unicode characters that editors are leaving out that make the matching fail
But that doesn't really explain why the module_mapping stuff fixes it I guess
I just know I've had that happen in completely other things where a file looked right and wasn't until you actually inspected the encoded form that you saw why it didn't work.
r
Worth checking the dependencies (also with --transitive)
Copy code
./pants dependencies src/python/foursquare/mongo/mongo_ops.py
b
Can either of y'all share the file that reproduces the issue? Feel free to remove as much noise while still being able to reproduce 😉
Oh and what version of Pants
h
It would take me a while to recover so I'll leave it to Tansy (was prototyping a lot at the time and now cleaning it all up). I was on 2.9.0
d
I'm on 2.9 and just deleted the contents of the file, leaving only the import. I'll see if it repros.
What's especially weird is that in this directory I have multiple files that 
import pymongo
and they work just fine, but this one doesn't.
This one was doing some weirdness with
Copy code
import pymongo
import pymongo.errors
from pymongo.thing import SomeThingElse
Which was gross, so I fixed it to only have 
import pymongo
I thought maybe it had cached some bad state, but I've been running it with 
--no-pantsd
and 
--no-local-cache
to check and that didn't fix it either.
b
I have a suspicion for @dry-analyst-73584’s issue, but it wouldn't explain @high-yak-85899’s or why it worked for some files but not others
d
Ok, here is the full output of my pants run.
output.txt
And here is the file that is having issues. I deleted the body of the file, but it still repros just fine.
mongo_ops.py
And the BUILD file. The relevant bit is just 
python_sources()
BUILD2
Some of the files in the same directory import stuff like this:
Copy code
# coding=utf-8
# Copyright 2014 Foursquare Labs Inc. All Rights Reserved.

from __future__ import absolute_import, division, print_function

import datetime
import logging
import os
import shlex
import shutil
import socket
import subprocess
import urllib2
import xmlrpclib

import argparse
import motor
import pymongo
import supervisor.supervisord
import supervisor.xmlrpc
from tornado import gen
import tornado.ioloop
import tornado.web
import toro
And are working just fine.
b
So I think there's a bug in 
pants
specifically for 
pymongo
I can reproduce on 
main
.
👀 1
@dry-analyst-73584 for the other files that 
import pymongo
and are working fine: • I assume some tests are importing those files and not failing to import • Do you know if any file in those passing tests directly or indirectly import either 
bson
or 
gridfs
?
d
None of them are importing it directly. transitively I'm not so sure about yet.
b
I think this would fix your issue: https://github.com/pantsbuild/pants/pull/14284
I'll make a PR to backport to 2.9.
(It doesn't explain @high-yak-85899’s issue though 😞 )
h
Isn't that PR the same as me adjusting 
module_mappings
on my own locally?
d
Thanks Joshua! Any idea what causes certain things to not import like this?
b
Yes-ish. However 
requests
isn't in Pants' default list, so it isn't the same issue 😕
@dry-analyst-73584 So as best I can tell, the tests that fail only ever 
import pymongo
. The linked PR is Pants' list of "these packages map to these modules". It's used for cases like 
pyyaml
where the package name is 
pyyaml
but the module name is 
yaml
. Before the PR, 
pants
though the 
pymongo
package only contained the modules 
bson
and 
gridfs
. So if you imported those (directly or indirectly) 
pymongo
would be in your tests' env. If you didn't import either of those (and just imported 
pymongo
) Pants wouldn't know where 
pymongo
came from.
👍 2
FWIW I added the https://www.pantsbuild.org/docs/reference-python-infer#section-unowned-dependency-behavior flag to catch these things early (when doing dep inference) instead of late (when running tests)
❤️ 1
d
Awesome! Thank you!
h
Cool, thanks for the fix @bitter-ability-32190! Will merge when CI goes green.
Still unclear on @high-yak-85899’s issue, that one's perplexing
➕ 1
b
h
@dry-analyst-73584 meanwhile the workaround is to update that mapping in your pants.toml, ya?
h
That mapping I did is in the BUILD file.
d
👍 Adding 
dependencies=["pymongo"]
also worked
h
Yeah but that will only fix things for that one target, so it's a land mine waiting to happen elsewhere
d
Ah, fair enough
h
The fix I did goes at the top level where 
python_requirements()
is defined so it applies to all files. Certainly not a fix for all modules and doesn't help in the case where you make your own 
python_requirement
much closer to the target that consumes it.
➕ 1
d
👍 We won't be making any other python_requirements anywhere else, so this should do just fine! Thanks!
Fun fact: the file with the import issue is referencing mongo clusters that haven't existed in ~3 years. The tests have been passing this whole time because they were cached in V1 pants. Hooray V2 for helping us find some extremely dead code.
h
hahahaha that is hilarious
b
A cache that has existed for 3 years is more impressive to me 😮
d
We had a build that ran all our tests without the cache - “test-cache-sanity” but it got turned off a few years ago and no one knows why. I think V2 has a better handle on caching the results of network calls?
h
IIRC it only caches the results of network calls when the result matches a known sha.
2 Views
Open in Slack
PreviousNext
Powered by