https://pantsbuild.org/ logo
Powered by
d

dry-analyst-73584

01/27/2022, 7:29 PM
It looks like pants is correctly inferring all our deps except one (pymongo), for reasons I do not understand. my requirements.txt has 
pymongo==3.7.1
and it's being imported as 
import pymongo
in a bunch of places, but for some reason when I run tests I'm getting
Copy code
ImportError while importing test module '/private/var/folders/yv/1kp6r8h57p91lxsszfdppwsw0000gn/T/process-executionGbSusG/test/python/foursquare_test/mongo/lib/handles_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
test/python/foursquare_test/mongo/lib/handles_test.py:10: in <module>
    from foursquare.mongo import mongo_ops
src/python/foursquare/mongo/mongo_ops.py:13: in <module>
    import pymongo
E   ImportError: No module named pymongo
It seems to be specific to that 
mongo_ops.py
file. Everything that depends on mongo_ops.py fails with the import error. But every other file that has 
import pymongo
is working fine. I can fix it by editing the BUILD file:
Copy code
python_sources(
    dependencies = [
        "3rdparty/python:pymongo",
    ],
)
But it seems extremely odd that I would need to.
h

high-yak-85899

01/27/2022, 7:30 PM
I had this happen with 
requests
Still unknown why but at least it's unblocking.
d

dry-analyst-73584

01/27/2022, 7:31 PM
That's bizarre. I wonder what's going on.
➕ 1
h

high-yak-85899

01/27/2022, 7:32 PM
a really naive guess is that there are hidden unicode characters that editors are leaving out that make the matching fail
But that doesn't really explain why the module_mapping stuff fixes it I guess
I just know I've had that happen in completely other things where a file looked right and wasn't until you actually inspected the encoded form that you saw why it didn't work.
r

refined-addition-53644

01/27/2022, 7:35 PM
Worth checking the dependencies (also with --transitive)
Copy code
./pants dependencies src/python/foursquare/mongo/mongo_ops.py
b

bitter-ability-32190

01/27/2022, 7:37 PM
Can either of y'all share the file that reproduces the issue? Feel free to remove as much noise while still being able to reproduce 😉
Oh and what version of Pants
h

high-yak-85899

01/27/2022, 7:41 PM
It would take me a while to recover so I'll leave it to Tansy (was prototyping a lot at the time and now cleaning it all up). I was on 2.9.0
d

dry-analyst-73584

01/27/2022, 7:42 PM
I'm on 2.9 and just deleted the contents of the file, leaving only the import. I'll see if it repros.
What's especially weird is that in this directory I have multiple files that 
import pymongo
and they work just fine, but this one doesn't.
This one was doing some weirdness with
Copy code
import pymongo
import pymongo.errors
from pymongo.thing import SomeThingElse
Which was gross, so I fixed it to only have 
import pymongo
I thought maybe it had cached some bad state, but I've been running it with 
--no-pantsd
and 
--no-local-cache
to check and that didn't fix it either.
b

bitter-ability-32190

01/27/2022, 7:47 PM
I have a suspicion for @dry-analyst-73584’s issue, but it wouldn't explain @high-yak-85899’s or why it worked for some files but not others
d

dry-analyst-73584

01/27/2022, 7:53 PM
Ok, here is the full output of my pants run.
output.txt
And here is the file that is having issues. I deleted the body of the file, but it still repros just fine.
mongo_ops.py
And the BUILD file. The relevant bit is just 
python_sources()
BUILD2
Some of the files in the same directory import stuff like this:
Copy code
# coding=utf-8
# Copyright 2014 Foursquare Labs Inc. All Rights Reserved.

from __future__ import absolute_import, division, print_function

import datetime
import logging
import os
import shlex
import shutil
import socket
import subprocess
import urllib2
import xmlrpclib

import argparse
import motor
import pymongo
import supervisor.supervisord
import supervisor.xmlrpc
from tornado import gen
import tornado.ioloop
import tornado.web
import toro
And are working just fine.
b

bitter-ability-32190

01/27/2022, 7:59 PM
So I think there's a bug in 
pants
specifically for 
pymongo
I can reproduce on 
main
.
👀 1
@dry-analyst-73584 for the other files that 
import pymongo
and are working fine: • I assume some tests are importing those files and not failing to import • Do you know if any file in those passing tests directly or indirectly import either 
bson
or 
gridfs
?
d

dry-analyst-73584

01/27/2022, 8:07 PM
None of them are importing it directly. transitively I'm not so sure about yet.
b

bitter-ability-32190

01/27/2022, 8:09 PM
I think this would fix your issue: https://github.com/pantsbuild/pants/pull/14284
I'll make a PR to backport to 2.9.
(It doesn't explain @high-yak-85899’s issue though 😞 )
h

high-yak-85899

01/27/2022, 8:11 PM
Isn't that PR the same as me adjusting 
module_mappings
on my own locally?
d

dry-analyst-73584

01/27/2022, 8:11 PM
Thanks Joshua! Any idea what causes certain things to not import like this?
b

bitter-ability-32190

01/27/2022, 8:12 PM
Yes-ish. However 
requests
isn't in Pants' default list, so it isn't the same issue 😕
@dry-analyst-73584 So as best I can tell, the tests that fail only ever 
import pymongo
. The linked PR is Pants' list of "these packages map to these modules". It's used for cases like 
pyyaml
where the package name is 
pyyaml
but the module name is 
yaml
. Before the PR, 
pants
though the 
pymongo
package only contained the modules 
bson
and 
gridfs
. So if you imported those (directly or indirectly) 
pymongo
would be in your tests' env. If you didn't import either of those (and just imported 
pymongo
) Pants wouldn't know where 
pymongo
came from.
👍 2
FWIW I added the https://www.pantsbuild.org/docs/reference-python-infer#section-unowned-dependency-behavior flag to catch these things early (when doing dep inference) instead of late (when running tests)
❤️ 1
d

dry-analyst-73584

01/27/2022, 8:17 PM
Awesome! Thank you!
h

happy-kitchen-89482

01/27/2022, 8:17 PM
Cool, thanks for the fix @bitter-ability-32190! Will merge when CI goes green.
Still unclear on @high-yak-85899’s issue, that one's perplexing
➕ 1
b

bitter-ability-32190

01/27/2022, 8:21 PM
h

happy-kitchen-89482

01/27/2022, 8:47 PM
@dry-analyst-73584 meanwhile the workaround is to update that mapping in your pants.toml, ya?
h

high-yak-85899

01/27/2022, 8:48 PM
That mapping I did is in the BUILD file.
d

dry-analyst-73584

01/27/2022, 8:49 PM
👍 Adding 
dependencies=["pymongo"]
also worked
h

happy-kitchen-89482

01/27/2022, 8:53 PM
Yeah but that will only fix things for that one target, so it's a land mine waiting to happen elsewhere
d

dry-analyst-73584

01/27/2022, 8:54 PM
Ah, fair enough
h

high-yak-85899

01/27/2022, 8:55 PM
The fix I did goes at the top level where 
python_requirements()
is defined so it applies to all files. Certainly not a fix for all modules and doesn't help in the case where you make your own 
python_requirement
much closer to the target that consumes it.
➕ 1
d

dry-analyst-73584

01/27/2022, 8:58 PM
👍 We won't be making any other python_requirements anywhere else, so this should do just fine! Thanks!
Fun fact: the file with the import issue is referencing mongo clusters that haven't existed in ~3 years. The tests have been passing this whole time because they were cached in V1 pants. Hooray V2 for helping us find some extremely dead code.
h

happy-kitchen-89482

01/28/2022, 1:01 AM
hahahaha that is hilarious
b

bitter-ability-32190

01/28/2022, 1:09 AM
A cache that has existed for 3 years is more impressive to me 😮
d

dry-analyst-73584

01/28/2022, 5:39 PM
We had a build that ran all our tests without the cache - “test-cache-sanity” but it got turned off a few years ago and no one knows why. I think V2 has a better handle on caching the results of network calls?
h

happy-kitchen-89482

01/28/2022, 5:43 PM
IIRC it only caches the results of network calls when the result matches a known sha.
2 Views
Powered by