https://pantsbuild.org/ logo
Powered by
g

gentle-guitar-84783

03/09/2022, 7:27 PM
qq: I've added a dependency to requirements.txt but the 
python_requirements()
macro doesn't seem to be picking it up when I run my tests? do I need to refresh the cache or something or at the minimum how can I debug this situation (is there a goal to validate what the macro expands to?)
h

hundreds-father-404

03/09/2022, 7:28 PM
Hi, where is the requirements.txt file located specifically?
g

gentle-guitar-84783

03/09/2022, 7:29 PM
in my 
src/python
root the other dependencies there are working just fine
there also doesn't seem to be a clean goal to clear up any cache of the requirements
h

hundreds-father-404

03/09/2022, 7:32 PM
Okay I suspect your intuition is right about caching -- specifically that the Pants daemon is not recognizing the change. Try 
rm -rf .pids
to force Pantsd to restart What's going on is that 
python_requirements
was using an old macro system that predates the Rules API, and it does not properly use the file watching everything else uses with Pants. There's an option 
[GLOBAL].pantsd_invalidation_globs
that forces Pantsd to restart when the file changes, but the default doesn't include every location like 
src/python/reqs.txt
We fixed this in Pants 2.10, which is very close to a stable release 🙂
g

gentle-guitar-84783

03/09/2022, 7:33 PM
ok awesome, I literally just tried that and it worked
🙌 1
so I just need to add 
pantsd_invalidation_globs = ["/src/python/requirements.txt"]
to fix this in the future?
h

hundreds-father-404

03/09/2022, 7:34 PM
It'd be great to confirm that 
rm -rf .pids
does fix this. If so, then two options: 1. Stick with your current release and add 
pantsd_invalidation_globs.add = ["src/python/reqs.txt"]
2. Use Pants 2.10.0rc3 and follow the instructions to switch to the fixed "target generator" mechanism
Great! Really sorry for the trouble, this has tripped up several people and I'm very excited for 2.10 to be released so it stops being so confusing
g

gentle-guitar-84783

03/09/2022, 7:35 PM
yup, have to add another dependency, I'll test now that the invalidation glob works
also qq while we're on 3rdparty deps: inference doesn't seem to work with 
from lib import mod
statements, will that be fixed in the future or not planned to be supported?
invalidation globs seems to works, thanks for the help
❤️ 1
h

hundreds-father-404

03/09/2022, 7:37 PM
Hm, it should be! Pants understands both 
import path.to.module
and from 
from path.to.module import Foo
Have you seen https://www.pantsbuild.org/docs/troubleshooting#import-errors-and-missing-dependencies for how to debug missing imports? https://www.pantsbuild.org/docs/troubleshooting#import-errors-and-missing-dependencies
👍 1
👀 1
g

gentle-guitar-84783

03/09/2022, 7:39 PM
I'll check it out, for some reason it's not inferring pyspark correctly
which is weird because it looks like it's inferring it correctly for other sources 🤔
h

hundreds-father-404

03/09/2022, 7:40 PM
what is the import for pyspark? I wonder if you need to set up 
module_mapping
g

gentle-guitar-84783

03/09/2022, 7:42 PM
Copy code
E   ModuleNotFoundError: No module named 'pyspark.sql.SparkSession'
python_requirement() dependency:
Copy code
src/python:pyspark
I think inference worked fine for other modules (I do need a 
module_mapping
for some other dependencies
h

hundreds-father-404

03/09/2022, 7:48 PM
hmmm fishy. purely for the sake of debugging, could you try other imports like 
import pyspark
, then run 
./pants dependencies path/to/foo.py
to see if it shows up?
also you could try running 
./pants peek src/python:pyspark
to make sure that target looks good
g

gentle-guitar-84783

03/09/2022, 7:50 PM
Copy code
❯ ./pants peek src/python:pyspark
[
  {
    "address": "src/python:pyspark",
    "target_type": "python_requirement",
    "dependencies": [
      "src/python:requirements.txt"
    ],
    "dependencies_raw": [
      ":requirements.txt"
    ],
    "description": null,
    "modules": null,
    "requirements": [
      "pyspark==3.1.2"
    ],
    "tags": null,
    "type_stub_modules": null
  }
]
so inference definitely failed, I wonder if it's because it's a pex target? does it only work for sources?
is there more documentation on inference? the only thing I've seen is in the third party deps page and it's pretty sparse
h

hundreds-father-404

03/09/2022, 7:51 PM
what do you mean pex target? the import is happening in a source file right?
g

gentle-guitar-84783

03/09/2022, 7:53 PM
yeah
h

hundreds-father-404

03/09/2022, 7:54 PM
I don't think we have more detailed docs, but you could check out the code to see what's happening: 1. For each file, we run this parser to grab your import statements https://github.com/pantsbuild/pants/blob/2.9.x/src/python/pants/backend/python/dependency_inference/import_parser.py 2. We create a module mapping of third-party and first-party code. The third-party code looks like this https://github.com/pantsbuild/pants/blob/93037980e22ba711a2594f9e43ad0356141087da/src/python/pants/backend/python/dependency_inference/module_mapper.py#L287-L368 3. We then look up for each import if it's in either module mapping, but don't infer if there's ambiguity. https://github.com/pantsbuild/pants/blob/93037980e22ba711a2594f9e43ad0356141087da/src/python/pants/backend/python/dependency_inference/module_mapper.py#L396-L443
btw have you tried explicitly adding the pyspark target to the dependencies of the 
python_source
/ 
python_sources
target? To confirm that fixes things
g

gentle-guitar-84783

03/09/2022, 7:55 PM
ok this is weird, what I meant by pex target was that I was checking 
./pants dependencies src/python/path:target
which wasn't showing the dependencies correctly (which makes sense I guess since the target's dependency is just the file) but when I tried it by pointing at the file the correct dependencies are showing up, which means I'm prob importing it incorrectly in my tests
s/correctly/what I expected/
h

hundreds-father-404

03/09/2022, 7:56 PM
Oh yeah for the pex target, you'd want to use 
dependencies --transitive
g

gentle-guitar-84783

03/09/2022, 7:56 PM
oh nice, ok that's cool
h

hundreds-father-404

03/09/2022, 7:56 PM
https://www.pantsbuild.org/docs/project-introspection has more on queries like that, including 
dependees
goal 🙂
g

gentle-guitar-84783

03/09/2022, 7:57 PM
that'll be helpful to debug why the test isn't importing the right dependencies
awesome, thank you so much Eric
❤️ 1
this should be enough for me to figure out where the hell I went wrong
h

hundreds-father-404

03/09/2022, 7:58 PM
You're welcome! I'd love to figure this out - it's crucial for Pants to be ergonomic that dep inference works well, so we're eager to know of any bugs, or ways to make docs & errors more clear
g

gentle-guitar-84783

03/09/2022, 7:59 PM
yeah so it looks like the inference worked correctly, but I'm still getting an import module error, which leads me to believe I'm just importing incorrectly
actually it looks like it's not picking up transitive dependencies?
dependencies --transitive
is showing the correct dependencies, but it doesn't seem to be building the requirements when running the tests?
h

hundreds-father-404

03/09/2022, 8:07 PM
Hm
but it doesn't seem to be building the requirements when running the tests?
What do you mean? Is this based on Pants's logging when building requirements.txt for example?
g

gentle-guitar-84783

03/09/2022, 8:12 PM
I wrote a small noop test that just imports pyspark, worked as expected, but you can see the output of it building the dependency no such thing happens when it runs my other test that depends on a file that depends on pyspark
h

hundreds-father-404

03/09/2022, 8:14 PM
hmm, you could try running with 
--no-local-cache --no-pantsd
to force Pants to rebuild the requirements PEX? That's a mission-critical bug if that actually makes a difference..but to be sure
g

gentle-guitar-84783

03/09/2022, 8:17 PM
oh maybe it was the local cache, it seems to be collecting pyspark now
nope, never mind, still got the same error 🤔
I'm trying to package the pex now to see if there's an issue there, I'm getting a weird error
Copy code
WARNING: Discarding <https://files.pythonhosted.org/packages/89/db/e18cfd78e408de957821ec5ca56de1250645b05f8523d169803d8df35a64/pyspark-3.1.2.tar.gz#sha256=5e25ebb18756e9715f4d26848cc7e558035025da74b4fc325a0ebc05ff538e65> (from <https://pypi.org/simple/pyspark/>) (requires-python:>=3.6). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement pyspark==3.1.2
👀 1
this is interesting since my interpreter constraint is 
CPython==3.7.*
h

hundreds-father-404

03/09/2022, 8:19 PM
But it looks like Pyspark was included in the requirements.pex? Here's another thing you can try, 
--no-process-cleanup
https://www.pantsbuild.org/docs/troubleshooting#debug-tip-inspect-the-sandbox-with---no-process-cleanup A PEX file can be inspected with 
unzip
. There is a file 
PEX-INFO
at the root that can say if pyspark is included. So you can inspect requirements.pex
g

gentle-guitar-84783

03/09/2022, 8:22 PM
oh blah, I think I know what's going on, this might be a problem with 
pyenv
I might need to bootstrap the interpreter by hand
👀 1
h

hundreds-father-404

03/09/2022, 8:24 PM
Huh, how come?
g

gentle-guitar-84783

03/09/2022, 8:25 PM
I changed two things, not sure which fixed the issue, but it is fixed 1. I changed the interpreter to 3.7.10 2. I changed the search_path to 
["<PYENV>"]
More than likely it was the first change that fixed this issue, but the pex built correctly, gonna run the tests again
based on the warning I assume it was picking up the wrong interpreter somehow, not sure why
test is still failing with an importerror though
^^^ that's without --no-local-cache and --no-pantsd let me test with those flags
yup, still same error
h

hundreds-father-404

03/09/2022, 8:29 PM
K how about 
--no-process-cleanup
?
g

gentle-guitar-84783

03/09/2022, 8:29 PM
ran both commands with no-process-cleanup
h

hundreds-father-404

03/09/2022, 8:30 PM
were you able to follow the linked instructions to inspect the relevant sandbox and use 
unzip
to look at the 
requirements.pex
?
g

gentle-guitar-84783

03/09/2022, 8:31 PM
it's definitely building it (or trying)
👍 1
and yeah it's in the pex
h

hundreds-father-404

03/09/2022, 8:35 PM
Okay then I have no idea what's going on. Sounds like Pants is correctly detecting the dependency & building it. At that point, this is no longer in Pants's control and it's the underlying Pytest process / Python machinery
👍 1
g

gentle-guitar-84783

03/09/2022, 8:36 PM
yeah, it's prob something I did wrong, I'll start from scratch and see what's going on, thanks for your help Eric
❤️ 1
you gave me enough tools that I don't need to flounder around as much 😛
🙌 2
2 Views
Powered by