gentle-guitar-84783
03/09/2022, 7:27 PMpython_requirements()
macro doesn't seem to be picking it up when I run my tests? do I need to refresh the cache or something or at the minimum how can I debug this situation (is there a goal to validate what the macro expands to?)hundreds-father-404
03/09/2022, 7:28 PMgentle-guitar-84783
03/09/2022, 7:29 PMsrc/python
root
the other dependencies there are working just finehundreds-father-404
03/09/2022, 7:32 PMrm -rf .pids
to force Pantsd to restart
What's going on is that python_requirements
was using an old macro system that predates the Rules API, and it does not properly use the file watching everything else uses with Pants. There's an option [GLOBAL].pantsd_invalidation_globs
that forces Pantsd to restart when the file changes, but the default doesn't include every location like src/python/reqs.txt
We fixed this in Pants 2.10, which is very close to a stable release 🙂gentle-guitar-84783
03/09/2022, 7:33 PMpantsd_invalidation_globs = ["/src/python/requirements.txt"]
to fix this in the future?hundreds-father-404
03/09/2022, 7:34 PMrm -rf .pids
does fix this. If so, then two options:
1. Stick with your current release and add pantsd_invalidation_globs.add = ["src/python/reqs.txt"]
2. Use Pants 2.10.0rc3 and follow the instructions to switch to the fixed "target generator" mechanismgentle-guitar-84783
03/09/2022, 7:35 PMfrom lib import mod
statements, will that be fixed in the future or not planned to be supported?hundreds-father-404
03/09/2022, 7:37 PMimport path.to.module
and from from path.to.module import Foo
Have you seen https://www.pantsbuild.org/docs/troubleshooting#import-errors-and-missing-dependencies for how to debug missing imports? https://www.pantsbuild.org/docs/troubleshooting#import-errors-and-missing-dependenciesgentle-guitar-84783
03/09/2022, 7:39 PMhundreds-father-404
03/09/2022, 7:40 PMmodule_mapping
gentle-guitar-84783
03/09/2022, 7:42 PME ModuleNotFoundError: No module named 'pyspark.sql.SparkSession'
src/python:pyspark
module_mapping
for some other dependencieshundreds-father-404
03/09/2022, 7:48 PMimport pyspark
, then run ./pants dependencies path/to/foo.py
to see if it shows up?./pants peek src/python:pyspark
to make sure that target looks goodgentle-guitar-84783
03/09/2022, 7:50 PM❯ ./pants peek src/python:pyspark
[
{
"address": "src/python:pyspark",
"target_type": "python_requirement",
"dependencies": [
"src/python:requirements.txt"
],
"dependencies_raw": [
":requirements.txt"
],
"description": null,
"modules": null,
"requirements": [
"pyspark==3.1.2"
],
"tags": null,
"type_stub_modules": null
}
]
hundreds-father-404
03/09/2022, 7:51 PMgentle-guitar-84783
03/09/2022, 7:53 PMhundreds-father-404
03/09/2022, 7:54 PMpython_source
/ python_sources
target? To confirm that fixes thingsgentle-guitar-84783
03/09/2022, 7:55 PM./pants dependencies src/python/path:target
which wasn't showing the dependencies correctly (which makes sense I guess since the target's dependency is just the file)
but when I tried it by pointing at the file the correct dependencies are showing up, which means I'm prob importing it incorrectly in my testshundreds-father-404
03/09/2022, 7:56 PMdependencies --transitive
gentle-guitar-84783
03/09/2022, 7:56 PMhundreds-father-404
03/09/2022, 7:56 PMdependees
goal 🙂gentle-guitar-84783
03/09/2022, 7:57 PMhundreds-father-404
03/09/2022, 7:58 PMgentle-guitar-84783
03/09/2022, 7:59 PMdependencies --transitive
is showing the correct dependencies, but it doesn't seem to be building the requirements when running the tests?hundreds-father-404
03/09/2022, 8:07 PMbut it doesn't seem to be building the requirements when running the tests?What do you mean? Is this based on Pants's logging when building requirements.txt for example?
gentle-guitar-84783
03/09/2022, 8:12 PMhundreds-father-404
03/09/2022, 8:14 PM--no-local-cache --no-pantsd
to force Pants to rebuild the requirements PEX? That's a mission-critical bug if that actually makes a difference..but to be suregentle-guitar-84783
03/09/2022, 8:17 PMWARNING: Discarding <https://files.pythonhosted.org/packages/89/db/e18cfd78e408de957821ec5ca56de1250645b05f8523d169803d8df35a64/pyspark-3.1.2.tar.gz#sha256=5e25ebb18756e9715f4d26848cc7e558035025da74b4fc325a0ebc05ff538e65> (from <https://pypi.org/simple/pyspark/>) (requires-python:>=3.6). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement pyspark==3.1.2
CPython==3.7.*
hundreds-father-404
03/09/2022, 8:19 PM--no-process-cleanup
https://www.pantsbuild.org/docs/troubleshooting#debug-tip-inspect-the-sandbox-with---no-process-cleanup
A PEX file can be inspected with unzip
. There is a file PEX-INFO
at the root that can say if pyspark is included. So you can inspect requirements.pexgentle-guitar-84783
03/09/2022, 8:22 PMpyenv
I might need to bootstrap the interpreter by handhundreds-father-404
03/09/2022, 8:24 PMgentle-guitar-84783
03/09/2022, 8:25 PM["<PYENV>"]
More than likely it was the first change that fixed this issue, but the pex built correctly, gonna run the tests againhundreds-father-404
03/09/2022, 8:29 PM--no-process-cleanup
?gentle-guitar-84783
03/09/2022, 8:29 PMhundreds-father-404
03/09/2022, 8:30 PMunzip
to look at the requirements.pex
?gentle-guitar-84783
03/09/2022, 8:31 PMhundreds-father-404
03/09/2022, 8:35 PMgentle-guitar-84783
03/09/2022, 8:36 PM