https://pantsbuild.org/ logo
Join Slack
Powered by
Trying to understand how `python_library` /`python...
# development
w
Trying to understand how 
python_library
/`python_tests` get resolved as we move our tests to v2 seeing an interesting issue when bringing in our common test utils
current source tree. is:
Copy code
src/
   testing/
   watch/
   common/
     watch_join_subscription/
      BUILD
that spacing is weird: the join directory is a sibling of the rest
BUILD file contents:
Untitled
process execution dir:
Untitled
the 
integration
and 
end_2_end
folders are part of 
src/testing
but they are promoted to the root unlike the other lib dependencies that get their own directories
this is causing errors:
Copy code
from testing.dataframe_test_case import DataframeTestCase
E   ModuleNotFoundError: No module named 'testing'
why would this behave differently
h
Hello! Have you configured 
[source].root_patterns
by chance in 
pants.toml
?
w
i have!
Copy code
[source]
root_patterns = [
  'src',
  'build-support/plugins'
]
h
Cool, so then that is not an issue that the default source roots include 
/
(top-level). Thanks for confirming
w
one thig is that our testing bundle is the rare. python library with out a 
setup_py
field
but im not clear that is a distincttion that mmatters
h
one thig is that our testing bundle is the rare. python library with out a setup_py field
That’s totally fine. Most of our 
python_library
targets internally don’t have a 
provides=setup_py
field. And in fact, in 2.0, we made a distinct 
python_distribution
target specifically for 
setup-py
. It sounds like a common misconception is that 
python_library
corresponds to something you export to end users, e.g. the whole package. It can be that, but it need not be. It’s really any code that is source code, even if purely internal and never packaged. It’s probably too late in the game for us to change this, but I wonder if 
python_sources
might have been a clearer name than 
python_library
.
w
oh wait a minute i think you just pointed me in the right direction. 
src/testing
used to be 
testing
and i added it to the source root before deciding to move it into 
src
i forgot to save my pants.toml 🙈
💯 1
h
Okay awesome, that sounds very plausible. From Pants’s perspective, with source roots, it doesn’t care about 
python_library
vs 
jvm_library
vs 
resources
etc. It always strips the source root. (Or, more recently, sets PYTHONPATH to avoid stripping so that you keep the full filename in the tool’s output) The only exception is a 
files()
target. That always preserves its source roots.
w
yup makes sense looks like i got some more passing. test targets
❤️ 1
will continue to work through it
🤞 1
h
Let us know if we can help with anything 🙂 I’ll be releasing 1.30.1 today
w
is there a way to limit. paralleliism on the test run?
h
There is. There’s probably a better spot for us to document this, but check out https://www.pantsbuild.org/docs/using-pants-in-ci#pantscitoml If you’re using 1.30, you’ll want to limit 
[python-setup].resolver_jobs
. We set it to a sensible default in 2.0, but didn’t in 1.30 because you want a higher number if you’re using the v1 engine, but lower if using the v2 engine
w
yeah we run a lot of tests with Spark contexts and spinninng up a ton of them definitely taxes the system
👍 1
probablly want to control the parallelism
to make it use half the cores or someething
h
Ik this isn’t helpful right now and Chicken And The Egg problem, but, you’ll likely find you need resolve requirements a lot in 1.30, which is frustrating and slow. That’s solved in 2.0 if you use the constraints file to only resolve once
w
gotcha. the resolver isnt the problem in this case tho! which is nice
❤️ 1
last question: is there a way to set common env vars for all tests targets
we currently run tests in v1 like 
ENV1=val1 ENV2=val2 ./pants test <targets>
obviously that no longer works
h
Yes in 2.0, unfortunately not in 1.30 😬 It sounds like that might be a blocker?
w
yeah 😞
this is unfortuate because part of the reason we're doing this is that moto is yet again creating conflicts in v1 when trying to resolve itself with mutliple targets running
since we only. use moto during tests. we. figured that v2 test transition would be the easiest path forward
h
Agreed, thinking about if there is any workaround and asked a couple others if they know of any. Worst case, we’ll cherry-pick into 1.30. It’s not a very clean cherry-pick, but ik the jump from 1.30 to 2.0 is a big one to do in one go, and we’re happy to do what we can to try to make the migration smoother.
h
@wonderful-iron-54019 what kinds of environment variables do you need to pass through to your tests?
w
Copy code
OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES \
        SPARK_MASTER=local[1] \
        TZ=UTC \
        ENV=dev \
        DATA_BUCKET=test-bucket \
        EXTERNAL_DATA_BUCKET=external-bucket \
        ./pants --no-v1 test ${PANTS_TARGETS} \
        --pytest-args="${FILTER_OPTION} ${MARKS_OPTION}"
👍 2
that's our command
h
Okay, I’ll backport this feature today to 1.30. We didn’t have all the same infrastructure for “uncachable rules” in 1.30, so there’s going to be a limitation that when you use Pantsd, changes to your env var won’t get picked up in the same Pantsd runs. But, if you restart Pantsd, it will pick it up. I don’t suspect this will impact you, as it looks like you’re hardcoding the env vars, but it’s a gotcha when iterating locally. (Fixed in 2.0)
w
yup that's ok
appreciate it
❤️ 1
feel like ive been. so high maintenance 😩
gimme a shout once the backport/release is complette
will i need to pass the envvars. a different way?
h
if you're having issues, it's likely other people are too. we think of this as you helping us make a better piece of software 🙂
w
or is just popping those on the command sufficientt
h
To the contrary, thank you for sharing all this feedback. Our general philosophy is that if you are having a certain issue with your builds, it’s almost guaranteed another org will have the same issue. Again, we know jumping from 1.30 to 2.0 is not great, and lots of Chicken And Egg. It’s helpful for us to know what features are worthy enough to backport.
h
in 2.0 there's a new pants option 
--test-extra-env-vars
that lets you explicitly specify what environment variables you want to use in the test
w
and will. that move over to the backport?
h
so you'd specify 
--test-extra-env-vars='[TZ=UTC, ENV=dev]'
, etc.
h
will i need to pass the envvars. a different way?
See https://www.pantsbuild.org/docs/python-test-goal#setting-environment-variables You could either use an allowlist approach, or you could haradcode them in 
<http://pants.ci|pants.ci>.toml
rather than having them set in the script
w
yeah that would be ideal tbh
the 
toml
path
h
yeah you can specify that option in the toml file, like any other pants option
h
and will. that move over to the backport?
Yep. The backport is to add that subsection I linked to “Setting environment variables”, with the nuance about Pantsd caching not working how we’d want for this feature.
w
perfect
we're off pantsd. for the. time being so that nuance is noted
but willl likely not affect us
h
Cool, sg. We’ve been putting lots of polish into Pantsd in 2.0, including better signal handling and working on allowing you to use concurrent runs. You’ll almost certainly want it on in 2.0, but makes sense for 1.30
FYI we decided to still release 1.30.1 today, so this backport will go into 1.30.2rc0 (hopefully today right after)
❤️ 1
w
i'll stay tuned 🙂
h
w
nice nice nice
h
Okay release prep: https://github.com/pantsbuild/pants/pull/10846 Gonna log off for volunteering, but will finish the 1.30.1 and 1.30.2rc0 releases tonight. (And 2.0.0b1 if all goes right..)
w
Got all my tests. passing! Thanks for the help folks 🙏🏼
🔥 2
Open in Slack
PreviousNext
Powered by