https://pantsbuild.org/ logo
w

wonderful-iron-54019

09/23/2020, 5:05 PM
Trying to understand how
python_library
/`python_tests` get resolved as we move our tests to v2 seeing an interesting issue when bringing in our common test utils
current source tree. is:
Copy code
src/
   testing/
   watch/
   common/
     watch_join_subscription/
      BUILD
that spacing is weird: the join directory is a sibling of the rest
BUILD file contents:
process execution dir:
the
integration
and
end_2_end
folders are part of
src/testing
but they are promoted to the root unlike the other lib dependencies that get their own directories
this is causing errors:
Copy code
from testing.dataframe_test_case import DataframeTestCase
E   ModuleNotFoundError: No module named 'testing'
why would this behave differently
h

hundreds-father-404

09/23/2020, 5:15 PM
Hello! Have you configured
[source].root_patterns
by chance in
pants.toml
?
w

wonderful-iron-54019

09/23/2020, 5:16 PM
i have!
Copy code
[source]
root_patterns = [
  'src',
  'build-support/plugins'
]
h

hundreds-father-404

09/23/2020, 5:17 PM
Cool, so then that is not an issue that the default source roots include
/
(top-level). Thanks for confirming
w

wonderful-iron-54019

09/23/2020, 5:18 PM
one thig is that our testing bundle is the rare. python library with out a
setup_py
field
but im not clear that is a distincttion that mmatters
h

hundreds-father-404

09/23/2020, 5:20 PM
one thig is that our testing bundle is the rare. python library with out a setup_py field
That’s totally fine. Most of our
python_library
targets internally don’t have a
provides=setup_py
field. And in fact, in 2.0, we made a distinct
python_distribution
target specifically for
setup-py
. It sounds like a common misconception is that
python_library
corresponds to something you export to end users, e.g. the whole package. It can be that, but it need not be. It’s really any code that is source code, even if purely internal and never packaged. It’s probably too late in the game for us to change this, but I wonder if
python_sources
might have been a clearer name than
python_library
.
w

wonderful-iron-54019

09/23/2020, 5:21 PM
oh wait a minute i think you just pointed me in the right direction.
src/testing
used to be
testing
and i added it to the source root before deciding to move it into
src
i forgot to save my pants.toml 🙈
💯 1
h

hundreds-father-404

09/23/2020, 5:23 PM
Okay awesome, that sounds very plausible. From Pants’s perspective, with source roots, it doesn’t care about
python_library
vs
jvm_library
vs
resources
etc. It always strips the source root. (Or, more recently, sets PYTHONPATH to avoid stripping so that you keep the full filename in the tool’s output) The only exception is a
files()
target. That always preserves its source roots.
w

wonderful-iron-54019

09/23/2020, 5:23 PM
yup makes sense looks like i got some more passing. test targets
❤️ 1
will continue to work through it
🤞 1
h

hundreds-father-404

09/23/2020, 5:25 PM
Let us know if we can help with anything 🙂 I’ll be releasing 1.30.1 today
w

wonderful-iron-54019

09/23/2020, 5:27 PM
is there a way to limit. paralleliism on the test run?
h

hundreds-father-404

09/23/2020, 5:28 PM
There is. There’s probably a better spot for us to document this, but check out https://www.pantsbuild.org/docs/using-pants-in-ci#pantscitoml If you’re using 1.30, you’ll want to limit
[python-setup].resolver_jobs
. We set it to a sensible default in 2.0, but didn’t in 1.30 because you want a higher number if you’re using the v1 engine, but lower if using the v2 engine
w

wonderful-iron-54019

09/23/2020, 5:29 PM
yeah we run a lot of tests with Spark contexts and spinninng up a ton of them definitely taxes the system
👍 1
probablly want to control the parallelism
to make it use half the cores or someething
h

hundreds-father-404

09/23/2020, 5:30 PM
Ik this isn’t helpful right now and Chicken And The Egg problem, but, you’ll likely find you need resolve requirements a lot in 1.30, which is frustrating and slow. That’s solved in 2.0 if you use the constraints file to only resolve once
w

wonderful-iron-54019

09/23/2020, 5:31 PM
gotcha. the resolver isnt the problem in this case tho! which is nice
❤️ 1
last question: is there a way to set common env vars for all tests targets
we currently run tests in v1 like
ENV1=val1 ENV2=val2 ./pants test <targets>
obviously that no longer works
h

hundreds-father-404

09/23/2020, 5:37 PM
Yes in 2.0, unfortunately not in 1.30 😬 It sounds like that might be a blocker?
w

wonderful-iron-54019

09/23/2020, 5:37 PM
yeah 😞
this is unfortuate because part of the reason we're doing this is that moto is yet again creating conflicts in v1 when trying to resolve itself with mutliple targets running
since we only. use moto during tests. we. figured that v2 test transition would be the easiest path forward
h

hundreds-father-404

09/23/2020, 5:43 PM
Agreed, thinking about if there is any workaround and asked a couple others if they know of any. Worst case, we’ll cherry-pick into 1.30. It’s not a very clean cherry-pick, but ik the jump from 1.30 to 2.0 is a big one to do in one go, and we’re happy to do what we can to try to make the migration smoother.
h

hundreds-breakfast-49010

09/23/2020, 5:50 PM
@wonderful-iron-54019 what kinds of environment variables do you need to pass through to your tests?
w

wonderful-iron-54019

09/23/2020, 5:50 PM
Copy code
OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES \
        SPARK_MASTER=local[1] \
        TZ=UTC \
        ENV=dev \
        DATA_BUCKET=test-bucket \
        EXTERNAL_DATA_BUCKET=external-bucket \
        ./pants --no-v1 test ${PANTS_TARGETS} \
        --pytest-args="${FILTER_OPTION} ${MARKS_OPTION}"
👍 2
that's our command
h

hundreds-father-404

09/23/2020, 5:53 PM
Okay, I’ll backport this feature today to 1.30. We didn’t have all the same infrastructure for “uncachable rules” in 1.30, so there’s going to be a limitation that when you use Pantsd, changes to your env var won’t get picked up in the same Pantsd runs. But, if you restart Pantsd, it will pick it up. I don’t suspect this will impact you, as it looks like you’re hardcoding the env vars, but it’s a gotcha when iterating locally. (Fixed in 2.0)
w

wonderful-iron-54019

09/23/2020, 5:53 PM
yup that's ok
appreciate it
❤️ 1
feel like ive been. so high maintenance 😩
gimme a shout once the backport/release is complette
will i need to pass the envvars. a different way?
h

hundreds-breakfast-49010

09/23/2020, 5:55 PM
if you're having issues, it's likely other people are too. we think of this as you helping us make a better piece of software 🙂
w

wonderful-iron-54019

09/23/2020, 5:55 PM
or is just popping those on the command sufficientt
h

hundreds-father-404

09/23/2020, 5:55 PM
To the contrary, thank you for sharing all this feedback. Our general philosophy is that if you are having a certain issue with your builds, it’s almost guaranteed another org will have the same issue. Again, we know jumping from 1.30 to 2.0 is not great, and lots of Chicken And Egg. It’s helpful for us to know what features are worthy enough to backport.
h

hundreds-breakfast-49010

09/23/2020, 5:56 PM
in 2.0 there's a new pants option
--test-extra-env-vars
that lets you explicitly specify what environment variables you want to use in the test
w

wonderful-iron-54019

09/23/2020, 5:56 PM
and will. that move over to the backport?
h

hundreds-breakfast-49010

09/23/2020, 5:56 PM
so you'd specify
--test-extra-env-vars='[TZ=UTC, ENV=dev]'
, etc.
h

hundreds-father-404

09/23/2020, 5:57 PM
will i need to pass the envvars. a different way?
See https://www.pantsbuild.org/docs/python-test-goal#setting-environment-variables You could either use an allowlist approach, or you could haradcode them in
<http://pants.ci|pants.ci>.toml
rather than having them set in the script
w

wonderful-iron-54019

09/23/2020, 5:57 PM
yeah that would be ideal tbh
the
toml
path
h

hundreds-breakfast-49010

09/23/2020, 5:57 PM
yeah you can specify that option in the toml file, like any other pants option
h

hundreds-father-404

09/23/2020, 5:57 PM
and will. that move over to the backport?
Yep. The backport is to add that subsection I linked to “Setting environment variables”, with the nuance about Pantsd caching not working how we’d want for this feature.
w

wonderful-iron-54019

09/23/2020, 5:57 PM
perfect
we're off pantsd. for the. time being so that nuance is noted
but willl likely not affect us
h

hundreds-father-404

09/23/2020, 6:02 PM
Cool, sg. We’ve been putting lots of polish into Pantsd in 2.0, including better signal handling and working on allowing you to use concurrent runs. You’ll almost certainly want it on in 2.0, but makes sense for 1.30
FYI we decided to still release 1.30.1 today, so this backport will go into 1.30.2rc0 (hopefully today right after)
❤️ 1
w

wonderful-iron-54019

09/23/2020, 6:48 PM
i'll stay tuned 🙂
h

hundreds-father-404

09/23/2020, 7:13 PM
w

wonderful-iron-54019

09/23/2020, 7:15 PM
nice nice nice
h

hundreds-father-404

09/23/2020, 10:05 PM
Okay release prep: https://github.com/pantsbuild/pants/pull/10846 Gonna log off for volunteering, but will finish the 1.30.1 and 1.30.2rc0 releases tonight. (And 2.0.0b1 if all goes right..)
w

wonderful-iron-54019

09/24/2020, 1:46 PM
Got all my tests. passing! Thanks for the help folks 🙏🏼
🔥 2