https://pantsbuild.org/ logo
#general
Title
# general
s

strong-toothbrush-37759

03/13/2022, 8:34 PM
Hello it’s me again 😄, so after some tinkering with the settings and removing `[source].root_patterns = ["src"]`it looks like Pants is trying to build PEX-files for my tests (YEAY 🙌 ) when I run
./pants test src/core/tests/test_companies.py
for example. Question #1: In the example-django repo, the tests were placed in a
models_test.py
. When I have nested testing folders for submodules of a module, what is the best way to formulate a BUILD file inside of the module? Do I adjust the
core
BUILD file with
sources=["tests/**/test_*.py"]
or should I not worry at all and let tailor create all the BUILD-files? Question #2: I’m getting an error which I think tells me that I need to configure
root_patterns
correctly. The error is:
Copy code
Exception: String("Can only merge Directories with no duplicates, but found 2 duplicate entries in :\n\n`pyproject.toml` 1.) file digest=9d7d5d9df3e4ae5a4842a380891776c10adfc04ffdf0d48728db8ff4e7a05943 size=1006 ... 2.) file digest=55e6c945374e7d8f58b1d90e4dc00dee93a906850e953db2512938ead8b84fef size=994
I’m gonna post an example of our project-structure inside the thread here to not spam #general too much. The question is: How do I setup
source-roots
correctly? I’ve read the documentation about source-roots a couple of times now, but I’m not really sure what’s the optimal way. Maybe this error is again me missing an error inside of our
pyproject.toml
, but I hope it’s not 😅 .
1
f

fast-nail-55400

03/13/2022, 8:40 PM
Generally you would have a
BUILD
file in each module’s directory with
python_sources()
(and
python_tests(name="tests")
as applicable. Note: If you run
./pants tailor
, then Pants will write out these
BUILD
files for you automatically.
s

strong-toothbrush-37759

03/13/2022, 8:40 PM
Here’s a simplified example of the directory structure:
Copy code
src/ - This is the django root folder
  api/
  core/
    models/
      tests/
        test_companies.py - company unit-tests
      companies.py
    tests/
      test_companies.py - company integration-tests
    admin.py
    apps.py
    tasks.py
  another_module1/
  another_module2/
I’d like to be able to import like this
from core.models import Company
. In the past we adjusted the
__init__.py
inside of the
core/models
folder to look like this:
Copy code
__all__ = [
    # companies.py
    "Company",
]

from .companies import (
    Company
)
As I understand Pants, this adjustment could be a thing of the past. But that’s a question for another time. As always: Help is highly appreciated 💚
@fast-nail-55400 thanks for your response! I did run
./pants tailor
and it created all the necessary BUILD-files for me to start the tests. This is totally fine and I’d like to avoid manually adjusting auto-generated files. In any case this is me asking “should I adjust the generated BUILD-files at all or just let tailor to its job?“. I think that maybe my
root_patterns
need to be adjusted. Right now I don’t have anything setup (which allowed me to start running
./pants test src/core/tests/test_companies.py
although this runs into the error mentioned above). My roots look like this:
Copy code
❯ ./pants roots
.
src
f

fast-nail-55400

03/13/2022, 8:50 PM
Do you have multiple
pyproject.toml
at all?
(trying to understand why Pants encountered
pyproject.toml
with different content)
s

strong-toothbrush-37759

03/13/2022, 8:50 PM
No, I think that’s the roots reading
.
and
src/../
Copy code
❯ find . -name 'pyproject.toml' | xargs wc -l
      55 ./pyproject.toml
      13 ./src/venv/lib/python3.9/site-packages/spacy/tests/package/pyproject.toml
      68 total
🤔
f

fast-nail-55400

03/13/2022, 8:52 PM
I assume that venv is not checked into source control. Can you add
src/venv/**
to the --pants-ignore option?
(
[GLOBAL].pants_ignore
in the
pants.toml
)
(also I’m not that familiar with some of the more “nitty gritty” details of Pants/Python support, so this is just a guess on my part)
s

strong-toothbrush-37759

03/13/2022, 8:55 PM
I did add
"/src/venv/**",
to the
pants_ignore
array but it didn’t change the outcome of the command.
I’m trying to get into it 🙂 and getting to know other people that are active around here is always a good feeling. Thanks for trying to help @fast-nail-55400 ❤️
f

fast-nail-55400

03/13/2022, 8:57 PM
The maintainers with more experience with Pants/Python should be online once work week starts again (so tomorrow).
👌 1
🙏 1
s

strong-toothbrush-37759

03/13/2022, 8:59 PM
Here are the contents of my
pyproject.toml
if that does matter to somebody: https://pastebin.com/QUz5bT35
h

happy-kitchen-89482

03/14/2022, 12:07 AM
It shouldn't be a problem to have a source root inside another source root. For a given file, pants will take the longest prefix that matches
What determines what your source roots should be configured to be is how you import
In your case, if core is a child of src and you want to "from core.models import..." then src is a source root
So it seems that your source roots are configured correctly
It's fine to edit the tailored BUILD files, although of course the less you have to do that the better
👌 1
So having BUILD files inside the tests subfolder is fine
Re question #2, what is the full stacktrace? (Run with --print-stacktrace)
s

strong-toothbrush-37759

03/14/2022, 6:37 AM
Here is the complete stacktrace:
Copy code
❯ ./pants test src/core/tests/test_companies.py --print-stacktrace
07:36:56.15 [INFO] Initialization options changed: reinitializing scheduler...
07:36:56.81 [INFO] Scheduler initialized.
07:37:00.77 [ERROR] 1 Exception encountered:

Engine traceback:
  in select
  in pants.core.goals.test.run_tests
  in pants.backend.python.goals.pytest_runner.run_python_test (src/core/tests/test_companies.py:../tests)
  in pants.backend.python.goals.pytest_runner.setup_pytest_for_target
Traceback (no traceback):
  <pants native internals>
Exception: String("Can only merge Directories with no duplicates, but found 2 duplicate entries in :\n\n`pyproject.toml`: 1.) file digest=9d7d5d9df3e4ae5a4842a380891776c10adfc04ffdf0d48728db8ff4e7a05943 size=1006:\n\n[tool.black]\nline-length = 100\ntarget-version = [ \"py38\",]\ninclude = \"\\\\.pyi?$\"\nforce-exclude = \"/(\\n    .git\\n  | .hg\\n  | .eggs\\n  | .mypy_cache\\n  | .tox\\n  | .venv\\n  | .venv\\n  | _build\\n  | buck-out\\n  | build\\n  | dist\\n  | migrations\\n)/\\n\"\n\n[tool.isort]\nprofile = \"black\"\nline_length = 100\nsections = [ \"FUTURE\", \"STDLIB\", \"THIRDPARTY\", \"FIRSTPARTY\", \"LOCALFOLDER\",]\nno_lines_before = \"LOCALFOLDER\"\nskip_glob = [ \"*/migrations/*.py\",]\nsrc_paths = [ \"src\", \".\",]\nknown_thirdparty = [ \"django\",]\n\n[tool.coverage.run]\nomit = [ \"*/tests/*\", \"*/migrations/*\", \"*/prototypes/*\", \"*/site-packages/*\", \"*/test_*\", \"scripts/*\", \"docker/*\", \"config/*\", \"pytest.pex/*\",]\nrelative_files = true\n\n[tool.coverage.report]\nprecision = 2\n\n[tool.pytest.ini_options]\npython_files = \"tests.py test_*.py *_tests.py\"\nDJANGO_SETTINGS_MODULE = \"config.settings_testing\"\nmarkers = [ \"slow: marks tests as slow (deselect with '-m \\\"not slow\\\"')\", \"jwt_scope: sets required scope for api test\",]\nenv = [ \"ENVIRONMENT=test\",]\n\n\n`pyproject.toml`: 2.) file digest=55e6c945374e7d8f58b1d90e4dc00dee93a906850e953db2512938ead8b84fef size=994:\n\n[tool.black]\nline-length = 100\ntarget-version = ['py38']\ninclude = '\\.pyi?$'\nforce-exclude = \"\"\"\n/(\n    .git\n  | .hg\n  | .eggs\n  | .mypy_cache\n  | .tox\n  | .venv\n  | .venv\n  | _build\n  | buck-out\n  | build\n  | dist\n  | migrations\n)/\n\"\"\"\n\n[tool.isort]\nprofile = \"black\"\nline_length = 100\nsections = [\"FUTURE\", \"STDLIB\", \"THIRDPARTY\", \"FIRSTPARTY\", \"LOCALFOLDER\"]\nno_lines_before = \"LOCALFOLDER\"\nskip_glob = [\"*/migrations/*.py\"]\nsrc_paths = [\"src\", \".\"]\nknown_thirdparty = [\"django\"]\n\n[tool.coverage.run]\nomit = [\n    \"*/tests/*\",\n    \"*/migrations/*\",\n    \"*/prototypes/*\",\n    \"*/site-packages/*\",\n    \"*/test_*\",\n    \"scripts/*\",\n    \"docker/*\",\n    \"config/*\",\n]\n\n[tool.coverage.report]\nprecision = 2\n\n[tool.pytest.ini_options]\npython_files = \"tests.py test_*.py *_tests.py\"\nDJANGO_SETTINGS_MODULE= \"config.settings_testing\"\nmarkers = [\n    \"slow: marks tests as slow (deselect with '-m \\\"not slow\\\"')\",\n    \"jwt_scope: sets required scope for api test\",\n]\nenv = [\n    \"ENVIRONMENT=test\",\n]\n")
So I kind of narrowed it down to the
[tool.pytest.ini_options]
section. Whenever this section exists inside of my
pyproject.toml
, Pants will throw this error. Weird is that when I comment it out, Pants still recognizes it and throws the same error 😅
h

happy-kitchen-89482

03/14/2022, 5:49 PM
Does that section exist in both pyproject.tomls?
s

strong-toothbrush-37759

03/14/2022, 5:53 PM
I think I only have one pyproject.toml 😅
h

happy-kitchen-89482

03/14/2022, 6:29 PM
Oh, in that case something truly silly is happening 🙂
I will look into this today
Sorry for the trouble! You're hitting some unusual issues
❤️ 1
s

strong-toothbrush-37759

03/14/2022, 6:42 PM
It really feels like I‘m pushing the wrong buttons 😅. Thank you very much!
@happy-kitchen-89482 If you need help or anything, I’m up for another 2 hours trying to understand Third-party dependencies 🙌
👍 1
h

happy-kitchen-89482

03/14/2022, 7:55 PM
Extremely weird that these have different digests and sizes...
s

strong-toothbrush-37759

03/14/2022, 7:59 PM
That’s what I was confused about as well. When I replaced the
\\n
with new-lines both files seemed identical.
h

happy-kitchen-89482

03/14/2022, 8:01 PM
When I look at the error above, I see, e.g., that one file has
target-version = [ \"py38\",]
and the other has
target-version = ['py38']
Note the double quotes and comma vs single quotes and no comma
👀 1
s

strong-toothbrush-37759

03/14/2022, 8:03 PM
They are actually single quotes, I’m gonna try changing to double-quotes to see if that helps 😄
Seems like the
include = '\.pyi?$'
is causing the error 🙄
btw. this is taken from black itself 😄
Okay that was a bit quick I guess. It’s still throwing the same error when I try to use the pytest config in it:
Copy code
[tool.pytest.ini_options]
python_files = "tests.py test_*.py *_tests.py"
DJANGO_SETTINGS_MODULE= "config.settings_testing"
markers = [
    "slow: marks tests as slow (deselect with '-m \"not slow\"')",
    "jwt_scope: sets required scope for api test",
]
env = [
    "ENVIRONMENT=test",
]
Maybe because of the escaped quotes?
As soon as I put
[tool.pytest.ini_options]
in the pyproject.toml it stops working. Is there some pytest magic trying to parse the file or something? 🤔
h

happy-kitchen-89482

03/14/2022, 8:19 PM
Yes, we detect that
pyproject.toml
is relevant to
pytest
runs by looking for a
tool.pytest
table in the file
But we don't parse the TOML, we just regex
Or actually just search for the exact string
which explains why commenting it out didn't help
🙌 1
But this shouldn't be a problem
what we're seeing is two versions of the same file
s

strong-toothbrush-37759

03/14/2022, 8:22 PM
Which in turn explains why regexing the file could be a problem, because it contains a regex itself (?)
h

happy-kitchen-89482

03/14/2022, 8:22 PM
One with single quotes, one with double quotes
No, that part seems to be working correctly
s

strong-toothbrush-37759

03/14/2022, 8:23 PM
Does this somehow parse the file?
h

happy-kitchen-89482

03/14/2022, 8:23 PM
No, that just fetches the file content
s

strong-toothbrush-37759

03/14/2022, 8:26 PM
Could this somehow be related to the Snapshot mechanism? As we’re seeing two different files = actual content vs. maybe snapshot content
h

happy-kitchen-89482

03/14/2022, 8:26 PM
No, Snapshot just grabs file content
What we're seeing is two differently-formatted versions of the same pyproject.toml
You're absolutely sure you have only one in the repo?
1
s

strong-toothbrush-37759

03/14/2022, 8:27 PM
I just checked again:
Copy code
❯ find . -name 'pyproject.toml' | xargs wc -l
      44 ./pyproject.toml
      13 ./src/venv/lib/python3.9/site-packages/spacy/tests/package/pyproject.toml
      57 total
The other one should be unrelated:
Copy code
❯ cat ./src/venv/lib/python3.9/site-packages/spacy/tests/package/pyproject.toml
[build-system]
requires = [
    "setuptools",
    "cython>=0.25,<3.0",
    "cymem>=2.0.2,<2.1.0",
    "preshed>=3.0.2,<3.1.0",
    "murmurhash>=0.28.0,<1.1.0",
    "thinc>=8.0.12,<8.1.0",
    "blis>=0.4.0,<0.8.0",
    "pathy",
    "numpy>=1.15.0",
]
build-backend = "setuptools.build_meta"
h

happy-kitchen-89482

03/14/2022, 8:28 PM
OK
Any chance you can post a simple repo that reproduces this?
And can you file a bug report here? https://github.com/pantsbuild/pants/issues/new/choose
So that we don't get lost in Slack history... 🙂
Sorry for the inconvenience!
🙏 1
s

strong-toothbrush-37759

03/14/2022, 8:29 PM
Sure I can try
I created a reproduction using your
example-django
project. Give me a second to upload.
Here’s the issue with all necessary information: https://github.com/pantsbuild/pants/issues/14776 Don’t hesitate to come back to me, thanks for digging in advance!
h

happy-kitchen-89482

03/14/2022, 9:48 PM
Thanks!