https://pantsbuild.org/ logo
#general
Title
# general
e

echoing-farmer-15630

03/11/2021, 8:01 PM
Rather foolish-feeling question; how does one handle tests in a separate directory from source? I may be missing something; I have (actual filenames changed):
Copy code
mylib/
  requirements.txt
  BUILD
  mylib/
    python source files...
    BUILD
  tests/
    python test files with lots of 'from mylib import...'
    BUILD
mylib/BUILD
just has a
python_requirements
target to build the requirements in requirements.txt.
mylib/mylib/BUILD
just has a
python_library
target with
sources=["**.py", "/***.*py"]
, and
mylib/tests/BUILD
has a
python_tests
target with
sources=["***/*.py"]
and
dependencies=['//mylib/mylib']
. But when I do
.pants test mylib/tests
, it looks like it resolves third-party requirements (and first-party; no complaints until pytest starts), but all of the
from mylib import ...
statements get me
cannot import name XXX from mylib (unknown location)
. So there's something I'm missing in how dependencies are resolved or copied or some such. Doing
pants dependencies mylib/tests
gets me entries like
mylib/mylib/submodule/__init__.py:../../mylib
, if that helps at all. I feel like I'm missing somethang that's just stonking obvious, but I'm not clever enough to diagnose it, it seems.
w

witty-crayon-22786

03/11/2021, 8:05 PM
your question relates to the layout of “source roots”, so good to see https://www.pantsbuild.org/docs/source-roots if you haven’t already
in this case, if your top-level
mylib
folder is at the root of the repository (there is a default source root at the root of the repository), then that might be the only source root in play
but that source root would result in your non-test code being named
mylib.mylib
if you created a source root in the top-level
mylib
folder (either by installing a pattern, or creating a marker file: see that page), then you’d get just
mylib
e

echoing-farmer-15630

03/11/2021, 9:20 PM
Aha! I had missed the source-roots business; thank you very much and apologies for the basic question. I'm fairly certain that's the issue (apologies for the delay; zoom meetings make laggards of us all). Unfortunately, as part of flailing to diagnose earlier, I reinstalled
pants
(by removing my
~/.cache/pants
directory and re-bootstrapping) and am now getting the dreaded
no module named pex.bin
error, which is most vexing; I fixed it earlier but can't for the life of me remember how. I can happily
import pex.bin
both from a local python shell and a shell if I activate the virtualenv at
~/.cache/pants/setup/bootstrap-Linux-x86_64/2.2.1rc0_py38
so I'm utterly mystified.
👀 1
❤️ 1
h

hundreds-father-404

03/11/2021, 9:22 PM
Gr I don't remember how that one is fixed either, nor what causes it What's the full error?
e

echoing-farmer-15630

03/11/2021, 9:26 PM
Was 2.2.0, now trying 2.2.1rc0; Here's the trace:
Copy code
❯ ./pants repl
/home/vputz/.cache/pants/setup/bootstrap-Linux-x86_64/2.2.1rc0_py38
15:23:07.41 [INFO] Completed: Building requirements.pex
15:23:07.41 [ERROR] Exception caught: (pants.engine.internals.scheduler.ExecutionError)
  File "/home/vputz/.cache/pants/setup/bootstrap-Linux-x86_64/2.2.1rc0_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 276, in run
    engine_result = self._run_v2(goals)
  File "/home/vputz/.cache/pants/setup/bootstrap-Linux-x86_64/2.2.1rc0_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 190, in _run_v2
    return self._maybe_run_v2_body(goals, poll=False)
  File "/home/vputz/.cache/pants/setup/bootstrap-Linux-x86_64/2.2.1rc0_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 207, in _maybe_run_v2_body
    return self.graph_session.run_goal_rules(
  File "/home/vputz/.cache/pants/setup/bootstrap-Linux-x86_64/2.2.1rc0_py38/lib/python3.8/site-packages/pants/init/engine_initializer.py", line 131, in run_goal_rules
    exit_code = self.scheduler_session.run_goal_rule(
  File "/home/vputz/.cache/pants/setup/bootstrap-Linux-x86_64/2.2.1rc0_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 563, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "/home/vputz/.cache/pants/setup/bootstrap-Linux-x86_64/2.2.1rc0_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 531, in _raise_on_error
    raise ExecutionError(

Exception message: 1 Exception encountered:

  ProcessExecutionFailure: Process 'Building requirements.pex' failed with exit code 1.
stdout:

stderr:
Traceback (most recent call last):
  File "/home/vputz/.cache/pants/named_caches/pex_root/unzipped_pexes/1c83f6fbdf4d7132c83e9cff7eb2c16d22feee03/.bootstrap/pex/pex.py", line 487, in execute
    exit_code = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/home/vputz/.cache/pants/named_caches/pex_root/unzipped_pexes/1c83f6fbdf4d7132c83e9cff7eb2c16d22feee03/.bootstrap/pex/pex.py", line 404, in _wrap_coverage
    return runner(*args)
  File "/home/vputz/.cache/pants/named_caches/pex_root/unzipped_pexes/1c83f6fbdf4d7132c83e9cff7eb2c16d22feee03/.bootstrap/pex/pex.py", line 435, in _wrap_profiling
    return runner(*args)
  File "/home/vputz/.cache/pants/named_caches/pex_root/unzipped_pexes/1c83f6fbdf4d7132c83e9cff7eb2c16d22feee03/.bootstrap/pex/pex.py", line 543, in _execute
    return self.execute_entry(self._pex_info.entry_point)
  File "/home/vputz/.cache/pants/named_caches/pex_root/unzipped_pexes/1c83f6fbdf4d7132c83e9cff7eb2c16d22feee03/.bootstrap/pex/pex.py", line 645, in execute_entry
    return runner(entry_point)
  File "/home/vputz/.cache/pants/named_caches/pex_root/unzipped_pexes/1c83f6fbdf4d7132c83e9cff7eb2c16d22feee03/.bootstrap/pex/pex.py", line 660, in execute_pkg_resources
    runner = entry.resolve()
  File "/home/vputz/.cache/pants/named_caches/pex_root/unzipped_pexes/1c83f6fbdf4d7132c83e9cff7eb2c16d22feee03/.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/__init__.py", line 2481, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
ModuleNotFoundError: No module named 'pex.bin'
Ugh, sorry; that's horrible.
w

witty-crayon-22786

03/11/2021, 9:27 PM
@echoing-farmer-15630: are you running pants from inside of a separate virtualenv?
e

echoing-farmer-15630

03/11/2021, 9:31 PM
I've tried it both ways, but with a new terminal window, I still get it:
Copy code
Welcome to fish, the friendly interactive shell
Type `help` for instructions on how to use fish

~ on ☁️  (us-west-2) 
❯ which python
/usr/bin/python

~ on ☁️  (us-west-2) 
❯ cd projects/forge

forge on  feature/vputz-try-pants [!?] via ⬢ v12.19.0 via 🐍 v3.8.6 on ☁️  (us-west-2) 
❯ ./pants repl
/home/vputz/.cache/pants/setup/bootstrap-Linux-x86_64/2.2.1rc0_py38
15:27:52.55 [INFO] Completed: Building requirements.pex
15:27:52.55 [ERROR] Exception caught: (pants.engine.internals.scheduler.ExecutionError)
...
although... HM. Okay, a data point for this; in the process of diagnosing, I switched
interpreter_constraints = ["CPython==3.8.*"]
(which wasn't working at the time) to
3.9.*
, which didn't work, and now I switched it back and it restarted pantsd and is now semi-magically working again:
Copy code
forge on  feature/vputz-try-pants [!?] via ⬢ v12.19.0 via 🐍 v3.8.6 on ☁️  (us-west-2) 
❯ ./pants repl
/home/vputz/.cache/pants/setup/bootstrap-Linux-x86_64/2.2.1rc0_py38
15:29:34.34 [INFO] initializing pantsd...
15:29:34.99 [INFO] pantsd initialized.
15:29:36.51 [INFO] Completed: Building requirements.pex
Python 3.8.6 (default, Jan 27 2021, 15:42:20) 
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>>
So I apologize for the vexing bug reports; this seems peculiar to me.
👀 1
The good news is that your source roots advice was right on (and well appreciated).
w

witty-crayon-22786

03/11/2021, 9:34 PM
very sorry for the trouble! i’m with Eric: i believe that there have been fixes for the
pex.bin
issue, but am having trouble locating them.
@echoing-farmer-15630: if you manage to get back in this state, i have an idea of what to try
e

echoing-farmer-15630

03/11/2021, 9:37 PM
Not to worry; apologies for not writing down my first fix (I had thought it was sourcing the pants venv and installing pex, but it's there on both counts). I tried activating another virtualenv and 2.2.1rc0 with python 3.8.6 still worked with
pants repl
. Thanks so much for your help; if I can replicated it I'll start another thread. Now to chip away at the library aliases and other requirements I haven't figured out yet!
👍 1
w

witty-crayon-22786

03/11/2021, 9:40 PM
ok, found it: https://github.com/pantsbuild/pants/issues/11211 … but yea, if you see it again, please pass
--pex-verbosity=9
and add some info there, or start a new thread
sounds good: thanks!
h

hundreds-father-404

03/11/2021, 9:44 PM
Now to chip away at the library aliases and other requirements I haven't figured out yet!
I'm not sure what part that's referring to, but we're happy to help 🙂
2 Views