https://pantsbuild.org/ logo
Join Slack
Powered by
Hi everyone. I am migrating my monorepo to pants a...
# general
p
Hi everyone. I am migrating my monorepo to pants and one of my pytest integration tests is using pytest-xprocess to invoke fastapi test app using uvicorn. The problem is that pants cannot find uvicorn command in the generated pex environment. I tried to use it indirectly with python -m uvicorn but it's using my global python. What's the best way to instruct pants to use python or uvicorn in the pex environment? NB: I have all dependencies required by the test in the BUILD file.
šŸ‘€ 1
f
not sure if I understood correctly, but perhaps you could package a 
pex_binary
with 
uvicorn
and make it accessible to your test? https://www.pantsbuild.org/docs/reference-python_test#coderuntime_package_dependenciescode
so inside your Python test you have something like:
Copy code
def test_case():
  result = subprocess.run("uvicorn --run foo")
  assert result...
and you want to make sure that the 
uvicorn
comes from a PEX file in your test sandbox, not from some random place on the machine, right?
here's an example with tabulate:
Copy code
python_requirement(
    name="tabulate",
    requirements=["tabulate"],
    entry_point="tabulate:_main",
)

pex_binary(
    name="runner",
    dependencies=[":tabulate"],
    entry_point="tabulate:_main",
)
and now you can do
Copy code
āÆ dist/runner.pex --help
Usage: tabulate [options] [FILE ...]

Pretty-print tabular data.
...
p
yes, you understand it correctly, OK, will try this
šŸ‘ 1
hm, including the directives for requirements and pex binary wasn't enough, it still doesnt work. since I am completely new to using pex, how do you generate dist/runner.pex with pants?
f
oh I am very sorry, I missed a step! Now, in your 
python_tests
target you'd need to tell Pants that you want to make a package accessible
you can test packaging the 
runner.pex
with 
pants package //:runner
with 
//:runner
being the target address
so you can confirm it does package and you can then run locally from the 
dist
directory
once it works,
Copy code
python_tests(
    name="test-name",
    ...
    runtime_package_dependencies=[
        "//:runner", # path to your pex_binary target from the root of the repo
    ],
)
making sure you can run the pex from the 
dist
after 
pants package //:runner
will help to confirm you can run the 
uvicorn
when it's going to be accessible to a test
it's often the matter of finding the right entry point, I think it's https://github.com/encode/uvicorn/blob/master/pyproject.toml#L47C1-L48C30
p
thanks, I managed to package and test the runner.pex. It's working fine with just uvicorn as entrypoint or the full path. but the pytest is still failing and getting uvicorn command not found, even though I included the runner in runtime deps.
f
okay, let me try to repro locally...
I am able to run the workflow similar to yours locally, but using plain subprocess:
Copy code
def test_tabulate():
    #import pdb; pdb.set_trace()
    ls_result = subprocess.Popen(["ls"], stderr=subprocess.PIPE, stdout=subprocess.PIPE)
    ls_output = ls_result.stdout.readlines()
    tabulate_result = subprocess.Popen(["python3", "tabulate-for-tests.pex", "--help"], stderr=subprocess.PIPE, stdout=subprocess.PIPE)
    tabulate_output = tabulate_result.stdout.read().decode()
    assert 'Pretty-print tabular data' in tabulate_output
and
Copy code
python_requirement(
    name="tabulate",
    requirements=["tabulate"],
    entry_point="tabulate:_main",
)

pex_binary(
    name="tabulate-for-tests",
    dependencies=[":tabulate"],
    entry_point="tabulate:_main",
)
and
Copy code
python_tests(
    name="tests",
    runtime_package_dependencies=[
        "//:tabulate-for-tests", # path to your pex_binary target from the root of the repo
    ],
)
if you uncomment the 
pdb
line, you can run 
pants test --debug ::
and step through to see what's inside your sandbox
you can also ask Pants to leave the sandbox on failure https://www.pantsbuild.org/docs/reference-global#keep_sandboxes with 
pants --keep-sandboxes=on_failure test ::
and you can inspect your sandbox
I think the pex file is placed not in the root of the dist but somewhere in the directory hierarchy (where you declared your 
pex_binary
), once we know where it's stored, we can provide the path in 
["python3", "tabulate-for-tests.pex"...
you can read more about it here https://www.pantsbuild.org/docs/python-test-goal#testing-your-packaging-pipeline to clarify, if your 
pex_binary
is declared in 
helloworld/greet/BUILD
, then when packaged:
Copy code
āÆ tree dist          
dist
ā”œā”€ā”€ helloworld.greet
ā”‚   ā””ā”€ā”€ tabulate-for-tests.pex
this is the default behavior. This can be adjusted with https://www.pantsbuild.org/docs/reference-pex_binary#codeoutput_pathcode, if necessary
p
Thanks for the repro. I am using the default settings, so the pex binary is generated in tmp folder. I managed to locate it but whether I use 
python3 path/to/runner.pex
or just 
path/to/runner.pex
it has problem to locate a working python interpreter now, despite having python interpreter constraints setup in pants.toml with search_path to PYENV and other tests running normally, Idk
f
so the pex binary is generated in tmp folder
I assume you mean not 
/tmp
but rather something like 
/tmp/pants-sandbox-vwutM9
? Just checking šŸ™‚
it has problem to locate a working python interpreter now
Oh that's great, so you are able to locate the pex now from your Python test, right?
despite having python interpreter constraints setup in pants.toml with search_path to PYENV
would you be able to share the command you run from your test? I can try to repro locally
Oh that's great, so you are able to locate the pex now from your Python test, right?
I mean you can confirm you can run from you test some shell command on that file, e.g.
Copy code
head -1 foo/bar.pex
#!/usr/bin/env python3.11
p
yup, that /tmp/sandbox-etc yes I am able to locate it and then execute /tmp/pants-sandbox/path/to/runner.pex in the subprocess like
Copy code
[
 "python",
 "{os.getenv('PWD')}/services.runner.tests/runner.pex",
]
I also tried just
Copy code
[
 "{os.getenv('PWD')}/services.runner.tests/runner.pex",
]
f
you know that you don't have to name it 
runner.pex
, right? Just checking šŸ™‚
p
I know, just lazy to come up with names
f
fair enough šŸ˜„
p
It's just a single pex_binary for now, so runner will do just fine šŸ˜„
f
hm, and when you run
Copy code
[
 "python",
 "{os.getenv('PWD')}/services.runner.tests/runner.pex",
]
in a subprocess or call or whatever, what's the error you get?
p
Failed to find compatible interpreter on path $PATH Examined the following working interpreters: 1.) /home/adrian/.pyenv/versions/miniconda3-3.8-4.9.2/bin/python3.8 CPython==3.8.2 2.) /usr/bin/python2.7 CPython==2.7.18 3.) /usr/bin/python3.8 CPython==3.8.10 No working interpreter compatible with the requested constraints was found: Version matches CPython==3.8
f
hm, what do you have in 
pants.toml
file?
Copy code
[python]
interpreter_constraints = ["==3.9.*"]
your 
pex_binary
should default to the IC set in 
pants.toml
, see https://www.pantsbuild.org/docs/reference-pex_binary#codeinterpreter_constraintscode
I am worried about 
Version matches CPython==3.8
whether it should instead be 
==3.8.*
šŸ˜•
p
Copy code
[python]
enable_resolves = true
interpreter_constraints = ["CPython==3.8"]

[python-bootstrap]
search_path = ["<PYENV>"]
this is in my pants.toml, I found in another repo
other tests work normally, only this one with pex_binary has a problem
f
thanks! So you run 
pants test path/to/test.py
and you get
Copy code
Failed to find compatible interpreter on path $PATH
...
error?
p
I run 
pants test path/to/tests/::
and one fails and the rest passes
f
Gotcha, thanks. I wonder if we could experiment setting the path explicitly with shebang to your 
/usr/bin/python3
to check it can find a specific interpreter https://www.pantsbuild.org/docs/python-interpreter-compatibility#changing-the-interpreter-search-path and https://www.pantsbuild.org/docs/reference-pex_binary#codeshebangcode
p
adding
Copy code
pex_binary(
    ...
    shebang="/usr/bin/python3"
)
Copy code
[python-bootstrap]
search_path = ["<PYENV>", "/usr/bin/python3"]
also didnt help
f
and what's the error message exactly you got this time?
p
the error is still the same
f
can you try
Copy code
[
 "/usr/bin/python3",
 "{os.getenv('PWD')}/services.runner.tests/runner.pex",
]
p
still the same error
f
šŸ˜ž
I'll let my brain work on this in background then šŸ˜„
p
haha, OK, thanks for the effort anyway
at least I learned something new today
f
oh! Can you try running 
pants --no-pantsd --no-local-cache test path/to/tests/::
and then I'll go šŸ˜„
I wonder if there's anything dodgy going with the cached paths to the Python interpreters, I remember I had a few issues a while ago
p
nope, didnt help, still the same error
šŸ˜ž
e
@fresh-cat-90827 was correct to worry about ==3.8 that zero fills the patch version and will fail to match. Definitely use ==3.8.*
Open in Slack
PreviousNext
Powered by