https://pantsbuild.org/ logo
#general
Title
# general
c

clever-table-90691

07/14/2020, 4:43 PM
Hi there, I’ve got a question regarding 
./pant awslambda
  , we are currently on PANTS
v1.28.0
When pants is building an awlambda target, it seems to be always using python 2.7 even there is python 3.8 installed on our CI/CD environment, we’ve also specified the 
interpreter_constraints = ["CPython>=3.7"]
 in the pants.ci.toml file Is there a way to enforce it to use python 3.8? Here is part of the trace stack:
Copy code
pid: 14794 -> /usr/bin/python2.7 /tmp/process-executionAmQ08c/pex_root/pip.pex/d3323a933395f1fea4b421b92e99bd2c3d524e6c --disable-pip-version-check --isolated --no-python-version-warning -q --cache-dir /tmp/process-executionAmQ08c/pex_root download --dest /tmp/tmpsuxiy3/manylinux2014_x86_64-cp-38-cp38 --index-url <https://pypi.org/simple/> --header Cache-Control:max-age=3600 --retries 5 --timeout 15 --platform manylinux2014_x86_64 --implementation cp --python-version 38 --abi cp38 --only-binary :all: --constraint constraints.txt boto3==1.12.20 idna<2.10 requests==2.23.0 ruamel.yaml==0.16.10 snowflake-sqlalchemy==1.2.3 raised Executing /usr/bin/python2.7 /tmp/process-executionAmQ08c/pex_root/pip.pex/d3323a933395f1fea4b421b92e99bd2c3d524e6c --disable-pip-version-check --isolated --no-python-version-warning -q --cache-dir /tmp/process-executionAmQ08c/pex_root download --dest /tmp/tmpsuxiy3/manylinux2014_x86_64-cp-38-cp38 --index-url <https://pypi.org/simple/> --header Cache-Control:max-age=3600 --retries 5 --timeout 15 --platform manylinux2014_x86_64 --implementation cp --python-version 38 --abi cp38 --only-binary :all: --constraint constraints.txt boto3==1.12.20 idna<2.10 requests==2.23.0 ruamel.yaml==0.16.10 snowflake-sqlalchemy==1.2.3 failed with 1
we’ve also tried with the flag
--python-setup-interpreter-constraints="['CPython>=3.8']"
but it’s still using python 2.7
h

hundreds-father-404

07/14/2020, 4:49 PM
Hi, what is the
python_awslambda
target definition in your BUILD file?
c

clever-table-90691

07/14/2020, 4:57 PM
the runtime is
runtime="python3.8"
h

hundreds-father-404

07/14/2020, 5:07 PM
So from the stack trace, it looks like Pex is getting all the right commands to build a PEX with Python 3.8. The python2.7 piece is a little confusing that it’s not actually what your PEX is being built with. Rather, it’s what PEX is using to run itself. This is like how both Pants and MyPy need to run with Python 3, but they can work on Python 2 code. Pex can run with Python 2.7, and still build Python 3 PEXes What is the specific error you’re getting / the full stack trace?
c

clever-table-90691

07/14/2020, 5:59 PM
stacktrace
the stack trace is too long.. here’s the last part where it’s printing the error
👍 1
Copy code
ERROR: No matching distribution found for ruamel.ordereddict; platform_python_implementation == "CPython" and python_version <= "2.7" (from ruamel.yaml==0.16.10->-c constraints.txt (line 53))
👀 1
the
./pants awslambda
command completes without error when we ran it in a venv with python 3.8
w

witty-crayon-22786

07/14/2020, 6:45 PM
so, the
Copy code
pid: 14794 -> /usr/bin/python2.7 /tmp/process-executionAmQ08c/pex_root/pip.pex/d3323a933395f1fea4b421b92e99bd2c3d524e6c
bit does indicate that pex is using pip under python 2.7 to build this target
so i would wonder: what does the target definition look like in your
BUILD
file? do you override the constraints on the target?
and if not, does the AWS plugin do that, perhaps? cc @happy-kitchen-89482
h

hundreds-father-404

07/14/2020, 6:49 PM
does the AWS plugin do that
It does not. We determine interpreter constraints based on the transitive closure of targets. We merge them into a single set of constraints But because we set
--platform
, that doesn’t even matter.
--platform
overrides interpreter constraints with Pex. You can’t specify both at the same time. And in the original Pex command, it looks like we indeed are setting
--platform
correctly.
the ./pants awslambda command completes without error when we ran it in a venv with python 3.8
I think this is likely more relevant than the
compatibility
field or the
interpreter_constraints
option. I suspect there’s a bug with the venv somehow leaking.
w

witty-crayon-22786

07/14/2020, 6:53 PM
@clever-table-90691: if you’re able to run with
./pants -ldebug ...
it should render a lot more information about how pex is being invoked.
@hundreds-father-404: if the venv leaked, what i would expect to happen here would be for pex to fail to find an interpreter matching the constraint, rather than to proceed with 2.7
so i expect that the setting isn’t being applied (correctly), perhaps
c

clever-table-90691

07/15/2020, 11:48 AM
ldebug
w

witty-crayon-22786

07/15/2020, 4:41 PM
Copy code
[2020-07-15T11:40:40.222Z] pex: Selecting runtime interpreter
[2020-07-15T11:40:40.222Z] pex: Using the current interpreter /usr/bin/python2.7 since no constraints have been specified and PYTHONPATH is not set.
[2020-07-15T11:40:40.222Z] pex: Selecting runtime interpreter: 0.1ms
… so it looks like however you are setting the interpreter_constraints is not sticking
i don’t see a file named
<http://pants.ci|pants.ci>.toml
in use in your output though… should it be?
c

clever-table-90691

07/15/2020, 5:46 PM
hey @witty-crayon-22786, thanks for looking into this Do you mean?
export PANTS_CONFIG_FILES=<http://pants.ci|pants.ci>.toml
w

witty-crayon-22786

07/15/2020, 5:48 PM
got it. ok: so, you might consider confirming that that is resulting in the options you expect: the debug input indicates that the constraint is not being applied: running
PANTS_CONFIG_FILES=<http://pants.ci|pants.ci>.toml ./pants help-advanced python-setup
should show you the option applied
if you do see the option being applied, then this might be a bug in the awslambda plugin, potentially
c

clever-table-90691

07/15/2020, 6:21 PM
3.8 is default
Screen Shot 2020-07-15 at 19.20.36.png
h

hundreds-father-404

07/15/2020, 6:24 PM
Thanks, Maryanne. @witty-crayon-22786 it looks like
lambdex.pex
, i.e. the pex that runs the Lambdex tool, does not actually have constraints applied. We set it to
lambdex.get_requirement_specs()
, but that results in None, meaning we have no constriants. However, that is different than when we resolve requirements, and it looks like the issue is that resolving requirements is not doing the right thing.
w

witty-crayon-22786

07/15/2020, 6:28 PM
ok. if you have the context on it, would you mind opening the issue?
h

hundreds-father-404

07/15/2020, 6:29 PM
Well, I’m still unclear why that would be an issue. And I still believe that when you set
--platform
,
--interpreter-constraints
no longer apply. We had that wrong originally, and it took John walking me through it to fix that in Pants.
w

witty-crayon-22786

07/15/2020, 6:37 PM
h

hundreds-father-404

07/17/2020, 7:42 AM
Hey Maryanne, I’m going to try to reproduce this tomorrow/today and figure out what’s going on. Are you able to please share a little more about your environment, eg which Linux version and the output of
which -a python
? This is Pants 1.28, right? It’d also help if you can share the constraints file if y’all are able to, even if some of it is stripped.
Oh, and what you mean by running Pants in a venv? I’m not sure what you mean by doing that vs how you normally run Pants?
7 Views