hey folks having some real trouble building a pex ...
# pex
hey folks having some real trouble building a pex file do to some
dependency version issues
following command continually gives me
ModuleNotFoundError: No module named 'pkg_resources.py31compat'
in my build container
pex --interpreter-constraint='CPython>=3.7' -i <https://artifactory.us-east-1.com/pypi/simple> thanos[cli]==11.3.2 setuptools==45.2.0 -o thanos.pex -m thanos
works fine on my machine
build container platform string is:
note: thanos is a private library
i can confirm that the specified version of setuptools does have the module in question:
Copy code
# pip list | grep setuptools
setuptools                 45.2.0
setuptools-scm             3.4.3
# # python
Python 3.7.9 (default, Aug 18 2020, 06:24:24)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pkg_resources.py31compat
kind of at the end of my rope here 😞
Okay one idea to further isolate the problem. It looks like this problematic dep is a transitive dep, and
might be a red herring. With the Pex CLI, try installing just that bad transitive dep and nothing else If that works, could you please copy the command you’re using? It’ll help us to reproduce
yeah can reproduce with:
pex --interpreter-constraint='CPython>=3.7' -i <https://artifactory.us-east-1.bamgrid.net/api/pypi/dataengineering-pypi/simple> boto3 -o boto3.pex -m boto3
mm. interesting. i’ll see if i can repro
building in the
public image
if that's helpful
well, an image derived from that image
i’m not in exactly that image, but in an ubuntu image using 3.8,
~/src/venvs/pex/bin/pex --interpreter-constraint='CPython>=3.7' boto3
~/src/venvs/pex/bin/pex --interpreter-constraint='CPython>=3.7' boto3 setuptools==45.2.0
are both fine
(both load correctly, and in the setuptools case i’m able to
import pkg_resources.py31compat
@wonderful-iron-54019: do you think you could describe the minimal repro in a ticket? it sounds like it’s β€œboto3 doesn’t install in a particular docker image”?
…a pex ticket, preferably
and this is all in the context of the
that ships with pants 1.30 i think?
but also repro'dd with the latest version
the default python in that image appears to be 3.5
we install python3.7
one sec i'll put up a ticket
got it.
(still waiting for this volunteering call to start...) What happens if you try installing directly via Pip? Pex vendors Pip, so that can help isolate if Pex is doing something wrong
installling what? boto3?
Yes. Using the pip3.7 that corresponds to the interpreter. Use the -i setting too (I think it’s that in pip)
one sec, paring down our Dockerfile for the ticket and trying to reproduce
Pip won’t build a Pex file of course, nor decide which interpreter to use, but will install the problematic distribution
(but i will say that alll these are installed in our container without problems before i tried running thiscommmandd
πŸ‘ 1
Copy code
# pip install boto3
Collecting boto3
  Downloading boto3-1.14.63.tar.gz (97 kB)
     |β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 97 kB 2.0 MB/s
Collecting botocore<1.18.0,>=1.17.63
  Downloading botocore-1.17.63-py2.py3-none-any.whl (6.6 MB)
     |β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 6.6 MB 4.2 MB/s
Collecting jmespath<1.0.0,>=0.7.1
  Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting s3transfer<0.4.0,>=0.3.0
  Downloading s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
     |β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 69 kB 5.7 MB/s
Collecting urllib3<1.26,>=1.20; python_version != "3.4"
  Downloading urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
     |β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 127 kB 17.4 MB/s
Collecting docutils<0.16,>=0.10
  Downloading docutils-0.15.2-py3-none-any.whl (547 kB)
     |β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 547 kB 16.0 MB/s
Collecting python-dateutil<3.0.0,>=2.1
  Downloading python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
     |β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 227 kB 5.8 MB/s
Collecting six>=1.5
  Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
Building wheels for collected packages: boto3
  Building wheel for boto3 (setup.py) ... done
  Created wheel for boto3: filename=boto3-1.14.63-py2.py3-none-any.whl size=127856 sha256=236c64173f95a8916dfc182ae1e27bbb2bff95c7c9997b686211288b62a7794e
  Stored in directory: /root/.cache/pip/wheels/23/8c/0f/3bc284d5679e59e94fb5606c3e3f086a539e9e26ec61311fe8
Successfully built boto3
Installing collected packages: urllib3, jmespath, docutils, six, python-dateutil, botocore, s3transfer, boto3
Successfully installed boto3-1.14.63 botocore-1.17.63 docutils-0.15.2 jmespath-0.10.0 python-dateutil-2.8.1 s3transfer-0.3.3 six-1.15.0 urllib3-1.25.10
seems fine
thanks, looking
if i may: why do you explicitly install setuptools along with pex?
…i’ll just work through the repro, thanks. but my knee jerk reaction is: it’s not surprising that boto3 cannot find the setuptools that is β€œnext to” pex, because pex is trying to isolate you from anything that isn’t supposed to be inside the pex file
Installing setuptools was my idea due to suspecting that it was missing, as often happens with things like isoet. See the thread in general for context
sure, but you wouldn’t install it there. you’d add it to the pex file, i think.
(downloading things to repro)
@wonderful-iron-54019: commented with some workarounds
the latter thing (β€œrun pex from a pex”) is what pants 2.x/v2 always does
ok will try the. site packages since upgrading tto v2 not an option atm
Yea. You can explicitly run pex from a pex as well: see the second point on the ticket.
not sure how i would do that in context of a custom pants plugin?
would i make a pex request and then a second one?
oh. hm. unclear. i guess you’re already using the v2 plugin api?
yea, if you’re able to remove
Copy code
would do that for now. i’ll validate that the second workaround is an actual fix.
yea, double checked that both workarounds on the ticket work. i think eric is likely to have more context on precisely which Pants APIs you’re using, but maybe we don’t have to worry about that if the site-packages solution is enough.
I don’t follow this recommendation. Where is this being set atm? Either workaround, I think that we need to make changes to the core
type in the Plugin API. We’ll cherry-pick that into 1.30
see the dockerfile in the description of the ticket.
if you remove that flag, the commands in the description work.
Ah. So, the idea is to change the docker image, and that should cause Pex and Pants to do the right thing, right? Are you able to change the image JP?
hey sorry dropped off last night. yes i can change the image, tho im not sure what implication that is going to have for the rest of our build, if everything works it should be fine
will keep y'all posted, thanks for the help here
❀️ 1
can confirm that the pex resolution works without the
flag, will run the container through the rest of the build steps
🀞 1
happy to announce we just got this out to prod
πŸŽ‰ 1
thanks for all your help once again
huzzah! thanks for the very thorough bug report.
❀️ 1
you got it!