https://pantsbuild.org/ logo
Join Slack
Powered by
Hey, <following up on this thread>, I tried to bui...
# general
f
Hey, following up on this thread, I tried to build a wheel for a python package that only has an sdist, and pants doesn't seem to be able to find it, even though I can see the extra index URL and I get the package when I run 
pip download
with the same settings. Can I get some help? Details in thread.
✅ 1
👀 1
h
Awaiting details 🙂 Or are they the same as in the thread from a few weeks ago?
f
I am using this command to build the wheel:
Copy code
pip3 wheel -w . \
  --build-option --plat-name=manylinux2014_x86_64 \
  --build-option --python-tag=cp39 \
  --build-option --py-limited-api=cp39 \
  --only-binary ":all:" --no-deps \
  jax-jaxlib-v0.3.2/
and producing a wheel called 
jax-0.3.2-cp39-none-manylinux2014_x86_64.whl
- which I have uploaded to an artifactory repository. I have also tried naming the wheel 
jax-0.3.2-cp39-cp39-manylinux2014_x86_64.whl
and 
jax-0.3.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
and I verified that I can download the package using pip:
Copy code
pip3 download jax==0.3.2 --index-url "https://*******@delfidiagnostics.jfrog.io/artifactory/api/pypi/product_pipelines_pypi_dev/simple" --platform manylinux2014_x86_64 --implementation cp --python-version 39 --abi cp39 --only-binary :all: jax==0.3.2
However, when pants tries to do the same operation on our CI server, I get an error:
Copy code
[ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Building src.stages.bin_mixture/exe.pex with 12 requirements: [...] jax==0.3.2, [...]' failed with exit code 1.
stdout:

stderr:
ERROR: Could not find a version that satisfies the requirement jax==0.3.2
ERROR: No matching distribution found for jax==0.3.2
pid 2031 -> /home/runner/.cache/pants/named_caches/pex_root/venvs/33f4ac952e01f0714dcebef0f716941c259ca3ef/d3a92f1ce0047120f06ca0e3e9d47bf9e990ce51/pex --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /home/runner/.cache/pants/named_caches/pex_root --log /tmp/process-executionnEus9I/.tmp/tmp7vqidq11/pip.log download --dest /tmp/process-executionnEus9I/.tmp/tmpozayljtj/manylinux2014_x86_64-cp-39-cp39 --platform manylinux2014_x86_64 --implementation cp --python-version 39 --abi cp39 --only-binary :all: [...] jax==0.3.2 [...] --index-url <https://pypi.org/simple/> --extra-index-url ***<http://delfidiagnostics.jfrog.io/artifactory/api/pypi/product_pipelines_pypi_dev/simple|delfidiagnostics.jfrog.io/artifactory/api/pypi/product_pipelines_pypi_dev/simple> --retries 5 --timeout 15 exited with 1 and STDERR:
None
the build file looks like this:
Copy code
python_requirements()

python_sources(
    name="lib",
    sources=["src/*.py"],
    dependencies=[
        'src/common:lib',
...
        'src/stages/bin_mixture:jax',
    ]
)
pex_binary(
    name="exe",
    entry_point="src/bin_mixture.py",
    dependencies=[':lib'],
    shebang="/usr/bin/env python3",
    platforms=[
        "current",
        "manylinux2014_x86_64-cp-39-cp39",
    ]
)

docker_image(
    name="docker",
    image_tags=["{build_args.GIT_BRANCH}", "{build_args.GIT_COMMIT}", "{build_args.RELEASE_VERSION}"],
    dependencies=[
        "src/common/docker:python_base",
        ":exe",
    ]
)
and 
jax==0.3.2
is in requirements.txt
one last thing I am going to check is that ** in the extra index url - I have been assuming that github actions is obscuring the secrets in our log output, but I will double check that
h
Thanks for the details, and sorry for the trouble! Can you post the pants.toml snippet where you set up the custom index?
f
we're actually using an environment variable for that:
Copy code
export PANTS_PYTHON_REPOS_INDEXES="+['https://$ARTIFACTORY_USER:$ARTIFACTORY_PASS@delfidiagnostics.jfrog.io/artifactory/api/pypi/$ARTIFACTORY_REPO/simple']"
h
That should work
OK, let's see...
So, ~this does not work, at least in a Pants sandbox dir:
Copy code
pip --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /home/runner/.cache/pants/named_caches/pex_root --log /tmp/process-executionnEus9I/.tmp/tmp7vqidq11/pip.log download --dest /tmp/process-executionnEus9I/.tmp/tmpozayljtj/manylinux2014_x86_64-cp-39-cp39 --platform manylinux2014_x86_64 --implementation cp --python-version 39 --abi cp39 --only-binary :all: [...] jax==0.3.2 [...] --index-url <https://pypi.org/simple/> --extra-index-url ***<http://delfidiagnostics.jfrog.io/artifactory/api/pypi/product_pipelines_pypi_dev/simple|delfidiagnostics.jfrog.io/artifactory/api/pypi/product_pipelines_pypi_dev/simple> --retries 5 --timeout 15
But this, which is a subset of the same flags, does:
Copy code
pip download jax==0.3.2 --index-url "https://*******@delfidiagnostics.jfrog.io/artifactory/api/pypi/product_pipelines_pypi_dev/simple" --platform manylinux2014_x86_64 --implementation cp --python-version 39 --abi cp39 --only-binary :all: jax==0.3.2
So I guess the next step in diagnosing is to "bisect" which flag(s), when added, make this stop working
Can you first confirm that running the longer pip command above as a standalone command works? You may need to modify the tmpdirs it refers to I guess, but I don't imagine that affects the outcome
In particular, does it work or not when run on just 
jax==0.3.2
, without the other requirements you elided in the 
[...]
?
Going offline while my plane takes off, will check back in once airborne, assuming there is wifi on this thing
f
I removed the -q and --cache-dir arguments so I could see what was happening (and not have to blow away my existing pants cache) - confirmed it works with all the other flags, and all the other packages:
Copy code
pip3 --disable-pip-version-check --no-python-version-warning --exists-action a --isolated  download  --extra-index-url "https://*******@delfidiagnostics.jfrog.io/artifactory/api/pypi/product_pipelines_pypi_dev/simple" --platform manylinux2014_x86_64 --implementation cp --python-version 39 --abi cp39 --only-binary :all:  boto3==1.20.21 cryptography==3.4.8 jax==0.3.2 numpy==1.21.4 numpyro==0.9.1 overrides==6.1.0 pandas==1.3.5 scikit-learn==1.0.2 scipy==1.7.3 toml==0.10.2 typeguard==2.13.3 types-toml==0.10.1
Looking in indexes: <https://pypi.org/simple>, <https://build-pypi>:****@delfidiagnostics.jfrog.io/artifactory/api/pypi/product_pipelines_pypi_dev/simple
Collecting boto3==1.20.21
  Using cached boto3-1.20.21-py3-none-any.whl (131 kB)
Collecting cryptography==3.4.8
  Downloading cryptography-3.4.8-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.2/3.2 MB 5.6 MB/s eta 0:00:00
Collecting jax==0.3.2
  Downloading <https://delfidiagnostics.jfrog.io/artifactory/api/pypi/product_pipelines_pypi_dev/jax/0.3.2/jax-0.3.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl> (1.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 2.0 MB/s eta 0:00:00
Collecting numpy==1.21.4
  Downloading numpy-1.21.4-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 15.7/15.7 MB 12.1 MB/s eta 0:00:00
Collecting numpyro==0.9.1
  Using cached numpyro-0.9.1-py3-none-any.whl (283 kB)
Collecting overrides==6.1.0
  Using cached overrides-6.1.0-py3-none-any.whl (14 kB)
Collecting pandas==1.3.5
....etc....
Also I was able to confirm that the credentials are making it into pants from our CI system as expected. I'm going to try running an x86 ubuntu in docker and see if I can reproduce the error locally.
h
Hmm, interesting
So another thing to try is to run pants with 
--no-process-cleanup
, which will conserve the sandbox directories for processes, and print them out.
Then find the sandbox for the pex process and cd into it
then you can run the original process with the __run.sh script in that dir
And modify it appropriately to see what gives
it seems unlikely that 
-q
affects the outcome, but it is not impossible that 
--cache-dir
does
So trying it inside the sandbox, first exactly as-is to verify that it fails, and then modifying flags to see if that makes a difference
f
Well, this is unexpected... I was able to run pants on AWS on an x86 instance and it worked end to end, including resolving the dependency as expected. It must be in github actions. I'm sorry to have wasted your time on something that likely isn't pant's fault.
h
Huh, weird
Not a waste of time! We want to hear about any issues users are having.
So wait, previously you were running this on ARM?
Or on MacOS?
The fact that it worked on an x86 instance may be because it used the sdist
Would you be able to confirm that by, say, removing the extra index?
f
Yes, I have been developing on an M1 mac - when I removed the extra index on the x86 AWS instance, I got the failure back:
Copy code
ERROR: Could not find a version that satisfies the requirement jax==0.3.2
ERROR: No matching distribution found for jax==0.3.2
at least that matches my expectations - that pants/pex isn't using sdists
one really weird thing is that I seem to be able to build a PEX on my arm mac, even though jax doesn't have a wheel for that either . I don't understand how that is working.
I do see a wheel for it in my pants cache - named_caches/pex_root/wheels/34/fb/1e/[...]/jax-0.3.1-py3-none-any.whl - no idea how it got there. Does pants know how to build sdists after all?
h
Yes, Pants will gladly use an sdist if building for the current platform
f
ah! of course. that... makes sense.
h
Note that that cached wheel is 0.3.1 so that's not how your 0.3.2 got resolved
But your 
pex_binary
specifies an explicit non-current platform
So that requires wheels
f
totally fair. thanks. So it seems like pants is doing everything as expected, I just need to figure out what the difference is between my x86 instance and my github actions runner,
h
Still a little mysterious to me why it didn't work on your mac, given the wheel in the custom index
If you have the patience to keep debugging that, would love to figure it out
f
building the pex_binary has always failed my mac, without the extra index - sorry if I didn't make that clear
however, with the custom index, it does create the pex as expected
h
Huh
then I misunderstood what you're reporting above
At the top of this thread
Under what circumstances does that failure happen?
f
hey! sorry for not following up earlier but it turns out that my issue was actually around auth to artifactory. I was able to resolve it. Thanks again for all your help
h
Great, glad that was resolved!
Open in Slack
PreviousNext
Powered by