https://pantsbuild.org/ logo
Join Slack
Powered by
Hi all. Running into something that I hope someone...
# general
m
Hi all. Running into something that I hope someone can help me with. In my pants project, I have a dependency on lxml. I develop locally in an ubuntu-22.04 devcontainer. It installs OK locally. In GitHub actions, I see:
Copy code
Error: 5.50 [ERROR] 1 Exception encountered:

Engine traceback:
  in `package` goal

ProcessExecutionFailure: Process 'Build python_google_cloud_function artifact for src/misc/funda_cloud_fn:cloud_function' failed with exit code 1.
stdout:

stderr:
A distribution for lxml could not be resolved for cp311-cp311-linux_x86_64.
Found 1 distribution for lxml that do not apply:
1.) The wheel tags for lxml 5.2.2 are cp311-cp311-manylinux_2_28_x86_64 which do not match the supported tags of cp311-cp311-linux_x86_64:
cp311-cp311-manylinux2014_x86_64
... 113 more ...
Not sure how to go about solving this. Installing lxml in the same pipeline selects the correct version:
Copy code
Run pip install lxml
Collecting lxml
  Downloading lxml-5.2.2-cp311-cp311-manylinux_2_28_x86_64.whl.metadata (3.4 kB)
Downloading lxml-5.2.2-cp311-cp311-manylinux_2_28_x86_64.whl (5.0 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.0/5.0 MB 23.6 MB/s eta 0:00:00
Installing collected packages: lxml
Successfully installed lxml-5.2.2

Notice:  A new release of pip is available: 24.0 -> 24.1
Notice:  To update, run: pip install --upgrade pip
Anyone seen this before?
w
What version of ubuntu is your action running?
m
22.04
Same as local dev environment. Installing the lxml package also downloads the same manylinux as is available in the lockfile
w
And you don't have anything noteworthy in 
pants.ci.toml
? And both your devcontainer and CI have the same glibc version I guess? I don't really have many ideas here, as I don't use lxml much and haven't run into this. Just trying to check off some sanities
And which pants version?
m
Yeah fair. Appreciate it.
• Not using a 
pants.ci.toml
at the moment, just 
pants.toml
• Checking version atm
pants.toml
Copy code
[GLOBAL]
pants_version = "2.21.0"
backend_packages = [
  "pants.backend.docker",
  "pants.backend.python",
  "pants.backend.experimental.python",
  "pants.backend.build_files.fmt.black",  
  "pants.backend.python.lint.docformatter",
  "pants.backend.python.lint.black",
  "pants.backend.experimental.python.lint.ruff.check",
  "pants.backend.python.lint.isort",
  "pants.backend.python.typecheck.mypy",
  "pants.backend.google_cloud_function.python"
]
build_ignore = [ ".github/workflows/templates/*/*" ]

[source]
root_patterns = [
    "src",
    "tests"
]

[python]
interpreter_constraints = ["==3.11.*"]
enable_resolves = true

[python.resolves]
python-default = "3rdparty/python-default.lock"

[python-infer]
use_rust_parser = true

[docker]
env_vars = [
    "DOCKER_BUILDKIT=0",
]
build_args = ["VERSION"]
ldd version: ldd (Ubuntu GLIBC 2.35-0ubuntu3.8) 2.35 on both
w
Okay, I think there might be a newer 
pex
to use, which uses pip 24.1 (pex 2.5.0 or pex 2.6.0) - which I think has improved error messages. Maybe that could enlighten us? https://www.pantsbuild.org/2.23/reference/subsystems/pex-cli Again, this is a novel error to me as well - so I'm debugging. Maybe someone else has seen this, and knows 🤷
👀 1
m
Trying to figure out how to install pants pre-release but can't find it easily in the docs
Any pointers?
w
Ah, you don't need to - you can put that in your pants.toml Let me do it locally and I'll send the snippet
Its also a lot easier with the correct JSON schema 🙂
I believe it should be this
Copy code
[pex-cli]
version = "v2.6.0"
known_versions = [
    "v2.6.0|macos_arm64|bc8fc449fc8165c395b26b407cc6033cf25cf02a76b7ede54828d2e899c1ee97|4157988", 
    "v2.6.0|macos_x86_64|71690e672871b55323f5d6ef9a3fe9705f1668662652c4081080e7ab27d44de3|4157988", 
    "v2.6.0|linux_x86_64|71690e672871b55323f5d6ef9a3fe9705f1668662652c4081080e7ab27d44de3|4157988", 
    "v2.6.0|linux_arm64|71690e672871b55323f5d6ef9a3fe9705f1668662652c4081080e7ab27d44de3|4157988" 
]
✅ 1
If you try locally and remotely, I'd be curious if the error is different ?
m
Copy code
pants generate-lockfiles --resolve=python-default
15:28:13.39 [INFO] Initializing scheduler...
15:28:16.06 [INFO] Scheduler initialized.
15:28:17.05 [ERROR] 1 Exception encountered:

Engine traceback:
  in `generate-lockfiles` goal

IntrinsicError: Wrong digest for downloaded file: want Digest { hash: Fingerprint<71690e672871b55323f5d6ef9a3fe9705f1668662652c4081080e7ab27d44de3>, size_bytes: 4157988 } got Digest { hash: Fingerprint<bc8fc449fc8165c395b26b407cc6033cf25cf02a76b7ede54828d2e899c1ee97>, size_bytes: 4157988 }
Not sure what to make of that 😅
w
Oh, did I bad math that? Seemed to work
Hmm, seems to work for me - that's odd.. I'm on Mac though. Did I mis-copy the arch names?
m
My bad maybe. Should have specified: I'm on an M3 mac but running a linux devcontainer
ARM-based
w
Is it supposed to be linux_x86-64? hyphen instead of underscore?
m
Still bad unfortunately:
Copy code
15:37:23.43 [INFO] Initializing scheduler...
15:37:25.98 [INFO] Scheduler initialized.
15:37:26.79 [ERROR] 1 Exception encountered:

Engine traceback:
  in `generate-lockfiles` goal

IntrinsicError: Wrong digest for downloaded file: want Digest { hash: Fingerprint<71690e672871b55323f5d6ef9a3fe9705f1668662652c4081080e7ab27d44de3>, size_bytes: 4157988 } got Digest { hash: Fingerprint<bc8fc449fc8165c395b26b407cc6033cf25cf02a76b7ede54828d2e899c1ee97>, size_bytes: 4157988 }
I'll take a look later. Have to pick up my kid from kindergarten. Thanks for your help!
w
Oh! I for sure blundered the copy/paste - the first hash is the correct one, and I didn't copy/paste them into the other 3 architectures
I think this should work?
Copy code
[pex-cli]
version = "v2.6.0"
known_versions = [
    "v2.6.0|macos_arm64|bc8fc449fc8165c395b26b407cc6033cf25cf02a76b7ede54828d2e899c1ee97|4157988", 
    "v2.6.0|macos_x86_64|bc8fc449fc8165c395b26b407cc6033cf25cf02a76b7ede54828d2e899c1ee97|4157988", 
    "v2.6.0|linux_x86_64|bc8fc449fc8165c395b26b407cc6033cf25cf02a76b7ede54828d2e899c1ee97|4157988", 
    "v2.6.0|linux_arm64|bc8fc449fc8165c395b26b407cc6033cf25cf02a76b7ede54828d2e899c1ee97|4157988" 
]
✅ 1
m
I was curious what would happen if I just installed the library from a URL and this works in GH actions. I'm not sure why this happens.
w
Did pex 2.6 give anymore insight into the error?
(pip 24.1, more accurately)
m
No, the same error unfortunately
For future reference: added lxml as an URL dependency to 'pyproject.toml' for both aarch64 and x86_64 platforms:
Copy code
lxml = [
    {url = "<https://github.com/lxml/lxml/releases/download/lxml-5.2.2/lxml-5.2.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl>", markers = "sys_platform == 'linux' and platform_machine == 'x86_64'"},
    {url = "<https://github.com/lxml/lxml/releases/download/lxml-5.2.2/lxml-5.2.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl>", markers = "sys_platform == 'linux' and platform_machine == 'aarch64'"}
]
Not ideal but works.
b
Heya, I think this will be a problem with “complete platforms”. I’m inferring that you’re building a Google cloud function target. But please share your BUILD file if not. https://github.com/pantsbuild/pants/discussions/18756 discusses how to set a complete platform file for lambda, I think the same thing should work for GCF.
m
Hi @broad-processor-92400. THanks for this. Yes I'm building a GCF and this looks promising. I'll update you on the results!
Thanks for your help @broad-processor-92400 and @wide-midnight-78598
👍 1
FYI: approach shared by @broad-processor-92400 worked. Made some minor changes to get this to work on GCP. Macro becomes:
Copy code
def custom_python_google_cloud_function(**kwargs):
    kwargs.setdefault("dependencies", [])#.extend(
    #    [
    #        # FIXME <https://github.com/pantsbuild/pants/issues/15454>
    #        "!!path/to/requirements#mypy",
    #        "!!path/to/requirements#sqlalchemy-stubs",
    #    ]
    #)
    kwargs["complete_platforms"] = ["3rdparty/platforms:google_cloud_function_python_3_11"]  
    python_google_cloud_function(**kwargs)
'complete platforms' file is generated on GCF using the following code. See [discussion](https://github.com/pantsbuild/pants/discussions/18756).
Copy code
python
import sh
import functions_framework

@functions_framework.http
def hello_http(request):
    print(sh.pex3("interpreter", "inspect", "--markers", "--tags"))
    return {'statusCode': 200, 'body': "{}"}
👍 1
22 Views
Open in Slack
PreviousNext
Powered by