https://pantsbuild.org/ logo
Join Slack
Powered by
I have a pyproject.toml that depends on some local...
# general
c
I have a pyproject.toml that depends on some local wheels (internal 3rdparty deps). Whenever I generate the lockfile (pex) I see that it uses the paths on my local computer… this means that I can’t really commit that lockfile to git (and thus have to manually regenerate on CI). Is there a way to get around this? Would the answer be to use a different resolve for local wheels and only generate that lockfile every time?
e
Are you in a position to try 2.14.0a1?: https://pypi.org/project/pantsbuild.pants/2.14.0a1/ Although Pex has supported 
--path-mapping
for just this case for a long time now, Pants just picked up support in https://github.com/pantsbuild/pants/pull/16625
c
oh nice. I’m a little hestiant to try and alpha build, but am fine with waiting till 2.14 comes out
f
Could this be done with config file templating, using e.g. 
%(buildroot)
pre-2.14? For example,
Copy code
[python-repos]
# Use an absolute path to a directory containing `.whl` and/or sdist files.
find_links = ["file://%(buildroot)/dist"]
Or would that result in the same problem of writing the (templated) absolute path into the lockfile?
e
Yeah, the `%(buildroot)s`is expanded early by Pants before it's handed as a real absolute path to Pex. Pex doesn't try to guess and requires you give it a `--path-mapping`to indicate a path that should be symbolized.
👍 1
c
@enough-analyst-54434 is there a full example of how this is meant to work somewhere?
i’m trying this with pants 2.14:
Copy code
find_links = ["file://%(buildroot)s/prebuilt_wheels",]
path_mappings = ["WHEELS_DIR|%(buildroot)s/prebuilt_wheels"]
e
I'm not sure, but I don't think so. If that's not working a quick thing to try is to get rid of the 
file://
prefix in 
find_links
.
c
nvm, I figured it out. I thought you still had to use direct references to the file when defining dependencies
realized now that you just need to define the dep as normal, and it worked. thanks!
e
Excellent.
c
hmm, i’m seeing this in my lockfile, though:
Copy code
"path_mappings": {
    "WHEELS_DIR": null
  },
e
That sounds like a bug. Can you re-run with 
./pants generate-lockfiles -ldebug
and look for the command line including the word 
universal
? That will tell us if Pants hand's off badly to Pex or if Pex fumbles.
c
sure. fwiw, everything still works 🤷
Copy code
14:11:59.35 [DEBUG] spawned local process as Some(26736) for Process { argv: ["/Users/ashu/.pyenv/versions/3.10.3/bin/python", "./pex", "lock", "create", "--tmpdir", ".tmp", "--python-path", "/Users/ashu/.pyenv/versions/3.10.3/bin:/Users/ashu/.pyenv/versions/3.8.12/bin:/Users/ashu/.pyenv/versions/3.9.10/bin:/Users/ashu/.pyenv/versions/3.9.11/bin:/Users/ashu/.pyenv/versions/3.9.12/bin:/opt/python3", "--output=lock.json", "--no-emit-warnings", "--style=universal", "--resolver-version", "pip-2020-resolver", "--target-system", "linux", "--target-system", "mac", "--indent=2", "--no-pypi", "--index=<https://pypi.org/simple/>", "--find-links=file:///Users/ashu/Documents/backend/prebuilt_wheels", "--manylinux", "manylinux2014", "--path-mapping=WHEELS_DIR|/Users/ashu/Documents/backend/prebuilt_wheels", "--interpreter-constraint", "CPython==3.9.10", "pytest==7.0.1", "pytest-cov>=2.12,!=2.12.1,<3.1", "pytest-xdist>=2.5,<3", "fakeredis[lupa]", "freezegun", "pytest-asyncio", "pytest-django", "pytest-env", "pytest-freezegun", "pytest-mock"], env: {"CPPFLAGS": "-I/opt/homebrew/opt/openssl/include -I/opt/homebrew/opt/openblas/include", "GRPC_PYTHON_BUILD_SYSTEM_OPENSSL": "1", "LANG": "en_US.UTF-8", "LDFLAGS": "-L/opt/homebrew/opt/openssl/lib -L/opt/homebrew/opt/openblas/lib", "PATH": "/Users/ashu/.pyenv/shims:/Users/ashu/.yarn/bin:/Users/ashu/.config/yarn/global/node_modules/.bin:/Users/ashu/.nvm/versions/node/v14.17.6/bin:/Users/ashu/.go/bin:/opt/homebrew/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/go/bin", "PEX_IGNORE_RCFILES": "true", "PEX_PYTHON_PATH": "/Users/ashu/.pyenv/versions/3.10.3/bin:/Users/ashu/.pyenv/versions/3.8.12/bin:/Users/ashu/.pyenv/versions/3.9.10/bin:/Users/ashu/.pyenv/versions/3.9.11/bin:/Users/ashu/.pyenv/versions/3.9.12/bin:/opt/python3", "PEX_ROOT": ".cache/pex_root", "PEX_SCRIPT": "pex3"}, working_directory: None, input_digests: InputDigests { complete: DirectoryDigest { digest: Digest { hash: Fingerprint<17cfe36e90b7216296f0651d9c72ef150c00d24bb1b99c8fe07134012366ce0b>, size_bytes: 158 }, tree: "Some(..)" }, nailgun: DirectoryDigest { digest: Digest { hash: Fingerprint<e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855>, size_bytes: 0 }, tree: "Some(..)" }, input_files: DirectoryDigest { digest: Digest { hash: Fingerprint<17cfe36e90b7216296f0651d9c72ef150c00d24bb1b99c8fe07134012366ce0b>, size_bytes: 158 }, tree: "Some(..)" }, immutable_inputs: {}, use_nailgun: {} }, output_files: {RelativePath("lock.json")}, output_directories: {}, timeout: None, execution_slot_variable: None, concurrency_available: 0, description: "Generate lockfile for pytest", level: Info, append_only_caches: {CacheName("pex_root"): RelativePath(".cache/pex_root")}, jdk_home: None, platform_constraint: None, cache_scope: PerSession, remote_cache_speculation_delay: 0ns }
@enough-analyst-54434 I’m also running into errors where the hashes of the wheels being built on different machines are different (expectedly), and pants is complaining. is there a way to ignore the hashes?
e
There is not.
Ok, 
--path-mapping=WHEELS_DIR|/Users/ashu/Documents/backend/prebuilt_wheels
looks like Pants is doing the right thing when creating the lock. You say, though that the lock file json has a path_mappings with null valued entries?
c
correct
so i’d have to upload the wheels as part of my git repo?
e
I have not paged back in full context, but if you want to stand a chance at a consistent resolve / lock file, you must have a stable set of wheels - no matter where they come from.
c
got it. I guess since our wheels are platform independent its fine. I’m not entirely sure how it would work for a platfoorm dependent wheel, though
it looks like 
pip wheel
is non-deterministic, which is annoying
e
Have you tried https://reproducible-builds.org/docs/source-date-epoch/? I think I did this before and found it doesn't help with 
pip wheel
but worth a quick try.
Ok. Sorry for the delay there. This is my own code and I forget it!
The null is correct! So the key is the name of the path mapping that must be defined to use the lock, the null value is an optional descrption of the path mapping that Pants doesn't currently fill in.
The idea was that if you had 3 path mappings, you could add text to say what each was to help someone set up the proper mappings when they later went to use the lock on some other machine with - perhaps - wildly different paths.
So:
Copy code
"path_mappings": {
  "WHEELS_DIR": "XYZ Corp standard wheels.",
  "ML_WHEELS_DIR": "XYZ Corp GPU-specific wheels."
}
7 Views
Open in Slack
PreviousNext
Powered by