hundreds-father-404
03/07/2022, 9:19 PMechoing-farmer-15630
03/08/2022, 12:58 PMFor the current interpreter at /usr/local/bin/python3.9 the full platform string is manylinux_2_28_x86_64-cp-3.9.9-cp39
... which is what you get from `pex3 interpreter inspect --all --verbose --indent 4:
{
"path": "/usr/local/bin/python3.9",
"version": "3.9.9",
"requirement": "CPython==3.9.9",
"platform": "manylinux_2_28_x86_64-cp-3.9.9-cp39",
"venv": false
}
I tried "pex3 interpreter inspect --markers --tags" and got... well, showers of things, which didn't help.
Building my pex setting platforms
to the above string errored out with PyYAML==5.4.1
(https://pypi.org/project/PyYAML/5.4.1/#files) which has cp39 manylinux1 and manylinux2014 wheels, so I can't even get past something that seems to have compatible wheels much less the problems I'm going to have when I get to source-only distributions (build a wheel in pants somehow?).
I know pants is relatively new to the Docker scene, but "build a pex that can run on docker" seems a useful thing, even if we need to spin up a container to do it. But I can't sell this to my team unless they can build locally (the old version was built entirely within Docker so worked just fine). If that's the way forward, is there a "known best way" to build a pants repository within docker that maintains some of the pants goodness like caching etc?echoing-farmer-15630
03/08/2022, 1:12 PMpex --platform="manylinux_2_35_x86_64-cp-3.9.9-cp39" PyYAML==5.4.1
ERROR: Could not find a version that satisfies the requirement PyYAML==5.4.1 (from versions: none)
ERROR: No matching distribution found for PyYAML==5.4.1
... I can't find a way to make even a pex with just PyYAML... there are a few packages that seem to work (requests
) and some that absolutely refuse (MarkupSafe
). I can get the PyYAML version to get a prompt with
❯ pex --platform="manylinux2014_x86_64-cp-3.9.9-cp39" PyYAML
WARNING: attempting to run PEX with incompatible platforms!
Running on platform manylinux_2_35_x86_64-cp-3.9.9-cp39 but built for manylinux2014_x86_64-cp-3.9.9-cp39
...so there's evidently an issue with manylinux2014 vs manylinux 2.35...?echoing-farmer-15630
03/08/2022, 1:15 PMplatforms=["manylinux_2_28_x86_64-cp-3.9-cp39", "manylinux2014_x86_64-cp-3.9-cp39"],
but no dice, still couldn't find a compatible wheel in pants
)echoing-farmer-15630
03/08/2022, 1:37 PMlively-exabyte-12840
03/08/2022, 3:10 PMbitter-ability-32190
03/08/2022, 5:45 PMrapid-crayon-8232
03/08/2022, 5:46 PMpython_requirement(
name="tensorflow",
requirements=["tensorflow==2.3.0"],
modules=["tensorflow", "numpy", "scipy", "h5py"],
)
resources(name="config-files", sources=[...])
python_sources(
name="lib",
dependencies=[":c-extensions", ":config-files", "!//:numpy", ":tensorflow"],
)
and the warning:
18:41:06.42 [WARN] The target my_target:../../../legacy imports `numpy`, but Pants cannot safely infer a dependency because more than one target owns this module, so it is ambiguous which to use: ['//:numpy', 'my_target:tensorflow'].
Please explicitly include the dependency you want in the `dependencies` field of my_target:../../../legacy, or ignore the ones you do not want by prefixing with `!` or `!!` so that one or no targets are left.
Alternatively, you can remove the ambiguity by deleting/changing some of the targets so that only 1 target owns this module. Refer to <https://www.pantsbuild.org/v2.9/docs/troubleshooting#import-errors-and-missing-dependencies>.
average-australia-85137
03/08/2022, 8:02 PMalthough it behaves identically to the python_tests target generator and you can safely use that insteadinstead of
although it behaves identically to the python_sources target generator and you can safely use that insteadIt has the same default sources as python_tests just adding conftest:
expected_file_extensions = (".py", "") # Note that this does not include `.pyi`.
default = ("test_*.py", "*_test.py", "tests.py")
class PythonTestUtilsGeneratingSourcesField(PythonGeneratingSourcesBase):
default = ("conftest.py", "test_*.pyi", "*_test.pyi", "tests.pyi")
[1] https://www.pantsbuild.org/docs/reference-python_test_utils#codesourcescodehundreds-father-404
03/08/2022, 9:55 PM--changed-since --changed-dependees
for Go to handle changes to go.mod
and go.sum
https://github.com/pantsbuild/pants/issues/14736flat-zoo-31952
03/08/2022, 9:59 PM./pants filter
combined with --tag
doesn't do what you'd think when, say, passing a list of targets to it via xargs, or giving a list of literal targets:
# item with a few tags
❯ ./pants peek tests/config_validation/test_verify_alert_rules.py | jq -r '.[0].tags | join(" ")'
local config
# let's try filtering with another tag
❯ ./pants --tag=unit filter tests/config_validation/test_verify_alert_rules.py
tests/config_validation/test_verify_alert_rules.py
# oops! we need tag-regex
❯ ./pants filter --tag-regex='^unit$' tests/config_validation/test_verify_alert_rules.py
wide-zoo-86070
03/08/2022, 10:12 PM.cache/pants
and .pex
to other destination?freezing-quill-50372
03/09/2022, 1:52 AMpython_source_root
, but the codes are generated in the original directory (detailed settings are following in thread)plain-fireman-49959
03/09/2022, 9:37 AMcythonize
some files using Pants. These are my attempts so far using a python_distribution
target:
• Dist has only dependencies
on the source package, uses a generated setup.py
and provides=python_artifact
, I can build a package (without cython ofc)
• Adding a pyproject.toml
dependency (requiring ["setuptools", "wheel", "Cython"]
does create a target, but it doesn't cythonize
files
• Adding a minimal setup.py
with _ext_modules_=cythonize([Extension("*", ["*.pyx"])])
produced this error: ValueError: '*.pyx' doesn't match any files
Has anyone dealt with this before? The .pyx
file lives at the same level as the setup.py
, I tried globbing with directories too (src/test/test.pyx
) without success.
I can provide more details if required, this setup is as minimal as I can get it (even though it's part of a bigger project)wide-zoo-86070
03/09/2022, 5:03 PMgentle-guitar-84783
03/09/2022, 7:27 PMpython_requirements()
macro doesn't seem to be picking it up when I run my tests? do I need to refresh the cache or something or at the minimum how can I debug this situation (is there a goal to validate what the macro expands to?)witty-family-13337
03/10/2022, 10:28 AMjvm_artifact
definition like this would fail to be recognised as a dependency for other targets:
jvm_artifact(
name="enumeratum",
group="com.beachape",
artifact="enumeratum-circe_2.13",
version="1.7+",
packages=[
"enumeratum.*",
"enumeratum.values.*",
],
)
however, if we change the packages
field to be like the following:
packages=[
"enumeratum.**",
"enumeratum.values.**",
]
then the dependency gets resolved… what is the difference between the single *
and the **
in this context? Is it advised that we always use the **
for these cases?freezing-area-97131
03/10/2022, 11:03 AM.pants.d
, and when I build the pex again, I got this error:
10:54:36 00:03 [cache]
No cached artifacts for 202 targets.
Invalidated 202 targets.**** Failed to install coverage-6.3.2 (caused by: NonZeroExit("received exit code 1 during execution of `[u'/usr/bin/python2.7', '-s', '-', 'bdist_wheel', '--dist-dir=/tmp/tmp4CEKZy']` while trying to execute `[u'/usr/bin/python2.7', '-s', '-', 'bdist_wheel', '--dist-dir=/tmp/tmp4CEKZy']`",)
):
stdout:
stderr:
Traceback (most recent call last):
File "<stdin>", line 14, in <module>
File "<string>", line 80
classifier_list.append(f"Development Status :: {devstat}")
^
SyntaxError: invalid syntax
Although I have specified the dependencies in the BUILD
file and the version in requirement.txt
, it still cannot build successfully. Anyone has suggestion?nice-florist-55958
03/10/2022, 1:19 PMpackage
goal.
Me/CI host/some others previously onboarded:
taymarti$ ./pants --print-stacktrace -ldebug --pex-verbosity=9 --no-pantsd package proj/libs/blade
18:19:50.84 [INFO] Wrote dist/morganstanley-casper-blade-0.1.5a2.tar.gz
18:19:50.84 [INFO] Wrote dist/morganstanley_casper_blade-0.1.5a2-py3-none-any.whl
New Onboardees:
hanzhw$ ./pants --print-stacktrace -ldebug --pex-verbosity=9 --no-pantsd package proj/libs/blade
18:11:54.45 [INFO] Completed: Building build_backend.pex from setuptools_default_lockfile.txt
When I ran in debug mode, we see at the very end:
`180447.28 [DEBUG] Completed: package
goal`
18:04:47.29 [DEBUG] computed 1 nodes in 3.834978 seconds. there are 7100 total nodes.
Here is the slimmed down output (a ...
is where I cut out text to make it short enough for the message -- if the full log is useful, I can create a link to it)
hanzhw$ ./pants --print-stacktrace -ldebug --pex-verbosity=9 --no-pantsd package proj/libs/blade
18:04:40.62 [DEBUG] File handle limit is: 10000
18:04:42.54 [DEBUG] Launching 1 roots (poll=false).
18:04:42.56 [DEBUG] computed 1 nodes in 0.014291 seconds. there are 7 total nodes.
18:04:42.95 [WARN] DEPRECATED: option 'interpreter_search_paths' in scope 'python' will be removed in version 2.10.0.dev0.
Moved to `[python-bootstrap] search_path`.
18:04:42.98 [DEBUG] File handle limit is: 10000
18:04:43.38 [DEBUG] specs are: Specs(address_specs=AddressSpecs(literals=(AddressLiteralSpec(path_component='proj/libs/blade', target_component=None, generated_component=None),), globs=(), filter_by_global_options=True), filesystem_specs=FilesystemSpecs(file_includes=(), dir_includes=(), ignores=()))
18:04:43.38 [DEBUG] changed_options are: ChangedOptions(since=None, diffspec=None, dependees=<DependeesOption.NONE: 'none'>)
18:04:43.45 [DEBUG] Launching 1 roots (poll=false).
18:04:43.45 [DEBUG] computed 1 nodes in 0.003152 seconds. there are 12 total nodes.
18:04:43.45 [DEBUG] requesting <class 'pants.core.goals.package.Package'> to satisfy execution of `package` goal
18:04:43.45 [DEBUG] Launching 1 roots (poll=false).
18:04:43.47 [DEBUG] Completed: Find targets from input specs
18:04:43.47 [WARN] DEPRECATED: option 'interpreter_search_paths' in scope 'python' will be removed in version 2.10.0.dev0.
Moved to `[python-bootstrap] search_path`.
18:04:43.48 [DEBUG] Completed: Determining the entry points for a `python_distribution` target
18:04:43.48 [DEBUG] Completed: Determining the entry points for a `python_distribution` target
18:04:44.30 [DEBUG] Completed: Find all targets in the project
18:04:44.31 [DEBUG] Completed: Find all Python targets in project
18:04:44.31 [DEBUG] Completed: Creating map of third party targets to Python modules
18:04:44.48 [DEBUG] Completed: Creating map of first party Python targets to Python modules
18:04:44.48 [DEBUG] Completed: pants.backend.python.dependency_inference.module_mapper.merge_first_party_module_mappings
18:04:44.51 [DEBUG] Completed: Downloading: DownloadFile(url='file:///v/region/na/appl/feg/rates/data/dev/nytools/casper/train/pantsbuild/binaries/pex_v2.1.61', expected_digest=FileDigest('8072340969ad517279f153551f34d6c43ba51f7984... (34 characters truncated)
18:04:44.51 [DEBUG] Completed: Extracting an archive file
18:04:44.51 [DEBUG] Completed: pants.core.util_rules.external_tool.download_external_tool
........
pex: Executing: /ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/bin/python3.7 -s -E /tmp/process-execution7nAJfJ/.tmp/tmpzp6_27gh -c import hashlib, os, sys
python = os.path.realpath(sys.executable)
print(python)
hasher = hashlib.sha256()
with open(python, "rb") as fp:
for chunk in iter(lambda: fp.read(8192), b""):
hasher.update(chunk)
print(hasher.hexdigest())
pex: PEX.run invoking /ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/bin/python3.7 -s -E /tmp/process-execution7nAJfJ/.tmp/tmpzp6_27gh -c import hashlib, os, sys
python = os.path.realpath(sys.executable)
print(python)
hasher = hashlib.sha256()
with open(python, "rb") as fp:
for chunk in iter(lambda: fp.read(8192), b""):
hasher.update(chunk)
print(hasher.hexdigest())
pex: Executing: /ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/bin/python3.7 -s -E /tmp/process-execution7nAJfJ/.tmp/tmpzp6_27gh -c import hashlib, os, sys
python = os.path.realpath(sys.executable)
print(python)
hasher = hashlib.sha256()
with open(python, "rb") as fp:
for chunk in iter(lambda: fp.read(8192), b""):
hasher.update(chunk)
print(hasher.hexdigest())
pex: Installed VendorImporter(root='/tmp/process-execution7nAJfJ/.tmp/tmpzp6_27gh/.bootstrap', importables=(_Importable(module='attr', is_pkg=True, path='pex/vendor/_vendored/attrs', prefix='pex.third_party'), _Importable(module='pyparsing', is_pkg=False, path='pex/vendor/_vendored/packaging', prefix='pex.third_party'), _Importable(module='packaging', is_pkg=True, path='pex/vendor/_vendored/packaging', prefix='pex.third_party'), _Importable(module='pkg_resources', is_pkg=True, path='pex/vendor/_vendored/setuptools', prefix='pex.third_party')))
pex: pex.third_party.attr imported via _Loader(module_name='pex.third_party.attr', vendor_module_name='pex.vendor._vendored.attrs.attr')
......
pex: creating PythonIdentity from encoded: {"abi_tag": "cp37m", "base_prefix": "/ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys", "binary": "/a/stor116ncs1.new-york.ms.com/sc19303/s96008/hanzhw/.cache/pants/named_caches/pex_root/venvs/29ce536258b55e43b46f5316f30b1f4881319563/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e.d2918d4275cd40728943cdbeb27e419b/bin/python3.7", "configured_macosx_deployment_target": "", "env_markers": {"implementation_name": "cpython", "implementation_version": "3.7.5", "os_name": "posix", "platform_machine": "x86_64", "platform_python_implementation": "CPython", "platform_release": "3.10.0-1160.42.2.el7.x86_64", "platform_system": "Linux", "platform_version": "#1 SMP Tue Aug 31 20:15:00 UTC 2021", "python_full_version": "3.7.5", "python_version": "3.7", "sys_platform": "linux"}, "platform_tag": "manylinux_2_17_x86_64", "prefix": "/a/stor116ncs1.new-york.ms.com/sc19303/s96008/hanzhw/.cache/pants/named_caches/pex_root/venvs/29ce536258b55e43b46f5316f30b1f4881319563/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e.d2918d4275cd40728943cdbeb27e419b", "python_tag": "cp37", "supported_tags": [["cp37", …]], "version": [3, 7, 5]}
pex: Re-writing /a/stor116ncs1.new-york.ms.com/sc19303/s96008/hanzhw/.cache/pants/named_caches/pex_root/venvs/29ce536258b55e43b46f5316f30b1f4881319563/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e.d2918d4275cd40728943cdbeb27e419b/bin/activate
pex: Re-writing /a/stor116ncs1.new-york.ms.com/sc19303/s96008/hanzhw/.cache/pants/named_caches/pex_root/venvs/29ce536258b55e43b46f5316f30b1f4881319563/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e.d2918d4275cd40728943cdbeb27e419b/bin/activate.csh
pex: Re-writing /a/stor116ncs1.new-york.ms.com/sc19303/s96008/hanzhw/.cache/pants/named_caches/pex_root/venvs/29ce536258b55e43b46f5316f30b1f4881319563/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e.d2918d4275cd40728943cdbeb27e419b/bin/activate.fish
pex: Seeding local caches for build_backend.pex :: Creating venv from build_backend.pex :: Searching dependency cache: /tmp/process-executionSLuaCY/.tmp/tmp19xs29u6/.deps
/a/stor116ncs1.new-york.ms.com/sc19303/s96008/hanzhw/.cache/pants/named_caches/pex_root/installed_wheels/f5f24693b0e9ee59a7c54d4d4dcd46e0714353cc/pex-2.1.61-py2.py3-none-any.whl/pex/pex_bootstrapper.py:493: PEXWarning: The venv for /tmp/process-executionSLuaCY/.tmp/tmp19xs29u6 at /a/stor116ncs1.new-york.ms.com/sc19303/s96008/hanzhw/.cache/pants/named_caches/pex_root/venvs/29ce536258b55e43b46f5316f30b1f4881319563/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e has script shebangs of '#!/a/stor116ncs1.new-york.ms.com/sc19303/s96008/hanzhw/.cache/pants/named_caches/pex_root/venvs/s/d599bbf3/venv/bin/python3.7 -sE' with 129 characters. On some systems this may be too long and cause problems running the venv scripts. You may be able adjust PEX_ROOT from /a/stor116ncs1.new-york.ms.com/sc19303/s96008/hanzhw/.cache/pants/named_caches/pex_root to a shorter path as a work-around.
pex_root=pex_info.pex_root,
pex: Seeding local caches for build_backend.pex: 537.6ms
pex: Creating venv from build_backend.pex: 537.4ms
pex: Searching dependency cache: /tmp/process-executionSLuaCY/.tmp/tmp19xs29u6/.deps: 0.1ms
18:04:47.28 [DEBUG] Completed: pants.backend.python.util_rules.pex.build_pex
18:04:47.28 [DEBUG] Completed: Hit: Local cache lookup: Run setuptools.build_meta:__legacy__ for proj/libs/blade:blade
18:04:47.28 [DEBUG] Completed: Scheduling: Run setuptools.build_meta:__legacy__ for proj/libs/blade:blade
18:04:47.28 [DEBUG] Completed: `package` goal
18:04:47.29 [DEBUG] computed 1 nodes in 3.834978 seconds. there are 7100 total nodes.
nice-florist-55958
03/10/2022, 1:29 PMrealpath
of the pants.toml
(SCM root) file inside pants.toml
file itself? We have a need to use file://
instead of external URLs, but this only works with absolute paths. Right now we are pointing to a shared network drive, but my preference is for this to point to files checked into SCM.
More generally, is there documentation of all accessible variables? I'm talking about things like {version}
. It doesn't look like Bash or Python are available in .toml
files.tall-country-45957
03/10/2022, 1:34 PMechoing-farmer-15630
03/10/2022, 3:26 PMADD
and probably COPY
instruction of the form ADD ./my_file /tmp/install/my_file
where my_file
is listed as part of a files
dependency works fine on buildbarn and buildgrid but on buildfarm generates the error
#8 ERROR: "/my_file" not found: not found
------
> [4/5] ADD ./my_file /tmp/install/my_file:
------
failed to compute cache key: "/my_file" not found: not found
...it could be I'm doing something strange, but given that it works on 2/3 build systems but fails on the third is interesting. Wanted to at least raise it for awareness but I'm not sure if I should file an issue given that I'm not sure which product the issue is with!
I'm not sure given the size of generated artifacts if remote execution or caching even makes all that much sense for a lot of pants builds since that's an awful lot of I/O, but it's certainly been a worthwhile experiment. I still may try it for our CI solution.
One of the buildfarm folks raised the idea that it may be worthwhile to add something to pants
to disable remote execution (and maybe remote caching) for only certain steps, such as building pexes via remote execution but build the docker containers locally (also useful because as @fast-nail-55400 has mentioned in another thread, docker caches are machine-specific, so it may make sense to optionally turn off remote execution for docker builds; I was doing the buildfarms locally with a mounted /var/run/docker.sock
so it was using my local machine anyway).echoing-farmer-15630
03/10/2022, 3:42 PMdocker_image
target depends on a pex_binary
target, could one tag the pex_binary
targets with one tag and the docker_image
tags with another, build the pex_binary
targets with remote execution on and then use those generated artifacts to build the docker_image
targets locally without recreating the pex targets?gentle-guitar-84783
03/10/2022, 8:06 PMrelocated_files
target, meaning I want to set src='{cwd}', dest=''
rather than having to write out the boilerplate myselffresh-cat-90827
03/11/2022, 11:46 AMpants
that lives in the repo root is updated, how would one force Pants to re-bootstrap itself using the latest file? (see https://www.pantsbuild.org/docs/upgrade-tips#check-for-updates-to-the-pants-script). Blowing the ~/.cache/pants
doesn’t feel very friendly 😕 I’m on 2.9.0
and after making changes to the pants
file that would cause Pants to fail bootstrapping, Pants still runs, ignoring the changes in the pants
file.curved-television-6568
03/11/2022, 11:53 AMfirst_party_dependency_version_scheme
field/option but for 3rdparty deps of a python_distribution
so that consumers of my lib are not tied down to exact versions of the transitive deps, as that makes it harder to maintain in case there are multiple libraries depending on the same 3rdparty deps, when released at different times, there could be compatible differences in patch versions for instance.acceptable-football-32760
03/11/2022, 2:38 PM<http://some.path.to|some.path.to>.pex.binary
) under dist/
configuarble to be /
-separated instead? The practicality comes from the fact that we share bazel and pants repos in one codebase and it would be nice for use to refer to the results of the different builds using a unified way.
Thank you!hundreds-father-404
03/11/2022, 5:01 PMhigh-yak-85899
03/11/2022, 7:21 PMstrong-toothbrush-37759
03/12/2022, 3:54 PMsrc/ - This is the django root folder
api/
core/
another_module1/
another_module2/
setup_core.py - This is building a core-package we upload and use in apps/project-z
manage.py
... etc
apps/
project-x/ - TypeScript Project
project-y/ - JavaScript Project
project-z/ - Python Project
As I’m trying to get Pants setup only for the src
directory. I configured root_patterns
to only contain '/src'
. I’m still wondering about the perfect configuration but this was my starting-point.
My findings:
1. ./pants tailor
does not seem to respect [source].root_patterns
. When I run tailor
it prompts me with a `NoSourceRootError: No source root found for `apps/heka/manage.py`` . Why is that exactly?
2. Removing the root_patterns
configuration ./pants tailor
runs successfully but quits with this warning: `164908.52 [INFO] Filesystem changed during run: retrying TailorGoal
in 500ms...` . Since I did not change anything, but pants did, is it possible it is tripping over its own changes here?
3. Playing around with the setup for fmt
and lint
, I figured that pants trips over requirements that come from github, i.e. we are using a library de_core_news_md
which is pulled by spaCy and cached by our CI using this url: <https://github.com/explosion/spacy-models/releases/download/de_core_news_md-3.1.0/de_core_news_md-3.1.0.tar.gz#egg=de_core_news_md==3.1.0>
. After running ./pants tailor
every following command halts immediately with the following error:
16:53:31.83 [ERROR] 1 Exception encountered:
MappingError: Failed to parse ./apps/prometheus/BUILD:
Invalid requirement '<https://github.com/explosion/spacy-models/releases/download/de_core_news_md-3.1.0/de_core_news_md-3.1.0.tar.gz#egg=de_core_news_md==3.1.0>' in apps/prometheus/requirements.txt at line 37: Parse error at "'://githu'": Expected stringEnd
Maybe someone can give me a hint in the right direction for incremental adoption, any help is appreciated 🙂.