https://pantsbuild.org/ logo
Join Slack
Powered by
I've been setting up other users on our team w/ ou...
# general
n
I've been setting up other users on our team w/ our new Pants-based repo. There is a step in our setup script where our internal packages are installed in edit mode, and it's here I discovered newly onboarded people have issues w/ the 
package
goal. Me/CI host/some others previously onboarded: 
taymarti$ ./pants --print-stacktrace -ldebug --pex-verbosity=9 --no-pantsd package proj/libs/blade
18:19:50.84 [INFO] Wrote dist/morganstanley-casper-blade-0.1.5a2.tar.gz
18:19:50.84 [INFO] Wrote dist/morganstanley_casper_blade-0.1.5a2-py3-none-any.whl
New Onboardees: 
hanzhw$ ./pants --print-stacktrace -ldebug --pex-verbosity=9 --no-pantsd package proj/libs/blade
18:11:54.45 [INFO] Completed: Building build_backend.pex from setuptools_default_lockfile.txt
When I ran in debug mode, we see at the very end: `180447.28 [DEBUG] Completed: 
package
goal` 
18:04:47.29 [DEBUG] computed 1 nodes in 3.834978 seconds. there are 7100 total nodes.
Here is the slimmed down output (a 
...
is where I cut out text to make it short enough for the message -- if the full log is useful, I can create a link to it)
Copy code
hanzhw$ ./pants --print-stacktrace -ldebug --pex-verbosity=9 --no-pantsd package proj/libs/blade
18:04:40.62 [DEBUG] File handle limit is: 10000
18:04:42.54 [DEBUG] Launching 1 roots (poll=false).
18:04:42.56 [DEBUG] computed 1 nodes in 0.014291 seconds. there are 7 total nodes.
18:04:42.95 [WARN] DEPRECATED: option 'interpreter_search_paths' in scope 'python' will be removed in version 2.10.0.dev0.

Moved to `[python-bootstrap] search_path`.
18:04:42.98 [DEBUG] File handle limit is: 10000
18:04:43.38 [DEBUG] specs are: Specs(address_specs=AddressSpecs(literals=(AddressLiteralSpec(path_component='proj/libs/blade', target_component=None, generated_component=None),), globs=(), filter_by_global_options=True), filesystem_specs=FilesystemSpecs(file_includes=(), dir_includes=(), ignores=()))
18:04:43.38 [DEBUG] changed_options are: ChangedOptions(since=None, diffspec=None, dependees=<DependeesOption.NONE: 'none'>)
18:04:43.45 [DEBUG] Launching 1 roots (poll=false).
18:04:43.45 [DEBUG] computed 1 nodes in 0.003152 seconds. there are 12 total nodes.
18:04:43.45 [DEBUG] requesting <class 'pants.core.goals.package.Package'> to satisfy execution of `package` goal
18:04:43.45 [DEBUG] Launching 1 roots (poll=false).
18:04:43.47 [DEBUG] Completed: Find targets from input specs
18:04:43.47 [WARN] DEPRECATED: option 'interpreter_search_paths' in scope 'python' will be removed in version 2.10.0.dev0.

Moved to `[python-bootstrap] search_path`.
18:04:43.48 [DEBUG] Completed: Determining the entry points for a `python_distribution` target
18:04:43.48 [DEBUG] Completed: Determining the entry points for a `python_distribution` target
18:04:44.30 [DEBUG] Completed: Find all targets in the project
18:04:44.31 [DEBUG] Completed: Find all Python targets in project
18:04:44.31 [DEBUG] Completed: Creating map of third party targets to Python modules
18:04:44.48 [DEBUG] Completed: Creating map of first party Python targets to Python modules
18:04:44.48 [DEBUG] Completed: pants.backend.python.dependency_inference.module_mapper.merge_first_party_module_mappings
18:04:44.51 [DEBUG] Completed: Downloading: DownloadFile(url='file:///v/region/na/appl/feg/rates/data/dev/nytools/casper/train/pantsbuild/binaries/pex_v2.1.61', expected_digest=FileDigest('8072340969ad517279f153551f34d6c43ba51f7984... (34 characters truncated)
18:04:44.51 [DEBUG] Completed: Extracting an archive file
18:04:44.51 [DEBUG] Completed: pants.core.util_rules.external_tool.download_external_tool
........
pex: Executing: /ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/bin/python3.7 -s -E /tmp/process-execution7nAJfJ/.tmp/tmpzp6_27gh -c import hashlib, os, sys

python = os.path.realpath(sys.executable)
print(python)

hasher = hashlib.sha256()
with open(python, "rb") as fp:
  for chunk in iter(lambda: fp.read(8192), b""):
      hasher.update(chunk)
print(hasher.hexdigest())

pex: PEX.run invoking /ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/bin/python3.7 -s -E /tmp/process-execution7nAJfJ/.tmp/tmpzp6_27gh -c import hashlib, os, sys

python = os.path.realpath(sys.executable)
print(python)

hasher = hashlib.sha256()
with open(python, "rb") as fp:
  for chunk in iter(lambda: fp.read(8192), b""):
      hasher.update(chunk)
print(hasher.hexdigest())

pex: Executing: /ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/bin/python3.7 -s -E /tmp/process-execution7nAJfJ/.tmp/tmpzp6_27gh -c import hashlib, os, sys

python = os.path.realpath(sys.executable)
print(python)

hasher = hashlib.sha256()
with open(python, "rb") as fp:
  for chunk in iter(lambda: fp.read(8192), b""):
      hasher.update(chunk)
print(hasher.hexdigest())

pex: Installed VendorImporter(root='/tmp/process-execution7nAJfJ/.tmp/tmpzp6_27gh/.bootstrap', importables=(_Importable(module='attr', is_pkg=True, path='pex/vendor/_vendored/attrs', prefix='pex.third_party'), _Importable(module='pyparsing', is_pkg=False, path='pex/vendor/_vendored/packaging', prefix='pex.third_party'), _Importable(module='packaging', is_pkg=True, path='pex/vendor/_vendored/packaging', prefix='pex.third_party'), _Importable(module='pkg_resources', is_pkg=True, path='pex/vendor/_vendored/setuptools', prefix='pex.third_party')))
pex: pex.third_party.attr imported via _Loader(module_name='pex.third_party.attr', vendor_module_name='pex.vendor._vendored.attrs.attr')
......

pex: creating PythonIdentity from encoded: {"abi_tag": "cp37m", "base_prefix": "/ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys", "binary": "/a/stor116ncs1.new-york.ms.com/sc19303/s96008/hanzhw/.cache/pants/named_caches/pex_root/venvs/29ce536258b55e43b46f5316f30b1f4881319563/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e.d2918d4275cd40728943cdbeb27e419b/bin/python3.7", "configured_macosx_deployment_target": "", "env_markers": {"implementation_name": "cpython", "implementation_version": "3.7.5", "os_name": "posix", "platform_machine": "x86_64", "platform_python_implementation": "CPython", "platform_release": "3.10.0-1160.42.2.el7.x86_64", "platform_system": "Linux", "platform_version": "#1 SMP Tue Aug 31 20:15:00 UTC 2021", "python_full_version": "3.7.5", "python_version": "3.7", "sys_platform": "linux"}, "platform_tag": "manylinux_2_17_x86_64", "prefix": "/a/stor116ncs1.new-york.ms.com/sc19303/s96008/hanzhw/.cache/pants/named_caches/pex_root/venvs/29ce536258b55e43b46f5316f30b1f4881319563/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e.d2918d4275cd40728943cdbeb27e419b", "python_tag": "cp37", "supported_tags": [["cp37", …]], "version": [3, 7, 5]}
pex: Re-writing /a/stor116ncs1.new-york.ms.com/sc19303/s96008/hanzhw/.cache/pants/named_caches/pex_root/venvs/29ce536258b55e43b46f5316f30b1f4881319563/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e.d2918d4275cd40728943cdbeb27e419b/bin/activate
pex: Re-writing /a/stor116ncs1.new-york.ms.com/sc19303/s96008/hanzhw/.cache/pants/named_caches/pex_root/venvs/29ce536258b55e43b46f5316f30b1f4881319563/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e.d2918d4275cd40728943cdbeb27e419b/bin/activate.csh
pex: Re-writing /a/stor116ncs1.new-york.ms.com/sc19303/s96008/hanzhw/.cache/pants/named_caches/pex_root/venvs/29ce536258b55e43b46f5316f30b1f4881319563/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e.d2918d4275cd40728943cdbeb27e419b/bin/activate.fish
pex: Seeding local caches for build_backend.pex :: Creating venv from build_backend.pex :: Searching dependency cache: /tmp/process-executionSLuaCY/.tmp/tmp19xs29u6/.deps
/a/stor116ncs1.new-york.ms.com/sc19303/s96008/hanzhw/.cache/pants/named_caches/pex_root/installed_wheels/f5f24693b0e9ee59a7c54d4d4dcd46e0714353cc/pex-2.1.61-py2.py3-none-any.whl/pex/pex_bootstrapper.py:493: PEXWarning: The venv for /tmp/process-executionSLuaCY/.tmp/tmp19xs29u6 at /a/stor116ncs1.new-york.ms.com/sc19303/s96008/hanzhw/.cache/pants/named_caches/pex_root/venvs/29ce536258b55e43b46f5316f30b1f4881319563/9a128dacefb3843fa45de2c0dc225c7ee1cb4d0e has script shebangs of '#!/a/stor116ncs1.new-york.ms.com/sc19303/s96008/hanzhw/.cache/pants/named_caches/pex_root/venvs/s/d599bbf3/venv/bin/python3.7 -sE' with 129 characters. On some systems this may be too long and cause problems running the venv scripts. You may be able adjust PEX_ROOT from /a/stor116ncs1.new-york.ms.com/sc19303/s96008/hanzhw/.cache/pants/named_caches/pex_root to a shorter path as a work-around.
  pex_root=pex_info.pex_root,
pex: Seeding local caches for build_backend.pex: 537.6ms
pex:   Creating venv from build_backend.pex: 537.4ms
pex:     Searching dependency cache: /tmp/process-executionSLuaCY/.tmp/tmp19xs29u6/.deps: 0.1ms

18:04:47.28 [DEBUG] Completed: pants.backend.python.util_rules.pex.build_pex
18:04:47.28 [DEBUG] Completed: Hit: Local cache lookup: Run setuptools.build_meta:__legacy__ for proj/libs/blade:blade
18:04:47.28 [DEBUG] Completed: Scheduling: Run setuptools.build_meta:__legacy__ for proj/libs/blade:blade
18:04:47.28 [DEBUG] Completed: `package` goal
18:04:47.29 [DEBUG] computed 1 nodes in 3.834978 seconds. there are 7100 total nodes.
e
There is a step in our setup script where our internal packages are installed in edit mode
I'm surprised this works at all. Edit mode means edit mode in this venv over here IIUC and Pants does everything possible to isolate itself from the environment generally and any particular venv on your machine in particular. So even if an edit mode install works (doesn't error) is it actually happening in the venv you need it to happen in?
n
Sorry, should have clarified. We only use Pants for packaging the library as a convenience for getting a valid setup.py file. Then we just direct Pip to do an editable install (no more Pants involvement) w/ this setup file and cleanup everything after.
Oh, one more thing. The basic goals like 
version
, 
list
, 
count-loc
, etc. all work for these users. I haven't done a more thorough deep-dive yet--I'll try to do that today--it will be interesting to see if the goals that rely on 
package
such as 
run
and 
test
work or not.
e
We only use Pants for packaging the library as a convenience for getting a valid setup.py file. Then we just direct Pip to do an editable install (no more Pants involvement)
Aha - OK, Makes sense. Thanks for clarifying that. I'll stare at that output a bit more then with that in mind... So 
run
and 
test
do not rely on 
package
except under very special circumstances you configure explicitly in BUILD files. Is that something you're aware of and have already done?
Ok, so to your original post - what version of Pants is this? / Is this just a UI output bug for new onboardees? In other words, have you checked the 
dist/
dir for new onboardees to see if the packages were output there? I ask about Pants version because we've had some console UI bugs that involved some items not being flushed towards end of run that have been fixed in more recent versions.
n
Version is 
2.9.0
and it's a real issue; 
dist
gets created, but there is nothing there after the command completes.
e
Hrm.
So, remind me: is your Pants setup to have its caches (default ~/.cache/pants) on a shared NFS mount?
I mean 
/a/stor116ncs1.new-york.ms.com/sc19303/s96008/hanzhw/.cache/pants
looks NFS but it does not look shared. It looks like it belongs to 
hanzhw
but I want to make sure.
n
Correct, these are all local user installs / usages of Pants. Indeed, the paths are all NFS, but not shared, and the same setup as mine when I run locally.
e
Ok. If you don't mind comparing success output. I run the equivalent in the pants repo and see this tail content:
Copy code
^jsirois@gill ~/dev/pantsbuild/pants ((release_2.9.0)) $ ./pants --print-stacktrace -ldebug --pex-verbosity=9 --no-pantsd package src/python/pants:pants-packaged
...
pex: Seeding local caches for build_backend.pex: 358.9ms
pex:   Creating venv from build_backend.pex: 358.7ms
pex:     Searching dependency cache: /tmp/process-executionfSL3Gz/.tmp/tmpesl1pw3g/.deps: 3.8ms
pex:       Adding setuptools 57.5.0: 0.1ms
pex:       Adding wheel 0.37.1: 0.1ms
pex:     Resolving setuptools==57.5.0; python_version >= "3.6": 60.3ms
pex:     Resolving wheel==0.37.1; (python_version >= "2.7" and python_full_version < "3.0.0") or python_full_version >= "3.5.0": 3.6ms

06:30:52.65 [DEBUG] Completed: pants.backend.python.util_rules.pex.build_pex
06:30:52.65 [DEBUG] Running Run setuptools.build_meta:__legacy__ for src/python/pants:pants-packaged under semaphore with concurrency id: 16
06:30:52.70 [DEBUG] spawned local process as Some(25799) for Process { argv: ["../build_backend.pex_pex_shim.sh", "backend_shim.py"], env: {}, working_directory: Some(RelativePath("chroot")), input_digests: InputDigests { complete: Digest { hash: Fingerprint<415b2b75e66324fd10f3f3fb1a49f708d102071be9681b6c65b8dc192dc7370d>, size_bytes: 392 }, nailgun: Digest { hash: Fingerprint<e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855>, size_bytes: 0 }, input_files: Digest { hash: Fingerprint<415b2b75e66324fd10f3f3fb1a49f708d102071be9681b6c65b8dc192dc7370d>, size_bytes: 392 }, immutable_inputs: {}, use_nailgun: [] }, output_files: {}, output_directories: {RelativePath("dist")}, timeout: None, execution_slot_variable: None, description: "Run setuptools.build_meta:__legacy__ for src/python/pants:pants-packaged", level: Debug, append_only_caches: {CacheName("pex_root"): RelativePath(".cache/pex_root")}, jdk_home: None, platform_constraint: None, cache_scope: Successful }
06:30:56.96 [DEBUG] Completed: Run setuptools.build_meta:__legacy__ for src/python/pants:pants-packaged
06:30:56.96 [DEBUG] Completed: Scheduling: Run setuptools.build_meta:__legacy__ for src/python/pants:pants-packaged
06:30:56.99 [INFO] Wrote dist/pantsbuild.pants-2.9.0-cp37-cp37m-linux_x86_64.whl
06:30:56.99 [DEBUG] Completed: `package` goal
06:30:56.99 [DEBUG] computed 1 nodes in 7.315042 seconds. there are 17990 total nodes.
06:30:57.00 [DEBUG] Async completion is disabled: waiting for workunit callbacks to complete...
06:30:57.00 [DEBUG] Starting new HTTPS connection (1): spire.bugout.dev:443
06:30:57.41 [DEBUG] <https://spire.bugout.dev:443> "POST /humbug/reports HTTP/1.1" 200 None
^jsirois@gill ~/dev/pantsbuild/pants ((release_2.9.0)) $
A few suspicious differences from your detailed failure output: 1. You: 
DEBUG] Completed: Hit: Local cache lookup: Run setuptools.build_meta:__legacy__ for proj/libs/blade:blade
Me: 
[DEBUG] Completed: Run setuptools.build_meta:__legacy__ for src/python/pants:pants-packaged
I get this same output on every (repeated) run. Note that you have 
Hit: Local cache lookup
and I do not. 2. Me:
Copy code
pex: Seeding local caches for build_backend.pex: 358.9ms
pex:   Creating venv from build_backend.pex: 358.7ms
pex:     Searching dependency cache: /tmp/process-executionfSL3Gz/.tmp/tmpesl1pw3g/.deps: 3.8ms
pex:       Adding setuptools 57.5.0: 0.1ms
pex:       Adding wheel 0.37.1: 0.1ms
pex:     Resolving setuptools==57.5.0; python_version >= "3.6": 60.3ms
pex:     Resolving wheel==0.37.1; (python_version >= "2.7" and python_full_version < "3.0.0") or python_full_version >= "3.5.0": 3.6ms
You:
Copy code
pex: Seeding local caches for build_backend.pex: 537.6ms
pex:   Creating venv from build_backend.pex: 537.4ms
pex:     Searching dependency cache: /tmp/process-executionSLuaCY/.tmp/tmp19xs29u6/.deps: 0.1ms
So 1st stop - I'd like to drill in on why we're seeing different types of logging. 1st stop there is pants.toml config. How much of that can you share?
n
I can share the entire thing, 1sec. For the difference in logging, what do you mean? I pasted 3 outputs: 1. user: taymarti w/ no debug/verbose flags 2. user: hanzhw w/ no debug/verbose flags 3. user: hanzhw w/ debug/verbose flags The difference in (1) and (2) is the final message (pants wrote packages to dist vs. pex backend build completed). (3) doesn't really have a comparison, it's just the long-form of (2). Would it be helpful for me to run #4 like #3 but with me?
e
I think it would be helpful to have full output for all relevant cases in 1 compact spot if you can arrange that. If an issue works, great. But however you can.
So, by full output I mean the command line run and all following output and that same command line run for each case - success and failure. This will give me full apples-apples and help drilling in on the detailed differences.
Its disconcerting that I'm not seeing ~identical output for sub-sections of the run when I'm using 2.9.0 and all the same command line arguments as your last example save for the target. So having full logs + commands for each case will be useful.
n
Copy code
[GLOBAL]
ca_certs_path = "./pantsbuild/ms-ca-chain.pem"

pants_version = "2.9.0"

colors = true

pantsd = false

backend_packages = [
   "pants.backend.python",
   "pants.backend.experimental.python.lint.autoflake",
   "pants.backend.python.lint.isort",
   "pants.backend.python.lint.black",
   "pants.backend.python.typecheck.mypy",
]

pants_ignore = [
   "scripts/",
   "casper/proto/",
   "**.venv**/",
   "docs/",
   "setup.py",
   "!dist/docs/",
   "proj/user/**"
]

[anonymous-telemetry]
enabled = false

[source]
root_patterns = [
   "/3rdparty",
   # QUESTION: Why does this not work and separate app/lib qualifiers are needed?
   #"/proj/**/src",
   #"/proj/**/tests",
   "/proj/libs/**/src",
   "/proj/libs/**/tests",
   "/proj/apps/**/src",
   "/proj/apps/**/tests",
]

[python-repos]
# indexes = [ "" ] # Redacted

[download-pex-bin]
url_template = "file://%(buildroot)s/pantsbuild/binaries/pex_{version}"

[scc]
url_template = "file://%(buildroot)s/pantsbuild/binaries/scc-{version}-{platform}.zip"

[python]
tailor_ignore_solitary_init_files = false
interpreter_constraints = [
   "CPython>=3.6,<4",
]
interpreter_search_paths = [
   "/ms/dist/python/PROJ/core/3.7.5/exec/bin",
]
requirement_constraints = "3rdparty/constraints.txt"

[python-infer]
inits = true

[test]
output = "all"
use_coverage = true
extra_env_vars = ["LD_LIBRARY_PATH", "SYS_LOC"]

[pytest]
args = ["-vv", "--pdb"]

[coverage-py]
report = ["raw", "xml", "html", "json", "console"]

[mypy]
args = ["--linecount-report=reports", "--xml-report=reports", "--xslt-html-report=reports"]
extra_requirements = ["lxml==4.6.4"]
lockfile = "3rdparty/mypy_lockfile.txt"
Ok, I'll compile the full outputs and start an issue.
e
OK - great, Thank you.
n
Sent you a DM.
h
Thanks, can you link to the issue?
For posterity?
n
I think the culprit lies at the end during the build pex phase. In my run:
Copy code
pex: Hashing pex
pex: Hashing pex: 742.5ms
pex: Isolating pex
pex: Isolating pex: 1.6ms
pex: Building pex :: Resolving distributions (setuptools_default_lockfile.txt) :: Resolving requirements. :: Calculating project names for direct requirements:
  PyPIRequirement(line=LogicalLine(raw_text='setuptools==57.5.0; python_version >= "3.6" \\\n    --hash=sha256:60d78588f15b048f86e35cdab73003d8b21dd45108ee61a6693881a427f22073 \\\n    --hash=sha256:d9d3266d50f59c6967b9312844470babbdb26304fe740833a5f8d89829ba3a24\n', processed_text='setuptools==57.5.0; python_version >= "3.6" --hash=sha256:60d78588f15b048f86e35cdab73003d8b21dd45108ee61a6693881a427f22073 --hash=sha256:d9d3266d50f59c6967b9312844470babbdb26304fe740833a5f8d89829ba3a24', source='/tmp/process-executionRYiECS/setuptools_default_lockfile.txt', start_line=18, end_line=20), requirement=Requirement.parse('setuptools==57.5.0; python_version >= "3.6"'), editable=False)
  PyPIRequirement(line=LogicalLine(raw_text='wheel==0.37.1; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.5.0") \\\n    --hash=sha256:4bdcd7d840138086126cd09254dc6195fb4fc6f01c050a1d7236f2630db1d22a \\\n    --hash=sha256:e9a504e793efbca1b8e0e9cb979a249cf4a0a7b5b8c9e8b65a5e39d49529c1c4\n', processed_text='wheel==0.37.1; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.5.0") --hash=sha256:4bdcd7d840138086126cd09254dc6195fb4fc6f01c050a1d7236f2630db1d22a --hash=sha256:e9a504e793efbca1b8e0e9cb979a249cf4a0a7b5b8c9e8b65a5e39d49529c1c4', source='/tmp/process-executionRYiECS/setuptools_default_lockfile.txt', start_line=21, end_line=23), requirement=Requirement.parse('wheel==0.37.1; (python_version >= "2.7" and python_full_version < "3.0.0") or python_full_version >= "3.5.0"'), editable=False)
pex: Building pex :: Resolving distributions (setuptools_default_lockfile.txt) :: Resolving requirements. :: Installing:
  InstallRequest(target=DistributionTarget(interpreter=PythonInterpreter('/ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/bin/python3.7', PythonIdentity('/ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/bin/python3.7', 'cp37', 'cp37m', 'manylinux_2_17_x86_64', (3, 7, 5)))), wheel_path='/tmp/process-executionRYiECS/.tmp/tmpmc1petv9/ms.dist.python.PROJ.core.3.7.5-0..exec.@sys.bin.python3.7/wheel-0.37.1-py2.py3-none-any.whl', fingerprint='86aa63600ec29372a677394df507f70953e44962')
  InstallRequest(target=DistributionTarget(interpreter=PythonInterpreter('/ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/bin/python3.7', PythonIdentity('/ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/bin/python3.7', 'cp37', 'cp37m', 'manylinux_2_17_x86_64', (3, 7, 5)))), wheel_path='/tmp/process-executionRYiECS/.tmp/tmpmc1petv9/ms.dist.python.PROJ.core.3.7.5-0..exec.@sys.bin.python3.7/setuptools-57.5.0-py3-none-any.whl', fingerprint='3d985bbcccef78d812efe081dc4e21174fae56a0')
pex: Using cached installation of wheel-0.37.1-py2.py3-none-any.whl at /a/nyn183f2/vol/nyn183f2v18/u_t1445795813/taymarti/.cache/pants/named_caches/pex_root/installed_wheels/86aa63600ec29372a677394df507f70953e44962/wheel-0.37.1-py2.py3-none-any.whl
pex: Using cached installation of setuptools-57.5.0-py3-none-any.whl at /a/nyn183f2/vol/nyn183f2v18/u_t1445795813/taymarti/.cache/pants/named_caches/pex_root/installed_wheels/3d985bbcccef78d812efe081dc4e21174fae56a0/setuptools-57.5.0-py3-none-any.whl
pex: Spawning a maximum of 10 parallel jobs to process:
  
pex: Building pex: 3463.0ms
pex:   Resolving distributions (setuptools_default_lockfile.txt): 3462.5ms
pex:     Resolving requirements.: 3462.1ms
pex:       Resolving for:
  DistributionTarget(interpreter=PythonInterpreter('/ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/bin/python3.7', PythonIdentity('/ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/bin/python3.7', 'cp37', 'cp37m', 'manylinux_2_17_x86_64', (3, 7, 5)))): 3143.3ms
pex:       Calculating project names for direct requirements:
  PyPIRequirement(line=LogicalLine(raw_text='setuptools==57.5.0; python_version >= "3.6" \\\n    --hash=sha256:60d78588f15b048f86e35cdab73003d8b21dd45108ee61a6693881a427f22073 \\\n    --hash=sha256:d9d3266d50f59c6967b9312844470babbdb26304fe740833a5f8d89829ba3a24\n', processed_text='setuptools==57.5.0; python_version >= "3.6" --hash=sha256:60d78588f15b048f86e35cdab73003d8b21dd45108ee61a6693881a427f22073 --hash=sha256:d9d3266d50f59c6967b9312844470babbdb26304fe740833a5f8d89829ba3a24', source='/tmp/process-executionRYiECS/setuptools_default_lockfile.txt', start_line=18, end_line=20), requirement=Requirement.parse('setuptools==57.5.0; python_version >= "3.6"'), editable=False)
  PyPIRequirement(line=LogicalLine(raw_text='wheel==0.37.1; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.5.0") \\\n    --hash=sha256:4bdcd7d840138086126cd09254dc6195fb4fc6f01c050a1d7236f2630db1d22a \\\n    --hash=sha256:e9a504e793efbca1b8e0e9cb979a249cf4a0a7b5b8c9e8b65a5e39d49529c1c4\n', processed_text='wheel==0.37.1; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.5.0") --hash=sha256:4bdcd7d840138086126cd09254dc6195fb4fc6f01c050a1d7236f2630db1d22a --hash=sha256:e9a504e793efbca1b8e0e9cb979a249cf4a0a7b5b8c9e8b65a5e39d49529c1c4', source='/tmp/process-executionRYiECS/setuptools_default_lockfile.txt', start_line=21, end_line=23), requirement=Requirement.parse('wheel==0.37.1; (python_version >= "2.7" and python_full_version < "3.0.0") or python_full_version >= "3.5.0"'), editable=False): 0.2ms
pex:       Installing:
  InstallRequest(target=DistributionTarget(interpreter=PythonInterpreter('/ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/bin/python3.7', PythonIdentity('/ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/bin/python3.7', 'cp37', 'cp37m', 'manylinux_2_17_x86_64', (3, 7, 5)))), wheel_path='/tmp/process-executionRYiECS/.tmp/tmpmc1petv9/ms.dist.python.PROJ.core.3.7.5-0..exec.@sys.bin.python3.7/wheel-0.37.1-py2.py3-none-any.whl', fingerprint='86aa63600ec29372a677394df507f70953e44962')
  InstallRequest(target=DistributionTarget(interpreter=PythonInterpreter('/ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/bin/python3.7', PythonIdentity('/ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/bin/python3.7', 'cp37', 'cp37m', 'manylinux_2_17_x86_64', (3, 7, 5)))), wheel_path='/tmp/process-executionRYiECS/.tmp/tmpmc1petv9/ms.dist.python.PROJ.core.3.7.5-0..exec.@sys.bin.python3.7/setuptools-57.5.0-py3-none-any.whl', fingerprint='3d985bbcccef78d812efe081dc4e21174fae56a0'): 306.4ms
In other's run, almost all of this is missing, so presumably nothing is to be disted:
Copy code
pex: Hashing pex
pex: Hashing pex: 557.7ms
pex: Isolating pex
pex: Isolating pex: 0.6ms
pex: Building pex: 603.8ms
pex:   Resolving distributions (setuptools_default_lockfile.txt): 603.3ms
pex:     Resolving requirements.: 603.3ms
pex:       Resolving for:
  DistributionTarget(interpreter=PythonInterpreter('/ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/bin/python3.7', PythonIdentity('/ms/dist/python/PROJ/core/3.7.5-0/.exec/@sys/bin/python3.7', 'cp37', 'cp37m', 'manylinux_2_17_x86_64', (3, 7, 5)))): 593.1ms
e
I'll be away for an hour or so before I can dig further with this + your email, but I should have asked the obvious even though I'm sure you've also wracked your brain over this: Is there anything you can think of different between your machine / your account / your setup from others?
n
All of us are setup virtually the same. I even tried nuking everything and going through the onboarding myself using the same host, and still getting the different results. All the more surprising in light of the CI host/uid working, which is setup substantially differently from us.
If you can advise on all the possible locations pants/pex/etc. would put stuff, we can do an experiment later today: 1. Nuke everything, including our local Git workspaces, virtual environments, Pants, etc. 2. Login to the same Linux hosts 3. Run the same repo onboarding scripts 4. Run the same Pants commands If we still see the same differences, then I guess the next step if there is no obvious cause, is to try upgrading Pants/Pex to see if that helps. Right now I only know to nuke 
~/.cache/pants
and 
~/.pex
. We could also remove Pip's cache too. Anything else?
h
When running purely via Pants, it should be only 
~/.cache/pants
and 
<build root>/.pants.d
https://www.pantsbuild.org/docs/troubleshooting#how-to-change-your-cache-directory 
~/.pex
should only be if you run Pex directly
n
So NFS must be a seriously %$@#'ed up replica of local file systems. Yet again it's apparently the underlying issue, but why it affects some users and not others I have no idea yet. We did the above experiment (twice) and got the same result originally reported. Then I just decided to do export 
HOME=/var/tmp/hanzhw
and everything worked. I'm documenting the verbose output for NFS vs. local file system right now and will send along in our email chain. I'm really at a loss here on what to do though. Everything at MS is over NFS...and not really clear what specifically about NFS is the issue in this case as it doesn't look related at all to cache locking as in the thread we had a few weeks ago.
h
n
@hundreds-father-404 You can thank John for having the foresight to create such a catch-all issue and me for the source of them 🙂
😂 1
❤️ 1
Update: On Friday I started on-boarding a number of new people. 5 had no issues and 3 had the same problems described in this thread. Briefly recapping the problems described in this thread: 1. For packaging a 
python_distribution
, PEX (not Pants) apparently sees nothing to package and so no artifacts are written to 
./dist
and the goal completes without error (a side-by-side comparison of the diff'd outputs and the full logs have been made available to the core developers). 2. For packaging 
pex_binary
, this being roughly what I think is happening, Pants correctly resolves the dependencies but PEX complains they don't exist when trying to build the PEX file, and the goal in this case ends with an uncaught exception. I did not send a diff'd output of this goal yet, but will do-so soon. 3. Goals not involving PEX all work fine, such as 
count-loc
, 
list
, 
dependencies
, etc. It is believed NFS is a direct cause, though it's unclear exactly why because the behavior occurs for some users and not others, even for (presumably) identically setup users. The errors seen here are similar to others encountered when using NFS, and they go away when the user resets their home directory to local storage (e.g., 
export HOME=/var/tmp/$USER
before running 
./pants
). With that temporary solution in mind, we moved the 
./pants
script to 
./pantsbuild/pants
and then substituted a "filesystem switch" script in its place. Users that can successfully package a dummy 
pex_binary
will use their NFS home directory, and those for which it fails will switch to 
/var/tmp/$USER
. Details are below in case they are of interest to anyone in the future. The main disadvantage to this is that any caches/installs will be lost and need to be re-created after regularly scheduled host cleanup or if a user logs into a different host. On the flip side, 
pantsd
becomes a possibility now. ---- ./pants
Copy code
#!/bin/ksh
 
if [[ ! -f ./pantsbuild/filesystem.txt ]]; then
      ./scripts/determine_pants_filesystem.sh
fi
 
FILESYSTEM=$(cat ./pantsbuild/filesystem.txt)
if [[ $FILESYSTEM != "local" && $FILESYSTEM != "nfs" ]]; then
      echo "Invalid file system for pants: ${FILESYSTEM}"
      exit 1
fi
if [[ $FILESYSTEM == "local" ]]; then
      export HOME=/var/tmp/$USER
      mkdir -p $HOME
fi
 
if [[ ! -f $HOME/.pants.rc ]]; then
      ./scripts/generate_pantsrc.sh
fi
 
echo "Running Pants build system on ${FILESYSTEM} home directory ${HOME}"
./pantsbuild/pants.pex "$@"
./scripts/determine_pants_filesystem.sh
Copy code
#!/bin/ksh

# The preference is to use NFS
echo "nfs" > ./pantsbuild/filesystem.txt

PEX=./dist/proj.apps.oasnb/oasnb.pex
if [[ -f $PEX ]]; then
      rm $PEX
fi

./pants package proj/apps/oasnb
STATUS=$?
if [[ $STATUS != 0 || ! -f $PEX ]]; then
      echo "local" > ./pantsbuild/filesystem.txt
fi
./scripts/generate_pantsrc.sh
Copy code
#!/bin/ksh

PIP_INDEX_URL=$(./scripts/mspypi_index_url.sh | tail -n1)
echo "
[python-repos]
indexes = [
   '${PIP_INDEX_URL}'
]
" > $HOME/.pants.rc
Open in Slack
PreviousNext
Powered by