https://pantsbuild.org/ logo
Join Slack
Powered by
Hello, I'm running into some issues with pants `2....
# general
c
Hello, I'm running into some issues with pants 
2.10.1rc0
(and 
2.10.0
) where a 
python_distribution
will generate a 
.tar.gz
which gets enumerated by 
isolate_local_dist_wheels
and subsequently fails when 
unzip -Z1
returns a non 0 exit code because it has no idea what to do with a 
.tar.gz
I'm not sure if this is from something in our config/BUILDs, but this seems like a pretty critical failure
The problem also goes away if you change internal dependencies to only depend on the actual targets, not the 
python_distribution
, however there's a lot of dependencies I'd have to clean up, and relying on the dist is conceptually cleaner regardless
h
c
Yeah, that appears to be exactly the problem
h
If so, the workaround as described there is to set 
sdist=False
on the python_distribution, so it doesn't generate the sdist in the first place. But that is only a hack, this bug (as you can see from my comments on it) is very befuddling because we filter on the 
.whl
suffix, so I cannot understand how this is happening at all...
Do you happen to have an easy repro of this to share?
e
The direct implication is that DigestSubset has a bug. Has that been explored?
c
unfortunately it's basically my companies monorepo, I can try to throw something together. Alternatively, attempt to debug live
h
yeah I mean the simplest explanation, from staring at the code, is a DigestSubset bug, but I have not been able to reproduce it.
c
is there a way to a repo of pants instead of the distributed ones?
ie so I can toss in some prints/pdbs?
h
That would be great! Yes, there is a way to run Pants directly from its sources
(scrambles for the link)
😂 1
c
neato, please hold
h
Specifically, if 
wheels_snapshot.files
contains anything that doesn't end with 
.whl
then that is pretty bad...
c
waiting on cargo doing it's thing, but yeah I added log statement there. Could I expect breakpoints to work or is that not going to play nicely with the Rust runtime?
breakpoints meaning 
import pdb; pdb.set_trace()
h
Breakpoints should work, but your stacktraces will be uninformative
Because they will disappear into the rust engine
c
it'll probably eat the stdout/stdin and not actually be interactive then 🙂
h
If the print statement shows that the filtering isn't working, one quick thing to try is to replace that line with:
Copy code
wheels_digest = await Get(Digest, DigestSubset(dist.digest, PathGlobs(["**/*.whl"])))
wheels_snapshot = await Get(Snapshot, Digest, wheels_digest)
And see if that makes a difference. It shouldn't, but we're already through the looking glass at this point.
w
https://pantsbuild.slack.com/archives/C046T6T9U/p1650389498493749?thread_ts=1650388000.877219&cid=C046T6T9U if you run with 
--no-pantsd
you’ll be more likely to succeed in `pdb`*, yea. as to 
print
statements: they’ll be rendered, but prefer log statements in order to get better formatting
c
Copy code
12:36:04.35 [DEBUG] Wheels snapshot: ['<redacted>-py36.py3
7.py38.py39-none-any.whl', '<redacted>.tar.gz']
uh oh 😬
from this statement: on line 57 
logger.debug("Wheels snapshot: %s", wheels)
the above code fix didn't change anything
Copy code
wheels_digest = await Get(Digest, DigestSubset(dist.digest, PathGlobs(["**/*.whl"])))
wheels_snapshot = await Get(Snapshot, Digest, wheels_digest)
this one
Where does the logic for resolving 
DigestSubset
live?
e
h
@calm-ambulance-65371 Really appreciate the debugging help here BTW!
c
np
❤️ 1
h
If you want to throw some print statements in the rust code that would be awesome, but potentially also noisy
👍 1
c
is there a way to eliminate concurrency?
nvm figured it out
rust build is so sloooow
c
c
I am unfamiliar with the globbing library used, but here's the output from my print:
Copy code
Globs:PreparedPathGlobs {
    include: [
        PathGlobIncludeEntry {
            input: GlobParsedSource(
                "**/*.whl",
            ),
            globs: [
                DirWildcard {
                    canonical_dir: Dir(
                        "",
                    ),
                    symbolic_path: "",
                    wildcard: Pattern {
                        original: "*",
                        tokens: [
                            AnySequence,
                        ],
                        is_recursive: false,
                    },
                    remainder: [
                        Pattern {
                            original: "**",
                            tokens: [
                                AnyRecursiveSequence,
                            ],
                            is_recursive: true,
                        },
                        Pattern {
                            original: "*.whl",
                            tokens: [
                                AnySequence,
                                Char(
                                    '.',
                                ),
                                Char(
                                    'w',
                                ),
                                Char(
                                    'h',
                                ),
                                Char(
                                    'l',
                                ),
                            ],
                            is_recursive: false,
                        },
                    ],
                },
                Wildcard {
                    canonical_dir: Dir(
                        "",
                    ),
                    symbolic_path: "",
                    wildcard: Pattern {
                        original: "*.whl",
                        tokens: [
                            AnySequence,
                            Char(
                                '.',
                            ),
                            Char(
                                'w',
                            ),
                            Char(
                                'h',
                            ),
                            Char(
                                'l',
                            ),
                        ],
                        is_recursive: false,
                    },
                },
            ],
        },
    ],
    exclude: GitignoreStyleExcludes {
        patterns: [],
        gitignore: Gitignore {
            set: GlobSet {
                len: 0,
                strats: [],
            },
            root: "",
            globs: [],
            num_ignores: 0,
            num_whitelists: 0,
            matches: None,
        },
    },
    strict_match_behavior: Ignore,
    conjunction: AnyMatch,
    patterns: [
        Pattern {
            original: "**/*.whl",
            tokens: [
                AnyRecursiveSequence,
                AnySequence,
                Char(
                    '.',
                ),
                Char(
                    'w',
                ),
                Char(
                    'h',
                ),
                Char(
                    'l',
                ),
            ],
            is_recursive: true,
        },
    ],
}, 
Digest:Digest {
    hash: Fingerprint<934af979fdb5a823fc6de3e173933a839503470e4b796a608731c01a741ab70f>,
    size_bytes: 350,
}
sorry guys, I don't know rust very well, so I can't do much beyond this. If you think of anything you need me to print and share here, let me know. I'm happy to do that
I added in the 
sdist=false
to my 
python_distributions
and it seems to get me past that. Now I'm seeing another error:
Copy code
ERROR: Package 'common' requires a different Python: 3.8.10 not in '>=3.6'
seems kinda odd lol
is toolchain labs hiring? 😛
h
Re the debugging, it would be great to print 
path_stats
and 
files
in the rust code, right at the end before the call to 
DigestTrie::from_path_stats
👍 1
That error is odd indeed!
c
there's perhaps some confusion, I'm working off of the 2.10.1rc0 branch, so 
snapshot_glob_match
looks a bit different: https://github.com/pantsbuild/pants/blob/release_2.10.1rc0/src/rust/engine/fs/store/src/snapshot_ops.rs#L502
h
Oh, huh
c
perhaps need to backport that merge?
h
Looks like this problem may go away in 2.11.x
Not sure if we can backport it cleanly. @witty-crayon-22786?
c
I'll see if switching to 2.11.0rc3 works
cargo builds aggressively
h
You can go back to consuming 2.11.0rc3 from a built distribution, no?
w
it will not backport at all, unfortunately. sits on top of tons of changes =/
c
yes, but not if I want to debug print 🙂
it seems 
BinaryPathRequest
moved 😐
c
into src/python/pants/core/util_rules/system_binaries.py
h
If things work in 2.11.0rc3 then we won't need debug prints 🙂
I strongly suspect this was the issue
c
I'm having trouble with 
DuplicateNameError
due to 
python_requirements
apparently there's a naming conflict but I can't give the 
python_requirements
a different name 😕
Copy code
DuplicateNameError: A target already exists at 'common/BUILD' with name 'common' and target type 'python_requirements'. The 'python_distribution' target cannot use the same name.
Further, it seems we can't support multiple 
python_requirements
targets at the same level (rather common for things such as test requirements) in pants 
2.11
this is a complete blocker for upgrading, and considering there's a pretty critical issue in 
2.10
and possibly older versions, I'm not sure what we can do at this point
h
Ah, there is an annoying upgrade dance re 
python_requirements
, @hundreds-father-404 knows the details. I think it's in the error message?
c
there's not really any error message
docs on what I have to do would be appreciated
h
Yeah, sorry about that, for reasons I can't recall off hand that upgrade had an extra bit of complexity. This should help: https://www.pantsbuild.org/v2.11/docs/python-third-party-dependencies#first-turn-off-old-style-macros
Apologies for how truly annoying this is
Once you set
Copy code
[GLOBAL]
use_deprecated_python_macros = false
You can add 
name=
to your 
python_requirements
to differentiate them
h
Further, it seems we can't support multiple python_requirements targets at the same level (rather common for things such as test requirements) in pants 2.11
Before pants 2.10, this was indeed impossible. It was a major issue we wanted to solve. It is possible in 2.10+ via 
use_deprecated_python_macros
. That defaults to true in 2.10, but false in 2.11 There is indeed an awkward dance to migrate to the new system. The instructions should have been logged to you, but here they are https://github.com/pantsbuild/pants/blob/f50c6744243e0d8eac0d8e9fb715a6290585f397/src/python/pants/option/global_options.py#L1653
c
huh, we've been stuck on 
2.2
for awhile and have had multiple 
python_requirements
living along-side each other for awhile. Idk if we've actually been using their generated targets though
also thank you @happy-kitchen-89482!
h
If you are on 2.2 still, then it works as long as they don't include any of the same requirements 🤷 (fixing this has been a major crusade for me the past few months)
What's blocking upgrading past 2.2?
c
we're trying to get off 2.2 for a few reasons, it's very old at this piont
➕ 1
we were previously blocked by dep issues due to a hard req on python 3.5
now we're free, but we ran into some dep issues with the new pip dependency resolving algorithm
👍 1
h
Please do feel free to keep reaching out for help while upgrading. We've made a lot of substantial improvements since 2.2
c
will do, thanks for the help!
❤️ 1
One question:
Copy code
You have set interpreter constraints (`CPython<4,>=3.7`) that are not compatible with those used to generate the lockfile (`CPython<4,>=3.8 OR CPython>=3.6,>=3.8`). You can fix this by not setting `[black].interpreter_constraints`, or by using a new custom lockfile.
any idea why I'm getting that when I just generated the lockfile?
h
What goal are you running? Which pants version also?
c
update-build-files --fix-python-macros
2.11.0rc3
use_deprecated_python_macros = false
h
might be this https://github.com/pantsbuild/pants/issues/14912 😕 you can use 
--no-fmt
, or try setting 
[black].interpreter_constraints
to 3.6+ and regenerate lockfile
c
setting the 
[black].interpreter_constraints
and regenerating appeared to fix it
👍 1
h
Cool. Then I think it is that one issue. Sorry for the broken window!
c
no worries, thanks much for the help!
now to fix all 4 million 3rd party dependencies for our monorepo...
🔥 1
h
Hopefully pants automates the majority of that 😱 benjy has already said this, but again our apologies for this deprecation dance
c
it would do something like 
common:requests
-> 
common:requirements#requests
for me?
h
it should! You will need to look closely at the ERROR logs to add 
name=
to the 
python_requirements()
, but no manual changes for call sites. Just the definition
And do let us know if that is not the case. Several organizations have successfully migrated to the new target generator mechanism, so we think that we have now handled all the edge cases (including pants.toml). But it's possible there are more edges
c
looks like it's not happy (?) with my partially manually completed work:
Copy code
17:19:37.39 [ERROR] 1 Exception encountered:

  MappingError: Failed to parse ./interfaces/BUILD:
__call__() got an unexpected keyword argument 'name'
that BUILD looks like:
Copy code
python_requirements(name="requirements")

python_sources()
h
Yes, it is not happy with a partial state. You need to update the global option now
c
But then I get:
Copy code
17:25:08.13 [ERROR] 1 Exception encountered:

  ValueError: `--update-build-files-fix-python-macros` specified when `[GLOBAL].use_deprecated_python_macros` is already set to false, which means that there is nothing left to fix.
it lies!
🙂
h
It should be safe to change the global option as long as you have run the update build files goal already. And make sure you preserve those error log messages in case you have not finished them
you only need to run that one time
i.e. don't permanently set 
--fix-python-macros
Let me know if you want to hop on a video chat
c
I appreciate the offer, but I'm about to log off for the day. I'll pick this back up tomorrow though. Is there some command besides 
update-build-files --update-build-files-fix-python-macros
that I should be running to update the dependencies? I may just have gotten myself into a bad state and need to rework things myself 🙂
again, I really appreciate the help!
h
should be only the instructions at https://github.com/pantsbuild/pants/blob/f50c6744243e0d8eac0d8e9fb715a6290585f397/src/python/pants/option/global_options.py#L1653. As long as you follow all those instructions—including changing the global option after running the updater and applying each of the error log changes—should be good to go
Do you already have >1 
python_requirements
in a single directory? There is this TODO that could give you trouble https://github.com/pantsbuild/pants/blob/b1fa13f8c0788f11564ef486e2f4fb5f9c56d2e8/src/python/pants/backend/python/macros/deprecation_fixers.py#L128
c
ok back at this, fixed up all our 3rd party deps., but now I'm running into some odd errors. Example:
Copy code
4: numpy
    Required by:
      FingerprintedDistribution(distribution=<redacted>-3136395a3be17fe9145c4a8a3d3f2d9c82fa6349 (/home/noah/.cache/pants/named_caches/pex_root/installed_wheels/537d7fc53f4f80dc839310442f7d14a3ee8b01f8ab2d929f987f1ac55cbeaf30/<redacted>-3136395a3be17fe9145c4a8a3d3f2d9c82fa6349-py3-none-any.whl), fingerprint='6e7c538bf420421cd2668a8e1db727ea763104aef000bfa4d08fec0751891228')
      FingerprintedDistribution(distribution=<redacted>-3136395a3be17fe9145c4a8a3d3f2d9c82fa6349 (/home/noah/.cache/pants/named_caches/pex_root/installed_wheels/19769ef6f634442291de1989e8ece3158aa195aa05b985ff08a85c4dab4af640/<redacted>-3136395a3be17fe9145c4a8a3d3f2d9c82fa6349-py38-none-any.whl), fingerprint='d5159cf4c0d5b014f94d9b896d527ffeeb0e2596957a38aba5bd391d9b8548a1')
    But this pex had no 'numpy' distributions.
👀 1
I have a bunch of these for our 3rd party deps
h
huh....if you temporarily rename the folder 
/home/noah/.cache/pants/named_caches
?
c
same thing
completely wiping ~/.cache/pants doesn't help either 😞
h
How are you installing things? Are you on pants 2.11 and using lock files for example?
c
oh, actually, it might be because we're using a 
constraints.txt
. Will try removing
nope
I have lockfiles for setuptools, pytest, and black. Our code (now) has no lock files
h
It should be fine to keep using constraints.txt. Soon, it would be good to switch to the new lock files, but it's best to do that as a dedicated change once things are already stable
How about using 
-ldebug
and/or 
--no-process-cleanup
? I am curious if this is something obvious like that we are not correctly saying to install numpy because eg deps are wrong, vs something else
c
I think we might just be somehow stuck in dep hell, not sure it's Pants' fault
h
How so? Were things working before?
c
yes, but I think there may have been some silent issues
not sure
Copy code
14:29:35.31 [ERROR] 1 Exception encountered:
ncy id: 3, and concurrency: 1
  ProcessExecutionFailure: Process 'Building pytest_runner.pex' failed with exit code 1.                                       _runner.pex under semaphore with concurre
stdout:
14:00:23.69 [INFO] Preserving local process execution dir /tmp/process-executionCURx
stderr:
Appears to be failing when making the pytest_runner pex
what's odd is some tests pass
nvm, tests aren't running
h
is there more to the error?
c
Copy code
Traceback (most recent call last):
  File "/home/noah/.cache/pants/named_caches/pex_root/unzipped_pexes/aecac44135d419f932725eb33af2df4856517e99/.bootstrap/pex/pex.py", line 504, in execute
    exit_value = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/home/noah/.cache/pants/named_caches/pex_root/unzipped_pexes/aecac44135d419f932725eb33af2df4856517e99/.bootstrap/pex/pex.py", line 409, in _wrap_coverage
    return runner(*args)
  File "/home/noah/.cache/pants/named_caches/pex_root/unzipped_pexes/aecac44135d419f932725eb33af2df4856517e99/.bootstrap/pex/pex.py", line 440, in _wrap_profiling
    return runner(*args)
  File "/home/noah/.cache/pants/named_caches/pex_root/unzipped_pexes/aecac44135d419f932725eb33af2df4856517e99/.bootstrap/pex/pex.py", line 560, in _execute
    return self.execute_entry(self._pex_info.entry_point)
  File "/home/noah/.cache/pants/named_caches/pex_root/unzipped_pexes/aecac44135d419f932725eb33af2df4856517e99/.bootstrap/pex/pex.py", line 696, in execute_entry
    return self.execute_pkg_resources(entry_point)
  File "/home/noah/.cache/pants/named_caches/pex_root/unzipped_pexes/aecac44135d419f932725eb33af2df4856517e99/.bootstrap/pex/pex.py", line 728, in execute_pkg_resources
    return runner()
  File "/home/noah/.cache/pants/named_caches/pex_root/installed_wheels/fae568165f31fc90f0e75717c4271e0cfdd33bef60060cee0ed9d51279de33a0/pex-2.1.79-py2.py3-none-any.whl/pex/bin/pex.py", line 740, in main
    do_main(
  File "/home/noah/.cache/pants/named_caches/pex_root/installed_wheels/fae568165f31fc90f0e75717c4271e0cfdd33bef60060cee0ed9d51279de33a0/pex-2.1.79-py2.py3-none-any.whl/pex/bin/pex.py", line 761, in do_main
    pex_builder = build_pex(
  File "/home/noah/.cache/pants/named_caches/pex_root/installed_wheels/fae568165f31fc90f0e75717c4271e0cfdd33bef60060cee0ed9d51279de33a0/pex-2.1.79-py2.py3-none-any.whl/pex/bin/pex.py", line 686, in build_pex
    pex_builder.set_script(options.script)
  File "/home/noah/.cache/pants/named_caches/pex_root/installed_wheels/fae568165f31fc90f0e75717c4271e0cfdd33bef60060cee0ed9d51279de33a0/pex-2.1.79-py2.py3-none-any.whl/pex/pex_builder.py", line 348, in set_script
    distributions.update(PEX(pex, interpreter=self._interpreter).resolve())
  File "/home/noah/.cache/pants/named_caches/pex_root/installed_wheels/fae568165f31fc90f0e75717c4271e0cfdd33bef60060cee0ed9d51279de33a0/pex-2.1.79-py2.py3-none-any.whl/pex/orderedset.py", line 45, in update
    for key in iterable:
  File "/home/noah/.cache/pants/named_caches/pex_root/installed_wheels/fae568165f31fc90f0e75717c4271e0cfdd33bef60060cee0ed9d51279de33a0/pex-2.1.79-py2.py3-none-any.whl/pex/pex.py", line 121, in resolve
    for dist in env.resolve():
  File "/home/noah/.cache/pants/named_caches/pex_root/installed_wheels/fae568165f31fc90f0e75717c4271e0cfdd33bef60060cee0ed9d51279de33a0/pex-2.1.79-py2.py3-none-any.whl/pex/environment.py", line 492, in resolve
    for fingerprinted_distribution in self.resolve_dists(all_reqs)
  File "/home/noah/.cache/pants/named_caches/pex_root/installed_wheels/fae568165f31fc90f0e75717c4271e0cfdd33bef60060cee0ed9d51279de33a0/pex-2.1.79-py2.py3-none-any.whl/pex/environment.py", line 579, in resolve_dists
    raise ResolveError(
pex.environment.ResolveError: Failed to resolve requirements from PEX environment @ /home/noah/.cache/pants/named_caches/pex_root/unzipped_pexes/b1f697e212f61b6154e26d9885061a3836dfb691.
Needed cp38-cp38-manylinux_2_31_x86_64 compatible dependencies for:
 1: Flask
    Required by:
      FingerprintedDistribution(distribution=<redacted>-3136395a3be17fe9145c4a8a3d3f2d9c82fa6349 (/home/noah/.cache/pants/named_caches/pex_root/installed_wheels/b7a06d2474b32d3876949dd74da83c93f060b4a4aaf5f4b434a17085a8b40344/<redacted>-3136395a3be17fe9145c4a8a3d3f2d9c82fa6349-py3-none-any.whl), fingerprint='8e7c49e6be260ac829d0c9058bf09b8d90039ba5b010b47d7d04623778ffaf25')
    But this pex had no 'Flask' distributions.
h
Does your constraints file definitely have flask?
lmk if you want to hop on a call
c
it's not in the constraints file at all
we only have versions that must be pinned in our constraints as it's company-wide in a monorepo
h
ohhhhhhhhhhh, one sec
c
it is pinned to a version in the respective requirements.txt
h
set this to false. It should not be necessary to set that, but I want to see if it fixes it https://www.pantsbuild.org/docs/reference-python#section-resolve-all-constraints
c
we already had that set actually
🤔 1
h
This is 2.11.0rc3 right? Maybe try 2.10.1rc0 -- your 
python_requirement
changes should still work We made some changes in 2.11 so that lock files generated by pex are much more efficient. Perhaps we broke a code path
👍 1
your python_requirement changes should still work
you may need to explicitly set 
[GLOBAL].use_deprecated_python_macros = false
c
same issue (I had to do a bunch of other stuff to get 2.10 to work)
😔 1
Will try to get more info
where is the 
FingerprintedDistribution
defined?
I'm not finding it in the pants codebase
h
I think that that is pex code
👍 1
bump that using 
-ldebug
may be useful: make sure pants is correctly telling Pex to install things. you may need to use 
--no-pantsd --no-local-cache
c
I confirmed that tests are actually running, it appears to fail at the end. Possibly when trying to do coverage?
👀 1
h
Strange. Could you try turning off coverage to confirm that?
c
Did not effect it. The debug log seems to indicate the 
test
goal is completing:
Copy code
16:20:23.03 [DEBUG] Completed: `test` goal
16:20:23.03 [DEBUG] Completed: Run Pytest
16:20:23.03 [DEBUG] Canceled: Run Pytest
16:20:23.03 [DEBUG] computed 1 nodes in 41.582558 seconds. there are 16938 total nodes.
16:20:23.03 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Building pytest_runner.pex' failed with exit code 1.
stdout:
h
Are you running test on a single file or multiple?
c
we have a bunch of modules, I'm running something like 
./pants test module::
(excluded the flags)
I'm seeing failing tests for expected reasons, so it's not that it can't run anything
oh weird, the generated .whl files don't appear to honor the 
constraints.txt
(unzipped and looking at the METADATA file)
h
It might be helpful if you can make that more precise to figure out the exact file that is failing. If you have the option 
--print-stacktrace
set, it should show you a list of rules with the file name in parentheses Note that pants will build multiple pytest_runner.py files. It sounds like some tests work and some down
7 Views
Open in Slack
PreviousNext
Powered by