https://pantsbuild.org/ logo
#general
Title
# general
p

plain-carpet-73994

02/22/2023, 12:50 AM
I'm trying to use neptune.ai but have to fix it in two ways: ā€¢ The package in
requirements.txt
is
neptune-client
so I need a
module_mapping
to let Pants know it gets imported as just
neptune
ā€¢ The package fails to declare its dependency on
setuptools
so I need to add that via a
python_requirements
Somehow these two aren't playing nicely together. Here's what I have:
Copy code
python_requirement(
    name="setuptools",
    requirements=["setuptools"],
)

python_requirement(
    name="neptune-client",
    requirements=["neptune-client"],
    dependencies=[":setuptools"],
)

# Module mappings, etc.

python_requirements(
   name='reqs',
   module_mapping={
      ...
      ':neptune-client': ['neptune'],
   },
)
I've tried a bunch of variations like having the
python_requirement
have its
requirements
come after the module mapping bit, having the module mapping be from
neptune-client
without the leading
:
, etc. and every combination ends up with an error, though the errors change as I blindly throw things at the wall hoping something will stick. Guidance would be much appreciated!
b

broad-processor-92400

02/22/2023, 12:53 AM
Is there a
requirements.txt
that
python_requirements
is loading? Does this also include
neptune-client
?
p

plain-carpet-73994

02/22/2023, 12:53 AM
Yes. Actually, I have a freeze file that Pants generated from
requirements.txt
Copy code
[python.resolves]
default = "3rdparty/python/default.lock"
and it is in the lockfile
b

broad-processor-92400

02/22/2023, 12:57 AM
Ah, okay, the trick here might be to only use
python_requirements
, and not have the separate `python_requirement`s. As it is, I think you'll be (trying) to declare duplicate dependencies, that is, have
neptune-client
loaded into your Pants resolve twice. If
setuptools
is in the
requirements.txt
, you might do something what we do:
Copy code
python_requirements(
    name="reqs",
    overrides={
        "neptune-client": {"dependencies": ["reqs#setuptools"]},
    },
    module_mapping={
        "neptune-client": ["neptune"],
    },
)
p

plain-carpet-73994

02/22/2023, 12:57 AM
AH! I might have fixed it. I added a
modules=
to the
python_requirement
to say that
neptune-client
provides the
neptune
module and then removed it from the module_mappings bit. I think that's working (my unit test is failing but I think now it's bad code šŸ™‚
b

broad-processor-92400

02/22/2023, 12:58 AM
Yeah, that probably works too, although you'll need to be careful about having two
python_requirement
for
neptune-client
(the
python_requirements
target is a helper that generates the "real"
python_requirement
s from a file)
p

plain-carpet-73994

02/22/2023, 12:59 AM
I only hae 1 now and it seems to be working (even got the test to pass). It looks like this now:
Copy code
python_requirement(
    name="neptune-client",
    requirements=["neptune-client"],
    dependencies=[":setuptools"],
    modules=['neptune', 'neptune.new'],
)

python_requirements(
   name='reqs',
   module_mapping={
      # neptune/neptune-client no longer appear in here
   },
)
Does that seem safe?
b

broad-processor-92400

02/22/2023, 1:00 AM
Does
neptune-client
appear in the separate
requirements.txt
file that
python_requirements
is loading?
p

plain-carpet-73994

02/22/2023, 1:00 AM
yes.
Is there any other way to get it into my lock file?
That is, have good, fully transitive dependency resolution of its transitie deps along with the transitive deps of all my other dependencies?
b

broad-processor-92400

02/22/2023, 1:04 AM
It's currently being introduced twice, once for the manual
python_requirement
, and once for the
python_requirement
that's automatically generated by
python_requirements
. This may cause problems in future. So, options: 1. remove
neptune-client
from
requirements.txt
, so that you've only got the manual
python_requirement
2. remove the manual
python_requirement
and use the overrides and module-mapping as I demonstrate above Either way will appear in the Pants lockfile. I'd probably encourage option 2, so that dependencies are (mostly) in one place. (You could even put
setuptools
into the
requirements.txt
too, and avoid the manual
python_requirement
for it.)
p

plain-carpet-73994

02/22/2023, 1:05 AM
I see. Lemme give that a try. Thanks for the help @broad-processor-92400!
hmmm... @broad-processor-92400 that's not working for me.
b

broad-processor-92400

02/22/2023, 1:09 AM
It may be handy to start simple (without the module mappings or overrides) and run
./pants list path/to/targets:
to see the names of the targets being generated to be able to refer to them correctly. (
path/to/targets
is the directory of the
BUILD
file with these contents. if it's the root,
./pants list //:
, I think)
p

plain-carpet-73994

02/22/2023, 1:09 AM
If I do:
Copy code
python_requirement(
    name="setuptools",
    requirements=["setuptools"],
)

python_requirements(
   name='reqs',
   module_mapping={
      ...
      'neptune-client': ['neptune'],
   },
   overrides = {
      'neptune-client': {
         'dependencies': 'reqs#setuptools',
      }
   },
)
I get:
Copy code
AddressParseException: Failed to parse address spec `#`: error at 1:2: expected a non-empty generated target name to follow a `#`.
which I think makes sense.
setuptools
isn't in my
requirements.txt
(though I'm starting to think it should be).
b

broad-processor-92400

02/22/2023, 1:10 AM
With a separate setuptools, referring to it by its target name
:setuptools
should work
(but, having them all in one place makes it nice and uniform, if there's any other dependency overrides required)
p

plain-carpet-73994

02/22/2023, 1:11 AM
I tried that (the
:setuptools
) and got:
Copy code
UnsupportedWildcardError: The address `:` from the `dependencies` field from the target //:reqs#neptune-client ended in a wildcard (`:`), which is not supported.
yeah. Let me trying moving setuptools into requirements.txt and then doing everything via overrides....
b

broad-processor-92400

02/22/2023, 1:11 AM
Reading that message makes me wonder if you had a typo, like
setuptools:
(
:
at the end); it should be
:setuptools
at the start
p

plain-carpet-73994

02/22/2023, 1:13 AM
I didn't. Triple checked, deleted and re-added, etc. I thought that was strange. Trying the alternative now but it's gonna be a while; have to regenerate the lockfile and that takes forever with my set of dependencies (lookin' at you PyTorch!)
b

broad-processor-92400

02/22/2023, 1:13 AM
huh, weird. Ah, yeah, lockfile speed is annoying
p

plain-carpet-73994

02/22/2023, 1:14 AM
ug. Now my freeze attempt is failing:
Copy code
MappingError: Failed to parse ./BUILD:
expression cannot contain assignment, perhaps you meant "=="? (<string>, line 9)
line 9 is:
Copy code
overrides={
contents are:
Copy code
overrides={
      'neptune-client': { 'dependencies': 'reqs#setuptools' }
    }
b

broad-processor-92400

02/22/2023, 1:15 AM
I think the error might be elsewhere, that looks like a normal Python syntax error
(the BUILD files are literally Python)
p

plain-carpet-73994

02/22/2023, 1:16 AM
This is the entire contents of my BUILD file:
Copy code
python_requirements(
   name='reqs',
   module_mapping={
      'pyyaml': ['yaml'],
      'Pillow': ['PIL'],
      'imageio': ['imageio.v3'],
      'neptune-client': ['neptune'],
   },
   overrides={
      'neptune-client': { 'dependencies': 'reqs#setuptools' }
      'torchmetrics': { 'dependencies': 'reqs#setuptools' },
   },
   type_stubs_module_mapping={
      'types-PyYAML': ['yaml'],
      'types-Pillow': ['PIL'],
   }
)
ah! Was missing the
,
on the next line (line 10) after the
neptune-client
override.
lockfile slowly generating....
nope. that failed:
Copy code
AddressParseException: Failed to parse address spec `#`: error at 1:2: expected a non-empty generated target name to follow a `#`.
Full contents of `BUILD`:
Copy code
python_requirements(
   name='reqs',
   module_mapping={
      'pyyaml': ['yaml'],
      'Pillow': ['PIL'],
      'imageio': ['imageio.v3'],
      'neptune-client': ['neptune'],
   },
   overrides={
      'neptune-client': { 'dependencies': 'reqs#setuptools' },
   },
   type_stubs_module_mapping={
      'types-PyYAML': ['yaml'],
      'types-Pillow': ['PIL'],
   }
)
b

broad-processor-92400

02/22/2023, 1:23 AM
I don't know if I got the names of the targets correct. Using
./pants list
is a good way for you to understand what's being generated: https://pantsbuild.slack.com/archives/C046T6T9U/p1677028145118729?thread_ts=1677027041.399219&amp;cid=C046T6T9U Let me know if you have questions about the output
Standard debugging technique as well: make the feedback cycle shorter. E.g. commenting out the irrelevant parts of the requirements.txt (like pytorch) should help make iterating on just this bit faster.
p

plain-carpet-73994

02/22/2023, 1:24 AM
Copy code
$ ./pants list //: 
...
//:reqs#neptune-client
...
//:reqs#setuptools
...
Adding a leading
:
was bad. Maybe I need the full
//:
prefix. Trying that.
nope. It doesn't like
//:
either.
Standard debugging technique as well: make the feedback cycle shorter. E.g. commenting out the irrelevant parts of the requirements.txt (like pytorch) should help make iterating on just this bit faster.
Makes sense but I think that'd actually take longer; I have a big mono-repo with a lot of interdependencies. Figuring out what subset of those, transitively, are releant to this seems hard...
So the following do not work:
Copy code
'neptune-client': { 'dependencies': '//:reqs#setuptools' },
Copy code
'neptune-client': { 'dependencies': ':reqs#setuptools' },
Copy code
'neptune-client': { 'dependencies': 'reqs#setuptools' },
All result in different errors. Output of
pants list
would seem to imply that one of those is correct.
b

broad-processor-92400

02/22/2023, 1:28 AM
Ah, I see what you mean. I was thinking we should focus on getting the BUILD file syntax correct for
neptune-client
, and thus only
neptune-client
and
setuptools
are relevant. Once that's set up right, you can uncomment everything to see if test passing etc.
Hmmm, maybe there's bugs with target generation at the root of a repository šŸ¤” Can you make a reduced example with just
neptune-client
and
setuptools
?
p

plain-carpet-73994

02/22/2023, 1:31 AM
trying that now. Commented out a bunch of stuff. Our setup is a little funky. Unfortunately, PyTorch differentiates its CUDA and CPU versions only by the extra-index-url; the pip requirement name, extras, etc. are all identical. So, with some help from y'all at Pants, we came up with a bit of a funky setup with env variables and a little bit of wrapper automation. I could be that's causing issues though that setup has been pretty stable for almost a year now.
Strange. So with everything commented out the freeze succeeds. But
pants test
fails:
Copy code
$ ./pants test src/ml_logger/neptune/::
17:32:28.12 [ERROR] 1 Exception encountered:

  InvalidSpecPathError: Invalid address / from the `dependencies` field from the target //:reqs#neptune-client. Cannot use absolute paths.
And
pants test
fails with all 3 variants of the address of setuptools above (no prefix,
:
prefix,
//:
prefix).
Is the problem maybe that
reqs
is referring to itself here? That is, our main dependency block is
reqs
but we're then using that in the overrides block in that target.
BTW: i'm guessing you're a pants employee and it's past the end of your workday. If you want to pick this up tomorrow that's totally understandable.
It's very strange that
./pants generate-lockfiles
works fine but
./pants test
does not. I'm getting those syntax errors from the latter but not the former.
AHA! I tracked down at least part of it. I had:
Copy code
'neptune-client': { 'dependencies': 'reqs#setuptools' },
It needs to be:
Copy code
'neptune-client': { 'dependencies': ['reqs#setuptools'] },
That is, the dependencies are a list. If not, it treats the string as an iterable and it's trying to resolve
r
,
e
,
q
,
#
, etc. But
pants test
does that while
generate-lockfiles
does not.
b

broad-processor-92400

02/22/2023, 1:54 AM
Ooh, that bad from pants. If you have a chance to file an issue about that lack of validation, please do, or else I'll give it a go later.
BTW: i'm guessing you're a pants employee and it's past the end of your workday. If you want to pick this up tomorrow that's totally understandable.
I'm just a random person from Australia, not employed by Toolchain šŸ˜„
p

plain-carpet-73994

02/22/2023, 1:54 AM
Wow! Then thanks even more for your time and help. MUCH appreciated @broad-processor-92400!
b

broad-processor-92400

02/24/2023, 3:48 AM
p

plain-carpet-73994

02/24/2023, 6:14 PM
Thanks!!
4 Views