https://pantsbuild.org/ logo
Join Slack
Powered by
I'm trying to use <neptune.ai> but have to fix it ...
# general
p
I'm trying to use neptune.ai but have to fix it in two ways: ā€¢ The package in 
requirements.txt
is 
neptune-client
so I need a 
module_mapping
to let Pants know it gets imported as just 
neptune
ā€¢ The package fails to declare its dependency on 
setuptools
so I need to add that via a 
python_requirements
Somehow these two aren't playing nicely together. Here's what I have:
Copy code
python_requirement(
    name="setuptools",
    requirements=["setuptools"],
)

python_requirement(
    name="neptune-client",
    requirements=["neptune-client"],
    dependencies=[":setuptools"],
)

# Module mappings, etc.

python_requirements(
   name='reqs',
   module_mapping={
      ...
      ':neptune-client': ['neptune'],
   },
)
I've tried a bunch of variations like having the 
python_requirement
have its 
requirements
come after the module mapping bit, having the module mapping be from 
neptune-client
without the leading 
:
, etc. and every combination ends up with an error, though the errors change as I blindly throw things at the wall hoping something will stick. Guidance would be much appreciated!
b
Is there a 
requirements.txt
that 
python_requirements
is loading? Does this also include 
neptune-client
?
p
Yes. Actually, I have a freeze file that Pants generated from 
requirements.txt
Copy code
[python.resolves]
default = "3rdparty/python/default.lock"
and it is in the lockfile
b
Ah, okay, the trick here might be to only use 
python_requirements
, and not have the separate `python_requirement`s. As it is, I think you'll be (trying) to declare duplicate dependencies, that is, have 
neptune-client
loaded into your Pants resolve twice. If 
setuptools
is in the 
requirements.txt
, you might do something what we do:
Copy code
python_requirements(
    name="reqs",
    overrides={
        "neptune-client": {"dependencies": ["reqs#setuptools"]},
    },
    module_mapping={
        "neptune-client": ["neptune"],
    },
)
p
AH! I might have fixed it. I added a 
modules=
to the 
python_requirement
to say that 
neptune-client
provides the 
neptune
module and then removed it from the module_mappings bit. I think that's working (my unit test is failing but I think now it's bad code šŸ™‚
b
Yeah, that probably works too, although you'll need to be careful about having two 
python_requirement
for 
neptune-client
(the 
python_requirements
target is a helper that generates the "real" 
python_requirement
s from a file)
p
I only hae 1 now and it seems to be working (even got the test to pass). It looks like this now:
Copy code
python_requirement(
    name="neptune-client",
    requirements=["neptune-client"],
    dependencies=[":setuptools"],
    modules=['neptune', 'neptune.new'],
)

python_requirements(
   name='reqs',
   module_mapping={
      # neptune/neptune-client no longer appear in here
   },
)
Does that seem safe?
b
Does 
neptune-client
appear in the separate 
requirements.txt
file that 
python_requirements
is loading?
p
yes.
Is there any other way to get it into my lock file?
That is, have good, fully transitive dependency resolution of its transitie deps along with the transitive deps of all my other dependencies?
b
It's currently being introduced twice, once for the manual 
python_requirement
, and once for the 
python_requirement
that's automatically generated by 
python_requirements
. This may cause problems in future. So, options: 1. remove 
neptune-client
from 
requirements.txt
, so that you've only got the manual 
python_requirement
2. remove the manual 
python_requirement
and use the overrides and module-mapping as I demonstrate above Either way will appear in the Pants lockfile. I'd probably encourage option 2, so that dependencies are (mostly) in one place. (You could even put 
setuptools
into the 
requirements.txt
too, and avoid the manual 
python_requirement
for it.)
p
I see. Lemme give that a try. Thanks for the help @broad-processor-92400!
hmmm... @broad-processor-92400 that's not working for me.
b
It may be handy to start simple (without the module mappings or overrides) and run 
./pants list path/to/targets:
to see the names of the targets being generated to be able to refer to them correctly. (
path/to/targets
is the directory of the 
BUILD
file with these contents. if it's the root, 
./pants list //:
, I think)
p
If I do:
Copy code
python_requirement(
    name="setuptools",
    requirements=["setuptools"],
)

python_requirements(
   name='reqs',
   module_mapping={
      ...
      'neptune-client': ['neptune'],
   },
   overrides = {
      'neptune-client': {
         'dependencies': 'reqs#setuptools',
      }
   },
)
I get:
Copy code
AddressParseException: Failed to parse address spec `#`: error at 1:2: expected a non-empty generated target name to follow a `#`.
which I think makes sense. 
setuptools
isn't in my 
requirements.txt
(though I'm starting to think it should be).
b
With a separate setuptools, referring to it by its target name 
:setuptools
should work
(but, having them all in one place makes it nice and uniform, if there's any other dependency overrides required)
p
I tried that (the 
:setuptools
) and got:
Copy code
UnsupportedWildcardError: The address `:` from the `dependencies` field from the target //:reqs#neptune-client ended in a wildcard (`:`), which is not supported.
yeah. Let me trying moving setuptools into requirements.txt and then doing everything via overrides....
b
Reading that message makes me wonder if you had a typo, like 
setuptools:
(
:
at the end); it should be 
:setuptools
at the start
p
I didn't. Triple checked, deleted and re-added, etc. I thought that was strange. Trying the alternative now but it's gonna be a while; have to regenerate the lockfile and that takes forever with my set of dependencies (lookin' at you PyTorch!)
b
huh, weird. Ah, yeah, lockfile speed is annoying
p
ug. Now my freeze attempt is failing:
Copy code
MappingError: Failed to parse ./BUILD:
expression cannot contain assignment, perhaps you meant "=="? (<string>, line 9)
line 9 is:
Copy code
overrides={
contents are:
Copy code
overrides={
      'neptune-client': { 'dependencies': 'reqs#setuptools' }
    }
b
I think the error might be elsewhere, that looks like a normal Python syntax error
(the BUILD files are literally Python)
p
This is the entire contents of my BUILD file:
Copy code
python_requirements(
   name='reqs',
   module_mapping={
      'pyyaml': ['yaml'],
      'Pillow': ['PIL'],
      'imageio': ['imageio.v3'],
      'neptune-client': ['neptune'],
   },
   overrides={
      'neptune-client': { 'dependencies': 'reqs#setuptools' }
      'torchmetrics': { 'dependencies': 'reqs#setuptools' },
   },
   type_stubs_module_mapping={
      'types-PyYAML': ['yaml'],
      'types-Pillow': ['PIL'],
   }
)
ah! Was missing the 
,
on the next line (line 10) after the 
neptune-client
override.
lockfile slowly generating....
nope. that failed:
Copy code
AddressParseException: Failed to parse address spec `#`: error at 1:2: expected a non-empty generated target name to follow a `#`.
Full contents of `BUILD`:
Copy code
python_requirements(
   name='reqs',
   module_mapping={
      'pyyaml': ['yaml'],
      'Pillow': ['PIL'],
      'imageio': ['imageio.v3'],
      'neptune-client': ['neptune'],
   },
   overrides={
      'neptune-client': { 'dependencies': 'reqs#setuptools' },
   },
   type_stubs_module_mapping={
      'types-PyYAML': ['yaml'],
      'types-Pillow': ['PIL'],
   }
)
b
I don't know if I got the names of the targets correct. Using 
./pants list
is a good way for you to understand what's being generated: https://pantsbuild.slack.com/archives/C046T6T9U/p1677028145118729?thread_ts=1677027041.399219&amp;cid=C046T6T9U Let me know if you have questions about the output
Standard debugging technique as well: make the feedback cycle shorter. E.g. commenting out the irrelevant parts of the requirements.txt (like pytorch) should help make iterating on just this bit faster.
p
Copy code
$ ./pants list //: 
...
//:reqs#neptune-client
...
//:reqs#setuptools
...
Adding a leading 
:
was bad. Maybe I need the full 
//:
prefix. Trying that.
nope. It doesn't like 
//:
either.
Standard debugging technique as well: make the feedback cycle shorter. E.g. commenting out the irrelevant parts of the requirements.txt (like pytorch) should help make iterating on just this bit faster.
Makes sense but I think that'd actually take longer; I have a big mono-repo with a lot of interdependencies. Figuring out what subset of those, transitively, are releant to this seems hard...
So the following do not work:
Copy code
'neptune-client': { 'dependencies': '//:reqs#setuptools' },
Copy code
'neptune-client': { 'dependencies': ':reqs#setuptools' },
Copy code
'neptune-client': { 'dependencies': 'reqs#setuptools' },
All result in different errors. Output of 
pants list
would seem to imply that one of those is correct.
b
Ah, I see what you mean. I was thinking we should focus on getting the BUILD file syntax correct for 
neptune-client
, and thus only 
neptune-client
and 
setuptools
are relevant. Once that's set up right, you can uncomment everything to see if test passing etc.
Hmmm, maybe there's bugs with target generation at the root of a repository šŸ¤” Can you make a reduced example with just 
neptune-client
and 
setuptools
?
p
trying that now. Commented out a bunch of stuff. Our setup is a little funky. Unfortunately, PyTorch differentiates its CUDA and CPU versions only by the extra-index-url; the pip requirement name, extras, etc. are all identical. So, with some help from y'all at Pants, we came up with a bit of a funky setup with env variables and a little bit of wrapper automation. I could be that's causing issues though that setup has been pretty stable for almost a year now.
Strange. So with everything commented out the freeze succeeds. But 
pants test
fails:
Copy code
$ ./pants test src/ml_logger/neptune/::
17:32:28.12 [ERROR] 1 Exception encountered:

  InvalidSpecPathError: Invalid address / from the `dependencies` field from the target //:reqs#neptune-client. Cannot use absolute paths.
And 
pants test
fails with all 3 variants of the address of setuptools above (no prefix, 
:
prefix, 
//:
prefix).
Is the problem maybe that 
reqs
is referring to itself here? That is, our main dependency block is 
reqs
but we're then using that in the overrides block in that target.
BTW: i'm guessing you're a pants employee and it's past the end of your workday. If you want to pick this up tomorrow that's totally understandable.
It's very strange that 
./pants generate-lockfiles
works fine but 
./pants test
does not. I'm getting those syntax errors from the latter but not the former.
AHA! I tracked down at least part of it. I had:
Copy code
'neptune-client': { 'dependencies': 'reqs#setuptools' },
It needs to be:
Copy code
'neptune-client': { 'dependencies': ['reqs#setuptools'] },
That is, the dependencies are a list. If not, it treats the string as an iterable and it's trying to resolve 
r
, 
e
, 
q
, 
#
, etc. But 
pants test
does that while 
generate-lockfiles
does not.
b
Ooh, that bad from pants. If you have a chance to file an issue about that lack of validation, please do, or else I'll give it a go later.
BTW: i'm guessing you're a pants employee and it's past the end of your workday. If you want to pick this up tomorrow that's totally understandable.
I'm just a random person from Australia, not employed by Toolchain šŸ˜„
p
Wow! Then thanks even more for your time and help. MUCH appreciated @broad-processor-92400!
b
p
Thanks!!
9 Views
Open in Slack
PreviousNext
Powered by