https://pantsbuild.org/ logo
#general
Title
# general
w

wonderful-iron-54019

08/18/2020, 4:02 PM
Hey folks, qq here: I'm working on
pipenv_requirements
build alias to source python requirements from pipenv using the
PythonRequirements
class as a model. its working like a charm. However, we've traditionally kept our
Pipfile.lock
file at the project root but, for ease of migration, i'd still like the build addresses for the requirements to go to
3rdparty/python:<req>
is there a way i can use the
_parse_context
to get this to happen?
relatedly, im not sure i understand the purpose the target defined here: https://github.com/pantsbuild/pants/blob/7268985491bb935c6c9845726ff91981211910fe/src/python/pants/backend/python/python_requirements.py#L86-L92 I presume its for dependency resolution, i.e triggereing a cache miss when the requirments file is changed
h

hundreds-father-404

08/18/2020, 5:34 PM
Hey good morning / good afternoon.
However, we’ve traditionally kept our Pipfile.lock file at the project root but, for ease of migration, i’d still like the build addresses for the requirements to go to 3rdparty/python:<req> is there a way i can use the
_parse_context
to get this to happen?
I would recommend that you perhaps ignore the
rel_path
from the parse context and that you always look at the build root for the file. You would call your new macro in
3rdparty/python/BUILD
, which would result in the new
python_requirement_library
targets that get generated from your macro being created there, in
3rdparty/python
. Rather than
os.path.join(self._parse_context.rel_path, requirements_relpath)
, use
get_buildroot()
from
pants.base.environment
, so something like
os.path.join(get_buildroot(), "Pipfile.lock")
You could also then make it more generic by having your macro take some parameter like
path_from_buildroot
.
relatedly, im not sure i understand the purpose the target defined here:
Indeed, that target is being used so that invalidation works properly. Without adding a dependency on all the generated
python_requirement_library
targets, Pants would not know that they’re invalidated upon changes to the Pipfile. Previously, this dependency was expressed by creating a
files()
target. But, that resulted in the requirements.txt file showing up more than we wanted to, as it was included wherever else
files()
targets would be pulled in. So, Benjy created
_python_requirement_library
to better model the situation. Afaict, you can reuse that same target type, no need to add a new
_pipfile_requirement_library
or anything.
w

wonderful-iron-54019

08/18/2020, 5:37 PM
ok im down with always look at the root ("im actually defining the
pipenv_requirements
in
3rdparty/python
and setting the rel to
../../Pipfile.lock
with no issues. Where the problem comes in is in defining the
_python_requirement_library
target since the sources fielld being passed is
['../../Pipfile.lock']
and PEX is not happy with that
but it seems that might cause invalidartion issues
h

hundreds-father-404

08/18/2020, 5:39 PM
Hm where is that showing up with Pex? If you use
os.path.join(get_buildroot(), "Pipfile.lock")
, I think you’d had have an absolute path and Pex would be fine
w

wonderful-iron-54019

08/18/2020, 5:39 PM
yeah let me revert one sec
BOST (big ol's stack trace incoming)
ok so i think that may be a path forward
h

hundreds-father-404

08/18/2020, 5:43 PM
so i think that may be a path forward
Which part? The
get_buildroot()
suggestion?
w

wonderful-iron-54019

08/18/2020, 5:44 PM
yes! now let me ask you this, if i wanted to design this in a way that might get pulled into the main project, would something like a
resolve_from_buildroot
option on the build_alias be acceptable to toggle this behavior on and off?
or would you prefer that for pipenv the alias would always resolve from root
h

hundreds-father-404

08/18/2020, 5:48 PM
To get pulled into the main project (which would be great!), we’d want it to be generic. I think the default behavior would be like
python_requirements
that the
Pipfile.lock
is a sibling file to where your BUILD file is. For any new users, if they have a top-level
Pipfile.lock
, we would encourage them to use a top-level
BUILD
and specify reqs via
//:my-req
, rather than
3rdparty/python:req
. From there, though, the macro could have options like
resolve_from_buildroot
and/or
requirements_relpath
to do what you’re describing.
w

wonderful-iron-54019

08/18/2020, 5:48 PM
yup that makes sense
attempting the whole resolve option now, stay tuned
🤞 1
ok. so it seems to work in the first case, want to. test invalidation
circling back here, im not quite sure how to make this work with
resolve_from_buildroot
and properly invalidate
resolving the pipfile works! but the invalidation target reports:
Unmatched glob from 3rdparty/python:Pipfile.lock's sources field: "3rdparty/python/Pipfile.lock"
👍 1
h

hundreds-father-404

08/18/2020, 7:42 PM
Ahh, hm. @witty-crayon-22786 is it possible for a CAFO to create an object in an ancestor folder? JP is trying to create a
_python_requirements_file
defined in the build root from a CAFO that gets called in
3rdparty/python/BUILD
. He wants all the generated
python_requirement_library
targets to still be in
3rdparty/python
, though.
w

witty-crayon-22786

08/18/2020, 7:43 PM
it is not.
😞 1
sorry. that’s one of the weirdest old v1 APIs that we haven’t been able to improve yet
w

wonderful-iron-54019

08/18/2020, 7:44 PM
the
python_requirement_library
?
w

witty-crayon-22786

08/18/2020, 7:44 PM
the “CAFO” API
h

hundreds-father-404

08/18/2020, 7:44 PM
Bummer. This is less automatic, but I think this will solve your specific problem, JP. Manually create a
_python_requirements_file
target in your top level
BUILD
for
Pipfile.lock
. Check that in. Then, in your macro, have it declare an explicit dependency on
//:pipfile
for every generated
python_requirement_library
.
w

wonderful-iron-54019

08/18/2020, 7:44 PM
ah
h

hundreds-father-404

08/18/2020, 7:44 PM
Where CAFO = context-aware object factory
w

wonderful-iron-54019

08/18/2020, 7:45 PM
yeah i think that'll work
was hoping to have something more general though
i quite like the
3rdparty
construction but Pipfile's just 'make sense' in the root especially if you plan to run things off pants
👍 1
i'll think on this
h

hundreds-father-404

08/18/2020, 7:46 PM
If you still want it to be general, you could maybe have an option on the macro called
pipfile_target: Optional[str]
. If unset, then generate the
_python_requirement_library
in the same directory as the macro. Otherwise, use the provided explicit dep. This does leak far more than the original design, but is somewhat general.
w

wonderful-iron-54019

08/18/2020, 7:46 PM
at any rate, its working functionally
❤️ 1
👖 1
Hi folks, based on this convo, just opened the following: https://github.com/pantsbuild/pants/pull/10654/files Eager to hear your feedback
3 Views