https://pantsbuild.org/ logo
Join Slack
Powered by
Can I trigger goals from within my plugin? For ex...
# plugins
h
Can I trigger goals from within my plugin? For example, I'm building a custom package goal and need to trigger the normal package goal on some targets first.
h
Yep! And the cache will be shared across those two goals Although you don't quite trigger the goal_rule itself, you trigger the downstream rules that goal would be invoking to get the same effect. See core/goals/package.py. You'll copy a decent chunk of that code, iirc the await MultiGet(Get(BuiltArtifact, BuildArtifactRequest)) I'm afk (holiday in US today), but lmk if that doesn't fully make sense and one of us can try to give more clear instructions
👍 1
h
I tried but not making much sense to me. In my BUILD file I have the following: 
*dash_distribution*(
name="dist",
libraries = [
"lib_dash",
"lib_test_support"
]
)
In my goal rule I have: 
@goal_rule
 
async def *dash_package*(console: Console, targets: Targets) -> DashPackage:
libraries = []
for target in targets:
if target.has_field(LibrariesField):
libraries = target[LibrariesField].value
So I can get the list of "libraries" no problem. Now what I would like to do is trigger the equivalent of: 
./pants package lib_dash
./pants package lib_test_support
After that I need to move a bunch of files around and copy the wheels from the above two packages into the output folder.
h
Question, do you mind sharing the bigger objective you're after? It sounds like you will want to be able to do 
./pants package path/to:dash_distribution
, and that will create a Dash executable which possibly includes artifacts built from running 
./pants package
on those? Very similar to the 
archive
type we have: https://www.pantsbuild.org/docs/resources#archive-create-a-zip-or-tar-file
h
Sure. We are working with Plotly Dash enterprise. We are going to have multiple independent dash applications in the repository. In Dash enterprise, each app gets it's own git repository for deployment. When developing, testing etc I would like to use pants to run a dash app locally. Once it's ready to be officially deployed, I need to copy sources, wheels, requirements.txt into the app's dash repository and commit the changes.
I created a bash script that does this, but thought it would be nice if I could do it directly in pants.
h
Once it's ready to be officially deployed, I need to copy sources, wheels, requirements.txt into the app's dash repository and commit the changes.
This is the part that you are writing a plugin for, right? What does the final "product"/artifact look like? Is it a zipped folder for example?
It sounds like indeed the goal is to run 
./pants package path/to:dash_distribution
, only that that will need to create several wheels based on 
python_distribution
targets first. Is that right? I don't think you necessarily want a separate goal 
dash-package
. You can do that if you want, but a goal of Pants is to give a consistent interface for users so that they don't need to remember whether it's 
./pants pex-package
vs. 
./pants setup-py
vs. 
./pants dash-package
, etc
h
Yes that is the part I'm trying to create. Its just a folder containing something like this: report_app .gitignore .git packages lib_dash-0.0.0-py3-none-any.whl lib_test_support-0.0.0-py3-none-any.whl app.py requirements.txt
👍 1
I was going to create the output in "dist" within the root of the repo.
Then either as part of or after generating these folders there would be a git push to deploy them to dash.
h
Cool. So to hook into the 
package
goal, you'll want to follow the guide on https://www.pantsbuild.org/docs/plugins-package-goal Ignore the part about "sources" for your new target type, that was a bad idea and we haven't finished fixing the guidance. Almost certainly, 
dash_distribution
should not have any 
sources
field.
Then either as part of or after generating these folders there would be a git push to deploy them to dash.
Sounds good. To scope it for now, I would recommend not adding the git push logic yet. Only automate the step of creating that directory. That can be a followup
h
kk - will look at that. I was trying to add my own goal. Something like: ./pants dash-package report_app So I shouldn't do that then?
h
You can do a dedicated 
dash-package
, but it's not as elegant and means your users now need to remember when to use 
package
vs. 
dash-package
Can you please share the portion of your bash script that is running 
./pants package
to create the wheels? It'll help to understand how to give advice on how to run that in your new rule
h
It's actually a python script that looks like this right now: 
def *package*(target):
dist_folder = 'dist/' + target
if not os.path.exists(dist_folder):
os.makedirs(dist_folder)
prepare_dist_folder(dist_folder)
# Copy root folder files.
 
for f in glob.glob(target + '/*.py') + ['requirements.txt']:
if not f.endswith('_test.py') and not f.endswith('conftest.py'):
shutil.copy(f, dist_folder)
# Copy assets.
 
assets_path = target + "/assets"
if os.path.exists(assets_path):
dist_assets_path = dist_folder + "/assets/"
if not os.path.exists(dist_assets_path):
os.makedirs(dist_assets_path)
shutil.copy(assets_path, dist_assets_path)
with *open*(target + '/BUILD_DEPS', 'r') as fd:
dependencies = fd.read().split(',')
with *open*(dist_folder + '/requirements.txt', 'a') as requirements:
for dependency in dependencies:
# Package dependency.
 
os.system('./pants package :' + dependency)
# copy dependencies to dist folder
 
wheel_name = dependency + "-0.0.0-py3-none-any.whl"
dist_packages_folder = dist_folder + '/packages/'
os.makedirs(dist_packages_folder, exist_ok=True)
shutil.copy('dist/' + wheel_name, dist_packages_folder)
# Append wheels to requirements.txt
 
requirements.write('./packages/' + wheel_name)
h
Thanks for sharing! So, are you running 
./pants package
on 
python_distribution
targets? That's the part I'm confused on: are you making wheels for your first-party distributions or also for third-party dependencies? If the latter, did you create a dedicated 
python_distribution
target for each requirement in your requirements.txt?
h
So I'm running os.system('./pants package :' + dependency) to build each library that the targeted app depends on. Then I'm copying the wheels into the packages folder of the output directory. Then I'm appending the wheels to the requirements.txt that was also copied into the output folder.
The wheels are just for first-party (my libraries).
💯 1
👍 1
h
Perfect, thanks! So, I found a much better example for how to call 
./pants package
in your new rule. https://github.com/pantsbuild/pants/blob/c55fd827252fb1288c1f1d1c7e0af78b551336e1/src/python/pants/core/target_types.py#L285-L291 Where 
package_targets
is 
Iterable[Target]
, meaning it can be the targets that you were collecting into a list with your 
target.has_field(LibrariesField)
snippet Basically, this snippet is going to convert the 
Target
objects into things called `FieldSet`s that are ready to work with the rule to create wheels. In the next line, you then invoke that rule
You'll get back a collection of 
BuiltPackage
objects, defined in 
package.py
. Each of those has a 
digest: Digest
property, which will give you access to the final result through the file system API https://www.pantsbuild.org/docs/rules-api-file-system
h
Taking a look ...
Wow - really lost. I tried the following: 
class *DashDependencies*(*SpecialCasedDependencies*):
alias = "dash_dependencies"
*help* = ("")
class *DashDistributionTarget*(*Target*):
"""Dash App Deployment."""
alias = "dash_distribution"
core_fields = (
*COMMON_TARGET_FIELDS,
OutputPathField,
DashDependencies,
)
*@dataclass*(frozen=True)
class *DashFieldSet*(*PackageFieldSet*):
required_fields = ()
dependencies: DashDependencies
output_path: OutputPathField
*@rule*(level=LogLevel.DEBUG)
async def *dash_package*(field_set: DashFieldSet) -> BuiltPackage:
dependency_targets = await Get(Targets, UnparsedAddressInputs, field_<http://set.dependencies.to|set.dependencies.to>_unparsed_address_inputs())
package_field_sets_per_target = await Get(
FieldSetsPerTarget, FieldSetsPerTargetRequest(PackageFieldSet, dependency_targets)
)
packages = await MultiGet(
Get(BuiltPackage, PackageFieldSet, field_set)
for field_set in package_field_sets_per_target.field_sets
)
for p in packages:
*print*(p)
I get back 161333.03 [ERROR] Exception caught: (pants.engine.internals.scheduler.ExecutionError) File "/home/greg/.cache/pants/setup/bootstrap-Linux-x86_64/2.1.1_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 255, in run engine_result = self._run_v2(goals) File "/home/greg/.cache/pants/setup/bootstrap-Linux-x86_64/2.1.1_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 182, in _run_v2 return self._maybe_run_v2_body(goals, poll=False) File "/home/greg/.cache/pants/setup/bootstrap-Linux-x86_64/2.1.1_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 199, in _maybe_run_v2_body return self.graph_session.run_goal_rules( File "/home/greg/.cache/pants/setup/bootstrap-Linux-x86_64/2.1.1_py38/lib/python3.8/site-packages/pants/init/engine_initializer.py", line 125, in run_goal_rules exit_code = self.scheduler_session.run_goal_rule( File "/home/greg/.cache/pants/setup/bootstrap-Linux-x86_64/2.1.1_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 569, in run_goal_rule self._raise_on_error([t for _, t in throws]) File "/home/greg/.cache/pants/setup/bootstrap-Linux-x86_64/2.1.1_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 537, in _raise_on_error raise ExecutionError( Exception message: 1 Exception encountered: StopIteration: (Use --print-stacktrace to see more error details.)
3 Views
Open in Slack
PreviousNext
Powered by