Now I m going to post new questions about `platform specific Pants #plugins

Now I'm going to post new questions about `platfor...

rhythmic-morning-87313

05/25/2022, 6:23 AM

Now I'm going to post new questions about

platform_specific_resources()

target plugin here. There seems to be multiple ways to specify separate source lists for each platform, but at least I don't want to declare per-platform classes for targets or fields. I'm thinking about doing:

Copy code

platform_specific_resources(
  name="binaries1",
  platform="linux-x86_64",
  sources=[...],
)
platform_specific_resources(
  name="binaries2",
  platform="linux-arm64",
  sources=[...],
)
python_sources(
  name="lib",
  dependencies=[":binaries1", ":binaries2"],
  ...
)

where each

binaries1

and

binaries2

target generates empty resources or fully hydrated resources depending on the subsystem option like

--select-platform

. Is it possible to write such an optional sources?

✅ 1

rhythmic-morning-87313

05/25/2022, 8:21 AM

hmm

rhythmic-morning-87313

05/25/2022, 8:21 AM

instead of creating

platform_specific_resources

, could i try https://www.pantsbuild.org/docs/target-api-extending-targets to add an optional "platfom" field to

resources

rhythmic-morning-87313

05/25/2022, 8:22 AM

in such case, how do i customize the "rule"?

curved-television-6568

05/25/2022, 8:38 AM

Using a target plugin field ought to be able to achieve what you describe above. Rough sketch:

Copy code

resources(
  name="binaries1",
  sources=[...],
)
resources(
  name="binaries2",
  sources=[...],
)
resources(
  name="platform-binaries",
  platforms={
    "linux-arm64": ":binaries1",
    "linux-x86_64": ":binaries2",
  },
)
python_sources(
  name="lib",
  dependencies=[":platform-binaries"],
  ...
)

The idea being, using dependency injection for resources with a

platforms

field, based on some provided option, would allow you to selectively add resources to your Python

lib

. https://www.pantsbuild.org/docs/target-api-extending-targets Unfortunately I didn’t find any documentation for how dependency inference/injection works, so will have to glean that from source. One example of dependency injection is from the docker backend (with which I’m most familiar) https://github.com/pantsbuild/pants/blob/1c1c0f14b04d6398b9f541ba557cecad8e51ec19/src/python/pants/backend/docker/util_rules/dependencies.py

👀 1

curved-television-6568

05/25/2022, 8:43 AM

Actually, I think perhaps the custom

platforms

field is better targeted for the generic

target

target, as a dependency selector there. So

Copy code

target(
  name="platform-binaries",
  platforms={
    "linux-arm64": ":binaries1",
    "linux-x86_64": ":binaries2",
  },
)

https://github.com/pantsbuild/pants/blob/1c1c0f14b04d6398b9f541ba557cecad8e51ec19/src/python/pants/core/target_types.py#L488-L498

👀 1

rhythmic-morning-87313

05/25/2022, 8:46 AM

ahha

rhythmic-morning-87313

05/25/2022, 8:46 AM

that seems to be a good idea

rhythmic-morning-87313

05/25/2022, 9:04 AM

image.png

rhythmic-morning-87313

05/25/2022, 9:05 AM

Trying this, but I'd like to know how to make such "dict"-like field and access it inside the rule

curved-television-6568

05/25/2022, 9:08 AM

There’s a

DictStringToStringField

https://github.com/pantsbuild/pants/blob/1c1c0f14b04d6398b9f541ba557cecad8e51ec19/src/python/pants/engine/target.py#L1716

👍 1

curved-television-6568

05/25/2022, 9:10 AM

From the rule, then you’d pick out the value for the specific platform, and resolve that to an address to be injected as dependency, using `await Get(Address, UnparsedAddressInputs, …)`: https://github.com/pantsbuild/pants/blob/1c1c0f14b04d6398b9f541ba557cecad8e51ec19/src/python/pants/engine/internals/graph.py#L1084-L1098

curved-television-6568

05/25/2022, 9:11 AM

Example: https://github.com/pantsbuild/pants/blob/1c1c0f14b04d6398b9f541ba557cecad8e51ec19/src/python/pants/backend/docker/util_rules/dependencies.py#L30-L38

curved-television-6568

05/25/2022, 9:12 AM

The owning address is used as reference for relative addresses

curved-television-6568

05/25/2022, 9:16 AM

Hum, also, I think your

inject_for

must reference a field based on the actual

Dependencies

field type. And given that the

GenericTarget

doesn’t have a specific dependencies field type, the dependency injection you’re going for here would apply to all targets.. unless that is preferable, you’d want to have a dedicated target with a specific dependencies field for which to apply dependency injection on. (could simple mimic the GenericTarget but with a custom dependencies field)…

curved-television-6568

05/25/2022, 9:18 AM

So my idea above using the generic

target

was not the best.

rhythmic-morning-87313

05/26/2022, 5:51 AM

image.png

rhythmic-morning-87313

05/26/2022, 5:51 AM

Hmm...

rhythmic-morning-87313

05/26/2022, 5:51 AM

image.png

rhythmic-morning-87313

05/26/2022, 5:51 AM

it seems that my rule is not executed yet

rhythmic-morning-87313

05/26/2022, 6:10 AM

hmmm

rhythmic-morning-87313

05/26/2022, 6:11 AM

i think it's because

inject_for

expects a str-list

Dependencies

-like field, not

DictStringToStringField

rhythmic-morning-87313

05/26/2022, 6:12 AM

Somehow I need to make my own

Dependencies

field impl.

rhythmic-morning-87313

05/26/2022, 6:16 AM

🤔 ...

rhythmic-morning-87313

05/26/2022, 6:18 AM

somehow I need to "transform"

PlatformDependenciesField

to pass

if isinstance(request.field, inject_request_type.inject_for)

...

rhythmic-morning-87313

05/26/2022, 6:45 AM

hm..

rhythmic-morning-87313

05/26/2022, 6:45 AM

image.png

rhythmic-morning-87313

05/26/2022, 6:56 AM

hmmmm

rhythmic-morning-87313

05/26/2022, 6:56 AM

i think we are misunderstanding the dependency injection overall

curved-television-6568

05/26/2022, 7:32 AM

Yeah, I’ve not explained it properly. If you’re able to share your work, I could take a look

curved-television-6568

05/26/2022, 7:34 AM

Don’t worry if it feels like it’s a mess, it’s expected at this point while figuring stuff out 😉

rhythmic-morning-87313

05/26/2022, 7:45 AM

https://github.com/lablup/backend.ai/blob/monorepo-for-server/tools/pants-plugins/selective_resources/register.py

rhythmic-morning-87313

05/26/2022, 7:45 AM

https://github.com/lablup/backend.ai/blob/monorepo-for-server/src/ai/backend/runner/BUILD

rhythmic-morning-87313

05/26/2022, 7:46 AM

I'm testing with

./pants --selective-resources-platform=linux_x86_64 -ldebug dependencies --transitive src/ai/backend/runner:resources

curved-television-6568

05/26/2022, 7:47 AM

👀

rhythmic-morning-87313

05/26/2022, 7:48 AM

the rule function is incomplete and includes misuse of some APIs, but at the first place I'm trying to let it get executed.

curved-television-6568

05/26/2022, 7:54 AM

I’ve got an errand to run, but will be back looking at this momentarily.

curved-television-6568

05/26/2022, 8:16 AM

OK, yeah, I’m gonna make some tweaks, see what you make of them…

👀 1

curved-television-6568

05/26/2022, 8:46 AM

https://github.com/lablup/backend.ai/pull/425/files

curved-television-6568

05/26/2022, 8:47 AM

Copy code

$ ./pants --selective-resources-platform=linux_arm64 dependencies --transitive src/ai/backend/runner:resources
10:47:13.93 [INFO] Initializing scheduler...
10:47:14.06 [INFO] Scheduler initialized.
10:47:14.10 [INFO] ---- request: InjectPlatformSpecificDependenciesRequest(dependencies_field=<class 'selective_resources.register.PlatformSpecificDependenciesField'>(alias='dependencies', address=src/ai/backend/runner:platform-binaries, value=None, default=None))
10:47:14.10 [INFO] ---- selected platform: <Platform.linux_arm64: 'linux_arm64'>
10:47:14.10 [INFO] === Platforms: FrozenDict({'linux_x86_64': ':linux-x86_64-binaries', 'linux_arm64': ':linux-arm64-binaries'})

src/ai/backend/runner/DO_NOT_STORE_PERSISTENT_FILES_HERE.md:resources
src/ai/backend/runner/dropbear.glibc.aarch64.bin:linux-arm64-binaries
src/ai/backend/runner/dropbear.musl.aarch64.bin:linux-arm64-binaries
src/ai/backend/runner/dropbearconvert.glibc.aarch64.bin:linux-arm64-binaries
src/ai/backend/runner/dropbearconvert.musl.aarch64.bin:linux-arm64-binaries
src/ai/backend/runner/dropbearkey.glibc.aarch64.bin:linux-arm64-binaries
src/ai/backend/runner/dropbearkey.musl.aarch64.bin:linux-arm64-binaries
src/ai/backend/runner/entrypoint.sh:resources
src/ai/backend/runner/krunner-extractor.img.aarch64.tar.xz:linux-arm64-binaries
src/ai/backend/runner/krunner-extractor.sh:resources
src/ai/backend/runner/libbaihook.alpine3.8.aarch64.so:linux-arm64-binaries
src/ai/backend/runner/libbaihook.centos7.6.aarch64.so:linux-arm64-binaries
src/ai/backend/runner/libbaihook.ubuntu20.04.aarch64.so:linux-arm64-binaries
src/ai/backend/runner/logo.svg:resources
src/ai/backend/runner/scp.alpine3.8.aarch64.bin:linux-arm64-binaries
src/ai/backend/runner/scp.centos7.6.aarch64.bin:linux-arm64-binaries
src/ai/backend/runner/scp.ubuntu16.04.aarch64.bin:linux-arm64-binaries
src/ai/backend/runner/scp.ubuntu18.04.aarch64.bin:linux-arm64-binaries
src/ai/backend/runner/scp.ubuntu20.04.aarch64.bin:linux-arm64-binaries
src/ai/backend/runner/sftp-server.alpine3.8.aarch64.bin:linux-arm64-binaries
src/ai/backend/runner/sftp-server.centos7.6.aarch64.bin:linux-arm64-binaries
src/ai/backend/runner/sftp-server.ubuntu16.04.aarch64.bin:linux-arm64-binaries
src/ai/backend/runner/sftp-server.ubuntu18.04.aarch64.bin:linux-arm64-binaries
src/ai/backend/runner/sftp-server.ubuntu20.04.aarch64.bin:linux-arm64-binaries
src/ai/backend/runner/su-exec.alpine3.8.aarch64.bin:linux-arm64-binaries
src/ai/backend/runner/su-exec.centos7.6.aarch64.bin:linux-arm64-binaries
src/ai/backend/runner/su-exec.ubuntu16.04.aarch64.bin:linux-arm64-binaries
src/ai/backend/runner/su-exec.ubuntu18.04.aarch64.bin:linux-arm64-binaries
src/ai/backend/runner/su-exec.ubuntu20.04.aarch64.bin:linux-arm64-binaries
src/ai/backend/runner/tmux.glibc.aarch64.bin:linux-arm64-binaries
src/ai/backend/runner/tmux.musl.aarch64.bin:linux-arm64-binaries
src/ai/backend/runner:platform-binaries
src/ai/backend/runner:version

🙏 1

curved-television-6568

05/26/2022, 8:47 AM

Copy code

$ ./pants --selective-resources-platform=linux_x86_64 dependencies --transitive src/ai/backend/runner:resources
10:47:27.07 [INFO] ---- request: InjectPlatformSpecificDependenciesRequest(dependencies_field=<class 'selective_resources.register.PlatformSpecificDependenciesField'>(alias='dependencies', address=src/ai/backend/runner:platform-binaries, value=None, default=None))
10:47:27.07 [INFO] ---- selected platform: <Platform.linux_x86_64: 'linux_x86_64'>
10:47:27.07 [INFO] === Platforms: FrozenDict({'linux_x86_64': ':linux-x86_64-binaries', 'linux_arm64': ':linux-arm64-binaries'})

src/ai/backend/runner/DO_NOT_STORE_PERSISTENT_FILES_HERE.md:resources
src/ai/backend/runner/dropbear.glibc.x86_64.bin:linux-x86_64-binaries
src/ai/backend/runner/dropbear.musl.x86_64.bin:linux-x86_64-binaries
src/ai/backend/runner/dropbearconvert.glibc.x86_64.bin:linux-x86_64-binaries
src/ai/backend/runner/dropbearconvert.musl.x86_64.bin:linux-x86_64-binaries
src/ai/backend/runner/dropbearkey.glibc.x86_64.bin:linux-x86_64-binaries
src/ai/backend/runner/dropbearkey.musl.x86_64.bin:linux-x86_64-binaries
src/ai/backend/runner/entrypoint.sh:resources
src/ai/backend/runner/krunner-extractor.img.x86_64.tar.xz:linux-x86_64-binaries
src/ai/backend/runner/krunner-extractor.sh:resources
src/ai/backend/runner/libbaihook.alpine3.8.x86_64.so:linux-x86_64-binaries
src/ai/backend/runner/libbaihook.centos7.6.x86_64.so:linux-x86_64-binaries
src/ai/backend/runner/libbaihook.ubuntu18.04.x86_64.so:linux-x86_64-binaries
src/ai/backend/runner/libbaihook.ubuntu20.04.x86_64.so:linux-x86_64-binaries
src/ai/backend/runner/logo.svg:resources
src/ai/backend/runner/scp.alpine3.8.x86_64.bin:linux-x86_64-binaries
src/ai/backend/runner/scp.centos7.6.x86_64.bin:linux-x86_64-binaries
src/ai/backend/runner/scp.ubuntu16.04.x86_64.bin:linux-x86_64-binaries
src/ai/backend/runner/scp.ubuntu18.04.x86_64.bin:linux-x86_64-binaries
src/ai/backend/runner/scp.ubuntu20.04.x86_64.bin:linux-x86_64-binaries
src/ai/backend/runner/sftp-server.alpine3.8.x86_64.bin:linux-x86_64-binaries
src/ai/backend/runner/sftp-server.centos7.6.x86_64.bin:linux-x86_64-binaries
src/ai/backend/runner/sftp-server.ubuntu16.04.x86_64.bin:linux-x86_64-binaries
src/ai/backend/runner/sftp-server.ubuntu18.04.x86_64.bin:linux-x86_64-binaries
src/ai/backend/runner/sftp-server.ubuntu20.04.x86_64.bin:linux-x86_64-binaries
src/ai/backend/runner/su-exec.alpine3.8.x86_64.bin:linux-x86_64-binaries
src/ai/backend/runner/su-exec.centos7.6.x86_64.bin:linux-x86_64-binaries
src/ai/backend/runner/su-exec.ubuntu16.04.x86_64.bin:linux-x86_64-binaries
src/ai/backend/runner/su-exec.ubuntu18.04.x86_64.bin:linux-x86_64-binaries
src/ai/backend/runner/su-exec.ubuntu20.04.x86_64.bin:linux-x86_64-binaries
src/ai/backend/runner/tmux.glibc.x86_64.bin:linux-x86_64-binaries
src/ai/backend/runner/tmux.musl.x86_64.bin:linux-x86_64-binaries
src/ai/backend/runner:platform-binaries
src/ai/backend/runner:version

rhythmic-morning-87313

05/26/2022, 8:48 AM

oh........ nice! +_+

❤️ 1

curved-television-6568

05/26/2022, 8:49 AM

I should’ve linked you to this nugget of docs, I think would’ve made things more clear from the outset: https://github.com/pantsbuild/pants/blob/770c90af52380cab30211c2cc8aa7df3bcae38be/src/python/pants/engine/target.py#L2464-L2502 😉

curved-television-6568

05/26/2022, 8:49 AM

It was a fun exercise 🙂

🙌 1

rhythmic-morning-87313

05/26/2022, 8:50 AM

i've read the exactly that part of

target.py

but it didn't help me..

curved-television-6568

05/26/2022, 8:51 AM

Oh, ok. Maybe with my changes, it will become more clear, then, maybe 😁

👍 1

rhythmic-morning-87313

05/26/2022, 8:51 AM

image.png

curved-television-6568

05/26/2022, 8:51 AM

If it does, please do suggest edits to that doc string to make it more helpful.

rhythmic-morning-87313

05/26/2022, 8:52 AM

so i needed to have both of these two fields

rhythmic-morning-87313

05/26/2022, 8:52 AM

i was going to make it one thing and that made me to get confused

rhythmic-morning-87313

05/26/2022, 8:53 AM

PlatformSpecificDependenciesField

is something like a virtual field to trigger our own rule processing

rhythmic-morning-87313

05/26/2022, 8:53 AM

right?

curved-television-6568

05/26/2022, 8:53 AM

You need a regular dependencies field as target for your injection, and then you want something else as source for the injection logic. If that makes sense.

👍 1

curved-television-6568

05/26/2022, 8:53 AM

PlatformSpecificDependenciesField

is our “regular” dep field, in the above sentence.

curved-television-6568

05/26/2022, 8:54 AM

So, there’s nothing special about it, more than that the injection is implemented based on the type of the dependencies field. So you can “hook into” any kind dependencies dynamically.

curved-television-6568

05/26/2022, 8:54 AM

Then we add other custom fields to provide data for our injection logic.

curved-television-6568

05/26/2022, 8:55 AM

I think that last bit may be what was missing from the inejction doc string…

👍 1

rhythmic-morning-87313

05/26/2022, 8:55 AM

yeah, now i understand how it works

💯 1

rhythmic-morning-87313

05/26/2022, 8:55 AM

that's right!

rhythmic-morning-87313

05/26/2022, 8:56 AM

i began to miss around when i started to think about "how to merge

DictStringToStringField

and

Dependencies

?" ...

rhythmic-morning-87313

05/26/2022, 8:56 AM

anyway, thanks so much!

❤️ 1

curved-television-6568

05/26/2022, 8:56 AM

Yeah, I guess it’s rather complicated until all the pieces fit correctly…

curved-television-6568

05/26/2022, 8:57 AM

you’re so welcome 🙂

rhythmic-morning-87313

05/26/2022, 8:57 AM

i hope that this will be the final piece for my long journey to the mono-repo migration which took about one month.....

👍 1

rhythmic-morning-87313

05/26/2022, 8:58 AM

(i had many other work so could not work on this for full-time, but anyway ...)

rhythmic-morning-87313

05/26/2022, 8:58 AM

i'll be back in the night and continue over your work!

👍 1

5 Views

Open in Slack

Previous Next