https://pantsbuild.org/ logo
#general
Title
# general
h

high-egg-2153

03/10/2021, 9:03 PM
Is there anyway to force pants to rebuild everything when packaging? I need to increment the version number of my libraries before deployment. I created a plugin as per the documentation that does the trick. For now it reads the version from a file in the repo root. The plugin adds the version to the PKG-INFO and setup.py of all my wheels. However, when I increment the version number and re-run package, it seems to just use a cached version. I tried making the file containing the version number a dependency in the python_distribution target thinking it would see a change and rebuild, but no luck.
h

hundreds-father-404

03/10/2021, 9:07 PM
Hm, I'm afk but try --no-pantsd to isolate if it's being cached to disk vs memoized in memory
h

high-egg-2153

03/10/2021, 9:15 PM
Yup that works
h

hundreds-father-404

03/10/2021, 9:19 PM
Huh. I would have expected the file watcher to pick up the change 🤔 if you have a moment, it would be helpful to add log statements to your plugin https://www.pantsbuild.org/docs/rules-api-logging If the rule is memoized, it won't log what's inside because the rule won't execute. So, you could add a log to before where you read the file with PathGlobs for example, then after, to isolate where the memorization is happening when you don't expect it. That will help us figure out if there's a legit Pants bug
h

high-egg-2153

03/10/2021, 9:25 PM
I already did have logging in the plugin. After the first run no more logs. With -no-pantsd I see the logging everytime.
w

witty-crayon-22786

03/10/2021, 9:37 PM
which API are you using to read the file in the plugin?
@high-egg-2153: ^
h

high-egg-2153

03/10/2021, 9:46 PM
Right now in the plugin I'm just using os.read, which I'll eventually change. But the thing is is that the plugin doesn't even get called again after the initial package.
w

witty-crayon-22786

03/10/2021, 9:47 PM
that’s the source of the bug
👍 1
you need to use a rule-aware API to read files: see https://www.pantsbuild.org/docs/rules-api-file-system
h

high-egg-2153

03/10/2021, 9:48 PM
kk - I did however make the version file a dependency in the python_library target. Thought that would cause it to detect the change. Guess not. Ok I'll read the version file the proper way and see if that works
w

witty-crayon-22786

03/10/2021, 9:48 PM
Pants uses filesystem watching to know when to re-run
@rules
, but in order to know what to watch, specific APIs need to be used to access the filesystem
👍 1
h

high-egg-2153

03/10/2021, 9:50 PM
Makes sense. Is there a way to just read a loose file or do I have to make it a dependency in the distribution target?
w

witty-crayon-22786

03/10/2021, 9:50 PM
in this case you probably want to do something like:
Copy code
contents = await Get(DigestContents, PathGlobs(["my/version/file.txt"]))
version_content = contents[0].content.decode()
h

high-egg-2153

03/10/2021, 9:51 PM
Trying now
That works perfect. Once again you guys are awesome. Thanks!!
❤️ 1
w

witty-crayon-22786

03/10/2021, 9:55 PM
sorry for the trouble… we’d eventually like to make it harder to accidentally use
os.read
/ etc with better sandboxing
h

high-egg-2153

03/10/2021, 9:57 PM
No worries. I knew I was supposed to use the file API but took a shortcut. My bad.
b

busy-vase-39202

03/11/2021, 12:07 AM
Would it be useful to cite os.read in the docs, or elaborate on what other calls are implied in the docs by "work with the filesystem"? https://www.pantsbuild.org/docs/rules-api-file-system
It is not safe to use os.path or pathlib.Path like you normally would because this can break caching. Instead, Pants has several mechanisms to work with the file system in a safe and concurrent way.
w

witty-crayon-22786

03/11/2021, 5:49 PM
yea, will mention. thanks!
done.
12 Views