https://pantsbuild.org/ logo
Join Slack
Powered by
I've noticed what I think is a bug in my recent `p...
# development
h
I've noticed what I think is a bug in my recent 
pants binary
work
if I run 
./pants binary <some target>
(i.e. running the v1 version of it)
and then immediately thereafter run 
./pants binary --v2 --no-v1 <same target>
the resulting pex written to 
dist
seems to be corrupt
a
nice
h
but I can fix the problem by deleting that pex and re-running the v2 command
a
yeah ok
h
curious if anyone else is seeing this
a
i would just make sure to delete any expected output file?
h
yeah that might be the cleanest way to fix this
a
i don't think we need to try to merge pex files
not sure what that bug is though it wouldn't be impossible to fix
just maybe not now idk
h
so this is coming from the underlying rust 
materialize_directories
probably doing something wrong
a
hm
h
like trying ot merge the newly-created pex with the pex already on the filesystem?
a
that might be more important then
well, that sometimes can happen when you use zip libraries
and you open a file
and don't specify truncating the existing file
a lot of code isn't written to handle that case and things can break
how corrupt was the pex? was it a valid zip?
h
Untitled
Untitled
it did seem to be a valid zip
a
could you try running with 
--no-compile
or is it already doing that
h
like, pass that flag to the pex?
a
i feel like i ran into this a week ago with another pex thing i was doing
yes
h
when I run it manually?
a
well
no
when it's created
h
hm what does that flag do?
a
it avoids compiling pyc files
if you look into that entry in the pex
Copy code
> unzip -p /home/gregs/code/pants/dist/main.pex .bootstrap/pex/__init__.py
what does that print
because i've had pex write pyc files to that file before i think
i don't have the repro on hand, but i could probably find it
h
Copy code
gregs@gregs-dev-laptop ~/c/p/dist> unzip -p /home/gregs/code/pants/dist/main.pex .bootstrap/pex/__init__.py
.bootstrap/pex/__init__.py:  mismatching "local" filename (.bootstrap/_pex/__init__.pyc),
         continuing with "central" filename version
3
n]@sdS)Nrrrootstrap/_pex/__init__.p<module>s⏎
a
and 
--no-compile
fixed that
there we go
i have no clue what "local" and "central" filenames are
this might be avoidable if we add 
--no-compile
h
hm, ok
a
but i believe this looks maybe like pex doing something wrong
h
I wonder if it would be better to just explicitly delete any files with the same name in dist, before overwriting them
a
but also if we just make sure to delete output files beforehand, that would work not just for this issue
yeah
h
I dunno if 
materialize_directories
lets us do that already
a
like we can avoid this with any program that interacts with files this is a good long-term choice i think
h
maybe there's an overwrite flag?
a
maybe we should always do it whenever we materialize?
it feels like an optimization to not overwrite the file that we may not need write now
right now
h
hm, is there a way to do this iwth the APIs in the engine already?
in 
store/src/lib.rs
, we have 
materialize_file
that actually does the disk-writing
but I'm not sure what hte behavior of the flag is if the file already exists on-disk
a
oh
yeah
also
well
i might try just building the pex with 
--no-compile
in pants and see if that fixes the issue
h
rm'ing the file and building it again (in v2) fixes the issue, I'd rather make the semantics of file materialization be correct
a
i think the issue might be separate
because the only reason i can think of for the pex to be corrupt is that the file previously existed in dist, and was somehow snapshotted and used as the input to the pex creation
and where we materialize the file doesn't seem like it would affect that
since 
--no-compile
fixes the issue, we know it's not because file materialization itself is corrupting the file
h
hm, that definitely shouldn't be happening
the pex should be getting created in a tmpdir
a
yes, and if the one in dist was snapshotted and used as the input, then it might also be in that tmpdir
h
we shouldn't be snapshotting things in 
dist
, right?
unless this is a cas bug?
a
i don't think we should be
i also cannot explain why pex is acting as if the output file is already there
^if that hypothesis is correct
h
adding that flag fixes the issue
at pex creation time
a
if we use 
--no-process-execution-cleanup-local-dirs
we can look at the output pex, but that doesn't let us see the input
if we run with 
-ldebug
we might get the process execution request printed out?
and then we could view the contents of the 
input_files
digest with 
fs_util
h
one thing I was doing when I was oriiginally debugging this
was taking the 
Digest
we pass to ExecuteProcessRequest
and requesting a 
Snapshot
from that and printing the snapshot
which gives:
Copy code
SNAP: Snapshot(directory_digest<Exactly(Digest)>=Digest(fingerprint=d8f1dacca264009f32c6fb5171e9c7f561dd2d2e6fde1c80593848a66872c9db, serialized_bytes_length=168), files<Exactly(tuple)>=('pex', 'source_files/3rdparty/python/requirements.txt', 'source_files/example/__init__.py', 'source_files/example/hello/__init__.py', 'source_files/example/hello/greet/__init__.py', 'source_files/example/hello/greet/greet.py', 'source_files/example/hello/main/__init__.py', 'source_files/example/hello/main/main.py'), dirs<Exactly(tuple)>=('source_files', 'source_files/3rdparty', 'source_files/3rdparty/python', 'source_files/example', 'source_files/example/hello', 'source_files/example/hello/greet', 'source_files/example/hello/main'))
it also seems like the 
no-process-execution-cleanup-local-dirs
flag isn't doing anything
and I'm not sure why
a
check the output! it should print out a directory
it's in 
/tmp
now
it's not obvious i had to figure it out
Open in Slack
PreviousNext
Powered by