Bug 1235936 - rpm-ostree composed tree names files on /boot differently on each generation, leading to large delta between versions
Summary: rpm-ostree composed tree names files on /boot differently on each generation,...
Keywords:
Status: ASSIGNED
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: rpm-ostree-client
Version: 7.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Colin Walters
QA Contact: atomic-bugs@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-06-26 06:44 UTC by Leandro Santiago
Modified: 2020-07-03 16:57 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)

Description Leandro Santiago 2015-06-26 06:44:03 UTC
Description of problem:
I am using rpm-ostree to generate a system based on centos7 and using delta updates. But those delta files are very big (~32MB), even when I had changed only one or two lines in a script between two versions (or even if I haven't changed at all).

By running ostree diff, I got this:

(...small-changes-on-scripts...)
D    /boot/initramfs-3.10.0-229.7.2.el7.x86_64.img-c8c66111612920826e08d9b10a09624be3d41cedb44ea9cc40afe7b0cc74f124
D    /boot/vmlinuz-3.10.0-229.7.2.el7.x86_64-c8c66111612920826e08d9b10a09624be3d41cedb44ea9cc40afe7b0cc74f124
D    /usr/lib/ostree-boot/initramfs-3.10.0-229.7.2.el7.x86_64.img-c8c66111612920826e08d9b10a09624be3d41cedb44ea9cc40afe7b0cc74f124
D    /usr/lib/ostree-boot/vmlinuz-3.10.0-229.7.2.el7.x86_64-c8c66111612920826e08d9b10a09624be3d41cedb44ea9cc40afe7b0cc74f124
A    /boot/initramfs-3.10.0-229.7.2.el7.x86_64.img-755d6f3609c093448a21efdadb8ef173f26d596eec6ad8194f83f20566618ba3
A    /boot/vmlinuz-3.10.0-229.7.2.el7.x86_64-755d6f3609c093448a21efdadb8ef173f26d596eec6ad8194f83f20566618ba3
A    /usr/lib/ostree-boot/initramfs-3.10.0-229.7.2.el7.x86_64.img-755d6f3609c093448a21efdadb8ef173f26d596eec6ad8194f83f20566618ba3
A    /usr/lib/ostree-boot/vmlinuz-3.10.0-229.7.2.el7.x86_64-755d6f3609c093448a21efdadb8ef173f26d596eec6ad8194f83f20566618ba3

The files vmlinuz-3.10.0-229.7.2.el7.x86_64 and initramfs-3.10.0-229.7.2.el7.x86_64.img have the same content in both versions, but as rpm-ostree added different suffixes on their names, they are considered different, and as they are big in size, the delta files are big too.

Version-Release number of selected component (if applicable):

How reproducible:

Run rpm-ostree compose tree two times to the same repository and then check the difference between those two trees with ostree diff.


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:
vmlinuz and initramfs with same name on both versions, if they haven't changed.

Additional info:
I am using the most recent rpm-ostree from the centos atomic7 repository.

Comment 1 Leandro Santiago 2015-06-26 08:56:37 UTC
(In reply to Leandro Santiago from comment #0)
> Description of problem:
> I am using rpm-ostree to generate a system based on centos7 and using delta
> updates. But those delta files are very big (~32MB), even when I had changed
> only one or two lines in a script between two versions (or even if I haven't
> changed at all).
> 
> By running ostree diff, I got this:
> 
> (...small-changes-on-scripts...)
> D   
> /boot/initramfs-3.10.0-229.7.2.el7.x86_64.img-
> c8c66111612920826e08d9b10a09624be3d41cedb44ea9cc40afe7b0cc74f124
> D   
> /boot/vmlinuz-3.10.0-229.7.2.el7.x86_64-
> c8c66111612920826e08d9b10a09624be3d41cedb44ea9cc40afe7b0cc74f124
> D   
> /usr/lib/ostree-boot/initramfs-3.10.0-229.7.2.el7.x86_64.img-
> c8c66111612920826e08d9b10a09624be3d41cedb44ea9cc40afe7b0cc74f124
> D   
> /usr/lib/ostree-boot/vmlinuz-3.10.0-229.7.2.el7.x86_64-
> c8c66111612920826e08d9b10a09624be3d41cedb44ea9cc40afe7b0cc74f124
> A   
> /boot/initramfs-3.10.0-229.7.2.el7.x86_64.img-
> 755d6f3609c093448a21efdadb8ef173f26d596eec6ad8194f83f20566618ba3
> A   
> /boot/vmlinuz-3.10.0-229.7.2.el7.x86_64-
> 755d6f3609c093448a21efdadb8ef173f26d596eec6ad8194f83f20566618ba3
> A   
> /usr/lib/ostree-boot/initramfs-3.10.0-229.7.2.el7.x86_64.img-
> 755d6f3609c093448a21efdadb8ef173f26d596eec6ad8194f83f20566618ba3
> A   
> /usr/lib/ostree-boot/vmlinuz-3.10.0-229.7.2.el7.x86_64-
> 755d6f3609c093448a21efdadb8ef173f26d596eec6ad8194f83f20566618ba3
> 
> The files vmlinuz-3.10.0-229.7.2.el7.x86_64 and
> initramfs-3.10.0-229.7.2.el7.x86_64.img have the same content in both
> versions, but as rpm-ostree added different suffixes on their names, they
> are considered different, and as they are big in size, the delta files are
> big too.
> 
> Version-Release number of selected component (if applicable):
> 
> How reproducible:
> 
> Run rpm-ostree compose tree two times to the same repository and then check
> the difference between those two trees with ostree diff.
> 
> 
> Steps to Reproduce:
> 1.
> 2.
> 3.
> 
> Actual results:
> 
> 
> Expected results:
> vmlinuz and initramfs with same name on both versions, if they haven't
> changed.
> 
> Additional info:
> I am using the most recent rpm-ostree from the centos atomic7 repository.
A fix on my comment: I am actually using rpm-ostree from fedora22 but I tested with centos7 + atomic7-testing and the results were the same.

Investigating better now I can see that the file vmlinux- is the same across the versions (I checked out it from several commits and the file hasn't changed), but initramfs changes across the builds as is't it seems to be generated during tree composing time.

The suffix is calculated by "cat vmlinuz-* initramfs-* | sha256sum", so if any of those files change, the sum changes too.

One partial solution would be using only the kernel checksum to compute the suffix, so only the initramfs would be used during the delta creation. 

It would reduce its size a little bit, as here vmlinuz is 5MB while initramfs is 17MB.

But I suspect having two kernels with the same checksum would generate more problems than solution to the rest of ostree working way.

Comment 2 Colin Walters 2015-06-26 12:26:05 UTC
Yes, every tree change currently rebuilds the initramfs because we're being conservative.

See: https://bugzilla.redhat.com/show_bug.cgi?id=1098457

which we could teach rpm-ostree to use.  This way, we'd regenerate the initramfs if bash or systemd change, but not if e.g. docker changes.

Comment 3 Leandro Santiago 2015-06-26 13:26:09 UTC
(In reply to Colin Walters from comment #2)
> Yes, every tree change currently rebuilds the initramfs because we're being
> conservative.
> 
> See: https://bugzilla.redhat.com/show_bug.cgi?id=1098457
> 
> which we could teach rpm-ostree to use.  This way, we'd regenerate the
> initramfs if bash or systemd change, but not if e.g. docker changes.

As each tree composing is independent, perhaps there is some way to generate initramfs which are identical (at binary level) from the same input. Perhaps not saving the creation timestamps nor other information from runtime (as mktemp), so the image creation would be deterministic.

I don't know if it's possible with tar/gzip, just a suggestion.

Comment 5 Leandro Santiago 2015-10-07 15:26:21 UTC
This issue can be solved by using cpio 2.12 and some recent version of dracut (I used 041, borrowed from fedora 22 on my centos7 based system), by adding the option --reproducible when calling dracut.

Comment 6 Dusty Mabe 2017-06-14 15:46:45 UTC
This is fixed in upstream fedora but depends on BZ1386662 and BZ1386675 in RHEL

Comment 8 Steve Milner 2018-02-12 15:10:50 UTC
This is still on going waiting on https://bugzilla.redhat.com/show_bug.cgi?id=1386675


Note You need to log in before you can comment on or make changes to this bug.