Bug 1389842

Summary: upgrade from 7.2.7 to 7.3 not using available static deltas
Product: Red Hat Enterprise Linux 7 Reporter: Micah Abbott <miabbott>
Component: rhel-server-atomicAssignee: Colin Walters <walters>
Status: CLOSED CURRENTRELEASE QA Contact: atomic-bugs <atomic-bugs>
Severity: unspecified Docs Contact: Yoana Ruseva <yruseva>
Priority: unspecified    
Version: 7.3Keywords: Extras
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Known Issue
Doc Text:
Consequence: Upgrading to RHELAH 7.3 from RHELAH 7.2.7 may not use static deltas Workaround (if any): If the system contains the refspec 'rhel-atomic-host:rhel-atomic-host/7/x86_64/standard' in the output of 'ostree refs', the user should delete the refspec with the command 'ostree refs --delete rhel-atomic-host:rhel-atomic-host/7/x86_64/standard' and perform the upgrade. Result: Using 'atomic host upgrade' to upgrade 7.3 should retrieve static deltas. The output from the command should look similar to: # atomic host upgrade Updating from: rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard 21 delta parts, 15 loose fetched; 214444 KiB transferred in 73 seconds
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-05-03 17:59:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Comment 2 Colin Walters 2016-10-31 16:44:05 UTC
I was getting deltas downloaded in a quick test here, but now I'm seeing:

Receiving delta parts: 21/21 1.0 MB/s 322.3 MB/219.6 MB

Which showed no signs of stopping.  `systemctl restart rpm-ostreed && atomic host upgrade` fixed it.

Comment 3 Colin Walters 2016-10-31 17:17:18 UTC
Yeah, something is clearly going badly wrong here.  I get in the end:

21 delta parts, 6231 loose fetched; 405931 KiB transferred in 381 seconds

Why did we fetch over 6000 loose objects?

Comment 4 Micah Abbott 2016-11-01 16:14:51 UTC
After debugging with @walters a bit this morning, it looks like this is a side effect of BZ#1233358

After registering with subscription-manager, we end up with two refspecs that use the same branch, but on different remotes:

# ostree refs
ostree/1/1/0
ostree/1/1/1
rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard
rhel-atomic-host:rhel-atomic-host/7/x86_64/standard


When the client goes looking for deltas, it incorrectly tries to find a delta for the commit associated with the refspec 'rhel-atomic-host:rhel-atomic-host/7/x86_64/standard'.  This can be teased out with the 'ostree pull --require-static-deltas' command.

# rpm-ostree status
State: idle
Deployments:
● rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard
       Version: 7.2.7 (2016-09-09 18:43:35)
        Commit: 347c3f5eb641e69fc602878c646cf42c4bcd5d9f36847a1f24ff8f3ec80f17b1
        OSName: rhel-atomic-host

  rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard
       Version: 7.2.6-1 (2016-08-18 16:01:03)
        Commit: dae35767902aad07b087d359be20f234d244da79fdd4734cd2fbc3ee39b12cf8
        OSName: rhel-atomic-host

# env OSTREE_DEBUG_HTTP=1 ostree pull --require-static-deltas rhel-atomic-host-ostree rhel-atomic-host/7/x86_64/standard
...
error: Static deltas required, but none found for dae35767902aad07b087d359be20f234d244da79fdd4734cd2fbc3ee39b12cf8 to 90c9735becfff1c55c8586ae0f2c904bc0928f042cd4d016e9e0e2edd16e5e97


To work around this, we can delete the "old" refspec and then try to perform the upgrade again.  And then we will see the deltas being fetched.

# ostree refs --delete rhel-atomic-host:rhel-atomic-host/7/x86_64/standard

# rpm-ostree upgrade
Updating from: rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard

21 delta parts, 15 loose fetched; 214444 KiB transferred in 73 seconds

Comment 5 Micah Abbott 2017-05-03 17:59:35 UTC
We have doc text explaining a workaround and this issue does not exist in the current release, so I am closing this.