Bug 1207292

Summary: interrupting 'atomic host upgrade' multiple times can result in corrupt file objects
Product: Red Hat Enterprise Linux 7 Reporter: Micah Abbott <miabbott>
Component: rhel-server-atomicAssignee: Colin Walters <walters>
Status: CLOSED ERRATA QA Contact: Martin Jenner <mjenner>
Severity: high Docs Contact:
Priority: unspecified    
Version: 7.1CC: breilly, bsarathy, gscrivan, lmiksik, sct, sjenning
Target Milestone: rcKeywords: Extras
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: ostree-2015.5-2.atomic.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-05-15 16:16:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1200876    

Description Micah Abbott 2015-03-30 15:47:38 UTC 
Description of problem:
When 'atomic host upgrade' is interrupted multiple times, subsequent tries to finish the upgrade ultimately fail

Version-Release number of selected component (if applicable):

* 2015-02-19 20:26:26     7.1.0     5799825b36     rhel-atomic-host     rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard   

How reproducible:



Steps to Reproduce:
1.  Boot GA cloud image using KVM
2.  Register via 'subman' to stage
3.  Use 'atomic host upgrade' to initiate upgrade
4.  Interrupt upgrade via ctrl-c
5.  Repeat upgrade/interrupt process


Actual results:
Subsequent attempts to upgrade are met with errors about 'Invalid compressed data' or 'Corrupted file object'

Expected results:
Upgrade should be successful


Additional info:




-bash-4.2# atomic host status
  TIMESTAMP (UTC)         VERSION   ID             OSNAME               REFSPEC                                                        
* 2015-02-19 20:26:26     7.1.0     5799825b36     rhel-atomic-host     rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard     
-bash-4.2# atomic host upgrade
Updating from: rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard

Receiving objects: 36% (136/376) 678.9 kB/s 8.1 MB           ^C
-bash-4.2# 
-bash-4.2# atomic host upgrade
Updating from: rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard

Receiving objects: 26% (62/230) 3.6 MB/s 32.5 MB^C
-bash-4.2# atomic host status                                                                                                                                                                                      
  TIMESTAMP (UTC)         VERSION   ID             OSNAME               REFSPEC                                                        
* 2015-02-19 20:26:26     7.1.0     5799825b36     rhel-atomic-host     rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard     
-bash-4.2# atomic host upgrade
Updating from: rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard

Receiving objects: 28% (47/165) 5.5 MB/s 27.3 MB
error: Invalid compressed data
-bash-4.2# ^C
-bash-4.2# atomic host status                                                                                                                                                                                      
  TIMESTAMP (UTC)         VERSION   ID             OSNAME               REFSPEC                                                        
* 2015-02-19 20:26:26     7.1.0     5799825b36     rhel-atomic-host     rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard     
-bash-4.2# atomic host upgrade
Updating from: rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard

Receiving objects: 36% (43/117) 5.8 MB/s 34.6 MB^C
-bash-4.2# atomic host status                                                                                                                                                                                      
  TIMESTAMP (UTC)         VERSION   ID             OSNAME               REFSPEC                                                        
* 2015-02-19 20:26:26     7.1.0     5799825b36     rhel-atomic-host     rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard     
-bash-4.2# atomic host upgrade
Updating from: rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard


error: Corrupted file object 2353a510588164315e981e15492bf56d6b3c135bba3266a8663a684fadac2caf (actual checksum is fc29f67122ff1086ec755d7e7d0f02d4236c64e5c5242fbb41f523c37c90ae31)
-bash-4.2# atomic host upgrade
Updating from: rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard

Receiving objects: 13% (8/58) 11.9 MB/s 23.8 MB
error: Invalid compressed data
-bash-4.2# atomic host upgrade
Updating from: rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard

Receiving objects: 8% (4/45) 24.6 MB/s 24.6 MB
error: Invalid compressed data
Comment 1 Colin Walters 2015-03-30 15:51:21 UTC 
If this does occur, a workaround is:

    rm /ostree/repo/tmp/* -rf
Comment 2 Micah Abbott 2015-03-30 15:58:16 UTC 
It appears that if you keep retrying the upgrade, the process will eventually complete.

In the session below, it took 8 retries.


-bash-4.2# atomic host status
  TIMESTAMP (UTC)         VERSION   ID             OSNAME               REFSPEC                                                        
* 2015-02-19 20:26:26     7.1.0     5799825b36     rhel-atomic-host     rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard     
-bash-4.2# atomic host upgrade
Updating from: rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard

Receiving metadata objects: 79/(estimating) 1.4 MB/s 12.8 MB^C 
-bash-4.2# atomic host status                                                                                                                                                                                      
  TIMESTAMP (UTC)         VERSION   ID             OSNAME               REFSPEC                                                        
* 2015-02-19 20:26:26     7.1.0     5799825b36     rhel-atomic-host     rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard     
-bash-4.2# atomic host upgrade
Updating from: rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard

Receiving objects: 36% (102/278) 2.7 MB/s 23.9 MB^C        
-bash-4.2# atomic host status                                                                                                                                                                                      
  TIMESTAMP (UTC)         VERSION   ID             OSNAME               REFSPEC                                                        
* 2015-02-19 20:26:26     7.1.0     5799825b36     rhel-atomic-host     rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard     
-bash-4.2# atomic host upgrade
Updating from: rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard

Receiving objects: 10% (19/174) 8.0 MB/s 15.9 MB
error: Corrupted file object 76f3a8b44a4fbb50926587d275ceb3fca4e11be547d0769601a4ee13ba05c66d (actual checksum is 973cbf44a2b806ab94010dbdeddbb41b7bcab63e336a1b88f238677ee50970e4)
-bash-4.2# atomic host upgrade
Updating from: rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard

Receiving objects: 14% (22/151) 7.3 MB/s 21.8 MB
error: Invalid compressed data
-bash-4.2# atomic host upgrade
Updating from: rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard

Receiving objects: 13% (18/129) 10.4 MB/s 20.8 MB
error: Corrupted file object 9ba1d5e663eec196ce24dea92ae270b438f1c45b8b7f16c9c9a05b537b87427d (actual checksum is a503f0d5dc5a6d4407a88de837432ccff319dee859a438394ca5f46b82697392)
-bash-4.2# atomic host upgrade
Updating from: rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard

Receiving objects: 16% (18/106) 11.7 MB/s 23.3 MB
error: Corrupted file object 8d8e5e62c6d01ddb30e4c45684b4d0645dded0884698a3a7450ec2a47254b880 (actual checksum is 441af61594097852f01d0589eeeefa250813b3b7b9e5743cd1557f552c789232)
-bash-4.2# atomic host upgrade
Updating from: rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard

Receiving objects: 17% (15/84) 15.2 MB/s 30.4 MB
error: Corrupted file object 76f3a8b44a4fbb50926587d275ceb3fca4e11be547d0769601a4ee13ba05c66d (actual checksum is 4a13707b8d1d2b11ba8a36a8d58f5120200908360abcf4c16341209960f1cb9c)
-bash-4.2# atomic host upgrade
Updating from: rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard

Receiving objects: 30% (18/60) 15.7 MB/s 31.4 MB
error: Invalid compressed data
-bash-4.2# atomic host upgrade
Updating from: rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard

Receiving objects: 91% (34/37) 8.0 MB/s 55.9 MB 
error: Invalid compressed data
-bash-4.2# atomic host upgrade
Updating from: rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard

Receiving objects: 33% (1/3) 3.6 MB/s 118.3 MB
error: Invalid compressed data
-bash-4.2# atomic host upgrade
Updating from: rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard

0 metadata, 2 content objects fetched; 64852 KiB transferred in 34 seconds
Copying /etc changes: 27 modified, 4 removed, 57 added
Transaction complete; bootconfig swap: yes deployment count change: 1
Changed:
  bind-libs-32:9.9.4-18.el7_1.1.x86_64
  bind-libs-lite-32:9.9.4-18.el7_1.1.x86_64
  bind-license-32:9.9.4-18.el7_1.1.noarch
  bind-utils-32:9.9.4-18.el7_1.1.x86_64
  binutils-2.23.52.0.1-30.el7_1.1.x86_64
  dnsmasq-2.66-13.el7_1.x86_64
  docker-1.5.0-27.el7.x86_64
  docker-python-1.0.0-21.el7.x86_64
  dracut-033-241.el7_1.1.x86_64
  dracut-network-033-241.el7_1.1.x86_64
  etcd-0.4.6-0.13.el7.x86_64
  flannel-0.2.0-7.el7.x86_64
  freetype-2.4.11-10.el7_1.1.x86_64
  libgudev1-208-20.el7_1.2.x86_64
  libipa_hbac-1.12.2-58.el7_1.6.x86_64
  libsss_idmap-1.12.2-58.el7_1.6.x86_64
  libsss_nss_idmap-1.12.2-58.el7_1.6.x86_64
  mdadm-3.3.2-2.el7_1.1.x86_64
  openssh-6.6.1p1-12.el7_1.x86_64
  openssh-clients-6.6.1p1-12.el7_1.x86_64
  openssh-server-6.6.1p1-12.el7_1.x86_64
  openssl-1:1.0.1e-42.el7_1.4.x86_64
  openssl-libs-1:1.0.1e-42.el7_1.4.x86_64
  python-sssdconfig-1.12.2-58.el7_1.6.noarch
  python-websocket-client-0.14.1-64.el7.noarch
  selinux-policy-3.13.1-23.el7_1.3.noarch
  selinux-policy-targeted-3.13.1-23.el7_1.3.noarch
  sssd-1.12.2-58.el7_1.6.x86_64
  sssd-ad-1.12.2-58.el7_1.6.x86_64
  sssd-client-1.12.2-58.el7_1.6.x86_64
  sssd-common-1.12.2-58.el7_1.6.x86_64
  sssd-common-pac-1.12.2-58.el7_1.6.x86_64
  sssd-ipa-1.12.2-58.el7_1.6.x86_64
  sssd-krb5-1.12.2-58.el7_1.6.x86_64
  sssd-krb5-common-1.12.2-58.el7_1.6.x86_64
  sssd-ldap-1.12.2-58.el7_1.6.x86_64
  sssd-proxy-1.12.2-58.el7_1.6.x86_64
  systemd-208-20.el7_1.2.x86_64
  systemd-libs-208-20.el7_1.2.x86_64
  systemd-sysv-208-20.el7_1.2.x86_64
  tzdata-2015b-1.el7.noarch
Added:
  atomic-0-0.8.git4ff7dbd.el7.x86_64
Upgrade prepared for next boot; run "systemctl reboot" to start a reboot
-bash-4.2# atomic host status
  TIMESTAMP (UTC)         VERSION   ID             OSNAME               REFSPEC                                                        
  2015-03-25 19:26:39     7.1.1     cceedd50f3     rhel-atomic-host     rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard     
* 2015-02-19 20:26:26     7.1.0     5799825b36     rhel-atomic-host     rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard
Comment 10 Colin Walters 2015-04-02 15:10:08 UTC 
*** Bug 1207650 has been marked as a duplicate of this bug. ***
Comment 12 Colin Walters 2015-04-04 21:00:30 UTC 
https://github.com/GNOME/ostree/pull/84
Comment 13 Colin Walters 2015-04-06 23:51:16 UTC 
https://github.com/GNOME/ostree/pull/85 is also related to this.
Comment 14 Colin Walters 2015-05-15 16:16:39 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2015:0992