Bug 1417534 - unmodified configuration files should be updated during update.
Summary: unmodified configuration files should be updated during update.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-node-ng
Version: 4.0.6
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ovirt-4.1.1
: ---
Assignee: Ryan Barry
QA Contact: Huijuan Zhao
URL:
Whiteboard:
: 1419435 1425194 (view as bug list)
Depends On:
Blocks: 1418179 1421098
TreeView+ depends on / blocked
 
Reported: 2017-01-30 06:56 UTC by Germano Veit Michel
Modified: 2017-04-20 19:02 UTC (History)
16 users (show)

Fixed In Version: imgbased-0.9.17-0.1.el7ev
Doc Type: Bug Fix
Doc Text:
Previously, imgbased copied /etc from old layers into new layers in order to keep configuration changes between upgrades. This behavior differed from that of RPM, in that unmodified configuration files would be preserved across imgbased upgrades rather than being replaced. In this release, imgbased compares the sums of files to the originals that are kept per-layer in /usr/share/factory/etc. As a result, the unmodified configuration files are now handled as required.
Clone Of:
: 1418179 (view as bug list)
Environment:
Last Closed: 2017-04-20 19:02:41 UTC
oVirt Team: Node
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2017:1114 0 normal SHIPPED_LIVE redhat-virtualization-host bug fix and enhancement update 2017-04-20 22:57:46 UTC
oVirt gerrit 71403 0 'None' 'MERGED' 'osupdater: be smarter about migrating /etc' 2019-11-22 11:22:08 UTC

Description Germano Veit Michel 2017-01-30 06:56:52 UTC 
Description of problem:

As we have seen on bugzilla #1416278, many configurations files are not updated when a new image is installed.

How reproducible:
100%

Steps to Reproduce:
1. Install RHVH 4.0.4, check timestamps/contents of configuration files
   * /etc/vdsm for example
2. Attach entitlements, update to 4.0-20170104.1 and reboot
3. Compare timestamps/contents of configuration files

A notable file between these two versions is logger.conf, 4.0.6 and 4.0.4 contents are different, but the upgraded node doesn't have the newer configurations.

Pretty much every config file is with the older image timestamp after upgrade.

# ll /etc/vdsm/
total 32
-rw-r--r--. 1 root root  1747 Sep  7 21:14 logger.conf
drwxr-xr-x. 2 root root    17 Oct 13 00:02 logrotate
-rw-r--r--. 1 root root  3161 Sep  7 21:14 mom.conf
drwxr-xr-x. 2 root root  4096 Oct 13 00:02 mom.d
-rw-r--r--. 1 root root   619 Sep  7 21:14 svdsm.logger.conf
-rw-r--r--. 1 root root 12725 Sep  7 21:14 vdsm.conf
drwxr-xr-x. 2 root root     6 Sep  7 21:14 vdsm.conf.d

Whereas in a fresh 4.0-20170104.1 install:

# ll /etc/vdsm/
total 40
-rw-r--r--. 1 root root  1825 Dec 14 20:52 logger.conf
drwxr-xr-x. 2 root root  4096 Jan  5 01:33 logrotate
-rw-r--r--. 1 root root  3166 Dec 14 20:52 mom.conf
drwxr-xr-x. 2 root root  4096 Jan  5 01:33 mom.d
-rw-r--r--. 1 root root   619 Dec 14 20:52 svdsm.logger.conf
-rw-r--r--. 1 root root 12655 Dec 14 20:52 vdsm.conf
drwxr-xr-x. 2 root root  4096 Dec 14 20:52 vdsm.conf.d

Version-Release number of selected component (if applicable):
4.0-20170104.1

Expected results:
Files should be rolled forward to the latest version.

Additional info:
Files were not manually modified
Comment 1 Ryan Barry 2017-01-30 13:43:56 UTC 
I just want to point out a potential risk here --

Systems upgraded from:

4.0.4 -> 4.0.6 -> some other version

Will not carry this change across. In a manner similar to the way RPM treats .rpmnew files, we can compare against a "good" file (in /usr/share/factory, which is put aside when the image is built). However, a 4.0.6 system will see logger.conf as already modified, and any upgrades from that image will keep holding onto the old logger.conf.

We can also check "rpm -V", but will see the same result -- logger.conf already differs.

I'll try to think of a way to work around this, but we'll definitely need a release note.
Comment 4 Ryan Barry 2017-01-31 13:33:42 UTC 
Example:

* User installs RHV 4.0.4
* User upgrades to 4.0.6 (with a new logger.conf)
* The upgrade copies all of /etc
* logger.conf now differs from what is in the rpmdb and /usr/share/factory/etc
* User upgrades to 4.0.6-1 or 4.0.7
* logger.conf (and potentially other files) differ from what's in the rpmdb because they were blindly copied, even though they were never modified, and will not be copied by the new patch.
* In this case, relying on the rpmdb is not reliable, because the file is modified.

The difficulty is that it was modified by imgbased copying the file, but differentiating this from an intentional change programatically is almost impossible. We have the option of going back through the rpmdb on old imgbased layers to see if the files match the rpmdb on any of those, but this seems risky, and it adds a lot of runtime to the upgrade.

I'm still investigating to see whether the tradeoff of going through old layers is worth the additional runtime/complexity.
Comment 7 Ryan Barry 2017-02-06 14:30:52 UTC 
*** Bug 1419435 has been marked as a duplicate of this bug. ***
Comment 9 Ryan Barry 2017-02-28 14:52:23 UTC 
*** Bug 1425194 has been marked as a duplicate of this bug. ***
Comment 10 Huijuan Zhao 2017-03-13 07:00:42 UTC 
The contents of the actual files in /etc/vdsm/ are rolled forward to the latest version. But timestamps are not same.

Test version:
From:
RHVH-4.0-20160919.1-RHVH-x86_64-dvd1.iso
To:
redhat-virtualization-host-4.1-20170308.1.x86_64
imgbased-0.9.17-0.1.el7ev.noarch

Test steps:
1. Install RHVH-4.0-20160919.1-RHVH-x86_64-dvd1.iso, check timestamps/contents of configuration files
   * /etc/vdsm for example
2. Login RHVH and setup local repos, update to redhat-virtualization-host-4.1-20170308.1 and reboot
3. Check timestamps/contents of configuration files
   # ls -lh /etc/vdsm
4. Clean install redhat-virtualization-host-4.1-20170308.1, check timestamps/contents of configuration files
   # ls -lh /etc/vdsm

Test results:
diff the files in /etc/vdsm/, step 3 and step4 are the same, step3 and step1 are different. So the unmodified content is rolled forward to the latest version during upgrade.

But timestamps are not same between step 3 and step4:
After step 1:
# ls -lh /etc/vdsm
total 32K
-rw-r--r--. 1 root root 1.8K Sep  7  2016 logger.conf
drwxr-xr-x. 2 root root   17 Sep 19 20:16 logrotate
-rw-r--r--. 1 root root 3.1K Sep  7  2016 mom.conf
drwxr-xr-x. 2 root root 4.0K Sep 19 20:16 mom.d
-rw-r--r--. 1 root root  619 Sep  7  2016 svdsm.logger.conf
-rw-r--r--. 1 root root  13K Sep  7  2016 vdsm.conf
drwxr-xr-x. 2 root root    6 Sep  7  2016 vdsm.conf.d

After step3:
# ls -lh /etc/vdsm/
total 32K
-rw-r--r--. 1 root root 1.6K Mar  2 13:37 logger.conf
drwxr-xr-x. 2 root root   17 Mar  9 02:35 logrotate
-rw-r--r--. 1 root root 3.1K Mar  2 13:37 mom.conf
drwxr-xr-x. 2 root root 4.0K Mar  9 02:35 mom.d
-rw-r--r--. 1 root root  619 Mar  2 13:37 svdsm.logger.conf
-rw-r--r--. 1 root root  14K Mar  2 13:37 vdsm.conf
drwxr-xr-x. 2 root root    6 Mar  2 13:37 vdsm.conf.d

After step4:
# ls -lh /etc/vdsm
total 40K
-rw-r--r--. 1 root root 1.6K Mar  2 08:37 logger.conf
drwxr-xr-x. 2 root root 4.0K Mar  8 21:35 logrotate
-rw-r--r--. 1 root root 3.1K Mar  2 08:37 mom.conf
drwxr-xr-x. 2 root root 4.0K Mar  8 21:35 mom.d
-rw-r--r--. 1 root root  619 Mar  2 08:37 svdsm.logger.conf
-rw-r--r--. 1 root root  14K Mar  2 08:37 vdsm.conf
drwxr-xr-x. 2 root root 4.0K Mar  2 08:37 vdsm.conf.d

Ryan, should I verify this bug, and report a new bug to track timestamps issue?
Thanks!
Comment 11 Ryan Barry 2017-03-13 18:14:17 UTC 
These times look similar enough that I'd guess it's a timezone issue.

What does timedatectl show before and after upgrading?
Comment 12 Ryan Barry 2017-03-13 20:22:39 UTC 
I just tried updating:

2016-0916.0
2016-1116.1
2017-0308.1

[root@localhost ~]# imgbase layout
timedatrhvh-4.0-0.20161116.0
 +- rhvh-4.0-0.20161116.0+1
rhvh-4.1-0.20170309.0
 +- rhvh-4.1-0.20170309.0+1
[root@localhost ~]# timedatectl
      Local time: Mon 2017-03-13 13:21:53 MST
  Universal time: Mon 2017-03-13 20:21:53 UTC
        RTC time: Mon 2017-03-13 20:21:53
       Time zone: America/Phoenix (MST, -0700)
     NTP enabled: yes
NTP synchronized: yes
 RTC in local TZ: no
      DST active: n/a
[root@localhost ~]# ls -l /etc/vdsm
total 28
-rw-r--r--. 1 root root  1551 Mar  2 06:37 logger.conf
drwxr-xr-x. 2 root root    18 Mar  8 19:35 logrotate
-rw-r--r--. 1 root root  3166 Mar  2 06:37 mom.conf
drwxr-xr-x. 2 root root   154 Mar  8 19:35 mom.d
-rw-r--r--. 1 root root   619 Mar  2 06:37 svdsm.logger.conf
-rw-r--r--. 1 root root 14240 Mar  2 06:37 vdsm.conf
drwxr-xr-x. 2 root root     6 Mar  2 06:37 vdsm.conf.d


Clean install:

[root@localhost ~]# imgbase layout
rhvh-4.1-0.20170308.0
 +- rhvh-4.1-0.20170308.0+1
[root@localhost ~]# timedatectl
      Local time: Mon 2017-03-13 13:19:47 MST
  Universal time: Mon 2017-03-13 20:19:47 UTC
        RTC time: Mon 2017-03-13 20:19:46
       Time zone: America/Phoenix (MST, -0700)
     NTP enabled: yes
NTP synchronized: yes
 RTC in local TZ: no
      DST active: n/a
[root@localhost ~]# ls -l /etc/vdsm
total 32
-rw-r--r--. 1 root root  1551 Mar  2 06:37 logger.conf
drwxr-xr-x. 2 root root    17 Mar  8 13:21 logrotate
-rw-r--r--. 1 root root  3166 Mar  2 06:37 mom.conf
drwxr-xr-x. 2 root root  4096 Mar  8 13:21 mom.d
-rw-r--r--. 1 root root   619 Mar  2 06:37 svdsm.logger.conf
-rw-r--r--. 1 root root 14240 Mar  2 06:37 vdsm.conf
drwxr-xr-x. 2 root root     6 Mar  2 06:37 vdsm.conf.d
Comment 13 Huijuan Zhao 2017-03-14 09:08:05 UTC 
Yes, I think you are right, this is a timezone issue. When I setup Time zone: America/New_York, the timestamps are the same between step3 and step4 in comment 10.

Test version and steps are same as comment 10.

1. After step3(upgrade from 4.0-20160919.0 to 4.1-20170308.1):
[root@ibm-x3650m5-04 ~]# imgbase w
[INFO] You are on rhvh-4.1-0.20170309.0+1
[root@ibm-x3650m5-04 ~]# imgbase layout
rhvh-4.0-0.20160919.0
 +- rhvh-4.0-0.20160919.0+1
rhvh-4.1-0.20170309.0
 +- rhvh-4.1-0.20170309.0+1
[root@ibm-x3650m5-04 ~]# timedatectl
      Local time: Tue 2017-03-14 05:00:10 EDT
  Universal time: Tue 2017-03-14 09:00:10 UTC
        RTC time: Tue 2017-03-14 05:00:10
       Time zone: America/New_York (EDT, -0400)
     NTP enabled: yes
NTP synchronized: yes
 RTC in local TZ: yes
      DST active: yes
 Last DST change: DST began at
                  Sun 2017-03-12 01:59:59 EST
                  Sun 2017-03-12 03:00:00 EDT
 Next DST change: DST ends (the clock jumps one hour backwards) at
                  Sun 2017-11-05 01:59:59 EDT
                  Sun 2017-11-05 01:00:00 EST
[root@ibm-x3650m5-04 ~]# ls -l /etc/vdsm
total 32
-rw-r--r--. 1 root root  1551 Mar  2 08:37 logger.conf
drwxr-xr-x. 2 root root    17 Mar  8 21:35 logrotate
-rw-r--r--. 1 root root  3166 Mar  2 08:37 mom.conf
drwxr-xr-x. 2 root root  4096 Mar  8 21:35 mom.d
-rw-r--r--. 1 root root   619 Mar  2 08:37 svdsm.logger.conf
-rw-r--r--. 1 root root 14240 Mar  2 08:37 vdsm.conf
drwxr-xr-x. 2 root root     6 Mar  2 08:37 vdsm.conf.d

2. After step4(clean install 4.1-20170308.1):
[root@ibm-x3650m5-04 ~]# imgbase w
[INFO] You are on rhvh-4.1-0.20170309.0+1
[root@ibm-x3650m5-04 ~]# imgbase layout
rhvh-4.1-0.20170309.0
 +- rhvh-4.1-0.20170309.0+1
[root@ibm-x3650m5-04 ~]# timedatectl
      Local time: Tue 2017-03-14 04:02:40 EDT
  Universal time: Tue 2017-03-14 08:02:40 UTC
        RTC time: Tue 2017-03-14 08:02:40
       Time zone: America/New_York (EDT, -0400)
     NTP enabled: yes
NTP synchronized: yes
 RTC in local TZ: no
      DST active: yes
 Last DST change: DST began at
                  Sun 2017-03-12 01:59:59 EST
                  Sun 2017-03-12 03:00:00 EDT
 Next DST change: DST ends (the clock jumps one hour backwards) at
                  Sun 2017-11-05 01:59:59 EDT
                  Sun 2017-11-05 01:00:00 EST
[root@ibm-x3650m5-04 ~]# ls -l /etc/vdsm
total 40
-rw-r--r--. 1 root root  1551 Mar  2 08:37 logger.conf
drwxr-xr-x. 2 root root  4096 Mar  8 21:35 logrotate
-rw-r--r--. 1 root root  3166 Mar  2 08:37 mom.conf
drwxr-xr-x. 2 root root  4096 Mar  8 21:35 mom.d
-rw-r--r--. 1 root root   619 Mar  2 08:37 svdsm.logger.conf
-rw-r--r--. 1 root root 14240 Mar  2 08:37 vdsm.conf
drwxr-xr-x. 2 root root  4096 Mar  2 08:37 vdsm.conf.d

So this bug is fixed in imgbased-0.9.17-0.1.el7ev.noarch, change the status to VERIFIED.
Comment 14 errata-xmlrpc 2017-04-20 19:02:41 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1114
Note You need to log in before you can comment on or make changes to this bug.