By default, imgbased tries to keep 2 bases of the image by deleting the oldest base However, after updating the node twice, imgbase fails to delete the oldest base and throws an exception on missing _filter_overflow in GarbageCollector.
I can not reproduce this issue, did I miss some steps? Thanks. Test version: # imgbase layout rhvh-4.0-0.20160817.0 +- rhvh-4.0-0.20160817.0+1 rhvh-4.0-0.20161116.0 +- rhvh-4.0-0.20161116.0+1 rhvh-4.1-0.20170208.0 +- rhvh-4.1-0.20170208.0+1 Test steps: 1. Install build1 redhat-virtualization-host-4.0-20160817.0 2. Add rhvh4.0 to rhvm4.0, add NFS storage to rhvh in rhvm side 3. Download update rpm, update rhvh to rhvh-4.0-0.20161116.0 # yum install redhat-virtualization-host-image-update-4.0-20161116.1.el7_3.noarch.rpm 4. Reboot and login new build2 rhvh-4.0-0.20161116.0 Download update rpm, update rhvh to rhvh-4.1-0.20170208.0 # yum install redhat-virtualization-host-image-update-4.1-20170208.0.el7_3.noarch.rpm 5. Reboot and login new build3 rhvh-4.1-0.20170208.0 6. Delete oldest imgbase rhvh-4.0-0.20160817.0 # lvremove /dev/rhvh_ibm-x3650m5-04/rhvh-4.0-0.20160817.0+1 Logical volume "rhvh-4.0-0.20160817.0+1" successfully removed # lvremove /dev/rhvh_ibm-x3650m5-04/rhvh-4.0-0.20160817.0 Logical volume "rhvh-4.0-0.20160817.0" successfully removed Test results: After step6, delete oldest imgbase successfully. # imgbase layout rhvh-4.0-0.20161116.0 +- rhvh-4.0-0.20161116.0+1 rhvh-4.1-0.20170208.0 +- rhvh-4.1-0.20170208.0+1 # lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert pool00 rhvh_ibm-x3650m5-04 twi-aotz-- 525.38g 1.20 1.13 rhvh-4.0-0.20161116.0 rhvh_ibm-x3650m5-04 Vri---tz-k 4.06g pool00 rhvh-4.0-0.20161116.0+1 rhvh_ibm-x3650m5-04 Vwi---tz-- 4.06g pool00 rhvh-4.0-0.20161116.0 rhvh-4.1-0.20170208.0 rhvh_ibm-x3650m5-04 Vri---tz-k 4.06g pool00 rhvh-4.1-0.20170208.0+1 rhvh_ibm-x3650m5-04 Vwi-aotz-- 4.06g pool00 rhvh-4.1-0.20170208.0 38.23 root rhvh_ibm-x3650m5-04 Vwi---tz-- 510.34g pool00 swap rhvh_ibm-x3650m5-04 -wi-ao---- 15.44g var rhvh_ibm-x3650m5-04 Vwi-aotz-- 15.00g pool00 1.34
I inst(In reply to Huijuan Zhao from comment #1) > I can not reproduce this issue, did I miss some steps? > Thanks. > > Test version: > # imgbase layout > rhvh-4.0-0.20160817.0 > +- rhvh-4.0-0.20160817.0+1 > rhvh-4.0-0.20161116.0 > +- rhvh-4.0-0.20161116.0+1 > rhvh-4.1-0.20170208.0 > +- rhvh-4.1-0.20170208.0+1 > > Test steps: > 1. Install build1 redhat-virtualization-host-4.0-20160817.0 > 2. Add rhvh4.0 to rhvm4.0, add NFS storage to rhvh in rhvm side > 3. Download update rpm, update rhvh to rhvh-4.0-0.20161116.0 > # yum install > redhat-virtualization-host-image-update-4.0-20161116.1.el7_3.noarch.rpm > 4. Reboot and login new build2 rhvh-4.0-0.20161116.0 > Download update rpm, update rhvh to rhvh-4.1-0.20170208.0 > # yum install > redhat-virtualization-host-image-update-4.1-20170208.0.el7_3.noarch.rpm > 5. Reboot and login new build3 rhvh-4.1-0.20170208.0 > 6. Delete oldest imgbase rhvh-4.0-0.20160817.0 > # lvremove /dev/rhvh_ibm-x3650m5-04/rhvh-4.0-0.20160817.0+1 > Logical volume "rhvh-4.0-0.20160817.0+1" successfully removed > > # lvremove /dev/rhvh_ibm-x3650m5-04/rhvh-4.0-0.20160817.0 > Logical volume "rhvh-4.0-0.20160817.0" successfully removed > > > Test results: > After step6, delete oldest imgbase successfully. > > # imgbase layout > rhvh-4.0-0.20161116.0 > +- rhvh-4.0-0.20161116.0+1 > rhvh-4.1-0.20170208.0 > +- rhvh-4.1-0.20170208.0+1 > > # lvs > LV VG Attr LSize Pool > Origin Data% Meta% Move Log Cpy%Sync Convert > pool00 rhvh_ibm-x3650m5-04 twi-aotz-- 525.38g > 1.20 1.13 > rhvh-4.0-0.20161116.0 rhvh_ibm-x3650m5-04 Vri---tz-k 4.06g pool00 > > rhvh-4.0-0.20161116.0+1 rhvh_ibm-x3650m5-04 Vwi---tz-- 4.06g pool00 > rhvh-4.0-0.20161116.0 > rhvh-4.1-0.20170208.0 rhvh_ibm-x3650m5-04 Vri---tz-k 4.06g pool00 > > rhvh-4.1-0.20170208.0+1 rhvh_ibm-x3650m5-04 Vwi-aotz-- 4.06g pool00 > rhvh-4.1-0.20170208.0 38.23 > root rhvh_ibm-x3650m5-04 Vwi---tz-- 510.34g pool00 > > swap rhvh_ibm-x3650m5-04 -wi-ao---- 15.44g > > var rhvh_ibm-x3650m5-04 Vwi-aotz-- 15.00g pool00 > 1.34 This error should happen in step 4. You have 3 snapshots, while there should be only 2. Also, you shouldn't lvremove manually rhvh-4.0-0.20160817.0, but instead use: imgbase base --remove rhvh-4.0-0.20160817.0 which didn't work previous to this fix.
*** Bug 1421687 has been marked as a duplicate of this bug. ***
(In reply to Yuval Turgeman from comment #2) > This error should happen in step 4. You have 3 snapshots, while there > should be only 2. Also, you shouldn't lvremove manually > rhvh-4.0-0.20160817.0, but instead use: > > imgbase base --remove rhvh-4.0-0.20160817.0 > > which didn't work previous to this fix. Still can not completely reproduce this issue. Could you please tell which 3 builds did you use? Or your testing steps? I tested with another older rhvh 4.1 build rhvh-4.1-0.20170202.0: 1. In step4, there are still 3 snapshots # imgbase layout rhvh-4.0-0.20160817.0 +- rhvh-4.0-0.20160817.0+1 rhvh-4.0-0.20161116.0 +- rhvh-4.0-0.20161116.0+1 rhvh-4.1-0.20170202.0 +- rhvh-4.1-0.20170202.0+1 [root@dhcp-10-16 home]# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert pool00 rhvh_dhcp-10-16 twi-aotz-- 906.71g 1.03 1.12 rhvh-4.0-0.20160817.0 rhvh_dhcp-10-16 Vwi---tz-k 891.68g pool00 root rhvh-4.0-0.20160817.0+1 rhvh_dhcp-10-16 Vwi-a-tz-- 891.68g pool00 rhvh-4.0-0.20160817.0 0.36 rhvh-4.0-0.20161116.0 rhvh_dhcp-10-16 Vri---tz-k 4.06g pool00 rhvh-4.0-0.20161116.0+1 rhvh_dhcp-10-16 Vwi-aotz-- 4.06g pool00 rhvh-4.0-0.20161116.0 61.36 rhvh-4.1-0.20170202.0 rhvh_dhcp-10-16 Vri-a-tz-k 4.06g pool00 36.47 rhvh-4.1-0.20170202.0+1 rhvh_dhcp-10-16 Vwi-a-tz-- 4.06g pool00 rhvh-4.1-0.20170202.0 47.55 root rhvh_dhcp-10-16 Vwi-a-tz-- 891.68g pool00 0.25 swap rhvh_dhcp-10-16 -wi-ao---- 7.75g var rhvh_dhcp-10-16 Vwi-aotz-- 15.00g pool00 1.19 2. In step6, # imgbase base --remove rhvh-4.0-0.20160817.0 Nothing output after run this delete command. But "#imgbase layout" is failed after then: # imgbase layout Traceback (most recent call last): File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/usr/lib/python2.7/site-packages/imgbased/__main__.py", line 51, in <module> CliApplication() File "/usr/lib/python2.7/site-packages/imgbased/__init__.py", line 82, in CliApplication app.hooks.emit("post-arg-parse", args) File "/usr/lib/python2.7/site-packages/imgbased/hooks.py", line 120, in emit cb(self.context, *args) File "/usr/lib/python2.7/site-packages/imgbased/plugins/core.py", line 172, in post_argparse print(layout.dumps()) File "/usr/lib/python2.7/site-packages/imgbased/plugins/core.py", line 200, in dumps return self.app.imgbase.layout() File "/usr/lib/python2.7/site-packages/imgbased/imgbase.py", line 148, in layout return self.naming.layout() File "/usr/lib/python2.7/site-packages/imgbased/naming.py", line 109, in layout tree = self.tree(lvs) File "/usr/lib/python2.7/site-packages/imgbased/naming.py", line 224, in tree bases[img.base.nvr].layers.append(img) KeyError: <NVR rhvh-4.0-0.20160817.0 />
*** Bug 1421732 has been marked as a duplicate of this bug. ***
This is exactly the problem, there should be only 2 bases after the second update, and there are 3. Could you please attach /tmp/imgbased.log after the second update (after step 4, and before the reboot) ? This fix addresses the imgbase base --remove as well since they are caused by the same bug.
Yuval, thanks for your explanation. I thought we can support many upgrades which is just limited by disk size. So it means we will keep only two latest layers after upgrade in future(so this bug is RFE)? Please refer to attachment for imgbased.log
Created attachment 1250159 [details] Comment 7 : imgbased.log
(In reply to Huijuan Zhao from comment #7) > Yuval, thanks for your explanation. I thought we can support many upgrades > which is just limited by disk size. > So it means we will keep only two latest layers after upgrade in future(so > this bug is RFE)? > > Please refer to attachment for imgbased.log Yes today we are supposed to keep the two latest bases/layers. Please take a look at the end of the imgbased.log you attached, that's the error this bug fixes as well as imgbase base --remove :)
(In reply to Yuval Turgeman from comment #9) > Yes today we are supposed to keep the two latest bases/layers. Please take > a look at the end of the imgbased.log you attached, that's the error this > bug fixes as well as imgbase base --remove :) Yes, so QE can reproduce this issue. Thanks Yuval:)
Still encounter issues in redhat-virtualization-host-4.1-20170222.0. Test version: Build1: redhat-virtualization-host-4.0-20160919.0 Build2: redhat-virtualization-host-4.0-20161116.0 Build3: redhat-virtualization-host-4.1-20170222.0 # imgbase layout rhvh-4.0-0.20160919.0 +- rhvh-4.0-0.20160919.0+1 rhvh-4.0-0.20161116.0 +- rhvh-4.0-0.20161116.0+1 rhvh-4.1-0.20170223.0 +- rhvh-4.1-0.20170223.0+1 Test steps: 1. Install build1 redhat-virtualization-host-4.0-20160919.0 2. Reboot and login build1, download redhat-virtualization-host-image-update-4.0-20161116.1.el7_3.noarch.rpm and update: # yum install redhat-virtualization-host-image-update-4.0-20161116.1.el7_3.noarch.rpm 3. Reboot and login build2 rhvh-4.0-0.20161116.0, setup local repos, update to build3 redhat-virtualization-host-4.1-20170222.0: # yum update 4. Reboot and focus on boot entry Actual results: 1. In step3, there is warning info during update "#yum update". After update,there are 3 bases. # yum update Loaded plugins: imgbased-warning, product-id, search-disabled-repos, subscription-manager This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. Warning: yum operations are not persisted across upgrades! Repo rhel-7-server-rhvh-4-source-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877.pem Repo rhel-7-server-rhvh-4-source-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877-key.pem Repo rhel-7-server-rhvh-4-beta-debug-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877.pem Repo rhel-7-server-rhvh-4-beta-debug-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877-key.pem Repo rhel-7-server-rhvh-4-debug-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877.pem Repo rhel-7-server-rhvh-4-debug-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877-key.pem Repo rhel-7-server-rhvh-4-beta-source-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877.pem Repo rhel-7-server-rhvh-4-beta-source-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877-key.pem Repo rhel-7-server-rhvh-4-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877.pem Repo rhel-7-server-rhvh-4-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877-key.pem Repo rhel-7-server-rhvh-4-beta-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877.pem Repo rhel-7-server-rhvh-4-beta-rpms forced skip_if_unavailable=True due to: /etc/pki/entitlement/5858345830546115877-key.pem rhel-7-server-rhvh-4-beta-debug-rpms | 2.9 kB 00:00:00 rhel-7-server-rhvh-4-beta-rpms | 2.9 kB 00:00:00 rhel-7-server-rhvh-4-beta-source-rpms | 2.9 kB 00:00:00 rhel-7-server-rhvh-4-debug-rpms | 2.9 kB 00:00:00 rhel-7-server-rhvh-4-rpms | 2.9 kB 00:00:00 rhel-7-server-rhvh-4-source-rpms | 2.9 kB 00:00:00 (1/6): rhel-7-server-rhvh-4-beta-debug-rpms/primary_db | 1.7 kB 00:00:00 (2/6): rhel-7-server-rhvh-4-source-rpms/primary_db | 4.1 kB 00:00:00 (3/6): rhel-7-server-rhvh-4-beta-rpms/primary_db | 9.3 kB 00:00:00 (4/6): rhel-7-server-rhvh-4-debug-rpms/primary_db | 1.7 kB 00:00:00 (5/6): rhel-7-server-rhvh-4-rpms/primary_db | 9.3 kB 00:00:00 (6/6): rhel-7-server-rhvh-4-beta-source-rpms/primary_db | 4.1 kB 00:00:00 Resolving Dependencies --> Running transaction check ---> Package redhat-virtualization-host-image-update.noarch 0:4.1-20170222.0.el7_3 will be obsoleting ---> Package redhat-virtualization-host-image-update-placeholder.noarch 0:4.0-5.2.el7 will be obsoleted --> Finished Dependency Resolution Dependencies Resolved ====================================================================================================================================================================== Package Arch Version Repository Size ====================================================================================================================================================================== Installing: redhat-virtualization-host-image-update noarch 4.1-20170222.0.el7_3 rhel-7-server-rhvh-4-beta-rpms 548 M replacing redhat-virtualization-host-image-update-placeholder.noarch 4.0-5.2.el7 Transaction Summary ====================================================================================================================================================================== Install 1 Package Total download size: 548 M Is this ok [y/d/N]: y Downloading packages: redhat-virtualization-host-image-update-latest.rpm | 548 MB 00:00:07 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : redhat-virtualization-host-image-update-4.1-20170222.0.el7_3.noarch 1/2 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 lsof: no pwd entry for UID 992 cp: warning: source file ‘/var/cache/yum/x86_64/7RedHatVirtualizationHost/rhel-7-server-rhvh-4-beta-rpms/packages/redhat-virtualization-host-image-update-latest.rpm’ specified more than once Erasing : redhat-virtualization-host-image-update-placeholder-4.0-5.2.el7.noarch 2/2 Verifying : redhat-virtualization-host-image-update-4.1-20170222.0.el7_3.noarch 1/2 Verifying : redhat-virtualization-host-image-update-placeholder-4.0-5.2.el7.noarch 2/2 Installed: redhat-virtualization-host-image-update.noarch 0:4.1-20170222.0.el7_3 Replaced: redhat-virtualization-host-image-update-placeholder.noarch 0:4.0-5.2.el7 Complete! [root@dhcp-10-16 ~]# imgbase w [INFO] You are on rhvh-4.0-0.20161116.0+1 [root@dhcp-10-16 ~]# imgbase layout rhvh-4.0-0.20160919.0 +- rhvh-4.0-0.20160919.0+1 rhvh-4.0-0.20161116.0 +- rhvh-4.0-0.20161116.0+1 rhvh-4.1-0.20170223.0 +- rhvh-4.1-0.20170223.0+1 2. In step4, there is not boot entry of build3 rhvh-4.1-0.20170223.0, only 2 boot entry of rhvh-4.0-0.20160919.0 and rhvh-4.0-0.20161116.0. And login rhvh-4.0-0.20161116.0, encounter Bug 1426151. Expected results: 1. In step3, there should be no error info during update. After upgrade, there should be only 2 latest bases. 2. In step4, there should be boot entry of rhvh-4.1-0.20170223.0 and rhvh-4.0-0.20161116.0 So this bug is not fixed in redhat-virtualization-host-4.1-20170222.0, change the status to ASSIGNED.
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.
Created attachment 1258290 [details] comment 11: imgbased.log
Yuval, according to Comment 9, this bug should be RFE, could you please set it as RFE and add RFE to the subject? QE need to modify test cases according to this bug. Thanks!
(In reply to Huijuan Zhao from comment #14) > Yuval, according to Comment 9, this bug should be RFE, could you please set > it as RFE and add RFE to the subject? QE need to modify test cases according > to this bug. > > Thanks! It is not an RFE, this bug makes the upgrade fail. It seems like you encounter something else, though. Can you please attach /tmp/imgbased.log ?
(In reply to Yuval Turgeman from comment #15) > > It is not an RFE, this bug makes the upgrade fail. It seems like you > encounter something else, though. Can you please attach /tmp/imgbased.log ? ok, but should we open a new RFE bug to track this modification from supporting many bases/layers to two bases/layers? Please refer to attachment "Comment 11: imgbased.log" for /tmp/imgbased.log.
I dont think we should add a new RFE because it isn't an RFE, this was a bug that was fixed. The bug reported in comment 11 is a new one, if you'd look at imgbased.log, you'd see that rsync is running out of space which raises a ConfigMigrationError, causing the upgrade to fail. I think we need to open a new bug for comment 11.
Moving back to QE as per comment #17
(In reply to Yuval Turgeman from comment #17) > I dont think we should add a new RFE because it isn't an RFE, this was a bug > that was fixed. The bug reported in comment 11 is a new one, if you'd look > at imgbased.log, you'd see that rsync is running out of space which raises a > ConfigMigrationError, causing the upgrade to fail. I think we need to open > a new bug for comment 11. ok, thanks Yuval, I opened a new Bug 1427468, and it blocks this bug verification. So I will verify this bug after Bug 1427468 is resolved.
Test version: Build1: redhat-virtualization-host-4.0-20160919.0 Build2: redhat-virtualization-host-4.0-20161116.0 Build3: redhat-virtualization-host-4.1-20170308.1 Test steps: 1. Install build1 redhat-virtualization-host-4.0-20160919.0 2. Reboot and login build1, download redhat-virtualization-host-image-update-4.0-20161116.1.el7_3.noarch.rpm and update: # yum install redhat-virtualization-host-image-update-4.0-20161116.1.el7_3.noarch.rpm 3. Reboot and login build2 rhvh-4.0-0.20161116.0, setup local repos, update to build3 redhat-virtualization-host-4.1-20170308.1: # yum update 4. Reboot and focus on boot entry 5. Login rhvh-4.1-20170308.1, ssh it from other host Test results: 1. In step3, there is no error info during update "#yum update" and upgrade successful. After update,there are 2 bases/layers. [root@dell-per730-35 ~]# imgbase layout rhvh-4.0-0.20161116.0 +- rhvh-4.0-0.20161116.0+1 rhvh-4.1-0.20170309.0 +- rhvh-4.1-0.20170309.0+1 2. In step4, there are two latest build boot entries. ------------------------ rhvh-4.1-0.20170309.0 rhvh-4.0-0.20161116.0 ------------------------ 3. In step5, there is error with sshd service, ssh rhvh 4.1 failed from other host: [huzhao@dhcp-8-245 ~]$ ssh root.131.65 ssh_dispatch_run_fatal: Connection reset by peer This issue is tracked by Bug 1427468. According to test results 1 and 2, this bug is fixed in redhat-virtualization-host-4.1-20170308.1. Test results 3 ssh issue is tracked by bug 1427468. So change the bug status to VERIFIED.
Agreed, no RFE