Hide Forgot
Description of problem: lvm related failed messages observed during update of device-mapper-event package with 7.3-RC-1.3 Version-Release number of selected component (if applicable): 7.3-RC-1.3 How reproducible: Once Steps to Reproduce: 1.Have 7.2 based machines (in my case i had 3.2 RHGS installed on top of RHEL 7.2) 2.Have the lv's created on the machines. 3.Try to update the machine using 7.3-RC-1.3 repo [root@dhcp43-110 ~]# cd /etc/yum.repos.d/ [root@dhcp43-110 yum.repos.d]# ll | grep 73 -rw-r--r--. 1 root root 310 Oct 7 2016 73_RC.repo [root@dhcp43-110 yum.repos.d]# cat 73_RC.repo [rhel73] name=rhel73 baseurl=http://download-node-02.eng.bos.redhat.com/rel-eng/RHEL-7.3-RC-1.3/compose/Server/x86_64/os/ enabled=1 gpgcheck=0 [rhel73_optional] name=rhel73_opt baseurl=http://download-node-02.eng.bos.redhat.com/rel-eng/RHEL-7.3-RC-1.3/compose/Server-optional/x86_64/os/ enabled=1 gpgcheck=0 4. Observe that during update, while updating device-mapper-event package, following failed messages are seen: Updating : 7evice-mapper-event-1.02.135-1.el7.x86_64 182/763 Failed to reinstate monitoring for device LVM-ui0n8ItBYepaiZeatj6SvljJJ5IwG2UF5Muzgik9YwCPx8EsCp21z1vO5mBF5MRK-tpool. Failed to reinstate monitoring for device LVM-GeD5Pjh6nkLBW01W9pZBfyKEnzO0j1eX3eokS5TUiYCLIPxy4SnW6Mcdwp1TzTl7-tpool. Failed to reinstate monitoring for device LVM-BrIeE0TQVkTroSs62FBLHOCHNs66ssxyJVi1RzmuFTB11LkoMbsGjh5By952CQzd-tpool. Failed to reinstate monitoring for device LVM-tjcQvwdrQwOp3pyRGoiFcVui3KnZcX2VfjywdDPQlah9lHwSJ8M1ONc8ew5b6l6X-tpool. Failed to reinstate monitoring for device LVM-fhmJlBwY8tkoIAWQJexq3BiTReUVqI8nOjwsxhHMVEU8PvBdiR3UaxVWFcP2970s-tpool. Failed to reinstate monitoring for device LVM-5aAJ6TeLtp9fb4LWNn1j1unZaB2z22KAJ9tGmeZiN4Sk11e1iZUUq9NnffJHHR0u-tpool. Failed to reinstate monitoring for device LVM-zPDX7xNobouuLCA0obfcoQaOx5D1OeJMmOXjVzWRhB8zDkpAKfFuOcXE1tSaQNLI-tpool. Failed to reinstate monitoring for device LVM-S37ZaCgXectZNzw46IKUhvVUQWflJuHNUasnfJuIG9H3ZlBsQWoaTJjcLhH1jadU-tpool. Failed to reinstate monitoring for device LVM-XDxysuCHHumfNX4x7xtO73BA63o04EGjzUWtEHHdyHnaeB42JE5hcG5Qijpbx18F-tpool. Failed to reinstate monitoring for device LVM-rh25IAzTG3iPHjPfPmGbT8wlDzymf0snG9knYczDe1s9oH0FKXz0hNgMPmHdAyhf-tpool. Updating : 7:lvm2-libs-2.02.166-1.el7.x86_64 183/76 5. The update was successful and no issues were seen during update other than these failed messages. Actual results: lvm related failed messages observed during update of device-mapper-event with 7.3-RC-1.3 Expected results: There should not be any failures. Additional info:
OK, reproduced. The problem is here: [0] rhel7-a/~ # systemctl status -l dm-event * dm-event.service - Device-mapper event daemon Loaded: loaded (/usr/lib/systemd/system/dm-event.service; disabled; vendor preset: enabled) Active: active (running) since Fri 2016-10-07 14:47:55 CEST; 8min ago Docs: man:dmeventd(8) Main PID: 1533 (dmeventd) CGroup: /system.slice/dm-event.service `-1533 /usr/sbin/dmeventd -f Oct 07 14:47:55 rhel7-a dmeventd[1533]: dmeventd libdevmapper-event-lvm2thin.so dlopen failed: /lib64/liblvm2cmd.so.2.02: symbol dm_tree_node_size_changed, version DM_1_02_107 not defined in file libdevmapper.so.1.02 with link time reference. When we released lvm2/device-mapper-event in RHEL7.2, we backported "dm_tree_node_size_changed" from libdm version 1.02.110 to 1.02.107 (which is a base for libdm in 7.2). We didn't add the patch for 7.3 to register that function as being introduced in version 1.02.107 instead of 1.02.110. So to fix this, we need to add this patch for 7.3+ libdm releases (just moving dm_tree_node_size_changed to proper version in exported symbols file): [0] rhel7-a/~/rpmbuild/SOURCES # cat a.patch libdm/.exported_symbols.DM_1_02_107 | 1 + libdm/.exported_symbols.DM_1_02_110 | 1 - 2 files changed, 1 insertion(+), 1 deletion(-) diff --git a/libdm/.exported_symbols.DM_1_02_107 b/libdm/.exported_symbols.DM_1_02_107 index 89d3464..0c7b7af 100644 --- a/libdm/.exported_symbols.DM_1_02_107 +++ b/libdm/.exported_symbols.DM_1_02_107 @@ -13,3 +13,4 @@ dm_stats_create_region dm_stats_driver_supports_histogram dm_stats_get_histogram dm_stats_get_region_nr_histogram_bins +dm_tree_node_size_changed diff --git a/libdm/.exported_symbols.DM_1_02_110 b/libdm/.exported_symbols.DM_1_02_110 index eba5625..da742ee 100644 --- a/libdm/.exported_symbols.DM_1_02_110 +++ b/libdm/.exported_symbols.DM_1_02_110 @@ -1,3 +1,2 @@ dm_report_compact_given_fields dm_hold_control_dev -dm_tree_node_size_changed However, even with this patch, I'm still getting: [0] rhel7-a/~ # systemctl status dm-event * dm-event.service - Device-mapper event daemon Loaded: loaded (/usr/lib/systemd/system/dm-event.service; disabled; vendor preset: enabled) Active: active (running) since Fri 2016-10-07 15:23:28 CEST; 5min ago Docs: man:dmeventd(8) Main PID: 1532 (dmeventd) CGroup: /system.slice/dm-event.service `-1532 /usr/sbin/dmeventd -f Oct 07 15:23:28 rhel7-a dmeventd[1532]: dmeventd libdevmapper-event-lvm2thin.so dlopen failed: /lib64/libdevmapper-event-lvm2.so.2.02: undefined symbol: dmeventd_debug. So there's more incompatible changes here - in this case, it's the "dmeventd_debug" variable which got removed since libdm v1.02.107. This variable was defined as "extern int dmeventd_debug" in dmeventd plugin code (daemons/dmeventd/plugins/lvm2/dmeventd_lvm.c) while it's definition is in dmeventd code (daemons/dmeventd/dmeventd.c). Will need to inspect further how to deal with this. We should fix this in 7.3.z if possible because otherwise we lose monitoring on package update from 7.2 to 7.3+ (where dmeventd -R is called to take the new dmeventd version and take over any existing monitoring).
(In reply to Peter Rajnoha from comment #2) > So there's more incompatible changes here - in this case, it's the > "dmeventd_debug" variable which got removed since libdm v1.02.107. This > variable was defined as "extern int dmeventd_debug" in dmeventd plugin code > (daemons/dmeventd/plugins/lvm2/dmeventd_lvm.c) while it's definition is in > dmeventd code (daemons/dmeventd/dmeventd.c). The other problem here is that plugins for dmeventd (e.g. that plugin for thin monitoring thin pools) is part of lvm2-libs package. Now when looking at the sequence during update, we have: [0] raw/~/b # rpm -F -v *.rpm Preparing packages... device-mapper-libs-1.02.135-2.1.fc26.x86_64 device-mapper-1.02.135-2.1.fc26.x86_64 device-mapper-event-libs-1.02.135-2.1.fc26.x86_64 device-mapper-event-1.02.135-2.1.fc26.x86_64 Failed to reinstate monitoring for device LVM-Aq9KPExgGvRABXbRwAm5kxSq8MVUCMUkAvy3SxmFFPurb95XUp18EY57eFPzxn8u. lvm2-libs-2.02.166-2.1.fc26.x86_64 lvm2-2.02.166-2.1.fc26.x86_64 ... Which means that device-mapper-event (which contains the dmeventd -R reload hook) is updated *before* lvm2-libs (which contains dmeventd plugins for thin, mirror, snapshots...). We need to make sure that the dmeventd -R is called after new plugins are installed. The best way to do this is to move the dmeventd -R from device-mapper-event package's "post" rpm script to "posttrans" which happens even all postun scripts which means we're even pass the lvm2-libs package upgrade: [0] raw/~/c # rpm -F -v *.rpm Preparing packages... device-mapper-libs-1.02.135-2.1.fc26.x86_64 device-mapper-1.02.135-2.1.fc26.x86_64 device-mapper-event-libs-1.02.135-2.1.fc26.x86_64 device-mapper-event-1.02.135-2.1.fc26.x86_64 --> "post" script for new device-mapper-event goes here lvm2-libs-2.02.166-2.1.fc26.x86_64 lvm2-2.02.166-2.1.fc26.x86_64 lvm2-2.02.130-1.fc23.x86_64 lvm2-libs-2.02.130-1.fc23.x86_64 device-mapper-event-1.02.107-1.fc23.x86_64 --> "postun" script for old device-mapper-event goes here --> "posttrans" script for new device-mapper-event goes here device-mapper-event-libs-1.02.107-1.fc23.x86_64 device-mapper-libs-1.02.107-1.fc23.x86_64 device-mapper-1.02.107-1.fc23.x86_64
So two fixes here: - the one mentioned in comment #2 (moving the dm_tree_node_size_changed to proper .exported_symbols.DM_1_02_107 for proper symbol version registration) - the one mentioned in comment #3 (moving dmeventd -R call for reload from post to posttrans rpm script for device-mapper-event package)
Got this error message on update to 7.3 GA. What is the implication of the error message, and are there any workarounds until this is fixed ?
(In reply to prasun.gera from comment #12) > Got this error message on update to 7.3 GA. What is the implication of the > error message, and are there any workarounds until this is fixed ? Please contact your Red Hat Support representative. Thanks
Verified with latest rpms. Initial setup (7.2GA): 3.10.0-327.el7.x86_64 lvm2-2.02.130-5.el7 BUILT: Wed Oct 14 15:27:29 CEST 2015 lvm2-libs-2.02.130-5.el7 BUILT: Wed Oct 14 15:27:29 CEST 2015 lvm2-cluster-2.02.130-5.el7 BUILT: Wed Oct 14 15:27:29 CEST 2015 device-mapper-1.02.107-5.el7 BUILT: Wed Oct 14 15:27:29 CEST 2015 device-mapper-libs-1.02.107-5.el7 BUILT: Wed Oct 14 15:27:29 CEST 2015 device-mapper-event-1.02.107-5.el7 BUILT: Wed Oct 14 15:27:29 CEST 2015 device-mapper-event-libs-1.02.107-5.el7 BUILT: Wed Oct 14 15:27:29 CEST 2015 device-mapper-persistent-data-0.5.5-1.el7 BUILT: Thu Aug 13 16:58:10 CEST 2015 # lvs -a LV VG Attr LSize Pool Origin Data% Meta% ... POOL vg twi-aotz-- 5.00g 0.00 0.73 [POOL_tdata] vg Twi-ao---- 5.00g [POOL_tmeta] vg ewi-ao---- 8.00m [lvol0_pmspare] vg ewi------- 8.00m thinlv vg Vwi-a-tz-- 1.00g POOL 0.00 # systemctl is-active dm-event active BEFORE PATCH: update lvm2 to 7.3RC packages: # yum update lvm2 ... Updating : 7:device-mapper-event-1.02.135-1.el7.x86_64 4/18 Failed to reinstate monitoring for device LVM-Jjc0sYveZf6ZbrbyKUdrZb5aY5NFX3R0n0luNUK3kTfj8fQ0WXppgnu2a3FMMnCU-tpool. Updating : 7:lvm2-libs-2.02.166-1.el7.x86_64 ... log: ... Apr 11 11:16:25 virt-378 systemd: Starting Device-mapper event daemon... Apr 11 11:16:25 virt-378 dmeventd[15710]: dmeventd libdevmapper-event-lvm2thin.so dlopen failed: /lib64/libdevmapper-event-lvm2.so.2.02: undefined symbol: dmeventd_debug. Apr 11 11:16:25 virt-378 yum[15680]: Updated: 7:device-mapper-event-1.02.135-1.el7.x86_64 ... AFTER PATCH: update to lvm2-2.02.169-3 - no errors observed # systemctl status dm-event -l ● dm-event.service - Device-mapper event daemon Loaded: loaded (/usr/lib/systemd/system/dm-event.service; disabled; vendor preset: enabled) Active: active (running) since Tue 2017-04-11 11:29:05 CEST; 3min 35s ago Docs: man:dmeventd(8) Main PID: 1464 (dmeventd) CGroup: /system.slice/dm-event.service └─1464 /usr/sbin/dmeventd -f Apr 11 11:29:05 virt-378.cluster-qe.lab.eng.brq.redhat.com systemd[1]: Started Device-mapper event daemon. Apr 11 11:29:05 virt-378.cluster-qe.lab.eng.brq.redhat.com systemd[1]: Starting Device-mapper event daemon... Apr 11 11:29:05 virt-378.cluster-qe.lab.eng.brq.redhat.com dmeventd[1464]: dmeventd ready for processing. Apr 11 11:29:05 virt-378.cluster-qe.lab.eng.brq.redhat.com lvm[1464]: Monitoring thin pool vg-POOL-tpool. 3.10.0-640.el7.x86_64 lvm2-2.02.169-3.el7 BUILT: Wed Mar 29 16:17:46 CEST 2017 lvm2-libs-2.02.169-3.el7 BUILT: Wed Mar 29 16:17:46 CEST 2017 lvm2-cluster-2.02.169-3.el7 BUILT: Wed Mar 29 16:17:46 CEST 2017 device-mapper-1.02.138-3.el7 BUILT: Wed Mar 29 16:17:46 CEST 2017 device-mapper-libs-1.02.138-3.el7 BUILT: Wed Mar 29 16:17:46 CEST 2017 device-mapper-event-1.02.138-3.el7 BUILT: Wed Mar 29 16:17:46 CEST 2017 device-mapper-event-libs-1.02.138-3.el7 BUILT: Wed Mar 29 16:17:46 CEST 2017 device-mapper-persistent-data-0.7.0-0.1.rc6.el7 BUILT: Mon Mar 27 17:15:46 CEST 2017
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:2222