Bug 1382688
| Summary: | lvm related failed messages observed during update of device-mapper-event with 7.3-RC-1.3 | |||
|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Shashank Raj <sraj> | |
| Component: | lvm2 | Assignee: | Peter Rajnoha <prajnoha> | |
| lvm2 sub component: | Thin Provisioning | QA Contact: | cluster-qe <cluster-qe> | |
| Status: | CLOSED ERRATA | Docs Contact: | Milan Navratil <mnavrati> | |
| Severity: | high | |||
| Priority: | urgent | CC: | agk, cmarthal, heinzm, jbrassow, lmanasko, mkolaja, msnitzer, mthacker, mzywusko, pasik, pprakash, prajnoha, prasun.gera, prockai, rbednar, salmy, shivgupt, snagar, thornber, tlavigne, yizhan, zkabelac | |
| Version: | 7.3 | Keywords: | ZStream | |
| Target Milestone: | rc | |||
| Target Release: | --- | |||
| Hardware: | x86_64 | |||
| OS: | Linux | |||
| Whiteboard: | ||||
| Fixed In Version: | lvm2-2.02.169-1.el7 | Doc Type: | Bug Fix | |
| Doc Text: |
LVM2 library incompatibilities no longer cause device monitoring to fail and be lost during an upgrade
Due to a bug in the lvm2-2.02.166-1.el package, released in Red Hat Enterprise Linux 7.3, the library was incompatible with earlier versions of Red Hat Enterprise Linux 7. The incompatibility could cause device monitoring to fail and be lost during an upgrade. As a consequence, device failures could go unnoticed (RAID), or out-of-space conditions were not handled properly (thin-p). This update fixes the incompatibility, and the logical volume monitoring now works as expected.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 1383833 (view as bug list) | Environment: | ||
| Last Closed: | 2017-08-01 21:49:49 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1383833 | |||
|
Description
Shashank Raj
2016-10-07 12:15:45 UTC
OK, reproduced. The problem is here:
[0] rhel7-a/~ # systemctl status -l dm-event
* dm-event.service - Device-mapper event daemon
Loaded: loaded (/usr/lib/systemd/system/dm-event.service; disabled; vendor preset: enabled)
Active: active (running) since Fri 2016-10-07 14:47:55 CEST; 8min ago
Docs: man:dmeventd(8)
Main PID: 1533 (dmeventd)
CGroup: /system.slice/dm-event.service
`-1533 /usr/sbin/dmeventd -f
Oct 07 14:47:55 rhel7-a dmeventd[1533]: dmeventd libdevmapper-event-lvm2thin.so dlopen failed: /lib64/liblvm2cmd.so.2.02: symbol dm_tree_node_size_changed, version DM_1_02_107 not defined in file libdevmapper.so.1.02 with link time reference.
When we released lvm2/device-mapper-event in RHEL7.2, we backported "dm_tree_node_size_changed" from libdm version 1.02.110 to 1.02.107 (which is a base for libdm in 7.2).
We didn't add the patch for 7.3 to register that function as being introduced in version 1.02.107 instead of 1.02.110. So to fix this, we need to add this patch for 7.3+ libdm releases (just moving dm_tree_node_size_changed to proper version in exported symbols file):
[0] rhel7-a/~/rpmbuild/SOURCES # cat a.patch
libdm/.exported_symbols.DM_1_02_107 | 1 +
libdm/.exported_symbols.DM_1_02_110 | 1 -
2 files changed, 1 insertion(+), 1 deletion(-)
diff --git a/libdm/.exported_symbols.DM_1_02_107 b/libdm/.exported_symbols.DM_1_02_107
index 89d3464..0c7b7af 100644
--- a/libdm/.exported_symbols.DM_1_02_107
+++ b/libdm/.exported_symbols.DM_1_02_107
@@ -13,3 +13,4 @@ dm_stats_create_region
dm_stats_driver_supports_histogram
dm_stats_get_histogram
dm_stats_get_region_nr_histogram_bins
+dm_tree_node_size_changed
diff --git a/libdm/.exported_symbols.DM_1_02_110 b/libdm/.exported_symbols.DM_1_02_110
index eba5625..da742ee 100644
--- a/libdm/.exported_symbols.DM_1_02_110
+++ b/libdm/.exported_symbols.DM_1_02_110
@@ -1,3 +1,2 @@
dm_report_compact_given_fields
dm_hold_control_dev
-dm_tree_node_size_changed
However, even with this patch, I'm still getting:
[0] rhel7-a/~ # systemctl status dm-event
* dm-event.service - Device-mapper event daemon
Loaded: loaded (/usr/lib/systemd/system/dm-event.service; disabled; vendor preset: enabled)
Active: active (running) since Fri 2016-10-07 15:23:28 CEST; 5min ago
Docs: man:dmeventd(8)
Main PID: 1532 (dmeventd)
CGroup: /system.slice/dm-event.service
`-1532 /usr/sbin/dmeventd -f
Oct 07 15:23:28 rhel7-a dmeventd[1532]: dmeventd libdevmapper-event-lvm2thin.so dlopen failed: /lib64/libdevmapper-event-lvm2.so.2.02: undefined symbol: dmeventd_debug.
So there's more incompatible changes here - in this case, it's the "dmeventd_debug" variable which got removed since libdm v1.02.107. This variable was defined as "extern int dmeventd_debug" in dmeventd plugin code (daemons/dmeventd/plugins/lvm2/dmeventd_lvm.c) while it's definition is in dmeventd code (daemons/dmeventd/dmeventd.c). Will need to inspect further how to deal with this.
We should fix this in 7.3.z if possible because otherwise we lose monitoring on package update from 7.2 to 7.3+ (where dmeventd -R is called to take the new dmeventd version and take over any existing monitoring).
(In reply to Peter Rajnoha from comment #2) > So there's more incompatible changes here - in this case, it's the > "dmeventd_debug" variable which got removed since libdm v1.02.107. This > variable was defined as "extern int dmeventd_debug" in dmeventd plugin code > (daemons/dmeventd/plugins/lvm2/dmeventd_lvm.c) while it's definition is in > dmeventd code (daemons/dmeventd/dmeventd.c). The other problem here is that plugins for dmeventd (e.g. that plugin for thin monitoring thin pools) is part of lvm2-libs package. Now when looking at the sequence during update, we have: [0] raw/~/b # rpm -F -v *.rpm Preparing packages... device-mapper-libs-1.02.135-2.1.fc26.x86_64 device-mapper-1.02.135-2.1.fc26.x86_64 device-mapper-event-libs-1.02.135-2.1.fc26.x86_64 device-mapper-event-1.02.135-2.1.fc26.x86_64 Failed to reinstate monitoring for device LVM-Aq9KPExgGvRABXbRwAm5kxSq8MVUCMUkAvy3SxmFFPurb95XUp18EY57eFPzxn8u. lvm2-libs-2.02.166-2.1.fc26.x86_64 lvm2-2.02.166-2.1.fc26.x86_64 ... Which means that device-mapper-event (which contains the dmeventd -R reload hook) is updated *before* lvm2-libs (which contains dmeventd plugins for thin, mirror, snapshots...). We need to make sure that the dmeventd -R is called after new plugins are installed. The best way to do this is to move the dmeventd -R from device-mapper-event package's "post" rpm script to "posttrans" which happens even all postun scripts which means we're even pass the lvm2-libs package upgrade: [0] raw/~/c # rpm -F -v *.rpm Preparing packages... device-mapper-libs-1.02.135-2.1.fc26.x86_64 device-mapper-1.02.135-2.1.fc26.x86_64 device-mapper-event-libs-1.02.135-2.1.fc26.x86_64 device-mapper-event-1.02.135-2.1.fc26.x86_64 --> "post" script for new device-mapper-event goes here lvm2-libs-2.02.166-2.1.fc26.x86_64 lvm2-2.02.166-2.1.fc26.x86_64 lvm2-2.02.130-1.fc23.x86_64 lvm2-libs-2.02.130-1.fc23.x86_64 device-mapper-event-1.02.107-1.fc23.x86_64 --> "postun" script for old device-mapper-event goes here --> "posttrans" script for new device-mapper-event goes here device-mapper-event-libs-1.02.107-1.fc23.x86_64 device-mapper-libs-1.02.107-1.fc23.x86_64 device-mapper-1.02.107-1.fc23.x86_64 So two fixes here: - the one mentioned in comment #2 (moving the dm_tree_node_size_changed to proper .exported_symbols.DM_1_02_107 for proper symbol version registration) - the one mentioned in comment #3 (moving dmeventd -R call for reload from post to posttrans rpm script for device-mapper-event package) Got this error message on update to 7.3 GA. What is the implication of the error message, and are there any workarounds until this is fixed ? (In reply to prasun.gera from comment #12) > Got this error message on update to 7.3 GA. What is the implication of the > error message, and are there any workarounds until this is fixed ? Please contact your Red Hat Support representative. Thanks Verified with latest rpms.
Initial setup (7.2GA):
3.10.0-327.el7.x86_64
lvm2-2.02.130-5.el7 BUILT: Wed Oct 14 15:27:29 CEST 2015
lvm2-libs-2.02.130-5.el7 BUILT: Wed Oct 14 15:27:29 CEST 2015
lvm2-cluster-2.02.130-5.el7 BUILT: Wed Oct 14 15:27:29 CEST 2015
device-mapper-1.02.107-5.el7 BUILT: Wed Oct 14 15:27:29 CEST 2015
device-mapper-libs-1.02.107-5.el7 BUILT: Wed Oct 14 15:27:29 CEST 2015
device-mapper-event-1.02.107-5.el7 BUILT: Wed Oct 14 15:27:29 CEST 2015
device-mapper-event-libs-1.02.107-5.el7 BUILT: Wed Oct 14 15:27:29 CEST 2015
device-mapper-persistent-data-0.5.5-1.el7 BUILT: Thu Aug 13 16:58:10 CEST 2015
# lvs -a
LV VG Attr LSize Pool Origin Data% Meta%
...
POOL vg twi-aotz-- 5.00g 0.00 0.73
[POOL_tdata] vg Twi-ao---- 5.00g
[POOL_tmeta] vg ewi-ao---- 8.00m
[lvol0_pmspare] vg ewi------- 8.00m
thinlv vg Vwi-a-tz-- 1.00g POOL 0.00
# systemctl is-active dm-event
active
BEFORE PATCH:
update lvm2 to 7.3RC packages:
# yum update lvm2
...
Updating : 7:device-mapper-event-1.02.135-1.el7.x86_64 4/18
Failed to reinstate monitoring for device LVM-Jjc0sYveZf6ZbrbyKUdrZb5aY5NFX3R0n0luNUK3kTfj8fQ0WXppgnu2a3FMMnCU-tpool.
Updating : 7:lvm2-libs-2.02.166-1.el7.x86_64
...
log:
...
Apr 11 11:16:25 virt-378 systemd: Starting Device-mapper event daemon...
Apr 11 11:16:25 virt-378 dmeventd[15710]: dmeventd libdevmapper-event-lvm2thin.so dlopen failed: /lib64/libdevmapper-event-lvm2.so.2.02: undefined symbol: dmeventd_debug.
Apr 11 11:16:25 virt-378 yum[15680]: Updated: 7:device-mapper-event-1.02.135-1.el7.x86_64
...
AFTER PATCH:
update to lvm2-2.02.169-3 - no errors observed
# systemctl status dm-event -l
● dm-event.service - Device-mapper event daemon
Loaded: loaded (/usr/lib/systemd/system/dm-event.service; disabled; vendor preset: enabled)
Active: active (running) since Tue 2017-04-11 11:29:05 CEST; 3min 35s ago
Docs: man:dmeventd(8)
Main PID: 1464 (dmeventd)
CGroup: /system.slice/dm-event.service
└─1464 /usr/sbin/dmeventd -f
Apr 11 11:29:05 virt-378.cluster-qe.lab.eng.brq.redhat.com systemd[1]: Started Device-mapper event daemon.
Apr 11 11:29:05 virt-378.cluster-qe.lab.eng.brq.redhat.com systemd[1]: Starting Device-mapper event daemon...
Apr 11 11:29:05 virt-378.cluster-qe.lab.eng.brq.redhat.com dmeventd[1464]: dmeventd ready for processing.
Apr 11 11:29:05 virt-378.cluster-qe.lab.eng.brq.redhat.com lvm[1464]: Monitoring thin pool vg-POOL-tpool.
3.10.0-640.el7.x86_64
lvm2-2.02.169-3.el7 BUILT: Wed Mar 29 16:17:46 CEST 2017
lvm2-libs-2.02.169-3.el7 BUILT: Wed Mar 29 16:17:46 CEST 2017
lvm2-cluster-2.02.169-3.el7 BUILT: Wed Mar 29 16:17:46 CEST 2017
device-mapper-1.02.138-3.el7 BUILT: Wed Mar 29 16:17:46 CEST 2017
device-mapper-libs-1.02.138-3.el7 BUILT: Wed Mar 29 16:17:46 CEST 2017
device-mapper-event-1.02.138-3.el7 BUILT: Wed Mar 29 16:17:46 CEST 2017
device-mapper-event-libs-1.02.138-3.el7 BUILT: Wed Mar 29 16:17:46 CEST 2017
device-mapper-persistent-data-0.7.0-0.1.rc6.el7 BUILT: Mon Mar 27 17:15:46 CEST 2017
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:2222 |