RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1382688 - lvm related failed messages observed during update of device-mapper-event with 7.3-RC-1.3
Summary: lvm related failed messages observed during update of device-mapper-event wit...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: lvm2
Version: 7.3
Hardware: x86_64
OS: Linux
urgent
high
Target Milestone: rc
: ---
Assignee: Peter Rajnoha
QA Contact: cluster-qe@redhat.com
Milan Navratil
URL:
Whiteboard:
Depends On:
Blocks: 1383833
TreeView+ depends on / blocked
 
Reported: 2016-10-07 12:15 UTC by Shashank Raj
Modified: 2021-09-03 12:54 UTC (History)
22 users (show)

Fixed In Version: lvm2-2.02.169-1.el7
Doc Type: Bug Fix
Doc Text:
LVM2 library incompatibilities no longer cause device monitoring to fail and be lost during an upgrade Due to a bug in the lvm2-2.02.166-1.el package, released in Red Hat Enterprise Linux 7.3, the library was incompatible with earlier versions of Red Hat Enterprise Linux 7. The incompatibility could cause device monitoring to fail and be lost during an upgrade. As a consequence, device failures could go unnoticed (RAID), or out-of-space conditions were not handled properly (thin-p). This update fixes the incompatibility, and the logical volume monitoring now works as expected.
Clone Of:
: 1383833 (view as bug list)
Environment:
Last Closed: 2017-08-01 21:49:49 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:2222 0 normal SHIPPED_LIVE lvm2 bug fix and enhancement update 2017-08-01 18:42:41 UTC

Description Shashank Raj 2016-10-07 12:15:45 UTC 
Description of problem:

lvm related failed messages observed during update of device-mapper-event package with 7.3-RC-1.3

Version-Release number of selected component (if applicable):

7.3-RC-1.3

How reproducible:

Once

Steps to Reproduce:
1.Have 7.2 based machines (in my case i had 3.2 RHGS installed on top of RHEL 7.2)

2.Have the lv's created on the machines. 

3.Try to update the machine using 7.3-RC-1.3 repo

[root@dhcp43-110 ~]# cd /etc/yum.repos.d/ 

[root@dhcp43-110 yum.repos.d]# ll | grep 73
-rw-r--r--. 1 root root    310 Oct  7  2016 73_RC.repo

[root@dhcp43-110 yum.repos.d]# cat 73_RC.repo 
[rhel73]
name=rhel73
baseurl=http://download-node-02.eng.bos.redhat.com/rel-eng/RHEL-7.3-RC-1.3/compose/Server/x86_64/os/
enabled=1
gpgcheck=0

[rhel73_optional]
name=rhel73_opt
baseurl=http://download-node-02.eng.bos.redhat.com/rel-eng/RHEL-7.3-RC-1.3/compose/Server-optional/x86_64/os/
enabled=1
gpgcheck=0

4. Observe that during update, while updating device-mapper-event package, following failed messages are seen:

  Updating   : 7evice-mapper-event-1.02.135-1.el7.x86_64                 182/763
Failed to reinstate monitoring for device LVM-ui0n8ItBYepaiZeatj6SvljJJ5IwG2UF5Muzgik9YwCPx8EsCp21z1vO5mBF5MRK-tpool.
Failed to reinstate monitoring for device LVM-GeD5Pjh6nkLBW01W9pZBfyKEnzO0j1eX3eokS5TUiYCLIPxy4SnW6Mcdwp1TzTl7-tpool.
Failed to reinstate monitoring for device LVM-BrIeE0TQVkTroSs62FBLHOCHNs66ssxyJVi1RzmuFTB11LkoMbsGjh5By952CQzd-tpool.
Failed to reinstate monitoring for device LVM-tjcQvwdrQwOp3pyRGoiFcVui3KnZcX2VfjywdDPQlah9lHwSJ8M1ONc8ew5b6l6X-tpool.
Failed to reinstate monitoring for device LVM-fhmJlBwY8tkoIAWQJexq3BiTReUVqI8nOjwsxhHMVEU8PvBdiR3UaxVWFcP2970s-tpool.
Failed to reinstate monitoring for device LVM-5aAJ6TeLtp9fb4LWNn1j1unZaB2z22KAJ9tGmeZiN4Sk11e1iZUUq9NnffJHHR0u-tpool.
Failed to reinstate monitoring for device LVM-zPDX7xNobouuLCA0obfcoQaOx5D1OeJMmOXjVzWRhB8zDkpAKfFuOcXE1tSaQNLI-tpool.
Failed to reinstate monitoring for device LVM-S37ZaCgXectZNzw46IKUhvVUQWflJuHNUasnfJuIG9H3ZlBsQWoaTJjcLhH1jadU-tpool.
Failed to reinstate monitoring for device LVM-XDxysuCHHumfNX4x7xtO73BA63o04EGjzUWtEHHdyHnaeB42JE5hcG5Qijpbx18F-tpool.
Failed to reinstate monitoring for device LVM-rh25IAzTG3iPHjPfPmGbT8wlDzymf0snG9knYczDe1s9oH0FKXz0hNgMPmHdAyhf-tpool.
  Updating   : 7:lvm2-libs-2.02.166-1.el7.x86_64                           183/76

5. The update was successful and no issues were seen during update other than these failed messages.

Actual results:

lvm related failed messages observed during update of device-mapper-event with 7.3-RC-1.3

Expected results:

There should not be any failures.

Additional info:
Comment 2 Peter Rajnoha 2016-10-07 13:34:27 UTC 
OK, reproduced. The problem is here:

[0] rhel7-a/~ # systemctl status -l dm-event
* dm-event.service - Device-mapper event daemon
   Loaded: loaded (/usr/lib/systemd/system/dm-event.service; disabled; vendor preset: enabled)
   Active: active (running) since Fri 2016-10-07 14:47:55 CEST; 8min ago
     Docs: man:dmeventd(8)
 Main PID: 1533 (dmeventd)
   CGroup: /system.slice/dm-event.service
           `-1533 /usr/sbin/dmeventd -f

Oct 07 14:47:55 rhel7-a dmeventd[1533]: dmeventd libdevmapper-event-lvm2thin.so dlopen failed: /lib64/liblvm2cmd.so.2.02: symbol dm_tree_node_size_changed, version DM_1_02_107 not defined in file libdevmapper.so.1.02 with link time reference.

When we released lvm2/device-mapper-event in RHEL7.2, we backported "dm_tree_node_size_changed" from libdm version 1.02.110 to 1.02.107 (which is a base for libdm in 7.2).

We didn't add the patch for 7.3 to register that function as being introduced in version 1.02.107 instead of 1.02.110. So to fix this, we need to add this patch for 7.3+ libdm releases (just moving dm_tree_node_size_changed to proper version in exported symbols file):

[0] rhel7-a/~/rpmbuild/SOURCES # cat a.patch 
 libdm/.exported_symbols.DM_1_02_107 | 1 +
 libdm/.exported_symbols.DM_1_02_110 | 1 -
 2 files changed, 1 insertion(+), 1 deletion(-)

diff --git a/libdm/.exported_symbols.DM_1_02_107 b/libdm/.exported_symbols.DM_1_02_107
index 89d3464..0c7b7af 100644
--- a/libdm/.exported_symbols.DM_1_02_107
+++ b/libdm/.exported_symbols.DM_1_02_107
@@ -13,3 +13,4 @@ dm_stats_create_region
 dm_stats_driver_supports_histogram
 dm_stats_get_histogram
 dm_stats_get_region_nr_histogram_bins
+dm_tree_node_size_changed
diff --git a/libdm/.exported_symbols.DM_1_02_110 b/libdm/.exported_symbols.DM_1_02_110
index eba5625..da742ee 100644
--- a/libdm/.exported_symbols.DM_1_02_110
+++ b/libdm/.exported_symbols.DM_1_02_110
@@ -1,3 +1,2 @@
 dm_report_compact_given_fields
 dm_hold_control_dev
-dm_tree_node_size_changed


However, even with this patch, I'm still getting: 

[0] rhel7-a/~ # systemctl status dm-event
* dm-event.service - Device-mapper event daemon
   Loaded: loaded (/usr/lib/systemd/system/dm-event.service; disabled; vendor preset: enabled)
   Active: active (running) since Fri 2016-10-07 15:23:28 CEST; 5min ago
     Docs: man:dmeventd(8)
 Main PID: 1532 (dmeventd)
   CGroup: /system.slice/dm-event.service
           `-1532 /usr/sbin/dmeventd -f

Oct 07 15:23:28 rhel7-a dmeventd[1532]: dmeventd libdevmapper-event-lvm2thin.so dlopen failed: /lib64/libdevmapper-event-lvm2.so.2.02: undefined symbol: dmeventd_debug.

So there's more incompatible changes here - in this case, it's the "dmeventd_debug" variable which got removed since libdm v1.02.107. This variable was defined as "extern int dmeventd_debug" in dmeventd plugin code (daemons/dmeventd/plugins/lvm2/dmeventd_lvm.c) while it's definition is in dmeventd code (daemons/dmeventd/dmeventd.c). Will need to inspect further how to deal with this.

We should fix this in 7.3.z if possible because otherwise we lose monitoring on package update from 7.2 to 7.3+ (where dmeventd -R is called to take the new dmeventd version and take over any existing monitoring).
Comment 3 Peter Rajnoha 2016-10-10 10:53:14 UTC 
(In reply to Peter Rajnoha from comment #2)
> So there's more incompatible changes here - in this case, it's the
> "dmeventd_debug" variable which got removed since libdm v1.02.107. This
> variable was defined as "extern int dmeventd_debug" in dmeventd plugin code
> (daemons/dmeventd/plugins/lvm2/dmeventd_lvm.c) while it's definition is in
> dmeventd code (daemons/dmeventd/dmeventd.c).

The other problem here is that plugins for dmeventd (e.g. that plugin for thin monitoring thin pools) is part of lvm2-libs package. Now when looking at the sequence during update, we have:

[0] raw/~/b # rpm -F -v *.rpm   
Preparing packages...
device-mapper-libs-1.02.135-2.1.fc26.x86_64
device-mapper-1.02.135-2.1.fc26.x86_64
device-mapper-event-libs-1.02.135-2.1.fc26.x86_64
device-mapper-event-1.02.135-2.1.fc26.x86_64
Failed to reinstate monitoring for device LVM-Aq9KPExgGvRABXbRwAm5kxSq8MVUCMUkAvy3SxmFFPurb95XUp18EY57eFPzxn8u.
lvm2-libs-2.02.166-2.1.fc26.x86_64
lvm2-2.02.166-2.1.fc26.x86_64
...

Which means that device-mapper-event (which contains the dmeventd -R reload hook) is updated *before* lvm2-libs (which contains dmeventd plugins for thin, mirror, snapshots...). We need to make sure that the dmeventd -R is called after new plugins are installed.

The best way to do this is to move the dmeventd -R from device-mapper-event package's "post" rpm script to "posttrans" which happens even all postun scripts which means we're even pass the lvm2-libs package upgrade:

[0] raw/~/c # rpm -F -v *.rpm
Preparing packages...
device-mapper-libs-1.02.135-2.1.fc26.x86_64
device-mapper-1.02.135-2.1.fc26.x86_64
device-mapper-event-libs-1.02.135-2.1.fc26.x86_64
device-mapper-event-1.02.135-2.1.fc26.x86_64

--> "post" script for new device-mapper-event goes here

lvm2-libs-2.02.166-2.1.fc26.x86_64
lvm2-2.02.166-2.1.fc26.x86_64
lvm2-2.02.130-1.fc23.x86_64
lvm2-libs-2.02.130-1.fc23.x86_64
device-mapper-event-1.02.107-1.fc23.x86_64

--> "postun" script for old device-mapper-event goes here
--> "posttrans" script for new device-mapper-event goes here

device-mapper-event-libs-1.02.107-1.fc23.x86_64
device-mapper-libs-1.02.107-1.fc23.x86_64
device-mapper-1.02.107-1.fc23.x86_64
Comment 4 Peter Rajnoha 2016-10-10 10:56:31 UTC 
So two fixes here:

- the one mentioned in comment #2 (moving the dm_tree_node_size_changed to proper .exported_symbols.DM_1_02_107 for proper symbol version registration)

- the one mentioned in comment #3 (moving dmeventd -R call for reload from post to posttrans rpm script for device-mapper-event package)
Comment 12 prasun.gera 2016-11-03 20:25:54 UTC 
Got this error message on update to 7.3 GA. What is the implication of the error message, and are there any workarounds until this is fixed ?
Comment 13 Steve Almy 2016-11-03 21:25:16 UTC 
(In reply to prasun.gera from comment #12)
> Got this error message on update to 7.3 GA. What is the implication of the
> error message, and are there any workarounds until this is fixed ?

Please contact your Red Hat Support representative.  Thanks
Comment 16 Roman Bednář 2017-04-11 09:41:31 UTC 
Verified with latest rpms. 


Initial setup (7.2GA):

3.10.0-327.el7.x86_64

lvm2-2.02.130-5.el7    BUILT: Wed Oct 14 15:27:29 CEST 2015
lvm2-libs-2.02.130-5.el7    BUILT: Wed Oct 14 15:27:29 CEST 2015
lvm2-cluster-2.02.130-5.el7    BUILT: Wed Oct 14 15:27:29 CEST 2015
device-mapper-1.02.107-5.el7    BUILT: Wed Oct 14 15:27:29 CEST 2015
device-mapper-libs-1.02.107-5.el7    BUILT: Wed Oct 14 15:27:29 CEST 2015
device-mapper-event-1.02.107-5.el7    BUILT: Wed Oct 14 15:27:29 CEST 2015
device-mapper-event-libs-1.02.107-5.el7    BUILT: Wed Oct 14 15:27:29 CEST 2015
device-mapper-persistent-data-0.5.5-1.el7    BUILT: Thu Aug 13 16:58:10 CEST 2015


# lvs -a
  LV              VG            Attr       LSize   Pool Origin Data%  Meta%
  ...                 
  POOL            vg            twi-aotz--   5.00g             0.00   0.73                            
  [POOL_tdata]    vg            Twi-ao----   5.00g                                                    
  [POOL_tmeta]    vg            ewi-ao----   8.00m                                                    
  [lvol0_pmspare] vg            ewi-------   8.00m                                                    
  thinlv          vg            Vwi-a-tz--   1.00g POOL        0.00  

# systemctl is-active dm-event
active


BEFORE PATCH:
update lvm2 to 7.3RC packages:

# yum update lvm2
...
  Updating   : 7:device-mapper-event-1.02.135-1.el7.x86_64                                                       4/18 
Failed to reinstate monitoring for device LVM-Jjc0sYveZf6ZbrbyKUdrZb5aY5NFX3R0n0luNUK3kTfj8fQ0WXppgnu2a3FMMnCU-tpool.
  Updating   : 7:lvm2-libs-2.02.166-1.el7.x86_64  
...

log:
...
Apr 11 11:16:25 virt-378 systemd: Starting Device-mapper event daemon...
Apr 11 11:16:25 virt-378 dmeventd[15710]: dmeventd libdevmapper-event-lvm2thin.so dlopen failed: /lib64/libdevmapper-event-lvm2.so.2.02: undefined symbol: dmeventd_debug.
Apr 11 11:16:25 virt-378 yum[15680]: Updated: 7:device-mapper-event-1.02.135-1.el7.x86_64
...


AFTER PATCH:
update to lvm2-2.02.169-3 - no errors observed

# systemctl status dm-event -l
● dm-event.service - Device-mapper event daemon
   Loaded: loaded (/usr/lib/systemd/system/dm-event.service; disabled; vendor preset: enabled)
   Active: active (running) since Tue 2017-04-11 11:29:05 CEST; 3min 35s ago
     Docs: man:dmeventd(8)
 Main PID: 1464 (dmeventd)
   CGroup: /system.slice/dm-event.service
           └─1464 /usr/sbin/dmeventd -f

Apr 11 11:29:05 virt-378.cluster-qe.lab.eng.brq.redhat.com systemd[1]: Started Device-mapper event daemon.
Apr 11 11:29:05 virt-378.cluster-qe.lab.eng.brq.redhat.com systemd[1]: Starting Device-mapper event daemon...
Apr 11 11:29:05 virt-378.cluster-qe.lab.eng.brq.redhat.com dmeventd[1464]: dmeventd ready for processing.
Apr 11 11:29:05 virt-378.cluster-qe.lab.eng.brq.redhat.com lvm[1464]: Monitoring thin pool vg-POOL-tpool.


3.10.0-640.el7.x86_64

lvm2-2.02.169-3.el7    BUILT: Wed Mar 29 16:17:46 CEST 2017
lvm2-libs-2.02.169-3.el7    BUILT: Wed Mar 29 16:17:46 CEST 2017
lvm2-cluster-2.02.169-3.el7    BUILT: Wed Mar 29 16:17:46 CEST 2017
device-mapper-1.02.138-3.el7    BUILT: Wed Mar 29 16:17:46 CEST 2017
device-mapper-libs-1.02.138-3.el7    BUILT: Wed Mar 29 16:17:46 CEST 2017
device-mapper-event-1.02.138-3.el7    BUILT: Wed Mar 29 16:17:46 CEST 2017
device-mapper-event-libs-1.02.138-3.el7    BUILT: Wed Mar 29 16:17:46 CEST 2017
device-mapper-persistent-data-0.7.0-0.1.rc6.el7    BUILT: Mon Mar 27 17:15:46 CEST 2017
Comment 17 errata-xmlrpc 2017-08-01 21:49:49 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2222
Note You need to log in before you can comment on or make changes to this bug.