Bug 1182048 - [3.5-7.0]Hide error messages(device-mapper: multipath: error getting device) display while rhevh 7.0 login
Summary: [3.5-7.0]Hide error messages(device-mapper: multipath: error getting device) ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-node
Version: 3.5.0
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: ovirt-3.6.3
: 3.6.0
Assignee: Anatoly Litovsky
QA Contact: Ying Cui
URL:
Whiteboard:
: 1293484 (view as bug list)
Depends On:
Blocks: rhev35gablocker
TreeView+ depends on / blocked
 
Reported: 2015-01-14 10:59 UTC by Ying Cui
Modified: 2016-03-09 14:25 UTC (History)
18 users (show)

Fixed In Version:
Doc Type: Known Issue
Doc Text:
On some Red Hat Enterprise Virtualization Hypervisor hosts, a device-mapper error (device-mapper: table: 253:6: multipath: error getting device) is displayed on the login screen. This error does not have a functional impact on those hosts.
Clone Of:
Environment:
Last Closed: 2016-03-09 14:25:02 UTC
oVirt Team: Node


Attachments (Terms of Use)
varlog (144.99 KB, application/x-gzip)
2015-01-14 11:00 UTC, Ying Cui
no flags Details
screen_output (320.76 KB, text/plain)
2015-01-14 11:02 UTC, Ying Cui
no flags Details
sosreport (4.96 MB, application/x-xz)
2015-01-14 11:19 UTC, Ying Cui
no flags Details
part sof dmesg and /dev (39.72 KB, image/png)
2015-01-14 11:56 UTC, Fabian Deutsch
no flags Details
varlog_20150119 (390.36 KB, application/x-gzip)
2015-01-21 11:14 UTC, Ying Cui
no flags Details
sosreport_20150119 (5.14 MB, application/x-xz)
2015-01-21 11:15 UTC, Ying Cui
no flags Details
screenoutput_20150119 (234.23 KB, text/plain)
2015-01-21 11:16 UTC, Ying Cui
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:0378 normal SHIPPED_LIVE ovirt-node bug fix and enhancement update for RHEV 3.6 2016-03-09 19:06:36 UTC
oVirt gerrit 36971 master MERGED Add product info in %post, but inside the chroot Never
oVirt gerrit 36987 master MERGED Don't glob the initramfs, use explicit kernel versions Never
oVirt gerrit 37099 ovirt-3.5 MERGED Don't glob the initramfs, use explicit kernel versions Never

Description Ying Cui 2015-01-14 10:59:24 UTC
Description:
RHEVH 7.0 installed successful, and while login rhevh 7.0, there display error messages, not sure whether impact function or not.

=======
Red Hat Enterprise Virtualization Hypervisor release 7.0 (20150106.0.el7ev)
Kernel 3.10.0-123.13.2.el7.x86_64 on an x86_64 (ttyS0)

Please login as 'admin' to configure the node
localhost login: [   40.781316] device-mapper: multipath service-time: version 0.2.0 loaded
[   40.788514] device-mapper: table: 253:6: multipath: error getting device
[   40.795228] device-mapper: ioctl: error adding target to table
[   40.803269] device-mapper: table: 253:6: multipath: error getting device
[   40.809979] device-mapper: ioctl: error adding target to table
[   77.192223] systemd-readahead[746]: Failed to open pack file: Read-only file system
=======

Test version:
# rpm -q kernel device-mapper ovirt-node dracut systemd device-mapper-multipath
kernel-3.10.0-123.13.2.el7.x86_64
device-mapper-1.02.84-14.el7.x86_64
ovirt-node-3.1.0-0.40.20150105git69f34a6.el7.noarch
dracut-033-161.el7_0.173.x86_64
systemd-208-11.el7_0.5.x86_64
device-mapper-multipath-0.4.9-66.el7.x86_64
# cat /etc/system-release
Red Hat Enterprise Virtualization Hypervisor release 7.0 (20150106.0.el7ev)

Test steps:
1. installed rhevh via TUI.
2. after rhevh boot, can see _login_

Actual result:
There display device-mapper multipath: error getting device.

Expected result:
Do not display such error in rhevh 7.0 login.

Comment 1 Ying Cui 2015-01-14 11:00:07 UTC
Created attachment 979947 [details]
varlog

Comment 2 Ying Cui 2015-01-14 11:02:34 UTC
Created attachment 979948 [details]
screen_output

Comment 3 Ying Cui 2015-01-14 11:19:03 UTC
Created attachment 979949 [details]
sosreport

Comment 4 Ying Cui 2015-01-14 11:26:05 UTC
Till now we have not identified the obvious effect on rhevh real function. but this error display on rhevh login page, that is not good for user experience. consider GA blocker.

Comment 5 Fabian Deutsch 2015-01-14 11:56:32 UTC
Created attachment 979967 [details]
part sof dmesg and /dev

The screenshot shows that mpath is complaining about a device which does not exist (according to the minor:major number).

Comment 6 Fabian Deutsch 2015-01-14 11:57:00 UTC
Ben, have you seen this error before?

Comment 7 Ben Marzinski 2015-01-15 18:18:29 UTC
What's happening here is that multipath is configured to use some paths, but for some reason it can't created a device with them (most likely because they are already in use).  Unfortunately, when the kernel prints this message, it doesn't have the information on which paths those where any more.

The reason that a non-existent dm device is being shown is due to how device mapper works.  First it creates the dm device, 253:6 in this case.  Then it tries to load the table.  If the table load fails, multipath will remove the useless dm device. But for a brief bit, there was a dm device 253:6.  I will look in the logs to see if I can figure out what device multipath was trying to load.

If I can't find anything useful in the logs, you may be able to find out by simply running

# multipath -v3

If the devices are still there and still in use by something other than multipath, you should see the same messages in the log, and multipath will tell you that it is ignoring the device because it is already in use.

Comment 8 Fabian Deutsch 2015-01-19 13:49:55 UTC
Thanks Ben.

It looks like this is yet another symptom of the build problem we were seeing with our latest build.

Basically the initrd multipath was wrong, which also lead to a in general broken multipath. I can not reproduce this anymore on a build containing the fixes.

Comment 9 Fabian Deutsch 2015-01-19 14:29:08 UTC

*** This bug has been marked as a duplicate of bug 1182551 ***

Comment 10 Ying Cui 2015-01-21 11:13:13 UTC
The bug 1182551 is fixed on rhevh 7.0 20150119.0.1.el7ev build yet.
But we still encounter this bug on 20150119.0.1.el7ev build, so I have to reopen this bug to more investigation.

Red Hat Enterprise Virtualization Hypervisor release 7.0 (20150119.0.1.el7ev)
Kernel 3.10.0-123.13.2.el7.x86_64 on an x86_64 (ttyS0)

Please login as 'admin' to configure the node
localhost login: [   57.795013] e1000e: em1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[   57.802465] IPv6: ADDRCONF(NETDEV_CHANGE): em1: link becomes ready
[   71.064811] device-mapper: multipath service-time: version 0.2.0 loaded
[   71.071596] device-mapper: table: 253:6: multipath: error getting device
[   71.078311] device-mapper: ioctl: error adding target to table
[  100.654467] systemd-readahead[816]: Failed to open pack file: Read-only file system

# rpm -q ovirt-node dracut systemd device-mapper-multipath  
device-mapper
ovirt-node-3.2.1-5.el7.noarch
dracut-033-161.el7_0.173.x86_64
systemd-208-11.el7_0.6.x86_64
device-mapper-multipath-0.4.9-66.el7.x86_64
device-mapper-1.02.84-14.el7.x86_64

Comment 11 Ying Cui 2015-01-21 11:14:39 UTC
Created attachment 982277 [details]
varlog_20150119

already move quiet and add rd.debug in grub then capture this var/log

Comment 12 Ying Cui 2015-01-21 11:15:41 UTC
Created attachment 982278 [details]
sosreport_20150119

Comment 13 Ying Cui 2015-01-21 11:16:21 UTC
Created attachment 982279 [details]
screenoutput_20150119

Comment 14 Fabian Deutsch 2015-01-21 14:56:06 UTC
It took some time to figur it out.
multipath -v3 does trigger the error, but the erorr is only printed on the console, not on a ssh terminal:

[root@alhost ~]# multipath -v3
Jan 21 14:53:59 | multipath.conf +5, invalid keyword: getuid_callout
Jan 21 14:53:59 | multipath.conf +18, invalid keyword: getuid_callout
Jan 21 14:53:59 | multipath.conf +37, invalid keyword: getuid_callout
Jan 21 14:53:59 | loading /lib64/multipath/libcheckdirectio.so checker
Jan 21 14:53:59 | loading /lib64/multipath/libprioconst.so prioritizer
Jan 21 14:53:59 | sda: not found in pathvec
Jan 21 14:53:59 | sda: mask = 0x3f
Jan 21 14:53:59 | sda: dev_t = 8:0
Jan 21 14:53:59 | sda: size = 41943040
Jan 21 14:53:59 | sda: vendor = ATA     
Jan 21 14:53:59 | sda: product = QEMU HARDDISK   
Jan 21 14:53:59 | sda: rev = 2.1.
Jan 21 14:53:59 | sda: h:b:t:l = 6:0:0:0
Jan 21 14:53:59 | sda: tgt_node_name = ata-7.00
Jan 21 14:53:59 | sda: path state = running

Jan 21 14:53:59 | sda: 2610 cyl, 255 heads, 63 sectors/track, start at 0
Jan 21 14:53:59 | sda: serial = 1234                
Jan 21 14:53:59 | sda: get_state
Jan 21 14:53:59 | sda: path checker = directio (internal default)
Jan 21 14:53:59 | sda: checker timeout = 30000 ms (sysfs setting)
Jan 21 14:53:59 | directio: starting new request
Jan 21 14:53:59 | directio: io finished 4096/0
Jan 21 14:53:59 | sda: state = up
Jan 21 14:53:59 | sda: uid_attribute = ID_SERIAL (internal default)
Jan 21 14:53:59 | sda: uid = QEMU_HARDDISK_1234 (udev)
Jan 21 14:53:59 | sda: detect_prio = 1 (config file default)
Jan 21 14:53:59 | sda: prio = const (internal default)
Jan 21 14:53:59 | sda: prio =  (internal default)
Jan 21 14:53:59 | sda: const prio = 1
Jan 21 14:53:59 | sr0: device node name blacklisted
Jan 21 14:53:59 | loop0: device node name blacklisted
Jan 21 14:53:59 | loop1: device node name blacklisted
Jan 21 14:53:59 | loop2: device node name blacklisted
Jan 21 14:53:59 | loop3: device node name blacklisted
Jan 21 14:53:59 | loop4: device node name blacklisted
Jan 21 14:53:59 | dm-0: device node name blacklisted
Jan 21 14:53:59 | dm-1: device node name blacklisted
Jan 21 14:53:59 | dm-2: device node name blacklisted
Jan 21 14:53:59 | dm-3: device node name blacklisted
Jan 21 14:53:59 | dm-4: device node name blacklisted
Jan 21 14:53:59 | dm-5: device node name blacklisted
===== paths list =====
uuid               hcil    dev dev_t pri dm_st chk_st vend/prod/rev           
QEMU_HARDDISK_1234 6:0:0:0 sda 8:0   1   undef ready  ATA     ,QEMU HARDDISK  
Jan 21 14:53:59 | sda: ownership set to QEMU_HARDDISK_1234
Jan 21 14:53:59 | sda: not found in pathvec
Jan 21 14:53:59 | sda: mask = 0xc
Jan 21 14:53:59 | sda: path state = running

Jan 21 14:53:59 | sda: get_state
Jan 21 14:53:59 | directio: starting new request
Jan 21 14:53:59 | directio: io finished 4096/0
Jan 21 14:53:59 | sda: state = up
Jan 21 14:53:59 | sda: const prio = 1
Jan 21 14:53:59 | QEMU_HARDDISK_1234: pgfailover = -1 (internal default)
Jan 21 14:53:59 | QEMU_HARDDISK_1234: pgpolicy = failover (internal default)
Jan 21 14:53:59 | QEMU_HARDDISK_1234: selector = service-time 0 (internal default)
Jan 21 14:53:59 | QEMU_HARDDISK_1234: features = 0 (config file default)
Jan 21 14:53:59 | QEMU_HARDDISK_1234: hwhandler = 0 (internal default)
Jan 21 14:53:59 | QEMU_HARDDISK_1234: rr_weight = 1 (internal default)
Jan 21 14:53:59 | QEMU_HARDDISK_1234: minio = 1 rq (config file default)
Jan 21 14:53:59 | QEMU_HARDDISK_1234: no_path_retry = -1 (config file default)
Jan 21 14:53:59 | QEMU_HARDDISK_1234: pg_timeout = NONE (internal default)
Jan 21 14:53:59 | QEMU_HARDDISK_1234: fast_io_fail_tmo = 5 (config file default)
Jan 21 14:53:59 | QEMU_HARDDISK_1234: dev_loss_tmo = 30 (config file default)
Jan 21 14:53:59 | QEMU_HARDDISK_1234: retain_attached_hw_handler = 1 (config file default)
Jan 21 14:53:59 | QEMU_HARDDISK_1234: remove queue_if_no_path from '0'
Jan 21 14:53:59 | QEMU_HARDDISK_1234: assembled map [0 0 1 1 service-time 0 1 1 8:0 1]
Jan 21 14:53:59 | QEMU_HARDDISK_1234: set ACT_CREATE (map does not exist)

== Here does the error appear on the console ==
([   71.071596] device-mapper: table: 253:6: multipath: error getting device)

Jan 21 14:53:59 | QEMU_HARDDISK_1234: domap (0) failure for create/reload map
Jan 21 14:53:59 | QEMU_HARDDISK_1234: ignoring map
Jan 21 14:53:59 | const prioritizer refcount 1
Jan 21 14:53:59 | directio checker refcount 1
Jan 21 14:53:59 | unloading const prioritizer
Jan 21 14:53:59 | unloading directio checker

Ben, can you explain that output?

Comment 15 Fabian Deutsch 2015-01-21 14:57:18 UTC
/etc/multipath/wwids is empty

Comment 16 Ben Marzinski 2015-01-21 16:12:29 UTC
Do you know if /etc/multipath.conf has find_multipaths set?  It doesn't look like it.  If it was, you should see some messages about why this device should be either allowed or not based on the find_multipaths rules.

Comment 17 Fabian Deutsch 2015-01-21 16:32:52 UTC
You are right, it is not set. perfect finding. Then it's an effect of bug 1173290

Comment 18 Fabian Deutsch 2015-01-26 10:05:35 UTC
TestOnly, because there is no code change required from our side.

Comment 19 Fabian Deutsch 2015-01-26 10:18:55 UTC
Does not look like this will be in for 3.5.0, moving this out.

Comment 20 Fabian Deutsch 2015-02-11 09:09:27 UTC
Reducing the priority, because it looks like a visual problem only.

Comment 21 Ying Cui 2015-05-11 12:19:01 UTC
Clarified this issue only happen on RHEV-H 7, did not exist on RHEV-H 6.

Tested on rhevh 7.0-20150127.0 3.5.0 GA version, and Tested on rhevh 7.1-20150420.0 3.5.1 GA version, both exist this issue.

Comment 22 Sam Yangsao 2015-05-18 14:46:04 UTC
Confirmed on RHEV-H 7.1 20150505

[root@rhev1 ~]# uname -a
Linux rhev1.yangsao.net 3.10.0-229.1.2.el7.x86_64 #1 SMP Fri Mar 6 17:12:08 EST 2015 x86_64 x86_64 x86_64 GNU/Linux

[root@rhev1 ~]# cat /etc/redhat-release
Red Hat Enterprise Virtualization Hypervisor 7.1 (20150505.0.el7ev)

[root@rhev1 ~]# dmesg |grep multi
[    1.330223] device-mapper: multipath: version 1.7.0 loaded
[  107.679155] device-mapper: multipath service-time: version 0.2.0 loaded
[  107.679801] device-mapper: table: 253:6: multipath: error getting device
[  133.191197] device-mapper: table: 253:9: multipath: error getting device
[  143.768872] device-mapper: table: 253:9: multipath: error getting device

Comment 23 Fabian Deutsch 2015-05-27 12:47:26 UTC
We can try to hide these error messages, which does not solve the problem, but the symptomps

Comment 24 Charlie Inglese 2015-12-09 22:56:02 UTC
(In reply to Fabian Deutsch from comment #23)
> We can try to hide these error messages, which does not solve the problem,
> but the symptomps

Fabian,
We are running into this issue as well, and would like to know how to suppress/hide these error messages. Can you provide some more details as to how we can do that on our end?

Comment 25 Fabian Deutsch 2015-12-14 08:26:23 UTC
This was not investigated, but these errors can maybe be hidden by adjusting the syslog configuration, the downside of this is that maybe other dm related errors will be hidden as well.

Comment 28 Ying Cui 2015-12-24 07:47:37 UTC
*** Bug 1293484 has been marked as a duplicate of this bug. ***

Comment 29 Ying Cui 2015-12-24 07:51:20 UTC
We have to reopen this bug due to customer ticket in bug 1293484.

Comment 30 Ying Cui 2016-03-02 03:01:52 UTC
Re-tested this issue on RHEV-H 3.6 RC1 build(rhev-hypervisor7-7.2-20160225.0), this bug is gone, did not exist in current 3.6 RC1 RHEV-H. 

Tested and confirmed on RHEV-H 3.5.8 build, the bug still exist.

So if we plan to not fix in 3.5.z, then we need to consider closed this bug as next release. Thanks.

Comment 31 Fabian Deutsch 2016-03-02 11:58:40 UTC
Thanks Ying.

We are not going to fix this bug in 3.5.z

Comment 32 Ying Cui 2016-03-03 12:53:57 UTC
According to comment 30, verified this bug on rhev-hypervisor7-7.2-20160225.0.

Comment 34 errata-xmlrpc 2016-03-09 14:25:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0378.html


Note You need to log in before you can comment on or make changes to this bug.