Description: RHEVH 7.0 installed successful, and while login rhevh 7.0, there display error messages, not sure whether impact function or not. ======= Red Hat Enterprise Virtualization Hypervisor release 7.0 (20150106.0.el7ev) Kernel 3.10.0-123.13.2.el7.x86_64 on an x86_64 (ttyS0) Please login as 'admin' to configure the node localhost login: [ 40.781316] device-mapper: multipath service-time: version 0.2.0 loaded [ 40.788514] device-mapper: table: 253:6: multipath: error getting device [ 40.795228] device-mapper: ioctl: error adding target to table [ 40.803269] device-mapper: table: 253:6: multipath: error getting device [ 40.809979] device-mapper: ioctl: error adding target to table [ 77.192223] systemd-readahead[746]: Failed to open pack file: Read-only file system ======= Test version: # rpm -q kernel device-mapper ovirt-node dracut systemd device-mapper-multipath kernel-3.10.0-123.13.2.el7.x86_64 device-mapper-1.02.84-14.el7.x86_64 ovirt-node-3.1.0-0.40.20150105git69f34a6.el7.noarch dracut-033-161.el7_0.173.x86_64 systemd-208-11.el7_0.5.x86_64 device-mapper-multipath-0.4.9-66.el7.x86_64 # cat /etc/system-release Red Hat Enterprise Virtualization Hypervisor release 7.0 (20150106.0.el7ev) Test steps: 1. installed rhevh via TUI. 2. after rhevh boot, can see _login_ Actual result: There display device-mapper multipath: error getting device. Expected result: Do not display such error in rhevh 7.0 login.
Created attachment 979947 [details] varlog
Created attachment 979948 [details] screen_output
Created attachment 979949 [details] sosreport
Till now we have not identified the obvious effect on rhevh real function. but this error display on rhevh login page, that is not good for user experience. consider GA blocker.
Created attachment 979967 [details] part sof dmesg and /dev The screenshot shows that mpath is complaining about a device which does not exist (according to the minor:major number).
Ben, have you seen this error before?
What's happening here is that multipath is configured to use some paths, but for some reason it can't created a device with them (most likely because they are already in use). Unfortunately, when the kernel prints this message, it doesn't have the information on which paths those where any more. The reason that a non-existent dm device is being shown is due to how device mapper works. First it creates the dm device, 253:6 in this case. Then it tries to load the table. If the table load fails, multipath will remove the useless dm device. But for a brief bit, there was a dm device 253:6. I will look in the logs to see if I can figure out what device multipath was trying to load. If I can't find anything useful in the logs, you may be able to find out by simply running # multipath -v3 If the devices are still there and still in use by something other than multipath, you should see the same messages in the log, and multipath will tell you that it is ignoring the device because it is already in use.
Thanks Ben. It looks like this is yet another symptom of the build problem we were seeing with our latest build. Basically the initrd multipath was wrong, which also lead to a in general broken multipath. I can not reproduce this anymore on a build containing the fixes.
*** This bug has been marked as a duplicate of bug 1182551 ***
The bug 1182551 is fixed on rhevh 7.0 20150119.0.1.el7ev build yet. But we still encounter this bug on 20150119.0.1.el7ev build, so I have to reopen this bug to more investigation. Red Hat Enterprise Virtualization Hypervisor release 7.0 (20150119.0.1.el7ev) Kernel 3.10.0-123.13.2.el7.x86_64 on an x86_64 (ttyS0) Please login as 'admin' to configure the node localhost login: [ 57.795013] e1000e: em1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None [ 57.802465] IPv6: ADDRCONF(NETDEV_CHANGE): em1: link becomes ready [ 71.064811] device-mapper: multipath service-time: version 0.2.0 loaded [ 71.071596] device-mapper: table: 253:6: multipath: error getting device [ 71.078311] device-mapper: ioctl: error adding target to table [ 100.654467] systemd-readahead[816]: Failed to open pack file: Read-only file system # rpm -q ovirt-node dracut systemd device-mapper-multipath device-mapper ovirt-node-3.2.1-5.el7.noarch dracut-033-161.el7_0.173.x86_64 systemd-208-11.el7_0.6.x86_64 device-mapper-multipath-0.4.9-66.el7.x86_64 device-mapper-1.02.84-14.el7.x86_64
Created attachment 982277 [details] varlog_20150119 already move quiet and add rd.debug in grub then capture this var/log
Created attachment 982278 [details] sosreport_20150119
Created attachment 982279 [details] screenoutput_20150119
It took some time to figur it out. multipath -v3 does trigger the error, but the erorr is only printed on the console, not on a ssh terminal: [root@alhost ~]# multipath -v3 Jan 21 14:53:59 | multipath.conf +5, invalid keyword: getuid_callout Jan 21 14:53:59 | multipath.conf +18, invalid keyword: getuid_callout Jan 21 14:53:59 | multipath.conf +37, invalid keyword: getuid_callout Jan 21 14:53:59 | loading /lib64/multipath/libcheckdirectio.so checker Jan 21 14:53:59 | loading /lib64/multipath/libprioconst.so prioritizer Jan 21 14:53:59 | sda: not found in pathvec Jan 21 14:53:59 | sda: mask = 0x3f Jan 21 14:53:59 | sda: dev_t = 8:0 Jan 21 14:53:59 | sda: size = 41943040 Jan 21 14:53:59 | sda: vendor = ATA Jan 21 14:53:59 | sda: product = QEMU HARDDISK Jan 21 14:53:59 | sda: rev = 2.1. Jan 21 14:53:59 | sda: h:b:t:l = 6:0:0:0 Jan 21 14:53:59 | sda: tgt_node_name = ata-7.00 Jan 21 14:53:59 | sda: path state = running Jan 21 14:53:59 | sda: 2610 cyl, 255 heads, 63 sectors/track, start at 0 Jan 21 14:53:59 | sda: serial = 1234 Jan 21 14:53:59 | sda: get_state Jan 21 14:53:59 | sda: path checker = directio (internal default) Jan 21 14:53:59 | sda: checker timeout = 30000 ms (sysfs setting) Jan 21 14:53:59 | directio: starting new request Jan 21 14:53:59 | directio: io finished 4096/0 Jan 21 14:53:59 | sda: state = up Jan 21 14:53:59 | sda: uid_attribute = ID_SERIAL (internal default) Jan 21 14:53:59 | sda: uid = QEMU_HARDDISK_1234 (udev) Jan 21 14:53:59 | sda: detect_prio = 1 (config file default) Jan 21 14:53:59 | sda: prio = const (internal default) Jan 21 14:53:59 | sda: prio = (internal default) Jan 21 14:53:59 | sda: const prio = 1 Jan 21 14:53:59 | sr0: device node name blacklisted Jan 21 14:53:59 | loop0: device node name blacklisted Jan 21 14:53:59 | loop1: device node name blacklisted Jan 21 14:53:59 | loop2: device node name blacklisted Jan 21 14:53:59 | loop3: device node name blacklisted Jan 21 14:53:59 | loop4: device node name blacklisted Jan 21 14:53:59 | dm-0: device node name blacklisted Jan 21 14:53:59 | dm-1: device node name blacklisted Jan 21 14:53:59 | dm-2: device node name blacklisted Jan 21 14:53:59 | dm-3: device node name blacklisted Jan 21 14:53:59 | dm-4: device node name blacklisted Jan 21 14:53:59 | dm-5: device node name blacklisted ===== paths list ===== uuid hcil dev dev_t pri dm_st chk_st vend/prod/rev QEMU_HARDDISK_1234 6:0:0:0 sda 8:0 1 undef ready ATA ,QEMU HARDDISK Jan 21 14:53:59 | sda: ownership set to QEMU_HARDDISK_1234 Jan 21 14:53:59 | sda: not found in pathvec Jan 21 14:53:59 | sda: mask = 0xc Jan 21 14:53:59 | sda: path state = running Jan 21 14:53:59 | sda: get_state Jan 21 14:53:59 | directio: starting new request Jan 21 14:53:59 | directio: io finished 4096/0 Jan 21 14:53:59 | sda: state = up Jan 21 14:53:59 | sda: const prio = 1 Jan 21 14:53:59 | QEMU_HARDDISK_1234: pgfailover = -1 (internal default) Jan 21 14:53:59 | QEMU_HARDDISK_1234: pgpolicy = failover (internal default) Jan 21 14:53:59 | QEMU_HARDDISK_1234: selector = service-time 0 (internal default) Jan 21 14:53:59 | QEMU_HARDDISK_1234: features = 0 (config file default) Jan 21 14:53:59 | QEMU_HARDDISK_1234: hwhandler = 0 (internal default) Jan 21 14:53:59 | QEMU_HARDDISK_1234: rr_weight = 1 (internal default) Jan 21 14:53:59 | QEMU_HARDDISK_1234: minio = 1 rq (config file default) Jan 21 14:53:59 | QEMU_HARDDISK_1234: no_path_retry = -1 (config file default) Jan 21 14:53:59 | QEMU_HARDDISK_1234: pg_timeout = NONE (internal default) Jan 21 14:53:59 | QEMU_HARDDISK_1234: fast_io_fail_tmo = 5 (config file default) Jan 21 14:53:59 | QEMU_HARDDISK_1234: dev_loss_tmo = 30 (config file default) Jan 21 14:53:59 | QEMU_HARDDISK_1234: retain_attached_hw_handler = 1 (config file default) Jan 21 14:53:59 | QEMU_HARDDISK_1234: remove queue_if_no_path from '0' Jan 21 14:53:59 | QEMU_HARDDISK_1234: assembled map [0 0 1 1 service-time 0 1 1 8:0 1] Jan 21 14:53:59 | QEMU_HARDDISK_1234: set ACT_CREATE (map does not exist) == Here does the error appear on the console == ([ 71.071596] device-mapper: table: 253:6: multipath: error getting device) Jan 21 14:53:59 | QEMU_HARDDISK_1234: domap (0) failure for create/reload map Jan 21 14:53:59 | QEMU_HARDDISK_1234: ignoring map Jan 21 14:53:59 | const prioritizer refcount 1 Jan 21 14:53:59 | directio checker refcount 1 Jan 21 14:53:59 | unloading const prioritizer Jan 21 14:53:59 | unloading directio checker Ben, can you explain that output?
/etc/multipath/wwids is empty
Do you know if /etc/multipath.conf has find_multipaths set? It doesn't look like it. If it was, you should see some messages about why this device should be either allowed or not based on the find_multipaths rules.
You are right, it is not set. perfect finding. Then it's an effect of bug 1173290
TestOnly, because there is no code change required from our side.
Does not look like this will be in for 3.5.0, moving this out.
Reducing the priority, because it looks like a visual problem only.
Clarified this issue only happen on RHEV-H 7, did not exist on RHEV-H 6. Tested on rhevh 7.0-20150127.0 3.5.0 GA version, and Tested on rhevh 7.1-20150420.0 3.5.1 GA version, both exist this issue.
Confirmed on RHEV-H 7.1 20150505 [root@rhev1 ~]# uname -a Linux rhev1.yangsao.net 3.10.0-229.1.2.el7.x86_64 #1 SMP Fri Mar 6 17:12:08 EST 2015 x86_64 x86_64 x86_64 GNU/Linux [root@rhev1 ~]# cat /etc/redhat-release Red Hat Enterprise Virtualization Hypervisor 7.1 (20150505.0.el7ev) [root@rhev1 ~]# dmesg |grep multi [ 1.330223] device-mapper: multipath: version 1.7.0 loaded [ 107.679155] device-mapper: multipath service-time: version 0.2.0 loaded [ 107.679801] device-mapper: table: 253:6: multipath: error getting device [ 133.191197] device-mapper: table: 253:9: multipath: error getting device [ 143.768872] device-mapper: table: 253:9: multipath: error getting device
We can try to hide these error messages, which does not solve the problem, but the symptomps
(In reply to Fabian Deutsch from comment #23) > We can try to hide these error messages, which does not solve the problem, > but the symptomps Fabian, We are running into this issue as well, and would like to know how to suppress/hide these error messages. Can you provide some more details as to how we can do that on our end?
This was not investigated, but these errors can maybe be hidden by adjusting the syslog configuration, the downside of this is that maybe other dm related errors will be hidden as well.
*** Bug 1293484 has been marked as a duplicate of this bug. ***
We have to reopen this bug due to customer ticket in bug 1293484.
Re-tested this issue on RHEV-H 3.6 RC1 build(rhev-hypervisor7-7.2-20160225.0), this bug is gone, did not exist in current 3.6 RC1 RHEV-H. Tested and confirmed on RHEV-H 3.5.8 build, the bug still exist. So if we plan to not fix in 3.5.z, then we need to consider closed this bug as next release. Thanks.
Thanks Ying. We are not going to fix this bug in 3.5.z
According to comment 30, verified this bug on rhev-hypervisor7-7.2-20160225.0.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-0378.html