Bug 679423

Summary: RHEV Manager: Can't attach/detach additional Storage Domain
Product: Red Hat Enterprise Linux 6 Reporter: Idan Mansano <imansano>
Component: device-mapper-multipathAssignee: LVM and device-mapper development team <lvm-team>
Status: CLOSED WORKSFORME QA Contact: Red Hat Kernel QE team <kernel-qe>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 6.1CC: abaron, acathrow, agk, bazulay, bmarzins, danken, dkenigsb, dwysocha, heinzm, iheim, mbroz, prajnoha, prockai, zkabelac
Target Milestone: rcKeywords: TestBlocker
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-03-10 20:05:39 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 565939    
Attachments:
Description Flags
vdsm.log
none
log from /var/log/messages
none
dmesg none

Description Idan Mansano 2011-02-22 14:35:05 UTC
Created attachment 480147 [details]
vdsm.log

Description of problem:
When attaching/detaching additional storage domain to an existing Data Center, the Data Center crashes, and we see various errors on the host.

Version-Release number of selected component (if applicable):
RHEL 6.1
Kernel-2.6.32-117.el6.x86_64
vdsm-4.9-47.el6.x86_64
device-mapper-multipath-0.4.9-38.el6.x86_64
lvm2-2.02.83-2.el6.x86_64
RHEVM - Build IC96

Steps to Reproduce:
RHEVM Flow:
1. Create iSCSI Storage domain, and attach it to Data center.
2. Create another iSCSI storage domain, and attach it to the same Data Center.
3. The Data center will crash, and kernel errors will be shown in /var/log/messages
  
Additional info:
1. Attached LOGS.
2. For further information, please contact Dan Kenigsberg.

Comment 1 Idan Mansano 2011-02-22 14:37:01 UTC
Created attachment 480148 [details]
log from /var/log/messages

Comment 2 Idan Mansano 2011-02-22 14:37:28 UTC
Created attachment 480149 [details]
dmesg

Comment 4 Ben Marzinski 2011-02-23 19:04:38 UTC
I'm trying to figure out what's going on here.  I don't know what kernel errors you are talking about.  There are 30 hours of logs, so it's quite possible that I missed them.  If you could give me some pointers to when you tried to attach the other storage domain, so I knew where to look for the errors, that would be a big help. There are a lot of multipath messages, but I didn't see any that looked obviously problematic without knowing more details.

For instance, all those

Feb 22 12:10:02 nari12 multipathd: dm-11: remove map (uevent)
Feb 22 12:10:02 nari12 multipathd: dm-11: devmap not registered, can't remove

are udev ending remove messages for a device that multipathd has already removed. I don't know why udev sends so many remove uevents, but sometimes is just seems to.

Messages like:

Feb 22 12:10:02 nari12 kernel: device-mapper: table: 253:11: multipath: error getting device

Ususally happen because multipath hasn't blacklisted some device that is already in use, and so it's trying to create a multipath device on it, but failing since the device is already in use.

There's lots of other messages which could be errors if something's happening that's not supposed to, but mostly they are just notification messages that a path  came up or down, or was added or removed.

The vdsm log is 8 megs long, and I'm not sure what I'm looking for in it.

Comment 6 Idan Mansano 2011-02-27 15:30:16 UTC
hmm, with a newer kernel, this bug was not reproduced...

kernel-2.6.32-118.el6.x86_64
iscsi-initiator-utils-6.2.0.872-17.el6.x86_64
device-mapper-multipath-0.4.9-38.el6.x86_64

Comment 7 Ayal Baron 2011-03-10 20:05:39 UTC
(In reply to comment #6)
> hmm, with a newer kernel, this bug was not reproduced...

Closing.

> 
> kernel-2.6.32-118.el6.x86_64
> iscsi-initiator-utils-6.2.0.872-17.el6.x86_64
> device-mapper-multipath-0.4.9-38.el6.x86_64