Bug 1372727 - Cannot find master domain
Summary: Cannot find master domain
Keywords:
Status: CLOSED UPSTREAM
Alias: None
Product: vdsm
Classification: oVirt
Component: SuperVDSM
Version: 4.16.27
Hardware: x86_64
OS: Linux
unspecified
medium vote
Target Milestone: ovirt-4.0.5
: ---
Assignee: Liron Aravot
QA Contact: Aharon Canan
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-09-02 13:44 UTC by Csiga
Modified: 2016-09-11 10:13 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-09-11 10:13:06 UTC
oVirt Team: Storage
tnisan: ovirt-4.0.z?
rule-engine: planning_ack?
rule-engine: devel_ack?
rule-engine: testing_ack?

Attachments (Terms of Use)
Debug log (33.66 KB, text/plain)
2016-09-02 13:44 UTC, Csiga 		no flags Details
Engine.log (51.09 KB, text/plain)
2016-09-05 09:35 UTC, Csiga 		no flags Details
Answer 1 (26.38 KB, image/png)
2016-09-05 11:03 UTC, Csiga 		no flags Details
Answer 2 (8.91 KB, image/png)
2016-09-05 11:03 UTC, Csiga 		no flags Details
Answer 3 (40.43 KB, image/png)
2016-09-05 11:04 UTC, Csiga 		no flags Details
View All Add an attachment (proposed patch, testcase, etc.)

Description Csiga 2016-09-02 13:44:00 UTC 
Created attachment 1197215 [details]
Debug log

Description of problem:

After reboot the vdsm can't connect to storage:

Host node1 cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Dataplex1. Setting Host state to Non-Operational.


VDSM node1 command failed: Cannot find master domain: 'spUUID=0b48ce08-1efb-4450-a76b-8f770abe1130, msdUUID=932612a9-a493-4134-b67b-607bd2cfecec'

Version-Release number of selected component (if applicable):

Centos 6
Vdsm 4.16.27

How reproducible:

I was uprading the the node and was a reboot on the host.

Steps to Reproduce:
1. Maintenance
2. yum update
3. Reboot

Additional info:

I attached the vdsm log.


Best Regards,
Cs
Comment 1 Liron Aravot 2016-09-05 06:56:44 UTC 
Please attach the engine.log of the ovirt-engine as well.
Comment 2 Csiga 2016-09-05 09:35:05 UTC 
Created attachment 1197829 [details]
Engine.log
Comment 3 Csiga 2016-09-05 09:35:53 UTC 
Dear Liron Aravot,

I have attached the engine log.

Best Regards,
Sandor Kalmanczi
Comment 4 Liron Aravot 2016-09-05 10:16:53 UTC 
Thanks Sandor,

The host has problem accessing the master storage domain (see [1]):
1. How many hosts do you have in that data center?
2. what is the status of the master storage domain?
3. what is the master storage domain storage type?

thanks,
Liron

[1] Thread-541::DEBUG::2016-09-02 15:20:03,868::blockSD::336::Storage.Misc.excCmd::(readlines) /bin/dd iflag=direct skip=0 bs=2048 if=/dev/932612a9-a493-4134-b67b-607bd2cfecec/metadata count=1 (cwd None)
Thread-541::DEBUG::2016-09-02 15:20:03,879::blockSD::336::Storage.Misc.excCmd::(readlines) FAILED: <err> = "/bin/dd: opening `/dev/932612a9-a493-4134-b67b-607bd2cfecec/metadata': Permission denied\n"; <rc> = 1
Thread-541::ERROR::2016-09-02 15:20:03,879::sdc::169::Storage.StorageDomainCache::(_findUnfetchedDomain) Error while looking for domain `932612a9-a493-4134-b67b-607bd2cfecec`
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 164, in _findUnfetchedDomain
    return mod.findDomain(sdUUID)
  File "/usr/share/vdsm/storage/blockSD.py", line 1360, in findDomain
    return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/blockSD.py", line 403, in __init__
    metadata = selectMetadata(sdUUID)
  File "/usr/share/vdsm/storage/blockSD.py", line 374, in selectMetadata
    if len(mdProvider) > 0:
  File "/usr/share/vdsm/storage/persistentDict.py", line 65, in __len__
    return len(self.keys())
  File "/usr/share/vdsm/storage/persistentDict.py", line 110, in keys
    return list(self.__iter__())
  File "/usr/share/vdsm/storage/persistentDict.py", line 107, in __iter__
    self._dict.__iter__())
  File "/usr/share/vdsm/storage/persistentDict.py", line 225, in __iter__
    with self._accessWrapper():
  File "/usr/lib64/python2.6/contextlib.py", line 16, in __enter__
    return self.gen.next()
  File "/usr/share/vdsm/storage/persistentDict.py", line 152, in _accessWrapper
    self.refresh()
  File "/usr/share/vdsm/storage/persistentDict.py", line 230, in refresh
    lines = self._metaRW.readlines()
  File "/usr/share/vdsm/storage/blockSD.py", line 336, in readlines
    m = misc.readblock(self.metavol, self._offset, self._size)
  File "/usr/share/vdsm/storage/misc.py", line 251, in readblock
    raise se.MiscBlockReadException(name, offset, size)
MiscBlockReadException: Internal block device read failure: 'name=/dev/932612a9-a493-4134-b67b-607bd2cfecec/metadata, offset=0, size=2048'
Thread-542::DEBUG::2016-09-02 15:20:03,882::__init__::298::IOProcessClient::(_run) Starting IOProcess...
Comment 5 Csiga 2016-09-05 11:03:24 UTC 
Created attachment 1197873 [details]
Answer 1
Comment 6 Csiga 2016-09-05 11:03:50 UTC 
Created attachment 1197874 [details]
Answer 2
Comment 7 Csiga 2016-09-05 11:04:17 UTC 
Created attachment 1197875 [details]
Answer 3
Comment 8 Csiga 2016-09-05 11:06:27 UTC 
Dear Liron,

1. How many hosts do you have in that data center?
I have four hosts in that data center.

2. what is the status of the master storage domain?
Status of the master storage is "Active".

3. what is the master storage domain storage type?
Type of the master storage is "Fibre Channel".

I have attached more information.

Best Regards,
Sandor Kalmanczi
Comment 9 Csiga 2016-09-09 08:05:25 UTC 
Dear Liron,

There was a CentOS update:

And this update solved this problem:

Loaded plugins: fastestmirror, rhnplugin
This system is receiving updates from RHN Classic or Red Hat Satellite.
Setting up Update Process
Loading mirror speeds from cached hostfile
* base: ftp.freepark.org
* epel: mirror.atomki.mta.hu
* extras: ftp.freepark.org
* ovirt-3.6: mirror.slu.cz
* ovirt-3.6-epel: mirror.atomki.mta.hu
* ovirt-3.6-jpackage-6.0-generic: sunsite.informatik.rwth-aachen.de
* updates: ftp.freepark.org
Resolving Dependencies
--> Running transaction check
---> Package glusterfs.x86_64 0:3.7.13-1.el6 will be updated
---> Package glusterfs.x86_64 0:3.7.14-1.el6 will be an update
---> Package glusterfs-api.x86_64 0:3.7.13-1.el6 will be updated
---> Package glusterfs-api.x86_64 0:3.7.14-1.el6 will be an update
---> Package glusterfs-cli.x86_64 0:3.7.13-1.el6 will be updated
---> Package glusterfs-cli.x86_64 0:3.7.14-1.el6 will be an update
---> Package glusterfs-client-xlators.x86_64 0:3.7.13-1.el6 will be updated
---> Package glusterfs-client-xlators.x86_64 0:3.7.14-1.el6 will be an update
---> Package glusterfs-fuse.x86_64 0:3.7.13-1.el6 will be updated
---> Package glusterfs-fuse.x86_64 0:3.7.14-1.el6 will be an update
---> Package glusterfs-libs.x86_64 0:3.7.13-1.el6 will be updated
---> Package glusterfs-libs.x86_64 0:3.7.14-1.el6 will be an update
---> Package glusterfs-rdma.x86_64 0:3.7.13-1.el6 will be updated
---> Package glusterfs-rdma.x86_64 0:3.7.14-1.el6 will be an update
---> Package libgudev1.x86_64 0:147-2.73.el6_8.1 will be updated
---> Package libgudev1.x86_64 0:147-2.73.el6_8.2 will be an update
---> Package libudev.x86_64 0:147-2.73.el6_8.1 will be updated
---> Package libudev.x86_64 0:147-2.73.el6_8.2 will be an update
---> Package udev.x86_64 0:147-2.73.el6_8.1 will be updated
---> Package udev.x86_64 0:147-2.73.el6_8.2 will be an update
epel/filelists_db | 7.9 MB 00:00
ovirt-3.6-centos-gluster37/filelists_db | 29 kB 00:00
ovirt-3.6-epel/filelists_db | 7.9 MB 00:00
updates/filelists_db | 1.5 MB 00:00
--> Finished Dependency Resolution

Dependencies Resolved

===================================================================================================================================================================
Package Arch Version Repository Size
===================================================================================================================================================================
Updating:
glusterfs x86_64 3.7.14-1.el6 ovirt-3.6-centos-gluster37 444 k
glusterfs-api x86_64 3.7.14-1.el6 ovirt-3.6-centos-gluster37 79 k
glusterfs-cli x86_64 3.7.14-1.el6 ovirt-3.6-centos-gluster37 164 k
glusterfs-client-xlators x86_64 3.7.14-1.el6 ovirt-3.6-centos-gluster37 1.2 M
glusterfs-fuse x86_64 3.7.14-1.el6 ovirt-3.6-centos-gluster37 123 k
glusterfs-libs x86_64 3.7.14-1.el6 ovirt-3.6-centos-gluster37 336 k
glusterfs-rdma x86_64 3.7.14-1.el6 ovirt-3.6-centos-gluster37 57 k
libgudev1 x86_64 147-2.73.el6_8.2 updates 65 k
libudev x86_64 147-2.73.el6_8.2 updates 78 k
udev x86_64 147-2.73.el6_8.2 updates 358 k

Transaction Summary
===================================================================================================================================================================
Upgrade 10 Package(s)

Total download size: 2.8 M
Is this ok [y/N]: y
Downloading Packages:
(1/10): glusterfs-3.7.14-1.el6.x86_64.rpm | 444 kB 00:00
(2/10): glusterfs-api-3.7.14-1.el6.x86_64.rpm | 79 kB 00:00
(3/10): glusterfs-cli-3.7.14-1.el6.x86_64.rpm | 164 kB 00:00
(4/10): glusterfs-client-xlators-3.7.14-1.el6.x86_64.rpm | 1.2 MB 00:00
(5/10): glusterfs-fuse-3.7.14-1.el6.x86_64.rpm | 123 kB 00:00
(6/10): glusterfs-libs-3.7.14-1.el6.x86_64.rpm | 336 kB 00:00
(7/10): glusterfs-rdma-3.7.14-1.el6.x86_64.rpm | 57 kB 00:00
(8/10): libgudev1-147-2.73.el6_8.2.x86_64.rpm | 65 kB 00:00
(9/10): libudev-147-2.73.el6_8.2.x86_64.rpm | 78 kB 00:00
(10/10): udev-147-2.73.el6_8.2.x86_64.rpm | 358 kB 00:00
-------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 1.5 MB/s | 2.8 MB 00:01
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Updating : glusterfs-libs-3.7.14-1.el6.x86_64 1/20
Updating : glusterfs-3.7.14-1.el6.x86_64 2/20
Updating : glusterfs-client-xlators-3.7.14-1.el6.x86_64 3/20
Updating : libudev-147-2.73.el6_8.2.x86_64 4/20
Updating : libgudev1-147-2.73.el6_8.2.x86_64 5/20
Updating : glusterfs-api-3.7.14-1.el6.x86_64 6/20
Updating : glusterfs-fuse-3.7.14-1.el6.x86_64 7/20
Updating : glusterfs-rdma-3.7.14-1.el6.x86_64 8/20
Updating : glusterfs-cli-3.7.14-1.el6.x86_64 9/20
Updating : udev-147-2.73.el6_8.2.x86_64 10/20
Cleanup : glusterfs-fuse-3.7.13-1.el6.x86_64 11/20
Cleanup : glusterfs-api-3.7.13-1.el6.x86_64 12/20
Cleanup : glusterfs-rdma-3.7.13-1.el6.x86_64 13/20
Cleanup : glusterfs-3.7.13-1.el6.x86_64 14/20
Cleanup : glusterfs-client-xlators-3.7.13-1.el6.x86_64 15/20
Cleanup : glusterfs-cli-3.7.13-1.el6.x86_64 16/20
Cleanup : libgudev1-147-2.73.el6_8.1.x86_64 17/20
Cleanup : glusterfs-libs-3.7.13-1.el6.x86_64 18/20
Cleanup : libudev-147-2.73.el6_8.1.x86_64 19/20
Cleanup : udev-147-2.73.el6_8.1.x86_64 20/20
Verifying : glusterfs-client-xlators-3.7.14-1.el6.x86_64 1/20
Verifying : libudev-147-2.73.el6_8.2.x86_64 2/20
Verifying : glusterfs-api-3.7.14-1.el6.x86_64 3/20
Verifying : udev-147-2.73.el6_8.2.x86_64 4/20
Verifying : glusterfs-libs-3.7.14-1.el6.x86_64 5/20
Verifying : glusterfs-rdma-3.7.14-1.el6.x86_64 6/20
Verifying : glusterfs-fuse-3.7.14-1.el6.x86_64 7/20
Verifying : glusterfs-cli-3.7.14-1.el6.x86_64 8/20
Verifying : libgudev1-147-2.73.el6_8.2.x86_64 9/20
Verifying : glusterfs-3.7.14-1.el6.x86_64 10/20
Verifying : glusterfs-libs-3.7.13-1.el6.x86_64 11/20
Verifying : glusterfs-client-xlators-3.7.13-1.el6.x86_64 12/20
Verifying : glusterfs-3.7.13-1.el6.x86_64 13/20
Verifying : libudev-147-2.73.el6_8.1.x86_64 14/20
Verifying : glusterfs-cli-3.7.13-1.el6.x86_64 15/20
Verifying : glusterfs-api-3.7.13-1.el6.x86_64 16/20
Verifying : udev-147-2.73.el6_8.1.x86_64 17/20
Verifying : libgudev1-147-2.73.el6_8.1.x86_64 18/20
Verifying : glusterfs-fuse-3.7.13-1.el6.x86_64 19/20
Verifying : glusterfs-rdma-3.7.13-1.el6.x86_64 20/20

Updated:
glusterfs.x86_64 0:3.7.14-1.el6 glusterfs-api.x86_64 0:3.7.14-1.el6 glusterfs-cli.x86_64 0:3.7.14-1.el6 glusterfs-client-xlators.x86_64 0:3.7.14-1.el6
glusterfs-fuse.x86_64 0:3.7.14-1.el6 glusterfs-libs.x86_64 0:3.7.14-1.el6 glusterfs-rdma.x86_64 0:3.7.14-1.el6 libgudev1.x86_64 0:147-2.73.el6_8.2
libudev.x86_64 0:147-2.73.el6_8.2 udev.x86_64 0:147-2.73.el6_8.2

Best Regards,
Sandor Kalmanczi
Comment 10 Allon Mureinik 2016-09-11 10:13:06 UTC 
Closing, based on this comment.
Note You need to log in before you can comment on or make changes to this bug.