Description of problem: After storage problem on host, host not show correct metadata Version-Release number of selected component (if applicable): ovirt-host-deploy-1.4.0-1.el7ev.noarch How reproducible: Always Steps to Reproduce: 1. Deploy hosted-engine on two hosts 2. Block connection to storage from host with engine vm(iptables -I INPUT -s ip_of_storage -j DROP) 3. Wait until vm will start on second host 4. Restore connection to storage on first host(iptables -D INPUT 1) 5. Wait 10 minute Actual results: Output of hosted-engine vm status on first host =============================================== --== Host 1 status ==-- Status up-to-date : False Hostname : cyan-vdsf.qa.lab.tlv.redhat.com Host ID : 1 Engine status : unknown stale-data Score : 3400 stopped : False Local maintenance : False crc32 : f55d2339 Host timestamp : 86510 --== Host 2 status ==-- Status up-to-date : True Hostname : rose05.qa.lab.tlv.redhat.com Host ID : 2 Engine status : {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "paused"} Score : 3000 stopped : False Local maintenance : False crc32 : dbde7db1 Host timestamp : 87326 Output of hosted-engine vm status on second host =============================================== --== Host 1 status ==-- Status up-to-date : True Hostname : cyan-vdsf.qa.lab.tlv.redhat.com Host ID : 1 Engine status : {"health": "good", "vm": "up", "detail": "up"} Score : 3348 stopped : False Local maintenance : False crc32 : ff8239d4 Host timestamp : 86947 --== Host 2 status ==-- Status up-to-date : True Hostname : rose05.qa.lab.tlv.redhat.com Host ID : 2 Engine status : {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "paused"} Score : 3000 stopped : False Local maintenance : False crc32 : 14c95eef Host timestamp : 87337 Expected results: Same correct output from two hosts and also paused vm from first host must be killed Additional info: you can start looking from 2015-11-02 11:10
Created attachment 1088516 [details] first host logs
Created attachment 1088517 [details] second host logs
It happen to me on ISCSI storage.
The second host (rose05) can access the storage and does see the data. It just thinks the first host is dead: MainThread::INFO::2015-11-02 13:14:50,097::state_machine::171::ovirt_hosted_engi ne_ha.agent.hosted_engine.HostedEngine::(refresh) Host cyan-vdsf.qa.lab.tlv.redh at.com (id 1): {'hostname': 'cyan-vdsf.qa.lab.tlv.redhat.com', 'alive': False, ' host-id': 1, 'engine-status': {'health': 'good', 'vm': 'up', 'detail': 'up'}, 's core': 3400, 'stopped': False, 'maintenance': False, 'crc32': '961b357c', 'host- ts': 87361} The important field here is 'alive': False. No idea why though.
Whiteboard?
This bug is not marked for z-stream, yet the milestone is for a z-stream version, therefore the milestone has been reset. Please set the correct milestone or add the z-stream flag.
So this is actually pretty weird, the writes from cyan are not visible to rose, but the writes from rose are visible to cyan. Both hosts see their own writes just fine. Cyan (host no 1): [root@cyan-vdsf ~]# hexdump -C /rhev/data-center/mnt/blockSD/d38153c3-0cfa-452f-9619-adadc00e27d7/ha_agent/hosted-engine.metadata 00000000 6d 61 69 6e 74 65 6e 61 6e 63 65 3d 30 00 00 00 |maintenance=0...| 00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00001000 31 7c 31 7c 31 38 39 39 38 37 7c 31 7c 33 34 30 |1|1|189987|1|340| 00001010 30 7c 7b 22 68 65 61 6c 74 68 22 3a 20 22 67 6f |0|{"health": "go| 00001020 6f 64 22 2c 20 22 76 6d 22 3a 20 22 75 70 22 2c |od", "vm": "up",| 00001030 20 22 64 65 74 61 69 6c 22 3a 20 22 75 70 22 7d | "detail": "up"}| 00001040 7c 63 79 61 6e 2d 76 64 73 66 2e 71 61 2e 6c 61 ||cyan-vdsf.qa.la| 00001050 62 2e 74 6c 76 2e 72 65 64 68 61 74 2e 63 6f 6d |b.tlv.redhat.com| 00001060 7c 30 7c 30 7c 38 63 30 37 34 37 62 61 00 00 00 ||0|0|8c0747ba...| 00001070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00001200 6d 65 74 61 64 61 74 61 5f 70 61 72 73 65 5f 76 |metadata_parse_v| 00001210 65 72 73 69 6f 6e 3d 31 0a 6d 65 74 61 64 61 74 |ersion=1.metadat| 00001220 61 5f 66 65 61 74 75 72 65 5f 76 65 72 73 69 6f |a_feature_versio| 00001230 6e 3d 31 0a 74 69 6d 65 73 74 61 6d 70 3d 31 38 |n=1.timestamp=18| 00001240 39 39 38 37 20 28 54 75 65 20 4e 6f 76 20 20 33 |9987 (Tue Nov 3| 00001250 20 31 36 3a 30 37 3a 33 30 20 32 30 31 35 29 0a | 16:07:30 2015).| 00001260 68 6f 73 74 2d 69 64 3d 31 0a 73 63 6f 72 65 3d |host-id=1.score=| 00001270 33 34 30 30 0a 6d 61 69 6e 74 65 6e 61 6e 63 65 |3400.maintenance| 00001280 3d 46 61 6c 73 65 0a 73 74 61 74 65 3d 45 6e 67 |=False.state=Eng| 00001290 69 6e 65 55 70 0a 73 74 6f 70 70 65 64 3d 46 61 |ineUp.stopped=Fa| 000012a0 6c 73 65 0a 00 00 00 00 00 00 00 00 00 00 00 00 |lse.............| 000012b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00002000 31 7c 31 7c 31 39 30 33 38 37 7c 32 7c 33 30 30 |1|1|190387|2|300| 00002010 30 7c 7b 22 72 65 61 73 6f 6e 22 3a 20 22 53 74 |0|{"reason": "St| 00002020 6f 72 61 67 65 20 6f 66 20 56 4d 20 69 73 20 6c |orage of VM is l| 00002030 6f 63 6b 65 64 2e 20 49 73 20 61 6e 6f 74 68 65 |ocked. Is anothe| 00002040 72 20 68 6f 73 74 20 61 6c 72 65 61 64 79 20 73 |r host already s| 00002050 74 61 72 74 69 6e 67 20 74 68 65 20 56 4d 3f 22 |tarting the VM?"| 00002060 2c 20 22 68 65 61 6c 74 68 22 3a 20 22 62 61 64 |, "health": "bad| 00002070 22 2c 20 22 76 6d 22 3a 20 22 61 6c 72 65 61 64 |", "vm": "alread| 00002080 79 5f 6c 6f 63 6b 65 64 22 2c 20 22 64 65 74 61 |y_locked", "deta| 00002090 69 6c 22 3a 20 22 64 6f 77 6e 22 7d 7c 72 6f 73 |il": "down"}|ros| 000020a0 65 30 35 2e 71 61 2e 6c 61 62 2e 74 6c 76 2e 72 |e05.qa.lab.tlv.r| 000020b0 65 64 68 61 74 2e 63 6f 6d 7c 30 7c 30 7c 65 66 |edhat.com|0|0|ef| 000020c0 32 34 30 34 33 36 00 00 00 00 00 00 00 00 00 00 |240436..........| 000020d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00002200 6d 65 74 61 64 61 74 61 5f 70 61 72 73 65 5f 76 |metadata_parse_v| 00002210 65 72 73 69 6f 6e 3d 31 0a 6d 65 74 61 64 61 74 |ersion=1.metadat| 00002220 61 5f 66 65 61 74 75 72 65 5f 76 65 72 73 69 6f |a_feature_versio| 00002230 6e 3d 31 0a 74 69 6d 65 73 74 61 6d 70 3d 31 39 |n=1.timestamp=19| 00002240 30 33 38 37 20 28 54 75 65 20 4e 6f 76 20 20 33 |0387 (Tue Nov 3| 00002250 20 31 36 3a 30 37 3a 33 31 20 32 30 31 35 29 0a | 16:07:31 2015).| 00002260 68 6f 73 74 2d 69 64 3d 32 0a 73 63 6f 72 65 3d |host-id=2.score=| 00002270 33 30 30 30 0a 6d 61 69 6e 74 65 6e 61 6e 63 65 |3000.maintenance| 00002280 3d 46 61 6c 73 65 0a 73 74 61 74 65 3d 45 6e 67 |=False.state=Eng| 00002290 69 6e 65 44 6f 77 6e 0a 73 74 6f 70 70 65 64 3d |ineDown.stopped=| 000022a0 46 61 6c 73 65 0a 00 00 00 00 00 00 00 00 00 00 |False...........| 000022b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 08000000 Rose (host no. 2): [root@rose05 ~]# hexdump -C /rhev/data-center/mnt/blockSD/d38153c3-0cfa-452f-9619-adadc00e27d7/ha_agent/hosted-engine.metadata 00000000 6d 61 69 6e 74 65 6e 61 6e 63 65 3d 30 00 00 00 |maintenance=0...| 00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00001000 31 7c 31 7c 38 37 33 36 31 7c 31 7c 33 34 30 30 |1|1|87361|1|3400| 00001010 7c 7b 22 68 65 61 6c 74 68 22 3a 20 22 67 6f 6f ||{"health": "goo| 00001020 64 22 2c 20 22 76 6d 22 3a 20 22 75 70 22 2c 20 |d", "vm": "up", | 00001030 22 64 65 74 61 69 6c 22 3a 20 22 75 70 22 7d 7c |"detail": "up"}|| 00001040 63 79 61 6e 2d 76 64 73 66 2e 71 61 2e 6c 61 62 |cyan-vdsf.qa.lab| 00001050 2e 74 6c 76 2e 72 65 64 68 61 74 2e 63 6f 6d 7c |.tlv.redhat.com|| 00001060 30 7c 30 7c 39 36 31 62 33 35 37 63 00 00 00 00 |0|0|961b357c....| 00001070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00001200 6d 65 74 61 64 61 74 61 5f 70 61 72 73 65 5f 76 |metadata_parse_v| 00001210 65 72 73 69 6f 6e 3d 31 0a 6d 65 74 61 64 61 74 |ersion=1.metadat| 00001220 61 5f 66 65 61 74 75 72 65 5f 76 65 72 73 69 6f |a_feature_versio| 00001230 6e 3d 31 0a 74 69 6d 65 73 74 61 6d 70 3d 38 37 |n=1.timestamp=87| 00001240 33 36 31 20 28 4d 6f 6e 20 4e 6f 76 20 20 32 20 |361 (Mon Nov 2 | 00001250 31 31 3a 33 37 3a 30 31 20 32 30 31 35 29 0a 68 |11:37:01 2015).h| 00001260 6f 73 74 2d 69 64 3d 31 0a 73 63 6f 72 65 3d 33 |ost-id=1.score=3| 00001270 34 30 30 0a 6d 61 69 6e 74 65 6e 61 6e 63 65 3d |400.maintenance=| 00001280 46 61 6c 73 65 0a 73 74 61 74 65 3d 45 6e 67 69 |False.state=Engi| 00001290 6e 65 55 70 0a 73 74 6f 70 70 65 64 3d 46 61 6c |neUp.stopped=Fal| 000012a0 73 65 0a 00 00 00 00 00 00 00 00 00 00 00 00 00 |se..............| 000012b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00002000 31 7c 31 7c 31 39 30 33 38 37 7c 32 7c 33 30 30 |1|1|190387|2|300| 00002010 30 7c 7b 22 72 65 61 73 6f 6e 22 3a 20 22 53 74 |0|{"reason": "St| 00002020 6f 72 61 67 65 20 6f 66 20 56 4d 20 69 73 20 6c |orage of VM is l| 00002030 6f 63 6b 65 64 2e 20 49 73 20 61 6e 6f 74 68 65 |ocked. Is anothe| 00002040 72 20 68 6f 73 74 20 61 6c 72 65 61 64 79 20 73 |r host already s| 00002050 74 61 72 74 69 6e 67 20 74 68 65 20 56 4d 3f 22 |tarting the VM?"| 00002060 2c 20 22 68 65 61 6c 74 68 22 3a 20 22 62 61 64 |, "health": "bad| 00002070 22 2c 20 22 76 6d 22 3a 20 22 61 6c 72 65 61 64 |", "vm": "alread| 00002080 79 5f 6c 6f 63 6b 65 64 22 2c 20 22 64 65 74 61 |y_locked", "deta| 00002090 69 6c 22 3a 20 22 64 6f 77 6e 22 7d 7c 72 6f 73 |il": "down"}|ros| 000020a0 65 30 35 2e 71 61 2e 6c 61 62 2e 74 6c 76 2e 72 |e05.qa.lab.tlv.r| 000020b0 65 64 68 61 74 2e 63 6f 6d 7c 30 7c 30 7c 65 66 |edhat.com|0|0|ef| 000020c0 32 34 30 34 33 36 00 00 00 00 00 00 00 00 00 00 |240436..........| 000020d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00002200 6d 65 74 61 64 61 74 61 5f 70 61 72 73 65 5f 76 |metadata_parse_v| 00002210 65 72 73 69 6f 6e 3d 31 0a 6d 65 74 61 64 61 74 |ersion=1.metadat| 00002220 61 5f 66 65 61 74 75 72 65 5f 76 65 72 73 69 6f |a_feature_versio| 00002230 6e 3d 31 0a 74 69 6d 65 73 74 61 6d 70 3d 31 39 |n=1.timestamp=19| 00002240 30 33 38 37 20 28 54 75 65 20 4e 6f 76 20 20 33 |0387 (Tue Nov 3| 00002250 20 31 36 3a 30 37 3a 33 31 20 32 30 31 35 29 0a | 16:07:31 2015).| 00002260 68 6f 73 74 2d 69 64 3d 32 0a 73 63 6f 72 65 3d |host-id=2.score=| 00002270 33 30 30 30 0a 6d 61 69 6e 74 65 6e 61 6e 63 65 |3000.maintenance| 00002280 3d 46 61 6c 73 65 0a 73 74 61 74 65 3d 45 6e 67 |=False.state=Eng| 00002290 69 6e 65 44 6f 77 6e 0a 73 74 6f 70 70 65 64 3d |ineDown.stopped=| 000022a0 46 61 6c 73 65 0a 00 00 00 00 00 00 00 00 00 00 |False...........| 000022b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 08000000
Please note the difference in the timestamp in the first metadata block as the hosts see it: 189987 (correct value) vs. 87361 (value seen at rose05). The same field from rose05 is properly seen by both hosts - 190387. This looks like an iscsi layer issue to me, because I can't explain why two different hosts see different data while directly accessing the same LV over iscsi.
And I see many instances of the following on both hosts. It might be related. [190939.616454] ovirt-ha-broker: sending ioctl 5401 to a partition! [190940.215608] device-mapper: table: 253:7: multipath: error getting device [190940.222671] device-mapper: ioctl: error adding target to table [190940.429911] device-mapper: table: 253:7: multipath: error getting device [190940.436967] device-mapper: ioctl: error adding target to table
Here is the standard storage layout we use. It comes from a different host as the test machine was already reinstalled so the IDs won't be the same. We always prepare the layout in the same way though. [root@host1 blockSD]# ls -R ... ./0537afcd-398e-4bc1-9ed9-e656d2576f79/ha_agent: total 8 lrwxrwxrwx. 1 vdsm kvm 132 Oct 30 12:18 hosted-engine.lockspace -> /var/run/vdsm/storage/0537afcd-398e-4bc1-9ed9-e656d2576f79/977decce-f985-4c29-a0fd-2bb2c462c554/92e88aff-00bb-4b8c-8e72-7cdc94b6b6e6 lrwxrwxrwx. 1 vdsm kvm 132 Oct 30 12:18 hosted-engine.metadata -> /var/run/vdsm/storage/0537afcd-398e-4bc1-9ed9-e656d2576f79/ffcf6a9b-be1f-403f-8da5-ed97a3333644/eec5ab35-c816-4dbc-9ec1-d1f61db498dc ... [root@host1 blockSD]# ls -al /var/run/vdsm/storage/0537afcd-398e-4bc1-9ed9-e656d2576f79/ffcf6a9b-be1f-403f-8da5-ed97a3333644/eec5ab35-c816-4dbc-9ec1-d1f61db498dc lrwxrwxrwx. 1 vdsm kvm 78 Oct 30 12:15 /var/run/vdsm/storage/0537afcd-398e-4bc1-9ed9-e656d2576f79/ffcf6a9b-be1f-403f-8da5-ed97a3333644/eec5ab35-c816-4dbc-9ec1-d1f61db498dc -> /dev/0537afcd-398e-4bc1-9ed9-e656d2576f79/eec5ab35-c816-4dbc-9ec1-d1f61db498dc [root@host1 blockSD]# ls -al /var/run/vdsm/storage/0537afcd-398e-4bc1-9ed9-e656d2576f79/977decce-f985-4c29-a0fd-2bb2c462c554/92e88aff-00bb-4b8c-8e72-7cdc94b6b6e6 lrwxrwxrwx. 1 vdsm kvm 78 Oct 30 12:15 /var/run/vdsm/storage/0537afcd-398e-4bc1-9ed9-e656d2576f79/977decce-f985-4c29-a0fd-2bb2c462c554/92e88aff-00bb-4b8c-8e72-7cdc94b6b6e6 -> /dev/0537afcd-398e-4bc1-9ed9-e656d2576f79/92e88aff-00bb-4b8c-8e72-7cdc94b6b6e6 [root@host1 ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 10G 0 disk └─360014052ab520caf86d44d0a0906fdf9 253:4 0 10G 0 mpath ├─c27f03a2--35fd--45dc--aee2--97787bb9cfbb-metadata 253:11 0 512M 0 lvm ├─c27f03a2--35fd--45dc--aee2--97787bb9cfbb-outbox 253:13 0 128M 0 lvm ├─c27f03a2--35fd--45dc--aee2--97787bb9cfbb-leases 253:16 0 2G 0 lvm ├─c27f03a2--35fd--45dc--aee2--97787bb9cfbb-ids 253:19 0 128M 0 lvm ├─c27f03a2--35fd--45dc--aee2--97787bb9cfbb-inbox 253:21 0 128M 0 lvm ├─c27f03a2--35fd--45dc--aee2--97787bb9cfbb-master 253:24 0 1G 0 lvm ├─c27f03a2--35fd--45dc--aee2--97787bb9cfbb-a241f588--d097--49be--af17--cc75cf338c29 253:27 0 128M 0 lvm └─c27f03a2--35fd--45dc--aee2--97787bb9cfbb-6e621077--b291--46bb--8dcb--8bb443d62196 253:32 0 128M 0 lvm sdb 8:16 0 40G 0 disk └─36001405fd3ac2af0b994c92b5ca0e0bb 253:9 0 40G 0 mpath sdc 8:32 0 40G 0 disk └─3600140504aac743668c468193a0ab064 253:5 0 40G 0 mpath ├─4cef71f3--6d3b--4225--bded--7bceaf46df30-metadata 253:10 0 512M 0 lvm ├─4cef71f3--6d3b--4225--bded--7bceaf46df30-outbox 253:12 0 128M 0 lvm ├─4cef71f3--6d3b--4225--bded--7bceaf46df30-leases 253:26 0 2G 0 lvm ├─4cef71f3--6d3b--4225--bded--7bceaf46df30-ids 253:31 0 128M 0 lvm ├─4cef71f3--6d3b--4225--bded--7bceaf46df30-inbox 253:36 0 128M 0 lvm ├─4cef71f3--6d3b--4225--bded--7bceaf46df30-master 253:40 0 1G 0 lvm ├─4cef71f3--6d3b--4225--bded--7bceaf46df30-d7f4fe14--3522--4df9--9b4b--8bc2e5e8cf13 253:44 0 1G 0 lvm ├─4cef71f3--6d3b--4225--bded--7bceaf46df30-68d67e85--898e--4853--a41a--df977622e1b1 253:47 0 128M 0 lvm ├─4cef71f3--6d3b--4225--bded--7bceaf46df30-1e709e9c--a325--45aa--9556--dde4273d0d81 253:50 0 128M 0 lvm ├─4cef71f3--6d3b--4225--bded--7bceaf46df30-8b29a2cb--710e--40e3--9247--d017e919e0d5 253:53 0 10G 0 lvm ├─4cef71f3--6d3b--4225--bded--7bceaf46df30-8f1a34f6--32af--4159--83b7--3d0c478cbd66 253:55 0 128M 0 lvm └─4cef71f3--6d3b--4225--bded--7bceaf46df30-ffb892be--b61c--4412--9556--c49d4f3934b7 253:56 0 128M 0 lvm sdd 8:48 0 40G 0 disk └─360014057855c2eb0dbe4d7fa60a324ae 253:7 0 40G 0 mpath ├─1b9072dc--44a9--4540--9370--1210924ed325-metadata 253:15 0 512M 0 lvm ├─1b9072dc--44a9--4540--9370--1210924ed325-outbox 253:17 0 128M 0 lvm ├─1b9072dc--44a9--4540--9370--1210924ed325-leases 253:20 0 2G 0 lvm ├─1b9072dc--44a9--4540--9370--1210924ed325-ids 253:23 0 128M 0 lvm ├─1b9072dc--44a9--4540--9370--1210924ed325-inbox 253:25 0 128M 0 lvm ├─1b9072dc--44a9--4540--9370--1210924ed325-master 253:28 0 1G 0 lvm ├─1b9072dc--44a9--4540--9370--1210924ed325-7fffa255--06c6--4144--90b8--06190e99f575 253:33 0 128M 0 lvm ├─1b9072dc--44a9--4540--9370--1210924ed325-61cb123f--7654--4b5b--b129--1b0538b0d1a6 253:38 0 128M 0 lvm └─1b9072dc--44a9--4540--9370--1210924ed325-7608e402--902c--4a47--b161--66496c8e7f5e 253:42 0 1G 0 lvm sde 8:64 0 40G 0 disk └─3600140539585f2cceea46558c6bdca9a 253:6 0 40G 0 mpath ├─de1d37cd--ec33--4f99--89ca--62a2e2e8134e-metadata 253:14 0 512M 0 lvm ├─de1d37cd--ec33--4f99--89ca--62a2e2e8134e-outbox 253:18 0 128M 0 lvm ├─de1d37cd--ec33--4f99--89ca--62a2e2e8134e-leases 253:22 0 2G 0 lvm ├─de1d37cd--ec33--4f99--89ca--62a2e2e8134e-ids 253:30 0 128M 0 lvm ├─de1d37cd--ec33--4f99--89ca--62a2e2e8134e-inbox 253:35 0 128M 0 lvm ├─de1d37cd--ec33--4f99--89ca--62a2e2e8134e-master 253:39 0 1G 0 lvm ├─de1d37cd--ec33--4f99--89ca--62a2e2e8134e-e2d8bca9--7af7--4508--87ae--4917c057097d 253:43 0 1G 0 lvm ├─de1d37cd--ec33--4f99--89ca--62a2e2e8134e-b4aa4220--1778--4490--b196--45ee31e6204f 253:46 0 128M 0 lvm ├─de1d37cd--ec33--4f99--89ca--62a2e2e8134e-0c6c57b4--85d9--4ca4--9ece--815e50769044 253:49 0 128M 0 lvm └─de1d37cd--ec33--4f99--89ca--62a2e2e8134e-5acbfe66--e8f7--4461--bc3f--0a18f62fb19b 253:52 0 10G 0 lvm sdf 8:80 0 40G 0 disk └─36001405882c9ec443204715b7f04b6c6 253:8 0 40G 0 mpath ├─0537afcd--398e--4bc1--9ed9--e656d2576f79-metadata 253:29 0 512M 0 lvm ├─0537afcd--398e--4bc1--9ed9--e656d2576f79-outbox 253:34 0 128M 0 lvm ├─0537afcd--398e--4bc1--9ed9--e656d2576f79-leases 253:37 0 2G 0 lvm ├─0537afcd--398e--4bc1--9ed9--e656d2576f79-ids 253:41 0 128M 0 lvm ├─0537afcd--398e--4bc1--9ed9--e656d2576f79-inbox 253:45 0 128M 0 lvm ├─0537afcd--398e--4bc1--9ed9--e656d2576f79-master 253:48 0 1G 0 lvm ├─0537afcd--398e--4bc1--9ed9--e656d2576f79-f37d620d--6642--4c67--b33b--7579ba7fc833 253:51 0 1G 0 lvm ├─0537afcd--398e--4bc1--9ed9--e656d2576f79-92e88aff--00bb--4b8c--8e72--7cdc94b6b6e6 253:54 0 128M 0 lvm ├─0537afcd--398e--4bc1--9ed9--e656d2576f79-eec5ab35--c816--4dbc--9ec1--d1f61db498dc 253:57 0 128M 0 lvm └─0537afcd--398e--4bc1--9ed9--e656d2576f79-34c7ba83--6f13--4def--87d5--c0d7d78f6699 253:58 0 10G 0 lvm sr0 11:0 1 1024M 0 rom vda 252:0 0 20G 0 disk ├─vda1 252:1 0 500M 0 part /boot └─vda2 252:2 0 19.5G 0 part ├─rootvg-lv_root 253:0 0 15.2G 0 lvm / ├─rootvg-lv_swap 253:1 0 2G 0 lvm [SWAP] ├─rootvg-lv_tmp 253:2 0 2G 0 lvm /tmp └─rootvg-lv_home 253:3 0 200M 0 lvm /home [root@host1 ~]# dmsetup table 4cef71f3--6d3b--4225--bded--7bceaf46df30-1e709e9c--a325--45aa--9556--dde4273d0d81: 0 262144 linear 253:5 10756096 4cef71f3--6d3b--4225--bded--7bceaf46df30-ids: 0 262144 linear 253:5 5775360 0537afcd--398e--4bc1--9ed9--e656d2576f79-outbox: 0 262144 linear 253:8 1318912 4cef71f3--6d3b--4225--bded--7bceaf46df30-metadata: 0 1048576 linear 253:5 270336 4cef71f3--6d3b--4225--bded--7bceaf46df30-master: 0 2097152 linear 253:5 6299648 rootvg-lv_swap: 0 4194304 linear 252:2 4605952 rootvg-lv_root: 0 31948800 linear 252:2 8800256 4cef71f3--6d3b--4225--bded--7bceaf46df30-ffb892be--b61c--4412--9556--c49d4f3934b7: 0 262144 linear 253:5 32251904 0537afcd--398e--4bc1--9ed9--e656d2576f79-inbox: 0 262144 linear 253:8 6037504 1b9072dc--44a9--4540--9370--1210924ed325-leases: 0 4194304 linear 253:7 1581056 1b9072dc--44a9--4540--9370--1210924ed325-61cb123f--7654--4b5b--b129--1b0538b0d1a6: 0 262144 linear 253:7 8658944 de1d37cd--ec33--4f99--89ca--62a2e2e8134e-inbox: 0 262144 linear 253:6 6037504 4cef71f3--6d3b--4225--bded--7bceaf46df30-outbox: 0 262144 linear 253:5 1318912 4cef71f3--6d3b--4225--bded--7bceaf46df30-8f1a34f6--32af--4159--83b7--3d0c478cbd66: 0 262144 linear 253:5 31989760 3600140504aac743668c468193a0ab064: 0 83886080 multipath 0 0 1 1 service-time 0 1 2 8:32 1 1 de1d37cd--ec33--4f99--89ca--62a2e2e8134e-5acbfe66--e8f7--4461--bc3f--0a18f62fb19b: 0 20971520 linear 253:6 11018240 0537afcd--398e--4bc1--9ed9--e656d2576f79-f37d620d--6642--4c67--b33b--7579ba7fc833: 0 2097152 linear 253:8 8396800 1b9072dc--44a9--4540--9370--1210924ed325-7fffa255--06c6--4144--90b8--06190e99f575: 0 262144 linear 253:7 8396800 c27f03a2--35fd--45dc--aee2--97787bb9cfbb-6e621077--b291--46bb--8dcb--8bb443d62196: 0 262144 linear 253:4 8658944 de1d37cd--ec33--4f99--89ca--62a2e2e8134e-b4aa4220--1778--4490--b196--45ee31e6204f: 0 262144 linear 253:6 10493952 de1d37cd--ec33--4f99--89ca--62a2e2e8134e-leases: 0 4194304 linear 253:6 1581056 c27f03a2--35fd--45dc--aee2--97787bb9cfbb-leases: 0 4194304 linear 253:4 1581056 1b9072dc--44a9--4540--9370--1210924ed325-master: 0 2097152 linear 253:7 6299648 3600140539585f2cceea46558c6bdca9a: 0 83886080 multipath 0 0 1 1 service-time 0 1 2 8:64 1 1 c27f03a2--35fd--45dc--aee2--97787bb9cfbb-a241f588--d097--49be--af17--cc75cf338c29: 0 262144 linear 253:4 8396800 4cef71f3--6d3b--4225--bded--7bceaf46df30-d7f4fe14--3522--4df9--9b4b--8bc2e5e8cf13: 0 2097152 linear 253:5 8396800 c27f03a2--35fd--45dc--aee2--97787bb9cfbb-inbox: 0 262144 linear 253:4 6037504 1b9072dc--44a9--4540--9370--1210924ed325-outbox: 0 262144 linear 253:7 1318912 360014052ab520caf86d44d0a0906fdf9: 0 20971520 multipath 0 0 1 1 service-time 0 1 2 8:0 1 1 0537afcd--398e--4bc1--9ed9--e656d2576f79-leases: 0 4194304 linear 253:8 1581056 0537afcd--398e--4bc1--9ed9--e656d2576f79-92e88aff--00bb--4b8c--8e72--7cdc94b6b6e6: 0 262144 linear 253:8 10493952 0537afcd--398e--4bc1--9ed9--e656d2576f79-metadata: 0 1048576 linear 253:8 270336 4cef71f3--6d3b--4225--bded--7bceaf46df30-leases: 0 4194304 linear 253:5 1581056 4cef71f3--6d3b--4225--bded--7bceaf46df30-8b29a2cb--710e--40e3--9247--d017e919e0d5: 0 20971520 linear 253:5 11018240 1b9072dc--44a9--4540--9370--1210924ed325-7608e402--902c--4a47--b161--66496c8e7f5e: 0 2097152 linear 253:7 8921088 de1d37cd--ec33--4f99--89ca--62a2e2e8134e-master: 0 2097152 linear 253:6 6299648 0537afcd--398e--4bc1--9ed9--e656d2576f79-eec5ab35--c816--4dbc--9ec1--d1f61db498dc: 0 262144 linear 253:8 10756096 c27f03a2--35fd--45dc--aee2--97787bb9cfbb-master: 0 2097152 linear 253:4 6299648 1b9072dc--44a9--4540--9370--1210924ed325-inbox: 0 262144 linear 253:7 6037504 de1d37cd--ec33--4f99--89ca--62a2e2e8134e-outbox: 0 262144 linear 253:6 1318912 1b9072dc--44a9--4540--9370--1210924ed325-metadata: 0 1048576 linear 253:7 270336 rootvg-lv_home: 0 409600 linear 252:2 4196352 de1d37cd--ec33--4f99--89ca--62a2e2e8134e-0c6c57b4--85d9--4ca4--9ece--815e50769044: 0 262144 linear 253:6 10756096 1b9072dc--44a9--4540--9370--1210924ed325-ids: 0 262144 linear 253:7 5775360 360014057855c2eb0dbe4d7fa60a324ae: 0 83886080 multipath 0 0 1 1 service-time 0 1 2 8:48 1 1 de1d37cd--ec33--4f99--89ca--62a2e2e8134e-metadata: 0 1048576 linear 253:6 270336 rootvg-lv_tmp: 0 4194304 linear 252:2 2048 c27f03a2--35fd--45dc--aee2--97787bb9cfbb-outbox: 0 262144 linear 253:4 1318912 4cef71f3--6d3b--4225--bded--7bceaf46df30-inbox: 0 262144 linear 253:5 6037504 4cef71f3--6d3b--4225--bded--7bceaf46df30-68d67e85--898e--4853--a41a--df977622e1b1: 0 262144 linear 253:5 10493952 de1d37cd--ec33--4f99--89ca--62a2e2e8134e-ids: 0 262144 linear 253:6 5775360 c27f03a2--35fd--45dc--aee2--97787bb9cfbb-ids: 0 262144 linear 253:4 5775360 0537afcd--398e--4bc1--9ed9--e656d2576f79-34c7ba83--6f13--4def--87d5--c0d7d78f6699: 0 20971520 linear 253:8 11018240 36001405fd3ac2af0b994c92b5ca0e0bb: 0 83886080 multipath 0 0 1 1 service-time 0 1 2 8:16 1 1 0537afcd--398e--4bc1--9ed9--e656d2576f79-master: 0 2097152 linear 253:8 6299648 36001405882c9ec443204715b7f04b6c6: 0 83886080 multipath 0 0 1 1 service-time 0 1 2 8:80 1 1 de1d37cd--ec33--4f99--89ca--62a2e2e8134e-e2d8bca9--7af7--4508--87ae--4917c057097d: 0 2097152 linear 253:6 8396800 0537afcd--398e--4bc1--9ed9--e656d2576f79-ids: 0 262144 linear 253:8 5775360 c27f03a2--35fd--45dc--aee2--97787bb9cfbb-metadata: 0 1048576 linear 253:4 270336 [root@host1 ~]# dmsetup ls --tree 4cef71f3--6d3b--4225--bded--7bceaf46df30-1e709e9c--a325--45aa--9556--dde4273... └─3600140504aac743668c468193a0ab064 (253:5) └─ (8:32) 4cef71f3--6d3b--4225--bded--7bceaf46df30-ids (253:31) └─3600140504aac743668c468193a0ab064 (253:5) └─ (8:32) 0537afcd--398e--4bc1--9ed9--e656d2576f79-outbox (253:34) └─36001405882c9ec443204715b7f04b6c6 (253:8) └─ (8:80) 4cef71f3--6d3b--4225--bded--7bceaf46df30-metadata (253:10) └─3600140504aac743668c468193a0ab064 (253:5) └─ (8:32) 4cef71f3--6d3b--4225--bded--7bceaf46df30-master (253:40) └─3600140504aac743668c468193a0ab064 (253:5) └─ (8:32) rootvg-lv_swap (253:1) └─ (252:2) rootvg-lv_root (253:0) └─ (252:2) 4cef71f3--6d3b--4225--bded--7bceaf46df30-ffb892be--b61c--4412--9556--c49d4f3... └─3600140504aac743668c468193a0ab064 (253:5) └─ (8:32) 0537afcd--398e--4bc1--9ed9--e656d2576f79-inbox (253:45) └─36001405882c9ec443204715b7f04b6c6 (253:8) └─ (8:80) 1b9072dc--44a9--4540--9370--1210924ed325-leases (253:20) └─360014057855c2eb0dbe4d7fa60a324ae (253:7) └─ (8:48) 1b9072dc--44a9--4540--9370--1210924ed325-61cb123f--7654--4b5b--b129--1b0538b... └─360014057855c2eb0dbe4d7fa60a324ae (253:7) └─ (8:48) de1d37cd--ec33--4f99--89ca--62a2e2e8134e-inbox (253:35) └─3600140539585f2cceea46558c6bdca9a (253:6) └─ (8:64) 4cef71f3--6d3b--4225--bded--7bceaf46df30-outbox (253:12) └─3600140504aac743668c468193a0ab064 (253:5) └─ (8:32) 4cef71f3--6d3b--4225--bded--7bceaf46df30-8f1a34f6--32af--4159--83b7--3d0c478... └─3600140504aac743668c468193a0ab064 (253:5) └─ (8:32) de1d37cd--ec33--4f99--89ca--62a2e2e8134e-5acbfe66--e8f7--4461--bc3f--0a18f62... └─3600140539585f2cceea46558c6bdca9a (253:6) └─ (8:64) 0537afcd--398e--4bc1--9ed9--e656d2576f79-f37d620d--6642--4c67--b33b--7579ba7... └─36001405882c9ec443204715b7f04b6c6 (253:8) └─ (8:80) 1b9072dc--44a9--4540--9370--1210924ed325-7fffa255--06c6--4144--90b8--06190e9... └─360014057855c2eb0dbe4d7fa60a324ae (253:7) └─ (8:48) c27f03a2--35fd--45dc--aee2--97787bb9cfbb-6e621077--b291--46bb--8dcb--8bb443d... └─360014052ab520caf86d44d0a0906fdf9 (253:4) └─ (8:0) de1d37cd--ec33--4f99--89ca--62a2e2e8134e-b4aa4220--1778--4490--b196--45ee31e... └─3600140539585f2cceea46558c6bdca9a (253:6) └─ (8:64) de1d37cd--ec33--4f99--89ca--62a2e2e8134e-leases (253:22) └─3600140539585f2cceea46558c6bdca9a (253:6) └─ (8:64) c27f03a2--35fd--45dc--aee2--97787bb9cfbb-leases (253:16) └─360014052ab520caf86d44d0a0906fdf9 (253:4) └─ (8:0) 1b9072dc--44a9--4540--9370--1210924ed325-master (253:28) └─360014057855c2eb0dbe4d7fa60a324ae (253:7) └─ (8:48) c27f03a2--35fd--45dc--aee2--97787bb9cfbb-a241f588--d097--49be--af17--cc75cf3... └─360014052ab520caf86d44d0a0906fdf9 (253:4) └─ (8:0) 4cef71f3--6d3b--4225--bded--7bceaf46df30-d7f4fe14--3522--4df9--9b4b--8bc2e5e... └─3600140504aac743668c468193a0ab064 (253:5) └─ (8:32) c27f03a2--35fd--45dc--aee2--97787bb9cfbb-inbox (253:21) └─360014052ab520caf86d44d0a0906fdf9 (253:4) └─ (8:0) 1b9072dc--44a9--4540--9370--1210924ed325-outbox (253:17) └─360014057855c2eb0dbe4d7fa60a324ae (253:7) └─ (8:48) 0537afcd--398e--4bc1--9ed9--e656d2576f79-leases (253:37) └─36001405882c9ec443204715b7f04b6c6 (253:8) └─ (8:80) 0537afcd--398e--4bc1--9ed9--e656d2576f79-92e88aff--00bb--4b8c--8e72--7cdc94b... └─36001405882c9ec443204715b7f04b6c6 (253:8) └─ (8:80) 0537afcd--398e--4bc1--9ed9--e656d2576f79-metadata (253:29) └─36001405882c9ec443204715b7f04b6c6 (253:8) └─ (8:80) 4cef71f3--6d3b--4225--bded--7bceaf46df30-leases (253:26) └─3600140504aac743668c468193a0ab064 (253:5) └─ (8:32) 4cef71f3--6d3b--4225--bded--7bceaf46df30-8b29a2cb--710e--40e3--9247--d017e91... └─3600140504aac743668c468193a0ab064 (253:5) └─ (8:32) 1b9072dc--44a9--4540--9370--1210924ed325-7608e402--902c--4a47--b161--66496c8... └─360014057855c2eb0dbe4d7fa60a324ae (253:7) └─ (8:48) de1d37cd--ec33--4f99--89ca--62a2e2e8134e-master (253:39) └─3600140539585f2cceea46558c6bdca9a (253:6) └─ (8:64) 0537afcd--398e--4bc1--9ed9--e656d2576f79-eec5ab35--c816--4dbc--9ec1--d1f61db... └─36001405882c9ec443204715b7f04b6c6 (253:8) └─ (8:80) c27f03a2--35fd--45dc--aee2--97787bb9cfbb-master (253:24) └─360014052ab520caf86d44d0a0906fdf9 (253:4) └─ (8:0) 1b9072dc--44a9--4540--9370--1210924ed325-inbox (253:25) └─360014057855c2eb0dbe4d7fa60a324ae (253:7) └─ (8:48) de1d37cd--ec33--4f99--89ca--62a2e2e8134e-outbox (253:18) └─3600140539585f2cceea46558c6bdca9a (253:6) └─ (8:64) 1b9072dc--44a9--4540--9370--1210924ed325-metadata (253:15) └─360014057855c2eb0dbe4d7fa60a324ae (253:7) └─ (8:48) rootvg-lv_home (253:3) └─ (252:2) de1d37cd--ec33--4f99--89ca--62a2e2e8134e-0c6c57b4--85d9--4ca4--9ece--815e507... └─3600140539585f2cceea46558c6bdca9a (253:6) └─ (8:64) 1b9072dc--44a9--4540--9370--1210924ed325-ids (253:23) └─360014057855c2eb0dbe4d7fa60a324ae (253:7) └─ (8:48) de1d37cd--ec33--4f99--89ca--62a2e2e8134e-metadata (253:14) └─3600140539585f2cceea46558c6bdca9a (253:6) └─ (8:64) rootvg-lv_tmp (253:2) └─ (252:2) c27f03a2--35fd--45dc--aee2--97787bb9cfbb-outbox (253:13) └─360014052ab520caf86d44d0a0906fdf9 (253:4) └─ (8:0) 4cef71f3--6d3b--4225--bded--7bceaf46df30-inbox (253:36) └─3600140504aac743668c468193a0ab064 (253:5) └─ (8:32) 4cef71f3--6d3b--4225--bded--7bceaf46df30-68d67e85--898e--4853--a41a--df97762... └─3600140504aac743668c468193a0ab064 (253:5) └─ (8:32) de1d37cd--ec33--4f99--89ca--62a2e2e8134e-ids (253:30) └─3600140539585f2cceea46558c6bdca9a (253:6) └─ (8:64) c27f03a2--35fd--45dc--aee2--97787bb9cfbb-ids (253:19) └─360014052ab520caf86d44d0a0906fdf9 (253:4) └─ (8:0) 0537afcd--398e--4bc1--9ed9--e656d2576f79-34c7ba83--6f13--4def--87d5--c0d7d78... └─36001405882c9ec443204715b7f04b6c6 (253:8) └─ (8:80) 36001405fd3ac2af0b994c92b5ca0e0bb (253:9) └─ (8:16) 0537afcd--398e--4bc1--9ed9--e656d2576f79-master (253:48) └─36001405882c9ec443204715b7f04b6c6 (253:8) └─ (8:80) de1d37cd--ec33--4f99--89ca--62a2e2e8134e-e2d8bca9--7af7--4508--87ae--4917c05... └─3600140539585f2cceea46558c6bdca9a (253:6) └─ (8:64) 0537afcd--398e--4bc1--9ed9--e656d2576f79-ids (253:41) └─36001405882c9ec443204715b7f04b6c6 (253:8) └─ (8:80) c27f03a2--35fd--45dc--aee2--97787bb9cfbb-metadata (253:11) └─360014052ab520caf86d44d0a0906fdf9 (253:4) └─ (8:0)
OK, we reproduced the issue again. This time the updates are not propagated from rose05 to cyan (so the change to the second metadata block is not seen by cyan). Those two commands were executed at the same time. We update the fields once per 10 seconds so a minute difference is too much. [root@rose05 ~]# dd iflag=sync iflag=nocache bs=1M count=1 if=/rhev/data-center/mnt/blockSD/d38153c3-0cfa-452f-9619-adadc00e27d7/ha_agent/hosted-engine.metadata of=dump.bin; hexdump -C dump.bin 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0204172 s, 51.4 MB/s 00000000 6d 61 69 6e 74 65 6e 61 6e 63 65 3d 30 00 00 00 |maintenance=0...| 00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00001000 31 7c 31 7c 37 33 36 32 37 7c 31 7c 32 34 34 30 |1|1|73627|1|2440| 00001010 7c 7b 22 72 65 61 73 6f 6e 22 3a 20 22 62 61 64 ||{"reason": "bad| 00001020 20 76 6d 20 73 74 61 74 75 73 22 2c 20 22 68 65 | vm status", "he| 00001030 61 6c 74 68 22 3a 20 22 62 61 64 22 2c 20 22 76 |alth": "bad", "v| 00001040 6d 22 3a 20 22 75 70 22 2c 20 22 64 65 74 61 69 |m": "up", "detai| 00001050 6c 22 3a 20 22 70 61 75 73 65 64 22 7d 7c 63 79 |l": "paused"}|cy| 00001060 61 6e 2d 76 64 73 66 2e 71 61 2e 6c 61 62 2e 74 |an-vdsf.qa.lab.t| 00001070 6c 76 2e 72 65 64 68 61 74 2e 63 6f 6d 7c 30 7c |lv.redhat.com|0|| 00001080 30 7c 65 65 36 66 36 66 62 37 00 00 00 00 00 00 |0|ee6f6fb7......| 00001090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00001200 6d 65 74 61 64 61 74 61 5f 70 61 72 73 65 5f 76 |metadata_parse_v| 00001210 65 72 73 69 6f 6e 3d 31 0a 6d 65 74 61 64 61 74 |ersion=1.metadat| 00001220 61 5f 66 65 61 74 75 72 65 5f 76 65 72 73 69 6f |a_feature_versio| 00001230 6e 3d 31 0a 74 69 6d 65 73 74 61 6d 70 3d 37 33 |n=1.timestamp=73| 00001240 36 32 37 20 28 57 65 64 20 4e 6f 76 20 20 34 20 |627 (Wed Nov 4 | 00001250 31 33 3a 33 39 3a 33 38 20 32 30 31 35 29 0a 68 |13:39:38 2015).h| 00001260 6f 73 74 2d 69 64 3d 31 0a 73 63 6f 72 65 3d 32 |ost-id=1.score=2| 00001270 34 34 30 0a 6d 61 69 6e 74 65 6e 61 6e 63 65 3d |440.maintenance=| 00001280 46 61 6c 73 65 0a 73 74 61 74 65 3d 45 6e 67 69 |False.state=Engi| 00001290 6e 65 53 74 61 72 74 69 6e 67 0a 73 74 6f 70 70 |neStarting.stopp| 000012a0 65 64 3d 46 61 6c 73 65 0a 00 00 00 00 00 00 00 |ed=False........| 000012b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00002000 31 7c 31 7c 37 33 32 39 39 7c 32 7c 33 34 30 30 |1|1|73299|2|3400| 00002010 7c 7b 22 68 65 61 6c 74 68 22 3a 20 22 67 6f 6f ||{"health": "goo| 00002020 64 22 2c 20 22 76 6d 22 3a 20 22 75 70 22 2c 20 |d", "vm": "up", | 00002030 22 64 65 74 61 69 6c 22 3a 20 22 75 70 22 7d 7c |"detail": "up"}|| 00002040 72 6f 73 65 30 35 2e 71 61 2e 6c 61 62 2e 74 6c |rose05.qa.lab.tl| 00002050 76 2e 72 65 64 68 61 74 2e 63 6f 6d 7c 30 7c 30 |v.redhat.com|0|0| 00002060 7c 31 65 31 64 62 35 32 62 00 00 00 00 00 00 00 ||1e1db52b.......| 00002070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00002200 6d 65 74 61 64 61 74 61 5f 70 61 72 73 65 5f 76 |metadata_parse_v| 00002210 65 72 73 69 6f 6e 3d 31 0a 6d 65 74 61 64 61 74 |ersion=1.metadat| 00002220 61 5f 66 65 61 74 75 72 65 5f 76 65 72 73 69 6f |a_feature_versio| 00002230 6e 3d 31 0a 74 69 6d 65 73 74 61 6d 70 3d 37 33 |n=1.timestamp=73| 00002240 32 39 39 20 28 57 65 64 20 4e 6f 76 20 20 34 20 |299 (Wed Nov 4 | 00002250 31 33 3a 33 39 3a 34 33 20 32 30 31 35 29 0a 68 |13:39:43 2015).h| 00002260 6f 73 74 2d 69 64 3d 32 0a 73 63 6f 72 65 3d 33 |ost-id=2.score=3| 00002270 34 30 30 0a 6d 61 69 6e 74 65 6e 61 6e 63 65 3d |400.maintenance=| 00002280 46 61 6c 73 65 0a 73 74 61 74 65 3d 45 6e 67 69 |False.state=Engi| 00002290 6e 65 55 70 0a 73 74 6f 70 70 65 64 3d 46 61 6c |neUp.stopped=Fal| 000022a0 73 65 0a 00 00 00 00 00 00 00 00 00 00 00 00 00 |se..............| 000022b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00100000 [root@cyan-vdsf ~]# dd iflag=sync iflag=nocache bs=1M count=1 if=/rhev/data-center/mnt/blockSD/d38153c3-0cfa-452f-9619-adadc00e27d7/ha_agent/hosted-engine.metadata of=dump.bin; hexdump -C dump.bin 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00419074 s, 250 MB/s 00000000 6d 61 69 6e 74 65 6e 61 6e 63 65 3d 30 00 00 00 |maintenance=0...| 00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00001000 31 7c 31 7c 37 33 36 32 37 7c 31 7c 32 34 34 30 |1|1|73627|1|2440| 00001010 7c 7b 22 72 65 61 73 6f 6e 22 3a 20 22 62 61 64 ||{"reason": "bad| 00001020 20 76 6d 20 73 74 61 74 75 73 22 2c 20 22 68 65 | vm status", "he| 00001030 61 6c 74 68 22 3a 20 22 62 61 64 22 2c 20 22 76 |alth": "bad", "v| 00001040 6d 22 3a 20 22 75 70 22 2c 20 22 64 65 74 61 69 |m": "up", "detai| 00001050 6c 22 3a 20 22 70 61 75 73 65 64 22 7d 7c 63 79 |l": "paused"}|cy| 00001060 61 6e 2d 76 64 73 66 2e 71 61 2e 6c 61 62 2e 74 |an-vdsf.qa.lab.t| 00001070 6c 76 2e 72 65 64 68 61 74 2e 63 6f 6d 7c 30 7c |lv.redhat.com|0|| 00001080 30 7c 65 65 36 66 36 66 62 37 00 00 00 00 00 00 |0|ee6f6fb7......| 00001090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00001200 6d 65 74 61 64 61 74 61 5f 70 61 72 73 65 5f 76 |metadata_parse_v| 00001210 65 72 73 69 6f 6e 3d 31 0a 6d 65 74 61 64 61 74 |ersion=1.metadat| 00001220 61 5f 66 65 61 74 75 72 65 5f 76 65 72 73 69 6f |a_feature_versio| 00001230 6e 3d 31 0a 74 69 6d 65 73 74 61 6d 70 3d 37 33 |n=1.timestamp=73| 00001240 36 32 37 20 28 57 65 64 20 4e 6f 76 20 20 34 20 |627 (Wed Nov 4 | 00001250 31 33 3a 33 39 3a 33 38 20 32 30 31 35 29 0a 68 |13:39:38 2015).h| 00001260 6f 73 74 2d 69 64 3d 31 0a 73 63 6f 72 65 3d 32 |ost-id=1.score=2| 00001270 34 34 30 0a 6d 61 69 6e 74 65 6e 61 6e 63 65 3d |440.maintenance=| 00001280 46 61 6c 73 65 0a 73 74 61 74 65 3d 45 6e 67 69 |False.state=Engi| 00001290 6e 65 53 74 61 72 74 69 6e 67 0a 73 74 6f 70 70 |neStarting.stopp| 000012a0 65 64 3d 46 61 6c 73 65 0a 00 00 00 00 00 00 00 |ed=False........| 000012b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00002000 31 7c 31 7c 37 33 31 36 34 7c 32 7c 33 34 30 30 |1|1|73164|2|3400| 00002010 7c 7b 22 68 65 61 6c 74 68 22 3a 20 22 67 6f 6f ||{"health": "goo| 00002020 64 22 2c 20 22 76 6d 22 3a 20 22 75 70 22 2c 20 |d", "vm": "up", | 00002030 22 64 65 74 61 69 6c 22 3a 20 22 75 70 22 7d 7c |"detail": "up"}|| 00002040 72 6f 73 65 30 35 2e 71 61 2e 6c 61 62 2e 74 6c |rose05.qa.lab.tl| 00002050 76 2e 72 65 64 68 61 74 2e 63 6f 6d 7c 30 7c 30 |v.redhat.com|0|0| 00002060 7c 39 36 37 39 65 38 65 63 00 00 00 00 00 00 00 ||9679e8ec.......| 00002070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00002200 6d 65 74 61 64 61 74 61 5f 70 61 72 73 65 5f 76 |metadata_parse_v| 00002210 65 72 73 69 6f 6e 3d 31 0a 6d 65 74 61 64 61 74 |ersion=1.metadat| 00002220 61 5f 66 65 61 74 75 72 65 5f 76 65 72 73 69 6f |a_feature_versio| 00002230 6e 3d 31 0a 74 69 6d 65 73 74 61 6d 70 3d 37 33 |n=1.timestamp=73| 00002240 31 36 34 20 28 57 65 64 20 4e 6f 76 20 20 34 20 |164 (Wed Nov 4 | 00002250 31 33 3a 33 37 3a 32 39 20 32 30 31 35 29 0a 68 |13:37:29 2015).h| 00002260 6f 73 74 2d 69 64 3d 32 0a 73 63 6f 72 65 3d 33 |ost-id=2.score=3| 00002270 34 30 30 0a 6d 61 69 6e 74 65 6e 61 6e 63 65 3d |400.maintenance=| 00002280 46 61 6c 73 65 0a 73 74 61 74 65 3d 45 6e 67 69 |False.state=Engi| 00002290 6e 65 55 70 0a 73 74 6f 70 70 65 64 3d 46 61 6c |neUp.stopped=Fal| 000022a0 73 65 0a 00 00 00 00 00 00 00 00 00 00 00 00 00 |se..............| 000022b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00100000
[root@rose05 ~]# lsblk; dmsetup table; dmsetup ls --tree NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 465.8G 0 disk ├─sda1 8:1 0 200M 0 part /boot ├─sda2 8:2 0 2G 0 part [SWAP] └─sda3 8:3 0 463.6G 0 part / sdb 8:16 0 50G 0 disk └─3600a0980383035596d2b47677a426b61 253:0 0 50G 0 mpath ├─d38153c3--0cfa--452f--9619--adadc00e27d7-metadata 253:1 0 512M 0 lvm ├─d38153c3--0cfa--452f--9619--adadc00e27d7-outbox 253:2 0 128M 0 lvm ├─d38153c3--0cfa--452f--9619--adadc00e27d7-leases 253:3 0 2G 0 lvm ├─d38153c3--0cfa--452f--9619--adadc00e27d7-ids 253:4 0 128M 0 lvm ├─d38153c3--0cfa--452f--9619--adadc00e27d7-inbox 253:5 0 128M 0 lvm ├─d38153c3--0cfa--452f--9619--adadc00e27d7-master 253:6 0 1G 0 lvm ├─d38153c3--0cfa--452f--9619--adadc00e27d7-376d4800--8b81--4326--9f51--0c0e97890e67 253:7 0 25G 0 lvm ├─d38153c3--0cfa--452f--9619--adadc00e27d7-de655474--3a01--46fc--bca9--cfe092fbeb70 253:8 0 128M 0 lvm ├─d38153c3--0cfa--452f--9619--adadc00e27d7-3e5d50e0--02a1--421b--b249--4b561dfefd80 253:9 0 128M 0 lvm └─d38153c3--0cfa--452f--9619--adadc00e27d7-4b44d4c3--4a8f--486b--954b--680fab94144c 253:10 0 1G 0 lvm d38153c3--0cfa--452f--9619--adadc00e27d7-master: 0 2097152 linear 253:0 6293504 3600a0980383035596d2b47677a426b61: 0 104857600 multipath 3 pg_init_retries 50 retain_attached_hw_handler 1 alua 1 1 service-time 0 1 2 8:16 1 1 d38153c3--0cfa--452f--9619--adadc00e27d7-outbox: 0 262144 linear 253:0 1312768 d38153c3--0cfa--452f--9619--adadc00e27d7-376d4800--8b81--4326--9f51--0c0e97890e67: 0 52428800 linear 253:0 11012096 d38153c3--0cfa--452f--9619--adadc00e27d7-metadata: 0 1048576 linear 253:0 264192 d38153c3--0cfa--452f--9619--adadc00e27d7-inbox: 0 262144 linear 253:0 6031360 d38153c3--0cfa--452f--9619--adadc00e27d7-ids: 0 262144 linear 253:0 5769216 d38153c3--0cfa--452f--9619--adadc00e27d7-leases: 0 4194304 linear 253:0 1574912 d38153c3--0cfa--452f--9619--adadc00e27d7-de655474--3a01--46fc--bca9--cfe092fbeb70: 0 262144 linear 253:0 10487808 d38153c3--0cfa--452f--9619--adadc00e27d7-4b44d4c3--4a8f--486b--954b--680fab94144c: 0 2097152 linear 253:0 8390656 d38153c3--0cfa--452f--9619--adadc00e27d7-3e5d50e0--02a1--421b--b249--4b561dfefd80: 0 262144 linear 253:0 10749952 d38153c3--0cfa--452f--9619--adadc00e27d7-master (253:6) └─3600a0980383035596d2b47677a426b61 (253:0) └─ (8:16) d38153c3--0cfa--452f--9619--adadc00e27d7-outbox (253:2) └─3600a0980383035596d2b47677a426b61 (253:0) └─ (8:16) d38153c3--0cfa--452f--9619--adadc00e27d7-376d4800--8b81--4326--9f51--0c0e97890e67 (253:7) └─3600a0980383035596d2b47677a426b61 (253:0) └─ (8:16) d38153c3--0cfa--452f--9619--adadc00e27d7-metadata (253:1) └─3600a0980383035596d2b47677a426b61 (253:0) └─ (8:16) d38153c3--0cfa--452f--9619--adadc00e27d7-inbox (253:5) └─3600a0980383035596d2b47677a426b61 (253:0) └─ (8:16) d38153c3--0cfa--452f--9619--adadc00e27d7-ids (253:4) └─3600a0980383035596d2b47677a426b61 (253:0) └─ (8:16) d38153c3--0cfa--452f--9619--adadc00e27d7-leases (253:3) └─3600a0980383035596d2b47677a426b61 (253:0) └─ (8:16) d38153c3--0cfa--452f--9619--adadc00e27d7-de655474--3a01--46fc--bca9--cfe092fbeb70 (253:8) └─3600a0980383035596d2b47677a426b61 (253:0) └─ (8:16) d38153c3--0cfa--452f--9619--adadc00e27d7-4b44d4c3--4a8f--486b--954b--680fab94144c (253:10) └─3600a0980383035596d2b47677a426b61 (253:0) └─ (8:16) d38153c3--0cfa--452f--9619--adadc00e27d7-3e5d50e0--02a1--421b--b249--4b561dfefd80 (253:9) └─3600a0980383035596d2b47677a426b61 (253:0) └─ (8:16)
[root@cyan-vdsf ~]# lsblk; dmsetup table; dmsetup ls --tree NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 74.5G 0 disk ├─sda1 8:1 0 200M 0 part /boot ├─sda2 8:2 0 2G 0 part [SWAP] └─sda3 8:3 0 72.3G 0 part / sdb 8:16 0 50G 0 disk └─3600a0980383035596d2b47677a426b61 253:0 0 50G 0 mpath ├─d38153c3--0cfa--452f--9619--adadc00e27d7-metadata 253:1 0 512M 0 lvm ├─d38153c3--0cfa--452f--9619--adadc00e27d7-outbox 253:2 0 128M 0 lvm ├─d38153c3--0cfa--452f--9619--adadc00e27d7-leases 253:3 0 2G 0 lvm ├─d38153c3--0cfa--452f--9619--adadc00e27d7-ids 253:4 0 128M 0 lvm ├─d38153c3--0cfa--452f--9619--adadc00e27d7-inbox 253:5 0 128M 0 lvm ├─d38153c3--0cfa--452f--9619--adadc00e27d7-master 253:6 0 1G 0 lvm ├─d38153c3--0cfa--452f--9619--adadc00e27d7-3e5d50e0--02a1--421b--b249--4b561dfefd80 253:7 0 128M 0 lvm ├─d38153c3--0cfa--452f--9619--adadc00e27d7-de655474--3a01--46fc--bca9--cfe092fbeb70 253:8 0 128M 0 lvm ├─d38153c3--0cfa--452f--9619--adadc00e27d7-4b44d4c3--4a8f--486b--954b--680fab94144c 253:9 0 1G 0 lvm └─d38153c3--0cfa--452f--9619--adadc00e27d7-376d4800--8b81--4326--9f51--0c0e97890e67 253:10 0 25G 0 lvm sr0 11:0 1 1024M 0 rom d38153c3--0cfa--452f--9619--adadc00e27d7-master: 0 2097152 linear 253:0 6293504 3600a0980383035596d2b47677a426b61: 0 104857600 multipath 3 pg_init_retries 50 retain_attached_hw_handler 1 alua 1 1 service-time 0 1 2 8:16 1 1 d38153c3--0cfa--452f--9619--adadc00e27d7-outbox: 0 262144 linear 253:0 1312768 d38153c3--0cfa--452f--9619--adadc00e27d7-376d4800--8b81--4326--9f51--0c0e97890e67: 0 52428800 linear 253:0 11012096 d38153c3--0cfa--452f--9619--adadc00e27d7-metadata: 0 1048576 linear 253:0 264192 d38153c3--0cfa--452f--9619--adadc00e27d7-inbox: 0 262144 linear 253:0 6031360 d38153c3--0cfa--452f--9619--adadc00e27d7-ids: 0 262144 linear 253:0 5769216 d38153c3--0cfa--452f--9619--adadc00e27d7-leases: 0 4194304 linear 253:0 1574912 d38153c3--0cfa--452f--9619--adadc00e27d7-de655474--3a01--46fc--bca9--cfe092fbeb70: 0 262144 linear 253:0 10487808 d38153c3--0cfa--452f--9619--adadc00e27d7-4b44d4c3--4a8f--486b--954b--680fab94144c: 0 2097152 linear 253:0 8390656 d38153c3--0cfa--452f--9619--adadc00e27d7-3e5d50e0--02a1--421b--b249--4b561dfefd80: 0 262144 linear 253:0 10749952 d38153c3--0cfa--452f--9619--adadc00e27d7-master (253:6) └─3600a0980383035596d2b47677a426b61 (253:0) └─ (8:16) d38153c3--0cfa--452f--9619--adadc00e27d7-outbox (253:2) └─3600a0980383035596d2b47677a426b61 (253:0) └─ (8:16) d38153c3--0cfa--452f--9619--adadc00e27d7-376d4800--8b81--4326--9f51--0c0e97890e67 (253:10) └─3600a0980383035596d2b47677a426b61 (253:0) └─ (8:16) d38153c3--0cfa--452f--9619--adadc00e27d7-metadata (253:1) └─3600a0980383035596d2b47677a426b61 (253:0) └─ (8:16) d38153c3--0cfa--452f--9619--adadc00e27d7-inbox (253:5) └─3600a0980383035596d2b47677a426b61 (253:0) └─ (8:16) d38153c3--0cfa--452f--9619--adadc00e27d7-ids (253:4) └─3600a0980383035596d2b47677a426b61 (253:0) └─ (8:16) d38153c3--0cfa--452f--9619--adadc00e27d7-leases (253:3) └─3600a0980383035596d2b47677a426b61 (253:0) └─ (8:16) d38153c3--0cfa--452f--9619--adadc00e27d7-de655474--3a01--46fc--bca9--cfe092fbeb70 (253:8) └─3600a0980383035596d2b47677a426b61 (253:0) └─ (8:16) d38153c3--0cfa--452f--9619--adadc00e27d7-4b44d4c3--4a8f--486b--954b--680fab94144c (253:9) └─3600a0980383035596d2b47677a426b61 (253:0) └─ (8:16) d38153c3--0cfa--452f--9619--adadc00e27d7-3e5d50e0--02a1--421b--b249--4b561dfefd80 (253:7) └─3600a0980383035596d2b47677a426b61 (253:0) └─ (8:16)
[root@rose05 ~]# ls -al /rhev/data-center/mnt/blockSD/d38153c3-0cfa-452f-9619-adadc00e27d7/ha_agent/hosted-engine.metadata lrwxrwxrwx. 1 vdsm kvm 132 Nov 4 12:50 /rhev/data-center/mnt/blockSD/d38153c3-0cfa-452f-9619-adadc00e27d7/ha_agent/hosted-engine.metadata -> /var/run/vdsm/storage/d38153c3-0cfa-452f-9619-adadc00e27d7/0080728c-9e07-40e4-bfa9-0abe94f01bed/3e5d50e0-02a1-421b-b249-4b561dfefd80 [root@rose05 ~]# ls -al /var/run/vdsm/storage/d38153c3-0cfa-452f-9619-adadc00e27d7/0080728c-9e07-40e4-bfa9-0abe94f01bed/3e5d50e0-02a1-421b-b249-4b561dfefd80 lrwxrwxrwx. 1 vdsm kvm 78 Nov 3 17:18 /var/run/vdsm/storage/d38153c3-0cfa-452f-9619-adadc00e27d7/0080728c-9e07-40e4-bfa9-0abe94f01bed/3e5d50e0-02a1-421b-b249-4b561dfefd80 -> /dev/d38153c3-0cfa-452f-9619-adadc00e27d7/3e5d50e0-02a1-421b-b249-4b561dfefd80
[root@cyan-vdsf ~]# ls -al /rhev/data-center/mnt/blockSD/d38153c3-0cfa-452f-9619-adadc00e27d7/ha_agent/hosted-engine.metadata lrwxrwxrwx. 1 vdsm kvm 132 Nov 4 13:29 /rhev/data-center/mnt/blockSD/d38153c3-0cfa-452f-9619-adadc00e27d7/ha_agent/hosted-engine.metadata -> /var/run/vdsm/storage/d38153c3-0cfa-452f-9619-adadc00e27d7/0080728c-9e07-40e4-bfa9-0abe94f01bed/3e5d50e0-02a1-421b-b249-4b561dfefd80 [root@cyan-vdsf ~]# ls -al /var/run/vdsm/storage/d38153c3-0cfa-452f-9619-adadc00e27d7/0080728c-9e07-40e4-bfa9-0abe94f01bed/3e5d50e0-02a1-421b-b249-4b561dfefd80 lrwxrwxrwx. 1 vdsm kvm 78 Nov 3 17:13 /var/run/vdsm/storage/d38153c3-0cfa-452f-9619-adadc00e27d7/0080728c-9e07-40e4-bfa9-0abe94f01bed/3e5d50e0-02a1-421b-b249-4b561dfefd80 -> /dev/d38153c3-0cfa-452f-9619-adadc00e27d7/3e5d50e0-02a1-421b-b249-4b561dfefd80
[root@rose05 ~]# uname -a; cat /etc/redhat-release Linux rose05.qa.lab.tlv.redhat.com 3.10.0-327.el7.x86_64 #1 SMP Thu Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 7.2 (Maipo) [root@cyan-vdsf ~]# uname -a; cat /etc/redhat-release Linux cyan-vdsf.qa.lab.tlv.redhat.com 3.10.0-327.el7.x86_64 #1 SMP Thu Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 7.2 (Maipo)
Ok so I have good and bad news: the good news is that the following command returns proper data on both hosts: dd iflag=direct iflag=sync bs=1M count=1 if=/rhev/data-center/mnt/blockSD/d38153c3-0cfa-452f-9619-adadc00e27d7/ha_agent/hosted-engine.metadata of=dump.bin; hexdump -C dump.bin The bad news is that we disabled direct mode on LVM because of http://bugs.python.org/issue5396. We might be able to fix this using https://pypi.python.org/pypi/directio but the library is in alpha stage. The relevant lines in the code are in: ovirt_hosted_engine_ha.lib.storage_backends.FilesystemBackend#direct_io and ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker#get_raw_stats_for_service_type
And I have even better news, it is possible using mmap and I think I have working patch now.
oVirt 3.6.0 has been released on November 4th, re-targeting to 3.6.1 since this bug has been marked as urgent
Verified on ovirt-hosted-engine-ha-1.3.3-1.el7ev.noarch After block connection to storage and restore it, hosted-engine --vm-status show correct information on both hosts.
According to verification status and target milestone this issue should be fixed in oVirt 3.6.1. Closing current release.