Bug 1091140
| Summary: | VM <some name> is down. Exit message: cannot read header '/rhev/data-center/mnt/glusterSD/<host>:_GlusterStorage/<datacenter>/images/ < harddrive img>': Input/output error. | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Retired] oVirt | Reporter: | Filip Goll <filip.goll> | ||||
| Component: | ovirt-engine-core | Assignee: | Sahina Bose <sabose> | ||||
| Status: | CLOSED CANTFIX | QA Contact: | Pavel Stehlik <pstehlik> | ||||
| Severity: | medium | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 3.4 | CC: | acathrow, amureini, bugs, filip.goll, fsimonce, gklein, iheim, michal.skrivanek, sabose, spandura, yeylon | ||||
| Target Milestone: | --- | Flags: | amureini:
needinfo+
|
||||
| Target Release: | 3.5.0 | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | gluster | ||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2014-07-08 08:30:34 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | Gluster | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
Created attachment 889513 [details]
node hardware
Fede, could this be related to the missing links issues we've been seeing? (In reply to Allon Mureinik from comment #2) > Fede, could this be related to the missing links issues we've been seeing? Input/output errors on gluster generally means a split-brain. (In reply to Federico Simoncelli from comment #3) > (In reply to Allon Mureinik from comment #2) > > Fede, could this be related to the missing links issues we've been seeing? > > Input/output errors on gluster generally means a split-brain. Fabulous. So how do we proceed here? Hi Filip are you familiar with gluster? Can you check if the file has been affected by a split brain? Hi Fede - youre right. Split-brain on both disk drives: #gluster volume heal GlusterStorage info split-brain 2014-05-15 08:23:48 /a166ea56-2c22-43ce-9c5c-57414a468914/dom_md/ids 2014-05-15 08:23:48 /a166ea56-2c22-43ce-9c5c-57414a468914/images/b79cecdd-f452-4e0c-be56-fe28d207462a/29e886b0-12b6-4b6d-bf5f-7ff44dc1082b 2014-05-15 08:23:48 /a166ea56-2c22-43ce-9c5c-57414a468914/images/03500661-8ae8-4454-b013-1facc2b2fb93/358a7349-9e28-4bc6-a528-a160996e7509 glusterfshd.log [2014-05-15 08:23:48.364535] E [afr-self-heal-common.c:197:afr_sh_print_split_brain_log] 0-GlusterStorage-replicate-0: Unable to self-heal contents of '<gfid:dd5c28f7-6eb5-43aa-8cf1-90c402711e77>' (possible split-brain). Please delete the file from all but the preferred subvolume.- Pending matrix: [ [ 0 99777 ] [ 31 0 ] ] [2014-05-15 08:23:48.368185] E [afr-self-heal-common.c:197:afr_sh_print_split_brain_log] 0-GlusterStorage-replicate-0: Unable to self-heal contents of '<gfid:77f5927e-d977-47e8-a53a-3e117439efee>' (possible split-brain). Please delete the file from all but the preferred subvolume.- Pending matrix: [ [ 0 1089493 ] [ 267 0 ] ] [2014-05-15 08:23:48.372100] E [afr-self-heal-common.c:197:afr_sh_print_split_brain_log] 0-GlusterStorage-replicate-0: Unable to self-heal contents of '<gfid:3c9ea6a4-7a5d-4a36-82b9-afb59efb40e1>' (possible split-brain). Please delete the file from all but the preferred subvolume.- Pending matrix: [ [ 0 13344 ] [ 59 0 ] ] (In reply to Filip Goll from comment #6) > Hi Fede - youre right. Split-brain on both disk drives: Based on this comment - moving to gluster. Sahina, can one of the gluster guys take a look please? Thanks! (In reply to Filip Goll from comment #6) > Hi Fede - youre right. Split-brain on both disk drives: Based on this comment - moving to gluster. Sahina, can one of the gluster guys take a look please? Thanks! [Removing needinfo from Flip - was added by mistake.] This is an automated message: This bug has been re-targeted from 3.4.2 to 3.5.0 since neither priority nor severity were high or urgent. Please re-target to 3.4.3 if relevant. If the files are in split brain state, the available option would be to fix the split brain situation manually. Adding Shwetha to help here. https://github.com/gluster/glusterfs/blob/master/doc/split-brain.md - Please follow this to manually fix the split brain. Unfortunately, there's nothing to be done from oVirt to fix this, so closing this as a CANT FIX. |
Description of problem: After reboot ovirt manager machine this error messages on two virt-machines. Version-Release number of selected component (if applicable): running on ovirt-engine-3.4.0-1.el6 engine.log: 2014-04-25 03:52:06,059 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-19) [52bf6671] Running command: RunVmCommand internal: false. Entities affected : ID: 80a0e341-b7c1-4ce9-9e68-3140890251d2 Type: VM 2014-04-25 03:52:06,114 INFO [org.ovirt.engine.core.bll.scheduling.policyunits.HaReservationWeightPolicyUnit] (org.ovirt.thread.pool-6-thread-19) [52bf6671] Started HA reservation scoring method 2014-04-25 03:52:06,157 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoDirectoryVDSCommand] (org.ovirt.thread.pool-6-thread-19) [52bf6671] START, IsoDirectoryVDSCommand( storagePoolId = 00000002-0002-0002-0002-000000000164, ignoreFailoverLimit = false), log id: 261d4e4d 2014-04-25 03:52:06,158 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoDirectoryVDSCommand] (org.ovirt.thread.pool-6-thread-19) [52bf6671] FINISH, IsoDirectoryVDSCommand, return: \\172.30.1.7\CD, log id: 261d4e4d 2014-04-25 03:52:06,158 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-19) [52bf6671] Running VM with attached cd \\172.30.1.7\CD/RHEV-toolsSetup_3.3_11.iso 2014-04-25 03:52:06,179 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.IsoPrefixVDSCommand] (org.ovirt.thread.pool-6-thread-19) [52bf6671] START, IsoPrefixVDSCommand(HostName = SRV02.pcport.eu, HostId = 71afcf49-0d16-45e4-ad6d-18900f6ed1c1, storagePoolId=00000002-0002-0002-0002-000000000164), log id: 381f1413 2014-04-25 03:52:06,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.IsoPrefixVDSCommand] (org.ovirt.thread.pool-6-thread-19) [52bf6671] FINISH, IsoPrefixVDSCommand, return: /rhev/data-center/mnt/172.30.1.100:_NFS-INSTALL/aa139b8c-833c-4823-bed2-7496b8c8317b/images/11111111-1111-1111-1111-111111111111, log id: 381f1413 2014-04-25 03:52:06,202 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-19) [52bf6671] START, CreateVmVDSCommand(HostName = SRV02.pcport.eu, HostId = 71afcf49-0d16-45e4-ad6d-18900f6ed1c1, vmId=80a0e341-b7c1-4ce9-9e68-3140890251d2, vm=VM [Azmera-bck]), log id: 37f1dbba 2014-04-25 03:52:06,223 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-19) [52bf6671] START, CreateVDSCommand(HostName = SRV02.pcport.eu, HostId = 71afcf49-0d16-45e4-ad6d-18900f6ed1c1, vmId=80a0e341-b7c1-4ce9-9e68-3140890251d2, vm=VM [Azmera-bck]), log id: 8a088fc 2014-04-25 03:52:06,255 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-19) [52bf6671] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=6144,kvmEnable=true,smp=4,vmType=kvm,emulatedMachine=rhel6.5.0,keyboardLayout=en-us,memGuaranteedSize=6144,nice=0,display=vnc,smartcardEnable=false,tabletEnable=true,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,maxVCpus=160,timeOffset=3600,transparentHugePages=true,vmId=80a0e341-b7c1-4ce9-9e68-3140890251d2,devices=[{specParams={vram=32768, heads=1}, device=cirrus, type=video, deviceId=159d125b-042c-4d23-b7ee-a10af96692d6}, {shared=false, iface=ide, index=2, specParams={path=}, path=/rhev/data-center/mnt/172.30.1.100:_NFS-INSTALL/aa139b8c-833c-4823-bed2-7496b8c8317b/images/11111111-1111-1111-1111-111111111111/RHEV-toolsSetup_3.3_11.iso, device=cdrom, type=disk, readonly=true, deviceId=f0b82000-2425-4a9a-87b1-1ad8fa945a7e}, {shared=false, index=0, volumeID=29e886b0-12b6-4b6d-bf5f-7ff44dc1082b, propagateErrors=off, format=raw, type=disk, iface=virtio, bootOrder=1, domainID=a166ea56-2c22-43ce-9c5c-57414a468914, imageID=b79cecdd-f452-4e0c-be56-fe28d207462a, specParams={}, optional=false, device=disk, poolID=00000002-0002-0002-0002-000000000164, readonly=true, deviceId=b79cecdd-f452-4e0c-be56-fe28d207462a}, {shared=false, volumeID=424125d8-69eb-48aa-bb17-9ec380d81399, iface=virtio, imageID=c1654080-7ad6-4fc0-ab1f-7a9366f76efb, domainID=a166ea56-2c22-43ce-9c5c-57414a468914, specParams={}, optional=false, propagateErrors=off, device=disk, poolID=00000002-0002-0002-0002-000000000164, format=raw, type=disk, readonly=false, deviceId=c1654080-7ad6-4fc0-ab1f-7a9366f76efb}, {nicModel=pv, specParams={}, macAddr=00:01:a4:a0:3d:32, device=bridge, linkActive=true, type=interface, filter=vdsm-no-mac-spoofing, network=SERVERS, deviceId=d4002bc4-d588-4f54-a610-7208301d8e00}],acpiEnable=true,vmName=Azmera-bck,cpuType=Nehalem,custom={} 2014-04-25 03:52:06,260 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-19) [52bf6671] FINISH, CreateVDSCommand, log id: 8a088fc 2014-04-25 03:52:06,293 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-19) [52bf6671] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 37f1dbba 2014-04-25 03:52:06,294 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-19) [52bf6671] Lock freed to object EngineLock [exclusiveLocks= key: 80a0e341-b7c1-4ce9-9e68-3140890251d2 value: VM , sharedLocks= ] 2014-04-25 03:52:06,303 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-19) [52bf6671] Correlation ID: 52bf6671, Job ID: 3b0a089d-385b-47e0-8091-ee3bdefa2577, Call Stack: null, Custom Event ID: -1, Message: VM Azmera-bck was started by goll (Host: SRV02.pcport.eu). 2014-04-25 03:52:09,140 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-43) START, GlusterVolumesListVDSCommand(HostName = SRV08.pcport.eu, HostId = 077dc166-bc49-4dba-bbf4-525dc6e659aa), log id: eb9d6f9 2014-04-25 03:52:09,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-31) START, DestroyVDSCommand(HostName = SRV02.pcport.eu, HostId = 71afcf49-0d16-45e4-ad6d-18900f6ed1c1, vmId=80a0e341-b7c1-4ce9-9e68-3140890251d2, force=false, secondsToWait=0, gracefully=false), log id: 68ce7b41 2014-04-25 03:52:09,285 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-31) FINISH, DestroyVDSCommand, log id: 68ce7b41 2014-04-25 03:52:09,342 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM Azmera-bck is down. Exit message: cannot read header '/rhev/data-center/mnt/glusterSD/172.30.1.8:_GlusterStorage/a166ea56-2c22-43ce-9c5c-57414a468914/images/b79cecdd-f452-4e0c-be56-fe28d207462a/29e886b0-12b6-4b6d-bf5f-7ff44dc1082b': Input/output error. 2014-04-25 03:52:09,344 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-31) Running on vds during rerun failed vm: null 2014-04-25 03:52:09,354 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-31) VM Azmera-bck (80a0e341-b7c1-4ce9-9e68-3140890251d2) is running in db and not running in VDS SRV02.pcport.eu 2014-04-25 03:52:09,427 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-31) Rerun vm 80a0e341-b7c1-4ce9-9e68-3140890251d2. Called from vds SRV02.pcport.eu 2014-04-25 03:52:09,439 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-42) Correlation ID: 52bf6671, Job ID: 3b0a089d-385b-47e0-8091-ee3bdefa2577, Call Stack: null, Custom Event ID: -1, Message: Failed to run VM Azmera-bck on Host SRV02.pcport.eu. 2014-04-25 03:52:09,455 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-42) Lock Acquired to object EngineLock [exclusiveLocks= key: 80a0e341-b7c1-4ce9-9e68-3140890251d2 value: VM , sharedLocks= ] 2014-04-25 03:52:09,460 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-42) START, IsVmDuringInitiatingVDSCommand( vmId = 80a0e341-b7c1-4ce9-9e68-3140890251d2), log id: 324c4454 2014-04-25 03:52:09,460 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-42) FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 324c4454 2014-04-25 03:52:09,523 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-42) Running command: RunVmCommand internal: false. Entities affected : ID: 80a0e341-b7c1-4ce9-9e68-3140890251d2 Type: VM 2014-04-25 03:52:09,574 INFO [org.ovirt.engine.core.bll.scheduling.policyunits.HaReservationWeightPolicyUnit] (org.ovirt.thread.pool-6-thread-42) Started HA reservation scoring method 2014-04-25 03:52:09,614 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoDirectoryVDSCommand] (org.ovirt.thread.pool-6-thread-42) START, IsoDirectoryVDSCommand( storagePoolId = 00000002-0002-0002-0002-000000000164, ignoreFailoverLimit = false), log id: 3b47a9bb 2014-04-25 03:52:09,615 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoDirectoryVDSCommand] (org.ovirt.thread.pool-6-thread-42) FINISH, IsoDirectoryVDSCommand, return: \\172.30.1.7\CD, log id: 3b47a9bb 2014-04-25 03:52:09,615 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-42) Running VM with attached cd \\172.30.1.7\CD/RHEV-toolsSetup_3.3_11.iso 2014-04-25 03:52:09,634 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.IsoPrefixVDSCommand] (org.ovirt.thread.pool-6-thread-42) START, IsoPrefixVDSCommand(HostName = SRV03.pcport.eu, HostId = 126416fd-413e-4fb3-bf35-ed3cd08895ca, storagePoolId=00000002-0002-0002-0002-000000000164), log id: 5dd85264 2014-04-25 03:52:09,635 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.IsoPrefixVDSCommand] (org.ovirt.thread.pool-6-thread-42) FINISH, IsoPrefixVDSCommand, return: /rhev/data-center/mnt/172.30.1.100:_NFS-INSTALL/aa139b8c-833c-4823-bed2-7496b8c8317b/images/11111111-1111-1111-1111-111111111111, log id: 5dd85264 2014-04-25 03:52:09,658 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-42) START, CreateVmVDSCommand(HostName = SRV03.pcport.eu, HostId = 126416fd-413e-4fb3-bf35-ed3cd08895ca, vmId=80a0e341-b7c1-4ce9-9e68-3140890251d2, vm=VM [Azmera-bck]), log id: 2c6777b0 2014-04-25 03:52:09,679 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-42) START, CreateVDSCommand(HostName = SRV03.pcport.eu, HostId = 126416fd-413e-4fb3-bf35-ed3cd08895ca, vmId=80a0e341-b7c1-4ce9-9e68-3140890251d2, vm=VM [Azmera-bck]), log id: 1f3ed745 2014-04-25 03:52:09,711 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-42) org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=6144,kvmEnable=true,smp=4,vmType=kvm,emulatedMachine=rhel6.5.0,keyboardLayout=en-us,memGuaranteedSize=6144,nice=0,display=vnc,smartcardEnable=false,tabletEnable=true,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,maxVCpus=160,timeOffset=3600,transparentHugePages=true,vmId=80a0e341-b7c1-4ce9-9e68-3140890251d2,devices=[{specParams={vram=32768, heads=1}, device=cirrus, type=video, deviceId=159d125b-042c-4d23-b7ee-a10af96692d6}, {shared=false, iface=ide, index=2, specParams={path=}, path=/rhev/data-center/mnt/172.30.1.100:_NFS-INSTALL/aa139b8c-833c-4823-bed2-7496b8c8317b/images/11111111-1111-1111-1111-111111111111/RHEV-toolsSetup_3.3_11.iso, device=cdrom, type=disk, readonly=true, deviceId=f0b82000-2425-4a9a-87b1-1ad8fa945a7e}, {shared=false, index=0, volumeID=29e886b0-12b6-4b6d-bf5f-7ff44dc1082b, propagateErrors=off, format=raw, type=disk, iface=virtio, bootOrder=1, domainID=a166ea56-2c22-43ce-9c5c-57414a468914, imageID=b79cecdd-f452-4e0c-be56-fe28d207462a, specParams={}, optional=false, device=disk, poolID=00000002-0002-0002-0002-000000000164, readonly=true, deviceId=b79cecdd-f452-4e0c-be56-fe28d207462a}, {shared=false, volumeID=424125d8-69eb-48aa-bb17-9ec380d81399, iface=virtio, imageID=c1654080-7ad6-4fc0-ab1f-7a9366f76efb, domainID=a166ea56-2c22-43ce-9c5c-57414a468914, specParams={}, optional=false, propagateErrors=off, device=disk, poolID=00000002-0002-0002-0002-000000000164, format=raw, type=disk, readonly=false, deviceId=c1654080-7ad6-4fc0-ab1f-7a9366f76efb}, {nicModel=pv, specParams={}, macAddr=00:01:a4:a0:3d:32, device=bridge, linkActive=true, type=interface, filter=vdsm-no-mac-spoofing, network=SERVERS, deviceId=d4002bc4-d588-4f54-a610-7208301d8e00}],acpiEnable=true,vmName=Azmera-bck,cpuType=Nehalem,custom={} 2014-04-25 03:52:09,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-42) FINISH, CreateVDSCommand, log id: 1f3ed745 2014-04-25 03:52:09,733 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-42) FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 2c6777b0 2014-04-25 03:52:09,733 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-42) Lock freed to object EngineLock [exclusiveLocks= key: 80a0e341-b7c1-4ce9-9e68-3140890251d2 value: VM , sharedLocks= ] 2014-04-25 03:52:09,739 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-42) Correlation ID: 52bf6671, Job ID: 3b0a089d-385b-47e0-8091-ee3bdefa2577, Call Stack: null, Custom Event ID: -1, Message: VM Azmera-bck was started by goll (Host: SRV03.pcport.eu). 2014-04-25 03:52:09,878 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-43) FINISH, GlusterVolumesListVDSCommand, return: {ffa882e6-fa14-489b-9a01-1363392be0ef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@5e7d10ac}, log id: eb9d6f9 2014-04-25 03:52:10,263 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-37) START, DestroyVDSCommand(HostName = SRV03.pcport.eu, HostId = 126416fd-413e-4fb3-bf35-ed3cd08895ca, vmId=80a0e341-b7c1-4ce9-9e68-3140890251d2, force=false, secondsToWait=0, gracefully=false), log id: fcc242b 2014-04-25 03:52:10,312 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-37) FINISH, DestroyVDSCommand, log id: fcc242b 2014-04-25 03:52:10,342 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM Azmera-bck is down. Exit message: Bad volume specification {'index': 0, 'iface': 'virtio', 'reqsize': '0', 'format': 'raw', 'bootOrder': '1', 'volumeID': '29e886b0-12b6-4b6d-bf5f-7ff44dc1082b', 'apparentsize': '64424509440', 'imageID': 'b79cecdd-f452-4e0c-be56-fe28d207462a', 'specParams': {}, 'readonly': 'true', 'domainID': 'a166ea56-2c22-43ce-9c5c-57414a468914', 'optional': 'false', 'deviceId': 'b79cecdd-f452-4e0c-be56-fe28d207462a', 'truesize': '64424509440', 'poolID': '00000002-0002-0002-0002-000000000164', 'device': 'disk', 'shared': 'false', 'propagateErrors': 'off', 'type': 'disk'}. 2014-04-25 03:52:10,344 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-37) Running on vds during rerun failed vm: null 2014-04-25 03:52:10,345 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-37) VM Azmera-bck (80a0e341-b7c1-4ce9-9e68-3140890251d2) is running in db and not running in VDS SRV03.pcport.eu 2014-04-25 03:52:10,364 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-37) Rerun vm 80a0e341-b7c1-4ce9-9e68-3140890251d2. Called from vds SRV03.pcport.eu 2014-04-25 03:52:10,375 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-7) Correlation ID: 52bf6671, Job ID: 3b0a089d-385b-47e0-8091-ee3bdefa2577, Call Stack: null, Custom Event ID: -1, Message: Failed to run VM Azmera-bck on Host SRV03.pcport.eu. 2014-04-25 03:52:10,392 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-7) Lock Acquired to object EngineLock [exclusiveLocks= key: 80a0e341-b7c1-4ce9-9e68-3140890251d2 value: VM , sharedLocks= ] 2014-04-25 03:52:10,396 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-7) START, IsVmDuringInitiatingVDSCommand( vmId = 80a0e341-b7c1-4ce9-9e68-3140890251d2), log id: 5bdb0768 2014-04-25 03:52:10,396 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-7) FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 5bdb0768 2014-04-25 03:52:10,456 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-7) Running command: RunVmCommand internal: false. Entities affected : ID: 80a0e341-b7c1-4ce9-9e68-3140890251d2 Type: VM 2014-04-25 03:52:10,506 INFO [org.ovirt.engine.core.bll.scheduling.policyunits.HaReservationWeightPolicyUnit] (org.ovirt.thread.pool-6-thread-7) Started HA reservation scoring method 2014-04-25 03:52:10,526 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoDirectoryVDSCommand] (org.ovirt.thread.pool-6-thread-7) START, IsoDirectoryVDSCommand( storagePoolId = 00000002-0002-0002-0002-000000000164, ignoreFailoverLimit = false), log id: 71dbcab0 2014-04-25 03:52:10,527 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoDirectoryVDSCommand] (org.ovirt.thread.pool-6-thread-7) FINISH, IsoDirectoryVDSCommand, return: \\172.30.1.7\CD, log id: 71dbcab0 2014-04-25 03:52:10,527 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-7) Running VM with attached cd \\172.30.1.7\CD/RHEV-toolsSetup_3.3_11.iso 2014-04-25 03:52:10,528 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.IsoPrefixVDSCommand] (org.ovirt.thread.pool-6-thread-7) START, IsoPrefixVDSCommand(HostName = SRV01.pcport.eu, HostId = c9831bf0-c40c-4343-9b09-2a65c700f0c5, storagePoolId=00000002-0002-0002-0002-000000000164), log id: 14d82336 2014-04-25 03:52:10,529 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.IsoPrefixVDSCommand] (org.ovirt.thread.pool-6-thread-7) FINISH, IsoPrefixVDSCommand, return: /rhev/data-center/mnt/172.30.1.100:_NFS-INSTALL/aa139b8c-833c-4823-bed2-7496b8c8317b/images/11111111-1111-1111-1111-111111111111, log id: 14d82336 2014-04-25 03:52:10,533 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-7) START, CreateVmVDSCommand(HostName = SRV01.pcport.eu, HostId = c9831bf0-c40c-4343-9b09-2a65c700f0c5, vmId=80a0e341-b7c1-4ce9-9e68-3140890251d2, vm=VM [Azmera-bck]), log id: 6ec2f2 2014-04-25 03:52:10,535 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-7) START, CreateVDSCommand(HostName = SRV01.pcport.eu, HostId = c9831bf0-c40c-4343-9b09-2a65c700f0c5, vmId=80a0e341-b7c1-4ce9-9e68-3140890251d2, vm=VM [Azmera-bck]), log id: 3dcbf1f4 2014-04-25 03:52:10,583 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-7) org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=6144,kvmEnable=true,smp=4,vmType=kvm,emulatedMachine=rhel6.5.0,keyboardLayout=en-us,memGuaranteedSize=6144,nice=0,display=vnc,smartcardEnable=false,tabletEnable=true,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,maxVCpus=160,timeOffset=3600,transparentHugePages=true,vmId=80a0e341-b7c1-4ce9-9e68-3140890251d2,devices=[{specParams={vram=32768, heads=1}, device=cirrus, type=video, deviceId=159d125b-042c-4d23-b7ee-a10af96692d6}, {shared=false, iface=ide, index=2, specParams={path=}, path=/rhev/data-center/mnt/172.30.1.100:_NFS-INSTALL/aa139b8c-833c-4823-bed2-7496b8c8317b/images/11111111-1111-1111-1111-111111111111/RHEV-toolsSetup_3.3_11.iso, device=cdrom, type=disk, readonly=true, deviceId=f0b82000-2425-4a9a-87b1-1ad8fa945a7e}, {shared=false, index=0, volumeID=29e886b0-12b6-4b6d-bf5f-7ff44dc1082b, propagateErrors=off, format=raw, type=disk, iface=virtio, bootOrder=1, domainID=a166ea56-2c22-43ce-9c5c-57414a468914, imageID=b79cecdd-f452-4e0c-be56-fe28d207462a, specParams={}, optional=false, device=disk, poolID=00000002-0002-0002-0002-000000000164, readonly=true, deviceId=b79cecdd-f452-4e0c-be56-fe28d207462a}, {shared=false, volumeID=424125d8-69eb-48aa-bb17-9ec380d81399, iface=virtio, imageID=c1654080-7ad6-4fc0-ab1f-7a9366f76efb, domainID=a166ea56-2c22-43ce-9c5c-57414a468914, specParams={}, optional=false, propagateErrors=off, device=disk, poolID=00000002-0002-0002-0002-000000000164, format=raw, type=disk, readonly=false, deviceId=c1654080-7ad6-4fc0-ab1f-7a9366f76efb}, {nicModel=pv, specParams={}, macAddr=00:01:a4:a0:3d:32, device=bridge, linkActive=true, type=interface, filter=vdsm-no-mac-spoofing, network=SERVERS, deviceId=d4002bc4-d588-4f54-a610-7208301d8e00}],acpiEnable=true,vmName=Azmera-bck,cpuType=Nehalem,custom={} 2014-04-25 03:52:10,591 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-7) FINISH, CreateVDSCommand, log id: 3dcbf1f4 2014-04-25 03:52:10,603 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-7) FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 6ec2f2 2014-04-25 03:52:10,603 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-7) Lock freed to object EngineLock [exclusiveLocks= key: 80a0e341-b7c1-4ce9-9e68-3140890251d2 value: VM , sharedLocks= ] 2014-04-25 03:52:10,609 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-7) Correlation ID: 52bf6671, Job ID: 3b0a089d-385b-47e0-8091-ee3bdefa2577, Call Stack: null, Custom Event ID: -1, Message: VM Azmera-bck was started by goll (Host: SRV01.pcport.eu). 2014-04-25 03:52:11,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-42) START, DestroyVDSCommand(HostName = SRV01.pcport.eu, HostId = c9831bf0-c40c-4343-9b09-2a65c700f0c5, vmId=80a0e341-b7c1-4ce9-9e68-3140890251d2, force=false, secondsToWait=0, gracefully=false), log id: c1be7df 2014-04-25 03:52:11,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler_Worker-42) FINISH, DestroyVDSCommand, log id: c1be7df 2014-04-25 03:52:11,333 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-42) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM Azmera-bck is down. Exit message: cannot read header '/rhev/data-center/mnt/glusterSD/172.30.1.8:_GlusterStorage/a166ea56-2c22-43ce-9c5c-57414a468914/images/b79cecdd-f452-4e0c-be56-fe28d207462a/29e886b0-12b6-4b6d-bf5f-7ff44dc1082b': Input/output error. 2014-04-25 03:52:11,334 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-42) Running on vds during rerun failed vm: null 2014-04-25 03:52:11,335 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-42) VM Azmera-bck (80a0e341-b7c1-4ce9-9e68-3140890251d2) is running in db and not running in VDS SRV01.pcport.eu 2014-04-25 03:52:11,360 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-42) Rerun vm 80a0e341-b7c1-4ce9-9e68-3140890251d2. Called from vds SRV01.pcport.eu 2014-04-25 03:52:11,374 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-35) Correlation ID: 52bf6671, Job ID: 3b0a089d-385b-47e0-8091-ee3bdefa2577, Call Stack: null, Custom Event ID: -1, Message: Failed to run VM Azmera-bck on Host SRV01.pcport.eu. 2014-04-25 03:52:11,389 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-35) Correlation ID: 52bf6671, Job ID: 3b0a089d-385b-47e0-8091-ee3bdefa2577, Call Stack: null, Custom Event ID: -1, Message: Failed to run VM Azmera-bck (User: goll).