Attachment 703829 Details for Bug 916485 – engine.log

engine.log

engine.log (text/plain), 32.83 KB, created by cshao on 2013-02-28 07:31:33 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: cshao

Created: 2013-02-28 07:31:33 UTC

Size: 32.83 KB

patch

obsolete

>2013-02-28 14:24:52,669 ERROR [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-22) VDS::handleNetworkException Server failed to respond,  vds_id = d876eea4-816c-11e2-b1f7-e76995059d3b, vds_name = hp-bl465cg5-02.qe.lab.eng.nay.redhat.com, error = VDSNetworkException: 
>2013-02-28 14:24:52,683 INFO  [org.ovirt.engine.core.bll.VdsEventListener] (pool-3-thread-46) ResourceManager::vdsNotResponding entered for Host d876eea4-816c-11e2-b1f7-e76995059d3b, 10.66.73.4
>2013-02-28 14:24:52,703 WARN  [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (pool-3-thread-46) [267eeaab] CanDoAction of action VdsNotRespondingTreatment failed. Reasons:VDS_FENCING_DISABLED
>2013-02-28 14:25:00,001 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) Checking autorecoverable hosts
>2013-02-28 14:25:00,003 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) Autorecovering 0 hosts
>2013-02-28 14:25:00,003 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) Checking autorecoverable hosts done
>2013-02-28 14:25:00,004 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) Autorecovering storage domains is disabled, skipping
>2013-02-28 14:30:00,001 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-49) Checking autorecoverable hosts
>2013-02-28 14:30:00,003 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-49) Autorecovering 0 hosts
>2013-02-28 14:30:00,004 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-49) Checking autorecoverable hosts done
>2013-02-28 14:30:00,004 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-49) Autorecovering storage domains is disabled, skipping
>2013-02-28 14:35:00,000 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-11) Checking autorecoverable hosts
>2013-02-28 14:35:00,003 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-11) Autorecovering 0 hosts
>2013-02-28 14:35:00,003 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-11) Checking autorecoverable hosts done
>2013-02-28 14:35:00,004 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-11) Autorecovering storage domains is disabled, skipping
>2013-02-28 14:35:46,542 INFO  [org.ovirt.engine.core.bll.RemoveVdsCommand] (ajp-/127.0.0.1:8702-6) Lock Acquired to object EngineLock [exclusiveLocks= key: c68a151e-816b-11e2-8d6a-c3d98783a924 value: VDS
>, sharedLocks= ]
>2013-02-28 14:35:46,555 INFO  [org.ovirt.engine.core.bll.RemoveVdsCommand] (pool-3-thread-42) [311729f] Running command: RemoveVdsCommand internal: false. Entities affected :  ID: c68a151e-816b-11e2-8d6a-c3d98783a924 Type: VDS
>2013-02-28 14:35:46,592 INFO  [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (pool-3-thread-42) [311729f] START, RemoveVdsVDSCommand( HostId = c68a151e-816b-11e2-8d6a-c3d98783a924), log id: 6078791
>2013-02-28 14:35:46,592 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-3-thread-42) [311729f] vdsManager::disposing
>2013-02-28 14:35:46,592 INFO  [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (pool-3-thread-42) [311729f] FINISH, RemoveVdsVDSCommand, log id: 6078791
>2013-02-28 14:35:46,593 INFO  [org.ovirt.engine.core.bll.RemoveVdsCommand] (pool-3-thread-42) [311729f] Lock freed to object EngineLock [exclusiveLocks= key: c68a151e-816b-11e2-8d6a-c3d98783a924 value: VDS
>, sharedLocks= ]
>2013-02-28 14:35:49,032 INFO  [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-3-thread-42) [4ba2a7d9] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected :  ID: d876eea4-816c-11e2-b1f7-e76995059d3b Type: VDS
>2013-02-28 14:35:49,065 INFO  [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-3-thread-42) [4ba2a7d9] Running command: MaintananceVdsCommand internal: true. Entities affected :  ID: d876eea4-816c-11e2-b1f7-e76995059d3b Type: VDS
>2013-02-28 14:35:49,069 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-42) [4ba2a7d9] START, SetVdsStatusVDSCommand(HostName = hp-bl465cg5-02.qe.lab.eng.nay.redhat.com, HostId = d876eea4-816c-11e2-b1f7-e76995059d3b, status=Maintenance, nonOperationalReason=NONE), log id: 2f7d55aa
>2013-02-28 14:36:01,751 INFO  [org.ovirt.engine.core.bll.storage.RemoveStoragePoolCommand] (ajp-/127.0.0.1:8702-7) [1b2f0e9d] Running command: RemoveStoragePoolCommand internal: false. Entities affected :  ID: 86cbc979-468e-42b0-99e9-b3b1afeb31fd Type: StoragePool
>2013-02-28 14:36:20,859 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-42) [4ba2a7d9] FINISH, SetVdsStatusVDSCommand, log id: 2f7d55aa
>2013-02-28 14:36:55,965 INFO  [org.ovirt.engine.core.bll.RemoveVdsGroupCommand] (pool-3-thread-42) [624608d9] Running command: RemoveVdsGroupCommand internal: false. Entities affected :  ID: 5401aed0-814d-11e2-b232-a7aac05aca28 Type: VdsGroups
>2013-02-28 14:37:03,873 INFO  [org.ovirt.engine.core.bll.storage.AddEmptyStoragePoolCommand] (ajp-/127.0.0.1:8702-6) [3a270dde] Running command: AddEmptyStoragePoolCommand internal: false. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
>2013-02-28 14:37:18,776 INFO  [org.ovirt.engine.core.bll.AddVdsGroupCommand] (ajp-/127.0.0.1:8702-11) [4cf97689] Running command: AddVdsGroupCommand internal: false. Entities affected :  ID: 7cdaa6bd-0030-40e3-9d52-4b519ebf98fe Type: StoragePool
>2013-02-28 14:40:00,000 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-100) Checking autorecoverable hosts
>2013-02-28 14:40:00,009 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-100) Autorecovering 0 hosts
>2013-02-28 14:40:00,009 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-100) Checking autorecoverable hosts done
>2013-02-28 14:40:00,009 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-100) Autorecovering storage domains is disabled, skipping
>2013-02-28 14:44:57,146 INFO  [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-1) Checking if user admin@internal is an admin, result true
>2013-02-28 14:44:57,147 INFO  [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginAdminUserCommand internal: false.
>2013-02-28 14:45:00,000 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [b6dff4e] Checking autorecoverable hosts
>2013-02-28 14:45:00,006 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [b6dff4e] Autorecovering 0 hosts
>2013-02-28 14:45:00,006 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [b6dff4e] Checking autorecoverable hosts done
>2013-02-28 14:45:00,006 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [b6dff4e] Autorecovering storage domains is disabled, skipping
>2013-02-28 14:48:01,263 INFO  [org.ovirt.engine.core.bll.RegisterVdsQuery] (ajp-/127.0.0.1:8702-13) Running Command: RegisterVds
>2013-02-28 14:48:01,364 INFO  [org.ovirt.engine.core.bll.AddVdsCommand] (pool-3-thread-42) [4a421081] Running command: AddVdsCommand internal: true. Entities affected :  ID: 99408929-82cf-4dc7-a532-9d998063fa95 Type: VdsGroups
>2013-02-28 14:48:01,387 INFO  [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (pool-3-thread-42) [7963efba] Running command: AddVdsSpmIdCommand internal: true. Entities affected :  ID: cb6ce23a-8172-11e2-ab7d-e700038e6d40 Type: VDS
>2013-02-28 14:48:01,394 INFO  [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (pool-3-thread-42) [7963efba] Lock freed to object EngineLock [exclusiveLocks= key: a3d6b688-7f10-11e2-8bba-6b35298a1796 value: REGISTER_VDS
>, sharedLocks= ]
>2013-02-28 14:48:01,394 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (pool-3-thread-42) [7963efba] Cannot get vdsManager for vdsid=cb6ce23a-8172-11e2-ab7d-e700038e6d40
>2013-02-28 14:48:01,396 INFO  [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (pool-3-thread-42) [7963efba] START, RemoveVdsVDSCommand(HostName = dell-per510-01.qe.lab.eng.nay.redhat.com, HostId = cb6ce23a-8172-11e2-ab7d-e700038e6d40), log id: 6f734b48
>2013-02-28 14:48:01,396 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (pool-3-thread-42) [7963efba] Cannot get vdsManager for vdsid=cb6ce23a-8172-11e2-ab7d-e700038e6d40
>2013-02-28 14:48:01,396 INFO  [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (pool-3-thread-42) [7963efba] FINISH, RemoveVdsVDSCommand, log id: 6f734b48
>2013-02-28 14:48:01,396 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (pool-3-thread-42) [7963efba] Cannot get vdsManager for vdsid=cb6ce23a-8172-11e2-ab7d-e700038e6d40
>2013-02-28 14:48:01,398 INFO  [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (pool-3-thread-42) [7963efba] START, AddVdsVDSCommand(HostName = dell-per510-01.qe.lab.eng.nay.redhat.com, HostId = cb6ce23a-8172-11e2-ab7d-e700038e6d40), log id: 7c45509d
>2013-02-28 14:48:01,398 INFO  [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (pool-3-thread-42) [7963efba] AddVds - entered , starting logic to add VDS cb6ce23a-8172-11e2-ab7d-e700038e6d40
>2013-02-28 14:48:01,417 INFO  [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (pool-3-thread-42) [7963efba] AddVds - VDS cb6ce23a-8172-11e2-ab7d-e700038e6d40 was added, will try to add it to the resource manager
>2013-02-28 14:48:01,418 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-3-thread-42) [7963efba] Eneterd VdsManager:constructor
>2013-02-28 14:48:01,418 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-3-thread-42) [7963efba] vdsBroker(10.66.110.242,54,321)
>2013-02-28 14:48:01,419 INFO  [org.ovirt.engine.core.vdsbroker.ResourceManager] (pool-3-thread-42) [7963efba] ResourceManager::AddVds - VDS cb6ce23a-8172-11e2-ab7d-e700038e6d40 was added to the Resource Manager
>2013-02-28 14:48:01,419 INFO  [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (pool-3-thread-42) [7963efba] FINISH, AddVdsVDSCommand, log id: 7c45509d
>2013-02-28 14:48:01,439 INFO  [org.ovirt.engine.core.bll.RegisterVdsQuery] (ajp-/127.0.0.1:8702-13) RegisterVdsQuery::Register - Registered a new VDS pending approval - Name: dell-per510-01.qe.lab.eng.nay.redhat.com, Hostname: 10.66.110.242, UniqueID: 4C4C4544-0058-3810-8033-B2C04F363358
>2013-02-28 14:48:01,451 INFO  [org.ovirt.engine.core.register.RegisterServlet] (ajp-/127.0.0.1:8702-13) Succeeded to run RegisterVds.
>2013-02-28 14:49:34,002 INFO  [org.ovirt.engine.core.bll.ChangeVDSClusterCommand] (ajp-/127.0.0.1:8702-12) [6b403c6b] Running command: ChangeVDSClusterCommand internal: false. Entities affected :  ID: cb6ce23a-8172-11e2-ab7d-e700038e6d40 Type: VDS,  ID: 4c695a14-8171-11e2-8cb6-bb00e0e99e8c Type: VdsGroups
>2013-02-28 14:49:34,027 INFO  [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-12) [6b403c6b] Running command: AddVdsSpmIdCommand internal: true. Entities affected :  ID: cb6ce23a-8172-11e2-ab7d-e700038e6d40 Type: VDS
>2013-02-28 14:49:34,036 INFO  [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-12) [6b403c6b] Lock freed to object EngineLock [exclusiveLocks= key: 7cdaa6bd-0030-40e3-9d52-4b519ebf98fe value: REGISTER_VDS
>, sharedLocks= ]
>2013-02-28 14:49:34,147 INFO  [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp-/127.0.0.1:8702-8) [7b919c5b] Running command: UpdateVdsCommand internal: false. Entities affected :  ID: cb6ce23a-8172-11e2-ab7d-e700038e6d40 Type: VDS
>2013-02-28 14:49:34,341 INFO  [org.ovirt.engine.core.bll.ApproveVdsCommand] (pool-3-thread-42) [34b9491a] Running command: ApproveVdsCommand internal: false. Entities affected :  ID: cb6ce23a-8172-11e2-ab7d-e700038e6d40 Type: VDS
>2013-02-28 14:49:34,341 INFO  [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-42) [34b9491a] Before Installation pool-3-thread-42, Powerclient/oVirtNode case: setting status to installing
>2013-02-28 14:49:34,344 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-42) [34b9491a] START, SetVdsStatusVDSCommand(HostName = dell-per510-01.qe.lab.eng.nay.redhat.com, HostId = cb6ce23a-8172-11e2-ab7d-e700038e6d40, status=Installing, nonOperationalReason=NONE), log id: 5eee8430
>2013-02-28 14:49:34,355 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-42) [34b9491a] FINISH, SetVdsStatusVDSCommand, log id: 5eee8430
>2013-02-28 14:49:34,357 INFO  [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-42) [34b9491a] Before Installation pool-3-thread-42
>2013-02-28 14:49:34,357 INFO  [org.ovirt.engine.core.bll.OVirtInstaller] (pool-3-thread-42) [34b9491a] Installation of 10.66.110.242. Executing oVirt installation stage. (Stage: Starting Host installation)
>2013-02-28 14:49:34,357 INFO  [org.ovirt.engine.core.bll.OVirtInstaller] (pool-3-thread-42) [34b9491a] Installation of 10.66.110.242. Executing oVirt installation stage. (Stage: Connecting to Host)
>2013-02-28 14:49:34,461 INFO  [org.ovirt.engine.core.bll.VdsInstaller] (pool-3-thread-42) [34b9491a] Installation of 10.66.110.242. Received message: <BSTRAP component='RHEV_INSTALL' status='OK' message='Connected to Host 10.66.110.242 with SSH key fingerprint: eb:3f:b8:0f:12:fc:5f:57:5f:94:12:c0:f5:d6:5a:23'/>. FYI. (Stage: Connecting to Host)
>2013-02-28 14:49:35,084 INFO  [org.ovirt.engine.core.bll.VdsInstaller] (pool-3-thread-42) [34b9491a] Installation of 10.66.110.242. Successfully connected to server ssh. (Stage: Connecting to Host)
>2013-02-28 14:49:35,084 INFO  [org.ovirt.engine.core.bll.OVirtInstaller] (pool-3-thread-42) [34b9491a] Installation of 10.66.110.242. Executing oVirt installation stage. (Stage: Running upgrade / reinstall script on oVirt Node)
>2013-02-28 14:49:35,084 INFO  [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-3-thread-42) SSH execute 10.66.110.242 '/usr/share/vdsm-reg/vdsm-gen-cert -O "rhevhhh" 10.66.110.242 9b05959f-8873-4134-995d-f4a13e52b3e3'
>2013-02-28 14:49:35,573 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-73) Initializing Host: dell-per510-01.qe.lab.eng.nay.redhat.com
>2013-02-28 14:49:37,008 INFO  [org.ovirt.engine.core.bll.RemoveVdsCommand] (ajp-/127.0.0.1:8702-6) Lock Acquired to object EngineLock [exclusiveLocks= key: d876eea4-816c-11e2-b1f7-e76995059d3b value: VDS
>, sharedLocks= ]
>2013-02-28 14:49:37,019 INFO  [org.ovirt.engine.core.bll.RemoveVdsCommand] (pool-3-thread-46) [3fdee7bd] Running command: RemoveVdsCommand internal: false. Entities affected :  ID: d876eea4-816c-11e2-b1f7-e76995059d3b Type: VDS
>2013-02-28 14:49:37,034 INFO  [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (pool-3-thread-46) [3fdee7bd] START, RemoveVdsVDSCommand( HostId = d876eea4-816c-11e2-b1f7-e76995059d3b), log id: 37f37934
>2013-02-28 14:49:37,034 INFO  [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-3-thread-46) [3fdee7bd] vdsManager::disposing
>2013-02-28 14:49:37,034 INFO  [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (pool-3-thread-46) [3fdee7bd] FINISH, RemoveVdsVDSCommand, log id: 37f37934
>2013-02-28 14:49:37,035 INFO  [org.ovirt.engine.core.bll.RemoveVdsCommand] (pool-3-thread-46) [3fdee7bd] Lock freed to object EngineLock [exclusiveLocks= key: d876eea4-816c-11e2-b1f7-e76995059d3b value: VDS
>, sharedLocks= ]
>2013-02-28 14:49:37,727 INFO  [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-30) Installation of 10.66.110.242. Received message: <BSTRAP component='Encryption setup' status='OK'/>. FYI. (Stage: Running upgrade / reinstall script on oVirt Node)
>2013-02-28 14:49:37,744 INFO  [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-30) Installation of 10.66.110.242. Received message: <BSTRAP component='RHEV_INSTALL' status='OK'/>. Stage completed. (Stage: Running upgrade / reinstall script on oVirt Node)
>2013-02-28 14:49:37,768 INFO  [org.ovirt.engine.core.bll.OVirtInstaller] (pool-3-thread-42) [34b9491a] Installation of 10.66.110.242. Executing oVirt installation stage. (Stage: Downloading certificate request from Host)
>2013-02-28 14:49:37,769 INFO  [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-3-thread-42) SSH receive 10.66.110.242:'/tmp/cert_9b05959f-8873-4134-995d-f4a13e52b3e3.req' '/etc/pki/ovirt-engine/requests/cert_9b05959f-8873-4134-995d-f4a13e52b3e3.req')
>2013-02-28 14:49:37,823 INFO  [org.ovirt.engine.core.bll.VdsInstaller] (pool-3-thread-42) [34b9491a] Installation of 10.66.110.242. successfully done sftp operation ( Stage: Downloading certificate request from Host)
>2013-02-28 14:49:37,823 INFO  [org.ovirt.engine.core.bll.VdsInstaller] (pool-3-thread-42) [34b9491a]  DownloadCertificateRequest ended:true
>2013-02-28 14:49:37,823 INFO  [org.ovirt.engine.core.bll.OVirtInstaller] (pool-3-thread-42) [34b9491a] Installation of 10.66.110.242. Executing oVirt installation stage. (Stage: Sign certificate request and generate certificate)
>2013-02-28 14:49:38,830 INFO  [org.ovirt.engine.core.bll.VdsInstaller] (pool-3-thread-42) [34b9491a]  SignCertificateRequest ended:true
>2013-02-28 14:49:38,830 INFO  [org.ovirt.engine.core.bll.OVirtInstaller] (pool-3-thread-42) [34b9491a] Installation of 10.66.110.242. Executing oVirt installation stage. (Stage: Upload signed sertificate to Host)
>2013-02-28 14:49:38,852 INFO  [org.ovirt.engine.core.bll.VdsInstaller] (pool-3-thread-42) [34b9491a] Installation of 10.66.110.242. successfully done sftp operation ( Stage: Upload signed sertificate to Host)
>2013-02-28 14:49:38,852 INFO  [org.ovirt.engine.core.bll.VdsInstaller] (pool-3-thread-42) [34b9491a]  UploadSignedCertificate ended:true
>2013-02-28 14:49:38,852 INFO  [org.ovirt.engine.core.bll.OVirtInstaller] (pool-3-thread-42) [34b9491a] Installation of 10.66.110.242. Executing oVirt installation stage. (Stage: Upload Cerficate Autority to Host)
>2013-02-28 14:49:38,909 INFO  [org.ovirt.engine.core.bll.VdsInstaller] (pool-3-thread-42) [34b9491a] Installation of 10.66.110.242. successfully done sftp operation ( Stage: Upload Cerficate Autority to Host)
>2013-02-28 14:49:38,909 INFO  [org.ovirt.engine.core.bll.OVirtInstaller] (pool-3-thread-42) [34b9491a] Installation of 10.66.110.242. Executing oVirt installation stage, sending SSH Command /usr/share/vdsm-reg/vdsm-complete -c 'ssl=true' 9b05959f-8873-4134-995d-f4a13e52b3e3 0. (Stage: Running second installation script on Host)
>2013-02-28 14:49:38,909 INFO  [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-3-thread-42) SSH execute 10.66.110.242 '/usr/share/vdsm-reg/vdsm-complete -c 'ssl=true' 9b05959f-8873-4134-995d-f4a13e52b3e3 0'
>2013-02-28 14:49:43,228 INFO  [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-30) Installation of 10.66.110.242. Received message: <BSTRAP component='instCert' status='OK'/>. FYI. (Stage: Running second installation script on Host)
>2013-02-28 14:49:43,239 INFO  [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-30) Installation of 10.66.110.242. Received message: <BSTRAP component='CoreDump' status='OK'/>. FYI. (Stage: Running second installation script on Host)
>2013-02-28 14:49:43,246 INFO  [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-30) Installation of 10.66.110.242. Received message: <BSTRAP component='cleanAll' status='OK'/>. FYI. (Stage: Running second installation script on Host)
>2013-02-28 14:49:43,254 INFO  [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-30) Installation of 10.66.110.242. Received message: <BSTRAP component='VDS Configuration' status='OK'/>. FYI. (Stage: Running second installation script on Host)
>2013-02-28 14:49:43,269 INFO  [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-30) Installation of 10.66.110.242. Received message: <BSTRAP component='Restart' status='OK' message='Restarting vdsmd service' />. FYI. (Stage: Running second installation script on Host)
>2013-02-28 14:49:43,283 INFO  [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-30) Installation of 10.66.110.242. Received message: <BSTRAP component='RHEV_INSTALL' status='OK'/>. Stage completed. (Stage: Running second installation script on Host)
>2013-02-28 14:49:43,296 INFO  [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-42) [34b9491a] After Installation pool-3-thread-42
>2013-02-28 14:49:43,298 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-42) [34b9491a] START, SetVdsStatusVDSCommand(HostName = dell-per510-01.qe.lab.eng.nay.redhat.com, HostId = cb6ce23a-8172-11e2-ab7d-e700038e6d40, status=NonResponsive, nonOperationalReason=NONE), log id: 6d9f4071
>2013-02-28 14:49:43,308 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-42) [34b9491a] FINISH, SetVdsStatusVDSCommand, log id: 6d9f4071
>2013-02-28 14:49:43,310 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-42) [34b9491a] START, SetVdsStatusVDSCommand(HostName = dell-per510-01.qe.lab.eng.nay.redhat.com, HostId = cb6ce23a-8172-11e2-ab7d-e700038e6d40, status=Unassigned, nonOperationalReason=NONE), log id: 34f56a9f
>2013-02-28 14:49:43,320 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-42) [34b9491a] FINISH, SetVdsStatusVDSCommand, log id: 34f56a9f
>2013-02-28 14:49:43,606 WARN  [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-80) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = cb6ce23a-8172-11e2-ab7d-e700038e6d40 : dell-per510-01.qe.lab.eng.nay.redhat.com, VDS Network Error, continuing.
>VDSNetworkException: 
>2013-02-28 14:49:46,864 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-47) [3b85bab0] Running command: InitVdsOnUpCommand internal: true.
>2013-02-28 14:49:46,883 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-47) [3b85bab0] No string for UNASSIGNED type. Use default Log
>2013-02-28 14:49:46,887 INFO  [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-3-thread-42) [3b85bab0] MultipleActionsRunner of type MigrateVmToServer invoked with no actions
>2013-02-28 14:49:46,896 INFO  [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-47) [51930182] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected :  ID: cb6ce23a-8172-11e2-ab7d-e700038e6d40 Type: VDS
>2013-02-28 14:49:46,908 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-47) [420dc7d1] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: cb6ce23a-8172-11e2-ab7d-e700038e6d40 Type: VDS
>2013-02-28 14:50:00,000 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-57) Checking autorecoverable hosts
>2013-02-28 14:50:00,003 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-57) Autorecovering 0 hosts
>2013-02-28 14:50:00,003 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-57) Checking autorecoverable hosts done
>2013-02-28 14:50:00,003 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-57) Autorecovering storage domains is disabled, skipping
>2013-02-28 14:50:59,287 INFO  [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-3-thread-42) [4796be59] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected :  ID: cb6ce23a-8172-11e2-ab7d-e700038e6d40 Type: VDS
>2013-02-28 14:50:59,290 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-42) [4796be59] START, SetVdsStatusVDSCommand(HostName = dell-per510-01.qe.lab.eng.nay.redhat.com, HostId = cb6ce23a-8172-11e2-ab7d-e700038e6d40, status=PreparingForMaintenance, nonOperationalReason=NONE), log id: 7dcf3d61
>2013-02-28 14:50:59,302 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-42) [4796be59] FINISH, SetVdsStatusVDSCommand, log id: 7dcf3d61
>2013-02-28 14:50:59,339 INFO  [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-3-thread-42) [4796be59] Running command: MaintananceVdsCommand internal: true. Entities affected :  ID: cb6ce23a-8172-11e2-ab7d-e700038e6d40 Type: VDS
>2013-02-28 14:51:00,272 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-70) vds::Updated vds status from Preparing for Maintenance to Maintenance in database,  vds = cb6ce23a-8172-11e2-ab7d-e700038e6d40 : dell-per510-01.qe.lab.eng.nay.redhat.com
>2013-02-28 14:51:10,287 INFO  [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp-/127.0.0.1:8702-5) [542fd4ed] Running command: UpdateVdsCommand internal: false. Entities affected :  ID: cb6ce23a-8172-11e2-ab7d-e700038e6d40 Type: VDS
>2013-02-28 14:51:10,313 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp-/127.0.0.1:8702-5) [542fd4ed] START, SetVdsStatusVDSCommand(HostName = dell-per510-01.qe.lab.eng.nay.redhat.com, HostId = cb6ce23a-8172-11e2-ab7d-e700038e6d40, status=Installing, nonOperationalReason=NONE), log id: 3526c26f
>2013-02-28 14:51:10,324 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp-/127.0.0.1:8702-5) [542fd4ed] FINISH, SetVdsStatusVDSCommand, log id: 3526c26f
>2013-02-28 14:51:10,332 INFO  [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-46) [e001947] Running command: InstallVdsCommand internal: true. Entities affected :  ID: cb6ce23a-8172-11e2-ab7d-e700038e6d40 Type: VDS
>2013-02-28 14:51:10,332 INFO  [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-46) [e001947] Before Installation pool-3-thread-46, Powerclient/oVirtNode case: setting status to installing
>2013-02-28 14:51:10,334 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-46) [e001947] START, SetVdsStatusVDSCommand(HostName = dell-per510-01.qe.lab.eng.nay.redhat.com, HostId = cb6ce23a-8172-11e2-ab7d-e700038e6d40, status=Installing, nonOperationalReason=NONE), log id: 2789ac8a
>2013-02-28 14:51:10,364 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-46) [e001947] FINISH, SetVdsStatusVDSCommand, log id: 2789ac8a
>2013-02-28 14:51:10,370 INFO  [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-46) [e001947] Before Installation pool-3-thread-46
>2013-02-28 14:51:10,370 INFO  [org.ovirt.engine.core.bll.OVirtUpgrader] (pool-3-thread-46) [e001947] Installation of 10.66.110.242. Executing oVirt reinstall/upgrade stage. (Stage: Starting Host installation)
>2013-02-28 14:51:10,370 INFO  [org.ovirt.engine.core.bll.OVirtUpgrader] (pool-3-thread-46) [e001947] Installation of 10.66.110.242. Executing oVirt reinstall/upgrade stage. (Stage: Connecting to Host)
>2013-02-28 14:51:10,488 INFO  [org.ovirt.engine.core.bll.VdsInstaller] (pool-3-thread-46) [e001947] Installation of 10.66.110.242. Received message: <BSTRAP component='RHEV_INSTALL' status='OK' message='Connected to Host 10.66.110.242 with SSH key fingerprint: eb:3f:b8:0f:12:fc:5f:57:5f:94:12:c0:f5:d6:5a:23'/>. FYI. (Stage: Connecting to Host)
>2013-02-28 14:51:10,543 INFO  [org.ovirt.engine.core.bll.VdsInstaller] (pool-3-thread-46) [e001947] Installation of 10.66.110.242. Successfully connected to server ssh. (Stage: Connecting to Host)
>2013-02-28 14:51:10,544 INFO  [org.ovirt.engine.core.bll.OVirtUpgrader] (pool-3-thread-46) [e001947] Installation of 10.66.110.242. Executing oVirt reinstall/upgrade stage. (Stage: Upload upgrade ISO to oVirt Node)
>2013-02-28 14:51:32,148 INFO  [org.ovirt.engine.core.bll.VdsInstaller] (pool-3-thread-46) [e001947] Installation of 10.66.110.242. successfully done sftp operation ( Stage: Upload upgrade ISO to oVirt Node)
>2013-02-28 14:51:32,148 INFO  [org.ovirt.engine.core.bll.OVirtUpgrader] (pool-3-thread-46) [e001947] Installation of 10.66.110.242. Executing oVirt reinstall/upgrade stage. (Stage: Running upgrade / reinstall script on oVirt Node)
>2013-02-28 14:51:32,148 INFO  [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-3-thread-46) SSH execute 10.66.110.242 '/usr/share/vdsm-reg/vdsm-upgrade'
>2013-02-28 14:51:34,696 INFO  [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-31) Installation of 10.66.110.242. Received message: <BSTRAP component='RHEL_INSTALL' status='OK'message='vdsm daemon stopped for upgrade process!'/>. FYI. (Stage: Running upgrade / reinstall script on oVirt Node)
>2013-02-28 14:51:34,698 ERROR [org.ovirt.engine.core.bll.InstallerMessages] (NioProcessor-31) Installation of Host. Received illegal XML from Host. Message: org.ovirt.engine.core.compat.CompatException: Element type "BSTRAP" must be followed by either attribute specifications, ">" or "/>"., Exception: {2}
>2013-02-28 14:51:34,702 INFO  [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-31) Installation of 10.66.110.242. Received message: <BSTRAP component='setMountPoint' status='OK' message='Mount succeeded.'/>. FYI. (Stage: Running upgrade / reinstall script on oVirt Node)
>2013-02-28 14:51:34,711 ERROR [org.ovirt.engine.core.bll.VdsInstaller] (NioProcessor-31) Installation of 10.66.110.242. Received message: <BSTRAP component='RHEV_INSTALL' status='FAIL'/>. Error occured. (Stage: Running upgrade / reinstall script on oVirt Node)
>2013-02-28 14:51:34,748 ERROR [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (pool-3-thread-46) SSH error running command 10.66.110.242:'/usr/share/vdsm-reg/vdsm-upgrade': java.io.IOException: Command returned failure code 1 during SSH session 10.66.110.242:22' '/usr/share/vdsm-reg/vdsm-upgrade'
>	at org.ovirt.engine.core.utils.ssh.SSHClient.executeCommand(SSHClient.java:442) [engine-utils.jar:]
>	at org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH.executeCommand(VdsInstallerSSH.java:387) [engine-utils.jar:]
>	at org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH.executeCommand(VdsInstallerSSH.java:426) [engine-utils.jar:]
>	at org.ovirt.engine.core.bll.OVirtUpgrader.RunStage(OVirtUpgrader.java:54) [engine-bll.jar:]
>	at org.ovirt.engine.core.bll.VdsInstaller.Install(VdsInstaller.java:280) [engine-bll.jar:]
>	at org.ovirt.engine.core.bll.InstallVdsCommand.executeCommand(InstallVdsCommand.java:110) [engine-bll.jar:]
>	at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:876) [engine-bll.jar:]
>	at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:973) [engine-bll.jar:]
>	at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1372) [engine-bll.jar:]
>	at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:168) [engine-utils.jar:]
>	at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:107) [engine-utils.jar:]
>	at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:991) [engine-bll.jar:]
>	at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:291) [engine-bll.jar:]
>	at org.ovirt.engine.core.bll.MultipleActionsRunner.executeValidatedCommands(MultipleActionsRunner.java:182) [engine-bll.jar:]
>	at org.ovirt.engine.core.bll.MultipleActionsRunner.RunCommands(MultipleActionsRunner.java:162) [engine-bll.jar:]
>	at org.ovirt.engine.core.bll.MultipleActionsRunner$1.run(MultipleActionsRunner.java:84) [engine-bll.jar:]
>	at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:64) [engine-utils.jar:]
>	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_09-icedtea]
>	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_09-icedtea]
>	at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_09-icedtea]
>	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_09-icedtea]
>	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_09-icedtea]
>	at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea]
>
>2013-02-28 14:51:34,750 ERROR [org.ovirt.engine.core.bll.VdsInstaller] (pool-3-thread-46) [e001947] Installation of 10.66.110.242 has failed. Failure details: SSH command failed while executing at host '10.66.110.242', refer to logs for further information. (Stage: Running upgrade / reinstall script on oVirt Node)
>2013-02-28 14:51:34,750 INFO  [org.ovirt.engine.core.bll.InstallerMessages] (pool-3-thread-46) [e001947] VDS message: SSH command failed while executing at host '10.66.110.242', refer to logs for further information
>2013-02-28 14:51:34,750 INFO  [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-46) [e001947] After Installation pool-3-thread-46
>2013-02-28 14:51:34,753 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-46) [e001947] START, SetVdsStatusVDSCommand(HostName = dell-per510-01.qe.lab.eng.nay.redhat.com, HostId = cb6ce23a-8172-11e2-ab7d-e700038e6d40, status=InstallFailed, nonOperationalReason=NONE), log id: 1c7cfd08
>2013-02-28 14:51:34,763 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-46) [e001947] FINISH, SetVdsStatusVDSCommand, log id: 1c7cfd08
>2013-02-28 14:55:00,001 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [b6dff4e] Checking autorecoverable hosts
>2013-02-28 14:55:00,008 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [b6dff4e] Autorecovering 0 hosts
>2013-02-28 14:55:00,008 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [b6dff4e] Checking autorecoverable hosts done
>2013-02-28 14:55:00,008 INFO  [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [b6dff4e] Autorecovering storage domains is disabled, skipping

Actions: View

Attachments on bug 916485: 703829 | 703830 | 704089