Attachment 850996 Details for Bug 1054140 – agent log

agent log

agent.log (text/x-log), 8.47 KB, created by movciari on 2014-01-16 10:26:03 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: movciari

Created: 2014-01-16 10:26:03 UTC

Size: 8.47 KB

patch

obsolete

>MainThread::INFO::2014-01-13 15:42:04,112::agent::50::ovirt_hosted_engine_ha.agent.agent.Agent::(run) ovirt-hosted-engine-ha agent 1.0.0-0.1.beta3 started
>MainThread::INFO::2014-01-13 15:42:04,195::hosted_engine::201::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: 10.34.63.136
>MainThread::INFO::2014-01-13 15:42:04,195::hosted_engine::363::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection
>MainThread::INFO::2014-01-13 15:42:04,196::brokerlink::67::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(connect) Failed to connect to broker: [Errno 2] No such file or directory
>MainThread::INFO::2014-01-13 15:42:04,196::brokerlink::68::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(connect) Retrying broker connection...
>MainThread::INFO::2014-01-13 15:42:05,197::brokerlink::126::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor ping, options {'addr': '10.34.63.254'}
>MainThread::INFO::2014-01-13 15:42:05,199::brokerlink::137::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 40839312
>MainThread::INFO::2014-01-13 15:42:05,201::brokerlink::126::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mgmt-bridge, options {'use_ssl': 'true', 'bridge_name': 'rhevm', 'address': '0'}
>MainThread::INFO::2014-01-13 15:42:05,212::brokerlink::137::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 40840080
>MainThread::INFO::2014-01-13 15:42:05,213::brokerlink::126::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mem-free, options {'use_ssl': 'true', 'address': '0'}
>MainThread::INFO::2014-01-13 15:42:05,216::brokerlink::137::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 40840016
>MainThread::INFO::2014-01-13 15:42:05,216::brokerlink::126::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor cpu-load-no-engine, options {'use_ssl': 'true', 'vm_uuid': '686fd9a2-72ba-4e01-ad15-5056d529f94a', 'address': '0'}
>MainThread::INFO::2014-01-13 15:42:05,220::brokerlink::137::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 40840720
>MainThread::INFO::2014-01-13 15:42:05,220::brokerlink::126::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor engine-health, options {'use_ssl': 'true', 'vm_uuid': '686fd9a2-72ba-4e01-ad15-5056d529f94a', 'address': '0'}
>MainThread::INFO::2014-01-13 15:42:05,224::brokerlink::137::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140572467894736
>MainThread::INFO::2014-01-13 15:42:05,225::hosted_engine::386::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Broker initialized, all submonitors started
>MainThread::INFO::2014-01-13 15:42:05,769::hosted_engine::454::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Ensuring lease for lockspace hosted-engine, host id 1 is acquired (file: /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_movciarik_nfs04/baf372ef-ef59-4149-808c-c2f35fdeff11/ha_agent/hosted-engine.lockspace)
>MainThread::ERROR::2014-01-13 15:42:05,773::hosted_engine::482::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) cannot get lock on host id 1: host already holds lock on a different host id
>Traceback (most recent call last):
>  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 458, in _initialize_sanlock
>    host_id, lease_file)
>SanlockException: (22, 'Sanlock lockspace add failure', 'Invalid argument')
>MainThread::WARNING::2014-01-13 15:42:05,790::hosted_engine::333::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Error while monitoring engine: Failed to initialize sanlock: cannot get lock on host id 1: host already holds lock on a different host id
>MainThread::WARNING::2014-01-13 15:42:05,791::hosted_engine::336::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Unexpected error
>Traceback (most recent call last):
>  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 322, in start_monitoring
>    self._initialize_sanlock()
>  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 484, in _initialize_sanlock
>    .format(msg))
>Exception: Failed to initialize sanlock: cannot get lock on host id 1: host already holds lock on a different host id
>MainThread::INFO::2014-01-13 15:42:05,791::hosted_engine::353::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Sleeping 60 seconds
>MainThread::INFO::2014-01-13 15:43:06,247::hosted_engine::454::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Ensuring lease for lockspace hosted-engine, host id 1 is acquired (file: /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_movciarik_nfs04/baf372ef-ef59-4149-808c-c2f35fdeff11/ha_agent/hosted-engine.lockspace)
>MainThread::ERROR::2014-01-13 15:43:06,248::hosted_engine::482::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) cannot get lock on host id 1: host already holds lock on a different host id
>Traceback (most recent call last):
>  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 458, in _initialize_sanlock
>    host_id, lease_file)
>SanlockException: (22, 'Sanlock lockspace add failure', 'Invalid argument')
>MainThread::WARNING::2014-01-13 15:43:06,248::hosted_engine::333::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Error while monitoring engine: Failed to initialize sanlock: cannot get lock on host id 1: host already holds lock on a different host id
>MainThread::WARNING::2014-01-13 15:43:06,248::hosted_engine::336::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Unexpected error
>Traceback (most recent call last):
>  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 322, in start_monitoring
>    self._initialize_sanlock()
>  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 484, in _initialize_sanlock
>    .format(msg))
>Exception: Failed to initialize sanlock: cannot get lock on host id 1: host already holds lock on a different host id
>MainThread::INFO::2014-01-13 15:43:06,249::hosted_engine::353::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Sleeping 60 seconds
>MainThread::INFO::2014-01-13 15:44:06,673::hosted_engine::454::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Ensuring lease for lockspace hosted-engine, host id 1 is acquired (file: /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_movciarik_nfs04/baf372ef-ef59-4149-808c-c2f35fdeff11/ha_agent/hosted-engine.lockspace)
>MainThread::ERROR::2014-01-13 15:44:06,674::hosted_engine::482::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) cannot get lock on host id 1: host already holds lock on a different host id
>Traceback (most recent call last):
>  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 458, in _initialize_sanlock
>    host_id, lease_file)
>SanlockException: (22, 'Sanlock lockspace add failure', 'Invalid argument')
>MainThread::WARNING::2014-01-13 15:44:06,675::hosted_engine::333::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Error while monitoring engine: Failed to initialize sanlock: cannot get lock on host id 1: host already holds lock on a different host id
>MainThread::WARNING::2014-01-13 15:44:06,675::hosted_engine::336::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Unexpected error
>Traceback (most recent call last):
>  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 322, in start_monitoring
>    self._initialize_sanlock()
>  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 484, in _initialize_sanlock
>    .format(msg))
>Exception: Failed to initialize sanlock: cannot get lock on host id 1: host already holds lock on a different host id
>MainThread::ERROR::2014-01-13 15:44:06,675::hosted_engine::350::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Shutting down the agent because of 3 failures in a row!
>MainThread::INFO::2014-01-13 15:44:06,676::agent::107::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Agent shutting down

Actions: View

Attachments on bug 1054140: 850985 | 850996 | 850997