hello we now install ovirt with 3 node hosted-self engine on our machine. After rebooting the machine,we run hosted-engine --vm-start. But we get the result: [root@ovirt-node1 ~]# hosted-engine --vm-status --== Host 1 status ==-- conf_on_shared_storage : True Status up-to-date : True Hostname : ovirt-node1 Host ID : 1 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 3400 stopped : False Local maintenance : False crc32 : 59048bf5 local_conf_timestamp : 672 Host timestamp : 657 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=657 (Tue Jul 25 20:40:34 2017) host-id=1 score=3400 vm_conf_refresh_time=672 (Tue Jul 25 20:40:48 2017) conf_on_shared_storage=True maintenance=False state=EngineDown stopped=False --== Host 2 status ==-- conf_on_shared_storage : True Status up-to-date : True Hostname : 192.168.71.129 Host ID : 2 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 3400 stopped : False Local maintenance : False crc32 : 7cc5adfa local_conf_timestamp : 666 Host timestamp : 651 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=651 (Tue Jul 25 20:40:33 2017) host-id=2 score=3400 vm_conf_refresh_time=666 (Tue Jul 25 20:40:47 2017) conf_on_shared_storage=True maintenance=False state=EngineDown stopped=False --== Host 3 status ==-- conf_on_shared_storage : True Status up-to-date : True Hostname : 192.168.71.130 Host ID : 3 Engine status : {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "up"} Score : 3400 stopped : False Local maintenance : False crc32 : 4c8a8e16 local_conf_timestamp : 686 Host timestamp : 672 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=672 (Tue Jul 25 20:40:31 2017) host-id=3 score=3400 vm_conf_refresh_time=686 (Tue Jul 25 20:40:46 2017) conf_on_shared_storage=True maintenance=False state=EngineStarting stopped=False [root@ovirt-node1 ~]# we can't go to the engione login web,the web page show 404 - Not Found. how can we fix it?
And the engine log in the engine VM shows : 2017-07-25 22:36:28,030-04 ERROR [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 54) [] Failed to initialize backend: org.jboss.weld.exceptions.WeldException: WELD-000049: Unable to invoke protected void org.ovirt.engine.core.bll.scheduling.AffinityRulesEnforcementManager.wakeup() on org.ovirt.engine.core.bll.scheduling.AffinityRulesEnforcementManager@5ea69caf at org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.invokeMethods(DefaultLifecycleCallbackInvoker.java:100) [weld-core-impl-2.3.5.Final.jar:2.3.5.Final] at org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.postConstruct(DefaultLifecycleCallbackInvoker.java:81) [weld-core-impl-2.3.5.Final.jar:2.3.5.Final] at org.jboss.weld.injection.producer.BasicInjectionTarget.postConstruct(BasicInjectionTarget.java:126) [weld-core-impl-2.3.5.Final.jar:2.3.5.Final] at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:162) [weld-core-impl-2.3.5.Final.jar:2.3.5.Final] at org.jboss.weld.context.AbstractContext.get(AbstractContext.java:96) [weld-core-impl-2.3.5.Final.jar:2.3.5.Final] at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:101) [weld-core-impl-2.3.5.Final.jar:2.3.5.Final] at org.jboss.weld.bean.ContextualInstanceStrategy$ApplicationScopedContextualInstanceStrategy.get(ContextualInstanceStrategy.java:141) [weld-core-impl-2.3.5.Final.jar:2.3.5.Final] at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) [weld-core-impl-2.3.5.Final.jar:2.3.5.Final] at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:742) [weld-core-impl-2.3.5.Final.jar:2.3.5.Final] at org.jboss.weld.bean.builtin.InstanceImpl.getBeanInstance(InstanceImpl.java:189) [weld-core-impl-2.3.5.Final.jar:2.3.5.Final] at org.jboss.weld.bean.builtin.InstanceImpl.get(InstanceImpl.java:100) [weld-core-impl-2.3.5.Final.jar:2.3.5.Final] at org.ovirt.engine.core.bll.ServiceLoader.load(ServiceLoader.java:34) [bll.jar:] at org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean.create(InitBackendServicesOnStartupBean.java:96) [bll.jar:] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_121] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_121] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_121] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_121] at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:96) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doLifecycleInterception(Jsr299BindingsInterceptor.java:114) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:103) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437) at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73) [weld-core-impl-2.3.5.Final.jar:2.3.5.Final] at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.weld.injection.WeldInjectionInterceptor.processInvocation(WeldInjectionInterceptor.java:53) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ee.component.AroundConstructInterceptorFactory$1.processInvocation(AroundConstructInterceptorFactory.java:28) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.weld.injection.WeldInterceptorInjectionInterceptor.processInvocation(WeldInterceptorInjectionInterceptor.java:56) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.weld.ejb.Jsr299BindingsCreateInterceptor.processInvocation(Jsr299BindingsCreateInterceptor.java:100) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:275) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.ejb3.tx.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:349) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.ejb3.tx.LifecycleCMTTxInterceptor.processInvocation(LifecycleCMTTxInterceptor.java:68) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.weld.injection.WeldInjectionContextInterceptor.processInvocation(WeldInjectionContextInterceptor.java:43) [wildfly-weld-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.singleton.StartupCountDownInterceptor.processInvocation(StartupCountDownInterceptor.java:25) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356) at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:161) at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:134) at org.jboss.as.ee.component.BasicComponent.createInstance(BasicComponent.java:88) at org.jboss.as.ejb3.component.singleton.SingletonComponent.getComponentInstance(SingletonComponent.java:124) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.ejb3.component.singleton.SingletonComponent.start(SingletonComponent.java:138) [wildfly-ejb3-10.1.0.Final.jar:10.1.0.Final] at org.jboss.as.ee.component.ComponentStartService$1.run(ComponentStartService.java:54) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_121] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_121] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_121] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_121] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_121] at org.jboss.threads.JBossThread.run(JBossThread.java:320) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_121] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_121] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_121] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_121] at org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.invokeMethods(DefaultLifecycleCallbackInvoker.java:98) [weld-core-impl-2.3.5.Final.jar:2.3.5.Final] ... 65 more Caused by: org.springframework.jdbc.UncategorizedSQLException: CallableStatementCallback; uncategorized SQLException for SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}]; SQL state [XX001]; error code [0]; ERROR: could not read block 20 in file "base/16385/19110": read only 0 of 8192 bytes Where: SQL statement "INSERT INTO audit_log ( LOG_TIME, log_type, log_type_name, severity, message, user_id, user_name, vds_id, vds_name, vm_id, vm_name, vm_template_id, vm_template_name, storage_pool_id, storage_pool_name, storage_domain_id, storage_domain_name, cluster_id, cluster_name, correlation_id, job_id, quota_id, quota_name, gluster_volume_id, gluster_volume_name, call_stack, brick_id, brick_path ) VALUES ( v_log_time, v_log_type, v_log_type_name, v_severity, v_message, v_user_id, v_user_name, v_vds_id, v_vds_name, v_vm_id, v_vm_name, v_vm_template_id, v_vm_template_name, v_storage_pool_id, v_storage_pool_name, v_storage_domain_id, v_storage_domain_name, v_cluster_id, v_cluster_name, v_correlation_id, v_job_id, v_quota_id, v_quota_name, v_gluster_volume_id, v_gluster_volume_name, v_call_stack, v_brick_id, v_brick_path )" PL/pgSQL function insertauditlog(integer,timestamp with time zone,integer,character varying,integer,text,uuid,character varying,uuid,character varying,uuid,character varying,uuid,character varying,uuid,character varying,uuid,character varying,uuid,character varying,uuid,character varying,character varying,uuid,uuid,character varying,text,boolean,uuid,text) line 9 at SQL statement; nested exception is org.postgresql.util.PSQLException: ERROR: could not read block 20 in file "base/16385/19110": read only 0 of 8192 bytes Where: SQL statement "INSERT INTO audit_log ( LOG_TIME, log_type, log_type_name, severity, message, user_id, user_name, vds_id, vds_name, vm_id, vm_name, vm_template_id, vm_template_name, storage_pool_id, storage_pool_name, storage_domain_id, storage_domain_name, cluster_id, cluster_name, correlation_id, job_id, quota_id, quota_name, gluster_volume_id, gluster_volume_name, call_stack, brick_id, brick_path ) VALUES ( v_log_time, v_log_type, v_log_type_name, v_severity, v_message, v_user_id, v_user_name, v_vds_id, v_vds_name, v_vm_id, v_vm_name, v_vm_template_id, v_vm_template_name, v_storage_pool_id, v_storage_pool_name, v_storage_domain_id, v_storage_domain_name, v_cluster_id, v_cluster_name, v_correlation_id, v_job_id, v_quota_id, v_quota_name, v_gluster_volume_id, v_gluster_volume_name, v_call_stack, v_brick_id, v_brick_path )" PL/pgSQL function insertauditlog(integer,timestamp with time zone,integer,character varying,integer,text,uuid,character varying,uuid,character varying,uuid,character varying,uuid,character varying,uuid,character varying,uuid,character varying,uuid,character varying,uuid,character varying,character varying,uuid,uuid,character varying,text,boolean,uuid,text) line 9 at SQL statement at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:84) [spring-jdbc.jar:4.2.4.RELEASE] at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81) [spring-jdbc.jar:4.2.4.RELEASE] at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81) [spring-jdbc.jar:4.2.4.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:1094) [spring-jdbc.jar:4.2.4.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.call(JdbcTemplate.java:1130) [spring-jdbc.jar:4.2.4.RELEASE] at org.springframework.jdbc.core.simple.AbstractJdbcCall.executeCallInternal(AbstractJdbcCall.java:405) [spring-jdbc.jar:4.2.4.RELEASE] at org.springframework.jdbc.core.simple.AbstractJdbcCall.doExecute(AbstractJdbcCall.java:365) [spring-jdbc.jar:4.2.4.RELEASE] at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:198) [spring-jdbc.jar:4.2.4.RELEASE] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:135) [dal.jar:] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:130) [dal.jar:] at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeModification(SimpleJdbcCallsHandler.java:76) [dal.jar:] at org.ovirt.engine.core.dao.AuditLogDaoImpl.save(AuditLogDaoImpl.java:169) [dal.jar:] at org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector.saveToDb(AuditLogDirector.java:95) [dal.jar:] at org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector.log(AuditLogDirector.java:81) [dal.jar:] at org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector.log(AuditLogDirector.java:70) [dal.jar:] at org.ovirt.engine.core.bll.scheduling.AffinityRulesEnforcementManager.wakeup(AffinityRulesEnforcementManager.java:51) [bll.jar:] ... 70 more Caused by: org.postgresql.util.PSQLException: ERROR: could not read block 20 in file "base/16385/19110": read only 0 of 8192 bytes Where: SQL statement "INSERT INTO audit_log ( LOG_TIME, log_type, log_type_name, severity, message, user_id, user_name, vds_id, vds_name, vm_id, vm_name, vm_template_id, vm_template_name, storage_pool_id, storage_pool_name, storage_domain_id, storage_domain_name, cluster_id, cluster_name, correlation_id, job_id, quota_id, quota_name, gluster_volume_id, gluster_volume_name, call_stack, brick_id, brick_path ) VALUES ( v_log_time, v_log_type, v_log_type_name, v_severity, v_message, v_user_id, v_user_name, v_vds_id, v_vds_name, v_vm_id, v_vm_name, v_vm_template_id, v_vm_template_name, v_storage_pool_id, v_storage_pool_name, v_storage_domain_id, v_storage_domain_name, v_cluster_id, v_cluster_name, v_correlation_id, v_job_id, v_quota_id, v_quota_name, v_gluster_volume_id, v_gluster_volume_name, v_call_stack, v_brick_id, v_brick_path )" PL/pgSQL function insertauditlog(integer,timestamp with time zone,integer,character varying,integer,text,uuid,character varying,uuid,character varying,uuid,character varying,uuid,character varying,uuid,character varying,uuid,character varying,uuid,character varying,uuid,character varying,character varying,uuid,uuid,character varying,text,boolean,uuid,text) line 9 at SQL statement at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2157) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1886) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255) at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:555) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:417) at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:410) at org.jboss.jca.adapters.jdbc.CachedPreparedStatement.execute(CachedPreparedStatement.java:303) at org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.execute(WrappedPreparedStatement.java:442) at org.springframework.jdbc.core.JdbcTemplate$6.doInCallableStatement(JdbcTemplate.java:1133) [spring-jdbc.jar:4.2.4.RELEASE] at org.springframework.jdbc.core.JdbcTemplate$6.doInCallableStatement(JdbcTemplate.java:1130) [spring-jdbc.jar:4.2.4.RELEASE] at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:1078) [spring-jdbc.jar:4.2.4.RELEASE] ... 82 more It seems like sql issue?
org.postgresql.util.PSQLException: ERROR: could not read block 20 in file "base/16385/19110": read only 0 of 8192 bytes Suggests an issue with database or filesystem. Could you provide the gluster mount logs for engine volume as well as gluster brick logs?
Are you able to provide requested info?
Closing this as there's no data provided to look into this further
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days