Description of problem: External Provider connections in 3.5.0-0.20.el6ev fail at testing keystone authentication due to unknown/unfiltered keystone token attribute 'audit_ids'. Version-Release number of selected component (if applicable): 3.5.0-0.20.el6ev How reproducible: always Steps to Reproduce: 1. Connect to Keystone on a Juno-based RHEL OSP installation 2. Attempt to validate connection, it fails each time. 3. Actual results: Get log entries like this: 2014-12-30 21:27:45,534 ERROR [org.ovirt.engine.core.bll.provider.TestProviderConnectivityCommand] (ajp-/127.0.0.1:8702-12) [3e1a39a5] Command org.ovirt.engine.core.bll.provider.TestProvider ConnectivityCommand throw Vdc Bll exception. With error message VdcBLLException: (Failed with error PROVIDER_FAILURE and code 5050) 2014-12-30 21:27:45,541 DEBUG [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-57) Before releasing a lock EngineLock [exclusiveLocks= key: 4a23ff16-a07e-49a3-ae80-f84892b20059 value: VDS_INIT , sharedLocks= ] 2014-12-30 21:27:45,541 DEBUG [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (DefaultQuartzScheduler_Worker-57) The exclusive lock for key 4a23ff16-a07e-49a3-ae80-f84892b20059VDS_INIT is released and lock is removed from map 2014-12-30 21:27:45,535 DEBUG [org.ovirt.engine.core.bll.provider.TestProviderConnectivityCommand] (ajp-/127.0.0.1:8702-12) [3e1a39a5] Command org.ovirt.engine.core.bll.provider.TestProviderConnectivityCommand throw Vdc Bll exception: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: (Failed with error PROVIDER_FAILURE and code 5050) at org.ovirt.engine.core.bll.provider.OpenStackImageProviderProxy.testConnection(OpenStackImageProviderProxy.java:94) [bll.jar:] at org.ovirt.engine.core.bll.provider.TestProviderConnectivityCommand.executeCommand(TestProviderConnectivityCommand.java:36) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1168) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1307) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1932) [bll.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:] at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1331) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:344) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:430) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:411) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:369) [bll.jar:] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_71] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_71] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_71] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_71] at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) [jboss-as-ee.jar:7.4.2.Final-redhat-2] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) [jboss-as-ee.jar:7.4.2.Final-redhat-2] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:74) [jboss-as-weld.jar:7.4.2.Final-redhat-2] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:84) [jboss-as-weld.jar:7.4.2.Final-redhat-2] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:97) [jboss-as-weld.jar:7.4.2.Final-redhat-2] at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) [jboss-as-ee.jar:7.4.2.Final-redhat-2] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.ovirt.engine.core.bll.interceptors.ThreadLocalSessionCleanerInterceptor.injectWebContextToThreadLocal(ThreadLocalSessionCleanerInterceptor.java:13) [bll.jar:] at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source) [:1.7.0_71] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_71] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_71] at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:89) [jboss-as-ee.jar:7.4.2.Final-redhat-2] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) [jboss-as-ee.jar:7.4.2.Final-redhat-2] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3.jar:7.4.2.Final-redhat-2] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:73) [jboss-as-weld.jar:7.4.2.Final-redhat-2] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation.jar:1.1.2.Final-redhat-1 ... snip ... at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:490) at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926) at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_71] Caused by: org.jboss.resteasy.spi.ReaderException: org.codehaus.jackson.map.exc.UnrecognizedPropertyException: Unrecognized field "audit_ids" (Class com.woorea.openstack.keystone.model.Token), not marked as ignorable at [Source: org.jboss.resteasy.core.interception.MessageBodyReaderContextImpl$InputStreamWrapper@5ac343ac; line: 1, column: 276] (through reference chain: com.woorea.openstack.keystone.model.Access["token"]->com.woorea.openstack.keystone.model.Token["audit_ids"]) at org.jboss.resteasy.client.core.BaseClientResponse.readFrom(BaseClientResponse.java:469) [resteasy-jaxrs.jar:] at org.jboss.resteasy.client.core.BaseClientResponse.getEntity(BaseClientResponse.java:377) [resteasy-jaxrs.jar:] at org.jboss.resteasy.client.core.BaseClientResponse.getEntity(BaseClientResponse.java:350) [resteasy-jaxrs.jar:] at org.jboss.resteasy.client.core.BaseClientResponse.getEntity(BaseClientResponse.java:344) [resteasy-jaxrs.jar:] at com.woorea.openstack.connector.RESTEasyResponse.getEntity(RESTEasyResponse.java:25) [resteasy-connector.jar:] at com.woorea.openstack.base.client.OpenStackClient.execute(OpenStackClient.java:67) [openstack-client.jar:] at com.woorea.openstack.base.client.OpenStackRequest.execute(OpenStackRequest.java:98) [openstack-client.jar:] at com.woorea.openstack.keystone.utils.KeystoneTokenProvider.getAccessByTenant(KeystoneTokenProvider.java:30) [keystone-client.jar:] at com.woorea.openstack.keystone.utils.KeystoneTokenProvider$1.getToken(KeystoneTokenProvider.java:47) [keystone-client.jar:] at com.woorea.openstack.base.client.OpenStackClient.request(OpenStackClient.java:47) [openstack-client.jar:] at com.woorea.openstack.base.client.OpenStackClient.execute(OpenStackClient.java:66) [openstack-client.jar:] at org.ovirt.engine.core.bll.provider.OpenStackImageProviderProxy.testConnection(OpenStackImageProviderProxy.java:92) [bll.jar:] ... 123 more Caused by: org.codehaus.jackson.map.exc.UnrecognizedPropertyException: Unrecognized field "audit_ids" (Class com.woorea.openstack.keystone.model.Token), not marked as ignorable at [Source: org.jboss.resteasy.core.interception.MessageBodyReaderContextImpl$InputStreamWrapper@5ac343ac; line: 1, column: 276] (through reference chain: com.woorea.openstack.keystone.model.Access["token"]->com.woorea.openstack.keystone.model.Token["audit_ids"]) at org.codehaus.jackson.map.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:53) [jackson-mapper-asl.jar:1.9.9.redhat-3] at org.codehaus.jackson.map.deser.StdDeserializationContext.unknownFieldException(StdDeserializationContext.java:267) [jackson-mapper-asl.jar:1.9.9.redhat-3] at org.codehaus.jackson.map.deser.std.StdDeserializer.reportUnknownProperty(StdDeserializer.java:673) [jackson-mapper-asl.jar:1.9.9.redhat-3] at org.codehaus.jackson.map.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:659) [jackson-mapper-asl.jar:1.9.9.redhat-3] at org.codehaus.jackson.map.deser.BeanDeserializer.handleUnknownProperty(BeanDeserializer.java:1365) [jackson-mapper-asl.jar:1.9.9.redhat-3] at org.codehaus.jackson.map.deser.BeanDeserializer._handleUnknown(BeanDeserializer.java:725) [jackson-mapper-asl.jar:1.9.9.redhat-3] at org.codehaus.jackson.map.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:703) [jackson-mapper-asl.jar:1.9.9.redhat-3] at org.codehaus.jackson.map.deser.BeanDeserializer.deserialize(BeanDeserializer.java:580) [jackson-mapper-asl.jar:1.9.9.redhat-3] at org.codehaus.jackson.map.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:299) [jackson-mapper-asl.jar:1.9.9.redhat-3] at org.codehaus.jackson.map.deser.SettableBeanProperty$FieldProperty.deserializeAndSet(SettableBeanProperty.java:579) [jackson-mapper-asl.jar:1.9.9.redhat-3] at org.codehaus.jackson.map.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:697) [jackson-mapper-asl.jar:1.9.9.redhat-3] at org.codehaus.jackson.map.deser.BeanDeserializer.deserialize(BeanDeserializer.java:580) [jackson-mapper-asl.jar:1.9.9.redhat-3] at org.codehaus.jackson.map.ObjectMapper._unwrapAndDeserialize(ObjectMapper.java:2802) [jackson-mapper-asl.jar:1.9.9.redhat-3] at org.codehaus.jackson.map.ObjectMapper._readValue(ObjectMapper.java:2702) [jackson-mapper-asl.jar:1.9.9.redhat-3] at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1315) [jackson-mapper-asl.jar:1.9.9.redhat-3] at org.codehaus.jackson.jaxrs.JacksonJsonProvider.readFrom(JacksonJsonProvider.java:419) at org.jboss.resteasy.core.interception.MessageBodyReaderContextImpl.proceed(MessageBodyReaderContextImpl.java:106) [resteasy-jaxrs.jar:] at org.jboss.resteasy.client.core.BaseClientResponse.readFrom(BaseClientResponse.java:433) [resteasy-jaxrs.jar:] ... 134 more Expected results: successful external provider connection to a Juno-based OpenStack installation Additional info: I don't have much time left to keep my test environment around in a non-functioning state. But I can currently give someone short-term access to my env for testing, it's in Boston in a lab.
Seems like it is a known issue, addressed by the following patch of the openstack-java-sdk: https://github.com/woorea/openstack-java-sdk/pull/166 Until it is merged, and there is a release with that, we can't address the issue in the engine side. All I can recommend is to work with icehouse and not juno until this issue is resolved in the openstack-java-sdk.
The upstream pull request has been merged and openstack-java-sdk-3.0.6 should reach the mirrors within 2 business day.
(In reply to Federico Simoncelli from comment #13) > (In reply to Oved Ourfali from comment #12) > > fsimonce - what should be the version to depend on now? > > I've mentioned it in comment #5 openstack-java-sdk-3.0.6 I've verified locally by [1] that openstack-java-sdk-3.0.6 fixed authentication issue against keystone. Federico, could you produce rpms in koji updates-testing ? [1] http://gerrit.ovirt.org/#/c/36837/
please dup to 3.5 so I can build rhevm-dependencies with this.
(In reply to Moti Asayag from comment #16) > (In reply to Federico Simoncelli from comment #13) > > (In reply to Oved Ourfali from comment #12) > > > fsimonce - what should be the version to depend on now? > > > > I've mentioned it in comment #5 openstack-java-sdk-3.0.6 > > I've verified locally by [1] that openstack-java-sdk-3.0.6 fixed > authentication issue against keystone. > > Federico, could you produce rpms in koji updates-testing ? The official builds are available here: https://admin.fedoraproject.org/updates/openstack-java-sdk please add karma to promote to updates.
Works ok with ovirt-engine-backend-3.6.0-0.0.master.20150412172301.git55ba764.el7.centos.noarch
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2016-0398.html