Bug 2063562

Summary:	mod_auth_openidc fails with IPv6 OIDCMemCacheServers
Product:	Red Hat Enterprise Linux 8	Reporter:	François Rigault <frigo>
Component:	apr-util	Assignee:	Luboš Uhliarik <luhliari>
Status:	VERIFIED ---	QA Contact:	icesalov
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	8.4	CC:	icesalov, jorton, spoore, thalman
Target Milestone:	rc	Keywords:	Triaged
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	apr-util-1.6.1-8.el8	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	2073460 2168760 (view as bug list)		Environment:
Last Closed:		Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	2073460, 2168760

Description François Rigault 2022-03-13 18:21:05 UTC

Description of problem:
deploying RHOSP 16.2 with "enable-federation-openidc.yaml"
authentication into Horizon fails and keystone logs:

[Sun Mar 13 17:42:18.981474 2022] [auth_openidc:error] [pid 1695] [client fd00:0:0:cc::233:54928] oidc_cache_memcache_log_status_error: apr_memcache_getp returned an error: [Address family for hostname not supported]; check your that your memcache server is available/accessible., referer: https://10.xxx/

Replacing
  OIDCMemCacheServers "[fd00:0:0:cc::233]:11211"
with
  OIDCMemCacheServers "127.0.0.1:11211"
in /var/lib/config-data/puppet-generated/keystone/etc/httpd/conf.d/10-keystone_wsgi.conf worksaround the issue (memcached is listening on both).


Version-Release number of selected component (if applicable):
https://access.redhat.com/containers/#/registry.access.redhat.com/rhosp16/openstack-keystone/images/16.2.1-6,
mod_auth_openidc-2.3.7-8.module+el8.4.0+9707+f2438af7.x86_64


How reproducible:
Using OpenStack + IPv6 on internal_api network + enabling Openid 

Steps to Reproduce:
1. 
2.
3.

Actual results:


Expected results:


Additional info:
maybe this is missing:
https://github.com/zmartzone/mod_auth_openidc/commit/0c7f5c2c2dbb6157e729ebc214a4d577e633903f

Comment 1 François Rigault 2022-03-17 14:56:45 UTC

workaround above works for a stack with a single controller.
Unfortunately when there are 3 controllers we read

  OIDCMemCacheServers "[fd00:0:0:cc::c]:11211 [fd00:0:0:cc::d]:11211 [fd00:0:0:cc::3]:11211"

and I guess we are supposed to update all three memcached. To make the workaround works, we need another workaround:
parameter_defaults:
  ExtraConfig:
    tripleo::haproxy::haproxy_defaults_override:
      balance: source

This gives the client more luck to hit back the same keystone server where the initial key was stored.

Comment 2 Tomas Halman 2022-07-01 11:19:13 UTC

Thanks to recent rebase this should be already fixed. We just have to validate the bug and then we can close it.

Comment 3 François Rigault 2022-07-04 19:39:29 UTC

thanks! any way you could build a version against RHEL 8.4? keystone containers ships version 2.3.7-8 which is the latest on 8.4.

Comment 4 Tomas Halman 2022-10-05 15:25:54 UTC

I checked this issue with latest build (mod_auth_openidc-2.4.9.4-1). Suggested patch is included but unfortunately it does not solve the problem.
I will take a closer look this quarter.

Tom

Comment 5 Tomas Halman 2022-12-21 12:21:29 UTC

After further investigation I found out that this bug is actually in apr-util library.

There are two places in memcache/apr_memcache.c where is protocol family set to AF_INET (APR_INET) and that enforces IPv4 protocol later on.
This bug is present in both versions 1.6 and 1.7 (rhel9) as well.

Following patch fixes the issue:

--------------- cut here ---------------
diff -up apr-util-1.6.1/memcache/apr_memcache.c.fix apr-util-1.6.1/memcache/apr_memcache.c
--- apr-util-1.6.1/memcache/apr_memcache.c.fix  2022-12-21 09:45:59.058400657 +0100
+++ apr-util-1.6.1/memcache/apr_memcache.c      2022-12-21 09:47:13.035230983 +0100
@@ -290,9 +290,9 @@ static apr_status_t conn_connect(apr_mem
     apr_status_t rv = APR_SUCCESS;
     apr_sockaddr_t *sa;
 #if APR_HAVE_SOCKADDR_UN
-    apr_int32_t family = conn->ms->host[0] != '/' ? APR_INET : APR_UNIX;
+    apr_int32_t family = conn->ms->host[0] != '/' ? APR_UNSPEC : APR_UNIX;
 #else
-    apr_int32_t family = APR_INET;
+    apr_int32_t family = APR_UNSPEC;
 #endif
 
     rv = apr_sockaddr_info_get(&sa, conn->ms->host, family, conn->ms->port, 0, conn->p);
@@ -328,9 +328,9 @@ mc_conn_construct(void **conn_, void *pa
     apr_pool_t *tp;
     apr_memcache_server_t *ms = params;
 #if APR_HAVE_SOCKADDR_UN
-    apr_int32_t family = ms->host[0] != '/' ? APR_INET : APR_UNIX;
+    apr_int32_t family = ms->host[0] != '/' ? APR_UNSPEC : APR_UNIX;
 #else
-    apr_int32_t family = APR_INET;
+    apr_int32_t family = APR_UNSPEC;
 #endif
 
     rv = apr_pool_create(&np, pool);
--------------- cut here ---------------

HTH
Tomáš

Comment 6 Tomas Halman 2022-12-21 12:23:53 UTC

Moving to apr-util

Tomáš