Bug 1576436

Summary: Containerzed barbican with Octavia - Listener with secret failes to be created.
Product: Red Hat OpenStack Reporter: Alexander Stafeyev <astafeye>
Component: openstack-octaviaAssignee: Carlos Goncalves <cgoncalves>
Status: CLOSED WORKSFORME QA Contact: Alexander Stafeyev <astafeye>
Severity: medium Docs Contact:
Priority: high    
Version: 13.0 (Queens)CC: bcafarel, cgoncalves, ihrachys, lpeer, majopela, nyechiel
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-05-15 21:34:49 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1553520    

Description Alexander Stafeyev 2018-05-09 12:58:44 UTC
Description of problem:
In order to configure TLS termination listener we need to use barbican secret flag . 

With non cotnainerized barbican a secret id was acceptable and listener was created ( see https://bugzilla.redhat.com/show_bug.cgi?id=1553520) 

I installed containerized barbican and the secret id nor secret container di is not acceptable. 

Version-Release number of selected component (if applicable):


How reproducible:
100%

Steps to Reproduce:
1. Deploy openstack with Octavia and containerized Barbican. 
2. Create ssl keys and certs and create Barbican secret. 
3. Try to create a https termination listener
4. Create a secret container and try to create the same listener with the container parameters. 



Actual results:
# openstack loadbalancer listener create --protocol-port 443 --protocol TERMINATED_HTTPS --name listener1 --default-tls-container=$(openstack secret list | awk '/ tls_secret1 / {print $2}') lb-ter-tls
Could not retrieve certificate: ['http://10.0.0.108:9311/v1/secrets/d2fbbce2-62db-45b1-9578-e349865bd686'] (HTTP 400) (Request-ID: req-78a8fe3f-a34f-4a97-b1b4-fc68101d0d45)


Expected results:
Lestener should be created

Additional info:


Deployment info: 

Overcloud deploy command : 
openstack overcloud deploy \
--timeout 100 \
--templates /usr/share/openstack-tripleo-heat-templates \
--stack overcloud \
--libvirt-type kvm \
--ntp-server clock.redhat.com \
-e /home/stack/virt/config_lvm.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/octavia.yaml \
-e /home/stack/virt/hostnames.yml \
-e /home/stack/virt/nodes_data.yaml \
--environment-file /usr/share/openstack-tripleo-heat-templates/environments/services-docker/barbican.yaml \
--environment-file /usr/share/openstack-tripleo-heat-templates/environments/barbican-backend-simple-crypto.yaml \
-e /home/stack/virt/extra_templates.yaml \
-e /home/stack/virt/docker-images.yaml \
--log-file overcloud_deployment_2.log


(overcloud) [stack@undercloud-0 ~]$ 
(overcloud) [stack@undercloud-0 ~]$ cat /home/stack/virt/extra_templates.yaml
parameter_defaults:
    BarbicanSimpleCryptoGlobalDefault: true


SSL info: 


# openssl pkcs12 -export -inkey /tmp/octavia-ssl/client.key -in /tmp/octavia-ssl/client-.pem -certfile /tmp/octavia-ssl/ca_01.pem -passout pass:qwerty123 -out server.p12
# openstack secret store --name='tls_secret1' -t 'application/octet-stream' -e 'base64' --payload="$(base64 < server.p12)"
# openstack user list | grep octav
# openstack acl user add -u d60a19f3157541e3a4a534971bc42148 $(openstack secret list | awk '/ tls_secret1 / {print $2}')
# openstack loadbalancer create --name lb-ter-tls --vip-network-id 0d8ac6f6-c7cb-4f24-b9c8-e4c56a1a17ae


The "d60a19f3157541e3a4a534971bc42148" id is octavia user id. 



The Octavia logs: 
[root@controller-0 ~]# tailf /var/log/containers/octavia/api.log 
2018-05-08 11:16:59.051 1 INFO octavia.certificates.manager.barbican [req-5c91dc92-95bd-408d-8e7e-ffa4e874353f - 467d248e3ac5410a98b3d041caa7370c - default default] Loading certificate secret http://10.0.0.108:9311/v1/secrets/d2fbbce2-62db-45b1-9578-e349865bd686 from Barbican.
2018-05-08 11:16:59.122 1 INFO octavia.certificates.manager.barbican_legacy [req-5c91dc92-95bd-408d-8e7e-ffa4e874353f - 467d248e3ac5410a98b3d041caa7370c - default default] Loading certificate container http://10.0.0.108:9311/v1/secrets/d2fbbce2-62db-45b1-9578-e349865bd686 from Barbican.
2018-05-08 11:16:59.150 1 ERROR octavia.certificates.manager.barbican_legacy [req-5c91dc92-95bd-408d-8e7e-ffa4e874353f - 467d248e3ac5410a98b3d041caa7370c - default default] Error getting cert http://10.0.0.108:9311/v1/secrets/d2fbbce2-62db-45b1-9578-e349865bd686: Unknown container type "".



Saving a secret into container did not help. In the bug mentioned in the begging it worked with secret ID (reminder). Here I will try with continer id: 

I saved the secret in a container but the result is the same: 

(overcloud) [stack@undercloud-0 ~]$ openstack secret list 
+------------------------------------------------------------------------+-------------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+
| Secret href                                                            | Name        | Created                   | Status | Content types                             | Algorithm | Bit length | Secret type | Mode | Expiration |
+------------------------------------------------------------------------+-------------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+
| http://10.0.0.108:9311/v1/secrets/d2fbbce2-62db-45b1-9578-e349865bd686 | tls_secret1 | 2018-05-08T10:37:54+00:00 | ACTIVE | {u'default': u'application/octet-stream'} | aes       |        256 | opaque      | cbc  | None       |
+------------------------------------------------------------------------+-------------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+
(overcloud) [stack@undercloud-0 ~]$ openstack  secret container create --name container1 --secret http://10.0.0.108:9311/v1/secrets/d2fbbce2-62db-45b1-9578-e349865bd686 
+----------------+---------------------------------------------------------------------------+
| Field          | Value                                                                     |
+----------------+---------------------------------------------------------------------------+
| Container href | http://10.0.0.108:9311/v1/containers/99033c41-5437-448d-80f6-de9d18ecc3d6 |
| Name           | container1                                                                |
| Created        | None                                                                      |
| Status         | ACTIVE                                                                    |
| Type           | generic                                                                   |
| Secrets        | None                                                                      |
| Consumers      | None                                                                      |
+----------------+---------------------------------------------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ 
(overcloud) [stack@undercloud-0 ~]$ 
(overcloud) [stack@undercloud-0 ~]$ 
(overcloud) [stack@undercloud-0 ~]$ 
(overcloud) [stack@undercloud-0 ~]$ 
(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer listener create --protocol-port 443 --protocol TERMINATED_HTTPS --name listener1 --default-tls-container=http://10.0.0.108:9311/v1/secrets/d2fbbce2-62db-45b1-9578-e349865bd686 lb-ter-tls
Could not retrieve certificate: ['http://10.0.0.108:9311/v1/secrets/d2fbbce2-62db-45b1-9578-e349865bd686'] (HTTP 400) (Request-ID: req-b91b62a3-e0b3-44bc-be48-94ffb732c202)
(overcloud) [stack@undercloud-0 ~]$ 




Same error and logs : 

2018-05-08 15:07:56.719 1 INFO octavia.certificates.manager.barbican [req-b91b62a3-e0b3-44bc-be48-94ffb732c202 - 467d248e3ac5410a98b3d041caa7370c - default default] Loading certificate secret http://10.0.0.108:9311/v1/secrets/d2fbbce2-62db-45b1-9578-e349865bd686 from Barbican.
2018-05-08 15:07:57.635 1 INFO octavia.certificates.manager.barbican_legacy [req-b91b62a3-e0b3-44bc-be48-94ffb732c202 - 467d248e3ac5410a98b3d041caa7370c - default default] Loading certificate container http://10.0.0.108:9311/v1/secrets/d2fbbce2-62db-45b1-9578-e349865bd686 from Barbican.
2018-05-08 15:07:57.657 1 ERROR octavia.certificates.manager.barbican_legacy [req-b91b62a3-e0b3-44bc-be48-94ffb732c202 - 467d248e3ac5410a98b3d041caa7370c - default default] Error getting cert http://10.0.0.108:9311/v1/secrets/d2fbbce2-62db-45b1-9578-e349865bd686: Unknown container type "".

Comment 1 Carlos Goncalves 2018-05-15 21:34:49 UTC
Cannot reproduce. I think the issue you're having is due to the fact you're setting a password for file server.p12 ("-passout pass:qwerty123"). Please check your setup and reopen this rhbz if needed.


Steps I used to reproduce with OSP13 (puddle 2018-05-10.3) + Octavia and Barbican containerized:


openstack overcloud deploy \
--timeout 100 \
--templates /usr/share/openstack-tripleo-heat-templates \
--stack overcloud \
--libvirt-type kvm \
--ntp-server clock.redhat.com \
-e /home/stack/virt/config_lvm.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/octavia.yaml \
-e /home/stack/virt/inject-trust-anchor.yaml \
-e /home/stack/virt/hostnames.yml \
-e /home/stack/virt/debug.yaml \
-e /home/stack/virt/nodes_data.yaml \
-e /home/stack/virt/barbican.yaml \
-e /home/stack/virt/docker-images.yaml \
--environment-file /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml \
--environment-file /usr/share/openstack-tripleo-heat-templates/environments/services-docker/barbican.yaml \
--environment-file /usr/share/openstack-tripleo-heat-templates/environments/barbican-backend-simple-crypto.yaml \
--log-file overcloud_deployment_54.log


[stack@undercloud-0 ~]$ cat /home/stack/virt/barbican.yaml
parameter_defaults:
    BarbicanSimpleCryptoGlobalDefault: true



(overcloud) [stack@undercloud-0 ~]$ openstack user show octavia
+---------------------+----------------------------------+
| Field               | Value                            |
+---------------------+----------------------------------+
| domain_id           | default                          |
| email               | octavia@localhost                |
| enabled             | True                             |
| id                  | 197f1542d32248c99a08f22f35e2080d |
| name                | octavia                          |
| options             | {}                               |
| password_expires_at | None                             |
+---------------------+----------------------------------+
(overcloud) [stack@undercloud-0 octavia-ssl]$ openssl pkcs12 -export -inkey server.key -in server.crt -certfile ca-chain.crt -passout pass: -out server.p12
(overcloud) [stack@undercloud-0 octavia-ssl]$ openstack secret store --name='tls_secret1' -t 'application/octet-stream' -e 'base64' --payload="$(base64 < server.p12)"
(overcloud) [stack@undercloud-0 octavia-ssl]$ openstack acl user add -u 197f1542d32248c99a08f22f35e2080d $(openstack secret list | awk '/ tls_secret1 / {print $2}')
(overcloud) [stack@undercloud-0 octavia-ssl]$ openstack loadbalancer listener create --protocol-port 443 --protocol TERMINATED_HTTPS --name listener1 --default-tls-container=$(openstack secret list | awk '/ tls_secret1 / {print $2}') lb1
(overcloud) [stack@undercloud-0 octavia-ssl]$ openstack loadbalancer listener show listener1
+---------------------------+------------------------------------------------------------------------+
| Field                     | Value                                                                  |
+---------------------------+------------------------------------------------------------------------+
| admin_state_up            | True                                                                   |
| connection_limit          | -1                                                                     |
| created_at                | 2018-05-15T21:18:47                                                    |
| default_pool_id           | None                                                                   |
| default_tls_container_ref | http://10.0.0.103:9311/v1/secrets/00a8ac26-33d0-4d35-99d4-4813c4c77507 |
| description               |                                                                        |
| id                        | 421886f9-5102-4c0f-8d95-732ec623a46b                                   |
| insert_headers            | None                                                                   |
| l7policies                |                                                                        |
| loadbalancers             | 063ecfe3-92b7-4554-881e-49362fadc85a                                   |
| name                      | listener1                                                              |
| operating_status          | ONLINE                                                                 |
| project_id                | 18cae82661624a12bd4c5b908044fcea                                       |
| protocol                  | TERMINATED_HTTPS                                                       |
| protocol_port             | 443                                                                    |
| provisioning_status       | ACTIVE                                                                 |
| sni_container_refs        | []                                                                     |
| updated_at                | 2018-05-15T21:18:54                                                    |
+---------------------------+------------------------------------------------------------------------+


==> /var/log/containers/octavia/worker.log <==
2018-05-15 21:18:47.510 23 INFO octavia.controller.queue.endpoint [-] Creating listener '421886f9-5102-4c0f-8d95-732ec623a46b'...
2018-05-15 21:18:47.556 23 INFO octavia.certificates.manager.barbican [req-b9678dd5-7afa-45f6-b00c-58cbc404fca4 - 18cae82661624a12bd4c5b908044fcea - - -] Loading certificate secret http://10.0.0.103:9311/v1/secrets/00a8ac26-33d0-4d35-99d4-4813c4c77507 from Barbican.