Bug 1596942 - Barbican - Creating secret using `` openstack secret store`` is failing `` Failed to establish a new connection: [Errno 111] Connection refused',))``
Summary: Barbican - Creating secret using `` openstack secret store`` is failing `` Fa...
Keywords:
Status: CLOSED DUPLICATE of bug 1559105
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-barbican
Version: 13.0 (Queens)
Hardware: All
OS: All
medium
medium
Target Milestone: ---
: ---
Assignee: Michele Baldessari
QA Contact: Prasanth Anbalagan
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-06-30 13:01 UTC by Nilesh
Modified: 2018-07-04 15:01 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-07-04 15:01:58 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Nilesh 2018-06-30 13:01:10 UTC
Here's the complete procedure to enable barbican on an existing cloud (copying the exact commands I ran for reference):

1. Deploy overcloud
    
~~~
(overcloud) [stack@undercloud-0 ~]$ cat overcloud_deploy.sh
#!/bin/bash

openstack overcloud deploy \
--timeout 100 \
--templates /usr/share/openstack-tripleo-heat-templates \
--stack overcloud \
--libvirt-type kvm \
--ntp-server clock.redhat.com \
-e /home/stack/virt/config_lvm.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /home/stack/virt/hostnames.yml \
-e /home/stack/virt/debug.yaml \
-e /home/stack/virt/nodes_data.yaml \
-e /home/stack/virt/docker-images.yaml \
--log-file overcloud_deployment_63.log
(overcloud) [stack@undercloud-0 ~]$ 
~~~




2. custom params

~~~
(overcloud) [stack@undercloud-0 ~]$ cat configure-barbican.yaml
---
parameter_defaults:
  BarbicanSimpleCryptoGlobalDefault: true
(overcloud) [stack@undercloud-0 ~]$ 
~~~




3. Prepare new images, include custom_params.yaml and the relevant tht files.

~~~
(overcloud) [stack@undercloud-0 ~]$ cat prepares_new_container_images_for_barbican
#!/bin/bash

openstack overcloud container image prepare \
--namespace registry.access.redhat.com/rhosp13 \
--tag latest \
--push-destination 192.168.24.1:8787 \
--output-images-file ~/container-images-with-barbican.yaml \
-e /home/stack/virt/config_lvm.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /home/stack/virt/hostnames.yml \
-e /home/stack/virt/debug.yaml \
-e /home/stack/virt/nodes_data.yaml \
-e /home/stack/virt/docker-images.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services/barbican.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/barbican-backend-simple-crypto.yaml \
-e /home/stack/configure-barbican.yaml 
(overcloud) [stack@undercloud-0 ~]$ 
~~~


4. upload container images to undercloud registry

~~~
(overcloud) [stack@undercloud-0 ~]$ openstack overcloud container image upload --debug --config-file container-images-with-barbican.yaml(overcloud) [stack@undercloud-0 ~]$ 
~~~



5. prepare the new environment file


~~~
(overcloud) [stack@undercloud-0 ~]$ cat prepares_new_env_file_for_barbican
#!/bin/bash

openstack overcloud container image prepare \
--tag latest \
--namespace 192.168.24.1:8787/rhosp13 \
--output-env-file ~/container-parameters-with-barbican.yaml \
-e /home/stack/virt/config_lvm.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /home/stack/virt/hostnames.yml \
-e /home/stack/virt/debug.yaml \
-e /home/stack/virt/nodes_data.yaml \
-e /home/stack/virt/docker-images.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services/barbican.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/barbican-backend-simple-crypto.yaml \
-e /home/stack/configure-barbican.yaml 
(overcloud) [stack@undercloud-0 ~]$ 
~~~


6. update overcloud 

~~~
(overcloud) [stack@undercloud-0 ~]$ cat overcloud_deploy_with_barbican.sh
#!/bin/bash

openstack overcloud deploy \
--timeout 100 \
--templates /usr/share/openstack-tripleo-heat-templates \
--stack overcloud \
--libvirt-type kvm \
--ntp-server clock.redhat.com \
-e /home/stack/virt/config_lvm.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /home/stack/virt/hostnames.yml \
-e /home/stack/virt/debug.yaml \
-e /home/stack/virt/nodes_data.yaml \
-e /home/stack/virt/docker-images.yaml \
-e /home/stack/container-parameters-with-barbican.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services/barbican.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/barbican-backend-simple-crypto.yaml \
-e /home/stack/configure-barbican.yaml \
--log-file overcloud_deployment_63.log
(overcloud) [stack@undercloud-0 ~]$ 
~~~


7. Check heat stack-list

~~~
(undercloud) [stack@undercloud-0 ~]$ heat stack-list
WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead
+--------------------------------------+------------+-----------------+----------------------+----------------------+----------------------------------+
| id                                   | stack_name | stack_status    | creation_time        | updated_time         | project                          |
+--------------------------------------+------------+-----------------+----------------------+----------------------+----------------------------------+
| 00ed5d0a-9c09-4db2-b147-282e9058fddf | overcloud  | UPDATE_COMPLETE | 2018-06-29T13:49:10Z | 2018-06-30T10:43:24Z | e0ec08746b694d4b8a8a45db95ca0db0 |
+--------------------------------------+------------+-----------------+----------------------+----------------------+----------------------------------+
(undercloud) [stack@undercloud-0 ~]$ 
~~~


8. Endpoints created 

~~~
(overcloud) [stack@undercloud-0 ~]$ openstack endpoint list |grep -i barbican
| 1671984f717e46e9ba39143040f0f340 | regionOne | barbican     | key-manager    | True    | internal  | http://172.17.1.15:9311                       |
| 72c51082a2f64fd08ffd9f295d75abcc | regionOne | barbican     | key-manager    | True    | public    | http://10.0.0.104:9311                        |
| c4b48a09820f4e6da953cc798f475535 | regionOne | barbican     | key-manager    | True    | admin     | http://172.17.1.15:9311                       |
(overcloud) [stack@undercloud-0 ~]$ 
~~~


Succesfully installed and configure Barbican but while testing - 

~~~
(overcloud) [stack@undercloud-0 ~]$ openstack secret store --name secret01 --payload secretkey 
Failed to contact the endpoint at http://10.0.0.104:9311 for discovery. Fallback to using that endpoint as the base url.
Unable to establish connection to http://10.0.0.104:9311/secrets/: HTTPConnectionPool(host='10.0.0.104', port=9311): Max retries exceeded with url: /secrets/ (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f09b6fbf7d0>: Failed to establish a new connection: [Errno 111] Connection refused',))
(overcloud) [stack@undercloud-0 ~]$ 
~~~

~~~
(overcloud) [stack@undercloud-0 ~]$ openstack endpoint list  |grep -i barbican 
| 1671984f717e46e9ba39143040f0f340 | regionOne | barbican     | key-manager    | True    | internal  | http://172.17.1.15:9311                       |
| 72c51082a2f64fd08ffd9f295d75abcc | regionOne | barbican     | key-manager    | True    | public    | http://10.0.0.104:9311                        |
| c4b48a09820f4e6da953cc798f475535 | regionOne | barbican     | key-manager    | True    | admin     | http://172.17.1.15:9311                       |
(overcloud) [stack@undercloud-0 ~]$ 
~~~


It should be listen to all endpoints :- 
~~~
[root@controller-0 ~]# netstat -lntpu |grep -i 9311
tcp        0      0 172.17.1.24:9311        0.0.0.0:*               LISTEN      51603/httpd         
[root@controller-0 ~]# 
~~~


* Restarted haproxy on all controller's 

~~~
[root@controller-1 ~]# docker ps |grep -i haproxy
bf6c4d234903        192.168.24.1:8787/rhosp13/openstack-haproxy:pcmklatest                  "/bin/bash /usr/lo..."   22 hours ago        Up 22 hours                                      haproxy-bundle-docker-1
[root@controller-1 ~]# 
[root@controller-1 ~]# 
[root@controller-1 ~]# docker restart bf6c4d234903
bf6c4d234903
[root@controller-1 ~]# docker ps |grep -i haproxy
bf6c4d234903        192.168.24.1:8787/rhosp13/openstack-haproxy:pcmklatest                  "/bin/bash /usr/lo..."   22 hours ago        Up 2 seconds                                     haproxy-bundle-docker-1
[root@controller-1 ~]# 
~~~

* Now its listing on all endpoints 
~~~
[root@controller-1 ~]# netstat -lunpta|grep 9311  |grep LISTEN
tcp        0      0 172.17.1.15:9311        0.0.0.0:*               LISTEN      356869/haproxy      
tcp        0      0 10.0.0.104:9311         0.0.0.0:*               LISTEN      356869/haproxy      
tcp        0      0 172.17.1.19:9311        0.0.0.0:*               LISTEN      1006403/httpd       
[root@controller-1 ~]# 
~~~


* Now able to create secrets

Comment 3 Ade Lee 2018-07-04 14:18:53 UTC
This is not a Barbican issue, but rather an HAProxy one, given that an HAProxy restart fixes the issue.

IIRC, there was recently a fix to restart HAProxy in this case.  Redirecting to PIDONE.

Comment 4 Michele Baldessari 2018-07-04 15:01:58 UTC

*** This bug has been marked as a duplicate of bug 1559105 ***


Note You need to log in before you can comment on or make changes to this bug.