Bug 1973674

*** Bug 1979840 has been marked as a duplicate of this bug. ***

jfrancoa got an additional error, which might be related:

    2021-07-28 16:15:18 | TASK [copy certificate, chgrp, restart haproxy] ********************************
    2021-07-28 16:15:18 | Wednesday 28 July 2021  16:15:13 +0000 (0:00:00.104)       0:02:02.950 ********
    2021-07-28 16:15:18 | skipping: [controller-1] => (item=)  => {"ansible_loop_var": "item", "changed": false, "item": "", "skip_reason": "Conditional result was False"}
    2021-07-28 16:15:18 | changed: [controller-0] => (item=40edb9869494) => {"ansible_loop_var": "item", "changed": true, "cmd": "set -e\nif podman ps -f \"id=40edb9869494\" --format \"{{.Names}}\" | grep -q \"^haproxy-bundle\"; then\n  tar -c /etc/pki/tls/private/overcloud_endpoint.pem | podman exec -i 40edb9869494 tar -C / -xv\nelse\n  podman cp /etc/pki/tls/private/overcloud_endpoint.pem 40edb9869494:/etc/pki/tls/private/overcloud_endpoint.pem\nfi\npodman exec --user root 40edb9869494 chgrp haproxy /etc/pki/tls/private/overcloud_endpoint.pem\npodman kill --signal=HUP 40edb9869494\n", "delta": "0:00:00.943979", "end": "2021-07-28 16:15:14.860717", "item": "40edb9869494", "rc": 0, "start": "2021-07-28 16:15:13.916738", "stderr": "", "stderr_lines": [], "stdout": "40edb9869494f55d9f81ceb5f83ab20cf5e221872c521e84caa3c40adaa481e5", "stdout_lines": ["40edb9869494f55d9f81ceb5f83ab20cf5e221872c521e84caa3c40adaa481e5"]}
    2021-07-28 16:15:18 | changed: [controller-0] => (item=ce11eae222e0) => {"ansible_loop_var": "item", "changed": true, "cmd": "set -e\nif podman ps -f \"id=ce11eae222e0\" --format \"{{.Names}}\" | grep -q \"^haproxy-bundle\"; then\n  tar -c /etc/pki/tls/private/overcloud_endpoint.pem | podman exec -i ce11eae222e0 tar -C / -xv\nelse\n  podman cp /etc/pki/tls/private/overcloud_endpoint.pem ce11eae222e0:/etc/pki/tls/private/overcloud_endpoint.pem\nfi\npodman exec --user root ce11eae222e0 chgrp haproxy /etc/pki/tls/private/overcloud_endpoint.pem\npodman kill --signal=HUP ce11eae222e0\n", "delta": "0:00:00.909961", "end": "2021-07-28 16:15:16.085611", "item": "ce11eae222e0", "rc": 0, "start": "2021-07-28 16:15:15.175650", "stderr": "", "stderr_lines": [], "stdout": "ce11eae222e0e564021a1594d8ee8d8afd793235cb323136a6f857ded75e28d5", "stdout_lines": ["ce11eae222e0e564021a1594d8ee8d8afd793235cb323136a6f857ded75e28d5"]}
    2021-07-28 16:15:18 | changed: [controller-0] => (item=dde4c18bb88f) => {"ansible_loop_var": "item", "changed": true, "cmd": "set -e\nif podman ps -f \"id=dde4c18bb88f\" --format \"{{.Names}}\" | grep -q \"^haproxy-bundle\"; then\n  tar -c /etc/pki/tls/private/overcloud_endpoint.pem | podman exec -i dde4c18bb88f tar -C / -xv\nelse\n  podman cp /etc/pki/tls/private/overcloud_endpoint.pem dde4c18bb88f:/etc/pki/tls/private/overcloud_endpoint.pem\nfi\npodman exec --user root dde4c18bb88f chgrp haproxy /etc/pki/tls/private/overcloud_endpoint.pem\npodman kill --signal=HUP dde4c18bb88f\n", "delta": "0:00:00.999568", "end": "2021-07-28 16:15:17.389579", "item": "dde4c18bb88f", "rc": 0, "start": "2021-07-28 16:15:16.390011", "stderr": "", "stderr_lines": [], "stdout": "dde4c18bb88f050f7631d4c2f00ede1ab812c6f5fcde999b241ce2ab48cb0b13", "stdout_lines": ["dde4c18bb88f050f7631d4c2f00ede1ab812c6f5fcde999b241ce2ab48cb0b13"]}
    2021-07-28 16:15:18 |
    2021-07-28 16:15:18 | failed: [controller-0] (item=cbd66eb3e0ce) => {"ansible_loop_var": "item", "changed": true, "cmd": "set -e\nif podman ps -f \"id=cbd66eb3e0ce\" --format \"{{.Names}}\" | grep -q \"^haproxy-bundle\"; then\n  tar -c /etc/pki/tls/private/overcloud_endpoint.pem | podman exec -i cbd66eb3e0ce tar -C / -xv\nelse\n  podman cp /etc/pki/tls/private/overcloud_endpoint.pem cbd66eb3e0ce:/etc/pki/tls/private/overcloud_endpoint.pem\nfi\npodman exec --user root cbd66eb3e0ce chgrp haproxy /etc/pki/tls/private/overcloud_endpoint.pem\npodman kill --signal=HUP cbd66eb3e0ce\n", "delta": "0:00:00.536844", "end": "2021-07-28 16:15:18.252000", "item": "cbd66eb3e0ce", "msg": "non-zero return code", "rc": 2, "start": "2021-07-28 16:15:17.715156", "stderr": "tar: Removing leading `/' from member names\ntar: This does not look like a tar archive\ntar: Exiting with failure status due to previous errors\ntime=\"2021-07-28T16:15:18Z\" level=error msg=\"read unixpacket @->/var/run/libpod/socket/2642350cdc5fbc67a2061a1471ec74c55f17e16bf4e474510fa0d819dada1628/attach: read: connection reset by peer\"\nError: non zero exit code: 2: OCI runtime error", "stderr_lines": ["tar: Removing leading `/' from member names", "tar: This does not look like a tar archive", "tar: Exiting with failure status due to previous errors", "time=\"2021-07-28T16:15:18Z\" level=error msg=\"read unixpacket @->/var/run/libpod/socket/2642350cdc5fbc67a2061a1471ec74c55f17e16bf4e474510fa0d819dada1628/attach: read: connection reset by peer\"", "Error: non zero exit code: 2: OCI runtime error"], "stdout": "", "stdout_lines": []}
    2021-07-28 16:15:18 |
    2021-07-28 16:15:18 | NO MORE HOSTS LEFT *************************************************************
    2021-07-28 16:15:18 |


I can trigger the tar error by truncating the tar archive, but this does not trigger the "read: connection reset by peer" error.

I've filed https://bugzilla.redhat.com/show_bug.cgi?id=1988330 for the error from comment #6

I can see on BZ #2008418 (the 16.2 version of this bug) that it has a target milestone of z2. Can we update this BZ with the same information for 16.1, so I can tell the customer when to expect it?

Thanks.

Tested with :
$ rpm -qa|grep openstack-tripleo-heat-templates
openstack-tripleo-heat-templates-11.3.2-1.20220114223345.el8ost.noarch

Before scale down:
(undercloud) [stack@undercloud-0 ~]$ openstack server list
+--------------------------------------+--------------+--------+------------------------+----------------+------------+
| ID                                   | Name         | Status | Networks               | Image          | Flavor     |
+--------------------------------------+--------------+--------+------------------------+----------------+------------+
| b59ee548-2f6b-4131-b915-a2b56065798d | controller-2 | ACTIVE | ctlplane=192.168.24.9  | overcloud-full | controller |
| 1ee28c02-504a-40d8-b8a5-d2c12db204c8 | controller-0 | ACTIVE | ctlplane=192.168.24.10 | overcloud-full | controller |
| d7fa4f32-79fd-4844-92ed-9f2178af8e54 | controller-1 | ACTIVE | ctlplane=192.168.24.32 | overcloud-full | controller |
| 1427a856-6371-49b1-9898-3f68592dff8b | compute-1    | ACTIVE | ctlplane=192.168.24.22 | overcloud-full | compute    |
| f66b055d-b7d8-4ccf-a2b1-958497a79855 | compute-0    | ACTIVE | ctlplane=192.168.24.42 | overcloud-full | compute    |
+--------------------------------------+--------------+--------+------------------------+----------------+------------+


After scale down:
(undercloud) [stack@undercloud-0 ~]$ openstack server list
+--------------------------------------+--------------+--------+------------------------+----------------+------------+
| ID                                   | Name         | Status | Networks               | Image          | Flavor     |
+--------------------------------------+--------------+--------+------------------------+----------------+------------+
| b59ee548-2f6b-4131-b915-a2b56065798d | controller-2 | ACTIVE | ctlplane=192.168.24.9  | overcloud-full | controller |
| 1ee28c02-504a-40d8-b8a5-d2c12db204c8 | controller-0 | ACTIVE | ctlplane=192.168.24.10 | overcloud-full | controller |
| d7fa4f32-79fd-4844-92ed-9f2178af8e54 | controller-1 | ACTIVE | ctlplane=192.168.24.32 | overcloud-full | controller |
| 1427a856-6371-49b1-9898-3f68592dff8b | compute-1    | ACTIVE | ctlplane=192.168.24.22 | overcloud-full | compute    |
+--------------------------------------+--------------+--------+------------------------+----------------+------------+


During scale up:
[root@controller-0 ~]# podman restart haproxy-bundle-podman-0
b40a511b3a094e0f6cb12d81393c7a44bb1f88a2cff5c0047925d5e9a60fa04c

After Scale up:
(undercloud) [stack@undercloud-0 ~]$ openstack server list
+--------------------------------------+--------------+--------+------------------------+----------------+------------+
| ID                                   | Name         | Status | Networks               | Image          | Flavor     |
+--------------------------------------+--------------+--------+------------------------+----------------+------------+
| 81e8bc2f-a845-4ecb-a0f4-ab423aa5ecce | compute-2    | ACTIVE | ctlplane=192.168.24.34 | overcloud-full | compute    |
| b59ee548-2f6b-4131-b915-a2b56065798d | controller-2 | ACTIVE | ctlplane=192.168.24.9  | overcloud-full | controller |
| 1ee28c02-504a-40d8-b8a5-d2c12db204c8 | controller-0 | ACTIVE | ctlplane=192.168.24.10 | overcloud-full | controller |
| d7fa4f32-79fd-4844-92ed-9f2178af8e54 | controller-1 | ACTIVE | ctlplane=192.168.24.32 | overcloud-full | controller |
| 1427a856-6371-49b1-9898-3f68592dff8b | compute-1    | ACTIVE | ctlplane=192.168.24.22 | overcloud-full | compute    |
+--------------------------------------+--------------+--------+------------------------+----------------+------------+

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.1.8 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:0986