Bug 1869001

Summary: Tripleo-ipa: "openstack overcloud delete" command does not trigger IPA entries cleanup for nodes
Product: Red Hat OpenStack Reporter: Marian Krcmarik <mkrcmari>
Component: python-tripleoclientAssignee: Grzegorz Grasza <ggrasza>
Status: CLOSED ERRATA QA Contact: Jeremy Agee <jagee>
Severity: medium Docs Contact:
Priority: medium    
Version: 16.1 (Train)CC: ggrasza, hbrock, jschluet, jslagle, mburns, rmascena
Target Milestone: z3Keywords: Triaged, ZStream
Target Release: 16.1 (Train on RHEL 8.2)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: python-tripleoclient-12.3.2-1.20200914164930.el8ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-12-15 18:36:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Marian Krcmarik 2020-08-15 05:08:14 UTC
Description of problem:
"openstack overcloud delete" command does not trigger IPA entries cleanup for nodes If deleted stack is deployed with tripleo-ipa. It does however work with novajoin method - IPA host entries are cleaned up.

If the playbook itself is trigger, It does work and entries are cleaned up (ansible-playbook -i generated-inventory.yaml /usr/share/ansible/tripleo-playbooks/cli-cleanup-ipa). It does not seem to work with "openstack overcloud delete"

The following steps were executed on EDGE DCN HCI (Multistack env with 3 stacks, 1 central with control plane and 2 DCNs with HCI compute nodes) env:
(undercloud) [stack@site-undercloud-0 ~]$ openstack stack list
+--------------------------------------+------------+----------------------------------+-----------------+----------------------+--------------+
| ID                                   | Stack Name | Project                          | Stack Status    | Creation Time        | Updated Time |
+--------------------------------------+------------+----------------------------------+-----------------+----------------------+--------------+
| d9938100-0331-4820-815a-9b5bd2c82f98 | dcn1       | 1f9d194eb4214c81a6cd94e1769c5e7b | CREATE_COMPLETE | 2020-08-14T07:54:17Z | None         |
| 43d154ca-eaf4-4ec9-b3ab-36ef79d0a227 | central    | 1f9d194eb4214c81a6cd94e1769c5e7b | CREATE_COMPLETE | 2020-08-14T06:50:55Z | None         |
+--------------------------------------+------------+----------------------------------+-----------------+----------------------+--------------+
(undercloud) [stack@site-undercloud-0 ~]$ openstack overcloud delete dcn1
Are you sure you want to delete this overcloud [y/N]? y
/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_delete.py:136: ResourceWarning: unclosed file <_io.BufferedReader name=6>
  python_interpreter=python_interpreter)
Undeploying stack dcn1...
Waiting for messages on queue 'tripleo' with no timeout.
Deleting plan dcn1...
Success.
sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=4, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('192.168.24.2', 46066)>
sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('192.168.24.2', 48534)>
sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('192.168.24.2', 60424), raddr=('192.168.24.2', 13989)>
(undercloud) [stack@site-undercloud-0 ~]$ openstack overcloud delete central
Are you sure you want to delete this overcloud [y/N]? y
/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_delete.py:136: ResourceWarning: unclosed file <_io.BufferedReader name=6>
  python_interpreter=python_interpreter)
Undeploying stack central...
Waiting for messages on queue 'tripleo' with no timeout.
Deleting plan central...
Success.
sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=4, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('192.168.24.2', 57942)>
sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('192.168.24.2', 60426)>
sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('192.168.24.2', 45426), raddr=('192.168.24.2', 13989)>
(undercloud) [stack@site-undercloud-0 ~]$ ipa host-find
----------------
51 hosts matched
----------------
  Host name: central-computehci0-0.internalapi.redhat.local
  Principal name: host/central-computehci0-0.internalapi.redhat.local
  Principal alias: host/central-computehci0-0.internalapi.redhat.local

  Host name: central-computehci0-0.redhat.local
  Principal name: host/central-computehci0-0.redhat.local
  Principal alias: host/central-computehci0-0.redhat.local
  SSH public key fingerprint: SHA256:JRToTm4Q4xP8L7vPiHJhehDjDPtHV5jHG4EJb1v0gbI (ecdsa-sha2-nistp256), SHA256:jOIBmj+webEBe4ITscIl1JUZ3oKy9tipvhQjmORHlIE (ssh-ed25519),
                              SHA256:Js0FC63+psNDabzOMJQJuc+6b6erQQHGolTsyjEEO6Q (ssh-rsa)

  Host name: central-computehci0-1.internalapi.redhat.local
  Principal name: host/central-computehci0-1.internalapi.redhat.local
  Principal alias: host/central-computehci0-1.internalapi.redhat.local

  Host name: central-computehci0-1.redhat.local
  Principal name: host/central-computehci0-1.redhat.local
  Principal alias: host/central-computehci0-1.redhat.local
  SSH public key fingerprint: SHA256:tmIiSbjIjZDa23yQllzgTeTD1dt2KApXoqG6Sa6gg3U (ecdsa-sha2-nistp256), SHA256:TzLyy+RhPHqnIVky9afA0b3/Fbwc4KhNa/H9I7zS26o (ssh-ed25519),
                              SHA256:zaHMECLchwXhbM2YJD5qWiyyAG9QMJOKGcM0Nz/01SU (ssh-rsa)

  Host name: central-computehci0-2.internalapi.redhat.local
  Principal name: host/central-computehci0-2.internalapi.redhat.local
  Principal alias: host/central-computehci0-2.internalapi.redhat.local

  Host name: central-computehci0-2.redhat.local
  Principal name: host/central-computehci0-2.redhat.local
  Principal alias: host/central-computehci0-2.redhat.local
  SSH public key fingerprint: SHA256:GGVnGEow1+0pwIiGzqNilivgmo7VMY6D1n25BpQN5tU (ecdsa-sha2-nistp256), SHA256:IyVlsLmCRAAYEG9ceDeNEj5Ym0kDLQj9OdEUl8EBp/A (ssh-ed25519),
                              SHA256:KWF2sf9XdnkG0NOPlw7Y27j7o8mEiy9IK8T0L3/G3dw (ssh-rsa)

  Host name: central-controller0-0.ctlplane.redhat.local
  Principal name: host/central-controller0-0.ctlplane.redhat.local
  Principal alias: host/central-controller0-0.ctlplane.redhat.local

  Host name: central-controller0-0.external.redhat.local
  Principal name: host/central-controller0-0.external.redhat.local
  Principal alias: host/central-controller0-0.external.redhat.local

  Host name: central-controller0-0.internalapi.redhat.local
  Principal name: host/central-controller0-0.internalapi.redhat.local
  Principal alias: host/central-controller0-0.internalapi.redhat.local

  Host name: central-controller0-0.redhat.local
  Principal name: host/central-controller0-0.redhat.local
  Principal alias: host/central-controller0-0.redhat.local
  SSH public key fingerprint: SHA256:pjwqf4xs0L/KCWh1xMHh8J3lMyHKH8O08pgt0cxUKGQ (ecdsa-sha2-nistp256), SHA256:4Kh089r347xQJBMKlqPwM86/tRqWhLgVSCHTcm7rXXw (ssh-ed25519),
                              SHA256:4QQXEbv1N9cTjfj03gSJTQDjfWM4AMsXX5t4AE2u6Ag (ssh-rsa)

  Host name: central-controller0-0.storage.redhat.local
  Principal name: host/central-controller0-0.storage.redhat.local
  Principal alias: host/central-controller0-0.storage.redhat.local

  Host name: central-controller0-0.storagemgmt.redhat.local
  Principal name: host/central-controller0-0.storagemgmt.redhat.local
  Principal alias: host/central-controller0-0.storagemgmt.redhat.local

  Host name: central-controller0-1.ctlplane.redhat.local
  Principal name: host/central-controller0-1.ctlplane.redhat.local
  Principal alias: host/central-controller0-1.ctlplane.redhat.local

  Host name: central-controller0-1.external.redhat.local
  Principal name: host/central-controller0-1.external.redhat.local
  Principal alias: host/central-controller0-1.external.redhat.local

  Host name: central-controller0-1.internalapi.redhat.local
  Principal name: host/central-controller0-1.internalapi.redhat.local
  Principal alias: host/central-controller0-1.internalapi.redhat.local

  Host name: central-controller0-1.redhat.local
  Principal name: host/central-controller0-1.redhat.local
  Principal alias: host/central-controller0-1.redhat.local
  SSH public key fingerprint: SHA256:dKuEeMyqpphRdmhGtC8guIaFmzRIy1xGLs2DK7mGaAk (ssh-ed25519), SHA256:Vfo8EE3JRACk4HO2mMamcO3x9zAUfoTUF+XyNSwcTko (ecdsa-sha2-nistp256),
                              SHA256:l65MQMS1sGIJT6sX3NaNnzlXSLf4FB1v7Lf8N3R7GmI (ssh-rsa)

  Host name: central-controller0-1.storage.redhat.local
  Principal name: host/central-controller0-1.storage.redhat.local
  Principal alias: host/central-controller0-1.storage.redhat.local

  Host name: central-controller0-1.storagemgmt.redhat.local
  Principal name: host/central-controller0-1.storagemgmt.redhat.local
  Principal alias: host/central-controller0-1.storagemgmt.redhat.local

  Host name: central-controller0-2.ctlplane.redhat.local
  Principal name: host/central-controller0-2.ctlplane.redhat.local
  Principal alias: host/central-controller0-2.ctlplane.redhat.local

  Host name: central-controller0-2.external.redhat.local
  Principal name: host/central-controller0-2.external.redhat.local
  Principal alias: host/central-controller0-2.external.redhat.local

  Host name: central-controller0-2.internalapi.redhat.local
  Principal name: host/central-controller0-2.internalapi.redhat.local
  Principal alias: host/central-controller0-2.internalapi.redhat.local

  Host name: central-controller0-2.redhat.local
  Principal name: host/central-controller0-2.redhat.local
  Principal alias: host/central-controller0-2.redhat.local
  SSH public key fingerprint: SHA256:K3OQac9UFViPgkNR8woT62u29QT5vBDdk3c3X2+DWxY (ecdsa-sha2-nistp256), SHA256:7XsC5a+CtJyWlArVcdBccQfKDkFbwl/K3XINj57lLAU (ssh-ed25519),
                              SHA256:4hJta+LtaDAP/0LZdg5Hrs8JPxCAXV8oRaNpXYk762M (ssh-rsa)

  Host name: central-controller0-2.storage.redhat.local
  Principal name: host/central-controller0-2.storage.redhat.local
  Principal alias: host/central-controller0-2.storage.redhat.local

  Host name: central-controller0-2.storagemgmt.redhat.local
  Principal name: host/central-controller0-2.storagemgmt.redhat.local
  Principal alias: host/central-controller0-2.storagemgmt.redhat.local

  Host name: dcn1-computehci1-0.ctlplane.redhat.local
  Principal name: host/dcn1-computehci1-0.ctlplane.redhat.local
  Principal alias: host/dcn1-computehci1-0.ctlplane.redhat.local

  Host name: dcn1-computehci1-0.internalapi.redhat.local
  Principal name: host/dcn1-computehci1-0.internalapi.redhat.local
  Principal alias: host/dcn1-computehci1-0.internalapi.redhat.local

  Host name: dcn1-computehci1-0.redhat.local
  Principal name: host/dcn1-computehci1-0.redhat.local
  Principal alias: host/dcn1-computehci1-0.redhat.local
  SSH public key fingerprint: SHA256:RGsoOoBWjWhmVh1t6TJQqXs4hcVtL0Mu6Xld6C+7pKM (ecdsa-sha2-nistp256), SHA256:cSf2bfjIoe7tXArn/G2rGkkoLYIIs6vg88zpqfx3EQE (ssh-ed25519),
                              SHA256:2ZuD3LW9BMCXbFhelwyyIy4Mf1dAE2xkF5juDFHlb2Q (ssh-rsa)

  Host name: dcn1-computehci1-0.storage.redhat.local
  Principal name: host/dcn1-computehci1-0.storage.redhat.local
  Principal alias: host/dcn1-computehci1-0.storage.redhat.local

  Host name: dcn1-computehci1-0.storagemgmt.redhat.local
  Principal name: host/dcn1-computehci1-0.storagemgmt.redhat.local
  Principal alias: host/dcn1-computehci1-0.storagemgmt.redhat.local

  Host name: dcn1-computehci1-1.ctlplane.redhat.local
  Principal name: host/dcn1-computehci1-1.ctlplane.redhat.local
  Principal alias: host/dcn1-computehci1-1.ctlplane.redhat.local

  Host name: dcn1-computehci1-1.internalapi.redhat.local
  Principal name: host/dcn1-computehci1-1.internalapi.redhat.local
  Principal alias: host/dcn1-computehci1-1.internalapi.redhat.local

  Host name: dcn1-computehci1-1.redhat.local
  Principal name: host/dcn1-computehci1-1.redhat.local
  Principal alias: host/dcn1-computehci1-1.redhat.local
  SSH public key fingerprint: SHA256:3Fg6Lu5O3kkwKC8TxGwY1AnFTqmAUiBwlPsYWaeeZM0 (ecdsa-sha2-nistp256), SHA256:fJv1zuvVhCDWT1Ci1f7yaX4OaisWRRprzYsC8TszQcg (ssh-ed25519),
                              SHA256:IrPWNvfaFa5jRvc//4yFgyFhya5eiO03ORrNA5OEPZY (ssh-rsa)

  Host name: dcn1-computehci1-1.storage.redhat.local
  Principal name: host/dcn1-computehci1-1.storage.redhat.local
  Principal alias: host/dcn1-computehci1-1.storage.redhat.local

  Host name: dcn1-computehci1-1.storagemgmt.redhat.local
  Principal name: host/dcn1-computehci1-1.storagemgmt.redhat.local
  Principal alias: host/dcn1-computehci1-1.storagemgmt.redhat.local

  Host name: dcn1-computehci1-2.ctlplane.redhat.local
  Principal name: host/dcn1-computehci1-2.ctlplane.redhat.local
  Principal alias: host/dcn1-computehci1-2.ctlplane.redhat.local

  Host name: dcn1-computehci1-2.internalapi.redhat.local
  Principal name: host/dcn1-computehci1-2.internalapi.redhat.local
  Principal alias: host/dcn1-computehci1-2.internalapi.redhat.local

  Host name: dcn1-computehci1-2.redhat.local
  Principal name: host/dcn1-computehci1-2.redhat.local
  Principal alias: host/dcn1-computehci1-2.redhat.local
  SSH public key fingerprint: SHA256:dRJZ2+UwVUohHyCSdFMuMO/J6ti5lOCyWwsSeMV96Mw (ecdsa-sha2-nistp256), SHA256:fkMe/wiDU+Ek/C9wl9xqtHxtg2AMINgQzMCfMXAbBUo (ssh-ed25519),
                              SHA256:AVOTEC6B4bQk8GMRvPOpFks+W+vY9msbg8TQuhTlNug (ssh-rsa)

  Host name: dcn1-computehci1-2.storage.redhat.local
  Principal name: host/dcn1-computehci1-2.storage.redhat.local
  Principal alias: host/dcn1-computehci1-2.storage.redhat.local

  Host name: dcn1-computehci1-2.storagemgmt.redhat.local
  Principal name: host/dcn1-computehci1-2.storagemgmt.redhat.local
  Principal alias: host/dcn1-computehci1-2.storagemgmt.redhat.local

  Host name: dcn1-computehciscaleout1-0.ctlplane.redhat.local
  Principal name: host/dcn1-computehciscaleout1-0.ctlplane.redhat.local
  Principal alias: host/dcn1-computehciscaleout1-0.ctlplane.redhat.local

  Host name: dcn1-computehciscaleout1-0.internalapi.redhat.local
  Principal name: host/dcn1-computehciscaleout1-0.internalapi.redhat.local
  Principal alias: host/dcn1-computehciscaleout1-0.internalapi.redhat.local

  Host name: dcn1-computehciscaleout1-0.redhat.local
  Principal name: host/dcn1-computehciscaleout1-0.redhat.local
  Principal alias: host/dcn1-computehciscaleout1-0.redhat.local
  SSH public key fingerprint: SHA256:iQdwA6bUvNwWm4gLcboAJNIFtCz+wtZBd4s4R2wUSqE (ecdsa-sha2-nistp256), SHA256:7A32u8shT2Cnowot7rl5Hps4CwTHs08fwPXrsycxxIk (ssh-ed25519),
                              SHA256:GSBHKE4jH7UEunCZqOua7waLSqGCxO746LUoQBs5N00 (ssh-rsa)

  Host name: dcn1-computehciscaleout1-0.storage.redhat.local
  Principal name: host/dcn1-computehciscaleout1-0.storage.redhat.local
  Principal alias: host/dcn1-computehciscaleout1-0.storage.redhat.local

  Host name: dcn1-computehciscaleout1-0.storagemgmt.redhat.local
  Principal name: host/dcn1-computehciscaleout1-0.storagemgmt.redhat.local
  Principal alias: host/dcn1-computehciscaleout1-0.storagemgmt.redhat.local

  Host name: overcloud.ctlplane.redhat.local
  Principal name: host/overcloud.ctlplane.redhat.local
  Principal alias: host/overcloud.ctlplane.redhat.local

  Host name: overcloud.internalapi.redhat.local
  Principal name: host/overcloud.internalapi.redhat.local
  Principal alias: host/overcloud.internalapi.redhat.local

  Host name: overcloud.redhat.local
  Principal name: host/overcloud.redhat.local
  Principal alias: host/overcloud.redhat.local

  Host name: overcloud.storage.redhat.local
  Principal name: host/overcloud.storage.redhat.local
  Principal alias: host/overcloud.storage.redhat.local

  Host name: overcloud.storagemgmt.redhat.local
  Principal name: host/overcloud.storagemgmt.redhat.local
  Principal alias: host/overcloud.storagemgmt.redhat.local

  Host name: site-freeipa-0.redhat.local
  Principal name: host/site-freeipa-0.redhat.local
  Principal alias: host/site-freeipa-0.redhat.local
  SSH public key fingerprint: SHA256:VNyTq/h9StFqeF4MwkFN1o0uC/SJsliu3DVPhKB4fV8 (ecdsa-sha2-nistp256), SHA256:SZ2GgM/QNYxvpA+PigUqfiFVQCscqwgfN2BD+sa4emk (ssh-ed25519),
                              SHA256:aIdLQ04ZpvT5IbkFbwlBL4RxMW5Lm+s1TFPUaF4BL7k (ssh-rsa)

  Host name: site-undercloud-0.redhat.local
  Principal name: host/site-undercloud-0.redhat.local
  Principal alias: host/site-undercloud-0.redhat.local
  SSH public key fingerprint: SHA256:ks3RPO1IXGMC6E5Ce4be1mw4RiYubmUkt/wQuI58p7A (ecdsa-sha2-nistp256), SHA256:JPwm/chcykLYmGnxdLlqAsFe5wDriGD3eIa/iUMVLAQ (ssh-ed25519),
                              SHA256:9NfuhRS0lxz6Aqz2AT6x7Y5KNXTl0LL/WQamSkzAa5M (ssh-rsa)
-----------------------------
Number of entries returned 51
-----------------------------

However The entries are not there if the same command is used but stack was deployed with novajoin method for TLS-E

Version-Release number of selected component (if applicable):
16.1

How reproducible:
Always

Steps to Reproduce:
1. Deploy stack with TLS-E (tripleo-ipa)
2. Try to delete the stack with "openstack overcloud delete"

Actual results:
IPA host entries are not cleaned up

Expected results:
Host from deleted overcloud should not have entries in "ipa host-find"

Additional info:
The failing command is recommended to use for cleanup in docs:
https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/features/tls-everywhere.html#deleting-overclouds

Comment 1 Grzegorz Grasza 2020-09-01 15:01:11 UTC
We tried this on a simple deployment with OSP 16.1 (RHOS-16.1-RHEL-8-20200821.n.0) but was unable to reproduce this. Can you provide us with the environment that contains the error?

Comment 19 errata-xmlrpc 2020-12-15 18:36:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.1.3 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:5413