Doesn't seem that we're seeing the same results in CI pipelines:
https://zuul.opendev.org/t/openstack/build/aac5ea93080a4001882e09dd331c56e9/log/logs/undercloud/home/zuul/ansible.log#716

Can you re-run the Ansible execution with -vvv to capture the debug logs? (You can just run the ansible-playbook-command.sh script in the config-download directory with -vvv).
Are you passing in any custom firewall rules?

no custom firewall rules and it doesn't seem to be getting stuck on the update (so it looks to be only happening on the initial deployment) .. We are at the OSP17.1 hackfest, I'll try to get someone who has not yet deployed their overcloud to start the process with the --debug in the script and will try to capture that.

I saw this hang for minutes on my deployment too -- it looked like it was waiting for a slow-running "iptables -t filter -L INPUT" to run on a compute node.  Maybe struggling to run reverse DNS lookups?  Running "iptables -t filter -L INPUT -n" there returned instantly.

another observation is that the process gets stuck even longer on the more complex deployments - for example the process has been waiting for over 2 hours on the DCN style deployment with 3 times as many subnets and IPs

Brendan,

Here is a debug log from another run. 
http://chrisj.cloud/ansible.log.tar.bz

Seems to be fine on the Compute nodes:
2023-08-23 11:54:40,122 p=89502 u=stack n=ansible | 2023-08-23 11:54:40.122193 | fa163e1d-f3ca-004e-1d59-000000003703 |    SUMMARY | overcloud-computehci-0 | tripleo_firewall : Manage firewall rules | 27.44s
2023-08-23 11:54:40,122 p=89502 u=stack n=ansible | 2023-08-23 11:54:40.122654 | fa163e1d-f3ca-004e-1d59-000000003703 |    SUMMARY | overcloud-computehci-1 | tripleo_firewall : Manage firewall rules | 27.02s
2023-08-23 11:54:40,123 p=89502 u=stack n=ansible | 2023-08-23 11:54:40.123053 | fa163e1d-f3ca-004e-1d59-000000003703 |    SUMMARY | overcloud-computehci-2 | tripleo_firewall : Manage firewall rules | 26.62s

Just the Controller nodes that it appears to be taking ages on:
2023-08-23 11:54:40,116 p=89502 u=stack n=ansible | 2023-08-23 11:54:40.115993 | fa163e1d-f3ca-004e-1d59-000000003703 |    SUMMARY | overcloud-controller-1 | tripleo_firewall : Manage firewall rules | 3335.07s
2023-08-23 11:54:40,116 p=89502 u=stack n=ansible | 2023-08-23 11:54:40.116415 | fa163e1d-f3ca-004e-1d59-000000003703 |    SUMMARY | overcloud-controller-0 | tripleo_firewall : Manage firewall rules | 3072.81s
2023-08-23 11:54:40,117 p=89502 u=stack n=ansible | 2023-08-23 11:54:40.116855 | fa163e1d-f3ca-004e-1d59-000000003703 |    SUMMARY | overcloud-controller-2 | tripleo_firewall : Manage firewall rules | 2229.57s

The nodes aren't being rebooted or something are they? I noticed it also complains about them being UNREACHABLE at some point too:
2023-08-23 11:26:18,143 p=89502 u=stack n=ansible | 2023-08-23 11:26:18.143012 | fa163e1d-f3ca-004e-1d59-000000003703 | UNREACHABLE | Manage firewall rules | overcloud-controller-2
2023-08-23 11:40:22,216 p=89502 u=stack n=ansible | 2023-08-23 11:40:22.215375 | fa163e1d-f3ca-004e-1d59-000000003703 | UNREACHABLE | Manage firewall rules | overcloud-controller-0
2023-08-23 11:44:44,994 p=89502 u=stack n=ansible | 2023-08-23 11:44:44.994172 | fa163e1d-f3ca-004e-1d59-000000003703 | UNREACHABLE | Manage firewall rules | overcloud-controller-1
2023-08-23 11:54:40,116 p=89502 u=stack n=ansible | 2023-08-23 11:54:40.115993 | fa163e1d-f3ca-004e-1d59-000000003703 |    SUMMARY | overcloud-controller-1 | tripleo_firewall : Manage firewall rules | 3335.07s
2023-08-23 11:54:40,116 p=89502 u=stack n=ansible | 2023-08-23 11:54:40.116415 | fa163e1d-f3ca-004e-1d59-000000003703 |    SUMMARY | overcloud-controller-0 | tripleo_firewall : Manage firewall rules | 3072.81s
2023-08-23 11:54:40,117 p=89502 u=stack n=ansible | 2023-08-23 11:54:40.116855 | fa163e1d-f3ca-004e-1d59-000000003703 |    SUMMARY | overcloud-controller-2 | tripleo_firewall : Manage firewall rules | 2229.57s


Maybe we'll need sosreports from those controllers along with the config-download directory from the director node.

Just dumping some context behind this bug.

We changed the tooling to manage firewall rules from OSP16 to 17. In OSP16 and older versions we used puppet
(firewall_rule type from puppetlabs-firewall), while in OSP17 we use ansible (built-in iptables module).
The core difference between iptables module  by ansible and one by ansible is how the tooling checks whether
a requested rule already exists or not before inserting it. The firewall_rule resource type in puppetlabs-firewall
implements the 'prefetch' interface, which means the provide first gets list of all existing rules, using
a single iptables-save command. On the other hand, the iptables module in ansible is not such smart and
checks existence of chain and rule by `iptables -C` and `iptables -L` in every single insert/delete operation.
This would be the reason why the task takes longer time in case there are more rules being configured.

I agree with Eric about what he mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=2233300#c3 .
My current guess is that these iptables command is stuck at DNS resolution because -n is not added to these
list commands.

My initial question here would be
 - Do these overcloud nodes have any DNS servers configured ?
  - If yes, are they really reachable from these overcloud nodes ?
  - If no, is the issue reproduced even when you prepare DNS server ?

If the problem can be reproduced in the deployment WITHOUT DNS server then we probably have to modify
the built-in iptables module somehow to avoid that DNS resolution

Hi team,

My customer will deploy a new OSP17.1 deployment. If we suspect iptables.py in ansible, they are willing to test the possible fix.
In this discussion, we suspect that ansible's current entry search method is not sophisticated.
I think the modification points are

def get_chain_policy(iptables_path, module, params):
    cmd = push_arguments(iptables_path, '-L', params, make_rule=False)

ref. https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/iptables.py#L731-L732

def check_chain_present(iptables_path, module, params):
    cmd = push_arguments(iptables_path, '-L', params, make_rule=False)

ref. https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/iptables.py#L754-L755

The current passed argument, '-L', should be '-n -L'?

If yes, I'll share the modification to the customer and get the result from them.

Best regards,
Keigo Noha

Keigo,

I expect that change would work, and may be worth trying, but I can't guarantee that change is
really acceptable from general ansible's PoV because the DNS resolution is required to support
usage of hostname instead of IPs.

Also, we first need to understand this point, which I mentioned earlier in my comment, to understand
this is an ansigle bug or an environment problem.

> My initial question here would be
>  - Do these overcloud nodes have any DNS servers configured ?
>   - If yes, are they really reachable from these overcloud nodes ?
>   - If no, is the issue reproduced even when you prepare DNS server ?
>
> If the problem can be reproduced in the deployment WITHOUT DNS server then we probably have to modify
> the built-in iptables module somehow to avoid that DNS resolution

Thanks for looking into this. Generally in our labs we are running in a disconnected fashion with no real DNS serving those labs. We've been doing this since OSP7. Instead the local /etc/hosts has always been generated by tripleo to satisfy the DNS needs. If that is no longer acceptable architecture, we will change and probably figure out how to make undercloud to also act as a DNS server (since that's the only host accessible to all the overcloud nodes in our labs). 
I am not sure why would computes not be affected .. is it possible /etc/hosts gets generated there quicker?

With that being said adding -n parameter to iptables_path sounds like a reasonable fix. I don't see the downside to doing this.

It's true it does not have downside in our usage, but as I said it breaks a different usage.
Because the module causing the problem is not our own module but comes from ansible-core, the change
may not be accepted.

@Chris
Is it possible that you capture following items and share these ?

1. sosreport from
 - controller nodes
 - one compute node
 - undercloud

2. deployment templates and the actual command line of `openstack overcloud deploy`

3. files under /home/stack/overcloud-deploy

If the environment can be shared we can connect to the nodes to look into details next week.

So, Takashi has since left Red Hat. But allow me to summarise the conclusions here:

1. Ideally, we need a valid DNS server. But I understand there are test scenarios where you don't necessarily need one. In that case, it's either going to be slow. Or, 

2. We need to fix iptables and / or Ansible to not do DNS lookups. However, in either of these two cases, they fall outside of the OpenStack scope and a BZ would need to be raised with the relevant component team. Probably Ansible would be easier to address compared to iptables I would imagine.

The recommendation for anyone hitting this issue is to use a valid DNS server, or configure a valid DNS service to respond to those DNS queries.

Hi Brendan,

Thank you for your comment on this bugzilla.
I don't collect the number of customers who deploy RHOSP in disconnected environment.
However, in some cases, we saw there were several customer who deployed RHOSP without DNS servers.
Unfortunately, we don't have a statement in our doc which describes DNS server is required.

What do you think if we open this issue to ansible for fixing issue?

Best regards,
Keigo Noha

Hey Keigo,

We can open an issue with Ansible and see if they want to improve the iptables handling when DNS servers aren't reachable. 

We should probably also mention that a valid DNS server is required in the docs. Even if you were using Satellite for the disconnected environment, you would still need to resolve the Satellite servers FQDN to pull packages and containers.

*** Bug 2265072 has been marked as a duplicate of this bug. ***

*** Bug 2247573 has been marked as a duplicate of this bug. ***

https://pkgs.devel.redhat.com/cgit/rpms/openstack-ansible-core/commit/?h=rhos-17.1-rhel-9-trunk&id=9e1c21519e98e1dc8b2fba591e09b6953ad36de7

still needs to be backported to rhos-17.1-rhel-9

the tripleo-ansible patch https://code.engineering.redhat.com/gerrit/c/tripleo-ansible/+/451168 has already merged on  rhos-17.1-patches

Next steps would be:

backport https://pkgs.devel.redhat.com/cgit/rpms/openstack-ansible-core/commit/?h=rhos-17.1-rhel-9-trunk&id=9e1c21519e98e1dc8b2fba591e09b6953ad36de7 to rhos-17.1-rhel-9

build tripleo-ansible with https://code.engineering.redhat.com/gerrit/c/tripleo-ansible/+/451168

@lhh Is release delivery going to take care of those, or is it expected the DFG does it?

reassigning as this should be DF owned for dev and qe

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Red Hat OpenStack Platform 17.1 (openstack-ansible-core) security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2024:2733