Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2049421

Summary: nm-online doesn't wait for internal dnsmasq instances to be functional before considering the network is up
Product: Red Hat Enterprise Linux 8 Reporter: Renaud Métrich <rmetrich>
Component: NetworkManagerAssignee: Thomas Haller <thaller>
Status: CLOSED ERRATA QA Contact: Filip Pokryvka <fpokryvk>
Severity: high Docs Contact:
Priority: high    
Version: 8.5CC: bgalvani, ferferna, lrintel, rkhan, sukulkar, thaller, till, vbenes
Target Milestone: rcKeywords: Triaged
Target Release: ---Flags: pm-rhel: mirror+
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: NetworkManager-1.39.2-1.el8 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-11-08 10:07:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2090344    

Description Renaud Métrich 2022-02-02 08:35:45 UTC 
Description of problem:

When NetworkManager is configured with multiple internal dnsmasq instances, initializing the dnsmasq instances takes some time but nm-online doesn't consider the services at all, which leads to having a race condition where the remote mounts are fired while DNS is not yet functional.
In KCS https://access.redhat.com/solutions/4879291, I propose a solution consisting in pinging the DNS servers present in /etc/resolv.conf, but this doesn't work in this particular case because the DNS server is set to 127.0.0.1 by NetworkManager, which is reachable and doesn't mean dnsmasq services are yet available.

Hence we really need to have something in NetworkManager that guarantees DNS are functional.

Ideally this should not be part of NetworkManager-wait-online which is used to reach network-online.target.
Instead , I see two alternatives:
1. having a new service, e.g. NetworkManager-dns-online, which would be used to reach nss-lookup.target
or
2. having a "NetworkManager DNS generator" systemd generator that would create runtime systemd services, one per dnsmasq instance, that would be in charge of making sure the dnsmasq instance is operational before reaching nss-lookup.target

The advantage of solution 2 is it's easily extensible from systemd's perspective.
The drawback of solution 2 is generating runtime services would require reloading systemd configuration afterwards.


Version-Release number of selected component (if applicable):

NetworkManager-1.32.10-4.el8.x86_64


How reproducible:

Always on customer setup


Steps to Reproduce:
1. Have multiple dnsmasq instances in /etc/NetworkManager/dnsmasq.d forwarding to some DNS servers but also defining their own records
2. Set up the node to be a NFS server exporting paths to some specific systems defining using a name

Actual results:

Fail to perform NFS exports at boot defined by names

Expected results:

DNS functional at time network-online.target is reached
Comment 1 Thomas Haller 2022-02-02 08:55:18 UTC 
> Have multiple dnsmasq instances in /etc/NetworkManager/dnsmasq.d

What means to have multiple dnsmasq instances? I presume, you are configuring `[main].dns=dnsmasq` in NetworkManager.conf in your case. But there is only one dnsmasq instance.



How long NM-w-o blocks, strongly depends on your configuration.

Could you please provide a complete level=TRACE log of a boot that shows this problem, to see what exactly is happening.
See https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/blob/main/contrib/fedora/rpm/NetworkManager.conf#L27 for info about logging.
Comment 2 Renaud Métrich 2022-02-02 09:05:21 UTC 
Actually yes, having a single dnsmasq instance is there, sorry for the confusion.
I'm requesting the log with tracing enabled.
Comment 5 Thomas Haller 2022-02-04 22:23:34 UTC 
Renaud, yes, that's right. A bug :)

All information present. Thank you.

(this needs fixing).
Comment 12 errata-xmlrpc 2022-11-08 10:07:32 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (NetworkManager bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:7680