Bug 2106242 - passt: can find dns search list on the guest with "--no-dhcp-search" option
Summary: passt: can find dns search list on the guest with "--no-dhcp-search" option
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: passt
Version: 9.1
Hardware: All
OS: All
medium
medium
Target Milestone: rc
: ---
Assignee: Stefano Brivio
QA Contact: Lei Yang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-07-12 07:26 UTC by Quan Wenli
Modified: 2023-06-30 06:20 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-02-23 23:43:22 UTC
Type: ---
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-127456 0 None None None 2022-07-12 07:37:55 UTC

Description Quan Wenli 2022-07-12 07:26:35 UTC
Description of problem:

run passt with -no-dhcp-search option, still can find the DNS serach list on the guest

Version-Release number of selected component (if applicable):


How reproducible:
passt-0.git.2022_06_08.d7d467f-0.el9.x86_64

Steps to Reproduce:
1.[test@dell-per440-18 ~]$ /usr/local/bin/passt --no-dhcp-search
Outbound interface: eno1
ARP:
    address: 2c:ea:7f:71:b6:ee
DHCP:
    assign: 10.73.114.95
    mask: 255.255.254.0
    router: 10.73.115.254
DNS:
    10.73.2.107
    10.73.2.108
    10.66.127.10
DNS search list:
    lab.eng.pek2.redhat.com
NDP/DHCPv6:
    assign: 2620:52:0:4972:2eea:7fff:fe71:b6ee
    router: fe80::cee1:9402:8b35:be41
    our link-local: fe80::2eea:7fff:fe71:b6ee
DNS search list:
    lab.eng.pek2.redhat.com
UNIX domain socket bound at /tmp/passt_1.socket

You can now start qrap:
    ./qrap 5 kvm ... -net socket,fd=5 -net nic,model=virtio
or directly qemu, patched with:
    qemu/0001-net-Allow-also-UNIX-domain-sockets-to-be-used-as-net.patch
as follows:
    kvm ... -net socket,connect=/tmp/passt_1.socket -net nic,model=virtio
2.boot up guest

[test@dell-per440-18 ~]$PATH=$PATH:/usr/libexec
[test@dell-per440-18 ~]$ qrap 5 qemu-kvm -m 16059 -cpu host -smp 6 -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=./rhel900-64-virtio.qcow2 -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0 -nographic -serial stdio -nodefaults -device virtio-net-pci,netdev=hostnet0,x-txburst=16384 -netdev socket,fd=5,id=hostnet0

3. on guest: 

[root@dell-per440-18 ~]# sed 's/\. / /g' /etc/resolv.conf | sed 's/\.$//g' | sed -n 's/^search \(.*\)/\1/p' | tr ' \n' ',' | sed 's/\(.*\'
lab.eng.pek2.redhat.com,

Actual results:

Could not find dns search list 

Expected results:

Additional info:

Comment 1 Stefano Brivio 2022-07-13 06:29:37 UTC
Could you please collect a guest-side packet capture by -p (--pcap) argument,
such as "-p dns_search.pcap"? I'm trying to understand over which protocol the
DNS search list was assigned to narrow down the issue, as it might be sent over
DHCP, DHCPv6 or NDP.

Comment 3 Quan Wenli 2022-07-13 10:58:51 UTC
(In reply to Stefano Brivio from comment #1)
> Could you please collect a guest-side packet capture by -p (--pcap) argument,
> such as "-p dns_search.pcap"? I'm trying to understand over which protocol
> the
> DNS search list was assigned to narrow down the issue, as it might be sent
> over
> DHCP, DHCPv6 or NDP.

I have uploaded the dns_search.pcap. please check it.

Comment 4 Stefano Brivio 2022-07-13 14:54:30 UTC
(In reply to Quan Wenli from comment #3)
> 
> I have uploaded the dns_search.pcap. please check it.

Thanks! There, we see:

- a DHCP ACK message in frame 4, which doesn't contain any DNS domain search information (that would be option 119, missing here)

- a Router Advertisement in frame 33, which also doesn't contain any DNS domain search information (that would be option type 31, missing)

- a DHCPv6 reply in frame 43, which doesn't contain DNS domain search information either (that would be option 24, missing)

So I don't see any way passt is sharing any DNS domain search list with the guest.

Could it be that NetworkManager is filling that in by means of static configuration, or stored configuration from previous attempts? It might be worth to check for any such information persistently stored under /var/lib/NetworkManager/.

Comment 5 Quan Wenli 2022-07-18 06:34:45 UTC
(In reply to Stefano Brivio from comment #4)
> (In reply to Quan Wenli from comment #3)
> > 
> > I have uploaded the dns_search.pcap. please check it.
> 
> Thanks! There, we see:
> 
> - a DHCP ACK message in frame 4, which doesn't contain any DNS domain search
> information (that would be option 119, missing here)
> 
> - a Router Advertisement in frame 33, which also doesn't contain any DNS
> domain search information (that would be option type 31, missing)
> 
> - a DHCPv6 reply in frame 43, which doesn't contain DNS domain search
> information either (that would be option 24, missing)
> 
> So I don't see any way passt is sharing any DNS domain search list with the
> guest.

But there could see "DNS search list" in passt output in comment #0, is it corret?

> 
> Could it be that NetworkManager is filling that in by means of static
> configuration, or stored configuration from previous attempts? It might be
> worth to check for any such information persistently stored under
/var/lib/NetworkManager/.

Even stop the NetworkManager service, still could see the dhs search list. 
[root@dell-per440-18 NetworkManager]# sed 's/\. / /g' /etc/resolv.conf | sed 's/\.$//g' | sed -n 's/^search \(.*\)/\1/p' | tr ' \n' ',' | sed 's/\(.*\)/\1\n/g'
lab.eng.pek2.redhat.com,

[root@dell-per440-18 NetworkManager]# ls
internal-b9a0c277-d395-47fa-b0c5-3f1779033606-eth0.lease  seen-bssids
NetworkManager-intern.conf                                timestamps
NetworkManager.state


[root@dell-per440-18 NetworkManager]# cat internal-b9a0c277-d395-47fa-b0c5-3f1779033606-eth0.lease
# This is private data. Do not parse.
ADDRESS=10.73.114.95

[root@dell-per440-18 NetworkManager]# cat NetworkManager.state
[main]
NetworkingEnabled=true
WirelessEnabled=true
WWANEnabled=true

[root@dell-per440-18 NetworkManager]# cat timestamps
[timestamps]
b9a0c277-d395-47fa-b0c5-3f1779033606=1658125232

files NetworkManager-intern.conf and see-bssids are empty

Comment 6 Stefano Brivio 2022-07-18 21:19:35 UTC
(In reply to Quan Wenli from comment #5)
> (In reply to Stefano Brivio from comment #4)
> > (In reply to Quan Wenli from comment #3)
> > > 
> > > I have uploaded the dns_search.pcap. please check it.
> > 
> > Thanks! There, we see:
> > 
> > - a DHCP ACK message in frame 4, which doesn't contain any DNS domain search
> > information (that would be option 119, missing here)
> > 
> > - a Router Advertisement in frame 33, which also doesn't contain any DNS
> > domain search information (that would be option type 31, missing)
> > 
> > - a DHCPv6 reply in frame 43, which doesn't contain DNS domain search
> > information either (that would be option 24, missing)
> > 
> > So I don't see any way passt is sharing any DNS domain search list with the
> > guest.
> 
> But there could see "DNS search list" in passt output in comment #0, is it
> corret?

Yes, it's expected: the DNS search list from the host is printed in any case, but it won't be sent over DHCP, NDP and DHCPv6 if you give --no-dhcp-search.

> > Could it be that NetworkManager is filling that in by means of static
> > configuration, or stored configuration from previous attempts? It might be
> > worth to check for any such information persistently stored under
> /var/lib/NetworkManager/.
> 
> Even stop the NetworkManager service, still could see the dhs search list. 
> [root@dell-per440-18 NetworkManager]# sed 's/\. / /g' /etc/resolv.conf | sed
> 's/\.$//g' | sed -n 's/^search \(.*\)/\1/p' | tr ' \n' ',' | sed
> 's/\(.*\)/\1\n/g'
> lab.eng.pek2.redhat.com,

So it's probably not related to DHCP or DHCPv6, otherwise I would expect NetworkManager to remove the entry when the service is stopped. The only remaining possibilities I see are a stale value received via NDP (option type 31) or a pre-configured value. Two ideas:

- is rdnssd installed on the system? If yes, is it running?

- could you print /etc/resolv.conf from the image while the guest is not running? For example:

  virt-cat -a rhel900-64-virtio.qcow2 /etc/resolv.conf

Comment 14 Lei Yang 2023-02-23 05:45:54 UTC
Hello Stefano

After QE tested, both passt-0.git.2022_06_08.d7d467f-0.el9.x86_64 and latest version: passt-0^20230222.g4ddbcb9-1.el9,x86_64 are all reproduced the Comment 0 problem. So could you please help to double confirm if the current bug got a fix or this is not a bug. Thanks in advance.


Best Regards
Lei

Comment 15 Laurent Vivier 2023-02-23 08:32:53 UTC
(In reply to Lei Yang from comment #14)
> Hello Stefano
> 
> After QE tested, both passt-0.git.2022_06_08.d7d467f-0.el9.x86_64 and latest
> version: passt-0^20230222.g4ddbcb9-1.el9,x86_64 are all reproduced the
> Comment 0 problem. So could you please help to double confirm if the current
> bug got a fix or this is not a bug. Thanks in advance.
> 

Hi Lei,

I'm not able to reproduce the problem. With --no-dhcp-search, I have "search .".

But my guest is a Fedora 36 and resolv.conf is managed by systemd, not NetworkManager as yours.

Perhaps the problem comes from the guest OS that keeps a cached value somewhere?

Comment 16 Lei Yang 2023-02-23 11:28:47 UTC
(In reply to Laurent Vivier from comment #15)
> (In reply to Lei Yang from comment #14)
> > Hello Stefano
> > 
> > After QE tested, both passt-0.git.2022_06_08.d7d467f-0.el9.x86_64 and latest
> > version: passt-0^20230222.g4ddbcb9-1.el9,x86_64 are all reproduced the
> > Comment 0 problem. So could you please help to double confirm if the current
> > bug got a fix or this is not a bug. Thanks in advance.
> > 
> 
> Hi Lei,
> 
> I'm not able to reproduce the problem. With --no-dhcp-search, I have "search
> .".
> 
> But my guest is a Fedora 36 and resolv.conf is managed by systemd, not
> NetworkManager as yours.
> 
Hello Stefano

> Perhaps the problem comes from the guest OS that keeps a cached value
> somewhere?

I can not sure this issues root cause. But I tested this bug on Fedora 37 and it couldn't reproduce it. So this bug only hit on the rhel guest,therefore how should we deal with the current bug? Based on fedora test results to close, or based on rhel results to keep the current state to fix it in the future.

Thanks
Lei

Comment 17 Stefano Brivio 2023-02-23 15:55:36 UTC
Lei, sorry for the delay, I wanted to look into normative references on the subject (RFCs) and then I forgot about this.

One thing we know from the captures Wenli shared is that, if --no-dhcp-search is given, DHCP option 119, router advertisement option type 31, and DHCPv6 option 24 are all missing from the captures (see also comment #4). So, as expected, passt is not sending DNS search options with --no-dhcp-search.

At this point, we need to check what NetworkManager or systemd-resolve might do on start, stop and reboot. But there's no RFC defining the lifetime of options. RFC 2131 indicates the T1 and T2 times only as validity of the IP address lease, and RFC 2132 doesn't define a general "expiration" of options. The same applies to NDP and DHCPv6.

So we don't have an indication telling us that it's right or wrong to keep the DNS domain search across reboots, and different implementations can even behave differently.

For this reason, I would close this as NOTABUG. As long as passt isn't *setting* the DNS domain search, I don't think we actually have an issue. What do you think?

Comment 18 Lei Yang 2023-02-23 23:43:22 UTC
(In reply to Stefano Brivio from comment #17)
> Lei, sorry for the delay, I wanted to look into normative references on the
> subject (RFCs) and then I forgot about this.
> 
> One thing we know from the captures Wenli shared is that, if
> --no-dhcp-search is given, DHCP option 119, router advertisement option type
> 31, and DHCPv6 option 24 are all missing from the captures (see also comment
> #4). So, as expected, passt is not sending DNS search options with
> --no-dhcp-search.
> 
> At this point, we need to check what NetworkManager or systemd-resolve might
> do on start, stop and reboot. But there's no RFC defining the lifetime of
> options. RFC 2131 indicates the T1 and T2 times only as validity of the IP
> address lease, and RFC 2132 doesn't define a general "expiration" of
> options. The same applies to NDP and DHCPv6.
> 
> So we don't have an indication telling us that it's right or wrong to keep
> the DNS domain search across reboots, and different implementations can even
> behave differently.
> 
Hello Stefano

> For this reason, I would close this as NOTABUG. As long as passt isn't
> *setting* the DNS domain search, I don't think we actually have an issue.
> What do you think?

Thanks for you help explain clearly. Agree with your point, move to "NOTABUG".


Note You need to log in before you can comment on or make changes to this bug.