Bug 1392815

Summary: [ocp-on-osp] /etc/resolv.conf was reset after rebooting the instances on RHEL-7.3
Product: OpenShift Container Platform Reporter: Gan Huang <ghuang>
Component: Reference ArchitectureAssignee: scollier
Status: CLOSED WONTFIX QA Contact: Gan Huang <ghuang>
Severity: high Docs Contact:
Priority: high    
Version: 3.4.0CC: aos-bugs, jokerman, mmccomas, tsedovic
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-02-22 14:44:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Gan Huang 2016-11-08 09:50:33 UTC
Description of problem:
Creating a heat stack which specify "volume_quota: 3" in env file with RHEL-7.3 hosts, that option would result in rebooting the instances during creating the stack. Then the stack would failed due to the dnsname server of those instances were missing because of the reboot.

Version-Release number of selected component (if applicable):
openshift-on-openstack-0.9.5-1.el7.centos.noarch

How reproducible:
always

Steps to Reproduce:
1. Specify "volume_quota: 3" and RHEL-7.3 image in env file
2. Creat a stack
3.

Actual results:
Stack failed when running the ansible playbook:

Ansible log on bastion host:
    "msg": "http://xxx.redhat.com/rhel/server/7/7Server/x86_64/os/rhel-7-server-rpms/Packages/iptables-services-1.4.21-17.el7.x86_64.rpm: [Errno 14] curl#6 - \"Could not resolve host: xxx.redhat.com; Unknown error\"\nTrying other mirror.\n\n\nError downloading packages:\n  iptables-services-1.4.21-17.el7.x86_64: [Errno 256] No more mirrors to try.\n\n", 
    "rc": 1, 
    "results": [
        "iptables-1.4.21-17.el7.x86_64 providing iptables is already installed", 
        "Loaded plugins: search-disabled-repos\nResolving Dependencies\n--> Running transaction check\n---> Package iptables-services.x86_64 0:1.4.21-17.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package                  Arch          Version              Repository    Size\n================================================================================\nInstalling:\n iptables-services        x86_64        1.4.21-17.el7        rhel7         50 k\n\nTransaction Summary\n================================================================================\nInstall  1 Package\n\nTotal download size: 50 k\nInstalled size: 24 k\nDownloading packages:\n"
    ]
}

NO MORE HOSTS LEFT *************************************************************
	to retry, use: --limit @/var/lib/ansible/playbooks/main.retry

PLAY RECAP *********************************************************************
flannel-73-openshift-infra-0.example.com : ok=61   changed=7    unreachable=0    failed=0   
flannel-73-openshift-master-0.example.com : ok=74   changed=10   unreachable=0    failed=1   
flannel-73-openshift-node-h7kt27x6.example.com : ok=61   changed=7    unreachable=0    failed=0   
localhost                  : ok=21   changed=15   unreachable=0    failed=0   

Check on flannel-73-openshift-master-0.example.com, nameservers were missing.
$ cat /etc/resolv.conf 
# Generated by NetworkManager
search example.com


Expected results:


Additional info:

Comment 1 Gan Huang 2016-11-09 03:26:36 UTC
"skip_dns: true" also can reproduce this issue.


"skip_dns: true" will install and config dnsmqsq in OpenShift cluster, thus it will trigger to restart NetworkManager during installing OpenShift, and /etc/resolv.conf would be reset as well.

Comment 2 Jan Provaznik 2016-11-14 08:48:26 UTC
this might be related to docker version:

docker-1.10.3-46.el7.10.x86_64 (rhel72 - works)
root@test-ocp34-master-0: ~ # nmcli con show docker0|grep -i dns
ipv4.dns:                              
ipv4.dns-search:                        
ipv4.ignore-auto-dns:                   no
ipv6.dns:                              
ipv6.dns-search:                        
ipv6.ignore-auto-dns:                   no
 
 
docker-selinux-1.10.3-57.el7.x86_64 (rhel73 - networkmanager overwrites resolv.conf)
[root@bug-openshift-node-15ok67md etc]#  nmcli con show 5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03|grep -i dns
ipv4.dns:                              
ipv4.dns-search:                        
ipv4.dns-options:                       (default)
ipv4.dns-priority:                      0
ipv4.ignore-auto-dns:                   yes
ipv6.dns:                              
ipv6.dns-search:                        
ipv6.dns-options:                       (default)
ipv6.dns-priority:                      0
ipv6.ignore-auto-dns:                   no

Comment 3 Jan Provaznik 2016-11-14 09:20:44 UTC
It's unrelated to docker interface. We set "PEERDNS=no" for all interfaces but it seems that for new NetworkManager its meaning is ""never add automatic (DHCP, PPP, VPN, etc.) nameservers to resolv.conf" instead of the expected "never touch resolv.conf".

Putting "dns=none" into [main] section of /etc/NetworkManager/NetworkManager.conf has desired effect.

Comment 5 Gan Huang 2016-11-22 07:53:35 UTC
This is related to NetworkManager indeed.

#RHEL-7.2

/etc/resolv.conf won't be overided in below version:
NetworkManager-1.0.6-27.el7.x86_64
NetworkManager-libnm-1.0.6-27.el7.x86_64
NetworkManager-team-1.0.6-27.el7.x86_64
NetworkManager-config-server-1.0.6-27.el7.x86_64
NetworkManager-tui-1.0.6-27.el7.x86_64

/etc/resolv.conf would be overided in below version:
NetworkManager-tui-1.0.6-31.el7_2.x86_64
NetworkManager-1.0.6-31.el7_2.x86_64
NetworkManager-team-1.0.6-31.el7_2.x86_64
NetworkManager-config-server-1.0.6-31.el7_2.x86_64
NetworkManager-libnm-1.0.6-31.el7_2.x86_64


#RHEL-7.3

/etc/resolv.conf would be overided in RHEL-7.3
NetworkManager-1.4.0-12.el7.x86_64
NetworkManager-team-1.4.0-12.el7.x86_64
NetworkManager-libnm-1.4.0-12.el7.x86_64
NetworkManager-tui-1.4.0-12.el7.x86_64
NetworkManager-config-server-1.4.0-12.el7.x86_64

Comment 6 scollier 2018-02-22 14:44:37 UTC
Team,  Closing this as the heat templates have been deprecated.  Future funcionality and integration capabilities will be moving to openshift-ansible moving forward.