Bug 1371004

Summary: atomic-openshift-node service wasn't reload on containerized env.
Product: OpenShift Container Platform Reporter: Anping Li <anli>
Component: Cluster Version OperatorAssignee: Devan Goodwin <dgoodwin>
Status: CLOSED ERRATA QA Contact: Anping Li <anli>
Severity: high Docs Contact:
Priority: high    
Version: 3.3.0CC: aos-bugs, bleanhar, jokerman, mmccomas
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1371708 (view as bug list) Environment:
Last Closed: 2016-09-27 09:46:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1371708    

Description Anping Li 2016-08-29 06:57:14 UTC
Description of problem:
During upgrade, the '-v /sys:/sys:ro' was changed to ' -v /sys:/sys:rw' in atomic-openshift-node.service. But the change doesn't take affect for systemctl daemon-reload missing.


Version-Release number of selected component (if applicable):
atomic-openshift-utils-3.3.16-1.git.0.9ed3fd2.el7.noarch

How reproducible:
always

Steps to Reproduce:
1. install containerized OSE 3.2 
2. upgrade to OCP 3.3
3. Check the node service
4. Check the process



Actual results:
3. [root@ip-172-18-6-19 ~]# systemctl status atomic-openshift-node
● atomic-openshift-node.service
   Loaded: loaded (/etc/systemd/system/atomic-openshift-node.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2016-08-29 02:00:45 EDT; 35min ago
  Process: 11181 ExecStop=/usr/bin/docker stop atomic-openshift-node (code=exited, status=0/SUCCESS)
  Process: 11405 ExecStartPost=/usr/bin/sleep 10 (code=exited, status=0/SUCCESS)
  Process: 11399 ExecStartPre=/usr/bin/docker rm -f atomic-openshift-node (code=exited, status=1/FAILURE)
 Main PID: 11404 (docker-current)
   Memory: 9.0M
   CGroup: /system.slice/atomic-openshift-node.service
           └─11404 /usr/bin/docker-current run --name atomic-openshift-node --rm --privileged --net=host --pid=host --env-file=/etc/sysconfig/atomic-openshift-node -v /:/rootfs:ro -e CONFIG_FILE=/etc/origin/n...

Aug 29 02:35:54 ip-172-18-6-19.ec2.internal atomic-openshift-node[11404]: I0829 02:35:54.881403   11452 iptables.go:380] running iptables -C [INPUT -t filter -p udp -m multiport --dports 4789 -m co... -j ACCEPT]
Aug 29 02:35:54 ip-172-18-6-19.ec2.internal atomic-openshift-node[11404]: I0829 02:35:54.885507   11452 iptables.go:380] running iptables -C [INPUT -t filter -i tun0 -m comment --comment traffic fr... -j ACCEPT]
Aug 29 02:35:54 ip-172-18-6-19.ec2.internal atomic-openshift-node[11404]: I0829 02:35:54.887370   11452 iptables.go:380] running iptables -C [FORWARD -t filter -d 10.2.0.0/16 -j ACCEPT]
Aug 29 02:35:54 ip-172-18-6-19.ec2.internal atomic-openshift-node[11404]: I0829 02:35:54.889711   11452 iptables.go:380] running iptables -C [FORWARD -t filter -s 10.2.0.0/16 -j ACCEPT]
Aug 29 02:35:54 ip-172-18-6-19.ec2.internal atomic-openshift-node[11404]: I0829 02:35:54.892443   11452 node_iptables.go:77] syncIPTableRules took 16.093266ms
Aug 29 02:35:55 ip-172-18-6-19.ec2.internal atomic-openshift-node[11404]: I0829 02:35:55.348931   11452 generic.go:181] GenericPLEG: Relisting
Aug 29 02:35:55 ip-172-18-6-19.ec2.internal atomic-openshift-node[11404]: I0829 02:35:55.352589   11452 docker.go:394] Docker Container: /atomic-openshift-master is not managed by kubelet.
Aug 29 02:35:55 ip-172-18-6-19.ec2.internal atomic-openshift-node[11404]: I0829 02:35:55.352603   11452 docker.go:394] Docker Container: /atomic-openshift-node is not managed by kubelet.
Aug 29 02:35:55 ip-172-18-6-19.ec2.internal atomic-openshift-node[11404]: I0829 02:35:55.352608   11452 docker.go:394] Docker Container: /openvswitch is not managed by kubelet.
Aug 29 02:35:55 ip-172-18-6-19.ec2.internal atomic-openshift-node[11404]: I0829 02:35:55.611693   11452 kubelet.go:2607] SyncLoop (housekeeping)
Warning: atomic-openshift-node.service changed on disk. Run 'systemctl daemon-reload' to reload units.

4. [root@ip-172-18-6-19 ~]# ps -ef|grep atomic-openshift-node
root      2428  2374  0 02:55 pts/0    00:00:00 grep --color=auto atomic-openshift-node
root     11404     1  0 02:00 ?        00:00:01 /usr/bin/docker-current run --name atomic-openshift-node --rm --privileged --net=host --pid=host --env-file=/etc/sysconfig/atomic-openshift-node -v /:/rootfs:ro -e CONFIG_FILE=/etc/origin/node/node-config.yaml -e OPTIONS=--loglevel=5 -e HOST=/rootfs -e HOST_ETC=/host-etc -v /var/lib/origin:/var/lib/origin:rslave -v /etc/origin/node:/etc/origin/node -v /etc/origin/cloudprovider:/etc/origin/cloudprovider -v /etc/localtime:/etc/localtime:ro -v /etc/machine-id:/etc/machine-id:ro -v /run:/run -v /sys:/sys:ro -v /usr/bin/docker:/usr/bin/docker:ro -v /var/lib/docker:/var/lib/docker -v /lib/modules:/lib/modules -v /etc/origin/openvswitch:/etc/openvswitch -v /etc/origin/sdn:/etc/openshift-sdn -v /etc/systemd/system:/host-etc/systemd/system -v /var/log:/var/log -v /dev:/dev --volume=/usr/bin/docker-current:/usr/bin/docker-current:ro --volume=/etc/sysconfig/docker:/etc/sysconfig/docker:ro openshift3/node:v3.3.0.26

Expected results:
'-v /sys:/sys:rw' take affect.

Additional info:

Comment 3 Anping Li 2016-08-30 04:31:43 UTC
Verified and pass.

[root@anli-working data]# diff pre_migrate_1472527608/host6node.example.com/etc/systemd/system/atomic-openshift-node.service post_migrate_1472527608/host6node.example.com/etc/systemd/system/atomic-openshift-node.service 
16c16
< ExecStart=/usr/bin/docker run --name atomic-openshift-node --rm --privileged --net=host --pid=host --env-file=/etc/sysconfig/atomic-openshift-node -v /:/rootfs:ro -e CONFIG_FILE=${CONFIG_FILE} -e OPTIONS=${OPTIONS} -e HOST=/rootfs -e HOST_ETC=/host-etc -v /var/lib/origin:/var/lib/origin -v /etc/origin/node:/etc/origin/node -v /etc/localtime:/etc/localtime:ro -v /etc/machine-id:/etc/machine-id:ro -v /run:/run -v /sys:/sys:ro -v /usr/bin/docker:/usr/bin/docker:ro -v /var/lib/docker:/var/lib/docker -v /lib/modules:/lib/modules -v /etc/origin/openvswitch:/etc/openvswitch -v /etc/origin/sdn:/etc/openshift-sdn -v /etc/systemd/system:/host-etc/systemd/system -v /var/log:/var/log -v /dev:/dev $DOCKER_ADDTL_BIND_MOUNTS openshift3/node:${IMAGE_VERSION}
---
> ExecStart=/usr/bin/docker run --name atomic-openshift-node --rm --privileged --net=host --pid=host --env-file=/etc/sysconfig/atomic-openshift-node -v /:/rootfs:ro -e CONFIG_FILE=${CONFIG_FILE} -e OPTIONS=${OPTIONS} -e HOST=/rootfs -e HOST_ETC=/host-etc -v /var/lib/origin:/var/lib/origin -v /etc/origin/node:/etc/origin/node -v /etc/localtime:/etc/localtime:ro -v /etc/machine-id:/etc/machine-id:ro -v /run:/run -v /sys:/sys:rw -v /usr/bin/docker:/usr/bin/docker:ro -v /var/lib/docker:/var/lib/docker -v /lib/modules:/lib/modules -v /etc/origin/openvswitch:/etc/openvswitch -v /etc/origin/sdn:/etc/openshift-sdn -v /etc/systemd/system:/host-etc/systemd/system -v /var/log:/var/log -v /dev:/dev $DOCKER_ADDTL_BIND_MOUNTS openshift3/node:${IMAGE_VERSION}

Comment 5 errata-xmlrpc 2016-09-27 09:46:37 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1933