Bug 1253624

Summary: After fencing, controller gets empty configuration files
Product: Red Hat OpenStack Reporter: Pablo Iranzo Gómez <pablo.iranzo>
Component: rhel-osp-installerAssignee: Mike Burns <mburns>
Status: CLOSED WORKSFORME QA Contact: Omri Hochman <ohochman>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.0 (Juno)CC: amedeo.salvati, amuller, chrisw, jcoufal, kbasil, lpeer, mburns, nyechiel, pablo.iranzo, rhos-maint, sknauss, srevivo
Target Milestone: z5Keywords: ZStream
Target Release: Installer   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-09-27 12:34:09 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1273812    

Description Pablo Iranzo Gómez 2015-08-14 09:38:23 UTC 
We've simulated a kernel panic on one controller node --> echo c > /proc/sysrq-trigger, but after that, when this node go up all neutron configuration files are empty, e.g.:

root@host:/etc/neutron$ cat /etc/neutron/neutron.conf 
[DEFAULT]
host = neutron-n-1
root@host:/etc/neutron$


and so, for this reason neutron on that node won't be up (for example because he can't login to rabbitmq server).

Please, could you verify for what reason configuration files are wiped on node after kernel panic|fencing?
Comment 3 Pablo Iranzo Gómez 2015-08-14 09:39:05 UTC 
From support case:

I have reproduced the issue once by doing

1) 3 node HA setup
2) poweroff/poweron one controller node.
3) when system came up again most of the neutron configs has been changed by crudini to :

# cat dhcp_agent.ini fwaas_driver.ini l3_agent.ini lbaas_agent.ini metadata_agent.ini neutron.conf 
[DEFAULT]
host = neutron-n-2
[DEFAULT]
host = neutron-n-2
[DEFAULT]
host = neutron-n-2
[DEFAULT]
host = neutron-n-2
[DEFAULT]
host = neutron-n-2
[DEFAULT]
host = neutron-n-2

Before restart I added auditd rules to monitor crudini and neutron.conf:

type=SYSCALL msg=audit(1439366108.113:8046): arch=c000003e syscall=59 success=yes exit=0 a0=cbb240 a1=cbadc0 a2=cbb260 a3=7ffce33934b0 items=3 ppid=6350 pid=6354 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="crudini" exe="/u
sr/bin/python2.7" subj=system_u:system_r:cluster_t:s0 key="crudinit-watcher"
type=EXECVE msg=audit(1439366108.113:8046): argc=6 a0="/usr/bin/python" a1="/usr/bin/crudini" a2="--del" a3="/etc/neutron/dhcp_agent.ini" a4="DEFAULT" a5="host"
type=CWD msg=audit(1439366108.113:8046):  cwd="/var/lib/pacemaker/cores"
type=PATH msg=audit(1439366108.113:8046): item=0 name="/usr/bin/crudini" inode=35389945 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:bin_t:s0 objtype=NORMAL
type=PATH msg=audit(1439366108.113:8046): item=1 name="/usr/bin/python" inode=33671557 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:bin_t:s0 objtype=NORMAL
type=PATH msg=audit(1439366108.113:8046): item=2 name="/lib64/ld-linux-x86-64.so.2" inode=67160115 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:ld_so_t:s0 objtype=NORMAL
type=SYSCALL msg=audit(1439366108.169:8047): arch=c000003e syscall=59 success=yes exit=0 a0=cac2b0 a1=cab4c0 a2=cac2d0 a3=7fff9f8d05b0 items=3 ppid=6350 pid=6361 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="crudini" exe="/u
sr/bin/python2.7" subj=system_u:system_r:cluster_t:s0 key="crudinit-watcher"
type=EXECVE msg=audit(1439366108.169:8047): argc=6 a0="/usr/bin/python" a1="/usr/bin/crudini" a2="--del" a3="/etc/neutron/fwaas_driver.ini" a4="DEFAULT" a5="host"
type=CWD msg=audit(1439366108.169:8047):  cwd="/var/lib/pacemaker/cores"
type=PATH msg=audit(1439366108.169:8047): item=0 name="/usr/bin/crudini" inode=35389945 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:bin_t:s0 objtype=NORMAL
type=PATH msg=audit(1439366108.169:8047): item=1 name="/usr/bin/python" inode=33671557 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:bin_t:s0 objtype=NORMAL
type=PATH msg=audit(1439366108.169:8047): item=2 name="/lib64/ld-linux-x86-64.so.2" inode=67160115 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:ld_so_t:s0 objtype=NORMAL
type=SYSCALL msg=audit(1439366108.219:8048): arch=c000003e syscall=59 success=yes exit=0 a0=1f751d0 a1=1f74d50 a2=1f751f0 a3=7fff6e8cc130 items=3 ppid=6350 pid=6368 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="crudini" exe=
"/usr/bin/python2.7" subj=system_u:system_r:cluster_t:s0 key="crudinit-watcher"
type=EXECVE msg=audit(1439366108.219:8048): argc=6 a0="/usr/bin/python" a1="/usr/bin/crudini" a2="--del" a3="/etc/neutron/l3_agent.ini" a4="DEFAULT" a5="host"
type=CWD msg=audit(1439366108.219:8048):  cwd="/var/lib/pacemaker/cores"
type=PATH msg=audit(1439366108.219:8048): item=0 name="/usr/bin/crudini" inode=35389945 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:bin_t:s0 objtype=NORMAL
type=PATH msg=audit(1439366108.219:8048): item=1 name="/usr/bin/python" inode=33671557 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:bin_t:s0 objtype=NORMAL
type=PATH msg=audit(1439366108.219:8048): item=2 name="/lib64/ld-linux-x86-64.so.2" inode=67160115 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:ld_so_t:s0 objtype=NORMAL
type=SYSCALL msg=audit(1439366108.272:8049): arch=c000003e syscall=59 success=yes exit=0 a0=c732b0 a1=c724c0 a2=c732d0 a3=7fff67b04a90 items=3 ppid=6350 pid=6375 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="crudini" exe="/u
sr/bin/python2.7" subj=system_u:system_r:cluster_t:s0 key="crudinit-watcher"
type=EXECVE msg=audit(1439366108.272:8049): argc=6 a0="/usr/bin/python" a1="/usr/bin/crudini" a2="--del" a3="/etc/neutron/lbaas_agent.ini" a4="DEFAULT" a5="host"
type=CWD msg=audit(1439366108.272:8049):  cwd="/var/lib/pacemaker/cores"
type=PATH msg=audit(1439366108.272:8049): item=0 name="/usr/bin/crudini" inode=35389945 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:bin_t:s0 objtype=NORMAL
type=PATH msg=audit(1439366108.272:8049): item=1 name="/usr/bin/python" inode=33671557 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:bin_t:s0 objtype=NORMAL
type=PATH msg=audit(1439366108.272:8049): item=2 name="/lib64/ld-linux-x86-64.so.2" inode=67160115 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:ld_so_t:s0 objtype=NORMAL
type=SYSCALL msg=audit(1439366108.324:8050): arch=c000003e syscall=59 success=yes exit=0 a0=10f52b0 a1=10f44c0 a2=10f52d0 a3=7ffe229e7e90 items=3 ppid=6350 pid=6382 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="crudini" exe=
"/usr/bin/python2.7" subj=system_u:system_r:cluster_t:s0 key="crudinit-watcher"
type=EXECVE msg=audit(1439366108.324:8050): argc=6 a0="/usr/bin/python" a1="/usr/bin/crudini" a2="--del" a3="/etc/neutron/metadata_agent.ini" a4="DEFAULT" a5="host"
type=CWD msg=audit(1439366108.324:8050):  cwd="/var/lib/pacemaker/cores"
type=PATH msg=audit(1439366108.324:8050): item=0 name="/usr/bin/crudini" inode=35389945 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:bin_t:s0 objtype=NORMAL
type=PATH msg=audit(1439366108.324:8050): item=1 name="/usr/bin/python" inode=33671557 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:bin_t:s0 objtype=NORMAL
type=PATH msg=audit(1439366108.324:8050): item=2 name="/lib64/ld-linux-x86-64.so.2" inode=67160115 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:ld_so_t:s0 objtype=NORMAL
type=SYSCALL msg=audit(1439366108.384:8051): arch=c000003e syscall=59 success=yes exit=0 a0=223d1d0 a1=223cd50 a2=223d1f0 a3=7ffe209be2f0 items=3 ppid=6350 pid=6389 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="crudini" exe=
"/usr/bin/python2.7" subj=system_u:system_r:cluster_t:s0 key="crudinit-watcher"
type=EXECVE msg=audit(1439366108.384:8051): argc=6 a0="/usr/bin/python" a1="/usr/bin/crudini" a2="--del" a3="/etc/neutron/neutron.conf" a4="DEFAULT" a5="host"
type=CWD msg=audit(1439366108.384:8051):  cwd="/var/lib/pacemaker/cores"
type=PATH msg=audit(1439366108.384:8051): item=0 name="/usr/bin/crudini" inode=35389945 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:bin_t:s0 objtype=NORMAL
type=PATH msg=audit(1439366108.384:8051): item=1 name="/usr/bin/python" inode=33671557 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:bin_t:s0 objtype=NORMAL
type=PATH msg=audit(1439366108.384:8051): item=2 name="/lib64/ld-linux-x86-64.so.2" inode=67160115 dev=fd:00 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:ld_so_t:s0 objtype=NORMAL
type=SYSCALL msg=audit(1439366108.449:8052): arch=c000003e syscall=59 success=yes exit=0 a0=25881b0 a1=2587eb0 a2=25881d0 a3=7ffee164fdb0 items=3 ppid=6350 pid=6396 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="crudini" exe=
"/usr/bin/python2.7" subj=system_u:system_r:cluster_t:s0 key="crudinit-watcher"
type=EXECVE msg=audit(1439366108.449:8052): argc=6 a0="/usr/bin/python" a1="/usr/bin/crudini" a2="--del" a3="/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini" a4="DEFAULT" a5="host"
type=CWD msg=audit(1439366108.449:8052):  cwd="/var/lib/pacemaker/cores"


Since I had not disabled puppet :( most of the configs got fixed:

# ll
total 40
-rw-r--r--. 1 root root     184 Jul 24 18:47 api-paste.ini
-rw-r-----. 1 root neutron  370 Aug 12 10:29 dhcp_agent.ini
-rw-r-----. 1 root neutron   29 Aug 12 09:59 fwaas_driver.ini
-rw-r-----. 1 root neutron  382 Aug 12 10:29 l3_agent.ini
-rw-r-----. 1 root neutron   29 Aug 12 09:59 lbaas_agent.ini
-rw-r-----. 1 root neutron  385 Aug 12 10:29 metadata_agent.ini
-rw-r-----. 1 root neutron 1840 Aug 12 10:32 neutron.conf
Comment 5 Assaf Muller 2015-08-17 12:33:50 UTC 
Moved component to OSP installer, if configuration files are empty this isn't a Neutron issue.
Comment 6 Pablo Iranzo Gómez 2015-08-17 14:10:40 UTC 
Assaf, Mike:

what does the installer has to do here if this happens after a node has been fenced?

Thanks,
Pablo
Comment 8 Assaf Muller 2015-08-17 14:33:54 UTC 
Either a pacemaker issue, or an issue with how the installer configures pacemaker, but not a Neutron issue so that's why I moved the component.
Comment 9 Pablo Iranzo Gómez 2015-09-04 09:57:59 UTC 
Hi Mike,

Any update on #6 ?

Customer asked today about ETA for this during todayś TAM call

Thanks,
Pablo