Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1475505

Summary: undercloud+ssl keepalived fails because of selinux: read error on modprobe
Product: Red Hat OpenStack Reporter: Gonéri Le Bouder <goneri>
Component: openstack-tripleo-heat-templatesAssignee: Emilien Macchi <emacchi>
Status: CLOSED DUPLICATE QA Contact: Gurenko Alex <agurenko>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 10.0 (Newton)CC: bperkins, dsavinea, mburns, rhel-osp-director-maint
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1475888 (view as bug list) Environment:
Last Closed: 2017-07-27 18:22:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1475888    
Attachments:
Description Flags
keepalived.conf
none
lsmod
none
undercloud.conf none

Description Gonéri Le Bouder 2017-07-26 19:46:42 UTC
Created attachment 1305015 [details]
keepalived.conf

Description of problem:


Note: I affect the bug to openstack-tripleo-heat-templates because this is what has been done for a similar issue (BZ1470209).

Version-Release number of selected component (if applicable):

puppet-keepalived-0.0.1-0.11.bbca37agit.el7ost.noarch
keepalived-1.3.5-1.el7.x86_64
openstack-tripleo-image-elements-5.2.0-2.el7ost.noarch
puppet-tripleo-5.6.0-4.el7ost.noarch
python-tripleoclient-5.4.2-1.el7ost.noarch
openstack-tripleo-0.0.8-0.2.4de13b3git.el7ost.noarch
openstack-tripleo-puppet-elements-5.3.0-1.el7ost.noarch
openstack-tripleo-ui-1.2.0-1.el7ost.noarch
openstack-tripleo-validations-5.1.1-1.el7ost.noarch
openstack-tripleo-common-5.4.2-2.el7ost.noarch
openstack-tripleo-heat-templates-5.2.0-25.el7ost.noarch

How reproducible:

Deploy an undercloud with the undercloud.conf attached to this bug.

Actual results:

/var/log/audit/audit.log is flooded with that, this at =~ 30 new line per second:
type=PROCTITLE msg=audit(1501098005.407:171803): proctitle=2F7573722F7362696E2F6B656570616C69766564002D44
type=ANOM_ABEND msg=audit(1501098005.407:171804): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:keepalived_t:s0 pid=27316 comm="keepalived" reason="memory violation" sig=11
type=AVC msg=audit(1501098005.433:171805): avc:  denied  { read } for  pid=27317 comm="keepalived" name="modprobe" dev="proc" ino=19572 scontext=system_u:system_r:keepalived_t:s0 tcontext=system_u:object_r:usermodehelper_t:s0 tclass=file
type=SYSCALL msg=audit(1501098005.433:171805): arch=c000003e syscall=2 success=no exit=-13 a0=7f5f845ccb77 a1=0 a2=1 a3=7ffc1e1ed710 items=0 ppid=23507 pid=27317 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="keepalived" exe="/usr/sbin/keepalived" subj=system_u:system_r:keepalived_t:s0 key=(null)
type=PROCTITLE msg=audit(1501098005.433:171805): proctitle=2F7573722F7362696E2F6B656570616C69766564002D44
type=ANOM_ABEND msg=audit(1501098005.433:171806): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:keepalived_t:s0 pid=27317 comm="keepalived" reason="memory violation" sig=11
type=AVC msg=audit(1501098005.461:171807): avc:  denied  { read } for  pid=27318 comm="keepalived" name="modprobe" dev="proc" ino=19572 scontext=system_u:system_r:keepalived_t:s0 tcontext=system_u:object_r:usermodehelper_t:s0 tclass=file
type=SYSCALL msg=audit(1501098005.461:171807): arch=c000003e syscall=2 success=no exit=-13 a0=7f5f845ccb77 a1=0 a2=1 a3=7ffc1e1ed710 items=0 ppid=23507 pid=27318 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="keepalived" exe="/usr/sbin/keepalived" subj=system_u:system_r:keepalived_t:s0 key=(null)

"journalctl -u keepalived" returns this error (30 lines per second):

Jul 26 15:41:11 directorvm.rpc.rackspace.com Keepalived[23507]: VRRP child process(30677) died: Respawning
Jul 26 15:41:11 directorvm.rpc.rackspace.com Keepalived[23507]: VRRP child process(30678) died: Respawning
Jul 26 15:41:11 directorvm.rpc.rackspace.com Keepalived[23507]: VRRP child process(30679) died: Respawning
Jul 26 15:41:11 directorvm.rpc.rackspace.com Keepalived[23507]: VRRP child process(30680) died: Respawning
Jul 26 15:41:11 directorvm.rpc.rackspace.com Keepalived[23507]: VRRP child process(30681) died: Respawning
Jul 26 15:41:11 directorvm.rpc.rackspace.com Keepalived[23507]: VRRP child process(30682) died: Respawning
Jul 26 15:41:11 directorvm.rpc.rackspace.com Keepalived[23507]: VRRP child process(30683) died: Respawning
Jul 26 15:41:11 directorvm.rpc.rackspace.com Keepalived[23507]: VRRP child process(30684) died: Respawning
Jul 26 15:41:11 directorvm.rpc.rackspace.com Keepalived[23507]: VRRP child process(30685) died: Respawning
Jul 26 15:41:11 directorvm.rpc.rackspace.com Keepalived[23507]: VRRP child process(30686) died: Respawning
Jul 26 15:41:11 directorvm.rpc.rackspace.com Keepalived[23507]: VRRP child process(30688) died: Respawning
Jul 26 15:41:11 directorvm.rpc.rackspace.com Keepalived[23507]: VRRP child process(30690) died: Respawning


The VIP are not mounted. Last time they were but unreachable.

Expected results:

VIP should be available. Log should but quiet.


Additional info:

Comment 1 Gonéri Le Bouder 2017-07-26 19:47:07 UTC
Created attachment 1305016 [details]
lsmod

Comment 2 Gonéri Le Bouder 2017-07-26 19:47:25 UTC
Created attachment 1305017 [details]
undercloud.conf

Comment 3 Gonéri Le Bouder 2017-07-26 20:03:08 UTC
The undercloud deployment succeed all the time. The first issue happens when we try to add the ironic nodes, the nodes are added but it after tries to reach zaqar on port 9000 and return an error:

openstack baremetal import --json /home/stack/instackenv.json
Handshake status 502

Comment 5 Gonéri Le Bouder 2017-07-26 22:48:38 UTC
I can avoid the problem if I reboot without selinux and lsmod returns a different list of modules. I will try to identify tomorrow which module is the culprit.
For the record, the configuration was working fine 6 months ago.

Comment 6 Gonéri Le Bouder 2017-07-27 14:03:20 UTC
(gdb) set follow-fork-mode child
(gdb) c
Continuing.
[New process 3717]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f84892ec840 (LWP 3717)]
0x0000000000000000 in ?? ()
(gdb) bt ful
#0  0x0000000000000000 in ?? ()
No symbol table info available.
#1  0x000055997297f096 in remove_ipsets () at vrrp_ipset.c:244
        session = <optimized out>
#2  0x000055997296fd85 in vrrp_complete_init () at vrrp.c:2458
        l = 0x55997304ab40
        ol = <optimized out>
        sl = <optimized out>
        e = 0x0
        oe = <optimized out>
        se = <optimized out>
        vrrp = <optimized out>
        sgroup = <optimized out>
        old_sgroup = <optimized out>
        l_o = <optimized out>
        e_o = <optimized out>
        next = <optimized out>
        vrrp_o = <optimized out>
        ifindex = <optimized out>
        ifindex_o = <optimized out>
        max_mtu_len = <optimized out>
#3  0x0000559972965be1 in start_vrrp () at vrrp_daemon.c:264
No locals.
#4  0x0000559972966034 in start_vrrp_child () at vrrp_daemon.c:537
        pid = <optimized out>
        syslog_ident = <optimized out>
#5  0x0000559972966138 in vrrp_respawn_thread (thread=<optimized out>) at vrrp_daemon.c:452
        pid = <optimized out>
#6  0x000055997298b1c5 in thread_call (thread=0x7ffec96714b0) at scheduler.c:846
No locals.
#7  launch_scheduler () at scheduler.c:871
        thread = {id = 55530, type = 5 '\005', next = 0x0, prev = 0x0, master = 0x559973049e40, func = 0x5599729660e0 <vrrp_respawn_thread>, arg = 0x0, sands = {tv_sec = 1501164211, tv_usec = 285121}, u = {val = 3682, fd = 3682, c = {pid = 3682, status = 11}}}
#8  0x0000559972951242 in keepalived_main (argc=2, argv=<optimized out>) at main.c:976
        report_stopped = true
        uname_buf = {sysname = "Linux", '\000' <repeats 59 times>, nodename = "directorvm.rpc.rackspace.com", '\000' <repeats 36 times>, release = "3.10.0-663.el7.x86_64", '\000' <repeats 43 times>, version = "#1 SMP Tue May 2 16:00:29 EDT 2017", '\000' <repeats 30 times>, 
          machine = "x86_64", '\000' <repeats 58 times>, domainname = "(none)", '\000' <repeats 58 times>}
        end = 0x7ffec96715d8 "-663.el7.x86_64"
        buf_len = <optimized out>
#9  0x00007f8486d04c05 in __libc_start_main (main=0x55997294fe40 <main>, argc=2, ubp_av=0x7ffec9671998, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffec9671988) at ../csu/libc-start.c:274
        result = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {0, 5405040621749117544, 94117540724293, 140732277397904, 0, 0, -5404569138252635544, -5473137511157751192}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x7f84890fd1b3 <_dl_init+275>, 0x7f8489311148}, data = {prev = 0x0, cleanup = 0x0, canceltype = -1995451981}}}
        not_first_call = <optimized out>
#10 0x000055997294fe6e in _start ()
No symbol table info available.

Comment 7 Gonéri Le Bouder 2017-07-27 18:22:57 UTC

*** This bug has been marked as a duplicate of bug 1449769 ***