Bug 1264533

Summary: Oracle Grid 12c Failed to stat() POSIX shared memory segment
Product: Red Hat Enterprise Linux 7 Reporter: Daniel Yeisley <dyeisley>
Component: selinux-policyAssignee: Miroslav Grepl <mgrepl>
Status: CLOSED DUPLICATE QA Contact: BaseOS QE Security Team <qe-baseos-security>
Severity: high Docs Contact:
Priority: unspecified    
Version: 7.2CC: dyeisley, fdanapfe, lvrabec, martin.marques, mgrepl, mmalik, plautrba, pvrabec, ssekidde, tbowling, tmichael, troels
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-12-03 16:00:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
selinux error log none

Description Daniel Yeisley 2015-09-18 17:36:35 UTC
Description of problem:
I've had a lot of difficulty getting Oracle Grid 12c to work on RHEL 7.2.  

Version-Release number of selected component (if applicable):
> selinux-policy-3.13.1-49.el7.noarch
> selinux-policy-targeted-3.13.1-49.el7.noarch

How reproducible:


Steps to Reproduce:
1. I started by installing RHEL 7.1 to verify that I wasn't dealing with hardware or user errors.  
2. Installed Oracle 12c Grid, Oracle 12c Database on a 2-node cluster and ran Oracle automated stress test against it. No issues.
3. Then I pointed yum at the RHEL-7.2-20150917.0 repo and updated to the following packages on one of the nodes:
> dracut-033-346.el7.x86_64
> dracut-config-rescue-033-346.el7.x86_64
> dracut-network-033-346.el7.x86_64
> initscripts-9.49.30-1.el7.x86_64
> kmod-20-5.el7.x86_64
> libgudev1-219-14.el7.x86_64
> selinux-policy-3.13.1-49.el7.noarch
> selinux-policy-targeted-3.13.1-49.el7.noarch
> systemd-219-14.el7.x86_64
> systemd-libs-219-14.el7.x86_64
> systemd-sysv-219-14.el7.x86_64
I wanted to rule out dracut before installing the kernel and it required systemd to be updated.

4. I rebooted the system and started the test.  I saw the following in /var/log/messages
Sep 18 13:22:40 veritas5 systemd-logind: Failed to stat() POSIX shared memory segment ora_+ASM2_219086852_13: Permission denied
Sep 18 13:22:40 veritas5 systemd-logind: Failed to stat() POSIX shared memory segment ora_+ASM2_219086852_12: Permission denied
Sep 18 13:22:40 veritas5 systemd-logind: Failed to stat() POSIX shared memory segment ora_+ASM2_219086852_11: Permission denied
Sep 18 13:22:40 veritas5 systemd-logind: Failed to stat() POSIX shared memory segment ora_+ASM2_219086852_10: Permission denied
Sep 18 13:22:40 veritas5 systemd-logind: Failed to stat() POSIX shared memory segment ora_+ASM2_219086852_9: Permission denied
Sep 18 13:22:40 veritas5 systemd-logind: Failed to stat() POSIX shared memory segment ora_+ASM2_219086852_8: Permission denied
Sep 18 13:22:40 veritas5 systemd-logind: Failed to stat() POSIX shared memory segment ora_+ASM2_219086852_7: Permission denied
Sep 18 13:22:40 veritas5 systemd-logind: Failed to stat() POSIX shared memory segment ora_+ASM2_219086852_6: Permission denied
Sep 18 13:22:40 veritas5 systemd-logind: Failed to stat() POSIX shared memory segment ora_+ASM2_219086852_5: Permission denied
Sep 18 13:22:40 veritas5 systemd-logind: Failed to stat() POSIX shared memory segment ora_+ASM2_219086852_4: Permission denied

[root@veritas5 ~]# /sbin/ausearch --input-logs -sv no -m AVC -m USER_AVC -m SELINUX_ERR
...
time->Fri Sep 18 13:22:40 2015
type=SYSCALL msg=audit(1442596960.926:840): arch=c000003e syscall=262 success=no exit=-13 a0=12 a1=7ffabe781583 a2=7ffff5467380 a3=100 items=0 ppid=1 pid=747 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="systemd-logind" exe="/usr/lib/systemd/systemd-logind" subj=system_u:system_r:systemd_logind_t:s0 key=(null)
type=AVC msg=audit(1442596960.926:840): avc:  denied  { getattr } for  pid=747 comm="systemd-logind" path="/dev/shm/ora_+ASM2_219086852_3" dev="tmpfs" ino=103059 scontext=system_u:system_r:systemd_logind_t:s0 tcontext=system_u:object_r:tmpfs_t:s0 tclass=file
----
time->Fri Sep 18 13:22:40 2015
type=SYSCALL msg=audit(1442596960.926:841): arch=c000003e syscall=262 success=no exit=-13 a0=12 a1=7ffabe7815b3 a2=7ffff5467380 a3=100 items=0 ppid=1 pid=747 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="systemd-logind" exe="/usr/lib/systemd/systemd-logind" subj=system_u:system_r:systemd_logind_t:s0 key=(null)
type=AVC msg=audit(1442596960.926:841): avc:  denied  { getattr } for  pid=747 comm="systemd-logind" path="/dev/shm/ora_+ASM2_219086852_2" dev="tmpfs" ino=103058 scontext=system_u:system_r:systemd_logind_t:s0 tcontext=system_u:object_r:tmpfs_t:s0 tclass=file
----
time->Fri Sep 18 13:22:40 2015
type=SYSCALL msg=audit(1442596960.926:842): arch=c000003e syscall=262 success=no exit=-13 a0=12 a1=7ffabe7815e3 a2=7ffff5467380 a3=100 items=0 ppid=1 pid=747 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="systemd-logind" exe="/usr/lib/systemd/systemd-logind" subj=system_u:system_r:systemd_logind_t:s0 key=(null)
type=AVC msg=audit(1442596960.926:842): avc:  denied  { getattr } for  pid=747 comm="systemd-logind" path="/dev/shm/ora_+ASM2_219086852_1" dev="tmpfs" ino=103057 scontext=system_u:system_r:systemd_logind_t:s0 tcontext=system_u:object_r:tmpfs_t:s0 tclass=file
----
time->Fri Sep 18 13:22:40 2015
type=SYSCALL msg=audit(1442596960.939:844): arch=c000003e syscall=262 success=no exit=-13 a0=12 a1=7ffabe781613 a2=7ffff5467380 a3=100 items=0 ppid=1 pid=747 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="systemd-logind" exe="/usr/lib/systemd/systemd-logind" subj=system_u:system_r:systemd_logind_t:s0 key=(null)
type=AVC msg=audit(1442596960.939:844): avc:  denied  { getattr } for  pid=747 comm="systemd-logind" path="/dev/shm/ora_+ASM2_219086852_0" dev="tmpfs" ino=103056 scontext=system_u:system_r:systemd_logind_t:s0 tcontext=system_u:object_r:tmpfs_t:s0 tclass=file
----
time->Fri Sep 18 13:22:40 2015
type=SYSCALL msg=audit(1442596960.939:845): arch=c000003e syscall=262 success=no exit=-13 a0=12 a1=7ffabe781643 a2=7ffff5467380 a3=100 items=0 ppid=1 pid=747 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="systemd-logind" exe="/usr/lib/systemd/systemd-logind" subj=system_u:system_r:systemd_logind_t:s0 key=(null)
type=AVC msg=audit(1442596960.939:845): avc:  denied  { getattr } for  pid=747 comm="systemd-logind" path="/dev/shm/ora_+ASM2_219054083_0" dev="tmpfs" ino=103052 scontext=system_u:system_r:systemd_logind_t:s0 tcontext=system_u:object_r:tmpfs_t:s0 tclass=file

Actual results:


Expected results:


Additional info:

Comment 2 Miroslav Grepl 2015-09-18 18:13:57 UTC
Daniel,
any chance to test it with permissive mode if we can get more AVCs?

Comment 3 Daniel Yeisley 2015-09-21 17:09:47 UTC
Created attachment 1075542 [details]
selinux error log

I generated the attached log with the following command: /sbin/ausearch --input-logs -sv no -m AVC -m USER_AVC -m SELINUX_ERR

Comment 4 Miroslav Grepl 2015-09-30 08:12:18 UTC
Do you know what process creates /dev/shm/ora_+ASM2_223477764_257? 

Does everything work if you add a local policy?

Re-tets in permissive and run

#setenforce 1;setenforce 0
#ausearch -m avc -ts recent |audit2allow -M mypol
#semodule -i mypol.pp

Thank you.

Comment 5 Daniel Yeisley 2015-10-01 16:05:14 UTC
This issue is related to a change made to systemd.  

* logind will now automatically remove all IPC objects owned
  by a user if she or he fully logs out. This makes sure that
  users who are logged out cannot continue to consume IPC
  resources. This covers SysV memory, semaphores and message 
  queues as well as POSIX shared memory and message 
  queues. Traditionally SysV and POSIX IPC had no life-cycle
  limits, with this functionality this is corrected. This may
  be turned off using the RemoveIPC= switch of logind.conf.

I added the switch mentioned in the changelog and restarted the logind service.

I installed Oracle 12c Grid and Oracle 12c database and started a stress test against it.  I don't see any complaints in /var/log/messages or selinux issues.

[root@localhost ~]# ausearch -m avc -ts recent
<no matches>
[root@localhost ~]# grep POSIX /var/log/messages 
Sep 30 17:03:14 veritas5 systemd: Mounted POSIX Message Queue File System.
[root@localhost ~]#

Comment 6 Miroslav Grepl 2015-10-01 18:38:10 UTC
(In reply to Daniel Yeisley from comment #5)
> This issue is related to a change made to systemd.  
> 
> * logind will now automatically remove all IPC objects owned
>   by a user if she or he fully logs out. This makes sure that
>   users who are logged out cannot continue to consume IPC
>   resources. This covers SysV memory, semaphores and message 
>   queues as well as POSIX shared memory and message 
>   queues. Traditionally SysV and POSIX IPC had no life-cycle
>   limits, with this functionality this is corrected. This may
>   be turned off using the RemoveIPC= switch of logind.conf.
> 
> I added the switch mentioned in the changelog and restarted the logind
> service.
> 
> I installed Oracle 12c Grid and Oracle 12c database and started a stress
> test against it.  I don't see any complaints in /var/log/messages or selinux
> issues.
> 
> [root@localhost ~]# ausearch -m avc -ts recent
> <no matches>
> [root@localhost ~]# grep POSIX /var/log/messages 
> Sep 30 17:03:14 veritas5 systemd: Mounted POSIX Message Queue File System.
> [root@localhost ~]#

So if I understand correctly you are not able to reproduce it again, right?

Comment 7 Daniel Yeisley 2015-10-01 19:01:13 UTC
(In reply to Miroslav Grepl from comment #6)
> (In reply to Daniel Yeisley from comment #5)
> > This issue is related to a change made to systemd.  
> > 
> > * logind will now automatically remove all IPC objects owned
> >   by a user if she or he fully logs out. This makes sure that
> >   users who are logged out cannot continue to consume IPC
> >   resources. This covers SysV memory, semaphores and message 
> >   queues as well as POSIX shared memory and message 
> >   queues. Traditionally SysV and POSIX IPC had no life-cycle
> >   limits, with this functionality this is corrected. This may
> >   be turned off using the RemoveIPC= switch of logind.conf.
> > 
> > I added the switch mentioned in the changelog and restarted the logind
> > service.
> > 
> > I installed Oracle 12c Grid and Oracle 12c database and started a stress
> > test against it.  I don't see any complaints in /var/log/messages or selinux
> > issues.
> > 
> > [root@localhost ~]# ausearch -m avc -ts recent
> > <no matches>
> > [root@localhost ~]# grep POSIX /var/log/messages 
> > Sep 30 17:03:14 veritas5 systemd: Mounted POSIX Message Queue File System.
> > [root@localhost ~]#
> 
> So if I understand correctly you are not able to reproduce it again, right?

If I set "RemoveIPC=yes" in /etc/systemd/logind.conf with selinux=enforcing then yes I can reproduce it.  Setting "RemoveIPC=no" makes it go away.

I think what this requires is documentation.  Oracle Grid users should set the RemoveIPC=no in /etc/systemd/logind.conf.

Comment 8 Miroslav Grepl 2015-10-02 06:55:19 UTC
Yes we need to have fixes for this option. 

But how is Oracle 12c Grid started?

Comment 9 Daniel Yeisley 2015-10-06 13:00:40 UTC
(In reply to Miroslav Grepl from comment #8)
> Yes we need to have fixes for this option. 
> 
> But how is Oracle 12c Grid started?

I guess I don't understand what needs to be fixed.  As long as RemoveIPC=no is set then there is no issue.  I'm ready to close this as NOTABUG.

Comment 10 Miroslav Grepl 2015-10-12 07:51:06 UTC
(In reply to Daniel Yeisley from comment #9)
> (In reply to Miroslav Grepl from comment #8)
> > Yes we need to have fixes for this option. 
> > 
> > But how is Oracle 12c Grid started?
> 
> I guess I don't understand what needs to be fixed.  As long as RemoveIPC=no
> is set then there is no issue.  I'm ready to close this as NOTABUG.

My point is it could happen also for another cases if RemoveIPC=Yes is used.

Comment 11 Frank Danapfel 2015-11-27 11:41:34 UTC
(In reply to Miroslav Grepl from comment #10)
> (In reply to Daniel Yeisley from comment #9)
> > (In reply to Miroslav Grepl from comment #8)
> > > Yes we need to have fixes for this option. 
> > > 
> > > But how is Oracle 12c Grid started?
> > 
> > I guess I don't understand what needs to be fixed.  As long as RemoveIPC=no
> > is set then there is no issue.  I'm ready to close this as NOTABUG.
> 
> My point is it could happen also for another cases if RemoveIPC=Yes is used.

Yes, SAP sw also uses POSIX shared memory segments extensively, therefore having them removed automatically when the user running the SAP processes logs out will hurt customers running SAP as well.

Since this is a change from previous RHEL releases I would say this is a reression ant therefore should be fixed asap.

Comment 12 Frank Danapfel 2015-12-02 14:48:26 UTC
This looks to be the same issue as https://bugzilla.redhat.com/show_bug.cgi?id=1284588

There have now also been reports from customers running SAP software on RHEL7 that after upgrading to RHEL 7.2 their SAP installations are broken because of this, therefore this change should be reverted asap.

Comment 13 Daniel Yeisley 2015-12-02 15:07:30 UTC
(In reply to Frank Danapfel from comment #12)
> This looks to be the same issue as
> https://bugzilla.redhat.com/show_bug.cgi?id=1284588
> 
> There have now also been reports from customers running SAP software on
> RHEL7 that after upgrading to RHEL 7.2 their SAP installations are broken
> because of this, therefore this change should be reverted asap.

Yes, they appear to be the same issue.  I got around it by setting RemoveIPC=no in /etc/systemd/logind.conf and restarting the service.

Comment 14 Frank Danapfel 2015-12-02 15:11:06 UTC
OK, looks like a Z-Stream fix is on the way:
https://bugzilla.redhat.com/show_bug.cgi?id=1286031

Comment 15 Terry Bowling 2015-12-03 15:58:27 UTC
can possibly close this as a duplicate of bz1284588 ?

Comment 16 Daniel Yeisley 2015-12-03 16:00:18 UTC
(In reply to Terry Bowling from comment #15)
> can possibly close this as a duplicate of bz1284588 ?

Yes.

*** This bug has been marked as a duplicate of bug 1284588 ***