Bug 478892

Summary: AVC storm brings setroubleshoot deamon down on its own AVC
Product: [Fedora] Fedora Reporter: Jaroslav Franek <jarin.franek>
Component: setroubleshootAssignee: Daniel Walsh <dwalsh>
Status: CLOSED WONTFIX QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: low    
Version: 10CC: bloch, dwalsh, jdennis, mgrepl
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-12-18 07:30:15 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 517000    

Description Jaroslav Franek 2009-01-05 21:16:56 UTC
Description of problem:

Suddenly I got a storm of AVCs from logwatch and related cat, perl commands. After about 160 of AVCs in about 2 minutes the setroubleshoot daemon generated AVCs on its own and died. It did not cleanup pid file. It damaged one of its files /var/lib/setroubleshoot/audit_listener_database.xml - losts all memory on the storm events.


Version-Release number of selected component (if applicable):
setroubleshoot-2.0.12-3.fc10.noarch

How reproducible:
Probably when logwatch invoked by cron. Not the first occurrence of the issue.
I cannot say I did anything in particular to fire the storm. I did not play with SELinux security contexts either.

Steps to Reproduce:
1. I just "sit and wait", I guess it is invoked by cron from time to time
2.
3.
  
Actual results:

Storm of AVCs begin

Jan  5 21:17:46 localhost setroubleshoot: SELinux is preventing 0logwatch (logwatch_t) "sys_resource" logwatch_t. For complete SELinux messages. run sealert -l 14d6d154-3566-42e2-bb34-2f30831ea173
Jan  5 21:17:47 localhost setroubleshoot: SELinux is preventing cat (logwatch_t) "sys_resource" logwatch_t. For complete SELinux messages. run sealert -l 14d6d154-3566-42e2-bb34-2f30831ea173
Jan  5 21:17:48 localhost setroubleshoot: SELinux is preventing cat (logwatch_t) "sys_resource" logwatch_t. For complete SELinux messages. run sealert -l 14d6d154-3566-42e2-bb34-2f30831ea173
Jan  5 21:17:49 localhost setroubleshoot: SELinux is preventing cat (logwatch_t) "sys_resource" logwatch_t. For complete SELinux messages. run sealert -l 14d6d154-3566-42e2-bb34-2f30831ea173
Jan  5 21:17:50 localhost setroubleshoot: SELinux is preventing perl (logwatch_t) "sys_resource" logwatch_t. For complete SELinux messages. run sealert -l 14d6d154-3566-42e2-bb34-2f30831ea173
Jan  5 21:17:50 localhost setroubleshoot: SELinux is preventing cat (logwatch_t) "sys_resource" logwatch_t. For complete SELinux messages. run sealert -l 14d6d154-3566-42e2-bb34-2f30831ea173
Jan  5 21:17:51 localhost setroubleshoot: SELinux is preventing cat (logwatch_t) "sys_resource" logwatch_t. For complete SELinux messages. run sealert -l 14d6d154-3566-42e2-bb34-2f30831ea173
...


After counting about 160 of these, about two minutes later, setroubleshoot deamon got into difficulties on its own:


Jan  5 21:19:57 localhost setroubleshoot: [program.ERROR] setroubleshoot generated AVC, exiting to avoid recursion, context=system_u:system_r:setroubleshootd_t:s0, AVC scontext=system_u:system_r:s
etroubleshootd_t:s0
Jan  5 21:19:57 localhost setroubleshoot: SELinux is preventing perl (logwatch_t) "sys_resource" logwatch_t. For complete SELinux messages. run sealert -l 14d6d154-3566-42e2-bb34-2f30831ea173
Jan  5 21:19:57 localhost setroubleshoot: [program.ERROR] audit event#012node=localhost.localdomain type=AVC msg=audit(1231186797.217:345): avc:  denied  { sys_resource } for  pid=2661 comm="setroubleshootd" capability=24 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=system_u:system_r:setroubleshootd_t:s0 tclass=capability#012#012node=localhost.localdomain type=AVC msg=audit(123
1186797.217:345): avc:  denied  { sys_resource } for  pid=2661 comm="setroubleshootd" capability=24 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=system_u:system_r:setroubleshootd_t:s0
tclass=capability#012#012node=localhost.localdomain type=AVC msg=audit(1231186797.217:345): avc:  denied  { sys_resource } for  pid=2661 comm="setroubleshootd" capability=24 scontext=system_u:syst
em_r:setroubleshootd_t:s0 tcontext=system_u:system_r:setroubleshootd_t:s0 tclass=capability#012#012node=localhost.localdomain type=SYSCALL msg=audit(1231186797.217:345): arch=40000003 syscall=4 success=yes exit=12288 a0=e a1=b52919c a2=3000 a3=3000 items=0 ppid=1 pid=2661 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="setroubleshootd" exe="/usr/bin/python" subj=system_u:system_r:setroubleshootd_t:s0 key=(null)
Jan  5 21:19:57 localhost setroubleshoot: [program.ERROR] setroubleshoot generated AVC, exiting to avoid recursion, context=system_u:system_r:setroubleshootd_t:s0, AVC scontext=system_u:system_r:setroubleshootd_t:s0
Jan  5 21:19:57 localhost setroubleshoot: [program.ERROR] audit event#012node=localhost.localdomain type=AVC msg=audit(1231186797.225:346): avc:  denied  { sys_resource } for  pid=2661 comm="setroubleshootd" capability=24 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=system_u:system_r:setroubleshootd_t:s0 tclass=capability#012#012node=localhost.localdomain type=SYSCALL msg=audit
(1231186797.225:346): arch=40000003 syscall=4 success=yes exit=3974 a0=e a1=b8093000 a2=f86 a3=f86 items=0 ppid=1 pid=2661 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="setroubleshootd" exe="/usr/bin/python" subj=system_u:system_r:setroubleshootd_t:s0 key=(null)
Jan  5 21:19:57 localhost setroubleshoot: SELinux is preventing perl (logwatch_t) "sys_resource" logwatch_t. For complete SELinux messages. run sealert -l 14d6d154-3566-42e2-bb34-2f30831ea173
// the storm log ends here

// trying the very first one
# sealert -l 14d6d154-3566-42e2-bb34-2f30831ea173
failed to connect to server: Connection refused

# services --status-all | grep setroubleshootd
setroubleshootd dead but pid file exists

// trying to restart
# service setroubleshoot restart
Stopping setroubleshootd:                                  [FAILED]
Starting setroubleshootd:                                  [  OK  ]
[root@localhost etc]# /var/lib/setroubleshoot/audit_listener_database.xml:155: parser error : Extra content at the end of the document
rname="franekj">
^
Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/setroubleshoot/rpc.py", line 940, in handle_client_io
    self.receiver.feed(data)
  File "/usr/lib/python2.5/site-packages/setroubleshoot/rpc.py", line 762, in feed
    self.process()
  File "/usr/lib/python2.5/site-packages/setroubleshoot/rpc.py", line 754, in process
    self.dispatchFunc(self.header, self.body)
  File "/usr/lib/python2.5/site-packages/setroubleshoot/rpc.py", line 978, in default_request_handler
    return_args = method_ptr(*args)
  File "/usr/lib/python2.5/site-packages/setroubleshoot/server.py", line 266, in logon
    self.database.add_user(username)
  File "/usr/lib/python2.5/site-packages/setroubleshoot/analyze.py", line 460, in add_user
    self.mark_modified()
  File "/usr/lib/python2.5/site-packages/setroubleshoot/analyze.py", line 340, in mark_modified
    self.save()
  File "/usr/lib/python2.5/site-packages/setroubleshoot/analyze.py", line 327, in save
    self.sigs.write_xml('sigs', self.filepath)
  File "/usr/lib/python2.5/site-packages/setroubleshoot/xml_serialize.py", line 310, in write_xml
    f.close()
KeyboardInterrupt


# sealert -l 14d6d154-3566-42e2-bb34-2f30831ea173
query_alerts error (1003): id (14d6d154-3566-42e2-bb34-2f30831ea173) not found

// losts its memory...



Expected results:

Logwatch should not generate AVCs.
The setroubleshoot daemon should not generate AVC.
The setroubleshoot deamon should handle its own AVC death better (cleanup PID file, not damage its data files)

Additional info:

Comment 1 Daniel Walsh 2009-01-05 22:18:52 UTC
The problem here is a bad kernel that is causing bogus avc messages one of them on setroubleshoot,  setroubleshoot dies when it sees an avc about itself since it does not want to go into an infinite loop.

So
Logwatch should not generate AVCs.
The setroubleshoot daemon should not generate AVC.

Are both caused by a bad kernel.

The setroubleshoot deamon should handle its own AVC death better (cleanup PID
file, not damage its data files)

This is a bug.

Comment 2 Jaroslav Franek 2009-01-06 20:39:56 UTC
I agree. Just for reference, the rogue kernel is logged under Bug #478299. Different AVCs but I am sure the cause is the same - ext4 (and missing patch in the kernel).

As for the setroubleshoot daemon, I guess calling sys.exit(0) on AVC from a context of a worker thread is probably causing the 'damaged files' - another thread got interrupted while writing data. The storm of AVCs might have kept multiple threads occupied.

Comment 4 Bug Zapper 2009-11-18 10:40:09 UTC
This message is a reminder that Fedora 10 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 10.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '10'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 10's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 10 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Comment 5 Bug Zapper 2009-12-18 07:30:15 UTC
Fedora 10 changed to end-of-life (EOL) status on 2009-12-17. Fedora 10 is 
no longer maintained, which means that it will not receive any further 
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.