Bug 985142
| Summary: | audit freezes the system on disk error | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Ondrej Moriš <omoris> |
| Component: | selinux-policy | Assignee: | Miroslav Grepl <mgrepl> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Ondrej Moriš <omoris> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 7.0 | CC: | dwalsh, mgrepl, mmalik, mvadkert, omoris, sgrubb |
| Target Milestone: | beta | ||
| Target Release: | --- | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | selinux-policy-3.12.1-69.el7 | Doc Type: | Bug Fix |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2014-06-13 09:52:46 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 717785, 978512 | ||
|
Description
Ondrej Moriš
2013-07-16 22:49:23 UTC
THis is an SELinux issue, can you run the test in permissive mode and gather the AVC's. We particularly want to see what auditd_t is trying to do to systemd. I'll also will look at auditd code to see if there is some room for improvement after trying to change the run level. Gathering AVC might be a problem, since the situation is completely different in permissive mode. I see only the following AVC:
----
time->Wed Jul 17 10:56:14 2013
type=SYSCALL msg=audit(1374072974.945:29): arch=c000003e syscall=1 success=yes exit=209 a0=4 a1=7fc3772c0000 a2=d1 a3=d0 items=0 ppid=1 pid=2390 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 ses=4294967295 tty=(none) comm="auditd" exe="/usr/sbin/auditd" subj=system_u:system_r:auditd_t:s0 key=(null)
type=AVC msg=audit(1374072974.945:29): avc: denied { append } for pid=2390 comm="auditd" path="/var/log/audit/audit.log" dev="dm-1" ino=2368001 scontext=system_u:system_r:auditd_t:s0 tcontext=system_u:object_r:games_data_t:s0 tclass=file
Which seems to be perfectly correct.
The problem appears only if audit.log cannot be opened for writing, which is the case only in enforcing mode I think.
Any suggestions how to get more information you may find helpful?
Ondrej, could you run systemd in DEBUG mode? We could see more info in # journalctl -x Well, I have booted with systemd in debug and when a disk_error is hit, systemd keeps printing tons of the following messages: Jul 18 18:20:52 hp-dl360gen8-01.rhts.eng.bos.redhat.com systemd[1]: SELinux policy denies access. Jul 18 18:20:52 hp-dl360gen8-01.rhts.eng.bos.redhat.com systemd[1]: SELinux access check scon=system_u:system_r:auditd_t:s0 tcon=system_u:object_r:systemd_unit_file_t:s0 tclass=service perm=start path=/usr/lib/systemd/system/rescue.target cmdline=(null): -13 and Jul 18 18:20:52 hp-dl360gen8-01.rhts.eng.bos.redhat.com auditd[1776]: The audit daemon is now changing the system to single user mode due to previously mention... There are thousands of both these messages (after a few minutes /v/l/m is about 200 MB and keeps growing until the system goes down). Then (later) there is: Jul 18 18:20:52 hp-dl360gen8-01.rhts.eng.bos.redhat.com kernel: Out of memory: Kill process 1509 (dhclient) score 0 or sacrifice child I am sorry, but I see nothing else. commit 1f239dd385a2f13d36f3ac38698e8525a1c2d796
Author: Miroslav Grepl <mgrepl>
Date: Mon Jul 22 08:55:23 2013 +0200
Make auditd working if audit is configured to perform SINGLE action on disk error
Will it also work if auditd attempts to halt the system? I have tried to reproduce the issues with selinux-policy-3.12.1-69.el7.noarch and it is still there: Aug 7 15:00:57 cc-toe3 kernel: [ 1085.847453] Code: 62 85 00 8b 0d a5 00 be 00 85 d2 48 63 f0 0f 84 32 01 00 00 41 8d 3c 16 39 f9 0f 86 26 01 00 00 45 85 e4 0f 84 b5 00 00 00 85 c0 <0f> 84 ad 00 00 00 48 8b 15 46 b2 92 00 49 89 f2 49 29 d2 4f 8d Aug 7 15:00:57 cc-toe3 kernel: [ 1087.617085] cmac rmd160 crypto_null camellia_generic camellia_x86_64 lzo cast6_generic cast5_generic cast_common deflate zlib_deflate cts gcm ccm serpent_sse2_x86_64 serpent_generic blowfish_generic blowfish_x86_64 blowfish_common twofish_generic ip6table_filter twofish_x86_64_3way twofish_x86_64 i p6_tables twofish_common xcbc ebtable_nat ebtables sha512_generic des_generic ah6 ah4 esp6 esp4 xfrm4_mode_beet xfrm4_tunnel tunnel4 xfrm4_mode_tunnel xfrm4_mode_transport xfrm6_mode_transport xfrm6_mode_ro xfrm6_mode_beet xfrm6_mode_tunnel ipcomp ipcomp6 xfrm6_tunnel tunnel6 xfrm_ipcomp af_key ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc sg mperf kvm_amd kvm amd64_edac_mod e1000e edac_mce_amd ptp sp5100_tco microcode pcspkr serio_raw edac_core k10temp i2c_piix4 pps_core hpilo hpwdt bnx2 acpi_power_meter xfs libcrc32c sr_mod c drom i2c_algo_bit sd_mod drm_kms_helper crc_t10dif ata_generic ttm ahci pata_acpi libahci pata_atiixp drm libata i2c_core hpsa dm_mirror dm_region_hash dm_log dm_mod Aug 7 15:00:57 cc-toe3 kernel: [ 1088.188720] CPU: 1 PID: 3839 Comm: sshd Not tainted 3.10.0-5.el7.bz983949.1.x86_64 #1 Aug 7 15:00:57 cc-toe3 kernel: [ 1088.234226] Hardware name: HP ProLiant DL385 G7, BIOS A18 03/19/2012 Aug 7 15:00:57 cc-toe3 kernel: [ 1088.271683] task: ffff88023873e420 ti: ffff880238cae000 task.ti: ffff880238cae000 Aug 7 15:00:57 cc-toe3 kernel: [ 1088.315166] RIP: 0010:[<ffffffff810d8dad>] [<ffffffff810d8dad>] audit_log_start+0x9d/0x430 Aug 7 15:00:57 cc-toe3 kernel: [ 1088.363961] RSP: 0018:ffff880238cafae8 EFLAGS: 00000206 Aug 7 15:00:57 cc-toe3 kernel: [ 1088.394658] RAX: 000000000000ea60 RBX: 00000001000aa880 RCX: 0000000000000141 Aug 7 15:00:57 cc-toe3 kernel: [ 1088.436683] RDX: 00000001000be735 RSI: 000000000000ea60 RDI: 0000000000000140 Aug 7 15:00:57 cc-toe3 kernel: [ 1088.477679] RBP: ffff880238cafb60 R08: ffff880238cafb28 R09: 0000000000000000 Aug 7 15:00:57 cc-toe3 kernel: [ 1088.519230] R10: fffffffefff5032b R11: 000000000000be7b R12: 0000000000000000 Aug 7 15:00:57 cc-toe3 kernel: [ 1088.562318] R13: 0000000000000286 R14: ffff88013bc97960 R15: ffffffff815f89da Aug 7 15:00:57 cc-toe3 kernel: [ 1088.603742] FS: 00007fd00dd40840(0000) GS:ffff88013bc80000(0000) knlGS:0000000000000000 Aug 7 15:00:57 cc-toe3 kernel: [ 1088.650891] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b Aug 7 15:00:57 cc-toe3 kernel: [ 1088.683759] CR2: 00007fd00afffbe0 CR3: 0000000228f80000 CR4: 00000000000006e0 Aug 7 15:00:57 cc-toe3 kernel: [ 1088.725558] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 7 15:00:57 cc-toe3 kernel: [ 1088.768518] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 7 15:00:57 cc-toe3 kernel: [ 1088.809264] Stack: Aug 7 15:00:57 cc-toe3 kernel: [ 1088.821342] 0000000000000000 000000d000000964 ffff88023873e420 ffff88013a3cd9c0 Aug 7 15:00:57 cc-toe3 kernel: [ 1088.865345] 0000000000000000 0000000000000000 ffff88023873e420 ffffffff810942f0 Aug 7 15:00:57 cc-toe3 kernel: [ 1088.908541] dead000000100100 dead000000200200 ffff880238cafba8 ffff88023873e420 Aug 7 15:00:57 cc-toe3 kernel: [ 1088.949848] Call Trace: Aug 7 15:00:57 cc-toe3 kernel: [ 1088.963537] [<ffffffff810942f0>] ? wake_up_state+0x20/0x20 Aug 7 15:00:57 cc-toe3 kernel: [ 1088.994471] [<ffffffff810d995f>] audit_log_common_recv_msg+0x4f/0xa0 Aug 7 15:00:57 cc-toe3 kernel: [ 1089.031300] [<ffffffff810da02d>] audit_receive_msg+0x67d/0x8c0 Aug 7 15:00:57 cc-toe3 kernel: [ 1089.063664] [<ffffffff81289235>] ? sock_has_perm+0x75/0x90 Aug 7 15:00:57 cc-toe3 kernel: [ 1089.095969] [<ffffffff810da2c0>] audit_receive+0x50/0xa0 Aug 7 15:00:57 cc-toe3 kernel: [ 1089.127704] [<ffffffff8151902d>] netlink_unicast+0xdd/0x190 Aug 7 15:00:57 cc-toe3 kernel: [ 1089.160423] [<ffffffff815193a6>] netlink_sendmsg+0x2c6/0x6c0 Aug 7 15:00:57 cc-toe3 kernel: [ 1089.193433] [<ffffffff814d6a49>] sock_sendmsg+0x99/0xd0 Aug 7 15:00:57 cc-toe3 kernel: [ 1089.224382] [<ffffffff8115acd3>] ? handle_pte_fault+0x93/0xa40 Aug 7 15:00:57 cc-toe3 kernel: [ 1089.258859] [<ffffffff811832ba>] ? kmem_cache_alloc+0x1ba/0x1d0 Aug 7 15:00:57 cc-toe3 kernel: [ 1089.293964] [<ffffffff814d6fa1>] SYSC_sendto+0x121/0x1c0 Aug 7 15:00:57 cc-toe3 kernel: [ 1089.325636] [<ffffffff811acac5>] ? do_vfs_ioctl+0x305/0x520 Aug 7 15:00:57 cc-toe3 kernel: [ 1089.356439] [<ffffffff8128c27e>] ? file_has_perm+0x8e/0xa0 Aug 7 15:00:57 cc-toe3 kernel: [ 1089.387870] [<ffffffff814d798e>] SyS_sendto+0xe/0x10 Aug 7 15:00:57 cc-toe3 kernel: [ 1089.415424] [<ffffffff81604459>] system_call_fastpath+0x16/0x1b However with this version I do not see any of the previous messages: Jul 16 18:08:58 hp-dl360gen8-01 systemd[1]: SELinux policy denies access. Jul 16 18:08:58 hp-dl360gen8-01 auditd[15791]: The audit daemon is now changing the system to single user mode due to previously mentioned write error And do you have systemd in DEBUG mode? We need to see additional info for Jul 16 18:08:58 hp-dl360gen8-01 systemd[1]: SELinux policy denies access. Any other AVC messages? (In reply to Miroslav Grepl from comment #9) > And do you have systemd in DEBUG mode? > > We need to see additional info for > > > Jul 16 18:08:58 hp-dl360gen8-01 systemd[1]: SELinux policy denies access. As I mentioned in Comment #8, I do not see those message anymore. (In reply to Daniel Walsh from comment #10) > Any other AVC messages? Unfortunately, I do not see any, as I mentioned earlier, in permissive mode there are no AVC messages at all and in enforcing mode I do not see them neither. My understanding is that the original bug had two parts, the first part (selinux one) was fixed (comment #6) but another part (kernel?) remains broken. Maybe we should consider changing the component or opening another bug. I can invite you to my testing machine so that you see the issue in more detail if needed. Ondro, the problem is you are still getting Jul 16 18:08:58 hp-dl360gen8-01 systemd[1]: SELinux policy denies access. Jul 16 18:08:58 hp-dl360gen8-01 auditd[15791]: The audit daemon is now changing the system to single user mode due to previously mentioned write error right? (In reply to Miroslav Grepl from comment #12) > Ondro, > the problem is you are still getting > > Jul 16 18:08:58 hp-dl360gen8-01 systemd[1]: SELinux policy denies access. > Jul 16 18:08:58 hp-dl360gen8-01 auditd[15791]: The audit daemon is now > changing the system to single user mode due to previously mentioned write > error > > right? No, this is fixed, I do not see them anymore. I saw these in the original bug description, but it was fixed with your fixes. The problem is that the system is still crashing with a call trace mentioned in comment #8. By the way: Please notice that there we are simulating a disk error by selinux (changing security context of /var/log/audit/audit.log). Therefore, in permissive mode such method to invoke disk error does not work and hence everything is fine and there are no AVC messages. That is the reason why we need enforcing mode to reproduce this issue. What about aforementioned call trace? Isn't there anything useful for locating the problem? So...why is this bug changed to audit? The audit daemon is definitely not the problem. It issues the command to go to runlevel 1 just as if you typed it on the command line. The trace above is a kernel call trace. Someone from the kernel should look at it, but function names are missing so its probably not very useful. Is this a new test? Did it work on RHEL6? The reason I ask is that in discussion with eric paris, we think that what's happening is the auditd has write error, issues command for systemd, goes back to listening, sees the avc, tries to write it, perm denied, issues new command to systemd, goes back to listening, sees new avc, and around we go over and over until the system hangs. Did this work in RHEL6? kernel-3.10.0-9.el7.x86_64 systemd-206-4.el7.x86_64 audit-2.3.2-1.el7.x86_64 selinux-policy-3.12.1-70.el7.noarch Works just fine for me. It goes to single user mode and asks for a pword.... This request was resolved in Red Hat Enterprise Linux 7.0. Contact your manager or support representative in case you have further questions about the request. |