Description of problem:
  When running on the MRG Realtime kernel, systemtap scripts are causing "BUG: scheduling while atomic" messages.

Version-Release number of selected component (if applicable):
  RHEL 6
  Red Hat Enterprise MRG 2.5 Realtime Kernel
  systemtap-2.5-5.el6.x86_64

How reproducible:
  Always

Steps to Reproduce:
  1. Install MRG Realtime kernel on RHEL6

  2. Save the following script as test.stap:
  %<---- test.stap ----
  #!/usr/bin/stap
  
  probe vm.pagefault {
	  printf("%s\n", probefunc());
  }
  %<---- test.stap ----

  3. run test.stap script:
  # stap test.stap

Actual results:

  Kernel hits "scheduling while atomic" BUG. For example:

BUG: scheduling while atomic: makewhatis/2878/0x00000003
Modules linked in: stap_3e768d9dd19a5493d042da8393d971a0_2841(O) autofs4 ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state ip6table_filter ip6_tables ipv6 ppdev microcode pcspkr parport_pc parport e1000 joydev sg snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc i2c_piix4 i2c_core ext4 jbd2 mbcache sd_mod crc_t10dif sr_mod cdrom sym53c8xx scsi_transport_spi floppy pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: stap_3e768d9dd19a5493d042da8393d971a0_2831]
CPU: 2 PID: 2878 Comm: makewhatis Tainted: G        W  O 3.10.33-rt32.45.el6rt.x86_64 #1
Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
 0000000000000002 ffff880158509b88 ffffffff81575e1e ffff880158509b98
 ffffffff8108362a ffff880158509c28 ffffffff8157737c ffff880158509bc8
 ffff880149f6c000 ffff880149f6c000 ffff880149f6c010 ffff880149f6dfd8
Call Trace:
 <#DB>  [<ffffffff81575e1e>] dump_stack+0x19/0x1b
 [<ffffffff8108362a>] __schedule_bug+0x5a/0x70
 [<ffffffff8157737c>] __schedule+0x7cc/0x800
 [<ffffffff81577594>] schedule+0x34/0xa0
 [<ffffffff8157899d>] rt_spin_lock_slowlock+0x10d/0x350
 [<ffffffff8114cd21>] ? handle_mm_fault+0x1/0x680
 [<ffffffff81579296>] rt_spin_lock+0x26/0x30
 [<ffffffffa05aeab3>] stp_print_flush+0x63/0x1d0 [stap_3e768d9dd19a5493d042da8393d971a0_2841]
 [<ffffffff8114cd21>] ? handle_mm_fault+0x1/0x680
 [<ffffffffa05b8ddf>] probe_2058+0x3bf/0x4e8 [stap_3e768d9dd19a5493d042da8393d971a0_2841]
 [<ffffffffa05b5609>] enter_kprobe_probe+0x1d9/0x3f0 [stap_3e768d9dd19a5493d042da8393d971a0_2841]
 [<ffffffff8157ca86>] kprobe_exceptions_notify+0x4a6/0x570
 [<ffffffff8114cd21>] ? handle_mm_fault+0x1/0x680
 [<ffffffff8114cd20>] ? handle_pte_fault+0xbf0/0xbf0
 [<ffffffff8157da05>] notifier_call_chain+0x55/0x80
 [<ffffffff8157da78>] __atomic_notifier_call_chain+0x48/0x70
 [<ffffffff8157dab6>] atomic_notifier_call_chain+0x16/0x20
 [<ffffffff8157daee>] notify_die+0x2e/0x30
 [<ffffffff8157a7bb>] do_int3+0x4b/0x120
 [<ffffffff81579eab>] int3+0x2b/0x40
 [<ffffffff8114cd20>] ? handle_pte_fault+0xbf0/0xbf0
 <<EOE>>  [<ffffffff8157d69c>] ? __do_page_fault+0x24c/0x550
 [<ffffffffa05af6d5>] ? __stp_tf_get_map_entry+0xa5/0xd0 [stap_3e768d9dd19a5493d042da8393d971a0_2841]
 [<ffffffffa05b66dd>] ? __stp_utrace_task_finder_target_syscall_exit+0x6d/0x280 [stap_3e768d9dd19a5493d042da8393d971a0_2841]
 [<ffffffffa05b3860>] ? utrace_report_syscall_exit+0xf0/0x100 [stap_3e768d9dd19a5493d042da8393d971a0_2841]
 [<ffffffff8157d9ae>] do_page_fault+0xe/0x10
 [<ffffffff8157cd48>] do_async_page_fault+0x28/0xc0
 [<ffffffff81579f78>] async_page_fault+0x28/0x30

Expected results:
 The "BUG scheduling while atomic" should not happen.

Additional info:

This error occurs because the module created by systemtap is calling spin_lock() in the atomic context. Although it's correct for RHEL6 kernel (non-rt kernel), as spinlocks are converted to rt_mutex on MRG Realtime Kernel, when a contention happens the rt_mutex puts the task to sleep, causing the error. For the RT kernel, these spinlocks should be converted to raw spinlocks.

Fortunately, this problem was fixed on mainstream systemtap's repo:

GIT rego: git://sourceware.org/git/systemtap.git

# git format-patch -9 d4e048442b3aea10dfc63b1f4b7cd8f8deb58a62
0001-stp-rt-locking-helper-api.patch
0002-stp-rt-replace-__stp_tf_map_lock-rd-wr-lock-with-stp.patch
0003-stp-rt-replace-__stp_tf_vma_lock-rd-wr-lock-with-stp.patch
0004-stp-rt-replace-utrace-lock-with-stp-style-raw-lock.patch
0005-stp-rt-replace-utrace_struct-lock-to-stp-style-raw-l.patch
0006-stp-rt-replace-__stp_tf_task_work_list_lock-to-stp-r.patch
0007-stp-rt-replace-addr_map_lock-rd-wr-lock-with-stp-typ.patch
0008-stp-rt-replace-stp_print-lock-with-stp-style-lock.patch
0009-stp-rt-fix-preemptible-bug_on-for-STAT_GET_CPU.patch

I backported this patches to the RHEL6 system tap, and the error stopped to occur. Although some patches does not applies directly, they are easy to fix with small changes.