Description of problem:

When CPU isolation is used, such as via the cpu-partitioning tuned profile, it is possible for isolated CPUs to be interrupted via kernel IPIs initiated by non-isolated CPUs.  There are many different ways that this can happen but a few have been diagnosed using the rt-trace-bpf tool:

caused by NetworkManager:
64359.052209596    NetworkManager       0    1405     smp_call_function_many_cond (cpu=0, func=do_kernel_range_flush)
        smp_call_function_many_cond+0x1
        smp_call_function+0x39
        on_each_cpu+0x2a
        flush_tlb_kernel_range+0x7b
        __purge_vmap_area_lazy+0x70
        _vm_unmap_aliases.part.42+0xdf
        change_page_attr_set_clr+0x16a
        set_memory_ro+0x26
        bpf_int_jit_compile+0x2f9
        bpf_prog_select_runtime+0xc6
        bpf_prepare_filter+0x523
        sk_attach_filter+0x13
        sock_setsockopt+0x92c
        __sys_setsockopt+0x16a
        __x64_sys_setsockopt+0x20
        do_syscall_64+0x87
        entry_SYSCALL_64_after_hwframe+0x65

caused by the mgag200 kernel module:
238903.096535737   kworker/0:1          0    88579    smp_call_function_many_cond (cpu=0, func=do_flush_tlb_all)
        smp_call_function_many_cond+0x1
        smp_call_function+0x39
        on_each_cpu+0x2a
        flush_tlb_kernel_range+0x48
        __purge_vmap_area_lazy+0x70
        free_vmap_area_noflush+0xf2
        remove_vm_area+0x93
        __vunmap+0x59
        drm_gem_shmem_vunmap+0x6d
        mgag200_handle_damage+0x62
        mgag200_simple_display_pipe_update+0x69
        drm_atomic_helper_commit_planes+0xb3
        drm_atomic_helper_commit_tail+0x26
        commit_tail+0xc6
        drm_atomic_helper_commit+0x103
        drm_atomic_helper_dirtyfb+0x20e
        drm_fb_helper_damage_work+0x228
        process_one_work+0x18f
        worker_thread+0x30
        kthread+0x15d
        ret_from_fork+0x1f

Tracing on the isolated CPUs shows preemptions such as this:

58118.769286 |   18)  <...>-128143  |               |  smp_call_function_interrupt() {
58118.769286 |   18)  <...>-128143  |               |    irq_enter() {
58118.769287 |   18)  <...>-128143  |   0.101 us    |      preempt_count_add();
58118.769288 |   18)  <...>-128143  |   0.968 us    |    }
58118.769288 |   18)  <...>-128143  |               |    generic_smp_call_function_single_interrupt() {
58118.769289 |   18)  <...>-128143  |               |      flush_smp_call_function_queue() {
58118.769289 |   18)  <...>-128143  |               |        do_flush_tlb_all() {
58118.769290 |   18)  <...>-128143  |   0.453 us    |          native_flush_tlb_global();
58118.769291 |   18)  <...>-128143  |   1.439 us    |        }
58118.769292 |   18)  <...>-128143  |   2.402 us    |      }
58118.769292 |   18)  <...>-128143  |   3.223 us    |    }
58118.769292 |   18)  <...>-128143  |               |    irq_exit() {
58118.769293 |   18)  <...>-128143  |   0.077 us    |      preempt_count_sub();
58118.769294 |   18)  <...>-128143  |   0.201 us    |      idle_cpu();
58118.769295 |   18)  <...>-128143  |               |      tick_nohz_irq_exit() {
58118.769295 |   18)  <...>-128143  |   0.164 us    |        ktime_get();
58118.769296 |   18)  <...>-128143  |               |        __tick_nohz_full_update_tick() {
58118.769296 |   18)  <...>-128143  |   0.079 us    |          check_tick_dependency();
58118.769297 |   18)  <...>-128143  |   0.074 us    |          check_tick_dependency();
58118.769298 |   18)  <...>-128143  |   0.070 us    |          check_tick_dependency();
58118.769299 |   18)  <...>-128143  |   0.101 us    |          check_tick_dependency();
58118.769300 |   18)  <...>-128143  |   1.458 us    |          tick_nohz_next_event();
58118.769302 |   18)  <...>-128143  |   0.082 us    |          tick_nohz_stop_tick();
58118.769303 |   18)  <...>-128143  |   6.229 us    |        }
58118.769303 |   18)  <...>-128143  |   8.124 us    |      }
58118.769303 |   18)  <...>-128143  | + 10.872 us   |    }
58118.769304 |   18)  <...>-128143  | + 17.471 us   |  }




Version-Release number of selected component (if applicable):

4.18.0-348.12.2.rt7.143.el8_5.x86_64


How reproducible:

Easily


Steps to Reproduce:
1.  Boot the system using an RT kernel and the cpu-partitioning tuned profile
2.  Run a workload that measures latency, such as oslat, on the isolated CPUs
3.  Trace the kernel activity on the isolated CPUs while the workload is running


Actual results:

Latency spikes caused by IPI processing will be observed on the isolated CPUs when there is no need to handle the IPI at that moment.


Expected results:

No needless IPI processing should occur on the isolated CPUs -- for example, for a 100% userspace workload such as oslat there is no need to enter the kernel and service the IPI until a necessary kernel entry occurs (ie. system call, timer interrupt, etc.).


Additional info: