Description of problem: During kernel-2.6.18-340.el5 kerneltier1 tests all tracepoints operational tests for xen kernels have failed. An example test is: https://beaker.engineering.redhat.com/recipes/645126#task8065613 An example output for from probing a tracepoint is : __stp_utrace_task_finder_target_syscall_entry __stp_utrace_task_finder_target_syscall_exit stap_start_task_finder stap_stop_task_finder __stp_tf_vma_new_entry __stp_tf_vma_release_entry stap_initialize_vma_map stap_destroy_vma_map __stp_tf_vma_map_hash __stp_tf_get_vma_map_entry_internal __stp_tf_get_vma_map_entry_end_internal stap_add_vma_map_info stap_extend_vma_map_info stap_remove_vma_map_info stap_find_vma_map_info stap_find_vma_map_info_user stap_drop_vma_maps _stp_vma_match_vdso _stp_vma_exec_cb _stp_vma_mmap_cb _stp_vma_munmap_cb _stp_vma_init _stp_vma_done _stp_kmodule_relocate _stp_umodule_relocate _stp_kmod_sec_lookup _stp_umod_lookup _stp_kallsyms_lookup _stp_build_id_check _stp_module_check _stp_kmodule_check _stp_usermodule_check _stp_snprint_addr _stp_print_addr _stp_kmodule_update_address upper_bound lookup_addr_aux lookup_bad_addr add_bad_addr_entry delete_bad_addr_entry _stp_stat_calc_buckets needed_space _stp_bucket_to_val _stp_val_to_bucket _stp_stat_print_histogram_buf _stp_stat_print_histogram __stp_stat_add _stp_stat_init _stp_stat_del _stp_stat_add _stp_stat_get_cpu _stp_stat_clear_data _stp_stat_get _stp_stat_clear param_set_int64_t param_get_int64_t init_module cleanup_module nd_set_link nd_get_link poll_wait init_poll_funcptr get_fd_set set_fd_set zero_fd_set relay_write __relay_write relay_reserve subbuf_start_reserve __stp_relay_switch_subbuf __stp_relay_wakeup_readers __stp_relay_wakeup_timer __stp_relay_timer_init _stp_transport_get_state _stp_transport_data_fs_overwrite __stp_relay_dropped_open __stp_relay_dropped_read __stp_relay_subbuf_start_callback __stp_relay_remove_buf_file_callback __stp_relay_create_buf_file_callback _stp_transport_data_fs_start _stp_transport_data_fs_stop _stp_transport_data_fs_close _stp_transport_data_fs_init _stp_data_write_reserve _stp_data_entry_data _stp_data_write_commit _stp_ctl_write_fs _stp_register_ctl_channel_fs _stp_unregister_ctl_channel_fs _stp_mempool_destroy _stp_mempool_init _stp_mempool_alloc _stp_mempool_free _stp_do_relocation _stp_module_notifier _stp_ctl_write_cmd _stp_ctl_alloc_special_buffers _stp_ctl_get_buffer _stp_ctl_free_buffer _stp_ctl_send _stp_ctl_send_notify _stp_ctl_read_cmd _stp_ctl_open_cmd _stp_ctl_close_cmd _stp_ctl_poll_cmd _stp_register_ctl_channel _stp_unregister_ctl_channel __check__stp_bufsize _stp_handle_start In file included from /usr/share/systemtap/runtime/print.c:17, from /usr/share/systemtap/runtime/runtime_context.h:16, from /tmp/stapIoHPcy/stap_ae5d188fc80435b7416b109bf50bd50b_678_src.c:51: /usr/share/systemtap/runtime/transport/transport.c: In function ‘_stp_handle_start’: /usr/share/systemtap/runtime/transport/transport.c:115: error: implicit declaration of function ‘alloc_vm_area’ cc1: warnings being treated as errors /usr/share/systemtap/runtime/transport/transport.c:115: warning: assignment makes pointer from integer without a cast /usr/share/systemtap/runtime/transport/transport.c:117: error: implicit declaration of function ‘free_vm_area’ _stp_cleanup_and_exit _stp_request_exit _stp_detach _stp_attach _stp_ctl_work_callback _stp_transport_close _stp_transport_init _stp_lock_inode _stp_unlock_inode _stp_lock_transport_dir _stp_unlock_transport_dir _stp_get_root_dir _stp_remove_root_dir _stp_get_module_dir _stp_transport_fs_init _stp_transport_fs_close _stp_handle_tzinfo _stp_handle_privilege_credentials _stp_handle_remote_id skip_atoi number number_size _stp_vsprint_char _stp_vsprint_char_size _stp_vsprint_memory _stp_vsprint_memory_size check_binary_precision _stp_vsprint_binary _stp_vsprint_binary_size _stp_vsnprintf _stp_print_init _stp_print_cleanup stp_print_flush _stp_print_flush _stp_reserve_bytes _stp_unreserve_bytes _stp_print_binary _stp_printf _stp_print _stp_print_char _stp_print_kernel_info _stp_vlog _stp_warn _stp_exit _stp_error _stp_softerror _stp_dbug stp_unlock_probe stp_lock_probe function_exit probe_2023 enter_real_tracepoint_probe_0 systemtap_module_init systemtap_module_refresh systemtap_module_exit Execution times (seconds) preprocessing : 0.11 (41%) usr 0.08 (30%) sys 0.17 (31%) wall 2346 kB (23%) ggc lexical analysis : 0.04 (15%) usr 0.10 (37%) sys 0.13 (24%) wall 0 kB ( 0%) ggc parser : 0.08 (30%) usr 0.08 (30%) sys 0.22 (40%) wall 5938 kB (58%) ggc tree gimplify : 0.04 (15%) usr 0.01 ( 4%) sys 0.02 ( 4%) wall 1365 kB (13%) ggc TOTAL : 0.27 0.27 0.55 10325 kB make[2]: *** [/tmp/stapIoHPcy/stap_ae5d188fc80435b7416b109bf50bd50b_678_src.o] Error 1 make[1]: *** [_module_/tmp/stapIoHPcy] Error 2 Spawn waitpid result (0x200): 2 WARNING: kbuild exited with status: 2 Pass 4: compiled C into "stap_ae5d188fc80435b7416b109bf50bd50b_678.ko" in 2580usr/1180sys/4110real ms. Pass 4: compilation failed. Try again with another '--vp 0001' option. Running rm -rf /tmp/stapIoHPcy Spawn waitpid result (0x0): 0 Removed temporary directory "/tmp/stapIoHPcy" Version-Release number of selected component (if applicable): ***** Current Running Kernel Package = kernel-xen-2.6.18-340.el5.i686 ***** ***** Installed systemtap version = systemtap-client-1.8-5.el5.i386 systemtap-devel-1.8-5.el5.i386 ***** ***** Current Running Distro = Red Hat Enterprise Linux Server release 5.9 Beta (Tikanga) ***** How reproducible: Very Steps to Reproduce: 1. Install and boot into kernel-2.6.18-340.el5 xen kernel 2. stap -c "sleep 1" -vvvve 'probe kernel.trace("activate_task") { exit() }' 3. (you can replace activate_task with any other function to trace Actual results: Expected results: Additional info:
The cause appears to be an unexpected interaction between an old kernel, a younger kernel's mm bug, and systemtap's workaround for the latter, and a later cleanup of this workaround. This problem does not indicate a problem with this xen kernel build. For http://sourceware.org/bugzilla/show_bug.cgi?id=9740, it was reported in 2009 that kernel 2.6.29 had a mm bug that could be worked around by a dummy call to alloc_vm_area / free_vm_area. This dummy call was activated by a stap runtime autoconf test that would trigger this only if the respective functions could be found & compiled. Forward two years. commit 08c29612 converted the compilability test (runtime/autoconf-vm-area.c) to a kernel-exported-symbol test, that checked instead whether the given symbols were SYMBOL_EXPORT'd by the kernel. This is approximately the same test for modern kernels. However, for this 2.6.18 xen kernel, it is a false positive. It does export alloc_vm_area, but does not define it in linux/vmalloc.h (where expected), and since the new test doesn't include compilability, stap's runtime assumes necessity & availability. However, xen includes alloc_vm_area only in some top secret header file xen/driver_util.h. That in turn leads to the compilation failure reported. This commit was included in systemtap 1.7 and past. One possible cure would be to make the whole PR9740 workaround conditional on a fairly narrow band of kernel versions, around 2.6.29 +- a few.
Patch coming soon for upstream http://sourceware.org/bugzilla/show_bug.cgi?id=14611
This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux release for currently deployed products. This request is not yet committed for inclusion in a release.
Jeff, just waiting on qa-ack.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2013-0058.html