Bug 2096536
| Summary: | stap --remote libvirt:// does not work as expected | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 9 | Reporter: | XinhuaLi <xili> |
| Component: | systemtap | Assignee: | Frank Ch. Eigler <fche> |
| systemtap sub component: | system-version | QA Contact: | qe-baseos-tools-bugs |
| Status: | CLOSED MIGRATED | Docs Contact: | |
| Severity: | unspecified | ||
| Priority: | unspecified | CC: | lberk, mcermak, mjw, scox, wcohen |
| Version: | 9.0 | Keywords: | MigratedToJIRA, Triaged |
| Target Milestone: | rc | Flags: | pm-rhel:
mirror+
|
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-09-22 16:00:21 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
This bug is filed against gcc-toolset-11. For GTS, this functionality was dropped per bz1965277. So for GTS, this is not expected to work. However, I've tested this using systemtap-4.7-1.el8 and systemtap-4.7-2.el9 (in the host role) and in both cases I've reproduced the problem. stap command pipes to a stapvirt command that is the libvirt interface. The remote is running a stapsh tool that receives the requests. The initial commands are sent and received okay (strace output on remote) butthe stap kernel module is not received. stapvirt seems has a quite similar layout to virsh-console. It sets up callbacks for stdin, stdout, stream. All is fine for short packets: 1. Send file command to remote, stdin_event gets event stdin_event:1000 fd 0 events 0x1 stbuf_off 0 stbuf 0x7ffd0e99f848 termbuf_off 58 buf "file 926928 stap_1f18b4b" 2. stream_event gets event stream_event:1067 events 0x2 stbuf_off 0 stbuf 0x7ffd0e99f848 buf "file 926928 stap_1f18b4b" but stream_event is not triggered when kernel module is being sent in chunks 3. stdin_event reads first chunk of kernel module (which does virStreamEventUpdateCallback when it notices buffer contents) stdin_event:1000 fd 0 events 0x1 stbuf_off 0 stbuf 0x7ffd0e99f848 termbuf_off 1024 buf "^?ELF^B^A^A" 4. but stream_event is never triggered The libvirt event loop is using the glib event loop mechanism. Somehow the stream_event is not getting invoked. Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug. This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there. Due to differences in account names between systems, some fields were not replicated. Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information. To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer. You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like: "Bugzilla Bug" = 1234567 In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information. |
Description of problem: Execute from HOST Works well if we use ssh connection # stap --remote-prefix --remote root.100.30 -ve 'probe begin { printf("Hello from VM3-RHEL9-DEV!\n") }' Pass 1: parsed user script and 496 library scripts using 387992virt/148384res/16740shr/131468data kb, in 430usr/50sys/484real ms. Pass 2: analyzed script: 1 probe, 0 functions, 0 embeds, 0 globals using 392216virt/153116res/17004shr/135692data kb, in 30usr/0sys/32real ms. Pass 3: using cached /root/.systemtap/cache/f2/stap_f2f4984a8595cb3047ff21e278bfd589_877.c Pass 4: using cached /root/.systemtap/cache/f2/stap_f2f4984a8595cb3047ff21e278bfd589_877.ko Pass 5: starting run. 0: Hello from VM3-RHEL9-DEV! <<== WE CAN SEE HELLO ^CPass 5: run completed in 20usr/0sys/3359real ms. Can not work is we use libvirt # stap --remote-prefix --remote libvirt://VM3-RHEL9-DEV -ve 'probe begin { printf("Hello from VM3-RHEL9-DEV!\n") }' Pass 1: parsed user script and 496 library scripts using 388028virt/148540res/16860shr/131504data kb, in 420usr/60sys/486real ms. Pass 2: analyzed script: 1 probe, 0 functions, 0 embeds, 0 globals using 392252virt/153212res/17060shr/135728data kb, in 40usr/10sys/33real ms. Pass 3: using cached /root/.systemtap/cache/f2/stap_f2f4984a8595cb3047ff21e278bfd589_877.c Pass 4: using cached /root/.systemtap/cache/f2/stap_f2f4984a8595cb3047ff21e278bfd589_877.ko Pass 5: starting run. <<== NOTHING HAPPEN WE CAN NOT SEE HELLO Version-Release number of selected component (if applicable): HOST INFO systemtap-runtime-virthost-4.6-12.el9_0.x86_64 systemtap-sdt-devel-4.6-12.el9_0.x86_64 systemtap-runtime-4.6-12.el9_0.x86_64 systemtap-client-4.6-12.el9_0.x86_64 systemtap-devel-4.6-12.el9_0.x86_64 systemtap-4.6-12.el9_0.x86_64 systemtap-server-4.6-12.el9_0.x86_64 libvirt-libs-8.0.0-8.el9_0.x86_64 libvirt-glib-4.0.0-3.el9.x86_64 libvirt-dbus-1.4.1-5.el9.x86_64 libvirt-client-8.0.0-8.el9_0.x86_64 python3-libvirt-8.0.0-1.el9.x86_64 libvirt-daemon-8.0.0-8.el9_0.x86_64 libvirt-daemon-driver-storage-core-8.0.0-8.el9_0.x86_64 libvirt-daemon-driver-network-8.0.0-8.el9_0.x86_64 libvirt-daemon-driver-nwfilter-8.0.0-8.el9_0.x86_64 libvirt-daemon-config-nwfilter-8.0.0-8.el9_0.x86_64 libvirt-daemon-config-network-8.0.0-8.el9_0.x86_64 libvirt-daemon-driver-storage-disk-8.0.0-8.el9_0.x86_64 libvirt-daemon-driver-storage-mpath-8.0.0-8.el9_0.x86_64 libvirt-daemon-driver-storage-scsi-8.0.0-8.el9_0.x86_64 libvirt-daemon-driver-storage-iscsi-8.0.0-8.el9_0.x86_64 libvirt-daemon-driver-storage-rbd-8.0.0-8.el9_0.x86_64 libvirt-daemon-driver-storage-logical-8.0.0-8.el9_0.x86_64 libvirt-daemon-driver-storage-8.0.0-8.el9_0.x86_64 libvirt-daemon-driver-interface-8.0.0-8.el9_0.x86_64 libvirt-daemon-driver-qemu-8.0.0-8.el9_0.x86_64 libvirt-daemon-driver-secret-8.0.0-8.el9_0.x86_64 libvirt-daemon-driver-nodedev-8.0.0-8.el9_0.x86_64 libvirt-8.0.0-8.el9_0.x86_64 VM3-RHEL9-DEV INFO systemtap-runtime-4.6-12.el9_0.x86_64 systemtap-client-4.6-12.el9_0.x86_64 systemtap-devel-4.6-12.el9_0.x86_64 systemtap-4.6-12.el9_0.x86_64 systemtap-server-4.6-12.el9_0.x86_64 systemtap-runtime-virtguest-4.6-12.el9_0.x86_64 systemtap-initscript-4.6-12.el9_0.x86_64 systemtap-runtime-virthost-4.6-12.el9_0.x86_64 systemtap-debugsource-4.6-12.el9_0.x86_64 systemtap-debuginfo-4.6-12.el9_0.x86_64 systemtap-runtime-debuginfo-4.6-12.el9_0.x86_64 How reproducible: Execute command Can not work is we use libvirt # stap --remote-prefix --remote libvirt://VM3-RHEL9-DEV -ve 'probe begin { printf("Hello from VM3-RHEL9-DEV!\n") }' Pass 1: parsed user script and 496 library scripts using 388028virt/148540res/16860shr/131504data kb, in 420usr/60sys/486real ms. Pass 2: analyzed script: 1 probe, 0 functions, 0 embeds, 0 globals using 392252virt/153212res/17060shr/135728data kb, in 40usr/10sys/33real ms. Pass 3: using cached /root/.systemtap/cache/f2/stap_f2f4984a8595cb3047ff21e278bfd589_877.c Pass 4: using cached /root/.systemtap/cache/f2/stap_f2f4984a8595cb3047ff21e278bfd589_877.ko Pass 5: starting run. <<== NOTHING HAPPEN WE CAN NOT SEE HELLO Steps to Reproduce: 1. 2. 3. Actual results: Expected results: It should work if use libvirt connection and we should see the output Additional info: Execute from HOST Works well if we use ssh connection # stap --remote-prefix --remote root.100.30 -ve 'probe begin { printf("Hello from VM3-RHEL9-DEV!\n") }' Pass 1: parsed user script and 496 library scripts using 387992virt/148384res/16740shr/131468data kb, in 430usr/50sys/484real ms. Pass 2: analyzed script: 1 probe, 0 functions, 0 embeds, 0 globals using 392216virt/153116res/17004shr/135692data kb, in 30usr/0sys/32real ms. Pass 3: using cached /root/.systemtap/cache/f2/stap_f2f4984a8595cb3047ff21e278bfd589_877.c Pass 4: using cached /root/.systemtap/cache/f2/stap_f2f4984a8595cb3047ff21e278bfd589_877.ko Pass 5: starting run. 0: Hello from VM3-RHEL9-DEV! <<== WE CAN SEE HELLO ^CPass 5: run completed in 20usr/0sys/3359real ms. Can not work is we use libvirt # stap --remote-prefix --remote libvirt://VM3-RHEL9-DEV -ve 'probe begin { printf("Hello from VM3-RHEL9-DEV!\n") }' Pass 1: parsed user script and 496 library scripts using 388028virt/148540res/16860shr/131504data kb, in 420usr/60sys/486real ms. Pass 2: analyzed script: 1 probe, 0 functions, 0 embeds, 0 globals using 392252virt/153212res/17060shr/135728data kb, in 40usr/10sys/33real ms. Pass 3: using cached /root/.systemtap/cache/f2/stap_f2f4984a8595cb3047ff21e278bfd589_877.c Pass 4: using cached /root/.systemtap/cache/f2/stap_f2f4984a8595cb3047ff21e278bfd589_877.ko Pass 5: starting run. <<== NOTHING HAPPEN WE CAN NOT SEE HELLO stapvirt info # stapvirt query VM3-RHEL9-DEV Name: VM3-RHEL9-DEV UUID: a719c10c-0b9b-4e3d-bfa8-af5bee183e44 State: running ID: 63 Type: persistent Permanent Ports: 2 Hotplugging: supported # stapvirt port-list VM3-RHEL9-DEV /var/lib/libvirt/qemu/VM3-RHEL9-DEV.org.systemtap.stapsh.0.sock /var/lib/libvirt/qemu/VM3-RHEL9-DEV.org.systemtap.stapsh.1.sock On VM3-RHEL9-DEV # systemctl status stapsh.stapsh.0.service ● stapsh.stapsh.0.service - SystemTap stapsh on org.systemtap.stapsh.0 Loaded: loaded (/usr/lib/systemd/system/stapsh@.service; static) Active: active (running) since Tue 2022-06-14 14:41:10 CST; 52min ago Docs: man:stap man:stapsh Main PID: 73714 (stapsh) Tasks: 1 (limit: 20701) Memory: 176.0K CPU: 30ms CGroup: /system.slice/system-stapsh.slice/stapsh.stapsh.0.service └─73714 /usr/bin/stapsh -l /dev/virtio-ports/org.systemtap.stapsh.0 Jun 14 14:41:10 RHEL9DEV systemd[1]: Started SystemTap stapsh on org.systemtap.stapsh.0. #lspci |grep console 09:00.0 Communication controller: Red Hat, Inc. Virtio console (rev 01)