Created attachment 907503 [details] logs_rhevh Description of problem: After loading dummy module it is not possible to unload it properly on RHEV-H. On RHEL it works just fine (see attached logs). This represents a problem for automated tests, where we need to work with more network interfaces than the machine physically has. Test was conducted on freshly installed hosts [root@localhost ~]# cat /etc/redhat-release Red Hat Enterprise Virtualization Hypervisor release 6.5 (20140603.2.el6ev) #no dummmy module loaded after boot [root@localhost ~]# lsmod | grep --color -i dummy #no dummy interface yet [root@localhost ~]# ip a l 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:1a:4a:51:b2:87 brd ff:ff:ff:ff:ff:ff inet 10.34.66.40/24 brd 10.34.66.255 scope global eth0 inet6 2620:52:0:2242:21a:4aff:fe51:b287/64 scope global dynamic valid_lft 2591799sec preferred_lft 604599sec inet6 fe80::21a:4aff:fe51:b287/64 scope link valid_lft forever preferred_lft forever #load dummy module [root@localhost ~]# modprobe dummy numdummies=2 #module loaded [root@localhost ~]# lsmod | grep --color -i dummy dummy 2101 0 #dummy interfaces visible [root@localhost ~]# ip a l 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:1a:4a:51:b2:87 brd ff:ff:ff:ff:ff:ff inet 10.34.66.40/24 brd 10.34.66.255 scope global eth0 inet6 2620:52:0:2242:21a:4aff:fe51:b287/64 scope global dynamic valid_lft 2591758sec preferred_lft 604558sec inet6 fe80::21a:4aff:fe51:b287/64 scope link valid_lft forever preferred_lft forever 4: dummy0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN link/ether fa:5d:e8:a8:d9:68 brd ff:ff:ff:ff:ff:ff 5: dummy1: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN link/ether 8a:d6:e3:52:78:46 brd ff:ff:ff:ff:ff:ff #try to unload dummy module [root@localhost ~]# modprobe -r dummy #did not work [root@localhost ~]# lsmod | grep --color -i dummy dummy 2101 0 #all dummy interfaces removed except for dummy0 [root@localhost ~]# ip a l 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:1a:4a:51:b2:87 brd ff:ff:ff:ff:ff:ff inet 10.34.66.40/24 brd 10.34.66.255 scope global eth0 inet6 2620:52:0:2242:21a:4aff:fe51:b287/64 scope global dynamic valid_lft 2591738sec preferred_lft 604538sec inet6 fe80::21a:4aff:fe51:b287/64 scope link valid_lft forever preferred_lft forever 6: dummy0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN link/ether 72:fc:39:c7:36:51 brd ff:ff:ff:ff:ff:ff How reproducible: 100% Actual results: on unloading dummy module dummy0 interfaces still remains in system and prevents dummy module to be unloaded Expected results: on unloading dummy module, all dummy interfaces are gone same way as on RHEL Additional info: workaround is available if you remove dummy device using IP command [root@localhost ~]# ip link del dev dummy0 dummy module can be unloaded [root@localhost ~]# modprobe -r dummy [root@localhost ~]# lsmod | grep --color -i dummy
Created attachment 907504 [details] logs_rhel
Could it be that dummy0 is used by something (bridge/bond)? Do dmesg or /var/log/messages give a clue on why `modprobe -r` failed? On another matter, the test could iterate over all dummy* device and delete them, instead of pulling the kernel module under their feet.
It seems that what happens is that dummy module gets removed (modprobe -r does not really fail), but immediately after rmmod happens it is loaded again and dummy0 is created. So no matter how many interfaces you've created before, only dummy0 remains after modprobe -r dummy. This appears in messages after unloading the module Jun 12 06:44:47 localhost kernel: Loading kernel module for a network device with CAP_SYS_MODULE (deprecated). Use CAP_NET_ADMIN and alias netdev-dummy0 instead (should I open the bug about the CAP_SYS_MODULE (deprecated) ??) Dummy0 should not be used by anything else since it is non-existent after the boot and it is just created and removed. It is possible that modprobe manipulates it somehow since there are rules for dummy{0,1} [root@localhost ~]# cat /etc/modprobe.d/dist.conf | grep dummy alias dummy0 dummy alias dummy1 dummy install dummy0 /sbin/modprobe -o dummy0 --ignore-install dummy install dummy1 /sbin/modprobe -o dummy1 --ignore-install dummy but same rules are in RHEL where it works [root@dell-r210ii-07 ~]# cat /etc/modprobe.d/dist.conf | grep dummy alias dummy0 dummy alias dummy1 dummy install dummy0 /sbin/modprobe -o dummy0 --ignore-install dummy install dummy1 /sbin/modprobe -o dummy1 --ignore-install dummy All in all I can handle it in the test, but I think it should properly, not with workarounds.
It is interesting, in a way, to find the difference between rhev-h and rhel in this regard. But I do not see it as an important question, as it does not affect rhev functionality.
(In reply to Martin Pavlik from comment #3) > It seems that what happens is that dummy module gets removed (modprobe -r > does not really fail), but immediately after rmmod happens it is loaded > again and dummy0 is created. So no matter how many interfaces you've created > before, only dummy0 remains after modprobe -r dummy. > > This appears in messages after unloading the module > > Jun 12 06:44:47 localhost kernel: Loading kernel module for a network device > with CAP_SYS_MODULE (deprecated). Use CAP_NET_ADMIN and alias netdev-dummy0 > instead > > (should I open the bug about the CAP_SYS_MODULE (deprecated) ??) > > Dummy0 should not be used by anything else since it is non-existent after > the boot and it is just created and removed. It is possible that modprobe > manipulates it somehow since there are rules for dummy{0,1} > > [root@localhost ~]# cat /etc/modprobe.d/dist.conf | grep dummy > alias dummy0 dummy > alias dummy1 dummy > install dummy0 /sbin/modprobe -o dummy0 --ignore-install dummy > install dummy1 /sbin/modprobe -o dummy1 --ignore-install dummy > > but same rules are in RHEL where it works > > [root@dell-r210ii-07 ~]# cat /etc/modprobe.d/dist.conf | grep dummy > alias dummy0 dummy > alias dummy1 dummy > install dummy0 /sbin/modprobe -o dummy0 --ignore-install dummy > install dummy1 /sbin/modprobe -o dummy1 --ignore-install dummy > > > All in all I can handle it in the test, but I think it should properly, not > with workarounds. # modinfo dummy filename: /lib/modules/2.6.32-431.20.5.el6.x86_64/kernel/drivers/net/dummy.ko alias: rtnl-link-dummy license: GPL srcversion: 578D860A932A87B109C715A depends: vermagic: 2.6.32-431.20.5.el6.x86_64 SMP mod_unload modversions parm: numdummies:Number of dummy pseudo devices (int) What happens if you change in /etc/modprobe.conf/dist.conf to the alias of dummy module name? alias rtnl-link-dummy0 dummy alias rtnl-link-dummy1 dummy instead of: alias dummy0 dummy alias dummy1 dummy
Seems to work #config BEFORE [root@localhost ~]# grep dummy /etc/modprobe.d/dist.conf alias dummy0 dummy alias dummy1 dummy install dummy0 /sbin/modprobe -o dummy0 --ignore-install dummy install dummy1 /sbin/modprobe -o dummy1 --ignore-install dummy #config AFTER [root@localhost ~]# grep dummy /etc/modprobe.d/dist.conf alias rtnl-link-dummy0 dummy alias rtnl-link-dummy1 dummy install dummy0 /sbin/modprobe -o dummy0 --ignore-install dummy install dummy1 /sbin/modprobe -o dummy1 --ignore-install dummy [root@localhost ~]# modprobe dummy numdummies=2 [root@localhost ~]# lsmod | grep --color -i dummy dummy 2101 0 [root@localhost ~]# modprobe -r dummy [root@localhost ~]# lsmod | grep --color -i dummy #returned empty output dummy interfaces were removed
(In reply to Martin Pavlik from comment #6) > Seems to work > > #config BEFORE > [root@localhost ~]# grep dummy /etc/modprobe.d/dist.conf > alias dummy0 dummy > alias dummy1 dummy > install dummy0 /sbin/modprobe -o dummy0 --ignore-install dummy > install dummy1 /sbin/modprobe -o dummy1 --ignore-install dummy > > #config AFTER > [root@localhost ~]# grep dummy /etc/modprobe.d/dist.conf > alias rtnl-link-dummy0 dummy > alias rtnl-link-dummy1 dummy > install dummy0 /sbin/modprobe -o dummy0 --ignore-install dummy > install dummy1 /sbin/modprobe -o dummy1 --ignore-install dummy > > [root@localhost ~]# modprobe dummy numdummies=2 > > [root@localhost ~]# lsmod | grep --color -i dummy > dummy 2101 0 > > > [root@localhost ~]# modprobe -r dummy > > [root@localhost ~]# lsmod | grep --color -i dummy > #returned empty output > > dummy interfaces were removed Thanks for the feedback. I have discussed with David Shea about this report and looks like we still have something triggering the dummy module even after the modprobe -r. Configured dist.conf with logger: ===================================== alias dummy0 dummy alias dummy1 dummy install dummy0 /bin/logger 'Loading dummy module via dummy0'; /sbin/modprobe -o dummy0 --ignore-install dummy install dummy1 /bin/logger 'Loading dummy module via dummy1'; /sbin/modprobe -o dummy1 --ignore-install dummy install dummy /bin/logger 'Loading dummy module via dummy'; /sbin/modprobe --ignore-install dummy # modprobe dummy # modprobe -r dummy # tail -f /var/log/messages Aug 5 21:18:33 localhost root: Loading dummy module via dummy Aug 5 21:18:37 localhost root: Unloading dummy module via dummy Aug 5 21:18:37 localhost logger: Loading dummy module via dummy ^^^^ auto loading Aug 5 21:18:37 localhost kernel: Loading kernel module for a network device with CAP_SYS_MODULE (deprecated). Use CAP_NET_ADMIN and alias netdev-dummy0 instead More investigation needed.
Hi Harald, Do you mind to take a look into this bug? Could it be related to udev/dracut? Doing the alias as comment#5 made the trick. Thanks!
I cannot reproduce this. # fgrep dummy /etc/modprobe.d/dist.conf alias dummy0 dummy alias dummy1 dummy install dummy0 /sbin/modprobe -o dummy0 --ignore-install dummy install dummy1 /sbin/modprobe -o dummy1 --ignore-install dummy # modprobe dummy numdummies=2 # lsmod | grep --color -i dummy dummy 1433 0 # ip a l … 4: dummy0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN link/ether 8a:ed:ab:98:cd:30 brd ff:ff:ff:ff:ff:ff 5: dummy1: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN link/ether 4e:1f:81:b3:09:e9 brd ff:ff:ff:ff:ff:ff … # modprobe -r dummy # lsmod | grep --color -i dummy # ip a l ... cleanly removed. Maybe you have a rogue udev rule or some strange hotplug scripts. Please try to run # udevadm control --log-priority=debug and in another console: # udevadm monitor Attach the output of the monitor, the udevd output in /var/log/messages and the output of: # modprobe dummy numdummies=2 # for i in \ /devices/virtual/net/dummy0/queues/rx-0 \ /devices/virtual/net/dummy0/queues/tx-0 \ /devices/virtual/net/dummy0 \ /module/dummy ; do udevadm test --action="remove" "$i"; done # modprobe -r dummy
@Harald on which RHEV-H build are you trying? Maybe it got fixed in newer RHEV-H builds.
(In reply to Martin Pavlik from comment #10) > @Harald > > on which RHEV-H build are you trying? Maybe it got fixed in newer RHEV-H > builds. Tried on an old RHEL-6
(In reply to Harald Hoyer from comment #11) > (In reply to Martin Pavlik from comment #10) > > @Harald > > > > on which RHEV-H build are you trying? Maybe it got fixed in newer RHEV-H > > builds. > > Tried on an old RHEL-6 It doesn't happen on RHEL or RHEV-H 7, only on RHEV-H 6. Do you mind to try: rhev-hypervisor6-6.6-20141218.0 https://brewweb.devel.redhat.com/buildinfo?buildID=404971 After install, use admin user and pass provided in the install. After that, press F2 to go to shell and execute: # modprobe dummy numdummies=2 # lsmod | grep -i dummy # ip l # modprobe -r dummy # lsmod | grep -i dummy # ip l Thanks!
I am closing this bug as this report is not affecting the RHEV-H and it's not supported action as we don't provide such tasks via TUI. For the record, this load/remove dummy modules worked in RHEV-H7 from my tests.