Bug 484990

Summary: net-snmp reports misleading interface information - missing peth0 and dup eth0
Product: Red Hat Enterprise Linux 5 Reporter: Bryan Mason <bmason>
Component: xenAssignee: Michal Novotny <minovotn>
Status: CLOSED NOTABUG QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 5.3CC: areis, gasmith, jdenemar, martin.wilck, minovotn, tao, xen-maint
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-11-24 11:56:56 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 499522    
Attachments:
Description Flags
The "patch" applied by us
none
My snmpd config file
none
Patch to refresh interface information in snmpd none

Description Bryan Mason 2009-02-10 23:53:55 UTC
Description of problem:

    In a Xen bridging setup, net-snmp reports wrong information about
    network interfaces.

Version-Release number of selected component (if applicable):

    xen-3.0.3-80.el5.x86_64
    net-snmp-5.3.2.2-5.el5.x86_64

How reproducible:

    100%

Steps to Reproduce:

    1. Boot a system with xend and snmpd enabled
    2. run snmpwalk
  
Actual results:

    # snmpwalk -v2c -c local localhost ifDescr
    IF-MIB::ifDescr.1 = STRING: lo
    IF-MIB::ifDescr.2 = STRING: eth1
    IF-MIB::ifDescr.3 = STRING: eth0
    IF-MIB::ifDescr.4 = STRING: sit0
    IF-MIB::ifDescr.5 = STRING: virbr0
    IF-MIB::ifDescr.6 = STRING: vif0.0
    IF-MIB::ifDescr.7 = STRING: eth0
    IF-MIB::ifDescr.8 = STRING: vif0.1
    IF-MIB::ifDescr.9 = STRING: veth1
    IF-MIB::ifDescr.10 = STRING: vif0.2
    IF-MIB::ifDescr.11 = STRING: veth2
    IF-MIB::ifDescr.12 = STRING: vif0.3
    IF-MIB::ifDescr.13 = STRING: veth3
    IF-MIB::ifDescr.14 = STRING: vif0.4
    IF-MIB::ifDescr.15 = STRING: veth4
    IF-MIB::ifDescr.16 = STRING: xenbr0
    
    Note eth0 is listed twice in ifDescr.3 and ifDescr.7.

Expected results:

    # snmpwalk -v2c -c local localhost ifDescr
    IF-MIB::ifDescr.1 = STRING: lo
    IF-MIB::ifDescr.2 = STRING: eth1
    IF-MIB::ifDescr.3 = STRING: peth0
    IF-MIB::ifDescr.4 = STRING: sit0
    IF-MIB::ifDescr.5 = STRING: virbr0
    IF-MIB::ifDescr.6 = STRING: vif0.0
    IF-MIB::ifDescr.7 = STRING: eth0
    IF-MIB::ifDescr.8 = STRING: vif0.1
    IF-MIB::ifDescr.9 = STRING: veth1
    IF-MIB::ifDescr.10 = STRING: vif0.2
    IF-MIB::ifDescr.11 = STRING: veth2
    IF-MIB::ifDescr.12 = STRING: vif0.3
    IF-MIB::ifDescr.13 = STRING: veth3
    IF-MIB::ifDescr.14 = STRING: vif0.4
    IF-MIB::ifDescr.15 = STRING: veth4
    IF-MIB::ifDescr.16 = STRING: xenbr0

    Note ifDescr.3 is now correctly listed as "peth0".

Additional info:

    There seem two be several different ways to resolve this:

    1) Modify the snmpd (and possibly snmptrapd) init scripts to start
       snmpd (and snmptrapd) after xend starts.  Something like:


       # diff -Nau /etc/rc.d/init.d/snmpd /etc/rc.d/init.d/snmpd.new 
       --- /etc/rc.d/init.d/snmpd       2009-02-10 14:36:18.000000000 -0800
       +++ /etc/rc.d/init.d/snmpd.new   2009-02-10 14:28:20.000000000 -0800
       @@ -1,7 +1,7 @@
        #!/bin/bash
        # ucd-snmp init file for snmpd
        #
       -# chkconfig: - 50 50
       +# chkconfig: - 99 50
        # description: Simple Network Management Protocol (SNMP) Daemon
        #
        # processname: /usr/sbin/snmpd

    2) Add some code to the xend init script to restart snmpd if so
       configured:

       [root@why ~]# diff -Nau /etc/sysconfig/xend.original /etc/sysconfig/xend
       --- /etc/sysconfig/xend.original  2009-02-10 14:37:40.000000000 -0800
       +++ /etc/sysconfig/xend           2009-02-10 14:50:10.000000000 -0800
       @@ -12,3 +12,6 @@

        # Running xenstored on tmpfs can help performance
        #XENSTORED_TMPFS=yes
       +
       +# Restart snmpd after starting xend?
       +RESTART_SNMPD=yes

       [root@why init.d]# diff -Nau xend.original xend
       --- /etc/rc.d/init.d/xend.original  2009-02-10 14:36:43.000000000 -0800
       +++ /etc/rc.d/init.d/xend           2009-02-10 14:45:12.000000000 -0800
       @@ -75,6 +75,10 @@
    
           /usr/sbin/xend start
           await_daemons_up
       +
       +   if [ "$RESTART_SNMPD" = "yes" ]; then
       +       service snmpd condrestart
       +   fi
           ;;
       stop)
           eho -n $"Stopping $prog: "

    3) Add some udev rules to restart snmpd when interfaces change.
       It would seem that 60-net already has a rule to call
       /etc/sysconfig/network-scripts/net.hotplug whenever something
       changes in the "net" subsystem, so maybe just an addition to
       net.hotplug would do the trick.

Comment 2 Bryan Mason 2009-02-11 00:32:20 UTC
For what it's worth, I hacked /etc/sysconfig/network-scripts/net.hotplug to add the following line

    echo `date`: $ACTION $INTERFACE >> /tmp/net.hotplug.log

so I could see what kinds of things were happening in that file.  After a reboot, the following entries appear in the log:

    # cat /tmp/net.hotplug.log 
    Tue Feb 10 16:27:35 PST 2009: remove virbr0
    Tue Feb 10 16:29:17 PST 2009: add sit0
    Tue Feb 10 16:29:29 PST 2009: add virbr0
    Tue Feb 10 16:29:35 PST 2009: add vif0.0
    Tue Feb 10 16:29:35 PST 2009: add veth0
    Tue Feb 10 16:29:36 PST 2009: add vif0.2
    Tue Feb 10 16:29:36 PST 2009: add vif0.3
    Tue Feb 10 16:29:36 PST 2009: add veth3
    Tue Feb 10 16:29:36 PST 2009: add vif0.4
    Tue Feb 10 16:29:36 PST 2009: add veth4
    Tue Feb 10 16:29:36 PST 2009: add vif0.1
    Tue Feb 10 16:29:36 PST 2009: add veth2
    Tue Feb 10 16:29:36 PST 2009: add veth1
    Tue Feb 10 16:29:38 PST 2009: add xenbr0

Comment 3 Jan Safranek 2009-02-20 14:01:30 UTC
Net-snmp cannot handle renaming of network devices. Snmpd reads interface names only once and caches them. If the interface name changes, snmpd does not read the new one. It's known problem and upstream does not know how to solve it.

As a workaround, you can try to start xen service before snmpd. Restarting snmpd when any interface changes name is bad, snmpd caches lot of information internally, which gets lost after restart. Most of snmpd counters would report misleading values.

Comment 4 Martin Wilck 2009-02-26 12:45:50 UTC
From my issue tracker comment from Nov. 11 (IT 217289):


Thus, our RFE is as follows:

1) For the interface renaming problem:

   a) have snmpd handle interface renaming and renumbering cleanly, at least for the limited scope of Xen bridging
   b) if that can't be done (as we expect), implement a workaround similar to the attached "fsc-xen-snmpd-fix" package (restart snmpd after starting/stopping the Xen bridges).


Jan's statement basically says "a) is impossible".

Thus our RFE boils down to b). AFAICS, the suggestions in comment #1 would be a good starting point.

Comment 5 Martin Wilck 2009-02-26 12:51:11 UTC
Created attachment 333321 [details]
The "patch" applied by us

This is the solution that we have employed for our customers. It is attached to the IT problem already, but I'd like to discuss it here as well.

The idea is simply to restart snmpd after applying any changes with the xen network-bridge script. Similar in spirit to comment #1, but IMO a more general solution which works also if the user just starts/stops XEN network bridging.

Comment 10 Michal Novotny 2009-10-22 12:22:54 UTC
Well, I did try to reproduce it using:
xen-3.0.3-96.el5
net-snmp-5.3.2.2-7.el5_4.2

and I was unable to reproduce it. My community is public (not local) for testing and my network device is eth2 (not eth0)...

# snmpwalk -v2c -c public localhost ifDescr
IF-MIB::ifDescr.1 = STRING: lo                         
IF-MIB::ifDescr.2 = STRING: eth1                       
IF-MIB::ifDescr.3 = STRING: peth2                      
IF-MIB::ifDescr.4 = STRING: sit0
IF-MIB::ifDescr.5 = STRING: virbr0
IF-MIB::ifDescr.6 = STRING: vif0.0
IF-MIB::ifDescr.7 = STRING: veth0
IF-MIB::ifDescr.8 = STRING: vif0.1
IF-MIB::ifDescr.9 = STRING: veth1
IF-MIB::ifDescr.10 = STRING: vif0.2
IF-MIB::ifDescr.11 = STRING: eth2
IF-MIB::ifDescr.12 = STRING: vif0.3
IF-MIB::ifDescr.13 = STRING: veth3
IF-MIB::ifDescr.14 = STRING: vif0.4
IF-MIB::ifDescr.15 = STRING: veth4
IF-MIB::ifDescr.16 = STRING: xenbr0
# service xend restart
restart xend:                                              [  OK  ]
# snmpwalk -v2c -c public localhost ifDescr
IF-MIB::ifDescr.1 = STRING: lo
IF-MIB::ifDescr.2 = STRING: eth1
IF-MIB::ifDescr.3 = STRING: peth2
IF-MIB::ifDescr.4 = STRING: sit0
IF-MIB::ifDescr.5 = STRING: virbr0
IF-MIB::ifDescr.6 = STRING: vif0.0
IF-MIB::ifDescr.7 = STRING: veth0
IF-MIB::ifDescr.8 = STRING: vif0.1
IF-MIB::ifDescr.9 = STRING: veth1
IF-MIB::ifDescr.10 = STRING: vif0.2
IF-MIB::ifDescr.11 = STRING: eth2
IF-MIB::ifDescr.12 = STRING: vif0.3
IF-MIB::ifDescr.13 = STRING: veth3
IF-MIB::ifDescr.14 = STRING: vif0.4
IF-MIB::ifDescr.15 = STRING: veth4
IF-MIB::ifDescr.16 = STRING: xenbr0
#

As you can see there is no dup eth2 and peth2 persists here so I can find no issue here. The package version used can be available by yum in RHEL 5.4. Also if there is newer version of xen package, it is usually available at http://people.redhat.com/jdenemar/xen ...

Could you try with versions described in this BZ please Bryan?

Thanks,
Michal

Comment 11 Martin Wilck 2009-10-22 12:44:33 UTC
It appears that snmpd is now started after xend at boot time under 5.4. What if you just run "service xend stop" and do the snmpwalk?

Comment 12 Bryan Mason 2009-10-22 20:54:18 UTC
Created attachment 365781 [details]
My snmpd config file

I can still reproduce this after upgrading my RHEL 5.3 test system to RHEL 5.4.  If I run snmpwalk right after rebooting, I see two eth0's:

    # snmpwalk -v2c -c local localhost ifDesc
    IF-MIB::ifDescr.1 = STRING: lo
    IF-MIB::ifDescr.2 = STRING: eth0
    IF-MIB::ifDescr.3 = STRING: eth1
    IF-MIB::ifDescr.4 = STRING: sit0
    IF-MIB::ifDescr.5 = STRING: virbr0
    IF-MIB::ifDescr.6 = STRING: vif0.0
    IF-MIB::ifDescr.7 = STRING: eth0
    IF-MIB::ifDescr.8 = STRING: vif0.1
    IF-MIB::ifDescr.9 = STRING: veth1
    IF-MIB::ifDescr.10 = STRING: vif0.2
    IF-MIB::ifDescr.11 = STRING: veth2
    IF-MIB::ifDescr.12 = STRING: vif0.3
    IF-MIB::ifDescr.13 = STRING: veth3
    IF-MIB::ifDescr.14 = STRING: vif0.4
    IF-MIB::ifDescr.15 = STRING: veth4
    IF-MIB::ifDescr.16 = STRING: xenbr0
    
    # rpm -q xen net-snmp
    xen-3.0.3-96.el5.x86_64
    net-snmp-5.3.2.2-7.el5_4.2.x86_64

I'm attaching my snmpd.conf.

Comment 13 Michal Novotny 2009-10-23 08:11:47 UTC
Bryan, that is really strange. I used your snmpd.conf file and restarted snmp daemon, more exactly I did this:

# service snmpd start
Starting snmpd:                                            [  OK  ]
# snmpwalk -v2c -c local localhost ifDesc              
IF-MIB::ifDescr.1 = STRING: lo                                     
IF-MIB::ifDescr.2 = STRING: eth1                                   
IF-MIB::ifDescr.3 = STRING: peth2                                  
IF-MIB::ifDescr.4 = STRING: sit0                                   
IF-MIB::ifDescr.5 = STRING: virbr0                                 
IF-MIB::ifDescr.6 = STRING: vif0.0                                 
IF-MIB::ifDescr.7 = STRING: veth0                                  
IF-MIB::ifDescr.8 = STRING: vif0.1                                 
IF-MIB::ifDescr.9 = STRING: veth1                                  
IF-MIB::ifDescr.10 = STRING: vif0.2                                
IF-MIB::ifDescr.11 = STRING: eth2                                  
IF-MIB::ifDescr.12 = STRING: vif0.3                                
IF-MIB::ifDescr.13 = STRING: veth3                                 
IF-MIB::ifDescr.14 = STRING: vif0.4                                
IF-MIB::ifDescr.15 = STRING: veth4                                 
IF-MIB::ifDescr.16 = STRING: xenbr0                                
# service xend restart     
restart xend:                                              [  OK  ]
# snmpwalk -v2c -c local localhost ifDesc              
IF-MIB::ifDescr.1 = STRING: lo                                     
IF-MIB::ifDescr.2 = STRING: eth1                                   
IF-MIB::ifDescr.3 = STRING: peth2                                  
IF-MIB::ifDescr.4 = STRING: sit0                                   
IF-MIB::ifDescr.5 = STRING: virbr0                                 
IF-MIB::ifDescr.6 = STRING: vif0.0
IF-MIB::ifDescr.7 = STRING: veth0
IF-MIB::ifDescr.8 = STRING: vif0.1
IF-MIB::ifDescr.9 = STRING: veth1
IF-MIB::ifDescr.10 = STRING: vif0.2
IF-MIB::ifDescr.11 = STRING: eth2
IF-MIB::ifDescr.12 = STRING: vif0.3
IF-MIB::ifDescr.13 = STRING: veth3
IF-MIB::ifDescr.14 = STRING: vif0.4
IF-MIB::ifDescr.15 = STRING: veth4
IF-MIB::ifDescr.16 = STRING: xenbr0
# service xend stop
Stopping xend:                                             [  OK  ]
# service xend start
Starting xend:                                             [  OK  ]
# snmpwalk -v2c -c local localhost ifDesc
IF-MIB::ifDescr.1 = STRING: lo
IF-MIB::ifDescr.2 = STRING: eth1
IF-MIB::ifDescr.3 = STRING: peth2
IF-MIB::ifDescr.4 = STRING: sit0
IF-MIB::ifDescr.5 = STRING: virbr0
IF-MIB::ifDescr.6 = STRING: vif0.0
IF-MIB::ifDescr.7 = STRING: veth0
IF-MIB::ifDescr.8 = STRING: vif0.1
IF-MIB::ifDescr.9 = STRING: veth1
IF-MIB::ifDescr.10 = STRING: vif0.2
IF-MIB::ifDescr.11 = STRING: eth2
IF-MIB::ifDescr.12 = STRING: vif0.3
IF-MIB::ifDescr.13 = STRING: veth3
IF-MIB::ifDescr.14 = STRING: vif0.4
IF-MIB::ifDescr.15 = STRING: veth4
IF-MIB::ifDescr.16 = STRING: xenbr0
#

As you can see, I see no problem here. Could you give me exact steps to reproduce incl. information when daemons are stopped/started etc... ?

Thanks,
Michal

Comment 14 Michal Novotny 2009-10-23 08:32:21 UTC
Oh, also, in addition to comment #13: When I stop the daemon and I issue SNMPWALK (ie. before I start xend again), SNMPWALK is showing exactly the same lines like in the comment above.

Michal

Comment 15 Martin Wilck 2009-10-23 11:55:28 UTC
snmpd never changes the numbering or naming of interfaces. Thus *if you start snmpd before xen's network-bridge script starts*, you will see e.g.

    IF-MIB::ifDescr.1 = STRING: lo
    IF-MIB::ifDescr.2 = STRING: eth0

If snmpd is started now, it will not change these 2 entries (eth0 will become peth0 without snmpd noticing) but add the new bridging interface entries, yielding:

    IF-MIB::ifDescr.3 = STRING: vif0.0
    IF-MIB::ifDescr.4 = STRING: eth0

Now you have iface 2 and 4 both by the name "eth0".

On the other hand, if network-bridge runs before snmpd is started, you get a listing as in the previous comment, which won't change even if the network-bridge is stopped. (Well, interfaces may vanish when the bridge is stopped. But the important thing is that ifDescr2 is "peth0" and stays so). restarting network-bridge has no effect while snmpd is running.

>Oh, also, in addition to comment #13: When I stop the daemon and I issue
>SNMPWALK (ie. before I start xend again), SNMPWALK is showing exactly the same
>lines like in the comment above.

That is strange, because the vifX.Y and vethX.Y should have disappeared. But it makes sense that the physical interface is still displayed as "peth0" while in realtiy the name is "eth0" if the bridge isn't running.

Comment 16 Bryan Mason 2009-10-23 18:52:10 UTC
Hi Michal,

My reproduction steps are:

    1) Enable snmpd ("chkconfig snmpd on")
    2) Rebot
    3) Log on
    4) Run snmpwalk.

According to /var/log/messages, snmpd started at 14:02:26

    Oct 22 14:02:26 why snmpd[2764]: error finding row index in
        _ifXTable_container_row_restore
    Oct 22 14:02:26 why last message repeated 10 times
    Oct 22 14:02:26 why snmpd[2764]: error finding row index in
        _ifXTable_container_row_restore
    Oct 22 14:02:26 why snmpd[2764]: NET-SNMP version 5.3.2.2
    Oct 22 14:02:55 why snmpd[2764]: IfIndex of an interface
        changed. Such interface s will appear multiple times in
        IF-MIB.
    Oct 22 14:02:55 why snmpd[2764]: IfIndex of an interface
        changed. Such interfaces will appear multiple times in IF-MIB.
    Oct 22 14:02:55 why snmpd[2764]: Name of an interface
        changed. Such interfaces will keep its old name in IF-MIB.
    Oct 22 14:02:55 why snmpd[2764]: Duplicate IPv6 address detected,
        some interfaces may not be visible in IP-MIB
    Oct 22 14:03:03 why snmpd[2764]: Connection from UDP:
        [127.0.0.1]:37905
    Oct 22 14:03:03 why snmpd[2764]: Received SNMP packet(s) from UDP:
        [127.0.0.1]:37905
    Oct 22 14:03:03 why snmpd[2764]: Connection from UDP:
        [127.0.0.1]:37905

According to /var/log/xen/xend.log, xend started at 14:02:38

    [2009-10-22 14:02:38 xend 3524] INFO (SrvDaemon:283) Xend Daemon
        started

Another way to reproduce this is to disable both snmpd and xen, reboot the system, start snmpd, then start xen:

    # service snmpd status
    snmpd is stopped
    # service xend status
    xend is stopped

    # service snmpd start
    Starting snmpd:                                            [  OK  ]

    # snmpwalk -v2c -c local localhost ifDesc
    IF-MIB::ifDescr.1 = STRING: lo
    IF-MIB::ifDescr.2 = STRING: eth1
    IF-MIB::ifDescr.3 = STRING: eth0
    IF-MIB::ifDescr.4 = STRING: sit0
    IF-MIB::ifDescr.5 = STRING: virbr0

    # service xend start
    Starting xend:                                             [  OK  ]

    # snmpwalk -v2c -c local localhost ifDesc
    IF-MIB::ifDescr.1 = STRING: lo
    IF-MIB::ifDescr.2 = STRING: eth1
    IF-MIB::ifDescr.3 = STRING: eth0
    IF-MIB::ifDescr.4 = STRING: sit0
    IF-MIB::ifDescr.5 = STRING: virbr0
    IF-MIB::ifDescr.6 = STRING: vif0.0
    IF-MIB::ifDescr.7 = STRING: eth0
    IF-MIB::ifDescr.8 = STRING: vif0.1
    IF-MIB::ifDescr.9 = STRING: veth1
    IF-MIB::ifDescr.10 = STRING: vif0.2
    IF-MIB::ifDescr.11 = STRING: veth2
    IF-MIB::ifDescr.12 = STRING: vif0.3
    IF-MIB::ifDescr.13 = STRING: veth3
    IF-MIB::ifDescr.14 = STRING: vif0.4
    IF-MIB::ifDescr.15 = STRING: veth4
    IF-MIB::ifDescr.16 = STRING: xenbr0
    
I waited for about 45 seconds after starting xend before running snmpwalk the second time.

The main thing that counts for reproduction is that snmpd has to start before xend has had a chance to configure the network bridge.

~ Bryan

Comment 17 Michal Novotny 2009-10-26 08:34:01 UTC
(In reply to comment #15)
> snmpd never changes the numbering or naming of interfaces. Thus *if you start
> snmpd before xen's network-bridge script starts*, you will see e.g.
> 
>     IF-MIB::ifDescr.1 = STRING: lo
>     IF-MIB::ifDescr.2 = STRING: eth0
> 
> If snmpd is started now, it will not change these 2 entries (eth0 will become
> peth0 without snmpd noticing) but add the new bridging interface entries,
> yielding:
> 
>     IF-MIB::ifDescr.3 = STRING: vif0.0
>     IF-MIB::ifDescr.4 = STRING: eth0
> 
> Now you have iface 2 and 4 both by the name "eth0".
> 
> On the other hand, if network-bridge runs before snmpd is started, you get a
> listing as in the previous comment, which won't change even if the
> network-bridge is stopped. (Well, interfaces may vanish when the bridge is
> stopped. But the important thing is that ifDescr2 is "peth0" and stays so).
> restarting network-bridge has no effect while snmpd is running.
> 
> >Oh, also, in addition to comment #13: When I stop the daemon and I issue
> >SNMPWALK (ie. before I start xend again), SNMPWALK is showing exactly the same
> >lines like in the comment above.
> 
> That is strange, because the vifX.Y and vethX.Y should have disappeared. But it
> makes sense that the physical interface is still displayed as "peth0" while in
> realtiy the name is "eth0" if the bridge isn't running.  

I'll try to reproduce using the steps from Bryan's last comment. The truth is I am using bridge and I have never tried enabling SNMPd using chkconfig and rebooting the machine so I can try it now.

Comment 18 Michal Novotny 2009-10-26 08:40:24 UTC
Well, I did try setting the system as described in comment #16. When I grep /var/log/messages for snmpd evidence I get:

[root@mig ~]# cat /var/log/messages | grep snmpd
Oct 26 09:34:47 mig snmpd[6331]: Connection from UDP: [127.0.0.1]:47995
Oct 26 09:34:47 mig snmpd[6331]: Received SNMP packet(s) from UDP: [127.0.0.1]:47995
Oct 26 09:34:47 mig snmpd[6331]: Connection from UDP: [127.0.0.1]:47995
Oct 26 09:34:49 mig snmpd[6331]: Connection from UDP: [127.0.0.1]:32998
Oct 26 09:34:49 mig snmpd[6331]: Received SNMP packet(s) from UDP: [127.0.0.1]:32998
Oct 26 09:34:49 mig snmpd[6331]: Connection from UDP: [127.0.0.1]:32998
Oct 26 09:35:08 mig snmpd[6331]: Received TERM or STOP signal...  shutting down...
Oct 26 09:37:31 mig snmpd[3532]: error finding row index in _ifXTable_container_row_restore
Oct 26 09:37:31 mig snmpd[3532]: NET-SNMP version 5.3.2.2
Oct 26 09:38:00 mig snmpd[3532]: IfIndex of an interface changed. Such interfaces will appear multiple times in IF-MIB.
Oct 26 09:38:00 mig snmpd[3532]: Name of an interface changed. Such interfaces will keep its old name in IF-MIB.
Oct 26 09:38:00 mig snmpd[3532]: Duplicate IPv6 address detected, some interfaces may not be visible in IP-MIB

When I grep /var/log/xen/xend.log I get:

[root@mig ~]# cat /var/log/xen/xend.log | grep started
[2009-10-26 09:37:37 xend 4290] INFO (SrvDaemon:283) Xend Daemon started

When I log on and try `snmpwalk -v2c -c local localhost ifDescr` I get:

IF-MIB::ifDescr.1 = STRING: lo
IF-MIB::ifDescr.2 = STRING: eth1
IF-MIB::ifDescr.3 = STRING: eth2
IF-MIB::ifDescr.4 = STRING: sit0
IF-MIB::ifDescr.5 = STRING: virbr0
IF-MIB::ifDescr.6 = STRING: vif0.0
IF-MIB::ifDescr.7 = STRING: veth0
IF-MIB::ifDescr.8 = STRING: vif0.1
IF-MIB::ifDescr.9 = STRING: veth1
IF-MIB::ifDescr.10 = STRING: vif0.2
IF-MIB::ifDescr.11 = STRING: eth2
IF-MIB::ifDescr.12 = STRING: vif0.3
IF-MIB::ifDescr.13 = STRING: veth3
IF-MIB::ifDescr.14 = STRING: vif0.4
IF-MIB::ifDescr.15 = STRING: veth4
IF-MIB::ifDescr.16 = STRING: xenbr0

So I see there is a problem with ifDescr.3 as it's showing eth2 instead of peth2. Thanks for the steps to reproduce Bryan. I'll investigate this...

Michal

Comment 19 Michal Novotny 2009-10-26 09:35:33 UTC
(In reply to comment #18)
> Well, I did try setting the system as described in comment #16. When I grep
> /var/log/messages for snmpd evidence I get:
> 
> [root@mig ~]# cat /var/log/messages | grep snmpd
> Oct 26 09:34:47 mig snmpd[6331]: Connection from UDP: [127.0.0.1]:47995
> Oct 26 09:34:47 mig snmpd[6331]: Received SNMP packet(s) from UDP:
> [127.0.0.1]:47995
> Oct 26 09:34:47 mig snmpd[6331]: Connection from UDP: [127.0.0.1]:47995
> Oct 26 09:34:49 mig snmpd[6331]: Connection from UDP: [127.0.0.1]:32998
> Oct 26 09:34:49 mig snmpd[6331]: Received SNMP packet(s) from UDP:
> [127.0.0.1]:32998
> Oct 26 09:34:49 mig snmpd[6331]: Connection from UDP: [127.0.0.1]:32998
> Oct 26 09:35:08 mig snmpd[6331]: Received TERM or STOP signal...  shutting
> down...
> Oct 26 09:37:31 mig snmpd[3532]: error finding row index in
> _ifXTable_container_row_restore
> Oct 26 09:37:31 mig snmpd[3532]: NET-SNMP version 5.3.2.2
> Oct 26 09:38:00 mig snmpd[3532]: IfIndex of an interface changed. Such
> interfaces will appear multiple times in IF-MIB.
> Oct 26 09:38:00 mig snmpd[3532]: Name of an interface changed. Such interfaces
> will keep its old name in IF-MIB.
> Oct 26 09:38:00 mig snmpd[3532]: Duplicate IPv6 address detected, some
> interfaces may not be visible in IP-MIB
> 
> When I grep /var/log/xen/xend.log I get:
> 
> [root@mig ~]# cat /var/log/xen/xend.log | grep started
> [2009-10-26 09:37:37 xend 4290] INFO (SrvDaemon:283) Xend Daemon started
> 
> When I log on and try `snmpwalk -v2c -c local localhost ifDescr` I get:
> 
> IF-MIB::ifDescr.1 = STRING: lo
> IF-MIB::ifDescr.2 = STRING: eth1
> IF-MIB::ifDescr.3 = STRING: eth2
> IF-MIB::ifDescr.4 = STRING: sit0
> IF-MIB::ifDescr.5 = STRING: virbr0
> IF-MIB::ifDescr.6 = STRING: vif0.0
> IF-MIB::ifDescr.7 = STRING: veth0
> IF-MIB::ifDescr.8 = STRING: vif0.1
> IF-MIB::ifDescr.9 = STRING: veth1
> IF-MIB::ifDescr.10 = STRING: vif0.2
> IF-MIB::ifDescr.11 = STRING: eth2
> IF-MIB::ifDescr.12 = STRING: vif0.3
> IF-MIB::ifDescr.13 = STRING: veth3
> IF-MIB::ifDescr.14 = STRING: vif0.4
> IF-MIB::ifDescr.15 = STRING: veth4
> IF-MIB::ifDescr.16 = STRING: xenbr0
> 
> So I see there is a problem with ifDescr.3 as it's showing eth2 instead of
> peth2. Thanks for the steps to reproduce Bryan. I'll investigate this...
> 
> Michal  

Well, I did try to investigate this one and I found out that if I restart snmpd after xend is started (with wait for vif devices to appear in snmpwalk output) everything is OK and ifDescr.3 appears as peth2 instead of eth2, ie.

[root@mig ~]#  snmpwalk -v2c -c local localhost ifDescr
IF-MIB::ifDescr.1 = STRING: lo
IF-MIB::ifDescr.2 = STRING: eth1
IF-MIB::ifDescr.3 = STRING: peth2
IF-MIB::ifDescr.4 = STRING: sit0
IF-MIB::ifDescr.5 = STRING: virbr0
IF-MIB::ifDescr.6 = STRING: vif0.0
IF-MIB::ifDescr.7 = STRING: veth0
IF-MIB::ifDescr.8 = STRING: vif0.1
IF-MIB::ifDescr.9 = STRING: veth1
IF-MIB::ifDescr.10 = STRING: vif0.2
IF-MIB::ifDescr.11 = STRING: eth2
IF-MIB::ifDescr.12 = STRING: vif0.3
IF-MIB::ifDescr.13 = STRING: veth3
IF-MIB::ifDescr.14 = STRING: vif0.4
IF-MIB::ifDescr.15 = STRING: veth4
IF-MIB::ifDescr.16 = STRING: xenbr0

This one is OK I guess so more investigation is needed but I wonder if this is really a Xen issue and not SNMPWALK one, eg. whether SNMPd forgets to refresh information since snmpd restart after xend is started is sufficient to rename ifDescr.3 from eth2 to peth2...

Michal

Comment 22 Michal Novotny 2009-10-27 10:05:35 UTC
Created attachment 366227 [details]
Patch to refresh interface information in snmpd

Please post only patch files and not whole RPMs to solve (if it's not the dependency package from another component). This is the patch to refresh interface information in snmpd. Since this is not so simple to implement to net-snmp as described in comment #3, I finally made a patch to restart snmpd after xend is started and veth devices are set (ie. xend bridge has started). The code will run only after xend bridge is created if any and no code is run when bridge is unavailable (ie. it's set to /bin/true)...

Michal

Comment 23 Michal Novotny 2009-10-27 10:09:00 UTC
(In reply to comment #22)
> Created an attachment (id=366227) [details]
> Patch to refresh interface information in snmpd
> 
> Please post only patch files and not whole RPMs to solve (if it's not the
> dependency package from another component). This is the patch to refresh
> interface information in snmpd. Since this is not so simple to implement to
> net-snmp as described in comment #3, I finally made a patch to restart snmpd
> after xend is started and veth devices are set (ie. xend bridge has started).
> The code will run only after xend bridge is created if any and no code is run
> when bridge is unavailable (ie. it's set to /bin/true)...
> 
> Michal  

Sorry, I mean vif devices, not veth...

Michal

Comment 24 Martin Wilck 2009-10-27 11:01:17 UTC
(In reply to comment #22)
> Please post only patch files and not whole RPMs to solve (if it's not the
> dependency package from another component).

I guess you're referring to my attachment #333321 [details] - that was only meant for your information, to give a hint what needs to be done. We (Fujitsu, being unable to patch RHEL directly) have been using this package as a workaround for our customers.

That said, your patch looks ok to me.

Comment 25 Michal Novotny 2009-11-02 08:06:01 UTC
(In reply to comment #24)
> (In reply to comment #22)
> > Please post only patch files and not whole RPMs to solve (if it's not the
> > dependency package from another component).
> 
> I guess you're referring to my attachment #333321 [details] - that was only meant for
> your information, to give a hint what needs to be done. We (Fujitsu, being
> unable to patch RHEL directly) have been using this package as a workaround for
> our customers.
> 
> That said, your patch looks ok to me.  

Hi Martin,
you are right, I was referring to your attachment #333321 [details]. In fact my patch basically checks for snmpd presence and if present (and started) it restarts snmpd after xen vif interfaces has been activated...

Michal

Comment 30 Michal Novotny 2009-11-23 15:09:33 UTC
The kbase article has been published at: http://kbase.redhat.com/faq/docs/DOC-21309

Thanks,
Michal

Comment 31 Michal Novotny 2009-11-24 11:56:56 UTC
Well, I'm going to close it as NOTABUG. In fact this is not a bug in Xen but it's a bug in SNMP daemon, this Kbase article provides necessary information about that issue. If anything, feel free to reopen it...

Comment 32 Martin Wilck 2009-11-24 12:05:41 UTC
(In reply to comment #30)
> The kbase article has been published at:
> http://kbase.redhat.com/faq/docs/DOC-21309

I am unable to read this document

"It appears you're not allowed to view what you requested. You might contact your administrator if you think this is a mistake."

I think this should be public, right?

Comment 33 Michal Novotny 2009-11-24 12:39:27 UTC
(In reply to comment #32)
> (In reply to comment #30)
> > The kbase article has been published at:
> > http://kbase.redhat.com/faq/docs/DOC-21309
> 
> I am unable to read this document
> 
> "It appears you're not allowed to view what you requested. You might contact
> your administrator if you think this is a mistake."
> 
> I think this should be public, right?  

Well, you need to login to kbase site to be able to see it. It is in English Sketch Space and no approval should be required now (it should be already visible) so the only thing you need to do is login to the site.

Comment 34 Martin Wilck 2009-11-24 12:42:37 UTC
I can't read it, although I am logged in into RHN.

Comment 35 Michal Novotny 2009-11-24 13:30:26 UTC
(In reply to comment #34)
> I can't read it, although I am logged in into RHN.  

You need to have access to Kbase site, not RHN.

Comment 36 Gary Smith 2009-11-24 13:44:41 UTC
Hi


I've notified Martin via the associated Issue Tracker what's taking place with this BZ, and explained the availability and text of the kbase.


Regards, Gary

Comment 37 Michal Novotny 2009-11-24 13:48:59 UTC
(In reply to comment #36)
> Hi
> 
> 
> I've notified Martin via the associated Issue Tracker what's taking place with
> this BZ, and explained the availability and text of the kbase.
> 
> 
> Regards, Gary  

Ok, thanks Gary. This kbase should be visible to people having access to kbase

Comment 41 Paolo Bonzini 2010-04-08 15:41:46 UTC
This bug was closed during 5.5 development and it's being removed from the internal tracking bugs (which are now for 5.6).