RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1283191 - mkdumprd fails on RHEV hosts with running VMs
Summary: mkdumprd fails on RHEV hosts with running VMs
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: kexec-tools
Version: 6.7
Hardware: All
OS: Linux
urgent
high
Target Milestone: rc
: ---
Assignee: Xunlei Pang
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks: 1172231 1305481
TreeView+ depends on / blocked
 
Reported: 2015-11-18 12:32 UTC by Julio Entrena Perez
Modified: 2019-12-16 05:06 UTC (History)
19 users (show)

Fixed In Version: kexec-tools-2.0.0-292.el6
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1305481 (view as bug list)
Environment:
Last Closed: 2016-05-10 19:12:09 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 1571553 0 None None None Never
Red Hat Product Errata RHBA-2016:0734 0 normal SHIPPED_LIVE kexec-tools bug fix and enhancement update 2016-05-10 22:28:16 UTC

Description Julio Entrena Perez 2015-11-18 12:32:47 UTC
Description of problem:
mkdumprd fails on RHEV hosts with running VMs due to the 

# service kdump restart
Stopping kdump:                                            [  OK  ]
Detected change(s) the following file(s):
  
  /etc/kdump.conf
Rebuilding /boot/initrd-2.6.32-573.7.1.el6.x86_64kdump.img
The ifcfg-vnet0 or ifcfg-xxx which contains DEVICE=vnet0 field doesn't exist.
Failed to run mkdumprd
Starting kdump:                                            [FAILED]

The vnet<x> network interfaces of the running guests seem to confuse mkdumprd:

# ifconfig | egrep -v ^\ \|^$\|^vlan\|^eth1\.\|^lo
eth0      Link encap:Ethernet  HWaddr 00:10:18:73:B8:8D  
rhevm     Link encap:Ethernet  HWaddr 00:10:18:73:B8:8D  
vnet0     Link encap:Ethernet  HWaddr FE:01:A4:AD:FE:CA  
vnet1     Link encap:Ethernet  HWaddr FE:01:A4:AD:FE:CB

Version-Release number of selected component (if applicable):
kexec-tools-2.0.0-286.el6.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Start some guests in a virt host (tested on a RHEV host)
2. Update /etc/kdump.conf
3. Restart kdump service

Actual results:
Rebuilding /boot/initrd-2.6.32-573.7.1.el6.x86_64kdump.img
The ifcfg-vnet0 or ifcfg-xxx which contains DEVICE=vnet0 field doesn't exist.
Failed to run mkdumprd
Starting kdump:                                            [FAILED]


Expected results:
Rebuilding /boot/initrd-2.6.32-573.7.1.el6.x86_64kdump.img
Starting kdump:                                            [  OK  ]


Additional info:
KCS article https://access.redhat.com/solutions/1571553 has a suggestion that helps mkdumprd to complete successfully in this scenario.

Comment 1 Minfei Huang 2015-11-20 06:16:31 UTC
(In reply to Julio Entrena Perez from comment #0)
> Description of problem:
> mkdumprd fails on RHEV hosts with running VMs due to the 
> 
> # service kdump restart
> Stopping kdump:                                            [  OK  ]
> Detected change(s) the following file(s):
>   
>   /etc/kdump.conf
> Rebuilding /boot/initrd-2.6.32-573.7.1.el6.x86_64kdump.img
> The ifcfg-vnet0 or ifcfg-xxx which contains DEVICE=vnet0 field doesn't exist.
> Failed to run mkdumprd
> Starting kdump:                                            [FAILED]
> 
> The vnet<x> network interfaces of the running guests seem to confuse
> mkdumprd:
> 
> # ifconfig | egrep -v ^\ \|^$\|^vlan\|^eth1\.\|^lo
> eth0      Link encap:Ethernet  HWaddr 00:10:18:73:B8:8D  
> rhevm     Link encap:Ethernet  HWaddr 00:10:18:73:B8:8D  
> vnet0     Link encap:Ethernet  HWaddr FE:01:A4:AD:FE:CA  

Could you mind setuping ifcfg for device vnet0? Then you can try it again.

Thanks
Minfei

Comment 2 Julio Entrena Perez 2015-11-20 09:15:39 UTC
(In reply to Minfei Huang from comment #1)
> 
> Could you mind setuping ifcfg for device vnet0? Then you can try it again.

The interfaces are already UP since the VMs are running:

# ifconfig | grep -A2 vnet
vnet0     Link encap:Ethernet  HWaddr FE:01:A4:AD:FE:CA  
          inet6 addr: fe80::fc01:a4ff:fead:feca/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
--
vnet1     Link encap:Ethernet  HWaddr FE:01:A4:AD:FE:CB  
          inet6 addr: fe80::fc01:a4ff:fead:fecb/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

# touch /etc/kdump.conf

# service kdump restart
Stopping kdump:                                            [  OK  ]
Detected change(s) the following file(s):
  
  /etc/kdump.conf
Rebuilding /boot/initrd-2.6.32-573.7.1.el6.x86_64kdump.img
The ifcfg-vnet0 or ifcfg-xxx which contains DEVICE=vnet0 field doesn't exist.
Failed to run mkdumprd
Starting kdump:                                            [FAILED]

Comment 3 Minfei Huang 2015-11-20 09:27:23 UTC
(In reply to Julio Entrena Perez from comment #2)
> (In reply to Minfei Huang from comment #1)
> > 
> > Could you mind setuping ifcfg for device vnet0? Then you can try it again.
> 
> The interfaces are already UP since the VMs are running:
> 
> # ifconfig | grep -A2 vnet
> vnet0     Link encap:Ethernet  HWaddr FE:01:A4:AD:FE:CA  
>           inet6 addr: fe80::fc01:a4ff:fead:feca/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
> --
> vnet1     Link encap:Ethernet  HWaddr FE:01:A4:AD:FE:CB  
>           inet6 addr: fe80::fc01:a4ff:fead:fecb/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
> 

Yes, this net device is up. What I mean is to write a persistent config in ifcfg-x. Kdump will use it to gather enough info to setup it in 2nd kernel.

Thanks
Minfei

Comment 4 Julio Entrena Perez 2015-11-20 09:29:37 UTC
(In reply to Julio Entrena Perez from comment #2)
> (In reply to Minfei Huang from comment #1)
> > 
> > Could you mind setuping ifcfg for device vnet0? Then you can try it again.
> 
Sorry, I misread your update.
vnet<x> interfaces are tap interfaces created for the vNICs of the running VMs.
Those interfaces are created and destroyed dynamically and consequently don't have an ifcfg file.
mkdumprd should ignore those interfaces.

Comment 5 Minfei Huang 2015-11-20 11:59:00 UTC
(In reply to Julio Entrena Perez from comment #4)
> (In reply to Julio Entrena Perez from comment #2)
> > (In reply to Minfei Huang from comment #1)
> > > 
> > > Could you mind setuping ifcfg for device vnet0? Then you can try it again.
> > 
> Sorry, I misread your update.
> vnet<x> interfaces are tap interfaces created for the vNICs of the running
> VMs.
> Those interfaces are created and destroyed dynamically and consequently
> don't have an ifcfg file.
> mkdumprd should ignore those interfaces.

Seems kdump will use ifcfg to setup network? Could you have a try?

Thanks
Minfei

Comment 7 Baoquan He 2015-12-18 09:30:15 UTC
Hi Cui Ying,

Could you help reproduce this bug on rhev host?

Comment 9 Xunlei Pang 2015-12-25 02:33:06 UTC
Seems the ip route accquired from the destination is through the guest's vnet, I think this is wrong, it should be one of the host's netdev. 

Could you please paste the content of your /etc/kdump.conf plus the output of following command both on host?
ip route get to <dest ip>

As an example, if using ssh kdump to "10.66.129.152", so in /etc/kdump.conf like:
ssh root.129.152
sshkey /root/.ssh/kdumprsa

And the command output is like:
# ip route get to 10.66.129.152
10.66.129.152 via 10.16.47.254 dev eth0  src 10.16.45.13
    cache  mtu 1500 advmss 1460 hoplimit 64

Comment 10 Julio Entrena Perez 2015-12-29 12:37:19 UTC
# cat /etc/kdump.conf | grep -v ^#

path /var/crash
core_collector makedumpfile -c --message-level 1 -d 31

Dump is to be produced locally so there's no <dest ip>.

Comment 11 Julio Entrena Perez 2015-12-29 12:44:09 UTC
Apologies, wrong host.

# cat /etc/kdump.conf | grep -v ^#

path /var/crash
core_collector makedumpfile -c --message-level 1 -d 31
fence_kdump_nodes rhevm1-375.usersys.redhat.com
fence_kdump_args -p 7410 -i 5

Looks like including the "fence_kdump_nodes" line causes the problem.

Comment 14 Xunlei Pang 2015-12-30 03:09:37 UTC
"rhevm" is a bridge, and it uses "eth1" for one of its physical interfaces. If I understand correctly, RHEV will create a new vnet<n> for every new guest<n> dynamically, and add "vnet<n>" to "rhevm" bridge, the kdump will simply fail when detecting no ifcfg-vnet<m> file created for the corresponding vnet<m> interface. 

I think for such bridges in RHEV, we can interpret only "eth1" for "rhevm" and ignore all the "vnet<m>" in the bridge, the bridge still works, and so does kdump on RHEV.

Comment 17 Xunlei Pang 2016-01-07 03:03:41 UTC
Could you please help start some VMs on "10.33.20.24 (root/redhat)", currently there're no vnet interfaces on the host, I want to gather some information to make a formal patch.

Thanks!

Comment 18 Julio Entrena Perez 2016-01-07 09:20:22 UTC
Sorry, done.

Comment 19 Xunlei Pang 2016-01-07 10:08:20 UTC
For vnet<x>, by what ways can I identify it as a virtual interface assigned to VMs, like through some file under /sys/class/net/vnet0/ ? Anyone knows that?

Comment 20 Julio Entrena Perez 2016-01-07 10:37:18 UTC
It should be safe to assume that any interface named vnet<x> is related to a virtual machine:

http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/conf/domain_conf.h;h=ae6d546978973539766ebb114e7be3a802c329fa;hb=HEAD#l1084

Comment 21 Xunlei Pang 2016-01-07 13:07:56 UTC
This is very helpful, thank you!

Comment 23 Xunlei Pang 2016-01-12 09:36:10 UTC
Hi,

On RHEV, is the bridge name "rhevm" also safe to use like vnet<n>?

Comment 24 Julio Entrena Perez 2016-01-12 09:43:44 UTC
rhevm is the default management network on RHEV hosts, it is used to communicate to the RHEV-M (manager) and it will _usually_ have the main IP address of the host.
If a host has to dump a vmcore over the network, it's _likely_ that this is the interface that should be used for that purpose.

Comment 33 Guo, Zhiyi 2016-03-03 08:37:56 UTC
(In reply to Xunlei Pang from comment #32)
> Hi Zhiyi,
> 
> It's a known issue, you can refer to the following link:
> https://bugzilla.redhat.com/show_bug.cgi?id=1284605
> 
> It's been fixed since Release 2.0.0-294 after this bug, so you try to test
> it 
> using newer release versions after "kexec-tools-2.0.0-294.el6.x86_64"
> 
> Thanks.

Thanks Xunlei,

Verify this bug on rhel 6.8 with kexec-tools-2.0.0-294.el6.x86_64:
[root@dhcp-10-61 ~]# service kdump restart
Stopping kdump:                                            [  OK  ]
Detected change(s) the following file(s):
  
  /etc/kdump.conf
Rebuilding /boot/initrd-2.6.32-621.el6.x86_64kdump.img
Starting kdump:                                            [  OK  ]

Comment 36 errata-xmlrpc 2016-05-10 19:12:09 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0734.html

Comment 37 Billy Woods 2016-06-28 20:06:08 UTC
Requesting that the bug be re-opened:

After updating to the new kexec, the customer is still unable to bring up kdump. 

The customer can bring down the tap device and start kdump; however, after a yum update kernel then kdump goes down and the only work around is to bring down the tap device again. Case # 01620459


Note You need to log in before you can comment on or make changes to this bug.