Bug 1384683

Hi Ben,

I am not a native speaker, could you speak of below things more understandable? I am trying to debug, and now code truly put proto=dhcp into initramfs, but I didn't get what is "instead of storing the fact the IP address for the kdump kernel should be ......." meaning?


Unfortunately, when we do this the initramfs generator used by kdump seems to ignore the fact that IP address was acquired via DHCP and instead of storing the fact the IP address for the kdump kernel should be acquired by DHCP it encodes the IP address at the time when the machine was booted up in the initramfs. 

Thanks
Baoquan

Besides could you change /lib/dracut/modules.d/99kdumpbase/module-setup.sh as below? I doubt there's something wrong inside kdump_static_ip(). Then the shell debugging print may tell what's going on.


--- module-setup.sh     2016-11-16 20:58:41.697426841 -0500
+++ module-setup.sh.orig        2016-11-16 20:58:00.156210910 -0500
@@ -324,7 +324,6 @@ get_ip_route_field()
 #$1: config values of net line in kdump.conf
 #$2: srcaddr of network device
 kdump_install_net() {
-set -x
     local _server _netdev _srcaddr _route _serv_tmp
     local config_val="$1"

@@ -359,7 +358,6 @@ set -x
         echo "kdumpnic=$(kdump_setup_ifname $_netdev)" > ${initdir}/etc/cmdline.d/60kdumpnic.conf
         echo "bootdev=$(kdump_setup_ifname $_netdev)" > ${initdir}/etc/cmdline.d/70bootdev.conf
     fi
-set +x
 }

In kdump shell, we use below code to check if an ip addr is permanent, namely a static configured IP. Otherwise it's a dynamically created IP, namely from dhcp.

_ipaddr=$(ip addr show dev $_netdev permanent | awk "/ $_srcaddr\/.* /{print \$2}")

However, when I checked my laptop, the virtual interface like virtual bridge for kvm and tunnel for vpn, they also behaved as statically configured, I mean permanent. In fact I didn't add their configuration information into /etc/sysconfig/network-scripts/ifcfg-xxxx to make them static. So I am wondering how it's working on your system, finally it makes dhcp created IP be identified as static configured. Testing it as comment 4 can tell if there's something wroong.

[bhe@x1 ~]$ ip a show virbr0 permanent
4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:c4:53:7e brd ff:ff:ff:ff:ff:ff
    inet 192.168.124.1/24 brd 192.168.124.255 scope global virbr0
       valid_lft forever preferred_lft forever

[bhe@x1 ~]$ ip a show tun0 permanent
22: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1412 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none 
    inet 10.72.6.154/32 scope global tun0
       valid_lft forever preferred_lft forever
    inet6 fe80::9403:d586:a7cf:50f9/64 scope link flags 800 
       valid_lft forever preferred_lft forever

If one specify an ip address as dump target it will definitely fail in case dhcp server changes. If on use hostname in kdump.conf then it should be a kdump scripts bug. Please double check and provide more infomatioin for debugging.

* kdump.conf
* kdump kernel console log
* how to reproduce it if possible

Thanks
Dave

It can't be reproduced on my local kvm guests. Just did as Comment 8 said.

There could be other deployment issues causing problem they saw. Since no further response and additional information provided, I would like to close this bug as INSUFFICIENT_DATA. That's it.

Below is cmdline content from decompressed kdump initramfs.

[root@localhost cmdline.d]# ll
total 16
-rw-r--r--. 1 root root 45 Aug 25 23:57 40ip.conf
-rw-r--r--. 1 root root 25 Aug 25 23:57 42dns.conf
-rw-r--r--. 1 root root  0 Aug 25 23:57 45route-static.conf
-rw-r--r--. 1 root root 14 Aug 25 23:57 60kdumpnic.conf
-rw-r--r--. 1 root root 13 Aug 25 23:57 70bootdev.conf
[root@localhost cmdline.d]# cat 40ip.conf 
 ip=ens3:dhcp
 ifname=ens3:52:54:00:0a:9a:7d
[root@localhost cmdline.d]# cat 42dns.conf 
nameserver=192.168.124.1
[root@localhost cmdline.d]# cat 45route-static.conf 
[root@localhost cmdline.d]# cat 60kdumpnic.conf 
kdumpnic=ens3
[root@localhost cmdline.d]# cat 70bootdev.conf 
bootdev=ens3
[root@localhost cmdline.d]#

(In reply to Baoquan He from comment #9)
> It can't be reproduced on my local kvm guests. Just did as Comment 8 said.
                                                                     ~7, I meant
> 
> There could be other deployment issues causing problem they saw. Since no
> further response and additional information provided, I would like to close
> this bug as INSUFFICIENT_DATA. That's it.
> 
> Below is cmdline content from decompressed kdump initramfs.
> 
> [root@localhost cmdline.d]# ll
> total 16
> -rw-r--r--. 1 root root 45 Aug 25 23:57 40ip.conf
> -rw-r--r--. 1 root root 25 Aug 25 23:57 42dns.conf
> -rw-r--r--. 1 root root  0 Aug 25 23:57 45route-static.conf
> -rw-r--r--. 1 root root 14 Aug 25 23:57 60kdumpnic.conf
> -rw-r--r--. 1 root root 13 Aug 25 23:57 70bootdev.conf
> [root@localhost cmdline.d]# cat 40ip.conf 
>  ip=ens3:dhcp
>  ifname=ens3:52:54:00:0a:9a:7d
> [root@localhost cmdline.d]# cat 42dns.conf 
> nameserver=192.168.124.1
> [root@localhost cmdline.d]# cat 45route-static.conf 
> [root@localhost cmdline.d]# cat 60kdumpnic.conf 
> kdumpnic=ens3
> [root@localhost cmdline.d]# cat 70bootdev.conf 
> bootdev=ens3
> [root@localhost cmdline.d]#

> [root@localhost cmdline.d]# cat 40ip.conf 
>  ip=ens3:dhcp
>  ifname=ens3:52:54:00:0a:9a:7d

The mac addresses are same for two guests? If no it should fail..

Interesting, mac is different, didn't fail.

Dropped into kdump shell, interface is up. udev rules expect a interface with mac in another kvm guest.

kdump:/etc/cmdline.d# cat /etc/udev/rules.d/80-ifname.rules 
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="52:54:00:0a:9a:7d", ATTR{type}=="1", NAME="ens3"
kdump:/etc/cmdline.d# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:83:47:b4 brd ff:ff:ff:ff:ff:ff
    inet 192.168.124.216/24 brd 192.168.124.255 scope global dynamic ens3
       valid_lft 3229sec preferred_lft 3229sec
    inet6 fe80::5054:ff:fe83:47b4/64 scope link 
       valid_lft forever preferred_lft forever

This results in a problem with all the dumps from all the clustered machines being named as if they come from 127.0.0.1 this of course makes it difficult to know from what machine the dump came from.

The SA's from the lab came up with a work around to the problem. I understand that this won't help you reproduce the reported problem (why you can't reproduce it mystifies me and I think that you should quit trying to use virtual machines and actually really netboot a real machine and then it will be obvious) but it may help resolve the larger issue.

diff -uNr kexec-tools_a/dracut-kdump.sh kexec-tools_b/dracut-kdump.sh
--- kexec-tools_a/dracut-kdump.sh       2016-11-23 10:30:14.162146000 -0800
+++ kexec-tools_b/dracut-kdump.sh       2016-11-23 12:58:58.116060000 -0800
@@ -114,6 +114,14 @@
     fi
 }
 
+get_host_name()
+{
+    _hostname=`cat /proc/cmdline | grep hostname | sed -r 's/.*hostname=(\w+).*/\1/'`
+    [ -z "$_hostname" ] && return 1
+    HOST_IP=$_hostname
+    return 0
+}
+
 get_host_ip()
 {
     local _host
@@ -173,10 +181,14 @@
 read_kdump_conf
 fence_kdump_notify
 
-get_host_ip
+# check to see if hostname is provided on cmdline first.  If not try to get host ip.
+get_host_name
 if [ $? -ne 0 ]; then
-    echo "kdump: get_host_ip exited with non-zero status!"
-    exit 1
+    get_host_ip
+    if [ $? -ne 0 ]; then
+        echo "kdump: get_host_ip exited with non-zero status!"
+        exit 1
+    fi
 fi
 
 if [ -z "$DUMP_INSTRUCTION" ]; then
diff -uNr kexec-tools_a/kdumpctl kexec-tools_b/kdumpctl
--- kexec-tools_a/kdumpctl      2016-11-23 10:30:14.196149000 -0800
+++ kexec-tools_b/kdumpctl      2016-11-23 11:06:24.771011000 -0800
@@ -128,6 +128,11 @@
                cmdline=`append_cmdline "${cmdline}" disable_cpu_apicid ${id}`
        fi
 
+        # Append hostname to cmdline so it can be used when writing out nfs/ssh vmcores
+       _host_name=`/usr/bin/hostname`
+
+       cmdline=`append_cmdline "${cmdline}" hostname ${_host_name}`
+
        echo $cmdline
 }

(In reply to Ben Woodard from comment #13)
> This results in a problem with all the dumps from all the clustered machines
> being named as if they come from 127.0.0.1 this of course makes it difficult
> to know from what machine the dump came from.

Good to know this, thanks for your update, Ben.

I haven't got the root cause, if possible could you try adding debug code as Comment 4 suggested. It could let me know what's done on your clustered system with network dumping config. Since you have got a work around, I will try a local system later to see if it can reproduce, as you said, not virtual machine. Just working on another issue, will update if any progress. And you can also paste the debug printing if you can try Comment 4.

Thanks
Baoquan

Hi,

We need investigate and check if we can fix it in rhel7.5, so we still don't get what happened exactly. 

Do you have any suggestion about the further fix? We might not be able to build a test environment to reproduce it. It will be appreciated if you can propose a fix patch with root cause, or further investigation, we can help review it.

Or you can add debug code in shell script as Comment 4 suggested, I can try to understand what happened.

Thanks
Baoquan

(In reply to Ben Woodard from comment #13)
> This results in a problem with all the dumps from all the clustered machines
> being named as if they come from 127.0.0.1 this of course makes it difficult
> to know from what machine the dump came from.

If you see a "127.0.0.1" that means it's a file system dumping, not network dumping. Even though the file system might be a network fs. So far, question is still not clear, root cause is not found. If still no reply, I will close this bug as NOTABUG because lack farther information.

Thanks
Baoquan

Having it use 127.0.0.1 for all FS dumping is the current problem. It writes to a network file system and that makes figuring out which dump is from which machine impossible. Comment #13 shows the patch that we are currently carrying to work around this issue.

Please either apply that patch or do something that achieves the same result.

(In reply to Ben Woodard from comment #17)
> Having it use 127.0.0.1 for all FS dumping is the current problem. It writes
> to a network file system and that makes figuring out which dump is from
> which machine impossible. Comment #13 shows the patch that we are currently
> carrying to work around this issue.

Yes, it might be a problem. Except of NFS and ssh, we don't know if those local fs system actually is going through a network transferring. Up to now, no one complained about this, including those FCOE and iSCSI. Only you raised a issue. The thing is that we at least let kdump know what fs systems are network fs system and need be exceptional to use the hostname.

> 
> Please either apply that patch or do something that achieves the same result.

Without knowledge about the problem and what we need do to fix, I can't apply the workaround fix. It's obviously not right and will stirrup the other correct fs system setting. So the point is clear, that we need to know what's the scenario and what's the problem, and what's the root cause, and next what we should do exactly.

Thanks
Baoquan

kdump was working fine in RHEL6 we had no problems with how it worked. We have been battling with how it works in RHEL7 since we first started playing with the RHEL7 beta.
Several other issues have been dealt with already.

Here there are two problems:
1) we generate the kdump initrd on one machine which has a disk and writable file systems and then we use it on all the compute nodes which do not have any disk and do not have writable system file systems. For a number of reasons there cannot be writable system filesystems in these machines. If when we generate the kdump-initrd it includes the interface MAC address for the interface in the machine on which we generate the kdump-initrd or the IP address for that interface in the kdump-initrd, then this kdump-initrd will not work on the compute nodes for which it is intended. We have had to patch around this by manually repacking the kdump-initrd after it was generated.

2) when a node crashes and the kdump kernel writes to one of our NFS filers, all the dumps appear with the address 127.0.0.1 rather than the actual machine's hostname or the IP address. (the kdump kernel gets its IP address from the DHCP server.) This creates a problem because we can't tell which dump came from which machine. Therefore, we have had to patch the code within the kdump script init the kdump-initrd to make it use IP of the machine which crashed rather than 127.0.0.1. The patch that we use is in comment 13.

Thanks, Ben. Then it's clear to us what's happened.

But I have to say that it might be a correct way to use kdump in your side. In kdump glue shell script, we always suppose it's run on the system which safety is worried about and we want to deploy kdump to try to collect the dumping vmcore when crash is triggered on. With this worry and deployment, we need the system information of the system itself, not from others. Imagine that you have another machine with more than 100 CPUs, TB of memory, many PCI/PCIe devices, kinds of iSCSI/FCOE storage disks and locate root fs on one of them, powerful ethernet network interface cards, now you want to dump to remote ssh server through NIC by default, and dump to rootfs if failed to dump by ssh. DOES it work with your copied initrd?

In fact, I said much in above just for clarifying that kdump initrd need be created on their own system where you worry about its safety. You don't tell the network info to kdump, how could you expect it to get correct into and tell you? As for the scenario, I need discuss with team and see if there's a method to make it.

Thanks
Baoquan

(In reply to Baoquan He from comment #20)
> Thanks, Ben. Then it's clear to us what's happened.
> 
> But I have to say that it might be a correct way to use kdump in your side.
                                    ~ NOT

> In kdump glue shell script, we always suppose it's run on the system which

(In reply to Baoquan He from comment #20)
> Thanks, Ben. Then it's clear to us what's happened.
> 
> But I have to say that it might not be a correct way to use kdump in your side.

We have been through this at least two times before in other bugs.

> In kdump glue shell script, we always suppose it's run on the system which
> safety is worried about and we want to deploy kdump to try to collect the
> dumping vmcore when crash is triggered on. 

I know that you really really want that to be true and it would make things easier for you if you could alway make that assumption but after long discussions bringing in many other parties, we finally won the argument over that.

I'm going to make state it as plainly as possible:
THAT IS _NOT_ A VALID ASSUMPTION FOR AN ENTERPRISE DISTRIBUTION. End of story. Go back and talk to PM about this again if you must. That is a done and decided position. Yes, I understand that the people who initially architected the kdump scripts for RHEL7 failed to take that into account when they designed kdump. I'm sorry that technical requirement wasn't explicitly clear when it was first implemented, I'm sorry that you have to deal with the mess. No one expected the kdump functionality to regress so badly between rhel6 and rhel7.

> With this worry and deployment,
> we need the system information of the system itself, not from others.
> Imagine that you have another machine with more than 100 CPUs, TB of memory,
> many PCI/PCIe devices, kinds of iSCSI/FCOE storage disks and locate root fs
> on one of them, powerful ethernet network interface cards, now you want to
> dump to remote ssh server through NIC by default, and dump to rootfs if
> failed to dump by ssh. DOES it work with your copied initrd?
> 

You or someone else working on kdump made the same argument last time. The argument is not convincing. You can always dream up hypothetical situations where it won't constructing a stawman argument. When we last went through this argument two of the most convincing points were: 
1) RHEL5 and RHEL6 had no problem with this.
2) The creation of the kdump-initrd is not for the ultra-general case where the kind of variation that you are putting forth in your strawman argument might crop up. These kdump-initrd's are built to be deployed on a homogeneous compute cluster and as such the hardware and software configuration of the machines on which the kdump-initrd is generated matches that of the compute node as closely as possible. The exception being that the machine generating the kdump-initrd has a writable /boot


> In fact, I said much in above just for clarifying that kdump initrd need be
> created on their own system where you worry about its safety. You don't tell
> the network info to kdump, how could you expect it to get correct into and
> tell you? As for the scenario, I need discuss with team and see if there's a
> method to make it.
> 
> Thanks
> Baoquan

One way I can think of is you need find one of those diskless systems and attach a disk to create a kdump initrd with network dump setting, then copy it to other diskless systems. And dhcp is necessary, otherwise they will get fixed IP addr.

I didn't notice comment 22 and made comment 23. So this is not a bug according to comment 19 and 20. And I am in modd to talk about the architecture of kdump now because there are many other issues to do now. If a re-architecturing kdump issue is needed, please open a new bug. No further comment about this bug. We fix bug, any bug, no matter what it is, kernel bug or user space bug, we try to find out the root cause, and try to work with people to find a better way to fix or work around. Not a yelling or childish crying.

It's not a bug.

(In reply to Ben Woodard from comment #22)
> (In reply to Baoquan He from comment #20)
> > Thanks, Ben. Then it's clear to us what's happened.
> > 
> > But I have to say that it might not be a correct way to use kdump in your side.
> 
> We have been through this at least two times before in other bugs.
> 
> > In kdump glue shell script, we always suppose it's run on the system which
> > safety is worried about and we want to deploy kdump to try to collect the
> > dumping vmcore when crash is triggered on. 
> 
> I know that you really really want that to be true and it would make things
> easier for you if you could alway make that assumption but after long
> discussions bringing in many other parties, we finally won the argument over
> that.
> 
> I'm going to make state it as plainly as possible:
> THAT IS _NOT_ A VALID ASSUMPTION FOR AN ENTERPRISE DISTRIBUTION. End of
> story. Go back and talk to PM about this again if you must. That is a done
> and decided position. Yes, I understand that the people who initially
> architected the kdump scripts for RHEL7 failed to take that into account
> when they designed kdump. I'm sorry that technical requirement wasn't
> explicitly clear when it was first implemented, I'm sorry that you have to
> deal with the mess. No one expected the kdump functionality to regress so
> badly between rhel6 and rhel7.

It may happen to work in RHEL6, but that does not means it is supported, if QE do not test it.

Question is, arguing about this do not help to solve the problem, we need *understand* and make *clear* what is the problem and if we can help. If it is a new feature then we should re-evaluate if we can do it, filing a new feature bug makes sense.