Bug 1170869 - Failed to register RHEL7.1 to rhevm3.5 as host's internet ip was lost
Summary: Failed to register RHEL7.1 to rhevm3.5 as host's internet ip was lost
Keywords:
Status: CLOSED DUPLICATE of bug 1154399
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: vdsm
Version: 3.5.0
Hardware: x86_64
OS: All
medium
high
Target Milestone: ---
: 3.5.3
Assignee: Dan Kenigsberg
QA Contact: Meni Yakove
URL:
Whiteboard: network
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-12-05 02:59 UTC by Liushihui
Modified: 2016-02-10 19:48 UTC (History)
17 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-04-08 12:25:35 UTC
oVirt Team: Network
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
vdsm log (5.34 KB, application/x-gzip)
2014-12-05 03:00 UTC, Liushihui
no flags Details
deploy-host log on rhevm (320.10 KB, text/plain)
2014-12-05 03:04 UTC, Liushihui
no flags Details
event on rhevm web UI (181.23 KB, image/png)
2014-12-05 03:05 UTC, Liushihui
no flags Details
deploy-host-log-20141208 (266.34 KB, text/plain)
2014-12-08 02:16 UTC, Liushihui
no flags Details
vdsm config before register (634 bytes, application/x-gzip)
2014-12-09 03:19 UTC, Liushihui
no flags Details
vdsm config after register (1.05 KB, application/x-gzip)
2014-12-09 03:20 UTC, Liushihui
no flags Details
vdsm log-20141209 (5.79 KB, application/x-gzip)
2014-12-09 03:21 UTC, Liushihui
no flags Details

Description Liushihui 2014-12-05 02:59:44 UTC
Description of problem:
During register rhel7.1 to rhevm3.5, the rhel7.1's network won't reachable after start vdsm.

Version-Release number of selected component (if applicable):
RHEL7.1-20141201.0-Server-x86_64
rhevm-3.5.0-0.21.el6ev.noarch
vdsm-4.16.7.5-1.el7ev.x86_64

How reproducible:
50%

Steps to Reproduce:
1. Configure repos and install vdsm package on rhel7.1. success to install vdsm package.
# yum install vdsm -y
2. In the rhevm web UI, add rhel7.1 to rhevm3.5, Check the events(choose host-->Events tab), Failed to register rhel7.1 to rhevm3.5, the events always in "starting vdsm".
3. Go to rhel7.1, check the network connection
[root@hp-z220-03 ~]# service vdsmd status
Redirecting to /bin/systemctl status  vdsmd.service
vdsmd.service - Virtual Desktop Server Manager
   Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled)
   Active: active (running) since Thu 2014-12-04 17:10:22 CST; 17h ago
  Process: 22677 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start (code=exited, status=0/SUCCESS)
 Main PID: 22878 (vdsm)
   CGroup: /system.slice/vdsmd.service
           └─22878 /usr/bin/python /usr/share/vdsm/vdsm
Dec 05 10:22:04 hp-z220-03.qe.lab.eng.nay.redhat.com vdsm[22878]: vdsm vds.MultiProtocolAcceptor WARNING Unrecognized protocol: ''
Dec 05 10:22:07 hp-z220-03.qe.lab.eng.nay.redhat.com vdsm[22878]: vdsm vds.MultiProtocolAcceptor WARNING Unrecognized protocol: ''

[root@hp-z220-03 ~]# ifconfig
eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether a0:b3:cc:f7:e5:fd  txqueuelen 1000  (Ethernet)
        RX packets 59227  bytes 37628734 (35.8 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 31881  bytes 4507792 (4.2 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 20  memory 0xf7c00000-f7c20000  

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 0  (Local Loopback)
        RX packets 7555  bytes 1169032 (1.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 7555  bytes 1169032 (1.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

rhevm: flags=4099<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::a2b3:ccff:fef7:e5fd  prefixlen 64  scopeid 0x20<link>
        ether 00:00:00:00:00:00  txqueuelen 0  (Ethernet)
        RX packets 11534  bytes 14864026 (14.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 8227  bytes 911753 (890.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

virbr0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.122.1  netmask 255.255.255.0  broadcast 192.168.122.255
        ether 46:8f:f8:6c:bb:e7  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Actual results:
On the rhel7.1, Although vdsmd service was start, rhel7.1's internet ip was lost.Please see the vdsm log and host-deploy log in the attachment.

Expected results:
Success to register rhel7.1 to rhevm, the network connection should normally.

Additional info:

Comment 1 Liushihui 2014-12-05 03:00:52 UTC
Created attachment 964960 [details]
vdsm log

Comment 2 Liushihui 2014-12-05 03:04:00 UTC
Created attachment 964961 [details]
deploy-host log on rhevm

Comment 3 Liushihui 2014-12-05 03:05:08 UTC
Created attachment 964962 [details]
event on rhevm web UI

Comment 4 Yaniv Bronhaim 2014-12-07 12:24:05 UTC
vdsmd is down. can you try to start it or run "/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start" and say if it runs smoothly?

can't find anything relevant in the logs

Comment 5 Liushihui 2014-12-08 02:14:44 UTC
Hi Yaniv,

 Restart the network(it can't get the ip address before as this bug), then start vdsmd as your suggestion, it still failed to register rhevh to rhevm as "Host 10.66.100.110 does not comply with the cluster Default networks, the following networks are missing on host: 'rhevm'"(please see the detail log in the attachment "deploy-host-log-20141208"). when I check the network, it still can show rhevm as the following and vdsmd service run normally:

[root@hp-z220-05 ~]# ifconfig
eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::1260:4bff:fe5b:2b19  prefixlen 64  scopeid 0x20<link>
        ether 10:60:4b:5b:2b:19  txqueuelen 1000  (Ethernet)
        RX packets 484530  bytes 57384142 (54.7 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 19261  bytes 3973294 (3.7 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 20  memory 0xf7c00000-f7c20000  

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 0  (Local Loopback)
        RX packets 1608  bytes 173688 (169.6 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1608  bytes 173688 (169.6 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

rhevm: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.66.100.110  netmask 255.255.252.0  broadcast 10.66.103.255
        inet6 fe80::1260:4bff:fe5b:2b19  prefixlen 64  scopeid 0x20<link>
        ether 10:60:4b:5b:2b:19  txqueuelen 0  (Ethernet)
        RX packets 405141  bytes 44180408 (42.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 18761  bytes 3805632 (3.6 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@hp-z220-05 ~]# systemctl status vdsmd
vdsmd.service - Virtual Desktop Server Manager
   Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled)
   Active: active (running) since Mon 2014-12-08 10:06:59 CST; 4min 30s ago
  Process: 8816 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start (code=exited, status=0/SUCCESS)
 Main PID: 8914 (vdsm)
   CGroup: /system.slice/vdsmd.service
           └─8914 /usr/bin/python /usr/share/vdsm/vdsm

Dec 08 10:06:59 hp-z220-05.qe.lab.eng.nay.redhat.com python[8907]: DIGEST-MD5 client mech dispose
Dec 08 10:06:59 hp-z220-05.qe.lab.eng.nay.redhat.com python[8907]: DIGEST-MD5 common mech dispose
Dec 08 10:06:59 hp-z220-05.qe.lab.eng.nay.redhat.com systemd[1]: Started Virtual Desktop Server Manager.
Dec 08 10:06:59 hp-z220-05.qe.lab.eng.nay.redhat.com python[8914]: DIGEST-MD5 client step 2
Dec 08 10:06:59 hp-z220-05.qe.lab.eng.nay.redhat.com python[8914]: DIGEST-MD5 parse_server_challenge()
Dec 08 10:06:59 hp-z220-05.qe.lab.eng.nay.redhat.com python[8914]: DIGEST-MD5 ask_user_info()
Dec 08 10:06:59 hp-z220-05.qe.lab.eng.nay.redhat.com python[8914]: DIGEST-MD5 client step 2
Dec 08 10:06:59 hp-z220-05.qe.lab.eng.nay.redhat.com python[8914]: DIGEST-MD5 ask_user_info()
Dec 08 10:06:59 hp-z220-05.qe.lab.eng.nay.redhat.com python[8914]: DIGEST-MD5 make_client_response()
Dec 08 10:06:59 hp-z220-05.qe.lab.eng.nay.redhat.com python[8914]: DIGEST-MD5 client step 3

Thanks.
Liushihui



(In reply to Yaniv Bronhaim from comment #4)
> vdsmd is down. can you try to start it or run
> "/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start" and say if it runs
> smoothly?
> 
> can't find anything relevant in the logs

Comment 6 Liushihui 2014-12-08 02:16:25 UTC
Created attachment 965672 [details]
deploy-host-log-20141208

Comment 7 Dan Kenigsberg 2014-12-08 15:01:09 UTC
Is it a plain installation or an upgrade?
From which version? The logs suggests that a "rhevm" bridge existed prior to the installation.

How was the IP address obtained? Static or DHCP?

Could you share your ifcfg files before and after the installation, as well as the contents of /var/{lib,run}/vdsm ?

Comment 8 Liushihui 2014-12-09 03:16:59 UTC
(In reply to Dan Kenigsberg from comment #7)
> Is it a plain installation or an upgrade?
==> It's a plain installation of RHEL7.1-20141204.2-Server-x86_64.

> From which version? The logs suggests that a "rhevm" bridge existed prior to
> the installation.
==> Yes, The rhevm bridge exist because I configured it before the installation. 
[root@hp-z220-07 run]# ifconfig
bond0: flags=5123<UP,BROADCAST,MASTER,MULTICAST>  mtu 1500
        ether f6:91:28:42:2f:9b  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether b4:b5:2f:e0:ce:88  txqueuelen 1000  (Ethernet)
        RX packets 110621  bytes 42344873 (40.3 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 32573  bytes 2871080 (2.7 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 20  memory 0xf7c00000-f7c20000  

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 0  (Local Loopback)
        RX packets 1380  bytes 149004 (145.5 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1380  bytes 149004 (145.5 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

rhevm: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.66.100.112  netmask 255.255.252.0  broadcast 10.66.103.255
        inet6 fe80::b6b5:2fff:fee0:ce88  prefixlen 64  scopeid 0x20<link>
        ether b4:b5:2f:e0:ce:88  txqueuelen 0  (Ethernet)
        RX packets 13133  bytes 31757639 (30.2 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 9616  bytes 1150794 (1.0 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

virbr0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.122.1  netmask 255.255.255.0  broadcast 192.168.122.255
        ether 92:59:c2:90:bf:9d  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

> How was the IP address obtained? Static or DHCP?
==> DHCP

> Could you share your ifcfg files before and after the installation, as well
> as the contents of /var/{lib,run}/vdsm ?
==> Please see the two configurations and vdsm log in the attachment.

Comment 9 Liushihui 2014-12-09 03:19:17 UTC
Created attachment 966049 [details]
vdsm config before register

Comment 10 Liushihui 2014-12-09 03:20:20 UTC
Created attachment 966050 [details]
vdsm config after register

Comment 11 Liushihui 2014-12-09 03:21:45 UTC
Created attachment 966051 [details]
vdsm log-20141209

Comment 12 Dan Kenigsberg 2014-12-09 10:09:39 UTC
Could you share your /etc/sysconfig/network-scripts/ifcfg-* files before and after the installation?


I suspect that what you describe is an expected (albeit surprising) behaviour: network interfaces that were defined out-of-band are not upgraded to the 3.5's "unified persistence".

Comment 13 Liushihui 2014-12-10 06:30:52 UTC
(In reply to Dan Kenigsberg from comment #12)
> Could you share your /etc/sysconfig/network-scripts/ifcfg-* files before and
> after the installation?
==>Before install vdsm package, I configured the "rhevm" bridge manually(see as the following). Meanwhile, these two files haven't changed after installation and registration.
[root@hp-z220-07 network-scripts]# cat /etc/sysconfig/network-scripts/ifcfg-eno1
# Generated by dracut initrd
NAME="eno1"
DEVICE="eno1"
ONBOOT=yes
NETBOOT=yes
UUID="70b1ddf0-3faa-4740-ba4d-ec83365b1d9d"
IPV6INIT=yes
#BOOTPROTO=dhcp
TYPE=Ethernet
BRIDGE=rhevm
[root@hp-z220-07 network-scripts]# cat /etc/sysconfig/network-scripts/ifcfg-rhevm 
DEVICE=rhevm
ONBOOT=yes
TYPE=Bridge
DELAY=0
BOOTPROTO=dhcp

> I suspect that what you describe is an expected (albeit surprising)
> behaviour: network interfaces that were defined out-of-band are not upgraded
> to the 3.5's "unified persistence".
==> I'm sorry I'm not sure about this. Do you means the network connection between rhel and rhevm must be down when rhel register to rhevm?

Comment 14 Liushihui 2014-12-10 06:35:38 UTC
Maybe you check it on my test env:
rhel:10.66.100.113
rhevm:10.66.79.84

Comment 15 Dan Kenigsberg 2014-12-10 12:10:20 UTC
BTW, may I ask why you have pre-created ifcfg-rhevm? Does thing work as they should when you do not do that?

Comment 16 Nir Yechiel 2014-12-10 12:17:50 UTC
This is not considered a blocker for 3.5 as the right flow should not be creating the rhevm bridge manually. Will wait for answers to commet #15 to see if we can target it into a z-stream. 

Thanks,
Nir

Comment 17 Liushihui 2014-12-11 07:18:22 UTC
RHEL7.1 can registered to RHEVM3.5 successfully if rhevm bridge hasn't been created manually.

Comment 18 Liushihui 2014-12-11 07:37:45 UTC
I'm sorry for the wrong way. because we always pre-created ifcfg-rhevm before register rhel to rhevm and it hasn't any problem on the old rhevm build(rhevm3.2,rhevm3.3, rhevm3.4).
(In reply to Dan Kenigsberg from comment #15)
> BTW, may I ask why you have pre-created ifcfg-rhevm? Does thing work as they
> should when you do not do that?

Comment 19 Dan Kenigsberg 2014-12-11 09:41:59 UTC
(In reply to Liushihui from comment #18)
> I'm sorry for the wrong way. because we always pre-created ifcfg-rhevm
> before register rhel to rhevm and it hasn't any problem on the old rhevm
> build(rhevm3.2,rhevm3.3, rhevm3.4).

No need to apologize; I am happy to see that there is no current need of pre-creating the bridge and this is only a case of institutionalized memory.

I'll reduce severity as a simple workaround exists.

Comment 20 Eyal Edri 2015-02-25 08:40:12 UTC
3.5.1 is already full with bugs (over 80), and since none of these bugs were added as urgent for 3.5.1 release in the tracker bug, moving to 3.5.2

Comment 21 Lior Vernia 2015-04-08 12:25:35 UTC
This looks like a very early report (possibly the first!) of the referred bug - please re-open if you think differently.

*** This bug has been marked as a duplicate of bug 1154399 ***


Note You need to log in before you can comment on or make changes to this bug.