Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1507939

Summary: [Scale] VMs losing IP address
Product: [oVirt] ovirt-engine Reporter: mlehrer
Component: GeneralAssignee: bugs <bugs>
Status: CLOSED DUPLICATE QA Contact: mlehrer
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.2.0CC: bugs, dagur, michal.skrivanek, mlehrer
Target Milestone: ---Flags: mlehrer: needinfo-
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-11-08 20:57:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Virt RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
VDSM log, and qemu logs of specific VMs that lost ip, and /var/log/message from 1 of the vms which lost IP after having an assignment. none

Description mlehrer 2017-10-31 14:06:12 UTC
Created attachment 1345938 [details]
VDSM log, and qemu logs of specific VMs that lost ip, and /var/log/message from 1 of the vms which lost IP after having an assignment.

Description of problem:

Some vms are losing IP address after IP assignment in 200 VM Scale Scenario.

The vms are assigned a custom mac address from the range via the cluster mac address settings, and in the Administration -> Configure tab.

200 VMs in total are created & started in batches of 15 via Ansible ovirt modules. 

The last 15 VMs start without IP address while the rest have an IP.
After 30 minutes, an additional 11 VMs also lost their assigned IP.
Applying dhclient, or restarting services or powering down/up the VM in the hopes of getting an IP address from the mac address doesn't succeed.



Version-Release number of selected component (if applicable):
vdsm-4.20.4-1.git08e939e.el7.centos.x86_64
libvirt-daemon-driver-network-3.2.0-14.el7_4.3.x86_64


How reproducible:
Inconsistent, as it seems to occur a variable amount of time.

Steps to Reproduce:
1. Import Template
2. Create & Run 200 VMs in batches of 15 wait 1 - 2 hours
3. Identify which VMs no longer have IP

Actual results:
Some VMs do not retain IP.

Expected results:
VM to receive IP in 'up' status, and to retain IP.

Additional info:
VDSM log, and qemu logs of specific VMs that lost ip, and
/var/log/message from 1 of the vms which lost IP after having an assignment.

Comment 1 Yaniv Kaul 2017-10-31 14:43:43 UTC
1. Please ensure you have a reliable DHCP server that can serve an IP to this whole range.
2. Please ensure that there are no duplicate MACs or VMs with no MAC assigned to their NICs.
3. Please ensure there are no network drops (snoop the DHCP traffic) and that the DHCP server handles the load well.

Comment 2 Michal Skrivanek 2017-11-01 09:14:56 UTC
in the attached logs there is nothing to indicate qemu/libvirt/ovirt issue

Comment 3 mlehrer 2017-11-01 10:40:13 UTC
(In reply to Yaniv Kaul from comment #1)
> 1. Please ensure you have a reliable DHCP server that can serve an IP to
> this whole range.

We will speak with IT

> 2. Please ensure that there are no duplicate MACs or VMs with no MAC
> assigned to their NICs.

Ok

> 3. Please ensure there are no network drops (snoop the DHCP traffic) and
> that the DHCP server handles the load well.

(In reply to Michal Skrivanek from comment #2)
> in the attached logs there is nothing to indicate qemu/libvirt/ovirt issue

Understood, we will speak with IT regarding possible issues with the custom mac address range that they manage and built for this specific environment.

Comment 4 Yaniv Kaul 2017-11-01 11:18:34 UTC
Re-instating NEEDINFO. Waiting for additional information from QE.

Comment 5 Daniel Gur 2017-12-25 09:52:31 UTC

*** This bug has been marked as a duplicate of bug 1512572 ***