Bug 1595549 - Ovirt Guest VMs are not starting after version update-4.2.4.5-1.el7
Summary: Ovirt Guest VMs are not starting after version update-4.2.4.5-1.el7
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: ovirt-engine
Classification: oVirt
Component: BLL.Virt
Version: 4.2.2.4
Hardware: x86_64
OS: Linux
urgent
urgent with 1 vote
Target Milestone: ---
: ---
Assignee: Michal Skrivanek
QA Contact: Liran Rotenberg
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-06-27 06:47 UTC by Arghya Adhya
Modified: 2023-09-14 04:30 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-11-05 09:15:03 UTC
oVirt Team: Storage
Embargoed:

Attachments (Terms of Use)
vdsm log (2.13 MB, text/plain)
2018-06-28 13:01 UTC, Arghya Adhya 		no flags Details
upgrade log (79.45 KB, text/plain)
2018-06-28 13:02 UTC, Arghya Adhya 		no flags Details
supervdsm log (11.34 MB, text/plain)
2018-06-28 13:03 UTC, Arghya Adhya 		no flags Details
qemu log (52.24 KB, text/plain)
2018-06-28 13:06 UTC, Arghya Adhya 		no flags Details
vsdm log (5.92 MB, text/plain)
2018-08-07 20:10 UTC, Vincent 		no flags Details
supervsdm.log vincent (671.41 KB, text/plain)
2018-08-07 20:18 UTC, Vincent 		no flags Details
View All

Description Arghya Adhya 2018-06-27 06:47:53 UTC 
Description of problem:

Yesterday I have updated oVirt new version(4.2.4.5-1.el7), after that my VMs are not running. When I try to start VMs it giving me below error:--

VM NDL-BUGZILLA is down with error. Exit message: internal error: qemu unexpectedly closed the monitor: 2018-06-27T06:30:33.403163Z qemu-kvm: -drive file=/rhev/data-center/mnt/192.168.1.100:_Ovirt-export_data/22d10021-f581-47de-98a4-fae5163da8dd/images/36572886-da1c-4603-80f8-46f3c658b817/e1491323-fec7-4d4b-a4de-1362a586bae7,format=qcow2,if=none,id=drive-ua-36572886-da1c-4603-80f8-46f3c658b817,serial=36572886-da1c-4603-80f8-46f3c658b817,cache=none,werror=stop,rerror=stop,aio=threads: Failed to lock byte 100
Unexpected error in raw_apply_lock_bytes() at block/file-posix.c:642:
2018-06-27T06:30:33.412809Z qemu-kvm: -drive file=/rhev/data-center/mnt/192.168.1.100:_Ovirt-export_data/22d10021-f581-47de-98a4-fae5163da8dd/images/36572886-da1c-4603-80f8-46f3c658b817/e1491323-fec7-4d4b-a4de-1362a586bae7,format=qcow2,if=none,id=drive-ua-36572886-da1c-4603-80f8-46f3c658b817,serial=36572886-da1c-4603-80f8-46f3c658b817,cache=none,werror=stop,rerror=stop,aio=threads: Failed to lock byte 100.


Version-Release number of selected component (if applicable):

4.2.4.5-1.el7

Please help me to run my VMs, if anything more information required inform me.
Comment 1 Doron Fediuck 2018-06-28 04:20:06 UTC 
Which vdsm version are you using? CentOS version?
Please provide upgrade, vdsm and qemu logs.
Comment 2 Arghya Adhya 2018-06-28 05:48:04 UTC 
OS info--3.10.0-862.3.3.el7.x86_64 #1 SMP Fri Jun 15 04:15:27 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

VDSM version-vdsm-4.20.32-1.el7.x86_64
Comment 3 Arghya Adhya 2018-06-28 05:51:00 UTC 
Upgrade log--

MainThread::DEBUG::2018-01-03 14:46:35,629::libvirtconnection::163::root::(get) trying to connect libvirt
MainThread::DEBUG::2018-01-03 14:46:39,026::cmdutils::150::root::(exec_cmd) lshw -json -disable usb -disable pcmcia -disable isapnp -disable ide -disable scsi -disable dmi -disable memory -disable cpuinfo (cwd None)
MainThread::DEBUG::2018-01-03 14:46:39,290::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-01-03 14:46:39,380::cmdutils::150::root::(exec_cmd) /sbin/tc qdisc show (cwd None)
MainThread::DEBUG::2018-01-03 14:46:39,419::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-01-03 14:46:39,422::cmdutils::150::root::(exec_cmd) /usr/share/openvswitch/scripts/ovs-ctl status (cwd None)
MainThread::DEBUG::2018-01-03 14:46:39,554::cmdutils::158::root::(exec_cmd) FAILED: <err> = ''; <rc> = 1
MainThread::INFO::2018-01-03 14:46:39,558::netconfpersistence::194::root::(_clearDisk) Clearing /var/lib/vdsm/staging/netconf/nets/ and /var/lib/vdsm/staging/netconf/bonds/
MainThread::WARNING::2018-01-03 14:46:39,558::fileutils::96::root::(rm_tree) Directory: /var/lib/vdsm/staging/netconf/nets/ already removed
MainThread::WARNING::2018-01-03 14:46:39,558::fileutils::96::root::(rm_tree) Directory: /var/lib/vdsm/staging/netconf/bonds/ already removed
MainThread::INFO::2018-01-03 14:46:39,559::netconfpersistence::139::root::(save) Saved new config RunningConfig({}, {}) to /var/lib/vdsm/staging/netconf/nets/ and /var/lib/vdsm/staging/netconf/bonds/
MainThread::DEBUG::2018-01-03 14:46:39,561::cmdutils::150::root::(exec_cmd) /usr/share/openvswitch/scripts/ovs-ctl status (cwd None)
MainThread::DEBUG::2018-01-03 14:46:39,579::cmdutils::158::root::(exec_cmd) FAILED: <err> = ''; <rc> = 1
MainThread::DEBUG::2018-01-03 15:02:05,561::libvirtconnection::163::root::(get) trying to connect libvirt
MainThread::DEBUG::2018-01-03 15:02:05,606::cmdutils::150::root::(exec_cmd) /usr/share/openvswitch/scripts/ovs-ctl status (cwd None)
MainThread::DEBUG::2018-01-03 15:02:05,640::cmdutils::158::root::(exec_cmd) FAILED: <err> = ''; <rc> = 1
MainThread::DEBUG::2018-01-03 15:43:13,374::libvirtconnection::163::root::(get) trying to connect libvirt
MainThread::INFO::2018-01-03 15:43:13,420::netconfpersistence::194::root::(_clearDisk) Clearing /var/lib/vdsm/staging/netconf/nets/ and /var/lib/vdsm/staging/netconf/bonds/
MainThread::INFO::2018-01-03 15:43:13,422::netconfpersistence::139::root::(save) Saved new config RunningConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/staging/netconf/nets/ and /var/lib/vdsm/staging/netconf/bonds/
MainThread::INFO::2018-01-03 15:43:13,423::netconfpersistence::194::root::(_clearDisk) Clearing /var/lib/vdsm/persistence/netconf/nets/ and /var/lib/vdsm/persistence/netconf/bonds/
MainThread::INFO::2018-01-03 15:43:13,425::netconfpersistence::139::root::(save) Saved new config PersistentConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/persistence/netconf/nets/ and /var/lib/vdsm/persistence/netconf/bonds/
MainThread::DEBUG::2018-01-03 15:43:13,425::cmdutils::150::root::(exec_cmd) /usr/share/openvswitch/scripts/ovs-ctl status (cwd None)
MainThread::DEBUG::2018-01-03 15:43:13,462::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-01-03 15:43:13,465::vsctl::58::root::(commit) Executing commands: /usr/bin/ovs-vsctl --oneline --format=json -- list Bridge -- list Port -- list Interface
MainThread::DEBUG::2018-01-03 15:43:13,465::cmdutils::150::root::(exec_cmd) /usr/bin/ovs-vsctl --oneline --format=json -- list Bridge -- list Port -- list Interface (cwd None)
MainThread::DEBUG::2018-01-03 15:43:13,545::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-01-03 15:43:13,548::vsctl::58::root::(commit) Executing commands: /usr/bin/ovs-vsctl --oneline --format=json -- set open . external-ids:ovn-bridge-mappings=""
MainThread::DEBUG::2018-01-03 15:43:13,561::cmdutils::150::root::(exec_cmd) /usr/bin/ovs-vsctl --oneline --format=json -- set open . 'external-ids:ovn-bridge-mappings=""' (cwd None)
MainThread::DEBUG::2018-01-03 15:43:13,637::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-01-03 19:08:32,200::libvirtconnection::163::root::(get) trying to connect libvirt
MainThread::INFO::2018-01-03 19:08:32,235::netconfpersistence::194::root::(_clearDisk) Clearing /var/lib/vdsm/staging/netconf/nets/ and /var/lib/vdsm/staging/netconf/bonds/
MainThread::INFO::2018-01-03 19:08:32,241::netconfpersistence::139::root::(save) Saved new config RunningConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/staging/netconf/nets/ and /var/lib/vdsm/staging/netconf/bonds/
MainThread::INFO::2018-01-03 19:08:32,241::netconfpersistence::194::root::(_clearDisk) Clearing /var/lib/vdsm/persistence/netconf/nets/ and /var/lib/vdsm/persistence/netconf/bonds/
MainThread::INFO::2018-01-03 19:08:32,241::netconfpersistence::139::root::(save) Saved new config PersistentConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/persistence/netconf/nets/ and /var/lib/vdsm/persistence/netconf/bonds/
MainThread::DEBUG::2018-01-03 19:08:32,242::cmdutils::150::root::(exec_cmd) /usr/share/openvswitch/scripts/ovs-ctl status (cwd None)
MainThread::DEBUG::2018-01-03 19:08:32,257::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-01-03 19:08:32,262::vsctl::58::root::(commit) Executing commands: /usr/bin/ovs-vsctl --oneline --format=json -- list Bridge -- list Port -- list Interface
MainThread::DEBUG::2018-01-03 19:08:32,263::cmdutils::150::root::(exec_cmd) /usr/bin/ovs-vsctl --oneline --format=json -- list Bridge -- list Port -- list Interface (cwd None)
MainThread::DEBUG::2018-01-03 19:08:32,297::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-01-03 19:08:32,299::vsctl::58::root::(commit) Executing commands: /usr/bin/ovs-vsctl --oneline --format=json -- set open . external-ids:ovn-bridge-mappings=""
MainThread::DEBUG::2018-01-03 19:08:32,299::cmdutils::150::root::(exec_cmd) /usr/bin/ovs-vsctl --oneline --format=json -- set open . 'external-ids:ovn-bridge-mappings=""' (cwd None)
MainThread::DEBUG::2018-01-03 19:08:32,333::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-01-04 01:26:01,549::libvirtconnection::163::root::(get) trying to connect libvirt
MainThread::INFO::2018-01-04 01:26:01,567::netconfpersistence::194::root::(_clearDisk) Clearing /var/lib/vdsm/staging/netconf/nets/ and /var/lib/vdsm/staging/netconf/bonds/
MainThread::INFO::2018-01-04 01:26:01,568::netconfpersistence::139::root::(save) Saved new config RunningConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/staging/netconf/nets/ and /var/lib/vdsm/staging/netconf/bonds/
MainThread::INFO::2018-01-04 01:26:01,568::netconfpersistence::194::root::(_clearDisk) Clearing /var/lib/vdsm/persistence/netconf/nets/ and /var/lib/vdsm/persistence/netconf/bonds/
MainThread::INFO::2018-01-04 01:26:01,569::netconfpersistence::139::root::(save) Saved new config PersistentConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/persistence/netconf/nets/ and /var/lib/vdsm/persistence/netconf/bonds/
MainThread::DEBUG::2018-01-04 01:26:01,569::cmdutils::150::root::(exec_cmd) /usr/share/openvswitch/scripts/ovs-ctl status (cwd None)
MainThread::DEBUG::2018-01-04 01:26:01,585::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-01-04 01:26:01,586::vsctl::58::root::(commit) Executing commands: /usr/bin/ovs-vsctl --oneline --format=json -- list Bridge -- list Port -- list Interface
MainThread::DEBUG::2018-01-04 01:26:01,586::cmdutils::150::root::(exec_cmd) /usr/bin/ovs-vsctl --oneline --format=json -- list Bridge -- list Port -- list Interface (cwd None)
MainThread::DEBUG::2018-01-04 01:26:01,619::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-01-04 01:26:01,621::vsctl::58::root::(commit) Executing commands: /usr/bin/ovs-vsctl --oneline --format=json -- set open . external-ids:ovn-bridge-mappings=""
MainThread::DEBUG::2018-01-04 01:26:01,621::cmdutils::150::root::(exec_cmd) /usr/bin/ovs-vsctl --oneline --format=json -- set open . 'external-ids:ovn-bridge-mappings=""' (cwd None)
MainThread::DEBUG::2018-01-04 01:26:01,653::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-01-10 11:42:48,306::libvirtconnection::163::root::(get) trying to connect libvirt
MainThread::INFO::2018-01-10 11:42:48,329::netconfpersistence::194::root::(_clearDisk) Clearing /var/lib/vdsm/staging/netconf/nets/ and /var/lib/vdsm/staging/netconf/bonds/
MainThread::INFO::2018-01-10 11:42:48,339::netconfpersistence::139::root::(save) Saved new config RunningConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/staging/netconf/nets/ and /var/lib/vdsm/staging/netconf/bonds/
MainThread::INFO::2018-01-10 11:42:48,339::netconfpersistence::194::root::(_clearDisk) Clearing /var/lib/vdsm/persistence/netconf/nets/ and /var/lib/vdsm/persistence/netconf/bonds/
MainThread::INFO::2018-01-10 11:42:48,340::netconfpersistence::139::root::(save) Saved new config PersistentConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/persistence/netconf/nets/ and /var/lib/vdsm/persistence/netconf/bonds/
MainThread::DEBUG::2018-01-10 11:42:48,340::cmdutils::150::root::(exec_cmd) /usr/share/openvswitch/scripts/ovs-ctl status (cwd None)
MainThread::DEBUG::2018-01-10 11:42:48,355::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-01-10 11:42:48,356::vsctl::58::root::(commit) Executing commands: /usr/bin/ovs-vsctl --oneline --format=json -- list Bridge -- list Port -- list Interface
MainThread::DEBUG::2018-01-10 11:42:48,356::cmdutils::150::root::(exec_cmd) /usr/bin/ovs-vsctl --oneline --format=json -- list Bridge -- list Port -- list Interface (cwd None)
MainThread::DEBUG::2018-01-10 11:42:48,390::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-01-10 11:42:48,391::vsctl::58::root::(commit) Executing commands: /usr/bin/ovs-vsctl --oneline --format=json -- set open . external-ids:ovn-bridge-mappings=""
MainThread::DEBUG::2018-01-10 11:42:48,392::cmdutils::150::root::(exec_cmd) /usr/bin/ovs-vsctl --oneline --format=json -- set open . 'external-ids:ovn-bridge-mappings=""' (cwd None)
MainThread::DEBUG::2018-01-10 11:42:48,424::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-01-11 11:38:59,568::libvirtconnection::163::root::(get) trying to connect libvirt
MainThread::INFO::2018-01-11 11:38:59,622::netconfpersistence::194::root::(_clearDisk) Clearing /var/lib/vdsm/staging/netconf/nets/ and /var/lib/vdsm/staging/netconf/bonds/
MainThread::INFO::2018-01-11 11:38:59,631::netconfpersistence::139::root::(save) Saved new config RunningConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/staging/netconf/nets/ and /var/lib/vdsm/staging/netconf/bonds/
MainThread::INFO::2018-01-11 11:38:59,632::netconfpersistence::194::root::(_clearDisk) Clearing /var/lib/vdsm/persistence/netconf/nets/ and /var/lib/vdsm/persistence/netconf/bonds/
MainThread::INFO::2018-01-11 11:38:59,634::netconfpersistence::139::root::(save) Saved new config PersistentConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/persistence/netconf/nets/ and /var/lib/vdsm/persistence/netconf/bonds/
MainThread::DEBUG::2018-01-11 11:38:59,644::cmdutils::150::root::(exec_cmd) /usr/share/openvswitch/scripts/ovs-ctl status (cwd None)
MainThread::DEBUG::2018-01-11 11:38:59,682::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-01-11 11:38:59,684::vsctl::58::root::(commit) Executing commands: /usr/bin/ovs-vsctl --oneline --format=json -- list Bridge -- list Port -- list Interface
MainThread::DEBUG::2018-01-11 11:38:59,685::cmdutils::150::root::(exec_cmd) /usr/bin/ovs-vsctl --oneline --format=json -- list Bridge -- list Port -- list Interface (cwd None)
MainThread::DEBUG::2018-01-11 11:38:59,767::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-01-11 11:38:59,770::vsctl::58::root::(commit) Executing commands: /usr/bin/ovs-vsctl --oneline --format=json -- set open . external-ids:ovn-bridge-mappings=""
MainThread::DEBUG::2018-01-11 11:38:59,771::cmdutils::150::root::(exec_cmd) /usr/bin/ovs-vsctl --oneline --format=json -- set open . 'external-ids:ovn-bridge-mappings=""' (cwd None)
MainThread::DEBUG::2018-01-11 11:38:59,852::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-02-13 13:00:44,535::libvirtconnection::167::root::(get) trying to connect libvirt
MainThread::INFO::2018-02-13 13:00:47,518::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/staging/netconf
MainThread::ERROR::2018-02-13 13:00:47,519::fileutils::53::root::(rm_file) Removing file: /var/lib/vdsm/staging/netconf failed
Comment 4 Arghya Adhya 2018-06-28 05:51:33 UTC 
upgrade log continue--
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/common/fileutils.py", line 48, in rm_file
    os.unlink(file_to_remove)
OSError: [Errno 21] Is a directory: '/var/lib/vdsm/staging/netconf'
MainThread::INFO::2018-02-13 13:00:47,527::netconfpersistence::150::root::(save) Saved new config RunningConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/staging/netconf/nets and /var/lib/vdsm/staging/netconf/bonds
MainThread::INFO::2018-02-13 13:00:47,529::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/persistence/netconf
MainThread::INFO::2018-02-13 13:00:47,533::netconfpersistence::150::root::(save) Saved new config PersistentConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/persistence/netconf/nets and /var/lib/vdsm/persistence/netconf/bonds
MainThread::DEBUG::2018-02-13 13:00:47,544::cmdutils::150::root::(exec_cmd) /usr/share/openvswitch/scripts/ovs-ctl status (cwd None)
MainThread::DEBUG::2018-02-13 13:00:47,589::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-02-13 13:00:47,597::vsctl::58::root::(commit) Executing commands: /usr/bin/ovs-vsctl --oneline --format=json -- list Bridge -- list Port -- list Interface
MainThread::DEBUG::2018-02-13 13:00:47,597::cmdutils::150::root::(exec_cmd) /usr/bin/ovs-vsctl --oneline --format=json -- list Bridge -- list Port -- list Interface (cwd None)
MainThread::DEBUG::2018-02-13 13:00:47,703::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-02-13 13:00:47,708::vsctl::58::root::(commit) Executing commands: /usr/bin/ovs-vsctl --oneline --format=json -- set open . external-ids:ovn-bridge-mappings=""
MainThread::DEBUG::2018-02-13 13:00:47,708::cmdutils::150::root::(exec_cmd) /usr/bin/ovs-vsctl --oneline --format=json -- set open . 'external-ids:ovn-bridge-mappings=""' (cwd None)
MainThread::DEBUG::2018-02-13 13:00:47,791::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-02-13 17:24:02,924::libvirtconnection::167::root::(get) trying to connect libvirt
MainThread::INFO::2018-02-13 17:24:02,955::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/staging/netconf
MainThread::INFO::2018-02-13 17:24:02,960::netconfpersistence::150::root::(save) Saved new config RunningConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/staging/netconf/nets and /var/lib/vdsm/staging/netconf/bonds
MainThread::INFO::2018-02-13 17:24:02,961::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/persistence/netconf
MainThread::INFO::2018-02-13 17:24:02,963::netconfpersistence::150::root::(save) Saved new config PersistentConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/persistence/netconf/nets and /var/lib/vdsm/persistence/netconf/bonds
MainThread::DEBUG::2018-02-13 17:24:02,963::cmdutils::150::root::(exec_cmd) /usr/share/openvswitch/scripts/ovs-ctl status (cwd None)
MainThread::DEBUG::2018-02-13 17:24:02,981::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-02-13 17:24:02,982::vsctl::58::root::(commit) Executing commands: /usr/bin/ovs-vsctl --oneline --format=json -- list Bridge -- list Port -- list Interface
MainThread::DEBUG::2018-02-13 17:24:02,982::cmdutils::150::root::(exec_cmd) /usr/bin/ovs-vsctl --oneline --format=json -- list Bridge -- list Port -- list Interface (cwd None)
MainThread::DEBUG::2018-02-13 17:24:03,019::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-02-13 17:24:03,021::vsctl::58::root::(commit) Executing commands: /usr/bin/ovs-vsctl --oneline --format=json -- set open . external-ids:ovn-bridge-mappings=""
MainThread::DEBUG::2018-02-13 17:24:03,021::cmdutils::150::root::(exec_cmd) /usr/bin/ovs-vsctl --oneline --format=json -- set open . 'external-ids:ovn-bridge-mappings=""' (cwd None)
MainThread::DEBUG::2018-02-13 17:24:03,057::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-02-13 17:40:46,929::libvirtconnection::167::root::(get) trying to connect libvirt
MainThread::INFO::2018-02-13 17:40:46,961::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/staging/netconf
MainThread::INFO::2018-02-13 17:40:46,971::netconfpersistence::150::root::(save) Saved new config RunningConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/staging/netconf/nets and /var/lib/vdsm/staging/netconf/bonds
MainThread::INFO::2018-02-13 17:40:46,972::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/persistence/netconf
MainThread::INFO::2018-02-13 17:40:46,974::netconfpersistence::150::root::(save) Saved new config PersistentConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/persistence/netconf/nets and /var/lib/vdsm/persistence/netconf/bonds
MainThread::DEBUG::2018-02-13 17:40:46,974::cmdutils::150::root::(exec_cmd) /usr/share/openvswitch/scripts/ovs-ctl status (cwd None)
MainThread::DEBUG::2018-02-13 17:40:46,991::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-02-13 17:40:46,992::vsctl::58::root::(commit) Executing commands: /usr/bin/ovs-vsctl --oneline --format=json -- list Bridge -- list Port -- list Interface
MainThread::DEBUG::2018-02-13 17:40:46,992::cmdutils::150::root::(exec_cmd) /usr/bin/ovs-vsctl --oneline --format=json -- list Bridge -- list Port -- list Interface (cwd None)
MainThread::DEBUG::2018-02-13 17:40:47,026::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-02-13 17:40:47,028::vsctl::58::root::(commit) Executing commands: /usr/bin/ovs-vsctl --oneline --format=json -- set open . external-ids:ovn-bridge-mappings=""
MainThread::DEBUG::2018-02-13 17:40:47,028::cmdutils::150::root::(exec_cmd) /usr/bin/ovs-vsctl --oneline --format=json -- set open . 'external-ids:ovn-bridge-mappings=""' (cwd None)
MainThread::DEBUG::2018-02-13 17:40:47,060::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-03-07 18:59:56,283::libvirtconnection::167::root::(get) trying to connect libvirt
MainThread::INFO::2018-03-07 18:59:57,578::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/staging/netconf
MainThread::INFO::2018-03-07 18:59:57,643::netconfpersistence::150::root::(save) Saved new config RunningConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/staging/netconf/nets and /var/lib/vdsm/staging/netconf/bonds
MainThread::INFO::2018-03-07 18:59:57,644::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/persistence/netconf
MainThread::INFO::2018-03-07 18:59:57,649::netconfpersistence::150::root::(save) Saved new config PersistentConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/persistence/netconf/nets and /var/lib/vdsm/persistence/netconf/bonds
MainThread::DEBUG::2018-03-07 18:59:57,649::cmdutils::150::root::(exec_cmd) /usr/share/openvswitch/scripts/ovs-ctl status (cwd None)
MainThread::DEBUG::2018-03-07 18:59:57,694::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-03-07 18:59:57,703::vsctl::58::root::(commit) Executing commands: /usr/bin/ovs-vsctl --oneline --format=json -- list Bridge -- list Port -- list Interface
MainThread::DEBUG::2018-03-07 18:59:57,704::cmdutils::150::root::(exec_cmd) /usr/bin/ovs-vsctl --oneline --format=json -- list Bridge -- list Port -- list Interface (cwd None)
MainThread::DEBUG::2018-03-07 18:59:57,810::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-03-07 18:59:57,814::vsctl::58::root::(commit) Executing commands: /usr/bin/ovs-vsctl --oneline --format=json -- set open . external-ids:ovn-bridge-mappings=""
MainThread::DEBUG::2018-03-07 18:59:57,815::cmdutils::150::root::(exec_cmd) /usr/bin/ovs-vsctl --oneline --format=json -- set open . 'external-ids:ovn-bridge-mappings=""' (cwd None)
MainThread::DEBUG::2018-03-07 18:59:57,898::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-03-08 16:50:01,265::libvirtconnection::167::root::(get) trying to connect libvirt
MainThread::INFO::2018-03-08 16:50:01,341::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/staging/netconf
MainThread::INFO::2018-03-08 16:50:01,351::netconfpersistence::150::root::(save) Saved new config RunningConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/staging/netconf/nets and /var/lib/vdsm/staging/netconf/bonds
MainThread::INFO::2018-03-08 16:50:01,351::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/persistence/netconf
MainThread::INFO::2018-03-08 16:50:01,353::netconfpersistence::150::root::(save) Saved new config PersistentConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/persistence/netconf/nets and /var/lib/vdsm/persistence/netconf/bonds
MainThread::DEBUG::2018-03-08 16:50:01,353::cmdutils::150::root::(exec_cmd) /usr/share/openvswitch/scripts/ovs-ctl status (cwd None)
MainThread::DEBUG::2018-03-08 16:50:01,371::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-03-08 16:50:01,372::vsctl::58::root::(commit) Executing commands: /usr/bin/ovs-vsctl --oneline --format=json -- list Bridge -- list Port -- list Interface
MainThread::DEBUG::2018-03-08 16:50:01,372::cmdutils::150::root::(exec_cmd) /usr/bin/ovs-vsctl --oneline --format=json -- list Bridge -- list Port -- list Interface (cwd None)
MainThread::DEBUG::2018-03-08 16:50:01,407::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-03-08 16:50:01,409::vsctl::58::root::(commit) Executing commands: /usr/bin/ovs-vsctl --oneline --format=json -- set open . external-ids:ovn-bridge-mappings=""
MainThread::DEBUG::2018-03-08 16:50:01,409::cmdutils::150::root::(exec_cmd) /usr/bin/ovs-vsctl --oneline --format=json -- set open . 'external-ids:ovn-bridge-mappings=""' (cwd None)
MainThread::DEBUG::2018-03-08 16:50:01,442::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-03-08 17:05:23,252::libvirtconnection::167::root::(get) trying to connect libvirt
MainThread::INFO::2018-03-08 17:05:23,306::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/staging/netconf
MainThread::INFO::2018-03-08 17:05:23,314::netconfpersistence::150::root::(save) Saved new config RunningConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/staging/netconf/nets and /var/lib/vdsm/staging/netconf/bonds
MainThread::INFO::2018-03-08 17:05:23,315::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/persistence/netconf
MainThread::INFO::2018-03-08 17:05:23,319::netconfpersistence::150::root::(save) Saved new config PersistentConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/persistence/netconf/nets and /var/lib/vdsm/persistence/netconf/bonds
MainThread::DEBUG::2018-03-08 17:05:23,319::cmdutils::150::root::(exec_cmd) /usr/share/openvswitch/scripts/ovs-ctl status (cwd None)
MainThread::DEBUG::2018-03-08 17:05:23,341::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-03-08 17:05:23,342::vsctl::58::root::(commit) Executing commands: /usr/bin/ovs-vsctl --oneline --format=json -- list Bridge -- list Port -- list Interface
MainThread::DEBUG::2018-03-08 17:05:23,343::cmdutils::150::root::(exec_cmd) /usr/bin/ovs-vsctl --oneline --format=json -- list Bridge -- list Port -- list Interface (cwd None)
MainThread::DEBUG::2018-03-08 17:05:23,380::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-03-08 17:05:23,382::vsctl::58::root::(commit) Executing commands: /usr/bin/ovs-vsctl --oneline --format=json -- set open . external-ids:ovn-bridge-mappings=""
MainThread::DEBUG::2018-03-08 17:05:23,382::cmdutils::150::root::(exec_cmd) /usr/bin/ovs-vsctl --oneline --format=json -- set open . 'external-ids:ovn-bridge-mappings=""' (cwd None)
MainThread::DEBUG::2018-03-08 17:05:23,417::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-03-08 17:42:00,671::libvirtconnection::167::root::(get) trying to connect libvirt
MainThread::INFO::2018-03-08 17:42:00,706::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/staging/netconf
MainThread::INFO::2018-03-08 17:42:00,715::netconfpersistence::150::root::(save) Saved new config RunningConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/staging/netconf/nets and /var/lib/vdsm/staging/netconf/bonds
MainThread::INFO::2018-03-08 17:42:00,716::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/persistence/netconf
MainThread::INFO::2018-03-08 17:42:00,718::netconfpersistence::150::root::(save) Saved new config PersistentConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/persistence/netconf/nets and /var/lib/vdsm/persistence/netconf/bonds
MainThread::DEBUG::2018-03-08 17:42:00,718::cmdutils::150::root::(exec_cmd) /usr/share/openvswitch/scripts/ovs-ctl status (cwd None)
MainThread::DEBUG::2018-03-08 17:42:00,736::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-03-08 17:42:00,737::vsctl::58::root::(commit) Executing commands: /usr/bin/ovs-vsctl --oneline --format=json -- list Bridge -- list Port -- list Interface
MainThread::DEBUG::2018-03-08 17:42:00,737::cmdutils::150::root::(exec_cmd) /usr/bin/ovs-vsctl --oneline --format=json -- list Bridge -- list Port -- list Interface (cwd None)
MainThread::DEBUG::2018-03-08 17:42:00,772::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-03-08 17:42:00,773::vsctl::58::root::(commit) Executing commands: /usr/bin/ovs-vsctl --oneline --format=json -- set open . external-ids:ovn-bridge-mappings=""
MainThread::DEBUG::2018-03-08 17:42:00,773::cmdutils::150::root::(exec_cmd) /usr/bin/ovs-vsctl --oneline --format=json -- set open . 'external-ids:ovn-bridge-mappings=""' (cwd None)
MainThread::DEBUG::2018-03-08 17:42:00,807::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-03-08 17:45:28,693::libvirtconnection::167::root::(get) trying to connect libvirt
MainThread::INFO::2018-03-08 17:45:28,741::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/staging/netconf
MainThread::INFO::2018-03-08 17:45:28,746::netconfpersistence::150::root::(save) Saved new config RunningConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/staging/netconf/nets and /var/lib/vdsm/staging/netconf/bonds
MainThread::INFO::2018-03-08 17:45:28,747::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/persistence/netconf
MainThread::INFO::2018-03-08 17:45:28,751::netconfpersistence::150::root::(save) Saved new config PersistentConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/persistence/netconf/nets and /var/lib/vdsm/persistence/netconf/bonds
MainThread::DEBUG::2018-03-08 17:45:28,752::cmdutils::150::root::(exec_cmd) /usr/share/openvswitch/scripts/ovs-ctl status (cwd None)
MainThread::DEBUG::2018-03-08 17:45:28,792::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-03-08 17:45:28,795::vsctl::58::root::(commit) Executing commands: /usr/bin/ovs-vsctl --oneline --format=json -- list Bridge -- list Port -- list Interface
MainThread::DEBUG::2018-03-08 17:45:28,795::cmdutils::150::root::(exec_cmd) /usr/bin/ovs-vsctl --oneline --format=json -- list Bridge -- list Port -- list Interface (cwd None)
MainThread::DEBUG::2018-03-08 17:45:28,880::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-03-08 17:45:28,885::vsctl::58::root::(commit) Executing commands: /usr/bin/ovs-vsctl --oneline --format=json -- set open . external-ids:ovn-bridge-mappings=""
MainThread::DEBUG::2018-03-08 17:45:28,885::cmdutils::150::root::(exec_cmd) /usr/bin/ovs-vsctl --oneline --format=json -- set open . 'external-ids:ovn-bridge-mappings=""' (cwd None)
MainThread::DEBUG::2018-03-08 17:45:28,969::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-04-03 17:57:59,429::libvirtconnection::167::root::(get) trying to connect libvirt
MainThread::INFO::2018-04-03 17:58:00,383::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/staging/netconf
MainThread::INFO::2018-04-03 17:58:00,392::netconfpersistence::150::root::(save) Saved new config RunningConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/staging/netconf/nets and /var/lib/vdsm/staging/netconf/bonds
MainThread::INFO::2018-04-03 17:58:00,393::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/persistence/netconf
MainThread::INFO::2018-04-03 17:58:00,394::netconfpersistence::150::root::(save) Saved new config PersistentConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/persistence/netconf/nets and /var/lib/vdsm/persistence/netconf/bonds
MainThread::DEBUG::2018-04-03 17:58:00,395::cmdutils::150::root::(exec_cmd) /usr/share/openvswitch/scripts/ovs-ctl status (cwd None)
MainThread::DEBUG::2018-04-03 17:58:00,413::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-04-03 17:58:00,414::vsctl::58::root::(commit) Executing commands: /usr/bin/ovs-vsctl --oneline --format=json -- list Bridge -- list Port -- list Interface
MainThread::DEBUG::2018-04-03 17:58:00,414::cmdutils::150::root::(exec_cmd) /usr/bin/ovs-vsctl --oneline --format=json -- list Bridge -- list Port -- list Interface (cwd None)
MainThread::DEBUG::2018-04-03 17:58:00,447::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-04-03 17:58:00,449::vsctl::58::root::(commit) Executing commands: /usr/bin/ovs-vsctl --oneline --format=json -- set open . external-ids:ovn-bridge-mappings=""
MainThread::DEBUG::2018-04-03 17:58:00,449::cmdutils::150::root::(exec_cmd) /usr/bin/ovs-vsctl --oneline --format=json -- set open . 'external-ids:ovn-bridge-mappings=""' (cwd None)
MainThread::DEBUG::2018-04-03 17:58:00,491::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-04-03 18:35:51,198::libvirtconnection::167::root::(get) trying to connect libvirt
MainThread::INFO::2018-04-03 18:35:52,929::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/staging/netconf
MainThread::INFO::2018-04-03 18:35:53,067::netconfpersistence::150::root::(save) Saved new config RunningConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/staging/netconf/nets and /var/lib/vdsm/staging/netconf/bonds
MainThread::INFO::2018-04-03 18:35:53,068::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/persistence/netconf
MainThread::INFO::2018-04-03 18:35:53,126::netconfpersistence::150::root::(save) Saved new config PersistentConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/persistence/netconf/nets and /var/lib/vdsm/persistence/netconf/bonds
MainThread::DEBUG::2018-04-03 18:35:53,126::cmdutils::150::root::(exec_cmd) /usr/share/openvswitch/scripts/ovs-ctl status (cwd None)
MainThread::DEBUG::2018-04-03 18:35:53,172::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-04-03 18:35:53,184::vsctl::58::root::(commit) Executing commands: /usr/bin/ovs-vsctl --oneline --format=json -- list Bridge -- list Port -- list Interface
MainThread::DEBUG::2018-04-03 18:35:53,185::cmdutils::150::root::(exec_cmd) /usr/bin/ovs-vsctl --oneline --format=json -- list Bridge -- list Port -- list Interface (cwd None)
MainThread::DEBUG::2018-04-03 18:35:53,286::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-04-03 18:35:53,291::vsctl::58::root::(commit) Executing commands: /usr/bin/ovs-vsctl --oneline --format=json -- set open . external-ids:ovn-bridge-mappings=""
MainThread::DEBUG::2018-04-03 18:35:53,291::cmdutils::150::root::(exec_cmd) /usr/bin/ovs-vsctl --oneline --format=json -- set open . 'external-ids:ovn-bridge-mappings=""' (cwd None)
MainThread::DEBUG::2018-04-03 18:35:53,374::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-04-04 19:36:20,241::libvirtconnection::167::root::(get) trying to connect libvirt
MainThread::INFO::2018-04-04 19:36:20,320::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/staging/netconf
MainThread::INFO::2018-04-04 19:36:20,517::netconfpersistence::150::root::(save) Saved new config RunningConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/staging/netconf/nets and /var/lib/vdsm/staging/netconf/bonds
MainThread::INFO::2018-04-04 19:36:20,518::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/persistence/netconf
MainThread::INFO::2018-04-04 19:36:20,576::netconfpersistence::150::root::(save) Saved new config PersistentConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/persistence/netconf/nets and /var/lib/vdsm/persistence/netconf/bonds
MainThread::DEBUG::2018-04-04 19:36:20,577::cmdutils::150::root::(exec_cmd) /usr/share/openvswitch/scripts/ovs-ctl status (cwd None)
MainThread::DEBUG::2018-04-04 19:36:20,617::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-04-04 19:36:20,619::vsctl::58::root::(commit) Executing commands: /usr/bin/ovs-vsctl --oneline --format=json -- list Bridge -- list Port -- list Interface
MainThread::DEBUG::2018-04-04 19:36:20,620::cmdutils::150::root::(exec_cmd) /usr/bin/ovs-vsctl --oneline --format=json -- list Bridge -- list Port -- list Interface (cwd None)
MainThread::DEBUG::2018-04-04 19:36:20,693::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-04-04 19:36:20,695::vsctl::58::root::(commit) Executing commands: /usr/bin/ovs-vsctl --oneline --format=json -- set open . external-ids:ovn-bridge-mappings=""
MainThread::DEBUG::2018-04-04 19:36:20,696::cmdutils::150::root::(exec_cmd) /usr/bin/ovs-vsctl --oneline --format=json -- set open . 'external-ids:ovn-bridge-mappings=""' (cwd None)
MainThread::DEBUG::2018-04-04 19:36:20,777::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-05-10 18:04:31,052::libvirtconnection::167::root::(get) trying to connect libvirt
MainThread::INFO::2018-05-10 18:04:31,099::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/staging/netconf
MainThread::INFO::2018-05-10 18:04:31,192::netconfpersistence::150::root::(save) Saved new config RunningConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'10.4.8.50', u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/staging/netconf/nets and /var/lib/vdsm/staging/netconf/bonds
MainThread::INFO::2018-05-10 18:04:31,193::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/persistence/netconf
MainThread::INFO::2018-05-10 18:04:31,251::netconfpersistence::150::root::(save) Saved new config PersistentConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'10.4.8.50', u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/persistence/netconf/nets and /var/lib/vdsm/persistence/netconf/bonds
MainThread::DEBUG::2018-05-10 18:04:31,251::cmdutils::150::root::(exec_cmd) /usr/share/openvswitch/scripts/ovs-ctl status (cwd None)
MainThread::DEBUG::2018-05-10 18:04:31,288::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-05-10 18:04:31,291::vsctl::58::root::(commit) Executing commands: /usr/bin/ovs-vsctl --oneline --format=json -- list Bridge -- list Port -- list Interface
MainThread::DEBUG::2018-05-10 18:04:31,291::cmdutils::150::root::(exec_cmd) /usr/bin/ovs-vsctl --oneline --format=json -- list Bridge -- list Port -- list Interface (cwd None)
MainThread::DEBUG::2018-05-10 18:04:31,364::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-05-10 18:04:31,369::vsctl::58::root::(commit) Executing commands: /usr/bin/ovs-vsctl --oneline --format=json -- set open . external-ids:ovn-bridge-mappings=""
MainThread::DEBUG::2018-05-10 18:04:31,369::cmdutils::150::root::(exec_cmd) /usr/bin/ovs-vsctl --oneline --format=json -- set open . 'external-ids:ovn-bridge-mappings=""' (cwd None)
MainThread::DEBUG::2018-05-10 18:04:31,440::cmdutils::158::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2018-05-10 19:06:28,238::libvirtconnection::167::root::(get) trying to connect libvirt
MainThread::INFO::2018-05-10 19:06:30,854::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/staging/netconf
MainThread::INFO::2018-05-10 19:06:30,930::netconfpersistence::150::root::(save) Saved new config RunningConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'10.4.8.50', u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/staging/netconf/nets and /var/lib/vdsm/staging/netconf/bonds
MainThread::INFO::2018-05-10 19:06:30,931::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/persistence/netconf
MainThread::INFO::2018-05-10 19:06:31,003::netconfpersistence::150::root::(save) Saved new config PersistentConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'10.4.8.50', u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/persistence/netconf/nets and /var/lib/vdsm/persistence/netconf/bonds
MainThread::DEBUG::2018-05-10 19:06:31,009::cmdutils::150::root::(exec_cmd) /usr/share/openvswitch/scripts/ovs-ctl status (cwd None)
MainThread::DEBUG::2018-05-10 19:06:31,048::cmdutils::158::root::(exec_cmd) FAILED: <err> = ''; <rc> = 1
MainThread::DEBUG::2018-05-10 20:08:37,775::libvirtconnection::167::root::(get) trying to connect libvirt
MainThread::INFO::2018-05-10 20:08:37,861::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/staging/netconf
MainThread::INFO::2018-05-10 20:08:38,026::netconfpersistence::150::root::(save) Saved new config RunningConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'10.4.8.50', u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/staging/netconf/nets and /var/lib/vdsm/staging/netconf/bonds
MainThread::INFO::2018-05-10 20:08:38,028::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/persistence/netconf
MainThread::INFO::2018-05-10 20:08:38,086::netconfpersistence::150::root::(save) Saved new config PersistentConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'10.4.8.50', u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/persistence/netconf/nets and /var/lib/vdsm/persistence/netconf/bonds
MainThread::DEBUG::2018-05-10 20:08:38,086::cmdutils::150::root::(exec_cmd) /usr/share/openvswitch/scripts/ovs-ctl status (cwd None)
MainThread::DEBUG::2018-05-10 20:08:38,125::cmdutils::158::root::(exec_cmd) FAILED: <err> = ''; <rc> = 1
MainThread::DEBUG::2018-05-14 18:06:59,693::libvirtconnection::167::root::(get) trying to connect libvirt
MainThread::INFO::2018-05-14 18:07:00,045::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/staging/netconf
MainThread::INFO::2018-05-14 18:07:00,319::netconfpersistence::150::root::(save) Saved new config RunningConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'10.4.8.50', u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/staging/netconf/nets and /var/lib/vdsm/staging/netconf/bonds
MainThread::INFO::2018-05-14 18:07:00,319::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/persistence/netconf
MainThread::INFO::2018-05-14 18:07:00,370::netconfpersistence::150::root::(save) Saved new config PersistentConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'10.4.8.50', u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/persistence/netconf/nets and /var/lib/vdsm/persistence/netconf/bonds
MainThread::DEBUG::2018-05-14 18:07:00,376::cmdutils::150::root::(exec_cmd) /usr/share/openvswitch/scripts/ovs-ctl status (cwd None)
MainThread::DEBUG::2018-05-14 18:07:00,393::cmdutils::158::root::(exec_cmd) FAILED: <err> = ''; <rc> = 1
MainThread::DEBUG::2018-05-14 19:36:50,749::libvirtconnection::167::root::(get) trying to connect libvirt
MainThread::INFO::2018-05-14 19:36:51,318::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/staging/netconf
MainThread::INFO::2018-05-14 19:36:51,600::netconfpersistence::150::root::(save) Saved new config RunningConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'10.4.8.50', u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/staging/netconf/nets and /var/lib/vdsm/staging/netconf/bonds
MainThread::INFO::2018-05-14 19:36:51,600::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/persistence/netconf
MainThread::INFO::2018-05-14 19:36:51,651::netconfpersistence::150::root::(save) Saved new config PersistentConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'10.4.8.50', u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/persistence/netconf/nets and /var/lib/vdsm/persistence/netconf/bonds
MainThread::DEBUG::2018-05-14 19:36:51,651::cmdutils::150::root::(exec_cmd) /usr/share/openvswitch/scripts/ovs-ctl status (cwd None)
MainThread::DEBUG::2018-05-14 19:36:51,667::cmdutils::158::root::(exec_cmd) FAILED: <err> = ''; <rc> = 1
MainThread::DEBUG::2018-05-14 20:04:13,537::libvirtconnection::167::root::(get) trying to connect libvirt
MainThread::INFO::2018-05-14 20:04:13,810::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/staging/netconf
MainThread::INFO::2018-05-14 20:04:14,113::netconfpersistence::150::root::(save) Saved new config RunningConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'10.4.8.50', u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/staging/netconf/nets and /var/lib/vdsm/staging/netconf/bonds
MainThread::INFO::2018-05-14 20:04:14,114::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/persistence/netconf
MainThread::INFO::2018-05-14 20:04:14,175::netconfpersistence::150::root::(save) Saved new config PersistentConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'10.4.8.50', u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/persistence/netconf/nets and /var/lib/vdsm/persistence/netconf/bonds
MainThread::DEBUG::2018-05-14 20:04:14,175::cmdutils::150::root::(exec_cmd) /usr/share/openvswitch/scripts/ovs-ctl status (cwd None)
MainThread::DEBUG::2018-05-14 20:04:14,191::cmdutils::158::root::(exec_cmd) FAILED: <err> = ''; <rc> = 1
MainThread::DEBUG::2018-06-14 20:16:18,949::libvirtconnection::167::root::(get) trying to connect libvirt
MainThread::INFO::2018-06-14 20:16:19,254::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/staging/netconf
MainThread::INFO::2018-06-14 20:16:19,484::netconfpersistence::150::root::(save) Saved new config RunningConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'10.4.8.50', u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/staging/netconf/nets and /var/lib/vdsm/staging/netconf/bonds
MainThread::INFO::2018-06-14 20:16:19,485::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/persistence/netconf
MainThread::INFO::2018-06-14 20:16:19,543::netconfpersistence::150::root::(save) Saved new config PersistentConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'10.4.8.50', u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/persistence/netconf/nets and /var/lib/vdsm/persistence/netconf/bonds
MainThread::DEBUG::2018-06-14 20:16:19,546::cmdutils::150::root::(exec_cmd) /usr/share/openvswitch/scripts/ovs-ctl status (cwd None)
MainThread::DEBUG::2018-06-14 20:16:19,583::cmdutils::158::root::(exec_cmd) FAILED: <err> = ''; <rc> = 1
MainThread::DEBUG::2018-06-16 14:39:29,308::libvirtconnection::167::root::(get) trying to connect libvirt
MainThread::INFO::2018-06-16 14:39:29,359::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/staging/netconf
MainThread::INFO::2018-06-16 14:39:29,544::netconfpersistence::150::root::(save) Saved new config RunningConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'10.4.8.50', u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/staging/netconf/nets and /var/lib/vdsm/staging/netconf/bonds
MainThread::INFO::2018-06-16 14:39:29,545::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/persistence/netconf
MainThread::INFO::2018-06-16 14:39:29,603::netconfpersistence::150::root::(save) Saved new config PersistentConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'10.4.8.50', u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/persistence/netconf/nets and /var/lib/vdsm/persistence/netconf/bonds
MainThread::DEBUG::2018-06-16 14:39:29,604::cmdutils::150::root::(exec_cmd) /usr/share/openvswitch/scripts/ovs-ctl status (cwd None)
MainThread::DEBUG::2018-06-16 14:39:29,646::cmdutils::158::root::(exec_cmd) FAILED: <err> = ''; <rc> = 1
MainThread::DEBUG::2018-06-26 18:56:26,589::libvirtconnection::167::root::(get) trying to connect libvirt
MainThread::INFO::2018-06-26 18:56:26,653::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/staging/netconf
MainThread::INFO::2018-06-26 18:56:26,732::netconfpersistence::150::root::(save) Saved new config RunningConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'10.4.8.50', u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/staging/netconf/nets and /var/lib/vdsm/staging/netconf/bonds
MainThread::INFO::2018-06-26 18:56:26,733::netconfpersistence::198::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/persistence/netconf
MainThread::INFO::2018-06-26 18:56:26,791::netconfpersistence::150::root::(save) Saved new config PersistentConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'10.4.8.50', u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}) to /var/lib/vdsm/persistence/netconf/nets and /var/lib/vdsm/persistence/netconf/bonds
MainThread::DEBUG::2018-06-26 18:56:26,791::cmdutils::150::root::(exec_cmd) /usr/share/openvswitch/scripts/ovs-ctl status (cwd None)
MainThread::DEBUG::2018-06-26 18:56:26,824::cmdutils::158::root::(exec_cmd) FAILED: <err> = ''; <rc> = 1
MainThread::DEBUG::2018-06-26 19:32:06,827::libvirtconnection::167::root::(get) trying to connect libvirt
MainThread::INFO::2018-06-26 19:32:09,252::netconfpersistence::227::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/staging/netconf
MainThread::INFO::2018-06-26 19:32:09,314::netconfpersistence::177::root::(save) Saved new config RunningConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'10.4.8.50', u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}, {}) to [/var/lib/vdsm/staging/netconf/nets,/var/lib/vdsm/staging/netconf/bonds,/var/lib/vdsm/staging/netconf/devices]
MainThread::INFO::2018-06-26 19:32:09,315::netconfpersistence::227::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/persistence/netconf
MainThread::INFO::2018-06-26 19:32:09,381::netconfpersistence::177::root::(save) Saved new config PersistentConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'10.4.8.50', u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}, {}) to [/var/lib/vdsm/persistence/netconf/nets,/var/lib/vdsm/persistence/netconf/bonds,/var/lib/vdsm/persistence/netconf/devices]
MainThread::DEBUG::2018-06-26 19:32:09,382::cmdutils::150::root::(exec_cmd) /usr/share/openvswitch/scripts/ovs-ctl status (cwd None)
MainThread::DEBUG::2018-06-26 19:32:09,422::cmdutils::158::root::(exec_cmd) FAILED: <err> = ''; <rc> = 1
MainThread::DEBUG::2018-06-27 10:53:28,162::libvirtconnection::167::root::(get) trying to connect libvirt
MainThread::INFO::2018-06-27 10:53:28,395::netconfpersistence::227::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/staging/netconf
MainThread::INFO::2018-06-27 10:53:28,662::netconfpersistence::177::root::(save) Saved new config RunningConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'10.4.8.50', u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}, {}) to [/var/lib/vdsm/staging/netconf/nets,/var/lib/vdsm/staging/netconf/bonds,/var/lib/vdsm/staging/netconf/devices]
MainThread::INFO::2018-06-27 10:53:28,663::netconfpersistence::227::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/persistence/netconf
MainThread::INFO::2018-06-27 10:53:28,724::netconfpersistence::177::root::(save) Saved new config PersistentConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'10.4.8.50', u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}, {}) to [/var/lib/vdsm/persistence/netconf/nets,/var/lib/vdsm/persistence/netconf/bonds,/var/lib/vdsm/persistence/netconf/devices]
MainThread::DEBUG::2018-06-27 10:53:28,724::cmdutils::150::root::(exec_cmd) /usr/share/openvswitch/scripts/ovs-ctl status (cwd None)
MainThread::DEBUG::2018-06-27 10:53:28,767::cmdutils::158::root::(exec_cmd) FAILED: <err> = ''; <rc> = 1
MainThread::DEBUG::2018-06-27 12:22:56,426::libvirtconnection::167::root::(get) trying to connect libvirt
MainThread::INFO::2018-06-27 12:22:56,718::netconfpersistence::227::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/staging/netconf
MainThread::INFO::2018-06-27 12:22:56,976::netconfpersistence::177::root::(save) Saved new config RunningConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'10.4.8.50', u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}, {}) to [/var/lib/vdsm/staging/netconf/nets,/var/lib/vdsm/staging/netconf/bonds,/var/lib/vdsm/staging/netconf/devices]
MainThread::INFO::2018-06-27 12:22:56,977::netconfpersistence::227::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/persistence/netconf
MainThread::INFO::2018-06-27 12:22:57,035::netconfpersistence::177::root::(save) Saved new config PersistentConfig({'ovirtmgmt': {u'ipv6autoconf': True, u'nameservers': [u'10.4.8.50', u'172.16.1.180'], u'nic': u'em1', u'ipaddr': u'10.4.8.80', u'switch': u'legacy', u'mtu': 1500, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'10.4.8.2', u'defaultRoute': True, u'bootproto': u'none'}, 'Backend': {u'ipv6autoconf': True, u'nameservers': [], u'nic': u'p3p1', u'ipaddr': u'192.168.1.21', u'netmask': u'255.255.255.0', u'mtu': 1500, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'gateway': u'192.168.1.100', u'defaultRoute': False, u'bootproto': u'none'}}, {}, {}) to [/var/lib/vdsm/persistence/netconf/nets,/var/lib/vdsm/persistence/netconf/bonds,/var/lib/vdsm/persistence/netconf/devices]
MainThread::DEBUG::2018-06-27 12:22:57,035::cmdutils::150::root::(exec_cmd) /usr/share/openvswitch/scripts/ovs-ctl status (cwd None)
MainThread::DEBUG::2018-06-27 12:22:57,075::cmdutils::158::root::(exec_cmd) FAILED: <err> = ''; <rc> = 1
Comment 5 Arghya Adhya 2018-06-28 05:54:16 UTC 
qemu log--

2018-06-27 05:30:55.834+0000: starting up libvirt version: 3.9.0, package: 14.el7_5.5 (CentOS BuildSystem <http://bugs.centos.org>, 2018-05-22-02:42:56, c1bm.rdu2.centos.org), qemu version: 2.10.0(qemu-kvm-ev-2.10.0-21.el7_5.3.1), hostname: host1.ndl.iitkgp.ac.in
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name guest=BACKUP-STAGE-DATAAPI,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-4-BACKUP-STAGE-DATAAPI/master-key.aes -machine pc-i440fx-rhel7.3.0,accel=kvm,usb=off,dump-guest-core=off -cpu Conroe -m size=8388608k,slots=16,maxmem=33554432k -realtime mlock=off -smp 1,maxcpus=16,sockets=16,cores=1,threads=1 -numa node,nodeid=0,cpus=0,mem=8192 -uuid ead91366-1e60-4858-85b3-b4ed21f6de97 -smbios 'type=1,manufacturer=oVirt,product=oVirt Node,version=7-5.1804.el7.centos.2,serial=4C4C4544-0044-4210-8046-B7C04F523432,uuid=ead91366-1e60-4858-85b3-b4ed21f6de97' -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-4-BACKUP-STAGE-DATAAPI/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2018-06-27T11:00:54,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-scsi-pci,id=ua-7538cc96-401d-490a-86fe-f233fea68291,bus=pci.0,addr=0x5 -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x6 -drive file=/rhev/data-center/mnt/192.168.1.100:_Ovirt-export_data/22d10021-f581-47de-98a4-fae5163da8dd/images/48132925-5a28-4bdf-bee4-8e14dd68ccd3/077101eb-36d6-4a5e-86e6-0ef0376d4dfe,format=raw,if=none,id=drive-ua-48132925-5a28-4bdf-bee4-8e14dd68ccd3,serial=48132925-5a28-4bdf-bee4-8e14dd68ccd3,cache=none,werror=stop,rerror=stop,aio=threads -device scsi-hd,bus=ua-7538cc96-401d-490a-86fe-f233fea68291.0,channel=0,scsi-id=0,lun=0,drive=drive-ua-48132925-5a28-4bdf-bee4-8e14dd68ccd3,id=ua-48132925-5a28-4bdf-bee4-8e14dd68ccd3,bootindex=1 -netdev tap,fd=30,id=hostua-50346e87-87b0-4407-8b81-66031571919a,vhost=on,vhostfd=32 -device virtio-net-pci,netdev=hostua-50346e87-87b0-4407-8b81-66031571919a,id=ua-50346e87-87b0-4407-8b81-66031571919a,mac=00:1a:4a:16:01:17,bus=pci.0,addr=0x3 -netdev tap,fd=33,id=hostua-4f341434-b018-4e64-aa9f-3e44e082b101,vhost=on,vhostfd=34 -device virtio-net-pci,netdev=hostua-4f341434-b018-4e64-aa9f-3e44e082b101,id=ua-4f341434-b018-4e64-aa9f-3e44e082b101,mac=00:1a:4a:16:01:18,bus=pci.0,addr=0x4 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/ead91366-1e60-4858-85b3-b4ed21f6de97.ovirt-guest-agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=ovirt-guest-agent.0 -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/ead91366-1e60-4858-85b3-b4ed21f6de97.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel2,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 -spice port=5900,tls-port=5901,addr=10.4.8.80,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,seamless-migration=on -device qxl-vga,id=ua-7da2c374-3d78-490b-bbd3-45e65ec606c4,ram_size=67108864,vram_size=33554432,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 -object rng-random,id=objua-3f23b997-f5b9-4f4a-8be3-799036e929c1,filename=/dev/urandom -device virtio-rng-pci,rng=objua-3f23b997-f5b9-4f4a-8be3-799036e929c1,id=ua-3f23b997-f5b9-4f4a-8be3-799036e929c1,bus=pci.0,addr=0x7 -msg timestamp=on
2018-06-27T05:30:56.212012Z qemu-kvm: -drive file=/rhev/data-center/mnt/192.168.1.100:_Ovirt-export_data/22d10021-f581-47de-98a4-fae5163da8dd/images/48132925-5a28-4bdf-bee4-8e14dd68ccd3/077101eb-36d6-4a5e-86e6-0ef0376d4dfe,format=raw,if=none,id=drive-ua-48132925-5a28-4bdf-bee4-8e14dd68ccd3,serial=48132925-5a28-4bdf-bee4-8e14dd68ccd3,cache=none,werror=stop,rerror=stop,aio=threads: 'serial' is deprecated, please use the corresponding option of '-device' instead
2018-06-27T05:30:56.235244Z qemu-kvm: -drive file=/rhev/data-center/mnt/192.168.1.100:_Ovirt-export_data/22d10021-f581-47de-98a4-fae5163da8dd/images/48132925-5a28-4bdf-bee4-8e14dd68ccd3/077101eb-36d6-4a5e-86e6-0ef0376d4dfe,format=raw,if=none,id=drive-ua-48132925-5a28-4bdf-bee4-8e14dd68ccd3,serial=48132925-5a28-4bdf-bee4-8e14dd68ccd3,cache=none,werror=stop,rerror=stop,aio=threads: Failed to lock byte 100
2018-06-27T05:30:56.236265Z qemu-kvm: warning: CPU(s) not present in any NUMA nodes: CPU 1 [socket-id: 1, core-id: 0, thread-id: 0], CPU 2 [socket-id: 2, core-id: 0, thread-id: 0], CPU 3 [socket-id: 3, core-id: 0, thread-id: 0], CPU 4 [socket-id: 4, core-id: 0, thread-id: 0], CPU 5 [socket-id: 5, core-id: 0, thread-id: 0], CPU 6 [socket-id: 6, core-id: 0, thread-id: 0], CPU 7 [socket-id: 7, core-id: 0, thread-id: 0], CPU 8 [socket-id: 8, core-id: 0, thread-id: 0], CPU 9 [socket-id: 9, core-id: 0, thread-id: 0], CPU 10 [socket-id: 10, core-id: 0, thread-id: 0], CPU 11 [socket-id: 11, core-id: 0, thread-id: 0], CPU 12 [socket-id: 12, core-id: 0, thread-id: 0], CPU 13 [socket-id: 13, core-id: 0, thread-id: 0], CPU 14 [socket-id: 14, core-id: 0, thread-id: 0], CPU 15 [socket-id: 15, core-id: 0, thread-id: 0]
2018-06-27T05:30:56.236298Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config, ability to start up with partial NUMA mappings is obsoleted and will be removed in future
2018-06-27T05:30:56.281317Z qemu-kvm: -device scsi-hd,bus=ua-7538cc96-401d-490a-86fe-f233fea68291.0,channel=0,scsi-id=0,lun=0,drive=drive-ua-48132925-5a28-4bdf-bee4-8e14dd68ccd3,id=ua-48132925-5a28-4bdf-bee4-8e14dd68ccd3,bootindex=1: Failed to lock byte 100
2018-06-27T05:30:56.281636Z qemu-kvm: -device scsi-hd,bus=ua-7538cc96-401d-490a-86fe-f233fea68291.0,channel=0,scsi-id=0,lun=0,drive=drive-ua-48132925-5a28-4bdf-bee4-8e14dd68ccd3,id=ua-48132925-5a28-4bdf-bee4-8e14dd68ccd3,bootindex=1: Failed to lock byte 100
Unexpected error in raw_apply_lock_bytes() at block/file-posix.c:642:
2018-06-27T05:30:56.281958Z qemu-kvm: -device scsi-hd,bus=ua-7538cc96-401d-490a-86fe-f233fea68291.0,channel=0,scsi-id=0,lun=0,drive=drive-ua-48132925-5a28-4bdf-bee4-8e14dd68ccd3,id=ua-48132925-5a28-4bdf-bee4-8e14dd68ccd3,bootindex=1: Failed to lock byte 100
2018-06-27 05:30:57.997+0000: shutting down, reason=failed
2018-06-27 06:29:37.379+0000: starting up libvirt version: 3.9.0, package: 14.el7_5.5 (CentOS BuildSystem <http://bugs.centos.org>, 2018-05-22-02:42:56, c1bm.rdu2.centos.org), qemu version: 2.10.0(qemu-kvm-ev-2.10.0-21.el7_5.3.1), hostname: host1.ndl.iitkgp.ac.in
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name guest=BACKUP-STAGE-DATAAPI,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-9-BACKUP-STAGE-DATAAPI/master-key.aes -machine pc-i440fx-rhel7.3.0,accel=kvm,usb=off,dump-guest-core=off -cpu Conroe -m size=8388608k,slots=16,maxmem=33554432k -realtime mlock=off -smp 1,maxcpus=16,sockets=16,cores=1,threads=1 -numa node,nodeid=0,cpus=0,mem=8192 -uuid ead91366-1e60-4858-85b3-b4ed21f6de97 -smbios 'type=1,manufacturer=oVirt,product=oVirt Node,version=7-5.1804.el7.centos.2,serial=4C4C4544-0044-4210-8046-B7C04F523432,uuid=ead91366-1e60-4858-85b3-b4ed21f6de97' -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-9-BACKUP-STAGE-DATAAPI/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2018-06-27T11:59:35,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-scsi-pci,id=ua-7538cc96-401d-490a-86fe-f233fea68291,bus=pci.0,addr=0x5 -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x6 -drive file=/rhev/data-center/mnt/192.168.1.100:_Ovirt-export_data/22d10021-f581-47de-98a4-fae5163da8dd/images/48132925-5a28-4bdf-bee4-8e14dd68ccd3/077101eb-36d6-4a5e-86e6-0ef0376d4dfe,format=raw,if=none,id=drive-ua-48132925-5a28-4bdf-bee4-8e14dd68ccd3,serial=48132925-5a28-4bdf-bee4-8e14dd68ccd3,cache=none,werror=stop,rerror=stop,aio=threads -device scsi-hd,bus=ua-7538cc96-401d-490a-86fe-f233fea68291.0,channel=0,scsi-id=0,lun=0,drive=drive-ua-48132925-5a28-4bdf-bee4-8e14dd68ccd3,id=ua-48132925-5a28-4bdf-bee4-8e14dd68ccd3,bootindex=1 -netdev tap,fd=35,id=hostua-50346e87-87b0-4407-8b81-66031571919a,vhost=on,vhostfd=37 -device virtio-net-pci,netdev=hostua-50346e87-87b0-4407-8b81-66031571919a,id=ua-50346e87-87b0-4407-8b81-66031571919a,mac=00:1a:4a:16:01:17,bus=pci.0,addr=0x3 -netdev tap,fd=38,id=hostua-4f341434-b018-4e64-aa9f-3e44e082b101,vhost=on,vhostfd=39 -device virtio-net-pci,netdev=hostua-4f341434-b018-4e64-aa9f-3e44e082b101,id=ua-4f341434-b018-4e64-aa9f-3e44e082b101,mac=00:1a:4a:16:01:18,bus=pci.0,addr=0x4 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/ead91366-1e60-4858-85b3-b4ed21f6de97.ovirt-guest-agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=ovirt-guest-agent.0 -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/ead91366-1e60-4858-85b3-b4ed21f6de97.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel2,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 -spice port=5901,tls-port=5902,addr=10.4.8.80,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,seamless-migration=on -device qxl-vga,id=ua-7da2c374-3d78-490b-bbd3-45e65ec606c4,ram_size=67108864,vram_size=33554432,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 -object rng-random,id=objua-3f23b997-f5b9-4f4a-8be3-799036e929c1,filename=/dev/urandom -device virtio-rng-pci,rng=objua-3f23b997-f5b9-4f4a-8be3-799036e929c1,id=ua-3f23b997-f5b9-4f4a-8be3-799036e929c1,bus=pci.0,addr=0x7 -msg timestamp=on
2018-06-27T06:29:37.709135Z qemu-kvm: -drive file=/rhev/data-center/mnt/192.168.1.100:_Ovirt-export_data/22d10021-f581-47de-98a4-fae5163da8dd/images/48132925-5a28-4bdf-bee4-8e14dd68ccd3/077101eb-36d6-4a5e-86e6-0ef0376d4dfe,format=raw,if=none,id=drive-ua-48132925-5a28-4bdf-bee4-8e14dd68ccd3,serial=48132925-5a28-4bdf-bee4-8e14dd68ccd3,cache=none,werror=stop,rerror=stop,aio=threads: 'serial' is deprecated, please use the corresponding option of '-device' instead
2018-06-27T06:29:37.720880Z qemu-kvm: -drive file=/rhev/data-center/mnt/192.168.1.100:_Ovirt-export_data/22d10021-f581-47de-98a4-fae5163da8dd/images/48132925-5a28-4bdf-bee4-8e14dd68ccd3/077101eb-36d6-4a5e-86e6-0ef0376d4dfe,format=raw,if=none,id=drive-ua-48132925-5a28-4bdf-bee4-8e14dd68ccd3,serial=48132925-5a28-4bdf-bee4-8e14dd68ccd3,cache=none,werror=stop,rerror=stop,aio=threads: Failed to lock byte 100
2018-06-27T06:29:37.722019Z qemu-kvm: warning: CPU(s) not present in any NUMA nodes: CPU 1 [socket-id: 1, core-id: 0, thread-id: 0], CPU 2 [socket-id: 2, core-id: 0, thread-id: 0], CPU 3 [socket-id: 3, core-id: 0, thread-id: 0], CPU 4 [socket-id: 4, core-id: 0, thread-id: 0], CPU 5 [socket-id: 5, core-id: 0, thread-id: 0], CPU 6 [socket-id: 6, core-id: 0, thread-id: 0], CPU 7 [socket-id: 7, core-id: 0, thread-id: 0], CPU 8 [socket-id: 8, core-id: 0, thread-id: 0], CPU 9 [socket-id: 9, core-id: 0, thread-id: 0], CPU 10 [socket-id: 10, core-id: 0, thread-id: 0], CPU 11 [socket-id: 11, core-id: 0, thread-id: 0], CPU 12 [socket-id: 12, core-id: 0, thread-id: 0], CPU 13 [socket-id: 13, core-id: 0, thread-id: 0], CPU 14 [socket-id: 14, core-id: 0, thread-id: 0], CPU 15 [socket-id: 15, core-id: 0, thread-id: 0]
2018-06-27T06:29:37.722054Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config, ability to start up with partial NUMA mappings is obsoleted and will be removed in future
2018-06-27T06:29:37.767319Z qemu-kvm: -device scsi-hd,bus=ua-7538cc96-401d-490a-86fe-f233fea68291.0,channel=0,scsi-id=0,lun=0,drive=drive-ua-48132925-5a28-4bdf-bee4-8e14dd68ccd3,id=ua-48132925-5a28-4bdf-bee4-8e14dd68ccd3,bootindex=1: Failed to lock byte 100
2018-06-27T06:29:37.767594Z qemu-kvm: -device scsi-hd,bus=ua-7538cc96-401d-490a-86fe-f233fea68291.0,channel=0,scsi-id=0,lun=0,drive=drive-ua-48132925-5a28-4bdf-bee4-8e14dd68ccd3,id=ua-48132925-5a28-4bdf-bee4-8e14dd68ccd3,bootindex=1: Failed to lock byte 100
Unexpected error in raw_apply_lock_bytes() at block/file-posix.c:642:
2018-06-27T06:29:37.767893Z qemu-kvm: -device scsi-hd,bus=ua-7538cc96-401d-490a-86fe-f233fea68291.0,channel=0,scsi-id=0,lun=0,drive=drive-ua-48132925-5a28-4bdf-bee4-8e14dd68ccd3,id=ua-48132925-5a28-4bdf-bee4-8e14dd68ccd3,bootindex=1: Failed to lock byte 100
2018-06-27 06:29:39.328+0000: shutting down, reason=failed
Comment 6 Arghya Adhya 2018-06-28 05:55:09 UTC 
vdsm log is huge, can I attach log file here?
Comment 7 Doron Fediuck 2018-06-28 12:52:08 UTC 
(In reply to Arghya Adhya from comment #6)
> vdsm log is huge, can I attach log file here?

Yes, logs should be attached to the bz as files in the attachments section.
Comment 8 Arghya Adhya 2018-06-28 13:01:58 UTC 
Created attachment 1455325 [details]
vdsm log
Comment 9 Arghya Adhya 2018-06-28 13:02:33 UTC 
Created attachment 1455326 [details]
upgrade log
Comment 10 Arghya Adhya 2018-06-28 13:03:37 UTC 
Created attachment 1455327 [details]
supervdsm log
Comment 11 Arghya Adhya 2018-06-28 13:06:28 UTC 
Created attachment 1455328 [details]
qemu log
Comment 12 Yaniv Lavi 2018-07-02 07:47:11 UTC 
Can you please review?
Comment 13 Sandro Bonazzola 2018-07-02 07:47:55 UTC 
Did you reboot the host after upgrade?
Comment 14 Arghya Adhya 2018-07-03 16:35:49 UTC 
yes, hosts are rebooted by hosted engine. I have got update notification in host tab in hosted engine GUI, then I update hosts from the GUI.
Comment 15 Arghya Adhya 2018-07-04 07:08:35 UTC 
Additional information

I have another 3 node ovirt setup with gluster fs running on their local HDD and for large size vm creation I took space from NAS also(mounted as data domain). Now I have face same problem in this setup after updating oVirt latest version. Inspecting this problem I have found if my vm ruining from gluster storage its working but if the vm running from the NFS storage I am getting same error. Pl. suggest how to solve this problem. My test environment is running form these 2 environment and some vms are not ruining (which storage from NAS) which are critical to my environment.
Comment 16 Arghya Adhya 2018-07-05 06:58:24 UTC 
Please give a solution, my environment is very critical. Should I roll back 4.2.2 version (that working fine) and stop update until next release?
Comment 17 Vincent 2018-08-07 20:00:20 UTC 
I am having a similar bug starting VMs on a new 4.2.5 host.  The same VM below starts fine on a 4.2 host in the same datacenter/cluster

OS Version: RHEL - 7 - 5.1804.el7.centos.2
OS Description: oVirt Node 4.2.5.1
Kernel Version: 3.10.0 - 862.9.1.el7.x86_64
KVM Version: 2.10.0 - 21.el7_5.4.1
LIBVIRT Version: libvirt-3.9.0-14.el7_5.6
VDSM Version: vdsm-4.20.35-1.el7
SPICE Version: 0.14.0 - 2.el7_5.4
GlusterFS Version: glusterfs-3.12.11-1.el7
CEPH Version: librbd1-0.94.5-2.el7
Open vSwitch Version: openvswitch-2.9.0-4.el7
Kernel Features: PTI: 1, IBRS: 0, RETP: 1

Steps to reproduce:

1. import centos image from Glance as template
2. Create new VM from this template
3. Create and attach disk and network interface
3. Run VM

Result:

VM CentOSPlayground is down with error. Exit message: internal error: qemu unexpectedly closed the monitor: 2018-08-07T19:32:17.947985Z qemu-kvm: -drive file=/rhev/data-center/mnt/DOMINIC.malonetuning.net:_VMS/d3165759-07c2-46ae-b7b8-b6226a929d68/images/6ae658b4-43a4-4980-9310-a70f40aabaea/9b617fb6-ad7a-41b4-860d-4b287f7b1601,format=qcow2,if=none,id=drive-ua-6ae658b4-43a4-4980-9310-a70f40aabaea,serial=6ae658b4-43a4-4980-9310-a70f40aabaea,cache=none,werror=stop,rerror=stop,aio=threads: Failed to lock byte 100 Unexpected error in raw_apply_lock_bytes() at block/file-posix.c:642: 2018-08-07T19:32:17.952908Z qemu-kvm: -drive file=/rhev/data-center/mnt/DOMINIC.malonetuning.net:_VMS/d3165759-07c2-46ae-b7b8-b6226a929d68/images/6ae658b4-43a4-4980-9310-a70f40aabaea/9b617fb6-ad7a-41b4-860d-4b287f7b1601,format=qcow2,if=none,id=drive-ua-6ae658b4-43a4-4980-9310-a70f40aabaea,serial=6ae658b4-43a4-4980-9310-a70f40aabaea,cache=none,werror=stop,rerror=stop,aio=threads: Failed to lock byte 100.

will upload my logs (vsdm log is massive)
Comment 18 Vincent 2018-08-07 20:10:30 UTC 
Created attachment 1474092 [details]
vsdm log

Vincent's vsdm.log
Comment 19 Vincent 2018-08-07 20:18:20 UTC 
Created attachment 1474094 [details]
supervsdm.log vincent
Comment 20 Arghya Adhya 2018-08-08 14:49:04 UTC 
Please give us solution so that we can update our environment, currently I downgrade my version to 4.2.2 and its working fine. Also your new version(4.2.5) having the same problem. Please resolve this problem.
Comment 21 Ryan Barry 2018-08-15 16:58:30 UTC 
Tal -

Any hints at storage? It's NFS only, and these messages look like problems with the disks:

2018-06-27T06:29:37.767319Z qemu-kvm: -device scsi-hd,bus=ua-7538cc96-401d-490a-86fe-f233fea68291.0,channel=0,scsi-id=0,lun=0,drive=drive-ua-48132925-5a28-4bdf-bee4-8e14dd68ccd3,id=ua-48132925-5a28-4bdf-bee4-8e14dd68ccd3,bootindex=1: Failed to lock byte 100
2018-06-27T06:29:37.767594Z qemu-kvm: -device scsi-hd,bus=ua-7538cc96-401d-490a-86fe-f233fea68291.0,channel=0,scsi-id=0,lun=0,drive=drive-ua-48132925-5a28-4bdf-bee4-8e14dd68ccd3,id=ua-48132925-5a28-4bdf-bee4-8e14dd68ccd3,bootindex=1: Failed to lock byte 100
Unexpected error in raw_apply_lock_bytes() at block/file-posix.c:642:
2018-06-27T06:29:37.767893Z qemu-kvm: -device scsi-hd,bus=ua-7538cc96-401d-490a-86fe-f233fea68291.0,channel=0,scsi-id=0,lun=0,drive=drive-ua-48132925-5a28-4bdf-bee4-8e14dd68ccd3,id=ua-48132925-5a28-4bdf-bee4-8e14dd68ccd3,bootindex=1: Failed to lock byte 100
Comment 22 Tal Nisan 2018-08-16 13:09:51 UTC 
Seems like image locking errors, Arghya, are you using shared disks by any chance?
Comment 23 Arghya Adhya 2018-08-16 14:10:43 UTC 
I am using NAS for the data storage.
Comment 24 Vincent 2018-08-16 16:44:14 UTC 
(In reply to Ryan Barry from comment #21)
> Tal -
> 
> Any hints at storage? It's NFS only, and these messages look like problems
> with the disks:
> 
> 2018-06-27T06:29:37.767319Z qemu-kvm: -device
> scsi-hd,bus=ua-7538cc96-401d-490a-86fe-f233fea68291.0,channel=0,scsi-id=0,
> lun=0,drive=drive-ua-48132925-5a28-4bdf-bee4-8e14dd68ccd3,id=ua-48132925-
> 5a28-4bdf-bee4-8e14dd68ccd3,bootindex=1: Failed to lock byte 100
> 2018-06-27T06:29:37.767594Z qemu-kvm: -device
> scsi-hd,bus=ua-7538cc96-401d-490a-86fe-f233fea68291.0,channel=0,scsi-id=0,
> lun=0,drive=drive-ua-48132925-5a28-4bdf-bee4-8e14dd68ccd3,id=ua-48132925-
> 5a28-4bdf-bee4-8e14dd68ccd3,bootindex=1: Failed to lock byte 100
> Unexpected error in raw_apply_lock_bytes() at block/file-posix.c:642:
> 2018-06-27T06:29:37.767893Z qemu-kvm: -device
> scsi-hd,bus=ua-7538cc96-401d-490a-86fe-f233fea68291.0,channel=0,scsi-id=0,
> lun=0,drive=drive-ua-48132925-5a28-4bdf-bee4-8e14dd68ccd3,id=ua-48132925-
> 5a28-4bdf-bee4-8e14dd68ccd3,bootindex=1: Failed to lock byte 100


My NFS shares are on a windows server on a RAID - totally healthy. My 4.2 hosts use the shares, VMs come up and go down and new VMs can be created with no issue. The issue only lies with 4.2.5 host.  What is the likelihood of this issue being looked at soon?  Can't keep going with Ovirt if I can't update my hosts!
Comment 25 Ryan Barry 2018-08-16 17:35:06 UTC 
Are any VMs sharing the same disk image, maybe from a template?

Vincent -

It's actively being looked at, but no root cause is apparent yet, other than disk locking failures, so that's what we're trying to figure out...
Comment 26 Arghya Adhya 2018-09-03 11:37:38 UTC 
yes we have some vm which are sharing same disk but I notice that vm which I create in NAS those are not starting but vms which I create using gluster storage those running. Even If I create new VM using my NAS storage space that also are not running.
Comment 27 Arghya Adhya 2018-09-10 15:42:34 UTC 
Toady I have updated my environment in your new version 4.2.6, but same problem exists there.
Comment 28 Vincent 2018-10-01 20:53:04 UTC 
I have made some progress. 

I created a new NFSV3 storage domain on the same windows storage server. 

I created some VMs and disks on this domain and they run normally. 

So the issue here is that Ovirt after 4.2.0 does not correctly support NFSV4 in some configurations. 

What logs can I provide to help rectify this issue?
Comment 29 Tal Nisan 2018-10-15 08:28:48 UTC 
Hi Vincent,
Please supply the Engine log and the VDSM log from the host on which you try to run the VM please
Comment 30 Fred Rolland 2018-10-22 08:11:48 UTC 
See similar issue:
From: https://bugzilla.redhat.com/show_bug.cgi?id=1624734#c2
=====
isn't it the problem with OFD? 
From RHEL 7.5 hyperviors there is new kind of locks for NFSv4 - OFD. If the NFS protocol in use is 4 and we have Windows NFS server, then qemu-kvm attempts to lock an already locked byte on the VM's disk. That is tracked by bug:

Bug 1589627 - Improve image locking error handling 
https://bugzilla.redhat.com/show_bug.cgi?id=1589627

It is likely that EMC storage (if it is some old Clarion or VNX) doesn't have POSIX lock splitting and merging semantics. RFC 5661, Section 18.10.4

When the client sends a LOCK operation that corresponds to a range
that the lock-owner has locked already (with the same or different
lock type), or to a sub-range of such a range, or to a byte-range
that includes multiple locks already granted to that lock-owner, in
whole or in part, and the server does not support such locking
operations (i.e., does not support POSIX locking semantics), the
server will return the error NFS4ERR_LOCK_RANGE.  In that case, the
client may return an error, or it may emulate the required
operations, using only LOCK for ranges that do not include any bytes
already locked by that lock-owner and LOCKU of locks held by that
lock-owner (specifying an exactly matching range and type).
=====
Comment 31 Tal Nisan 2018-11-05 09:15:03 UTC 
Hi Arghya, it seems like the error you are hitting is related to comment #30, it seems like the server does not support a feature in the NFS V4 protocol, consider moving to NFS V3 or contact your storage vendor for a solution in the server if possible.
I'm closing the bug for now, if you still hit that issue after trying out those suggestions please reopen
Comment 32 Red Hat Bugzilla 2023-09-14 04:30:25 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days
Note You need to log in before you can comment on or make changes to this bug.