Bug 1283628

Summary: [RHEV-H] - setupNetworks fail sometimes with error OSError: [Errno 16] Device or resource busy
Product: [oVirt] vdsm Reporter: Meni Yakove <myakove>
Component: CoreAssignee: Sagi Shnaidman <sshnaidm>
Status: CLOSED CURRENTRELEASE QA Contact: Meni Yakove <myakove>
Severity: high Docs Contact:
Priority: high    
Version: ---CC: bugs, danken, fdeutsch, mburman, sshnaidm, ylavi
Target Milestone: ovirt-3.6.1Keywords: Automation, Regression
Target Release: 4.17.12Flags: sshnaidm: needinfo-
fdeutsch: ovirt-3.6.z?
rule-engine: blocker?
myakove: testing_plan_complete+
myakove: planning_ack?
rule-engine: devel_ack+
rule-engine: testing_ack+
Hardware: x86_64   
OS: All   
Whiteboard: network
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1286442 1286443 (view as bug list) Environment:
Last Closed: 2015-12-16 12:21:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Network RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1283643, 1286442, 1286443    
Attachments:
Description Flags
engine, vdsm and supervdsm logs none

Description Meni Yakove 2015-11-19 12:59:45 UTC
Created attachment 1096671 [details]
engine, vdsm and supervdsm logs

Description of problem:
When calling setupNetwork the operation sometimes fails with error:
MainProcess|Thread-22381::ERROR::2015-11-18 14:44:30,017::utils::148::root::(rmFile) Removing file: /etc/sysconfig/network-scripts/ifcfg-case4_net1 failed
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 142, in rmFile
    os.unlink(fileToRemove)
OSError: [Errno 16] Device or resource busy: '/etc/sysconfig/network-scripts/ifcfg-case4_net1'
MainProcess|Thread-22381::ERROR::2015-11-18 14:44:30,018::supervdsmServer::118::SuperVdsm.ServerCallback::(wrapper) Error in setupNetworks
Traceback (most recent call last):
  File "/usr/share/vdsm/supervdsmServer", line 116, in wrapper
    res = func(*args, **kwargs)
  File "/usr/share/vdsm/supervdsmServer", line 241, in setupNetworks
    return setupNetworks(networks, bondings, **options)
  File "/usr/share/vdsm/network/api.py", line 942, in setupNetworks
    options, logger)
  File "/usr/share/vdsm/network/configurators/__init__.py", line 60, in __exit__
    leftover = self.rollback()
  File "/usr/share/vdsm/network/configurators/ifcfg.py", line 83, in rollback
    self.configApplier.restoreBackups()
  File "/usr/share/vdsm/network/configurators/ifcfg.py", line 466, in restoreBackups
    self.restoreAtomicBackup()
  File "/usr/share/vdsm/network/configurators/ifcfg.py", line 389, in restoreAtomicBackup
    utils.rmFile(confFilePath)
  File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 142, in rmFile
    os.unlink(fileToRemove)
OSError: [Errno 16] Device or resource busy: '/etc/sysconfig/network-scripts/ifcfg-case4_net1'

Version-Release number of selected component (if applicable):
Red Hat Enterprise Virtualization Hypervisor (Beta) release 7.2 (20151112.1.el7ev)

Steps to Reproduce:
1. Run TestSanityCase13 from network sanity automation

Comment 1 Red Hat Bugzilla Rules Engine 2015-11-23 16:17:02 UTC
This bug report has Keywords: Regression or TestBlocker.
Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP.

Comment 2 Meni Yakove 2015-11-24 12:01:32 UTC
Steps to Reproduce:

1. Create VM network and attach it to host NIC
2. Change the network to be non-VM
3. Change the network to be VM network

Comment 3 Sagi Shnaidman 2015-11-29 12:38:18 UTC
reproduction steps:

1. Install ovirt-node/RHEVH and connect to engine.
2. Change in the management network IP of the node to something different.

The node starts rolling back and fails because of error above.

Comment 4 Yaniv Lavi 2015-11-29 12:40:15 UTC
should this be POST?

Comment 5 Sagi Shnaidman 2015-11-29 13:16:01 UTC
yes
https://gerrit.ovirt.org/#/c/48841/1

Comment 6 Sandro Bonazzola 2015-12-01 15:25:45 UTC
This bug is referenced in 4.17.12 git log and has target release unset.
Please check

Comment 7 Michael Burman 2015-12-06 13:39:01 UTC
Verified on - 3.6.1.1-0.1.el6 with:
vdsm-4.17.12-0.el7ev.noarch
Red Hat Enterprise Virtualization Hypervisor (Beta) release 7.2 (20151201.2.el7ev)
ovirt-node-3.6.0-0.23.20151201git5eed7af.el7ev.noarch

Comment 8 Sandro Bonazzola 2015-12-16 12:21:20 UTC
According to verification status and target milestone this issue should be fixed in oVirt 3.6.1. Closing current release.