Bug 1383165

Summary: [WALA] waagent unexpectedly restart when create a VM base on a specialized image
Product: Red Hat Enterprise Linux 7 Reporter: Yuxin Sun <yuxisun>
Component: WALinuxAgentAssignee: Miroslav Rezanina <mrezanin>
Status: CLOSED CURRENTRELEASE QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.3CC: leiwang, salmy, wshi, yuxisun
Target Milestone: rcKeywords: Extras, Tracking
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: WALinuxAgent-2.2.4 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1429323 (view as bug list) Environment:
Last Closed: 2017-03-07 08:51:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1429323    

Description Yuxin Sun 2016-10-10 05:27:25 UTC
Description of problem:
There's an error log "WALinuxAgent-2.2.0 unexpectedly restarted" in the waagent.log when create a VM base on a specialized image. The specialized image is captured when the base VM is running.

Version-Release number of selected component (if applicable):
WALinuxAgent-2.2.0/2.1.6
RHEL Version: RHEL-7.3-20161005.0

How reproducible:
100%

Steps to Reproduce:
1. Prepare a VM with WALA-2.2.0 installed on ASM mode.
2. Capture a specialized image from this VM.(Capture when the VM is running)
3. Create a new VM base on this image.
4. Check the waagent.log
 
Actual Result:
"WALinuxAgent-2.2.0 unexpectedly restarted" message exists in the waagent.log.

/var/log/waagent.log:
2016/10/09 13:56:28.079588 INFO Start env monitor service.
2016/10/09 13:56:28.085487 INFO Configure routes
2016/10/09 13:56:28.091958 VERBOSE Found event file: /var/lib/waagent/events/1475992177311592.tld
2016/10/09 13:56:28.093109 INFO Gateway:None
2016/10/09 13:56:28.097999 INFO Routes:None
2016/10/09 13:56:28.106174 VERBOSE run cmd 'pidof dhclient'
2016/10/09 13:56:28.144945 VERBOSE Processed event file: /var/lib/waagent/events/1475992177311592.tld
2016/10/09 13:56:28.155009 VERBOSE Found event file: /var/lib/waagent/events/1475992588091745.tld
2016/10/09 13:56:28.185060 VERBOSE Processed event file: /var/lib/waagent/events/1475992588091745.tld
2016/10/09 13:56:28.195144 INFO Set block dev timeout: sda with timeout: 300
2016/10/09 13:56:28.196404 INFO WALinuxAgent-2.2.0 running as process 2103
2016/10/09 13:56:28.198425 VERBOSE Sleep 1 second to avoid throttling.
2016/10/09 13:56:28.212005 INFO Set block dev timeout: sdb with timeout: 300
2016/10/09 13:56:28.227663 INFO WALinuxAgent-2.2.0 unexpectedly restarted
2016/10/09 13:56:28.234527 ERROR Event: name=WALinuxAgent, op=Restart, message=WALinuxAgent-2.2.0 unexpectedly restarted
2016/10/09 13:56:28.246089 INFO Checking for agent family Prod updates


Expect Result:
waagent is not restarted unexpectedly.

Additional Info:
1. No functional impact is observed.
2. The root cause might be this:
   i). Only if the /var/lib/waagent/current_version file exists and the content is equal to the current wala version, the unexpectedly restart is touched off.
   ii). When the waagent service is stopping, waagent removes the current_version file. When the waagent service is starting, waagent create the current_version file.
   iii). Capture clone the OS disk when the VM is running, so the current_version is not removed. When a new VM is created base on this OS disk, the current_version file is already there and the content is equal to the current wala version.

Comment 1 Yuxin Sun 2016-12-15 11:14:29 UTC
Verified on WALinuxAgent-2.2.2 upstream.
Result: PASS

Actual result:
2016/12/15 11:11:53.986245 INFO WALinuxAgent-2.2.2 did not terminate cleanly
2016/12/15 11:11:53.994978 ERROR Event: name=WALinuxAgent, op=Restart, message=WALinuxAgent-2.2.2 did not terminate cleanly

Fix method: Change message "unexpectedly restarted" to "did not terminate cleanly".

Comment 2 Yuxin Sun 2017-03-06 03:15:56 UTC
Change the status to "POST" for developers to add it into ERRATA.