Bug 1023019 - Windows 2008 - Upgrade to JON3.2.ER3 freezes on 'Updating RHQ Agent Service'
Windows 2008 - Upgrade to JON3.2.ER3 freezes on 'Updating RHQ Agent Service'
Product: JBoss Operations Network
Classification: JBoss
Component: Upgrade (Show other bugs)
JON 3.2
Unspecified Unspecified
unspecified Severity high
: CR01
: JON 3.2.0
Assigned To: Jay Shaughnessy
Mike Foley
Depends On:
Blocks: 1010354 1012435 1037824
  Show dependency treegraph
Reported: 2013-10-24 09:09 EDT by Filip Brychta
Modified: 2014-01-02 15:39 EST (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1037824 (view as bug list)
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
all logs (63.96 KB, application/x-gzip)
2013-10-24 09:09 EDT, Filip Brychta
no flags Details
screen shot (100.16 KB, image/png)
2013-11-25 11:54 EST, Filip Brychta
no flags Details

  None (edit)
Description Filip Brychta 2013-10-24 09:09:08 EDT
Created attachment 815778 [details]
all logs

Description of problem:
During an upgrade of jon3.1.2.GA to jon3.2.ER3 on windows machine, the installer freezes after removing the old agent service. Killing the installer leads to rollback and system stays not upgraded.

Using services.msc i can see that old agent service is removed (the one from jon3.1.2.GA) but new agent service (which should be installed via rhqctl) is not there.

I have following set up:
OS - Windows 2008 server, 64b
jdk - oracle jre1.7.0_45 32b

Version-Release number of selected component (if applicable):
Version: 3.2.0.ER3
Build Number: c0742ed:90dd474

How reproducible:

Steps to Reproduce:
1. jon3.1.2.GA is installed under 'Administrator' user, agent service is run as 'Administrator'
2. stop jon server and jon agent
3. uninstall the jon server service: rhq-server.bat remove
4. rhqctl upgrade --from-server-dir c:\jon-server-3.1.2.GA --from-agent-dir c:\rhq-agent --run-data-migrator do-it

Actual results:
Upgrade is frozen on line 'RHQ Agent [rhqagent-WIN-2008] removed.'
Old agent service is removed but new agent service is not created.

Expected results:
New agent service is created and upgrade is finished successfully.

Additional info:
I tried to remove old agent service manualy (rhq-agent-wrapper.bat remove) before upgrade, but it didn't solved the problem. Upgrade was frozen and last lines were
00:55:33,753 INFO  [org.rhq.server.control.command.Upgrade] Updating RHQ Agent Service...
The rhqagent-WIN-2008 service is not installed - The specified service does not exist as an installed service. (0x424)
The rhqagent-WIN-2008 service is not installed - The specified service does not exist as an installed service. (0x424)

When i kill the installer i can see in process explorer that wrapper.exe which handles rhq-agent is still running. So the problem is probably in wrapper. 

Logs attached
Comment 1 Filip Brychta 2013-10-25 09:16:46 EDT
I retested this issue to be sure it is not caused by corrupted JDK installation as discussed in bug 1022620.
I reproduced this issue with correct JDK installation.
Comment 2 Jay Shaughnessy 2013-10-28 10:06:53 EDT
I'm fairly sure this was due to Bug 1022989, and possibly assisted by Bug 1022620.  The fixes for those should resolve this issue.  Setting MODIFIED for ER5 retest.
Comment 3 Simeon Pinder 2013-11-06 21:18:02 EST
Moving to ON_QA for test with new brew build.
Comment 4 Filip Brychta 2013-11-25 11:53:11 EST
Still the same behaviour on ER7. Attaching screen shot which shows "frozen" state and processes which handle rhq-agent
Comment 5 Filip Brychta 2013-11-25 11:54:34 EST
Created attachment 828758 [details]
screen shot
Comment 6 Jay Shaughnessy 2013-11-26 11:48:43 EST
After spending some time with Filip we discovered the issue here.  It's that the agent service install was stuck waiting for interactive response for the logon account password.

The problem arises when the 3.1.2 agent install was done using RUN_AS or RUN_AS_ME, with the password being supplied interactively via prompt.

The 3.2 upgrade (or agent install) for that matter, does not allow for interactive password prompt for the agent service's logon account.  Therefore, the following env vars must be set:

In fact, the RHQ_AGENT_PASSWORD_PROMPT should probably be obsolete.

This can be resolved with documentation in the near term, but the best fix is likely a code change such that we fail installs that set RHQ_RUN_AS or RHQ_RUN_AS_ME and don't set RHQ_AGENT_PASSWORD.

Note the same is true for SERVER and STORAGE windows service installs.

*** Advise as to whether code change or doc change for 3.2...
Comment 7 Jay Shaughnessy 2013-11-27 17:35:15 EST
master commit ecb4d5877a6c585f0900389bb5c0f6bf587fe6b7
Author: Jay Shaughnessy <jshaughn@redhat.com>
Date:   Wed Nov 27 17:33:24 2013 -0500

 rhqctl-based  installs will now exit if the password is not set when the
 option is specified. The RHQ_XXX_PASSWORD_PROMPT env vars are no
 longer relevant or documented for rhqctl-based installs. Note that
 the agent is typically installed standalone, outside of rhqctl, and in
 that case the RHQ_AGENT_PASSWORD_PROMPT env var is still relevant.

 Additionally, rhq-agent.bat had some newly discovered problems handling
 certain options (32-bit java install, I think), and has had a decent
 amount of reworking to deal with the issues.
Comment 8 Jay Shaughnessy 2013-12-02 10:46:49 EST
release/jon3.2.x commit d769ac141cc0ea09dbb5ffaeaec596a9633d7732
Author: Jay Shaughnessy <jshaughn@redhat.com>
Date:   Wed Nov 27 17:33:24 2013 -0500

 Cherry-Pick of master commit ecb4d5877a6c585f0900389bb5c0f6bf587fe6b7


Given the number of script changes full Windows install testing should be performed.  Particular to this fix, attempt rhqctl installs where only RHQ_XXX_RUN_AS or RUN_AS_ME is defined, but not  RHQ_XXX_PASSWORD.  The installs should exit with a useful message.  With RHQ_XXX_PASSWORD defined the install should proceed and the service should run as the defined user account (not the system default account).

Also perform non-rhqctl-based windows service installs of the agent. Ensure the RHQ_AGENT_PASSWORD_PROMPT is used properly in those situations (it is ignored by rhqctl).
Comment 9 Filip Brychta 2013-12-03 08:54:15 EST
I tried following scenario on ER7:
1-  unzip 
2-  set RHQ_SERVER_RUN_AS_ME=true
(note: i didn't set RHQ_AGENT_PASSWORD_PROMPT=false)
4-  rhqctl install

Installation was stuck on creating agent service without any useful message.

Is this resolved as a part of commit from comment 8 as well?
Comment 10 Filip Brychta 2013-12-03 09:14:43 EST
Correction of comment 9: I pasted there incorrect env variables. All env variables (steps 2 and 3) should be relevant for agent. So instead of RHQ_SERVER... and RHQ_STORAGE... should be RHQ_AGENT...
Comment 11 Jay Shaughnessy 2013-12-03 15:22:18 EST
There were too many changes to worry about ER7.  Please test with CR1, which has (or will have) the changes when it is released.
Comment 14 Simeon Pinder 2013-12-03 18:19:34 EST
Moving to ON_QA for testing in latest(CR1) brew build.
Comment 15 Filip Brychta 2013-12-05 05:57:46 EST
Verified on:
Version :	
Build Number :	

Verified following things for both the clean installation and upgrage:
 - installation/upgrade was successful for default user (Local System) 
 - installation/upgrade was successful for Administrator user (set RHQ_AGENT_RUN_AS_ME=true, set RHQ_AGENT_PASSWORD=***)
 - installation/upgrade was stopped with useful message when only RHQ_SERVER_RUN_AS_ME was set
 - non-rhqctl-based windows service installs of the agent works as expected

Note You need to log in before you can comment on or make changes to this bug.