Bug 1000612

Summary: RHQ Agent failed to discover postgres resources when postgres user alter its password
Product: [Other] RHQ Project Reporter: Roger Hui <rhui>
Component: PluginsAssignee: Jirka Kremser <jkremser>
Status: CLOSED WORKSFORME QA Contact: Mike Foley <mfoley>
Severity: high Docs Contact:
Priority: unspecified    
Version: unspecifiedCC: dowoods, hrupp, jkremser, loleary
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1073995 (view as bug list) Environment:
Last Closed: 2014-03-19 15:48:31 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1073995    

Description Roger Hui 2013-08-23 20:23:18 UTC 
Description of problem:

JON Agent failed to discover postgres database when we alter the postgres user password. I've tried both md5 or trust authentication same result. 


Version-Release number of selected component (if applicable):
RHEL 6.4 for both JON Server and Agent
JON 3.1.2
psql (8.4.13)


How reproducible:
Every time


Steps to Reproduce:
1. Set up a jon server and another server (remote one, localhost postgres won't hit this issue)
2. Install postgres 
3. /bin/su -c "psql -c \"ALTER USER postgres PASSWORD 'xxxx';\"" postgres
If the password hasn't altered, it will work. 
4. Observe the agent.log 
2013-08-20 17:21:43,735 INFO [ResourceDiscoveryComponent.invoker.daemon-1] (org.rhq.plugins.postgres.PostgresDiscoveryComponent)- Discovered a postgres process: ProcessScanResult: scan=[ProcessScan: query=[process|basename|match=^(postgres|postmaster)$,process|basename|nomatch|parent=^(postgres|postmaster)$], name=[unix]], info=[process: pid=[5903], name=[/usr/bin/postmaster], ppid=[1]]
2013-08-20 17:21:43,843 WARN [InventoryManager.discovery-1] (rhq.core.pc.inventory.InventoryManager)- Failure during discovery for [Postgres Server] Resources - failed after 109 ms.
java.lang.Exception: Discovery component invocation failed.
at org.rhq.core.pc.util.DiscoveryComponentProxyFactory$ComponentInvocationThread.call(DiscoveryComponentProxyFactory.java:297)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
at java.util.ArrayList.rangeCheck(ArrayList.java:604)
at java.util.ArrayList.get(ArrayList.java:382)
at org.rhq.plugins.postgres.PostgresDiscoveryComponent.getServerResourceName(PostgresDiscoveryComponent.java:334)
at org.rhq.plugins.postgres.PostgresDiscoveryComponent.createResourceDetails(PostgresDiscoveryComponent.java:168)
at org.rhq.plugins.postgres.PostgresDiscoveryComponent.discoverResources(PostgresDiscoveryComponent.java:142)
at sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.rhq.core.pc.util.DiscoveryComponentProxyFactory$ComponentInvocationThread.call(DiscoveryComponentProxyFactory.java:293)
... 5 more


Actual results:
JON won't discover the postgres resource on the inventory. 


Expected results:
JON should be able to handle this gracefully or have a way for user to enter the password in the connection settings before it blow up even in the discovery stage?
Comment 1 Roger Hui 2013-08-23 20:25:05 UTC 
Related to https://bugzilla.redhat.com/show_bug.cgi?id=1000059 but not supported on JON.
Comment 3 Jirka Kremser 2014-03-19 15:48:31 UTC 
Not able to reproduce on 3.2.0 GA:

I've installed a Postgresql and agent on remote machine. Changed the password of user postgres.. I got following error during availability scan (not during discovery phase, discovery went well):

2014-03-19 11:37:15,325 WARN  [InventoryManager.availability-1] (rhq.core.pc.inventory.AvailabilityExecutor)- Availability collection failed with exception on Resource[id=10181, uuid=205af431-6ff0-462a-b699-b32be48b5606, type={Postgres}Postgres Server, key=jdbc:postgresql://127.0.0.1:5432/postgres, name=postgres, parent=jkremser-agentonly.bc.jonqe.lab.eng.bos.redhat.com], availability will be reported as DOWN, reason=Failed to start component for Resource[id=10181, uuid=205af431-6ff0-462a-b699-b32be48b5606, type={Postgres}Postgres Server, key=jdbc:postgresql://127.0.0.1:5432/postgres, name=postgres, parent=jkremser-agentonly.bc.jonqe.lab.eng.bos.redhat.com] - Unable to open a test connection to the given database. JDBC url = null, username = postgres. Terminating connection pool (set lazyInit to true if you expect to start your database after your app). Original Exception: ------
org.postgresql.util.PSQLException: FATAL: password authentication failed for user "postgres"
	at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:398)
	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:173)
	at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:64)
	at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:136)
	at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:29)
	at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:21)
	at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:31)
	at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
	at org.postgresql.Driver.makeConnection(Driver.java:393)
	at org.postgresql.Driver.connect(Driver.java:267)
	at org.rhq.plugins.database.DriverDataSource.getConnection(DriverDataSource.java:96)
	at com.jolbox.bonecp.BoneCP.obtainRawInternalConnection(BoneCP.java:341)
	at com.jolbox.bonecp.BoneCP.<init>(BoneCP.java:416)
	at org.rhq.plugins.database.BasePooledConnectionProvider.<init>(BasePooledConnectionProvider.java:140)
	at org.rhq.plugins.postgres.PostgresPooledConnectionProvider.<init>(PostgresPooledConnectionProvider.java:40)
	at org.rhq.plugins.postgres.PostgresServerComponent.start(PostgresServerComponent.java:95)
	at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.rhq.core.pc.inventory.ResourceContainer$ComponentInvocation.call(ResourceContainer.java:759)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)





This ^ exception is, however, a reasonable result and it is easy to fix the system afterwards. One has to set the password under the Connection Settings to the same 'xxxx' as in "ALTER USER postgres PASSWORD 'xxxx'". 

Also the AuthN method for localhost (agent is colocated on the same box) in /var/lib/pgsql/data/pg_hba.conf must be set to md5, password or trust if using the non-default postgres password.