+++ This bug was initially created as a clone of Bug #1085033 +++

Description of problem:
I noticed that when rhnsearch is running it connects on the loopback to the database regardless of whats set in /etc/rhn.conf

Version-Release number of selected component (if applicable):
spacewalk-search-2.1.14-1.el6.noarch

How reproducible:
in my configuration its fairly easy I have two spacewalk servers running PostgreSQL 9.1 in hot spare syncronous replication mode. all of the processes except for taskomatic are running on both nodes.

Steps to Reproduce:
1.set db_host /etc/rhn/rhn.conf to an external host name
2. run /etc/init.d/rhn-search start
3. run ps ax |grep postgres|grep 127.0.0.1

you will see several connection to the loop back address

4. run /etc/init.d/rhn-search stop
5. run ps ax |grep postgres|grep 127.0.0.1

you will now see all of the PostgreSQL connections over the loop back have gone away


Actual results:

rhnsearchd is connecting to the database over the loopback


Expected results:

rhnsearchd should read the db_host variable in the configuration file and connect to the database host specified.


Additional info:

here is some of the log output from /var/log/rhn/search/rhn_search_daemon.log on the secondary host on which the database in in read only hot spare mode.

INFO   | jvm 1    | 2014/04/07 10:59:52 | 2014-04-07 10:59:52,915 [QuartzScheduler_Worker-3] ERROR org.quartz.core.ErrorLogger - Job (updateIndex.server threw an exception.
INFO   | jvm 1    | 2014/04/07 10:59:52 | org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: org.apache.ibatis.exceptions.PersistenceException:
INFO   | jvm 1    | 2014/04/07 10:59:52 | ### Error updating database.  Cause: org.postgresql.util.PSQLException: ERROR: cannot execute UPDATE in a read-only transaction
INFO   | jvm 1    | 2014/04/07 10:59:52 | ### The error may involve server.updateLastServer-Inline
INFO   | jvm 1    | 2014/04/07 10:59:52 | ### The error occurred while setting parameters
INFO   | jvm 1    | 2014/04/07 10:59:52 | ### SQL: UPDATE rhnIndexerWork SET last_id = ?, last_modified = ? where object_type = 'server'
INFO   | jvm 1    | 2014/04/07 10:59:52 | ### Cause: org.postgresql.util.PSQLException: ERROR: cannot execute UPDATE in a read-only transaction]
INFO   | jvm 1    | 2014/04/07 10:59:52 |       at org.quartz.core.JobRunShell.run(JobRunShell.java:227)
INFO   | jvm 1    | 2014/04/07 10:59:52 |       at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)
INFO   | jvm 1    | 2014/04/07 10:59:52 | Caused by: org.apache.ibatis.exceptions.PersistenceException:
INFO   | jvm 1    | 2014/04/07 10:59:52 | ### Error updating database.  Cause: org.postgresql.util.PSQLException: ERROR: cannot execute UPDATE in a read-only transaction
INFO   | jvm 1    | 2014/04/07 10:59:52 | ### The error may involve server.updateLastServer-Inline
INFO   | jvm 1    | 2014/04/07 10:59:52 | ### The error occurred while setting parameters
INFO   | jvm 1    | 2014/04/07 10:59:52 | ### SQL: UPDATE rhnIndexerWork SET last_id = ?, last_modified = ? where object_type = 'server'
INFO   | jvm 1    | 2014/04/07 10:59:52 | ### Cause: org.postgresql.util.PSQLException: ERROR: cannot execute UPDATE in a read-only transaction
INFO   | jvm 1    | 2014/04/07 10:59:52 |       at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:23)
INFO   | jvm 1    | 2014/04/07 10:59:52 |       at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:150)
INFO   | jvm 1    | 2014/04/07 10:59:52 |       at com.redhat.satellite.search.db.WriteQuery.update(WriteQuery.java:57)
INFO   | jvm 1    | 2014/04/07 10:59:52 |       at com.redhat.satellite.search.scheduler.tasks.GenericIndexTask.updateLastRecord(GenericIndexTask.java:112)
INFO   | jvm 1    | 2014/04/07 10:59:52 |       at com.redhat.satellite.search.scheduler.tasks.GenericIndexTask.execute(GenericIndexTask.java:76)
INFO   | jvm 1    | 2014/04/07 10:59:52 |       at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
INFO   | jvm 1    | 2014/04/07 10:59:52 |       ... 1 more
INFO   | jvm 1    | 2014/04/07 10:59:52 | Caused by: org.postgresql.util.PSQLException: ERROR: cannot execute UPDATE in a read-only transaction
INFO   | jvm 1    | 2014/04/07 10:59:52 |       at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2062)
INFO   | jvm 1    | 2014/04/07 10:59:52 |       at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1795)
INFO   | jvm 1    | 2014/04/07 10:59:52 |       at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
INFO   | jvm 1    | 2014/04/07 10:59:52 |       at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:479)
INFO   | jvm 1    | 2014/04/07 10:59:52 |       at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:367)
INFO   | jvm 1    | 2014/04/07 10:59:52 |       at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:360)
INFO   | jvm 1    | 2014/04/07 10:59:52 |       at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.execute(NewProxyPreparedStatement.java:362)
INFO   | jvm 1    | 2014/04/07 10:59:52 |       at org.apache.ibatis.executor.statement.PreparedStatementHandler.update(PreparedStatementHandler.java:41)
INFO   | jvm 1    | 2014/04/07 10:59:52 |       at org.apache.ibatis.executor.statement.RoutingStatementHandler.update(RoutingStatementHandler.java:66)
INFO   | jvm 1    | 2014/04/07 10:59:52 |       at org.apache.ibatis.executor.SimpleExecutor.doUpdate(SimpleExecutor.java:45)
INFO   | jvm 1    | 2014/04/07 10:59:52 |       at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:100)
INFO   | jvm 1    | 2014/04/07 10:59:52 |       at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:75)
INFO   | jvm 1    | 2014/04/07 10:59:52 |       at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:148)
INFO   | jvm 1    | 2014/04/07 10:59:52 |       ... 5 more

--- Additional comment from Jesse Weisner on 2014-08-27 21:39:11 CEST ---

This patch made rhnsearchd work with external PostgreSQL, but I haven't tested it with "embedded" PostgreSQL yet.

This patch is against the SRPM of the current stable version (2.2.8).

--- Additional comment from Tomas Lestach on 2014-09-23 11:12:43 CEST ---

Hello,

the patch does not seem to be correct.
The 'else' path at [1] constructs the 'db_name' according to your rhn.conf file.

What does
# egrep 'db_(host|name|port)' /etc/rhn/rhn.conf
say?

[1] https://github.com/spacewalkproject/spacewalk/blob/master/search-server/spacewalk-search/src/java/com/redhat/satellite/search/db/DatabaseManager.java#L76

--- Additional comment from Jesse Weisner on 2014-09-23 23:54:01 CEST ---

(In reply to Tomas Lestach from comment #2)
> Hello,
> 
> the patch does not seem to be correct.
> The 'else' path at [1] constructs the 'db_name' according to your rhn.conf
> file.
> 
> What does
> # egrep 'db_(host|name|port)' /etc/rhn/rhn.conf
> say?
> 
> [1]
> https://github.com/spacewalkproject/spacewalk/blob/master/search-server/
> spacewalk-search/src/java/com/redhat/satellite/search/db/DatabaseManager.
> java#L76

# egrep 'db_(host|name|port)' /etc/rhn/rhn.conf
db_name = spacewalk
db_host = pgsql.example.com
db_port = 5432

--- Additional comment from Jesse Weisner on 2014-09-24 00:04:06 CEST ---

Without adding db_host to the list of "options" (which is what I assume https://github.com/spacewalkproject/spacewalk/blob/master/search-server/spacewalk-search/src/java/com/redhat/satellite/search/db/DatabaseManager.java#L69 does), db_host is empty, so it can't get added to the jdbcUrl property at https://github.com/spacewalkproject/spacewalk/blob/master/search-server/spacewalk-search/src/config/com/redhat/satellite/search/db/config.xml#L19

This made rhn-search work for me, but I have no idea if it's the right solution 8-)

--- Additional comment from Tomas Lestach on 2014-09-24 12:14:04 CEST ---

That you for the provided information, now I see the issue.

spacewalk.git: 4073abaf7bc7f17b738bcc0ca64e657ab9d074bf