Bug 806028 - Installer fails, calls postgres sysinit script which will fail if it takes more than 2s to start
Installer fails, calls postgres sysinit script which will fail if it takes mo...
Status: CLOSED ERRATA
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Installer (Show other bugs)
6.0.0
Unspecified Unspecified
unspecified Severity high (vote)
: Unspecified
: --
Assigned To: Lukas Zapletal
Katello QA List
: Triaged
Depends On: 802346
Blocks:
  Show dependency treegraph
 
Reported: 2012-03-22 13:32 EDT by Jeff Weiss
Modified: 2014-11-09 17:52 EST (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
installing SE with kickstart using virt-install
Last Closed: 2012-05-15 15:10:09 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Jeff Weiss 2012-03-22 13:32:08 EDT
Description of problem:
see https://bugzilla.redhat.com/show_bug.cgi?id=795600#c8

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:
Comment 1 Lukas Zapletal 2012-03-23 04:27:57 EDT
Okay so the workaround is to modify the startup script and force it to never return failure when PID does not exist.

Then I will have to cherry pick also the second workaround that waits until PostgreSQL socket is ready, because initial database creation could also potentionaly fail:

https://bugzilla.redhat.com/show_bug.cgi?id=802346
Comment 2 Lukas Zapletal 2012-03-23 05:59:55 EDT
Pushing upstream:

c5f0990 806028 - postgres sysvinit script workaround

This workadound basically removes the if statement from the postgresql init script.

+  # RHBZ 800534 for RHEL 6.x - pg sysvinit script return non-zero ...
+  exec { "fix-pgsysvinit":
+    path        => "/usr/bin:/bin",
+    onlyif      => "grep '\"x\$pid\" != x' /etc/init.d/postgresql",
+    command     => "sed -i 's/\"x\$pid\" != x/1 = 1/g' /etc/init.d/postgresql"
+  }

I have reopened bug for RHEL 6.4+ so we could remove this workaround in next major release: https://bugzilla.redhat.com/show_bug.cgi?id=800534
Comment 4 Lukas Zapletal 2012-03-23 06:07:15 EDT
Setting depends on.
Comment 5 Corey Welton 2012-03-23 11:12:26 EDT
FYI, I saw something similar happen when timing out trying to start... httpd or tomcat6 or something. Re-running katello-configure worked just fine.
Comment 6 Lukas Zapletal 2012-03-23 12:20:20 EDT
@Corey - if you hit it again please report. We had similar issue with Candlepin (db initialization), but this should be resolved too.
Comment 7 Corey Welton 2012-03-29 09:28:47 EDT
Actually found my old log:

err: /Stage[main]/Candlepin::Service/Exec[wait for candlepin]/returns: change from notrun to 0 failed: /usr/bin/wget --spider --tries 10 --retry-connrefused --no-check-certificate https://localhost:8443/candlepin/ returned 8 instead of one of [0] at /usr/share/katello/install/puppet/modules/candlepin/manifests/service.pp:10

Again, rerunning worked. But I haven't seen it occur since. Will obviously bug it if it returns.
Comment 9 Lukas Zapletal 2012-03-30 04:57:31 EDT
@Corey - that is another issue, we should have a BZ on that (VERIFIED), maybe to reopen it.
Comment 10 Corey Welton 2012-04-02 16:49:27 EDT
Currently trying to tackle/test this; I've been experimenting with adding sleep statements (e.g., "sleep 5s") in various locations within the start/restart functions of the postgresql initscript, but I'm looking for more detail as to whether this would be sufficient to exercise the functionality.
Comment 11 Corey Welton 2012-04-03 13:20:13 EDT
Testing notes:  

In addition to the above, I've attempted to do the following - used the lines below script to run at the point when puppet was looking for the open postgresql connections.

  
iptables -A OUTPUT -p tcp --destination-port 5432 -s 127.0.0.1 -j DROP
iptables -L -v
echo Sleeping...
sleep 6s
iptables -A OUTPUT -p tcp --destination-port 5432 -s 127.0.0.1 -j ACCEPT
Comment 12 Corey Welton 2012-04-03 14:02:49 EDT
rather, this is the corrected script

iptables -A OUTPUT -p tcp --destination-port 5432 -s 127.0.0.1 -j DROP
iptables -L -v
echo Sleeping...
sleep 6s


after tailing the logfile and noticing it wait on postgres, i reenabled the port by restarting iptables (w/o the rule listed above).

End result was that puppet waited accordingly for postgresql to start up.

I've tried exhaustively to repro this, and short of ending up with a real-world scenario where I can see it take place in a repeatable manner, I'd posit that this has been tested as thoroughly as possible.

Marking QA Verified.
Comment 13 errata-xmlrpc 2012-05-15 15:10:09 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2012-0661.html

Note You need to log in before you can comment on or make changes to this bug.