Description of problem: I've seen multiple customers report problems getting an apache resource to run correctly sometimes...the service just fails to start. I finally had a spare minute to reproduce this on my end and try to troubleshoot it. Even with rgmanager debugging turned up, there isn't much useful being spit out to point to where the problem lies. So...I broke down and started running through the script by hand to see why it was failing. I was able to find that the httpd daemon was failing to start and spitting an error of "no listening sockets available, shutting down". Looking in the config file it generated in /etc/cluster/apache/<service>/httpd.conf I found that all the "Listen" lines were commented out. Back to the script, I started digging around trying to figure out how/why. I found the sed that commented out all of the existing "Listen" lines from httpd.conf, but couldn't immediately figure out why the processing just above that to write a valid "Listen" line for the cluster's VIP wasn't writing anything to the file. It turns out that I had a valid "Listen" line in /etc/httpd/conf/httpd.conf, but that the script wasn't handling it correctly. If you look in the default httpd.conf file, it shows: <snip> # Listen: Allows you to bind Apache to specific IP addresses and/or # ports, in addition to the default. See also the <VirtualHost> # directive. # # Change this to Listen on specific IP addresses as shown below to # prevent Apache from glomming onto all bound IP addresses (0.0.0.0) # #Listen 12.34.56.78:80 Listen 80 <snip> With a line of "Listen 80" in httpd.conf, apache.sh will correctly write a "Listen" line to it's new config file it is creating. However, if you instead comment this line and use one similar to the other example it shows above like "Listen 192.168.1.1:80", it will fail to properly handle this. The reason why is the grep on the end of the first for statement in this block of code inside generate_configFile(): for i in `"$APACHE_parseConfig" -D"$OCF_RESKEY_name" < "$originalConfigFile" | grep -P '(^Listen)|(^Port)' | grep -v ':'`; do port=`echo $i | sed 's/^Listen \(.*\)/\1/;s/^Port \(.*\)/\1/'`; IFS=$' '; for z in $ip_addresses; do echo "Listen $z:$port" >> "$generatedConfigFile"; done IFS=$'\n'; done; The "grep -v ':' in the for loop will parse out the line since it has a colon in it, hence $i is empty in the case where this is your only "Listen" line...which means the entire loop never runs, and it never writes a new "Listen" line into the generated config file. I made a couple changes to this loop. I believe my changes handle all previous possible situations, plus this case. Here is my new for loop: for i in `"$APACHE_parseConfig" -D"$OCF_RESKEY_name" < "$originalConfigFile" | grep -P '(^Listen)|(^Port)' `; do port=`echo $i | sed 's/^Listen \(.*\)/\1/;s/^Port \(.*\)/\1/'`; testcond=`echo $port|grep :` if [ $testcond ]; then port=`echo $port|awk -F : '{print $2}'` fi IFS=$' '; for z in $ip_addresses; do echo "Listen $z:$port" >> "$generatedConfigFile"; done IFS=$'\n'; done; I basically removed the "grep -v ':' from the first for loop. Then, I handled the case where if the line has a colon in it, to parse the port number out correctly. I've put this code into place in my /usr/share/cluster/apache.sh and tested it against multiple different /etc/httpd/conf/httpd.conf "Listen" configurations. Every variation I've tried appears to be working correctly...please let me know if I've made some kind of obvious error or missed a corner case. Version-Release number of selected component (if applicable): rgmanager-2.0.46-1.el5 How reproducible: Easily with only one "Listen" line in /etc/httpd/conf/httpd.conf that has an ip:port specified. Steps to Reproduce: 1) Place only one "Listen" line in /etc/httpd/conf/httpd.conf similar to "Listen 192.168.1.1:80" where there is a colon in the line (namely separating the ip and port). 2) Create a cluster service using the built in apache resource (that of course has an ip address in the service to start, too) 3) Try to start the service Actual results: Service fails to start Expected results: Service should start cleanly. Additional info: On a completely unrelated note...what on God's green earth kind of voodoo is trying to be accomplished with the setting, unsetting, juggling, and resetting of the IFS variable in this same block of code inside generate_configFile() ?
Changed components from "Red Hat Cluster Suite" 4 to "Red Hat Enterprise Linux 5" 5.3 Also, I decided to generate a quick patch in case it makes it easier to see what exactly I did. diff -up /usr/share/cluster/apache.sh.ORIG /usr/share/cluster/apache.sh --- /usr/share/cluster/apache.sh.ORIG 2008-12-03 16:17:54.000000000 -0500 +++ /usr/share/cluster/apache.sh 2009-03-11 16:28:49.000000000 -0400 @@ -135,8 +135,12 @@ EOT IFS_old="$IFS" IFS=$'\n' - for i in `"$APACHE_parseConfig" -D"$OCF_RESKEY_name" < "$originalConfigFile" | grep -P '(^Listen)|(^Port)' | grep -v ':'`; do + for i in `"$APACHE_parseConfig" -D"$OCF_RESKEY_name" < "$originalConfigFile" | grep -P '(^Listen)|(^Port)' `; do port=`echo $i | sed 's/^Listen \(.*\)/\1/;s/^Port \(.*\)/\1/'`; + testcond=`echo $port|grep :` + if [ $testcond ]; then + port=`echo $port|awk -F : '{print $2}'` + fi IFS=$' '; for z in $ip_addresses; do echo "Listen $z:$port" >> "$generatedConfigFile";
Created attachment 334860 [details] proposed patch
Verified error and after patching apache.sh it didn't work until I manually removed the /etc/cluster/apache/apache:httpd/httpd.conf files. You can double check by looking at those .conf's and seeing an uncommented Listen directive near the top of the file. If those don't exist, enable the service again and check for their presence.
I understand your problems and thanks for such detailed explanation. This have to be fixed but I'm not sure if changing user defined IP address is what should be done here. More appropriate solution for me will be to left these lines without changes and not bother with them. If only port number is specified (non-empty $i) then we will use it otherwise we will just use original line. What do you think about such solution?
I was mostly going off the assumption that we wanted to wipe the config file of all pre-configured Listen and Port lines, and then write our own for just the cluster configured ip addresses...simply because that's what is attempting to be done now. After the above block of code I changed, we then comment out all pre-existing Listen/Port/Pidfile lines: cat "$originalConfigFile" | sed 's/^Listen/### Listen/;s/^Port/### Port/;s/^PidFile/### PidFile/' | \ "$APACHE_parseConfig" -D"$OCF_RESKEY_name" >> "$generatedConfigFile" If you think this behavior is incorrect and would rather carry over/leave the user configured ip addresses from /etc/httpd/conf/httpd.conf in the config file we're building for the cluster resource, then I guess some more adjustment is needed. If the above is what you want to accomplish (or think is more proper behavior), I don't think my change is invalid since it still parses the port out of a Listen line that we don't handle and uses that to write a Listen line for the cluster address. Rather, I think the change you would want to make would be to the code I listed in this comment that blindly goes in and comments out all pre-existing Listen lines. Does this make sense, or did I just completely miss what you were saying?
Just as some extra input here... I'd rather the httpd.conf maintain the listening ip:port, since we already have to specify that file for the service to start anyway. This would also help if someone wanted to stack multiple daemons on a single node (in case of failover) by running different listening ports. (Unless the original script let us specify port, which I never noticed it did...) Thanks
Hi Henry, Original idea is that: 1) user will specify just port for apache 2) user will add IP resource (as sibling) to service group - that way you can have more resources with same IP address (e.g. apache + mysql) 3) resource agent will create a new apache configuration according to one provided by user. fix pid files, ... and also change port where to listen to set of IP address:port defined in service group Can you give me an example of your situation with stack multiple daemons?
Sure! So in a recent situation, I've had to setup a bunch of httpd's with a couple thousand vhosts each. These daemons listen on *:X, even though they are childed to a VIP (so I can route to a target IP based on a range). This lets me serve the data for thousands of different IPs without actually having to have that IP locally (local routes are configured to accept a block of IPs). If I have one of those daemons fail over (get stacked) they have to be on different ports or have an issue binding. Maybe we can just have the script NOT change the ip:port to the VIP if its a *:X directive in the httpd.conf?
After long decisions, I accept the attached patch. This resource agent is our first generation and perhaps it is time to try to create something better what will be used on real world use cases. If you will find some spare time, we can put ideas together and I can write it.
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2009-1339.html