Bug 694476 - [apache] The plugin doesn't reflect the true runtime configuration of apache server which forces monitoring to not be available in all cases
Summary: [apache] The plugin doesn't reflect the true runtime configuration of apache ...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: RHQ Project
Classification: Other
Component: Plugins
Version: 4.0.0
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: ---
: ---
Assignee: RHQ Project Maintainer
QA Contact: Corey Welton
URL:
Whiteboard:
Depends On: 702691
Blocks: jon3 700616 702690 rhq401 705004
TreeView+ depends on / blocked
 
Reported: 2011-04-07 13:29 UTC by Lukas Krejci
Modified: 2018-11-14 14:04 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 700616 (view as bug list)
Environment:
Last Closed: 2011-05-24 01:09:26 UTC
Embargoed:


Attachments (Terms of Use)

Description Lukas Krejci 2011-04-07 13:29:53 UTC
Description of problem:
The apache plugin doesn't take into account the conditional aspects of the configuration.

For example a VirtualHost that is nested inside an IfModule or IfDefine directive isn't discovered even if it is actually "in effect".

Note that nesting definitions inside conditional directives is a common practice even if VirtualHosts are not that usual to be nested (but it's not a rare occurence to see such configuration).

This makes the inventory not reflect the true state of the runtime configuration of the apache server.

Version-Release number of selected component (if applicable):
all

How reproducible:
always

Steps to Reproduce:
1. edit httpd.conf and nest a virtual host definition inside an ifmodule that checks for a module that IS being loaded
2. start the apache server
3. discover it using RHQ
  
Actual results:
the nested vhost isn't discovered

Expected results:
the nested vhost should be discovered if it is in effect.

Additional info:

This obviously is not limited to just vhosts but any directives.

The problem with IfDefine is that the values it uses are passed to the apache executable as parameters. Are we able to read those with our process scan?

Comment 1 Lukas Krejci 2011-04-08 13:20:50 UTC
Note that if we successfully implemented this, we could get rid of the imperfect matching rules between what's in the configuration files and what SNMP provides.

SNMP reports the vhost hostnames according to the values specified in the ServerName directives. But in the following (legal and realistic) example, the SNMP reports the same hostname and port for 2 different vhosts, which makes our current matching techniques unusable:


<VirtualHost _default_:80>
    DocumentRoot /var/www/common
    ServerName public-hostname.com:80
</VirtualHost>

<VirtualHost 10.1.1.1:80>
    DocumentRoot /var/www/special
    ServerName public-hostname.com:80
</VirtualHost>

The usecase for the above configuration is that the actual server is hidden behind a load-balancer / internet-facing server that relays the requests to our instance. Our instance is configured to report back the hostname of that 'front server'. But the front server has some rules out of our control to direct "special class" of requests to the "special" vhost, while all other will go to the common one.

Because we currently cannot assume that the vhosts discovered by the plugin are the same (because we leave out the conditionally defined ones) as the actual ones reported by SNMP (i.e. the runtime configuration), we cannot distinguish which of the 2 discovered vhosts corresponds to which in the SNMP report, because there are 2 records in the SNMP report with the same hostname and port.

If we successfully managed to parse out the actual runtime configuration, we could match the discovered vhosts and the SNMP records merely by their indices.

I think detecting the runtime configuration is doable if we watch out for several gotchas:

1) IfDefine can theoretically refer to -D variables hardcoded at compile time to the apache binary (but we should be able to detect those using the utilities we already have in the plugin code)

2) Names used in IfModule are potentially different to the ones in corresponding LoadModule directives but this is rare. We could solve this by having a map in the plugin configuration to cover for the exceptional cases which we wouldn't handle by default.

3) some modules are compiled in - the list of these can be retrieved by executing "httpd -l"

Comment 3 Lukas Krejci 2011-04-13 15:55:52 UTC
In the commit below I actually implemented 2 things at once. I updated the name of this bug to take that into account.

commit 24dd3ebadfca65618caad06ab45887a09d0f3972
Author: Lukas Krejci <lkrejci>
Date:   Wed Apr 13 17:50:24 2011 +0200

    BZ 694476 - We now correctly determine the runtime configuration of the apache server
    by taking into account the list of loaded modules and -D options passed to it on commandline.
    
    With this in place, it is possible to deterministically match the discovered vhost resources
    with the vhosts from the list returned by the SNMP plugin merely knowing the ordering in which
    the SNMP plugin returns the list.
    
    This commit contains both of the above fixes.

Comment 4 Lukas Krejci 2011-04-14 13:06:25 UTC
commit 5e6b9a0580eb8373a33e5054b184f4d91a674681
Author: Lukas Krejci <lkrejci>
Date:   Thu Apr 14 14:59:50 2011 +0200

    BZ 694476 - the user is now only expected to list the non-standard module mappings instead of all modules in the plugin config. This enables a clear and simple upgrade path from prev
    
    Also made the loaded module detection more robust.

Comment 5 Lukas Krejci 2011-04-20 13:53:10 UTC
commit c8f7388581b8d3c00c17ecf4732f77481c83a86e
Author: Lukas Krejci <lkrejci>
Date:   Wed Apr 20 15:01:33 2011 +0200

    BZ 694476 - Adding support for the IfVersion directive in detecting the runtime configuration.

Comment 6 Lukas Krejci 2011-04-28 18:27:50 UTC
This bug depends on bug 700461 because the apache plugin depends on up-to-date native process info being available so that it can determine the correct runtime configuration.

Comment 8 Lukas Krejci 2011-05-16 11:26:08 UTC
Additional commits to fix this bug:

43332d44c33e12ff5dfcfed715d53f7709697bfb
635df1088bb12a327b698af04cd71a2146a304ee
10261c194c6a411673e1b7b791183f06b8f44ddf
4bdfdfa198868d8e52b90a398843ddb058cc7d25

Comment 9 Lukas Krejci 2011-05-16 12:44:03 UTC
The above 4 commits have been cherry-picked to release-4.0.0 branch as:

d12ebcb50c3718399eb7247f73b4474eaac40c6d
b7d179f450b2a689af7fef01cd61ce47c4a22812
88ec1a2aa9fc0868921a3c1e280f4c8f85a7a57f
116848b1d4deac8af998e1a2e52813c616204162

Comment 10 Mike Foley 2011-05-16 14:14:28 UTC
from conference call 5/16/2011.  unit tests have been added  for the new configuration nesting by lukas.  lukas to define discrete tests for sunil to perform the verification.  verification to be done by sunil who has domain knowledge of this area from previous verifications.

Comment 11 Lukas Krejci 2011-05-16 16:41:35 UTC
things to test with this bug:

a) VHost nesting
================

Apache has 3 conditional directives: IfModule, IfVersion and IfDefine, please read up on the possible parameters to those in the apache docs: http://httpd.apache.org/docs/2.0/mod/quickreference.html

The plugin should support the full syntax of these and any number of nesting. As an example, this should be handled correctly:

<IfModule alias_module>
<IfModule !mod_proxy.c>

<VirtualHost 127.0.0.1:80>
</VirtualHost>

<IfVersion /2\.2\.1[7-8]/>
<VirtualHost 127.0.0.1:81>
</VirtualHost>
</IfVersion>

</IfModule>

<IfDefine RHQ_TESTING>
<VirtualHost 127.0.0.1:82>
</VirtualHost>
</IfDefine>

</IfModule>

Other examples can be found in the unit tests I created for this:
http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=tree;f=modules/plugins/apache/src/test/resources/runtime-config/conditional;hb=release-4.0.0

with the code for the tests located in http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=blob;f=modules/plugins/apache/src/test/java/org/rhq/plugins/apache/RuntimeConfigurationTest.java;hb=release-4.0.0

b) Inclusion order
==================

The Include directive, which is used to include other files into the apache server configuration, can accept glob patterns to specify more than 1 file at once. If there is more than 1 file matching the glob pattern, they are included in the alphabetical order. This was not the case in the previous versions of the plugin and needs to be tested. Again, there is a simple unit test for this with the config files in:
http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=tree;f=modules/plugins/apache/src/test/resources/runtime-config/incl-order;hb=release-4.0.0

In a nutshell, if there is an Include directive like this:

Include conf.d/*.conf

and the $APACHE_SERVER_ROOT/conf.d contains files a.conf, b.conf and c.conf, they should be included in alphabetical order.

c) Non-default modules handling
===============================

Apache uses 2 names to identify a module that are impossible to match without reading in the module libraries, which given the number of platforms we have to support is infeasible to do in java code.

In a LoadModule directive, that actually loads the module, the module is identified by a "module identifier", usually having a format of "foo_module". The IfModule directive uses the other name - the source file name - to identify a module, e.g. "mod_foo.c" (as of Apache 2.1, IfModule can also use the module identifier).

The plugin comes with the default list of "known modules" that is a collection of modules listed on http://httpd.apache.org/docs/2.2/mod/ or http://httpd.apache.org/docs/1.3/mod/ respectively + mod_jk and the snmp module identifiers. 

It is quite common though that custom modules are used that are not present in the default list - on Fedora 14, you can use the following command to see the list of the packages that provide additional modules to apache:

yum whatprovides '/usr/lib/httpd/modules/*'

pick and install some of them to see what their identifier and source files are.

Use some of them your apache configuration. If you use the module identifier for both LoadModule and IfModule directives, you should be able to discover the vhosts nested in those IfModule directives no problem. However, if you use the source file to identify the module in the IfModule directive, you should see a warning in the agent log (logged only once per agent lifetime) that the directive references an unknown module like:

2011-05-11 15:04:21,696 WARN  [ResourceDiscoveryComponent.invoker.daemon-1] (rhq.plugins.apache.util.RuntimeApacheConfiguration)- Encountered unknown module name in an IfModule directive: mod_unknown.c

The plugin handles such non-standard modules using the "Custom Module Names" property in its connection settings. In there you can specify the two names of the non-standard modules. After specifying the non-standard modules in that list, the warning is still going to appear in the logs once per agent lifetime but the module will be taken into account during the discovery of vhosts.

The warnings appear during the apache server discovery, where we have no notion of the custom modules and therefore we can only work with the list of known modules. Once the Custom Module Names list is filled in for a particular apache server, its resource configuration as well as the vhost discovery should accomodate to that list and  "correct itself".

So to summarize, here are steps to test:

1) Have 2 modules installed to apache that are not part of the standard distribution. Let's call them a and b.

2) Use the LoadModule directive to load module a but don't load module b.

3) Define 4 vhosts nested in IfModules along the lines of:
<IfModule a>
<VirtualHost 127.0.0.1:80>
</VirtualHost>
</IfModule>

<IfModule mod_a.c>
<VirtualHost 127.0.0.1:81>
</VirtualHost>
</IfModule>

<IfModule b>
<VirtualHost 127.0.0.1:82>
</VirtualHost>
</IfModule>

<IfModule mod_b.c>
<VirtualHost 127.0.0.1:83>
</VirtualHost>
</IfModule>

4) Discover the apache server, you should see 2 warnings in the logs about unknown modules mod_a.c and mod_b.c. The vhosts on port 80 should appear in the inventory, the vhosts on port 81-83 should NOT appear in the inventory.


5) Define the mapping between module a and mod_a.c and b and mod_b.c in the connection settings of the server.

6) Restart the agent and run discovery -f on its commandline.

7) vhost on port 81 should appear in the inventory, vhosts on port 82 and 83 should not appear in the inventory, another 2 warnings about the modules should appear.

d) Robust monitoring support 1
==============================

Unlike in previous versions of the plugin, it should always be able to identify the SNMP "index" that it can use to collect monitoring data.

1) Configure SNMP module in the apache server.

2) Set up 2 virtual hosts like this:

<VirtualHost _default_:8080>
  DocumentRoot /var/www/html
  ServerName monitoring-test:8080
</VirtualHost>

<VirtualHost 127.0.0.1:8080>
  DocumentRoot /var/www/html
  ServerName monitoring-test:8080
</VirtualHost>

3) Discover the apache server, both vhosts should appear in the inventory.

4) load http://127.0.0.1:8080 in your server and keep reloading it (or wget the url in a loop or whatever to create requests on that specific url).

5) You should see the load you created in the step 4 represented in the measurements collected on the second vhost and *only* on that.

e) Robust monitoring support 2
==============================

1) Repeat the tests from "Robust monitoring support 1".
2) Edit the apache configuration and swap the order of the vhost directives.
3) restart apache server, do NOT rediscover the server resource in RHQ
4) Do the "Robust monitoring support 1" tests again, all should still work.

f) Configuration support
========================

Even the nested vhosts should still support configuration reading and updates. The configuration is explicitly *NOT* supported for vhosts that contain additional conditional directives (i.e. if the virtual host definition contains any IfModule, IfVersion or IfDefine (even if indirectly Include-ed from another file), the attempt to read (or update) the resource configuration of the vhost will fail with an exception explaining the situation.

This is a change from the previous version of the plugin which could silently mess up the configuration.

g) The rest
===========

All other things should work as well in any of the above tests - the operations on the server resource should work, as well as child resource creation.

Comment 12 Lukas Krejci 2011-05-16 16:46:14 UTC
I forgot to mention - in the test "d) Robust Monitoring Support 1" - make sure that in addition to the virtual host definitions, the server itself listens on the ports you specify in the vhost directives, i.e. in the example given, make sure to have:

Listen 8080

directive in your apache config

Comment 13 Sunil Kondkar 2011-05-18 11:49:13 UTC
Tested on build#31 (Version: 4.0.1-SNAPSHOT Build Number: dc9a4b6)
 
The tests a) VHost nesting b) Inclusion order are working as expected.

Test d) Robust monitoring support 1
Configured SNMP module in the apache server and followed the steps.
Observed that both the vhosts appear in the inventory as expected.
Created a script which wgets the URL http://127.0.0.1:8080. The monitoring tab displays data for  403 responces per minute only on the vhost 127.0.0.1:8080

Test e) Robust monitoring support 2

Edited the apache configuration and swapped the order of the vhost directives. Restarted apache server, the "Robust monitoring support 1" tests still work.

Test f) Configuration support

Verified that the nested vhosts support configuration reading and updates.

To verify vhosts that contain additional conditional directives:

Created a vhost containing a IfModule as below:

<VirtualHost sunillaptop.pnq.redhat.com:80>
    ServerAdmin root@localhost
    DocumentRoot /var/www/html/sunil
    DirectoryIndex index2.html
    ServerName sunillaptop.pnq.redhat.com:80

<IfModule alias_module>
</IfModule>

</VirtualHost>


Clicking on configuration tab of the vhost containing a IfModule display Globally uncaught exception in UI and displays a warming message in server log as below:

2011-05-18 00:26:43,124 WARN  [org.rhq.enterprise.server.configuration.ConfigurationManagerBean] Could not get live configuration for resource [Resource[id=10682, type=Apache Virtual Host, key=sunillaptop.pnq.redhat.com:80|sunillaptop.pnq.redhat.com:80, name=sunillaptop.pnq.redhat.com:80, parent=sunillaptop:80]]org.rhq.core.clientapi.agent.PluginContainerException:Plugin Error: Resource Component for [Apache Virtual Host] Resource with id [10682]: An exception was thrown. -> org.rhq.core.util.exception.WrappedRemotingException:java.lang.IllegalStateException: Configuration of the virtual host [sunillaptop.pnq.redhat.com:80|sunillaptop.pnq.redhat.com:80] contains conditional blocks. This is not supported by this plugin. -> org.rhq.core.util.exception.WrappedRemotingException:Configuration of the virtual host [sunillaptop.pnq.redhat.com:80|sunillaptop.pnq.redhat.com:80] contains conditional blocks. This is not supported by this plugin.
2011-05-18 00:26:43,124 WARN  [org.rhq.enterprise.server.configuration.ConfigurationManagerBean] Could not get live resource configuration for resource [Resource[id=10682, type=Apache Virtual Host, key=sunillaptop.pnq.redhat.com:80|sunillaptop.pnq.redhat.com:80, name=sunillaptop.pnq.redhat.com:80, parent=sunillaptop:80]]; will assume latest resource configuration update is the current resource configuration.
2011-05-18 00:26:43,124 WARN  [org.rhq.enterprise.server.configuration.ConfigurationManagerBean] Could not get live resource configuration for resource [Resource[id=10682, type=Apache Virtual Host, key=sunillaptop.pnq.redhat.com:80|sunillaptop.pnq.redhat.com:80, name=sunillaptop.pnq.redhat.com:80, parent=sunillaptop:80]]; will assume latest resource configuration update is the current resource configuration.

This is as expected.

g) The rest

Did some operations and created child resources which works fine.


However, the test for C) Non-default modules handling is not working on my setup. The vhost 127.0.0.1:80 is not discovered. 

Discussed the issue with lukas and marking this bug as ON_DEV.

Comment 14 Lukas Krejci 2011-05-18 12:00:23 UTC
Added a fix for the failed test as mentioned above by Sunil.

master branch:

commit 241f8a1ee1fe40bdd9522cf97aba37846e36ed1c
Author: Lukas Krejci <lkrejci>
Date:   Wed May 18 13:55:51 2011 +0200

    BZ 694476 - fixing the handling of loaded modules.

commit d4edbc99bed2e836f91d50745207844970f52b6d
Author: Lukas Krejci <lkrejci>
Date:   Wed May 18 13:55:51 2011 +0200

    BZ 694476 - fixing the handling of loaded modules.

Comment 15 Lukas Krejci 2011-05-18 12:01:20 UTC
the second commit in the comment above was made to release-4.0.0 branch

Comment 16 Sunil Kondkar 2011-05-18 15:26:04 UTC
Verified on build32 (Version: 4.0.1-SNAPSHOT Build Number: d4edbc9)

verified the test for C) Non-default modules handling:

The vhost 127.0.0.1:80 and 127.0.0.1:81 are discovered and there are warnings in the logs about unknown modules.

The vhosts on port 82 and 83 did not appear in the inventory as expected.

Marking as verified.

Comment 17 Corey Welton 2011-05-24 01:09:26 UTC
Bookkeeping - closing bug - fixed in recent release.


Note You need to log in before you can comment on or make changes to this bug.