Bug 700616

Summary: [RHQ 3.0.1][apache] The plugin doesn't reflect the true runtime configuration of apache server which forces monitoring to not be available in all cases
Product: [Other] RHQ Project Reporter: Lukas Krejci <lkrejci>
Component: PluginsAssignee: Lukas Krejci <lkrejci>
Status: CLOSED CURRENTRELEASE QA Contact: Mike Foley <mfoley>
Severity: high Docs Contact:
Priority: urgent    
Version: 3.0.1CC: hrupp, loleary, skondkar
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 694476
: 702690 769956 (view as bug list) Environment:
Last Closed: 2013-09-02 07:14:10 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 694476, 700461    
Bug Blocks: 702690, 703637, 753904, 753929    

Description Lukas Krejci 2011-04-28 19:56:08 UTC 
This is a tracking bug for the port of the fix for 694476 over to RHQ 3.0.x branch.

+++ This bug was initially created as a clone of Bug #694476 +++

Description of problem:
The apache plugin doesn't take into account the conditional aspects of the configuration.

For example a VirtualHost that is nested inside an IfModule or IfDefine directive isn't discovered even if it is actually "in effect".

Note that nesting definitions inside conditional directives is a common practice even if VirtualHosts are not that usual to be nested (but it's not a rare occurence to see such configuration).

This makes the inventory not reflect the true state of the runtime configuration of the apache server.

Version-Release number of selected component (if applicable):
all

How reproducible:
always

Steps to Reproduce:
1. edit httpd.conf and nest a virtual host definition inside an ifmodule that checks for a module that IS being loaded
2. start the apache server
3. discover it using RHQ
  
Actual results:
the nested vhost isn't discovered

Expected results:
the nested vhost should be discovered if it is in effect.

Additional info:

This obviously is not limited to just vhosts but any directives.

The problem with IfDefine is that the values it uses are passed to the apache executable as parameters. Are we able to read those with our process scan?

--- Additional comment from lkrejci on 2011-04-08 09:20:50 EDT ---

Note that if we successfully implemented this, we could get rid of the imperfect matching rules between what's in the configuration files and what SNMP provides.

SNMP reports the vhost hostnames according to the values specified in the ServerName directives. But in the following (legal and realistic) example, the SNMP reports the same hostname and port for 2 different vhosts, which makes our current matching techniques unusable:


<VirtualHost _default_:80>
    DocumentRoot /var/www/common
    ServerName public-hostname.com:80
</VirtualHost>

<VirtualHost 10.1.1.1:80>
    DocumentRoot /var/www/special
    ServerName public-hostname.com:80
</VirtualHost>

The usecase for the above configuration is that the actual server is hidden behind a load-balancer / internet-facing server that relays the requests to our instance. Our instance is configured to report back the hostname of that 'front server'. But the front server has some rules out of our control to direct "special class" of requests to the "special" vhost, while all other will go to the common one.

Because we currently cannot assume that the vhosts discovered by the plugin are the same (because we leave out the conditionally defined ones) as the actual ones reported by SNMP (i.e. the runtime configuration), we cannot distinguish which of the 2 discovered vhosts corresponds to which in the SNMP report, because there are 2 records in the SNMP report with the same hostname and port.

If we successfully managed to parse out the actual runtime configuration, we could match the discovered vhosts and the SNMP records merely by their indices.

I think detecting the runtime configuration is doable if we watch out for several gotchas:

1) IfDefine can theoretically refer to -D variables hardcoded at compile time to the apache binary (but we should be able to detect those using the utilities we already have in the plugin code)

2) Names used in IfModule are potentially different to the ones in corresponding LoadModule directives but this is rare. We could solve this by having a map in the plugin configuration to cover for the exceptional cases which we wouldn't handle by default.

3) some modules are compiled in - the list of these can be retrieved by executing "httpd -l"

--- Additional comment from lkrejci on 2011-04-13 11:55:52 EDT ---

In the commit below I actually implemented 2 things at once. I updated the name of this bug to take that into account.

commit 24dd3ebadfca65618caad06ab45887a09d0f3972
Author: Lukas Krejci <lkrejci>
Date:   Wed Apr 13 17:50:24 2011 +0200

    BZ 694476 - We now correctly determine the runtime configuration of the apache server
    by taking into account the list of loaded modules and -D options passed to it on commandline.
    
    With this in place, it is possible to deterministically match the discovered vhost resources
    with the vhosts from the list returned by the SNMP plugin merely knowing the ordering in which
    the SNMP plugin returns the list.
    
    This commit contains both of the above fixes.

--- Additional comment from lkrejci on 2011-04-14 09:06:25 EDT ---

commit 5e6b9a0580eb8373a33e5054b184f4d91a674681
Author: Lukas Krejci <lkrejci>
Date:   Thu Apr 14 14:59:50 2011 +0200

    BZ 694476 - the user is now only expected to list the non-standard module mappings instead of all modules in the plugin config. This enables a clear and simple upgrade path from prev
    
    Also made the loaded module detection more robust.

--- Additional comment from lkrejci on 2011-04-20 09:53:10 EDT ---

commit c8f7388581b8d3c00c17ecf4732f77481c83a86e
Author: Lukas Krejci <lkrejci>
Date:   Wed Apr 20 15:01:33 2011 +0200

    BZ 694476 - Adding support for the IfVersion directive in detecting the runtime configuration.

--- Additional comment from lkrejci on 2011-04-28 14:27:50 EDT ---

This bug depends on bug 700461 because the apache plugin depends on up-to-date native process info being available so that it can determine the correct runtime configuration.
Comment 1 Lukas Krejci 2011-05-06 15:11:16 UTC 
Fixed by the following commits in release-3.0.1 branch:

a48c68385ba2fbd226fa62e52b944c2fbcf43515
069d5fc2d0e10278277223dd2c04639caa25e3a4
116411a3e10a5a2e5840d620a14326f5ca9974ce
3e93c6fb5d7dc6b2ab1522d97b845872afbfe1df
bd816f6c926b932a2b02349f46b7a00a79c57766
a2c00c6f1da359c6fbd51e391fb4422ada03ac9d
2632a30b0bc37eaabbac4e82afc8388411cb34dd
7d71991e8df59f75bdfe5c9c2929f8d27956b05c
258be003ddf7bf022611ef183ed60a7e7ee7ef18

Apart from a number of internal changes to support extracting the runtime configuration of the apache instance, there are 4 user-facing changes:

1) Virtual hosts inside IfModule, IfDefine or IfVersion directives (or combination thereof) are discovered as long as they are active in the current runtime configuration of the apache instance

2) The "SNMP WWW Service Index" property in the connection settings of the virtual hosts was re-introduced (it used to exist in read-only form in RHQ 1.x but was removed in RHQ 3.0.(0|1)). It is now writeable by the user so that it is possible to assign correct values to the RHQ3-discovered resources without re-inventorying them.

3) The Apache server resource has a new operation called "Detect SNMP WWW Service Index values for Virtual Hosts". This outputs a mapping from virtual host resource key to a corresponding SNMP WWW Service Index value as discovered by the plugin. This is to aid the users in assigning those values to individual inventoried vhost resources (the plugin is only able to assign the value during the discovery, an already inventoried resource has to be edited manually).

4) Reading or updating the configuration of virtual hosts that contain additional If* directives is not supported (the previous version of the plugin would be oblivious about the configuration contained inside the conditional directives and could therefore break the configuration altogether).

The SNMP WWW Service Index value for virtual hosts is dependent on the position of the virtual host in the configuration files. Therefore whenever a new vhost is added to the configuration of the apache instance (either manually or using RHQ), the values for all vhosts should be checked if they are still correct. If the new vhost is added in such a place in the configuration files that it is inserted "in the middle of" the pre-existing vhosts in the runtime configuration, the SNMP WWW Service Index values will have to be manually updated for all vhosts that end up *before* that vhost (this might seem paradoxical, but is a consequence of the fact that apache internally stores the vhosts in the reverse order they are defined in (with exception of the "main server" vhost that is always first)).

The runtime configuration of an apache instance is a union of all configuration files filtered by conditional If* directives. The union is produced by replacing each "Include" directive by the contents of the files it specifies (if there is more than 1 file included using a single directive, the files are included in the alphabetical order).

Note that the connection property and operation is only present in the RHQ 3 codebase. RHQ4 (bug 694476) doesn't need these because the value of the SNMP WWW Service Index is determined and automatically updated internally at runtime and no user intervention is needed.
Comment 2 Larry O'Leary 2011-05-06 16:08:54 UTC 
Patch isn't being picked up by JON 2.4.1 server. It is throwing the following error/exception:

2011-05-06 11:40:04,000 ERROR [org.rhq.enterprise.server.core.plugin.ProductPluginDeployer] Failed to register RHQ plugin file [file:/opt/jboss/jon/jon-server-2.4.1.GA/jbossas/server/default/deploy/rhq.ear/rhq-downloads/rhq-plugins/rhq-apache-plugin-3.0.1.GA.jar]
javax.ejb.EJBException: java.lang.NullPointerException
	at org.jboss.ejb3.tx.Ejb3TxPolicy.handleExceptionInOurTx(Ejb3TxPolicy.java:63)
	at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:83)
	at org.jboss.aspects.tx.TxInterceptor$RequiresNew.invoke(TxInterceptor.java:253)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:95)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
	at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:110)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:240)
	at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:210)
	at org.jboss.ejb3.stateless.StatelessLocalProxy.invoke(StatelessLocalProxy.java:84)
	at $Proxy364.registerPluginInNewTransaction(Unknown Source)
	at org.rhq.enterprise.server.resource.metadata.ResourceMetadataManagerBean.registerPlugin(ResourceMetadataManagerBean.java:299)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
	at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
	at org.rhq.enterprise.server.common.TransactionInterruptInterceptor.addCheckedActionToTransactionManager(TransactionInterruptInterceptor.java:77)
	at sun.reflect.GeneratedMethodAccessor110.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:118)
	at org.rhq.enterprise.server.authz.RequiredPermissionsInterceptor.checkRequiredPermissions(RequiredPermissionsInterceptor.java:156)
	at sun.reflect.GeneratedMethodAccessor109.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:118)
	at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79)
	at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:95)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
	at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:110)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:240)
	at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:210)
	at org.jboss.ejb3.stateless.StatelessLocalProxy.invoke(StatelessLocalProxy.java:84)
	at $Proxy364.registerPlugin(Unknown Source)
	at org.rhq.enterprise.server.core.plugin.ProductPluginDeployer.registerPluginJar(ProductPluginDeployer.java:504)
	at org.rhq.enterprise.server.core.plugin.ProductPluginDeployer.access$000(ProductPluginDeployer.java:56)
	at org.rhq.enterprise.server.core.plugin.ProductPluginDeployer$LatchedPluginDeploymentService.executeService(ProductPluginDeployer.java:613)
	at org.rhq.enterprise.server.core.concurrency.LatchedServiceController$LatchedService.run(LatchedServiceController.java:266)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.NullPointerException
	at org.rhq.enterprise.server.resource.metadata.ResourceMetadataManagerBean.updateResourcePluginConfiguration(ResourceMetadataManagerBean.java:978)
	at org.rhq.enterprise.server.resource.metadata.ResourceMetadataManagerBean.updatePluginConfiguration(ResourceMetadataManagerBean.java:955)
	at org.rhq.enterprise.server.resource.metadata.ResourceMetadataManagerBean.mergeExistingType(ResourceMetadataManagerBean.java:675)
	at org.rhq.enterprise.server.resource.metadata.ResourceMetadataManagerBean.updateType(ResourceMetadataManagerBean.java:657)
	at org.rhq.enterprise.server.resource.metadata.ResourceMetadataManagerBean.updateTypes(ResourceMetadataManagerBean.java:458)
	at org.rhq.enterprise.server.resource.metadata.ResourceMetadataManagerBean.updateTypes(ResourceMetadataManagerBean.java:463)
	at org.rhq.enterprise.server.resource.metadata.ResourceMetadataManagerBean.registerPluginInNewTransaction(ResourceMetadataManagerBean.java:353)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
	at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
	at org.rhq.enterprise.server.common.TransactionInterruptInterceptor.addCheckedActionToTransactionManager(TransactionInterruptInterceptor.java:77)
	at sun.reflect.GeneratedMethodAccessor110.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:118)
	at org.rhq.enterprise.server.authz.RequiredPermissionsInterceptor.checkRequiredPermissions(RequiredPermissionsInterceptor.java:156)
	at sun.reflect.GeneratedMethodAccessor109.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:118)
	at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79)
	... 68 more
Comment 3 Lukas Krejci 2011-05-09 20:23:45 UTC 
commit 5d9ebbfb39f859099048e24e7c23bc4e7cbe4ef1
Author: Lukas Krejci <lkrejci>
Date:   Mon May 9 20:37:04 2011 +0200

    BZ 700616 - using the legacy code to create the virtual host resource keys to ensure the backwards compatibility with previous versions of the plugin.

commit 34371638426c8d83444f9aa1e2e5ec622c157543
Author: Lukas Krejci <lkrejci>
Date:   Mon May 9 12:57:31 2011 +0200

    BZ 700616 - adding a default value to the new snmpWwwServiceIndex property so that updating the plugin actually works when there are some inventoried vhosts already on the server.
Comment 4 Lukas Krejci 2011-05-10 16:21:49 UTC 
Additional commits that bring release-3.0.1 branch in sync with the code that will go into the master branch:

commit 5e55fb232d2496961477debd9d5eaf94837d69d4
Author: Lukas Krejci <lkrejci>
Date:   Tue May 10 16:58:28 2011 +0200

    BZ 700616 - make sure to call getVersion() instead of resourceContext.getVersion() directly. This is just to workaround the possible null version being returned from the resourceContext in rare circumstances.

commit dc92e3b118d14e0aadaf2a2143b566f79662d6d9
Author: Lukas Krejci <lkrejci>
Date:   Tue May 10 16:57:15 2011 +0200

    BZ 700616 - ported over the code from master to handle the user provided module name <-> source file mappings for apache modules.
Comment 5 Lukas Krejci 2011-05-19 09:29:41 UTC 
Porting the final bugfix from bug 694476 to release-3.0.1 branch:

commit 434616bdc1fa49b90a03b54d6f1fed7bf9227c2d
Author: Lukas Krejci <lkrejci>
Date:   Wed May 18 13:55:51 2011 +0200

    BZ 694476 - fixing the handling of loaded modules.
Comment 6 Sunil Kondkar 2011-06-29 08:49:03 UTC 
Verified in build#51(release 301 branch)

Verified all the Apache tests as mentioned in the bug#694476).

Marking as verified.
Comment 7 Heiko W. Rupp 2013-09-02 07:14:10 UTC 
Bulk closing of issues that were VERIFIED, had no target release and where the status changed more than a year ago.