Bug 1145680

Summary: Syslog handler for audit log throws NPE after syslog server restart (UDP protocol)
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: Petr Kremensky <pkremens>
Component: Domain ManagementAssignee: James Perkins <jperkins>
Status: CLOSED CURRENTRELEASE QA Contact: Petr Kremensky <pkremens>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 6.4.0CC: brian.stansberry, dandread, jperkins, kkhan
Target Milestone: DR4   
Target Release: EAP 6.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-08-19 12:43:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
dummy syslog server none

Description Petr Kremensky 2014-09-23 13:39:50 UTC
Created attachment 940442 [details]
dummy syslog server

Description of problem:
 JBAS013408: Update of the management operation audit log failed in handler 'mysyslog': java.lang.NullPointerException is thrown once I restart the syslog server (UDP) and run some mgmt operation.

Version-Release number of selected component (if applicable):
 EAP 6.4.0.DR1.1

How reproducible:
 Always

Steps to Reproduce:
Use dummy UDP syslog server from attachment which listens on port 9998 and print messages to the output
 1. Start the dummy syslog server and clean EAP 6.4.0 in standalone mode
java org.jboss.qe.syslog.UDPSyslogListener
./standalone.sh 

 2. Configure a audit syslog handler 
./jboss-cli.sh -c
/core-service=management/access=audit/logger=audit-log:write-attribute(name=enabled,value=true)
batch
/core-service=management/access=audit/syslog-handler=mysyslog:add(formatter=json-formatter)
/core-service=management/access=audit/syslog-handler=mysyslog/protocol=udp:add(host=localhost,port=9998)
run-batch
/core-service=management/access=audit/logger=audit-log/handler=mysyslog:add()

# You should see that last operation was already received by dummy syslog server

 3. kill the dummy syslog server process and run some management operation
kill -9 `jps | grep UDPSyslogListener | awk '{ print $1 }'`
./jboss-cli.sh -c '/core-service=management/management-interface=http-interface:write-attribute(name=console-enabled, value=true)'

 4. start the syslog server again and run some other operation
java org.jboss.qe.syslog.UDPSyslogListener
./jboss-cli.sh -c '/core-service=management/management-interface=http-interface:write-attribute(name=console-enabled, value=true)'

Actual results:
15:22:41,791 ERROR [org.jboss.as.controller.management-operation] (management-handler-thread - 4) JBAS013408: Update of the management operation audit log failed in handler 'mysyslog': java.lang.NullPointerException
	at org.jboss.as.controller.audit.SyslogAuditLogHandler$TransportErrorManager.throwAsIoOrRuntimeException(SyslogAuditLogHandler.java:474) [jboss-as-controller-7.5.0.Final-redhat-2.jar:7.5.0.Final-redhat-2]
	at org.jboss.as.controller.audit.SyslogAuditLogHandler$TransportErrorManager.getAndThrowError(SyslogAuditLogHandler.java:468) [jboss-as-controller-7.5.0.Final-redhat-2.jar:7.5.0.Final-redhat-2]
	at org.jboss.as.controller.audit.SyslogAuditLogHandler.writeLogItem(SyslogAuditLogHandler.java:292) [jboss-as-controller-7.5.0.Final-redhat-2.jar:7.5.0.Final-redhat-2]
	at org.jboss.as.controller.audit.AuditLogHandler.writeLogItem(AuditLogHandler.java:81) [jboss-as-controller-7.5.0.Final-redhat-2.jar:7.5.0.Final-redhat-2]

Expected results:
 - No NPE (NPE is thrown every time (6.4.0) doing these operations on running server: start syslog, run mgmt op, stop syslog, run mgmt op, start syslog, run mgmt op -> NPE, stop syslog, ...)
 - message should be logged if the syslog server is ready

Additional info:
Doing same with EAP 6.3.0.GA gives me:
LogManager error of type WRITE_FAILURE: Could not write to syslog
java.net.PortUnreachableException: ICMP Port Unreachable
	at java.net.PlainDatagramSocketImpl.send(Native Method)
...
but only once after EAP start.

Comment 1 Petr Kremensky 2014-09-24 10:42:52 UTC
The NPE breaks also failure-count statistics once UDP protocol is used so handler is never automatically disabled.

Block the 9998 port for TCP protocol by:

ServerSocket listener = new ServerSocket(8080);
listener.accept();

Now add the UDP syslog handler (use configuration from description). 
Use following commands:
[1] mgmt operation: /core-service=management/management-interface=http-interface:write-attribute(name=console-enabled, value=true)
[2] syslog stats: ls -l /core-service=management/access=audit/syslog-handler=mysyslog

Every first invocation of management operation sets failure-count to 0 so syslog handler is never disabled due to max-failure-count
Every second invocation triggers NPE and set failure-count to 1

Comment 2 Petr Kremensky 2014-09-24 10:43:25 UTC
Correction:
ServerSocket listener = new ServerSocket(8080);
-->>
ServerSocket listener = new ServerSocket(9998);

Comment 3 Petr Kremensky 2014-10-09 12:41:24 UTC
Fix introduced a new warning message if the UDP syslog server is down. No failure-count statistic for UDP. Handler is able to reconnect immediately once the syslog server is up again.

Verified on EAP 6.4.0.DR4.

Comment 4 JBoss JIRA Server 2015-04-28 15:03:39 UTC
John Doyle <jdoyle> updated the status of jira EAP6-230 to Closed