Bug 911216

Summary: Notifier doesn't start because it can't find glassfish-jaxb/jaxb-impl.jar
Product: Red Hat Enterprise Virtualization Manager Reporter: Juan Hernández <juan.hernandez>
Component: ovirt-engine-notification-serviceAssignee: Juan Hernández <juan.hernandez>
Status: CLOSED CURRENTRELEASE QA Contact: Ilanit Stein <istein>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 3.2.0CC: acathrow, dyasny, iheim, istein, Rhev-m-bugs, rnori, ykaul, yzaslavs
Target Milestone: ---   
Target Release: 3.2.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: infra
Fixed In Version: sf10 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-06-11 09:20:09 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Infra RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 915537, 922807    

Description Juan Hernández 2013-02-14 15:32:58 UTC
Description of problem:

The notifier fails during startup because it can't find the glassfish-jaxb/jaxb-impl.jar file.


Version-Release number of selected component (if applicable):

Branh rhev-3.2, git hash 72ab0f.

How reproducible:

Always.


Steps to Reproduce:

Run /etc/rc.d/init.d/engine-notifier.d start.

  
Actual results:

Error: can't run without missing JAR file: /usr/share/java/glassfish-jaxb/jaxb-impl.jar


Expected results:

The notifier should start without errors.

Additional info:

Comment 1 Juan Hernández 2013-02-14 15:34:53 UTC
This is a leftover from the rebase from upstream. I am redoing the changes required downstream.

Comment 2 Yair Zaslavsky 2013-02-17 07:31:24 UTC
*** Bug 911143 has been marked as a duplicate of this bug. ***

Comment 4 Ilanit Stein 2013-02-28 08:06:30 UTC
Tested on sf-8:

1. engine-notifierd service is blocked from starting if $MAIL_SERVER is not defined. After defining it service is started:

Error: $MAIL_SERVER is not defined, please check for this in configuration file /etc/ovirt-engine/notifier/notifier.conf

2. When running start engine-notifierd service, for each line in /etc/ovirt-engine/notifier/notifier.conf, a line similar to the following is printed:

"/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `# This file holds the configuration of oVirt events notification daemon.': not a valid identifier"

Full log:
--------
/etc/rc.d/init.d/engine-notifierd is stopped
[root@istein-32 ~]# /etc/rc.d/init.d/engine-notifierd start
Starting engine-notifier: at Wed Feb 27 21:31:24 IST 2013/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `#': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `# This file holds the configuration of oVirt events notification daemon.': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `# The daemon consists of two major services: notification service and engine monitoring.': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `#': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `#-------------------------------------#': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `# Notification Service Configuration: #': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `#-------------------------------------#': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `# Interval (in seconds) between iterations of dispatching messages to subscribers. Default is 120 seconds.': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `#INTERVAL_IN_SECONDS=120': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `# The SMTP mail server address. Required.': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `# The default port of non-secured SMTP server is 25, for secured (ssl enabled) 465.': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `#MAIL_PORT=25': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `# Required if SSL enabled to authenticate the user. Used also to specify 'from' user address if mail server': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `# supports, when MAIL_FROM is not set. Address is in RFC822 format': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `#MAIL_USER=': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `# Required to authenticate the user if mail server requires authentication or if SSL is enabled': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `#MAIL_PASSWORD=': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `# Indicates whether SSL should be used to communicate with mail server. Default is false.': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `#MAIL_ENABLE_SSL=true': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `# If set to true, sends a message in HTML format. Default is false.': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `#HTML_MESSAGE_FORMAT=true': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `# Specifies 'from' address on sent mail in RFC822 format, if supported by mail server.': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `#MAIL_FROM=': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `# Specifies 'reply-to' address on sent mail in RFC822 format.': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `#MAIL_REPLY_TO=': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `# Amount of days to keep dispatched events on history table. If not set, events remain on history table.': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `#DAYS_TO_KEEP_HISTORY=30': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `# This parameter specifies how many days of old events are processed and sent': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `# when the notifier starts. If set to 2, for example, the notifier will': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `# process and send the events of the last two days, older events will just': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `# be marked as processed and won't be sent. The default is 0, so no old': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `# messages will be sent at all during startup.': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `#DAYS_TO_SEND_ON_STARTUP=0': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `#----------------------------------#': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `# Engine Monitoring Configuration: #': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `#----------------------------------#': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `# Interval (in seconds) between engine server monitoring iterations. Interval is being measured from': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `# the time an iteration is completed. Default is 300 seconds.': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `#ENGINE_INTERVAL_IN_SECONDS=300': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `# Number of retries to monitor server status per iteration. Default is 3 retries.': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `#ENGINE_MONITOR_RETRIES=3': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `# Time (in seconds) to wait between retries. Default is 30 seconds.': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `#ENGINE_TIMEOUT_IN_SECONDS=30': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `# If running Jboss in secure mode, should be set to true. Default is false.': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `#IS_HTTPS_PROTOCOL=true': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `# Specifies the protocol used by Jboss Configuration Connector when SSL is enabled. Default is 'TLS'.': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `#SSL_PROTOCOL=TLS': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `# If running Jboss in secure mode, and wishes to ignore SSL errors, should set to true. Default is false.': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `#SSL_IGNORE_CERTIFICATE_ERRORS=false': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `# If running Jboss in secure mode, and wishes to ignore hostname verification, should set to true. Default is false.': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `# If setting this property to 'true', SSL_IGNORE_CERTIFICATE_ERRORS considered to be set to 'true' as well: disabling host name': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `# verification means that it will not be verified against the certification, therefore certification errors are ignored.': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `#SSL_IGNORE_HOST_VERIFICATION=false': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `# Specifies whether to repeat auditing of failure messages of non-responding engine server. Default is false (meaning': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `# repeated failure messages will NOT be sent to the subscribers)': not a valid identifier
/usr/share/ovirt-engine/bin/engine-notifier.sh: line 68: declare: `#REPEAT_NON_RESPONSIVE_NOTIFICATION=false:': not a valid identifier

Error: $MAIL_SERVER is not defined, please check for this in configuration file /etc/ovirt-engine/notifier/notifier.conf
                                                           [FAILED]
[root@istein-32 ~]# /etc/rc.d/init.d/engine-notifierd status
/etc/rc.d/init.d/engine-notifierd is stopped

[root@istein-32 ~]# rpm -q rhevm
rhevm-3.2.0-9.el6ev.noarch

Comment 5 Juan Hernández 2013-02-28 11:21:48 UTC
If the MAIL_SERVER variable is defined then the notifier will start and work correctly, even if it prints all these error messages.

These error messages are a side effect of the change introduced to fix bug 883301.

I will try to avoid these messages.

Comment 6 Juan Hernández 2013-02-28 14:49:53 UTC
In order to solve this issue we need first to correct a syntax error that we currently have in the notifier.sh script (only downstream):

https://gerrit.eng.lab.tlv.redhat.com/6763

And then we need to change the mechanism that we use to check if lines in notifier.conf are empty or comments (upstream and downstream):

http://gerrit.ovirt.org/12528

Comment 9 Ilanit Stein 2013-03-13 10:13:22 UTC
Verified on sf-10

Comment 10 Ilanit Stein 2013-03-13 12:02:54 UTC
On a clean installed sf-10, where in /etc/ovirt-engine/notifier/notifier.conf, MAIL_SERVER parameter looks like this:

MAIL_SERVER=
The notifier service is not started, though it looks like it did:

root@istein-32 notifier]# /etc/rc.d/init.d/engine-notifierd start
Starting engine-notifier: at Wed Mar 13 13:27:14 IST 2013  [  OK  ]
[root@istein-32 notifier]# /etc/rc.d/init.d/engine-notifierd status
Notifier service is not running for pid 15182

notifier.log:

2013-03-13 12:11:29,839 INFO  [org.ovirt.engine.core.notifier.utils.NotificationConfigurator] Starting event notification service with configuration file: /etc/ovirt-engine/notifier/notifier.conf
2013-03-13 12:11:29,844 ERROR [org.ovirt.engine.core.notifier.Notifier] Failed to run the event notification service.
java.lang.IllegalArgumentException: Check configuration file, /etc/ovirt-engine/notifier/notifier.conf MAIL_SERVER is missing
        at org.ovirt.engine.core.notifier.Notifier.main(Notifier.java:55)
2013-03-13 13:27:14,676 INFO  [org.ovirt.engine.core.notifier.utils.NotificationConfigurator] Starting event notification service with configuration file: /etc/ovirt-engine/notifier/notifier.conf
2013-03-13 13:27:14,681 ERROR [org.ovirt.engine.core.notifier.Notifier] Failed to run the event notification service.
java.lang.IllegalArgumentException: Check configuration file, /etc/ovirt-engine/notifier/notifier.conf MAIL_SERVER is missing
        at org.ovirt.engine.core.notifier.Notifier.main(Notifier.java:55)

Comment 11 Yair Zaslavsky 2013-03-14 06:23:23 UTC
This is a separate issue.
I recommend you open a different bug on that.
Please also attach the configuration file of the notification service.

Comment 12 Ilanit Stein 2013-03-14 06:59:37 UTC
Marking this bug as verified. Another bug opened for comment 10: Bug 921405.

Comment 13 Itamar Heim 2013-06-11 09:20:09 UTC
3.2 has been released

Comment 14 Itamar Heim 2013-06-11 09:43:22 UTC
3.2 has been released