Bug 1515980

Created attachment 1356901 [details]
full snmptrapd.log file

Description of problem
======================

There is either single (or small subset) of MIB structure used in snmp trap
messages tendrl-notifier is sending for alerts.

This seems to be implemented as planned in upstream, looking into snmp
spec[1]:

> design and implementation of a browseable MIB or MIB for Tendrl
> is not currently planned;

> this will allow us to translate the potentially large number of Tendrl
> events and alerts into a very limited number of named element types

> third-party trap handlers may be required to perform some parsing on
> the generalized trap messages (we’ve found precedent for this in, e.g.,
> OpenNMS and Nagios),

SNMP trap message structure is not documented in upstream.

I'm creating this BZ to validate this design wrt downstream requirements.

[1] https://github.com/Tendrl/specifications/issues/185

Version-Release
===============

tendrl-notifier-1.5.4-2.el7rhgs.noarch

How reproducible
================

100 %

Steps to Reproduce
==================

1. Install RHGS WA using tendrl-ansible
2. Configure alerting to send events via snmp
3. Import gluster trusted storage pool with a volume
4. Stop and start again glusterd on some storage done
5. Stop adn start again whole volume
6. Check snmp trap messages you have received, eg:
   journalctl -u snmptrapd > snmptrapd.log

When qe playbook for alerting test setup is used:

https://github.com/usmqe/usmqe-setup/blob/master/test_setup.snmp.yml

one can check incoming snmp trap messages via:

# journalctl -u snmptrapd -fe

Actual results
==============

Checking snmptrapd log file, all alerts uses the same MIB structure:

> 'DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (0) 0:00:00.00        SNMPv2-MIB::snmpTrapOID.0 = OID: SNMPv2-MIB::coldStart        SNMPv2-SMI::private.2312.19.1.0 = STRING:

As can be demonstrated here:

```
$ grep '\[Tendrl Alert\]' snmptrapd.log | wc -l
38
$ grep 'DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (0) 0:00:00.00        SNMPv2-MIB::snmpTrapOID.0 = OID: SNMPv2-MIB::coldStart        SNMPv2-SMI::private.2312.19.1.0 = STRING:' snmptrapd.log | wc -l
38
```

To provide a full context for the example shown above, among these 38
messages, we have only 5 types of alerts:

```
$ grep 'STRING:' snmptrapd.log | sed 's/.*STRING: //' | sed 's/\(INFO\|WARNING\).*//' | sort | uniq                                                                                                                                                                                                                
"[Tendrl Alert] Afr Quorum State,                                                                                                                                                                                                                                                                                            
"[Tendrl Alert] Afr Subvol State,                                                                                                                                                                                                                                                                                            
"[Tendrl Alert] Brick Status,                                                                                                                                                                                                                                                                                                
"[Tendrl Alert] Peer Status,                                                                                                                                                                                                                                                                                                 
"[Tendrl Alert] Volume Status,  
```

Expected results
================

?

Additional info
===============

Options we have is to:

* consider if we need to document structure of snmp trap messages we are
  sending (especially if structure of a trap message is different in some
  cases)
* consider if we need to change the structure a bit, without creating our
  own MIB
* plan development of our own MIB for next version