“SEND”ing POSTEMSG to CA Service Operations Insight

So I recently uncovered a bug in our systems monitoring that I thought I would share. We recently migrated from IBM Tivoli to CA’s Service Operation Insight for our Enterprise monitoring solution.

This LOB (Line of Business) is a small one, and as we are building it as an autonomous business unit I decided to use the Virtual Center Server Appliance (vCSA 5.5). In doing so I was deviating from our normal Windows-centric approach to things. This meant that our trusty Windows batch files would not be able to be used and instead I took advantage of this to refine the logic into a streamlined BASH script.

OK enough of this foreplay, lets get to the meat of this. For those of you that don’t use it, postemsg is an app that will post an event to an event server. In our case that is now CA’s Service Operations Insight. We generally fire it as a result of a particular vCenter Alarm Action. This allows all of the vCenter Alarm info to be sent over to our Systems/Network Monitoring Team for triage.

The syntax of postemsg is very basic:

postemsg -S server | -f configuration file [-m message] [-r severity] [attribute=value...] class source
 

With that I added the following to my BASH script to send to SOI:

if [[ $VMWARE_ALARM_NEWSTATUS == "Yellow" ]]
then
SEVERITY="CRITICAL"
elif [[ $VMWARE_ALARM_NEWSTATUS == "Red" ]]
then
SEVERITY="DOWN"
fi

EXE="/usr/local/scripts/postemsg"
FOLDER="/usr/local/scripts"

...

$EXE -f ${FOLDER}/post.conf -r $SEVERITY -m "SEND TICKET TO Virt_Team $VMWARE_ALARM_EVENTDESCRIPTION" hostname=$VMWARE_ALARM_TARGET_NAME Scr_VM_Event lobvmprod

Simple enough right? I thought so too, that is until a member of the Systems/Network Monitoring Team contacted me to tell me that the Alert was not being received properly. He sent me the following image for reference.

CA_Bad_sanitized

The summary field only shows an “S”. After numerous iterations and testing the following has been found to be true:

If you use the keyword “END” anywhere in the string, all following characters are ignored. You have got to be kidding me.

With some further testing I thought I should just try the “Send” and was able to produce the following desired result.

CA_Good_sanitized

Success! For some reason SOI just treats “END” as just that, the END of the string.

This folks is what we call an epic FAIL.

I hope this helps someone as this was a thorn in the side of our Monitoring team for quite some time.

Leave a Reply

Your email address will not be published. Required fields are marked *

*