Recently whilst I was working onsite with a vROPS 6.7 customer helping them to create, tune and set up SNMP notifications we noticed that there was a longer than expected delay in Alerts firing from vROPS. Due to this, tickets in their 3rd party system (the SNMP destination) were  being created after a long time. This vROPS instance has existed for a while and been updated throughout all versions since 6.2.

Issue Description

On closer inspection of the affected objects we could see that the collection cycle was EXACTLY 20 minutes on the responsible vCenter adapter.  This was observed on the objects by picking individual metrics and looking at the collection intervals going back weeks.  As the spacing was 20 minutes exactly I initially suspected the collection interval on the adapter had been manually changed.

This long collection cycle was having a knock on effect with the vROPS alert firing which had an alert cycle set of ‘1’.  All in all the alert was taking 40 minutes to be generated. That is 20 minutes for collection of the metric and a further 20 for the alert to fire.

Checking the vSphere adapter for the particular vCenter using Inventory Explorer  I could see that it was set to the default of 5 minutes. In addition, the objects themselves had no collection settings overriding the default adapter.

 

 

This particular vCenter happened to be on the limit for its own sizing requirements (just over the Max number of Powered VMs for a medium sized deployment  https://kb.vmware.com/s/article/2106572  )  so I began to wonder if it was struggling to return values to vROPS.  This was discounted when I found another vCenter adapter working at exactly 10 minute intervals. In this case the second vCenter was correctly sized.

Resolution

Working with GSS, they helped us track down within vROPS that DEBUG logging had been turned on for a number of logs on the nodes where the adapter was running. Turning these values back to default settings allowed vROPS to start collecting again on the normal collection cycle with the alerts working as expected.  At some point in the past the customer had turned on DEBUG when working on an old issue and it clearly had been left on by mistake.

 

Additional Notes

* Interesting that the vCenter adapters had different behaviours. The most affected was one of 4 vCenter adapters. This one had the greatest number of objects and metrics collected. Perhaps that was a factor.

**  When changing the level of the logs back to INFO we noticed that setting this on one node in the Analytics Cluster automatically set this for all nodes. This is a change in behaviour (confirmed y GSS) from before.  Be careful in a multi node environment if you turn on DEBUG logging believing it to only be effective on one node.