[AGENT++] SNMP v3 get performance

[John McCaskey]

Everyone,

Using the patch I previously provided for per session v3MP objects I have my
polling engine running and stable using SNMP++ for all of our SNMP
requests.  I have 80 threads that get spawned, each create's it's own v3MP
object, and each checks a global queue for devices that need SNMP polling.
Each thread then once it has a device sets up a new snmp session, which
includes doing a usm->add_usm_user() call if the device is using v3, and
then issues SNMP get requests on groups of OIDs until it has gathered all
the data needed for the device.

This works well, but the performance when polling a large number of v3
devices is pretty bad.  For v1 I can poll several thousand devices in a
matter of a minute or two and performance is quite good on par with Net-SNMP
which we used previously.  However, if I have just a couple hundred v3
devices the performance starts to crawl.  Under my linux build I've been
using oprofile to gather some performance data and it seems that the bulk of
the time is being spent in the MD5 routines, I'm testing with a security
level of authPriv using MD5/DES.  Here is the output from oprofile:

CPU: P4 / Xeon with 2 hyper-threads, speed 2592.35 MHz (estimated)
Counted GLOBAL_POWER_EVENTS events (time during which processor is not
stopped) with a unit mask of 0x01 (mandatory) count 100000
samples % linenr info symbol name
9306146 64.4420 md5c.cpp:220 MD5Transform
3414196 23.6422 auth_priv.cpp:1040 AuthMD5::password_to_key(unsigned char
const*, unsigned int, unsigned char const*, un
signed int, unsigned char*, unsigned int*)
821898 5.6914 md5c.cpp:332 Decode

I'm interested in any suggestions on cutting down this time.  I know Jochen
already mentioned that I shouldn't keep calling add_usm_user if the user has
already been added previously, but given that each device may have any
user/password combination and I have no way of knowing whether a device may
use the same username as another, but have a different password I don't see
how I can avoid that.  It seems like maybe what I need is for the table of
added users inside snmp++ to be a little smarter and not lookup by just
username but also by password so that whenever I add it will reuse if an
exact match exists otherwise add the user again and perform the MD5
computations again.  Does that seem like the best approach to improving
performance?

John McCaskey
Klir Technologies, Inc.