[SNMP4J] Performance issue
Frank Fock
fock at agentpp.com
Tue Jun 4 23:31:33 CEST 2013
Hi Jeff,
I would expect the best result with asynchronous request handling and
a MultiThreadedMessageDispatcher. The latter is important, if you
response processing blocks (= takes some time).
While the receiving thread provides the response to your code, it cannot
process further messages to other listeners. Thus, a multi-thread dispatcher
can decouple this and increase the throughput.
Hope this helps.
Best regards,
Frank
Am 04.06.2013 23:09, schrieb Jeff Pople:
>
> Hi Frank,
>
> I'm struggling with a performance issue when querying large numbers of devices from a single process.
>
>
> If, for example, I simultaneously issue 20 separate 'get' requests to each of 100 different addresses and repeatedly do that on a half minute cycle, I see the CPU usage of my process plateau at a high level. If I increase the number of devices, I don't see much more of an increase in CPU but the time taken to fulfill each request increases on a roughly linear scale in relation to the number of devices/requests.
>
>
> I have tried using more than one Snmp instance with (as I expected) no obvious benefit and I have tried asynchronous requests (where I can reduce CPU load by introducing a wait for a response but this comes at the expense of overall response times for a given set of 'gets').
>
>
> Is it that ultimately all the responses to my many 'get' requests must arrive at a single port (161) on my network interface and that the Snmp reading thread can only process them so fast? Is there something I can do to increase throughput short of bulking up my queries?
>
> Any help appreciated.
>
> Thanks
>
> Jp
>
>
>
>
>
> ________________________________
> From: "snmp4j-request at agentpp.org" <snmp4j-request at agentpp.org>
> To: snmp4j at agentpp.org
> Sent: Monday, 3 June 2013, 20:43
> Subject: SNMP4J Digest, Vol 111, Issue 3
>
>
> Send SNMP4J mailing list submissions to
> snmp4j at agentpp.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://lists.agentpp.org/mailman/listinfo/snmp4j
> or, via email, send a message with subject or body 'help' to
> snmp4j-request at agentpp.org
>
> You can reach the person managing the list at
> snmp4j-owner at agentpp.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of SNMP4J digest..."
>
>
> Today's Topics:
>
> 1. Re: Performance hotspot in Cipher.getInstance() method call
> (Praveen Jain)
> 2. Re: V3 Not In Time Window and Client Clock Drift (Elise Atkins)
> 3. Re: V3 Not In Time Window and Client Clock Drift (Frank Fock)
> 4. Re: V3 Not In Time Window and Client Clock Drift (Elise Atkins)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Mon, 3 Jun 2013 18:24:19 +0530
> From: "Praveen Jain" <praveen.jain at esq.com>
> To: "'Frank Fock'" <fock at agentpp.com>, <snmp4j at agentpp.org>
> Subject: Re: [SNMP4J] Performance hotspot in Cipher.getInstance()
> method call
> Message-ID: <000501ce6059$79227420$6b675c60$@esq.com>
> Content-Type: text/plain; charset="us-ascii"
>
> Hi Frank,
>
> Thanks a lot for quick reply. I would really appreciate if you backport the
> fix into 1.x branch. Please let me know if there is a place where I can
> track that (any issue tracking system). In case the backporting idea is
> dropped for 1.x branch, please provide a patch so that I can apply the same
> on source code. Thanks.
>
> Regards,
> Praveen.
>
> -----Original Message-----
> From: snmp4j-bounces at agentpp.org [mailto:snmp4j-bounces at agentpp.org] On
> Behalf Of Frank Fock
> Sent: Monday, June 03, 2013 2:59 AM
> To: snmp4j at agentpp.org
> Subject: Re: [SNMP4J] Performance hotspot in Cipher.getInstance() method
> call
>
> Hi Praveen,
>
> SNMP4J is not caching the cipher object. Apparently, the performance
> overhead of retrieving a Cipher instance is not that great with newer Java
> versions.
> Nevertheless, I am currently working on a caching solution, which does not
> introduce a new bootleneek for multi-threaded usage for SNMP4J 2.x.
>
> If it will possible, I will also provide a backport to the 1.x branch.
>
> Best regards,
> Frank
>
>
> Am 30.05.2013 13:03, schrieb Praveen Jain:
>> We are using SNMP4J version 1.11.4 for a legacy application running in
>> Java
>> 1.4 environment. The application behaved slower than expected and
>> profiling showed that there is hotspot in PrivDES.encrypt() method.
>> This method in turn calls javax.crypto.Cipher.getInstance(String)
>> method (apparently on each message transmission) and
>> Cipher.getInstance(String) is the expensive and CPU intensive call. Can
> you please suggest if:
>> 1. This might be due wrong way of calling SNMP4J api (and also point
>> to a reference code for correct usage to prevent this problem)
>>
>> 2. Is this an inherent problem with Java 1.4 code?
>>
>> 3. Is this a problem with SNMP 1.11.4 code (it is not caching the
>> obtained cipher object)?
>>
>> Thanks in advance for the help.
>>
>>
>>
>> -Praveen
>>
>> _______________________________________________
>> SNMP4J mailing list
>> SNMP4J at agentpp.org
>> http://lists.agentpp.org/mailman/listinfo/snmp4j
> --
> ---
> AGENT++
> Maximilian-Kolbe-Str. 10
> 73257 Koengen, Germany
> https://agentpp.com
> Phone: +49 7024 8688230
> Fax: +49 7024 8688231
>
> _______________________________________________
> SNMP4J mailing list
> SNMP4J at agentpp.org
> http://lists.agentpp.org/mailman/listinfo/snmp4j
>
>
>
> ------------------------------
>
> Message: 2
> Date: Mon, 03 Jun 2013 11:52:14 -0400
> From: Elise Atkins <eatkins at tavve.com>
> To: Frank Fock <fock at agentpp.com>, snmp4j at agentpp.org
> Cc: chuckc <chuck at tavve.com>
> Subject: Re: [SNMP4J] V3 Not In Time Window and Client Clock Drift
> Message-ID: <51ACBBAE.8010105 at tavve.com>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> Frank,
>
> After looking at RFC 3414, I have a different interpretation. In the
> case 1) the RFC states that the local notion of the value of the
> snmpEngineTime is to be updated to the msgAuthoritativeEngineTime. This
> should mean that the response is in time regardless of how slow or fast
> the authoritative engine's time is running as long as it moves in the
> forward direction when the time check is done in 2).
>
> b) If the extracted value of msgAuthoritativeEngineID is not the
> same as the value snmpEngineID of the processing SNMP engine
> (meaning this is not the authoritative SNMP engine), then:
>
> 1) if at least one of the following conditions is true:
>
> - the extracted value of the msgAuthoritativeEngineBoots
> field is greater than the local notion of the value of
> snmpEngineBoots; or,
>
> - the extracted value of the msgAuthoritativeEngineBoots
> field is equal to the local notion of the value of
> snmpEngineBoots, and the extracted value of
> msgAuthoritativeEngineTime field is greater than the
> value of latestReceivedEngineTime,
>
> then the LCD entry corresponding to the extracted value of
> the msgAuthoritativeEngineID field is updated, by setting:
>
> - the local notion of the value of snmpEngineBoots to the
> value of the msgAuthoritativeEngineBoots field,*
> *
> *- the local notion of the value of snmpEngineTime to the
> value of the msgAuthoritativeEngineTime field, and*
>
> - the latestReceivedEngineTime to the value of the value of
> the msgAuthoritativeEngineTime field
>
> 2) if any of the following conditions is true, then the
> message is considered to be outside of the Time Window:
>
> - the local notion of the value of snmpEngineBoots is
> 2147483647;
>
> - the value of the msgAuthoritativeEngineBoots field is
> less than the local notion of the value of
> snmpEngineBoots; or,
>
> - the value of the msgAuthoritativeEngineBoots field is
> equal to the local notion of the value of
> snmpEngineBoots and the value of the
> msgAuthoritativeEngineTime field is more than 150
> seconds less than the local notion of the value of
> snmpEngineTime.
>
> From the above, I think if the reboots are the same and the
> msgAuthoritativeEngineTime is greater then the
> latestReceivedTimeEngineTime, the latestReceivedEngineTime should be
> updated. At that point if you fall down into 2) then the since the
>
> From release 2.1.2 UsmTimeTable starting at line 174
> if ((entry.getEngineBoots() < time.getEngineBoots()) ||
> ((entry.getEngineBoots() == time.getEngineBoots()) &&
> (time.getTimeDiff() + now >
> entry.getLatestReceivedTime() + 150)) ||
> (time.getEngineBoots() == 2147483647)) {
> if (logger.isDebugEnabled()) {
> logger.debug(
> "CheckTime: received message outside time window (non
> authoritative)");
> }
> return SnmpConstants.SNMPv3_USM_NOT_IN_TIME_WINDOW;
> }
> else {
> if ((entry.getEngineBoots() > time.getEngineBoots()) ||
> ((entry.getEngineBoots() == time.getEngineBoots()) &&
> (entry.getLatestReceivedTime() >
> time.getLatestReceivedTime()))) {
> /* time ok, update values */
> time.setEngineBoots(entry.getEngineBoots());
> time.setLatestReceivedTime(entry.getLatestReceivedTime());
> time.setTimeDiff(entry.getLatestReceivedTime() - now);
> }
>
> Frank Fock wrote:
>> Hi Elise,
>>
>> I would like to give you an update on the issue below.
>> I have checked the implementation, your findings, and the current RFC
>> 3414
>> and found out, that RFC 3414 precised the description of the incoming
>> message handling (section 3.2.7 pages 27-28).
>>
>> SNMP4J follows that description and is therefore conforming.
>> No changes will be applied.
>>
>> Could you please verify this on your side, too?
>>
>> Best regards,
>> Frank
>>
>> Am 23.05.2013 17:52, schrieb Elise Atkins:
>>> We have been successfully using snmp4j using v3 for both requests and
>>> traps but have recently run into a problem with a client whose engine
>>> time clock runs slow.
>>>
>>> Initial time synchronization goes ok and the SNMP v3 requests and
>>> responses flow properly. Eventually this client's engine time
>>> becomes more 150 seconds behind the time SNMP4j is expecting and the
>>> responses are marked as Not In Time. I have looked at RFC 2574
>>> Section 3.2 subsection 7b and the code in the UsmTimeTable class,
>>> checkTime method and have questions about the order of testing for
>>> timeliness.
>>>
>>> Based on the RFC, I would expect the code to test for the conditions
>>> in 1 first and update if needed before testing the conditions in 2
>>> but the code seems to test in the reverse order. Testing 1 and
>>> updating reboots and time first will allow the client's clock to
>>> drift (fast or slow) as long as it is always increasing and remain in
>>> the time window. Am I missing something here?
>>>
>>> Elise Atkins
>>>
>>> The RFC says:
>>>
>>> 3.2. Processing an Incoming SNMP Message
>>> 7) If the securityLevel indicates an authenticated message, then
>>> the local values of snmpEngineBoots, snmpEngineTime
>>> and latestReceivedEngineTime
>>> corresponding to the value of the msgAuthoritativeEngineID
>>> field are extracted from the Local Configuration Datastore.
>>>
>>> b) If the extracted value of msgAuthoritativeEngineID is not the
>>> same as the value snmpEngineID of the processing SNMP engine
>>> (meaning this is not the authoritative SNMP engine), then:
>>>
>>> 1) if at least one of the following conditions is true:
>>>
>>> - the extracted value of the msgAuthoritativeEngineBoots
>>> field is greater than the local notion of the value of
>>> snmpEngineBoots; or,
>>>
>>> - the extracted value of the msgAuthoritativeEngineBoots
>>> field is equal to the local notion of the value of
>>> snmpEngineBoots, and the extracted value of
>>> msgAuthoritativeEngineTime field is greater than the
>>> value of latestReceivedEngineTime,
>>>
>>> then the LCD entry corresponding to the extracted value
>>> of the msgAuthoritativeEngineID field is updated, by
>>> setting:
>>>
>>> - the local notion of the value of snmpEngineBoots to
>>> the value of the msgAuthoritativeEngineBoots field,
>>> - the local notion of the value of snmpEngineTime to
>>> the value of the msgAuthoritativeEngineTime field,
>>> and
>>> - the latestReceivedEngineTime to the value of the
>>> value of the msgAuthoritativeEngineTime field.
>>>
>>> 2) if any of the following conditions is true, then the
>>> message is considered to be outside of the Time Window:
>>>
>>> - the local notion of the value of snmpEngineBoots is
>>> 2147483647;
>>>
>>> - the value of the msgAuthoritativeEngineBoots field is
>>> less than the local notion of the value of
>>> snmpEngineBoots; or,
>>>
>>> - the value of the msgAuthoritativeEngineBoots field is
>>> equal to the local notion of the value of
>>> snmpEngineBoots and the value of the
>>> msgAuthoritativeEngineTime field is more than 150
>>> seconds less than the local notion of the value of
>>> snmpEngineTime.
>>>
>>> If the message is considered to be outside of the Time
>>> Window then an error indication (notInTimeWindow) is
>>> returned to the calling module.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> SNMP4J mailing list
>>> SNMP4J at agentpp.org
>>> http://lists.agentpp.org/mailman/listinfo/snmp4j
>
>
> ------------------------------
>
> Message: 3
> Date: Mon, 03 Jun 2013 21:17:57 +0200
> From: Frank Fock <fock at agentpp.com>
> To: elise.atkins at tavve.com
> Cc: chuckc <chuck at tavve.com>, Elise Atkins <eatkins at tavve.com>,
> snmp4j at agentpp.org
> Subject: Re: [SNMP4J] V3 Not In Time Window and Client Clock Drift
> Message-ID: <51ACEBE5.7070802 at agentpp.com>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> Elise,
>
> OK, I am convinced. I reservley checked the implementation with RFC3414 and
> overseen your initial point about the order of checks.
>
> I will provide a fix for it (ticket ID is SFJ-75).
>
> The SNMP4J 2.2.2 release (with that fix) is then planned for next week.
>
> Best regards,
> Frank
>
>
> Am 03.06.2013 17:52, schrieb Elise Atkins:
>> Frank,
>>
>> After looking at RFC 3414, I have a different interpretation. In the
>> case 1) the RFC states that the local notion of the value of the
>> snmpEngineTime is to be updated to the msgAuthoritativeEngineTime.
>> This should mean that the response is in time regardless of how slow
>> or fast the authoritative engine's time is running as long as it moves
>> in the forward direction when the time check is done in 2).
>>
>> b) If the extracted value of msgAuthoritativeEngineID is not the
>> same as the value snmpEngineID of the processing SNMP engine
>> (meaning this is not the authoritative SNMP engine), then:
>>
>> 1) if at least one of the following conditions is true:
>>
>> - the extracted value of the msgAuthoritativeEngineBoots
>> field is greater than the local notion of the value of
>> snmpEngineBoots; or,
>>
>> - the extracted value of the msgAuthoritativeEngineBoots
>> field is equal to the local notion of the value of
>> snmpEngineBoots, and the extracted value of
>> msgAuthoritativeEngineTime field is greater than the
>> value of latestReceivedEngineTime,
>>
>> then the LCD entry corresponding to the extracted value of
>> the msgAuthoritativeEngineID field is updated, by setting:
>>
>> - the local notion of the value of snmpEngineBoots to the
>> value of the msgAuthoritativeEngineBoots field,*
>> *
>> *- the local notion of the value of snmpEngineTime to the
>> value of the msgAuthoritativeEngineTime field, and*
>>
>> - the latestReceivedEngineTime to the value of the value of
>> the msgAuthoritativeEngineTime field
>>
>> 2) if any of the following conditions is true, then the
>> message is considered to be outside of the Time Window:
>>
>> - the local notion of the value of snmpEngineBoots is
>> 2147483647;
>>
>> - the value of the msgAuthoritativeEngineBoots field is
>> less than the local notion of the value of
>> snmpEngineBoots; or,
>>
>> - the value of the msgAuthoritativeEngineBoots field is
>> equal to the local notion of the value of
>> snmpEngineBoots and the value of the
>> msgAuthoritativeEngineTime field is more than 150
>> seconds less than the local notion of the value of
>> snmpEngineTime.
>>
>> From the above, I think if the reboots are the same and the
>> msgAuthoritativeEngineTime is greater then the
>> latestReceivedTimeEngineTime, the latestReceivedEngineTime should be
>> updated. At that point if you fall down into 2) then the since the
>>
>> From release 2.1.2 UsmTimeTable starting at line 174
>> if ((entry.getEngineBoots() < time.getEngineBoots()) ||
>> ((entry.getEngineBoots() == time.getEngineBoots()) &&
>> (time.getTimeDiff() + now >
>> entry.getLatestReceivedTime() + 150)) ||
>> (time.getEngineBoots() == 2147483647)) {
>> if (logger.isDebugEnabled()) {
>> logger.debug(
>> "CheckTime: received message outside time window (non
>> authoritative)");
>> }
>> return SnmpConstants.SNMPv3_USM_NOT_IN_TIME_WINDOW;
>> }
>> else {
>> if ((entry.getEngineBoots() > time.getEngineBoots()) ||
>> ((entry.getEngineBoots() == time.getEngineBoots()) &&
>> (entry.getLatestReceivedTime() >
>> time.getLatestReceivedTime()))) {
>> /* time ok, update values */
>> time.setEngineBoots(entry.getEngineBoots());
>> time.setLatestReceivedTime(entry.getLatestReceivedTime());
>> time.setTimeDiff(entry.getLatestReceivedTime() - now);
>> }
>>
>> Frank Fock wrote:
>>> Hi Elise,
>>>
>>> I would like to give you an update on the issue below.
>>> I have checked the implementation, your findings, and the current RFC
>>> 3414
>>> and found out, that RFC 3414 precised the description of the incoming
>>> message handling (section 3.2.7 pages 27-28).
>>>
>>> SNMP4J follows that description and is therefore conforming.
>>> No changes will be applied.
>>>
>>> Could you please verify this on your side, too?
>>>
>>> Best regards,
>>> Frank
>>>
>>> Am 23.05.2013 17:52, schrieb Elise Atkins:
>>>> We have been successfully using snmp4j using v3 for both requests
>>>> and traps but have recently run into a problem with a client whose
>>>> engine time clock runs slow.
>>>>
>>>> Initial time synchronization goes ok and the SNMP v3 requests and
>>>> responses flow properly. Eventually this client's engine time
>>>> becomes more 150 seconds behind the time SNMP4j is expecting and the
>>>> responses are marked as Not In Time. I have looked at RFC 2574
>>>> Section 3.2 subsection 7b and the code in the UsmTimeTable class,
>>>> checkTime method and have questions about the order of testing for
>>>> timeliness.
>>>>
>>>> Based on the RFC, I would expect the code to test for the conditions
>>>> in 1 first and update if needed before testing the conditions in 2
>>>> but the code seems to test in the reverse order. Testing 1 and
>>>> updating reboots and time first will allow the client's clock to
>>>> drift (fast or slow) as long as it is always increasing and remain
>>>> in the time window. Am I missing something here?
>>>>
>>>> Elise Atkins
>>>>
>>>> The RFC says:
>>>>
>>>> 3.2. Processing an Incoming SNMP Message
>>>> 7) If the securityLevel indicates an authenticated message, then
>>>> the local values of snmpEngineBoots, snmpEngineTime
>>>> and latestReceivedEngineTime
>>>> corresponding to the value of the msgAuthoritativeEngineID
>>>> field are extracted from the Local Configuration Datastore.
>>>>
>>>> b) If the extracted value of msgAuthoritativeEngineID is not the
>>>> same as the value snmpEngineID of the processing SNMP engine
>>>> (meaning this is not the authoritative SNMP engine), then:
>>>>
>>>> 1) if at least one of the following conditions is true:
>>>>
>>>> - the extracted value of the msgAuthoritativeEngineBoots
>>>> field is greater than the local notion of the value of
>>>> snmpEngineBoots; or,
>>>>
>>>> - the extracted value of the msgAuthoritativeEngineBoots
>>>> field is equal to the local notion of the value of
>>>> snmpEngineBoots, and the extracted value of
>>>> msgAuthoritativeEngineTime field is greater than the
>>>> value of latestReceivedEngineTime,
>>>>
>>>> then the LCD entry corresponding to the extracted value
>>>> of the msgAuthoritativeEngineID field is updated, by
>>>> setting:
>>>>
>>>> - the local notion of the value of snmpEngineBoots to
>>>> the value of the msgAuthoritativeEngineBoots field,
>>>> - the local notion of the value of snmpEngineTime to
>>>> the value of the msgAuthoritativeEngineTime field,
>>>> and
>>>> - the latestReceivedEngineTime to the value of the
>>>> value of the msgAuthoritativeEngineTime field.
>>>>
>>>> 2) if any of the following conditions is true, then the
>>>> message is considered to be outside of the Time Window:
>>>>
>>>> - the local notion of the value of snmpEngineBoots is
>>>> 2147483647;
>>>>
>>>> - the value of the msgAuthoritativeEngineBoots field is
>>>> less than the local notion of the value of
>>>> snmpEngineBoots; or,
>>>>
>>>> - the value of the msgAuthoritativeEngineBoots field is
>>>> equal to the local notion of the value of
>>>> snmpEngineBoots and the value of the
>>>> msgAuthoritativeEngineTime field is more than 150
>>>> seconds less than the local notion of the value of
>>>> snmpEngineTime.
>>>>
>>>> If the message is considered to be outside of the Time
>>>> Window then an error indication (notInTimeWindow) is
>>>> returned to the calling module.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> SNMP4J mailing list
>>>> SNMP4J at agentpp.org
>>>> http://lists.agentpp.org/mailman/listinfo/snmp4j
--
---
AGENT++
Maximilian-Kolbe-Str. 10
73257 Koengen, Germany
https://agentpp.com
Phone: +49 7024 8688230
Fax: +49 7024 8688231
More information about the SNMP4J
mailing list