Fwd: [SNMP4J] Threads are being blocked in snmp4j 1.8 RC1

[Jamie Bisotti]

---------- Forwarded message ----------
From: Jamie Bisotti <jbisotti at gmail.com>
Date: Jan 22, 2007 2:55 PM
Subject: Re: [SNMP4J] Threads are being blocked in snmp4j 1.8 RC1
To: Frank Fock <fock at agentpp.com>



On 1/22/07, Frank Fock <fock at agentpp.com> wrote:
>
> Hi Varma,
>
> For what reason are you using 250 threads??? That's about
> 250 MB memory footprint for the thread stack only. Do
> you have a system with 250 CPUs? Otherwise, those threads
> also waste thread context switching CPU cycles too.
>
> OK, back to the locked threads problem: The
> SyncResponseListener is created per request, thus there could
> not be two requests blocking on the same SyncResponseListener
> instance.
>
> The only situation where requests could block forever on
> the SyncResponseListener is when the close() method of the
> Snmp instance has been called *before* the request has
> been sent or if the internal Timer thread died or could not
> be started due to a OutOfMemoryException.
>
> Best regards,
> Frank
>
> varma datla wrote:
> > Hi Frank,
> > I am using the snmp4j 1.8 RC1 library (multi-threaded, synchronous) in a
> tomcat application with 250 threads. Also using 3 retries and 500,1000,5000
> as timeouts per request.
> > After a while, the whole application is blocked and  doing nothing. A
> JConsole lookup revealed that all the threads are waiting on
> org.snmp4j.Snmp$SyncResponseListener class.
> >
> > Any ideas?
> >
> > JConsole  output:
> >
> > Name: pool-1-thread-1
> > State: WAITING on org.snmp4j.Snmp$SyncResponseListener at 676bc7
> > Total blocked: 7,546  Total waited: 127,560
> >
> > Stack trace:
> > java.lang.Object.wait (Native Method)
> > java.lang.Object.wait(Object.java:474)
> > org.snmp4j.Snmp.send(Unknown Source)
> > org.snmp4j.Snmp.send(Unknown Source)
> > ..............
> > .............
> >
> >
> > Thank you and I appreciate any suggestions to overcome this problem.
> >
> > Varma
> >
> >
> > ---------------------------------
> > Now that's room service! Choose from over 150,000 hotels
> > in 45,000 destinations on Yahoo! Travel to find your fit.
> > _______________________________________________
> > SNMP4J mailing list
> > SNMP4J at agentpp.org
> > http://lists.agentpp.org/mailman/listinfo/snmp4j
>
> --
> AGENT++
> http://www.agentpp.com
> http://www.mibexplorer.com
> http://www.mibdesigner.com
>
> _______________________________________________
> SNMP4J mailing list
> SNMP4J at agentpp.org
> http://lists.agentpp.org/mailman/listinfo/snmp4j
>

Frank,

I work with Varma and I'm looking into this problem as well.  We are doing a
network device discovery, and performance is very important (interrogating
65K addresses in 15 minutes is a usecase).  We've played with the size of
our thread pool and 250 seems to be the "sweet spot"; many more or many less
results in a slower discovery.  To answer one of your questions though, in
some cases the code will be executing on a single proc and in some it will
be on multi-procs.

Getting back to the issue at hand, in your last response, you mentioned the
internal timer, so I went back to JConsole and here's what I found:


Name: Timer-1
State: BLOCKED on org.snmp4j.Snmp$SyncResponseListener at 1d040d2 owned by:
pool-1-thread-150
Total blocked: 1,211  Total waited: 329,080

Stack trace:
org.snmp4j.Snmp$SyncResponseListener.onResponse (Unknown Source)
org.snmp4j.Snmp$PendingRequest.run(Unknown Source)
java.util.TimerThread.mainLoop(Timer.java:512)
java.util.TimerThread.run(Timer.java:462)


As you can see, it is being blocked by pool-1-thread-150; so I took a look
at that thread:


Name: pool-1-thread-150
State: BLOCKED on org.snmp4j.Snmp$PendingRequest at 838c6e owned by: Timer-1
Total blocked: 7,622  Total waited: 127,540

Stack trace:
org.snmp4j.Snmp.send(Unknown Source)
org.snmp4j.Snmp.send(Unknown Source)
...
java.util.concurrent.ThreadPoolExecutor$Worker.runTask (
ThreadPoolExecutor.java:650)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java
:675)
java.lang.Thread.run(Thread.java:595)


So, Timer-1 is blocked by pool-1-thread-150, which is blocked by
Timer-1...dead-lock.

Any more ideas?  Thanks.


-- 
Jamie Bisotti


Oops!  Accidentally sent my reply straight to Frank instead of the mailing
list.

-- 
Jamie Bisotti