[SNMP4J] problem when running snmp walk for a long time

Sat Feb 2 03:03:16 CET 2008

Try 6 threads collecting 10 devices each. If it were me I would start
with 3x20 and move up from there. Also follow up on Frank's suggestion
to use asynchronous retrieval in which case all 60 devices could run in
one thread. Unfortunately I am not familiar with this myself or I would
offer a more concrete suggestion.

Also I have had good luck using the Eclipse IDE ( www.eclipse.org ) and
importing the SNMP4J projects from the Ant build file. I'm not familiar
with NetBeans so perhaps it already has a comprehensive capability of
this sort. The code is extremely well organized and once you get the
hang of it will be extremely easy to leverage.

Gene

YUAN Shuai wrote:
> Hi, Eugene
> Thanks for your idea.
> I know the concept of a thread pool, however, how can I use a thread
> pool to generate threads collecting data from nearly 60 devices at the
> specified same time? As the project requests that the data should be
> collected at almost the same time (only little variance allowed, and
> that is not avoidable), an async method is probably not acceptable,
> which may be the basic feature of a thread pool.
> I appreciate the idea of -Xss and I shall try it out, although the
> memory is no longer a problem (according to our test, the whole
> program will only cost less than 30MB, so turely the problem of
> Netbean is).
> :) Thank you again.
> 2008-02-01
> YUAN Shuai
> ------------------------------------------------------------------------
> *发件人：* Eugene R. Snider
> *发送时间：* 2008-01-31 11:20:39
> *收件人：* YUAN Shuai
> *抄送：* Frank Fock; snmp4j
> *主题：* Re: [SNMP4J] problem when running snmp walk for a long time
> I tend to agree with Frank's concerns about performance. Also I think
> you are running under a mistaken assumption, having 60 threads each
> performing a single task is not necessarily more concurrent, and may
> even be less concurrent, than fewer threads polling more devices. The
> context overhead of a thread switch is smaller than a process context
> switch and at best you are probably spending signficant time ( which you
> will not be able to account for ) in meaningless system activity. At
> worst you are thrashing and are actually impeding performance rather
> than enhancing it. While linux supports 10K+ threads without blinking
> (more or less and actually less) Java, all marketing crap aside, doesn't
> like them in the double digit counts.
> It is possible to reduce the amount of memory allocated for each thread
> with the -Xss command line switch, that might improve your memory
> allocation and possibly allow you to work around this specific problem.
> Gene
> YUAN Shuai wrote:
> > Hi,
> >
> > Thank you for your help.
> > Actually I've got no choice but create 60 threads at the same time,
> > because I have to collect data at every 0 and 30 second from all the
> devices.
> > The reason for doing this is to study the relationship of the
> devices according to the performance data.
> > So you see, I hava to split nearly 400 requests into about 60 jobs,
> each one containing 6 requests.
> > A distributed system will be a good choice, however I haven't gone
> that far :)
> >
> > And maybe this is important, finally I found the PROBLEM, Netbean's
> fault, actually.
> > When I packed the project and run it under Linux from 'java -jar',
> the program just never goes error again.
> >
> >
> >
> >
> >
> > 发件人： Frank Fock
> > 发送时间： 2008-01-26 04:15:15
> > 收件人： YUAN Shuai
> > 抄送： snmp4j
> > 主题： Re: [SNMP4J] problem when running snmp walk for a long time
> > Hi,
> > You should not create a Snmp session for each
> > request. The overhead is too big especially
> > if you are sending requests concurrently.
> > Running 60 threads for performing 60 concurrently
> > walks is a bad idea either! A thread pool with
> > as many threads as there are CPUs in the host
> > will provide best performance. You need such
> > a thread pool for processing the responses only.
> > Thus, using a multi-threaded message dispatcher
> > is all you need.
> > Always have in mind, that a thread needs about
> > one MByte of memory!
> > Best regards,
> > Frank
> > YUAN Shuai wrote:
> >
> >> hi all, here is the situation:
> >>
> >> In each thread, I put a snmp walk request in a while-loop, so that
> the walk can be done at every 0 and 30 second.
> >>
> >> Nearly 60 threads are created, and after running for about 3 hours,
> I got the following error and my other running applications just got
> errors too such as 'dll error' :
> >>
> >> Exception in thread "Thread-34" java.lang.OutOfMemoryError: unable
> to create new native thread
> >> at java.lang.Thread.start0(Native Method)
> >> at java.lang.Thread.start(Thread.java:597)
> >> at java.util.Timer.<init>(Timer.java:154)
> >> at java.util.Timer.<init>(Timer.java:122)
> >> at org.snmp4j.Snmp.<init>(Unknown Source)
> >> at org.snmp4j.Snmp.<init>(Unknown Source)
> >> at snmpacc.SNMPAccWalk.createSnmpSession(SNMPAccWalk.java:80)
> >> at snmpacc.SNMPAccWalk.walkLeaf(SNMPAccWalk.java:106)
> >> at scout.PERFWalker.run(PERFWalker.java:80)
> >> at java.lang.Thread.run(Thread.java:619)
> >>
> >> and
> >>
> >> java.net.SocketException: No buffer space available (maximum
> connections reached?): Socket creation failed
> >> at java.net.PlainDatagramSocketImpl.datagramSocketCreate(Native Method)
> >> at
> java.net.PlainDatagramSocketImpl.create(PlainDatagramSocketImpl.java:73)
> >> at java.net.DatagramSocket.createImpl(DatagramSocket.java:317)
> >> at java.net.DatagramSocket.<init>(DatagramSocket.java:208)
> >> at java.net.DatagramSocket.<init>(DatagramSocket.java:261)
> >> at java.net.DatagramSocket.<init>(DatagramSocket.java:234)
> >> at org.snmp4j.transport.DefaultUdpTransportMapping.<init>(Unknown
> Source)
> >> at snmpacc.SNMPAccWalk.createSnmpSession(SNMPAccWalk.java:79)
> >> at snmpacc.SNMPAccWalk.walkLeaf(SNMPAccWalk.java:106)
> >> at scout.PERFWalker.run(PERFWalker.java:80)
> >> at java.lang.Thread.run(Thread.java:619)
> >>
> >> However I am sure to invoke the close() method, and the following
> is my walkLeaf():
> >>
> >> public String walkLeaf(String oid) throws IOException {
> >> // System.out.println("walk oid:" + oid);
> >> Address address = new UdpAddress(IPAddress.toString() + "/161");
> >> Snmp snmp = createSnmpSession();
> >> Target target = createTarget();
> >> target.setVersion(getVersion(Version));
> >> target.setAddress(address);
> >> target.setRetries(retries);
> >> target.setTimeout(timeout);
> >> target.setMaxSizeRequestPDU(65535);
> >> snmp.listen();
> >> PDU request = new PDU();
> >> request.setType(PDU.GET);
> >> request.add(new VariableBinding(new OID(oid)));
> >>
> >> PDU response = null;
> >>
> >> ResponseEvent responseEvent = snmp.send(request, target);
> >> if (responseEvent == null) {
> >> return "no response";
> >> }
> >>
> >> // receive the result
> >> response = responseEvent.getResponse();
> >>
> >> // close the connection
> >> snmp.close();
> >> snmp = null;
> >>
> >> if (response == null) {
> >> return "no response";
> >> }
> >>
> >> return new String(response.get(0).getVariable().toString());
> >> }
> >>
> >> anyone get any idea dealing with this? THANK YOU!
> >>
> >> vmape, 2008.01.24
> >> _______________________________________________
> >> SNMP4J mailing list
> >> SNMP4J at agentpp.org
> >> http://lists.agentpp.org/mailman/listinfo/snmp4j
> >>
> >>
> ------------------------------------------------------------------------
> >>
> >> _______________________________________________
> >> SNMP4J mailing list
> >> SNMP4J at agentpp.org
> >> http://lists.agentpp.org/mailman/listinfo/snmp4j
> >>