[AGENT++] Race condition in AgentX subagent processing

[Jens Engel]

Hello Frank,

APPLIES-TO: Agent++ 3.5.23, AgentX++ 1.4.12b, SNMP++ 3.2.17 (2005-05-17
snapshot)
EFFECTS: AgentX subagent

The newest changes in the request processing synchronization seem to have
caused a RACE-CONDITION.


DESCRIPTION:
------------------------------------------------------------------------

Deletion of a AgentX request object occurs in 2 places and threads:

  1. In AgentXRequestList::receive() delete PredecessorRequestor when next
phase request is received (Thread:1 EventLoop).
  2. In SubAgentXMib::finalize(): (Thread:2)
     Calls AgentXRequestList::answer(req), then
SubAgentXMib::delete_request(req)

Sometimes the processing in Thread:2 is interrupted after or just before
end of AgentXRequestList::answer(req).

Thread:1 begins to run again and receives the next phase in a SetRequest.
This causes the Predecessor Request object to be deleted.

Thread:2 runs again and SubAgentXMib::delete_request(req) is called with a
dangling pointer (vtable pointer is sometimes OUT-OF-SCOPE).
The "delete req" may cause a SEGMENTATION-FAULT.

NOTES:
SEGMENTATION-FAULT occurs either in
  * SubAgentXMib::delete_request() or
  * AgentXRequestList::add_request() [new request uses memory of old
one???]

SUGGESTION:
It is probably better to move the delete-request functionality of
SubAgentXMib::delete_request()
to AgentXRequestList::answer() [synchronized, atomic].


HOW-TO-REPEAT:
-----------------------------------

SNMP client pseudo code:

    mibObject="sonetMediumLoopbackConfig"
    for n in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20;
    do
        for value in 1 2 3 4 5
        do
            snmpset ${mibObject}.10 = ${value}
            current_value =`snmpget ${mibObject}.10`
            assert current_value == value
        done
    done

NOTE:
It is sometimes necessary to run the script from 2 TTYs.


EXAMPLE LOGFILE (my format, with my debug tracepoints):
---------------------------------------------------------------
2005-06-06 18:30:34.361 WARN                    inspect:
AgentXRequestList::answer: pdu agentx.commitSet(type=9, transId=77630]
req=0x1bc6ecd8 in thread=2

2005-06-06 18:30:34.401 ERROR                   agentpp: AgentXRequestList:
request received (context)(tid)(pid)(siz)(type)(err)(status): (), (77630),
(77633), (0), (11), (0), (0)
2005-06-06 18:30:34.411 WARN                    inspect:
AgentXRequestList::receive agentx.cleanupSet(type=11, requestId=77630,
size=0):
2005-06-06 18:30:34.422 WARN                    inspect:
AgentXRequestList::receive[pdu=agentx.cleanupSet,
transId=77630].beforeDelete predecessor req(0x1bc6ecd8,
pdu=agentx.commitSet) in thread=1
2005-06-06 18:30:34.433 WARN                    inspect:
AgentXRequest::Dtor[this=0x1bc6ecd8, requestId=77630] called in thread=1
2005-06-06 18:30:34.445 WARN                    inspect:
AgentXRequestList::receive[pdu=agentx.cleanupSet,
transId=77630].afterDelete predecessor req in thread=1

2005-06-06 18:30:34.456 WARN                    inspect:
AgentXRequestList::answer[pdu=agentx.commitSet, transId=77630].afterSending
response in thread=2
2005-06-06 18:30:34.467 WARN                    inspect:
SubAgentXMib::delete_request[77630] req=0x1bc6ecd8 called in thread=2
2005-06-06 18:30:34.533 FATAL                   inspect: INSPECT-FAIL:
SubAgentXMib::delete_request.pre: avanguard(0xdeaddead) is not ALIVE.
2005-06-06 18:30:34.543 FATAL                   inspect: INSPECT-FAIL:
SubAgentXMib::delete_request.pre: readguard(0xdeaddead) is not ALIVE.
2005-06-06 18:30:34.594 FATAL               application: SEGV: SEGMENTATION
FAULT: Abort here.


Ciao,
Jens Engel