[AGENT++] Another bug in asn1.cpp on 64-bit Solaris platform - Counter64 values are wrong

[Alex Agranov]

I found another bug in asn1.cpp code on 64-bit Solaris platform that is caused by the wrong assumption that "sizeof(long) == 4" - while on this specific platform it's 8.

The problem causes wrong values for VarBinds of type Counter64 to be returned in the SNMP-RESPONSE (typically 0 is returned instead of any value).

The fix that I applied is pretty straightforward - I simply replaced "sizeof(long)" with "sizeof(int)" inside the asn_build_unsigned_int64() function (because "sizeof(int)==4" both for 32-bit and 64-bit Linux and Solaris - as per https://www.ibm.com/developerworks/linux/library/l-solar/).
However I wonder whether "sizeof()" is needed here at all - 'cause ASN seems to clearly define the size of its elements...

Anyway, I verified that the fix works fine on Linux IA-32 (RHEL 5) and Solaris 64-bit platforms.

Regards,
    Alex

*** asn1.cpp    Tue May 19 11:18:54 2009
--- asn1.orig   Wed Feb  4 15:53:48 2009

***************
*** 1014,1020 ****
    intsize = 8;
    low = cp->low;
    high = cp->high;
!   mask = 0xFFul << (8 * (sizeof(int) - 1));
    /* mask is 0xFF000000 on a big-endian machine */
    if ((unsigned char)((high & mask) >> (8 * (sizeof(long) - 1))) & 0x80){
      /* if MSB is set */
--- 1013,1019 ----
    intsize = 8;
    low = cp->low;
    high = cp->high;
!   mask = 0xFFul << (8 * (sizeof(long) - 1));
    /* mask is 0xFF000000 on a big-endian machine */
    if ((unsigned char)((high & mask) >> (8 * (sizeof(long) - 1))) & 0x80){
      /* if MSB is set */
***************
*** 1028,1040 ****
       * There should be no sequence of 9 consecutive 1's or 0's at the most
       * significant end of the integer.
       */
!     mask2 = 0x1FFul << ((8 * (sizeof(int) - 1)) - 1);
      /* mask2 is 0xFF800000 on a big-endian machine */
      while((((high & mask2) == 0) || ((high & mask2) == mask2))
          && intsize > 1){
        intsize--;
        high = (high << 8)
!       | ((low & mask) >> (8 * (sizeof(int) - 1)));
        low <<= 8;
      }
    }
--- 1027,1039 ----
       * There should be no sequence of 9 consecutive 1's or 0's at the most
       * significant end of the integer.
       */
!     mask2 = 0x1FFul << ((8 * (sizeof(long) - 1)) - 1);
      /* mask2 is 0xFF800000 on a big-endian machine */
      while((((high & mask2) == 0) || ((high & mask2) == mask2))
          && intsize > 1){
        intsize--;
        high = (high << 8)
!       | ((low & mask) >> (8 * (sizeof(long) - 1)));
        low <<= 8;
      }
    }
***************
*** 1049,1057 ****
      intsize--;
    }
    while(intsize--){
!     *data++ = (unsigned char)((high & mask) >> (8 * (sizeof(int) - 1)));
      high = (high << 8)
!       | ((low & mask) >> (8 * (sizeof(int) - 1)));
      low <<= 8;

    }
--- 1048,1056 ----
      intsize--;
    }
    while(intsize--){
!     *data++ = (unsigned char)((high & mask) >> (8 * (sizeof(long) - 1)));
      high = (high << 8)
!       | ((low & mask) >> (8 * (sizeof(long) - 1)));
      low <<= 8;

    }


This email and any files transmitted with it are confidential material. They are intended solely for the use of the designated individual or entity to whom they are addressed. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, use, distribution or copying of this communication is strictly prohibited and may be unlawful.

If you have received this email in error please immediately notify the sender and delete or destroy any copy of this message