Debug assertion failed: (unsigned)(c+1) <= 256, file isctype.c, line 68

I have compiled a debug version of Thunderbird using VC++ Express. I intermittently get the error in the Subject. When I kick into the debugger, the stack trace seems to indicate that the range check is actually failing in the _security_check_cookie() routine. Here's the stack trace:

msvcr80d.dll!1023c92d()
[Frames below may be incorrect and/or missing, no symbols loaded for msvcr80d.dll]
msvcr80d.dll!1023cd81()
> xpcom_core.dll!nsCRT::strtok(char * string=0x040db607, const char * delims=0x0371f398, char * * newStr=0x0371f398) Line 172 + 0xa bytes C++

Here's where the application code got to - it is just trying to return from the end of the function, and the VC++ compiler is adding in the check_cookie stuff:

return str == result NULL : result;
002B9B6F mov eax,dword ptr [ebp-30h]
002B9B72 sub eax,dword ptr [ebp-2Ch]
002B9B75 neg eax
002B9B77 sbb eax,eax
002B9B79 and eax,dword ptr [ebp-2Ch]
}
002B9B7C mov ecx,dword ptr [ebp-8]
002B9B7F xor ecx,ebp
002B9B81 call __security_check_cookie (328650h)
002B9B86 mov esp,ebp
002B9B88 pop ebp
002B9B89 ret

If you push down into the check_cookie(), you get into the two levels of the CRT debug library (msvcr80d.dll), and the range check seems to be failing somewhere in there. (No source code, of course, so can't tell what it is trying to do).

Is there a problem in the VC++ Express CRT libs Here's the version I'm running:

Microsoft Visual Studio 2005
Version 8.0.50727.42 (RTM.050727-4200)
Microsoft .NET Framework
Version 2.0.50727

Installed Edition: VC Express

Microsoft Visual C++ 2005 76542-000-0000011-00125

Answer this question

Debug assertion failed: (unsigned)(c+1) <= 256, file isctype.c, line 68

Roggan

Seams that you are overwritting the stack with a function.

Show us some source code.
ermnet
Turns out the problem was already found and fixed.

https://bugzilla.mozilla.org/show_bug.cgi id=251882

mel001212100121

Here's the source. It's from the mozilla codebase.

The entire codebase is browsable at http://lxr.mozilla.org/mozilla1.8.0/source/

131 ////////////////////////////////////////////////////////////////////////////////
132 // My lovely strtok routine
133 
134 #define IS_DELIM(m, c)     ((m)[(c) >> 3] & (1 << ((c) & 7)))
135 #define SET_DELIM(m, c)     ((m)[(c) >> 3] |= (1 << ((c) & 7)))
136 #define DELIM_TABLE_SIZE    32
137 
138 char* nsCRT::strtok(char* string, const char* delims, char* *newStr)
139 {
140  NS_ASSERTION(string, "Unlike regular strtok, the first argument cannot be null.");
141 
142  char delimTable[DELIM_TABLE_SIZE];
143  PRUint32 i;
144  char* result;
145  char* str = string;
146 
147  for (i = 0; i < DELIM_TABLE_SIZE; i++)
148   delimTable[i] = '\0';
149 
150  for (i = 0; delims[i]; i++) {
151   SET_DELIM(delimTable, NS_STATIC_CAST(PRUint8, delims[i]));
152  }
153  NS_ASSERTION(delims[i] == '\0', "too many delimiters");
154 
155  // skip to beginning
156  while (*str && IS_DELIM(delimTable, NS_STATIC_CAST(PRUint8, *str))) {
157   str++;
158  }
159  result = str;
160 
161  // fix up the end of the token
162  while (*str) {
163   if (IS_DELIM(delimTable, NS_STATIC_CAST(PRUint8, *str))) {
164    *str++ = '\0';
165    break;
166   }
167   str++;
168  }
169  *newStr = str;
170 
171  return str == result  NULL : result;
172 }

crillion
I am not sure, but I think that a char is signed. So if c contains a german Umlaut or another char >0x80 it is interpreted as a negative number. ((m)[(c) >> 3] will use than a negative index! This corrupts the stack!
Debug assertion failed: (unsigned)(c+1) <= 256, file isctype.c, line 68

Answer this question