Pat,
Thanks for your quick response. I've updated the MDAC. We went 2
weeks without a problem before the lastest incident. Below is the
IISState log:
Opened log file 'C:\iisstate\output\IISState-4416.log'
***********************
Starting new log output
IISState version 3.3.1
Sat Aug 21 15:01:06 2004
OS = Windows 2000
Executable: inetinfo.exe
PID = 4416
Note: Thread times are formatted as HH:MM:SS.ms
***********************
Thread ID: 0
System Thread ID: 3b4
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 0006f89c 7c5785d1 ntdll!ZwReadFile+0xb
01 0006f910 7c2e4cd9 KERNEL32!ReadFile+0x181
02 0006f93c 7c2e4b5f ADVAPI32!ScGetPipeInput+0x28
03 0006f9b8 7c2e6632 ADVAPI32!ScDispatcherLoop+0x4a
04 0006fbf4 01002884 ADVAPI32!StartServiceCtrlDispatcherA+0x7d
05 0006fd30 01001e94 inetinfo!StartDispatchTable+0x2f1
06 0006ff70 01002fbf inetinfo!main+0x654
07 0006ffc0 7c581af6 inetinfo!mainCRTStartup+0xff
08 0006fff0 00000000 KERNEL32!BaseProcessStart+0x3d
Thread ID: 1
System Thread ID: 9ec
Kernel Time: 0:0:0.31
User Time: 0:0:0.46
Thread Type: Other
# ChildEBP RetAddr
00 005dfd1c 7c573b28 ntdll!ZwWaitForSingleObject+0xb
01 005dfd44 7c573b50 KERNEL32!WaitForSingleObjectEx+0x71
02 005dfd54 6e6f1685 KERNEL32!WaitForSingleObject+0xf
03 005dfd70 01002440 iisadmin!ServiceEntry+0x156
04 005dffa4 7c2e4e9b inetinfo!InetinfoStartService+0x2bd
05 005dffb4 7c57438b ADVAPI32!ScSvcctrlThreadW+0xe
06 005dffec 00000000 KERNEL32!BaseThreadStart+0x52
Thread ID: 2
System Thread ID: 10dc
Kernel Time: 0:0:0.15
User Time: 0:0:0.15
Thread Type: Other
# ChildEBP RetAddr
00 0071fe5c 7c573c23 ntdll!ZwWaitForMultipleObjects+0xb
01 0071feac 77e119e6 KERNEL32!WaitForMultipleObjectsEx+0xea
02 0071ff08 77e11ace USER32!MsgWaitForMultipleObjectsEx+0x153
03 0071ff24 6e5a5a7c USER32!MsgWaitForMultipleObjects+0x1d
04 0071ff7c 780085bc IisRTL!SchedulerWorkerThread+0xa7
05 0071ffb4 7c57438b MSVCRT!_endthreadex+0xc1
06 0071ffec 00000000 KERNEL32!BaseThreadStart+0x52
Thread ID: 3
System Thread ID: 1134
Kernel Time: 0:0:0.46
User Time: 0:0:0.15
Thread Type: Other
# ChildEBP RetAddr
00 0075fe5c 7c573c23 ntdll!ZwWaitForMultipleObjects+0xb
01 0075feac 77e119e6 KERNEL32!WaitForMultipleObjectsEx+0xea
02 0075ff08 77e11ace USER32!MsgWaitForMultipleObjectsEx+0x153
03 0075ff24 6e5a5a7c USER32!MsgWaitForMultipleObjects+0x1d
04 0075ff7c 780085bc IisRTL!SchedulerWorkerThread+0xa7
05 0075ffb4 7c57438b MSVCRT!_endthreadex+0xc1
06 0075ffec 00000000 KERNEL32!BaseThreadStart+0x52
Thread ID: 4
System Thread ID: 9ac
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Possible ASP page. Possible DCOM activity
Executing Page: ASP.dll symbols not found. Unable to locate ASP page.
Continuing with other analysis.
No remote call being made
# ChildEBP RetAddr
00 00bbfe24 77d37ba7 ntdll!ZwReplyWaitReceivePortEx+0xb
01 00bbff74 77d37b4c RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x74
02 00bbff78 77d35924 RPCRT4!RecvLotsaCallsWrapper+0x9
03 00bbffa8 77d358d6 RPCRT4!BaseCachedThreadRoutine+0x4f
04 00bbffb4 7c57438b RPCRT4!ThreadStartRoutine+0x18
05 00bbffec 00000000 KERNEL32!BaseThreadStart+0x52
Thread ID: 5
System Thread ID: 9a8
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Other information: Thread is waiting for a lock to be released.
Looking for lock owner.
Owning thread System ID: 106c
Thread Type: Other
# ChildEBP RetAddr
00 00bffe78 77f8822a ntdll!ZwWaitForSingleObject+0xb
01 00bffeec 77f8819b ntdll!RtlpWaitForCriticalSection+0x9e
02 00bffef4 77f884e9 ntdll!RtlEnterCriticalSection+0x46
03 00bfff98 7c574072 ntdll!LdrUnloadDll+0x5f
04 00bfffa0 77ab5060 KERNEL32!FreeLibraryAndExitThread+0x1b
05 00bfffb4 7c57438b ole32!CRpcThreadCache::RpcWorkerThreadEntry+0x34
06 00bfffec 00000000 KERNEL32!BaseThreadStart+0x52
Thread ID: 6
System Thread ID: 106c
Kernel Time: 0:0:0.109
User Time: 0:0:0.31
*** WARNING: Unable to verify checksum for
D:\JRun4\lib\wsconfig\5\jrun.dll
*** ERROR: Symbol file could not be found. Defaulted to export
symbols for D:\JRun4\lib\wsconfig\5\jrun.dll -
Thread Type: Other
# ChildEBP RetAddr
00 00e4e9e4 74fd1394 ntdll!ZwWaitForSingleObject+0xb
01 00e4ea20 74fd160d msafd!SockWaitForSingleObject+0x1a8
02 00e4ea9c 750313fa msafd!WSPRecv+0x1e9
03 00e4ead8 750510a4 WS2_32!WSARecv+0x8a
04 00e4eb04 024b903b WSOCK32!recv+0x31
WARNING: Stack unwind information not available. Following frames may
be wrong.
05 00e4ebbc 0250004c jrun_24b0000!HttpFilterProc+0x708b
06 000186a0 00000000 0x250004c
Thread ID: 7
System Thread ID: 11e4
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: HTTP Listener
# ChildEBP RetAddr
00 00f8ff5c 7c573c73 ntdll!ZwRemoveIoCompletion+0xb
01 00f8ff88 6d7029ef KERNEL32!GetQueuedCompletionStatus+0x27
02 00f8ffb4 7c57438b ISATQ!I_AtqOplockThreadFunc+0x32
03 00f8ffec 00000000 KERNEL32!BaseThreadStart+0x52
Thread ID: 8
System Thread ID: 11d8
Kernel Time: 0:0:0.15
User Time: 0:0:0.0
Other information: Thread is waiting for a lock to be released.
Looking for lock owner.
Owning thread System ID: 11dc
Thread Type: HTTP Listener
# ChildEBP RetAddr
00 00fcefbc 77f8822a ntdll!ZwWaitForSingleObject+0xb
01 00fcf030 77f8819b ntdll!RtlpWaitForCriticalSection+0x9e
02 00fcf038 6ca81d42 ntdll!RtlEnterCriticalSection+0x46
03 00fcf0a0 6ca81ee3 lonsint!IISLogonNetUserW+0x91
04 00fcf0f0 769cef98 lonsint!IISLogonNetUserA+0x6a
05 00fcf15c 769ced34 INFOCOMM!ValidateUser+0x10b
06 00fcf6b8 769c1739 INFOCOMM!TsLogonUser+0x313
07 00fcf6ec 65f07611 INFOCOMM!TCP_AUTHENT::ClearTextLogon+0x67
08 00fcf888 65f03977 w3svc!HTTP_REQ_BASE::LogonUserA+0x67e
09 00fcfcac 65f0275a w3svc!HTTP_REQ_BASE::OnRestartRequest+0xa42
0a 00fcfcec 65f025a4 w3svc!HTTP_REQ_BASE::OnCompleteRequest+0x498
0b 00fcfd2c 65f02501 w3svc!HTTP_REQ_BASE::UnWrapRequest+0x1f4
0c 00fcfd4c 65f023b8 w3svc!HTTP_REQ_BASE::OnFillClientReq+0x7a
0d 00fcfec4 65f01d97 w3svc!HTTP_REQUEST::DoWork+0x99
0e 00fcfee4 65f06be5 w3svc!CLIENT_CONN::DoWork+0x1aa
0f 00fcff08 65f06b58 w3svc!CreateClient+0x7b
10 00fcff4c 6d701ad2 w3svc!W3OnConnectEx+0x118
11 00fcff80 6d7029a6 ISATQ!AtqpProcessContext+0x23e
12 00fcffb4 7c57438b ISATQ!AtqPoolThread+0x1a8
13 00fcffec 00000000 KERNEL32!BaseThreadStart+0x52
Thread ID: 9
System Thread ID: 11dc
Kernel Time: 0:0:0.15
User Time: 0:0:0.0
Other information: Thread is waiting for a lock to be released.
Looking for lock owner.
Owning thread System ID: 106c
Thread Type: HTTP Listener
# ChildEBP RetAddr
00 0104edf0 77f8822a ntdll!ZwWaitForSingleObject+0xb
01 0104ee64 77f8819b ntdll!RtlpWaitForCriticalSection+0x9e
02 0104ee6c 7c574114 ntdll!RtlEnterCriticalSection+0x46
03 0104eed0 7c57420c KERNEL32!GetModuleFileNameW+0xad
04 0104eefc 6ca815d1 KERNEL32!GetModuleFileNameA+0x44
05 0104f03c 6ca81d50 lonsint!L32pInitLsa+0x39
06 0104f0a0 6ca81ee3 lonsint!IISLogonNetUserW+0x9f
07 0104f0f0 769cef98 lonsint!IISLogonNetUserA+0x6a
08 0104f15c 769ced34 INFOCOMM!ValidateUser+0x10b
09 0104f6b8 769c1739 INFOCOMM!TsLogonUser+0x313
0a 0104f6ec 65f07611 INFOCOMM!TCP_AUTHENT::ClearTextLogon+0x67
0b 0104f888 65f03977 w3svc!HTTP_REQ_BASE::LogonUserA+0x67e
0c 0104fcac 65f0275a w3svc!HTTP_REQ_BASE::OnRestartRequest+0xa42
0d 0104fcec 65f025a4 w3svc!HTTP_REQ_BASE::OnCompleteRequest+0x498
0e 0104fd2c 65f02501 w3svc!HTTP_REQ_BASE::UnWrapRequest+0x1f4
0f 0104fd4c 65f023b8 w3svc!HTTP_REQ_BASE::OnFillClientReq+0x7a
10 0104fec4 65f01d97 w3svc!HTTP_REQUEST::DoWork+0x99
11 0104fee4 65f06be5 w3svc!CLIENT_CONN::DoWork+0x1aa
12 0104ff08 65f06b58 w3svc!CreateClient+0x7b
13 0104ff4c 6d701ad2 w3svc!W3OnConnectEx+0x118
14 0104ff80 6d7029a6 ISATQ!AtqpProcessContext+0x23e
15 0104ffb4 7c57438b ISATQ!AtqPoolThread+0x1a8
16 0104ffec 00000000 KERNEL32!BaseThreadStart+0x52
Thread ID: 10
System Thread ID: 11e0
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Possible ASP page. Possible DCOM activity
Executing Page: ASP.dll symbols not found. Unable to locate ASP page.
Continuing with other analysis.
No remote call being made
# ChildEBP RetAddr
00 011cfeb8 7c573c73 ntdll!ZwRemoveIoCompletion+0xb
01 011cfee4 77d31394 KERNEL32!GetQueuedCompletionStatus+0x27
02 011cff20 77d3e93f RPCRT4!COMMON_ProcessCalls+0x9e
03 011cff74 77d3e8c2 RPCRT4!LOADABLE_TRANSPORT::ProcessIOEvents+0x99
04 011cff78 77d35924 RPCRT4!ProcessIOEventsWrapper+0x9
05 011cffa8 77d358d6 RPCRT4!BaseCachedThreadRoutine+0x4f
06 011cffb4 7c57438b RPCRT4!ThreadStartRoutine+0x18
07 011cffec 00000000 KERNEL32!BaseThreadStart+0x52
Thread ID: 11
System Thread ID: 11e8
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Possible ASP page. Possible DCOM activity
Executing Page: ASP.dll symbols not found. Unable to locate ASP page.
Continuing with other analysis.
No remote call being made
# ChildEBP RetAddr
00 013cfe24 77d37ba7 ntdll!ZwReplyWaitReceivePortEx+0xb
01 013cff74 77d37b4c RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x74
02 013cff78 77d35924 RPCRT4!RecvLotsaCallsWrapper+0x9
03 013cffa8 77d358d6 RPCRT4!BaseCachedThreadRoutine+0x4f
04 013cffb4 7c57438b RPCRT4!ThreadStartRoutine+0x18
05 013cffec 00000000 KERNEL32!BaseThreadStart+0x52
Thread ID: 12
System Thread ID: c88
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 0144fd20 7c573c23 ntdll!ZwWaitForMultipleObjects+0xb
01 0144fd70 7c578f0d KERNEL32!WaitForMultipleObjectsEx+0xea
02 0144fd88 778322b2 KERNEL32!WaitForMultipleObjects+0x17
03 0144ffb4 7c57438b RTUTILS!TraceServerThread+0xde
04 0144ffec 00000000 KERNEL32!BaseThreadStart+0x52
Thread ID: 13
System Thread ID: 1064
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
*** ERROR: Symbol file could not be found. Defaulted to export
symbols for \\?\C:\IISDebugTools\IISCHAgent.dll -
Thread Type: Other
# ChildEBP RetAddr
00 0189fed8 7c573c73 ntdll!ZwRemoveIoCompletion+0xb
01 0189ff04 67306fab KERNEL32!GetQueuedCompletionStatus+0x27
WARNING: Stack unwind information not available. Following frames may
be wrong.
02 0189ffb4 7c57438b IISCHAgent!ConfigFileMonitor+0x15b
03 0189ffec 00000000 KERNEL32!BaseThreadStart+0x52
Thread ID: 14
System Thread ID: 1058
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 019dff18 7c573c23 ntdll!ZwWaitForMultipleObjects+0xb
01 019dff68 7c578f0d KERNEL32!WaitForMultipleObjectsEx+0xea
02 019dff80 6730649c KERNEL32!WaitForMultipleObjects+0x17
WARNING: Stack unwind information not available. Following frames may
be wrong.
03 019dffb4 7c57438b IISCHAgent!MonitorWorkerProcess+0xa3
04 019dffec 00000000 KERNEL32!BaseThreadStart+0x52
Thread ID: 15
System Thread ID: 1038
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: HTTP Compression Thread
# ChildEBP RetAddr
00 01a1ff5c 7c573b28 ntdll!ZwWaitForSingleObject+0xb
01 01a1ff84 7c573b50 KERNEL32!WaitForSingleObjectEx+0x71
02 01a1ff94 732c3366 KERNEL32!WaitForSingleObject+0xf
03 01a1ffb4 7c57438b compfilt!CompressionThread+0x29
04 01a1ffc0 00000000 KERNEL32!BaseThreadStart+0x52
Thread ID: 16
System Thread ID: 1018
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 01c9fe70 7c573c23 ntdll!ZwWaitForMultipleObjects+0xb
01 01c9fec0 77e119e6 KERNEL32!WaitForMultipleObjectsEx+0xea
02 01c9ff1c 77e11ace USER32!MsgWaitForMultipleObjectsEx+0x153
03 01c9ff38 65f09ccb USER32!MsgWaitForMultipleObjects+0x1d
04 01c9ff7c 78008454 w3svc!CMTACallbackThread::Thread+0x42
05 01c9ffb4 7c57438b MSVCRT!_endthread+0xc6
06 01c9ffec 00000000 KERNEL32!BaseThreadStart+0x52
Thread ID: 17
System Thread ID: 100c
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 01cdfea8 7c573c23 ntdll!ZwWaitForMultipleObjects+0xb
01 01cdfef8 77e119e6 KERNEL32!WaitForMultipleObjectsEx+0xea
02 01cdff54 77e11ace USER32!MsgWaitForMultipleObjectsEx+0x153
03 01cdff70 65f09d47 USER32!MsgWaitForMultipleObjects+0x1d
04 01cdffb4 7c57438b w3svc!OleHackThread+0x88
05 01cdffec 00000000 KERNEL32!BaseThreadStart+0x52
Thread ID: 18
System Thread ID: 11cc
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 01d5fce0 74fd1394 ntdll!ZwWaitForSingleObject+0xb
01 01d5fd1c 74fd3c59 msafd!SockWaitForSingleObject+0x1a8
02 01d5fe08 750312f5 msafd!WSPSelect+0x24e
03 01d5fe6c 6e2b3b6e WS2_32!select+0xe7
04 01d5ffb4 7c57438b inetsloc!SocketListenThread+0x51
05 01d5ffec 00000000 KERNEL32!BaseThreadStart+0x52
Thread ID: 19
System Thread ID: b4c
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: HTTP Listener
# ChildEBP RetAddr
00 01d9ff6c 7c573a4e ntdll!NtDelayExecution+0xb
01 01d9ff8c 7c573a22 KERNEL32!SleepEx+0x32
02 01d9ff98 6d7075e6 KERNEL32!Sleep+0xb
03 01d9ffb0 6d70791b ISATQ!ATQ_BMON_SET::BmonThreadFunc+0x4b
04 01d9ffb4 7c57438b ISATQ!BmonThreadFunc+0x9
05 01d9ffec 00000000 KERNEL32!BaseThreadStart+0x52
Thread ID: 20
System Thread ID: 11d0
Kernel Time: 0:0:0.15
User Time: 0:0:0.0
Other information: Thread is waiting for a lock to be released.
Looking for lock owner.
Owning thread System ID: 11dc
Thread Type: HTTP Listener
# ChildEBP RetAddr
00 023aefbc 77f8822a ntdll!ZwWaitForSingleObject+0xb
01 023af030 77f8819b ntdll!RtlpWaitForCriticalSection+0x9e
02 023af038 6ca81d42 ntdll!RtlEnterCriticalSection+0x46
03 023af0a0 6ca81ee3 lonsint!IISLogonNetUserW+0x91
04 023af0f0 769cef98 lonsint!IISLogonNetUserA+0x6a
05 023af15c 769ced34 INFOCOMM!ValidateUser+0x10b
06 023af6b8 769c1739 INFOCOMM!TsLogonUser+0x313
07 023af6ec 65f07611 INFOCOMM!TCP_AUTHENT::ClearTextLogon+0x67
08 023af888 65f03977 w3svc!HTTP_REQ_BASE::LogonUserA+0x67e
09 023afcac 65f0275a w3svc!HTTP_REQ_BASE::OnRestartRequest+0xa42
0a 023afcec 65f025a4 w3svc!HTTP_REQ_BASE::OnCompleteRequest+0x498
0b 023afd2c 65f02501 w3svc!HTTP_REQ_BASE::UnWrapRequest+0x1f4
0c 023afd4c 65f023b8 w3svc!HTTP_REQ_BASE::OnFillClientReq+0x7a
0d 023afec4 65f01d97 w3svc!HTTP_REQUEST::DoWork+0x99
0e 023afee4 65f06be5 w3svc!CLIENT_CONN::DoWork+0x1aa
0f 023aff08 65f06b58 w3svc!CreateClient+0x7b
10 023aff4c 6d701ad2 w3svc!W3OnConnectEx+0x118
11 023aff80 6d7029a6 ISATQ!AtqpProcessContext+0x23e
12 023affb4 7c57438b ISATQ!AtqPoolThread+0x1a8
13 023affec 00000000 KERNEL32!BaseThreadStart+0x52
Thread ID: 21
System Thread ID: ae8
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Other information: Thread is waiting for a lock to be released.
Looking for lock owner.
Owning thread System ID: 106c
Thread Type: HTTP Listener
# ChildEBP RetAddr
00 0236fc34 77f8822a ntdll!ZwWaitForSingleObject+0xb
01 0236fca8 77f8819b ntdll!RtlpWaitForCriticalSection+0x9e
02 0236fcb0 769c8488 ntdll!RtlEnterCriticalSection+0x46
03 0236fcd0 65f1dafb
INFOCOMM!IIS_SERVICE::FindAndReferenceEndpoint+0x20
04 0236fcfc 65f2b0a7
w3svc!HTTP_REQ_BASE::QueryW3InstanceAggressively+0x2f
05 0236fd2c 65f02501 w3svc!HTTP_REQ_BASE::UnWrapRequest+0x3a
06 0236fd4c 65f023b8 w3svc!HTTP_REQ_BASE::OnFillClientReq+0x7a
07 0236fec4 65f01d97 w3svc!HTTP_REQUEST::DoWork+0x99
08 0236fee4 65f06be5 w3svc!CLIENT_CONN::DoWork+0x1aa
09 0236ff08 65f06b58 w3svc!CreateClient+0x7b
0a 0236ff4c 6d701ad2 w3svc!W3OnConnectEx+0x118
0b 0236ff80 6d7029a6 ISATQ!AtqpProcessContext+0x23e
0c 0236ffb4 7c57438b ISATQ!AtqPoolThread+0x1a8
0d 0236ffec 00000000 KERNEL32!BaseThreadStart+0x52
Thread ID: 22
System Thread ID: b50
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Other information: Thread is waiting for a lock to be released.
Looking for lock owner.
Owning thread System ID: 106c
Thread Type: Other
# ChildEBP RetAddr
00 0261fc24 77f8822a ntdll!ZwWaitForSingleObject+0xb
01 0261fc98 77f8819b ntdll!RtlpWaitForCriticalSection+0x9e
02 0261fca0 77f8b536 ntdll!RtlEnterCriticalSection+0x46
03 0261fd1c 77f9ff3b ntdll!LdrpInitialize+0x1a7
04 00000000 00000000 ntdll!KiUserApcDispatcher+0x7
Closing open log file C:\iisstate\output\IISState-4416.log
"Pat [MSFT]" <patfilot RemoveThis @online.microsoft.com> wrote in message news:<#Jqt7hkeEHA.4092@TK2MSFTNGP10.phx.gbl>...
> Thread 16 is your shutdown thread. It basically tells all the requests to
> finish up and then calls Sleep() and waits for a bit then wakes up to see
> they (the requests) are done. If not, it Sleeps() again. So, it is
> basically looping in and out of sleep. The only thread that appears to be
> preventing this from working is thread 10, which is doing something inside
> ODBC. Unfortunately, the symbols for ODBC32 did not download (not your
> fault, that build may not be on the symbol server) so I can't tell why it
> would be blocked (bug? bad DB driver? etc.). I would start with updating
> MDAC to the latest version (www.microsoft.com/data) and see if that helps.
>
> There is no indication in the log that the issue is w/Cold Fusion or any
> site code. Of course, with a different log that may change.
>
> Pat<!-- ~MESSAGE_AFTER~ -->
>> Stay informed about: IIS hangs - IISState log