IIS State Log

I'm running into an issue with our IIS install where the service just
stops responding. I've run iisstate during one of these hangs which had
the following output:
-----------------------------------------------------------------------------------------------------------------------------------
Starting new log output
IISState version 3.3.1

Mon Mar 13 15:39:47 2006

OS = Windows 2003 Server
Executable: inetinfo.exe
PID = 4376

Note: Thread times are formatted as HH:MM:SS.ms

***********************

Thread ID: 0
System Thread ID: 1454
Kernel Time: 0:0:0.15
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 0006f9a8 77f4303b SharedUserData!SystemCallStub+0x4
01 0006f9ac 77e4905d ntdll!NtReadFile+0xc
02 0006fa14 77db51f1 kernel32!ReadFile+0x16c
03 0006fa40 77db5297 ADVAPI32!ScGetPipeInput+0x28
04 0006fab0 77dfa7f1 ADVAPI32!ScDispatcherLoop+0x4c
05 0006fcec 01002655 ADVAPI32!StartServiceCtrlDispatcherA+0x91
06 0006fe1c 010027ea inetinfo!StartDispatchTable+0x214
07 0006ff44 01003160 inetinfo!main+0x104
08 0006ffc0 77e4f38c inetinfo!mainCRTStartup+0x12f
09 0006fff0 00000000 kernel32!BaseProcessStart+0x23




Thread ID: 1
System Thread ID: fdc
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 0083fe18 77f43741 SharedUserData!SystemCallStub+0x4
01 0083fe1c 77e41817 ntdll!ZwWaitForSingleObject+0xc
02 0083fe8c 77e4168f kernel32!WaitForSingleObjectEx+0xac
03 0083fe9c 01002cf9 kernel32!WaitForSingleObject+0xf
04 0083ffb8 77e4a990 inetinfo!W3SVCThreadEntry+0x3b
05 0083ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 2
System Thread ID: d3c
Kernel Time: 0:0:0.78
User Time: 0:0:0.15
Thread Type: Other
# ChildEBP RetAddr
00 0087fcc4 77f43741 SharedUserData!SystemCallStub+0x4
01 0087fcc8 77e41817 ntdll!ZwWaitForSingleObject+0xc
02 0087fd38 77e4168f kernel32!WaitForSingleObjectEx+0xac
03 0087fd48 649f24ac kernel32!WaitForSingleObject+0xf
04 0087fd70 010023b6 iisadmin!ServiceEntry+0x214
05 0087ffa8 77db571b inetinfo!InetinfoStartService+0x2a6
06 0087ffb8 77e4a990 ADVAPI32!ScSvcctrlThreadA+0xe
07 0087ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 3
System Thread ID: 52c
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 00a7ff9c 77f4262b SharedUserData!SystemCallStub+0x4
01 00a7ffa0 77f6b5b2 ntdll!NtDelayExecution+0xc
02 00a7ffb8 77e4a990 ntdll!RtlpTimerThread+0x45
03 00a7ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 4
System Thread ID: 1290
Kernel Time: 0:0:0.0
User Time: 0:0:0.31
Thread Status: Thread is in a WAIT state.
Thread Type: Other
# ChildEBP RetAddr
00 00b8feb0 77f4372d SharedUserData!SystemCallStub+0x4
01 00b8feb4 77e41bfa ntdll!NtWaitForMultipleObjects+0xc
02 00b8ff5c 77e4b0e4 kernel32!WaitForMultipleObjectsEx+0x11a
03 00b8ff74 6e0b377a kernel32!WaitForMultipleObjects+0x17
04 00b8ffa0 6e0b6012 COADMIN!NOTIFY_CONTEXT::GetNextContext+0x68
05 00b8ffb8 77e4a990 COADMIN!NOTIFY_CONTEXT::NotifyThreadProc+0x62
06 00b8ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 5
System Thread ID: 1700
Kernel Time: 0:0:0.78
User Time: 0:0:0.62
Thread Status: Thread is in a WAIT state.
Thread Type: SMTP Service Worker Thread
# ChildEBP RetAddr
00 016efbc0 77f4372d SharedUserData!SystemCallStub+0x4
01 016efbc4 77e41bfa ntdll!NtWaitForMultipleObjects+0xc
02 016efc6c 77d076f5 kernel32!WaitForMultipleObjectsEx+0x11a
03 016efcc8 77d077f5 USER32!RealMsgWaitForMultipleObjectsEx+0x13f
04 016efce4 643f5723 USER32!MsgWaitForMultipleObjects+0x1d
05 016efd30 6b77305d INFOCOMM!IIS_SERVICE::StartServiceOperation+0x22f
06 016efd70 010023b6 SMTPSVC!ServiceEntry+0x129
07 016effa8 77db571b inetinfo!InetinfoStartService+0x2a6
08 016effb8 77e4a990 ADVAPI32!ScSvcctrlThreadA+0xe
09 016effec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 6
System Thread ID: 128c
Kernel Time: 0:0:0.78
User Time: 0:0:0.15
Thread Type: HTTP Listener
# ChildEBP RetAddr
00 01a2ff50 77f430c7 SharedUserData!SystemCallStub+0x4
01 01a2ff54 77e430bc ntdll!ZwRemoveIoCompletion+0xc
02 01a2ff80 63ec7235 kernel32!GetQueuedCompletionStatus+0x27
03 01a2ffb8 77e4a990 ISATQ!AtqPoolThread+0x40
04 01a2ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 7
System Thread ID: 15dc
Kernel Time: 0:0:0.156
User Time: 0:0:0.31
Thread Type: HTTP Listener
# ChildEBP RetAddr
00 01a6ff50 77f430c7 SharedUserData!SystemCallStub+0x4
01 01a6ff54 77e430bc ntdll!ZwRemoveIoCompletion+0xc
02 01a6ff80 63ec7235 kernel32!GetQueuedCompletionStatus+0x27
03 01a6ffb8 77e4a990 ISATQ!AtqPoolThread+0x40
04 01a6ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 8
System Thread ID: 1728
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Possible ASP page. Possible DCOM activity
Executing Page: ASP.dll symbols not found. Unable to locate ASP page.
Continuing with other analysis.

No remote call being made

# ChildEBP RetAddr
00 01e6fe20 77f4313f SharedUserData!SystemCallStub+0x4
01 01e6fe24 77c57b85 ntdll!NtReplyWaitReceivePortEx+0xc
02 01e6ff8c 77c60829 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x193
03 01e6ff90 77c60771 RPCRT4!RecvLotsaCallsWrapper+0x9
04 01e6ffb0 77c60857 RPCRT4!BaseCachedThreadRoutine+0x9c
05 01e6ffb8 77e4a990 RPCRT4!ThreadStartRoutine+0x17
06 01e6ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 9
System Thread ID: 1b8
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Possible ASP page. Possible DCOM activity
Executing Page: ASP.dll symbols not found. Unable to locate ASP page.
Continuing with other analysis.

No remote call being made

# ChildEBP RetAddr
00 01eafeb4 77f430c7 SharedUserData!SystemCallStub+0x4
01 01eafeb8 77e430bc ntdll!ZwRemoveIoCompletion+0xc
02 01eafee4 77c80bd1 kernel32!GetQueuedCompletionStatus+0x27
03 01eaff20 77c80a78 RPCRT4!COMMON_ProcessCalls+0x9f
04 01eaff8c 77c58159 RPCRT4!LOADABLE_TRANSPORT::ProcessIOEvents+0x115
05 01eaff90 77c60771 RPCRT4!ProcessIOEventsWrapper+0x9
06 01eaffb0 77c60857 RPCRT4!BaseCachedThreadRoutine+0x9c
07 01eaffb8 77e4a990 RPCRT4!ThreadStartRoutine+0x17
08 01eaffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 10
System Thread ID: 1650
Kernel Time: 0:0:0.0
User Time: 0:0:0.15
Thread Status: Thread is in a WAIT state.
Thread Type: Other
# ChildEBP RetAddr
00 01eefec8 77f4372d SharedUserData!SystemCallStub+0x4
01 01eefecc 77e41bfa ntdll!NtWaitForMultipleObjects+0xc
02 01eeff74 77e4b0e4 kernel32!WaitForMultipleObjectsEx+0x11a
03 01eeff8c 69532430 kernel32!WaitForMultipleObjects+0x17
04 01eeffb8 77e4a990 exstrace!RegNotifyThread+0x68
05 01eeffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 11
System Thread ID: 8d8
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: Other
# ChildEBP RetAddr
00 01f2fecc 77f4372d SharedUserData!SystemCallStub+0x4
01 01f2fed0 77e41bfa ntdll!NtWaitForMultipleObjects+0xc
02 01f2ff78 77e4b0e4 kernel32!WaitForMultipleObjectsEx+0x11a
03 01f2ff90 695319c0 kernel32!WaitForMultipleObjects+0x17
04 01f2ffb8 77e4a990 exstrace!WriteTraceThread+0x2f
05 01f2ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 12
System Thread ID: 2c4
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 0212ff18 77f43741 SharedUserData!SystemCallStub+0x4
01 0212ff1c 77e41817 ntdll!ZwWaitForSingleObject+0xc
02 0212ff8c 77e4168f kernel32!WaitForSingleObjectEx+0xac
03 0212ff9c 01738673 kernel32!WaitForSingleObject+0xf
04 0212ffb8 77e4a990 FCACHDLL!CScheduleThread::ScheduleThread+0x60
05 0212ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 13
System Thread ID: 150c
Kernel Time: 0:0:0.15
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: SMTP Service Worker Thread
# ChildEBP RetAddr
00 0226fec0 77f4372d SharedUserData!SystemCallStub+0x4
01 0226fec4 77e41bfa ntdll!NtWaitForMultipleObjects+0xc
02 0226ff6c 77e4b0e4 kernel32!WaitForMultipleObjectsEx+0x11a
03 0226ff84 6b78b2d6 kernel32!WaitForMultipleObjects+0x17
04 0226ffb8 77e4a990 SMTPSVC!TcpRegNotifyThread+0xdc
05 0226ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 14
System Thread ID: d10
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: SMTP Service Worker Thread
# ChildEBP RetAddr
00 022aff20 77f43741 SharedUserData!SystemCallStub+0x4
01 022aff24 77e41817 ntdll!ZwWaitForSingleObject+0xc
02 022aff94 77e4168f kernel32!WaitForSingleObjectEx+0xac
03 022affa4 6b78b17a kernel32!WaitForSingleObject+0xf
04 022affb8 77e4a990 SMTPSVC!FreeLibThread+0x2c
05 022affec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 15
System Thread ID: 1c4
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Possible ASP page. Possible DCOM activity
Executing Page: ASP.dll symbols not found. Unable to locate ASP page.
Continuing with other analysis.

No remote call being made

# ChildEBP RetAddr
00 02a2fe20 77f4313f SharedUserData!SystemCallStub+0x4
01 02a2fe24 77c57b85 ntdll!NtReplyWaitReceivePortEx+0xc
02 02a2ff8c 77c60829 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x193
03 02a2ff90 77c60771 RPCRT4!RecvLotsaCallsWrapper+0x9
04 02a2ffb0 77c60857 RPCRT4!BaseCachedThreadRoutine+0x9c
05 02a2ffb8 77e4a990 RPCRT4!ThreadStartRoutine+0x17
06 02a2ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 16
System Thread ID: 428
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Possible ASP page. Possible DCOM activity
Executing Page: ASP.dll symbols not found. Unable to locate ASP page.
Continuing with other analysis.

No remote call being made

# ChildEBP RetAddr
00 02a8fe20 77f4313f SharedUserData!SystemCallStub+0x4
01 02a8fe24 77c57b85 ntdll!NtReplyWaitReceivePortEx+0xc
02 02a8ff8c 77c60829 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x193
03 02a8ff90 77c60771 RPCRT4!RecvLotsaCallsWrapper+0x9
04 02a8ffb0 77c60857 RPCRT4!BaseCachedThreadRoutine+0x9c
05 02a8ffb8 77e4a990 RPCRT4!ThreadStartRoutine+0x17
06 02a8ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 17
System Thread ID: 4b0
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 02acff10 77f43741 SharedUserData!SystemCallStub+0x4
01 02acff14 77e41817 ntdll!ZwWaitForSingleObject+0xc
02 02acff84 77e4168f kernel32!WaitForSingleObjectEx+0xac
03 02acff94 029ad064 kernel32!WaitForSingleObject+0xf
04 02acffb8 77e4a990
aqueue!CSMTP_RETRY_HANDLER::RetryThreadRoutine+0xc1
05 02acffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 18
System Thread ID: 224
Kernel Time: 0:0:0.15
User Time: 0:0:0.15
Thread Status: Thread is in a WAIT state.
Thread Type: SMTP Service Worker Thread
# ChildEBP RetAddr
00 02b4fe84 77f4372d SharedUserData!SystemCallStub+0x4
01 02b4fe88 77e41bfa ntdll!NtWaitForMultipleObjects+0xc
02 02b4ff30 77e4b0e4 kernel32!WaitForMultipleObjectsEx+0x11a
03 02b4ff48 0298c648 kernel32!WaitForMultipleObjects+0x17
04 02b4ffa4 6b77e8ae aqueue!CConnMgr::GetNextConnection+0x1e1
05 02b4ffb8 77e4a990 SMTPSVC!PERSIST_QUEUE::QueueThreadRoutine+0x21
06 02b4ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 19
System Thread ID: 17e8
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: HTTP Listener
# ChildEBP RetAddr
00 02b8fe38 77f43741 SharedUserData!SystemCallStub+0x4
01 02b8fe3c 71b23ac3 ntdll!ZwWaitForSingleObject+0xc
02 02b8fe78 71b239d1 mswsock!SockWaitForSingleObject+0x19b
03 02b8ff3c 71c016c9 mswsock!WSPSelect+0x229
04 02b8ff8c 63ec4696 WS2_32!select+0xb9
05 02b8ffb4 63ec4700 ISATQ!ATQ_BMON_SET::BmonThreadFunc+0x22
06 02b8ffb8 77e4a990 ISATQ!BmonThreadFunc+0x9
07 02b8ffc4 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 20
System Thread ID: 17bc
Kernel Time: 0:0:0.15
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: Other
# ChildEBP RetAddr
00 02c2fcec 77f4372d SharedUserData!SystemCallStub+0x4
01 02c2fcf0 77f75297 ntdll!NtWaitForMultipleObjects+0xc
02 02c2ffb8 77e4a990 ntdll!RtlpWaitThread+0x158
03 02c2ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 21
System Thread ID: d9c
Kernel Time: 0:0:0.375
User Time: 0:0:0.328
Thread Type: Possible ASP page. Possible DCOM activity
Executing Page: ASP.dll symbols not found. Unable to locate ASP page.
Continuing with other analysis.

No remote call being made

# ChildEBP RetAddr
00 00abfe20 77f4313f SharedUserData!SystemCallStub+0x4
01 00abfe24 77c57b85 ntdll!NtReplyWaitReceivePortEx+0xc
02 00abff8c 77c60829 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x193
03 00abff90 77c60771 RPCRT4!RecvLotsaCallsWrapper+0x9
04 00abffb0 77c60857 RPCRT4!BaseCachedThreadRoutine+0x9c
05 00abffb8 77e4a990 RPCRT4!ThreadStartRoutine+0x17
06 00abffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 22
System Thread ID: ffc
Kernel Time: 0:0:0.328
User Time: 0:0:0.453
Thread Type: Possible ASP page. Possible DCOM activity
Executing Page: ASP.dll symbols not found. Unable to locate ASP page.
Continuing with other analysis.

No remote call being made

# ChildEBP RetAddr
00 00f8fe20 77f4313f SharedUserData!SystemCallStub+0x4
01 00f8fe24 77c57b85 ntdll!NtReplyWaitReceivePortEx+0xc
02 00f8ff8c 77c60829 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x193
03 00f8ff90 77c60771 RPCRT4!RecvLotsaCallsWrapper+0x9
04 00f8ffb0 77c60857 RPCRT4!BaseCachedThreadRoutine+0x9c
05 00f8ffb8 77e4a990 RPCRT4!ThreadStartRoutine+0x17
06 00f8ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 23
System Thread ID: 16b8
Kernel Time: 0:0:0.359
User Time: 0:0:0.390
Thread Type: Possible ASP page. Possible DCOM activity
Executing Page: ASP.dll symbols not found. Unable to locate ASP page.
Continuing with other analysis.

No remote call being made

# ChildEBP RetAddr
00 00f4fe20 77f4313f SharedUserData!SystemCallStub+0x4
01 00f4fe24 77c57b85 ntdll!NtReplyWaitReceivePortEx+0xc
02 00f4ff8c 77c60829 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x193
03 00f4ff90 77c60771 RPCRT4!RecvLotsaCallsWrapper+0x9
04 00f4ffb0 77c60857 RPCRT4!BaseCachedThreadRoutine+0x9c
05 00f4ffb8 77e4a990 RPCRT4!ThreadStartRoutine+0x17
06 00f4ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 24
System Thread ID: fd4
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 00ecff70 77f430c7 SharedUserData!SystemCallStub+0x4
01 00ecff74 77f7e6ae ntdll!ZwRemoveIoCompletion+0xc
02 00ecffb8 77e4a990 ntdll!RtlpWorkerThread+0x3b
03 00ecffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 25
System Thread ID: a34
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 00f0ff70 77f430c7 SharedUserData!SystemCallStub+0x4
01 00f0ff74 77f7e6ae ntdll!ZwRemoveIoCompletion+0xc
02 00f0ffb8 77e4a990 ntdll!RtlpWorkerThread+0x3b
03 00f0ffec 00000000 kernel32!BaseThreadStart+0x34

*****

Dump name is formatted as: PID-Timestamp.dmp

Creating C:\iisstate\output\4376-1142278788.dmp - mini user dump

*****

Closing open log file C:\iisstate\output\IISState-4376.log

Finished
-------------------------------------------------------------------------------------------------------
I'm wondering if anyone could suggest where I go from here?
Thanks in advance...