Windows 7 Serious memory leak in nonpaged pool (Irp tag)

Rafcio

Active Member
I have a serious memory leak in nonpaged pool, which happen to be Irp packets. Here is the whole story.

The system is Win7 Ultimate x64 which I primarily use to host some VirtualBox VMs. Around beginning of May I noticed that the box started to lock up and it was rock solid for almost a year before then. I discovered that the reason for lock ups is a memory leak that east up its 16 GB of memory in a few days. I started digging deeper and figured out that the nonpaged memory grows from the typical 500 MB or so to a few GBs in a couple of days. The poolmon tool pointed to Irp tag as the clear offender.

I thought it could be some update or new driver that got installed, so I restored the system from an earlier image. I went back to images as far back as November, October and September last year, but nothing helped. The system was working fine till about end of April, so I was very surprised that reverting back to the time the system was working OK did not fix the problem.

Anyway, further troubleshooting with driver verifier (log file analysis) did not point to any driver with suspicious amount of allocated memory. So the next step was to force a memory dump with driver verifier running and use !verifier kernel debugger extension to see the memory allocated.

First of all, driver verifier puts a lot of load on the system, so it stops responding after few hours with CPUs pegged at 100%. Also, the system is much slower when driver verifier is running and the memory leak happens at the much slower rate.

I recently forced a memory dump after about 8 hours of system uptime. The nonpaged memory was about 915 MB, so I'd expected a clear indication of what driver has plenty of memory allocated. Unfortunately not so.

The !verifier 1 provided this output:

Verify Level 418 ... enabled options are:
All pool allocations checked on unload
Io subsystem checking enabled
IRP Logging

Summary of All Verifier Statistics

RaiseIrqls 0x0
AcquireSpinLocks 0x18819b3d
Synch Executions 0x7b1a30
Trims 0x0

Pool Allocations Attempted 0x474127c6
Pool Allocations Succeeded 0x474127c6
Pool Allocations Succeeded SpecialPool 0x47589c
Pool Allocations With NO TAG 0xa
Pool Allocations Failed 0x0
Resource Allocations Failed Deliberately 0x0

Current paged pool allocations 0x18a9a for 08CA3440 bytes
Peak paged pool allocations 0x27a61 for 0A99EE20 bytes
Current nonpaged pool allocations 0x1916e for 039C57D0 bytes
Peak nonpaged pool allocations 0x19500 for 03AECF70 bytes

Driver Verification List

Entry State NonPagedPool PagedPool Module

fffffa800cb8d880 Loaded 00036530 00000090 hal.dll
fffffa800cb91740 Loaded 00000000 00000000 kdcom.dll
fffffa800cafd200 Loaded 00000000 00000000 mcupdate.dll
fffffa800cafd040 Loaded 00000000 00000000 PSHED.dll
fffffa800cb90740 Loaded 000196c0 000f2710 CLFS.SYS
fffffa800cb90580 Loaded 00000000 00305a80 CI.dll
fffffa800cb9af50 Loaded 00064e90 00003660 Wdf01000.sys
fffffa800cb9ad90 Loaded 00000650 000002d0 WDFLDR.SYS
fffffa800cb9abd0 Loaded 00090bc0 000050a0 ACPI.sys
fffffa800cb9aa10 Loaded 00000000 00000000 WMILIB.SYS
fffffa800cb9a820 Loaded 00000000 00000000 msisadrv.sys
fffffa800cb9a660 Loaded 0000e110 00015830 pci.sys
fffffa800cb9a470 Loaded 00000000 00000000 vdrvroot.sys
fffffa800cb9a290 Loaded 00004ec0 00000080 partmgr.sys
fffffa800cb9a1b0 Loaded 00000000 00000000 compbatt.sys
fffffa800cba3e50 Loaded 000001a0 00000130 BATTC.SYS
fffffa800cba3c70 Loaded 00000110 00000500 volmgr.sys
fffffa800cba3a90 Loaded 00008140 00004050 volmgrx.sys
fffffa800cba38b0 Loaded 00000000 00000000 pciide.sys
fffffa800cba36d0 Loaded 00000000 00000050 PCIIDEX.SYS
fffffa800cba3510 Loaded 000003d0 00000000 jraid.sys
fffffa800cba3320 Loaded 00007350 00000190 SCSIPORT.SYS
fffffa800cba3130 Loaded 00000000 00002110 mountmgr.sys
fffffa800cba4f40 Loaded 00000000 00000000 vmbus.sys
fffffa800cba4d60 Loaded 000020e0 00000000 winhv.sys
fffffa800cba4b80 Loaded 00000000 00000000 atapi.sys
fffffa800cba49a0 Loaded 00008ec0 00000000 ataport.SYS
fffffa800cba47e0 Loaded 00000000 00000000 amdxata.sys
fffffa800cba4600 Loaded 00847040 00ae4440 fltmgr.sys
fffffa800cba4410 Loaded 00000ba0 00000620 fileinfo.sys
fffffa800cba4230 Loaded 0001abf0 00000a80 stcvsm.sys
fffffa800cba5fa0 Loaded 004980f0 019f7d10 Ntfs.sys
fffffa800cba5dc0 Loaded 00012250 0000b2b0 msrpc.sys
fffffa800cba5bb0 Loaded 000002a0 00004560 ksecdd.sys
fffffa800cba59d0 Loaded 00010ae0 00000070 cng.sys
fffffa800cba57f0 Loaded 00000000 00002570 pcw.sys
fffffa800cba5610 Loaded 00000020 00000000 Fs_Rec.sys
fffffa800cba5430 Loaded 005015a0 00000420 ndis.sys
fffffa800cba5250 Loaded 00125a50 00000000 NETIO.SYS
fffffa800cba5070 Loaded 00000070 00000b00 ksecpkg.sys
fffffa800cba6f40 Loaded 00109730 00000000 tcpip.sys
fffffa800cba6d50 Loaded 000002e0 000030b0 fwpkclnt.sys
fffffa800cba6b60 Loaded 00000000 00000000 vmstorfl.sys
fffffa800cba6980 Loaded 0043c6e0 000001f0 volsnap.sys
fffffa800cba67a0 Loaded 00000000 00000000 spldr.sys
fffffa800cba6580 Loaded 000325c0 00000000 rdyboost.sys
fffffa800cba63a0 Loaded 00000000 00000000 NBVol.sys
fffffa800cba61c0 Loaded 00000000 00000000 NBVolUp.sys
fffffa800cba7fa0 Loaded 00000f20 00000410 mup.sys
fffffa800cba7db0 Loaded 00000000 00000000 hwpolicy.sys
fffffa800cba7bd0 Loaded 000003e0 00000000 fvevol.sys
fffffa800cba7a10 Loaded 000000c0 00000060 disk.sys
fffffa800cba77f0 Loaded 00035680 00000200 CLASSPNP.SYS
fffffa800cba7600 Loaded 00000000 00000000 AtiPcie64.sys
fffffa800cba7410 Loaded 00000000 00000000 ahcix64s.sys
fffffa800cba7220 Loaded 00bdf820 000001b0 storport.sys
fffffa800f9035d0 Loaded&Unloaded 000dd3b0 00000000 crashdmp.sys
fffffa800f5fb900 Loaded&Unloaded 00000000 00000000 dump_storport.sys
fffffa800f906920 Loaded&Unloaded 00000000 00000000 dump_ahcix64s.sys
fffffa800f4bfe70 Loaded&Unloaded 00004010 00000000 dump_dumpfve.sys
fffffa800f958010 Loaded 000137e0 00000030 cdrom.sys
fffffa800fc59150 Loaded 00005560 0320d390 SRTSP64.SYS
fffffa800f92f7f0 Loaded&Unloaded 000016c0 00000000 EX64.SYS
fffffa800f914570 Loaded 00016b90 0000de90 SYMEVENT64x86.SYS
fffffa800f985590 Loaded&Unloaded 00000000 00000000 ENG64.SYS
fffffa800fc4b030 Loaded 00000350 00000260 SRTSPX64.SYS
fffffa800fc60030 Loaded 00000000 00000000 Null.SYS
fffffa800fc62030 Loaded 00000000 00000000 Beep.SYS
fffffa800fb6c030 Loaded 00005180 00000060 watchdog.sys
fffffa800f936560 Loaded 000000c0 00000970 VIDEOPRT.SYS
fffffa800fb5d030 Loaded 00000000 000010f0 vga.sys
fffffa800f9b3480 Loaded 00000000 00000000 RDPCDD.sys
fffffa800f998480 Loaded 00000000 00000000 rdpencdd.sys
fffffa800fc60480 Loaded 00000000 00000000 rdprefmp.sys
fffffa800fb5d490 Loaded 00000370 000005d0 Msfs.SYS
fffffa800fc5b480 Loaded 00000f40 0000ea30 Npfs.SYS
fffffa800fb72490 Loaded 00004870 00000000 TDI.SYS
fffffa800fb6e490 Loaded 00165260 00000000 tdx.sys
fffffa800faf5060 Loaded 00207000 00000000 wpsdrvnt.sys
fffffa800fac7510 Loaded 0009cac0 00005d40 afd.sys
fffffa800fa4b0c0 Loaded 00024590 00000000 netbt.sys
fffffa800fadc8c0 Loaded 000006d0 00000000 wfplwf.sys
fffffa800fb5f030 Loaded 000015b0 00000000 pacer.sys
fffffa800f4407a0 Loaded 00002b30 00000000 vpcnfltr.sys
fffffa800f8f57e0 Loaded 000011f0 00000000 netbios.sys
fffffa800fab28b0 Loaded 00000260 00000000 wanarp.sys
fffffa800fd84040 Loaded 0000a480 00000080 vpcvmm.sys
fffffa800faae500 Loaded 00000000 00000000 VBoxUSBMon.sys
fffffa800fbb2040 Loaded 0000db80 00000000 VBoxDrv.sys
fffffa800fae4510 Loaded 000042c0 00000000 termdd.sys
fffffa800fbc3030 Loaded 00000000 00000000 SCDEmu.SYS
fffffa800fbc5030 Loaded 00000000 00000020 sbmount.SYS
fffffa800fbf3030 Loaded 000064e0 00003e70 rdbss.sys
fffffa800fba94f0 Loaded 00001d10 00000000 nsiproxy.sys
fffffa800fbe20a0 Loaded 00005260 00000080 mssmbios.sys
fffffa800fbb6470 Loaded 00001010 00001cf0 eeCtrl64.sys
fffffa800fb79e30 Loaded 00000f80 0000a9e0 EraserUtilRebootDrv.sys
fffffa800fb795f0 Loaded 00000000 000003f0 discache.sys
fffffa800fada4a0 Loaded 00002070 000027e0 csc.sys
fffffa800fada570 Loaded 000003e0 00000090 dfsc.sys
fffffa800fb2f6c0 Loaded 00000000 00000000 blbdrive.sys
fffffa800fde8030 Loaded 00000000 00000000 AsUpIO.sys
fffffa800fb70490 Loaded 00000000 00000000 AsIO.sys
fffffa800fdfb030 Loaded 000000c0 00000000 tunnel.sys
fffffa800fb61aa0 Loaded 00000bc0 00000000 amdppm.sys
fffffa800fb2f600 Loaded 0000c170 0004a450 atikmpag.sys
fffffa800fe1c8b0 Loaded 00167540 0120cd70 atikmdag.sys
fffffa800fe59030 Loaded 000107b0 01d776a0 dxgkrnl.sys
fffffa800fb964a0 Loaded 000c3f10 000d5ab0 dxgmms1.sys
fffffa800fbbe490 Loaded 0000e860 000000a0 HDAudBus.sys
fffffa800fbe6c60 Loaded 00000000 00000000 USBD.SYS
fffffa800fdd3750 Loaded 0007ebc0 000000f0 nusb3xhc.sys
fffffa800fade480 Loaded 00000000 00000000 usbfilter.sys
fffffa800fe30030 Loaded 00048e80 00000150 USBPORT.SYS
fffffa800fdff490 Loaded 00000000 00000000 usbohci.sys
fffffa800fdbb8b0 Loaded 00000000 00000000 usbehci.sys
fffffa800fe57040 Loaded 00000000 00000000 ASACPI.sys
fffffa800fe26030 Loaded 00002010 00000000 i8042prt.sys
fffffa800fe99030 Loaded 000000d0 00000000 L8042Kbd.sys
fffffa800fe99090 Loaded 00001a60 00000000 kbdclass.sys
fffffa800fe69030 Loaded 000005b0 00000000 L8042mou.Sys
fffffa800fa8f030 Loaded 00000ff0 00000000 LMouKE.Sys
fffffa800fdcc8b0 Loaded 000023c0 00000000 mouclass.sys
fffffa800fe5d030 Loaded 0000da40 00000000 1394ohci.sys
fffffa800feb6030 Loaded 0009bf50 00000000 Rt64win7.sys
fffffa800fdd5490 Loaded 00000030 000000b0 wmiacpi.sys
fffffa800fdbf480 Loaded 00000000 00000000 CompositeBus.sys
fffffa800fe26980 Loaded 00000020 00000000 AgileVpn.sys
fffffa800fe34310 Loaded 00000040 00000000 rasl2tp.sys
fffffa800feac060 Loaded 00000000 00000000 ndistapi.sys
fffffa800fed5030 Loaded 00002830 00000000 ndiswan.sys
fffffa80109b1930 Loaded 00000000 00000000 raspppoe.sys
fffffa800feb4040 Loaded 00000040 00000000 raspptp.sys
fffffa800fe52490 Loaded 000003b0 00000000 rassstp.sys
fffffa800feb0560 Loaded 00000000 000000d0 teamviewervpn.sys
fffffa800fe598b0 Loaded 000003f0 00000000 VBoxNetAdp.sys
fffffa800fe9f4c0 Loaded 00000000 00000000 rdpbus.sys
fffffa800feb09b0 Loaded 00000d80 00000000 VBoxNetFlt.sys
fffffa80109e3970 Loaded 001c3c80 00000000 teefer2.sys
fffffa800feb0e70 Loaded 00001200 00002540 ks.sys
fffffa800ffaa030 Loaded 00000000 00000000 swenum.sys
fffffa800ff15040 Loaded 00000000 00000080 amdiox64.sys
fffffa800ff30040 Loaded 00001a10 00000000 umbus.sys
fffffa800ff5e030 Loaded 000002e0 000004e0 usbrpm.sys
fffffa800ff7f030 Loaded 00000000 00000000 vpcusb.sys
fffffa800ff49040 Loaded 000049e0 00000000 vpchbus.sys
fffffa800ffd7030 Loaded 00000a30 00000080 nusb3hub.sys
fffffa800ffcb030 Loaded 00019430 000001a0 usbhub.sys
fffffa8010ac8030 Loaded 00004e90 00000000 NDProxy.SYS
fffffa8010c46230 Loaded 000000c0 000000b0 drmk.sys
fffffa8010c20890 Loaded 00004930 00007920 portcls.sys
fffffa8010c3cf20 Loaded 0000bf80 00000000 RtHDMIVX.sys
fffffa8010c6b7a0 Loaded 000001d0 00000000 ksthunk.sys
fffffa8010c6b570 Loaded 000294b0 00000e60 viahduaa.sys
fffffa8010ca0480 Loaded 00002760 00000000 61883.sys
fffffa8010d1c550 Loaded 00000ed0 00000000 avc.sys
fffffa8010d71450 Loaded 00000710 00000050 STREAM.SYS
fffffa80111e1040 Loaded 00001920 00000000 msdv.sys
fffffa80116ea030 Loaded 00002e00 00000000 HIDPARSE.SYS
fffffa8010d914d0 Loaded 00006a30 00000060 HIDCLASS.SYS
fffffa800ffa8030 Loaded 00001200 00000000 hidusb.sys
fffffa80116ea280 Loaded 00000590 00000000 AmUStor.SYS
fffffa8011087030 Loaded 00000000 00000000 Dxapi.sys
fffffa8011252030 Loaded 00014520 000000c0 win32k.sys
fffffa80114561a0 Loaded 00000000 00000000 monitor.sys
fffffa8011bd4b90 Loaded&Unloaded 00000000 00000000 TSDDD.dll
fffffa800f96b060 Loaded&Unloaded 00001030 00001010 cdd.dll
fffffa801100baf0 Loaded 00000000 00017dc0 luafv.sys
fffffa80111d18c0 Loaded 00000cc0 00018460 PDFsFilter.sys
fffffa8010ffae30 Loaded 0000eaa0 00000000 WudfPf.sys
fffffa80112437e0 Loaded 00000000 00000000 DefragFS.SYS
fffffa80116c8780 Loaded 00000040 00000000 lltdio.sys
fffffa80115bc520 Loaded 00000000 00000000 pnarp.sys
fffffa8011720af0 Loaded 00000000 00000000 purendis.sys
fffffa80117bf030 Loaded 00000090 00000090 rspndr.sys
fffffa80118bb6c0 Loaded 0001cba0 000024d0 HTTP.sys
fffffa8010907a10 Loaded 00000ce0 000008d0 bowser.sys
fffffa8011520220 Loaded 000000e0 00000000 mpsdrv.sys
fffffa8010968730 Loaded 000060f0 00001120 mrxsmb.sys
fffffa8011711bb0 Loaded 00001050 00000f00 mrxsmb10.sys
fffffa80109cfaf0 Loaded 00000000 00000000 mrxsmb20.sys
fffffa8011c460e0 Loaded 00000000 00000000 AODDriver2.sys
fffffa80104ac220 Loaded&Unloaded 006fa270 00000000 WpsHelper.sys
fffffa80118dc5a0 Loaded 00000000 00000000 cpuz133_x64.sys
fffffa8011d46a40 Loaded 00000000 00000000 cpuz135_x64.sys
fffffa8012276dd0 Loaded 00000000 000000b0 peauth.sys
fffffa8010442180 Loaded 00000000 00000050 secdrv.SYS
fffffa8011c50f00 Loaded 0007ae10 00063220 srvnet.sys
fffffa8012357bb0 Loaded 00006680 00000000 tcpipreg.sys
fffffa8010c19470 Loaded 00012550 00000620 srv2.sys
fffffa8012aef8b0 Loaded 0001a3a0 000018e0 srv.sys
fffffa80123b5c00 Loaded 00000000 00000000 TuneUpUtilitiesDriver64.sys
fffffa800d6aa730 Loaded 00006210 00000000 WUDFRd.sys
fffffa8012b230b0 Loaded 000004a0 00000070 rdpdr.sys
fffffa80116da920 Loaded 00000f60 00000000 tdtcp.sys
fffffa80111febd0 Loaded 00000000 00000000 tssecsrv.sys
fffffa8011186b50 Loaded 000000a0 00002c20 RDPWD.SYS
fffffa80112a59d0 Unloaded 00000000 00000000 spsys.sys
fffffa800dfbd180 Loaded 00000110 00000000 asyncmac.sys
fffffa8012b76010 Loaded 00000000 00000000 myfault.sys

I can't provide the !verifier 3 output, because it's 12 MB, but a script I wrote to provide certain statistics from that output for Irp+ tag listed this:

Driver: CLFS.SYS
NonPagedPool: 104128 bytes
PagedPool: 993040 bytes
Tags found: 10

Driver: Wdf01000.sys
NonPagedPool: 413328 bytes
PagedPool: 13920 bytes
Tags found: 4

Driver: partmgr.sys
NonPagedPool: 20160 bytes
PagedPool: 128 bytes
Tags found: 6

Driver: SCSIPORT.SYS
NonPagedPool: 29520 bytes
PagedPool: 400 bytes
Tags found: 2

Driver: fltmgr.sys
NonPagedPool: 8679488 bytes
PagedPool: 11420736 bytes
Tags found: 1

Driver: volsnap.sys
NonPagedPool: 4441824 bytes
PagedPool: 496 bytes
Tags found: 4

Driver: CLASSPNP.SYS
NonPagedPool: 218752 bytes
PagedPool: 512 bytes
Tags found: 155

Driver: cdrom.sys
NonPagedPool: 79840 bytes
PagedPool: 48 bytes
Tags found: 2

Driver: SRTSP64.SYS
NonPagedPool: 21856 bytes
PagedPool: 52482960 bytes
Tags found: 2

Driver: netbt.sys
NonPagedPool: 148880 bytes
PagedPool: 0 bytes
Tags found: 1

Driver: LMouKE.Sys
NonPagedPool: 4080 bytes
PagedPool: 0 bytes
Tags found: 1

Driver: nusb3hub.sys
NonPagedPool: 2608 bytes
PagedPool: 128 bytes
Tags found: 1

Driver: usbhub.sys
NonPagedPool: 103472 bytes
PagedPool: 416 bytes
Tags found: 7

Driver: avc.sys
NonPagedPool: 3792 bytes
PagedPool: 0 bytes
Tags found: 2

Driver: HIDCLASS.SYS
NonPagedPool: 27184 bytes
PagedPool: 96 bytes
Tags found: 3

Driver: AmUStor.SYS
NonPagedPool: 1424 bytes
PagedPool: 0 bytes
Tags found: 1

Driver: HTTP.sys
NonPagedPool: 117664 bytes
PagedPool: 9424 bytes
Tags found: 2

Driver: mrxsmb.sys
NonPagedPool: 24816 bytes
PagedPool: 4384 bytes
Tags found: 2

Driver: srvnet.sys
NonPagedPool: 503312 bytes
PagedPool: 406048 bytes
Tags found: 101

Driver: srv2.sys
NonPagedPool: 75088 bytes
PagedPool: 1568 bytes
Tags found: 23

Driver: tdtcp.sys
NonPagedPool: 3936 bytes
PagedPool: 0 bytes
Tags found: 12

There are no Irp tags anymore with driver verifier active (well, with the options I've enabled), but Irpt and Irp+, however Irpt tags cannot be found in memory dump at all, but Irp+ tags have the majority of memory allocations anyway based of poolmon output.

So I'm at a loss right now. I've exhausted all the troubleshooting options I knew about and still I have no clue what is the damned thing that is leaking the memory. I had some wild thoughts that it could be something hardware related, but when I boot to an XP maintenance partition there is no memory leak whatsoever.

Are there any kernel/drivers gurus here that can help me fix the memory leak issue?
Thanks.
 
Back
Top