Serious memory leak in nonpaged pool (Irp tag)

Discussion in 'Windows 7 Help and Support' started by Rafcio, Oct 5, 2012.

  1. Rafcio

    Rafcio Active Member

    Joined:
    Oct 5, 2012
    Messages:
    7
    Likes Received:
    0
    I have a serious memory leak in nonpaged pool, which happen to be Irp packets. Here is the whole story.

    The system is Win7 Ultimate x64 which I primarily use to host some VirtualBox VMs. Around beginning of May I noticed that the box started to lock up and it was rock solid for almost a year before then. I discovered that the reason for lock ups is a memory leak that east up its 16 GB of memory in a few days. I started digging deeper and figured out that the nonpaged memory grows from the typical 500 MB or so to a few GBs in a couple of days. The poolmon tool pointed to Irp tag as the clear offender.

    I thought it could be some update or new driver that got installed, so I restored the system from an earlier image. I went back to images as far back as November, October and September last year, but nothing helped. The system was working fine till about end of April, so I was very surprised that reverting back to the time the system was working OK did not fix the problem.

    Anyway, further troubleshooting with driver verifier (log file analysis) did not point to any driver with suspicious amount of allocated memory. So the next step was to force a memory dump with driver verifier running and use !verifier kernel debugger extension to see the memory allocated.

    First of all, driver verifier puts a lot of load on the system, so it stops responding after few hours with CPUs pegged at 100%. Also, the system is much slower when driver verifier is running and the memory leak happens at the much slower rate.

    I recently forced a memory dump after about 8 hours of system uptime. The nonpaged memory was about 915 MB, so I'd expected a clear indication of what driver has plenty of memory allocated. Unfortunately not so.

    The !verifier 1 provided this output:

    Verify Level 418 ... enabled options are:
    All pool allocations checked on unload
    Io subsystem checking enabled
    IRP Logging

    Summary of All Verifier Statistics

    RaiseIrqls 0x0
    AcquireSpinLocks 0x18819b3d
    Synch Executions 0x7b1a30
    Trims 0x0

    Pool Allocations Attempted 0x474127c6
    Pool Allocations Succeeded 0x474127c6
    Pool Allocations Succeeded SpecialPool 0x47589c
    Pool Allocations With NO TAG 0xa
    Pool Allocations Failed 0x0
    Resource Allocations Failed Deliberately 0x0

    Current paged pool allocations 0x18a9a for 08CA3440 bytes
    Peak paged pool allocations 0x27a61 for 0A99EE20 bytes
    Current nonpaged pool allocations 0x1916e for 039C57D0 bytes
    Peak nonpaged pool allocations 0x19500 for 03AECF70 bytes

    Driver Verification List

    Entry State NonPagedPool PagedPool Module

    fffffa800cb8d880 Loaded 00036530 00000090 hal.dll
    fffffa800cb91740 Loaded 00000000 00000000 kdcom.dll
    fffffa800cafd200 Loaded 00000000 00000000 mcupdate.dll
    fffffa800cafd040 Loaded 00000000 00000000 PSHED.dll
    fffffa800cb90740 Loaded 000196c0 000f2710 CLFS.SYS
    fffffa800cb90580 Loaded 00000000 00305a80 CI.dll
    fffffa800cb9af50 Loaded 00064e90 00003660 Wdf01000.sys
    fffffa800cb9ad90 Loaded 00000650 000002d0 WDFLDR.SYS
    fffffa800cb9abd0 Loaded 00090bc0 000050a0 ACPI.sys
    fffffa800cb9aa10 Loaded 00000000 00000000 WMILIB.SYS
    fffffa800cb9a820 Loaded 00000000 00000000 msisadrv.sys
    fffffa800cb9a660 Loaded 0000e110 00015830 pci.sys
    fffffa800cb9a470 Loaded 00000000 00000000 vdrvroot.sys
    fffffa800cb9a290 Loaded 00004ec0 00000080 partmgr.sys
    fffffa800cb9a1b0 Loaded 00000000 00000000 compbatt.sys
    fffffa800cba3e50 Loaded 000001a0 00000130 BATTC.SYS
    fffffa800cba3c70 Loaded 00000110 00000500 volmgr.sys
    fffffa800cba3a90 Loaded 00008140 00004050 volmgrx.sys
    fffffa800cba38b0 Loaded 00000000 00000000 pciide.sys
    fffffa800cba36d0 Loaded 00000000 00000050 PCIIDEX.SYS
    fffffa800cba3510 Loaded 000003d0 00000000 jraid.sys
    fffffa800cba3320 Loaded 00007350 00000190 SCSIPORT.SYS
    fffffa800cba3130 Loaded 00000000 00002110 mountmgr.sys
    fffffa800cba4f40 Loaded 00000000 00000000 vmbus.sys
    fffffa800cba4d60 Loaded 000020e0 00000000 winhv.sys
    fffffa800cba4b80 Loaded 00000000 00000000 atapi.sys
    fffffa800cba49a0 Loaded 00008ec0 00000000 ataport.SYS
    fffffa800cba47e0 Loaded 00000000 00000000 amdxata.sys
    fffffa800cba4600 Loaded 00847040 00ae4440 fltmgr.sys
    fffffa800cba4410 Loaded 00000ba0 00000620 fileinfo.sys
    fffffa800cba4230 Loaded 0001abf0 00000a80 stcvsm.sys
    fffffa800cba5fa0 Loaded 004980f0 019f7d10 Ntfs.sys
    fffffa800cba5dc0 Loaded 00012250 0000b2b0 msrpc.sys
    fffffa800cba5bb0 Loaded 000002a0 00004560 ksecdd.sys
    fffffa800cba59d0 Loaded 00010ae0 00000070 cng.sys
    fffffa800cba57f0 Loaded 00000000 00002570 pcw.sys
    fffffa800cba5610 Loaded 00000020 00000000 Fs_Rec.sys
    fffffa800cba5430 Loaded 005015a0 00000420 ndis.sys
    fffffa800cba5250 Loaded 00125a50 00000000 NETIO.SYS
    fffffa800cba5070 Loaded 00000070 00000b00 ksecpkg.sys
    fffffa800cba6f40 Loaded 00109730 00000000 tcpip.sys
    fffffa800cba6d50 Loaded 000002e0 000030b0 fwpkclnt.sys
    fffffa800cba6b60 Loaded 00000000 00000000 vmstorfl.sys
    fffffa800cba6980 Loaded 0043c6e0 000001f0 volsnap.sys
    fffffa800cba67a0 Loaded 00000000 00000000 spldr.sys
    fffffa800cba6580 Loaded 000325c0 00000000 rdyboost.sys
    fffffa800cba63a0 Loaded 00000000 00000000 NBVol.sys
    fffffa800cba61c0 Loaded 00000000 00000000 NBVolUp.sys
    fffffa800cba7fa0 Loaded 00000f20 00000410 mup.sys
    fffffa800cba7db0 Loaded 00000000 00000000 hwpolicy.sys
    fffffa800cba7bd0 Loaded 000003e0 00000000 fvevol.sys
    fffffa800cba7a10 Loaded 000000c0 00000060 disk.sys
    fffffa800cba77f0 Loaded 00035680 00000200 CLASSPNP.SYS
    fffffa800cba7600 Loaded 00000000 00000000 AtiPcie64.sys
    fffffa800cba7410 Loaded 00000000 00000000 ahcix64s.sys
    fffffa800cba7220 Loaded 00bdf820 000001b0 storport.sys
    fffffa800f9035d0 Loaded&Unloaded 000dd3b0 00000000 crashdmp.sys
    fffffa800f5fb900 Loaded&Unloaded 00000000 00000000 dump_storport.sys
    fffffa800f906920 Loaded&Unloaded 00000000 00000000 dump_ahcix64s.sys
    fffffa800f4bfe70 Loaded&Unloaded 00004010 00000000 dump_dumpfve.sys
    fffffa800f958010 Loaded 000137e0 00000030 cdrom.sys
    fffffa800fc59150 Loaded 00005560 0320d390 SRTSP64.SYS
    fffffa800f92f7f0 Loaded&Unloaded 000016c0 00000000 EX64.SYS
    fffffa800f914570 Loaded 00016b90 0000de90 SYMEVENT64x86.SYS
    fffffa800f985590 Loaded&Unloaded 00000000 00000000 ENG64.SYS
    fffffa800fc4b030 Loaded 00000350 00000260 SRTSPX64.SYS
    fffffa800fc60030 Loaded 00000000 00000000 Null.SYS
    fffffa800fc62030 Loaded 00000000 00000000 Beep.SYS
    fffffa800fb6c030 Loaded 00005180 00000060 watchdog.sys
    fffffa800f936560 Loaded 000000c0 00000970 VIDEOPRT.SYS
    fffffa800fb5d030 Loaded 00000000 000010f0 vga.sys
    fffffa800f9b3480 Loaded 00000000 00000000 RDPCDD.sys
    fffffa800f998480 Loaded 00000000 00000000 rdpencdd.sys
    fffffa800fc60480 Loaded 00000000 00000000 rdprefmp.sys
    fffffa800fb5d490 Loaded 00000370 000005d0 Msfs.SYS
    fffffa800fc5b480 Loaded 00000f40 0000ea30 Npfs.SYS
    fffffa800fb72490 Loaded 00004870 00000000 TDI.SYS
    fffffa800fb6e490 Loaded 00165260 00000000 tdx.sys
    fffffa800faf5060 Loaded 00207000 00000000 wpsdrvnt.sys
    fffffa800fac7510 Loaded 0009cac0 00005d40 afd.sys
    fffffa800fa4b0c0 Loaded 00024590 00000000 netbt.sys
    fffffa800fadc8c0 Loaded 000006d0 00000000 wfplwf.sys
    fffffa800fb5f030 Loaded 000015b0 00000000 pacer.sys
    fffffa800f4407a0 Loaded 00002b30 00000000 vpcnfltr.sys
    fffffa800f8f57e0 Loaded 000011f0 00000000 netbios.sys
    fffffa800fab28b0 Loaded 00000260 00000000 wanarp.sys
    fffffa800fd84040 Loaded 0000a480 00000080 vpcvmm.sys
    fffffa800faae500 Loaded 00000000 00000000 VBoxUSBMon.sys
    fffffa800fbb2040 Loaded 0000db80 00000000 VBoxDrv.sys
    fffffa800fae4510 Loaded 000042c0 00000000 termdd.sys
    fffffa800fbc3030 Loaded 00000000 00000000 SCDEmu.SYS
    fffffa800fbc5030 Loaded 00000000 00000020 sbmount.SYS
    fffffa800fbf3030 Loaded 000064e0 00003e70 rdbss.sys
    fffffa800fba94f0 Loaded 00001d10 00000000 nsiproxy.sys
    fffffa800fbe20a0 Loaded 00005260 00000080 mssmbios.sys
    fffffa800fbb6470 Loaded 00001010 00001cf0 eeCtrl64.sys
    fffffa800fb79e30 Loaded 00000f80 0000a9e0 EraserUtilRebootDrv.sys
    fffffa800fb795f0 Loaded 00000000 000003f0 discache.sys
    fffffa800fada4a0 Loaded 00002070 000027e0 csc.sys
    fffffa800fada570 Loaded 000003e0 00000090 dfsc.sys
    fffffa800fb2f6c0 Loaded 00000000 00000000 blbdrive.sys
    fffffa800fde8030 Loaded 00000000 00000000 AsUpIO.sys
    fffffa800fb70490 Loaded 00000000 00000000 AsIO.sys
    fffffa800fdfb030 Loaded 000000c0 00000000 tunnel.sys
    fffffa800fb61aa0 Loaded 00000bc0 00000000 amdppm.sys
    fffffa800fb2f600 Loaded 0000c170 0004a450 atikmpag.sys
    fffffa800fe1c8b0 Loaded 00167540 0120cd70 atikmdag.sys
    fffffa800fe59030 Loaded 000107b0 01d776a0 dxgkrnl.sys
    fffffa800fb964a0 Loaded 000c3f10 000d5ab0 dxgmms1.sys
    fffffa800fbbe490 Loaded 0000e860 000000a0 HDAudBus.sys
    fffffa800fbe6c60 Loaded 00000000 00000000 USBD.SYS
    fffffa800fdd3750 Loaded 0007ebc0 000000f0 nusb3xhc.sys
    fffffa800fade480 Loaded 00000000 00000000 usbfilter.sys
    fffffa800fe30030 Loaded 00048e80 00000150 USBPORT.SYS
    fffffa800fdff490 Loaded 00000000 00000000 usbohci.sys
    fffffa800fdbb8b0 Loaded 00000000 00000000 usbehci.sys
    fffffa800fe57040 Loaded 00000000 00000000 ASACPI.sys
    fffffa800fe26030 Loaded 00002010 00000000 i8042prt.sys
    fffffa800fe99030 Loaded 000000d0 00000000 L8042Kbd.sys
    fffffa800fe99090 Loaded 00001a60 00000000 kbdclass.sys
    fffffa800fe69030 Loaded 000005b0 00000000 L8042mou.Sys
    fffffa800fa8f030 Loaded 00000ff0 00000000 LMouKE.Sys
    fffffa800fdcc8b0 Loaded 000023c0 00000000 mouclass.sys
    fffffa800fe5d030 Loaded 0000da40 00000000 1394ohci.sys
    fffffa800feb6030 Loaded 0009bf50 00000000 Rt64win7.sys
    fffffa800fdd5490 Loaded 00000030 000000b0 wmiacpi.sys
    fffffa800fdbf480 Loaded 00000000 00000000 CompositeBus.sys
    fffffa800fe26980 Loaded 00000020 00000000 AgileVpn.sys
    fffffa800fe34310 Loaded 00000040 00000000 rasl2tp.sys
    fffffa800feac060 Loaded 00000000 00000000 ndistapi.sys
    fffffa800fed5030 Loaded 00002830 00000000 ndiswan.sys
    fffffa80109b1930 Loaded 00000000 00000000 raspppoe.sys
    fffffa800feb4040 Loaded 00000040 00000000 raspptp.sys
    fffffa800fe52490 Loaded 000003b0 00000000 rassstp.sys
    fffffa800feb0560 Loaded 00000000 000000d0 teamviewervpn.sys
    fffffa800fe598b0 Loaded 000003f0 00000000 VBoxNetAdp.sys
    fffffa800fe9f4c0 Loaded 00000000 00000000 rdpbus.sys
    fffffa800feb09b0 Loaded 00000d80 00000000 VBoxNetFlt.sys
    fffffa80109e3970 Loaded 001c3c80 00000000 teefer2.sys
    fffffa800feb0e70 Loaded 00001200 00002540 ks.sys
    fffffa800ffaa030 Loaded 00000000 00000000 swenum.sys
    fffffa800ff15040 Loaded 00000000 00000080 amdiox64.sys
    fffffa800ff30040 Loaded 00001a10 00000000 umbus.sys
    fffffa800ff5e030 Loaded 000002e0 000004e0 usbrpm.sys
    fffffa800ff7f030 Loaded 00000000 00000000 vpcusb.sys
    fffffa800ff49040 Loaded 000049e0 00000000 vpchbus.sys
    fffffa800ffd7030 Loaded 00000a30 00000080 nusb3hub.sys
    fffffa800ffcb030 Loaded 00019430 000001a0 usbhub.sys
    fffffa8010ac8030 Loaded 00004e90 00000000 NDProxy.SYS
    fffffa8010c46230 Loaded 000000c0 000000b0 drmk.sys
    fffffa8010c20890 Loaded 00004930 00007920 portcls.sys
    fffffa8010c3cf20 Loaded 0000bf80 00000000 RtHDMIVX.sys
    fffffa8010c6b7a0 Loaded 000001d0 00000000 ksthunk.sys
    fffffa8010c6b570 Loaded 000294b0 00000e60 viahduaa.sys
    fffffa8010ca0480 Loaded 00002760 00000000 61883.sys
    fffffa8010d1c550 Loaded 00000ed0 00000000 avc.sys
    fffffa8010d71450 Loaded 00000710 00000050 STREAM.SYS
    fffffa80111e1040 Loaded 00001920 00000000 msdv.sys
    fffffa80116ea030 Loaded 00002e00 00000000 HIDPARSE.SYS
    fffffa8010d914d0 Loaded 00006a30 00000060 HIDCLASS.SYS
    fffffa800ffa8030 Loaded 00001200 00000000 hidusb.sys
    fffffa80116ea280 Loaded 00000590 00000000 AmUStor.SYS
    fffffa8011087030 Loaded 00000000 00000000 Dxapi.sys
    fffffa8011252030 Loaded 00014520 000000c0 win32k.sys
    fffffa80114561a0 Loaded 00000000 00000000 monitor.sys
    fffffa8011bd4b90 Loaded&Unloaded 00000000 00000000 TSDDD.dll
    fffffa800f96b060 Loaded&Unloaded 00001030 00001010 cdd.dll
    fffffa801100baf0 Loaded 00000000 00017dc0 luafv.sys
    fffffa80111d18c0 Loaded 00000cc0 00018460 PDFsFilter.sys
    fffffa8010ffae30 Loaded 0000eaa0 00000000 WudfPf.sys
    fffffa80112437e0 Loaded 00000000 00000000 DefragFS.SYS
    fffffa80116c8780 Loaded 00000040 00000000 lltdio.sys
    fffffa80115bc520 Loaded 00000000 00000000 pnarp.sys
    fffffa8011720af0 Loaded 00000000 00000000 purendis.sys
    fffffa80117bf030 Loaded 00000090 00000090 rspndr.sys
    fffffa80118bb6c0 Loaded 0001cba0 000024d0 HTTP.sys
    fffffa8010907a10 Loaded 00000ce0 000008d0 bowser.sys
    fffffa8011520220 Loaded 000000e0 00000000 mpsdrv.sys
    fffffa8010968730 Loaded 000060f0 00001120 mrxsmb.sys
    fffffa8011711bb0 Loaded 00001050 00000f00 mrxsmb10.sys
    fffffa80109cfaf0 Loaded 00000000 00000000 mrxsmb20.sys
    fffffa8011c460e0 Loaded 00000000 00000000 AODDriver2.sys
    fffffa80104ac220 Loaded&Unloaded 006fa270 00000000 WpsHelper.sys
    fffffa80118dc5a0 Loaded 00000000 00000000 cpuz133_x64.sys
    fffffa8011d46a40 Loaded 00000000 00000000 cpuz135_x64.sys
    fffffa8012276dd0 Loaded 00000000 000000b0 peauth.sys
    fffffa8010442180 Loaded 00000000 00000050 secdrv.SYS
    fffffa8011c50f00 Loaded 0007ae10 00063220 srvnet.sys
    fffffa8012357bb0 Loaded 00006680 00000000 tcpipreg.sys
    fffffa8010c19470 Loaded 00012550 00000620 srv2.sys
    fffffa8012aef8b0 Loaded 0001a3a0 000018e0 srv.sys
    fffffa80123b5c00 Loaded 00000000 00000000 TuneUpUtilitiesDriver64.sys
    fffffa800d6aa730 Loaded 00006210 00000000 WUDFRd.sys
    fffffa8012b230b0 Loaded 000004a0 00000070 rdpdr.sys
    fffffa80116da920 Loaded 00000f60 00000000 tdtcp.sys
    fffffa80111febd0 Loaded 00000000 00000000 tssecsrv.sys
    fffffa8011186b50 Loaded 000000a0 00002c20 RDPWD.SYS
    fffffa80112a59d0 Unloaded 00000000 00000000 spsys.sys
    fffffa800dfbd180 Loaded 00000110 00000000 asyncmac.sys
    fffffa8012b76010 Loaded 00000000 00000000 myfault.sys

    I can't provide the !verifier 3 output, because it's 12 MB, but a script I wrote to provide certain statistics from that output for Irp+ tag listed this:

    Driver: CLFS.SYS
    NonPagedPool: 104128 bytes
    PagedPool: 993040 bytes
    Tags found: 10

    Driver: Wdf01000.sys
    NonPagedPool: 413328 bytes
    PagedPool: 13920 bytes
    Tags found: 4

    Driver: partmgr.sys
    NonPagedPool: 20160 bytes
    PagedPool: 128 bytes
    Tags found: 6

    Driver: SCSIPORT.SYS
    NonPagedPool: 29520 bytes
    PagedPool: 400 bytes
    Tags found: 2

    Driver: fltmgr.sys
    NonPagedPool: 8679488 bytes
    PagedPool: 11420736 bytes
    Tags found: 1

    Driver: volsnap.sys
    NonPagedPool: 4441824 bytes
    PagedPool: 496 bytes
    Tags found: 4

    Driver: CLASSPNP.SYS
    NonPagedPool: 218752 bytes
    PagedPool: 512 bytes
    Tags found: 155

    Driver: cdrom.sys
    NonPagedPool: 79840 bytes
    PagedPool: 48 bytes
    Tags found: 2

    Driver: SRTSP64.SYS
    NonPagedPool: 21856 bytes
    PagedPool: 52482960 bytes
    Tags found: 2

    Driver: netbt.sys
    NonPagedPool: 148880 bytes
    PagedPool: 0 bytes
    Tags found: 1

    Driver: LMouKE.Sys
    NonPagedPool: 4080 bytes
    PagedPool: 0 bytes
    Tags found: 1

    Driver: nusb3hub.sys
    NonPagedPool: 2608 bytes
    PagedPool: 128 bytes
    Tags found: 1

    Driver: usbhub.sys
    NonPagedPool: 103472 bytes
    PagedPool: 416 bytes
    Tags found: 7

    Driver: avc.sys
    NonPagedPool: 3792 bytes
    PagedPool: 0 bytes
    Tags found: 2

    Driver: HIDCLASS.SYS
    NonPagedPool: 27184 bytes
    PagedPool: 96 bytes
    Tags found: 3

    Driver: AmUStor.SYS
    NonPagedPool: 1424 bytes
    PagedPool: 0 bytes
    Tags found: 1

    Driver: HTTP.sys
    NonPagedPool: 117664 bytes
    PagedPool: 9424 bytes
    Tags found: 2

    Driver: mrxsmb.sys
    NonPagedPool: 24816 bytes
    PagedPool: 4384 bytes
    Tags found: 2

    Driver: srvnet.sys
    NonPagedPool: 503312 bytes
    PagedPool: 406048 bytes
    Tags found: 101

    Driver: srv2.sys
    NonPagedPool: 75088 bytes
    PagedPool: 1568 bytes
    Tags found: 23

    Driver: tdtcp.sys
    NonPagedPool: 3936 bytes
    PagedPool: 0 bytes
    Tags found: 12

    There are no Irp tags anymore with driver verifier active (well, with the options I've enabled), but Irpt and Irp+, however Irpt tags cannot be found in memory dump at all, but Irp+ tags have the majority of memory allocations anyway based of poolmon output.

    So I'm at a loss right now. I've exhausted all the troubleshooting options I knew about and still I have no clue what is the damned thing that is leaking the memory. I had some wild thoughts that it could be something hardware related, but when I boot to an XP maintenance partition there is no memory leak whatsoever.

    Are there any kernel/drivers gurus here that can help me fix the memory leak issue?
    Thanks.
     

Share This Page

Loading...