Hello everyone, I am Feng Hui, a researcher in this issue of Microsoft MVP Labs. This article mainly introduces how to use Windbg to analyze the memory problems in the application process, from the exploration of managed heap to unmanaged heap and the allocation of memory, let's explore together.
Recently, several friends have used our 161d2e54c9b0ee that the memory has skyrocketed during the export process. Next, let’s use windbg to see what caused it.
Let's first look at the current application memory footprint
address -summary
0:000> !address -summary
--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free 581 7df8`ef0c9000 ( 125.972 TB) 98.42%
<unknown> 1678 206`ffb9e000 ( 2.027 TB) 99.99% 1.58%
Image 950 0`064fd000 ( 100.988 MB) 0.00% 0.00%
Heap 58 0`050f6000 ( 80.961 MB) 0.00% 0.00%
Stack 156 0`04380000 ( 67.500 MB) 0.00% 0.00%
Other 11 0`019ad000 ( 25.676 MB) 0.00% 0.00%
TEB 52 0`00068000 ( 416.000 kB) 0.00% 0.00%
PEB 1 0`00001000 ( 4.000 kB) 0.00% 0.00%
--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_MAPPED 282 200`038a6000 ( 2.000 TB) 98.64% 1.56%
MEM_PRIVATE 1674 7`07184000 ( 28.111 GB) 1.35% 0.02%
MEM_IMAGE 950 0`064fd000 ( 100.988 MB) 0.00% 0.00%
--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE 581 7df8`ef0c9000 ( 125.972 TB) 98.42%
MEM_RESERVE 295 205`f8659000 ( 2.023 TB) 99.79% 1.58%
MEM_COMMIT 2611 1`188ce000 ( 4.384 GB) 0.21% 0.00%
--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_READWRITE 1595 1`0dc6c000 ( 4.215 GB) 0.20% 0.00%
PAGE_EXECUTE_READ 156 0`04d66000 ( 77.398 MB) 0.00% 0.00%
PAGE_READONLY 600 0`03851000 ( 56.316 MB) 0.00% 0.00%
PAGE_NOACCESS 99 0`021f2000 ( 33.945 MB) 0.00% 0.00%
PAGE_EXECUTE_READWRITE 19 0`0027b000 ( 2.480 MB) 0.00% 0.00%
PAGE_WRITECOPY 90 0`001a0000 ( 1.625 MB) 0.00% 0.00%
PAGE_READWRITE | PAGE_GUARD 52 0`0009e000 ( 632.000 kB) 0.00% 0.00%
--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free 189`0413c000 7c6b`01ed4000 ( 124.418 TB)
<unknown> 7dfb`2a153000 1f9`bd2ef000 ( 1.976 TB)
Image 7ffc`883c1000 0`009ba000 ( 9.727 MB)
Heap 183`0e9a1000 0`00f01000 ( 15.004 MB)
Stack 37`62980000 0`0017b000 ( 1.480 MB)
Other 183`77707000 0`01775000 ( 23.457 MB)
TEB 37`62600000 0`00002000 ( 8.000 kB)
PEB 37`627dd000 0`00001000 ( 4.000 kB)
MEM_COMMIT occupies 4.384G, then we use
eeheap -gc
to check the managed heap.
0:000> !eeheap -gc
GC Allocated Heap Size: Size: 0x11ac2568 (296494440) bytes.
GC Committed Heap Size: Size: 0x120e7000 (302936064) bytes.According to the memory, it seems that the problem is not here, a large amount of memory still appears in unmanaged. Let’s take a look at the Windows NT heap. In fact, most user heap allocators in Windows are built on the NT heap manager API (RtlAllocateHeap/RtlFreeHeap) in ntdll.dll, such as malloc/free and new in C. /delete, there are also SysAllocString in the COM framework and LocalAlloc, GlobalAlloc and HeapAlloc in Win32. Although these allocators will create different heaps to store their memory, they will eventually call NT in ntdll.dll Heap to achieve.
0:000> !heap -s
************************************************************************************************************************
NT HEAP STATS BELOW
************************************************************************************************************************
NtGlobalFlag enables following debugging aids for new heaps:
stack back traces
LFH Key : 0x7cfd4cc2db4ddb4d
Termination on corruption : ENABLED
Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast
(k) (k) (k) (k) length blocks cont. heap
-------------------------------------------------------------------------------------
0000018378fd0000 08000002 65128 15296 64928 1720 177 17 2 c LFH
External fragmentation 11 % (177 free blocks)
00000183775c0000 08008000 64 4 64 2 1 1 0 0
000001837aa90000 08001002 1280 108 1080 26 3 2 0 0 LFH
000001837ad20000 08001002 60 8 60 2 1 1 0 0
000001837aca0000 08041002 60 8 60 5 1 1 0 0
000001887bfd0000 08001002 60 20 60 1 2 1 0 0
000001830cf30000 08001002 3324 1364 3124 19 10 3 0 0 LFH
000001830ce30000 08001002 60 8 60 5 1 1 0 0
-------------------------------------------------------------------------------------The output results are as shown above, the NT heap content is so small... What is the reason... Okay, according to Maoni, it seems that there is a problem with the verification.
All user mode allocations on Windows ultimately obtain memory through VirtualAlloc, whether it is bitmaps or GC heap.
The difference is whether you call VirtualAlloc directly or use other methods to call it. If it is unmanaged memory, GC does not control it, and of course it does not know their existence.
GC does not manage these memory, so the code we wrote is still problematic. Let's go back and consider one more thing, "When the export is in progress, the memory will increase a lot, and the memory will decrease after the export is completed." Let's take a look at the code, as shown below. In fact, what we now understand is that these memories must have been "held" during our execution and have not been released.
app.MapGet("/excel", async content =>
string path = Path.Combine(Directory.GetCurrentDirectory(), "test.xlsx");
List<TestDto> list = new();
for (int i = 0; i < 400; i++)
list.Add(new TestDto
ImageUrl = "https://gimg2.baidu.com/image_search/src=http%3A%2F%2Fup.enterdesk.com%2Fedpic_source%2F53%2F0a%2Fda%2F530adad966630fce548cd408237ff200.jpg&refer=http%3A%2F%2Fup.enterdesk.com&app=2002&size=f9999,10000&q=a80&n=0&g=0n&fmt=jpeg?sec=1641193100&t=417a589da8c9ba3103ed74c33fbd6c70"
Stopwatch stopwatch = Stopwatch.StartNew();
ExcelExporter exporter = new ExcelExporter();
await exporter.Export(path, list);
stopwatch.Stop();
await content.Response.WriteAsync(stopwatch.Elapsed.TotalSeconds.ToString());
});According to the performance of memory and our theory, we continue to use Windbg to investigate. Now we can actually find that these objects were eventually recovered by the GC. With the theory, we continue to conceive. The GC knows which objects can be terminated, right? And they call their finalizers when they become unreachable, and use the finalization queue to record these finalized objects in the GC. So can we check it? As shown below, let's take a look.
0:000> !finalizequeue
----------------------------------
Statistics for all finalizable objects (including all objects ready for finalization):
MT Count TotalSize Class Name
00007ffc2dc23818 1 24 System.Net.Security.SafeCredentialReference
00007ffc2dac4238 1 24 System.WeakReference
00007ffc2d6eb908 1 24 System.WeakReference`1[[Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerOptions, Microsoft.AspNetCore.Server.Kestrel.Core]]
00007ffc2d6e4120 1 24 System.WeakReference`1[[System.Runtime.Loader.AssemblyLoadContext, System.Private.CoreLib]]
00007ffc2d572b68 1 24 System.WeakReference`1[[Microsoft.Extensions.DependencyInjection.ServiceProvider, Microsoft.Extensions.DependencyInjection]]
00007ffc2d429258 1 24 System.WeakReference`1[[System.IO.FileSystemWatcher, System.IO.FileSystem.Watcher]]
00007ffc2dd15c20 1 32 Microsoft.Win32.SafeHandles.SafeBCryptAlgorithmHandle
00007ffc2d6de4d8 1 32 Internal.Cryptography.Pal.Native.SafeLocalAllocHandle
00007ffc2d68fa00 1 32 Internal.Cryptography.Pal.Native.SafeCertStoreHandle
00007ffc2d3a5cc0 1 32 System.Net.Quic.Implementations.MsQuic.Internal.SafeMsQuicRegistrationHandle
00007ffc2db390c8 1 40 Interop+WinHttp+SafeWinHttpHandle
00007ffc2d69a420 1 40 Internal.Cryptography.Pal.Native.SafeCertContextHandle
00007ffc2d5bea18 1 40 System.Diagnostics.EventLog
00007ffc2dc29a38 1 48 System.Net.Security.SafeFreeCredential_SECURITY
00007ffc2d963f80 2 48 System.WeakReference`1[[System.Text.RegularExpressions.RegexReplacement, System.Text.RegularExpressions]]
00007ffc2d7a3750 2 48 System.WeakReference`1[[Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection, Microsoft.AspNetCore.Server.Kestrel.Core]]
00007ffc2d685e10 1 56 System.Runtime.CompilerServices.ConditionalWeakTable`2+Container[[System.Buffers.TlsOverPerCoreLockedStacksArrayPool`1+ThreadLocalArray[[System.Char, System.Private.CoreLib]][], System.Private.CoreLib],[System.Object, System.Private.CoreLib]]
00007ffc2d44c4d0 1 56 System.Runtime.CompilerServices.ConditionalWeakTable`2+Container[[System.Buffers.TlsOverPerCoreLockedStacksArrayPool`1+ThreadLocalArray[[System.Byte, System.Private.CoreLib]][], System.Private.CoreLib],[System.Object, System.Private.CoreLib]]
00007ffc2d96be68 1 64 CellStore`1[[System.Uri, System.Private.Uri]]
00007ffc2d96b780 1 64 FlagCellStore
00007ffc2d96af48 1 64 CellStore`1[[System.Object, System.Private.CoreLib]]
00007ffc2d96a5b8 1 64 CellStore`1[[OfficeOpenXml.ExcelCoreValue, Magicodes.IE.EPPlus]]
00007ffc2d6ddab8 2 64 Internal.Cryptography.Pal.Native.SafeChainEngineHandle
00007ffc2d69d528 2 64 Internal.Win32.SafeHandles.SafeRegistryHandle
00007ffc2d685bc8 2 64 Microsoft.Win32.SafeHandles.SafeWaitHandle
00007ffc2d685280 3 72 System.Threading.ThreadInt64PersistentCounter+ThreadLocalNodeFinalizationHelper
00007ffc2d5f5f50 3 72 System.Runtime.InteropServices.PosixSignalRegistration
00007ffc2d4299d0 1 72 Microsoft.Win32.SafeHandles.SafeFileHandle
00007ffc2d6e40b8 1 80 System.Runtime.Loader.DefaultAssemblyLoadContext
00007ffc2dac9ed0 2 96 PageIndex
00007ffc2d96d0c8 2 96 ColumnIndex
00007ffc2d464470 3 120 System.Gen2GcCallback
00007ffc2d40a620 1 120 System.IO.FileSystemWatcher
00007ffc2d96bc18 2 128 CellStore`1[[System.Int32, System.Private.CoreLib]]
00007ffc2dac20c8 2 144 System.Reflection.Emit.DynamicResolver
00007ffc2d680f10 3 144 System.Threading.LowLevelLock
00007ffc2d683c48 3 168 System.Threading.ThreadPoolWorkQueueThreadLocals
00007ffc2d681e80 1 176 System.Threading.LowLevelLifoSemaphore
00007ffc2dc25ef0 1 184 System.Collections.Concurrent.CDSCollectionETWBCLProvider
00007ffc2db8e658 1 184 System.Net.NetEventSource
00007ffc2db8c378 1 184 System.Net.NetEventSource
00007ffc2db38f90 1 184 System.Net.NetEventSource
00007ffc2d90c658 1 184 Microsoft.IO.RecyclableMemoryStreamManager+Events
00007ffc2d689b48 1 184 Microsoft.AspNetCore.Certificates.Generation.CertificateManager+CertificateManagerEventSource
00007ffc2d66f9f8 1 184 System.Diagnostics.Tracing.FrameworkEventSource
00007ffc2d66b720 1 184 System.Net.NetEventSource
00007ffc2d44d128 1 184 System.Buffers.ArrayPoolEventSource
00007ffc2d2e2ec8 1 184 System.Diagnostics.Tracing.NativeRuntimeEventSource
00007ffc2d694e10 1 192 System.Threading.Tasks.TplEventSource
00007ffc2d572ab0 1 192 Microsoft.Extensions.DependencyInjection.DependencyInjectionEventSource
00007ffc2d505f00 1 200 Microsoft.Extensions.Logging.EventSource.LoggingEventSource
00007ffc2db8ade8 1 224 System.Net.NameResolutionTelemetry
00007ffc2d428b08 7 224 System.Threading.PreAllocatedOverlapped
00007ffc2d563c78 1 232 System.Diagnostics.DiagnosticSourceEventSource
00007ffc2d61fe88 1 240 Microsoft.AspNetCore.Hosting.HostingEventSource
00007ffc2db6b788 8 256 System.Threading.TimerQueue+AppDomainTimerSafeHandle
00007ffc2d690270 1 280 System.Net.Sockets.SocketsTelemetry
00007ffc2db6bc80 1 296 System.Net.Http.HttpTelemetry
00007ffc2d68b998 1 336 Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelEventSource
00007ffc2dc21998 1 360 System.Net.Security.NetSecurityTelemetry
00007ffc2d2dae28 1 384 System.Diagnostics.Tracing.RuntimeEventSource
00007ffc2d66ad60 10 480 System.Net.Sockets.SafeSocketHandle
00007ffc2d2e0240 21 504 System.WeakReference`1[[System.Diagnostics.Tracing.EventSource, System.Private.CoreLib]]
00007ffc2d2b0538 9 648 System.Threading.Thread
00007ffc2d77a188 2 704 Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketReceiver
00007ffc2d90cec0 6 960 Microsoft.IO.RecyclableMemoryStream
00007ffc2d5fc658 10 1280 System.Net.Sockets.Socket
00007ffc2d68d898 4 1536 System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs
00007ffc2d2dc778 42 4704 System.Diagnostics.Tracing.EventSource+OverrideEventProvider
00007ffc2daec058 356 14240 System.Drawing.Bitmap
Total 553 objects
WOW! ! ! , Look at the above 356
System.Drawing.Bitmap
waiting for recycling, it seems that this is our influencing factor, let's check the code.
try
cell.Value = string.Empty;
Bitmap bitmap;
if (url.IsBase64StringValid())
bitmap = url.Base64StringToBitmap();
bitmap = Extension.GetBitmapByUrl(url);
if (bitmap == null)
cell.Value = ExporterHeaderList[colIndex].ExportImageFieldAttribute.Alt;
ExcelPicture pic = CurrentExcelWorksheet.Drawings.AddPicture(Guid.NewGuid().ToString(), bitmap);
AddImage((rowIndex + (ExcelExporterSettings.HeaderRowIndex > 1 ? ExcelExporterSettings.HeaderRowIndex : 0)),
colIndex - ignoreCount, pic, ExporterHeaderList[colIndex].ExportImageFieldAttribute.YOffset, ExporterHeaderList[colIndex].ExportImageFieldAttribute.XOffset);
CurrentExcelWorksheet.Row(rowIndex + 1).Height = ExporterHeaderList[colIndex].ExportImageFieldAttribute.Height;
pic.SetSize(ExporterHeaderList[colIndex].ExportImageFieldAttribute.Width * 7, ExporterHeaderList[colIndex].ExportImageFieldAttribute.Height);
catch (Exception)
cell.Value = ExporterHeaderList[colIndex].ExportImageFieldAttribute.Alt;
Use the Bitmap object in the ExcelPicture object. For online image sources, we will read and store it in the Bitmap, but we found that the object was not released, so a large number of Bitmaps have not been released. We use Let's deal with it.
using (ExcelPicture pic = CurrentExcelWorksheet.Drawings.AddPicture(Guid.NewGuid().ToString(), bitmap))
AddImage((rowIndex + (ExcelExporterSettings.HeaderRowIndex > 1 ? ExcelExporterSettings.HeaderRowIndex : 0)),
colIndex - ignoreCount, pic, ExporterHeaderList[colIndex].ExportImageFieldAttribute.YOffset, ExporterHeaderList[colIndex].ExportImageFieldAttribute.XOffset);
CurrentExcelWorksheet.Row(rowIndex + 1).Height = ExporterHeaderList[colIndex].ExportImageFieldAttribute.Height;
pic.SetSize(ExporterHeaderList[colIndex].ExportImageFieldAttribute.Width * 7, ExporterHeaderList[colIndex].ExportImageFieldAttribute.Height);
}A new object with a finalizer must be added to the finalization queue. This behavior is also called "registering for finalization". Of course, I also recommend that you choose to use the SOSEX extension plug-in, which provides content similar to finalization, which seems to be more intuitive, as shown below.
download http://www.stevestechspot.com/default.aspx
:000> .load D:\sosex_64\sosex.dll
This dump has no SOSEX heap index.
The heap index makes searching for references and roots much faster.
To create a heap index, run !bhi
0:000> !finq -stat
Generation 0:
Count Total Size Type
---------------------------------------------------------
54 2160 System.Drawing.Bitmap
54 objects, 2,160 bytes
Generation 1:
Count Total Size Type
---------------------------------------------------------
1 184 Microsoft.AspNetCore.Certificates.Generation.CertificateManager+CertificateManagerEventSource
1 336 Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelEventSource
4 1536 System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs
1 32 Internal.Cryptography.Pal.Native.SafeCertStoreHandle
1 280 System.Net.Sockets.SocketsTelemetry
1 192 System.Threading.Tasks.TplEventSource
1 40 Internal.Cryptography.Pal.Native.SafeCertContextHandle
2 64 Internal.Win32.SafeHandles.SafeRegistryHandle
2 64 Internal.Cryptography.Pal.Native.SafeChainEngineHandle
1 32 Internal.Cryptography.Pal.Native.SafeLocalAllocHandle
1 80 System.Runtime.Loader.DefaultAssemblyLoadContext
1 24 System.WeakReference`1[[System.Runtime.Loader.AssemblyLoadContext, System.Private.CoreLib]]
1 24 System.WeakReference`1[[Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerOptions, Microsoft.AspNetCore.Server.Kestrel.Core]]
2 704 Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketReceiver
2 48 System.WeakReference`1[[Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection, Microsoft.AspNetCore.Server.Kestrel.Core]]
1 184 Microsoft.IO.RecyclableMemoryStreamManager+Events
6 960 Microsoft.IO.RecyclableMemoryStream
2 48 System.WeakReference`1[[System.Text.RegularExpressions.RegexReplacement, System.Text.RegularExpressions]]
1 64 CellStore`1[[OfficeOpenXml.ExcelCoreValue, Magicodes.IE.EPPlus]]
1 64 CellStore`1[[System.Object, System.Private.CoreLib]]
1 64 FlagCellStore
2 128 CellStore`1[[System.Int32, System.Private.CoreLib]]
1 64 CellStore`1[[System.Uri, System.Private.Uri]]
2 96 ColumnIndex
2 144 System.Reflection.Emit.DynamicResolver
1 24 System.WeakReference
2 96 PageIndex
302 12080 System.Drawing.Bitmap
1 184 System.Net.NetEventSource
1 40 Interop+WinHttp+SafeWinHttpHandle
8 256 System.Threading.TimerQueue+AppDomainTimerSafeHandle
1 296 System.Net.Http.HttpTelemetry
1 224 System.Net.NameResolutionTelemetry
1 184 System.Net.NetEventSource
1 184 System.Net.NetEventSource
1 360 System.Net.Security.NetSecurityTelemetry
1 24 System.Net.Security.SafeCredentialReference
1 184 System.Collections.Concurrent.CDSCollectionETWBCLProvider
1 48 System.Net.Security.SafeFreeCredential_SECURITY
1 32 Microsoft.Win32.SafeHandles.SafeBCryptAlgorithmHandle