STL Custom Allocators

Tracking heap leaks - gflags and WinDbg - Create user mode stack trace database

This is example how to trace memory leaks using gflags and WinDbg.
I've some application called heaps1.exe which generates 1000000 (0xF4240) memory leaks elements of size 666 Bytes (0x29A).

1. Enable generating user mode stack trace database. You can also make it from GUI.
C:Program FilesDebugging Tools for Windows (x86)>gflags.exe /i heaps1.exe +ust

2. Run application and generate leaks.

3. Attach WinDbg to your process (heaps1.exe).

4. Load necessary symbols pdb.

5. Break into debugger if necessary.

6. And now magic words:

6a. At first check which heaps is leaking the most.

0:001> !heap -s
NtGlobalFlag enables following debugging aids for new heaps:
    validate parameters
    stack back traces
  Heap     Flags   Reserv  Commit  Virt   Free  List   UCR  Virt  Lock  Fast
                    (k)     (k)    (k)     (k) length      blocks cont. heap
00150000 58000062    1024     76     76     34     2     1    0      0   L 
00250000 58001062      64     24     24     11     1     1    0      0   L 
00260000 58008060      64     12     12     10     1     1    0      0     
00390000 58001062  883776 679796 679796      6     1     1    0      0   L 
003d0000 58001062      64     16     16      0     0     1    0      0   L 

OK. We see that heap 00390000 has outstanding amoutn of allocations.

6b. Let make statistics of this heap.

0:001> !heap -stat -h 00390000
 heap @ 00390000
group-by: TOTSIZE max-display: 20
    size     #blocks     total     ( %) (percent of total busy bytes)
    29a f4240 - 27b25a80  (99.99)
    2012 2 - 4024  (0.00)
    1504 1 - 1504  (0.00)
    800 2 - 1000  (0.00)
    204 3 - 60c  (0.00)
    214 2 - 428  (0.00)
    400 1 - 400  (0.00)
    5c 7 - 284  (0.00)
    16 19 - 226  (0.00)
    220 1 - 220  (0.00)
    1fc 1 - 1fc  (0.00)
    7c 4 - 1f0  (0.00)
    1c e - 188  (0.00)
    34 6 - 138  (0.00)
    98 2 - 130  (0.00)
    28 7 - 118  (0.00)
    88 2 - 110  (0.00)
    80 2 - 100  (0.00)
    fe 1 - fe  (0.00)
    2a 6 - fc  (0.00)
OK. Now we see that almost all allocations have size 0x29a (666 Bytes) and there is 0xf4240 allocations of this size.

6c. Let check who allocated these bytes. There will come gigantic flood. You can break it by pressing <ctrl>+<break>

!heap -flt s 29a

0:001> !heap -flt s 29a
    _HEAP @ 150000
    _HEAP @ 250000
    _HEAP @ 260000
    _HEAP @ 390000
      HEAP_ENTRY Size Prev Flags    UserPtr UserSize - state
        0039c9f8 0057 0000  [07]   0039ca00    0029a - (busy)
        0039ccb0 0057 0057  [07]   0039ccb8    0029a - (busy)
        0039cf68 0057 0057  [07]   0039cf70    0029a - (busy)
        0039d220 0057 0057  [07]   0039d228    0029a - (busy)
        0039d4d8 0057 0057  [07]   0039d4e0    0029a - (busy)
        0039d790 0057 0057  [07]   0039d798    0029a - (busy)
        0039da48 0057 0057  [07]   0039da50    0029a - (busy)
        0039dd00 0057 0057  [07]   0039dd08    0029a - (busy)
        0039dfb8 0057 0057  [07]   0039dfc0    0029a - (busy)
        0039e270 0057 0057  [07]   0039e278    0029a - (busy)
        0039e528 0057 0057  [07]   0039e530    0029a - (busy)
        0039e7e0 0057 0057  [07]   0039e7e8    0029a - (busy)
        3190b518 0057 0057  [07]   3190b520    0029a - (busy)
        3190b7d0 0057 0057  [07]   3190b7d8    0029a - (busy)
        3190ba88 0057 0057  [07]   3190ba90    0029a - (busy)
        3190bd40 0058 0057  [07]   3190bd48    0029a - (busy)
        3190c000 0057 0058  [07]   3190c008    0029a - (busy)
        3190c2b8 0057 0057  [07]   3190c2c0    0029a - (busy)
    _HEAP @ 3d0000

6d. And now see callstack of some random allocation.

0:001> !heap -p -a 0039d220
    address 0039d220 found in
    _HEAP @ 390000
      HEAP_ENTRY Size Prev Flags    UserPtr UserSize - state
        0039d220 0057 0000  [07]   0039d228    0029a - (busy)
        Trace: 030b
        7c96eed2 ntdll!RtlDebugAllocateHeap+0x000000e1
        7c94b394 ntdll!RtlAllocateHeapSlowly+0x00000044
        7c918f21 ntdll!RtlAllocateHeap+0x00000e64
        78583db8 MSVCR90!malloc+0x00000079
        78a34b52 mfc90u!operator new+0x00000033
        401450 Heaps1!CHeaps1Dlg::OnBnClickedButton1+0x00000010
        78a6d3e9 mfc90u!CCmdTarget::OnCmdMsg+0x00000124
        78a5de60 mfc90u!CPropertySheet::OnCmdMsg+0x0000001d
        78a3ff73 mfc90u!CWnd::OnCommand+0x00000092
        78a3f75b mfc90u!CWnd::OnWndMsg+0x00000066
        78a3f6ce mfc90u!CWnd::WindowProc+0x00000024
        78a3e2f4 mfc90u!AfxCallWndProc+0x000000a3
        78a3e580 mfc90u!AfxWndProc+0x00000037
        78a3c247 mfc90u!AfxWndProcBase+0x00000056
        7e418734 USER32!InternalCallWinProc+0x00000028
        7e418816 USER32!UserCallWinProcCheckWow+0x00000150
        7e428ea0 USER32!DispatchClientMessage+0x000000a3
        7e428eec USER32!__fnDWORD+0x00000024
        7c90e473 ntdll!KiUserCallbackDispatcher+0x00000013
        7e4292e3 USER32!SendMessageW+0x0000007f
        773f735f COMCTL32!Button_NotifyParent+0x0000003d
        773f7441 COMCTL32!Button_ReleaseCapture+0x000000d7
        773f9746 COMCTL32!Button_WndProc+0x00000887
        7e418734 USER32!InternalCallWinProc+0x00000028
        7e418816 USER32!UserCallWinProcCheckWow+0x00000150
        7e4189cd USER32!DispatchMessageWorker+0x00000306
        7e418a10 USER32!DispatchMessageW+0x0000000f
        7e4274ff USER32!IsDialogMessageW+0x00000572
        78a6c38d mfc90u!CWnd::IsDialogMessageW+0x00000032
        78a42739 mfc90u!CWnd::PreTranslateInput+0x0000002d
        78a5b244 mfc90u!CDialog::PreTranslateMessage+0x00000095
        78a40974 mfc90u!CWnd::WalkPreTranslateTree+0x00000023
6e. Let see how CHeaps1Dlg::OnBnClickedButton1 looks.

void CauseLeak()
    int nCount = 1000000;
    for (int i = 0; i < nCount; ++i)
        new char[666];

void CHeaps1Dlg::OnBnClickedButton1()

7. It is enough to fix bad code.

Heap fragmentation and Low-fragmentation Heap

1. Heap fragmentation

Fragmenting a heap is something I haven’t worried about for years. When you allocate and deallocate memory in certain patterns you can leave areas of unallocated memory stranded inamongst allocated memory. This can lead to the situation where you have, say, 10Mb of memory free, but yet an allocation for 256 bytes fails as although you have all this free memory, none if it is in a big enough continuous lump to give you your 256 bytes.

2. Low-fragmentation heap definition 

The LFH is not a separate heap. Instead, it is a policy that applications can enable for their heaps. When the LFH is enabled, the system allocates memory in certain predetermined sizes. When an application requests a memory allocation from a heap that has the LFH enabled, the system allocates the smallest block of memory that is large enough to contain the requested size. The system does not use the LFH for allocations larger than 16 KB, whether or not the LFH is enabled.

Applications that benefit most from the LFH are multi-threaded applications that allocate memory frequently and use a variety of allocation sizes under 16 KB. However, not all applications benefit from the LFH. To assess the effects of enabling the LFH in your application, use performance profiling data.

3. What is the cause of fragmentation

Imagine, for concreteness, a program that allocates memory in a loop like this:

* p1 = alloc(128)
* p2 = alloc(128)
* free(p1)
* p3 = alloc(96)
* (Keep p2 and p3 allocated.)
* Repeat

Under the classical model, when the request for 96 bytes comes in, the memory manager sees that 128-byte block (formerly known as p1) and splits it into two parts, a 96-byte block and a 32-byte block. The 96-byte block becomes block p3, and the 32-byte block sits around waiting for somebody to ask for 32 bytes (which never happens).

4. Low-fragmentation heap in Visual Studio 2010 and Windows Vista/7

Generally speaking, the low-fragmentation heap works pretty well for most classes of applications, and you should consider using it. (In fact, I'm told that the C runtime libraries have converted the default C runtime heap to be a low-fragmentation heap starting in Visual Studio 2010.)

Starting with Windows Vista, the system uses the low-fragmentation heap (LFH) as needed to service memory allocation requests. Applications do not need to enable the LFH for their heaps.

Why the low fragmentation heap (LFH) mechanism may be disabled on some computers that are running Windows Server 2003, Windows XP, or Windows 2000

Chris Valasek, Understanding the Low-Fragmentation Heap: From Allocation to Exploitation

Mark's Russinovich presentations from the 2010 Professional Developer's Conference

If you want to know the difference between System Committed memory and Process Committed memory, wondered what all those memory numbers shown by Task Manager really mean, or want to gain insight into the memory-related impact of a process, then this talk is for you. Using various memory analysis tools including: Process Explorer, VMMap, RAMMap, and others to highlight concepts throughout, the presentation starts with an overview of virtual memory management, describing types of process address space memory and how they impact system virtual memory. Then it dives into physical memory management, discussing how Windows manages process working sets, how it keeps track of physical memory, and how memory moves between different states.

Mysteries of Windows Memory Management, Part 1 
Mysteries of Windows Memory Management, Part 2

How a C++ compiler implements exception handling

The C++ standard, like any other feature in C++, doesn't say anything about how exception handling should be implemented. This means that every vendor is free to use any implementation as he sees fit. I will describe how VC++ implements this feature, but it should be a good study material for those as well who use other compilers or Operating Systems [1]. VC++ builds its exception handling support on top of structured exception handling (SEH) provided by Windows operating system [2].

Prevent Windows XP from searching inside ZIP and CAB files

Prevent Windows XP from searching inside ZIP and CAB files, without breaking Windows XP's native ZIP / CAB support

When you initiate a file search in Windows XP, it searches for files inside ZIP and CAB archives. Most people don't want this feature, as search causes delay if you have many ZIP or CAB files in the system. Note that Windows XP does not provide an option to disable searching inside these archives.

Utility discussed in this page can prevent Windows XP from searching inside ZIP and CAB files, whilst still maintaining Windows XP's native ZIP / CAB support.

Pitfalls of Object Oriented Programming

Sony Computer Entertainment Europe Research & Development Division, Tony Albrecht –Technical Consultant Developer Services

restrict (C++)

Jeszcze trochę na temat cache L1

Wojny walutowe,552624-Wojny-walutowe-.html



An Anti-Reverse Engineering Guide

  1. Breakpoints

  2. Timing Attacks

  3. Windows Internals

  4. Process Exploitation

  5. Anti-Dumping

  6. IA-32 Instruction Exploits

  7. OllyDBG Specific

  8. WinDBG Specific

  9. Other Techniques

What Do Exceptions Cost, In Theory?

Hardware breakpoints (from C++ code) on Intel

This is a debugging helper class which lets you set breakpoints on the fly from within code. This is mainly useful for the case where you have a variable that you know is getting trashed, but you have no idea who is trashing it. You can cause the debugger to break in at the very moment the variable is changed. The really cool thing is that this makes use of the Intel Pentium's built-in debug registers, which means that it really will stop no matter what code is executing, even if it's down in the NT kernel, in a different thread, or whatever.

Standing on the shoulders of the blue monster - Hardening Windows applications

Microsoft has implemented lots of useful functionality in Windows that they use in their own products. Many of these features can be used to enhance the security of third party applications, but not many developers or software architects know about them. This talk will detail some of the technical underpinnings of Windows features like UAC, IE protected mode and Terminal Serivces and show how they can be used to defend your own software from attack.

Multi-Core Support in Windows 7

Threads or Cores: Which Do You Need?

Porting to 64-bit Platforms

Boost: Intrusive and non-intrusive containers

The main difference between intrusive containers and non-intrusive containers is that in C++ non-intrusive containers store copies of values passed by the user. Containers use the Allocator template parameter to allocate the stored values.

On the other hand, an intrusive container does not store copies of passed objects, but it stores the objects themselves. The additional data needed to insert the object in the container must be provided by the object itself. For example, to insert MyClass in an intrusive container that implements a linked list, MyClass must contain the needed next and previous pointers:

Beyond Locks and Messages: The Future of Concurrent Programming

Message passing’s major flaw is the inversion of control–it is a moral equivalent of gotos in un-structured programming (it’s about time somebody said that message passing is considered harmful). MP still has its applications and, used in moderation, can be quite handy; but PGAS offers a much more straightforward programming model–its essence being the separation of implementation from algorithm. The Platonic ideal would be for the language to figure out the best parallel implementation for a particular algorithm. Since this is still a dream, the next best thing is getting rid of the interleaving of the two in the same piece of code.

Decoding the parameters of a thrown C++ exception (0xE06D7363)

Sekretna misja Terminatora

Protecting Your Code with Visual C++ Defenses

I know because I've googled it...

The Colbert ReportMon - Thurs 11:30pm / 10:30c
Charity Begins at 11:30
Colbert Report Full Episodes2010 ElectionFox News

Branch prediction in modern processors

Debugging with Visual Studio 2005/2008: Remote Debugging

Świetny tutorial. Polecam.

What's New in Visual C++ 2010

Breaking changes in Visual C++ 2010:

MC Chris - Fett's Vette

Processor Cache Effects

Example 1: Memory accesses and performance
Example 2: Impact of cache lines
Example 3: L1 and L2 cache sizes
Example 4: Instruction-level parallelism
Example 5: Cache associativity
Example 6: False cache line sharing
Example 7: Hardware complexities

Venkman JavaScript Debug

Venkman is the code name for Mozilla's JavaScript Debugger. Venkman aims to provide a powerful JavaScript debugging environment for Gecko-based browsers namely Firefox 3.x, the Netscape 7.x series of browsers, Netscape 9.x series, Mozilla Seamonkey 1.x and Mozilla Seamonkey 2.x. It does not include Gecko-based browsers such as K-Meleon 1.x, Galeon 2.x and Netscape 8.x. The debugger is available as an add-on package in XPI format. Venkman JavaScript Debugger has been provided as part of the Mozilla install distribution since October 3rd 2001.

Dense = good

Applications should also take care to avoid filling the cache with data that isn't needed. Remember, the cache is very small compared to main memory and every unnecessary data byte in the cache is taking up space that could be used beneficially by more relevant data. One simple way to avoid wasting cache space is to use smaller data types when possible. For example, a data structure might include an integer numeric field that ranges in value between 1 and 1000. Often a 32-bit INT would be used to store this value, but in this case that is overkill; a SHORT (two bytes) would work as well, and take up half the space.

Fault Tolerant Heap


Definitions of Non-blocking, Lock-free and Wait-free

Jed Duffy about messages and blocking

Overly aggressively admitting messages may seem like the right thing to do, until you’ve wedged yourself into some unforeseen inconsistent state.  You can avoid this by making each message handler atomic; see Argus.  But if you can't or don't have the discipline to do that, or aren't quite sure, you must not pump.  You either avoid pumping altogether or you selectively pump messages that do not touch the state encapsulated by the pump.  Or you lock access to state with a non-recursive lock and run the risk of deadlock.

How to run WinDbg to debug local machine

1. Set _NT_SYMBOL_PATH to something similiar

2. Download LiveKd v 3.12 or higher

3. Download and install WinDbg (6.11 or higher version)

4. Copy livekd to directory of WinDbg
(C:\Program Files\Debugging Tools for Windows (x86))

5. Run
livekd -w

6. Have a fun

P.S. If you have problems then temporary disable firewall to allow download symbols from Microsoft symbols server.

Tomasz Kulig