Heap Overflow Example in Java

What is Heap overflow ?

A heap overflow or heap overrun is a type of buffer overflow that occurs in the heap data area. Heap overflows are exploitable in a different manner to that of stack-based overflows. Memory on the heap is dynamically allocated at runtime and typically contains program data. Exploitation is performed by corrupting this data in specific ways to cause the application to overwrite internal structures such as linked list pointers. The canonical heap overflow technique overwrites dynamic memory allocation linkage (such as malloc metadata) and uses the resulting pointer exchange to overwrite a program function pointer.

For example, on older versions of Linux, two buffers allocated next to each other on the heap could result in the first buffer overwriting the second buffer’s metadata. By setting the in-use bit to zero of the second buffer and setting the length to a small negative value which allows null bytes to be copied, when the program calls free() on the first buffer it will attempt to merge these two buffers into a single buffer. When this happens, the buffer that is assumed to be freed will be expected to hold two pointers FD and BK in the first 8 bytes of the formerly allocated buffer. BK gets written into FD and can be used to overwrite a pointer.

What is a memory leak?

In computer science, a memory leak is a type of resource leak that occurs when a computer program incorrectly manages memory allocations[1] in a way that memory which is no longer needed is not released. A memory leak may also happen when an object is stored in memory but cannot be accessed by the running code.[2] A memory leak has symptoms similar to a number of other problems and generally can only be diagnosed by a programmer with access to the program’s source code.

How to simulate a heap overflow ?

Firstly , set Intellij’s heap memory to 10mb for test case.
(Configuration -> VM Arguments-> -Xmx10m).

single custom object that will fill the heap with dozens of it.

1. Create a reference to Custom object, that will stores object into the heap mem, and object reference of it into stack memory.
2. As long as you continue keeping the reference to objects under program loop, heap usage will increase to a point unless program becomes heavy and unresponsive.

3. App continue working and the Visual VMs heap-graph clearly shows that Garbage collector can not do its job to clean ram. Because all objects are referenced in the stack and it implies that they are not useless to thE GC.

free memory in the heap after some execution time

4.Resulting objects in the heap dump. 217k objects are swimming in the heap and cannot cleared by Garbage collector. This makes the programme works less performant as time goes by.

You can find full visual VM test code in my github account

https://gist.github.com/barron9/01143176a8c896dbfdf6582d14dc19a0