内存泄漏分析工具:tcmalloc

最近遇到一个内存泄漏的问题。由于代码量比较庞大,且使用了很多第三方库,部分第三方库还是商业闭源的,没法通过 code review 还排查了。对于闭源部分的库,静态代码检查工具也没法派上用场。所以一直在寻找一个能够追踪内存分配,释放,定位内存泄漏点的工具。尝试了 valgrind 和 gperftools。由于是嵌入式开发,内存,cpu 都比较有限。valgrind 资源占用太多,应用根本就没法跑起来。后面了解了 gperftools 的原理后,觉得内存,cpu 的占用应该是可以接受的,所以就一直折腾 gperftools,经过几天的折腾,总算靠 gperftools 这个工具找出了内存泄漏的真凶。下面介绍一下 gperftools 工具的使用方法。

1. 下载编译

先从 github 上下载最新版本的代码

1
2
3
qiushao@qiushao-pc:~/projects/opensources$ git clone https://github.com/gperftools/gperftools
qiushao@qiushao-pc:~/projects/opensources$ cd gperftools
qiushao@qiushao-pc:~/projects/opensources/gperftools$

我们先编译安装 pc 上使用的版本:

1
2
3
4
qiushao@qiushao-pc:~/projects/opensources/gperftools$ ./autogen.sh
qiushao@qiushao-pc:~/projects/opensources/gperftools$ ./configure
qiushao@qiushao-pc:~/projects/opensources/gperftools$ make -j
qiushao@qiushao-pc:~/projects/opensources/gperftools$ sudo make install

如果是需要交叉编译的话,则再重新编译。 ./configure 时指定目标平台,编译工具链即可。只需要编译,不需要安装,后面我们直接 copy so 库到板子上就行:

1
2
3
qiushao@qiushao-pc:~/projects/opensources/gperftools$ make clean
qiushao@qiushao-pc:~/projects/opensources/gperftools$ ./configure --host=arm-linux CXX=arm-himix200-linux-g++ CC=arm-himix200-linux-gcc
qiushao@qiushao-pc:~/projects/opensources/gperftools$ make -j

编译的结果在 .libs 目录下,gperftools 是一个工具集合,我们分析内存泄漏问题时,只需要使用其中的 libtcmalloc.so 即可。

1
2
3
4
5
qiushao@qiushao-pc:~/projects/opensources/gperftools$ file .libs/libtcmalloc.so*
.libs/libtcmalloc.so: symbolic link to libtcmalloc.so.4.5.5
.libs/libtcmalloc.so.4: symbolic link to libtcmalloc.so.4.5.5
.libs/libtcmalloc.so.4.5.5: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (GNU/Linux), dynamically linked, with debug_info, not stripped
qiushao@qiushao-pc:~/projects/opensources/gperftools$

2. tcmalloc 使用

我们先在 pc 上测试一下效果,如果能满足我们的需求的话,再到板子上进行测试。
先写个有内存泄漏的 demo:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
#include<stdio.h>
#include<stdlib.h>
#include<string.h>

char* get_buffer(size_t size) {
char *buffer = (char *)malloc(size); // 这里分配了内存,但后面 leak_memory 函数使用完之后没有释放,造成内存泄漏。
memset(buffer, 0, size);
return buffer;
}

void leak_memory() {
char *buffer = get_buffer(1024 * 1024);
sprintf(buffer, "do something with buffer");
printf("%s\n", buffer);
}

int main() {
printf("tcmalloc test");
for(int i=0; i < 10; i++) {
leak_memory();
}
return 0;
}

编译时需要加上调试信息 -g

1
qiushao@qiushao-pc:~/projects/test$ g++ -o tcmalloc_test -g tcmalloc_test.cpp

我们在写代码,和编译代码的时候都不需要 tcmalloc 的参与,我觉得这一点是做得非常好的。对代码完全没有侵入。
只需要按以下的方式执行就可以:

1
qiushao@qiushao-pc:~/projects/test$ LD_PRELOAD=/usr/local/lib/libtcmalloc.so HEAPCHECK=normal ./tcmalloc_test
  • LD_PRELOAD : 指定 libtcmalloc.so 的路径
  • HEAPCHECK: 指定检查等级。包括 minimal, normal, strict, draconian 这四种等级。具体差别参考文档: gperftools/docs/heap_checker.html 。如果 normal 没有检查出来问题的话,再用 draconian 试试。

3. 解析内存泄漏分析结果

程序的执行结果如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
qiushao@qiushao-pc:~/projects/test$ LD_PRELOAD=/usr/local/lib/libtcmalloc.so HEAPCHECK=normal ./tcmalloc_test 
WARNING: Perftools heap leak checker is active -- Performance may suffer
tcmalloc testdo something with buffer
do something with buffer
do something with buffer
do something with buffer
do something with buffer
do something with buffer
do something with buffer
do something with buffer
do something with buffer
do something with buffer
Have memory regions w/o callers: might report false leaks
Leak check _main_ detected leaks of 10485760 bytes in 10 objects
The 1 largest leaks:
*** WARNING: Cannot convert addresses to symbols in output below.
*** Reason: Cannot find 'pprof' (is PPROF_PATH set correctly?)
*** If you cannot fix this, try running pprof directly.
Leak of 10485760 bytes in 10 objects allocated from:
@ 55d4280a4732
@ 55d4280a4763
@ 55d4280a47d2
@ 7fc6cb53bb97
@ 55d4280a463a


If the preceding stack traces are not enough to find the leaks, try running THIS shell command:

pprof ./tcmalloc_test "/tmp/tcmalloc_test.14180._main_-end.heap" --inuse_objects --lines --heapcheck --edgefraction=1e-10 --nodefraction=1e-10 --gv

If you are still puzzled about why the leaks are there, try rerunning this program with HEAP_CHECK_TEST_POINTER_ALIGNMENT=1 and/or with HEAP_CHECK_MAX_POINTER_OFFSET=-1
If the leak report occurs in a small fraction of runs, try running with TCMALLOC_MAX_FREE_QUEUE_SIZE of few hundred MB or with TCMALLOC_RECLAIM_MEMORY=false, it might help find lea
Exiting with error code (instead of crashing) because of whole-program memory leaks
qiushao@qiushao-pc:~/projects/test$

tcmalloc 的内存泄漏检查生效时会有这个警告:WARNING: Perftools heap leak checker is active -- Performance may suffer
运行完程序之后,提示: Leak of 10485760 bytes in 10 objects allocated from: 。即有 10 处内存地址泄漏了,总共泄漏了 10485760 bytes。
tcmalloc 会在 /tmp 目录下生成一个 heap 文件:/tmp/tcmalloc_test.14180.main-end.heap。这个文件记录了所有的内存分配信息。
pprof 是 gperftools 自带的一个 perl 脚本,如果我们之前有 make install 的话,直接使用就行,没有 make install 的话,就得设置一下环境变量了。
我们可以通过 pprof 脚本来解析 heap 文件,获取每一个内存泄漏的位置及调用堆栈:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
qiushao@qiushao-pc:~/projects/test$ pprof ./tcmalloc_test "/tmp/tcmalloc_test.14180._main_-end.heap"  --lines  --text --stack
Using local file ./tcmalloc_test.
Using local file /tmp/tcmalloc_test.14180._main_-end.heap.
Total: 10 objects
Stacks:

10 (000055d4280a4732) /home/qiushao/projects/test/tcmalloc_test.cpp:6:get_buffer
(000055d4280a4762) /home/qiushao/projects/test/tcmalloc_test.cpp:12:leak_memory
(000055d4280a47d1) /home/qiushao/projects/test/tcmalloc_test.cpp:20:main
(00007fc6cb53bb96) /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310:__libc_start_main
(000055d4280a4639) ??:0:_start

Leak of 1048576 bytes in 1 objects allocated from:
@ 55d4280a4732 unknown
@ 000055d4280a4762 leak_memory /home/qiushao/projects/test/tcmalloc_test.cpp:12
@ 000055d4280a47d1 main /home/qiushao/projects/test/tcmalloc_test.cpp:20
@ 00007fc6cb53bb96 __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
@ 000055d4280a4639 _start ??:0
Leak of 1048576 bytes in 1 objects allocated from:
@ 55d4280a4732 unknown
@ 000055d4280a4762 leak_memory /home/qiushao/projects/test/tcmalloc_test.cpp:12
@ 000055d4280a47d1 main /home/qiushao/projects/test/tcmalloc_test.cpp:20
@ 00007fc6cb53bb96 __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
@ 000055d4280a4639 _start ??:0
Leak of 1048576 bytes in 1 objects allocated from:
@ 55d4280a4732 unknown
@ 000055d4280a4762 leak_memory /home/qiushao/projects/test/tcmalloc_test.cpp:12
@ 000055d4280a47d1 main /home/qiushao/projects/test/tcmalloc_test.cpp:20
@ 00007fc6cb53bb96 __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
@ 000055d4280a4639 _start ??:0
Leak of 1048576 bytes in 1 objects allocated from:
@ 55d4280a4732 unknown
@ 000055d4280a4762 leak_memory /home/qiushao/projects/test/tcmalloc_test.cpp:12
@ 000055d4280a47d1 main /home/qiushao/projects/test/tcmalloc_test.cpp:20
@ 00007fc6cb53bb96 __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
@ 000055d4280a4639 _start ??:0
Leak of 1048576 bytes in 1 objects allocated from:
@ 55d4280a4732 unknown
@ 000055d4280a4762 leak_memory /home/qiushao/projects/test/tcmalloc_test.cpp:12
@ 000055d4280a47d1 main /home/qiushao/projects/test/tcmalloc_test.cpp:20
@ 00007fc6cb53bb96 __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
@ 000055d4280a4639 _start ??:0
Leak of 1048576 bytes in 1 objects allocated from:
@ 55d4280a4732 unknown
@ 000055d4280a4762 leak_memory /home/qiushao/projects/test/tcmalloc_test.cpp:12
@ 000055d4280a47d1 main /home/qiushao/projects/test/tcmalloc_test.cpp:20
@ 00007fc6cb53bb96 __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
@ 000055d4280a4639 _start ??:0
Leak of 1048576 bytes in 1 objects allocated from:
@ 55d4280a4732 unknown
@ 000055d4280a4762 leak_memory /home/qiushao/projects/test/tcmalloc_test.cpp:12
@ 000055d4280a47d1 main /home/qiushao/projects/test/tcmalloc_test.cpp:20
@ 00007fc6cb53bb96 __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
@ 000055d4280a4639 _start ??:0
Leak of 1048576 bytes in 1 objects allocated from:
@ 55d4280a4732 unknown
@ 000055d4280a4762 leak_memory /home/qiushao/projects/test/tcmalloc_test.cpp:12
@ 000055d4280a47d1 main /home/qiushao/projects/test/tcmalloc_test.cpp:20
@ 00007fc6cb53bb96 __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
@ 000055d4280a4639 _start ??:0
Leak of 1048576 bytes in 1 objects allocated from:
@ 55d4280a4732 unknown
@ 000055d4280a4762 leak_memory /home/qiushao/projects/test/tcmalloc_test.cpp:12
@ 000055d4280a47d1 main /home/qiushao/projects/test/tcmalloc_test.cpp:20
@ 00007fc6cb53bb96 __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
@ 000055d4280a4639 _start ??:0
Leak of 1048576 bytes in 1 objects allocated from:
@ 55d4280a4732 unknown
@ 000055d4280a4762 leak_memory /home/qiushao/projects/test/tcmalloc_test.cpp:12
@ 000055d4280a47d1 main /home/qiushao/projects/test/tcmalloc_test.cpp:20
@ 00007fc6cb53bb96 __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
@ 000055d4280a4639 _start ??:0

10 100.0% 100.0% 10 100.0% get_buffer /home/qiushao/projects/test/tcmalloc_test.cpp:6
0 0.0% 100.0% 10 100.0% __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
0 0.0% 100.0% 10 100.0% _start ??:0
0 0.0% 100.0% 10 100.0% leak_memory /home/qiushao/projects/test/tcmalloc_test.cpp:12
0 0.0% 100.0% 10 100.0% main /home/qiushao/projects/test/tcmalloc_test.cpp:20
qiushao@qiushao-pc:~/projects/test$

其中:

  • --lines: 打印文件路径和行号。
  • --text : 以文本形式输出分析结果,还可以以 pdf, 调用图 gv 等形式输出。
  • --stack: 打印每一处泄漏的调用堆栈。

4. arm linux 上使用 tcmalloc

在开发板上使用 tcmalloc 跟 pc 上使用是差不多的。我们把交叉编译生成的 libtcmalloc.so* 库 copy 到板子上。我放到了 /vendor/lib 目录下。然后手动执行程序:

1
2
3
4
5
~ # mount -t nfs -o nolock -o tcp -o rsize=32768,wsize=32768 192.168.181.112:/media/qiushao/source-code/Hi3516/sourceCode/trunk  /mnt
~ # mount -o rw,remount /vendor
~ # cp -f /mnt/libtcmalloc.so* /vendor/lib/
~ # cp -f /mnt/tcmalloc_test /vendor/bin/
~ # LD_PRELOAD=/vendor/lib/libtcmalloc.so HEAPCHECK=normal /vendor/bin/tcmalloc_test

执行结果跟在 pc 上执行的是一样的:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
~ # LD_PRELOAD=/vendor/lib/libtcmalloc.so HEAPCHECK=normal /vendor/bin/tcmalloc_
test
WARNING: Perftools heap leak checker is active -- Performance may suffer
tcmalloc testdo something with buffer
do something with buffer
do something with buffer
do something with buffer
do something with buffer
do something with buffer
do something with buffer
do something with buffer
do something with buffer
do something with buffer
Have memory regions w/o callers: might report false leaks
Leak check _main_ detected leaks of 10485760 bytes in 10 objects
The 1 largest leaks:
*** WARNING: Cannot convert addresses to symbols in output below.
*** Reason: Cannot find 'pprof' (is PPROF_PATH set correctly?)
*** If you cannot fix this, try running pprof directly.
Leak of 10485760 bytes in 10 objects allocated from:


If the preceding stack traces are not enough to find the leaks, try running THIS shell command:

pprof /vendor/bin/tcmalloc_test "/tmp/tcmalloc_test.251._main_-end.heap" --inuse_objects --lines --heapcheck --edgefraction=1e-10 --nodefraction=1e-10 --gv

If you are still puzzled about why the leaks are there, try rerunning this program with HEAP_CHECK_TEST_POINTER_ALIGNMENT=1 and/or with HEAP_CHECK_MAX_POINTER_OFFSET=-1
If the leak report occurs in a small fraction of runs, try running with TCMALLOC_MAX_FREE_QUEUE_SIZE of few hundred MB or with TCMALLOC_RECLAIM_MEMORY=false, it might help
Exiting with error code (instead of crashing) because of whole-program memory leaks
~ #

同样也生成了 “/tmp/tcmalloc_test.251.main-end.heap” 文件。由于在板子上没法跑 pprof 脚本。这个脚本实际上是 perl 脚本,还依赖一大堆其他工具。
我们可以把 “/tmp/tcmalloc_test.251.main-end.heap” 文件 copy 到 pc 上,然后在 pc 用 pprof 进行分析。

在分析的时候可能会提示一些库找不到,我们需要把这些库从板子上也 copy 到 pc 上对应的目录。

5. tcmalloc 基本原理介绍

使用完 tcmalloc 之后,我们有以下几个疑问,只要把这几个疑问解决了,那我们对 tcmalloc 的基本原理也就清楚了。

    1. 如何记录内存分配,释放:
      tcmalloc 其实就是定义了一套自己的内存分配释放函数把标准库中的 malloc, alloc, free, new, delete 等替换掉。在分配内存的时候,通过 libunwind 获取调用堆栈,把内存分配调用堆栈记录下来。
    1. 如何替换标准库的内存分配,释放函数:
      LD_PRELOAD 是 Linux 系统的一个环境变量,它可以影响程序的运行时的链接,它允许你定义在程序运行前优先加载的动态链接库。这个功能主要就是用来有选择性的载入不同动态链接库中的相同函数。通过这个环境变量,我们可以在主程序和其动态链接库的中间加载别的动态链接库,甚至覆盖正常的函数库。一方面,我们可以以此功能来使用自己的或是更好的函数(无需别人的源码),而另一方面,我们也可以以向别人的程序注入程序,从而达到特定的目的。
    1. 如何在程序 main 函数结束后,执行内存泄漏分析:
      只要定义一个全局静态类变量就行,这个变量的构造函数会在 main 之前执行,析构函数会在 main 函数之后执行。

再详细的东西就不展开讨论了,想了解细节的同学,可以自己阅读代码。
更详细的使用细节说明,请阅读 gperftools 代码里面的 docs 文档。

参考资料: