[v2,1/2] mtrace: record backtrace of memory allocation/deallocation

Message ID 154268918372.10640.3813922125301329024.stgit@kyeongy-dl.ws.atlnz.lc
State New
Headers show
Series
  • [v2,1/2] mtrace: record backtrace of memory allocation/deallocation
Related show

Commit Message

Kyeong Yoo Nov. 20, 2018, 4:46 a.m.
When a memory leak is detected by mtrace, it is still difficult
to identify the exact source code causing that issue if the program
uses libraries or common functions.

Now a backtrace of the calling functions for memory allocation and
deallocation is also recorded. This new backtrace is displayed in
a separate line with '#' prefix prior to the original trace record
with '@' prefix.

This extra backtrace is turned off by default and only turned on
if valid number is set to the environment variable MALLOC_TRACE_LEVEL.
Valid value in MALLOC_TRACE_LEVEL is between 1 and 15, which indicates
the maximum number of recent function calls to display.

The companion mtrace.pl script is also updated to parse backtrace info.
And a new command argument "Maps" is added to mtrace.pl, which should
be a copy of the program's maps file ("/proc/PID/maps"). This helps
to interpret addresses from shared libraries.
Also to make mtrace more useful for cross-compilation, two new command
options are added: --prefix PREFIX and --addr2line ADDR2LINE.

Relevant documents are updated. Note typo in a related example for
mtrace() is also corrected.
---
 ChangeLog          |   19 +++++++
 NEWS               |    4 +
 malloc/mtrace.c    |   65 +++++++++++++++++++++++
 malloc/mtrace.pl   |  145 +++++++++++++++++++++++++++++++++++++++++-----------
 manual/memory.texi |   75 ++++++++++++++++++++++++---
 5 files changed, 271 insertions(+), 37 deletions(-)

Comments

Joseph Myers Nov. 20, 2018, 4:47 p.m. | #1
On Tue, 20 Nov 2018, Kyeong Yoo wrote:

> +To generate more detailed backtrace information of function calls to

> +@code{malloc} etc, set an environment variable named @code{MALLOC_TRACE_LEVEL}

> +with a number between 1 and 15. This indicates the maximum number of

> +function pointers to generate in the backtrace information.


@env for environment variables; two spaces after "." in Texinfo sources.

> +A list of pointers are printed with '#' mark before each memory


' is a right quote in Texinfo, so you need @samp{#} for proper quoting.

> +For cross-compilation, @code{mtrace} can generate meaningful output

> +with additional command line options @code{--prefix=PREFIX} and

> +@code{--addr2line=CMD} along with @code{/proc/<PID>maps} file copied

> +from the target device.


@option for command-line options, and @var for metasyntactic variables 
(lowercase inside @var, so @var{prefix} etc.).  @file for filenames.

I expect people will look at the patch in more detail once the copyright 
assignment is done.

https://git.savannah.gnu.org/cgit/gnulib.git/plain/doc/Copyright/request-assign.future

-- 
Joseph S. Myers
joseph@codesourcery.com
Kyeong Yoo Nov. 20, 2018, 11 p.m. | #2
Thanks for commenting on my patch.

On 21/11/18 05:47, Joseph Myers wrote:
> On Tue, 20 Nov 2018, Kyeong Yoo wrote:

> 

>> +To generate more detailed backtrace information of function calls to

>> +@code{malloc} etc, set an environment variable named @code{MALLOC_TRACE_LEVEL}

>> +with a number between 1 and 15. This indicates the maximum number of

>> +function pointers to generate in the backtrace information.

> 

> @env for environment variables; two spaces after "." in Texinfo sources.

I see. Will be fixed in a new revision.

> 

>> +A list of pointers are printed with '#' mark before each memory

> 

> ' is a right quote in Texinfo, so you need @samp{#} for proper quoting.

Thanks.

> 

>> +For cross-compilation, @code{mtrace} can generate meaningful output

>> +with additional command line options @code{--prefix=PREFIX} and

>> +@code{--addr2line=CMD} along with @code{/proc/<PID>maps} file copied

>> +from the target device.

> 

> @option for command-line options, and @var for metasyntactic variables

> (lowercase inside @var, so @var{prefix} etc.).  @file for filenames.

I see.

> 

> I expect people will look at the patch in more detail once the copyright

> assignment is done.

> 

> https://git.savannah.gnu.org/cgit/gnulib.git/plain/doc/Copyright/request-assign.future

I've sent an email for that.
>

Patch

diff --git a/ChangeLog b/ChangeLog
index d340866c43..1ea5f010f0 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,22 @@ 
+2018-11-20  Kyeong Yoo <kyeong.yoo@alliedtelesis.co.nz>
+
+	* malloc/mtrace.c: New environment variable MALLOC_TRACE_LEVEL is
+	defined to enable backtrace output from mtrace(). If the value set
+	in MALLOC_TRACE_LEVEL is between 1 and 15 when mtrace() is called,
+	extra backtrace info is recorded to the mtrace output.
+	* malloc/mtrace.c (tr_backtrace): Print backtrace of the function
+	calls up to the specified call level.
+	* malloc/mtrace.c (tr_freehook, tr_mallochook,tr_reallochook,
+	tr_memalignhook): Record backtrace if needed.
+	* malloc/mtrace.c (mtrace): Parse MALLOC_TRACE_LEVEL environment
+	variable. Valid number is between 1 and 15.
+	* malloc/mtrace.c (muntrace): Reset backtrace level to 0.
+	* malloc/mtrace.pl: Adjust to print backtrace of function calls.
+	* malloc/mtrace.pl: Accept new command argument for "Maps" which makes
+	possible to resolve addresses from shared libraries.
+	* malloc/mtrace.pl: Add "--prefix=PREFIX" and "--addr2line=CMD" options
+	for cross-compilation.
+
 2018-11-20  Joseph Myers  <joseph@codesourcery.com>
 
 	* conform/conformtest.py (ElementTest.run): Use unique identifiers
diff --git a/NEWS b/NEWS
index f488821af1..b025c49bfd 100644
--- a/NEWS
+++ b/NEWS
@@ -35,6 +35,10 @@  Major new features:
   different directory.  This is a GNU extension and similar to the
   Solaris function of the same name.
 
+* mtrace() records backtraces if MALLOC_TRACE_LEVEL environment variable
+  is set. mtrace Perl script is updated to specify maps file, prefix directory
+  and addr2line program to support cross-compilation case.
+
 Deprecated and removed features, and other changes affecting compatibility:
 
 * The glibc.tune tunable namespace has been renamed to glibc.cpu and the
diff --git a/malloc/mtrace.c b/malloc/mtrace.c
index 9064f209ec..5ab6c5d4fe 100644
--- a/malloc/mtrace.c
+++ b/malloc/mtrace.c
@@ -24,6 +24,7 @@ 
 # include <mcheck.h>
 # include <libc-lock.h>
 #endif
+#include <execinfo.h>
 
 #include <dlfcn.h>
 #include <fcntl.h>
@@ -44,8 +45,12 @@ 
 
 #define TRACE_BUFFER_SIZE 512
 
+#define MAX_BACKTRACE_LEVEL     15
+
 static FILE *mallstream;
 static const char mallenv[] = "MALLOC_TRACE";
+static const char mall_level_env[] = "MALLOC_TRACE_LEVEL";
+static int mall_trace_level = 0;
 static char *malloc_trace_buffer;
 
 __libc_lock_define_initialized (static, lock);
@@ -61,6 +66,9 @@  static void *(*tr_old_realloc_hook) (void *ptr, size_t size,
 static void *(*tr_old_memalign_hook) (size_t __alignment, size_t __size,
 				      const void *);
 
+static void *tr_mallochook (size_t size, const void *caller);
+static void *tr_reallochook (void *ptr, size_t size, const void *caller);
+
 /* This function is called when the block being alloc'd, realloc'd, or
    freed has an address matching the variable "mallwatch".  In a debugger,
    set "mallwatch" to the address of interest, then put a breakpoint on
@@ -108,6 +116,26 @@  tr_where (const void *caller, Dl_info *info)
     }
 }
 
+static void
+tr_backtrace (void)
+{
+  void *bt_addrs[MAX_BACKTRACE_LEVEL + 2];
+  size_t bt_size;
+
+  bt_size = __backtrace (bt_addrs, mall_trace_level + 2);
+
+  /* Print backtrace (skip the first two) */
+  if (bt_size > 2)
+    {
+      size_t i;
+
+      fprintf (mallstream, "# %p", bt_addrs[2]);
+      for (i = 3; i < bt_size; i++)
+        fprintf (mallstream, ",%p", bt_addrs[i]);
+      fprintf (mallstream, "\n");
+    }
+}
+
 static Dl_info *
 lock_and_info (const void *caller, Dl_info *mem)
 {
@@ -127,6 +155,18 @@  tr_freehook (void *ptr, const void *caller)
   if (ptr == NULL)
     return;
 
+  /* Print backtrace */
+  if (mall_trace_level > 0)
+    {
+      __free_hook = tr_old_free_hook;
+      __malloc_hook = tr_old_malloc_hook;
+      __realloc_hook = tr_old_realloc_hook;
+      tr_backtrace ();
+      __free_hook = tr_freehook;
+      __malloc_hook = tr_mallochook;
+      __realloc_hook = tr_reallochook;
+    }
+
   Dl_info mem;
   Dl_info *info = lock_and_info (caller, &mem);
   tr_where (caller, info);
@@ -160,6 +200,11 @@  tr_mallochook (size_t size, const void *caller)
     hdr = (void *) (*tr_old_malloc_hook)(size, caller);
   else
     hdr = (void *) malloc (size);
+
+  /* Print backtrace */
+  if (mall_trace_level > 0)
+    tr_backtrace ();
+
   __malloc_hook = tr_mallochook;
 
   tr_where (caller, info);
@@ -192,6 +237,11 @@  tr_reallochook (void *ptr, size_t size, const void *caller)
     hdr = (void *) (*tr_old_realloc_hook)(ptr, size, caller);
   else
     hdr = (void *) realloc (ptr, size);
+
+  /* Collect backtrace */
+  if (mall_trace_level > 0)
+    tr_backtrace ();
+
   __free_hook = tr_freehook;
   __malloc_hook = tr_mallochook;
   __realloc_hook = tr_reallochook;
@@ -236,6 +286,11 @@  tr_memalignhook (size_t alignment, size_t size, const void *caller)
     hdr = (void *) (*tr_old_memalign_hook)(alignment, size, caller);
   else
     hdr = (void *) memalign (alignment, size);
+
+  /* Collect backtrace */
+  if (mall_trace_level > 0)
+    tr_backtrace ();
+
   __memalign_hook = tr_memalignhook;
   __malloc_hook = tr_mallochook;
 
@@ -321,6 +376,15 @@  mtrace (void)
 			    __dso_handle);
             }
 #endif
+
+          /* Check backtrace level */
+          const char *level_str = getenv (mall_level_env);
+          if (level_str != NULL)
+            {
+              int num = atoi (level_str);
+              if (0 < num && num <= MAX_BACKTRACE_LEVEL)
+                mall_trace_level = num;
+            }
         }
       else
         free (mtb);
@@ -338,6 +402,7 @@  muntrace (void)
      file.  */
   FILE *f = mallstream;
   mallstream = NULL;
+  mall_trace_level = 0;
   __free_hook = tr_old_free_hook;
   __malloc_hook = tr_old_malloc_hook;
   __realloc_hook = tr_old_realloc_hook;
diff --git a/malloc/mtrace.pl b/malloc/mtrace.pl
index fe9f546000..22294bf20c 100644
--- a/malloc/mtrace.pl
+++ b/malloc/mtrace.pl
@@ -24,49 +24,56 @@  $VERSION = "@VERSION@";
 $PKGVERSION = "@PKGVERSION@";
 $REPORT_BUGS_TO = '@REPORT_BUGS_TO@';
 $progname = $0;
+$prefix = "";
+$addr2line_cmd = "addr2line";
+
+use Class::Struct;
+use File::Basename;
+use Getopt::Long;
+
+struct Map => {
+    start   => '$',         # start address
+    end     => '$',         # end address
+    path    => '$',         # library path
+};
+
+sub version {
+    print "mtrace $PKGVERSION$VERSION\n";
+    print "Copyright (C) 2018 Free Software Foundation, Inc.\n";
+    print "This is free software; see the source for copying conditions.  There is NO\n";
+    print "warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.\n";
+    print "Written by Ulrich Drepper <drepper\@gnu.org>\n";
+    exit 0;
+}
 
 sub usage {
-    print "Usage: mtrace [OPTION]... [Binary] MtraceData\n";
-    print "  --help       print this help, then exit\n";
-    print "  --version    print version number, then exit\n";
+    print "Usage: mtrace [OPTION]... [Binary] MtraceData [Maps]\n";
+    print "  --help           print this help, then exit\n";
+    print "  --version        print version number, then exit\n";
+    print "  --prefix=PREFIX  Root directory for library search (cross-compilation)\n";
+    print "  --addr2line=CMD  Specify addr2line command (cross-compilation)\n";
     print "\n";
     print "For bug reporting instructions, please see:\n";
     print "$REPORT_BUGS_TO.\n";
     exit 0;
 }
 
-# We expect two arguments:
+# We expect three arguments:
 #   #1: the complete path to the binary
 #   #2: the mtrace data filename
+#   #3: the memory map of the process (/proc/PID/maps)
 # The usual options are also recognized.
 
-arglist: while (@ARGV) {
-    if ($ARGV[0] eq "--v" || $ARGV[0] eq "--ve" || $ARGV[0] eq "--ver" ||
-	$ARGV[0] eq "--vers" || $ARGV[0] eq "--versi" ||
-	$ARGV[0] eq "--versio" || $ARGV[0] eq "--version") {
-	print "mtrace $PKGVERSION$VERSION\n";
-	print "Copyright (C) 2018 Free Software Foundation, Inc.\n";
-	print "This is free software; see the source for copying conditions.  There is NO\n";
-	print "warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.\n";
-	print "Written by Ulrich Drepper <drepper\@gnu.org>\n";
-
-	exit 0;
-    } elsif ($ARGV[0] eq "--h" || $ARGV[0] eq "--he" || $ARGV[0] eq "--hel" ||
-	     $ARGV[0] eq "--help") {
-	&usage;
-    } elsif ($ARGV[0] =~ /^-/) {
-	print "$progname: unrecognized option `$ARGV[0]'\n";
-	print "Try `$progname --help' for more information.\n";
-	exit 1;
-    } else {
-	last arglist;
-    }
-}
+GetOptions ("help"        => \&usage,
+            "version"     => \&version,
+            "prefix=s"    => \$prefix,        # string
+            "addr2line=s" => \$addr2line_cmd, # string
+           ) or die "Try `$progname --help' for more information.\n";
 
 if ($#ARGV == 0) {
     $binary="";
     $data=$ARGV[0];
-} elsif ($#ARGV == 1) {
+} elsif ($#ARGV == 1 or $#ARGV == 2) {
     $binary=$ARGV[0];
     $data=$ARGV[1];
 
@@ -84,10 +91,71 @@  if ($#ARGV == 0) {
 	}
 	close (LOCS);
     }
+
+    # Parse maps file
+    if (defined $ARGV[2]) {
+        open(MAPS, "<$ARGV[2]") || die "Cannot open maps file";
+        my $prog_base = basename($prog);
+        while (<MAPS>) {
+            chop;
+            my @cols = split (' ');
+            # Find executable memory segment for library (except the program itself)
+            if ($#cols == 5 and $cols[1] =~ /^..x.$/ and basename($cols[5]) ne $prog_base) {
+                my $path = $prefix . $cols[5];
+                if (-e $path and $cols[0] =~ /^(.*)-(.*)$/) {
+                    my $m = Map->new();
+                    $m->start(hex($1));
+                    $m->end(hex($2));
+                    $m->path($path);
+                    push(@maps, $m);
+                }
+            }
+        }
+        close (MAPS);
+    }
 } else {
     die "Wrong number of arguments, run $progname --help for help.";
 }
 
+sub find_address_map {
+    my $addr = pop(@_);
+    foreach my $m (@maps) {
+        return $m if ($m->start <= $addr and $addr <= $m->end);
+    }
+    return undef;
+}
+
+sub print_backtrace {
+    foreach my $addr (split (',', pop(@_))) {
+        printf ("  # %-16s   %s\n", $addr, &addr2line(hex($addr)));
+    }
+}
+
+sub addr2line {
+    my $addr = pop(@_);
+    return $cache{$addr} if (exists $cache{$addr});
+
+    my $executable;
+    my $searchaddr;
+    my $m = find_address_map($addr);
+    if ($m) {
+        $executable = $m->path;
+        $searchaddr = sprintf ("%#x", $addr - $m->start);
+    } else {
+        $executable = $binary;
+        $searchaddr = sprintf ("%#x", $addr);
+    }
+    if ($executable ne "" && open (ADDR, "$addr2line_cmd -e $executable $searchaddr|")) {
+        my $line = <ADDR>;
+        chomp $line;
+        close (ADDR);
+        $cache{$addr} = $line;
+        return $cache{$addr};
+    }
+    $cache{$addr} = "unknown";
+    return $cache{$addr};
+}
+
 sub location {
     my $str = pop(@_);
     return $str if ($str eq "");
@@ -95,7 +163,7 @@  sub location {
 	my $addr = $1;
 	my $fct = $2;
 	return $cache{$addr} if (exists $cache{$addr});
-	if ($binary ne "" && open (ADDR, "addr2line -e $binary $addr|")) {
+	if ($binary ne "" && open (ADDR, "$addr2line_cmd -e $binary $addr|")) {
 	    my $line = <ADDR>;
 	    chomp $line;
 	    close (ADDR);
@@ -116,7 +184,7 @@  sub location {
 	    $searchaddr = $addr;
 	    $prog = $binary;
 	}
-	if ($binary ne "" && open (ADDR, "addr2line -e $prog $searchaddr|")) {
+	if ($binary ne "" && open (ADDR, "$addr2line_cmd -e $prog $searchaddr|")) {
 	    my $line = <ADDR>;
 	    chomp $line;
 	    close (ADDR);
@@ -129,7 +197,7 @@  sub location {
     } elsif ($str =~ /^.*[[](0x[^]]*)]$/) {
 	my $addr = $1;
 	return $cache{$addr} if (exists $cache{$addr});
-	if ($binary ne "" && open (ADDR, "addr2line -e $binary $addr|")) {
+	if ($binary ne "" && open (ADDR, "$addr2line_cmd -e $binary $addr|")) {
 	    my $line = <ADDR>;
 	    chomp $line;
 	    close (ADDR);
@@ -144,6 +212,7 @@  sub location {
 }
 
 $nr=0;
+$bt="";
 open(DATA, "<$data") || die "Cannot open mtrace data file";
 while (<DATA>) {
     my @cols = split (' ');
@@ -167,9 +236,11 @@  while (<DATA>) {
 		printf ("+ %#0@XXX@x Alloc %d duplicate: %s %s\n",
 			hex($allocaddr), $nr, &location($addrwas{$allocaddr}),
 			$where);
+		&print_backtrace ($bt) if ($bt ne "");
 	    } elsif ($allocaddr =~ /^0x/) {
 		$allocated{$allocaddr}=$howmuch;
 		$addrwas{$allocaddr}=$where;
+		$backtrace{$allocaddr}=$bt if ($bt ne "");
 	    }
 	    last SWITCH;
 	}
@@ -177,9 +248,11 @@  while (<DATA>) {
 	    if (defined $allocated{$allocaddr}) {
 		undef $allocated{$allocaddr};
 		undef $addrwas{$allocaddr};
+		undef $backtrace{$allocaddr};
 	    } else {
 		printf ("- %#0@XXX@x Free %d was never alloc'd %s\n",
 			hex($allocaddr), $nr, &location($where));
+		&print_backtrace ($bt) if ($bt ne "");
 	    }
 	    last SWITCH;
 	}
@@ -187,9 +260,11 @@  while (<DATA>) {
 	    if (defined $allocated{$allocaddr}) {
 		undef $allocated{$allocaddr};
 		undef $addrwas{$allocaddr};
+		undef $backtrace{$allocaddr};
 	    } else {
 		printf ("- %#0@XXX@x Realloc %d was never alloc'd %s\n",
 			hex($allocaddr), $nr, &location($where));
+		&print_backtrace ($bt) if ($bt ne "");
 	    }
 	    last SWITCH;
 	}
@@ -198,9 +273,11 @@  while (<DATA>) {
 		printf ("+ %#0@XXX@x Realloc %d duplicate: %#010x %s %s\n",
 			hex($allocaddr), $nr, $allocated{$allocaddr},
 			&location($addrwas{$allocaddr}), &location($where));
+		&print_backtrace ($bt) if ($bt ne "");
 	    } else {
 		$allocated{$allocaddr}=$howmuch;
 		$addrwas{$allocaddr}=$where;
+		$backtrace{$allocaddr}=$bt if ($bt ne "");
 	    }
 	    last SWITCH;
 	}
@@ -213,6 +290,13 @@  while (<DATA>) {
 	    last SWITCH;
 	}
     }
+
+    # Save backtrace info (if any) for the next line
+    if ($cols[0] eq "#") {
+	$bt=$cols[1];
+    } else {
+	$bt="";
+    }
 }
 close (DATA);
 
@@ -229,6 +313,7 @@  if ($#addrs >= 0) {
 	    }
 	    printf ("%#0@XXX@x %#8x  at %s\n", hex($addr), $allocated{$addr},
 		    &location($addrwas{$addr}));
+	    &print_backtrace ($backtrace{$addr}) if (defined $backtrace{$addr});
 	}
     }
 }
diff --git a/manual/memory.texi b/manual/memory.texi
index a1435aad1a..25aefc4820 100644
--- a/manual/memory.texi
+++ b/manual/memory.texi
@@ -1709,6 +1709,11 @@  functions are traced and protocolled into the file.  There is now of
 course a speed penalty for all calls to the traced functions so tracing
 should not be enabled during normal use.
 
+To generate more detailed backtrace information of function calls to
+@code{malloc} etc, set an environment variable named @code{MALLOC_TRACE_LEVEL}
+with a number between 1 and 15. This indicates the maximum number of
+function pointers to generate in the backtrace information.
+
 This function is a GNU extension and generally not available on other
 systems.  The prototype can be found in @file{mcheck.h}.
 @end deftypefun
@@ -1830,13 +1835,13 @@  If you take a look at the output it will look similar to this:
 
 @example
 = Start
-@ [0x8048209] - 0x8064cc8
-@ [0x8048209] - 0x8064ce0
-@ [0x8048209] - 0x8064cf8
-@ [0x80481eb] + 0x8064c48 0x14
-@ [0x80481eb] + 0x8064c60 0x14
-@ [0x80481eb] + 0x8064c78 0x14
-@ [0x80481eb] + 0x8064c90 0x14
+@@ [0x8048209] - 0x8064cc8
+@@ [0x8048209] - 0x8064ce0
+@@ [0x8048209] - 0x8064cf8
+@@ [0x80481eb] + 0x8064c48 0x14
+@@ [0x80481eb] + 0x8064c60 0x14
+@@ [0x80481eb] + 0x8064c78 0x14
+@@ [0x80481eb] + 0x8064c90 0x14
 = End
 @end example
 
@@ -1915,6 +1920,62 @@  from line 33 in the source file @file{/home/drepper/tst-mtrace.c} four
 times without freeing this memory before the program terminates.
 Whether this is a real problem remains to be investigated.
 
+@node Interpreting the traces with backtrace
+@subsubsection Interpreting the traces with backtrace
+
+If backtrace information is generated by setting an environment variable
+named @code{MALLOC_TRACE_LEVEL} (for example,
+@code{export MALLOC_TRACE_LEVEL=3}), the output will look similar to this:
+
+@example
+= Start
+# 0x4009b4,0x400866,0x4008f3
+@@ [0x4009b4] - 0x184e280
+# 0x400879,0x4008f3,0x400923
+@@ [0x400879] + 0x184e280 0x14
+= End
+@end example
+
+A list of pointers are printed with '#' mark before each memory
+allocation/deallocation, which indicates the return addresses from the
+corresponding stack frame.
+
+@example
+drepper$ mtrace tst errlog
+- 0x0184e280 Free 19 was never alloc'd /home/drepper/tst.c:51
+  # 0x4009b4  /home/drepper/tst.c:51
+  # 0x400866  /home/drepper/tst.c:13
+  # 0x4008f3  /home/drepper/tst.c:26
+
+Memory not freed:
+-----------------
+   Address     Size     Caller
+0x0184e280     0x14  at /home/drepper/tst.c:13
+  # 0x400879   /home/drepper/tst.c:13
+  # 0x4008f3   /home/drepper/tst.c:26
+  # 0x400923   /home/drepper/tst.c:36
+@end example
+
+When this mtrace output is processed, it shows better picture of where
+the memory allocation/deallocation was called.
+
+@node Use mtrace for cross-compilation
+@subsubsection Use mtrace for cross-compilation
+
+For cross-compilation, @code{mtrace} can generate meaningful output
+with additional command line options @code{--prefix=PREFIX} and
+@code{--addr2line=CMD} along with @code{/proc/<PID>maps} file copied
+from the target device.
+
+For example, if you have cross-compiled libraries exists in @code{~/staging}
+and compiled for powerpc, you can run command like this:
+
+@example
+drepper$ mtrace --prefix=~/staging \
+  --addr2line=powerpc64-hardfloat-linux-gnu-addr2line \
+  tst errlog maps
+@end example
+
 @node Replacing malloc
 @subsection Replacing @code{malloc}