gdb: Print frame address in more cases

Message ID 20200304141258.17737-1-andrew.burgess@embecosm.com
State New
Headers show
Series
  • gdb: Print frame address in more cases
Related show

Commit Message

Andrew Burgess March 4, 2020, 2:12 p.m.
Change GDB to display the frame address for all inline frames except
the inner most frame.  This issue was discussed briefly on the mailing
list here:

   https://sourceware.org/ml/gdb-patches/2019-12/msg01015.html

Consider this stack setup from the test gdb.dwarf2/dw2-inline-many-frames.exp:

   #11     #10    #9     #8     #7     #6     #5     #4     #3     #2     #1     #0
   main -> aaa -> bbb -> ccc -> ddd -> eee -> fff -> ggg -> hhh -> iii -> jjj -> kkk
   \_______________________/    \________/    \______________________/    \________/
      Inline sequence #1          Normal          Inline sequence #2        Normal

When we stop in GDB and backtrace the stack is displayed like
this (I've replaced the file and line information with "..."):

  (gdb) bt
  #0  kkk () at ...
  #1  0x0000000000400494 in jjj () at ...
  #2  0x00000000004004b1 in iii () at ...
  #3  hhh () at ...
  #4  ggg () at ...
  #5  fff () at ...
  #6  0x0000000000400536 in eee () at ...
  #7  0x0000000000400519 in ddd () at ...
  #8  0x00000000004004f3 in ccc () at ...
  #9  bbb () at ...
  #10 aaa () at ...
  #11 main () at ...
  (gdb)

Notice that many frames are missing an address, take the sequence for
frames #6 to #2.  Frame #6 is a non-inline frame and has an address.
Frame #6 calls to frame #5 which is also non-inline.  Frames #4, #3,
and #2 are all inline within frame #5, and of these only frame #2 has
an address displayed.

This lack of address can be confusing, there's no clear indication why
the addresses are missing, a user needs to understand that the missing
address implies an inline frame, and can then infer the address based
on the address of frame #2.

After this patch the backtrace is now show like this:

  #0  kkk () at ...
  #1  0x0000000000400494 in jjj () at ...
  #2  0x00000000004004b1 in iii () at ...
  #3  0x00000000004004b1 in hhh () at ...
  #4  0x00000000004004b1 in ggg () at ...
  #5  0x00000000004004b1 in fff () at ...
  #6  0x0000000000400536 in eee () at ...
  #7  0x0000000000400519 in ddd () at ...
  #8  0x00000000004004f3 in ccc () at ...
  #9  0x00000000004004f3 in bbb () at ...
  #10 0x00000000004004f3 in aaa () at ...
  #11 0x00000000004004f3 in main () at ...

Now all frames (except #0) have an address.  Frames that are inline,
and previously lacked an address will now have the same address as the
inner-most inline frame in the sequence, so for the inline sequence #5
to #2, all frames have the same address as frame #2.  For the sequence
frame #11 to #8, all have the address of frame #8.

The duplicated address might also be confusing, though (personally) I
think it is slightly more obvious from the duplicated addresses that
the frames are inlined, however, if people strongly disagree and
prefer the no-address layout we could make this feature switchable.

One further location where the no address can crop up, is when the
user switches frame, previously we would see:

  (gdb) frame 3
  #3  hhh () at ...
  115	  return iii () + 1;
  (gdb)

After the patch we see:

  (gdb) frame 3
  #3  0x00000000004004b1 in hhh () at ...
  115	  return iii () + 1;
  (gdb)

There were a small number of tests that needed to have their expected
results updated, and I added a small section to the documentation to
help users understand what duplicate addresses in the backtrace might
mean.

gdb/ChangeLog:

	* stack.c (frame_show_address): Show the frame address for all but
	the inner-most inlined frame.

gdb/testsuite/ChangeLog:

	* gdb.dwarf2/dw2-inline-many-frames.exp: Update expected results.
	* gdb.dwarf2/dw2-inline-param.exp: Likewise.
	* gdb.guile/scm-frame-inline.exp: Likewise.
	* gdb.opt/inline-cmds.exp: Likewise.
	* gdb.python/py-frame-inline.exp: Likewise.

gdb/doc/ChangeLog:

	* gdb.texinfo (Backtrace): Mention duplicate addresses in the
	backtrace.
---
 gdb/ChangeLog                                       |  5 +++++
 gdb/doc/ChangeLog                                   |  5 +++++
 gdb/doc/gdb.texinfo                                 | 18 ++++++++++++++++++
 gdb/stack.c                                         |  2 +-
 gdb/testsuite/ChangeLog                             |  8 ++++++++
 gdb/testsuite/gdb.dwarf2/dw2-inline-many-frames.exp | 12 ++++++------
 gdb/testsuite/gdb.dwarf2/dw2-inline-param.exp       |  4 +++-
 gdb/testsuite/gdb.guile/scm-frame-inline.exp        |  2 +-
 gdb/testsuite/gdb.opt/inline-cmds.exp               | 14 +++++++-------
 gdb/testsuite/gdb.python/py-frame-inline.exp        |  6 +++---
 10 files changed, 57 insertions(+), 19 deletions(-)

-- 
2.14.5

Comments

Pedro Alves via Gdb-patches March 4, 2020, 7:53 p.m. | #1
On Wed, Mar 4, 2020 at 8:13 AM Andrew Burgess
<andrew.burgess@embecosm.com> wrote:
>

> Change GDB to display the frame address for all inline frames except

> the inner most frame.  This issue was discussed briefly on the mailing

> list here:

>

>    https://sourceware.org/ml/gdb-patches/2019-12/msg01015.html


Thanks! I think this makes more sense.

Christian

> Consider this stack setup from the test gdb.dwarf2/dw2-inline-many-frames.exp:

>

>    #11     #10    #9     #8     #7     #6     #5     #4     #3     #2     #1     #0

>    main -> aaa -> bbb -> ccc -> ddd -> eee -> fff -> ggg -> hhh -> iii -> jjj -> kkk

>    \_______________________/    \________/    \______________________/    \________/

>       Inline sequence #1          Normal          Inline sequence #2        Normal

>

> When we stop in GDB and backtrace the stack is displayed like

> this (I've replaced the file and line information with "..."):

>

>   (gdb) bt

>   #0  kkk () at ...

>   #1  0x0000000000400494 in jjj () at ...

>   #2  0x00000000004004b1 in iii () at ...

>   #3  hhh () at ...

>   #4  ggg () at ...

>   #5  fff () at ...

>   #6  0x0000000000400536 in eee () at ...

>   #7  0x0000000000400519 in ddd () at ...

>   #8  0x00000000004004f3 in ccc () at ...

>   #9  bbb () at ...

>   #10 aaa () at ...

>   #11 main () at ...

>   (gdb)

>

> Notice that many frames are missing an address, take the sequence for

> frames #6 to #2.  Frame #6 is a non-inline frame and has an address.

> Frame #6 calls to frame #5 which is also non-inline.  Frames #4, #3,

> and #2 are all inline within frame #5, and of these only frame #2 has

> an address displayed.

>

> This lack of address can be confusing, there's no clear indication why

> the addresses are missing, a user needs to understand that the missing

> address implies an inline frame, and can then infer the address based

> on the address of frame #2.

>

> After this patch the backtrace is now show like this:

>

>   #0  kkk () at ...

>   #1  0x0000000000400494 in jjj () at ...

>   #2  0x00000000004004b1 in iii () at ...

>   #3  0x00000000004004b1 in hhh () at ...

>   #4  0x00000000004004b1 in ggg () at ...

>   #5  0x00000000004004b1 in fff () at ...

>   #6  0x0000000000400536 in eee () at ...

>   #7  0x0000000000400519 in ddd () at ...

>   #8  0x00000000004004f3 in ccc () at ...

>   #9  0x00000000004004f3 in bbb () at ...

>   #10 0x00000000004004f3 in aaa () at ...

>   #11 0x00000000004004f3 in main () at ...

>

> Now all frames (except #0) have an address.  Frames that are inline,

> and previously lacked an address will now have the same address as the

> inner-most inline frame in the sequence, so for the inline sequence #5

> to #2, all frames have the same address as frame #2.  For the sequence

> frame #11 to #8, all have the address of frame #8.

>

> The duplicated address might also be confusing, though (personally) I

> think it is slightly more obvious from the duplicated addresses that

> the frames are inlined, however, if people strongly disagree and

> prefer the no-address layout we could make this feature switchable.

>

> One further location where the no address can crop up, is when the

> user switches frame, previously we would see:

>

>   (gdb) frame 3

>   #3  hhh () at ...

>   115     return iii () + 1;

>   (gdb)

>

> After the patch we see:

>

>   (gdb) frame 3

>   #3  0x00000000004004b1 in hhh () at ...

>   115     return iii () + 1;

>   (gdb)

>

> There were a small number of tests that needed to have their expected

> results updated, and I added a small section to the documentation to

> help users understand what duplicate addresses in the backtrace might

> mean.

>

> gdb/ChangeLog:

>

>         * stack.c (frame_show_address): Show the frame address for all but

>         the inner-most inlined frame.

>

> gdb/testsuite/ChangeLog:

>

>         * gdb.dwarf2/dw2-inline-many-frames.exp: Update expected results.

>         * gdb.dwarf2/dw2-inline-param.exp: Likewise.

>         * gdb.guile/scm-frame-inline.exp: Likewise.

>         * gdb.opt/inline-cmds.exp: Likewise.

>         * gdb.python/py-frame-inline.exp: Likewise.

>

> gdb/doc/ChangeLog:

>

>         * gdb.texinfo (Backtrace): Mention duplicate addresses in the

>         backtrace.

> ---

>  gdb/ChangeLog                                       |  5 +++++

>  gdb/doc/ChangeLog                                   |  5 +++++

>  gdb/doc/gdb.texinfo                                 | 18 ++++++++++++++++++

>  gdb/stack.c                                         |  2 +-

>  gdb/testsuite/ChangeLog                             |  8 ++++++++

>  gdb/testsuite/gdb.dwarf2/dw2-inline-many-frames.exp | 12 ++++++------

>  gdb/testsuite/gdb.dwarf2/dw2-inline-param.exp       |  4 +++-

>  gdb/testsuite/gdb.guile/scm-frame-inline.exp        |  2 +-

>  gdb/testsuite/gdb.opt/inline-cmds.exp               | 14 +++++++-------

>  gdb/testsuite/gdb.python/py-frame-inline.exp        |  6 +++---

>  10 files changed, 57 insertions(+), 19 deletions(-)

>

> diff --git a/gdb/doc/gdb.texinfo b/gdb/doc/gdb.texinfo

> index 32e419ead02..74178faee37 100644

> --- a/gdb/doc/gdb.texinfo

> +++ b/gdb/doc/gdb.texinfo

> @@ -8013,6 +8013,24 @@

>  either deduce that from other variables whose values depend on the one

>  you are interested in, or recompile without optimizations.

>

> +If your compiler has performed the function inlining optimisation

> +(@pxref{Inline Functions}) then this will be reflected in the program

> +counter addresses displayed in the backtrace.  In the following

> +modified example the function @code{expand_macro} has been inlined

> +within @code{expand_token}, as a result the program counter values for

> +frame 1 and frame 2 are the same.

> +

> +@smallexample

> +@group

> +#0  m4_traceon (obs=0x24eb0, argc=1, argv=0x2b8c8)

> +    at builtin.c:993

> +#1  0x6e38 in expand_macro (sym=<optimized out>) at macro.c:242

> +#2  0x6e38 in expand_token (obs=0x0, t=<optimized out>, td=0xf7fffb08)

> +    at macro.c:71

> +(More stack frames follow...)

> +@end group

> +@end smallexample

> +

>  @cindex backtrace beyond @code{main} function

>  @cindex program entry point

>  @cindex startup code, and backtrace

> diff --git a/gdb/stack.c b/gdb/stack.c

> index 266d771e35f..4570eed1a2f 100644

> --- a/gdb/stack.c

> +++ b/gdb/stack.c

> @@ -327,7 +327,7 @@ frame_show_address (struct frame_info *frame,

>         gdb_assert (inline_skipped_frames (inferior_thread ()) > 0);

>        else

>         gdb_assert (get_frame_type (get_next_frame (frame)) == INLINE_FRAME);

> -      return false;

> +      return frame_relative_level (frame) > 0;

>      }

>

>    return get_frame_pc (frame) != sal.pc;

> diff --git a/gdb/testsuite/gdb.dwarf2/dw2-inline-many-frames.exp b/gdb/testsuite/gdb.dwarf2/dw2-inline-many-frames.exp

> index 146af8c6ef7..f5890fb2aca 100644

> --- a/gdb/testsuite/gdb.dwarf2/dw2-inline-many-frames.exp

> +++ b/gdb/testsuite/gdb.dwarf2/dw2-inline-many-frames.exp

> @@ -350,15 +350,15 @@ gdb_test "bt" [multi_line \

>                    "#0  kkk \\(\\) at \[^\r\n\]+${srcfile}:${line_in_kkk}" \

>                    "#1  $hex in jjj \\(\\) at \[^\r\n\]+${srcfile}:${line_in_jjj}" \

>                    "#2  $hex in iii \\(\\) at \[^\r\n\]+${srcfile}:${line_in_iii}" \

> -                  "#3  hhh \\(\\) at \[^\r\n\]+${srcfile}:${line_in_hhh}" \

> -                  "#4  ggg \\(\\) at \[^\r\n\]+${srcfile}:${line_in_ggg}" \

> -                  "#5  fff \\(\\) at \[^\r\n\]+${srcfile}:${line_in_fff}" \

> +                  "#3  $hex in hhh \\(\\) at \[^\r\n\]+${srcfile}:${line_in_hhh}" \

> +                  "#4  $hex in ggg \\(\\) at \[^\r\n\]+${srcfile}:${line_in_ggg}" \

> +                  "#5  $hex in fff \\(\\) at \[^\r\n\]+${srcfile}:${line_in_fff}" \

>                    "#6  $hex in eee \\(\\) at \[^\r\n\]+${srcfile}:${line_in_eee}" \

>                    "#7  $hex in ddd \\(\\) at \[^\r\n\]+${srcfile}:${line_in_ddd}" \

>                    "#8  $hex in ccc \\(\\) at \[^\r\n\]+${srcfile}:${line_in_ccc}" \

> -                  "#9  bbb \\(\\) at \[^\r\n\]+${srcfile}:${line_in_bbb}" \

> -                  "#10 aaa \\(\\) at \[^\r\n\]+${srcfile}:${line_in_aaa}" \

> -                  "#11 main \\(\\) at \[^\r\n\]+${srcfile}:${line_in_main}" ]

> +                  "#9  $hex in bbb \\(\\) at \[^\r\n\]+${srcfile}:${line_in_bbb}" \

> +                  "#10 $hex in aaa \\(\\) at \[^\r\n\]+${srcfile}:${line_in_aaa}" \

> +                  "#11 $hex in main \\(\\) at \[^\r\n\]+${srcfile}:${line_in_main}" ]

>

>  # Now check we can use 'up' to inspect each frame correctly.

>  set patterns [list  \

> diff --git a/gdb/testsuite/gdb.dwarf2/dw2-inline-param.exp b/gdb/testsuite/gdb.dwarf2/dw2-inline-param.exp

> index 1c1e75619da..e27ef8f3b38 100644

> --- a/gdb/testsuite/gdb.dwarf2/dw2-inline-param.exp

> +++ b/gdb/testsuite/gdb.dwarf2/dw2-inline-param.exp

> @@ -55,4 +55,6 @@ if ![runto "*${break_at}"] {

>      return -1

>  }

>

> -gdb_test "bt" "#0  (0x\[0-9a-f\]+ in )?func \\(funcparam=<optimized out>\\)\r\n#1  main \\(mainparam=<optimized out>\\)\[^\r\n\]*"

> +gdb_test "bt" [multi_line \

> +                  "#0  ($hex in )?func \\(funcparam=<optimized out>\\)" \

> +                  "#1  $hex in main \\(mainparam=<optimized out>\\)\[^\r\n\]*" ]

> diff --git a/gdb/testsuite/gdb.guile/scm-frame-inline.exp b/gdb/testsuite/gdb.guile/scm-frame-inline.exp

> index 8a4d8f893de..70882f841d7 100644

> --- a/gdb/testsuite/gdb.guile/scm-frame-inline.exp

> +++ b/gdb/testsuite/gdb.guile/scm-frame-inline.exp

> @@ -37,7 +37,7 @@ gdb_continue_to_breakpoint "break-here"

>

>  gdb_test "info frame" "inlined into frame 1\r\n.*"

>

> -gdb_test "up" "#1  g .*"

> +gdb_test "up" "#1  $hex in g .*"

>

>  gdb_test "guile (print (frame-read-var (selected-frame) \"l\"))" \

>      "= 42"

> diff --git a/gdb/testsuite/gdb.opt/inline-cmds.exp b/gdb/testsuite/gdb.opt/inline-cmds.exp

> index aa8c8c6bfa0..de5eb1dfee5 100644

> --- a/gdb/testsuite/gdb.opt/inline-cmds.exp

> +++ b/gdb/testsuite/gdb.opt/inline-cmds.exp

> @@ -288,15 +288,15 @@ gdb_test_multiple "finish" "finish from marker" {

>  }

>  gdb_test "bt" "#0  main.*" "backtrace at main of outer_inline"

>  gdb_test "step" "outer_inline2 \\\(\\\) at .*" "enter outer_inline2"

> -gdb_test "bt" "#0  outer_inline2.*#1  main.*" "backtrace at outer_inline2"

> +gdb_test "bt" "#0  outer_inline2.*#1  $hex in main.*" "backtrace at outer_inline2 xx"

>  gdb_test "step" "outer_inline1 \\\(\\\) at .*" "enter outer_inline1 from outer_inline2"

>

>  set msg "backtrace at outer_inline1"

>  gdb_test_multiple "bt" $msg {

> -    -re "#0  outer_inline1.*#1  outer_inline2.*#2  main.*$gdb_prompt $" {

> +    -re "#0  outer_inline1.*#1  $hex in outer_inline2.*#2  $hex in main.*$gdb_prompt $" {

>         pass $msg

>      }

> -    -re "#0  $hex in outer_inline1.*#1  outer_inline2.*#2  main.*$gdb_prompt $" {

> +    -re "#0  $hex in outer_inline1.*#1  $hex in outer_inline2.*#2  $hex in main.*$gdb_prompt $" {

>         # Binutils PR gas/6717.  Gas moves .loc past .p2align and the

>         # leading nop of the inlined call appears to be on the same line

>         # as main's call to marker.

> @@ -306,17 +306,17 @@ gdb_test_multiple "bt" $msg {

>  }

>

>  gdb_test "step" "noinline \\\(\\\) at .*" "enter noinline from outer_inline1"

> -gdb_test "bt" "#0  noinline.*#1  .*outer_inline1.*#2  .*outer_inline2.*#3  main.*" "backtrace at noinline from outer_inline1"

> +gdb_test "bt" "#0  noinline.*#1  .*outer_inline1.*#2  .*outer_inline2.*#3  $hex in main.*" "backtrace at noinline from outer_inline1"

>  gdb_test "step" "inlined_fn \\\(\\\) at .*" "enter inlined_fn from noinline"

> -gdb_test "bt" "#0  inlined_fn.*#1  noinline.*#2  .*outer_inline1.*#3  .*outer_inline2.*#4  main.*" "backtrace at inlined_fn from noinline"

> +gdb_test "bt" "#0  inlined_fn.*#1  $hex in noinline.*#2  .*outer_inline1.*#3  .*outer_inline2.*#4  $hex in main.*" "backtrace at inlined_fn from noinline"

>  gdb_test "info frame" ".*inlined into frame.*" "inlined_fn from noinline inlined"

> -gdb_test "up" "#1  noinline.*" "up to noinline"

> +gdb_test "up" "#1  $hex in noinline.*" "up to noinline"

>  gdb_test "info frame" ".*\n called by frame.*" "noinline from outer_inline1 not inlined"

>  gdb_test "up" "#2  .*outer_inline1.*" "up to outer_inline1"

>  gdb_test "info frame" ".*inlined into frame.*" "outer_inline1 inlined"

>  gdb_test "up" "#3  .*outer_inline2.*" "up to outer_inline2"

>  gdb_test "info frame" ".*inlined into frame.*" "outer_inline2 inlined"

> -gdb_test "up" "#4  main.*" "up from outer_inline2"

> +gdb_test "up" "#4  $hex in main.*" "up from outer_inline2"

>  gdb_test "info frame" ".*\n caller of frame.*" "main not inlined"

>

>  gdb_exit

> diff --git a/gdb/testsuite/gdb.python/py-frame-inline.exp b/gdb/testsuite/gdb.python/py-frame-inline.exp

> index 71bffd375db..aa22001b0d8 100644

> --- a/gdb/testsuite/gdb.python/py-frame-inline.exp

> +++ b/gdb/testsuite/gdb.python/py-frame-inline.exp

> @@ -30,11 +30,11 @@ if ![runto main] then {

>  }

>

>  gdb_breakpoint [gdb_get_line_number "break-here"]

> -gdb_continue_to_breakpoint "Block break here."

> +gdb_continue_to_breakpoint "continue to 'break here', first time"

>

>  gdb_test "info frame" "inlined into frame 1\r\n.*"

>

> -gdb_test "up" "#1  g .*"

> +gdb_test "up" "#1  $hex in g .*"

>

>  gdb_test "python print (gdb.selected_frame().read_var('l'))" "\r\n42"

>

> @@ -48,7 +48,7 @@ gdb_test "python print (gdb.selected_frame().read_var('l'))" "\r\n42"

>  # the frame cache is flushed somehow after setting the limit, to force

>  # frame id recomputation.

>  gdb_test_no_output "set backtrace limit 1"

> -gdb_continue_to_breakpoint "Block break here."

> +gdb_continue_to_breakpoint "continue to 'break here', second time"

>

>  gdb_test "python print (gdb.newest_frame())" ".*"

>

> --

> 2.14.5

>
Pedro Alves via Gdb-patches March 13, 2020, 12:03 a.m. | #2
On Wed,  4 Mar 2020 14:12:58 +0000
Andrew Burgess <andrew.burgess@embecosm.com> wrote:
 
> The duplicated address might also be confusing, though (personally) I

> think it is slightly more obvious from the duplicated addresses that

> the frames are inlined, however, if people strongly disagree and

> prefer the no-address layout we could make this feature switchable.


I prefer the no-address layout.  I think it might be worth making it
switchable.

Kevin

Patch

diff --git a/gdb/doc/gdb.texinfo b/gdb/doc/gdb.texinfo
index 32e419ead02..74178faee37 100644
--- a/gdb/doc/gdb.texinfo
+++ b/gdb/doc/gdb.texinfo
@@ -8013,6 +8013,24 @@ 
 either deduce that from other variables whose values depend on the one
 you are interested in, or recompile without optimizations.
 
+If your compiler has performed the function inlining optimisation
+(@pxref{Inline Functions}) then this will be reflected in the program
+counter addresses displayed in the backtrace.  In the following
+modified example the function @code{expand_macro} has been inlined
+within @code{expand_token}, as a result the program counter values for
+frame 1 and frame 2 are the same.
+
+@smallexample
+@group
+#0  m4_traceon (obs=0x24eb0, argc=1, argv=0x2b8c8)
+    at builtin.c:993
+#1  0x6e38 in expand_macro (sym=<optimized out>) at macro.c:242
+#2  0x6e38 in expand_token (obs=0x0, t=<optimized out>, td=0xf7fffb08)
+    at macro.c:71
+(More stack frames follow...)
+@end group
+@end smallexample
+
 @cindex backtrace beyond @code{main} function
 @cindex program entry point
 @cindex startup code, and backtrace
diff --git a/gdb/stack.c b/gdb/stack.c
index 266d771e35f..4570eed1a2f 100644
--- a/gdb/stack.c
+++ b/gdb/stack.c
@@ -327,7 +327,7 @@  frame_show_address (struct frame_info *frame,
 	gdb_assert (inline_skipped_frames (inferior_thread ()) > 0);
       else
 	gdb_assert (get_frame_type (get_next_frame (frame)) == INLINE_FRAME);
-      return false;
+      return frame_relative_level (frame) > 0;
     }
 
   return get_frame_pc (frame) != sal.pc;
diff --git a/gdb/testsuite/gdb.dwarf2/dw2-inline-many-frames.exp b/gdb/testsuite/gdb.dwarf2/dw2-inline-many-frames.exp
index 146af8c6ef7..f5890fb2aca 100644
--- a/gdb/testsuite/gdb.dwarf2/dw2-inline-many-frames.exp
+++ b/gdb/testsuite/gdb.dwarf2/dw2-inline-many-frames.exp
@@ -350,15 +350,15 @@  gdb_test "bt" [multi_line \
 		   "#0  kkk \\(\\) at \[^\r\n\]+${srcfile}:${line_in_kkk}" \
 		   "#1  $hex in jjj \\(\\) at \[^\r\n\]+${srcfile}:${line_in_jjj}" \
 		   "#2  $hex in iii \\(\\) at \[^\r\n\]+${srcfile}:${line_in_iii}" \
-		   "#3  hhh \\(\\) at \[^\r\n\]+${srcfile}:${line_in_hhh}" \
-		   "#4  ggg \\(\\) at \[^\r\n\]+${srcfile}:${line_in_ggg}" \
-		   "#5  fff \\(\\) at \[^\r\n\]+${srcfile}:${line_in_fff}" \
+		   "#3  $hex in hhh \\(\\) at \[^\r\n\]+${srcfile}:${line_in_hhh}" \
+		   "#4  $hex in ggg \\(\\) at \[^\r\n\]+${srcfile}:${line_in_ggg}" \
+		   "#5  $hex in fff \\(\\) at \[^\r\n\]+${srcfile}:${line_in_fff}" \
 		   "#6  $hex in eee \\(\\) at \[^\r\n\]+${srcfile}:${line_in_eee}" \
 		   "#7  $hex in ddd \\(\\) at \[^\r\n\]+${srcfile}:${line_in_ddd}" \
 		   "#8  $hex in ccc \\(\\) at \[^\r\n\]+${srcfile}:${line_in_ccc}" \
-		   "#9  bbb \\(\\) at \[^\r\n\]+${srcfile}:${line_in_bbb}" \
-		   "#10 aaa \\(\\) at \[^\r\n\]+${srcfile}:${line_in_aaa}" \
-		   "#11 main \\(\\) at \[^\r\n\]+${srcfile}:${line_in_main}" ]
+		   "#9  $hex in bbb \\(\\) at \[^\r\n\]+${srcfile}:${line_in_bbb}" \
+		   "#10 $hex in aaa \\(\\) at \[^\r\n\]+${srcfile}:${line_in_aaa}" \
+		   "#11 $hex in main \\(\\) at \[^\r\n\]+${srcfile}:${line_in_main}" ]
 
 # Now check we can use 'up' to inspect each frame correctly.
 set patterns [list  \
diff --git a/gdb/testsuite/gdb.dwarf2/dw2-inline-param.exp b/gdb/testsuite/gdb.dwarf2/dw2-inline-param.exp
index 1c1e75619da..e27ef8f3b38 100644
--- a/gdb/testsuite/gdb.dwarf2/dw2-inline-param.exp
+++ b/gdb/testsuite/gdb.dwarf2/dw2-inline-param.exp
@@ -55,4 +55,6 @@  if ![runto "*${break_at}"] {
     return -1
 }
 
-gdb_test "bt" "#0  (0x\[0-9a-f\]+ in )?func \\(funcparam=<optimized out>\\)\r\n#1  main \\(mainparam=<optimized out>\\)\[^\r\n\]*"
+gdb_test "bt" [multi_line \
+		   "#0  ($hex in )?func \\(funcparam=<optimized out>\\)" \
+		   "#1  $hex in main \\(mainparam=<optimized out>\\)\[^\r\n\]*" ]
diff --git a/gdb/testsuite/gdb.guile/scm-frame-inline.exp b/gdb/testsuite/gdb.guile/scm-frame-inline.exp
index 8a4d8f893de..70882f841d7 100644
--- a/gdb/testsuite/gdb.guile/scm-frame-inline.exp
+++ b/gdb/testsuite/gdb.guile/scm-frame-inline.exp
@@ -37,7 +37,7 @@  gdb_continue_to_breakpoint "break-here"
 
 gdb_test "info frame" "inlined into frame 1\r\n.*"
 
-gdb_test "up" "#1  g .*"
+gdb_test "up" "#1  $hex in g .*"
 
 gdb_test "guile (print (frame-read-var (selected-frame) \"l\"))" \
     "= 42"
diff --git a/gdb/testsuite/gdb.opt/inline-cmds.exp b/gdb/testsuite/gdb.opt/inline-cmds.exp
index aa8c8c6bfa0..de5eb1dfee5 100644
--- a/gdb/testsuite/gdb.opt/inline-cmds.exp
+++ b/gdb/testsuite/gdb.opt/inline-cmds.exp
@@ -288,15 +288,15 @@  gdb_test_multiple "finish" "finish from marker" {
 }
 gdb_test "bt" "#0  main.*" "backtrace at main of outer_inline"
 gdb_test "step" "outer_inline2 \\\(\\\) at .*" "enter outer_inline2"
-gdb_test "bt" "#0  outer_inline2.*#1  main.*" "backtrace at outer_inline2"
+gdb_test "bt" "#0  outer_inline2.*#1  $hex in main.*" "backtrace at outer_inline2 xx"
 gdb_test "step" "outer_inline1 \\\(\\\) at .*" "enter outer_inline1 from outer_inline2"
 
 set msg "backtrace at outer_inline1"
 gdb_test_multiple "bt" $msg {
-    -re "#0  outer_inline1.*#1  outer_inline2.*#2  main.*$gdb_prompt $" {
+    -re "#0  outer_inline1.*#1  $hex in outer_inline2.*#2  $hex in main.*$gdb_prompt $" {
 	pass $msg
     }
-    -re "#0  $hex in outer_inline1.*#1  outer_inline2.*#2  main.*$gdb_prompt $" {
+    -re "#0  $hex in outer_inline1.*#1  $hex in outer_inline2.*#2  $hex in main.*$gdb_prompt $" {
 	# Binutils PR gas/6717.  Gas moves .loc past .p2align and the
 	# leading nop of the inlined call appears to be on the same line
 	# as main's call to marker.
@@ -306,17 +306,17 @@  gdb_test_multiple "bt" $msg {
 }
 
 gdb_test "step" "noinline \\\(\\\) at .*" "enter noinline from outer_inline1"
-gdb_test "bt" "#0  noinline.*#1  .*outer_inline1.*#2  .*outer_inline2.*#3  main.*" "backtrace at noinline from outer_inline1"
+gdb_test "bt" "#0  noinline.*#1  .*outer_inline1.*#2  .*outer_inline2.*#3  $hex in main.*" "backtrace at noinline from outer_inline1"
 gdb_test "step" "inlined_fn \\\(\\\) at .*" "enter inlined_fn from noinline"
-gdb_test "bt" "#0  inlined_fn.*#1  noinline.*#2  .*outer_inline1.*#3  .*outer_inline2.*#4  main.*" "backtrace at inlined_fn from noinline"
+gdb_test "bt" "#0  inlined_fn.*#1  $hex in noinline.*#2  .*outer_inline1.*#3  .*outer_inline2.*#4  $hex in main.*" "backtrace at inlined_fn from noinline"
 gdb_test "info frame" ".*inlined into frame.*" "inlined_fn from noinline inlined"
-gdb_test "up" "#1  noinline.*" "up to noinline"
+gdb_test "up" "#1  $hex in noinline.*" "up to noinline"
 gdb_test "info frame" ".*\n called by frame.*" "noinline from outer_inline1 not inlined"
 gdb_test "up" "#2  .*outer_inline1.*" "up to outer_inline1"
 gdb_test "info frame" ".*inlined into frame.*" "outer_inline1 inlined"
 gdb_test "up" "#3  .*outer_inline2.*" "up to outer_inline2"
 gdb_test "info frame" ".*inlined into frame.*" "outer_inline2 inlined"
-gdb_test "up" "#4  main.*" "up from outer_inline2"
+gdb_test "up" "#4  $hex in main.*" "up from outer_inline2"
 gdb_test "info frame" ".*\n caller of frame.*" "main not inlined"
 
 gdb_exit
diff --git a/gdb/testsuite/gdb.python/py-frame-inline.exp b/gdb/testsuite/gdb.python/py-frame-inline.exp
index 71bffd375db..aa22001b0d8 100644
--- a/gdb/testsuite/gdb.python/py-frame-inline.exp
+++ b/gdb/testsuite/gdb.python/py-frame-inline.exp
@@ -30,11 +30,11 @@  if ![runto main] then {
 }
 
 gdb_breakpoint [gdb_get_line_number "break-here"]
-gdb_continue_to_breakpoint "Block break here."
+gdb_continue_to_breakpoint "continue to 'break here', first time"
 
 gdb_test "info frame" "inlined into frame 1\r\n.*"
 
-gdb_test "up" "#1  g .*"
+gdb_test "up" "#1  $hex in g .*"
 
 gdb_test "python print (gdb.selected_frame().read_var('l'))" "\r\n42"
 
@@ -48,7 +48,7 @@  gdb_test "python print (gdb.selected_frame().read_var('l'))" "\r\n42"
 # the frame cache is flushed somehow after setting the limit, to force
 # frame id recomputation.
 gdb_test_no_output "set backtrace limit 1"
-gdb_continue_to_breakpoint "Block break here."
+gdb_continue_to_breakpoint "continue to 'break here', second time"
 
 gdb_test "python print (gdb.newest_frame())" ".*"