ABOUT
hung_task_all_cpu_backtrace
If this option is set, the kernel will send an NMI to all CPUs to dump their backtraces when a hung task
is detected. This file shows up if CONFIG_DETECT_HUNG_TASK and CONFIG_SMP are enabled.
0: Won’t show all CPUs backtraces when a hung task is detected. This is the default behavior
1: Will non-maskably interrupt all CPUs and dump their backtraces when a hung task is detected.
LINUX KERNEL RELATED PARAMETER
$cat /proc/sys/kernel/hung_task_all_cpu_backtrace
0
$echo 1 | sudo tee –append /proc/sys/kernel/hung_task_all_cpu_backtrace
[sudo] password for jeffrin:
1
$cat /proc/sys/kernel/hung_task_all_cpu_backtrace
1
$
PARAMETER RELATED
USERSPACE PROCESS BACKTRACE
(gdb) bt
#0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56
#1 0x00007ffff64197a4 in __internal_syscall_cancel
(a1=, a2=, a3=, a4=, a5=a5@entry=8, a6=a6@entry=0, nr=271) at ./nptl/cancellation.c:49
#2 0x00007ffff64197ed in __syscall_cancel
(a1=, a2=, a3=, a4=, a5=a5@entry=8, a6=a6@entry=0, nr=271) at ./nptl/cancellation.c:75
#3 0x00007ffff648f29e in __GI_ppoll
(fds=, nfds=, timeout=, sigmask=) at ../sysdeps/unix/sysv/linux/ppoll.c:42
#4 0x00007ffff7951b34 in ??? () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#5 0x00007ffff7952210 in g_main_context_iteration ()
at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#6 0x00007ffff76fe13d in g_application_run ()
at /usr/lib/x86_64-linux-gnu/libgio-2.0.so.0
#7 0x000055555580dfad in app_run ()
#8 0x0000555555641f3b in main ()
(gdb)
BACKTRACE FROM KERNEL RING BUFFER
echo 1 | sudo tee /proc/sys/kernel/sysrq
echo l | sudo tee /proc/sysrq-trigger.
dmesg
KERNEL SPACE BACKTRACE
[30208.196481] sysrq: Show backtrace of all active CPUs
[30208.196491] NMI backtrace for cpu 0
[30208.196494] CPU: 0 UID: 0 PID: 45306 Comm: tee Not tainted 6.18.20-rc1+ #61 PREEMPT(lazy)
[30208.196498] Hardware name: HP HP Laptop 15s-ey1xxx/8706, BIOS F.61 03/25/2022
[30208.196500] Call Trace:
[30208.196503] <TASK>code
[30208.196507] dump_stack_lvl+0x5d/0x80
[30208.196514] nmi_cpu_backtrace.cold+0x18/0x65
[30208.196518] ? __pfx_nmi_raise_cpu_backtrace+0x10/0x10
[30208.196524] nmi_trigger_cpumask_backtrace+0xf4/0x110
[30208.196529] __handle_sysrq.cold+0x8a/0x160
[30208.196534] write_sysrq_trigger+0x59/0xa0
[30208.196540] proc_reg_write+0x5a/0xb0
[30208.196544] vfs_write+0xce/0x480
[30208.196552] ksys_write+0x73/0xf0
[30208.196556] do_syscall_64+0x81/0x7f0
[30208.196562] ? srso_return_thunk+0x5/0x5f
[30208.196565] ? __x64_sys_openat+0x61/0xa0
[30208.196569] ? srso_return_thunk+0x5/0x5f
[30208.196571] ? do_syscall_64+0xb9/0x7f0
[30208.196576] ? srso_return_thunk+0x5/0x5f
[30208.196579] ? ksys_fadvise64_64+0x6c/0xa0
[30208.196584] ? srso_return_thunk+0x5/0x5f
[30208.196587] ? srso_return_thunk+0x5/0x5f
[30208.196589] ? do_syscall_64+0xb9/0x7f0
[30208.196593] ? srso_return_thunk+0x5/0x5f
[30208.196597] ? do_user_addr_fault+0x21a/0x690
[30208.196602] ? srso_return_thunk+0x5/0x5f
[30208.196605] ? srso_return_thunk+0x5/0x5f
[30208.196609] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[30208.196613] RIP: 0033:0x7f8875a3a7d2
[30208.196638] Code: 18 41 8b 93 08 03 00 00 59 5e 48 83 f8 fc 75 1a 83 e2 39 83 fa 08 75 12 e8 2b ff ff ff 0f 1f 00 49 89 ca 48 8b 44 24 20 0f 05 <48> 83 c4 18 c3 66 0f 1f 84 00 00 00 00 00 48 83 ec 10 ff 74 24 18
[30208.196640] RSP: 002b:00007fffda92a5b0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[30208.196644] RAX: ffffffffffffffda RBX: 000055b4a7c9c310 RCX: 00007f8875a3a7d2
[30208.196645] RDX: 0000000000000002 RSI: 00007fffda92a750 RDI: 0000000000000003
[30208.196647] RBP: 0000000000000002 R08: 0000000000000000 R09: 0000000000000000
[30208.196649] R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000002
[30208.196650] R13: 00007fffda92a750 R14: 00007fffda92a750 R15: 0000000000000002
[30208.196659] </TASK>
[30208.196688] Sending NMI from CPU 0 to CPUs 1-3:
[30208.196697] NMI backtrace for cpu 1
[30208.196699] CPU: 1 UID: 0 PID: 345 Comm: systemd-journal Not tainted 6.18.20-rc1+ #61 PREEMPT(lazy)
[30208.196702] Hardware name: HP HP Laptop 15s-ey1xxx/8706, BIOS F.61 03/25/2022
[30208.196703] RIP: 0033:0x7f51c5e71490
[30208.196710] Code: bd fd ff ff b8 f4 ff ff ff e9 40 fd ff ff 31 d2 e9 4c ff ff ff 89 da e9 45 ff ff ff e8 f9 8c e4 ff e8 f4 fb ff ff 0f 1f 40 00 0f 1e fa 41 54 49 89 d4 55 48 89 cd 53 48 89 fb e8 1a 91 ff ff
[30208.196712] RSP: 002b:00007ffc7ed4ca08 EFLAGS: 00000246
[30208.196714] RAX: 0000000000000000 RBX: 000056303827baf0 RCX: 0000000000000000
[30208.196716] RDX: 00007ffc7ed4ca18 RSI: 00007ffc7ed4ca20 RDI: 0000000000000000
[30208.196717] RBP: 00007ffc7ed4ca20 R08: 00007f51c62b3070 R09: 0000000000000000
[30208.196718] R10: 00007f51c62b3000 R11: 0000000000c1eb8a R12: 000056303827baf0
[30208.196720] R13: 0000000000000000 R14: ffffffffffffffff R15: 000056302ba21658
[30208.196721] FS: 00007f51c6172840 GS: 0000000000000000
[30208.196723] NMI backtrace for cpu 2 skipped: idling at io_idle+0x3/0x30
[30208.196736] NMI backtrace for cpu 3
[30208.196741] CPU: 3 UID: 0 PID: 806 Comm: in:imklog Not tainted 6.18.20-rc1+ #61 PREEMPT(lazy)
[30208.196749] Hardware name: HP HP Laptop 15s-ey1xxx/8706, BIOS F.61 03/25/2022
[30208.196752] RIP: 0010:vsnprintf+0x355/0x5e0
[30208.196761] Code: 0a 0f b6 03 49 89 df eb 27 4c 89 e0 48 8b 7c 24 30 48 8b 4c 24 20 4c 29 f8 4c 63 c0 48 39 cf 0f 82 cb 00 00 00 4c 01 44 24 30 <0f> b6 03 49 89 df 84 c0 0f 85 29 fd ff ff e9 09 fe ff ff 48 8b 44
[30208.196765] RSP: 0018:ffffd0d14ce93878 EFLAGS: 00000286
[30208.196771] RAX: ffffd0d14ce93992 RBX: ffffffff9358ebf2 RCX: 000000000000003e
[30208.196775] RDX: 0000000000000000 RSI: ffffffff9358ebf1 RDI: ffffd0d14ce93992
[30208.196779] RBP: ffffd0d14ce938e8 R08: 0000000000000001 R09: 00000000ffffffff
[30208.196782] R10: 0000000000000000 R11: 0000000000000004 R12: ffffffff9358ebf2
[30208.196785] R13: 0000000000000400 R14: 0000000000000400 R15: ffffffff9358ebf1
[30208.196789] FS: 00007f2e35a6b6c0(0000) GS:ffff8f48445af000(0000) knlGS:0000000000000000
[30208.196794] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[30208.196798] CR2: 000055b82387b970 CR3: 000000010a0e0000 CR4: 00000000003506f0
[30208.196802] Call Trace:
[30208.196805] <TASK>
[30208.196818] sprintf+0x5f/0x80
[30208.196828] ? srso_return_thunk+0x5/0x5f
[30208.196837] info_print_prefix+0x57/0xe0
[30208.196846] record_print_text+0x7a/0x170
[30208.196860] syslog_print+0x274/0x360
[30208.196869] ? __pfx_autoremove_wake_function+0x10/0x10
[30208.196886] do_syslog+0xd5/0x400
[30208.196900] ? srso_return_thunk+0x5/0x5f
[30208.196910] kmsg_read+0x57/0x70
[30208.196918] ? srso_return_thunk+0x5/0x5f
[30208.196924] ? __pfx_kmsg_read+0x10/0x10
[30208.196930] vfs_read+0xbf/0x390
[30208.196937] ? __seccomp_filter+0x42/0x590
[30208.196953] ksys_read+0x73/0xf0
[30208.196963] do_syscall_64+0x81/0x7f0
[30208.196972] ? srso_return_thunk+0x5/0x5f
[30208.196978] ? srso_return_thunk+0x5/0x5f
[30208.196984] ? wakeup_preempt+0x89/0xa0
[30208.196992] ? srso_return_thunk+0x5/0x5f
[30208.196998] ? ttwu_do_activate+0x86/0x250
[30208.197005] ? srso_return_thunk+0x5/0x5f
[30208.197010] ? update_rq_clock+0x81/0x180
[30208.197019] ? srso_return_thunk+0x5/0x5f
[30208.197024] ? try_to_wake_up+0x371/0x8b0
[30208.197032] ? srso_return_thunk+0x5/0x5f
[30208.197037] ? plist_del+0x5f/0xd0
[30208.197046] ? srso_return_thunk+0x5/0x5f
[30208.197052] ? wake_up_q+0x37/0x90
[30208.197059] ? srso_return_thunk+0x5/0x5f
[30208.197064] ? futex_wake+0xac/0x1c0
[30208.197077] ? srso_return_thunk+0x5/0x5f
[30208.197082] ? srso_return_thunk+0x5/0x5f
[30208.197088] ? __x64_sys_futex+0x12d/0x210
[30208.197101] ? srso_return_thunk+0x5/0x5f
[30208.197106] ? do_syscall_64+0xb9/0x7f0
[30208.197114] ? srso_return_thunk+0x5/0x5f
[30208.197120] ? switch_fpu_return+0x5b/0xe0
[30208.197128] ? srso_return_thunk+0x5/0x5f
[30208.197134] ? do_syscall_64+0x240/0x7f0
[30208.197146] ? srso_return_thunk+0x5/0x5f
[30208.197151] ? do_syscall_64+0xb9/0x7f0
[30208.197158] ? __irq_exit_rcu+0x3d/0xe0
[30208.197164] ? srso_return_thunk+0x5/0x5f
[30208.197172] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[30208.197177] RIP: 0033:0x7f2e35fd3ffe
[30208.197195] Code: 08 0f 85 15 47 ff ff 49 89 fb 48 89 f0 48 89 d7 48 89 ce 4c 89 c2 4d 89 ca 4c 8b 44 24 08 4c 8b 4c 24 10 4c 89 5c 24 08 0f 05 90 48 83 ec 08 bf 01 00 00 00 e8 e2 eb 05 00 bf 01 00 00 00 89
[30208.197199] RSP: 002b:00007f2e35a4a3b8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[30208.197205] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f2e35fd3ffe
[30208.197208] RDX: 0000000000001fa0 RSI: 00007f2e35a4ac30 RDI: 0000000000000005
[30208.197211] RBP: 00007f2e35a4b9c3 R08: 0000000000000000 R09: 0000000000000000
[30208.197214] R10: 0000000000000000 R11: 0000000000000246 R12: 000056345780d7a0
[30208.197217] R13: 00007f2e35a4ac30 R14: 0000000000001fa0 R15: 00007f2e35a4ac30
[30208.197232] </TASK>
SHELL CLASS ROOM
$pidof gimp
75203
$$sudo cat /proc/75203/stack
cat: /proc/75203/stack: Permission denied
$sudo cat /proc/75203/stack
[sudo] password for jeffrin:
[<0>] poll_schedule_timeout.constprop.0+0x37/0x90
[<0>] do_sys_poll+0x3fc/0x590
[<0>] __x64_sys_ppoll+0xdf/0x180
[<0>] do_syscall_64+0x81/0x7f0
[<0>] entry_SYSCALL_64_after_hwframe+0x76/0x7e
$kill -STOP 75203
$sudo cat /proc/75203/stack
[<0>] do_signal_stop+0x16f/0x200
[<0>] get_signal+0x230/0x920
[<0>] arch_do_signal_or_restart+0x41/0x260
[<0>] exit_to_user_mode_loop+0x8a/0x130
[<0>] do_syscall_64+0x206/0x7f0
[<0>] entry_SYSCALL_64_after_hwframe+0x76/0x7e
$kill -CONT 75203
$sudo cat /proc/75203/stack
[<0>] poll_schedule_timeout.constprop.0+0x37/0x90
[<0>] do_sys_poll+0x3fc/0x590
[<0>] __x64_sys_ppoll+0xdf/0x180
[<0>] do_syscall_64+0x81/0x7f0
[<0>] entry_SYSCALL_64_after_hwframe+0x76/0x7e
$
What is do_signal_stop?
do_signal_stop is an internal function in the Linux kernel (specifically in kernel/signal.c)
that handles group stops. This occurs when a process receives a signal like SIGSTOP, SIGTSTP, SIGTTIN, or
SIGTTOU, which tells the process to pause execution.
Its primary responsibilities include:
Transitioning State: It sets the process state to TASK_STOPPED.
Managing Group Stops: If one thread in a thread group initiates a stop, this function ensures
the entire group participates in the stop. Interaction with Debuggers: It handles special cases if a
process is being traced by a debugger (like ptrace).
Why is the address all zeros?
When you see 0000000000000000, it usually indicates that kernel pointer restriction is enabled on your system.
Security Feature: To prevent attackers from knowing where critical functions are located in memory
(which could be used for exploits), the kernel hides the real addresses.
kptr_restrict: This is controlled by the file /proc/sys/kernel/kptr_restrict. If the value
is set to 1 or 2, non-privileged users will only see zeros for kernel symbols.
Technical Context
The "t" in the output stands for a local symbol in the text (code) section of the kernel
binary. If it were a capital "T", it would be a global symbol. Are you currently
troubleshooting a kernel crash or a frozen process where this function appeared in a stack trace?
SOURCE: GOOGLE AI MODE
RELATED SOURCE CODE EXPOSURE
#ifdef CONFIG_SMP
{
.procname = "hung_task_all_cpu_backtrace",
.data = &sysctl_hung_task_all_cpu_backtrace,
.maxlen = sizeof(int),
.mode = 0644,
.proc_handler = proc_dointvec_minmax,
.extra1 = SYSCTL_ZERO,
.extra2 = SYSCTL_ONE,
},
#endif /* CONFIG_SMP */
RELATED FROM GOOGLE AI MODE
The hung_task_all_cpu_backtrace feature is not derived from a traditional academic research paper, but
rather from a Linux kernel development initiative designed to improve debugging for blocked tasks (D-
state). This mechanism was introduced to decouple panic events from CPU backtrace dumps, allowing
developers to see what all CPUs are doing when a hung task is detected.
Key Details of hung_task_all_cpu_backtrace
Purpose: To aid in debugging complex hang scenarios (e.g., deadlocks) by providing a snapshot
of the entire system state, not just the stuck task.
Mechanism: It introduces a new sysctl (kernel.hung_task_all_cpu_backtrace) analogous to the
approaches used for soft/hard lockups.
Implementation: When hung_task_warnings are still active, this feature triggers a backtrace of
all CPUs upon detecting a task that has been in an uninterruptible sleep for longer than
hung_task_timeout_secs.
Context: It is part of the kernel/hung_task.c framework in the Linux kernel.
Related Research and Concepts
Kernel Tracing: Techniques like Ftrace and other tracers are heavily used to debug and reverse-
engineer complex systems, often used to determine the execution frequency of kernel paths.
Deadlock Detection: Other studies focus on optimizing the performance of tracing to analyze
system bottlenecks.