Blog – TRUEANGLE

Featured

hung_task_all_cpu_backtrace( topic related to CPUs dumping their backtraces )

ABOUT

hung_task_all_cpu_backtrace

If this option is set, the kernel will send an NMI to all CPUs to dump their backtraces when a hung task is detected. This file shows up if CONFIG_DETECT_HUNG_TASK and CONFIG_SMP are enabled. 0: Won’t show all CPUs backtraces when a hung task is detected. This is the default behavior 1: Will non-maskably interrupt all CPUs and dump their backtraces when a hung task is detected.

LINUX KERNEL RELATED PARAMETER

$cat /proc/sys/kernel/hung_task_all_cpu_backtrace 0 $echo 1 | sudo tee –append /proc/sys/kernel/hung_task_all_cpu_backtrace [sudo] password for jeffrin: 1 $cat /proc/sys/kernel/hung_task_all_cpu_backtrace 1 $

PARAMETER RELATED

USERSPACE PROCESS BACKTRACE

(gdb) bt #0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 #1 0x00007ffff64197a4 in __internal_syscall_cancel (a1=, a2=, a3=, a4=, a5=a5@entry=8, a6=a6@entry=0, nr=271) at ./nptl/cancellation.c:49 #2 0x00007ffff64197ed in __syscall_cancel (a1=, a2=, a3=, a4=, a5=a5@entry=8, a6=a6@entry=0, nr=271) at ./nptl/cancellation.c:75 #3 0x00007ffff648f29e in __GI_ppoll (fds=, nfds=, timeout=, sigmask=) at ../sysdeps/unix/sysv/linux/ppoll.c:42 #4 0x00007ffff7951b34 in ??? () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0 #5 0x00007ffff7952210 in g_main_context_iteration () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0 #6 0x00007ffff76fe13d in g_application_run () at /usr/lib/x86_64-linux-gnu/libgio-2.0.so.0 #7 0x000055555580dfad in app_run () #8 0x0000555555641f3b in main () (gdb)

BACKTRACE FROM LINUX KERNEL SPACE

echo 1 | sudo tee /proc/sys/kernel/sysrq


echo l | sudo tee /proc/sysrq-trigger.
dmesg

[30208.196481] sysrq: Show backtrace of all active CPUs [30208.196491] NMI backtrace for cpu 0 [30208.196494] CPU: 0 UID: 0 PID: 45306 Comm: tee Not tainted 6.18.20-rc1+ #61 PREEMPT(lazy) [30208.196498] Hardware name: HP HP Laptop 15s-ey1xxx/8706, BIOS F.61 03/25/2022 [30208.196500] Call Trace: [30208.196503] <TASK>code [30208.196507] dump_stack_lvl+0x5d/0x80 [30208.196514] nmi_cpu_backtrace.cold+0x18/0x65 [30208.196518] ? __pfx_nmi_raise_cpu_backtrace+0x10/0x10 [30208.196524] nmi_trigger_cpumask_backtrace+0xf4/0x110 [30208.196529] __handle_sysrq.cold+0x8a/0x160 [30208.196534] write_sysrq_trigger+0x59/0xa0 [30208.196540] proc_reg_write+0x5a/0xb0 [30208.196544] vfs_write+0xce/0x480 [30208.196552] ksys_write+0x73/0xf0 [30208.196556] do_syscall_64+0x81/0x7f0 [30208.196562] ? srso_return_thunk+0x5/0x5f [30208.196565] ? __x64_sys_openat+0x61/0xa0 [30208.196569] ? srso_return_thunk+0x5/0x5f [30208.196571] ? do_syscall_64+0xb9/0x7f0 [30208.196576] ? srso_return_thunk+0x5/0x5f [30208.196579] ? ksys_fadvise64_64+0x6c/0xa0 [30208.196584] ? srso_return_thunk+0x5/0x5f [30208.196587] ? srso_return_thunk+0x5/0x5f [30208.196589] ? do_syscall_64+0xb9/0x7f0 [30208.196593] ? srso_return_thunk+0x5/0x5f [30208.196597] ? do_user_addr_fault+0x21a/0x690 [30208.196602] ? srso_return_thunk+0x5/0x5f [30208.196605] ? srso_return_thunk+0x5/0x5f [30208.196609] entry_SYSCALL_64_after_hwframe+0x76/0x7e [30208.196613] RIP: 0033:0x7f8875a3a7d2 [30208.196638] Code: 18 41 8b 93 08 03 00 00 59 5e 48 83 f8 fc 75 1a 83 e2 39 83 fa 08 75 12 e8 2b ff ff ff 0f 1f 00 49 89 ca 48 8b 44 24 20 0f 05 <48> 83 c4 18 c3 66 0f 1f 84 00 00 00 00 00 48 83 ec 10 ff 74 24 18 [30208.196640] RSP: 002b:00007fffda92a5b0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [30208.196644] RAX: ffffffffffffffda RBX: 000055b4a7c9c310 RCX: 00007f8875a3a7d2 [30208.196645] RDX: 0000000000000002 RSI: 00007fffda92a750 RDI: 0000000000000003 [30208.196647] RBP: 0000000000000002 R08: 0000000000000000 R09: 0000000000000000 [30208.196649] R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000002 [30208.196650] R13: 00007fffda92a750 R14: 00007fffda92a750 R15: 0000000000000002 [30208.196659] </TASK> [30208.196688] Sending NMI from CPU 0 to CPUs 1-3: [30208.196697] NMI backtrace for cpu 1 [30208.196699] CPU: 1 UID: 0 PID: 345 Comm: systemd-journal Not tainted 6.18.20-rc1+ #61 PREEMPT(lazy) [30208.196702] Hardware name: HP HP Laptop 15s-ey1xxx/8706, BIOS F.61 03/25/2022 [30208.196703] RIP: 0033:0x7f51c5e71490 [30208.196710] Code: bd fd ff ff b8 f4 ff ff ff e9 40 fd ff ff 31 d2 e9 4c ff ff ff 89 da e9 45 ff ff ff e8 f9 8c e4 ff e8 f4 fb ff ff 0f 1f 40 00 0f 1e fa 41 54 49 89 d4 55 48 89 cd 53 48 89 fb e8 1a 91 ff ff [30208.196712] RSP: 002b:00007ffc7ed4ca08 EFLAGS: 00000246 [30208.196714] RAX: 0000000000000000 RBX: 000056303827baf0 RCX: 0000000000000000 [30208.196716] RDX: 00007ffc7ed4ca18 RSI: 00007ffc7ed4ca20 RDI: 0000000000000000 [30208.196717] RBP: 00007ffc7ed4ca20 R08: 00007f51c62b3070 R09: 0000000000000000 [30208.196718] R10: 00007f51c62b3000 R11: 0000000000c1eb8a R12: 000056303827baf0 [30208.196720] R13: 0000000000000000 R14: ffffffffffffffff R15: 000056302ba21658 [30208.196721] FS: 00007f51c6172840 GS: 0000000000000000 [30208.196723] NMI backtrace for cpu 2 skipped: idling at io_idle+0x3/0x30 [30208.196736] NMI backtrace for cpu 3 [30208.196741] CPU: 3 UID: 0 PID: 806 Comm: in:imklog Not tainted 6.18.20-rc1+ #61 PREEMPT(lazy) [30208.196749] Hardware name: HP HP Laptop 15s-ey1xxx/8706, BIOS F.61 03/25/2022 [30208.196752] RIP: 0010:vsnprintf+0x355/0x5e0 [30208.196761] Code: 0a 0f b6 03 49 89 df eb 27 4c 89 e0 48 8b 7c 24 30 48 8b 4c 24 20 4c 29 f8 4c 63 c0 48 39 cf 0f 82 cb 00 00 00 4c 01 44 24 30 <0f> b6 03 49 89 df 84 c0 0f 85 29 fd ff ff e9 09 fe ff ff 48 8b 44 [30208.196765] RSP: 0018:ffffd0d14ce93878 EFLAGS: 00000286 [30208.196771] RAX: ffffd0d14ce93992 RBX: ffffffff9358ebf2 RCX: 000000000000003e [30208.196775] RDX: 0000000000000000 RSI: ffffffff9358ebf1 RDI: ffffd0d14ce93992 [30208.196779] RBP: ffffd0d14ce938e8 R08: 0000000000000001 R09: 00000000ffffffff [30208.196782] R10: 0000000000000000 R11: 0000000000000004 R12: ffffffff9358ebf2 [30208.196785] R13: 0000000000000400 R14: 0000000000000400 R15: ffffffff9358ebf1 [30208.196789] FS: 00007f2e35a6b6c0(0000) GS:ffff8f48445af000(0000) knlGS:0000000000000000 [30208.196794] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30208.196798] CR2: 000055b82387b970 CR3: 000000010a0e0000 CR4: 00000000003506f0 [30208.196802] Call Trace: [30208.196805] <TASK> [30208.196818] sprintf+0x5f/0x80 [30208.196828] ? srso_return_thunk+0x5/0x5f [30208.196837] info_print_prefix+0x57/0xe0 [30208.196846] record_print_text+0x7a/0x170 [30208.196860] syslog_print+0x274/0x360 [30208.196869] ? __pfx_autoremove_wake_function+0x10/0x10 [30208.196886] do_syslog+0xd5/0x400 [30208.196900] ? srso_return_thunk+0x5/0x5f [30208.196910] kmsg_read+0x57/0x70 [30208.196918] ? srso_return_thunk+0x5/0x5f [30208.196924] ? __pfx_kmsg_read+0x10/0x10 [30208.196930] vfs_read+0xbf/0x390 [30208.196937] ? __seccomp_filter+0x42/0x590 [30208.196953] ksys_read+0x73/0xf0 [30208.196963] do_syscall_64+0x81/0x7f0 [30208.196972] ? srso_return_thunk+0x5/0x5f [30208.196978] ? srso_return_thunk+0x5/0x5f [30208.196984] ? wakeup_preempt+0x89/0xa0 [30208.196992] ? srso_return_thunk+0x5/0x5f [30208.196998] ? ttwu_do_activate+0x86/0x250 [30208.197005] ? srso_return_thunk+0x5/0x5f [30208.197010] ? update_rq_clock+0x81/0x180 [30208.197019] ? srso_return_thunk+0x5/0x5f [30208.197024] ? try_to_wake_up+0x371/0x8b0 [30208.197032] ? srso_return_thunk+0x5/0x5f [30208.197037] ? plist_del+0x5f/0xd0 [30208.197046] ? srso_return_thunk+0x5/0x5f [30208.197052] ? wake_up_q+0x37/0x90 [30208.197059] ? srso_return_thunk+0x5/0x5f [30208.197064] ? futex_wake+0xac/0x1c0 [30208.197077] ? srso_return_thunk+0x5/0x5f [30208.197082] ? srso_return_thunk+0x5/0x5f [30208.197088] ? __x64_sys_futex+0x12d/0x210 [30208.197101] ? srso_return_thunk+0x5/0x5f [30208.197106] ? do_syscall_64+0xb9/0x7f0 [30208.197114] ? srso_return_thunk+0x5/0x5f [30208.197120] ? switch_fpu_return+0x5b/0xe0 [30208.197128] ? srso_return_thunk+0x5/0x5f [30208.197134] ? do_syscall_64+0x240/0x7f0 [30208.197146] ? srso_return_thunk+0x5/0x5f [30208.197151] ? do_syscall_64+0xb9/0x7f0 [30208.197158] ? __irq_exit_rcu+0x3d/0xe0 [30208.197164] ? srso_return_thunk+0x5/0x5f [30208.197172] entry_SYSCALL_64_after_hwframe+0x76/0x7e [30208.197177] RIP: 0033:0x7f2e35fd3ffe [30208.197195] Code: 08 0f 85 15 47 ff ff 49 89 fb 48 89 f0 48 89 d7 48 89 ce 4c 89 c2 4d 89 ca 4c 8b 44 24 08 4c 8b 4c 24 10 4c 89 5c 24 08 0f 05 90 48 83 ec 08 bf 01 00 00 00 e8 e2 eb 05 00 bf 01 00 00 00 89 [30208.197199] RSP: 002b:00007f2e35a4a3b8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 [30208.197205] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f2e35fd3ffe [30208.197208] RDX: 0000000000001fa0 RSI: 00007f2e35a4ac30 RDI: 0000000000000005 [30208.197211] RBP: 00007f2e35a4b9c3 R08: 0000000000000000 R09: 0000000000000000 [30208.197214] R10: 0000000000000000 R11: 0000000000000246 R12: 000056345780d7a0 [30208.197217] R13: 00007f2e35a4ac30 R14: 0000000000001fa0 R15: 00007f2e35a4ac30 [30208.197232] </TASK>

SHELL CLASS ROOM

$pidof gimp 75203 $$sudo cat /proc/75203/stack cat: /proc/75203/stack: Permission denied $sudo cat /proc/75203/stack [sudo] password for jeffrin: [<0>] poll_schedule_timeout.constprop.0+0x37/0x90 [<0>] do_sys_poll+0x3fc/0x590 [<0>] __x64_sys_ppoll+0xdf/0x180 [<0>] do_syscall_64+0x81/0x7f0 [<0>] entry_SYSCALL_64_after_hwframe+0x76/0x7e $kill -STOP 75203 $sudo cat /proc/75203/stack [<0>] do_signal_stop+0x16f/0x200 [<0>] get_signal+0x230/0x920 [<0>] arch_do_signal_or_restart+0x41/0x260 [<0>] exit_to_user_mode_loop+0x8a/0x130 [<0>] do_syscall_64+0x206/0x7f0 [<0>] entry_SYSCALL_64_after_hwframe+0x76/0x7e $kill -CONT 75203 $sudo cat /proc/75203/stack [<0>] poll_schedule_timeout.constprop.0+0x37/0x90 [<0>] do_sys_poll+0x3fc/0x590 [<0>] __x64_sys_ppoll+0xdf/0x180 [<0>] do_syscall_64+0x81/0x7f0 [<0>] entry_SYSCALL_64_after_hwframe+0x76/0x7e $

What is do_signal_stop?
do_signal_stop is an internal function in the Linux kernel (specifically in kernel/signal.c) 
that handles group stops. This occurs when a process receives a signal like SIGSTOP, SIGTSTP, SIGTTIN, or 
SIGTTOU, which tells the process to pause execution. 
Its primary responsibilities include:

Transitioning State: It sets the process state to TASK_STOPPED.
Managing Group Stops: If one thread in a thread group initiates a stop, this function ensures
the entire group participates in the stop. Interaction with Debuggers: It handles special cases if a
process is being traced by a debugger (like ptrace). 

Why is the address all zeros?
When you see 0000000000000000, it usually indicates that kernel pointer restriction is enabled on your system. 

Security Feature: To prevent attackers from knowing where critical functions are located in memory
  (which could be used for exploits), the kernel hides the real addresses.
kptr_restrict: This is controlled by the file /proc/sys/kernel/kptr_restrict. If the value
 is set to 1 or 2, non-privileged users will only see zeros for kernel symbols. 

Technical Context
The "t" in the output stands for a local symbol in the text (code) section of the kernel 
binary. If it were a capital "T", it would be a global symbol. Are you currently 
troubleshooting a kernel crash or a frozen process where this function appeared in a stack trace?




SOURCE: GOOGLE AI MODE

RELATED SOURCE CODE EXPOSURE

#ifdef CONFIG_SMP { .procname = "hung_task_all_cpu_backtrace", .data = &sysctl_hung_task_all_cpu_backtrace, .maxlen = sizeof(int), .mode = 0644, .proc_handler = proc_dointvec_minmax, .extra1 = SYSCTL_ZERO, .extra2 = SYSCTL_ONE, }, #endif /* CONFIG_SMP */

RELATED FROM GOOGLE AI MODE


The hung_task_all_cpu_backtrace feature is not derived from a traditional academic research paper, but 
rather from a Linux kernel development initiative designed to improve debugging for blocked tasks (D-
state).  This mechanism was introduced to decouple panic events from CPU backtrace dumps, allowing 
developers to see what all CPUs are doing when a hung task is detected.
 Key Details of hung_task_all_cpu_backtrace


Purpose: To aid in debugging complex hang scenarios (e.g., deadlocks) by providing a snapshot 
of the entire system state, not just the stuck task.

Mechanism: It introduces a new sysctl (kernel.hung_task_all_cpu_backtrace) analogous to the
 approaches used for soft/hard lockups.

Implementation: When hung_task_warnings are still active, this feature triggers a backtrace of 
all CPUs upon detecting a task that has been in an uninterruptible sleep for longer than 
hung_task_timeout_secs.
    
Context: It is part of the kernel/hung_task.c framework in the Linux kernel. 

Related Research and Concepts

Kernel Tracing: Techniques like Ftrace and other tracers are heavily used to debug and reverse-
engineer complex systems, often used to determine the execution frequency of kernel paths.
    
Deadlock Detection: Other studies focus on optimizing the performance of tracing to analyze 
system bottlenecks.

GOSPEL Matthew 13:47-53

Jesus said to the disciples:
“The Kingdom of heaven is like a net thrown into the sea,
which collects fish of every kind.
When it is full they haul it ashore
and sit down to put what is good into buckets.
What is bad they throw away.
Thus it will be at the end of the age.
The angels will go out and separate the wicked from the righteous
and throw them into the fiery furnace,
where there will be wailing and grinding of teeth.”

“Do you understand all these things?”
They answered, “Yes.”
And he replied,
“Then every scribe who has been instructed in the Kingdom of heaven
is like the head of a household who brings from his storeroom
both the new and the old.”
When Jesus finished these parables, he went away from there.

GOSPEL John 11:19-27

Many of the Jews had come to Martha and Mary
to comfort them about their brother [Lazarus, who had died].
When Martha heard that Jesus was coming,
she went to meet him;
but Mary sat at home.
Martha said to Jesus,
“Lord, if you had been here,
my brother would not have died.
But even now I know that whatever you ask of God,
God will give you.”
Jesus said to her,
“Your brother will rise.”
Martha said to him,
“I know he will rise,
in the resurrection on the last day.”
Jesus told her,
“I am the resurrection and the life;
whoever believes in me, even if he dies, will live,
and anyone who lives and believes in me will never die.
Do you believe this?”
She said to him, “Yes, Lord.
I have come to believe that you are the Christ, the Son of God,
the one who is coming into the world.”

GOSPEL Matthew 13:44-52

Jesus said to his disciples:
"The kingdom of heaven is like a treasure buried in a field,
which a person finds and hides again,
and out of joy goes and sells all that he has and buys that field. 
Again, the kingdom of heaven is like a merchant
searching for fine pearls. 
When he finds a pearl of great price,
he goes and sells all that he has and buys it. 
Again, the kingdom of heaven is like a net thrown into the sea,
which collects fish of every kind. 
When it is full they haul it ashore
and sit down to put what is good into buckets. 
What is bad they throw away. 
Thus it will be at the end of the age. 
The angels will go out and separate the wicked from the righteous
and throw them into the fiery furnace,
where there will be wailing and grinding of teeth.

"Do you understand all these things?" 
They answered, "Yes." 
And he replied,
"Then every scribe who has been instructed in the kingdom of heaven
is like the head of a household
who brings from his storeroom both the new and the old."

GOSPEL Matthew 20:20-28

The mother of the sons of Zebedee approached Jesus with her sons
and did him homage, wishing to ask him for something.
He said to her,
"What do you wish?"
She answered him,
"Command that these two sons of mine sit,
one at your right and the other at your left, in your Kingdom."
Jesus said in reply,
"You do not know what you are asking.
Can you drink the chalice that I am going to drink?"
They said to him, "We can."
He replied,
"My chalice you will indeed drink,
but to sit at my right and at my left, this is not mine to give
but is for those for whom it has been prepared by my Father."
When the ten heard this,
they became indignant at the two brothers.
But Jesus summoned them and said,
"You know that the rulers of the Gentiles lord it over them,
and the great ones make their authority over them felt.
But it shall not be so among you.
Rather, whoever wishes to be great among you shall be your servant;
whoever wishes to be first among you shall be your slave.
Just so, the Son of Man did not come to be served
but to serve and to give his life as a ransom for many."

GOSPEL Matthew 13:18-23

Jesus said to his disciples:
“Hear the parable of the sower.
The seed sown on the path is the one who hears the word of the Kingdom
without understanding it,
and the Evil One comes and steals away
what was sown in his heart.
The seed sown on rocky ground
is the one who hears the word and receives it at once with joy.
But he has no root and lasts only for a time.
When some tribulation or persecution comes because of the word,
he immediately falls away.
The seed sown among thorns is the one who hears the word,
but then worldly anxiety and the lure of riches choke the word
and it bears no fruit.
But the seed sown on rich soil
is the one who hears the word and understands it,
who indeed bears fruit and yields a hundred or sixty or thirtyfold.”

GOSPEL Matthew 13:10-17

The disciples approached Jesus and said,
“Why do you speak to the crowd in parables?”
He said to them in reply,
“Because knowledge of the mysteries of the Kingdom of heaven
has been granted to you, but to them it has not been granted.
To anyone who has, more will be given and he will grow rich;
from anyone who has not, even what he has will be taken away.
This is why I speak to them in parables, because
they look but do not see and hear but do not listen or understand.
Isaiah’s prophecy is fulfilled in them, which says:

You shall indeed hear but not understand,
you shall indeed look but never see.
Gross is the heart of this people,
they will hardly hear with their ears,
they have closed their eyes,
lest they see with their eyes
and hear with their ears
and understand with their hearts and be converted
and I heal them.

“But blessed are your eyes, because they see,
and your ears, because they hear.
Amen, I say to you, many prophets and righteous people
longed to see what you see but did not see it,
and to hear what you hear but did not hear it.”

GOSPEL Matthew 12:46-50

While Jesus was speaking to the crowds,
his mother and his brothers appeared outside,
wishing to speak with him.
Someone told him, “Your mother and your brothers are standing outside,
asking to speak with you.”
But he said in reply to the one who told him,
“Who is my mother? Who are my brothers?”
And stretching out his hand toward his disciples, he said,
“Here are my mother and my brothers.
For whoever does the will of my heavenly Father
is my brother, and sister, and mother.”

GOSPEL Matthew 12:14-21

The Pharisees went out and took counsel against Jesus
to put him to death.

When Jesus realized this, he withdrew from that place.
Many people followed him, and he cured them all,
but he warned them not to make him known.
This was to fulfill what had been spoken through Isaiah the prophet:

Behold, my servant whom I have chosen,
my beloved in whom I delight;
I shall place my Spirit upon him,
and he will proclaim justice to the Gentiles.
He will not contend or cry out,
nor will anyone hear his voice in the streets.
A bruised reed he will not break,
a smoldering wick he will not quench,
until he brings justice to victory.
And in his name the Gentiles will hope.

GOSPEL Matthew 11:25-27

At that time Jesus exclaimed: 
“I give praise to you, Father, Lord of heaven and earth,
for although you have hidden these things
from the wise and the learned
you have revealed them to the childlike.
Yes, Father, such has been your gracious will.
All things have been handed over to me by my Father.
No one knows the Son except the Father,
and no one knows the Father except the Son
and anyone to whom the Son wishes to reveal him.”

GOSPEL Matthew 11:20-24

Jesus began to reproach the towns
where most of his mighty deeds had been done,
since they had not repented.
“Woe to you, Chorazin! Woe to you, Bethsaida!
For if the mighty deeds done in your midst
had been done in Tyre and Sidon,
they would long ago have repented in sackcloth and ashes.
But I tell you, it will be more tolerable
for Tyre and Sidon on the day of judgment than for you.
And as for you, Capernaum:

Will you be exalted to heaven?
You will go down to the nether world.

For if the mighty deeds done in your midst had been done in Sodom,
it would have remained until this day.
But I tell you, it will be more tolerable
for the land of Sodom on the day of judgment than for you.”