Blog – TRUEANGLE

Featured

hung_task_all_cpu_backtrace( topic related to CPUs dumping their backtraces )

ABOUT

hung_task_all_cpu_backtrace

If this option is set, the kernel will send an NMI to all CPUs to dump their backtraces when a hung task is detected. This file shows up if CONFIG_DETECT_HUNG_TASK and CONFIG_SMP are enabled. 0: Won’t show all CPUs backtraces when a hung task is detected. This is the default behavior 1: Will non-maskably interrupt all CPUs and dump their backtraces when a hung task is detected.

LINUX KERNEL RELATED PARAMETER

$cat /proc/sys/kernel/hung_task_all_cpu_backtrace 0 $echo 1 | sudo tee –append /proc/sys/kernel/hung_task_all_cpu_backtrace [sudo] password for jeffrin: 1 $cat /proc/sys/kernel/hung_task_all_cpu_backtrace 1 $

PARAMETER RELATED

USERSPACE PROCESS BACKTRACE

(gdb) bt #0 __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56 #1 0x00007ffff64197a4 in __internal_syscall_cancel (a1=, a2=, a3=, a4=, a5=a5@entry=8, a6=a6@entry=0, nr=271) at ./nptl/cancellation.c:49 #2 0x00007ffff64197ed in __syscall_cancel (a1=, a2=, a3=, a4=, a5=a5@entry=8, a6=a6@entry=0, nr=271) at ./nptl/cancellation.c:75 #3 0x00007ffff648f29e in __GI_ppoll (fds=, nfds=, timeout=, sigmask=) at ../sysdeps/unix/sysv/linux/ppoll.c:42 #4 0x00007ffff7951b34 in ??? () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0 #5 0x00007ffff7952210 in g_main_context_iteration () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0 #6 0x00007ffff76fe13d in g_application_run () at /usr/lib/x86_64-linux-gnu/libgio-2.0.so.0 #7 0x000055555580dfad in app_run () #8 0x0000555555641f3b in main () (gdb)

BACKTRACE FROM LINUX KERNEL SPACE

echo 1 | sudo tee /proc/sys/kernel/sysrq


echo l | sudo tee /proc/sysrq-trigger.
dmesg

[30208.196481] sysrq: Show backtrace of all active CPUs [30208.196491] NMI backtrace for cpu 0 [30208.196494] CPU: 0 UID: 0 PID: 45306 Comm: tee Not tainted 6.18.20-rc1+ #61 PREEMPT(lazy) [30208.196498] Hardware name: HP HP Laptop 15s-ey1xxx/8706, BIOS F.61 03/25/2022 [30208.196500] Call Trace: [30208.196503] <TASK>code [30208.196507] dump_stack_lvl+0x5d/0x80 [30208.196514] nmi_cpu_backtrace.cold+0x18/0x65 [30208.196518] ? __pfx_nmi_raise_cpu_backtrace+0x10/0x10 [30208.196524] nmi_trigger_cpumask_backtrace+0xf4/0x110 [30208.196529] __handle_sysrq.cold+0x8a/0x160 [30208.196534] write_sysrq_trigger+0x59/0xa0 [30208.196540] proc_reg_write+0x5a/0xb0 [30208.196544] vfs_write+0xce/0x480 [30208.196552] ksys_write+0x73/0xf0 [30208.196556] do_syscall_64+0x81/0x7f0 [30208.196562] ? srso_return_thunk+0x5/0x5f [30208.196565] ? __x64_sys_openat+0x61/0xa0 [30208.196569] ? srso_return_thunk+0x5/0x5f [30208.196571] ? do_syscall_64+0xb9/0x7f0 [30208.196576] ? srso_return_thunk+0x5/0x5f [30208.196579] ? ksys_fadvise64_64+0x6c/0xa0 [30208.196584] ? srso_return_thunk+0x5/0x5f [30208.196587] ? srso_return_thunk+0x5/0x5f [30208.196589] ? do_syscall_64+0xb9/0x7f0 [30208.196593] ? srso_return_thunk+0x5/0x5f [30208.196597] ? do_user_addr_fault+0x21a/0x690 [30208.196602] ? srso_return_thunk+0x5/0x5f [30208.196605] ? srso_return_thunk+0x5/0x5f [30208.196609] entry_SYSCALL_64_after_hwframe+0x76/0x7e [30208.196613] RIP: 0033:0x7f8875a3a7d2 [30208.196638] Code: 18 41 8b 93 08 03 00 00 59 5e 48 83 f8 fc 75 1a 83 e2 39 83 fa 08 75 12 e8 2b ff ff ff 0f 1f 00 49 89 ca 48 8b 44 24 20 0f 05 <48> 83 c4 18 c3 66 0f 1f 84 00 00 00 00 00 48 83 ec 10 ff 74 24 18 [30208.196640] RSP: 002b:00007fffda92a5b0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001 [30208.196644] RAX: ffffffffffffffda RBX: 000055b4a7c9c310 RCX: 00007f8875a3a7d2 [30208.196645] RDX: 0000000000000002 RSI: 00007fffda92a750 RDI: 0000000000000003 [30208.196647] RBP: 0000000000000002 R08: 0000000000000000 R09: 0000000000000000 [30208.196649] R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000002 [30208.196650] R13: 00007fffda92a750 R14: 00007fffda92a750 R15: 0000000000000002 [30208.196659] </TASK> [30208.196688] Sending NMI from CPU 0 to CPUs 1-3: [30208.196697] NMI backtrace for cpu 1 [30208.196699] CPU: 1 UID: 0 PID: 345 Comm: systemd-journal Not tainted 6.18.20-rc1+ #61 PREEMPT(lazy) [30208.196702] Hardware name: HP HP Laptop 15s-ey1xxx/8706, BIOS F.61 03/25/2022 [30208.196703] RIP: 0033:0x7f51c5e71490 [30208.196710] Code: bd fd ff ff b8 f4 ff ff ff e9 40 fd ff ff 31 d2 e9 4c ff ff ff 89 da e9 45 ff ff ff e8 f9 8c e4 ff e8 f4 fb ff ff 0f 1f 40 00 0f 1e fa 41 54 49 89 d4 55 48 89 cd 53 48 89 fb e8 1a 91 ff ff [30208.196712] RSP: 002b:00007ffc7ed4ca08 EFLAGS: 00000246 [30208.196714] RAX: 0000000000000000 RBX: 000056303827baf0 RCX: 0000000000000000 [30208.196716] RDX: 00007ffc7ed4ca18 RSI: 00007ffc7ed4ca20 RDI: 0000000000000000 [30208.196717] RBP: 00007ffc7ed4ca20 R08: 00007f51c62b3070 R09: 0000000000000000 [30208.196718] R10: 00007f51c62b3000 R11: 0000000000c1eb8a R12: 000056303827baf0 [30208.196720] R13: 0000000000000000 R14: ffffffffffffffff R15: 000056302ba21658 [30208.196721] FS: 00007f51c6172840 GS: 0000000000000000 [30208.196723] NMI backtrace for cpu 2 skipped: idling at io_idle+0x3/0x30 [30208.196736] NMI backtrace for cpu 3 [30208.196741] CPU: 3 UID: 0 PID: 806 Comm: in:imklog Not tainted 6.18.20-rc1+ #61 PREEMPT(lazy) [30208.196749] Hardware name: HP HP Laptop 15s-ey1xxx/8706, BIOS F.61 03/25/2022 [30208.196752] RIP: 0010:vsnprintf+0x355/0x5e0 [30208.196761] Code: 0a 0f b6 03 49 89 df eb 27 4c 89 e0 48 8b 7c 24 30 48 8b 4c 24 20 4c 29 f8 4c 63 c0 48 39 cf 0f 82 cb 00 00 00 4c 01 44 24 30 <0f> b6 03 49 89 df 84 c0 0f 85 29 fd ff ff e9 09 fe ff ff 48 8b 44 [30208.196765] RSP: 0018:ffffd0d14ce93878 EFLAGS: 00000286 [30208.196771] RAX: ffffd0d14ce93992 RBX: ffffffff9358ebf2 RCX: 000000000000003e [30208.196775] RDX: 0000000000000000 RSI: ffffffff9358ebf1 RDI: ffffd0d14ce93992 [30208.196779] RBP: ffffd0d14ce938e8 R08: 0000000000000001 R09: 00000000ffffffff [30208.196782] R10: 0000000000000000 R11: 0000000000000004 R12: ffffffff9358ebf2 [30208.196785] R13: 0000000000000400 R14: 0000000000000400 R15: ffffffff9358ebf1 [30208.196789] FS: 00007f2e35a6b6c0(0000) GS:ffff8f48445af000(0000) knlGS:0000000000000000 [30208.196794] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [30208.196798] CR2: 000055b82387b970 CR3: 000000010a0e0000 CR4: 00000000003506f0 [30208.196802] Call Trace: [30208.196805] <TASK> [30208.196818] sprintf+0x5f/0x80 [30208.196828] ? srso_return_thunk+0x5/0x5f [30208.196837] info_print_prefix+0x57/0xe0 [30208.196846] record_print_text+0x7a/0x170 [30208.196860] syslog_print+0x274/0x360 [30208.196869] ? __pfx_autoremove_wake_function+0x10/0x10 [30208.196886] do_syslog+0xd5/0x400 [30208.196900] ? srso_return_thunk+0x5/0x5f [30208.196910] kmsg_read+0x57/0x70 [30208.196918] ? srso_return_thunk+0x5/0x5f [30208.196924] ? __pfx_kmsg_read+0x10/0x10 [30208.196930] vfs_read+0xbf/0x390 [30208.196937] ? __seccomp_filter+0x42/0x590 [30208.196953] ksys_read+0x73/0xf0 [30208.196963] do_syscall_64+0x81/0x7f0 [30208.196972] ? srso_return_thunk+0x5/0x5f [30208.196978] ? srso_return_thunk+0x5/0x5f [30208.196984] ? wakeup_preempt+0x89/0xa0 [30208.196992] ? srso_return_thunk+0x5/0x5f [30208.196998] ? ttwu_do_activate+0x86/0x250 [30208.197005] ? srso_return_thunk+0x5/0x5f [30208.197010] ? update_rq_clock+0x81/0x180 [30208.197019] ? srso_return_thunk+0x5/0x5f [30208.197024] ? try_to_wake_up+0x371/0x8b0 [30208.197032] ? srso_return_thunk+0x5/0x5f [30208.197037] ? plist_del+0x5f/0xd0 [30208.197046] ? srso_return_thunk+0x5/0x5f [30208.197052] ? wake_up_q+0x37/0x90 [30208.197059] ? srso_return_thunk+0x5/0x5f [30208.197064] ? futex_wake+0xac/0x1c0 [30208.197077] ? srso_return_thunk+0x5/0x5f [30208.197082] ? srso_return_thunk+0x5/0x5f [30208.197088] ? __x64_sys_futex+0x12d/0x210 [30208.197101] ? srso_return_thunk+0x5/0x5f [30208.197106] ? do_syscall_64+0xb9/0x7f0 [30208.197114] ? srso_return_thunk+0x5/0x5f [30208.197120] ? switch_fpu_return+0x5b/0xe0 [30208.197128] ? srso_return_thunk+0x5/0x5f [30208.197134] ? do_syscall_64+0x240/0x7f0 [30208.197146] ? srso_return_thunk+0x5/0x5f [30208.197151] ? do_syscall_64+0xb9/0x7f0 [30208.197158] ? __irq_exit_rcu+0x3d/0xe0 [30208.197164] ? srso_return_thunk+0x5/0x5f [30208.197172] entry_SYSCALL_64_after_hwframe+0x76/0x7e [30208.197177] RIP: 0033:0x7f2e35fd3ffe [30208.197195] Code: 08 0f 85 15 47 ff ff 49 89 fb 48 89 f0 48 89 d7 48 89 ce 4c 89 c2 4d 89 ca 4c 8b 44 24 08 4c 8b 4c 24 10 4c 89 5c 24 08 0f 05 90 48 83 ec 08 bf 01 00 00 00 e8 e2 eb 05 00 bf 01 00 00 00 89 [30208.197199] RSP: 002b:00007f2e35a4a3b8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 [30208.197205] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f2e35fd3ffe [30208.197208] RDX: 0000000000001fa0 RSI: 00007f2e35a4ac30 RDI: 0000000000000005 [30208.197211] RBP: 00007f2e35a4b9c3 R08: 0000000000000000 R09: 0000000000000000 [30208.197214] R10: 0000000000000000 R11: 0000000000000246 R12: 000056345780d7a0 [30208.197217] R13: 00007f2e35a4ac30 R14: 0000000000001fa0 R15: 00007f2e35a4ac30 [30208.197232] </TASK>

SHELL CLASS ROOM

$pidof gimp 75203 $$sudo cat /proc/75203/stack cat: /proc/75203/stack: Permission denied $sudo cat /proc/75203/stack [sudo] password for jeffrin: [<0>] poll_schedule_timeout.constprop.0+0x37/0x90 [<0>] do_sys_poll+0x3fc/0x590 [<0>] __x64_sys_ppoll+0xdf/0x180 [<0>] do_syscall_64+0x81/0x7f0 [<0>] entry_SYSCALL_64_after_hwframe+0x76/0x7e $kill -STOP 75203 $sudo cat /proc/75203/stack [<0>] do_signal_stop+0x16f/0x200 [<0>] get_signal+0x230/0x920 [<0>] arch_do_signal_or_restart+0x41/0x260 [<0>] exit_to_user_mode_loop+0x8a/0x130 [<0>] do_syscall_64+0x206/0x7f0 [<0>] entry_SYSCALL_64_after_hwframe+0x76/0x7e $kill -CONT 75203 $sudo cat /proc/75203/stack [<0>] poll_schedule_timeout.constprop.0+0x37/0x90 [<0>] do_sys_poll+0x3fc/0x590 [<0>] __x64_sys_ppoll+0xdf/0x180 [<0>] do_syscall_64+0x81/0x7f0 [<0>] entry_SYSCALL_64_after_hwframe+0x76/0x7e $

What is do_signal_stop?
do_signal_stop is an internal function in the Linux kernel (specifically in kernel/signal.c) 
that handles group stops. This occurs when a process receives a signal like SIGSTOP, SIGTSTP, SIGTTIN, or 
SIGTTOU, which tells the process to pause execution. 
Its primary responsibilities include:

Transitioning State: It sets the process state to TASK_STOPPED.
Managing Group Stops: If one thread in a thread group initiates a stop, this function ensures
the entire group participates in the stop. Interaction with Debuggers: It handles special cases if a
process is being traced by a debugger (like ptrace). 

Why is the address all zeros?
When you see 0000000000000000, it usually indicates that kernel pointer restriction is enabled on your system. 

Security Feature: To prevent attackers from knowing where critical functions are located in memory
  (which could be used for exploits), the kernel hides the real addresses.
kptr_restrict: This is controlled by the file /proc/sys/kernel/kptr_restrict. If the value
 is set to 1 or 2, non-privileged users will only see zeros for kernel symbols. 

Technical Context
The "t" in the output stands for a local symbol in the text (code) section of the kernel 
binary. If it were a capital "T", it would be a global symbol. Are you currently 
troubleshooting a kernel crash or a frozen process where this function appeared in a stack trace?




SOURCE: GOOGLE AI MODE

RELATED SOURCE CODE EXPOSURE

#ifdef CONFIG_SMP { .procname = "hung_task_all_cpu_backtrace", .data = &sysctl_hung_task_all_cpu_backtrace, .maxlen = sizeof(int), .mode = 0644, .proc_handler = proc_dointvec_minmax, .extra1 = SYSCTL_ZERO, .extra2 = SYSCTL_ONE, }, #endif /* CONFIG_SMP */

RELATED FROM GOOGLE AI MODE


The hung_task_all_cpu_backtrace feature is not derived from a traditional academic research paper, but 
rather from a Linux kernel development initiative designed to improve debugging for blocked tasks (D-
state).  This mechanism was introduced to decouple panic events from CPU backtrace dumps, allowing 
developers to see what all CPUs are doing when a hung task is detected.
 Key Details of hung_task_all_cpu_backtrace


Purpose: To aid in debugging complex hang scenarios (e.g., deadlocks) by providing a snapshot 
of the entire system state, not just the stuck task.

Mechanism: It introduces a new sysctl (kernel.hung_task_all_cpu_backtrace) analogous to the
 approaches used for soft/hard lockups.

Implementation: When hung_task_warnings are still active, this feature triggers a backtrace of 
all CPUs upon detecting a task that has been in an uninterruptible sleep for longer than 
hung_task_timeout_secs.
    
Context: It is part of the kernel/hung_task.c framework in the Linux kernel. 

Related Research and Concepts

Kernel Tracing: Techniques like Ftrace and other tracers are heavily used to debug and reverse-
engineer complex systems, often used to determine the execution frequency of kernel paths.
    
Deadlock Detection: Other studies focus on optimizing the performance of tracing to analyze 
system bottlenecks.

GOSPEL Matthew 5:38-42

Jesus said to his disciples:
"You have heard that it was said,
An eye for an eye and a tooth for a tooth.
But I say to you, offer no resistance to one who is evil.
When someone strikes you on your right cheek,
turn the other one to him as well.
If anyone wants to go to law with you over your tunic,
hand him your cloak as well.
Should anyone press you into service for one mile,
go with him for two miles.
Give to the one who asks of you,
and do not turn your back on one who wants to borrow."

GOSPEL Matthew 9:36—10:8

At the sight of the crowds, Jesus’ heart was moved with pity for them 
because they were troubled and abandoned,
like sheep without a shepherd.
Then he said to his disciples,
“The harvest is abundant but the laborers are few;
so ask the master of the harvest
to send out laborers for his harvest.”

Then he summoned his twelve disciples
and gave them authority over unclean spirits
to drive them out and to cure every disease and every illness.
The names of the twelve apostles are these:
first, Simon called Peter, and his brother Andrew;
James, the son of Zebedee, and his brother John;
Philip and Bartholomew, Thomas and Matthew the tax collector;
James, the son of Alphaeus, and Thaddeus;
Simon from Cana, and Judas Iscariot who betrayed him.

Jesus sent out these twelve after instructing them thus,
“Do not go into pagan territory or enter a Samaritan town.
Go rather to the lost sheep of the house of Israel.
As you go, make this proclamation: ‘The kingdom of heaven is at hand.’
Cure the sick, raise the dead, cleanse lepers, drive out demons.
Without cost you have received; without cost you are to give.”

GOSPEL Matthew 5:33-37

Jesus said to his disciples:
"You have heard that it was said to your ancestors,
Do not take a false oath,
but make good to the Lord all that you vow.
But I say to you, do not swear at all;
not by heaven, for it is God's throne;
nor by the earth, for it is his footstool;
nor by Jerusalem, for it is the city of the great King.
Do not swear by your head,
for you cannot make a single hair white or black.
Let your 'Yes' mean 'Yes,' and your 'No' mean 'No.'
Anything more is from the Evil One."

GOSPEL Matthew 11:25-30

At that time Jesus exclaimed:
"I give praise to you, Father, Lord of heaven and earth,
for although you have hidden these things
from the wise and the learned
you have revealed them to little ones.
Yes, Father, such has been your gracious will.
All things have been handed over to me by my Father. 
No one knows the Son except the Father,
and no one knows the Father except the Son
and anyone to whom the Son wishes to reveal him.

"Come to me, all you who labor and are burdened,
and I will give you rest.
Take my yoke upon you and learn from me,
for I am meek and humble of heart;
and you will find rest for yourselves. 
For my yoke is easy, and my burden light."

GOSPEL Matthew 5:20-26

Jesus said to his disciples: 
"I tell you, unless your righteousness surpasses that
of the scribes and Pharisees,
you will not enter into the Kingdom of heaven.

"You have heard that it was said to your ancestors,
You shall not kill; and whoever kills will be liable to judgment.
But I say to you, whoever is angry with his brother
will be liable to judgment,
and whoever says to his brother, Raqa,
will be answerable to the Sanhedrin,
and whoever says, 'You fool,' will be liable to fiery Gehenna.
Therefore, if you bring your gift to the altar,
and there recall that your brother
has anything against you,
leave your gift there at the altar,
go first and be reconciled with your brother,
and then come and offer your gift.
Settle with your opponent quickly while on the way to court with him.
Otherwise your opponent will hand you over to the judge,
and the judge will hand you over to the guard,
and you will be thrown into prison.
Amen, I say to you,
you will not be released until you have paid the last penny."

GOSPEL Matthew 5:17-19


Jesus said to his disciples:
"Do not think that I have come to abolish the law or the prophets.
I have come not to abolish but to fulfill.
Amen, I say to you, until heaven and earth pass away,
not the smallest letter or the smallest part of a letter
will pass from the law,
until all things have taken place.
Therefore, whoever breaks one of the least of these commandments
and teaches others to do so
will be called least in the Kingdom of heaven.
But whoever obeys and teaches these commandments
will be called greatest in the Kingdom of heaven."

GOSPEL Matthew 5:13-16

Jesus said to his disciples:
“You are the salt of the earth.
But if salt loses its taste, with what can it be seasoned?
It is no longer good for anything
but to be thrown out and trampled underfoot.
You are the light of the world.
A city set on a mountain cannot be hidden.
Nor do they light a lamp and then put it under a bushel basket;
it is set on a lampstand,
where it gives light to all in the house.
Just so, your light must shine before others,
that they may see your good deeds
and glorify your heavenly Father.”

GOSPEL Matthew 5:1-12

When Jesus saw the crowds, he went up the mountain,
and after he had sat down, his disciples came to him. 
He began to teach them, saying:

“Blessed are the poor in spirit,
for theirs is the Kingdom of heaven.
Blessed are they who mourn,
for they will be comforted.
Blessed are the meek,
for they will inherit the land.
Blessed are they who hunger and thirst for righteousness,
for they will be satisfied.
Blessed are the merciful,
for they will be shown mercy.
Blessed are the clean of heart,
for they will see God.
Blessed are the peacemakers,
for they will be called children of God.
Blessed are they who are persecuted for the sake of righteousness,
for theirs is the Kingdom of heaven.
Blessed are you when they insult you and persecute you
and utter every kind of evil against you falsely because of me.
Rejoice and be glad,
for your reward will be great in heaven.
Thus they persecuted the prophets who were before you.”

GOSPEL John 6:51-58

Jesus said to the Jewish crowds:
"I am the living bread that came down from heaven;
whoever eats this bread will live forever;
and the bread that I will give
is my flesh for the life of the world."

The Jews quarreled among themselves, saying,
"How can this man give us his flesh to eat?" 
Jesus said to them,
"Amen, amen, I say to you,
unless you eat the flesh of the Son of Man and drink his blood,
you do not have life within you. 
Whoever eats my flesh and drinks my blood
has eternal life,
and I will raise him on the last day. 
For my flesh is true food,
and my blood is true drink. 
Whoever eats my flesh and drinks my blood
remains in me and I in him. 
Just as the living Father sent me
and I have life because of the Father,
so also the one who feeds on me
will have life because of me. 
This is the bread that came down from heaven. 
Unlike your ancestors who ate and still died,
whoever eats this bread will live forever."

GOSPEL Mark 12:38-44

In the course of his teaching Jesus said,
“Beware of the scribes, who like to go around in long robes
and accept greetings in the marketplaces,
seats of honor in synagogues,
and places of honor at banquets.
They devour the houses of widows and, as a pretext,
recite lengthy prayers.
They will receive a very severe condemnation.”He sat down opposite the treasury
and observed how the crowd put money into the treasury.
Many rich people put in large sums.
A poor widow also came and put in two small coins worth a few cents. 
Calling his disciples to himself, he said to them,
“Amen, I say to you, this poor widow put in more
than all the other contributors to the treasury.
For they have all contributed from their surplus wealth,
but she, from her poverty, has contributed all she had,
her whole livelihood.”