Tools

1. GNU core utilities
http://savannah.gnu.org/projects/coreutils
2. Miscellaneous system utilities
http://userweb.kernel.org/~kzak/util-linux/
3. utilities for making and checking MS-DOS FAT filesystems
http://www.daniel-baumann.ch/software/dosfstools/
4. Undelete utility for the ext2 file system
http://e2undel.sourceforge.net
5. utilities for manipulating files in an ext2/ext3 filesystem
http://home.earthlink.net/~k_sheff/sw/e2tools/index.html
6. Determines file type using “magic” numbers
http://www.darwinsys.com/file/
7. filesystem activity monitoring tool
http://mytty.org/fspy
8. realtime filesystem monitoring program using inotify
http://iwatch.sourceforge.net/
9. Utilities for manipulating filesystem extended attributes
http://savannah.nongnu.org/projects/attr/
10. filename encoding conversion tool
http://www.j3e.de/linux/convmv/
11. Tool to help recover deleted files on ext3 filesystems
http://code.google.com/p/ext3grep/
12. The GNU Image Manipulation Program
http://www.gimp.org
13. A Scheme language interpreter
http://groups.csail.mit.edu/mac/ftpdir/scm/scm-5e7.zip
14. MPlayer’s Movie Encoder
http://www.mplayerhq.hu/
15. audio/video encoder, streaming server & audio/video file converter
http://ffmpeg.mplayerhq.hu/
16. Interactive Colorful IP LAN Monitor
http://iptraf.seul.org/
17. An utility for Directing compilation.
http://www.gnu.org/software/make/
18. GNU C compiler
http://gcc.gnu.org/
19. command-line network traffic analyzer
http://www.tcpdump.org/
20. interactive processes viewer
http://htop.sourceforge.net
21. FAT16/FAT32 filesystem resizer
http://sf.net/projects/fatresize
22. Guess PC disk partition table, find lost partitions
http://home.pages.de/~michab/gpart/
23. File Alteration Monitor
http://oss.sgi.com/projects/fam/
24. command-line utility that reports when files have been altered
http://fileschanged.sourceforge.net/

ls ( -L option )

UNIX Command

$ls -l vmlinuz
lrwxrwxrwx 1 root root 26 Dec  1 02:44 vmlinuz -> boot/vmlinuz-3.1.0-1-amd64
$ls -lL vmlinuz
-rw-r--r-- 1 root root 2725680 Nov 29 19:49 vmlinuz
$

 

UNIX Explanation

 -L, --dereference
when showing file information for a symbolic link, show information for the file the link references rather than for the link itself.

 

Theory Drop

In  computing, a symbolic  link (also  symlink or  soft link)  is a
special type of  file that contains a reference  to another file or
directory  in the form  of an  absolute or  relative path  and that
affects pathname resolution.[1] Symbolic links were already present
by  1978  in mini-computer  operating  systems  from  DEC and  Data
General's   RDOS.   Today  they   are   supported   by  the   POSIX
operating-system standard, most Unix-like operating systems such as
FreeBSD,  GNU/Linux,  and Mac  OS  X,  and  also Windows  operating
systems  such as Windows  Vista, Windows  7 and  to some  degree in
Windows 2000 and Windows  XP.  Symbolic links operate transparently
for most operations: programs which read or write to files named by
a symbolic link will behave  as if operating directly on the target
file.  However,  programs  that   need  to  handle  symbolic  links
specially (e.g., backup utilities) may identify and manipulate them
directly.

source : http://en.wikipedia.org/wiki/Symbolic_link

stackable filesystems . Wrapfs filesystem .

Classroom.

Wrapfs  is  a   full-fledged  stackable  null-layer  (or  loopback)
filesystem   that  simply   passes  all   operations   and  objects
(unmodified)  between  the VFS  and  the  lower filesystem.  Wrapfs
itself, however, is  not easy to write for one  main reason; it has
to  treat  the  lower filesystem  as  if  it  were the  VFS,  while
appearing to the  real Linux VFS as a  lower-level filesystem. This
dual  role requires  careful handling  of locks,  reference counts,
allocated  memory and  so on.  Luckily, someone  already  wrote and
maintains Wrapfs. Therefore, Wrapfs serves as an excellent template
for you to modify and add new functionality.

source : http://www.linuxjournal.com/article/6485

Get The Hang

int wrapfs_unlink(struct inode *dir,
                  struct dentry *dentry)
{
  int err = 0;
  struct inode *lower_dir;
  struct dentry *lower_dentry;
  lower_dir = get_lower_inode(dir);
  lower_dentry = get_lower_dentry(dentry);
  /* pre-call code can go here */
  err = lower_dir->i_op->unlink(lower_dir,
                                lower_dentry);
  /* post-call code can go here */
  return err;
}

source : http://www.linuxjournal.com/article/6485

What is a tracepoint ?

Topic

Tracepoint

Explanation

A tracepoint placed in code provides a hook to call a function (probe) that you can provide at runtime. A tracepoint can be "on" (a probe is connected to it) or "off" (no probe is attached). When a tracepoint is "off" it has no effect, except for adding a tiny time penalty (checking a condition for a branch) and space penalty (adding a few bytes for the function call at the end of the instrumented function and adds a data structure in a separate section). When a tracepoint is "on", the function you provide is called each time the tracepoint is executed, in the execution context of the caller. When the function provided ends its execution, it returns to the caller (continuing from the tracepoint site). source : linux-next/Documentation/trace/tracepoints.txt

Get The Hang

void bio_endio(struct bio *bio, int error) else if (!test_bit(BIO_UPTODATE, &bio->bi_flags)) error = -EIO; + if (test_bit(BIO_IN_FLIGHT, &bio->bi_flags)) { + struct request_queue *q = bdev_get_queue(bio->bi_bdev); + + trace_block_bio_complete(q, bio, error); + clear_bit(BIO_IN_FLIGHT, &bio->bi_flags); + } +

Endianness .. Little endian .. Big endian.

Topic

Endianness

Explanation

Endianess  is  the byte  order  of  the  number in  the  computer's
memory. The  number can have any  size of bits, bu  the most common
numbers used are 32 bits (4 bytes) and 16 bits (2 bytes).

source : http://www.rapidtables.com/prog/endianess.htm

Theory Drop

In computing, the term endian  or endianness refers to the ordering
of individually addressable sub-componentswithin the representation
of a larger data item  as stored in external memory (or, sometimes,
as  sent  on  a  serial  connection).  Each  sub-component  in  the
representation has a unique  degree of significance, like the place
value  of digits  in  a decimal  number.  These sub-components  are
typically  16-  or  32-bit   words,  8-bit  bytes,  or  even  bits.
Endianness is  a difference in data representation  at the hardware
level and may or may not be transparent at higher levels, depending
on factors such as the type of high level language used.


source : http://en.wikipedia.org/wiki/Endianness

Get The Hang

/* swap partition endianess hack... */
        if (swab32(swap_header->info.version) == 1) {
                swab32s(&swap_header->info.version);
                swab32s(&swap_header->info.last_page);
                swab32s(&swap_header->info.nr_badpages);
                for (i = 0; i info.nr_badpages; i++)
                        swab32s(&swap_header->info.badpages[i]);
        }

ls ( -I option ) do not list implied entries matching shell PATTERN

UNIX Command

$ls
bp  cfg.c  cfg.h  CVS  gtk  Makefile  sample.cfg  simpleproxy.c
$ls -I gtk
bp  cfg.c  cfg.h  CVS  Makefile  sample.cfg  simpleproxy.c
$ls -I gtk CVS
Entries  Repository  Root
$ls -I CVS gtk
main.c	Makefile  simpleproxy-gtk.c
$ls -I cfg.c
bp  cfg.h  CVS	gtk  Makefile  sample.cfg  simpleproxy.c
$ls -I bp
cfg.c  cfg.h  CVS  gtk	Makefile  sample.cfg  simpleproxy.c
$

UNIX Explanation

-I, --ignore=PATTERN
              do not list implied entries matching shell PATTERN

Theory Drop

In computer science,  pattern matching is the act  of checking some
sequence of  tokens for  the presence of  the constituents  of some
pattern. In contrast to  pattern recognition, the match usually has
to  be  exact. The  patterns  generally  have  the form  of  either
sequences  or tree  structures.  Uses of  pattern matching  include
outputting  the locations  (if any)  of  a pattern  within a  token
sequence, to output  some component of the matched  pattern, and to
substitute  the matching  pattern  with some  other token  sequence
(i.e., search and replace).

source : http://en.wikipedia.org/wiki/Pattern_matching

hung_task_timeout_secs ( modifying system parameters under /proc/sys file tree)

Linux Parameter 

$cat /proc/sys/kernel/hung_task_timeout_secs
120
$

Parameter Related


TEST-MAIL1 ~ #dmesg
[cut]
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
rm D ffff88107f472c40 0 16705 22512 0x00000000
ffff881014693810 0000000000000086 ffff881000000000 ffff88102013b040
0000000000012c40 ffff880471855fd8 0000000000012c40 ffff880471854010
ffff880471855fd8 0000000000012c40 ffff881017ff8e40 0000000100000000
Call Trace:
[] ? schedule_timeout+0x1ed/0x2d0
[] ? dlmlock+0x8a/0xda0 [ocfs2_dlm]
[] ? wait_for_common+0x12c/0x1a0
[] ? try_to_wake_up+0x280/0x280
[] ? __ocfs2_cluster_lock+0x1f0/0x780 [ocfs2]
[] ? wait_for_common+0x150/0x1a0
[] ? ocfs2_buffer_cached+0x8c/0x180 [ocfs2]
[] ? ocfs2_inode_lock_full_nested+0x126/0x540 [ocfs2]
[] ? ocfs2_lookup_lock_orphan_dir+0x6e/0x1b0 [ocfs2]
[] ? ocfs2_lookup_lock_orphan_dir+0x6e/0x1b0 [ocfs2]
[] ? ocfs2_prepare_orphan_dir+0x4a/0x290 [ocfs2]
[] ? ocfs2_unlink+0x6e1/0xbb0 [ocfs2]
[] ? may_link+0xda/0x170
[] ? vfs_unlink+0x9e/0x100
[] ? do_unlinkat+0x1a1/0x1d0
[] ? vfs_readdir+0xa0/0xe0
[] ? fsnotify_find_inode_mark+0x2b/0x40
[] ? dnotify_flush+0x54/0x110
[] ? filp_close+0x5c/0x90
[] ? system_call_fastpath+0x16/0x1b


Classroom

While  waiting for  read()  or write()  to/from  a file  descriptor
return, the process  will be put in a special  kind of sleep, known
as "D"  or "Disk Sleep". This  is special, because  the process can
not  be killed  or interrupted  while in  such a  state.  A process
waiting for  a return from  ioctl() would also  be put to  sleep in
this manner.

source related :http://stackoverflow.com/questions/1475683/linux-process-states

Parameter Code Internals


/*
 * Check whether a TASK_UNINTERRUPTIBLE does not get woken up for
 * a really long time (120 seconds). If that happens, print out
 * a warning.
 */
static void check_hung_uninterruptible_tasks(unsigned long timeout)
{
	int max_count = sysctl_hung_task_check_count;
	int batch_count = HUNG_TASK_BATCHING;
	struct task_struct *g, *t;

	/*
	 * If the system crashed already then all bets are off,
	 * do not report extra hung tasks:
	 */
	if (test_taint(TAINT_DIE) || did_panic)
		return;

	rcu_read_lock();
	do_each_thread(g, t) {
		if (!max_count--)
			goto unlock;
		if (!--batch_count) {
			batch_count = HUNG_TASK_BATCHING;
			rcu_lock_break(g, t);
			/* Exit if t or g was unhashed during refresh. */
			if (t->state == TASK_DEAD || g->state == TASK_DEAD)
				goto unlock;
		}
		/* use "==" to skip the TASK_KILLABLE tasks waiting on NFS */
		if (t->state == TASK_UNINTERRUPTIBLE)
			check_hung_task(t, timeout);
	} while_each_thread(g, t);
 unlock:
	rcu_read_unlock();
}


Related From Research Paper

Kernel  data collection  tools. Several  monitoring  facilities are
provided by  the Linux  kernel, which have  been exploited  in this
work. In  particular, we use  KProbes which inserts  breakpoints in
arbitrary binary code locations in charge of triggering user-defined
handler  functions. Handlers  can  be used  to collect  information
about internal kernel  variables; subsequently, kernel execution is
restored. Kdump is a tool  for failure data collection based on the
execution of  a secondary kernel,  namely capture kernel,  which is
preliminarily  loaded  into  a  reserved memory  region.  When  the
primary kernel fails, the capture  kernel is executed; then, it can
collect failure  data by reading  the main memory  state.  Built-in
hang  detection mechanisms. Several  hang detection  mechanisms are
available in the Linux OS,  which can be enabled by recompiling the
kernel.  In particular, the  following facilities  can be  used for
hang  detection: Soft  lockup detection,  i.e., the  kernel detects
whether a  "canary" task  is not scheduled  within a  timeout; Hard
lockup detection, i.e.,  if any CPU in the  system does not handles
local    timer    interrupt   for    longer    than   a    timeout;
Sleep-inside-spinlock   checking,  i.e.,  assertions   that  verify
whether there are spinlocks  that have been acquired before calling
a  sleeping function  (i.e., a  function during  which  the current
thread may block and be preempted by the scheduler); Checks on lock
API  usage, that  is: missing  lock initialization,  release  of an
already freed lock, release of a  lock by a thread or CPU different
from the lock holder, lock data structure corruption.

source : http://tinyurl.com/7pt5j9a

Assessment and Improvement of Hang Detection in the Linux Operating System
2009 28th IEEE International Symposium on Reliable Distributed Systems