[PATCH] udf: Fix deadlock when converting file from in-ICB one to normal one

[PATCH] udf: Fix deadlock when converting file from in-ICB one to normal one

During  BKL removal,  conversion  of files  from  in-ICB format  to
normal format got broken.  We call ->writepage with i_data_sem held
but  udf_get_block()  also acquires  i_data_sem  thus creating  A-A
deadlock.

We  fix   the  problem   by  dropping  i_data_sem   before  calling
->writepage() which is safe since i_mutex still protects us against
any  changes in  the  file.  Also fix  pagelock  - i_data_sem  lock
inversion  in   udf_expand_file_adinicb()  by  dropping  i_data_sem
before calling find_or_create_page().

Big Kernel Lock

Linux  contains  a  global   kernel  lock,  kernel_flag,  that  was
originally introduced  in kernel 2.0  as the only SMP  lock. During
2.2 and 2.4, much work went  into removing the global lock from the
kernel and replacing it  with finer-grained localized locks. Today,
the global  lock's use  is minimal. It  still exists,  however, and
developers need to be aware of it.

The global kernel lock is called  the big kernel lock or BKL. It is
a  spinning  lock  that  is recursive;  therefore  two  consecutive
requests for it will not deadlock  the process (as they would for a
spinlock).  Further,  a  process  can  sleep  and  even  enter  the
scheduler while  holding the  BKL. When a  process holding  the BKL
enters the  scheduler, the lock  is dropped so other  processes can
obtain it. These attributes of the BKL helped ease the introduction
of SMP  during the 2.0  kernel series. Today, however,  they should
provide plenty of reason not to use the lock.

source : http://www.linuxjournal.com/article/5833?page=0,2

mutex_init .

1. mutex_init

Initialize the mutex to unlocked state.

2. Classroom

The point of a mutex  is to synchronize two threads. When
you  have  two  threads  attempting to  access  a  single
resource, the general pattern  is to have the first block
of  code  attempting  access  to  set  the  mutex  before
entering the  code. When  the second code  block attempts
access,  it sees  the mutex  is set  and waits  until the
first block of code  is complete (and un-sets the mutex),
then continues.

source : http://stackoverflow.com/questions/34524/what-is-a-mutex

tcp_rmem – vector of 3 INTEGERs: min, default, max

ABOUT tcp_rmem

min: Minimal size of receive buffer used by TCP sockets.

It is  guaranteed to  each TCP socket,  even under  moderate memory
pressure.  Default: 1 page

default: initial size of receive  buffer used by TCP sockets.  This value  overrides net.core.rmem_
default  used by  other  protocols. Default: 87380  bytes. This value  results in window of  100535
with default setting  of tcp_adv_win_scale  and tcp_app_win:0 and  a bit less for default tcp_app_win.
See below about these variables.

max:  maximal  size of  receive  buffer  allowed for  automatically selected  receiver buffers  for  TCP
socket.  This  value does  not override  net.core.rmem_max.  Calling  setsockopt()  with SO_RCVBUF
disables automatic tuning of  that socket's receive buffer size, in which case this value is ignored.
Default: between 87380B and 4MB, depending on RAM size.

TYPICAL COMMANDLINE SESSION
[bash]
$cat /proc/sys/net/ipv4/tcp_rmem
4096 87380 2003296
$
[/bash]

TYPICAL NETWORK RELATED SOURCE CODE
[c]
/* the caller has to hold the sock lock */
static int rds_tcp_read_sock(struct rds_connection *conn, gfp_t gfp,
enum km_type km)
{
struct rds_tcp_connection *tc = conn->c_transport_data;
struct socket *sock = tc->t_sock;
read_descriptor_t desc;
struct rds_tcp_desc_arg arg;

/* It’s like glib in the kernel! */
arg.conn = conn;
arg.gfp = gfp;
arg.km = km;
desc.arg.data = &arg;
desc.error = 0;
desc.count = 1; /* give more than one skb per call */

tcp_read_sock(sock->sk, &desc, rds_tcp_data_recv);
rdsdebug("tcp_read_sock for tc %p gfp 0x%x returned %dn", tc, gfp,
desc.error);

return desc.error;
}
[/c]

RELATED KNOWLEDGE

TCP  performance depends  not upon  the transfer  rate  itself, but rather upon  the product  of the
transfer  rate and  the round-trip delay.  This "bandwidth*delay product"  measures the amount of data
that  would "fill the  pipe"; it  is the  buffer space  required at sender  and  receiver  to  obtain
maximum throughput  on  the  TCP connection over  the path, i.e., the amount  of unacknowledged data
that  TCP must  handle in  order to  keep the  pipeline  full.  TCP performance  problems  arise when
the  bandwidth*delay product  is large.  We refer to an Internet  path operating in this region as a
"long, fat  pipe", and a network  containing this path  as an "LFN" (pronounced "elephan(t)").
 

High-capacity  packet satellite  channels  (e.g., DARPA's  Wideband Net) are LFN's.   For example, a DS1
speed satellite  channel has a bandwidth*delay product of 10**6  bits or more; this corresponds to
100  outstanding  TCP segments  of  1200  bytes each.   Terrestrial fiber-optical paths will also fall
into the LFN class; for example, a cross-country  delay of  30 ms at  a DS3 bandwidth  (45Mbps) also
exceeds 10**6 bits.

LINK
https://tools.ietf.org/html/rfc1323

BusyBox – The Swiss Army Knife of Embedded Linux

UNIX Command 

$dpkg -L busybox
/.
/bin
/bin/busybox
/usr
/usr/share
/usr/share/man
/usr/share/man/man1
/usr/share/man/man1/busybox.1.gz
/usr/share/doc
/usr/share/doc/busybox
/usr/share/doc/busybox/copyright
/usr/share/doc/busybox/changelog.Debian.gz
$mount
/dev/sda1 on / type ext3 (rw,errors=remount-ro,commit=0)
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,size=5242880,mode=755,size=5242880,mode=755)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=755,size=10%,mode=755)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /run/shm type tmpfs (rw,nosuid,nodev,size=20%,mode=1777,size=20%,mode=1777)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620,gid=5,mode=620)
fusectl on /sys/fs/fuse/connections type fusectl (rw)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,noexec,nosuid,nodev)
$busybox mount
rootfs on / type rootfs (rw)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,relatime,size=991460k,nr_inodes=2478100,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=199580k,mode=755)
/dev/disk/by-uuid/26cca090-8a72-4443-859f-7a67b7188357 on / type ext3 (rw,relatime,errors=remount-ro,commit=5,barrier=1,data=ordered)
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,relatime,size=5120k,mode=755)
tmpfs on /run/shm type tmpfs (rw,nosuid,nodev,relatime,size=399156k)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
$

UNIX Explanation

BusyBox combines tiny versions of many common UNIX utilities into a
single  small executable. It  provides minimalist  replacements for
most  of the  utilities you  usually find  in GNU  coreutils, util-
linux, etc.  The utilities in BusyBox generally  have fewer options
than their full-featured GNU cousins; however, the options that are
included provide  the expected  functionality and behave  very much
like their GNU counterparts.

Why ?


BusyBox  has  been   written  with  size-optimization  and  limited
resources in mind.  It is  also extremely modular so you can easily
include  or exclude commands  (or features)  at compile  time. This
makes  it easy  to customize  your  embedded systems.  To create  a
working system, just  add /dev, /etc, and a  Linux kernel.  BusyBox
provides  a fairly  complete  POSIX environment  for  any small  or
embedded system.

fs: Make write(2) interruptible by a fatal signal

Kernel Space
linux-fsdevel@vger.kernel.org :


Currently write(2)  to a file  is not interruptible by  any signal.
Sometimes this is  desirable, e.g. when you want  to quickly kill a
process  hogging your  disk. Also,  with commit  499d05ecf990 ("mm:
Make  task in balance_dirty_pages()  killable"), it's  necessary to
abort the  current write accordingly  to avoid it  quickly dirtying
lots more pages at unthrottled rate.

balance_dirty_pages_ratelimited_nr - balance dirty memory state

Processes which  are dirtying memory  should call in here  once for
each page which was  newly dirtied.  The function will periodically
check  the system's  dirty  state and  will  initiate writeback  if
needed.  On really  big machines, get_writeback_state is expensive,
so try to avoid calling it too often (ratelimiting). But once we're
over the dirty memory limit  we decrease the ratelimiting by a lot,
to  prevent individual  processes  from overshooting  the limit  by
(ratelimit_pages) each.

Classroom

A memory-mapped file is a  segment of virtual memory which has been
assigned a direct byte-for-byte  correlation with some portion of a
file or file-like resource. This  resource is typically a file that
is physically  present on-disk,  but can also  be a  device, shared
memory  object,  or  other   resource  that  the  operating  system
canreference  through   a  file  descriptor.   Once  present,  this
correlation  between   the  file  and  the   memory  space  permits
applications  to treat  the mapped  portion as  if it  were primary
memory.


source: http://en.wikipedia.org/wiki/Memory-mapped_file

User Space
write - write to a file descriptor


A successful return  from write() does not make  any guarantee that
data  has  been  committed  to   disk.   In  fact,  on  some  buggy
implementations,  it  does  not   even  guarantee  that  space  has
successfully been reserved  for the data.  The only  way to be sure
is to call fsync(2) after you are done writing all your data.  If a
write() is  interrupted by  a signal handler  before any  bytes are
written,  then  the call  fails  with the  error  EINTR;  if it  is
interrupted  after at  least one  byte has  been written,  the call
succeeds, andreturns the number of bytes written.


fsync, fdatasync - synchronize a file's in-core state with storage device

fsync() transfers  ("flushes") all modified in-core  data of (i.e.,
modified buffer cache  pages for) the file referred  to by the file
descriptor  fd  to the  disk  device  (or  other permanent  storage
device) where that file resides.   The call blocks until the device
reports that the transfer  has completed.  It also flushes metadata
information  associated  with  the  file  (see  stat(2)).   Calling
fsync() does not necessarily ensure that the entry in the directory
containing the  file has also  reached disk.  For that  an explicit
fsync() on a file descriptor for the directory is also needed.

PXE Boot

Topic

PXE Boot

Explanation

The Preboot eXecution Environment (PXE, also known as Pre-Execution
Environment;  sometimes pronounced  "pixie") is  an  environment to
boot  computers using  a  network interface  independently of  data
storage devices  (like hard disks) or  installed operating systems.
PXE was introduced as part of the Wired for Management framework by
Intel and is described in the specification (version 2.1) published
by Intel and  Systemsoft on September 20, 1999.[1]  It makes use of
several  network  protocols   like  Internet  Protocol  (IP),  User
Datagram Protocol (UDP), Dynamic Host Configuration Protocol (DHCP)
and  Trivial File  Transfer Protocol  (TFTP) and  of  concepts like
Globally  Unique Identifier  (GUID), Universally  Unique Identifier
(UUID)  and  Universal Network  Device  Interface  and extends  the
firmware of  the PXE  client (the computer  to be  bootstrapped via
PXE) with  a set  of predefined Application  Programming Interfaces
(APIs).


The PXE protocol  is approximately a combination of  DHCP and TFTP,
albeit with  subtle modifications to  both. DHCP is used  to locate
the appropriate boot server or  servers, with TFTP used to download
the initial bootstrap program  and additional files.  To initiate a
PXE bootstrap  session the  PXE firmware broadcasts  a DHCPDISCOVER
packet extended  with PXE-specific options  (extended DHCPDISCOVER)
to port  67/UDP (DHCP  server port). The  PXE options  identify the
firmware as  capable of PXE, but  they will be  ignored by standard
DHCP  servers.  If  the  firmware  receives  DHCPOFFERs  from  such
servers, it may  configure itself by requesting one  of the offered
configurations.

source : http://en.wikipedia.org/wiki/Preboot_Execution_Environment

Get The Hang

 // next, check what the packet type was
        opt = request(43,71);
        if(opt == NULL)
        {
                logger->Event(LEVEL_INFO, "MakeReply", 1,
                  "Received proxy DHCP packet");
                pkttype = 1;
        }
        else
        {
                delete[] opt->data;
                delete opt;
                logger->Event(LEVEL_INFO, "MakeReply", 1,
                  "Received PXE request packet");
                pkttype = 2;
        }

bubble sort testing… 0.00006 Ver — 4000 No.’s related

$time php bubble.php

real	0m7.404s
user	0m7.012s
sys	0m0.080s
$time php bubble.php

real	0m7.333s
user	0m7.044s
sys	0m0.068s
$time php bubble.php

real	0m7.393s
user	0m7.068s
sys	0m0.100s
$time php bubble.php

real	0m7.345s
user	0m7.024s
sys	0m0.104s
$time php bubble.php

real	0m7.545s
user	0m7.224s
sys	0m0.080s
$time php bubble.php

real	0m7.275s
user	0m6.992s
sys	0m0.072s
$time php bubble.php

real	0m7.340s
user	0m7.044s
sys	0m0.072s
$

What is a TASK_KILLABLE state in Linux ?

ABOUT TASK_KILLABLE state

TASK_KILLABLE sleeping state.

Like most versions of Unix, Linux has two fundamental ways in which a process  can be put  to sleep. A
process which is placed  in the TASK_INTERRUPTIBLE  state  will sleep  until  either (1)  something
explicitly wakes it up, or (2) a non-masked signal is received. The TASK_UNINTERRUPTIBLE state, instead,
ignores signals; processes in that  state will  require an  explicit wakeup  before they  can run
again.   There are  advantages and  disadvantages to  each  type of sleep. Interruptible sleeps enable
faster response to signals, but they  make   the  programming   harder.  Kernel  code   which  uses
interruptible sleeps must always check to see whether it woke up as a result  of a signal, and, if  so,
clean up whatever  it was doing and return  -EINTR back  to user space.  The user-space  side, too,
must  realize  that  a  system  call was  interrupted  and  respond accordingly;  not all  user-space
programmers  are known  for their diligence in this regard. Making a sleep uninterruptible eliminates
these problems, but at the cost of being, well, uninterruptible. If the expected  wakeup event does  not
materialize, the  process will wait forever and there is usually nothing that anybody can do about
it  short  of rebooting  the  system. This  is  the  source of  the dreaded, unkillable process  which is
shown to be  in the "D" state by ps.


TASK_KILLABLE behaves like  TASK_UNINTERRUPTIBLE with the exception that fatal signals will interrupt the sleep.

TYPICAL PART OF A HEADER FILE RELATED (include/linux/sched.h)
[text]
/* Convenience macros for the sake of set_current_state: */
#define TASK_KILLABLE (TASK_WAKEKILL | TASK_UNINTERRUPTIBLE)
#define TASK_STOPPED (TASK_WAKEKILL | __TASK_STOPPED)
#define TASK_TRACED (TASK_WAKEKILL | __TASK_TRACED)

#define TASK_IDLE (TASK_UNINTERRUPTIBLE | TASK_NOLOAD)
[/text]

LINKS
https://lwn.net/Articles/288056/
https://www.ibm.com/developerworks/linux/library/l-task-killable/
https://elixir.bootlin.com/linux/latest/ident/TASK_KILLABLE
https://elixir.bootlin.com/linux/latest/source/include/linux/sched.h#L90