Zeeky

O file consists of data on the disk. A file descriptor is an inode that references the data on disk. Multiple inodes can point to the same data, giving the appearance of multiple files. If two inodes (created by "hard-linking" to a file) reference the same data, and one is deleted, the file is still there - only the inode is deleted. The data remains on disk until the last inode that refers to it is freed - then the disk space may be reused for new files. Inodes reference data blocks in disk through 12 direct blocks - each points to part of the disk. Following the direct blocks is an indirect block, a pointer to a block which contains pointers to data blocks. Then there is a doubly indirect block, a pointer to a block which contains pointers to blocks which point to data. There is also a triply-indirect block. This is how ext2 avoids fragmentation - the blocks have no need to be in linear order, so when they are not, the system does not bog down. The physical disks will have to seek just as they would with any other
filesystem if the file is not linear, but the filesystem itself has no performance degradation as a result of fragmentation. Random access is just as fast on a "fragmented" ext2 file as on an unfragmented one because the filesystem knows where all the blocks are and can fetch the millionth byte without having to read through and find out after the first hundred thousand that the rest of the file is elsewhere.

Now that we know a bit about how inodes work, we are in a better position to understand their weaknesses. The number of blocks that an inode can reference is fixed, so the maximum filesize depends on your blocksize, as shown in the following table:
          Filesystem block size: 1kB 2kB 4kB 8kB
          File size limit: 16GB 256GB 2048GB 2048GB
          Filesystem size limit: 2047GB 8192GB 16384GB 32768GB

However, the linux 2.4 kernel limits single block devices
(filesystems, partitions, what have you) to 2TB, even though ext2 can
handle larger filesystems with larger blocks. You do not sound like
you should have to worry about the maximum size of files.

You are starting to accumulate quite a few files in once place, so we
need to explore those limitations as well. There is a limit of 32k
(32768) subdirectories in a single directory, a limitation likely of
only academic interest, as many people don't even have that many files
(though huge mail servers may need to keep that in mind). The ext2
inode specification allows for over 100 trillion files to reside in a
single directory, however because of the current linked-list directory
implementation, only about 10-15 thousand files can realistically be
stored in a single directory. This is why systems such as Squid (
http://www.squid-cache.org ) use cache directories with many
subdirectories - searching through tens of thousands of files in one
directory is sloooooooow. There is however a hashed directory index
scheme under development which should allow up to about 1 million
files to be reasonably stored in a single directory. To overcome this
limitation, I would suggest you build subdirectories to store your
text files in, and implement some simple hash function within your
perl scripts to store the files in the subdirectories.

You say that about 5000 files are being created per year. Would it be
possible to create directories for each year, and put files in them on
that basis? That would keep you below the practical limits of ext2,
and should give easy-to-accomodate splitting of files amongst
directories. If the files need to be accessible as one chunk, or are
not reasonably sorted by date, I would suggest the hashing approach I
mentioned above. Your database project will also help you manage many
small data records, but still leaves you with the problem of how to
arrange your HTML files.

One possible solution to all your problems is to switch from ext2 to a
different filesystem. There are many different filesystems under
development, each with differing strengths and weaknesses. You may
want to keep your eye on the big three, Reiserfs, XFS, and JFS. XFS
is probably the most mature, followed by reiserfs, and JFS. However,
all three are less mature than ext.

So far, you're in fine shape - you've got plenty of inodes for several
years, operation. Unfortunately, there is no way to tune the number
of inodes once a filesystem is created - you have to recreate it with
new parameters. It looks to me like your only concern is the number
of files in one place.

Your question is a straightforward one in terms of data gathering, yet
there seems to be relatively little analysis to do. In any event, I
have not put $100 worth of time into this question yet, so if there is
any area you would like more clarifacion on, don't hesitate to ask -
I've barely scratched the surface of filesystem implementation, though
you quickly get bogged down in very technical aspects of the data
structures implementing the filesystem, which you may not care about.

> And as I understand it, the problem will be show up as performance
degradation. Correct?

Correct. The files are stored in a list, and searching the list is an
O(n) operation, so the more files there are, the longer it takes. A
hash table for files should be O(1) meaning it doesn't matter the
number of files. A real hash table though is O(n), except in this
case n is the number of files in each bin, which should be roughly
equal to the number of files total divided by the number of bins.
Basically, it can increase the useful limit from 10,000 to 1,000,000
by using 100 bins. Slightly less space-efficient (perhaps - depends
on implementation), but faster, which is useful.

When a file gets modified, the data gets written to the disk, and the
inode is updated to reflect the new disk blocks. If power dies
between writing data and updating disk blocks, there's data lost. If
the inode is written first, and the power dies, then the inode
references disk blocks that contain gibberish - also very bad. The
process of fscking (FileSystem ChecKing) takes forever because the
entire disk must be read searching for disk blocks that are gibberish
and disk blocks that got written but not added to the inode.

The way a journal works is that it adds extra steps to preserve what
is known as consistency - the inodes and the disk blocks are in sync.
First, a note is written which says "I'm going to update the disk
blocks now". Then the disk blocks are written. Then "I'm done
updating the disk blocks" and then "I'm going to update the inodes"
and then the inodes are updated, and then "I'm done updating inodes".
After a while the old journal entries get deleted because they're not
needed. If power dies between "going to update disk" and "updated
disk" then the data is invalid and is deleted. If the power dies
between "done updating disk" and "done updating inodes", the inodes
get updated and the data is saved - no need to read hundreds to
thousands of gigabytes of data. This is what is known as an atomic
update - the whole thing happens at once, or nothing happens.

The performance implication is that what used to be two operations
(write data, write inodes) is now six (will write data; write data;
wrote data; will write inodes; write inodes; wrote inodes) so things
could be a bit slower. This is not as bad as it sounds though because
the linux filesystem layer is very efficient and caches absolutely
everything. If you give a linux system 4GB of RAM and it only needs a
few hundred meg, it will use the rest to cache the filesystem which
makes it very fast. If only Windows did that instead of its braindead
insistence on writing memory contents to disk even when there are
hundreds of megabytes of RAM free!

In reality, it will be slower on I/O bound servers, but all the tiny
writes to be done will be cached by linux in system RAM, then by the
controller, then by the hard drives; a lightly-loaded system will have
plenty of downtime to write the updates when you're not using the
machine for anything else. If the system is heavily loaded, the disk
writes may take so long to propagate to the physical disks that the
journal operations are already undone (written, verified, and
discarded) so *nothing* ends up being written except your data. For
higher-end applications where data must be written to disk and not
cached, the 6 writes versus 2 becomes an issue, but is offset by very
expensive disk subsystems with large battery-backed redundant onboard
caches. It starts being mind-boggling after that.

5. Ok, now search for "user_frame" (without the inverted commas). You should see something like this:

Save the file and proceed to Step 2.

[root@ensim1 virtual]# find . -name mqueue -exec du -hsc {} \;
.
..
4.0k ./site10/fst/var/spool/mqueue
4.0k total
77G ./site11/fst/var/spool/mqueue
77G total
64k ./site12/fst/var/spool/mqueue
64k total
..
.
^C

[root@ensim1 virtual]# cd site11/fst/var/spool/mqueue
[root@ensim1 mqueue]# ls -lah
.
.
-rw------- 1 root mail 38MB Sep 11 14:19 dfh8B6Jpa13658
-rw------- 1 root root 38MB Sep 11 14:49 dfh8B6nsJ21407
-rw------- 1 root mail 38MB Sep 11 14:57 dfh8B6vFi23054
-rw------- 1 root root 38MB Sep 11 15:51 dfh8B7pAN03118
^C

[root@ensim1 mqueue]# ls -lah |grep 38MB |wc -l
2074
[root@ensim1 mqueue]#

This command will help you find the current culprit sites that are holding up the space. Then you can just delete away the queues. But, becarefull about deleting legitimate queues. You may want to delete only those which are huge in size.

Now, issue the following commands:

[root@ensim1 root]# cd /home/virtual
[root@ensim1 virtual]# for i in `sitelookup -a domain`;do if [ -s /home/virtual/$i/etc/mail/local-host-names ];then /bin/fixmqueue.sh $i;fi;done;

This command will convert all the /var/spool/mqueue directories of all the sites currently hosted for the system to save the queue files owned as the siteadmin user of the corresponding sites. This will prevent the system from saving the unnecessary queues as 'root' and wasting space.

The idea is to setgid the /var/spool/mqueue directory of the virtual site. By doing so, files saved in the mqueue directory will automatically be owned by the admin of the site thus restricting the mail queue space to the site's quota.

In order to save the queues as the siteadmin of the sites, the sendmail.cf of the site needs to be modified. By default ensim runs the sendmail for any or all sites as the user 'mail'. So, when it saves the queues it saves them as root:mail. To alter this to siteadmin:siteadmin, the "O DefaultUser=mail:mail" setting in sendmail.cf needs to be changed to "O DefaultUser=siteadmin:siteadmin". This is exactly what the above script does. The script above actually creates a backup of the sendmail.cf before it can make the changes. I used Perl to modify the sendmail.mc because I personally find it more convenient than 'sed' to modify files on the fly.

This script will automatically setgid the /var/spool/mqueue directory of any newly created site in both Ensim Basic and Pro (3.1.x and 3.5.x).

It can also be downloaded from: http://zort.org/codes/bash/ensim/virtDomain.sh.var-spool-mqueue

A zombie is already a dead process. No wonder it could not be killed eh? To put it in more elegant words, I found this explanation online:

> When I run top, I have a zombie process (it's a cron job that does my
> nightly backup) running and I need to kill it. I have tried kill -9
> , but it doesn't work. Any ideas? Thanx.

should do it. This works because if a process dies (e.g. crond) then init (process 1) inherits its children and init will do the wait correctly.

If your process table is showing an abnormally high number of defunct processes, and if the problem recurs even after rebooting, it is advisable to determine the cause. Look at the PPID (parent process ID) for the defunct processes and try to find out why its child processes are becoming defunct. Examining the process table (ps -elf), grepping for the parent process in the rc scripts, and making use of utilities such as u386mon or cpqmon should help.

Most likely the process has gone to sleep at a priority below PZERO, therefore signals will never reach the process and it will remain unkillable.

For further explanation of what this means, read the following.

To start with, processes run in two modes: user mode and kernel mode. When a process is in user mode it responds to interrupts and signals. When a process in user mode receives an interrupt, or a signal or makes a system call, it goes through a call gate to enter kernel mode and executes the kernel code (see fig. 1).

Once a process is in kernel mode it ignores all interrupts and signals until it is about to return to user mode. Most kernel functions execute quickly, and upon exiting kernel mode, they handle all interrupts and signals. After running for a maximum of one second, the process is preempted and is returned to the runqueue. Since kernel functions execute quickly this is not typically the cause of unkillable processes. To understand this, it is necessary to go into more detail about what goes on in kernel mode.

The following diagram (fig. 2) will be useful in understanding the details of what goes on in kernel mode. You may wish to print it out. A similar, easier to read, version of this diagram can be found on the cover of Maurice Bach's book _The Design of the UNIX Operating System_.

When a process is in kernel mode it may do a number of things. The first possibility is that it will just run some kernel code. In this case it will quickly run the code and then return to user mode or, if the program has finished, it will exit. Immediately after exiting a process is what is called a "zombie". Typically this will show up on your ps as a process. Usually the process that started this process (the parent) will clean up after the zombie (the now dead child process). If the parent has already exited then init will eventually clean up this zombie process.

Sometimes a process needs some resources that are not available at the time. If this is the case, the process is put to sleep. When a process goes to sleep it waits on an address and at a priority. This address is the value that appears in the field WCHAN when ps(C) is run with the -l option. This address is determined by the device driver that has been called and is typically the address of a local variable in the device driver.

If the process is sleeping at priority above PZERO, which is defined in /usr/include/sys/param.h, and the process receives a signal, the process is put back in the runqueue, and when it is run it returns to user mode. As it returns to user mode the signal is handled, and if a kill -9 has been sent, the process is killed.

If a process is sleeping at a priority below PZERO, signals will not cause the process to be woken up. The priority that a process sleeps at is determined by the device driver that has been called. The device driver should only put a process to sleep below PZERO if it is certain that the resource will be freed quickly so that the process can be woken up. If the process that you are trying to kill is sleeping below PZERO it will only be woken up when the resource it was waiting for has been freed. Once the process is woken up, it is put back on the run queue, and when it gets to run and as it returns to user mode, the signals are handled. If the process is never woken up by the driver the signals will never reach the process and it will remain unkillable.

The only other reason a process may be unkillable is if the process is being ptraced. The kernel will only ptrace a process on the behalf of a user process. Ptraces are for the most part only performed by debuggers.

In unix-like operating systems, ALL processes (apart from the first one) are created by other processes. To create a new process, a current process does a fork(2) system call. The kernel then creates the internal structures needed in the process table. Often, the parent process does a wait4(2) system call, which means it waits for the child process to finish. This means you can get a little info about the process after it finished, like cpu time, etc.

If you don't care when the process finishes, you have to explicitly say so, otherwise the kernel will keep the info in the process table expecting your process to eventually call wait4(2) or a similar function. A process that has finished (and so is using no memory) but has not yet been "reaped" is called a Zombie, and the kernel is keeping its process table entry alive.

Two ways to avoid creating Zombies (other than calling one of the wait() functions) include:

1. handling the SIGCHLD signal
   


2. fork(2) and then get the child to fork(2) again and then exit immediately, so that you've created a grandchild rather than a child.