Linux Performance Tuning

[osmansiddiq]

Linux Performance Tuning
Contributed by Fernando Apesteguia in Desktop
04-23-06

Fernando Apesteguia concludes: "When a distribution is packaged and delivered to clients, it is designed to be fully compatible with most of the computers available in the market. This is a very heterogeneous set of hardware (hard disks, video cards, network cards, etc.). So distribution vendors like Red Hat, SuSe, Mandriva and the rest of them choose some conservative configuration options to assure a successful installation."
Why tuning my system?

This is probably the first thing you want to know. When a distribution is packaged and delivered to clients, it is designed to be fully compatible with most of the computers available in the market. This is a very heterogeneous set of hardware (hard disks, video cards, network cards, etc.). So distribution vendors like Red Hat, SuSe, Mandriva and the rest of them choose some conservative configuration options to assure a successful installation.

For instance, probably you have a very advanced hard disk with some special features that are not being used due to standard configuration settings.

To summarize: your linux distribution goes well… but it can goes even better!

For instance, probably you have a very advanced hard disk with some special features that are not being used due to standard configuration settings.
How to know what goes bad?

Once you decide to optimize your system, it would be time to find which points can be improved. To achieve this, you can use these tools:
ps

This tool allows you to view what processes are running in your system. The range of the process you can see goes from your own ones to all-wide system processes. This command shows the PID for the process, the terminal associated, the accumulated time and the name of the process.
You can use ps to monitor what processes are running on your system and which are extensively using the CPU.

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND

fernape 2474 0.0 0.1 53944 1648 tty1 Ss 13:02 0:00 -bash
fernape 2577 0.0 0.1 52732 1180 tty1 S+ 13:02 0:00 /bin/sh /usr/X11R
fernape 2609 0.0 0.0 5476 720 tty1 S+ 13:02 0:00 xinit /etc/X11/xi
fernape 2801 0.0 1.1 134360 11768 tty1 S 13:03 0:00 /usr/bin/gnome-se
fernape 2854 0.0 0.0 8796 812 tty1 S 13:03 0:00 /usr/bin/dbus-lau
fernape 2899 0.1 0.9 73576 10112 tty1 S 13:03 0:01 /usr/libexec/gcon
fernape 2962 0.0 0.0 5244 972 tty1 S 13:03 0:00 /usr/bin/gnome-ke
fernape 3180 0.0 0.1 53952 1652 pts/0 Ss 13:07 0:00 bash
fernape 3249 0.0 0.1 53952 1628 pts/1 Ss 13:09 0:00 bash
fernape 3535 0.0 0.0 5440 828 pts/0 R+ 13:18 0:00 ps u

top

If you liked ps, you will like top. Top is a very efficient tool to keep track of processes continuously. The tool is invoked by running top at the prompt. With top, you can list processes ordered by CPU or memory usage. To know more about top, please read this article: Using Top More Efficiently.
vmstat

vmstat provides information about virtual memory, processes, cpu and more. It can be useful to monitor your global system statistics. This tool is a compilation of information that can be gathered from other well-known UNIX commands

[fernape@Hammer ~]$ vmstat

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----

r b swpd free buff cache si so bi bo in cs us sy id wa

0 0 0 598412 17444 222484 0 0 258 39 1040 211 6 2 86 7

time

This is a little but useful tool to measure execution times. It takes a command as an argument. Then executes it and shows a brief statistic about time spent by the process. This can help you to find out if a command takes more time than expected. Of course, this is specially useful if you are dealing with a program developed by you, so you can modify your sources and compile again to compare results.

[fernape@Hammer ~]$ time ls -R >/dev/null

real 0m0.117s
user 0m0.057s
sys 0m0.046s

x11perf

x11perf performs a benchmark over the X server. This command is invoked running x11perf from prompt. It can take a lot of parameters but running x11perf with -all parameter reveals a lot of information (but it will take a while...).

Obviously this information is not very useful if we can not compare it with other. Here is when x11perfcomp is in order. x11perfcomp is a shell script that compares two outputs generated by x11perf. Now, the way to analyze your X server performance is clear: take a snapshot of your server performance, make some changes in the configuration file and other system settings (remember that X window system uses shared memory and other IPC mechanisms that can affect performance), run x11perf again and finally, compare the results with x11perfcomp.

Below is a truncated output of x11perf. It is a very configurable tool, you can specify certain performance tests or tell him to run all available tests.

[fernape@Hammer ~]$ x11perf -all

x11perf - X11 performance program, version 1.5
The X.Org Foundation server version 60801000 on :0.0
from Hammer

Sun Apr 2 13:21:41 2006

Sync time adjustment is 0.0895 msecs.

40000000 reps @ 0.0001 msec (6890000.0/sec): Dot
40000000 reps @ 0.0001 msec (7910000.0/sec): Dot
40000000 reps @ 0.0001 msec (7890000.0/sec): Dot
40000000 reps @ 0.0001 msec (7790000.0/sec): Dot
40000000 reps @ 0.0001 msec (8290000.0/sec): Dot
200000000 trep @ 0.0001 msec (7720000.0/sec): Dot

50000000 reps @ 0.0001 msec (10100000.0/sec): 1x1 rectangle

Results can be saved by redirecting the output of x11perf to a file. I executed x11perf once and saved the results in test1 file, then I changed some settings in my xorg.conf file and executed the benchmark again saving results in test2 file. Then I compared them:

[fernape@Hammer ~]$ x11perfcomp test1 test2

1: test1
2: test2

1 2 Operation
-------- -------- ---------
8310000.0 8140000.0 Dot

It appears that my changes were not very appropriate.

When x11perf is used, a white window is showed in the screen as you can see in the picture below. To avoid benchmarking result modification, don't touch that window nor move the mouse over it. The window will disappear when the tests are finished.

hdparm

One of the most important devices in your machine is the hard drive. It is used whenever a new process is loaded, when a memory region is swapped from memory to hard disk and when it is restored from the swap partition to memory again. And because UNIX like systems makes an extensively use of files for almost everything, it is very important to set up the hard disk properly. Hdparm helps you in benchmarking and setting your hard disk.

For now I only show you how to perform a test for your hard disk. If you are impatient and want to know what parameters you can set, read the next section: “Tuning your system”

To perform the test, type (change /dev/hda to match your hard disk.):

[root@Hammer ~]# hdparm -tT /dev/hda

/dev/hda:

Timing cached reads: 1656 MB in 2.00 seconds = 828.13 MB/sec
Timing buffered disk reads: 102 MB in 3.03 seconds = 33.71 MB/sec

I executed hdparm from root account since it is a restricted tool. We can see transfer rates for cached reads and buffered disk reads. If you want to see your current settings, try:

[root@Hammer ~]# hdparm /dev/hda

/dev/hda:

multcount = 16 (on)
IO_support = 1 (32-bit)
unmaskirq = 1 (on)
using_dma = 1 (on)
keepsettings = 0 (off)
readonly = 0 (off)
readahead = 256 (on)
geometry = 65535/16/63, sectors = 80026361856, start = 0

I recommend you to run the test at least five times to take the mean value. And it is important not to use the rest of the system if it is possible, so the test will be not affected by other block disk requests.
Tuning your system

Now, we will see some tools that help us to fine tuning our system. We will also see some tips that can be useful to save memory and because of that, they are specially important for machines with no much memory.
Disabling system services

In a typical installation a lot of services are running to ensure maximum compatibility and usability experiences to the user. Since this is a good approach for most of people, sometimes, some services are not used at all while your system is up. Because of this, it is a good idea to disable these non used services. You can do this with several programs. In Red Hat based systems, you can use setup in console mode or serviceconf in a graphical session (showed below).

Some of the services you would like to disable are:

* Network Manager: chooses the best available connection (are you always connected and permanently moving?)
* bluetooth: if you usually don't work with bluetooth devices you can disable it.
* cron: disable it if you have not programmed tasks (maybe you would like to run these tasks for yourself instead of schedule them).
* cups: printer daemon (I have it disabled in my laptop since I don't print from it)
* iptables: the linux firewall. At your own risk.
* irda: Same as bluetooth for Infra Red communications
* irqbalance: disable if you are not working with an SMP system.
* nfs: you can disable it if you are not sharing your files through NFS.
* misc. servers (ssh, sftpd...): A lot of servers are launched by default. Do you really want your system becomes a server?
* vnc: Virtual Network Console. Disable if you don't need remote graphical access to your system.

In Red Hat systems, you can add and remove system services using the chkconfig command (for non graphical sessions) in this way:

chkconfig service on/off

Other systems have their own mechanisms for managing services. See your handbook.
Disabling X at booting.

You would like to disable X or not, depending of the type of user you are. I’m a fanatical of the command line, and I have disable X at boot time. When I start my system I log in a console do my work and shut it down. This can save a lot of RAM if you don’t mind to deal with bash (or the shell you use). If you want to disable X, go to the /etc/inittab file and change the line:

id:5:initdefault:

For this one:

id:3:initdefault:

The next time you boot your system, you will be in pure command line. This not only saves memory but it is faster for simply operations, e.g. If I want to insert a line in my movie list file I waste less time in console mode than in graphical mode, waiting for GNOME, Metacity and X server to be loaded.
Removing consoles

One of the cool features of a linux distribution is the availability of several terminals. These terminals are accessed with Ctrl+Fn keys and allows to log in more than once in the system. But sincerely, I have been using Linux since several years ago and I never needed more than three or maybe four terminals at a time. So you can save some memory by disabling some of them. Even more, if you usually use a graphical desktop, you can use one or more gnome-terminals or konsoles, so you can safely left only one additional terminal in your system and free the other ones.

To achieve this, the /etc/inittab file must be modified again. Just comment the lines that set the terminals:

# Run gettys in standard runlevels

1:2345:respawn:/sbin/mingetty tty1
2:2345:respawn:/sbin/mingetty tty2
3:2345:respawn:/sbin/mingetty tty3
4:2345:respawn:/sbin/mingetty tty4
5:2345:respawn:/sbin/mingetty tty5
6:2345:respawn:/sbin/mingetty tty6

Hdparm

At this moment we revisited hdparm. We have seen that it can show disk information but it can set some properties too. The most important options you can try are:

-d: enables/disables DMA transfers.
-X: fine tuning for the DMA transfers.
-A: for look ahead when reading disk blocks.
-c: enable IDE 32 bit support.
-C: very important for laptops (power management features)

As an example, I disable my DMA transfers, look ahead and 32 bit support and run a test:

[root@Hammer ~]# hdparm -A0 -d0 /dev/hda

/dev/hda:

setting using_dma to 0 (off)
setting drive read-lookahead to 0 (off)
using_dma = 0 (off)

[root@Hammer ~]# hdparm -tT /dev/hda

/dev/hda:
Timing cached reads: 1580 MB in 2.00 seconds = 789.33 MB/sec
Timing buffered disk reads: 8 MB in 3.20 seconds = 2.50 MB/sec

Notice that buffered disk reads have decreased dramatically! It is more than 15x worse (compared to the test performed above...).
Sysctl

Probably this is the best tool for tuning linux systems. It can configure a lot of system parameters while the linux kernel is running. It handles this, by reading and writing to kernel variables through procfs files.

The complete list of available variables is shown by typing:

[root@Hammer ~]# sysctl -a

It would be a waste of time to explain every variable in this long list. Instead of that, if you are curious, you can play with them (most of them are self-explanatory).

My advice: look at networking options.
Installing video card drivers

This is one of the devices that I’m sure you are not using all its capabilities if you are working with a stock configuration and installation. The biggest video card vendors, like Nvidia, ATI or Intel provides their own proprietary drivers for linux. This license issue is the reason by which they are not included in most of the distributions. Proprietary drivers can improve your video card performance very much. Let’s use x11perf again. I run a deepcopyplane500 test with the “Standard VGA” driver chosen by my stock installation. I saved the result in test1 file. Then I change to the ATI proprietary drivers for my ATI Radeon Mobility 9700 and run the test again saving the result in test2 file.

The use of x11perfcomp will reveal the big difference between the two drivers.

[fernape@Hammer ~]$ x11perfcomp test1 test2

1: test1
2: test2

1 2 Operation
-------- -------- ---------
7.7 54.2 Copy 500x500 n-bit deep plane

Almost 8x better performance with the ATI drivers!

As you can see, it is really a good idea to install the proper optimised driver for your video card.

Going further, you can even improve the performance of the driver. Nvidia and ATI, allows you to install a precompiled package or to generate a new one. The last option allows you to compile the driver for your specific machine instead of use a generic one.

If you choose the recompile option, you will need your kernel sources, but fortunately the installers make a good job and generate the package automatically without additional work.
Drivers for frequency scaling

If you are using a processor with variable frequency (e.g. Centrino or AMD64) you would like to properly set up your processor configuration to save battery charge. Cpufreq is a program that helps you to do this. It can manage Centrino and AMD64 processors, and allows you to control the frequency value by means of an interface under /sys. With cpufreq you can fix the frequency to the minimum value (max. saving), the maximum value (better performance), or let the driver choose when to increase or decrease the frequency.

To install this daemon, you will need libsysfs and a linux kernel capable of frequency scaling. Most of the modern distributions comes with cpufreq installed. Now it is time to configure it. In my case, it was not properly configured so my AMD was at 100% of use all the time.

Above a list of configurable files is showed:

[fernape@Hammer ~]$ ls /sys/devices/system/cpu/cpu0/cpufreq/

cpuinfo_max_freq scaling_available_governors scaling_governor cpuinfo_min_freq
scaling_cur_freq scaling_max_freq scaling_available_frequencies scaling_driver scaling_min_freq

See documentation for additional information. You can install the cpufreq applet to easily configure your driver.
I want more: Compiling your linux kernel (the easy way...)

The linux kernel is the core of the operating system. It is used permanently when you are working with your distribution. Even when you are executing a “hello world” program, is the kernel who loads the program, attends to memory requests and executes system calls to perform the actual work you can see.

Due to this, it is very important to have installed and properly configured (and optimized) your linux kernel. Compiling the kernel is a long and complex topic that can't be covered here in depth. However, you can be lucky if your distribution provides default configuration files for the most common architectures. This is the case of the Red Hat based systems. Let's see how to easily recompile our kernel:

First of all, I assume that you have your linux kernel sources properly installed in your machine.

Now, we move to the sources directory, typically something like:

/usr/src/linux-2.6.9/

At this moment, we want to start to configure our new kernel.

make gconfig

We could start to configure from scratch but this is not our objective. We could forget an important feature and then our kernel will be possibly broken. So, load a configuration file from “configs” subdirectory:

These configuration files ensure that you will obtain the same result you had with a stock kernel but with your specific processor optimizations. You can safely load this file and don't worry about anything. The picture below shows the configuration file loaded.

Even more, in my case, I'm sure my processor is an Athlon64 so I can set it in the options and deselect the Generic-x86-64 option.
Then save your settings and continue with the process:

make;
make modules;
make modules_install;

And if needed, create your initrd file.

If you have compiled and installed your kernel successfully, you will take advantage of your system architecture without a big effort.
Conclusions

Linux vendors make a great effort to build optimized versions of linux systems. However the success in installation is very important too. So, the vendors should estimate both factors. Due to this, some parts of the system are not ready to work at maximum performance. Our mission is to find these points and lead them to a better performance state.

Moreover, since vendors want linux become a wide-used operating system, sometimes they start a lot of services that are not usually used. As an advanced user, it is important to disable these services and features to save memory and other system resources.

The best strategy: if you are not using that feature you can safely disable it. You will save system resources.

Linux systems can be fully configured. Let's take advantage of this!
http://www.linuxforums.org/desktop/linu ... uning.html