HP-UX TCP/IP Performance White Paper, March 2008

36
If any CPU is saturated, it could potentially become a bottleneck for network performance, especially if the
saturated CPU is handling the interrupts from the network interface. Additional profiling can be taken using
Caliper to identify hot spots for CPU utilization.
6.2.1.1 Key CPU Performance Indicators in glance output:
The key glance CPU performance indicators are the following:
• User - % of time CPU spent by processes in user mode
• Sys - % of time CPU spent for processing system calls
• Intrpt - % of time CPU spent for processing interrupts
• CSwitch - % of time CPU spent for context switching
Abnormally high values in these values can potentially lead to a performance bottleneck. However, it is not
straightforward to tell what values are abnormal as there are numerous factors in determining these values:
whether the applications are compute-intensive or I/O intensive, speed of the CPU, whether the application
is interactive or batch processing, whether the application is a server or client, etc.
Heuristics can sometimes help identify abnormal values. For example, in typical OLTP environment the ratio
of 70% User and 10% system usage is ideal. Compute-bound environments may see ratios of 90% user to
10% system usage or more. Allowing a small percentage of idle CPU helps to provide for peaks or for
expected growth in usage. High values for system usage might indicate problems like memory thrashing or
SMP contention. On the other hand, high values for system usage do not necessarily indicate a
performance problem. They might only indicate that the application utilizes a lot of system calls.
6.2.1.2 Key factors for CPU saturation
The key factors that cause CPU saturation are the followings:
• Expensive system calls, for example fork(2), select(2), poll(2), etc.
• SMP contention involving spinlocks.
• Cache misses.
• Processing interrupts due to heavy network traffic.
• Context switching caused by process scheduling.
• Processing page faults, TLB misses, traps.
6.2.2 Monitoring CPU statistics using Caliper
Caliper is an extensive run-time performance analyzer that can be run on HP Integrity Servers with
Itanium2® processors for both kernel and applications. On HP Integrity Servers, caliper can be used to
efficiently identify CPU bottlenecks if CPUs are saturated by collecting cpu execution statistics. For example:
# caliper cpu -o cpu.txt my_program
This command runs my_program, measuring and reporting the overview metrics by taking one sample
every 8 milliseconds. By default, 125 low-level samples will be aggregated into one user-reported sample,
resulting in one aggregated sample collected per second. The text report is saved in the text file cpu.txt.
For the CPU measurement report, you may specify one or more comma-separated lists of predefined event
sets. Here are some key CPU events that can be monitored using caliper:
• cpi Provides metrics related to cycles per instruction (CPI).
• stall Provides metrics on primary CPU performance limiters by breaking the CPI into seven
components.