Level: Introductory
Carlos Ferreira, Product Manager, IBM
06 Aug 2004
In this whitepaper Carlos Ferreira provides information on how to configure your environment to optimize performance of the new IBM Rational ClearQuest Web.
The intent of this whitepaper is to provide IBM® Rational® ClearQuest® Web administrators with system monitoring, tuning, and sizing information. This document applies to ClearQuest Web version 2003.06.00.500 (not to previous versions of ClearQuest Web, which were based on Active Server Pages technology). In order to optimize the performance of ClearQuest Web effectively, it is important to understand the concepts described here. From a top level point of view, the primary areas to consider in a ClearQuest Web deployment are:
- ClearQuest Web Application
- Apache-provided with the Rational Web Platform (RWP)
- Tomcat-provided with the RWP
- ClearQuest Server (request manager and registry server)
- Server hardware and operating system configuration
- Network configuration
- License server configuration
In addition, these important considerations are covered in a separate Rational ClearQuest performance white paper entitled "Rational ClearQuest Performance Improvement" by James Tykal, Senior Technical Marketing Engineer:
- Schema design considerations
- Database configuration
In order to tune and size a ClearQuest Web application deployment effectively, you must first ensure that you have monitoring tools in place. They will measure critical performance parameters for each of the above areas, and are also useful for troubleshooting production performance problems.
Monitoring
Monitoring tools are dependent on the operating system on which you are deploying ClearQuest Web. The specific tool(s) you should use are naturally dependent on which ClearQuest Web component you are monitoring. In any case, your monitoring process should include storing results for specific configurations so that you can see performance trends over time.
Memory and CPU Usage
Operating system resources like process memory and CPU use are among the most important parameters to measure. Average CPU utilization of greater than 90% may be unhealthy-an indication that peak CPU usage of 100% is occurring. This may cause sporadically slow transaction response times and inconsistent memory usage patterns. To help you track this, Microsoft® Windows® comes with a performance monitoring tool that is available in the Microsoft Management Console. To launch the performance tool, type the following either in the Run dialog box or the at the MS-DOS prompt:
perfmon / WMI
For additional details on how to use this performance monitoring tool see the MSDN library performance tools page.
To measure UNIX® operating system resources, the top command is a useful performance monitoring tool. For additional details on how to use the top command, see: http://www.mcsr.olemiss.edu/cgi-bin/man-cgi?top+1 Table 1, following, shows some typical output of the top command.
Figure 1. Example output from UNIX top command table 1
Tables 2 and 3 show key ClearQuest Web processes to monitor.
Table 2. ClearQuest Web application processes on Windows
| Service | Process | Additional information |
| Rational Web Platform, Apache HTTP server | rwp.exe | There are two rwp.exe processes for fault tolerance; one is master, one is worker. If the worker process dies, the master starts up a new one. |
| Rational Web Platform, Tomcat servlet engine for ClearQuest Web |
| The properties file that controls these Java processes is |
| Rational Web Platform, Tomcat servlet engine for ReqWeb |
| ClearQuest Web does not use this service. |
Table 3. ClearQuest Server process on Windows
| Service | Processes | Additional information |
| Rational ClearQuest |
| The properties file that controls these Java® processes is |
| Rational ClearQuest |
| None. |
Renaming ClearQuest Java Processes on Windows for Easier Monitoring
In Tables 2 and 3 above, there are four different ClearQuest Web processes called java.exe, which makes it difficult to isolate and monitor specific ClearQuest processes. To clarify their function, you should rename the processes specified in the bootstrap files shown on Table 4. The three steps required to rename a ClearQuest Java process are:
1. Rename the Java executable in the bootstrap file that initiates the process.
2. Create a copy of the java.exe program with the new name.
3. Restart the services.
Table 4. Java process bootstrap properties files
| CQWJ Java Process | Bootstrap Filename and Location |
| Registry Server |
|
| Request Manager |
|
| Tomcat |
|
1. Modify the following section in each bootstrap file:
## This is the Java interpreter used for running Tomcat#wrapper.javabin=$(wrapper.java_home)\bin\java.exe |
2. Change the java.exe file name to something more meaningful, such as java_requestmgr.exe for the Request Manager. If you are working with a multiple Request Manager configuration, name each Request Manager process uniquely, such as java_requestmgr_A.exe.
3. Create copies of the actual java.exe program and give the copies the same names as the renamed processes. The java.exe program used by ClearQuest Web Application is in C:\Program Files\Rational\Common\Java\JRE\bin. Simply duplicate this program and rename the copy. Be sure to keep the original java.exe in this directory.
4. Restart all the ClearQuest Web processes to have the renamed processes take effect. You can use Windows Task Manager to examine the renamed processes. On UNIX, the command ps -ef | grep java will provide all the java processes that are running, along with their executable directory and process ID. The process ID can then be correlated with the process ID provided in the top command. Table 5 shows the Java process directory and the component to which it corresponds.
On UNIX, conversely, renaming the Java processes is not necessary because the process IDs can be determined from the information detailed in Table 5.
Table 5. ClearQuest Web application processes on UNIX
| Service | Process | Additional information |
| Rational Web Platform, Apache HTTP server |
| On UNIX, the rwp processes are collectively one instance of Apache. Apache consists of one master process and several worker processes (instead of one master and one multi-threaded worker like that running on Windows). You can tune the number of worker rwp processes by editing the |
| Rational Web Platform, Tomcat servlet engine |
| The word "arch" represents the architecture for example, sun5 for Sun Solaris. |
| Registry Server | java | Process ID can be found in the file |
| Request Manager | java | Process ID can be found in the file |
Users
When you are seeking to understand system performance, the number of concurrent users accessing the ClearQuest Web application during a performance test is an important metric to capture. Table 6 below provides access log information for Apache and Tomcat.
| Name | Description | Location |
|
| Apache access log containing the list of IP addresses that have accessed ClearQuest Web and when they accessed it. | Windows: |
|
| Tomcat access log | Windows: |
The access.log file contains the client IP addresses, as well as the times that the clients have accessed the application. Here is an example of the output:
192.168.223.1 - - [23/Aug/2003:17:36:37 -0400] "GET/cqweb/main?command=ExecuteQuery&queryName=Public+Queries%2FAll+Defects&dbid=33554655&pNm=Public+Queries&pDbid=33554440&rmsessionid=e8746e42-eb91-4773-bd99-c0a8df01e187 HTTP/1.1" 200 1571192.168.223.1 - - [23/Aug/2003:17:36:37 -0400] "GET/cqweb/main?command=ExecuteQueryWithoutPrompts&dbid=33554655&rmsessionid=e8746e42-eb91-4773-bd99-c0a8df01e187&bNewQuery=false&bNewSavedQuery=true&queryName=Public%20Queries%2FAll%20Defects HTTP/1.1" 200 519192.168.223.1 - - [23/Aug/2003:17:36:38 -0400] "GET/cqweb/main?command=GetWSContentFrame&dbid=33554655&qid=33554655&rmsessionid=e8746e42-eb91-4773-bd99-c0a8df01e187 HTTP/1.1" 200 1151192.168.223.1 - - [23/Aug/2003:17:36:38 -0400] "GET /common/html/null.html HTTP/1.1" 304 0 |
Other important values to monitor include:
- Active HTTP sessions
- Active ClearQuest sessions
- Database connections
- Worker threads
You can monitor active HTTP sessions-using the RWP Tomcat servlet access logging -- by modifying C:\Program Files\Rational\Common\rwp\conf\server.xml. For more information go to: http://jakarta.apache.org/tomcat/tomcat-4.1-doc/config/valve.html#Access%20Log%20Valve
You can log the following items:
- %a -- Remote IP address
- %A -- Local IP address
- %b -- Bytes sent, excluding HTTP headers, or '-' if zero
- %B -- Bytes sent, excluding HTTP headers
- %h -- Remote host name (or IP address if resolveHosts is false)
- %H -- Request protocol
- %l -- Remote logical username from identd (always returns '-')
- %m -- Request method (GET, POST, and so on)
- %p -- Local port on which this request was received
- %q -- Query string (prepended with a '?' if it exists)
- %r -- First line of the request (method and request URL)
- %s -- HTTP status code of the response
- %S -- User session ID
- %t -- Date and time, in Common Log Format
- %u -- Remote user that was authenticated (if any), else '-'
- %U -- Requested URL path
- %v -- Local server name
You can also monitor the current status of your HTTP Web server using the following URL: http://Hostname/server-status, where Hostname is the name of the server on which ClearQuest Web is deployed. The following information is provided:
- Restart Time: Thursday, 15-Apr-2004 11:05:27 Eastern Daylight Time
- Parent Server Generation: 0
- Server uptime: 6 minutes 26 seconds
- Total accesses: 67 - Total Traffic: 38 kB
- .944 requests/sec - 548 B/second - 580 B/request
- Number requests currently being processed, and number idle workers
The following additional information can be accessed by adding the ExtendedStatus On directive to the C:\Program Files\Rational\Common\rwp\conf\rwp.conf, as shown below.
##----------------------------------------------# Directives related to inquiring server status#----------------------------------------------ExtendedStatus On |
- Srv -- Child Server number or generation
- PID -- OS process ID
- Ac -- Number of accesses this connection / this child / this slot
- M -- Mode of operation
- SS -- Seconds since beginning of most recent request
- Req -- Milliseconds required to process most recent request
- Conn -- Kilobytes transferred this connection
- Child -- Megabytes transferred this child
- Slot -- Total megabytes transferred this slot
Performance
Other key indicators to consider when monitoring web applications include transaction response times and transaction throughput. There are many software tools available that allow you to simulate user transaction load on your ClearQuest Web deployment. Note that these tools often require that you record or build scripts that reflect your user transaction load profile. They also provide metrics regarding the number of transactions that succeed or fail, and the average, minimum, and maximum response times per transaction. Monitoring these transactions is an important step in being able to size your hardware for your production deployment effectively. In fact, IBM® Rational® Test Manager ™ was used to test the performance of the IBM Rational deployment of ClearQuest Web.
Tuning
Rational ClearQuest Web Application
The ClearQuest Web application and the ClearQuest Server are the two installable components of the latest version of ClearQuest Web. The ClearQuest Web application installs the RWP (including Apache version 2 and Tomcat version 4), which is the common web platform for serving the IBM® Rational® RequisteWeb®, ClearQuest Web, IBM® Rational® ClearCase® Web, and IBM® Rational® Project Console™ web clients. The ClearQuest Web Application is the presentation tier of the ClearQuest Web, deployed as a web application on the Tomcat servlet engine. Some ClearQuest Web static HTML and Javascript files are also deployed on the Apache Web Server.
When tuning the ClearQuest Web deployment, do not change too many configuration parameters at one time. After making a change, monitor the application using monitoring tools described in the previous section.
As opposed to the software configuration, the single biggest hardware issue affecting ClearQuest Web performance is RAM. If multiple applications are using a server that lacks available memory or CPU resources, the most likely result is poor performance. To ensure that there is adequate memory and CPU available for each ClearQuest process deployed on the server, use the monitoring tools described in the previous section to determine which processes are consuming memory and CPU. With this information, you can stop or move unnecessary processes to a different server to conserve memory or CPU. In addition, you can increase the maximum JVM memory for each of these processes to eliminate Out of Java Memory Error or OutOfMemoryException reported in the log files.
For the peak number of concurrent clients, you should ensure there is a sufficient memory for Apache, Tomcat, Request Manager, and Registry Server processes by looking at your process list using a tool such as top. If you deploy all of these applications on the same server, their total memory should not be more than 80% of the total available memory, since some memory needs to remain available for the operating system processes. Finally, run the latest stable release and patch level of the operating system that you choose. Many OS suppliers have introduced significant performance improvements to their TCP stacks and thread libraries in recent years.
Reduce the Amount of Logging
Logging to files requires the use of system resources: disk, memory, and CPU. Excessive logging degrades performance, and the sheer number of log entries can also make it more difficult to find problems. Logging should be set to
Tuning Apache
This section covers some of the more important tuning parameters of the Apache HTTP server. For a complete list of all the parameters that can affect performance, see the Apache Web site. You can obtain significant performance gains simply by ensuring that settings are correctly configured for the expected application load. Since every ClearQuest Web client's request goes to the Apache HTTP server first, its performance plays an important role in the overall system operation.
Secure Sockets Layer (SSL)
If SSL client authentication is configured, the server requests the client's certificate for any HTTPS (HTTP + SSL) request. This of course adds overhead and therefore decreases system performance. If you don't need SSL, you can disable SSL connections in the Apache Web server. See the New ClearQuest Administrator's Guide for information about enabling and disabling SSL Connections in Apache.
KeepAlive Directive 2
This directive enables HTTP persistent connections. The syntax is: KeepAlive On|Off Default: KeepAlive On
The Keep-Alive extension to HTTP/1.0 and the persistent connection feature of HTTP/1.1 provide long-lived HTTP sessions, which allows multiple requests to be sent over the same TCP connection. In some cases this has been shown to result in an almost 50% speedup in latency times for HTML documents with many images. To enable Keep-Alive connections, set KeepAlive On.
A Web server should never have to swap, as swapping increases the latency of each request beyond a point that users consider "fast enough". This causes users to hit stop and reload, further increasing the load. You can, and should, control the MaxClients setting to prevent your server from spawning so many children that it starts swapping. This directive specifies the maximum number of child processes that will be created to serve requests, and limits the number of simultaneous requests that will be served. Any connection attempts that are over the MaxClients limit will normally be queued, up to a number based on the ListenBacklog directive. You should set this to the maximum number of clients that your environment can manage without experiencing throughput degradation or a prohibitive increase of the response time.
ExpiresByType Directive
A major performance gain in high latency environments is possible if you add the Mod_Expires Module to the RWP, and modify the web application configuration to utilize the ExpiresByType directive. These directives will reduce the number of round trips from the client browser to the RWP Apache Web server. A web server has the option to tell the browser how long it should cache content. By default, the RWP Apache does not tell the browser anything about when the content should expire, which basically means that the browser determines (through its own heuristic algorithms) when the content will expire. When the browser checks for a new version, it sends a condition get request ("send me a new copy of this file if your copy is newer than mine"). However, since the server is rarely updated it almost always responds with a 304 response, which tells the browser it already has the latest version. All of these 304 requests are essentially wasted requests since there is no new data returned. The ExpiresByType directive may be added to the ClearQuest Web in a future release.
Tuning Tomcat
Setting the Servlet Queue Size
The maximum number of concurrent connections that a servlet is allowed to handle should typically be lower than the total number of http threads. This is because not all the requests take the same time to fulfill. Some requests may require a limited amount of computing power, while other HTTP requests may simply need an HTML page. ClearQuest Web includes the Apache and Tomcat versions that support the AJP 1.3 and 1.4 protocols, and the maxProcessors attribute of the \Rational\Common\rwp\conf\server.xml file
You can configure the available minimum and maximum Java Virtual Machine (JVM) memory used by Tomcat on Windows by modifying the C:\Program Files\Rational\Common\rwp\bin\jk_service.properties file. Modify the section shown below:
## JVM Options## Useful Options:# -Xms2m = Initial heap size, modify for desired size# -Xmx256m = Maximum heap size, modify for desired size |
For UNIX, modify the JAVA_OPTS definition in the setclasspath.sh script, which is in /opt/rational/common/rwp/bin.
As an example of how much memory you may require, performance tests completed using an example customer schema showed that approximately 400 kilobytes of memory per user was required for the CQ Web application session on the Tomcat servlet engine. The steady state memory usage without any users was approximately 60 megabytes.
Garbage Collection
The best way to optimize garbage collection is through testing. You'll want to set the minimum and maximum JVM memory size close to one another, because JVM maximum and minimum memory settings that are too far apart will cause the garbage collector to run less frequently, and will therefore require more time and CPU resources to collect more objects. Garbage collections with a frequency of less than one minute should produce a good result. To determine the duration between garbage collection intervals, measure the time between major decreases in Java process memory.
The ClearQuest Server includes two components: the Request Manager and the Registry Server. The Registry Server is a software load-balancer process that directs requests from the ClearQuest Web application (deployed on the Tomcat servlet engine) to the Request Manager(s). The Request Manager stores a user's ClearQuest session. The Request Manager is responsible for activities like submitting, modifying, viewing, and deleting queries and records in the ClearQuest database.
A thread is a unit of processing capacity available to perform computing within the JVM. The greater the number of threads the higher the CPU load. In cases where CPU load is high, and the ClearQuest Web application is CPU-bound, the number of threads should be reduced. In cases where CPU usage is low, on the other hand, you can increase the number of threads to handle more user or transaction load. When increasing the number of threads, check to make sure that there is adequate memory available for the Request Manager to handle the additional concurrent load. In order to understand how to modify the ClearQuest Web thread parameters described below, see the Rational ClearQuest Web Administrator's Guide (the Customizing the jtl.properties File section, as well as Table 5, jtl.properties File Locations).
MAXSERVICEREQUESTTHREADS -- This specifies the maximum number of concurrent ClearQuest Web application requests that the Request Manager can process. Although you can increase this value to try to minimize the queued time for multiple requests, the default value for this property is set to five based on the assumption that ClearQuest Web is deployed on a machine with one CPU. This is because testing suggests that this is the optimal number of threads for one CPU. On a single-CPU machine, setting this value to less than five might result in under-utilization of system resources, while increasing it to more than five might result in slight degradation in the response times.
If the ClearQuest Server is deployed on a single-CPU machine, use the default value. If the machine has more than one processor, then modify this setting to five times the number of CPUs, i.e., 5 * #CPUs. For example, if the ClearQuest Server is installed on a four-processor machine, modify this setting to 20.
MAXSERVICERESPONSETHREADS -- this is the number of threads sending responses back to the various requestors. The default value is two, again for a single-CPU machine. This value should be changed to reflect two times the number of CPUs. For example, on a four-CPU machine, this value should be set to eight.
Testing with an example customer schema on a four-CPU server with MAXSERVICEREQUESTTHREADS set to thirty, fourteen simultaneous users successfully preformed a submit action all within thirty seconds. In cases where MAXSERVICEREQUESTTHREADS is not set high enough, you may see errors like Failed to execute use case 'cq_getworkspace_folder'. String index out of range: -1.
RPCREQUESTTIMEOUTSECOND -- This setting defines the amount of time in seconds that the ClearQuest Web application waits for the Request Manager to processes requests. If the ClearQuest Web application does not receive the response from Request Manager in the time defined by this setting, it times out and notifies the user. The default value for this timeout is set to 90 seconds. Ideally, a request should not require 90 seconds to process. The Request Manager processes most of the requests it receives in just a few seconds. However, there might be some scenarios where the requests take a very long time. For example, a query that is searching for a word in the description area of all the defects might take longer then 90 seconds, and would result in a timeout. Similarly, submitting a defect that has a lot of hooks in it could result in a long processing time on the Request Manager, again resulting in a timeout.
Increasing this setting might provide an easy temporary resolution for the time-out error, but it would be the wrong solution for end users. For example, setting this value to 300 seconds would result in users having to wait five or more minutes for a query or record operation. The right solution is to find which queries, records, or hooks are taking a long time to execute, determine why, and resolve the problem. Ask yourself whether the filter criteria are badly defined and overly complex, or if there are there too many hooks into the record. Settling these types of issues could provide a permanent solution that is more responsive, requires less system resources, and has fewer time-out errors.
It is likely that you will have to modify the Request Manager memory to reflect your deployment characteristics. You can configure the available minimum and maximum JVM memory used by the ClearQuest Request Manager by modifying the file named C:\Program Files\Rational\ClearQuest\cqweb\cqserver\ requestmgr_service.properties. For UNIX, modify /opt/rational/clearquest/cqweb/cqserver/start_cqrm.sh Modify the section shown below:
## JVM Options## Useful Options:# -Xms2m = Initial heap size, modify for desired size# -Xmx256m = Maximum heap size, modify for desired size |
Again as a guideline, testing showed that the ClearQuest Request Manager steady-state memory usage with one user was approximately 60 megabytes. Performance tests done using an example customer schema showed that ClearQuest Request Manager required 5 to 12 megabytes of additional memory for each additional user doing 15 transactions per hour.
It is not likely that you will have to modify the Registry Server memory. Nevertheless, the available minimum and maximum JVM memory used by the ClearQuest Registry Server can be set by modifying C:\Program Files\Rational\ClearQuest\cqweb\cqregsvr\ cqregsvr_service.properties. For UNIX, modify /opt/rational/clearquest/cqweb/cqregsvr/start_cqreg.sh. Modify the section shown below:
## JVM Options## Useful Options:# -Xms2m = Initial heap size, modify for desired size# -Xmx256m = Maximum heap size, modify for desired size |
The ClearQuest Registry Server steady state memory usage with one user was approximately 20 megabytes. Performance tests done using an example customer schema showed that for each additional user, ClearQuest Registry Server needed a negligible amount of additional memory.
Tuning the Network Configuration
While this paper's intent is not to provide a comprehensive guide for tuning your network, this section will provide information on some quick ways to diagnose network configuration issues that may cause performance problems with the new ClearQuest Web.
Network Latency
Poor network performance may cause performance degradation of any web application. When troubleshooting performance problems with the new ClearQuest Web, you should verify network latency between:
- The computer making a ClearQuest Web client request and the Web Server
- The Web Server and ClearQuest Web Application Server
- ClearQuest Web Application Server and the Request Manager
- Request Manager and the Database Server
- The Registry Server and the Request Manager
- The Request Manager and the License Server
- All of the components utilized in the ClearQuest Web application and the Domain Naming Server
Since the new ClearQuest Web Application installs the Web Server (Apache) and servlet engine Server (Tomcat) on the same server, you don't have to verify item 2. However, this may be applicable in cases where a customer has installed a proxy web server between the web client making the HTTP request and the ClearQuest Web Application Server. These proxy web servers are sometimes used with firewalls for security purposes, or to provide a software load-balancer with multiple ClearQuest Web Clusters.
Determining Network Latency
Network latency can be determined by typing the ping command at the starting server and then the name of the target server between which you are trying to determine the latency. You must determine the latency between each server. The results below are from a ping command.
ping sus-ma1ratlt2Pinging sus-ma1ratlt2 [9.34.111.60] with 32 bytes of data:Reply from 9.34.111.60: bytes=32 time=125ms TTL=119Reply from 9.34.111.60: bytes=32 time=127ms TTL=119Reply from 9.34.111.60: bytes=32 time=125ms TTL=119Reply from 9.34.111.60: bytes=32 time=126ms TTL=119Ping statistics for 9.34.111.60: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),Approximate round trip times in milli-seconds: Minimum = 125ms, Maximum = 127ms, Average = 125ms |
If the combined network latency between the web client, ClearQuest Web Application, Request Manager, and the Database Server is greater than 200 milliseconds, ClearQuest Web transaction performance can degrade. Note that performance problems associated with ClearQuest Web login are often caused by latency between the Request Manager Server and the Rational Common License Server. In addition, ping statistics revealing packets being lost at a rate greater than 5 percent could also indicate degraded ClearQuest Web transaction performance. Consult your network administrator to resolve network latency and lost packets concerns.
Tuning the Rational Common License Server
ClearQuest Web uses the Rational Common License Server based on FlexLM from Globetrotter Software. Rational provides tools to monitor and administer your ClearQuest licenses. One tool is LMTools by Globetrotter Software, which provides License Server status and server diagnostics. To improve the performance of your ClearQuest deployment, ensure that there is good network connectivity between the Request Manager and ClearQuest License Server. As mentioned above, network connectivity should have a latency of less than 200 ms and very few or no dropped packets. Furthermore, performance testing has revealed that each additional license server causes the login transaction response time to reduce an additional four seconds. Also consider co-locating your license server with your Request Manager.
Additional time to check out a license is required at sites that have deployed the new ClearQuest Web with ClearQuest MultiSite. A MultiSite license is checked out when the user logs in. A ClearQuest license, on the other hand, is checked out when the user selects the user database.
Licenses are checked out only once. The ClearQuest Web Application does not have to have network connectivity to the license server, only the Request Manager does.
Hardware Sizing for a ClearQuest Web Deployment
Server sizing to meet performance needs is unique to each customer's deployment environment. Performance is highly dependent on a customer's database schema, usage, web server configuration, database configuration, and network configuration. The IBM Software Group does not give customer-specific recommendations for sizing or quantifying how much or what type of hardware a customer needs to meet specific ClearQuest Web transaction load or performance criteria. In order for customers to determine their hardware requirements to deploy ClearQuest Web, they should start with the minimum recommended hardware outlined in the installation guide. Then, they should execute performance tests to determine how many concurrent users can be accommodated while maintaining their desired transaction response times and transaction rates for a single-server environment.
Quantifying the number of concurrent users per server allows you to calculate how many servers you need to meet your anticipated peak concurrent user load. You should add more server capacity to allow for hardware failover or administrative maintenance. The reason for this approach is that one customer may do performance tests with a hardware configuration where they are able to achieve 50 concurrent users with a response time of 10 seconds. Another customer with the same hardware configuration could perform the same tests and determine that to maintain a response time of 5 seconds or less they can't have more than 25 concurrent users. The latter customer will have to buy twice the number of servers to meet their performance requirements. This is why it is important that each customer performs their own performance testing to determine specific hardware needs.
Capture Important Hardware Sizing Parameters
To effectively size your ClearQuest Web Application deployment, the following questions should be answered:
- What is the user population (that is, the total number of ClearQuest Web users in the user community)?
- What is the number of peak concurrent ClearQuest Web users? For medium to larger deployments with more than 100 total users, a good starting estimate is that 1 out of 5 users will be using ClearQuest web at any given time. For teams smaller than 20 users, this may be 2 out of 3 total ClearQuest Web users.
- What is the expected number of transactions per user? For example, login would be one transaction and running a query would be another transaction.
- The Load Model (see the following section for details) is the number of transactions generated by a population of users based on the expected behavior of those users. This is the transaction profile per the ClearQuest Web user group. What are the types of transactions you expect users to perform? Will they be primarily read or write database activities? In the future versions of ClearQuest, some transactions may be generated by web service usage. For example, a native ClearCase client that is leveraging the base ClearCase and ClearQuest integration may be submitting transactions to the ClearQuest Web server. This additional load on ClearQuest needs to be considered when doing server sizing.
- For each transaction type what is the allowable transaction response time?
- What is the estimated throughput? Throughput is the number of transactions that the system will perform in 1 second (this is known as TPS, transactions per second)
- What is the network topology and physical deployment architecture?
- Will a hardware or software load-balancer be used? If so, ensure that your performance test environment reflects these requirements.
- What is the network latency and bandwidth?
- Will SSL or a firewall be used? SSL will often have a negative impact on performance. Independent of ClearQuest Web, HTTPS performance for web applications has been documented by the industry to be up to 5 times slower with regard to transaction response times, and have up to 1/5 the system throughput. For more information, see Figure 1 at the following Web site: http://digital5.ece.tntech.edu/hexb/publications/https-STAR-03122003.pdf. If SSL will be used, the customer should consider buying hardware SSL accelerators, or buying higher end servers.
- Is failover or high availability required? If so, you will have to ensure that duplicate servers are provided to accommodate a hardware or software failure, or to allow maintenance work without discontinuing server operation.
Load Model
Here is an example of how to define the average and peak performance requirements for a ClearQuest Web deployment.
- Average performance is based on a steady-state of system load, assuming:
- Even distribution of user demand over a 9 or 24 hour time period
- Randomly distributed transactions over a 9 or 24 hour time period
- Peak performance is based on a non-uniform demand for system performance over the day:
- Heavier in the early morning and after lunch, for example
- Statistical peaks in demand based on random user factors
- Optimizing the system for peak performance (without degradation of response time) should be based on evaluating the price / performance impact of the infrastructure
Example Load Calculation
Here is an example of how to calculate average and peak transaction rates.
AVERAGE LOAD MODEL
User population 1500 users User profiles Sessions/day Transactions/session Heavy users 20% 12 10 Light users 80% 2 6% active per day 90% of Heavy users; 70% of Light usersWork Day 9 hours (all activity evenly distributed)User session is 20 minutes User Sessions per day is 4,920 = Heavy users 1500 users*20%*12 sessions/day * 90% Active Light users 1500 users*80%* 2 sessions/day * 70% ActiveTransactions per day is 42,480 = Heavy users 1500 users*20%*12 sessions/day * 90% Active * 10 Transactions/session Light users 1500 users*80%* 2 sessions/day * 70% Active * 6 Transactions/sessionConcurrent users is 182 = (20 * 4,920) / (9 * 60)Transactions per second 1.31 = (42,480 / 9 hr) * 1hr/60min*1min/60sec PEAK LOAD MODEL
- For user populations of more than 100 total users, a good starting peak load estimate is that 1 out of 5 users will be using CQ Web at the same time.
- For user populations smaller than 20 users, a peak load estimate is that 2 out of 3 total CQ Web users will be using CQ Web at the same time.
Concurrent users is 300 = 1500 / 5
Transactions per second 2.16 TPS = 300*(1.31/182)
Create and Execute Performance Tests
Based on the ClearQuest Web user requirements and sizing parameters, create a set of performance tests that reflect your environment. Start by configuring a group of ClearQuest Web Application and Request Manager Servers. Establish your monitoring tools and confirm that your testing environment reflects your production deployment environment. Then notify your web server, ClearQuest Server, and database server administrators that you will be executing performance tests. Have them monitor their servers during the performance tests. Gradually add ClearQuest Web user load at a consistent rate (such as one user per minute) and record key performance metrics at consistent time intervals. These include:
- Memory use for each ClearQuest Web component process
- CPU use for each ClearQuest Web component process
- Number of active HTTP sessions
- Number of active ClearQuest sessions
- Number of database connections
- Number of users
- Number of transactions per hour and total transactions
- Transaction response times
- Number of threads
Determine Bottlenecks and Tune ClearQuest Components
After executing the performance tests, review the monitoring data to determine which processes become resource constrained. For each such process determine what configuration parameters could improve performance (based on the tuning parameters outlined in the tuning section of this document). Adjust these parameters and execute performance tests again. During this process, you should:
- Optimize the number of ClearQuest Web Application Servers versus the number of ClearQuest Servers in your deployment
- Determine how many concurrent users and transactions per second your configuration can support while maintaining your desired level of service
- Add additional ClearQuest Servers to provide additional user and transaction capacity or failover support
- When the ClearQuest Web Application is the bottleneck, consider adding additional load-balanced ClearQuest Web Application Servers with their own dedicated ClearQuest Servers.
Determine the Required Number of Servers
For the optimized single-server configuration, execute performance testing utilizing different transaction loads. Graph the average transaction response time (y axis) versus transaction rate (x axis). Determine the maximum transaction rate that is achievable while maintaining an acceptable level of service transaction response time.
In order to maintain a response time of less than 10 seconds per transaction using the peak load example of 2.16 TPS, the max transaction rate cannot exceed .875 TPS. The number of servers required is the peak load divided by the max transaction rate: 2.16/.875 = 2.4. For this deployment, then, three ClearQuest servers are required. The single-server deployment in Figure 1 could be used if an additional ClearQuest Server B and C were added. If 4 or more ClearQuest servers are required, having two Figure 1deployments load-balanced would provide failover support in case Computer A was disabled. Having more than 5 ClearQuest servers load-balanced on a single ClearQuest Web application, on the other hand, may result in the ClearQuest Web application becoming the performance bottleneck.
Empirical Performance Metrics
Extensive performance, reliability, and scalability tests were performed with ClearQuest Web using an example customer ClearQuest Schema. The results of the performance tests are provided below. Two configurations were tested with this schema:
- A single-server configuration with ClearQuest and the Request Manager deployed
- Similar to the single-server configuration, however with two additional servers added (with the Request Manager deployed on each one)
Figure 1 following shows computer A with a suggested initial hardware configuration to start your performance tests. Other B servers are added to provide additional ClearQuest Server capacity until ClearQuest Web Server causes Server A's CPU or memory to become constrained. When this occurs, an additional ClearQuest Web Server could be added using a load-balancer. For instance, Apache provides a software load-balancer that could be configured as a proxy web server to load-balance requests to more than one ClearQuest Web server. This configuration, however, has not been tested yet, and it requires a significant amount of manual configuration steps that have not been documented.
Figure 1. Example single-server A deployment extended with additional ClearQuest Server B
The first test was a 16-hour ClearQuest Web application load performance test using an example customer schema. The hardware configuration used was the same as the above single-server A deployment without Computer B. The hardware consisted of dual Intel 2.4 Gigahertz Xeon processors and 2 gigabytes of memory. There were 50 concurrent users for a total of 750 transactions per hour. The transactions executed during the load tests consisted of 70% database reads and 30% database writes. The transaction types were:
- Login and Logout
- Create, modify, and view records.
- Create and run queries
Performance tests showed that approximately 7 megabytes of memory per user was required for the ClearQuest Request Manager Server. This value varied dramatically depending on the schema tested, as well as the quantity and types of transactions. The Request Manager server was not CPU bound. The ClearQuest Request Manager steady-state memory usage without any users was approximately 60 megabytes. The Request Manager memory usage is the hardware resource constraint for the ClearQuest Web application. Average response times for this customer example schema were less than 6 seconds per transaction.
The second test was also a 16-hour ClearQuest Web application load performance test using an example customer schema, but the deployment configuration was a single-server A with two additional servers, Request Manager Servers B and C. There were 100 concurrent users, each performing 15 transactions per hour. One server had the ClearQuest Web Application deployed with a Request Manager similar to Computer A in Figure 1. Two additional servers each had a Request Manager deployed. The single ClearQuest Web application was able to support these three Request Managers while maintaining an average response time of less than 6 seconds per transaction. Each server in this test had dual Intel 2.4 Gigahertz Xeon processors with 2 gigabytes of memory. The transactions executed during the load tests consisted of 70% database reads and 30% database writes. The transaction types were:
- Login and logout
- Create, modify, and view records.
- Create and run queries
Table 7. Hardware Requirements Using an Example Customer Schema
| Number Concurrent Users | Process | Memory | Number CPUs | Number of Transactions |
| 50 | Tomcat | 60 Megabytes | 2 | 750/hr |
| 100 | Tomcat | 80 Megabytes | 2 shared | 1500/hr |
Depending on the nature of your environment, you can use a combination of the monitoring, tuning, and sizing methods described in this paper not only to optimize the performance of IBM Rational ClearQuest Web, but also other web-based applications and web server processes
This website is published "as is". There is no warranty of any kind (express or implied) as to the operation of our site, the accuracy of the information or the services or products referred to on it. All warranties are excluded as far as permitted at law. Neither we nor any third party will be liable for any losses or damage that may result from use of the website or as a consequence of any inaccuracies in, or any omissions from, the information which it contains.
Great content, reminds me of another article I saw: Transaction Monitoring
ReplyDelete