Hi,
I really hope someone can help.
I have a 28 processes (associated with an application) that sometimes run fast (with a runtime of 30 minutes) and other times slow (up to 3 hours).
On nights that the application runs fast (30 minutes) the 28 processes are consuming less CPU (around 30% per process - and around 50% total for the server) compared to nights where it runs long (2 hours) (around 50% per process - and around 70% total for the server). The problem is that these are cobol programs and I do not have symbol files for them, and thus I cannot use Process Explorer (or something similiar) to inspect the threads to determine what is causing the use of additional cycles.
I have compared a couple of traces of a night when processing was fast to a night when processing was slow. The only difference that I could see between the traces are the CPU utilization of the 28 processes in question (the rest of the processes utilize the same amount of resource and it is the same amount of processes as well).
I have had a look at the following already (and could not determine the root cause):
1. Application server performance - every metric I used is in spec on a fast and a slow night - the only difference was the CPU utilization of the 28 processes in question
2. Database server performance - every metric I used is in spec on a fast and a slow night
3. Network performance - every metric I used is in spec on a fast and a slow night
4. SAN performance - every metric I used is in spec on a fast and a slow night
No errors are logged anywhere.
I have spent about two weeks trying to get to the bottom of this, but no luck yet. Please let me know if I need to post additional information (I have a lot, I am just uncertain what would be relevant on this forum and what not).
Tha application developers checked and confirmed that their application should be performing the same calculations everynight (on a similair amount of records - maybe a 1% variance).
How can I go about determining the root cause ?
The only things I can still think of is that the application is actualy performing more work on some evenings, but I do I verify this.
Thanks,
Scherrit