Performance Testing – Analyzing the AUT, Part 2

In my previous post, I talked about the need to consider more than just one of the common objectives in our performance test scenarios, such as hits/pages per second, transactions per second, specific transactions (such as Login) and peak concurrent users. The key point I wanted to make was that focusing on only one of any of these will almost certainly result in erroneous results, and there is nothing more damaging to the credibility of the performance testing professional than having to defend invalid test results.

Today I want to talk about concurrent users in more detail, as this is one of the more common test objectives and the subject of much debate within the performance testing community. The problem I have isn’t the goal itself, rather what the goal means and how to ensure that we’re using not only the right number of concurrent users, but under the right circumstances. The same applies whether you are running Peak, Average, Stress, or any other type of test whose definitions I will not get into here; there is already plenty of discussions on that topic.

So you want to test 100 concurrent users, but doing what?

The question we should be asking our business is not how many concurrent users they want to run, but how do those users behave? This question will have multiple factors, such as time between steps or requests or between iterations of a session. First we need to ask:

  • Are all transactions performed equally, or is there a weighting to each transaction?
  • Are all sessions roughly the same, or should there be a significant amount of randomization in their behavior?
  • Is there a compelling reason to include other transactions in the mix other than the “top ten”?

Scott Barber, of PerfTestPlus Inc., has a handy way of dealing with these that he outlines over at his blog (deep linking not allowed, just hit enter on the URL again), and its hard to shake the mnemonic once you’ve read it. Go ahead and read that, as well as the other 2 posts he has which are linked from that page. I’ll wait here…

You back? Great.

Once again, I don’t want to rehash something that has already been stated so eloquently. So instead I will offer my own experiences on the subject. First, don’t just look at the goal of X number of concurrent users and nothing else. That should be obvious to anyone who has ever run a performance test, but in case it isn’t I’ll explain why. Take an example where a script has 10 steps, executed in the same order with the same delay between steps, and once the script is finished it starts again from the beginning. That script might look something like this when executed with other scripts during a test.

Everyone in a straight line!

(In all my Word-art glory!)

Now lets look at the same basic script, but with a randomized ordering to 8 of the 10 steps (since Login and Logout have to happen at specific places), and with randomized time between steps, with a random delay between script iterations.

Anarchy, I tell you.

Guess which one is likely to be more realistic? (ethay econdsay iptscray <– pig latin answer key)

These two scenarios will have very different results, but only the second one will be realistic.

The information we tend to get from analytic tools such as Google Analytics or WebTrends show the average user session and not the standard deviation of those sessions’ duration and pacing. Also, a good tool should show us not only the top transactions or pages but also the percentage or weighting of those, so we don’t have to assume all users behave the same way. Of course there will be exceptions, such as a customer registration process which spans multiple pages in a specific sequence. These types of transactions can and should be grouped together in a single “business process” that is somehow represented in our scripts, and business processes treated as separate transactions to be randomized accordingly. I find the best way to get accurate information on exactly what users are doing in an application is by taking a sample of production logs and analyzing them myself, but there are a few tools out there capable of helping you with the work.

That’s it for now. Remember, keep it real and you will be able to stand by your test results with confidence!

Shane Evans

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: