Jon on Performance

CYA

 

As one moves toward Application Modernization and/or Service Oriented Architectures (SOA) the service-aware software professional needs to remember CYA: Cover Your Aggregation. The service your application provides is directly dependent on many factors, not the least of which is the service received from other services.  It can be said that the service that you’re providing is based on the aggregation of the services that you call.  If you pay attention to that, you can save yourself lots of aggravation.

Greater Than The Sum Of The Parts

Modernization and SOA tend to break up an application into lots of parts.  These parts may be spread around the Enterprise, out to external partners, or around the Internet. The good news is that interfacing to these services may be quite easy.  Enablers like SOAP, CGI, scripting languages, WSDL and other interface definitions, etc. hide complexity and layers of software and communications. They may be so easy to use that the designer and developer forget that each instance contributes to the service time and overhead of an application.

Trust, But Verify

 Application modernization and SOA is becoming a fact of life in enterprises.  There is no turning back.  To remain competitive one has to embrace the agility and functionality of these techniques. Yet, those of us in the high-availability, high-throughput, high-visibility transaction arena recognize that quality of service must remain our mantra. To that end, we must be able to understand and quantify the contribution that called services make to the service we’re providing. We need to measure the service we’re receiving, and confirm that those external services are meeting their SLA commitment to us.

Clock In, Clock Out

This is something the payments industry has always been on top of.  With money at stake, and with lots of external partners helping process transactions, it is mandatory that a well-run ATM/POS system or switch quantify the service it’s receiving. Timers have always been part of the application anyway, since there are strict rules about when transactions time out. A well-designed application records those timers as part of each transaction’s record.  Usually several times are recorded:

• Overall time, from when the transaction is received to when it completes processing and is returned to the originator;
• Authorizer time, from when the transaction is sent to the back end for processing and when it returns;
• Encryption time, if required;
• Queue time, if the application has to queue the transaction;
• Database time, if the application uses a DBMS.

Time The Enterprise

SOA environments will have lots of services, and also major DBMS systems.  While I’m sure that these are well run in your organization, the nature of IT guarantees that there will be periods of peak demand which may, repeat may, result in slower response.  As a service provider and a customer of other services you owe it to yourself to understand and measure everything you are depending upon.  In SOA, this could easily mean lots of services: Authentication, CRM, app-specific services like order creation, inventory management, finance, credit, shipping, etc.  Measure each of them.

Now That You’ve Got It, Review It

Collecting data without using it is a meaningless exercise.  You have two choices: Use the data for “post-mortem” analysis, which is useful but a waste of potential. Or, use the data to anticipate problems and discover worrisome trends and tune the services so that no one suffers.

Canary In The Coal Mine

Like the canary in the coal mine, the service times you’ve collected can help predict a problem before it hurts your transaction, or before your boss calls you. But it requires that you look at the data periodically. I’ve found that most of the time the Heinrich Ratio holds true: There are always a set of events which are precursors to major problems or outages.  If anyone had been looking they could have clearly seen the oncoming crisis.  So look!

Use Response Distributions

Figure 1 shows you a very effective technique for presenting response times, and what they may show.  It shows an 18-month history, by day.  This requires reporting only one summary row each day, and charting it.  The key here is that the data shows both the transactions by day, and the response distribution for that day.  NOT the average response time.  The response distribution in this case is:

• Red line: Percentage (right scale) of transactions completing in less than 2 seconds;
• Green line: between 2 and 5 seconds;
• Blue line: between 5 and 10 seconds;
• Purple line: Over 10 seconds.

So this chart shows that something changed in early January.  The 2 second line dropped from over 95% to less than 95%, and the 5 second line jumped to almost 10%.  Why?  What has changed in the environment?  Is it our system, or is it a service or back-end that we are calling?

The Buckets

When you write the report program to summarize the response times, I suggest that you select your buckets for the response distribution as follows:

• Ideal response
• Good enough response
• On the verge of broken
• Broken

As you plot this data each month, you will want to focus on the latter categories.  Are they changing?  Why are they changing?  Is it a steadily worsening situation?  Will it eventually start to break transactions?  And for the Enterprise SOA service, “Who else is affected?”

The Detail

Figure 2 shows the service or back-end level response detail.  This is the second level of plot you will need to do of the data which you’ve been collecting, if you’re tracing a problem.  This shows you by time of day what kind of response you are receiving from the services you are calling. This particular chart shows an extreme problem, since the mythical “bank1” is giving us terrible response every weekday around 9 a.m.

In an ideal system, a system with zero bottlenecks and zero queuing, every line on this chart would be flat.  Ideal systems don’t exist.  To state it another way, there is always the “next bottleneck” in a system.  You may simply have not hit it yet.

CYA: Covering Your Aggregation Will Save Aggravation

A properly-designed application, mindful of the services that it depends upon, can provide sufficient information for its management to predict problems before they become expensive.  Doing it right will CYA.

Opening the Window

Too Much Data!

One of the challenges when working with Windows systems is that there are LOTS of objects and counters.  Generally they are poorly documented.  If you look at the trade press, the articles that are written about Windows performance analysis are usually sketchy, designed for the novice.

Over the years we have taken the two-pronged approach: 1) Collect more data than you need, and 2) Cross-correlate that data to confirm that the counters are meaningful.  This approach has worked well for us across many architectures.

Balanced? Not!

This recent Windows project has shown us how things can get out of hand.  Here are some of our discoveries when looking at these systems.

Identical Boxes, Identical Software, Vastly Different Load

The first thing that jumped out of the data was a memory issue.  Nine application servers at each of two sites should have been performing identically.  However, the memory statistics showed that three of the application servers in the north had a severe memory problem.  The stats said that these servers had committed memory over 150% of real memory.  That in itself is not necessarily a problem.  However, these same servers were experiencing page-out rates from 300 to 900 per second.  The page-outs were happening continuously, with highest rates during the peak transaction times. That is a problem!

The other servers in the group had commited memory less than 100% of real, and had page-out rates peaking at much less than our three problem servers.

A little further investigation showed that a disk utility which was running on all systems had for some reason grown out of whack.  On all of the other systems, it was using 4MB of main memory.  On the three servers in questions it was using 2GB!

Thrashing

Those levels of page-outs have a severe impact on the response of the system.  When a transaction came in to the complex, it became the luck of the draw whether the transaction hit a good server or a bad one.  I can guarantee you that if it hit one of the challenged servers, it would take quite a bit longer to process.  The application would be competing with the disk utility for memory for programs and data, and this would take time. Since this is an interactive application with real people waiting for the web response, the consequences are real. 

Solution?

The client chose to reboot these servers and the broken service cleared itself up.  Since we work with this client on an "arms-length" basis, i.e no access to the systems, we never found out the exact cause of the problem, whether it was a bad control file or operator screwup.  The task for everyone concerned with these systems is to make sure it doesn't happen again.

We've just begun a new project: 26 Windows systems, encompassing RSA, application, and Web servers. Client wants to know if they have the oomph to handle the anticipated traffic for the next 2 years.

We find each project like this interesting, not only from the technical point of view, but also from what they tell us about the state of the art of system mamangement.

For example: The systems are supposed to be balanced. They're not! There's a 3X difference between the web traffic being handled by the highest versus the least-loaded.

More later...

Performance is Dead?

A recent article declared that we have “pretty much left behind the primary issue ... performance. Our systems have become blazingly fast...” That’s really good to know.  All of us concerned with performance can either retire, take the week off, or spend all our time surfing.  I’m sure our vendors would be happy to stop doing R&D to make faster chips, disks, and interconnects.  Whew!  Nirvana has been reached.

Or Maybe Not

Or maybe not. Of course, if you have infinitely deep pockets, you can always over-buy.  Money will solve any performance problem, if you know where to apply said cash.  But you know, times aren’t as flush as they once were.  IT budgets everywhere are constrained.  Dollars must be applied judiciously to solve or prevent IT problems.

The Basic IT Laws For Performance Analysts

Everyone knows Moore’s Law.  There are other laws and corollaries which provide a more realistic view of what we performance types have to deal with.  Here’s what quick search of Wikipedia  comes up with:

• The Great Moore’s Law Compensator: TGMLC, generally referred to as bloat, is the principle that successive generations of computer software acquire enough bloat to offset the performance gains predicted by Moore's Law.
• Hofstadter's Law: It always takes longer than you expect, even when you take into account Hofstadter's Law.
• Parkinson's Law: Work expands so as to fill the time available for its completion.
• Parkinson's Law Corollary: Data expands to fill the space available for storage.
• Parkinson's Second Law: Expenditures rise to meet income.
• Wirth's Law: Software is getting slower more rapidly than hardware becomes faster.
• Gate's Law (or Page’s Law): The speed of commercial software generally slows by fifty percent every 18 months thereby negating all the benefits of Moore's Law.
• Thatcher's Law: The unexpected happens. You had better prepare for it.
• The Jevons Paradox: Technological progress that increases the efficiency with which a resource is used tends to increase (rather than decrease) the rate of consumption of that resource.
• Amara's Law: We tend to overestimate the effect of a technology in the short run and underestimate the effect in the long run.
• Finagle's Law of Dynamic Negatives: If something can go wrong, it will go wrong, and at the worst possible moment.

Laws, Not Jokes

The rules above are amusing, I agree. However, the facts are there to confirm the validity of these rules.  We all know them: Software developers now write applications using languages that are not compiled.  Entire application systems are written using scripted languages. Java and scripted languages execute much more slowly than compiled languages.   Databases are used instead of flat files.

The industry has made a choice that hardware is inexpensive, and that software development is very expensive.  So we see applications being built that require dozens of frames, hundreds of processor cores, terabytes of main memory, and multiple terabytes of disk to implement.

On the other hand, it probably wouldn’t be possible to create some modern applications without the powerful tools found in the “Web 2.0” environment.

Virtual Rant

The loss of control implicit in the new software paradigm drives me up the wall.  In the old days, we performance types would obsess over a new operating system release, testing it to see if it added to the inherent overhead. We would have serious discussions with the application developers if their latest update used more resource per transaction. We’d compare clock speeds and channel bandwidths.

Now we despair over the loss of control and seeming indifference to overhead.  For example, I certainly understand the need for virtualization.  However, virtualization always comes at a price.  A hypervisor will always add to the overhead.  And when I hear of an environment where a system running Windows is the host OS, running a hypervisor, running daughter OS’s I just have to shake my head.

When my application’s performance is totally dependent on the implementation of a scripted language by a vendor or an open-source community and I can’t see under the covers to understand locks and resource consumption I worry.  I worry about “gotchas” like Java’s “stop-the-world” garbage collection and how it will affect my 100 TPS transaction stream.

When the disks for my mission-critical OLTP system are sharing the communications fabric, processors, cache, and perhaps drives of a SAN with major batch application systems, I worry.

Holistic Approach

In the new world of software and performance analysis we must take a new approach. We performance types have to become experts at “black box” analysis. We have to look at the entire environment we are given, and take a step back from the nanosecond-style analysis that we are used to doing.  There are simply too many factors outside our control. Our job becomes one where we must measure a complex environment, try to correlate usage to the business demand, and project what’s going to happen when that demand increases.

We also have to understand how outside-the-box competing systems and environments affect our critical system. And, we have to understand where there are constrains on scalability.

Nope, Performance Analysis Isn't Dead

But yes, it has changed significantly.

www.banbottlenecks.com

 

OS Latency

While traveling last week and presenting at a couple of regional conferences, I was asked about how to find a millisecond-level problem.  The questioner worked for an exchange, and was frustrated that most of his transactions were properly processed in the 3-5 ms. range, but occasionally some took longer.  He asked how he could use his tools to try to find what's causing the longer transactions.

We chatted a bit further and I admitted that I didn't have a solution for him.

Then I got to thinking about the issue.  He's using a fast modern processor, but an older operating system, designed for commercial transaction processing.  The OS is only recently updated to allow symmetric multiprocessing, and documentation isn't yet available on where the locks are or how deeply they're placed and their granularity.

The challenge is that every OS will occasionally turn out the lights on the application and proceed to do its own work.  That work can be substantial.  For example, periodically an OS will:

• Scan its memory page tables, looking for expired or modified pages.  It will destroy the expired pages, and flush out and perhaps destroy the modified pages.  With gigabyte memories, there are a lot of entries to check.
• Scan its disk cache pages, doing the same. 
• Scan other internal OS tables, doing cleanup.

And that's only if the application is stable, not starting processes, opening/closing files, paging, or doing disk I/O.  If the application is actively doing any of the above, the OS will be busier and will have greater latency.

What do I mean by OS latency?  It can be defined as the time between the instant that an application's requested data is available for processing and the instant that it actually is assigned a processor and allowed to run.

Let's say that I'm waiting on a disk I/O.  That I/O completes, causing an interrupt indicating that the disk block is in memory.  The event flags will be set indicating that the request is complete.  How long between that moment and the moment that the process can start processing the data is the latency.

A lot can happen between those events as mentioned above.  And at different levels.  The OS may continue processing the requests in the cache or queue for that disk.  If the disk is backed up, there may be significant processing to properly order and schedule the requests in the queue.  There may be other interrupts from other disks or comm devices which need processing. Then, if the interrupt processing is complete, process scheduling has to sort out which processes are ready to run, and allow them to proceed.  In the OS under discussion, disk and comm handlers are individual processes, typically running at a high priority, above the application.  They have to do their work before the application gets time.

All this will add up.  I wouldn't be surprised that the maximum latency delivered by this OS, even on a lightly-loaded system, could be on the order of 10 ms. or more. I don't think that OS designers publish those figures for commercial OS's.

Let's also keep in mind that transaction queuing is a factor.  Unless you have a single source for the transaction stream, presenting them one-by-one, you have the probability that multiple transactions will arrive at the same instant.  Don't forget that you're at the mercy of the arrival rate of those transactions. A 100 TPS average rate over a minute can mean that you had 6,000 transactions during that minute, arriving at even intervals, or that you had 6,000 transactions arriving in one second and nothing for the other 59 seconds.

The truth is somewhere in between, of course.  But if your application design and capacity planning are for evenly-spread transactions, expect that some transactions will occasionally be slow.

My recommendation is that you be generous in time-stamping the transaction at each stage as it goes through its path.  When the transaction has finished, record its information to disk and look at the data periodically to see what slowdowns are happening.  Some OS's will report on time-in-queue for messages between processes, and this should be recorded and analyzed.

Another possible way of discovering OS latency is by using performance tools, if their data-gathering granularity goes down to the resolution that you're looking for.  Let's say that you set simple data collection for CPU on a ten-second interval.  What you will probably see is that successive samples will vary from the clock by milliseconds, or more.  This can be taken as evidence of latency.  Of course, such sampling is affected by process priority and everything else happening on the box.

I was chatting with friend about this issue.  He had experience with another exchange regarding a similar problem.  His conversation with me was quite proactive, saying that if they could reproduce the problem (there's the rub!) there was a probability that the vendor engineers could find the cause.  In his particular experience, the exchange was able to reproduce the problem, and he was able to trace the cause to a piece of the IP stack that probably dated back to the days of 300 baud modems.  The buffering at this level of the stack was totally insufficient, causing queuing and dropped packets.  His team fixed the stack and the client was gratified.

On the other hand, when you're working with transactions that require such stringent response requirements, you probably need to find a very simple, low-overhead OS.  I'm reminded that Data General years ago (remember, I'm a greybeard) had an RTOS (Real Time OS) with a guaranteed maximum latency. In today's world there are OS's designed for embedding into devices for real-time (sub millisecond) response.

Hofstadter's Law: It always takes longer than you expect, even when you take into account Hofstadter's Law.

 

Hanging by a Thread

With over 40 years in the industry (yup, I’m one of those greybeards) and 20 years after forming my company Transaction Design, Inc. I have some tales to tell and some wisdom to share (you can be the judge of that).  I welcome feedback and questions. Reflecting what many of you are experiencing in your professional lives, my consulting practice covers multiple architectures, including NonStop, Windows, VOS, and Unix. My job, and I’m good at it, is helping clients anticipate and prevent slowdowns and outages.  I work with clients where there are literally thousands of dollars of transactions each second at stake.

The evolution of application systems demands an awareness of processing threads.  Using collections of computers forces the application designer to think about parallelism. The floor is littered with large, monolithic applications that, when ported to the new world, didn’t take advantage of the parallel nature of the architecture and failed to perform. Those apps had to be broken into smaller pieces that could be replicated and distributed across the resources available.  Once that was done there was much better hope that the application would be well matched to the architecture and perform well.

With multiple cores in each processor blade, the necessity of designing for parallelism becomes even more important.  Within each blade one finds symmetrical (shared memory) multiprocessing, as well as the traditional asymmetric multiprocessing across blades or boxes.

Even with well-designed applications there are occasionally situations where avoidable bottlenecks occur. The basic question is: “How many instances of process X or connection Y do I need to support the traffic I’m expecting?”

I like to use an analog to the highway system.  The difference between performance and capacity:

Performance is designing a highway, curvature, elevation, grade separation, signage, etc. so that cars can drive it at 60 mph.

Capacity is designing that same highway so cars can drive it at 60 mph during rush hour.

Obviously, you need more lanes to handle the traffic.  So, too, do you need more server processes, logical connections, etc. to handle the expected workload.  But how many?

The issue is not strictly throughput.  Throughput is a factor, of course.  And you must design for the anticipated peak demand, not the average demand.  As a friend said, “Never design a bridge for the average height of a sailboat’s mast.  That guarantees problems.”

When you know the throughput target, the real issue is concurrency.  At 100 tps, how many transactions will be “in flight” within the system at any given moment, and does the system have the resources to handle them?  The in-flight (concurrent) number is the product of the throughput and the response time.  So, if your target is 100 tps, and the average response is 0.5 seconds, you can expect (and must design for) 50 concurrent transactions.

But wait! I said never design for the average.  If you are going to design properly, design for the worst case response, so that if your disks or back-end authorizer slow down, you will still be able to process the transactions, just more slowly. If your experience shows that, for example, 4 seconds is still a reasonable response, and you’ve seen that from the back-end at busy times, then you need to design for 400 concurrent transactions.

To some of you this may be an obvious discussion, but I can assure you that this is a real issue at some sites.  We’re successfully completing a project with a Southern bank where the bottleneck was the capacity, both bandwidth and concurrent LU2 sessions, of an SNA link to their host.

Remember legacy comm, like SNA or X.25?  I realized that there was a generation gap when I was talking to a younger gentleman at a conference and he asked “What’s SNA?”

Designing for concurrency is a mandatory part of configuring the system.  Making sure that there are enough server processes and connections to handle the anticipated peak is absolutely key.  (For more about this issue, refer to my article “At The Mercy Of The Arrival Rate” http://www.banbottlenecks.com/mercy.html .)

One more, frequently-overlooked issue re parallelism: Batch!

Unfortunately, batch programs tend to be monolithic.  They tend to be single-process program sets which do heavy data manipulation in a sequential manner.  As such, they don’t take full advantage of multiprocessor architectures. Sometimes, they are in danger of breaking their “window,” i.e. running longer than they should and missing deadlines for completion.

Take the time to examine the batch stream and redesign portions of it so that the inherent concurrency can be used to advantage.  Look for programs in the batch sequence that can be run at the same time as other steps. 

If you’re using indexed files, or SQL, examine the batch process to see if it can be broken apart and executed in parallel based on a key range.  We’ve seen dramatic improvements in execution time with some judicious redesign of the basic batch stream.

A little bit of time looking at these issues now and making adjustments can save a lot of grief later.

Grading ourselves on a quiet holiday: A-

We’ve completed our monthly web conferences with our clients, following our January reports which cover December.  This is always the most interesting month of the year for us, because it’s the time we grade ourselves.  Most of our clients process point of sale transactions, so this is the time of the year with their highest volume. We grade ourselves on how well we’ve been able to anticipate and prevent client processing problems.

Each month we grade our clients’ systems, from A to F, on a dozen factors.  An “A” from us means “Adequate capacity.”  “B” means “Basically adequate.” Grade C means “Cause for concern,” or, “You had better have someone take a look at this problem because we think it’s going to hurt processing if it’s not fixed.”  D is “Deficient,” F is “Failed, too late!”

This year I would give us an “A-“.  Not because anyone failed or had processing problems that we could have foreseen. We deserve an A- because  we didn’t push one of our clients hard enough.  We didn’t ask enough questions, and cause a large enough stink.

They are a large client, lots of systems, processing point of sale. Unfortunately, they are also compartmentalized, and outsourced, and don’t communicate well enough among themselves. You know the pattern: Operations, sysadmin, capacity planning, and applications are all large departments.  We’ve been asking for years for some key information which would help us track and trend their application traffic statistics.  We’re just now beginning to get it.  We’ve offered our monthly web conference to them, but they haven’t attended in years. We know that they use the statistics we generate, and review our reports, and act on our warning emails and phone calls, but feedback is lacking.

When we are able to count and track the application statistics, we are able to correlate the system usage stats with the application traffic stats.  We are able to project those numbers, and answer the question “Will we make it through the next peak?”  Equally as important, we are able to look at the intra-day processing patterns, and the larger day/week/month/historical patterns, and answer the questions: “Does this make sense? Is usage proportional to the demand on the system?  Why is CPU or Disk or Comm activity at X per transaction at this time, and Y per transaction 3 hours later?”

If we had been able to do this for this particular client, we would have been able to detect well in advance that a user had been doing large DB restores at bad times during the day. S/he continued doing these during Christmas week, and the online application suffered.

No, we don’t monitor their systems in real time.  No, we aren’t involved with their operations policies, nor their security policy.  Yes, they hold some blame here too, perhaps the lion’s share.

But, as consultants, we should be pressing harder for the client to engage us in the dialogue that we offer them.  We should press harder to acquire the best information possible on the system and application.  We should press harder to make the Ban Bottlenecks process work superbly.

“A-“

Christmas is coming...

Is this a “white knuckle” time of the year for your organization?  If you are in banking or retail, the holiday season is the peak time for point of sale transactions.  It is the time when the most money (transaction value) is flowing through your systems. For our clients it is literally tens of thousands of dollars per second. It is also the time when these systems and networks are most susceptible to capacity problems.

As a performance manager or capacity planner, you must know and plan for the holiday peak.  For some organizations, that may be only 40% to 50% higher than the non-holiday traffic.  For other organizations, it may be two or three times normal.  We had one client where their holiday peak was five times normal!

 So, “white knuckles” or “cool and calm?”  Are you nervous because so much is at stake if something breaks?  Are you “flying blind” and simply don’t know if you have enough capacity? Or are you confident that your systems are ready?

Successful capacity managers know that they need to plan for only one half-hour peak a year.  If they have configured, optimized, and balanced for that peak time, the rest of the year will take care of itself.

So let’s talk about how to prepare for the peak.

First, count the transactions going through the system.  All of them.  Not just when you think the peak half-hour happens each week or month.  Our clients have proved that the peak half-hour moves.  Sometimes it’s noon, sometimes it’s 5:30 p.m.  Sometimes it’s Saturday, sometimes Monday, sometimes Friday. Expect the unexpected.  Count continuously.

Why do we use a half-hour?  Our experience is that POS transactions arrive in a nice bell-shaped curve over the course of a day. Occasionally we’ll see a bell with two peaks depending on the characteristics of the business.  Our experience also shows that the peak minute’s traffic is usually within 10% of the half-hour average traffic.  A half-hour interval gives 48 counts in a day, and that’s convenient to work with.

Now you have the data you need to analyze and do projections. We look at multiple growth ratios (and one additional factor) to try to figure out what the future will bring. We look at:

· Average transaction day each month, month-to-month growth, and growth over last year;

· Year to date transaction growth such as January – October this year versus the same period last year;

· Peak transaction day each month and growth over last year;

· Peak transaction half-hour each month and growth over last year.

Yes, this gives us several numbers to use, none of which agree.  So then we have to bring in the “additional factor:” Knowledge of the business! Is growth happening because of a merger? What does the business partner say?  Are there more business changes coming?  Will there be a special promotion?  What does s/he think the season will bring?

After reviewing all of the above, pick the most probable growth number using an educated guess.

One of our clients had a severe problem one year because they didn’t talk to their business partner.  They added a gift card product, and didn’t adequately plan for how successful the card would be.  Everything was in place for the card, but it was so wildly successful that the day after Christmas, when people came into the stores to use the card, the communications link to the authorizer didn’t have enough bandwidth and the system had severe problems.

OK, so now you have a number for growth.  Apply it to last year’s peak half-hour to come up with this year’s target.  And do one more thing:  Take last year’s December over October ratio, and apply it to this October.  How do the two numbers compare?  Take the larger.

What do you do with the new target?

Over the last year we have been collecting the details of the system during each month’s peak transaction half-hour. We know CPU, disk, memory, communications, and process usage for each month. What’s more, we use these numbers to compute the usage per transaction (or transaction per second) each month.

From this we have:

· Basic numbers to use for modeling, and

· An indication of whether the application is getting more or less efficient.

We once had a client who added velocity file synchronization between two active systems.  They did unit testing, but when they installed in production we showed them that the number of disk I/Os per transaction had doubled. We worked with them to rebalance the system.

So, it is reasonable to expect that if the transaction counts are expected to increase by 50% over October’s, it is also reasonable that all the other usage factors will do likewise.  If October’s CPU was 40% busy, we should expect to see 60%.  If the busiest disk in October was 70% busy, it will be 105%!  Oops!  Not good.  Time to look more closely at that disk and offload or stripe it.  If your legacy communications (SNA or X.25) will go over 50% of the bandwidth, time to upgrade it.  And so on…

Click here for sample of our peak half-hour analysis for the HP NonStop and Stratus VOS architectures. These reports don't include the process detail.

If you use our techniques your pagers will stay quiet during the holidays.

 

On Service Levels

During one of our monthly web conferences with a client, the subject of service levels came up.  They were getting poor service from one of their back-ends, and we found the source of the problem and quantified it.

Before I talk about how we found it, let’s talk about computer service level management in general.

What does “service level” mean?  It seems there is no single definition.

For some, service level means availability.  For example, 99% uptime.  But there is a further distinction required: Are we talking about box (operating system or hardware) availability or are we talking about application availability?  One can have 100% availability for the box, but only 97% application availability if you have a 4-hour “planned outage” each week for database maintenance.

Payments systems and customer-facing systems require 100% application availability. 

If you’re writing a specification regarding service level management based on availability, you may also want to include limits on downtime incidents per period and mean time to recover or repair. Nothing is more frustrating than something that breaks frequently even though it may have a quick recovery.

Payments systems and customer-facing systems require two additional measurements of service level: transaction success and response time. Both are extremely important.

Response time is a traditional measure of service level: i.e., 98% of the transactions must finish within 2 seconds.  Note that this is a distribution, not an average.  An average could be stated as “the average response time must be less than 2 seconds.”  Averages are terrible specifications of response. For example, one could have three transactions taking 3 seconds each and three taking 1 second each. The average response in this case is 2 seconds.  But this scenario wouldn’t meet the 98% criteria, since its distribution (3 at 3 secs and 3 at 1 sec) would be only 50%.

Transaction success rates should also be tracked. In the real world of multi-stage transaction paths, it’s possible to interrupt a transaction somewhere in the path, deny it, and return it within the required response time.  A denial or reversal reason code is always useful.

Happily, in the payments industry the standard packages* capture the data we need in their logs.  From these logs we can determine which institution or site sent the transaction, what response and success we gave them, to whom we sent the transaction, and what kind of response and success we’re getting from them.

*We have log analyzers for BASE24, eFunds Connix (both on Tandem – HP NonStop), ON/2 (Stratus VOS), and Open/2 (Windows, Unix). Other providers are welcome to talk to us.

We report and chart counts, averages, and distributions for the transaction stream as a whole, and for each source and destination (authorizer) of a transaction.  When we do response distributions, we categorize each transaction into one of several “buckets:” ideal response, good-enough, heading for trouble, and definitely broken response.  We watch for the outliers in the distribution.  They provide the early warning that something’s going wrong. If they’re there, we want to know why they’re there, and we will drill down to see when they occurred and who or what caused them.

Take a look at this page of example charts to see the progression of a problem diagnosis.

The bottom line: We saw the outliers increasing in our client’s transaction response. We were able to identify that it was an external problem, caused by increasingly slow or inconsistent response from one of their back-ends.  We were able to identify the back-end, and provide a history of the problem going back several months.  And we were able to identify the times of day that the back-end was having the problem.

If only I can get that back-end switch to hire us to help them!

On Continuous Improvement

Have you heard of the Heinrich Ratio, also known as the Iceberg Model?  It was originally postulated for the airline industry.  It says that for every 600 incidents, there will be 30 serious incidents, and 1 catastrophic incident. (The numbers vary by commentator.) To put it another way, for every disaster, there were 30 reported incidents which may have lead to the disaster, and 600 unreported incidents.

I firmly believe that the Heinrich Ratio applies to computer technology as well. Most system outages and slowdowns need not happen. They can be predicted.  Predicting them doesn't require rocket science, just due diligence.

In the world in which I work, that of customer-facing or money-handling systems (mission-critical, an overused term), an outage of even a minute or two can be extremely expensive.  If you're running a point of sale system, ATM network, or a financial switch, 100 transactions per second, with an average value of $10, means $1,000 per second of business through the system.  A one-minute system disruption means lost business of $60,000.

It may also mean a lost customer.  If I try to use a credit card and it gets rejected and I know that my credit is good, I may think twice about using that card again. Too many rejects and I'll drop the card.

How many times have you tried to use a website for ordering and were frustrated because it was too slow?  I recently just walked away from a site and chose a competing product because I wasn't able to get an order placed properly.

Given this, it makes a lot of sense to go attack the "Iceberg."

Yes, I know, all of your production software is perfect, no bugs or issues.  However, I suspect that it has some "unique features" that occasionally raise their heads.  And of course your operations staff never makes a mistake.  That's as it should be.

When we work with a client we search for incidents that could become the base of the iceberg.  We look for the intermittent problem which indicates a "unique feature," or for the ill-advised operator action that could lead to trouble in the future. Things like backups being run at the wrong time; a process that goes into a loop for a minute or two, but behaves normally most of the time; batch jobs competing with the on-line system during peak times; communications links that drop and resume unexpectedly.

Each month we talk to the client about what we find, and recommend measures for avoidance.

If we can minimize the base of the iceberg, reducing the base of potential mistakes, we will reduce the risk that a catastrophic outage will occur. We'll save money, and keep our customers happy.