main

• | Table of Contents
• | Index
• | Reviews
• | Reader Reviews
• | Errata
System Performance Tuning, Second Edition
By Mike Loukides, Gian-Paolo D. Musumeci
Publisher | : O'Reilly
Pub Date | : February 2002
ISBN | : 0-596-00284-X
Pages | : 336
Slots | : 1

System Performance Tuning covers two distinct areas: performance tuning, or the art of increasing performance for a specific application, and capacity planning, or deciding what hardware best fulfills a given role. Underpinning both subjects is the science of computer architecture. This book focuses on the operating system, the underlying hardware, and their interactions. For system administrators who want a hands-on introduction to system performance, this is the book to recommend.


Table of Contents

• | Table of Contents
• | Index
• | Reviews
• | Reader Reviews
• | Errata
System Performance Tuning, Second Edition
By Mike Loukides, Gian-Paolo D. Musumeci
Publisher | : O'Reilly
Pub Date | : February 2002
ISBN | : 0-596-00284-X
Pages | : 336
Slots | : 1

Copyright
Preface
 | Who Should Buy This Book?
 | A Note on Coverage
 | How to Read This Book
 | Organization
 | Typographic Conventions
 | Comments and Questions
 | Personal Comments and Acknowledgments
Chapter 1. An Introduction to Performance Tuning
 | Section 1.1. An Introduction to Computer Architecture
 | Section 1.2. Principles of Performance Tuning
 | Section 1.3. Static Performance Tuning
 | Section 1.4. Concluding Thoughts
Chapter 2. Workflow Management
 | Section 2.1. Workflow Characterization
 | Section 2.2. Workload Control
 | Section 2.3. Benchmarking
 | Section 2.4. Concluding Thoughts
Chapter 3. Processors
 | Section 3.1. Microprocessor Architecture
 | Section 3.2. Caching
 | Section 3.3. Process Scheduling
 | Section 3.4. Multiprocessing
 | Section 3.5. Peripheral Interconnects
 | Section 3.6. Processor Performance Tools
 | Section 3.7. Concluding Thoughts
Chapter 4. Memory
 | Section 4.1. Implementations of Physical Memory
 | Section 4.2. Virtual Memory Architecture
 | Section 4.3. Paging and Swapping
 | Section 4.4. Consumers of Memory
 | Section 4.5. Tools for Memory Performance Analysis
 | Section 4.6. Concluding Thoughts
Chapter 5. Disks
 | Section 5.1. Disk Architecture
 | Section 5.2. Interfaces
 | Section 5.3. Common Performance Problems
 | Section 5.4. Filesystems
 | Section 5.5. Tools for Analysis
 | Section 5.6. Concluding Thoughts
Chapter 6. Disk Arrays
 | Section 6.1. Terminology
 | Section 6.2. RAID Levels
 | Section 6.3. Software Versus Hardware
 | Section 6.4. A Summary of Disk Array Design
 | Section 6.5. Software RAID Implementations
 | Section 6.6. RAID Recipes
 | Section 6.7. Concluding Thoughts
Chapter 7. Networks
 | Section 7.1. Network Principles
 | Section 7.2. Physical Media
 | Section 7.3. Network Interfaces
 | Section 7.4. Network Protocols
 | Section 7.5. NFS
 | Section 7.6. CIFS via Unix
 | Section 7.7. Concluding Thoughts
Chapter 8. Code Tuning
 | Section 8.1. The Two Critical Approaches
 | Section 8.2. Techniques for Code Analysis
 | Section 8.3. Optimization Patterns
 | Section 8.4. Interacting with Compilers
 | Section 8.5. Concluding Thoughts
Chapter 9. Instant Tuning
 | Section 9.1. Top Five Tuning Tips
 | Section 9.2. Instant Tuning Recipes
Colophon
Index


Free Open Study

Copyright © 2002 O'Reilly & Associates, Inc. All rights reserved.

Printed in the United States of America.

Published by O'Reilly & Associates, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.

O'Reilly & Associates books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://safari.oreilly.com). For more information contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com.

The O'Reilly logo is a registered trademark of O'Reilly & Associates, Inc. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and O'Reilly & Associates, Inc. was aware of a trademark claim, the designations have been printed in caps or initial caps.

While every precaution has been taken in the preparation of this book, the publisher assumes no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein.


Preface

This book is about the art of tuning systems for optimal application performance. Within these pages, I'll discuss how to perform the actual tuning itself and get the most out of your applications. I'll explain the underlying algorithms behind these tunable parameters, so you can make intelligent decisions based on your own environment. I'll touch upon capacity planning as well, which describes how to appropriately size systems for different uses.


Who Should Buy This Book?

This book is primarily for people who are interested in optimizing their computer systems for performance. Those faced with purchasing decisions will find a clear description of how various system components interact with each other, as well as pointers on what to watch for. People writing or tuning code will find a concise explanation of modern compiler options, as well as an appreciation for how the underlying operating system manages their runtime applications. Finally, those who are simply interested in learning more about how their computers work may find the explanations within these pages to be interesting.


A Note on Coverage

This book is heavily oriented to the Solaris operating environment (up to and including Solaris 8) and Linux. However, this book emphasizes Solaris. There are a handful of reasons for this, which I would like to elucidate:

  • The majority of datacenter environments run Solaris; these environments tend to be the ones that require the most effort towards performance tuning.

  • Solaris machines tend to be more focused on performance. I suspect this is because Sun systems are more expensive than their Linux counterparts, on average. As a result, people tend to be a lot more picky about performance, so more work has been done in that area on Solaris. If your Linux box doesn't perform well enough, you can just buy another one and split up the workload -- it's cheap. If your several-million-dollar Ultra Enterprise 10000 doesn't perform well and your company is losing nontrivial sums of money every minute because of it, you call Sun Service and start demanding answers.

  • Finally, the tools available for performance analysis on Solaris are much stronger than those available on Linux. Part of this is that it's not glamorous to implement performance analysis frameworks; part is that sometimes you need to effect hardware changes to get the best performance data; part is that Linux is a relatively new operating system, developed by relatively few, whereas Solaris (and its predecessor SunOS, which provided a lot of feedback into the Solaris development process) is a rather old operating system, and Sun is a huge company. The end result is that because our ability to analyze performance problems in depth and to make intelligent tuning decisions is largely based on the data provided by our performance data-gathering tools (otherwise, every time we tune, we're just taking a shot in the dark) it is extremely difficult to tune Linux systems. This goes directly against one major principle of performance tuning (see Section 1.2.1).

I don't mean to indicate any bias for or against any one operating system.[1] The simple truth is that serious performance tuning on Linux is a headache right now, because the tools to facilitate understanding just aren't available.

[1] In the interest of full disclosure, the author was employed by Sun Microsystems for about half the time it took to write this book.

If you are running Linux, you absolutely must download and install the sysstat package, which provides (very limited) versions of iostat, mpstat, and sar. While far from being as complete as the Solaris versions, they are at least a way to get some information. As of this writing, they can be downloaded from the author's web site (Sebastien Godard) at http://perso.wanadoo.fr/sebastien.godard/.


How to Read This Book

This book is part reference and part story. Readers with some background in performance tuning and analysis may be tempted to immediately flip to a specific section (perhaps one concerning where they suspect a problem to be) and start using the book as a reference. I strongly discourage that.

This Book as a Story

To get the most out of this book, read it first as a story. Habits, ways of thinking, and approaches to problems are some of the most important things you can gain from this text, and these are to be soaked up over the course of hours, not gained by memorizing items from a list.

Many of the topics we'll discuss here may appear quite simple and straightforward. The truth is that they go to the foundations of modern computer architecture -- why computers work at all -- and are often are deceivingly complex. Don't be alarmed if you have to reread parts of the book; it takes time to absorb some of this material.

This Book as a Reference

Once you've read the story and understand the overall picture of how the pieces of the performance puzzle fit together, then this book will prove to be a useful reference. If you haven't read the story, you might look at one of the charts (such as the description of kernel memory tunables in Section 4.2.5.1 in Chapter 4) and think that you've understood the whole picture. If you believe that, there's a lot of complex background that you have just missed out on. After you've read this book as a story, then you'll have an appreciation for the issues surrounding complex problems.


Organization

This book consists of nine chapters:

Chapter 1, sets the stage for the rest of the book through a discussion of fundamental principles.

Chapter 2, describes performance management through understanding and limiting system workloads, as well as describing some common benchmarks.

Chapter 3, briefly discusses modern processor architecture, caching, multiprocessor systems, and how the operating system schedules tasks.

Chapter 4, explains how computer memory systems are organized, and how that management interacts with other parts of the system.

Chapter 5, describes disks in the simplest state: how hard drives function and what limitations they have that can be worked around.

Chapter 6, is in some senses a continuation of Chapter 5; it is a discussion of the technique of connecting multiple disks into a single logical unit.

Chapter 7, addresses issues surrounding network performance, from the physical layer through communications protocols like TCP/IP and file sharing systems like Samba and NFS.

Chapter 8, is a brief introduction to how to write fast code, as well as a summary of modern compiler usage.

Chapter 9, is a summary of some of the key points discussed throughout the book, presented as a reference for solving problems and configuring systems quickly.


Typographic Conventions

The following conventions are used in this book:

Constant Width

Used for source code samples and for quotations from source code within the text, including variable and function names. Constant width is also used for output printed by a computer and for the contents of files.

Constant Bold

Used for commands typed verbatim by the user.

Italic

Used for command names, directory names, and filenames. It is also used for new terms when they are defined.

Bold

Used for vectors (in the mathematical sense) and for command options.


Comments and Questions

We have tested and verified all of the information in this book to the best of our ability, but you may find that features have changed, that typos have crept in, or that we have made a mistake. Please let us know about what you find, as well as your suggestions for future editions, by contacting:

O'Reilly & Associates, Inc.
1005 Gravenstein Highway North
Sebastopol, CA 95472
(800)998-9938 (in the U.S. or Canada)
(707)829-0515 (international/local)
(707)829-0104 (fax)

You can also send us messages electronically. To be put on the mailing list or request a catalog, send email to:

info@oreilly.com

To ask technical questions or comment on the book, send email to:

bookquestions@oreilly.com

We have a web site for the book, where we'll list examples, errata, and any plans for future editions. You can access this page at:

http://www.oreilly.com/catalog/spt2/

For more information about this book and others, see the O'Reilly web site:

http://www.oreilly.com


Personal Comments and Acknowledgments

I am indebted to many people for their help and assistance as I wrote this book.

First and foremost, I would like to thank my editor, Mike Loukides. This book slipped by more than a few deadlines, and his patience, wonderful advice, and upbeat attitude were absolutely vital to this book being finished. I also owe a huge debt to my management teams, both at the University of Illinois (particularly Mona Heath and Ed Krol) and at Sun Microsystems (Dr. Keng-Tai Ko and Miroslav Klivansky). Their support, patience, and understanding proved invaluable.

My friends were a constant source of support during this affair, both in providing technical advice and being understanding of the "No, I can't go out, I have to work on the book" phrase that suddenly took over my vocabulary for two years. I would like to especially thank Keith Wessel, Kris Wehner, Majdi Abbas, Deb Fligor, Jay Kreibich, Adrian Cockcroft, Richard McDougall, Elizabeth Purcell, Tayfun Kocaglu, Paul Strong, Phil Stracchino, Christof Harper, and Ken MacInnis (in no particular order). Dr. Sanjay Patel was kind enough to admit a student with none of the prerequisites into his computer architectures course at the University of Illinois; his encouragement is very much appreciated. Rose Platt, Nancy Weber, and Dr. Robert Cahn opened their home to me and, in doing so, provided me with a safe retreat to sit and write for long breaks. Kate Secor kept me sane as I wrote the last half of this book: she jokes that she was my "good writing totem" as whenever she was around, I got a lot more written than otherwise.

It's hard for me to overstate the contributions of my family to this book, particularly my parents, Drs. Diane and Antonino Musumeci. It's somehow very reassuring to look at the people who raised you and realize that they have both written books -- how hard can it be, right?[2] My brothers Domenico and Walter kept me grounded in the real world, my grandfather Walter reminded me that there was a time before computers existed, and the family dog Gizmo was a constant source of amusement (it's hard to keep focused when a 15-pound Border Terrier is tearing around the house as fast as he can, and sometimes you need to be not focused).

[2] The answer is: very hard.

I would like to dedicate this book to my grandmother, Kathleen Mikolaitis.

Acknowledgments from Mike Loukides

First and foremost, I want to thank Gian-Paolo D. Musumeci for doing what I could never have done: write the second edition of this book. It's really his work. I'm proud of the first edition, but I have to confess that it is primarily a piece of journalism. I had access to a number of system experts (Doug Gilmore, Chris Ryland, Tan Bronson, and the rest of the Multiflow diaspora, among others), and reported their expertise. (Many people aren't aware that the book started as part of the Multiflow TRACE/UNIX documentation set, although Multiflow went out of business before it could be published.) Gian-Paolo is the real thing, where I'm just a journalist: he brings a depth of knowledge and experience to the book that I never had. As proud as I am of the first edition, the second edition is a much better book.


Chapter 1. An Introduction to Performance Tuning

There are a thousand hacking at the branches of evil to one who is striking at the root.

—Henry David Thoreau, 1854

Going faster seems to be a central part of human development. As little as three hundred years ago, the fastest you could expect to go was a few tens of miles an hour, aboard a fast clipper ship with a stiff wind behind you. Now, we must expand our view to things such as the fastest achievable speed while remaining in the Earth's atmosphere -- perhaps fifteen hundred miles an hour, twice the speed of sound, if you are a civilian without access to the latest, fastest military aircraft. The journey that used to take three weeks under sail from London to New York City now takes us as little as two and a half hours, sipping champagne the whole way.

This innate human desire to go fast is expressed in many ways: microwave ovens let us cook dinner quickly, high-performance automobiles and motorcycles give us a wonderful thrill, email lets us communicate at almost the speed of thought. But what happens when that email server is overwhelmed, as when we all log in at eight o'clock in the morning to check what has gone on while we've slept? Or when the procurement system for the company that distributes microwave ovens is only able to handle half of the workload, or when a mechanical engineer's CAD system runs so slowly that the car engine she's designing won't be ready in time for the new model year?

These are the problems facing us in performance tuning: it is the adaptation of the speed of a computer system to the speed requirements imposed by the real world.

Sometimes these problems are subtle, sometimes not. We must carefully consider the changes in the system that caused it to become unacceptably slow. These influences might come from within, such as a heavier load placed on the system, or from without, such as a new operating system revision subtly changing -- or utterly replacing -- an algorithm for resource management that is critically important to our software. The solutions are sometimes quick (a matter of adjusting one "knob" slightly), or sometimes slow and painful (a task for weeks of analysis, consultations with vendors, and careful redesign of an infrastructure).

When I originally started work on this book, it seemed straightforward: take Mike Loukides's excellent but dated first edition and overhaul it for modern computer systems. As I progressed further along, I realized that this book is actually about much more than simply performance tuning. It really covers two distinct areas:

Performance tuning

The art of increasing performance for a specific application set (also known as "squeezing blood from a stone").

Capacity planning

Deciding what hardware to purchase to fulfill a given role (also known as "fortune telling").

These areas are all underpinned by the science of computer architecture. This book does not concentrate on application design; rather, it focuses on the operating system, the underlying hardware, and their interactions.

To most systems administrators, a computer is really a black box. This is perfectly reasonable for many tasks: after all, it's certainly not necessary to understand how the operating system manages free memory to configure and maintain a mail server. However, in performance tuning -- which is, at heart, very much about the underlying hardware and how it is abstracted -- truly understanding the behavior of the system involves a detailed knowledge of the inner workings of the machine. In this chapter, we'll briefly discuss some of the most important concepts of computer architecture, and then go into the fundamental principles of performance tuning.


1.1 An Introduction to Computer Architecture

A full discussion of computer architecture is far beyond the level of this text. Periodically, we'll go into architectural matters, in order to provide the conceptual underpinnings of the system under discussion. However, if this sort of thing interests you, there are a great many excellent texts on the topic. Perhaps the most commonly used are two textbooks by John Hennessy and David Patterson: they are titled Computer Organization and Design: The Hardware/Software Interface and Computer Architecture: A Quantitative Approach (both published by Morgan Kaufmann).

In this section, we'll focus on the two most important general concepts of architecture: the general means by which we approach a problem (the levels of transformation), and the essential model around which computers are designed.

1.1.1 Levels of Transformation

When we approach a problem, we must reduce it to something that a computer can understand and work with: this might be anything from a set of logic gates, solving the fundamental problem of "How do we build a general-purpose computing machine?" to a few million bits worth of binary code. As we proceed through these logical steps, we transform the problem into a "simpler" one (at least from the computer's point of view). These steps are the levels of transformation.

1.1.1.1 Software: algorithms and languages

When faced with a problem where we think a computer will be of assistance, we first develop an algorithm for completing the task in question. Algorithms are, very simply, a repetitive set of instructions for performing a particular task -- for example, a clerk inspecting and routing incoming mail follows an algorithm for how to properly sort the mail.

This algorithm must then be translated by a programmer into a program written in a language.[1] Generally, this is a high-level language, such as C or Perl, although it might be a low-level language, such as assembler. The language layer exists to make our lives easier: the structure and grammar of high-level languages lets us easily write complex programs. This high-level language program, which is usually portable between different systems, is then transformed by a compiler into the low-level instructions required by a specific system. These instructions are specified by the Instruction Set Architecture.

[1] It has been conjectured that mathematicians are devices for transforming coffee into theorems. If this is true, then perhaps programmers are devices for transforming caffeine and algorithms into source code.

1.1.1.2 The Instruction Set Architecture

The Instruction Set Architecture, or ISA, is the fundamental language of the microprocessor: it defines the basic, indivisible instructions that we can execute. The ISA serves as the interface between software and hardware. Examples of instruction set architectures include IA-32, which is used by Intel and AMD CPUs; MIPS, which is implemented in the Silicon Graphics/MIPS R-series microprocessors (e.g., the R12000); and the SPARC V9 instruction set used by the Sun Microsystems UltraSPARC series.

1.1.1.3 Hardware: microarchitecture, circuits, and devices

At this level, we are firmly in the grasp of electrical and computer engineering. We concern ourselves with functional units of microarchitecture and the efficiency of our design. Below the microarchitectural level, we worry about how to implement the functional units through circuit design: the problems of electrical interference become very real. A full discussion of the hardware layer is far beyond us here; tuning the implementations of microprocessors is not something we are generally able to do.

1.1.2 The von Neumann Model

The von Neumann model has served as the basic design model for all modern computing systems: it provides a framework upon which we can hang the abstractions and flesh generated by the levels of transformation.[2] The model consists of four core components:

[2] A good book to read more about the von Neumann model is William Aspray's John von Neumann and the Origins of Modern Computing (MIT Press).

  • A memory system, which stores both instructions and data. This is known as a stored program computer. This memory is accessed by means of the memory address register (MAR), where the system puts the address of a location in memory, and a memory data register (MDR), where the memory subsystem puts the data stored at the requested location. I discuss memory in more detail in Chapter 4.

  • At least one processing unit, often known as the arithmetic and logic unit (ALU). The processing units are more commonly called the central processing unit (CPU).[3] It is responsible for the execution of all instructions. The processor also has a small amount of very fast storage space, called the register file. I discuss processors in detail in Chapter 3.

    [3] In modern implementations, the "CPU" includes both the central processing unit itself and the control unit.

  • A control unit, which is responsible for controlling cross-component operations. It maintains a program counter, which contains the next instruction to be loaded, and an instruction register, which contains the current instruction. The peculiarities of control design are beyond the scope of this text.

  • The system needs a nonvolatile way to store data, as well as ways to represent it to the user and to accept input. This is the domain of the input/output (I/O) subsystem. This book primarily concerns itself with disk drives as a mechanism for I/O; I discuss them in Chapter 5. I also discuss network I/O in Chapter 7.

Despite all the advances in computing over the last sixty years, they all fit into this framework. That is a very powerful statement: despite the fact that computers are orders of magnitude faster now, and being used in ways that weren't even imaginable at the end of the Second World War, the basic ideas, as formulated by von Neumann and his colleagues, are still applicable today.

1.1.3 Caches and the Memory Hierarchy

As you'll see in Section 1.2 later in this chapter, one of the principles of performance tuning is that there are always trade-offs. This problem was recognized by the pioneers in the field, and we still do not have a perfect solution today. In the case of data storage, we are often presented with the choice between cost, speed, and size. (Physical parameters, such as heat dissipation, also play a role, but for this discussion, they're usually subsumed into the other variables.) It is possible to build extremely large, extremely fast memory systems -- for example, the Cray 1S supercomputer used very fast static RAM exclusively for memory.[4] This is not something that can be adapted across the spectrum of computing devices.

[4] Heat issues with memory was the primary reason that the system was liquid cooled. The memory subsystem also comprised about three-quarters of the cost of the machine in a typical installation.

The problem we are trying to solve is that storage size tends to be inversely proportional to performance, particularly relative to the next highest level of price/performance. A modern microprocessor might have a cycle time measured in fractions of a nanosecond, while making the trip to main memory can easily be fifty times slower.

To try and work around this problem, we employ something known as the memory hierarchy. It is based on creating a tree of storage areas (Figure 1-1). At the top of the pyramid, we have very small areas of storage that are exceedingly fast. As we progress down the pyramid, things become increasingly slow, but correspondingly larger. At the foundation of the pyramid, we might have storage in a tape library: many terabytes, but it might take minutes to access the information we are looking for.

Figure 1-1. The memory hierarchy

From the point of view of the microprocessor, main memory is very slow. Anything that makes us go to main memory is bad -- unless we're going to main memory to prevent going to an even slower storage medium (such as disk).

The function of the pyramid is to cache the most frequently used data and instructions in the higher levels. For example, if we keep accessing the same file on tape, we might want to store a temporary copy on the next fastest level of storage (disk). We can similarly store a file we keep accessing from disk in main memory, taking advantage of main memory's substantial performance benefit over disk.

1.1.4 The Benefits of a 64-Bit Architecture

Companies that produced computer hardware and software often make a point of mentioning the size of their systems' address space (typically 32 or 64 bits). In the last five years, the shift from 32-bit to 64-bit microprocessors and operating systems has caused a great deal of hype to be generated by various marketing departments. The truth is that although in certain cases 64-bit architectures run significantly faster than 32-bit architectures, in general, performance is equivalent.

1.1.4.1 What does it mean to be 64-bit?

The number of "bits" refers to the width of a data path. However, what this actually means is subject to its context. For example, we might refer to a 16-bit data path (for example, UltraSCSI). This means that the interconnect can transfer 16 bits of information at a time. With all other things held constant, it would be twice as fast as an interconnect with a 8-bit data path.

The "bitness" of a memory system refers to how many wires are used to transfer a memory address. For example, if we had an 8-bit path to the memory address, and we wanted the 19th location in memory, we would turn on the appropriate wires (1, 2, and 5; we derive this from writing 19 in binary, which gives 00010011 -- everywhere there is a one, we turn on that wire). Note, however, that since we only have 8 bits worth of addressing, we are limited to 64 (28) addresses in memory. 32-bit systems are, therefore, limited to 4,294,967,296 (232) locations in memory. Since memory is typically accessible in 1-byte blocks, this means that the system can't directly access more than 4 GB of memory. The shift to 64-bit operating systems and hardware means that the maximum amount of addressable memory is about 16 petabytes (16777216 GB), which is probably sufficient for the immediately forseeable future.

Unfortunately, it's often not quite this simple in practice. A 32-bit SPARC system is actually capable of having more than 4 GB of memory installed, but, in Solaris, no single process can use more than 4 GB. This is because the hardware that controls memory management actually uses a 44-bit addressing scheme, but the Solaris operating system can only give any one process the amount of memory addressable in 32 bits.

1.1.4.2 Performance ramifications

The change from 32-bit to 64-bit architectures, then, expanded the size of main memory and the amount of memory a single process can have. An obvious question is, how did applications benefit from this? Here are some kinds of applications that benefitted from larger memory spaces:

  • Applications that could not use the most time-efficient algorithm for a problem because that algorithm would use more than 4 GB of memory.

  • Applications where caching large data sets is critically important, and therefore the more memory available to the process, the more can be cached.

  • Applications where the system is short on memory due to overwhelming utilization (many small processes). Note that in SPARC systems, this was not a problem: each process could only see 4 GB, but the system could have much more installed.

In general, the biggest winners from 64-bit systems are high-performance computing and corporate database engines. For the average desktop workstation, 32 bits is plenty.

Unfortunately, the change to 64-bit systems also meant that the underlying operating system and system calls needed to be modified, which sometimes resulted in a slight slowdown (for example, more data needs to be manipulated during pointer operations). This means that there may be a very slight performance penalty associated with running in 64-bit mode.


1.2 Principles of Performance Tuning

In this book, I present a few "rules of thumb." As an old IBM technical bulletin says, rules of thumb come from people who live out of town and who have no production experience.[5]

[5] MVS Performance Management, GG22-9351-00.

Keeping that in mind, I've found that much of the conceptual basis behind system performance tuning can be summarized in five principles: understand your environment, nothing is ever free, throughput and latency are distinct measurements, resources should not be overutilized, and one must design experiments carefully.

1.2.1 Principle 0: Understand Your Environment

If you don't understand your environment, you can't possibly fix the problem. This is why there is such an emphasis on conceptual material throughout this text: even though the specific details of how an algorithm is implemented might change, or even the algorithm itself, the abstract knowledge of what the problem is and how it is approached is still valid. You have a much more powerful tool at your disposal if you understand the problem than if you know a solution without understanding why the problem exists.

Certainly, at some level, problems are well beyond the scope of the average systems administrator: tuning network performance at the level discussed here does not really require an in-depth understanding of how the TCP/IP stack is implemented in terms of modules and function calls. If you are interested in the inner details of how the different variants of the Unix operating system work, there are a few excellent texts that I strongly suggest you read: Solaris Kernel Internals by Richard McDougall and James Mauro, The Design of the UNIX Operating System by Maurice J. Bach, and Operating Systems, Design and Implementation by Andrew Tanenbaum and Andrew Woodfull (all published by Prentice Hall).

The ultimate reference, of course, is the source code itself. As of this writing, the source code for all the operating systems I describe in detail (Solaris and Linux) is available for free.

1.2.2 Principle 1: TANSTAAFL!

TANSTAAFL means There Ain't No Such Thing As A Free Lunch.[6] At heart, performance tuning is about making trade-offs between various attributes. This usually consists of a list of three desirable attributes -- of which we can pick only two.

[6] With apologies to Robert A. Heinlein's "The Moon Is A Harsh Mistress."

One example comes from tuning the TCP network layer, where Nagle's algorithm provides a way to sacrifice latency, or the time required to deliver a single packet, in exchange for increased throughput, or how much data can be effectively pushed down the wire. (We'll discuss Nagle's algorithm in greater detail in Section 7.4.2.8.)

This principle often necessitates making real, significant, and difficult choices.

1.2.3 Principle 2: Throughput Versus Latency

In many ways, systems administrators who are evaluating computer systems are often like adolescent males evaluating cars. This is unfortunate. In both cases, there are a certain set of metrics, and we try to find the highest value for the most "important" metric. Typically, these are "bulk data throughput" for computers and "horsepower" for cars.

The lengths to which some people will go to obtain maximum horsepower from their four-wheeled vehicles are often somewhat ludicrous. A slight change in perspective would reveal that there are other facets to the performance gem. A lot of effort is spent in optimizing a single thing that may not actually be a problem. I would like to illustrate this by means of a simplistic comparison (please remember that we are comparing performance alone). Vehicle A puts out about 250 horsepower, whereas Vehicle B only produces about 95. One would assume that Vehicle A exhibits significantly "better" performance in the real world than Vehicle B. The astute reader may ask what the weights of the vehicles involved are: Vehicle A weighs about 3600 pounds, whereas Vehicle B weighs about 450. It is then apparent that Vehicle B is actually quite a bit faster (0-60 mph in about three and a half seconds, as opposed to Vehicle A's leisurely five and a half seconds). However, when we compare how fast the vehicles actually travel in crowded traffic down Highway 101 (which runs down the San Francisco Peninsula), Vehicle B wins by an even wider margin, since motorcycles in California are allowed to ride between the lanes ("lane splitting").[7]

[7] For the curious, Vehicle A is an Audi S4 sedan (2.7 liter twin-turbocharged V6). Vehicle B is a Honda VFR800FI Interceptor motorcycle (781cc normally-aspirated V4). Both vehicles are model year 2000.

Perhaps the most often neglected consideration in this world of tradeoffs is latency. For example, let's consider a fictional email server application -- call it SuperDuperMail. The SuperDuperMail marketing documentation claims that it is capable of handling over one million messages a hour. This might seem pretty reasonable: this rate is far faster than most companies need. In other words, the throughput is generally good. A different way of looking at the performance of this mail server would be to ask how long it takes to process a single message. After some pointed questions to the SuperDuperMail marketing department, they reveal that it takes half an hour to process a message. This appears contradictory: it would seem to indicate that the software can process at most two messages an hour. However, it turns out that the SuperDuperMail product is based on a series of bins internally, and moves messages to the next bin only when that bin is completely full. In this example, despite the fact that throughput is acceptable, the latency is horrible. Who wants to send a message to the person two offices down and have it take half an hour to get there?

1.2.4 Principle 3: Do Not Overutilize a Resource

Anyone who has driven on a heavily-travelled highway has seen a very common problem in computing emerge in a different area: the "minimum speed limit" signs often seem like a cruel joke! Clearly, there are many factors that go into designing and maintaining an interstate, particularly one that is heavily used by commuters: the peak traffic is significantly different than the average traffic, funding is always a problem, etc. Furthermore, adding another lane to the highway, assuming that space and funding is available, usually involves temporarily closing at least one lane of the active roadway. This invariably frustrates commuters even more. The drive to provide "sufficient" capacity is always there, but the barriers to implementing change are such that it often takes quite a while to add capacity.

In the abstract, the drive to expand is strongest when collapse is imminent or occuring. This principle usually makes sense: why should we build excess capacity when we aren't fully using what we have? Unfortunately, there are some cases where complete utilization is not optimal. This is true in computing, yet people often push their systems to 100% utilization before considering upgrades or expansion.

Overutilization is a dangerous thing. As a general rule of thumb, something should be no more than 70% busy or consumed at any given time: this will provide a margin of safety before serious degradation occurs.

1.2.5 Principle 4: Design Tests Carefully

I would like to explain this principle by means of a simple example of transferring a 130 MB file over Gigabit Ethernet using ftp.

% pwd

/home/jqpublic

% ls -l bigfile

-rw-------   1 jqpublic staff    134217728 Jul 10 20:18 bigfile

% ftp franklin

Connected to franklin.

220 franklin FTP server (SunOS 5.8) ready.

Name (franklin:jqpublic): jqpublic

331 Password required for jqpublic.

Password: <secret>

230 User jqpublic logged in.

ftp> bin

200 Type set to I.

ftp> prompt

Interactive mode off.

ftp> mput bigfile

200 PORT command successful.

150 Binary data connection for bigfile (192.168.254.2,34788).

226 Transfer complete.

local: bigfile remote: bigfile

134217728 bytes sent in 13 seconds (9781.08 Kbytes/s)

ftp> bye

221 Goodbye.

If you saw this sort of performance, you would probably be very upset: you were expecting to see a transfer rate on the order of 120 MB per second! Instead, you are barely getting a tenth of that. What on earth happened? Is there a parameter that's not set properly somewhere, did you get a bad network card? You could waste a lot of time searching for the answer to that question. The truth is that what you probably measured was either how fast you can push data out of /home, or how fast the remote host could accept that data.[8] The network layer is not the limiting factor here; it's something entirely different: it could be the CPU, the disks, the operating system, etc.

[8] Which is not necessarily the same as "How fast can this data be written to disk?" although it might be.

A great deal of this book's emphasis on explaining how and why things work is so that you can design experiments that enable you to measure what you think you are.

There is a huge amount of hearsay regarding performance analysis. Your only hope of making sense of it is to understand the issues, design tests, and gather data.

The moral of this example is to think very, very carefully when you design performance measurement experiments. All sorts of things might be going on just underneath the surface of the measurement that are causing you to measure something that has little bearing on what you're actually interested in measuring. If you really want to measure something, try and find a tool written specifically to test that component. Even then, be careful: it's very easy to get burnt.

Tuning Kernel Variables

Throughout this book, we will refer to making adjustments to specific kernel variables. Never perform kernel tuning on a live system if you can possibly help it; try and replicate the problem in a controlled laboratory environment and experiment with the tunables there. Even if you do have to make changes on a live system, test your changes extensively.

In Solaris, you can tune kernel variables on the fly with adb, or have them automatically set at boot time. We'll use maxpgio as an example (you can learn more about maxpgio in Section 4.2.5). First, you need to know whether you are booted into a 64-bit or 32-bit kernel. You can find this out by running isainfo -v. Here's an example from a 32-bit system:

% isainfo -v

32-bit sparc applications

And from a 64-bit system:

% isainfo -v

64-bit sparcv9 applications

32-bit sparc applications

In order to check the setting of a kernel variable, start up adb in kernel mode (-k). To view the contents of a variable on a 32-bit system, type the variable name followed by /D; on a 64-bit system, follow the variable name with /E. If you follow the variable name with an incorrect suffix, you will get meaningless output. Here's an example from a 32-bit system:

# adb -k

physmem 3bd6

maxpgio/D

maxpgio:

maxpgio:        40

And from a 64-bit system:

# adb -k

physmem 1f102

maxpgio/E

maxpgio:

maxpgio:        40

If you decide that you need to change the value of this variable, you can then make an immediate change to the variable. Drop out of adb by typing Ctrl-D, and restart it with adb -kw. To change the variable, you must again specify a suffix: use /W on 32-bit systems and /Z on 64-bit systems. This string should then be followed by 0t and the decimal value you wish to set the variable to.

For example, on a 32-bit system, let's change the value of maxpgio to 100:

# adb -kw

physmem 3bd6

maxpgio/W 0t100

maxpgio:        0x28            =       0x64

maxpgio/D

maxpgio:

maxpgio:        100

We can perform the same change on a 64-bit system:

# adb -kw

physmem 1f102

maxpgio/Z 0t100

maxpgio:        28              =       64

maxpgio/E

maxpgio:

maxpgio:        100

This change will not persist past a reboot. If you want to make the change permanent, edit /etc/system:

*

* change kernel variable at boot-time: 08-11-2001 by gdm

set maxpgio=100

In Linux, depending on whether the variable has been exposed in the /proc tree or not, you have two choices: you can either edit the kernel source code and build a new kernel with your changes, or you can edit the appropriate file in /proc. Editing files in proc is straightforward -- for example, let's say you wish to change the value of the min_percent variable in Linux 2.2, which controls the minimum percentage of system memory available for caching. This variable can be found in the /proc/sys/vm/buffermem file, and the format of that file is min_percent max_percent borrow_percent. We can change the value of min_percent quite straightforwardly:

# cat /proc/sys/vm/buffermem

2 10 60

# echo "5 10 60" > /proc/sys/vm/buffermem

# cat /proc/sys/vm/buffermem

5 10 60

These changes won't persist past a reboot, but you can put an entry in a startup script to perform the tuning automatically.

Again—be careful, and always test your changes!


1.3 Static Performance Tuning

This book largely focuses on the performance tuning of systems in dynamic conditions. I focus almost exclusively on how the system's performance corresponds to the stresses placed on the system. Another sort of performance tuning exists, however, that focuses on static (that is, workload-independent) factors. These factors tend to degrade system performance no matter what the workload is, and are not generally tied to resource contention problems.

By far the largest culprit in static performance issues are problems with naming services: the means by which information is retrieved about an entity. Examples of naming services are NIS+, LDAP, and DNS.

The symptoms are often vague: logins are slow, the windowing environment or a web browser feels sluggish, a new window locks at startup, or the X or CDE login screen hangs. Here are a few places to check, in approximate order of likelihood:

/etc/nsswitch.conf

This file is read only once for each process that is using a naming service, so a reboot may be necessary to make changes take effect.

/etc/resolv.conf

Are the nameservers and the domain specified correctly? Incorrect or lengthy (having many subcomponents) domain specifications may cause DNS to generate a great many requests. The nameservers should be sorted by latency of response.

Is the name service cache daemon (nscd) running?

This process caches name service-provided information for a significant boost to performance. This daemon is standard-issue on Solaris, and available for Linux. Historically, nscd has been blamed for many problems. Most of these problems have been long since fixed: you should probably run it. One exception is when troubleshooting, however, as it can mask underlying name service problems.

Another likely culprit is the Network File System (NFS). It is important to not nest mounts. For example, let's say my workstation mounts three directories from three distinct NFS servers:

alpha:/home

bravo:/home/projects

delta:/home/projects/system-performance-tuning-2nd-ed

Reading the file /home/projects/system-performance-tuning-2nd-ed/README requires all three NFS servers to be up, and generates quite a bit of excess traffic. It would be much faster to have separate mount points for everything.

1.3.1 Other Miscellaneous Things to Check

Duplicate IP addresses on a network, as well as many other host interface misconfigurations, can cause all sorts of problems. Try to maintain strict control over IP addressing. Sometimes cables fail or begin to generate errors: you can check the error rates for an interface with netstat -i.[9]

[9] I have run across these problems most in coaxial-cable (10base2) environments. These sorts of problems are often very hard to track down and are very frustrating. Buy good-quality cable, and label thoroughly.

Sometimes processors will fail. This will almost always induce a panic. However, in some high-end Sun systems (the E3500-E6500 and E10000, for example), the system will automatically reboot, attempt to isolate the fault, and carry on. The result is that a system could seem to spontaneously reboot, but come back up short a few processors (their failure had induced the reboot). It's good practice to use the psrinfo command to confirm that all your processors are online after a reboot, no matter what the cause.

There are many subtle hardware problems that can induce static performance problems. If you suspect that this is occuring, it's usually much easier on your sanity to simply replace the failed hardware. If that's not possible, expect a long and arduous troubleshooting session; it's almost always helpful to have a whiteboard handy to draw out a matrix of configurations to identify which situations work and which don't.


1.4 Concluding Thoughts

Approaching any complex task usually requires some measure of grounding in fundamentals, and a basic understanding of the underlying principles. We've touched upon what performance tuning is, the basic ideas behind computer architecture, the big questions surrounding 64-bit environments, static performance tuning techniques, and some fundamental principles of performance. While it's very difficult to approach the effectiveness of, say, ground school in pilot training, in an introductory chapter, I hope this chapter has given you some sense of the currency in which the rest of this book trades.

I have a suggestion for an exercise, which I think is particularly applicable if you are intent on reading this book cover-to-cover. Go back and reread the five principles of performance tuning (see Section 1.2 earlier in this chapter), put this book down for an hour, and think about what those principles mean and what their implications are. They are very broad, general concepts that are relevant far beyond the context in which they've been presented. Their application is left entirely to you -- in some sense the most difficult part -- although I hope you will find the rest of this book to be an effective guide. When you sit down to analyze a problem, ask yourself a few questions: do I understand what is going on? If not, how can I design meaningful tests to validate my theories? Am I, or my customers, searching for a "free lunch"? What tradeoffs am I making to get the performance I want? Am I overconsuming a resource? Are the metrics I am using to measure performance ones I have developed? Are these metrics measuring what I think they are?

These are all difficult questions, no matter how simple they may appear: they are innocent-looking trapdoors that lead into dark, confusing dungeons. It's not unusual to find eight or ten performance experts in a meeting discussing what exactly is going on in a system, formulating theories, and designing experiments to test those theories. If there is anything you take from this book, I earnestly hope it is those five principles. It's hard to go too far wrong if you use them as your guiding light.


Chapter 2. Workflow Management

The more precisely the position is determined, the less precisely the momentum is known in this instant, and vice versa.

—Werner von Heisenberg, 1927

The topic of "workflow management" is slippery: it can be interpreted to mean many different things. In this chapter, I describe practical means to accomplish the zeroth principle of performance tuning: understanding the environment as it exists. This is the beating heart of dynamic performance tuning. The rest of this book exists simply to improve your understanding about possible environments.

We concern ourselves, as mentioned, primarily with dynamic performance analysis: the system we are measuring is changing beneath us. It is, in some sense, like watching a pond. There might be a creek that flows into the pond; how does that affect the life in the pond? What happens when some beavers build a dam across that creek, when some children find the pond and throw rocks into it, or when someone dumps the ashes of old love letters in the middle of it?

To further complicate things, we are governed by an inviolable principle of physics. Heisenberg's Principle of Uncertainty says that no matter how carefully we try, we will always perturb the system when we measure it, and some piece of knowledge will remain outside of our grasp. We can minimize the perturbation, however, and throughout this chapter we'll concern ourselves with how significant of a perturbation our measurements are inducing.

Things at this point seem pretty daunting. Not only are we measuring something that's always changing in ways we don't understand, driven by factors we can't control, but our mere observation of the system induces further change in it! Why do we bother to measure, then? The answer is simple: without careful, methodical measurements, it becomes practically impossible to provide reasoned, analytical solutions to problems, because we don't understand why the problem exists or what happened. It's always better to know something about the situation. If we didn't strive to understand, we would be doomed to be Cargo Cult Systems Administrators, sitting at our desks with wooden keyboards and terminals,[1] waiting for the magicians to come back in their shiny silver birds at a few hundred dollars an hour to fix our problems for us. That's not a good solution for anyone but the magicians.

[1] Although I have often wished my pager or cellular phone was wooden, and therefore not able to ring.

In this chapter, I break workflow management into two distinct parts: workflow characterization and workflow control. It is important that they are presented in this order; control imposed haphazardly is a real mess, and many a consultant has made a small fortune cleaning up after it. Do not despair about the complexity implicit in this topic. There are many tools that can help us understand and, subsequently, restrict the loads placed on the systems we manage; many of them are minimally intrusive. I then move into benchmarks, as standardized ways of measuring the performance of various workloads on different systems. Consider this your primer on turning Cargo Cult wood into twenty-first century silicon.


2.1 Workflow Characterization

Performance experts are, in some sense, the gurus atop mountains of the computer world.[2] Every time some humble supplicant comes to them with a problem, the guru pokes around at things for a while, then sends the supplicant off to come up with more information. Characterization is the process of trying to gather as much information about the system as possible, so that trends and patterns can be determined. These patterns will prove to be vitally important the first time that performance falls through the floor; we will be able to piece together what the perturbations in the patterns are, and from that figure out what caused them -- rather like studying the wake of a passing ship to see what kind of vessel it was and where it's headed.

[2] To extend this analogy, the database performance people sometimes seem to be in outer space. I have three main points of evidence: it is absurdly expensive to talk to them, either you or they usually have to travel vast distances for an audience, and they speak in strange languages that I don't understand.

An analogy has been drawn between workload management and financial transactions. The first time I read of this was in Adrian Cockcroft's excellent book Sun Performance and Tuning (Prentice Hall). The essential idea is that workload management on computer systems is analogous to a department with a capital budget, staff, etc., performing a task. There are basically three possible outcomes:

  1. If there is no plan and no effective controls on the staff, then the staff will run wild, grabbing as much budget as they can to ensure that their own projects will be well-funded. Some staff will end up with no funding whatsoever, while other staff take up "fact-finding missions" to Maui. The project as a whole ends up a complete mess. (This is known as the "startup model.")

  2. Management is overzealous, and creates a huge bureaucratic staff to plan, assess the plans, and replan. The entire budget is consumed by this bureaucratic middle layer, which micromanages those responsible for actually doing the work, often by demanding daily status reports. The administrative overheads involved make it very difficult to spend any money. The work ends up a mess, because by the time it was supposed to be finished, it's barely been started. (This is known as the "government model.")

  3. In the ideal case, management balances and controls funding so that staff are constrained, but everyone has enough to get their work done. The bureaucratic overhead is kept to a minimum, and status reports are infrequent. The work is done on-time and within its budget. (This is known as the "dreamland model.")

The analogy to performance management is fairly straightforward. Instead of a capital budget, we have the currency of computing, if you will: processor cycles, disk I/O rates, network capacity, etc. We have to have some management over these resources, or we end up following the startup model, where actually getting any resources turns into a grabbing contest. However, if we manage things too much, we'll end up like the governmental model, where nobody can get any work done because we all spend too much time filling out paperwork. So, ideally, we should aim for a balanced approach: the dreamland model. Unfortunately, the dreamland model is aptly named. It's very hard to achieve, much less maintain, such a model in practice.

The first step in performance management has to be developing some processes to guide our decisions. This will involve setting concrete goals, identifying reproducibly measurable performance criteria, and determining the current status. This information then lets us go to management and negotiate for resources to improve the user's experience.

As a discipline, performance management started decades ago in the mainframe world, where everything is stunningly expensive. The payback from making tuning decisions based on real understanding was direct; the actual cost savings in the computing budget was often quite large. Now that the costs of computing have plummeted, it's getting harder to justify the time and effort spent in understanding and tuning. You have to focus more on the indirect advantages to analysis; haphazardly applying system upgrades might not accomplish anything for several months, whereas a careful study of the system and a reasoned upgrade might improve performance significantly. The value of the increased throughput over those several months may be much greater than the cost of the analysis.

There are three important tools we can use to improve our understanding of system behavior: the simple performance measurement commands that we are all fairly familiar with, process accounting, and sar's automated data collection facilities. In this chapter, I'll briefly touch on network pattern analysis.

2.1.1 Simple Commands

We have all used simple commands like iostat, vmstat, and mpstat. These tools are fundamental to performance analysis, and can provide a great deal of valuable information about what is going on in the system. One quick way to gather data about the runtime of a system is to set the interval on these commands to some fairly large amount of time (say, a few minutes), and then redirect the output of the command to a file. One problem with this is that it becomes very hard to keep track of the time that a specific data point was collected. It's straightforward to write some Perl to take care of this for you:

#!/usr/bin/perl

while (<>) { print localtime(  ) . ":  $_"; }

Example 2-1 shows the script in action on a Linux system.

Example 2-1. vmstat on a Linux system
# vmstat 5 | chrononome.pl

Sat Jun 30 00:37:28 2001:  procs               memory      swap      io     system    cpu

Sat Jun 30 00:37:28 2001: r  b  w  swpd  free  buff  cache  si  so   bi   bo  in  cs us sy id

Sat Jun 30 00:37:28 2001: 1  0  0  5472 26680  8420 177908   0  0    10   37  63  37  5  6 14

Sat Jun 30 00:37:33 2001: 0  0  0  5472 26576  8420 177908   0  0     0   20 163  37  1  0 99

Sat Jun 30 00:37:38 2001: 0  1  0  5472 26540  8420 177916   0  0     0   12 148  41  1  1 98

Sat Jun 30 00:37:43 2001: 0  0  0  5472 26904  8420 177920   0  0     2    5 141  43  1  2 98

Sat Jun 30 00:37:48 2001: 0  0  0  5472 26904  8420 177920   0  0     0    1 129  39  2  1 98

Sat Jun 30 00:37:53 2001: 0  0  0  5472 26904  8420 177920   0  0     0    0 124  36  0  1 99

Sat Jun 30 00:37:58 2001: 0  0  0  5472 26904  8420 177920   0  0     0   10 143  35  1  0 99

Some people run these commands with a very short interval via a crontab entry. This is a valid approach, but I am not personally fond of it. It depends on what you want to measure: if I run vmstat with a 1800-second interval, I expect each line to be the average virtual memory activity over the last half hour. However, if I run vmstat with a 2-second interval every half hour out of cron, I get a two-second snapshot of activity right at the half-hour mark. Both can be useful, but I would rather have longer-interval data, at least to start with. If it turns out that I am missing spikes in data, I can always start up another copy of the monitoring application and set the interval to something sufficient to catch the peaks and valleys. Long-term performance trends are not really captured well by increasing the collection interval; mostly you'll just have to do a lot of smoothing on a lot of data.

2.1.2 Process Accounting

Process accounting is a means by which the system gathers information on every process as it runs. This information consists of CPU utilization, disk I/O activity, memory consumption, and other useful tidbits. Perhaps the most useful part of process accounting is that it comes with mechanisms for determining patterns in usage, by means of the audit reporting system.

Some system administrators are scared off of process accounting because of fears of high overhead. Collecting the accounting data has essentially no impact on the system: the kernel always collects the accounting data, so the only extra overhead is the writing of a 40-byte record to the accounting logs. The log summary scripts, however, can take a significant amount of time to run, so they are best scheduled outside of peak hours.

2.1.2.1 Enabling process accounting

Starting system accounting is very simple. In Solaris, be sure that you have installed the optional packages that contain the process accounting functionality: they are SUNWaccu and SUNWaccr. As of Solaris 8, these packages are located on the second Solaris installation CD.

The first step is to link the startup and shutdown scripts:

# ln /etc/init.d/acct /etc/rc0.d/K22acct

# ln /etc/init.d/acct /etc/rc2.d/S22acct

You can then reboot the system, or start accounting immediately by running /etc/init.d/acct start. The second step is to add cron entries to the adm user for the summary reporting commands. Example 2-2 shows what to add.

Example 2-2. Adding cron entries
# min   hour    day     month   wkday   command

0       *       *       *       *       /usr/lib/acct/ckpacct

30      2       *       *       *       /usr/lib/acct/runacct 2> /var/adm/acct/nite/fd2log

30      9       *       *       5       /usr/lib/acct/monacct

The ckpacct program is used to check the size of the accounting file /usr/adm/pacct. The runacct command generates accounting information from the data, and the monacct generates "fiscal" reports for each user. These reports are stored in /var/adm/acct.

2.1.2.2 Reviewing accounting records

The most useful tool for reviewing accounting data is acctcom, which shows the immediate accounting data. It can be run in several modes: specifying -a will give you average statistics on each process that has been run, -t will provide the system/user time breakdowns for each process, -u user will show all the processes executed by a given user, and -C time will show all the processes that consumed more than time seconds of processor time. There are many other useful options to acctcom that may vary from system to system, so consult your manual pages for more information.

The system accounting programs also generate some files that can be reviewed at your leisure. These are done daily and at the end of every accounting period (defined as when monacct runs).

2.1.3 Automating sar

We will periodically talk about sar as a means of gathering performance data. sar can also be used to automatically collect data and store it for later review.

2.1.3.1 Enabling sar

Enabling automated data collection with sar is quite straightforward, and entails two steps: uncomment the relevant lines (namely, the last 13) in /etc/init.d/perf, and set up the system crontab file to support automated data recording.

Changing the system cron entries (located in /var/spool/crontabs/sys) entails making some decisions about when you'd like data to be recorded (via the sa1 command), and when you'd like data to be reported (by the sa2 command). By default, these are the entries:

# min   hour    day     month   wkday   command

# 0 * * * 0-6 /usr/lib/sa/sa1

# 20,40 8-17 * * 1-5 /usr/lib/sa/sa1

# 5 18 * * 1-5 /usr/lib/sa/sa2 -s 8:00 -e 18:01 -i 1200 -A

The first entry will write a sar record every hour on the hour, seven days a week. The second entry will write a sar record twice an hour, at 20 minutes and 40 minutes past the hour, during peak working hours of 8 A.M. to 5 P.M., Monday through Friday.

The third record is more complex. At five minutes past six o'clock, Monday through Friday, it will report on the data gathered between 8 A.M. (-s time) and one minute past six o'clock (-e time), at an interval of 1200 seconds, or 20 minutes (-i seconds), and it will report all data (-A option to sar).

2.1.3.2 Retrieving data

Retrieving data from the sar records is remarkably simple. You simply run sar, specifying the data you'd like to see by means of the normal option switches, and leave off the interval for the present day's data. You can use the -s starting-time and -e ending-time flags to control the time range of interest, and you can specify the day of interest by using -f /var/adm/sa/sa dd (where dd is the day of the month).

2.1.4 Virtual Adrian

Adrian Cockcroft is a distinguished engineer for Sun in the field of performance analysis. He, in coordination with Rich Pettit, developed a software suite for data collection and analysis on Solaris systems. It's described in great detail in their book, Sun Performance and Tuning (Prentice Hall). The book in general is very much worth reading, although it is slightly out of date, and the toolkit is also excellent. This package is called the SE Toolkit, and is located at http://www.setoolkit.com.

2.1.5 Network Pattern Analysis

Historically,[3] networks have not generally been a real-world performance limitation. Consequently, a great deal of effort has been placed on understanding and improving other aspects of the system. Because of the explosion of internetworked applications, the network layer is increasingly becoming a limiting factor. As a result, the tools in network performance analysis tend to not be quite as refined as those in other areas. How, then, can we approach the problem of understanding the network layer in our environment?

[3] That is, in the pre-web world.

A word of warning before we proceed: this section assumes some familiarity with the networking concepts discussed in Chapter 7.

I open this section by discussing three common traffic patterns observed in environments where applications communicate over TCP/IP; then, I briefly discuss some parameters of networked traffic, such as "How large is a packet, on average?" Finally, we'll discuss some of the tools that you can use to determine the patterns occuring on your own network.

A Brief Note on Terminology

I use a few terms in our discussion of network traffic that bear definition:

  • In a TCP/IP connection, the client is the end that opened the connection (the initiating endpoint). The server is the end that accepted the connection (the accepting endpoint). Note that this says absolutely nothing about what roles the machines are playing in anything other than the connection under examination!

  • Inbound traffic is moving from the client towards the server; outbound traffic is the reverse direction.

  • An payload-free packet is generally about 60 bytes on the wire (recall that there are various headers -- typically, Ethernet, IP, and TCP). These packets are a necessary part of how some functions (for example, TCP) work.

  • A small packet is one that is less than 400 bytes on the wire (including all headers).

  • A medium packet is one that is between 400 and 900 bytes.

  • A large packet is between 900 and 1,500 bytes (the maximum for Ethernet).

2.1.5.1 Pattern 1: request-response

The first pattern is that of request-response. The classic real-world examples of this are HTTP, email retrieval protocols such as POP3, and outbound SMTP transactions (delivered to a remote host). Figure 2-1 shows Pattern 1 traffic.

Figure 2-1. Pattern 1 traffic

The request-response pattern is characterized by generally high connection rates. However, fairly low connection rates are sometimes observed, particularly in modern web environments where HTTP session keepalives are being used extensively. The inbound packets are generally small; the outbound packets are generally medium-to-large, but generally there are only a few (less than twenty) of them.

2.1.5.2 Pattern 1B: inverse request-response

A variation on the first pattern is inverse request-response; it is typically seen in inbound SMTP transactions. The interesting part is that the inbound and outbound roles are reversed: the client initiates the majority of the data transfer. Figure 2-2 illustrates the pattern of 1B traffic.

Figure 2-2. Pattern 1B traffic

Connection rates are often moderate, and sometimes exhibit bursty behavior. Inbound packets are generally small in the initial part of the connection, but then become large. The outbound packets are small or payload-free.

2.1.5.3 Pattern 2: data transfer

The second pattern is typical of large amounts of data being transferred. This is most commonly seen in ftp traffic and file transfers during networked backups (see Figure 2-3).

Figure 2-3. Pattern 2 traffic

In general, the connection rate is very low. The inbound packet stream is essentially all payload-free packets, but the outbound packet stream consists almost entirely of full packets.

2.1.5.4 Pattern 3: message passing

The third and final pattern is that of message passing. This is most commonly seen in character-driven applications such as telnet, rlogin, and ssh, but it is also seen sometimes in database transaction schemes such as SQLnet. Some parallelized high-performance computing codes also exhibit this sort of pattern. Figure 2-4 illustrates Pattern 3.

Figure 2-4. Pattern 3 traffic

The connection rate is variable. It is generally low, but can be quite high -- it is very application-dependent. The characteristic to look for is that there are large numbers of small packets being pushed around. This sort of traffic tends to be very hard on a system, because the amount of work required to process a small packet is essentially the same as the amount of work required to process a large packet: you are doing the same amount of work for less payload.

Tuning systems for Pattern 3 workloads is very difficult. If you are faced with a Pattern 3 workload, you are sure the network is the limiting factor in performance, and you need a performance improvement, you have two real choices: either induce algorithmic change in the application so that it stops behaving like Pattern 3 and starts behaving more like Pattern 1, or invest in very low-latency network hardware.[4]

[4] Myrinet has a reputation for fast, low-latency network hardware. It is not widely deployed outside of specialty markets.

2.1.5.5 Packet size distributions

Gathering traces (via snoop or tcpdump, for example) of network activity can give you some very interesting information on what sort of patterns are occuring on your network. One of the more simple questions to ask is "If we draw the graph of packet size versus packet count, what does it look like?" In some of the work I've done on "real-world" systems on the Internet, I found that, remarkably, this graph tends to be trimodal (it has three distinct peaks). These systems are typically web and email servers.

  • The first peak is largely due to inbound traffic, and consists almost entirely of packets about 60 bytes in size. These are TCP acknowledgments flowing "backwards" from browsers towards the web server, acknowledging the receipt of chunks of data.

  • The second peak is almost entirely due to outbound traffic, and occurs at 1540 bytes on the wire. This represents full packets of data flowing away from the server towards the browsers.

  • The third peak took me a little while to figure out: it occurs at about 540 bytes, and is strongly biased towards outbound traffic. It turns out that this is due to certain Windows TCP/IP implementations, which set the "maximum segment size" for a TCP connection to 536 bytes. As a result, the other end cannot transmit any packet larger than 536 bytes back to the client.

If you're in a position where you can strongly influence the traffic on your network, this sort of information can drive great change. Properly tuning application algorithms and TCP stacks can eliminate things like the "middle peak" at 540 bytes, with a corresponding increase in efficiency.


2.2 Workload Control

The next step in the realm of workloads is imposing workload management. This step is often technically straightforward, but it can carry with it political problems. Forcibly attempting to control the users of a system is often not a good idea, and in my opinion should be restricted to cases where education and attempting to secure the cooperation of your user base has failed abjectly. If you are getting some pressure from management to do this, I think you would do well to get it in writing.

Why do I submit these warnings? It is significantly less of a problem in academic environments or in small companies, where everyone knows the systems administrator and it's easy to walk down to their office and ask them what's going on. However, in large professional environments or when the management of the users is divorced to a degree from the management of the administrators, things often get quite complex. It is often very frustrating to users (who, perhaps, designed the hardware or software that you are administrating) to have their abilities restricted: it reduces their flexibility in solving their own problems. If they appeal up through management, you could find yourself trying to explain what seemed like a reasonable technical decision at the time to a few very irate senior vice presidents. You really need to be careful if you are going to be forceful.

On the other hand, user education is one of the most powerful techniques available to you to control workloads. In this section, I discuss user education and the closely related area of written performance agreements, as well as some of the more direct techniques for limiting system resource consumption.

2.2.1 Education

The most powerful tool available to you in order to control the workloads on your systems is user education. Enforcing strict CPU time or disk quotas, while effective, often adds to a "resentment of the mystery" phenomenon. This leaves users feeling rather like medieval serfs: there are certain things they just can't do, like encourage the rain in a dry season, and all they can do is go and beg some rather mysterious people who usually live in caves to try and fix the problem for them. The end result is that the users get very frustrated.

A much better solution is to explain the problem to your users, how their actions induce it, and the solution to the problem. Many times, this sort of forthright discussion will produce the results you'd like, with much less of a headache.

2.2.1.1 Usage and performance agreements

One very basic approach to user education is to develop a usage and performance agreement. This is roughly akin to the "service-level agreements" that are common in the networking world: it is a document that sets out exactly what both sides expect from each other. It can be as formal or as informal as you want it to be. I have seen lengthy documents that looked like they'd gone out for legal review, and half-page descriptions that summarized what the agreed-upon metrics for performance were. In both cases, and in many cases in between, everyone was happy with what they had. I personally prefer short, technically dense documents (I think it's easier to get users to read them), but that's just a matter of taste.

There are many things that can be set out in a usage and performance agreement:

  • What is the minimum acceptable level of performance? That's a pretty loaded question. Ask your users to define it. Are there a certain number of concurrent jobs or users that must be supported? For each job, what is the "maximum runtime"? Can you agree upon a simple test suite that will determine whether, in the general case, performance is "acceptable"?

  • What mechanisms are in place to measure performance? What does the load on the system look like, and what characteristics does it exhibit before failing? Is there a way for users to tell when a machine is "full"?

  • What are the operational windows of the machine? When can maintainance be performed?

  • How are you, the systems administrator, to be notified of problems? If you are out of town or otherwise unavailable, do you have a backup?

  • How are they, the users, to be notified of problems? Is an entry in the message of the day sufficient? Does an email alias need to be created for announcements?

It's vitally important that you revisit these agreements periodically (say, once a quarter). This review may be very short if the document is still accurate, or it may be a bit longer; this is usually the case when the work being done has gotten more complex, or changed to pursue different problems.

It's been my experience that very few people actually take the time to create these sorts of agreements. This is, in my opinion, absurd. In addition to giving you solid information about what the users expect from their environment, it fosters good will and a cooperative spirit. Dealing with uncooperative users is one of the least pleasant aspects of being a systems administrator; unfortunately, it's probably part of your job. Take a few minutes to create an agreement with your users, and get management feedback to ensure your agreement is understood by everybody. It'll be enlightening.

2.2.2 The maxusers and the pt_cnt Parameters

When the Berkeley Standard Distribution (BSD) of Unix was being developed, it created a single tunable that would scale the size of several internal kernel tables and buffers. The scaling factor was directly correlated to the number of time-sharing users that the system had to support, so the parameter was called maxusers. In the modern world, there is no direct relationship between the number of users that a system supports and the value for maxusers. However, a set of parameters in Solaris are still derived from the value of maxusers, and some of them are related to workload control, as described in Table 2-1.

Table 2-1. Parameters derived from maxusers

Variable

Kernel resource

Default setting

max_nprocs

The maximum number of processes systemwide

10 + 16 x maxusers

reserved_procs

The number of processes, systemwide, reserved for the root user

5

maxuprc

The maximum number of processes per non-root user

max_nprocs -- reserved_procs

ndquot

The size of the quota table

(10 x maxusers) + max_nprocs

ncsize

The size of the DNLC(see Section 5.4.1.1)

4 x (max_nprocs + maxusers) + 320

ufs_ninode

The size of the inode cache

(see Section 5.4.2.7 in Chapter 5)

As ncsize

The maxusers parameter itself is automatically sized (unless you are running Solaris 2.3 or earlier). It is set to a value approximately equal to the number of megabytes of RAM in the system; the minimum limit is 8 and the maximum limit (without manual tuning) is 1024, which applies to any system with 1 GB or more of memory. You can set maxusers to a higher value manually in /etc/system, but it is limited to an overall maximum of 2048.

Contrary to popular suspicion, the maxusers parameter does not limit the number of current logins to the system; it just sizes a number of parameters that are generally related to the number of users a system can support. Of course, many things factor into the number of supportable users; maxusers is just intended to be a general knob. The variable that defines supportable users is pt_cnt. By default, this variable is set to 48, which can be a significant limit. There is a practical limit on the value of pt_cnt; namely, the format of the utmp file imposes a limit of 3844 telnet and 3844 rlogin sessions. You should probably keep pt_cnt under about three thousand, in order to avoid hitting other, nontunable system limits. After setting pt_cnt, you will need to create the entries in /dev/pts; this is best accomplished by rebooting with the reconfiguration flag (touch /reconfigure; reboot; this is identical to boot -r).

2.2.3 Limiting Users

Unfortunately, there comes a time when you have to impose limits. Sometimes these limits are set up because user education is very difficult, or because one bad apple can spoil the cider. My personal approach to user limitations is that they are best thought of as an emergency brake; when a (generous) parameter is exceeded, it's healthier to stop whatever is causing the problem rather than cleaning up after it later.

There are two basic approaches to limiting users: quotas and environment controls.

2.2.3.1 Quotas

One of the most common types of user limitations is disk quotas. Disk quotas exist to prevent a user from consuming more than their fair share of disk space, determined by what they're entitled to or what they've paid for.

Disk quotas are typically implemented in two stages. Once a user exceeds a specific threshold of utilization,[5] usually called the hard limit, they are not able to write any more data to disk until they remove some files and their disk utilization drops below that limit. At some lower level of utilization, there is another limit, which is usually called the soft limit. Once the user exceeds the soft limit, they can continue to write data up until they run into the hard limit; however, a timer starts. When the timer (which is usually measured in days) expires, the user cannot write any more files until they remove data and drop below the soft limit. This provides a way for users to temporarily consume large amounts of disk space.

[5] This utilization typically can either be measured in the amount of disk space consumed or in the number of files.

Enabling quotas is quite straightforward on both Solaris and Linux. On Solaris, you will need to install the accounting software packages (SUNWaccr and SUNWaccu) and link two startup files:

# ln /etc/init.d/acct /etc/rc0.d/K22acct

# ln /etc/init.d/acct /etc/rc2.d/S22acct

Then add these lines to adm's crontab file:

# min   hour    day     month   wkday   command

0    *   *    *   *   /usr/lib/acct/ckpacct

30   3   *    *   *   /usr/lib/acct/runacct 2> /var/adm/acct/nite/fd2log

30   9   *    *   5   /usr/lib/acct/monacct

Then enable process accounting by rebooting or by running /etc/init.d/acct start.

On Linux, the process is similar. The first step is to modify the system startup scripts to turn on process accounting at boot:

# enable process accounting

if [ -x /sbin/accton ]

then

    /sbin/accton /var/log/pacct

    echo "activating process accounting"

fi

You then need to create the accounting record file /var/log/pacct. This file should be owned by root and world-readable:

# touch /var/log/pacct

# chown root /var/log/pacct

# chmod 0644 /var/log/pacct

You can then either reboot or run /sbin/accton /var/log/pacct to turn on process data accounting.

Enterprising users can always come up with ways around disk quotas. I have seen users mail files to themselves, find directories on other filesystems (where quotas weren't being enforced) to hide things in, and even "bounce" files around on the network between multiple machines via some homegrown software. The more reasonable your disk quotas are, the less of a problem you will have with users creatively working around the problem. To return to the emergency brake analogy, think about the electronic speed governors installed on modern automobiles. If the governor is set to something like 140 miles per hour, most people will not ever bother to disable it;[6] they don't ever go that fast, so they may not even know it's there. However, if the speed governor is set to something like 80 miles per hour, a huge number of people will start trying to come up with ways around the problem of their engine cutting off as they try and pass someone on the freeway! The upshot is that the more reasonable you are with disk quotas, the less of a headache they are to enforce.

[6] Just like disk quotas, the enterprising driver can always disable a speed governor.

2.2.3.2 Environmental limits

A more subtle way of controlling users' processes is to impose limits on how much of a system resource their applications can consume. This is implemented at the shell level via the limit command in csh or ulimit in ksh and sh. Table 2-2 summarizes the most common of these parameters and their typical defaults

Table 2-2. Resource limits

Resource

Soft limit

Hard limit

cputime

Unlimited

Unlimited

filesize

Unlimited

Unlimited

datasize

2 GB

2 GB

stacksize

8 KB

2 GB

coredumpsize

Unlimited

Unlimited

descriptors

64

1024

memorysize

Unlimited

Unlimited

Users can increase their own limits up to the hard limit, which can usually be modified by the superuser. Older systems may have different limits imposed by hardware constraints on the data size and stack size parameters (for example, sun4c architecture machines are limited to 512 MB and 256 MB, respectively).

You can place changes to the limits in the default profiles for user's shells. One of the most common changes are to prevent core dumps from occuring by setting the coredumpsize limit to zero:

csh% limit coredumpsize 0

Remember, enterprising users can always find a way around these restrictions.

2.2.4 Complex Environments

In some complex environments, user workload management becomes a very big deal. You may have hundreds, if not thousands, of processors scattered amongst anywhere from one machine to a vast cluster of single-CPU workstations. The problem shifts away from avoiding overloading and toward obtaining optimal utilization across such a wide field of systems. This is a very complex area, and one that commercial vendors are just starting to approach. The leading idea now seems to be computational grids, which enable the distribution of work across geographically disparate areas. While a detailed discussion of these issues is beyond the scope of this text, I would like to offer a few points of advice. Chief amongst these is not to be afraid of gluing together tools into something that works for you. Some Perl wrapped around the process-accounting mechanism can produce nice summary statements of usage on a per-application or per-user basis, with "usage charges" assessed depending on when applications were run and on what machines. You can also, with some care, create simple batch-queuing programs: a user submits their job to a controlling process, which finds an idle system given a certain range of parameters and starts jobs on appropriate systems (e.g., certain machines are workstations and if otherwise idle can only be used between 8 P.M. and 6 A.M., or certain jobs require large amounts of physical memory and should only be run on machines with more than 2 GB of memory). Commercial software exists to perform these sorts of tasks.


2.3 Benchmarking

A benchmark measures the performance of a standard workload. By holding the workload fixed, you can vary the underlying system parameters to generate relative performance numbers.

In general, there are three types of benchmarks: component-specific benchmarks, like SPECint and SPECfp for microprocessors; whole-system benchmarks designed to emulate commercial environments, like the TPC series and SPECweb99, and user-developed benchmarks. In this section, I touch on these different approaches to benchmarking, providing examples of real-world benchmarks throughout.

This book is not about tuning systems to produce world-record benchmark results. In general, producing a world record requires an immense amount of work, strong support from the engineers responsible for the underlying subsystems, and an understanding that the end result is a world record (which may not necessarily be directly applicable to a customer environment).

2.3.1 MIPS and Megaflops

Before we get started in a discussion of benchmarks, we need to discuss a little bit of terminology and history: the MIPS and megaflops metrics.

2.3.1.1 MIPS

One of the first ways of estimating the performance of an application on a computer system was to look at how fast the system could execute instructions. However, this is a dismal metric. The execution rate of a processor (typically expressed in millions of instructions per second, or MIPS) is strongly correlated to the clock rate of the processor. It's probably safe to assume that a 5-MIPS CPU will outperform a 100-MIPS CPU, but a 50-MIPS processor might well actually outperform a 100-MIPS CPU. Using MIPS to compare the performance of different computers is flawed in a basic way: a million instructions on one microprocessor might not accomplish the same work as the same number of instructions on another. For example, let's say you have a processor that does all floating-point work in software, and takes 50 integer instructions to perform 1 floating-point operation; your 50-MIPS CPU now performs 1 million useful instructions per second. Coupled with this is the fact that different instructions can take different lengths of time to complete, especially on CISC processors. Were those million instructions "do-nothing" operations, or floating-point multiplies?

To make things worse, some vendors have redefined the meaning of "MIP" as their own internal metric. DEC was particularly famous for this; for a long time, the VAX 11/780 was considered as the standard "1 MIP" machine. DEC publicly stated that, for a reasonable job mix, the 11/780 was capable of executing about 470,000 instructions per second -- less than half a MIP. For this reason, you sometimes hear people refer to "VAX MIPS" or "VUPs" (VAX units of processing) to make the comparison clear.

The key point to take away from this is that MIPS is only a useful metric to compare processors from the same vendor that support the exact same instruction set with the same compilers. The fundamental weaknesses in MIPS as a performance metric have often led people to joke that it's really short for "Meaningless Indicator of Processor Speed."

2.3.1.2 Megaflops

Floating-point performance is often measured in terms of millions of floating-point instructions per second, often called megaflops or MFLOPS. A floating-point instruction is commonly regarded as one that executes on a microprocessor's specialized floating-point hardware; this includes floating-point additions, multiplications, comparisons, and format conversions. Less commonly included are square roots and divisions. Typically, though, megaflop rates are computed from some mixture of additions and multiplications. While it is considered misleading to present MIPS values, vendors regularly quote peak MFLOPS values. The peak MFLOPS for a processor is usually on the close order of an integer multiple (1, 2, or 4) of the clock rate. This is because optimizations in microprocessor hardware perform fast multiply-add operations (such as for accumulating a running sum of vector elements, like in a dot product of two matrices). As microprocessors become faster and faster, the peak MFLOPS value becomes less useful as a reasonable upper bound on application floating-point performance: memory bandwidth (how fast data can be moved) in and out of the processor becomes limiting.

2.3.2 Component-Specific Benchmarks

The first set of benchmarks I discuss are designed to stress specific areas of the system; typically, they are memory performance and/or microprocessor performance. I discuss two of the most common: the Linpack benchmark and the SPECcpu suite.

2.3.2.1 Linpack

One of the first commonly accepted benchmarks is the Linpack benchmark, written by Jack Dongarra, then of Argonne National Laboratory. Interestingly, the goal of Linpack was not to develop a benchmark; rather, it was intended to provide a high-performance library of routines for linear algebra problems. The Linpack benchmark derives megaflop rates by measuring the time and number of operations required by a few of these routines to solve a dense system of linear equations using Gaussian elimination. There are several versions of Linpack, differentiated by the size of the system of linear equations they worked with, the numerical precision required, and the ground rules for the benchmark. The most typically-quoted was given by the 100 x 100 matrix with double-precision results and strictly compiled FORTRAN (no hand optimizations were permitted).

In addition to the 100 x 100 benchmark, there is also a 1000 x 1000 benchmark, in which the system of equations can be solved via any method of the vendor's choice. The larger Linpack is an important benchmark because it tends to give a reasonable upper bound for the floating-point performance of very large, highly optimized scientific problems on a given system.

The heart of the 100 x 100 Linpack benchmark is a routine called daxpy, which scales a vector by a constant and adds it to another vector:[7]

[7] Daxpy is actually written in FORTRAN; it is represented here in C.

for (i = 0; i <= N, i++) {

     dy[i] = dy[i] + da * dx[i];

}

Each iteration of this loop performs two floating-point operations (a multiplication and an addition) and three memory operations (two loads and a store). It turns out that the Linpack benchmark stressed both the floating-point and memory performance of computer systems up through the early 1990s. When microprocessor cache sizes increased to a size where they could hold the entire benchmark, the Linpack 100 x 100 ceased to be a useful benchmark -- the data structures for the entire benchmark are only 320 KB. The first system that was widely credited with "breaking" the Linpack 100 x 100 was IBM's RS/6000 series. Unfortunately, optimization techniques prevent the Linpack 100 x 100 benchmark from being scaled up to the point where they stress the memory subsystem again.

So, the Linpack 100 x 100 went from being the most prevalent single-number benchmark in history to being relegated to a historical footnote in about two years, as vendors scaled their caches to fit the entire benchmark.

2.3.2.2 SPECint and SPECfp

In the late 1980s, a group then called the Systems Performance Evaluation Cooperative, or SPEC, formed to establish, maintain, and endorse a standardized set of benchmarks.[8] The first SPEC microprocessor benchmark was called SPEC89, and was revised in 1992, 1995, and 2000. By releasing new versions, the SPEC organization keeps current with the latest trends in hardware and compiler design, preventing vendors from becoming too good at tuning their systems to the benchmarks. While the most commonly quoted SPEC benchmarks regard CPU performance, SPEC publishes many standard benchmarks. For more information on SPEC, consult http://www.spec.org.

[8] They are now the Standard Performance Evaluation Corporation.

The first SPEC benchmark was SPEC89, which consisted of the geometric mean of the runtimes for 10 programs. It did not differentiate between floating-point and integer performance, which had the side effect of cloaking performance subtleties. For example, a system that did very poorly on integer performance but very well in floating-point would have the same SPECmark score as a system that performed in exactly the opposite way. Interestingly, when SPEC89 was released, it was not a particularly important benchmark; the different competing architectures of the time had very different performance characteristics, and no benchmark could effectively relate all of their performances. In the field of high-performance computing, the Linpack 100 x 100 code was completely dominant.

To improve the quality of the benchmark, SPEC revised it and released the successor as SPEC92. SPEC92 consisted of 20 test programs, broken up as 6 integer and 14 floating-point codes. The benchmark also reports two summary figures, SPECint92 and SPECfp92, which measure integer and floating-point performance, respectively. As the Linpack 100 x 100 benchmark started to fall, SPECfp92 became the battleground metric for floating-point performance. Around 1994, large caches and smart compilers broke the SPECfp92 benchmark. Thankfully, in 1995, SPEC again released a revision to the benchmark suite. It increased the size of the benchmarks so that they wouldn't fit into caches, greatly lengthened runtime, and became more like large user applications. A similar process occured in 2000, when SPEC again rescaled the benchmark.

The applications under test in the SPECcpu2000 benchmark are quite representative of typical user codes, and are summarized in Table 2-3.

Table 2-3. Codes under test in SPECcpu2000

Class

Code

Language

Description

Integer

164.gzip

C

Data compression

175.vpr

C

FPGA circuit placement and routing

176.gcc

C

C compiler

181.mcf

C

Minimum-cost network flow solver

186.crafty

C

Chess

197.parser

C

Natural language parser

252.eon

C++

Ray tracer

253.perlbmk

C

Perl

254.gap

C

Computational group theory

255.vortex

C

Object-oriented database

256.bzip2

C

Data compression

300.twolf

C

Circuit placement/routing simulator

Floating-point

168.wupwise

F77

Physics (quantum chromodynamics)

171.swim

F77

Shallow water modeling

172.mgrid

F77

Multigrid solver: 3D potential field

173.applu

F77

Partial differential equation solver

177.mesa

C

3D graphics

178.galgel

F90

Computational fluid dynamics

179.art

C

Image recognition

183.equake

C

Seismic wave propagation simulation

187.facerec

F90

Face recognition image processing

188.ammp

C

Computational chemistry

189.lucas

C

Prime number testing

191.fma3d

F90

Finite-element crash simulation

200.sixtrack

F77

Physics (high-energy accelerator design)

301.apsi

F77

Meteorology (pollutant distribution)

The SPEC benchmark reports three numbers for both integer and floating-point performance: the pure metric, the base metric, and the rate metric.

  • The base metric (e.g., SPECint_base95) measures the performance under a restrictive set of rules: applications may only use four compiler flags, the flags must be identical for all applications, etc. The idea is that the base rate will more accurately represent the typically observed performance of user applications.

  • The rate metric (e.g., SPECfp_rate95) measures the throughput of an entire, possibly multiprocessor system. The vendor can run as many concurrent copies of the benchmark as necessary; the rate metric is the aggregate across the entire system.

2.3.3 Commercial Workload Benchmarks

Of course, not all applications are computationally bound; in fact, most business applications stress almost every part of the system. In recognition of this fact, benchmarks have been developed to measure more typical user environments. Here, I discuss two of the most common of these commercial workload benchmarks: the TPC series, which focuses on database performance, and the SPECweb benchmarks for web servers.

2.3.3.1 TPC

The Transaction Processing Council, or TPC (http://www.tpc.org), is an industry-based organization representing computer system and database vendors. TPC provides a framework for measuring the relative performance of these systems. Performance is measured by the amount of time it takes the computer to complete transactions under various loads. The TPC results are extremely well regarded in the database field. Like SPEC, TPC has had to modify and scale their benchmarks over time, as user needs and technological capabilities have changed. Consequently, TPC-A and TPC-B have fallen out of favor; TPC-C and TCP-D are the most common, and TPC-W is gaining favor as an e-commerce benchmark.

There are seven benchmarks in the TPC series:

  • TPC-A was issued in November 1989, and is designed to measure performance in update-intensive, terminal-driven database environments, such as those seen in online transaction processing (OLTP) scenarios. These environments typically have multiple terminal sessions, significant disk I/O, moderate system and application execution time, and transaction integrity requirements. TPC-A is obsolete.

  • TPC-B was issued in August of 1990. Unlike TPC-A, TPC-B is not an OLTP benchmark. Rather, TPC-B is a database stress test. It tends to emphasize disk I/O without having any remote terminal sessions. TPC-B is obsolete.

  • TPC-C was approved in July 1992. TPC-C is an OLTP benchmark like TPC-A, but is significantly more complex; it implements multiple transaction types and a more complex database. TPC-C simulates a complete computing environment, similar to that of a wholesale supplier, where a population of users executes transactions against a database. The benchmark is based around the principal transactions of an order-entry environment: entering and delivering orders, recording payments, checking order status, and monitoring the level of stock on hand.

  • TPC-D was introduced in December of 1995, and is designed to measure decision-support performance. The test essentially consists of a database that is being slowly modified online while other processes are performing large queries, joins, and scans. TPC-D is obsolete.

  • TPC-H is a decision-support benchmark that consists of a suite of business-oriented ad hoc queries and concurrent data modifications, much like TPC-D. This benchmark was designed to measure the performance of decision-support systems that examine large amounts of data, execute very complex queries, and give answers to critical business questions.

  • TPC-R is another decision-support benchmark. It is very similar to TPC-H, except that it allows additional optimizations based on advance knowledge of the queries.

  • TPC-W is a benchmark designed to measure transactional web e-commerce performance. The workload simulates an Internet commerce environment driven by a transactional web server. There are multiple concurrent browser sessions, dynamic page generation with associated database accesses and updates, consistent web objects, online transaction execution modes, transaction integrity, and contention on data access and update.

2.3.3.2 SPECweb99

As the Web began to explode in popularity in the mid-1990s, SPEC released a benchmark for web performance, called SPECweb96. SPECweb96 was focused entirely on the speed at which static pages could be retrieved by a set of clients. SPECweb96 was eventually broken by a software technique pioneered by Sun Microsystems, called the Network Cache Accelerator, or NCA. NCA uses a kernel module to transparently cache static web content in a kernel memory buffer, and replies to HTTP document requests for documents in its cache without ever waking up the application web server.

By the end of the 1990s, there was an increasing trend toward dynamic content -- the precise contents of a page depended on form data or was in some other way personalized to the end user. Driven by this change, SPEC released a new revision to the SPECweb benchmarks, which is known as SPECweb99.

SPECweb99 measures the number of concurrent conforming connections per second; in order for a connection to conform, it needs to sustain between 400 and 320 Kbps. The transaction mix includes both static and dynamic content; the default mix is shown in Table 2-4.

Table 2-4. SPECweb99 transaction mixture

Request type

Percentage

Static GET

70.00%

Dynamic GET (plain)

12.45%

Dynamic GET (custom advertisement)

12.60%

Dynamic POST

4.80%

Dynamic POST calling CGI code

0.15%

The static portion of the workload models a hypothetical web content provider. The pages of content are represented by files of different sizes that are accessed with varying frequency; SPEC obtained the precise values of these parameters by inspecting the log files of a handful of busy web sites. The working set of content files fit into four classes, as illustrated in Table 2-5.

Table 2-5. Static content classes

Class

Size range

Size step

Percentage of hits

0

< 1 KB

0.1 KB

35%

1

< 10 KB

1 KB

50%

2

< 100 KB

10 KB

14%

3

< 1000 KB

100 KB

1%

The workload file set consists of a number of directories, each of which contains 9 files per class, for a total of 36 files per directory. The files in Class 0 are in increments of 0.1 KB, those in Class 1 are in increments of 1 KB, etc. For example, the files for Class 2 will be 10 KB, 20 KB, 30 KB, 40 KB, 50 KB, 60 KB, 70 KB, 80 KB, and 90 KB. Since larger web servers are expected to service more files, the exact size of the workload file set (the number of directories to be created) is determined by the number of simultaneous connections.

During a benchmark test, the class of file to be retrieved from a random directory is determined by a fixed distribution (illustrated in Table 2-4). A Zipf distribution is used to choose which file should be retrieved from the selected class.[9] The relative hit rates for each file in a class are shown in Table 2-6.

[9] The probability of selecting the nth item in a Zipf distribution is proportional to 1/n. Zipf distributions are usually associated with situations where there are many equal-cost alternatives, such as picking books at a public library.

Table 2-6. Relative hit rates of files in a given class

File number

Percentage

0

3.9%

1

5.9%

2

8.8%

3

17.7%

4

35.3%

5

11.8%

6

7.1%

7

5.0%

8

4.4%

SPECweb99 is an emerging benchmark, and vendors are starting to play a lot of tricks to improve performance that you might not want to play on a production system. If you're trying to use SPECweb99 to evaluate systems for purchase, please take them with a grain of salt, and read the reporting configuration very carefully; vendors are required to report exactly what hardware was used and what parameters were tuned. [10]

[10] Note that the way you tune a system when you're going for a world record can be quite different than the way you'd tune a system that you had to use every day in production. Vendors take world records very seriously.

2.3.4 User Benchmarks

The most meaningful benchmarks are the ones you develop that are relevant to your environment. In general, there are a few principles that you should follow to set up a good benchmark: they should be relevant to your typical problems, have a relatively short runtime, be extremely automated, and set rules for running the benchmark.

2.3.4.1 Choose your problem set

In general, pick problems that represent the spectrum of work you actually do. If, for example, you are buying a computer for genetic sequence analysis, you might want to pick a few representative sequences, each with a separate computational code. It's okay if most of your work uses one particular application -- just weight the performance of that application higher. You should also design the benchmark so that it tests not only the work that you are doing now, but the work that you plan on doing over the lifetime of the machine (say, three to five years). Try and benchmark the big memory, high computation problems, or a reasonable subset thereof, that you'll be doing five years from now.

2.3.4.2 Choose your runtime

Of course, it's hard to predict runtime. One of the best guidelines for this is a well respected, industry-standard benchmark like SPECfp2000. If your current system is rated at 100 SPECfp2000, a system that's rated at 1000 SPECfp2000 should go ten times as fast (work with me, here). A benchmark that took 20 minutes to run on your current system would be reduced to a trivial 2-minute run on the new system. As a rule of thumb, each "problem" should run for 10 to 20 minutes on the new hardware under consideration. It should take less than an hour or two to run the entire suite, if for no other reason than you will have to sit down and run it many, many times.

2.3.4.3 Automate heavily

You, or a vendor trying to get you to give them money, will be running this benchmark frequently. As a consequence, you will want things to be as automated as possible. The best way to do this is to write a wrapper (often a shell script) that will do everything from compiling the benchmark, to running it, to printing a summary of the results, along with a snapshot of the most important elements of the system configuration. These elements might include things like the number of configured processors, the amount of memory installed, or the compilation flags used.

2.3.4.4 Set benchmark runtime rules

It is important to set rules, especially if anyone other than you will be running the benchmark. There are a lot of questions to keep in mind when writing benchmark rules: are modifications allowed? Are code changes only allowed for portability, or can algorithms be replaced? What type of compiler optimizations are allowed? How closely must the benchmark results match your baseline results? In fact, you probably need some rules about the rules, just to make sure you have a clear understanding with the vendor.

We've only touched on the highlights of benchmark design. I strongly recommend that you refer to Kevin Dowd and Charles Severance's High Performance Computing (O'Reilly), for a much more in depth discussion of this topic.


2.4 Concluding Thoughts

I hope this chapter has provided some theoretical and practical insight into understanding and managing workflows. I've discussed means of understanding the workload impressed upon a system, as well as some means of enforcing limits upon that workload. I have also talked about benchmarks as means of comparing the performance of systems under standardized workloads.

Parts of this chapter may seem almost childlike in their simplicity. As we progress into the performance analysis of dynamic systems, we are presented with simple things that are actually complex. Part of this book's purpose is to examine some of the simplifications, so that the reasons behind become apparent. This exploration lets you make better decisions about where to direct your tuning efforts.


Chapter 3. Processors

Quote performance in terms of processor utilization, parallel speed-ups, or MFLOPS per dollar.

—D. H. Bailey, "Twelve ways to fool the masses when giving performance results on parallel supercomputers." Supercomputing Review, 1991

The increase in microprocessor performance over the last 20 years has been phenomenal. The prediction made by Gordon Moore, the founder of Intel, that transistor counts on microprocessors would double every 18 months (and thus performance would increase proportionately), has certainly held true, if not been exceeded. In the early 1980s, the fastest microprocessor from Intel was the 4004, which was clocked at less than 4 KHz; by comparison, in mid-2001, the latest generations of IA-32 processors from Intel and AMD are running at clock rates of about 2.0 GHz -- and execute more than one instruction per clock cycle (I survey some the key elements in microprocessor design later in this chapter). This jump is so huge that it is hard to comprehend. If automobile performance followed the same growth curve as microprocessors over the last 20 years, the successor to the DeLorean DMC-12 (1981-1983), which had a top speed of perhaps 140 miles per hour,[1] would have doubled 12 times: it would travel at almost 287 thousand miles per hour, which is equivalent to about 410 times the speed of sound, or about one-sixteenth of the speed of light. Even that's pretty difficult to comprehend. If a pack of bubble gum cost a quarter in 1983, and prices increased at the same rate as microprocessor performance, that same pack of gum would cost just over a thousand dollars.

[1] The DeLorean DMC-12, despite being positioned to compete against the Corvette as an American sports car, was quite heavy, due to its extensive use of dairy-grade stainless steel.

It's not clear whether this rate of increase is sustainable. It has often been predicted that the rate will have to slow. However, enterprising engineers have found ways to improve the technologies used for processor fabrication and design so that this has not yet become the case.

The fact that microprocessor performance increases have far outpaced the performance increases of the other components in a computer system is a unifying theme for this book: how do we make the other components go as fast as possible, to try and optimize the efficiency of the processor? Processor performance is without a doubt the most-quoted figure in computer performance. This can easily be verified simply by going to the local computer store and looking at their advertisements; the information in the largest print, often set apart and highlighted, is the clock rate of the microprocessor. It is much more appealing to have the latest 2.0-GHz processor than a few very fast hard drives. However, while it is absolutely true that microprocessor performance is a critical component of overall system performance, it is often overrated.

Despite the central importance of the microprocessor, to most system administrators it is little more than a replaceable black ceramic box with a heat sink attached. While it's true that most systems administration tasks don't require any significant knowledge of electrical or computer engineering, performance tuning is at heart about understanding how things work -- it's very difficult to improve something if you don't have any idea how it works. For processor performance, this foundation is microprocessor design. In this chapter, I move from discussing basic microprocessor architecture to a host of supporting areas: caches, which play a vital role in the performance of modern processors; process scheduling, or how the operating system decides what processes should have priority in using the CPU; multiprocessing; the interconnects used to connect processors to other processors and to peripheral devices; and, finally, I discuss some tools that can be used to monitor microprocessor performance.


3.1 Microprocessor Architecture

At a fundamental level, a microprocessor is the physical implementation of a set of rules. These rules, called instructions, specify exactly what tasks the microprocessor is allowed to perform. The set of all the rules together is called the instruction set; in combination with other information, they define the instruction set architecture, or ISA, which contains all the information you'd need to write a program that runs correctly. For a more detailed discussion of these abstract underpinnings, see Chapter 1.

In the last 20 years, microprocessor designers have split into two competing camps based on design philosophies. One camp espoused creating an instruction set in which individual instructions were very powerful, almost on the level of C primitives. These instructions are quite complex, and this camp came to be known as building complex instruction set computing (CISC) designs. The other camp decided to create instruction sets in which the individual instructions were minimal, but their simplicity allowed many optimizations to be performed. This is known as reduced instruction set computing (RISC) processor design. Both methods accomplish the same amount of work. The difference is that a RISC design will probably take more instructions, but those instructions will most likely execute in less time.

RISC has won the war of performance. Increasingly, CISC processor designs are becoming more RISC-like. For example, Intel's P6 processor core actually translates the complex IA-32 instructions into a much simpler internal format before executing them. Another good example is the Transmeta Crusoe processor, which uses specialized hardware (called code morphing) to, in part, translate IA-32 instructions into the Crusoe's internal instructions.

Lately, designers have started to add extra instructions, typically for graphics support. Sun's VIS and Intel's MMX extensions are good examples of this.[2] These instructions are based around the idea that while the smallest addressable piece of memory is typically 32 bits long, we might only operate on 8 of those bits, so we perform 4 memory operations and 4 arithmetic operations when the data would have fit in 1 memory operation. So, these extensions work by packing the full 32 bits of data in, so that only 1 memory operation and only 1 arithmetic operation is required. However, using these extensions to maximum effect involves quite a bit of effort on the part of the compiler design team. Figure 3-1 shows multimedia extension parallelism.

[2] Intel's IA-32 architecture is, as we've said, a CISC design. MMX is still a good example of a graphics-specialized instruction set, however.

Figure 3-1. Multimedia extension parallelism

So, are RISC processors with these extensions added on (for example, Sun's UltraSPARC) still RISC? The important thing to remember is that RISC was a means to an end -- building the fastest microprocessor possible within specific technological constraints -- not an end in and of itself. Some people have taken to saying that these "second generation" RISC processors aren't actually RISC at all: instead they implement fast instruction set computing (FISC), which perhaps defines their intent a little more clearly.

3.1.1 Clock Rates

Let's imagine our microprocessor as an Athenian trireme[3] moving through the Mediterranean. In order for the ship to move forward at peak efficiency, all the rowers must row at the same pace: their actions are synchronized by the beat of a large drum. In much the same way, everything on a microprocessor happens in synchronization by means of an external clock signal. This signal, which is typically cycling millions of times per second (one hertz is one cycle per second; processor clock signals are generally counted in megahertz (MHz), or millions of hertz), is usually generated by a clock crystal on the processor's logic board. In large datacenter systems, the clock signal is sometimes generated on a dedicated clock board. This clock rate is increased by a set multiplier to produce the internal clock rate, which is the number that is typically counted. This multiplier determines how fast a given processor will perform in comparison to other similar processors.

[3] A trireme is a seafaring vessel, typically a warship, characterized by three rows of oars. The Greek city-state of Athens was known for its naval strength, much as Sparta was known for its army.

I can't emphasize enough how dangerous it is to make performance comparisons based only on clock rate. Table 3-1 is a comparison of the IBM PowerPC 604e microprocessor, which is used in their RS/6000 line to the MIPS R10000, which is used by SGI.

Table 3-1. Processor/clock rate performance comparison

Processor

Clock rate

SPECint95

SPECfp95

IBM PowerPC 604e

332 MHz

12.9

6.2

IBM PowerPC 604e

266 MHz

9.2

5.8

MIPS R10000

250 MHz

12.4

9.7

We'll talk more about the SPEC benchmark suite later (also see Section 2.3.2.2 in Chapter 2), but a gross analysis of the data is possible here. From a simple comparison of the clock rates, it would seem that the MIPS R10000 processor, at a mere 250 MHz, should be slightly slower than the 266 MHz PowerPC, and 25% slower than the 332 MHz part. However, the benchmarks tell a very different tale: the R10000 is almost as fast as the 332 MHz PowerPC in integer performance, and bests it by a substantial margin in floating-point computation.

The astute reader will have noticed that Table 3-1 also provides evidence that increasing clock rate does not imply an equivalent increase in performance. While the 332 MHz IBM PowerPC processor has an almost 25% higher clock rate than the 266 MHz version, it only has a slight (6.8%) edge in floating-point performance. This is because increasing the clock rate of the processor tends to accentuate the constant and substantial delay involved in accessing memory. This is particularly an effect of floating-point benchmarks, because they tend to be rather large.

The overall limit of the clock signal is dependent on some rather esoteric parameters in the processor design, such as the underlying semiconductor technology. While it is possible to optimize for clock speed by adjusting these parameters, the real-world characteristics -- such as cost, heat, and power consumption -- of the resulting design may make it infeasible for the marketplace. However, the clock rate also depends on the complexity of work being done. It takes a certain amount of time for the processor to execute a specific task. The slowest path to take through the processor is called the critical path. Reducing the critical path, such as by simplifying the design, lets the clock speed go ever higher.

Another way of thinking about the performance of a processor is to consider how many cycles it takes to execute an instruction, on average. This is called the cycles per instruction (CPI); obviously, lower numbers are better.

3.1.2 Pipelining

Since we can't always increase the performance of a processor simply by increasing the clock rate, we must be more devious in our approach. One method involves placing two functional units in the design; for example, instead of having just one unit that can perform integer arithmetic, we'll have two.[4] However, this is rather expensive in terms of processor die space.

[4] A processor that uses this technique is called superscalar. I discuss this in Section 3.1.3 later in this chapter.

Instead, we might look closer at how a microprocessor operates. While an instruction is being retrieved from memory, the unit responsible for integer arithmetic is sitting idle; surely this isn't the best way to make a fast processor! For example, we might design a processor that breaks up instructions into five stages:

Instruction fetch (IF)

Retrieve the next instruction to be executed from memory.

Instruction decode (ID)

Figure out what the instruction actually does.

Execute (EX)

Execute the instruction.

Memory (MEM)

Write data to and read data from memory.

Writeback (WB)

Write computed results into registers.

A single instruction might not use all of these stages -- for example, a memory load instruction might not have any need of the circuitry provided in the execute stage.[5] So, let's try overlapping the execution of the instructions.

[5] Then again, it might. This depends on the particulars of the processor design.

The first instruction is initially fetched. After it moves onto the decode stage, the next instruction is fetched. As the processor continues to work, it occupies all of itself in this fashion, so that in a given instant there are five instructions being executed at once. This process is called pipelining.

In a five-stage pipeline such as we've described, we ideally get the results of an instruction every clock cycle. However, this does not mean that each instruction takes only one cycle to execute; rather, each one takes five cycles. It's important to be careful about this when you're thinking about CPI measurements; in a pipelined processor operating perfectly, the CPI is 1.0. This just means that, on average, one instruction is completed every clock cycle. So, pipelining does not increase the performance of a single instruction; rather, it increases the throughput of the microprocessor. In addition, as we break each instruction up into a set of stages, we reduce the complexity of each stage. The more stages we have, the simpler we can make each one, which means that we can probably increase the clock rate of the processor as well. Figure 3-2 illustrates instructions in flight through a pipeline.

Figure 3-2. Instructions in flight through a pipeline

Pipelining is one of the most dominant factors in the historical improvement of processor performance. However, it is not quite so simple as we have made it out to be. Sometimes, an instruction might take a particularly long time; for example, reading data from memory might take substantially longer if the data to be read doesn't reside in a cache. This causes the pipeline to "gum up," which is called a stall. Microprocessor designers go to great lengths to avoid pipeline stalls; stalls have a very detrimental effect on performance.

3.1.2.1 Variable-length instructions

The smooth functioning of our pipeline is dependent on all our instructions taking approximately the same amount of time. In a RISC design, each instruction is generally the same amount of data. However, because of their complexity, CISC instructions often vary in length. For example, a "return from routine" instruction is pretty compact, but a complex "multiply-add" instruction might be a lot longer. Figure 3-3 illustrates the difference between fixed-length instructions and variable-length instructions.

Figure 3-3. Fixed-length and variable-length instructions

The processor has no idea how long an instruction will be until it decodes it and finds out what it is. Since the processor has to fetch the instruction before it can be decoded, if the instruction is longer than what was fetched, the processor will have to go back to the instruction fetch stage and recover the rest. This involves a costly trip to memory, and stalls the pipeline. If, as in most RISC instruction sets, the length of each instruction is fixed by definition, then this problem doesn't occur, and the pipeline can flow more smoothly.

3.1.2.2 Branches

Another problem in pipelining is instructions known as conditional branches. A conditional branch specifies the next instruction to be executed, depending on a specific computation. Consider the following code snippet:

if (i == j) {

   delta++;

}

The if statement here is an example of a conditional branch. These are implemented in most RISC instruction sets as branch if equal (BEQ) and branch if not equal (BNE) instructions.

When the processor retrieves an instruction from memory, it has no idea what the instruction is. So, it fetches the next instruction in memory. By the time the processor knows that the instruction is a branch, it has finished decoding it. If the branch falls through (that is, the next instruction is the direction the branch goes), everything works well. However, if it turns out that the branch is taken (that is, the next instruction executed is not the instruction that was fetched), the processor needs to go back and remove all the pending instructions in the pipeline and fetch the next instruction we want to execute.

Unfortunately, analysis has shown that branch instructions occur on average about every seven instructions. If only half of our branches fall through, we lose about 20% of the pipeline's efficiency. Rather than take a penalty for clearing out the pipeline, many RISC processors implement a branch delay slot. This means that the instruction that goes into the pipeline right after the branch is either harmless or useful, whichever way the branch goes. For example, consider the pseudocode below:

loop:

   c--;

   i = j++;

   j = k + 4;

   l = m + n;

if (c == 0) goto loop;

Because adding m and n and storing the result in l does not depend on any earlier instructions in the loop, we can reorganize the instructions like this in order to use a branch delay slot:

loop:

   c--;

   i = j++;

   j = k + 4;

if (c == 0) goto loop;

l = m + n;

By the time the processor has decided which way the branch should go, that instruction is about to be fetched. If we can't find a good instruction to fill the branch delay slot, we simply insert a no-operation (NOP) instruction. This is generally handled by the assembler and is not dependent on the programmer. This is a neat way to reduce the impact of branch operation-related processor stalls; however, as processors became capable of running more concurrent instructions, more powerful approaches were needed.

If we could decrease the branch miss rate further, the pipeline would be more efficient. With this in mind, processor designers devised schemes that allow the processor to predict the direction of a branch. As an instruction is decoded, the processor notices that it is a branch and consults a table of the recent operation of the branch. Based on this recent behaviour, it makes a guess and immediately begins fetching instructions from the guessed location. If the prediction is wrong, we have to cancel all our current instructions. One trivially simple mechanism of branch prediction is to always assume that the branch that refers to an earlier address is always taken. This is surprisingly accurate, because many programs iterate over a loop many times:

for (i = 0; i <= 100000; i++) {

   j[i] = k[i] + j[i];

}

If the branch prediction is correct, branches are no more expensive than any other instruction. If not, we must stall the processor, but hopefully this won't occur very frequently. A simple branch-prediction scheme is over 95% correct, which significantly reduces the overall penalty of branch-related pipeline stalls.

3.1.3 The Second Generation of RISC Processor Design

The ideal first-generation RISC processor would be able to execute one instruction per clock cycle. Although a five-stage pipeline would take five instructions to fill completely, if we kept it full, the overall CPI would start to grow close to 1.0. Once this was approached in advanced first-generation designs, microprocessor design companies began to compete to see who could build the fastest RISC processor. These strategies fell into three basic areas: speed freaks, superscalar designs, and superpipelined designs.

Speed freak designs are based on optimizing a simple, small, efficient core for very high clock rates. This approach is epitomized by the Digital Equipment Corporation (DEC) Alpha processor, which has historically exhibited clock rates approximately twice that of the nearest competitor.

By adding more computational elements to the processor die, performance can be increased without increasing the clock rate. This is known as superscalar processor design. The number and type of operations that can be run concurrently is dependent on both the application, which must have some degree of parallelism, and the processor, which must have enough functional units and a way of keeping them busy. This idea seems simple, but it is quite difficult, on the part of both the compiler writer and the hardware engineer. Current state-of-the-art superscalar processors can issue six instructions per clock cycle. At peak performance, they achieve a CPI of 0.17!

One variation on superscalar processing is called very long instruction word (VLIW) design. In such a scheme, the processor itself is very simple, and lacks almost all of the advanced scheduling hardware found in superscalar processors. In a VLIW processor, such as the Transmeta Crusoe, instructions are grouped together into bundles (instruction words) of three or four.[6] The processor then loads each instruction word in sequence and executes all of the instructions in parallel on multiple functional units. This causes an incredible burden to be placed on the shoulders of the compiler writers: writing an optimizing compiler for a VLIW processor is exceptionally hard, and is still an area of active research. The Crusoe design lightens this burden by implementing what amounts to an on-the-fly recompiler in hardware.

[6] In the early VLIW processors built by Multiflow Computer, as many as 28 operations could be bundled in a single instruction word.

In RISC processors, simpler often means faster; perhaps some of the instruction steps in your pipeline are overly complex. The hallmark of a superpipelined design is to reduce per-step complexity by increasing the number of steps. If the reduced complexity allows the processor to execute faster, superpipelined processors can achieve the same performance improvements as superscalar processors. However, deep pipelines tend to mean that branches cannot be resolved until relatively late in the pipeline, which makes mispredicted branch stalls very expensive. Superpipelining can also be combined with other approaches, as is evidenced by the MIPS R8000, which was a superscalar design with a pair of deep pipelines.

Designing a fast computer is more than just a function of a fast microprocessor, however. Removing the bottlenecks in processing exposes more bottlenecks, particularly in the way that information is moved in and out of the CPU.


3.2 Caching

A major concern of our microprocessor optimization strategy is that pipeline stalls cause serious performance problems. If every memory request has to be satisfied from main memory, the pipeline will stall very frequently. Modern microprocessors universally implement caches to try and solve this problem. In some processors, this cache is organized as a single unit, called a unified cache architecture. More commonly, however, the cache is divided up into two areas: one for data and one for instructions. This is called a Harvard cache architecture. Harvard caches tend to be significantly more efficient in the real world, especially for small caches.

3.2.1 The Cache Hierarchy

Memory speed has increased at a much slower rate than processor speed: processor performance doubles roughly every eighteen months, but memory performance doubles roughly every seven years. The important thing to realize is that as processor speed has outpaced memory speed, even static RAM caches located off the processor die are not fast enough.[7] The total time required to move data to or from memory is more than just the memory's physical access time, as there is overhead in moving data between the chips, as well as keeping the caches synchronized with main memory. The most common mechanism for minimizing access time is to create a hierarchy of caches with steadily increasing size and steadily decreasing performance:

[7] We'll talk about the various types of memory in greater detail in Chapter 4.

  • The processor has separate caches for instructions and data; they are referred to as the i-cache and the d-cache, respectively. These are often about 16 KB each, and the fastest caches in the system. They are called level 1 or L1 caches.

  • There is a larger, combined data/instruction second-level cache, which is generally between 256 KB and 4 MB in size. A cache miss at the L1 cache can still produce a cache hit in this cache -- while it takes a few cycles longer it is still very efficient. This is the level 2 or L2 cache.

  • Sometimes a third-level cache is configured. It is usually 4 to 32 MB, and while it is still slower than the L2 cache, it is roughly an order of magnitude faster than main memory.

  • If cache misses occur at all these levels, main memory must be referenced. If the memory system is very loaded, there could be a long delay. We want to avoid that.

Most extremely high clock rate microprocessors have developed a three-level cache structure. An example of cache levels and their access times for a typical 1.0 GHz processor can be found in Table 3-2.

Table 3-2. Typical memory access speeds

Cache location and type

Access time (in ns)

Size

"Good" hit rate[8]

L1 on-chip

1-3

8-64 KB

>90%

L2 on-chip

6-18

1-8 MB

>50%

L3 off-chip

30-40

8-32 MB

>30%

Main memory

220

Very large

-

[8] This is best described as "the percentage of requests to the cache that can be fulfilled with data already in the cache."

Every time data is loaded from any lower cache level, it's loaded into the caches above it as well. When a reference to memory is located in a cache, we say we have a cache hit; when we have to "fall through" the cache and go look in the next level of the hierarchy, we have incurred a cache miss. Every cache miss incurs a miss cost. Predicting the miss cost is often complicated: newer processor designs like Sun's UltraSPARC have variable cache miss costs. In general, the miss cost is proportional to the difference in speed between the processor clock rate, the cache clock rate, and the main memory speed. It is very good from a performance viewpoint to hit the cache as often as possible. Let's look at the average memory-access time of our prototype system based on two distinct scenarios:

  • We hit the L1 cache 65% of the time, the L2 cache 25% of the time, and main memory 10% of the time.

    0.65(4ns)+0.25(5ns)+0.10(220ns) = 25.85ns

  • We hit the L1 cache 90% of the time, the L2 cache 8% of the time, and main memory 2% of the time.

    0.90(4ns)+0.08(5ns)+0.02(220ns) = 8.4ns

While 8.4 nanoseconds sounds awfully good, it's important to keep in mind that the clock cycle time of this system is on the order of a nanosecond. Nonetheless, as should be apparent, it is important for the cache hits to happen as much as possible and as high up in the hierarchy as fast as possible (Figure 3-4).

Figure 3-4. Performance as a function of L1 cache hit rate

3.2.2 Cache Organization and Operation

Caches are organized into equal-sized chunks called lines. A line typically contains four to sixteen sequential memory locations: although the data in a line is physically contiguous in main memory, two distinct cache lines can contain data that is quite separated in memory.

When data is requested from the cache, the system checks to see if the data is in one of the cache lines. If it is, then it is returned quickly; if not, the request falls through the next cache level (or to main memory). When the data is retrieved from the lower cache level, it is pulled into a line in the caches, discarding an existing line if necessary.

It is important to note that the data in a cache is not the "master copy" of the data. So, when data is modified in a cache, systems administrators have three choices:

  • Update the copy in the cache only, which is very fast. However, now the master copy and the cached copy differ. This is write-back behavior.

  • Update both copies immediately, which may be slow. This is efficient if the cost of writing to the cache is low. This is called write-through.

  • Throw away the cached copy and just update the master. This is only a good idea if the cost of writing to the cache is high.

Not all of main memory is cacheable: typically regions that involve I/O devices are uncacheable. This is determined by the address range, not by the data or instructions stored in the memory location.

3.2.3 Associativity

Every cache line is paired with the canonical data stored in main memory in a process called mapping. The cache maintains a record for each line, called a tag, of what memory addresses the cache line is associated with; the last access time is also stored in some designs. These maps can be constructed in three different ways:

Direct mapping

This is the most straightforward way of mapping main memory to cache lines. In this scheme, cache line zero stores integer multiples of the cache size. For a 32 KB cache, the memory addresses that were 0 modulo 32 KB (0, 32 KB, 64 KB, etc.) would be mapped against the first cache line. This is a cheap and simple method to implement, but it can cause decreased performance (called thrashing) when working with two memory areas that are an exact integer multiple of the cache size apart. For example, when copying the contents of an array A into an array B, and A and B are exactly 32 KB apart, A will be read, then B has to be pulled into the same cache line to be written; this incurs a lot of cache misses very quickly.

Fully associative mapping

Fully associative mapping allows any cache line to be mapped against any memory address. Unlike a direct mapping implementation, each cache line in a fully associative cache knows something about the data it contains. When the processor requests data, each of the cache lines is queried as to whether it contains the data. If none of them have it, then a cache miss occurs. The cache lines track when the last access time of the data occurred, so that when a line needs to be overwritten, the oldest data can be discarded. Fully associative caches tend to have high utilization percentages, but are very expensive.

Set associative caching

A set associative cache is a compromise between the direct-mapping and fully-associative methods. They consist of a grouping of two or four banks of direct-mapped cache. Every time a cache line needs to be replaced, there is more than one choice of where it can go. This prevents the cache thrashing problems that are observed in direct-mapped caching.

3.2.4 Locality and "Cache-Busters"

Caches are so successful at what they do because programs exhibit peculiar memory access characteristics, which are called temporal locality and spatial locality. This means that, after an instruction or data access, the next instruction executed or the next data bit fetched tends to be close in either space or time. Temporal locality depends on reusing something repeatedly; spatial locality depends on using things at the same time that are close to each other.

We call a piece of code that causes the cache functionality to fail a cache-buster. Typically, a cache-buster breaks the temporal/spatial locality that the cache relies on for efficient operation. Three of the most common are non-unit stride, linked lists, and cache-aligned block copies.

3.2.4.1 Unit stride

Caches work best when a program is working through memory sequentially. Let's say that we are using short (16-bit) integers, and our cache lines are 128 bits in length. When we reference the first integer, the next seven will be loaded into the cache line as well, and our next seven references will be satisfied very quickly. This is called unit stride, because the address of each successive data reference is one greater than the data reference previous. The following code exhibits unit stride:

for (i = 0; i <= 100000; i++) {

   j = j + Array[i];

}

If a program exhibits non-unit stride, its performance suffers because each memory reference requires a trip to main memory. This is an example of a non-unit stride loop:

for (i = 0; i <= 100000; i += 16) {

   j = j + Array[i];

}

Even though this code performs one-sixteenth of the work of the unit stride loop, the two loops perform identically! Noncached memory accesses dominate the performance of the non-unit stride loop. For more examples of loop optimization, see Section 8.3 in Chapter 8.

3.2.4.2 Linked lists

Let's think about a linked list with a few thousand entries. Each entry consists of some data and a link to the next element. We search our list by traversal: because the code for searching the list can be made very compact, it fits well in the instruction cache. However, every data request will be in a different cache line, and so every data access incurs a cache miss. If the size of the linked list exceeds our cache size and we are forced to start flushing cache lines to make room for the next element in the list, our next search attempt will not find the start of the list in the cache! The key point is that list elements are rarely congruent in memory; therefore, iterating over them breaks the spatial locality principle that our caches depend on.

The only solution to this problem is to devise a new algorithm. As processors get faster, the problem will only get worse, since the processor/cache clock rate and the memory access time disparity will only grow larger.

3.2.4.3 Cache-aligned block copy problems

The kernel spends much of its time copying and zeroing blocks of data, particularly in page-sized (typically 4 or 8 KB) chunks.[9] This data does not need to be cached, since it probably won't be used again immediately; in fact, it will simply serve to remove useful data from the cache. The most efficient way to copy a page-sized piece of data with write-back caching is to read a cache line, then write it as a block, as shown in the following steps:

[9] I discuss pages in Chapter 4.

  1. Read the first piece of the source, incurring a cache miss.

  2. Read the entire cache line of the source.

  3. Write the first piece to the target, incurring a cache miss.

  4. Write the entire cache line to the target.

  5. Repeat until finished. At some later stage, the target cache line will be flushed for more useful data.

If the target and source are separated by an exact multiple of the cache size, and the system uses a direct-mapped cache, both the source and destination will use the same cache line, which will cause a read and write miss for each entry in the line. This is horribly inefficient. Here's an example:

#define CACHESIZE 0x40000

char source[CACHESIZE], target[CACHESIZE];

for (i = 0; i < CACHESIZE; i++) {

   source[i] = target[i];

}

Support exists in processors to accelerate block copying. Sun's VIS instruction set implemented in the UltraSPARC-series processors has a block-copy instruction that bypasses the caches, while still maintaining consistency between the caches and main memory. The bcopy(3C) and memcpy(3C) function calls both use this mechanism on VIS-enabled processors automatically.

3.2.5 The Cache Size Anomaly

Based on what we have said so far, the general rule of "increased cache means increased performance" holds. If a piece of code operates in a cache-busting fashion, it may be that the code performs much faster without a processor cache at all! Even so, sometimes very strange performance results are observed, like a SPARCstation 5 Model 110 outperforming a SPARCstation 20 Model 85 in a large simulation. Why is this the case?

The SPARCstation 5 uses a microSPARC processor that was designed for cheap, efficient operation. It has a small cache, a high cache miss rate, but a very low cache miss cost. The SuperSPARC-II processor used in the SPARCstation 20 has a much larger cache, a much lower cache miss rate, and a much higher cache miss cost. In some applications (such as database processing), the large cache works efficiently, and the SPARCstation 20 is much faster, but in a simulation where the cache is not large enough to help performance at all, the low cache miss cost of the SPARCstation 5 makes it an extremely fast performer. The UltraSPARC design incorporated many of these features, which has a very low latency to main memory.

While I've given an example that uses older hardware, the potential for this sort of problem comes up quite frequently. For example, Celeron processors sometimes outperform their "professional" Pentium-II counterparts, because of a reduced cache miss cost. Some in-depth testing may be necessary to determine exactly what the cache access patterns of your applications are.

Another influence on cache effectiveness is something called page coloring, which refers to the algorithm used by the operating system kernel to decide where pages should be placed in the processor caches. For more information on this topic, see Section 4.2.6 in Chapter 4.


3.3 Process Scheduling

Another factor that affects processor performance is process scheduling. Process scheduling is how the operating system determines which process to run on the CPU. The next four sections look at different process scheduling models used in the Unix world.

3.3.1 The System V Model: The Linux Model

Linux implements a relatively simple, standard Unix scheduling model. It supports process preemption (interrupting a running process with a higher priority process), although its kernel is nonpreemptable.

The Linux kernel scheduling algorithm works by breaking the CPU time into units called epochs. In a given epoch, each process has a specified time quantum whose duration is computed when the epoch begins; therefore, each process can have a different quantum. This quantum represents the most CPU time that is consumable by that process during that epoch. When a process has exhausted its quantum, it is removed from the processor and replaced by another runnable process; of course, a process can be selected by the scheduler several times, as long as its quantum has not been exhausted (for example, it might have voluntarily relinquished the processor to wait for some disk I/O activity to complete).

When the scheduler is deciding what process to run next, it considers the priority of each process. There are two types of priority: static priority and dynamic priority. Static priority is assigned by users to real-time processes, and is never adjusted. Dynamic priority applies only to conventional processes, and is the sum of the base time quantum (also known as the base priority of the process) and the number of ticks of CPU time left to the process before its quantum expires in the current epoch. The static priority of a real-time process is always higher than any dynamic priority: the scheduler will only run a conventional process if there are no runnable real-time processes.

The epoch ends when all runnable processes have exhausted their quantum; the scheduler then recomputes the time quantum durations for all processes and a new epoch begins. The default quantum for each process, if it exhausted its quantum in the prior epoch, is the process's base time quantum. A new process inherits the base time quantum of its parent. Since process 0 (the swapper) has a default base time quantum of 20 ticks (about 200 milliseconds), each process by default has a base time quantum of equal size. The base time quantum value for a specific process is adjustable by the nice(3C) and setpriority(3C) system calls, as well as the nice command.

3.3.1.1 Finding a process's priority

Determining the priority of a running process can be done by observing the PRI column of the output of ps -el, as illustrated in Example 3-1.

Example 3-1. Finding a process's priority in Linux
% ps -el

  F S   UID   PID  PPID  C PRI  NI ADDR    SZ WCHAN  TTY          TIME CMD

100 S     0     1     0  0  60   0    -   326 do_sel ?        00:00:48 init

040 S     0     2     1  0  69   0    -     0 contex ?        00:00:00 keventd

040 S     0     3     1  0  69   0    -     0 kswapd ?        00:05:32 kswapd

040 S     0     4     1  0  69   0    -     0 krecla ?        00:00:00 kreclaimd

040 S     0     5     1  0  69   0    -     0 bdflus ?        00:24:43 bdflush

040 S     0     6     1  0  69   0    -     0 kupdat ?        00:23:01 kupdate

...

140 S     0 11602   488  0  68   0    -   745 tcp_da ?        00:00:00 sendmail

140 S     0 11727   488  0  68   0    -   745 tcp_da ?        00:00:00 sendmail

140 S     0 11793   488  0  69   0    -   745 tcp_da ?        00:00:00 sendmail

000 R   563 11877  5019  0  75   0    -   759 -      pts/3    00:00:00 ps
3.3.1.2 Adjusting a process's effective priority

Adjusting a process's effective priority is accomplished in a straightforward manner via the renice command. The renice command changes the absolute "niceness" value of the process. This value can range from +20 (the affected processes will only run when the system is otherwise idle) to -20 (the affected processes will be scheduled with priority). The default value is zero.

For example, let's look at a certain process (rhof ):

% ps -el |grep rhof

  F S   UID   PID  PPID  C PRI  NI ADDR    SZ WCHAN  TTY          TIME CMD

000 S   563  5620  5601  0  69   0    -   514 do_sel pts/6    00:00:00 rhof

Let's decrease the priority of this process to -20:

% renice -20 5620

5620: old priority 0, new priority 20

% ps -el |grep rhof

  F S   UID   PID  PPID  C PRI  NI ADDR    SZ WCHAN  TTY          TIME CMD

000 S   563  5620  5601  0  69  19    -   514 do_sel pts/6    00:00:00 rhof

Notice that the NI column value has become 19, reflecting the process's decreased priority. Nonroot users cannot increase the priority for their own processes, even if they were the ones who decreased them in the first place:

% ps -el |grep rhof

  F S   UID   PID  PPID  C PRI  NI ADDR    SZ WCHAN  TTY          TIME CMD

000 S   563  5865  5601  0  70   0    -   514 do_sel pts/6    00:00:00 rhof

% renice +5 5865

5865: old priority 0, new priority 5

% ps -el |grep rhof

  F S   UID   PID  PPID  C PRI  NI ADDR    SZ WCHAN  TTY          TIME CMD

000 S   563  5865  5601  0  71   5    -   514 do_sel pts/6    00:00:00 rhof

% renice +3 5865

renice: 5865: setpriority: Permission denied

In order to decrease the process's priority, we need to be root:

% sudo renice -20 5865

5865: old priority 0, new priority -20

% ps -el |grep rhof

  F S   UID   PID  PPID  C PRI  NI ADDR    SZ WCHAN  TTY          TIME CMD

000 S  1000  3274  3240  0  59 -20     -  1619 select pts/1    00:00:00 rhof

This provides a simple, easy way to adjust the priority of a running process.

3.3.1.3 Modifications for SMP systems

In a multiprocessor system, it is possible for an interesting conundrum to arise. Let's say there are two processors: one of them is idle, and the other is executing a process at relatively low priority. On the busy processor, a high priority process becomes executable. Should we execute the high priority process immediately on the idle CPU, in which case we have to do some work to migrate the cache lines for the process between processors, or do we defer it to run on the existing CPU? Linux addresses this issue by adopting an empirical rule related to the processor's cache size: the larger the processor's cache, the longer a process will wait for a piece of time on that processor.

3.3.2 Multilayered Scheduling Classes: The Solaris Model

One of the very nice features of the Solaris operating environment is the ability for administrators to control how the operating system schedules processes for execution. This allows fine-grained control over what processes are running at what priorities. One fundamental thing to keep in mind is that Solaris does not directly schedule processes; rather, it schedules the underlying kernel threads.

3.3.2.1 The Solaris threading model

Before we can continue our discussion of process scheduling, we need to discuss exactly what a kernel thread is. Solaris implements a multilayer threading model, which is intended to decouple the management of user-level threads from the kernel. User-level threads have their own priority scheme, and are scheduled by a scheduling thread that is created by the threading library when a multithreaded application is compiled and executed. This allows multithreaded applications to spawn thousands of threads without significantly loading the kernel; in fact, user threads aren't made visible to the kernel until they are bound to a lightweight process, or LWP, which has some state in the operating system. The scheduling thread is responsible for mapping the user threads into the LWP, which is associated with a kernel thread for execution on a processor.[10] Every LWP has a kernel thread, but every kernel thread need not have an LWP: some kernel threads are used exclusively by the operating system, and thus an LWP is not required.

[10] A nonthreaded application has one LWP and one kernel thread associated with it.

Periodically, some literature indicates that kernel threads and LWPs are the same thing. The truth is that they are distinctly different, but in the discussion of user processes, it's acceptable to think of them as one large entity. There is always, without exception, a one-to-one relationship between LWPs and kernel threads.

For the rest of our discussion of scheduling, the word "thread" will refer to kernel threads specifically.

3.3.2.2 Scheduling classes

Threads generally fall into five categories for scheduling: timesharing (ts), interactive (ia), kernel, real-time (rt), and interrupt. The relative priorities in each of these classes can be mapped into absolute priorities, as is summarized in Table 3-3.

Table 3-3. Process priority table layout in Solaris

Absolute priority range

Relative priority range

Scheduling queue

169-160 (or 109-100)

9-0

Interrupt threads (if real-time scheduling class is not loaded)

159-100

59-0

Real-time priorities

99-60

39-0

Kernel priorities

59-0

59-0

Timesharing and interactive priorities

The timeshare class is the default class that processes are scheduled in. However, the interactive class exists to improve interactive performance on windowing systems. This is accomplished by giving the foreground processes in the active window a higher priority. The processes that are chosen to be in the interactive class are those that come up under the windowing environment; remote processes are still scheduled in the timesharing class. However, the interactive and timesharing scheduling classes overlap in priority, so an interactive process at priority 59 is running at the same priority as a timesharing process at priority 59.

Let's discuss how the operating system decides the threads that should be run.

Notes on Real-Time Scheduling

Real-time scheduling refers to the means by which a process can be assured of having a slice of the processor when it needs it. This is a requirement for certain types of applications, such as robotics and flight control. In general, operating system support for real-time scheduling falls into three categories: none whatsoever, hard real-time support (in which all hard deadlines for applications are met), or soft real-time support (in which deadlines may be missed, but this is kept to a statistical minimum). Solaris is a soft real-time environment.

In a soft real-time environment, the response time must be finitely bounded (e.g., there must be an upper bound on the time during which the application can be off the processor -- in Solaris, this may be as long as five milliseconds). Unix environments typically do not implement real-time support because of two problems. First, page faults can induce unpredictable latency (see Section 4.3). Solaris solves this problem by locking all the pages of a real-time process in memory. The other problem is that in most Unix systems, the kernel is not preemptable, meaning that it can be removed from the processor and another process run. There are certain points at which the kernel cannot be preempted -- these are called nonpreemption points. Only special cases are nonpreemptable: the most common nonpreemptable case is when the kernel is executing in interrupt context. However, a higher priority interrupt can displace an interrupt that is in the process of being serviced.

3.3.2.3 The dispatcher

When a thread is ready to run, it is placed on a dispatch queue. Every processor has a separate set of dispatch queues. This state (waiting on the dispatch queue) is known as the RUN state. When a processor becomes free, the dispatcher takes the thread with the highest priority from the dispatch queue for that processor, and places it on the processor. The thread is then in the ONPROC state, and can perform computation.

A thread can leave the processor by several mechanisms. The first is that it performs computation until it reaches a time limit (determined by its priority); this is called the quantum. The quantum is usually inversely related to the priority (higher priority processes have smaller quanta), since higher priority processes are scheduled more frequently. When this happens, the thread's priority is set to a new level, which is referred to as the tqexp value. This level is almost always lower than the thread's priority when it started; this is done to ensure "fair share" scheduling.

Another way for the thread to leave the processor is if it blocks; for example, by waiting for an I/O to complete or for a lock to be released. This causes the thread to go into the SLEEP state. Threads that go to sleep while holding "critical" resources (such as reader/writer locks) are assigned a priority in the kernel class, so that they will have a chance to release their locks quickly. When processes on the sleep queues are returned to the processor, they have their priority set to a new value, called slpret. This is usually higher than the thread's earlier priority, so that the thread gets some execution time fairly quickly after it's gone to sleep.

The third and final way for a thread to leave the processor is if it is forcibly required to, usually by the kernel preempting the thread. If a preempted thread accrues enough time (determined by the maxwait parameter) of being starved for the processor, its priority is increased to the value specified by the lwait parameter. This provides a way of compensating threads that have been waiting on the dispatch queues for a long time after being preempted.

3.3.2.4 Checking a process's priority

You can find out what lightweight process and scheduling class a specific process is associated with by using the -L and -c switches, respectively, with /usr/bin/ps, as shown in Example 3-2.

Example 3-2. Viewing a process's priority in Solaris
% /usr/bin/ps -efcL

     UID   PID  PPID   LWP  NLWP  CLS PRI    STIME TTY     LTIME CMD

    root     0     0     1     1  SYS  96   Mar 20 ?        0:00 sched

    root     1     0     1     1   TS  58   Mar 20 ?        0:06 /etc/init -

    root     2     0     1     1  SYS  98   Mar 20 ?        0:00 pageout

    root     3     0     1     1  SYS  60   Mar 20 ?       22:37 fsflush

...

    root   200     1     1     7   TS  51   Mar 20 ?        0:00 /usr/sbin/nscd

    root   200     1     2     7   TS  58   Mar 20 ?        0:00 /usr/sbin/nscd

    root   200     1     3     7   TS  59   Mar 20 ?        0:00 /usr/sbin/nscd

    root   200     1     4     7   TS  58   Mar 20 ?        0:00 /usr/sbin/nscd

    root   200     1     5     7   TS  59   Mar 20 ?        0:00 /usr/sbin/nscd

    root   200     1     6     7   TS  59   Mar 20 ?        0:00 /usr/sbin/nscd

    root   200     1     7     7   TS  58   Mar 20 ?        0:01 /usr/sbin/nscd

    root   185     1     1     1   TS  48   Mar 20 ?        0:01 /usr/sbin/cron

...

    root  4897   164     1     1   TS  38 20:28:02 ?        0:00 in.telnetd

    root  5235  4899     1     1   TS  48 21:10:19 pts/1    0:00 /usr/bin/ps -cefL

     gdm  4899  4897     1     1   TS  58 20:28:02 pts/1    0:00 -csh

I have significantly trimmed the output of this command. Of particular interest here are the LWP, CLS, and PRI columns, which represent the lightweight process number, scheduling class, and priority, respectively. Of definite interest is the fact that multithreaded processes (for example, the name service cache daemon, /usr/sbin/nscd ) can exist on multiple LWPs concurrently (the number of LWPs a process is associated with is given by the NWLP column).

3.3.2.5 Tuning the dispatch tables

The dispatch tables are tunable via the dispadmin command. This is accomplished by writing the dispatch table for a specific class to a file, editing the file, and then reloading the file into the dispatch table.

To see what's available for editing, you can display the list of active scheduling classes by dispadmin -l:

% dispadmin -l

CONFIGURED CLASSES

==================

SYS     (System Class)

TS      (Time Sharing)

RT      (Real Time)

IA      (Interactive)

You can then dump the scheduling table by using dispadmin -c class -g:

% dispadmin -c TS -g

# Time Sharing Dispatcher Configuration

RES=1000

# ts_quantum  ts_tqexp  ts_slpret  ts_maxwait ts_lwait  PRIORITY LEVEL

       200         0        50           0        50        #     0

       200         0        50           0        50        #     1

       200         0        50           0        50        #     2

       200         0        50           0        50        #     3

...

       200         0        50           0        50        #     9

       160         0        51           0        51        #    10

       160         1        51           0        51        #    11

       160         2        51           0        51        #    12

       160         3        51           0        51        #    13

       160         4        51           0        51        #    14

       160         5        51           0        51        #    15

...

        40        42        58           0        59        #    52

        40        43        58           0        59        #    53

        40        44        58           0        59        #    54

        40        45        58           0        59        #    55

        40        46        58           0        59        #    56

        40        47        58           0        59        #    57

        40        48        58           0        59        #    58

        20        49        59       32000        59        #    59

The output of this table is significantly trimmed. Let's walk through it piece by piece.

The first line specifies the resolution of the time figures in the table below. The reciprocal of the resolution value is used to interpret the ts_quantum column in the dispatch table: as the value of RES increases, the ts_quantum values increase correspondingly. By default, RES is 1000. The reciprocal of RES is 0.001, which in fractions of a second represents one millisecond. By default, then, the units for the ts_quantum field are milliseconds. While the resolution value is changeable, it is usually advisable to leave it at the default.

I've already described the meanings of most of these columns in the discussions of how the dispatcher works (see Section 3.3.2.3 earlier in this chapter). I'll briefly summarize them here:

  • The ts_quantum field represents the maximum amount of time the thread can remain on the processor.

  • The ts_tqexp field specifies the priority the thread will be assigned following its removal from the processor after it uses its entire time quantum.

  • The ts_slpret field gives the priority a process will have after it returns to execution after going to sleep (e.g., waiting for an I/O to complete).

  • The ts_maxwait and ts_lswait fields control the compensation of a process (set to the priority given by ts_lswait) that has been preempted by a kernel-priority thread for a long period of time (greater than ts_maxwait).

Some features are of particular interest: the first is that the quantum values decrease correspondingly with the priority level. Another is that the maxwait value for priority level 59 is set very high; this ensures that every thread has a fair chance at being scheduled.

Because each priority class has a zero value in ts_maxwait for every priority except the highest level, all but the highest priority timesharing threads will have their priorities readjusted into the 50-59 range once a second or so. This has the effect of not penalizing a thread that is CPU-bound for an extended period. Threads that are CPU hogs will, over time, end up spending most of their time in the 0-9 priority range, as they will keep using up their time quantum and suffer from ts_tqexp priority readjustment. Once every second, they'll be bumped up again, but will only migrate back down if they continue to sustain their CPU-intensive behavior.

The real-time (RT) class has a much simpler table:

# Real Time Dispatcher Configuration

RES=1000

# TIME QUANTUM                    PRIORITY

# (rt_quantum)                      LEVEL

      1000                    #        0

      1000                    #        1

      1000                    #        2

      1000                    #        3

      1000                    #        4

...

       100                    #       55

       100                    #       56

       100                    #       57

       100                    #       58

       100                    #       59

This is simpler because the real-time thread scheduling does not involve any of the complex adjustments described previously. Real-time threads run until they stop or until their quantum expires (in case multiple real-time threads are pending). Also, note that even though the priority level range is from zero to 59, the real-time threads are scheduled at a higher priority than the kernel, as these relative priorities are mapped to absolute priorities via the map discussed previously.

You can tweak the dispatch tables by writing the output of dispadmin to a file (as shown), editing the file, and then loading it by running dispadmin -c class -s file.

The most common technique employed here is to fence off the top timesharing class, so that high priority CPU-intensive jobs run without being forced down to lower priority levels. You can accomplish this by editing the timesharing class dispatch table to look something like what's shown in Example 3-3.

Example 3-3. Creating a priority fence in the timesharing class
# Time Sharing Dispatcher Configuration

RES=1000

# ts_quantum  ts_tqexp  ts_slpret  ts_maxwait ts_lwait  PRIORITY LEVEL

...

        40        36        57           0        57        #    46

        40        37        58           0        58        #    47

        40        38        58           0        58        #    48

        40        39        58           0        58        #    49

        40        40        58           0        58        #    50

        40        41        58           0        58        #    51

        40        42        58           0        58        #    52

        40        43        58           0        58        #    53

        40        44        58           0        58        #    54

        40        45        58           0        58        #    55

        40        46        58           0        58        #    56

        40        47        58           0        58        #    57

        20        48        58       32000        58        #    58

       200        49        59           0        59        #    59

I have highlighted changed values. The net effect is that no process can naturally find its way into priority level 59 (this is why the ts_lwait parameter has been adjusted). Priority level 58 now functions the same as priority level 59: it has an unusually low quantum and a high value of ts_maxwait. By increasing the quantum of priority 59, threads in that class will run longer before they are forced to relinquish the CPU and look for other threads that need to run. In order to get a process into this priority level, you will need to run priocntl, as described in the next section.

There's one small catch with this scheme: you will need to flush any existing processes out of priority level 59. This is accomplished by creating a copy of Example 3-3, and changing the last line (for priority level 59) to this:

# Time Sharing Dispatcher Configuration

RES=1000

# ts_quantum  ts_tqexp  ts_slpret  ts_maxwait ts_lwait  PRIORITY LEVEL

...

       200         0         0           0         0        #    59

This ensures that any process in priority level 59 quickly departs from it. You can then use ps -ecL | grep 59 to find processes that are still running in priority 59; they may need to be stopped and restarted.

Please exercise care when adjusting the dispatch tables. It is possible to cause a great amount of damage to the stability and usability of a production system through this interface. Test thoroughly!

3.3.2.6 Adjusting priorities

You can directly adjust the priority of a process via the priocntl command. This is accomplished by specifying the -s flag, which indicates that you are setting a priority: either -p priority, which specifies the priority level to be set, or -i pid process-id, which specifies the process ID to be adjusted. For example, let's adjust the nscd process into priority level 59, after having adjusted the dispatch tables as described in Example 3-3 to fence off the timesharing priority level 59:

# /usr/bin/ps -ecL | grep 59

# /usr/bin/ps -ecL |/usr/xpg4/bin/grep -E 'nscd|PID'

   PID   LWP  CLS PRI TTY     LTIME CMD

   200     1   TS  51 ?        0:00 nscd

   200     2   TS  58 ?        0:00 nscd

   200     3   TS  58 ?        0:00 nscd

   200     4   TS  58 ?        0:00 nscd

   200     5   TS  58 ?        0:00 nscd

   200     6   TS  58 ?        0:00 nscd

   200     7   TS  58 ?        0:01 nscd

# priocntl -s -p 59 -i pid 200

# /usr/bin/ps -ecL |/usr/xpg4/bin/grep -E 'nscd|PID'

   PID   LWP  CLS PRI TTY     LTIME CMD

   200     1   TS  59 ?        0:00 nscd

   200     2   TS  59 ?        0:00 nscd

   200     3   TS  59 ?        0:00 nscd

   200     4   TS  59 ?        0:00 nscd

   200     5   TS  59 ?        0:00 nscd

   200     6   TS  59 ?        0:00 nscd

   200     7   TS  59 ?        0:01 nscd

The advantage of a trick like this is to ensure that high priority jobs get as much processor time as possible, regardless of the environment in which they are running.

The renice command, which we discussed earlier in this chapter in Section 3.3.1.2, provides another way of adjusting a process's priority, by providing a static offset into the priority tables. This offset is equal to the renice amount, but with the negative sign reversed. For example, a process that has been reniced to -20 recieves a +20 offset into the priority table (so its priority will never drop below 20).


3.4 Multiprocessing

Shared-memory multiprocessor systems were considered rather exotic in the mid-1980s, and they were expensive. Since then, we have seen hardware costs drop, much more robust multiprocessor support in operating systems, and an increased demand for affordable high-throughput systems at a workgroup and even desktop level. These multiprocessor systems are described as uniform memory access (UMA) architecture, which means that all physical memory is accessible to all processors at the same rate. Some large-scale systems use a nonuniform memory access (NUMA) architecture, in which a specific processor can access some bits of physical memory faster than others. Figure 3-5 illustrates the UMA and NUMA architectures.

Figure 3-5. UMA and NUMA architectures

We'll restrict our discussion here to UMA architectures.

3.4.1 Processor Communication

In order for the human body to function nominally, it is vital that information can be conveyed to and from the brain. Communication is also essential for successful operation of a multiprocessor system. If we draw an analogy between computers and the human body, and we think of the processor as the brain, the "spinal cord" facilitates the exchange of information to and from the processor to main memory and peripheral devices. This is facilitated in computer architecture by buses or crossbars, which also allow the arbitration of communication so that the components can figure out who can talk at any given point in time. It is important to note that we speak here only of interconnections between processors and main memory; for details on how interconnections between peripherals are handled, see Section 3.5 later in this chapter.

3.4.1.1 Buses

A bus consists of three parts: a protocol for communication, a set of parallel wires that link all the system components, and some supporting hardware. Buses are cheap and easy to build, but as it becomes more heavily populated and the load increases, the bus begins to be a performance bottleneck. As a result, cache size and performance is very important in bus-based multiprocessor configurations. Typically, this sort of architecture supports at most four processors, but larger ones will support a few tens of processors. It is possible that the bus or memory subsystem will become saturated before the maximum number of supported processors have been installed!

Buses implement either circuit-switched or packet-switched protocols. In a circuit-switched protocol, two devices establish a communications circuit over the bus and maintain that circuit until their transaction has completed. This is simple to implement, but the processor must wait if it is communicating with a slow device (such as main memory). These wait states reduce the bus throughput and waste clock cycles: the faster the processor clock rate, the more cycles wasted.

Older Sun systems used the MBus bus architecture, which is circuit-based.[11] You could actually configure four 50 MHz SuperSPARC modules with SuperCache in a SPARCserver 20, which was a remarkably fast desktop system for the time. Unfortunately, when 50 MHz processors were installed in a SPARCstation 20, it automatically reduced the MBus clock rate from 50 MHz to 40 MHz in order to support data synchronization between processors; it's too bad that the system that needed the fastest possible MBus didn't get it. Because of cooling limits in the SPARCstation 20 chassis, you could not configure processors faster than 50 MHz.

[11] The SPARCstation 10, 20, and the SPARCserver 6x0MP. Other systems, such as the SPARCstation LX, Classic, 4, and 5, are MBus-based, but do not have multiprocessor support. If uname -a returns sun4m, your machine uses the MBus architecture.

To get around the wasted cycles inherent in fast-slow device interactions on a circuit switched protocol like MBus, Sun implemented a packet-switched protocol for XBus, which was used on a per-board basis in the SPARCserver 1000 and 2000 lines as well as in the Cray SuperServer CS6400.[12] The boards were tied together via XDBus, which consisted of multiple interleaved XBuses. A packet-switched protocol relies upon breaking every bus transaction into a request packet and a response packet. This is hard to implement because every packet must contain an identifier of the issuing device, as well as the need for a particular component on the bus (such as the memory subsystem) to queue up requests if they come in too quickly.

[12] When Cray was purchased by Silicon Graphics in 1996, the division responsible for the SuperServer CS6400 was sold to Sun. This group later produced the Ultra Enterprise 10000 ("Starfire") system.

With the advent of the fast UltraSPARC processors, Sun designed a new interconnect architecture, called the UltraSPARC Port Architecture switched interconnect (UPA). It is a packet-switched architecture characterized by its implementation of a "switch" architecture in which all buses route through a centralized switch that redirects them appropriately, as well as sizing the bus to the device; for example, memory devices require a very wide bus, but I/O channels need not be so wide. This bus sizing keeps costs low. All ports on the switch run at the same clock rate, but some are master ports (processors, I/O controllers) that can both initiate and respond to requests, and some are slave ports (framebuffers, memory) that can only respond to requests. UPA is resistant to bus contention bottlenecks under heavy load, because each device has its own dedicated connection to the bus switch.

For the multiboard Ultra Enterprise server line, modifications were made to the UPA bus in order to support the multiboard system requirements.[13] This gave rise to the Gigaplane passive backplane, which is a distributed extension of the UPA bus techniques. It has support for optimizations that increase the efficiency of spinlocks and memory sharing.

[13] The Ultra Enterprise 250 and Ultra Enterprise 450 systems are single-board, and implement UPA.

The performance of these buses is detailed in Table 3-4.

Table 3-4. Sun multiprocessor bus/switch characteristics

Bus

Clock rate

Peak bandwidth (in MB/sec)

Peak read throughput (in MB/sec)

Peak write throughput (in MB/sec)

MBus

40 MHz

320

90

200

XBus (per board)

40 MHz

320

250

250

Single XDBus (SPARCserver 1000)

40 MHz

320

250

250

Dual XDBus (SPARCcenter 2000)

40 MHz

640

500

500

Quad XDBus (SuperServer CS6400)

55 MHz

1,760

1,375

1,375

UPA (UltraSPARC systems)

82 MHz

1,600

1,300

1,300

Gigaplane (Ultra Enterprise systems)

82 MHz

2,624

1,600

1,600

Gigaplane XB (Ultra Enterprise 10000)

100 MHz

Up to 10,622

Up to 12,800

Up to 12,800

3.4.1.2 Crossbars

Crossbars were developed to work around the performance problems in the single bus architecture. A crossbar is essentially a collection of several parallel buses, in which any component can access any other component through any path through the crossbar. In addition to connecting parties that wish to communicate, much like a switchboard operator, the crossbar must also arbitrate conflicts, such as those that arise when two processors try to access the same bit of memory. Because there is no single shared bus, performance is generally good, but because of the extra support hardware required, crossbars are typically more expensive, and only found in high-end multiprocessor systems.

One example of such an implementation is the Gigaplane XB crossbar used in the Sun Ultra Enterprise 10000 system. It is an extension of the Gigaplane passive backplane used in the other Ultra Enterprise systems; it increases the throughput of the architecture, but at a cost in price and memory latency. This increase in memory latency is caused by an additional layer of circuitry. Gigaplane XB does not feature the spinlock optimizations included in Gigaplane, because Gigaplane XB was developed by the Cray SuperServer team before their acquisition. Gigaplane is actually faster than Gigaplane XB when running within its bandwidth limitations; the full bandwidth of the XB crossbar is only available with a fully configured system, as illustrated in Table 3-5.

Table 3-5. Bandwidth variation as a function of configured processors

Architecture

Number of processors

Peak bandwidth (in MB/sec)

Latency (in ns)

UPA

1-4

1,600

170

Gigaplane

1-30

2,624

240

Gigaplane XB

1-12

13-20

21-28

29-36

37-44

45-52

53-60

61-64

1,333

2,666

4,000

5,331

6,664

7,997

9,330

10,622

400

In less than twenty processor configurations, the Gigaplane architecture has the edge in performance. In larger installations, the Gigaplane XB crossbar starts to shine, and we see much improved bandwidth: four times the bandwidth for twice the number of configured processors. In small installations (four processors or less), however, the latency of Gigaplane-based systems becomes a liability; because of the increased complexity of the switching times, it is approximately twice that of UPA-based systems. This means a four-processor single-board Ultra Enterprise server often outperforms a four-processor multiboard Ultra Enterprise server, and costs considerably less!

3.4.1.3 UltraSPARC-III systems: Fireplane

With the move to the UltraSPARC-III architecture, Sun developed a new interconnection technology, called Fireplane. Fireplane improves on the previous-generation UPA interconnect in both performance and feature set. Unlike the other interconnects we've discussed, Fireplane does not specify the device topology; instead, it describes the protocols that devices attached to the interconnect use to communicate. This gives it a good degree of flexibility for future uses. The other interesting new feature of Fireplane is that it supports having multiple outstanding transactions on the interconnect at once (up to fifteen 64-byte data transfer requests, for example, can be pending at once on the Sun Blade 1000).

The Sun Blade 1000 implements a packet-switched bus topology for both address and data paths. The data path is a 288-bit (256 bits of data, plus 32 bits of error correction information) bus, implemented via a group of six chips that are collectively known as the combined processor memory switch (CPMS). There are point-to-point connections between data path and devices connected to it, simplifying arbitration logic.

The memory controller is integrated onto the UltraSPARC-III processor, which reduces latency and improves bandwidth. The memory is addresed through a 144-bit-wide data path to a data switch chip, which has a 576-bit (512 bits of data, plus 64 bits of error correction information) path to memory. Interestingly, in a two-processor system, only one of the processors' memory controllers is active. This processor can directly access memory, whereas the other must access memory over the Fireplane interconnect.[14]

[14] All memory address requests go across the interconnect to facilitate cache coherency management.

The interconnect itself is capable of a peak bandwidth of 67.2 GB/second, with a sustained bandwidth of up to 9.6 GB/second (2.4 GB/second per CPU, up to a limit of 9.6 GB/second).

3.4.1.4 "Interconnectionless" architectures

Some UltraSPARC processors, which are typically used in lower-end systems, do not require a formal interconnect. These include the so-called "I-series" and "E-series" designs, such as the UltraSPARC-IIi and the UltraSPARC-IIe; I-series processors are highly integrated and optimized for price/performance, whereas the E-series processors are designed for embedded applications.

3.4.2 Operating System Multiprocessing

The majority of modern operating systems support multitasking. In fact, multitasking does not require more than one processor; it simply refers to the ability to run more than one process at a time. The system splits its time between them on the basis of context switching at fixed time intervals, I/O activity, or interrupts. On a multiprocessor system, individual processes will be split between both processors for an aggregate speedup (but without increasing the observed single-application performance). Most processes are self-contained,[15] but processes can play nicely with others and share information by means of the interprocess communication (IPC) mechanisms such as pipes and sockets or by intimately shared memory (ISM). We'll discuss ISM in greater detail in Section 4.4. It is generally a bad idea to implement IPC in an attempt to increase single-application performance on a UMA multiprocessor system; on very large scale parallel computing installations it may be appropriate, however.

[15] In terms of not sharing data with other processes. We won't worry about shared libraries and such for the purposes of this discussion.

3.4.3 Threads

A thread is not a process! A process begins its life as a single thread, but it can then add or remove threads as its execution progresses. These threads all share the same process memory space, but each thread also possesses a thread private area for its own local variables. Threads can improve single-application performance in two ways:

  • Each thread in a multithreaded process can be dispatched to a different processor in a multiprocessor system. This collaboration across multiple processors improves single-application performance.

  • Threads can be used to increase application throughput. A single-threaded web server would be stuck in a wait state every time it needed to fetch a document from disk. By multithreading the server, one thread can be waiting to handle a new request while another waits for the disk to provide data.

Because of this last point, it is quite practical to have a multithreaded process on a single processor; in fact, multithreading existed long before multiprocessing systems did. This technique is very useful for increasing throughput, but it's a bad idea to have more identical active threads than processors in a compute-intensive program.

3.4.4 Locking

It is critical to the smooth operation of a multiprocessor system that some mechanism of synchronization should exist. For example, say that we have two threads that each increment the same variable by executing the following assembler code:

LOAD      r1,Counter

ADD       r1,1

STORE     r1,Counter

Let's say that the initial value of Counter is 021400. What happens if, at the exact instant that the first thread completed the LOAD but before it executed the ADD and STORE, the operating system switched to executing the second thread? The second thread could execute all three instructions, resulting in Counter being 021401. When the operating system restarts the first thread, it will pick up where it left off and execute the ADD and STORE instructions, storing 021401 into Counter and giving it a terminal value of 021401 when it should be 021402. If the system has two processors, the situation is just as bad if not worse! Rather than relying on operating system intervention, if the two threads are executing the same instructions concurrently, they will perform the load-add-store sequence in perfect lockstep, and we will again have 021401 when we should have 021402 in Counter.

We need some way to make simple operations such as this atomic (indivisible), so that only one thread can execute these instructions at a time. We call areas like this critical sections.

On uniprocessor systems, this problem was solved by disabling interrupt servicing for the duration of the critical section, which prevents the thread from being interrupted. This won't work on multiprocessor systems, since each processor has its own interrupt mask, so we must be more devious in our approach.

All SPARC processors support an instruction called LDSTUB (load-store-unsigned-byte), which atomically reads a byte from memory into a register and writes 0xFF into memory. This instruction allows the establishment of mutual exclusion locks, or mutexes, which make sure that only one processor at a time can hold the lock. To tear down the lock, the thread simply writes 0x00 back to memory. If a processor can't get a lock, it may decide to spin until the lock becomes available. Spin locking is accomplished entirely within the processor's cache, so that it does not cause excess bus traffic. Spin locks are useful when the wait for the lock is expected to be short; if a longer wait is expected, the thread should sleep so that another job can be scheduled onto the processor. The SPARC V9 instruction set used on UltraSPARC processors includes an atomic compare-and-swap operation as well as the atomic load-store operation.

When the multiprocessing-enabled version of SunOS 4 was released for the SPARCserver 6x0-class systems, the uniprocessor interrupt-mask approach was replaced with a mutex-based approach: the call to disable interrupts was replaced by a mutex acquisition, and the call to re-enable interrupts was replaced with a mutex release. On a system with many processors that is under load, the mutex locking itself can actually become a bottleneck.[16] If this is the case, adding additional processors will hinder, rather than help, performance.

[16] I've never seen this behavior on a system with less than four CPUs, assuming a well-behaved application.

Interestingly, this phenomenon of performance decreasing as you add processors is not restricted to this particular case. It's possible for poor application programming to cause severe mutex contention. A large security company a friend of mine was affiliated with had a product written in C that ran on Solaris systems. On a uniprocessor system, this product performed adequately. When a second CPU was added, the performance of the application dropped by almost 30%. This is not at all unusual: poor application design and implementation is a possible, indeed a likely, root cause of poor performance on multiprocessor systems.

Sometimes all the threads in a process need to wait at a certain point until all of the threads are "caught up." For example, let's say that we have a time-based simulation of a car's frontend crumpling during an impact with a concrete wall.[17] It would not make sense for the threads to compute the state of the car at different points in time, since the threads working later would need to know what happened earlier. We can enforce this sort of synchronization with a construct known as a barrier. All threads must reach the barrier (say, finish processing one time interval's worth of data) before any are allowed to continue.

[17] I have, unfortunately, had occasion to attempt this experiment in real life. My personal experience indicates that simulations are a far less expensive way of determining that a car will protect its occupants from bodily harm.

3.4.5 Cache Influences on Multiprocessor Performance

Another major problem needs to be dealt with in multiprocessor system architecture. Let's say that we have two processors, CPU 0 and CPU 1. CPU 0 fetches a specific cache line from main memory into its cache. CPU 1 then fetches the same line into its cache. When CPU 0 writes to that cache line, it will update its local cache as well as main memory; however, there needs to be some way of notifying CPU 1 that its copy of the cache line is invalid and needs to be thrown away. This is accomplished via communications on the multiprocessor bus. Since communication must occur for most cache activity, it creates a good deal of traffic.

As a result, larger processor caches often cause a great performance increase on multiprocessor systems. References to main memory must go across the bus, which on a multiprocessor system is typically very busy.


3.5 Peripheral Interconnects

The last means of interconnection that we need to discuss is between the bus or crossbar that facilitates the communication between processors and main memory (see Section 3.4.1 earlier in this chapter) and the bus that interacts with peripheral devices.

There are two bus architectures in widespread use: Sun's SBus and the industry-standard Peripheral Component Interconnect, or PCI. SBus is being phased out in favor of the PCI standard, but I discuss it here in light of its very large installed base.

3.5.1 SBus

SBus and PCI are fairly similar architecturally: both are designed to be I/O buses (rather than having more general applications), have small form factors requiring a high degree of integration, and roughly similar performance characteristics. If PCI had been available in 1988 when Sun was developing SBus for the SPARCstation 1, SBus probably would not have been developed. Every Sun system implements SBus slightly differently, making a different set of choices between performance, cost, ease of implementation, etc. However, in practice, all current SBus implementations can meet almost all requirements.

SBus is a parallel-transfer (many bits are transferred concurrently) bus architecture. In the original specification, there were 32 address lines and 32 data lines; the Rev B.0 SBus specification permits the address lines to be used during the data cycle to transfer 64 bits; this is implemented in the SPARCstation 10SX, the SPARCstation 20, and all the UltraSPARC-based systems. This arrangement means that 64-bit and 32-bit SBus implementations share the same form factor. However, bus width (while an important factor) is often overestimated in its relevance to performance; very few devices can exceed the 32-bit bus capacity. These are largely very high performance graphics framebuffers and very fast network interfaces (faster than about 250 Mbits/second).

3.5.1.1 Clock speed

In most older Sun systems, the SBus clock is derived from the system clock, either directly or through a divider. Most SPARCstation 10 systems drive the system clock at 40 MHz, and the SBus clock is set at 20 MHz.[18] SPARCstation 20 systems operate the system clock at 50 MHz and the SBus at 25 MHz. The SPARCserver 1000 and SPARCcenter 2000 run the SBus at 20 MHz, independent of any other factors. The microSPARC and microSPARC-II systems run the SBus by dividing the processor clock. This is summarized in Table 3-6.

[18] The exception is in the early models (the SPARCstation 10 Model 20 and Model 30), which had system clocks of 33 MHz and 36 MHz respectively; the SBus clocks are set, then, to 16.5 Mhz or 18 MHz, respectively. Most of these systems have been upgraded to the newer revisions of the SPARCstation 10 boards.

Table 3-6. SBus clock rates for microSPARC systems

CPU clock

Divider

SBus clock

50 MHz

2:1

25 MHz

70 MHz

3:1

23.3 MHz

85 MHz

4:1

21.25 MHz

110 MHz

5:1

22 MHz

Most newer Sun systems (SPARCserver 1000E, SPARCcenter 2000E, and the UltraSPARC-based systems) drive the SBus at 25 MHz, regardless of any other factors.

3.5.1.2 Burst transfer size

The SBus architecture allows data to be transferred in bursts, which are defined as the amount of data that the bus can accept each time a unit on the bus arbitrates for the bus; it is roughly similar to the maximum transmission unit size on a network. Every time a data transfer is initiated, the bus interface hardware accepts one burst's worth of data; it then arbitrates for control of the bus, transmits the target address, and transfers as much data per clock cycle as the bus width allows. This influences the efficiency of bus transactions: on a platform that implements a 16-byte burst size, transferring 2 KB on the bus requires 128 bus arbitration/address/data transfer cycles, compared to just 32 of these cycles when using a 64-byte burst size.

SBus bursts are defined to be 1, 2, 4, 8, 16, 32, and 64 bytes in length. UltraSPARC systems and later MBus-based desktop systems also support a 128-byte burst length. The maximum supported burst size, however, depends on the hardware platform. The SPARCstation 2, SPARCstation 10, SPARCstation 20, and the SPARCsystem 600MP all support 32-byte bursts; all other non-UltraSPARC systems support 16-byte bursts. The UltraSPARC systems support all burst sizes. The burst size is negotiated at boot time, on a per-slot basis, to be the smaller of the maximum size supported by the host SBus device and the controller.

3.5.1.3 Transfer mode

Every SBus implementation offers at least programmed I/O (PIO) and consistent mode direct virtual memory access (DVMA).[19] In PIO, the processor uses load and store instructions to transfer data on the bus. This is primarily used for managing control and status registers on the SBus boards, but a handful of boards implement PIO for data transfer. Because it requires the processor's intervention and because the burst sizes are necessarily smaller (non-UltraSPARC processors define load and store instructions in quantities up to 8 bytes only; UltraSPARC processors have 64-byte block load/store instructions), PIO is almost always more expensive than DVMA. DVMA, however, has a fairly complex setup and teardown process for each I/O operation; therefore, small transfers are more efficiently handled by PIO.

[19] DVMA is very similar to the more common direct memory access (DMA) method. They differ only in that DVMA specifies the address of the memory transaction in virtual space, rather than the physical address.

Some SBus implementations provide an additional transfer mode, called streaming mode DVMA. Streaming mode DVMA provides significantly faster transfers, especially for reads; however, it requires explicit modifications to the driver for keeping transfer hardware consistent with shared memory. Most device drivers implement streaming mode support.

3.5.1.4 Summary of SBus implementations

Table 3-7 briefly summarizes the SBus implementations in many SPARC-based systems.

Table 3-7. SBus implementation details

Platform

Burst size

Bus width

Clock

Streaming?

Speed (read/write)

SPARCstation 1

16 bytes

32 bits

20 MHz

No

12/20 MB/sec

SPARCstation 1+, IPC

16 bytes

32 bits

25 MHz

No

15/25 MB/sec

SPARCstation 2, IPX

32 bytes

32 bits

20 MHz

No

15/32 MB/sec

SPARCstation 10, SPARCstation 600

32 bytes

32 bits

20 MHz

No

32/52 MB/sec

LX, Classic

16 bytes

32 bits

25 MHz

No

17/27 MB/sec

SPARCstation 4, SPARCstation 5

16 bytes

32 bits

21.25-25 MHz

No

36/55 MB/sec

SPARCstation 10 SX

128 bytes

32/64 bits

20 MHz

No

40/95 MB/sec

SPARCstation 20

128 bytes

32/64 bits

25 MHz

No

62/100 MB/sec

SPARCserver 1000, SPARCcenter 2000

64 bytes

32 bits

20 MHz

Yes

45/50 MB/sec

SPARCserver 1000E, SPARCcenter 2000E

64 bytes

32 bits

25 MHz

Yes

56/62 MB/sec

All UltraSPARC systems

128 bytes

32/64 bits

25 MHz

Yes

90/120 MB/sec

The SBus I/O boards used for the Ultra Enterprise series of systems are separate from the processor/memory boards. There are two types of SBus I/O boards; one is designed for graphics cards, whereas the other is more general purpose.

The general-purpose I/O SBus has a dual-ported Fibre Channel interface, a Fast Ethernet port, and a single-ended Fast Wide SCSI-2 port, as well as three 64-bit SBus slots. The board implements two independent SBuses: one is responsible for the Fast Ethernet and SCSI interfaces, along with one of the SBus slots. The other SBus services the Fibre Channel interfaces and the other two SBus slots.[20] The graphics I/O board replaces one of the SBus slots with a UPA slot; this means there is only a single SBus controller. Both the SBus slots are attached to this SBus, along with all the devices integrated into the I/O board. The UPA port is totally independent.

[20] There are actually three types of general-purpose SBus I/O cards. One type (501-4287) has a pair of 25 MB/s Fibre Channel interfaces, and supports only backplane speeds up to 83 MHz. Both of the other types have 100 MB/s Fibre Channel interfaces, implemented via GBIC slots; the only difference between these two is that one (501-4266) runs at 83 MHz, and the other (501-4883) can go up to 100 MHz. However, they all have two independent SBuses, which are broken up as described.

3.5.1.5 SBus card utilization

Table 3-8 provides a general idea of SBus cards and how much bandwidth they typically consume. Of particular interest is the fact that SBus framebuffers are very significant consumers of SBus bandwidth.

Table 3-8. SBus bandwidth utilization by card

Description

Typical bandwidth

Transfer type

Width

Fast/Wide SCSI-2

8-10 MB/sec

DVMA

32

25 MB/s Fibre Channel

16 MB/sec

DVMA

64

Fast/Wide SCSI-2 + Fast Ethernet (hme)

12 MB/sec

DVMA

64

155 Mb/s ATM (Revision 1)

10 MB/sec

DVMA

32

155 Mb/s ATM (Revision 2)

10 MB/sec

DVMA

64

655 Mb/s ATM

50-100 MB/sec

DVMA

64

Fast Ethernet (Revision 2)[21]

4 MB/sec

DVMA

64

Quad Fast Ethernet

20 MB/sec

DVMA

64

Gigabit Ethernet

40 MB/sec

DVMA

64

100 Mb/s FDDI

6-8 MB/sec

DVMA

32

16 Mb/s Token Ring

1 MB/sec

PIO

32

GX, GX+ framebuffer

4-12 MB/sec

PIO

32

TurboGX, TurboGX+ framebuffers

8-20 MB/sec

PIO

32

ZX framebuffers

20-30 MB/sec

DVMA

32

[21] Revision 1 Fast Ethernet cards, which use the be driver, are not supported past Solaris 2.6. If possible, replace them with Revision 2 cards, which use the hme driver.

An SBus should typically not be more than 80 percent utilized, in order to avoid bus contention and arbitration problems that may degrade performance.

3.5.2 PCI

PCI, or the Peripheral Component Interconnect bus, is probably the most widespread bus design in modern computer systems. It was initially introduced into the Intel-based PC world in 1994, and has largely replaced all the older peripheral bus architectures. It is now widely supported by workstation vendors as well. PCI is a standard under the auspices of the PCI Special Interest Group, or PCI SIG (http://www.pcisig.com).

PCI is a synchronous bus architecture, which means that all data transfers are performed relative to a system clock. The initial PCI specification permitted a maximum clock rate of 33 MHz, but the later Revision 2.1 specification extended this to 66 MHz. Most personal computers support only the 33 MHz standard; the primary application of 66 MHz slots is in very high speed networking (e.g., full-duplex Gigabit Ethernet, which greatly benefits from a 66 MHz slot speed). PCI supports a 32-bit multiplexed address and data bus,[22] as well as having architectural support for a 64-bit data bus through a longer connector slot; as with 66 MHz support, 64-bit cards are not widely available in the PC market (see Figure 3-6).

[22] This multiplexing allows for a reduced pin count on the PCI connector, which translates to a smaller physical size and lower cost. Typical 32-bit PCI boards use about 50 pins.

Figure 3-6. PCI connector types

The high speed of the PCI bus (up to 528 MB/second, at 64-bit data paths and a 66 MHz clock rate) limits the number of expansion slots on a single bus to no more than three or four slots due to electrical concerns. In order to permit an expansion bus to have more slots, the PCI SIG has defined a PCI-to-PCI bridge mechanism in PCI Revision 2.1. These bridges are chips that electrically isolate two PCI buses, while still allowing bus transfers to be forwarded from one bus to the other. Multiple bridge devices can be cascades to create a system with many PCI buses.

Multiboard Sun Enterprise systems support the PCI I/O board. These support two PCI cards per I/O board. Each PCI slot is on a separate PCI bus, and the Fast Ethernet and Fast/Wide SCSI interfaces are also on separate PCI buses.

3.5.2.1 PCI bus transactions

In PCI terminology, data is transferred between an initiator, which is the bus master, and a target, which is the bus slave. After the initiator arbitrates for the bus, the transfer itself consists of one address phase and any number of data phases. During the address phase of the transfer, the initiator signals what type of transfer is being done (read from memory, write to memory, I/O read, I/O write, etc.). Either the initiator or the target can insert wait states into the transfer; in addition, the initiator or target may terminate the transfer at any time.

PCI also supports a well-defined automatic configuration mechanism. Each PCI device includes a set of registers that contain configuration data. These registers define what the type of the card is (SCSI, Ethernet, a framebuffer, etc.), as well as who manufactured the card, what the interrupt level of the card is, and so on.

Finally, PCI supports both 5-volt and 3.3-volt signaling levels. However, most early PCI implementations were 5-volt only, and did not provide power on the 3.3-volt pins. Over more time, the 3.3-volt interface will be used more heavily. A "keying" scheme is incorporated in the PCI connector standard, in order to prevent inserting a board into a slot that doesn't supply the correct voltage.

3.5.2.2 CompactPCI

One variant of PCI, CompactPCI, was designed as an industrial-use bus for applications such as real-time machine control and data acquisition, instrumentation, and telecommunications. Compared to the standard, desktop PCI interface, CompactPCI supports twice as many slots (eight versus four), and offers a significantly more robust form factor. CompactPCI cards are designed to front-load into a card cage; this, along with more reliable pin-and-socket connectors, takes full advantage of CompactPCI's hot swap capability. CompactPCI devices should perform equivalently to comparable PCI devices. However, as CompactPCI cards are just beginning to be introduced into the wider workstation/server marketplace, their long term future is unclear.

3.5.3 A Summary of Peripheral Interconnects

Table 3-9 provides a summary of peripheral interconnects; the emphasis is on the PC hardware arena.

Table 3-9. Peripheral interconnect comparison

Bus type

Year

Width

Clock rate

Peak bandwidth

Burst transfers?

ISA

1984

16 bits

8 MHz

8 MB/sec

No

MCA

1987

32 bits

10 MHz

40 MB/sec

Yes

EISA

1988

32 bits

8 MHz

32 MB/sec

Yes

SBus

1988

32 or 64 bits

Up to 25 MHz

Up to 200 MB/sec

Yes

VESA

1992

32 bits

33 MHz

132 MB/sec

Yes

PCI

1992

32 or 64 bits

33 or 66 MHz

132 to 534 MB/sec

Yes

3.5.4 Interrupts in Linux

You can get a list of all the interrupts in use on a Linux system by inspecting the file /proc/interrupts:

% cat /proc/interrupts

           CPU0

  0:  216651906          XT-PIC  timer

  1:       1965          XT-PIC  keyboard

  2:          0          XT-PIC  cascade

  8:          1          XT-PIC  rtc

  9:   96370488          XT-PIC  eth0

 14:  381136978          XT-PIC  ide0

 15:   60751177          XT-PIC  ide1

NMI:          0

ERR:          0

The most useful way to adjust the interrupt distribution in a Linux system is the irqtune application. This application, which is available through http://www.best.com/~cae/irqtune/ at the time of this writing, allows you to specify certain interrupts that are serviced at higher priority. Interrupt priority is used to determine which interrupts are handled first when multiple interrupts occur concurrently. In general, interrupt priorities are assigned in decreasing order of IRQ; that is, the system timer (IRQ 0) has priority over all other IRQs. irqtune addresses this problem by giving priority 0 to the specified IRQs. To invoke it, just run irqtune and list the interrupts to prioritize. For example, to prioritize IRQs 9 and 14 (the eth0 and ide0 interfaces in the system described previously):

# irqtune 9 14

3.5.5 Interrupts in Solaris

Sun enterprise-class systems (the SPARCserver 1000, the SPARCcenter 2000, and Ultra Enterprise series) have many options for managing SBus devices. Since Solaris 2.3, the operating system has used a static interrupt distribution mechanism. This means that each I/O device is assigned to a specific processor; every time that device triggers an interrupt, the specific CPU that is assigned to the device handles the work. The interrupt assignment table is set up at boot time and every time a processor is set to, or removed from, the no-interrupt state (see Section 3.6.7 later in this chapter). Currently, there is no way to manually assign the interrupts from a specific device to a specific processor. The alternative, known as round robin distribution, causes each interrupt to be handled by the "next" processor. It is enabled by setting the do_robin kernel tunable to 1.

Round robin distribution causes the processor caches to be utilized much less efficiently, because a copy of the device-specific interrupt handling code must reside in each processor's cache, but it also results in interrupts being scheduled faster.

Static interrupt processing generally means that high-performance boards should be separated across as many processors and SBus interfaces as possible. The Quad Fast Ethernet (QFE) boards are particularly known for throwing many, many interrupts (up to about 20,000 per second under heavy load). Within an SBus, each slot is treated identically. Because of this, placing multiple instances of an SBus card on the same SBus can sometimes improve performance, because identical boards share the same interrupt processing code. On the other hand, you can easily overload the bandwidth capacity of the SBus. This arrangement improves the chances that the interrupt handling code is resident in the processor's cache. Also note that since interrupts are statically assigned, heavily loading a processor can restrict I/O performance from devices assigned to that processor. If one CPU is taking a large number of interrupts and that CPU is regularly at or near 100% utilization, moving SBus boards into another SBus may provide a significant boost. A simpler solution to try is to enable round robin scheduling.

One common approach to managing interrupt distributions in multiple-CPU systems is to use a small number of CPUs to handle all interrupts, and forbid other processors from taking interrupts. This is accomplished by using the psradm command to put processors into no-intr mode (see Section 3.6.7 later in this chapter). This command lets you segment your workload at a fairly fine level.


3.6 Processor Performance Tools

Concretely detecting whether a system is short of processor time is often more complicated than it first seems. Regular indications of CPU shortage can be attributed to many things: disk configuration problems, peripheral devices being bound to a single processor for interrupt management, processor starvation due to a lack of memory, etc. In a sense, the best way to tell when you're legitimately processor-bound is that you can exclude all other possibilities.

There is also a philosophical issue involved. Some people like seeing a low load average on their systems, or more than 50% of the processor idle. This makes some sense in terms of the system accomodating bursting load activity. Similarly, some people like seeing a load average that approaches the number of processors, meaning the system is running full-tilt. This also makes some sense, because you are using everything that you have paid for in terms of processor performance. My personal philosophy varies between the following: on a computational server, I would aim for full utilization of all processors; on a development system or a database server, I would aim for a degree of overcapacity to accomodate spikes in load.

There are quite a few tools for monitoring processor performance: the load average, vmstat, mpstat, prstat and/or top, and lockstat. There are also two tools in Solaris for managing processors in multiprocessor systems: psrinfo and psrset. We'll cover all of these in some detail.

3.6.1 The Load Average

The system understands a concept of the load average, which is expressed in the uptime command:

% uptime

  4:02pm  up 19 day(s),  9:28,  119 users,  load average: 1.22, 1.21, 1.25

The load average is represented as the average number of runnable jobs (the sum of the run queue length and the number of jobs currently running) over the last one, five, and fifteen minutes. This gives a quick estimate of how heavily loaded the system is, but it is not exclusively a measurement of processor load; a system that has a memory shortage will usually have a high load average.

The load average statistic is only meaningful when looked at in the context of the number of installed processors. As a general rule, if the load average is less than the number of installed processors, the system is not lacking processor power. For example, if the load average is 2.5 on a four-processor system, it is not an indication of a processor shortage.

3.6.2 Process Queues

One very powerful tool in our performance tuning arsenal is vmstat. I discuss it in pieces as we go along. Let's take a look at some sample vmstat output from a Solaris system (recall that vmstat takes as its first argument the time interval in seconds between displaying data, and that you must always discard the first line of output):

% vmstat 30

 procs     memory            page            disk          faults      cpu

 r b w   swap  free  re  mf pi po fr de sr s0 s1 -- --   in   sy   cs us sy id

 0 0 0   5928  1920   0 328 45 25 37 1360 2 4  8  0  0  306 1682  299 10  5 85

 0 0 0 1202648 17152  0 904  8  0  0 1200 0 7 36  0  0  547 5632  683 18 15 67

 0 0 0 1202744 19600  0 1136 0  0  0 1040 0 2 26  0  0  575 4542  752 33 12 55

The first column (highlighted) provides the state of the system's three process queues. In order, they are the run queue (r), the runnable-but-blocked queue (b) and the runnable-but-swapped queue (w). We'll discuss the swap queue's relevance, or lack thereof, in Section 4.5.2.1.

The run queue counts the number of processes that are runnable, but are not currently running. In this example, the system's processor is not overworked. As a general rule, if the run queue is consistently at or more than four times the number of configured processors in the system, you should probably consider increasing the available processors.

The runnable-but-blocked queue counts the number of processes that are otherwise runnable, but cannot be run because they are waiting for a block device transfer (generally a disk I/O operation).

The Linux vmstat looks slightly different, but displays the same information:

% vmstat 30

 procs                  memory    swap        io    system         cpu

 r b w  swpd  free  buff cache  si  so   bi   bo   in   cs  us  sy  id

 0 0 0 20068  9568 26524 50828   0   0    2    7    8    3   3   1   8

 0 0 0 20068  9568 26524 50828   0   0    0    0  115   22   8   4  89

 0 0 0 20068  9568 26524 50828   0   0    1    0  115   22   7   4  89

 0 0 0 20068  9568 26524 50828   0   0    0    0  114   20   7   4  89

3.6.3 Specific Breakdowns

Processor time is organized into four timed modes: system time, user time, I/O wait time, and idle time.[23] The idle time consists of what's left over when all other portions have had their fill. A program's normal operating state is user mode, but as it runs, it may generate requests for services that the kernel provides, such as I/O. These requests require the attention of the operating system, so the program switches into system mode, then returns to user mode when the request is complete. The time spent in these two modes is tabulated independently to give the user time and system time values, respectively. These two figures account for the majority of a process's execution time.

[23] Note that vmstat reports only the user time, system time, and idle time (wait time is summed in with idle time). In order to get separated values for wait time and idle time, use mpstat.

When a process waits for a block device data request to complete, it incurs I/O wait time. This brings up an important fact: when a process is blocked in this fashion, all idle time becomes wait time. If your idle time is zero, as reported by vmstat, the first thing you should check is if your system has I/O throughput problems.

Another good diagnostic event is looking at system time relative to user time. On NFS servers, which we will discuss in Section 7.5, it is normal for the system time to be much higher than the user time because NFS service is contained entirely inside the kernel.

You can obtain this sort of data on a per-process basis by means of the Solaris ptime command. Here is an example:

% /usr/proc/bin/ptime tar -cf miscellany.tar miscellany

real        7.402

user        0.044

sys         0.287

There are other commands, /bin/time and timex, which provide varying resolution. We'll discuss these commands in greater detail in Section 8.2.1.

There is a fair bit of system time incurred here because of the volume of I/O requests that must be made through the kernel facilities. There are many ways to get this sort of data on a systemwide level: vmstat and mpstat (which we'll discuss in more detail throughout this book) both provide it. The highlighted portion of the vmstat output is illustrative:

% vmstat 30

 procs     memory            page            disk          faults      cpu

 r b w   swap  free  re  mf pi po fr de sr s0 s1 -- --   in   sy  cs us sy id

 0 0 0   3296  2344   0 365 44 22 33  0  1  4  8  0  0  304  700  318 11  6 84

 0 0 0 1221816 99992  0 450  0  0  0  0  0  0  2  0  0  269 1791  465 19  6 76

 0 0 0 1222568 100688 0  83  0  0  0  0  0  0  8  0  0  254  591  389  2  1 98

This system exemplifies a system under light load. The idle percentage is fairly high and user time is greater than system time.

3.6.4 Multiprocessor Systems

The tools we've discussed present valid data for multiprocessor systems, but in order to get detailed per-processor statistics under Solaris, we need to turn to a more powerful tool: mpstat.

This is some sample mpstat output from a two-processor system:

% mpstat 30

CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl

  0  182   0  353   269   54  161   24   13    5    0   940   11   5   5  78

  1  182   1  531   134  116  157   23   13    6    0   931   10   6   5  79

CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl

  0    0   0 2856   228   28   72    0    6    0    0   454    2   2   4  92

  1    4   0    0    39   39   77    0    7    0    0    50    0   0   4  96

CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl

  0   80   0  515   489  288  108    2   11    1    0   275    1   6  37  56

  1   61   0   55    46   46  113    2    9    2    0   270    0   1  37  62

Here are some of the more interesting statistics:

  • intr is the number of interrupts handled by that processor; ithr is the number of interrupts as threads.

  • migr is the number of thread migrations. A thread migration occurs when a thread switches from running on one processor to running on some other processor.

  • smtx is the number of spins on mutexes; (in practical terms, this column represents the number of locks not acquired on the first try). We'll discuss this later in this chapter in Section 3.6.6.

  • syscl is the number of system calls.

  • usr, sys, wt, and idl are the user, system, wait, and idle percentages, on a per-processor basis. Note that vmstat provides one systemwide average only.

3.6.5 top and prstat

A popular and common tool for monitoring system-wide process activity is top, which provides a list of the most CPU-intensive processes running on a system, as well as some interesting statistics such as the load average, the number of processes, and the amount of memory and paging space in use. top is a very common and generally easy-to-understand application, so we won't discuss it further here. In the interest of completeness, this is what top output looks like:

% top

 10:27pm  up 24 days, 21:24,  9 users,  load average: 0.05, 0.06, 0.01

125 processes: 124 sleeping, 1 running, 0 zombie, 0 stopped

CPU states:  8.0% user,  9.1% system,  0.0% nice, 82.7% idle

Mem:  255596K av, 251056K used,   4540K free,      0K shrd,   3200K buff

Swap: 112444K av,      8K used, 112436K free                 99112K cached

  PID USER     PRI  NI  SIZE  RSS SHARE STAT  LIB %CPU %MEM   TIME COMMAND

 5752 gdm       17   0  1296 1296  1052 R       0 13.8  0.5   0:01 top

 3829 root       9   0  1252 1252  1072 S       0  1.1  0.4   0:06 sshd

 5753 root      10   0  1492 1492  1292 S       0  1.1  0.5   0:00 sendmail

    1 root       8   0   520  520   452 S       0  0.0  0.2   0:48 init

Unfortunately, the tradeoff for top's updating process list and other nice features is that it tends to consume quite a bit of processor time.[24] This has been known to substantially skew the behaviour of the system, so Solaris now ships with a similar tool, called prstat, that provides similar output. While it doesn't provide some of the nice information that top does about memory usage, it consumes very little processor time and doesn't skew the system's behaviour nearly as much:

[24] Rumor has it that this is where top got its name -- it often appears at the top of the CPU consumers list.

% prstat

   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP

  4585 gdm      1280K 1096K cpu0    48    0   0:00.00 0.4% prstat/1

  4554 gdm      1424K 1024K sleep   58    0   0:00.00 0.0% csh/1

   337 nobody     23M 6856K sleep   59    0   0:00.07 0.0% ns-slapd/15

  4552 root     1784K 1256K sleep   38    0   0:00.00 0.0% in.telnetd/1

...

Total: 33 processes, 143 lwps, load averages: 0.00, 0.00, 0.01

The upshot of this is that top is very nice, but you need to be careful about it skewing the system under observation because of its high overhead. On Solaris systems, prstat provides similar output at a much reduced cost in terms of processor time.

3.6.6 Lock Statistics

In addition to using mpstat for processor state information, we can also use it to provide meaningful data on interrupts and mutexes. The key measurement is the smtx column, which represents the number of instances in the unit time that the kernel failed to immediately obtain a mutex when it asked for one. If this number begins to exceed 250 times the number of configured processors, system time will begin to increase noticeably. Higher performance processors can cope with higher smtx values, e.g., UltraSPARC-III/Pentium 3 Xeon.

% mpstat 30

CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl

  0  182   0  353   269   54  161   24   13    5    0   940   11   5   5  78

  1  182   1  531   134  116  157   23   13    6    0   930   10   6   5  79

CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl

  0  112   0  365   259   55  141    6   12    2    0   486    6   3  10  81

  1  157   0  337    73   67  147    7   11    3    0   594    6   4   8  82

CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl

  0   97   0  598   229   26  135    6   11    2    0   389    5   3   5  87

  1  107   0   81    67   62  130    5   11    2    0   423    4   3   4  89

The best way to cope with this problem is to check with the vendor for any operating system upgrades that may have solved mutex contention issues, and then use the lockstat command to find out what mutexes are suffering. If you are consistently seeing very high processor utilization and the smtx values are high in your SMP system, replace your current processors with faster ones, rather than adding more!

To find suffering mutexes, lockstat uses a kernel lock monitoring facility. It must be invoked as root. The kernel can dynamically change its locks to collect information, then return to the normal, optimized locking code when the data collection is complete. lockstat can be invoked on a per-process basis or on a system-wide basis by calling it on the sleep command. I went through some grief trying to find a simple example of how to cause a large number of mutex spins to obtain a good example for illustration, and finally did so:[25]

[25] I set this up by running a number of infinitely looping C shell scripts equal to the number of configured processors on the system. I hope this is not actually what your system is doing.

% mpstat 10

CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl

  0    1   0    2   104    0   19    0    0    0    0    26    0   0   0 100

  1    1   0    1     8    6   19    0    0    0    0    37    0   0   0 100

CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl

  0    0   0    0   109    2   40   22    2  956    0 106954   47  53   0   0

  1    0   0    0    23    7   43   15    2 1117    0 107760   48  52   0   0

CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl

  0    0   0    0   116    0   57   26    2 1289    0 107204   47  53   0   0

  1    0   0    0    24    3   45   21    2 1099    0 107199   47  53   0   0

It should be obvious that this system is having serious mutex problems. We can use lockstat to try and examine what the system is doing:

# lockstat sleep 10

Adaptive mutex spin: 19132 events

Count indv cuml rcnt     spin Lock                   Caller

-------------------------------------------------------------------------------

 2716  14%  14% 1.00        1 flock_lock             cleanlocks+0x1c

 2388  12%  27% 1.00        3 0xe0452ab0             dev_get_dev_info+0x3d

 1905  10%  37% 1.00        3 0xe0452ab0             cdev_size+0x48

 1554   8%  45% 1.00        1 0xe07c9f30             vn_rele+0x22

 1307   7%  52% 1.00        5 stable_lock            specvp+0x8f

 1050   5%  57% 1.00        2 0xe0452ab0             mod_rele_dev_by_major+0x21

  995   5%  62% 1.00        2 0xe0452ab0             ddi_hold_installed_driver+0x36

...

-------------------------------------------------------------------------------

...

Unfortunately, lockstat is quite verbose by default. However, at first glance it appears that the system is spending most of its time working with the flock_lock lock, which is filesystem-related. From this point it's a matter of using truss to see what system calls in our application are generating these contentions.

If you have a suspicion as to what lock might be the one causing the problem, lockstat has the capability to search for a particular lock. This is accomplished with the -l lockname switch:

# lockstat -l flock_lock sleep 10

Adaptive mutex spin: 2594 events

Count indv cuml rcnt     spin Lock                   Caller

-------------------------------------------------------------------------------

 2594 100% 100% 1.00        1 flock_lock             cleanlocks+0x1c

-------------------------------------------------------------------------------

Adaptive mutex block: 9 events

Count indv cuml rcnt     nsec Lock                   Caller

-------------------------------------------------------------------------------

    9 100% 100% 1.00 64490487 flock_lock             cleanlocks+0x1c

-------------------------------------------------------------------------------

3.6.7 Controlling Processors in Solaris

Solaris 2.6 and later versions support commands to help the system administrator manage processors. They are of somewhat limited use in most performance tuning applications, but they are good to know about.

3.6.7.1 psrinfo

The psrinfo command displays information concerning the configured processors. It provides particularly useful information when the -v switch is applied.

% /usr/sbin/psrinfo -v

Status of processor 0 as of: 09/20/99 16:24:34

  Processor has been on-line since 09/01/99 06:33:49.

  The sparc processor operates at 200 MHz,

        and has a sparc floating point processor.

Status of processor 1 as of: 09/20/99 16:24:34

  Processor has been on-line since 09/01/99 06:33:52.

  The sparc processor operates at 200 MHz,

        and has a sparc floating point processor.

This utility has proved to be useful when dealing with clients who were not sure what sort of hardware they had, but wanted to have it tuned. Similar information can be obtained by examining the contents of /proc/cpuinfo in Linux.

3.6.7.2 psradm

You can change the operational status of processors with the psradm command. A processor can have the following states:

  • An on-line processor does computation[26] and may be interrupted by the system's I/O devices.

    [26] That is, an on-line processor processes processes.

  • An off-line processor does not do any computation and is not usually interruptible. In some cases, it may not be possible to disable interrupts for an off-line processor.

  • A no-intr processor performs computation, but is not interruptible.

At least one processor must be able to perform computation, and at least one processor must be able to be interrupted.

3.6.7.3 psrset

It is sometimes useful to create processor sets through psrset. A processor set allows you to bind a process to a particular group of processors, rather than just one. Processor sets are sometimes created by the system, in which case they will run both system processes and user processes, or they can be created by psrset, in which case they are only able to run processes that have been bound to that processor. Let's walk through an example of using psrset on a Sun Ultra Enterprise 3500 system, and then binding a process to that processor set.

The system I use for this example has eight 250 MHz UltraSPARC-II processors:

# psrinfo

6       on-line   since 11/15/01 16:47:33

7       on-line   since 11/15/01 16:47:40

10      on-line   since 11/15/01 16:47:40

11      on-line   since 11/15/01 16:47:40

14      on-line   since 11/15/01 16:47:40

15      on-line   since 11/15/01 16:47:40

18      on-line   since 11/15/01 16:47:40

19      on-line   since 11/15/01 16:47:40

You can use psrset -i to see what the defined processor sets on a system are (for a listing by processor instead of by set, you can use psrset -p). By default, there are none -- all processes are in a general execution pool. Let's create a processor set with four processors (those numbered 14, 15, 18, and 19) using psrset -c:

# psrset -i

# psrset -c 14 15 18 19

created processor set 1

processor 14: was not assigned, now 1

processor 15: was not assigned, now 1

processor 18: was not assigned, now 1

processor 19: was not assigned, now 1

# psrset -i

user processor set 1: processors 14 15 18 19

# psrset -p

processor 14: 1

processor 15: 1

processor 18: 1

processor 19: 1

If you want to add more processors to this set, you can use psrset -a set-id processor-list; similarly, you can remove them with psrset -r processor-list.

# psrset -i

user processor set 1: processors 14 15 18 19

# psrset -a 1 10 11

processor 10: was not assigned, now 1

processor 11: was not assigned, now 1

# psrset -i

user processor set 1: processors 14 15 18 19 10 11

# psrset -r 18 19

processor 18: was 1, now not assigned

processor 19: was 1, now not assigned

# psrset -i

user processor set 1: processors 14 15 10 11

You can completely remove a processor set with psrset -d set-id:

# psrset -i

user processor set 1: processors 14 15 10 11

# psrset -d 1

removed processor set 1

# psrset -i

#

If you are configuring a mixed-use system -- for example, you might have to support both software development and production computational codes on the same system -- you can use processor sets to segment your workload. For example, on an eight-processor system, you might create one two-processor set dedicated for compiles, one four-processor set for computational jobs, and leave the other two processors unassigned (and therefore available for interactive use). One thing that you might want to do in that case is to disallow the processors that are dedicated to compilation and computation from handling interrupts (e.g., for disk or network I/O). This can be done by running psrset -f set-id; alternatively, you might want to set aside a few processors just for handling interrupts, in which case you can turn on interrupt handling for a set by using psrset -n set-id. By default, all processors are available to handle interrupts. For example, let's create a four-CPU processor set and disable interrupt handling on it:

# psrinfo

6       on-line   since 11/15/01 16:47:33

7       on-line   since 11/15/01 16:47:40

10      on-line   since 11/15/01 16:47:40

11      on-line   since 11/15/01 16:47:40

14      on-line   since 11/15/01 18:26:23

15      on-line   since 11/15/01 18:26:23

18      on-line   since 11/15/01 18:26:23

19      on-line   since 11/15/01 18:26:23

# psrset -i

# psrset -c 14 15 18 19

created processor set 1

processor 14: was not assigned, now 1

processor 15: was not assigned, now 1

processor 18: was not assigned, now 1

processor 19: was not assigned, now 1

# psrset -f 1

# psrinfo

6       on-line   since 11/15/01 16:47:33

7       on-line   since 11/15/01 16:47:40

10      on-line   since 11/15/01 16:47:40

11      on-line   since 11/15/01 16:47:40

14      no-intr   since 11/15/01 18:27:09

15      no-intr   since 11/15/01 18:27:09

18      no-intr   since 11/15/01 18:27:09

19      no-intr   since 11/15/01 18:27:09

Note that deleting a processor set that had interrupts disabled will not have the side effect of reenabling interrupts on its member processors. You can always reenable interrupts on a per-CPU basis with psradm -n processor-list.

Now that we've created a processor set, how can we run processes on it? There are essentially two ways: we can use psrset -e set-id command to run a command directly on the processor set, or we can use psrset -b pid-list to bind processes to a set.[27] For example, let's run a multithreaded application on processor set 1:

[27] Actually, it binds the underlying lightweight processes to the set (for more information on lightweight processes, see Section 3.3.2.1 earlier in this chapter).

# psrset -i

user processor set 1: processors 14 15 18 19

# psrset -e 1 mtsieve &

[1] 3701

# mpstat 3

...

CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl

  6    0   0  747   401  301    0    0    0    0    0     0    0   0   0 100

  7    0   0    0   104  104   45    0    0    0    0    27    0   0   0 100

 10    0   0    0   100  100   13    0    0    0    0     9    0   0   0 100

 11    0   0    0   100  100   11    0    0    0    0     3    0   0   0 100

 14    0   0    0   111  100   11   11    0    0    0     0  100   0   0   0

 15    0   0    0   111  100   11   11    0    0    0     0  100   0   0   0

 18    0   0    0   111  100   11   11    0    0    0     0  100   0   0   0

 19    0   0    0   111  100   11   11    0    0    0     0  100   0   0   0

...

We can clearly see from the mpstat output that the CPUs in the processor set are completely saturated, but the other processors are totally idle.

3.6.8 Peripheral Interconnect Performance Tools

There are very few tools to monitor the performance of peripheral device interconnects. By far the best is the Solaris tool busstat, which lets you monitor the state of certain performance counters that have been embedded in Sun hardware. Because of the high degree of hardware support that is required, the capabilities of busstat may change over time, and from platform to platform. These examples were gathered on a Sun Ultra Enterprise 3500 running Solaris 8 7/01.

The best way to start using busstat is to see what devices are available for monitoring. This can be done via busstat -l:

# busstat -l

Busstat Device(s):

sbus0 sbus1 ac0 ac1 ac2 ac3 ac4

There are two sbus devices, which correspond to the two SBuses on this system's single SBus I/O card (see Section 3.5.1.4 earlier in this chapter). The ac devices correspond to the Address Controllers, which are on each installed board in the system. Each device has two performance instrumentation counters (PICs) that can count specific events. In order to get a list of the available events for each device, use busstat -e device:

# busstat -e sbus0

pic0

dvma_stream_rd

dvma_stream_wr

dvma_const_rd

dvma_const_wr

dvma_tlb_misses

dvma_stream_buf_mis

dvma_cycles

dvma_bytes_xfr

interrupts

upa_inter_nack

pio_reads

pio_writes

sbus_reruns

pio_cycles

pic1

...

Table 3-10 contains a description of the most important of these parameters.

Table 3-10. SBus performance counter events

Event

Description

dvma_stream_rd

The number of DVMA streaming reads issued

dvma_stream_wr

The number of DVMA streaming writes issued

dvma_const_rd

The number of DVMA consistent reads issued

dvma_const_wr

The number of DVMA consistent writes issued

dvma_tlb_misses

The number of transaction lookaside buffer (TLB) misses incurred during DVMA operations

dvma_stream_buf_mis

The number of DVMA streaming buffer read misses

dvma_cycles

The number of SBus cycles granted to DVMA operations

dvma_bytes_xfr

The number of bytes transferred across the SBus

interrupts

The number of interrupts generated by the SBus

upa_inter_nack

The number of nonacknowledged (on UPA) interrupts generated by the SBus

pio_reads

The number of programmed I/O reads issued

pio_writes

The number of programmed I/O writes issued

sbus_reruns

The number of SBus reruns

pio_cycles

The number of SBus cycles granted to PIO operations

And we can specify the counters that are to be examined by using busstat -w device,pic0=event,pic1=event. These counters will remain programmed until they are reset; the counters can only be programmed by the superuser.

These are obviously extremely detailed events, and may not be of great interest to anyone except driver developers or hardware engineers. However, one interesting thing to look at is the number of bytes transferred across the SBus. For example, we can use busstat to watch the activity on an SBus containing a network interface that is attached to a SunRay network appliance:[28]

[28] The SunRay is completely responsible for transmitting screen redrawing instructions on the host system. It functions somewhat like an X terminal.

# busstat -w sbus1,pic0=dvma_cycles,pic1=dvma_bytes_xfr 1

time dev    event0               pic0        event1               pic1

1    sbus1  dvma_cycles          1449        dvma_bytes_xfr       1940

2    sbus1  dvma_cycles          1361        dvma_bytes_xfr       1800

...

12   sbus1  dvma_cycles          1328        dvma_bytes_xfr       1800

13   sbus1  dvma_cycles          3469        dvma_bytes_xfr       6424

Up until this point, the system is simply sitting at the dtlogin screen. The small amount of data transfer is probably due to the "blinking" of the cursor. We then log in by typing a valid username and password:

14   sbus1  dvma_cycles          7282        dvma_bytes_xfr       19812

15   sbus1  dvma_cycles          7620        dvma_bytes_xfr       21172

16   sbus1  dvma_cycles          1399        dvma_bytes_xfr       2352

At this point we have successfully logged in, and the CDE "Welcome to Solaris" screen appears, followed by the user's desktop environment (twm and a few xterm processes):

17   sbus1  dvma_cycles          182874      dvma_bytes_xfr       590212

18   sbus1  dvma_cycles          1361        dvma_bytes_xfr       2004

19   sbus1  dvma_cycles          247957      dvma_bytes_xfr       814420

20   sbus1  dvma_cycles          2691        dvma_bytes_xfr       3688

21   sbus1  dvma_cycles          13542       dvma_bytes_xfr       25392

22   sbus1  dvma_cycles          7931        dvma_bytes_xfr       14248

After a little while, the SBus activity returns to near-zero levels, as there is only minimal graphical update activity occuring on the network:

31   sbus1  dvma_cycles          225         dvma_bytes_xfr       536

32   sbus1  dvma_cycles          597         dvma_bytes_xfr       760

Although this is a very limited example, it does show some of the powerful analysis made possible by busstat.

3.6.9 Advanced Processor Performance Statistics

The cpustat and cputrack tools in Solaris can be used to obtain extremely detailed statistics about processor performance. cpustat generates system-wide statistics, whereas cputrack lets you gather data about a specific process.

You can obtain the list of available events via cpustat -h. Some events are only available on specific PICs. Table 3-11 summarizes the various events, along with what PICs they are available on.

Table 3-11. CPU performance counter events

Event

Available PICs

Description

Cycle_cnt

0, 1

The number of accumulated clock cycles

Instr_cnt

0, 1

The number of completed instructions

Dispatch0_IC_Miss

0

The number of instruction cache misses

Dispatch0_mispred

1

The number of mispredicted branches (note that branch misprediction typically causes instructions in flight to be killed, so the number of pipeline bubbles is about twice as big as this count)

Dispatch0_storeBuf

0

The number of times that a store instruction was issued, but store buffer was full, and could not hold additional writes to memory

Dispatch0_FP_use

1

The number of times that an instruction depended on a floating-point result that was not yet available for reasons other than loading (that is, Dispatch0_FP_use and Load_use are mutually exclusive)

Load_use

0

The number of times that an instruction waiting to be executed required a load from memory that hadn't yet completed; stalls all instructions in the pipeline

Load_use_RAW

1

The number of times that a load-use hazard exists, but there is a read-after-write hazard on the oldest outstanding load. This indicates that load data is being delayed until an earlier store can be completed

IC_ref

0

The number of instruction cache references

IC_hit

1

The number of instruction cache hits

DC_rd

0

The number of data cache read references

DC_rd_hit

1

The number of data cache read hits

DC_wr

0

The number of data cache writes

DC_wr_hit

1

The number of data cache write hits

EC_ref

0

The number of E-Cache (L2 cache) references

EC_hit

1

The number of E-Cache hits

EC_write_hit_RD0

0

The number of E-Cache write hits that result in ownership being transferred

EC_wb

1

The number of E-Cache misses that incur writebacks to main memory

EC_snoop_inv

0

The number of E-Cache invalidations driven by bus snooping

EC_snoop_cb

1

The number of E-Cache copybacks driven by bus snooping

EC_rd_hit

0

The number of E-Cache read hits after a data cache miss

EC_ic_hit

1

The number of E-Cache read hits after an instruction cache miss

Again, some of these counters refer to very low-level performance structures, and will primarily be of use to people who are intimately familiar with the architecture of their processor. For more information about the UltraSPARC processor family architecture, consult the user guides published by Sun Microelectronics at http://www.sun.com/microelectronics/manuals/index.html.

One thing that is often interesting to compute is the number of cycles per instruction. For this example, I draw upon the simple Sieve of Eratosthenes application source code (see Section 8.2.4.2), with the minor modification that it will not display output (simply compute it). I run an experiment to compare the number of cycles per instruction required as a function of the compiler (Forte 6 Update 1) optimization level. Gather cputrack data for each process like this:

% cputrack -c pic0=Cycle_cnt,pic1=Instr_cnt sieve.o0

   time lwp      event      pic0      pic1

  1.046   1       tick 170075841 128955109

  2.056   1       tick 221955530 162910824

  3.056   1       tick 240994274  93274214

  4.056   1       tick 231824966  52868531

  5.056   1       tick 198733670  53220280

  6.056   1       tick  92304003  51971543

  6.522   1       exit 1192528514 564103611

We then compute, using the last line (the final statistics), the average number of cycles per instruction by dividing the pic0 entry (the number of cycles) by the pic1 entry (the number of instructions) to give a CPI of 2.114. As we increase the optimization, we get the results given here.

Compiler flags

Total instructions

Cycles per instruction

Total runtime (seconds)

-xO0

564,103,381

2.114

6.507

-xO2

168,642,699

5.022

5.480

-xO3

137,109,359

5.929

5.275

-fast -xO5

138,521,627

5.684

5.213

It's dangerous to overgeneralize these performance results, but it's interesting to watch the tremendous CPI jump as you move from no optimization to some optimization. Clearly, further optimization gives decreasing returns.

As we continue to further our understanding, we can use cputrack to try and answer some questions we might have about this application's runtime behavior. For example, we can compute the instruction cache hit rate for the unoptimized application:

%  ptime cputrack -c pic0=IC_ref,pic1=IC_hit sieve.o0

   time lwp      event      pic0      pic1

...

  6.475   1       exit 179350516 179325008

Therefore, we can compute that we have instruction cache hit rate of 99.986%, which is incredibly high. We can also determine the data cache read hit rate and data cache write hit rate:

% cputrack -c pic0=DC_rd,pic1=DC_rd_hit sieve.o0

   time lwp      event      pic0      pic1

...

  6.469   1       exit 128490011 128460760

real        6.509

user        0.038

sys         0.063

% cputrack -c pic0=DC_wr,pic1=DC_wr_hit sieve.o0

   time lwp      event      pic0      pic1

...

  6.471   1       exit  69216839  33654774

Finally, we end up with a data cache read hit rate of 99.977% and a write hit rate of 48.622%. At this point, things start to get interesting. We've been analyzing the unoptimized application, and our data cache write hit rate seems a little bit low. In fact, we might wonder whether it goes up as the optimization level increases. When we run that experiment, we find out that the write cache hit rate in the highest-optimized (and lowest-runtime) version has a write cache hit rate of only 0.031%. This seems confusing at first, but upon some consideration it makes sense: the application is performing at peak rates when it doesn't write often to the same memory location. This is a good illustration of how confusing examining these low-level counters can be without a great deal of knowledge about the application under analysis.


3.7 Concluding Thoughts

Microprocessors are some of the most complex devices per square centimeter that humanity has ever constructed, and understanding how they function is extremely challenging. I hope this chapter has given you some sense of that complexity, along with furthering your understanding and helping you prepare to measure a processor's performance in a dynamic system.

We've covered a lot, from the fundamentals of microprocessor design, to bus and crossbar architectures, to discussing caching and peripheral interconnects. This is a lot of information to grasp, but the payoff of understanding how these components function and interact with each other is commensurately large.


Chapter 4. Memory

Ideally one would desire an indefinitely large memory capacity such that any particular. . . word would be immediately available. . . .

—A. W. Burks, H. H. Goldstine, and J. von Neumann: Preliminary Discussion of the Logical Design of an Electronic Computing Instrument, 1946

Physical memory is a set of integrated circuits designed to store binary data. This storage has two characteristic properties: it is transient, as all stored information vanishes when electrical power is lost, and randomly accessible, meaning any bit can be accessed as fast as any other bit. In addition to physical memory, most systems implement virtual memory, which acts to manage physical memory and provide a simple interface to application developers. Virtual memory is consumed by the system kernel, filesystem caches, intimately shared memory, and processes.

Memory performance begins to affect overall system performance in two instances. The first instance occurs when the system is unable to retrieve and store data from physical memory fast enough, or when the system is forced to travel to main memory frequently. This sort of problem can be attacked by tuning the algorithm that is responsible or by buying a system with faster access to main memory. The second, and more likely, case is that the demand for physical memory by all currently running applications, including the kernel, exceeds the available amount. The system is then forced to begin paging , or writing unused pieces of memory to disk. If the low memory condition worsens, the memory consumed by entire processes will be written to disk, which is called swapping .[1] Memory conditions fall into four categories:

[1] Paging and swapping are often used interchangeably. However, they mean very different things in practice, so we'll be careful not to mix them.

  • Sufficient memory is available, and the system performs optimally.

  • Memory is constrained (one likely culprit, especially on older Solaris systems, is the filesystem cache). Performance begins to suffer as the system attempts to scavenge memory that is not in active use.

  • The system is legitimately short of memory. Performance suffers, especially on interactive processes.

  • Memory is critically scarce. Swapping begins to take place. System performance takes a heavy hit, and interactive performance becomes abysmal.

In this chapter, I'll describe how memory is physically implemented, the mechanics of how the system manages memory, and how paging and swapping function at a system level. I'll give you some tools you can use to analyze your memory use, and explain how to work with swap space. Finally, I'll cover mechanisms to address some common memory performance problems.


4.1 Implementations of Physical Memory

Let's start by looking at how memory is physically implemented in modern systems. All modern, fast memory implementations are accomplished via semiconductors,[2] of which there are two major types: dynamic random access memory (DRAM) and static random access memory (SRAM). The difference between them is how each memory cell is designed. Dynamic cells are charge-based, where each bit is represented by a charge stored in a tiny capacitor. The charge leaks away in a short period of time, so the memory must be continually refreshed to prevent data loss. The act of reading a bit also serves to drain the capacitor, so it's not possible to read that bit again until it has been refreshed. Static cells, however, are based on gates, and each bit is stored in four or six connected transistors. SRAM memories retain data as long as they have power; refreshing is not required. In general, DRAM is substantially cheaper and offers the highest densities of cells per chip; it is smaller, less power-intensive, and runs cooler. However, SRAM is as much as an order of magnitude faster, and therefore is used in high-performance environments. Interestingly, the Cray-1S supercomputer had a main memory constructed entirely from SRAM. The heat generated by the memory subsystem was the primary reason that system was liquid-cooled.

[2] In some cases, such as where resistance to ionizing radiation is required, magnetic core memory is still used.

There are two primary performance specifications for memory. The first represents the amount of time required to read or write a given location in memory, and is called the memory access time. The second, the memory cycle time, describes how frequently you can repeat a memory reference. They sound identical, but they are often quite different due to phenomena such as the need to refresh DRAM cells.

There is quite a gap between the speed of memory and the speed of microprocessors. In the early 1980s, the access time of commonly available DRAM was about 200 ns, which was shorter than the clock cycle of the commonly used 4.77 MHz (210 ns) microprocessors of the day. Fastforwarding two decades, the clock cycle time of the average home microprocessor is down to about a nanosecond (1 GHz), but memory access times are hovering around 50 ns.

Many different kinds of memory modules have been developed to improve system performance. I'll present a few here.

  • The oldest option currently in use is fast-page mode (FPM) memory. It implements the ability to read a full page (4 KB or 8 KB) of data during a single memory access cycle.

  • A refinement of this technique goes into extended data output (EDO) memory. The refinements are largely based on electrical modifications.

  • A more revolutionary change was implemented in synchronous DRAM (SDRAM) memory, which uses a clock to synchronize the input and output of signals. This clock is coordinated with the CPU clock, so the timings of all the components are synchronized. SDRAM also implements two memory banks on each module, which essentially doubles the memory throughput; it also allows multiple memory requests to be pending at once. A variation on SDRAM, called double-data rate SDRAM (DDR SDRAM) is able to read data on both the rising and falling edges of the clock, which doubles the data rate of the memory chip.

    SDRAM modules are usually described as PC66, PC100, or PC133, which refers to the clock rate they are driven at: 66 MHz (15 ns), 100 MHz (10 ns), or 133 MHz (8 ns), respectively. DDR SDRAM modules, however, are usually referred to by their peak throughput; a PC2100 module, for example, is theoretically capable of about 2.1 GB/sec peak throughput.

  • Direct Rambus (RDRAM) memory, however, takes an entirely different approach. It uses a narrow (16 bits wide) but extremely fast (600-800 MHz) path to memory, and allows sophisticated pipelining of operations. It is widely used in high-performance embedded applications (e.g., Sony's PlayStation 2 and the Nintendo64 consoles). RDRAM modules are usually called PC600, PC700, or PC800, which refers to their clock rate.

Workstations and servers tend to be able to interleave memory across banks, which is conceptually similar to disk striping (see Section 6.2.1); if we have 128 MB of memory configured as four 32 MB modules, we store one bit on each module in turn, rather than the first 32 MB on the first module, the second 32 MB on the second module, etc. This allows much higher performance because of the avoidance of cycle time delays. Interleaving is almost always done in power-of-two increments.[3]

[3] One exception is the Sun Ultra Enterprise hardware line, which will create 6-way interleaves.

It can also lead to confusion. Let's say we have a system with seven memory banks filled. The memory controller may well decide to make a four-way interleave, a two-way interleave, and a one-way interleave, which would mean that a memory-intensive process's performance would depend on where exactly it fell in memory. The best resource on how to configure your system's memory for optimum interleaving is the hardware vendor.


4.2 Virtual Memory Architecture

A virtual memory system exists to provide a framework for the system to manage memory on the behalf of various processes. The virtual memory system provides two primary benefits. It allows software developers to write to a simple memory model, which shields the programmer from the memory subsystem's hardware architecture and allows the use of memory sizes substantially greater than physical memory through backing stores. It also permits processes to have nonfragmented address spaces, regardless of how physical memory is organized or fragmented. In order to implement such a scheme, four key functions are required.

First, each process is presented with its own virtual address space; that is, it can "see" and potentially use a range of memory. This range of memory is equal to the maximum address size of the machine. For example, a process running on a 32-bit system will have a virtual address space of about 4 GB (232). The virtual memory system is responsible for managing the associations between the used portions of this virtual address space into physical memory.

Second, several processes might have substantial sharing between their address spaces. For example, let's say that two copies of the shell /bin/csh are running. Both copies will have separate virtual address spaces, each with their own copy of the executable itself, the libc library, and other shared resources. The virtual memory system transparently maps these shared segments to the same area of physical memory, so that multiple copies are not stored. This is analogous to making a hard link in a filesystem instead of duplicating the file.

However, sometimes physical memory will become insufficient to hold the used portions of all virtual address spaces. In this case, the virtual memory system selects less frequently used portions of memory and pushes them out to secondary storage (e.g., disk) in order to optimize the use of physical memory.

Finally, the virtual memory system plays one of the roles of an elementary school teacher: keeping the children in his care from interfering with each other's private things. Hardware facilities in the memory management unit perform this function by preventing a process from accessing memory outside its own address space.

4.2.1 Pages

Just like energy in quantum mechanics, memory is quantized; that is, it is organized into indivisible units. These units are called pages. The exact size of a page varies from system to system and is dependent on the processor's memory management unit (MMU) implementation. Large page sizes reduce MMU activity by reducing the number of page faults, and save kernel memory. However, they also waste some memory, since the system can only deal with memory in page-sized segments. A request for less than one full page of memory will be answered with the allocation of a full page. Typically, page sizes are 4 KB or 8 KB. On Solaris systems, you can find the page size via /usr/bin/pagesize or getpagesize(3C), while on Linux it is defined in the kernel header file asm/param.h as EXEC_PAGESIZE.

Page Sizes on Various Architectures

There are only three commonly encountered microprocessor designs that implement an 8 KB page size: the DEC Alpha, the original Sun SPARC processors (e.g., the Ross RT601/Cypress CY7C601/Texas Instruments TMS390C601A, which were used in the SPARCstation 2), and the Sun UltraSPARC designs. The Intel 80x86, the MIPS processors used in SGI systems, the Motorola/IBM PowerPC, and Sun's microSPARC and SuperSPARC series processors all use a 4 KB page size.

4.2.2 Segments

The pages that compose a process are grouped into several segments. Each process has at least four of these segments:

Executable text

Consists of the actual executable instructions in the binary. This is mapped from the on-disk binary and is read-only.

Executable data

Contains initialized variables present in the executable. This is mapped from the on-disk binary, but has permissions set to read/write/private (the private mapping ensures that changes to these variables during runtime are not reflected in the on-disk file, or in other processes sharing the same executable).

Heap space

Consists of memory allocated by means of malloc(3). This is called anonymous memory, because it has no mapping in the filesystem (we'll discuss it further in Section 4.3.2.1 later in this chapter).

Stack

Also allocated from anonymous memory.

4.2.3 Estimating Memory Requirements

Sometimes you have it easy, and a system only needs to run one commercial software package, for which the vendor has nicely told you precisely how much memory you need for optimal performance. Unfortunately, the real world is rarely so nice: the systems administrator in a typical computing environment has hundreds of different processes to worry about. Rather than try and solve every possible problem, we'll work out a general mechanism for thinking about how much memory is required for a given system.

The most important thing is to establish what a reasonable workload is on the system. Let's say that you have a system that's being used for serving typical low-end users: they run a shell, maybe an editor, a mailreader (probably pine or elm), or perhaps a newsreader. If you've been carefully monitoring your current environment, you'll be able to get a handle on how many of each sort of process is running at peak usage times. If you haven't, or if you're starting from scratch, you'll need to make a reasonable guess. Let's say that you've decided that there will be 50 users logged in at peak hours, and the usage pattern works out like this:

  • 5 invocations each of ksh and csh, 40 of tcsh

  • 25 pine processes and 10 elm processes

  • Other applications that defy categorization

We'll work through calculating the memory requirements of the csh processes. If we run pmap on a representative process, we get output that resembles this:

% pmap -x 7522

7522:   -csh

Address   Kbytes Resident Shared Private Permissions       Mapped File

0803C000      48      48       -      48 read/write/exec    [ stack ]

08048000     116     116     116       - read/exec         csh

08065000      16      16       -      16 read/write/exec   csh

08069000      72      72       -      72 read/write/exec    [ heap ]

DFF19000     548     444     408      36 read/exec         libc.so.1

DFFA2000      28      28       4      24 read/write/exec   libc.so.1

DFFA9000       4       4       -       4 read/write/exec    [ anon ]

DFFAB000       4       4       4       - read/exec

libmapmalloc.so.1

DFFAC000       8       8       -       8 read/write/exec

libmapmalloc.so.1

DFFAF000     148     136     136       - read/exec         libcurses.so.1

DFFD4000      28      28       -      28 read/write/exec   libcurses.so.1

DFFDB000      12       -       -       - read/write/exec    [ anon ]

DFFDF000       4       4       -       4 read/write/exec    [ anon ]

DFFE1000       4       4       4       - read/exec         libdl.so.1

DFFE3000     100     100     100       - read/exec         ld.so.1

DFFFC000      12      12       -      12 read/write/exec   ld.so.1

--------  ------  ------  ------  ------

total Kb    1152    1024     772     252

It looks like each process is taking about 1,150 KB of memory, with 1,024 KB resident in memory. Of that 1,024 KB, 772 KB is shared with other processes, and 252 KB is private. So, for five invocations, we could roughly estimate memory usage at roughly 2 MB (about 770 KB shared, plus five times about 250 KB). These are back-of-the-envelope calculations, but they are remarkably good at giving you an idea how much memory you need.

Remember, though, that memory is consumed by three other things (the filesystem cache, intimately shared memory, and the kernel) aside from processes! Unless you're running Oracle or another database application, you probably won't need to worry about intimately shared memory. As a rule of thumb, you can safely figure on about 32 MB for the kernel and other applications, plus another 16 MB if you're running a windowing system. If your users only access a few hundred megabytes of data, but access it frequently, it may be advisable to buy sufficient memory to cache the entire data set. This will drastically improve I/O performance to that information.

4.2.4 Address Space Layout

The exact details of a process's address space vary depending on the architecture it is implemented on. For Solaris systems, there are essentially four address space layouts:

  • SPARC V7 32-bit combined kernel and process address space (used on sun4c, sun4m, and sun4d architectures)

  • SPARC V9 32-bit separated kernel and process address space (used on sun4u machines)

  • SPARC V9 64-bit separated kernel and process address space (used on sun4u machines)

  • Intel IA-32 32-bit combined kernel and process address space (used on Solaris for Intel systems)

The SPARC V7 32-bit combined model maps the kernel's address space into the top of the process's address space. This means that the virtual address space available to the process is restricted by the amount consumed by the kernel's address space (256 MB on sun4c and sun4m architectures, and 512 MB on sun4d architectures). The kernel's address space is protected from the user's process by means of the the processor's privilege levels. The stack begins just beneath the kernel at address 0xEFFFC000 (0xDFFFE000 on sun4d systems) and grows downwards to address 0xEF7EA000 (0xDF7F9000 on sun4d), where libraries are loaded into memory. The text and data portions of the executable are loaded into the bottom of the address space, and the heap grows from the top of those upwards towards the libraries.

The UltraSPARC microprocessor architecture allows the kernel its own private address space, which removes the size limit on the kernel's address space.[4] Thus, for the 32-bit sun4u memory model, the stack begins at address 0xFFBEC000 and grows downwards until 0xFF3DC000 (the small space at the absolute top of the address space is reserved for the OpenBoot PROM). Otherwise, it is the same as the SPARC V7 models.

[4] This was a significant problem in the large pre-UltraSPARC systems, such as the SPARCcenter 2000E.

Although the UltraSPARC processor supports the SPARC V9 64-bit mode, the UltraSPARC-I and UltraSPARC-II processor implementations allow only 44-bit physical memory addresses. This creates a "hole" in the middle of the virtual address space, which spans addresses 0xFFFFF7FF.FFFFFFFF to 0x00000800.00000000.

The Intel architecture address space is similar to the sun4c/sun4m SPARC V7 model, in that is does not separate user space from kernel space. However, it has a significant difference: the stack is mapped beneath the executable segments (starting at address 0x8048000) and permitted to grow downwards to the very bottom of the address space.

4.2.5 The Free List

The free list is the mechanism by which the system dynamically manages the flow of memory between processes. Memory is taken from the free list by processes, and returned to it when the process exits or by the action of the page scanner, which tries to ensure that there is a small amount of memory free for immediate use at all times. Every time memory is requested, a page fault is incurred. There are three kinds of faults, which we'll discuss in more detail later in this section:

Minor page fault

Occurs every time a process needs some memory, or when a process tries to access a page that has been stolen by the page scanner but not recycled.

Major page fault

Occurs when a process tries to access a page that has been taken from it by the page scanner, completely recycled, and is now being used by another process. A major page fault is always preceded by a minor page fault.

Copy-on-write fault

Caused by a process trying to write to a page of memory that it shares with other processes.

Let's examine how the free list is managed under a Solaris system.

When a system boots, all of its memory is formed into pages, and a kernel data structure is created to hold their states. The kernel reserves a few megabytes of memory for itself and releases the rest to the free list. At some point, a process will request memory, and a minor page fault occurs. A page is then taken from the free list, zeroed, and made available to the process. This sort of behavior, in which memory is given out on an "as-needed" basis, has traditionally been called demand paging.

Pages are always taken from the head of the free list.

When the free list shrinks to a certain size (set to lotsfree in units of pages), the kernel wakes up the page scanner (also called the pagedaemon), which begins to search for pages that it can steal to replenish the free list. The page scanner implements a two-step algorithm order to avoid stealing pages that are being frequently accessed. The pagedaemon looks through memory in physical memory order and clears the MMU reference bit for each page. When a page is accessed, this bit is set. The page scanner then delays for a short while, waiting for pages to be accessed and their reference bits cleared. This delay is controlled by two parameters:

  • slowscan is the initial scan rate. Increasing this value causes the page scanner to run less unnecessary jobs, but do more work.

  • fastscan is the scan rate when the free list is completely empty.

The pagedaemon then goes through memory again. If a particular page still has the reference bit cleared, then the page hasn't been touched, so it is stolen for recycling. Think of memory as a circular track. A train rides on this track; as the engine in the front of the train passes a tie, that tie's reference bit is cleared. When the caboose gets around to a tie, if that tie's reference bit is still cleared, that page is recycled.

Some pages are very resistant to recycling, such as those that belong to the kernel and those that are shared between more than eight processes (this helps to keep shared libraries in memory). If the stolen page does not contain cached filesystem data, then it is moved to the page-out queue, marked as pending I/O, and eventually written out to swap space with a cluster of other pages. If the page is being used for filesystem caching, it is not written to disk. In either case, the page is placed on the end of the free list, but not cleared; the kernel remembers that this particular page still stores valid data.

If the average size of the free list over a 30-second interval is less than desfree, the kernel begins to take desperate measures to free memory: inactive processes are swapped out, and pages are written out immediately rather than being clustered. If the 5-second average of free memory is less than minfree, active processes start to be swapped out. The page scanner stops looking for memory to reclaim when the size of the free list increases above lotsfree.

The page scanner is governed by a parameter, maxpgio, which limits the rate at which I/O occurs to the paging devices. It is, by default, set very low (40 pages per second on sun4c, sun4m, and sun4u architectures, and 60 pages per second on sun4d), in order to prevent saturating the paging device with I/Os. This rate is inadequate for a modern system with fast disks, and should be set to 100 times the number of spindles of configured swap space. If a process tries to access a recycled page, another minor page fault is incurred. There are now two possibilities:

  • The page that was recycled has been placed back on the free list, but hasn't been reused yet. The kernel gives the page back to the process.

  • The page has been recycled completely, and is now in use by another process. A major page fault is incurred and the paged-out data is read into a new page, taken from the free list.

Since pages can be shared between multiple processes, special precautions need to be taken in order to ensure that changes to a page are maintained locally to that process. If a process tries to write to a shared page, it incurs a copy-on-write fault.[5] A page is taken from the free list, and a copy of the original shared page is made for the process. When a process completes, all of its nonshared pages are returned to the free list.

[5] Often called a COW fault; not to be confused with a ruminant falling into a chasm.

4.2.5.1 Virtual memory management in Linux

This mechanism is used in most modern memory management systems, but often with some minor changes. Here, I discuss the Linux 2.2 kernel, as at the time of this writing the 2.4 kernel was still undergoing significant revisions to the VM subsystem. The Linux kernel implements a different set of kernel variables that tune memory performance a little bit differently.

When the size of a Linux system's free list drops below the value of freepages.high, the system begins to gently page. When the available memory drops below the freepages.low value, the system pages heavily, and if the free list shrinks beneath freepages.min, only the kernel can allocate more memory. The freepages tunables are located in the /proc/sys/vm/freepages file, which is formatted as freepages.min freepages.low freepages.high.

The pagedaemon under Linux is called kswapd, which frees as many pages as necessary to get the system's free list back over the freepages.high mark. kswapd's behavior is controlled by three parameters, called tries_base, tries_min, and swap_cluster, located in that order in the /proc/sys/vm/kswapd file. The most important tunable is swap_cluster, which is the number of pages that kswapd will write out in a single turn. This value should be large enough so that kswapd does its I/O in large chunks, but small enough that it won't overload the disk I/O request queue. If you find yourself getting short on memory and paging heavily, you probably will want to experiment with increasing this value, to buy yourself more bandwidth to the paging space.

When a page fault is taken in Linux, multiple pages are actually read in to avoid multiple short trips to disk; the precise number of pages that are brought into memory is given by 2n, where n is the value of the page-cluster tunable (the only value in the /proc/sys/vm/page-cluster tunable file). It is pointless to set this value above 5.

There are two important things to take from the page lifecycle described here. The first is the manner by which the page scanner operates, and the parameters that define its behavior. The page scanner manages the ebb and flow of the free list by controlling paging, which can have a tremendous impact on performance. The second is the way pages are allocated and released; minor page faults occur when a process requests a page of memory, and major page faults occur if that requested page doesn't contain the data the process expects it to. Table 4-1 summarizes the kernel variables related to the behavior of the virtual memory system.

Table 4-1. A summary of memory-related kernel variables

Operating system

Variable

Effect

Solaris

lotsfree

The target size of the free list, in pages. Set by default, one sixty-fourth of physical memory, with a minimum size of 512 KB (64 or 128 pages on an 8 KB or 4 KB page size system, respectively).

The Linux equivalent is freemem/high.

desfree

The minimum pages of free memory, taken over an interval of 30 seconds, before swapping starts.

Set to one-half of lotsfree by default.

Roughly equivalent to Linux freemem/low.

minfree

The threshhold of minimum free memory, taken over a 5-second interval, before active swapping of processes starts. Represented in pages.

Set by default to one-half of desfree.

There is no equivalent parameter under Linux.

slowscan

The slowest scan rate (e.g., when scanning first starts) in pages.

Set by default to one sixty-fourth of the size of physical memory, with a minimum of 512 KB; that is, 64 pages on an 8 KB page size system, or 128 pages on a 4 KB page size system.

There is no Linux equivalent.

fastscan

The fastest scan rate (e.g., with the free list completely empty) in pages.

Set by default to half the physical memory, with a maximum of 64 MB; that is, 8,192 pages on an 8 KB page size system, or 16,384 pages on a 4 KB page size system.

There is no Linux equivalent.

maxpgio

The maximum number of I/Os per second that will be queued by the page scanner. Set by default to 40 on all systems except those based on the sun4d architecture, which has a default setting of 60.

There is no Linux equivalent.

Linux (2.2.x)

freepages.high

The target size of the free list in pages. When memory drops below this level, the system starts to gently page.

Set by default to 786.

The Solaris equivalent is lotsfree.

freepages.low

The threshhold for aggressive memory reclamation in pages.

By default, this is set to 512.

A rough Solaris equivalent is minfree.

freepages.min

The minimum number of pages on the free list. When this value is reached, only the kernel can allocate more memory.

Set to 256 by default.

There is no equivalent parameter under Solaris.

page-cluster

The number of pages that are read in on a page fault is given by 2page-cluster.

The default value is 4.

There is no equivalent parameter under Solaris.

4.2.6 Page Coloring

The precise way that pages are organized in processor caches can have a dramatic effect on application performance. The optimal placement of pages depends on how the application utilizes memory -- some applications access memory almost randomly, whereas others access memory in a strict sequential order -- so there is no single algorithm that provides universally optimal results.

The free list is actually organized into a number of specifically colored bins. The number of bins is given by the physical second-level cache size divided by the page size (for example, a system with a 64 KB L2 cache and an 8 KB page size would have 8 bins). When a page is returned to the free list, it is assigned to a specific bin. When a page is required, one is taken from a specific colored bin. That choice is made based on the virtual address of the requested page. (Recall that pages exist in physical memory, and are physically addressable, but processes only understand virtual addresses; a process incurs a page fault on a virtual address in its own address space, and that page fault is filled by a page with a physical address.) The algorithms we are discussing here decide how pages are assigned colors. There are three available page coloring algorithms in Solaris 7 and later:

  • The default algorithm (numbered 0) uses a hashing algorithm against the virtual address, in order to distribute pages as evenly as possible.

  • The first optional algorithm (numbered 1) assigns colors to pages so that physical addresses map directly to virtual addresses.

  • The second optional algorithm (numbered 2) assigns pages to bins in a round robin mechanism.

Another algorithm called Kessler's Best Bin (numbered 6) was supported on Ultra Enterprise 10000 systems running Solaris 2.5.1 and 2.6 only. It used a history-based mechanism to try and assign pages to the least-used bins.

The default paging algorithm was chosen because it consistently provides good performance. You are not likely to see any performance improvement from tuning these parameters on typical commercial workloads. However, scientific applications may see a significant performance gain -- some testing will reveal the page coloring algorithm that fits best with your application's memory usage patterns. You can change the algorithm by setting the system parameter consistent_coloring to the number of the algorithm you'd like to use.

4.2.7 Transaction Lookaside Buffers (TLB)

While we're discussing the physical to virtual address relationship, it's important to mention the transaction lookaside buffer (TLB). One of the jobs of the hardware memory management unit is to translate a virtual address, as provided by the operating system, into a physical address. It does this by looking up entries in a page translation table. The most recent translations are cached in the translation lookaside buffer. Intel and older SPARC processors (pre-UltraSPARC) use hardware mechanisms to populate the TLB, whereas the UltraSPARC architecture uses software algorithms. While a detailed discussion of the impact of the TLB on performance is a bit beyond the scope of this book, you should be generally aware of what a TLB is and what it does.


4.3 Paging and Swapping

Paging and swapping are terms that are often used interchangeably, but they are quite distinct. A system that is paging is writing selected, infrequently used pages of memory to disk, while a system that is swapping is writing entire processes from memory to disk. Let's say that you are working on your automobile, and you only have a small amount of space available for tools. Paging is equivalent to putting the 8 mm socket back in the toolchest so you have enough room for a pair of pliers; swapping is like putting your entire socket set away.

Many people feel that their systems should never nontrivially page (that is, perform paging on a pre-Solaris 8 system that is not simply filesystem activity). It's important to realize that paging and swapping allow the system to continue getting work done despite adverse memory conditions. Paging is not necessarily indicative of a problem; it is the action of the page scanner to try and increase the size of the free list by moving inactive pages to disk. A process, as a general rule, spends about 80% of its time running about 20% of its code; since the entire process doesn't need to be in memory at once, writing some pages out to disk won't affect performance substantially. Performance only begins to suffer when a memory shortage continues or worsens.

4.3.1 The Decline and Fall of Interactive Performance

Historically, Unix systems implemented a time-based swapping mechanism, whereby a process that was idle for more than 20 seconds would be swapped out; this isn't done anymore. Swapping is now used only to address the most severe memory shortages. If you come across a system where vmstat reports a nonzero swap queue, the only conclusion you can draw is that, at some indeterminate time in the past, the system was short enough on memory to swap out a process.

Memory shortages also tend to appear worse than they are because of the nature of the memory reclamation mechanism. When a system is desperately swapping jobs out, it tries to avoid a major performance decrease by keeping active jobs in memory for as long as possible. Unfortunately, programs that directly interact with users (such as shells, editors, or anything else that is dependent on user-supplied input) are inactive while waiting for the user to type something. As a result, these interactive processes are likely to be targeted as good places for memory reclamation efforts to be targeted. For example, when you pause momentarily in tying a command (such as when you're looking through the process table trying to find the ID of the process that is hogging all your memory so you can kill it), your shell will have to make the long trip from disk back into memory before your characters can be echoed back. Even worse, the disk subsystem is probably under heavy load from all the paging and swapping activity! The upshot is that in memory shortages, interactive performance falls through the floor.

This situation degrades even further if memory consumption continues. The system starts to slow down as disks become overloaded with paging and swapping, and the load average spirals higher; memory-based constraints quickly turn into I/O-based constraints.

4.3.2 Swap Space

Swap space (or, more accurately, paging space, since almost all of this sort of activity is paging rather than swapping) is a topic that tends to confuse people. Swap space serves three functions:

  • As a place to write pages of private memory (a paging store)

  • As a place for anonymous memory storage

  • As a place to store crash dumps[6]

    [6] By default, the very end of the first swap partition is used to store kernel crash dumps under Solaris 7.

This space is allocated from both spare physical memory and from a swap space on disk, be it a dedicated partition or a swapfile.

4.3.2.1 Anonymous memory

Swap space for anonymous memory is used in two stages. Anonymous memory reservations are taken out of disk-based swap, but allocations are taken out of physical memory-based swap. When anonymous memory is requested (via the malloc(3C) system call, for example), a reservation is made against swap, and a mapping is made against /dev/zero. Disk-based swap is used until none remains, in which case physical memory is used instead. Memory space that is mapped but never used stays reserved. This is common behavior in large database systems, and explains why applications such as Oracle require large amounts of disk-based swap, even though they are unlikely to allocate the entire reserved space.

When reserved pages are accessed for the first time, physical pages are taken from the free list, zeroed, and allocated rather than reserved. Should a page of anonymous memory be stolen by the page scanner, the data is written to disk-based swap space (that is, the allocation is moved from memory to disk), and the memory is freed for reuse.

4.3.2.2 Sizing swap space

There are many rules of thumb for how much swap space should be configured on a system, ranging from four times the amount of physical memory to half the amount of physical memory, and none of them are particularly good.

Under Solaris, one tool to get a long term picture of swap usage is /usr/sbin/swap -s. Here's an example from a workstation running Solaris 7 with 128 MB of physical memory and a 384 MB swap partition:

% swap -s

total: 12000k bytes allocated + 3512k reserved = 15512k used, 468904k available

Unfortunately, the names of these fields are misleading, if not downright incorrect. A more accurately labeled version would read:

total: 12000k bytes allocated + 3512k unallocated = 15512k reserved, 468904k available

When the available swap space reaches zero, the system will no longer be able to use more memory (until some is freed). From this report, it is obvious that the system has plenty of swap space configured.

You can also use /usr/sbin/swap -l to find out how much swap space is in use. Here's an example from the same system:

% swap -l

swapfile             dev  swaplo blocks   free

/dev/dsk/c1t1d0s1   32,129     16 788384 775136

In this case, there is a single swap device (a dedicated partition) with a total area of 688,384 512-byte blocks (about 384 MB). At the time this measurement was taken, virtually all of the swap space was free.

One tool you can use to monitor swap space usage over time is /usr/bin/sar -r interval, with interval specified in seconds:

% sar -r 3600

SunOS fermat 5.7 Generic sun4u    06/01/99

00:00:00 freemem freeswap

00:00:00     679   935313

01:00:00     680   937184

02:00:00     680   937184

sar reports the free memory in pages, and the free swap in disk blocks (really the amount of available swap space).

On Linux systems, you can get similar information from the file /proc/meminfo:

% cat /proc/meminfo | head -3 | grep -vi mem

        total:    used:    free:  shared: buffers:  cached:

Swap: 74657792 18825216 55832576

In general, if more than about half of your swap area is in use, you should consider increasing it. Disk is very cheap; skimping on swap space is only going to hurt you.

4.3.2.3 Organizing swap space

In order to minimize the impact of paging as much as possible, swap areas should be configured on the fastest possible disks. The swap partition should be on a low numbered slice (traditionally, slice one, with the root filesystem on slice zero); we'll explain why in Chapter 5. There is absolutely no reason, from a performance point of view, to place swap areas on slow disks, fast disks being accessed through slow controllers, fast disks that are already heavily loaded with I/O activity, or to have more than one swap area on a disk. The best strategy is to have a dedicated swapping area with multiple fast disks and controllers. If it is possible, try to put the swap area on a fast, lightly used disk. If it is not possible, however, try and put the swap file on the most heavily used partition. This minimizes the seek time to the disk; for a detailed discussion, see Section 5.4.10.1.

Solaris allows you to use remote files as swapping areas via NFS. This isn't a good way to achieve reasonable performance on modern systems. If you're configuring a diskless workstation, you have no choice unless you can survive without disk-based swap space. Measuring the level of I/O activity on a per-partition and per-disk basis is a topic that we'll cover in more detail in Section 5.5. The single most important issue in swap area placement is to put swap areas on low numbered partitions on fast, lightly used disks.

4.3.2.4 Swapfiles

Sometimes, you need to create a swap area in an "emergency" setting. The most effective way of doing this on a Solaris system is to use a swapfile, which is a file on disk that the system treats as swap space. In order to create one, you must first create an empty file with /usr/sbin/mkfile, then use /usr/bin/swap -a on the swapfile. Here's an example:

# mkfile 64m /swapfile

# /usr/bin/swap -a /swapfile

# /usr/bin/swap -l

swapfile             dev  swaplo blocks   free

/dev/dsk/c0t0d0s0   32,0      16 262944 232816

/swapfile              -      16  65520  65520

You may alternately specify the size of the swapfile to mkfile in blocks or kilobytes by specifying the units as b or k, respectively. You can remove the file from the swap space via /usr/bin/swap -d swapfile. Note that the swapfile will not be automatically activated at system boot.


4.4 Consumers of Memory

Memory is consumed by four things: the kernel, filesystem caches, processes, and intimately shared memory. When the system starts, it takes a small amount (generally less than 4 MB) of memory for itself. As it dynamically loads modules and requires additional memory, it claims pages from the free list. These pages are locked in physical memory, and cannot be paged out except in the most severe of memory shortages. Sometimes, on a system that is very short of memory, you can hear a pop from the speaker. This is actually the speaker being turned off as the audio device driver is being unloaded from the kernel. However, a module won't be unloaded if a process is actually using the device; otherwise, the disk driver could be paged out, causing difficulties. Occasionally, however, a system will experience a kernel memory allocation error. While there is a limit on the size of kernel memory,[7] the problem is caused by the kernel trying to get memory when the free list is completely exhausted. Since the kernel cannot always wait for memory to become available, this can cause operations to fail rather than be delayed. One of the subsystems that cannot wait for memory is the streams facility; if a large number of users try to log into a system at the same time, some logins may fail. Starting with Solaris 2.5.1, changes were made to expand the free list on large systems, which helps prevent the free list from ever being totally empty.

[7] This number is typically very large. On UltraSPARC-based Solaris systems, it is about 3.75 GB.

Processes have private memory to hold their stack space, heap, and data areas. The only way to see how much memory a process is actively using is to use /usr/proc/bin/pmap -x process-id, which is available in Solaris 2.6 and later releases.

Intimately shared memory is a technique for allowing the sharing of low-level kernel information about pages, rather than by sharing the memory pages themselves. This is a significant optimization in that it removes a great deal of redundant mapping information. It is of primary use in database applications such as Oracle, which benefit from having a very large shared memory cache. There are three special things worth noting about intimately shared memory. First, all the intimately shared memory is locked, and cannot ever be paged out. Second, the memory management structures that are usually created independently for each process are only created once, and shared between all processes. Third, the kernel tries to find large pieces of contiguous physical memory (4 MB) that can be used as large pages, which substantially reduces MMU overhead.

4.4.1 Filesystem Caching

The single largest consumer of memory is usually the filesystem-caching mechanism. In order for a process to read from or write to a file, the file needs to be buffered in memory. When this is happening, these pages are locked in memory. After the operation completes, the pages are unlocked and placed at the bottom of the free list. The kernel remembers the pages that store valid cached data. If the data is needed again, it is readily available in memory, which saves the system an expensive trip to disk. When a file is deleted or truncated, or if the kernel decides to stop caching a particular inode, any pages caching that data are placed at the head of the free list for immediate reuse. Most files, however, only become uncached upon the action of the page scanner. Data that has been modified in the memory caches is periodically written to data by fsflush on Solaris and bdflush on Linux, which we'll discuss a little later.

The amount of space used for this behavior is not tunable in Solaris; if you want to cache a large amount of filesystem data in memory, you simply need to buy a system with a lot of physical memory. Furthermore, since Solaris handles all its filesystem I/O by means of the paging mechanism, a large number of observed page-ins and page-outs is completely normal. In the Linux 2.2 kernel, this caching behavior is tunable: only a specific amount of memory is available for filesystem buffering. The min_percent variable controls the minimum percentage of system memory available for caching. The upper bound is not tunable. This variable can be found in the /proc/sys/vm/buffermem file. The format of that file is min_percent max_percent borrow_percent; note that max_percent and borrow_percent are not used.

4.4.2 Filesystem Cache Writes: fsflush and bdflush

Of course, the caching of files in memory is a huge performance boost; it often allows us to access main memory (a few hundred nanoseconds) when we would otherwise have to go all the way to disk (tens of milliseconds). Since the contents of a file can be operated upon in memory via the filesystem cache, it is important for data-reliability purposes to regularly write changed data to disk. Older Unix operating systems, like SunOS 4, would write the modified contents of memory to disk every 30 seconds. Solaris and Linux both implement a mechanism to spread this workload out, which is implemented by the fsflush and bdflush processes, respectively.

This mechanism can have substantial impacts on a system's performance. It also explains some unusual disk statistics.

4.4.2.1 Solaris: fsflush

The maximum age of any memory-resident modified page is set by the autoup variable, which is thirty seconds by default. It can be increased safely to several hundred seconds if necessary. Every tune_t_fsflushr seconds (by default, every five seconds), fsflush wakes up and checks a fraction of the total memory equal to tune_t_fsflushr divided by autoup (that is, by default, five-thirtieths, or one-sixth, of the system's total physical memory). It then flushes any modified entries it finds from the inode cache to disk; it can be disabled by setting doiflush to zero. The page-flushing mechanism can be totally disabled by setting dopageflush to zero, but this can have serious repercussions on data reliability in the event of a crash. Note that dopageflush and doiflush are complimentary, not mutually exclusive.

4.4.2.2 Linux: bdflush

Linux implements a slightly different mechanism, which is tuned via the values in the /proc/sys/vm/bdflush file. Unfortunately, the tunable behavior of the bdflush daemon has changed significantly from the 2.2 kernels to the 2.4 kernels. I discuss each in turn.

In Linux 2.2, if the percentage of the filesystem buffer cache that is "dirty" (that is, changed and needs to be flushed) exceeds bdflush.nfract, then bdflush wakes up. Setting this variable to a high value means that cache flushing can be delayed for quite a while, but it also means that when it does occur, a lot of disk I/O will happen at once. A lower value spreads out disk activity more evenly. bdflush will write out a number of buffer entries equal to bdflush.ndirty; a high value here causes sporadic, bursting I/O, but a small value can lead to a memory shortage, since bdflush isn't being woken up frequently enough. The system will wait for bdflush.age_buffer or bdflush.age_super, in hundredths of a second, before writing a dirty data block or dirty filesystem metadata block to disk. Here's a simple Perl script for displaying, in a pretty format, the values of the bdflush configuration file:

#!/usr/bin/perl

my ($nfract, $ndirty, $nrefill, $nref_dirt, $unused, $age_buffer, $age_super,

$unused, $unused) = split (/\s+/, `cat /proc/sys/vm/bdflush`, 9);

print "Current settings of bdflush kernel variables:\n";

print "nfract\t\t$nfract\tndirty\t\t$ndirty\tnrefill\t\t$nrefill\n\r";

print "nref_dirt\t$nref_dirt\tage_buffer\t$age_buffer\tage_super\t$age_super\n\r";

In Linux 2.4, about the only thing that didn't change was the fact that bdflush still wakes up if the percentage of the filesystem buffer cache that is dirty exceeds bdflush.nfract. The default value of bdflush.nfract (the first in the file) is 30%; the range is from 0 to 100%. The minimum interval between wakeups and flushes is determined by the bdflush.interval parameter (the fifth in the file), which is expressed in clock ticks.[8] The default value is 5 seconds; the minimum is 0 and the maximum is 600. The bdflush.age_buffer tunable (the sixth in the file) governs the maximum amount of time, in clock ticks, that the kernel will wait before flushing a dirty buffer to disk. The default value is 30 seconds, the minimum is 1 second, and the maximum is 6,000 seconds. The final parameter, bdflush.nfract_sync (the seventh in the file), governs the percentage of the buffer cache that must be dirty before bdflush will activate synchronously; in other words, it is the hard limit after which bdflush will force buffers to disk. The default is 60%. Here's a script to extract values for these bdflush parameters in Linux 2.4:

[8] There are typically 100 clock ticks per second.

#!/usr/bin/perl

my ($nfract, $unused, $unused, $unused, $interval, $age_buffer, $nfract_sync, $u

nused, $unused) = split (/\s+/, `cat /proc/sys/vm/bdflush`, 9);

print "Current settings of bdflush kernel variables:\n";

print "nfract $nfract\tinterval $interval\tage_buffer $age_buffer\tnfract_sync $

nfract_sync\n";

If the system has a very large amount of physical memory, fsflush and bdflush (we'll refer to them generically as flushing daemons) will have a lot of work to do every time they are woken up. However, most files that would have been written out by the flushing daemon have already closed by the time they're marked for flushing. Furthermore, writes over NFS are always performed synchronously, so the flushing daemon isn't required. In cases where the system is performing lots of I/O but not using direct I/O or synchronous writes, the performance of the flushing daemons becomes important. A general rule for Solaris systems is that if fsflush has consumed more than five percent of the system's cumulative nonidle processor time, autoup should be increased.

4.4.3 Interactions Between the Filesystem Cache and Memory

Because Solaris has an untunable filesystem caching mechanism, it can encounter problems under some specific instances. The source of the problem is that the kernel allows the filesystem cache to grow to the point where it begins to steal memory pages from user applications. This behavior not only shortchanges other potential consumers of memory, but it means that the filesystem performance becomes dominated by the rate at which the virtual memory subsystem can free memory.

There are two solutions to this problem: priority paging and the cyclic cache.

4.4.3.1 Priority paging

In order to address this issue, Sun introduced a new paging algorithm in Solaris 7, called priority paging, which places a boundary around the filesystem cache.[9] A new kernel variable, cachefree, is created, which scales with minfree, desfree, and lotsfree. The system attempts to keep cachefree pages of memory available, but frees filesystem cache pages only when the size of the free list is between cachefree and lotsfree.

[9] The algorithm was later backported to Solaris 2.6.

The effect is generally excellent. Desktop systems and on-line transaction processing (OLTP) environments tend to feel much more responsive, and much of the swap device activity is eliminated; computational codes that do a great deal of filesystem writing may see as much as a 300% performance increase. By default, priority paging has been disabled until sufficient end-user feedback on its performance is gathered. It will likely become the new algorithm in future Solaris releases. In order to use this new mechanism, you need Solaris 7, or 2.6 with kernel patch 105181-09. To enable the algorithm, set the priority_paging variable to 1. You can also implement the change on a live 32-bit system by setting the cachefree tunable to twice the value of lotsfree.

4.4.3.2 Cyclic caching

A more technically elegant solution to this problem has been implemented in Solaris 8, primarily due to efforts by Richard McDougall, a senior engineer at Sun Microsystems. No special procedures need be followed to enable it. At the heart of this mechanism is a straightforward rule: nondirty pages that are not mapped anywhere should be on the free list. This rule means that the free list now contains all the filesystem cache pages, which has far-reaching consequences:

  • Application startup (or other heavy memory consumption in a short period of time) can occur much faster, because the page scanner is not required to wake up and free memory.

  • Filesystem I/O has very little impact on other applications on the system.

  • Paging activity is reduced to zero and the page scanner is idle when sufficient memory is enabled.

As a result, analyzing a Solaris 8 system for a memory shortage is simple: if the page scanner reclaims any pages at all, there is a memory shortage. The mere activity of the page scanner means that memory is tight.

4.4.4 Interactions Between the Filesystem Cache and Disk

When data is being pushed out from memory to disk via fsflush, Solaris will try to gather modified pages that are adjacent to each other on disk, so that they can be written out in one continuous piece. This is governed by the maxphys kernel parameter. Set this parameter to a reasonably large value (1,048,576 is a good choice; it is the largest value that makes sense for a modern UFS filesystem). As we'll discuss in Section 6.6 in Chapter 6, a maxphys value of 1,048,576 with a 64 KB interlace size is sufficient to drive a 16-disk RAID 0 array to nearly full speed with a single file.

There is another case where memory and disk interact to give suboptimal performance. If your applications are constantly writing and rewriting files that are cached in memory, the in-memory filesystem cache is very effective. Unfortunately, the filesystem flushing process is regularly attempting to purge data out to disk, which may not be a good thing. For example, if your working set is 40 GB and fits entirely in available memory, with the default autoup value of 30, fsflush is attempting to synchronize up to 40 GB of data to disk every 30 seconds. Most disk subsystems cannot sustain 1.3 GB/second, which will mean that the application is throttled and waiting for disk I/O to complete, despite the fact that all of the working set is in memory!

There are three telltale signs for this case:

  • vmstat -p shows very low filesystem activity.

  • iostat -xtc shows constant disk write activity.

  • The application has a high wait time for file operations.

Increasing autoup (to, say, 840) and tune_t_fsflushr (to 120) will decrease the amount of data sent to disk, improving the chances of issuing a single larger I/O (rather than many smaller I/Os). You will also improve your chances of seeing write cancellation, when not every modification to a file is written to disk. The flip-side is that you run a higher risk of losing data in the case of server failure.


4.5 Tools for Memory Performance Analysis

Tools for memory performance analysis can be classed under three basic issues: how fast is memory, how constrained is memory in a given system, and how much memory does a specific process consume? In this section, I examine some tools for approaching each of these questions.

4.5.1 Memory Benchmarking

In general, monitoring memory performance is a function of monitoring memory restraints. Tools for providing benchmarks as to how fast the memory subsystem is do exist; they are largely of academic interest, as it is unlikely that much tuning will be able to increase these numbers. The one exception to this rule is that users can carefully tune interleaving as appropriate. Most systems handle interleaving by purely physical means, so you may have to purchase additional memory: consult your system hardware manual for more information.[10] Nonetheless, it is often important to be aware of relative memory subsystem performance in order to make meaningful comparisons.

[10] Keep in mind that having not enough slightly faster memory is probably worse than having enough slightly slower memory. Be careful.

4.5.1.1 STREAM

The STREAM tool is simple; it measures the time required to copy regions of memory. This measures "real-world" sustainable bandwidth, not the theoretical "peak bandwidth" that most computer vendors provide. It was developed by John McCalpin while he was a professor at the University of Delaware.

The benchmark itself is easy to run in single-processor mode (the multiprocessor mode is quite a bit more complex; consult the benchmark documentation for current details). Here's an example from an Ultra 2 Model 2200:

$ ./stream

-------------------------------------------------------------

This system uses 8 bytes per DOUBLE PRECISION word.

-------------------------------------------------------------

Array size = 1000000, Offset = 0

Total memory required = 22.9 MB.

Each test is run 10 times, but only

the *best* time for each is used.

-------------------------------------------------------------

Your clock granularity/precision appears to be 1 microseconds.

Each test below will take on the order of 40803 microseconds.

   (= 40803 clock ticks)

Increase the size of the arrays if this shows that

you are not getting at least 20 clock ticks per test.

-------------------------------------------------------------

WARNING -- The above is only a rough guideline.

For best results, please be sure you know the

precision of your system timer.

-------------------------------------------------------------

Function      Rate (MB/s)   RMS time     Min time     Max time

Copy:         226.1804       0.0709       0.0707       0.0716

Scale:        227.6123       0.0704       0.0703       0.0705

Add:          276.5741       0.0869       0.0868       0.0871

Triad:        239.6189       0.1003       0.1002       0.1007

The benchmarks obtained correspond to the summary in Table 4-2.

Table 4-2. STREAM benchmark types

Benchmark

Operation

Bytes per iteration

Copy

a[i] = b[i]

16

Scale

a[i] = q * b[i]

16

Add

a[i] = b[i] + c[i]

24

Triad

a[i] = b[i] + q * c[i]

24

It is also interesting to note that there are at least three ways, all in common use, of counting how much data is transferred in a single operation:

Hardware method

Counts how many bytes are physically transferred, as the hardware may move a different number of bytes than the user specified. This is because of cache behavior; when a store operation misses in a cache, many systems perform a write allocate, which means that the cache line containing that memory location is stored in the processor cache.[11]

[11] This is done in order to ensure coherency between main memory and the caches.

Bcopy method

Counts how many bytes get moved from one location in memory to another. If it takes your machine one second to read a certain number of bytes at one location and one more second to write the same number of bytes to another location, the resulting bandwidth is that number of bytes per second.

STREAM method

Counts how many bytes the user asked to be read and how many bytes the user asked to be written. For the simple "copy" test, this is precisely twice the number obtained by the bcopy method. This is done because some of the tests perform arithmetic; it makes sense to count both the data read into and the data written back from the CPU.

One of the nice things about STREAM is that it uses the same method of counting bytes, all the time, so it's safe to make comparisons.

STREAM is available in source code, so that it can be easily compiled. A list of benchmarks are also available. The STREAM home page is located at http://www.streambench.org.

4.5.1.2 lmbench

Another tool for measuring memory performance is lmbench. While lmbench is capable of many sorts of measurements, we'll focus on four specific ones. The first three measure bandwidth: the speed of memory reads, the speed of memory writes, and the speed of memory copies (via the bcopy method described previously). The last one is a measure of memory read latency. Let's briefly discuss what each of these benchmarks measures:

Memory copy benchmark

Allocates a large chunk of memory, fills it with zeroes, and then measures the time required to copy the first half of the chunk of memory into the second half. The results are reported in megabytes moved per second.

Memory read benchmark

Allocates a chunk of memory, fills it with zeroes, and then measures the time required to read that memory as a series of integer loads and adds; each 4-byte integer is loaded and added to an accumulator variable.

Memory write benchmark

Allocates a chunk of memory, fills it with zeroes, and then measures the time needed to write that memory as a series of 4-byte integer stores and increments.

Memory latency benchmark

Measures the time required to read a byte from memory. The results are reported in nanoseconds per load. The entire data memory hierarchy[12] is evaluated, including the latency to the primary caches, secondary caches, main memory, and the latency effects of a TLB miss. In order to derive the most information from the lmbench memory latency benchmarks, try plotting the data as a function of the latency versus the array size used for the test.

[12] Most notably, any separate instruction caches are not measured.

When you compile and run the benchmark, it will ask you a series of questions regarding what tests you would like to run. The suite is fairly well documented. The lmbench home page, which contains source code and more details about how the benchmark works, is located at http://www.bitmover.com/lm/lmbench/.

4.5.2 Examining Memory Usage System-Wide

It is important to understand a system's performance. The only way to get that understanding is through regular monitoring of data.

4.5.2.1 vmstat

vmstat is one of the most ubiquitous performance tools. There is one cardinal rule about vmstat, which you must learn and never forget: vmstat tries to present an average since boot on the first line. The first line of vmstat output is utter garbage and must be discarded.

With that in mind, Example 4-1 shows output under Solaris.

Example 4-1. vmstate output in Solaris
# vmstat 5

 procs     memory             page            disk         faults       cpu

 r b w  swap  free    re  mf pi po fr de sr s0 s1 s2 --  in   sy   cs  us sy id

 0 0 0  43248 49592   0   1  5  0  0  0  0  0  1  0  0  116  106   30  1  1  99

 0 0 0 275144 56936   0   1  0  0  0  0  0  2  0  0  0  120    5   19  0  1  99

 0 0 0 275144 56936   0   0  0  0  0  0  0  0  0  0  0  104    8   19  0  0 100

 0 0 0 275144 56936   0   0  0  0  0  0  0  0  0  0  0  103    9   20  0  0 100

The r, b, and w columns represent the number of processes that have been in the run queue, blocked for I/O resources (including paging), and processes that are runnable but swapped, respectively. If you ever see a system with a nonzero number in the w field, all that you can infer is that the system was, at some point in the past, low enough on memory to force swapping to occur. Here's the most important data you can glean from vmstat output:

swap

The amount of available swap space, in KB.

free

The amount of free memory (that is, the size of the free list) in KB. In Solaris 8, this number includes the amount of memory used for the filesystem cache; in prior releases, it does not, and therefore will be very small.

re

The number of pages reclaimed from the free list. A page had been stolen from a process, but then reclaimed by the process before it was recycled and given to another process.

mf

The number of minor page faults. These are fast as long as a page is available from the free list, because they can be resolved without performing a page-in.

pi, po

Page-ins and page-outs, respectively, in KB per second.

de

The short term memory deficit. If the value is nonzero, memory has been vanishing fast lately, and extra free memory will be reclaimed with the expectation that it'll be needed soon.

sr

The scan rate activity of the pagedaemon, in pages per second. This is the most critical memory-shortage indicator. In Solaris systems prior to 8, if this stays about 200 pages per second for a reasonably long period of time, you need more memory. If you are running Solaris 8 and this number is nonzero, you are short of memory.

On Linux, the output is a little bit different:

% vmstat 5

 procs                  memory    swap        io    system         cpu

 r b w  swpd  free  buff cache  si  so   bi   bo   in   cs  us  sy  id

 1 0 0 18372  8088 21828 56704   0   0    1    7   13    6   2   1   6

 0 0 0 18368  7900 21828 56708   1   0    0    8  119   42   6   3  91

 1 0 0 18368  7880 21828 56708   0   0    0   14  122   44   6   3  91

 0 0 0 18368  7880 21828 56708   0   0    0    5  113   24   2   2  96

 0 0 0 18368  7876 21828 56708   0   0    0    4  110   27   2   2  97

Note that while the w field is calculated, Linux never desperation-swaps. swpd, free, buff, and cache represented the amount of virtual memory used, the amount of idle memory, the amount of memory used as buffers, and the amount used as cache memory, respectively. There is generally very little to be gathered from a Linux vmstat output. The most important thing to watch is probably the si and so columns, which indicate the amount of swap-ins and swap-outs. If these grow large, you probably need to increase the kswapd swap_cluster kernel variable to buy yourself some additional bandwidth to and from swap, or purchase more physical memory (see Section 4.2.5.1 earlier in this chapter).

4.5.2.2 sar

sar (for system activity reporter) is, like vmstat, an almost ubiquitous performance-monitoring tool. It is particularly useful in that it can be adapted to gather data on its own for later perusal, as well as doing more focused, short-term data collection. In general, its output is comparable with vmstat's, although differently labeled.

In general, the syntax for invoking sar is sar -flags interval number. This causes a specific number of data points to be gathered every interval seconds. When looking at memory statistics, the most important flags are -g, -p, and -r. Here's an example of the output generated:

$ sar -gpr 5 100

SunOS islington.london-below.net 5.8 Generic_108528-03 sun4u    02/19/01

11:28:10  pgout/s ppgout/s pgfree/s pgscan/s %ufs_ipf

          atch/s  pgin/s ppgin/s  pflt/s  vflt/s slock/s

         freemem freeswap

...

11:28:50     0.00     0.00     0.00     0.00     0.00

          251.60    5.00    5.20 1148.20 2634.60    0.00

           86319  3304835

The most important output fields are summarized in Table 4-3.

Table 4-3. sar memory statistics fields

Flag

Field

Meaning

-g

pgout/s

Page-out requests per second

ppgout/s

Pages paged out per second

pgfree/s

Pages placed on the free list per second by the page scanner

pgscan/s

Pages scanned per second by the page scanner

%ufs_ipf

The percentage of cached filesystem pages taken off the free list while they still contained valid data; these pages are flushed and cannot be reclaimed (see Section 5.4.2.7)

-p

atch/s

Page faults per second that are satisfied by reclaiming a page from the free list (this is sometimes called an attach)

pgin/s

The number of page-in requests per second

ppgin/s

The number of pages paged in per second

pflt/s

The number of page faults caused by protection errors (illegal access to page, copy-on-write faults) per second

-r

freemem

The average amount of free memory

freeswap

The number of disk blocks available in paging space

4.5.2.3 memstat

Starting in the Solaris 7 kernel, Sun implemented new memory statistics to help evaluate system behavior. To access these statistics, you need a tool called memstat. While currently unsupported, it will hopefully be rolled into a Solaris release soon. This tool provides some wonderful functionality, and is available at the time of this writing from http://www.sun.com/sun-on-net/performance.html. Example 4-2 shows the sort of information it gives you.

Example 4-2. memstat
# memstat 5

memory ---------- paging----------executable- -anonymous---filesys  --  ---  cpu --

  free  re  mf  pi  po  fr de sr epi epo epf  api apo apf  fpi fpo fpf us sy wt  id

 49584   0   1   5   0   0  0  0   0   0   0    0   0   0    5   0   0  1  1  1  98

 56944   0   1   0   0   0  0  0   0   0   0    0   0   0    0   0   0  0  0  0 100

Much like vmstat, the first line of output is worthless and should be discarded. memstat breaks page-in, page-out, and page-free activity into three different categories: executable, anonymous, and file operations. Systems with plenty of memory should see very low activity in the epf and apo fields. Consistent activity in these fields indicates a memory shortage.

If you are running Solaris 7 or earlier and do not have priority paging enabled, however, executables and anonymous memory will be paged with even the smallest amount of filesystem I/O, once memory falls to or beneath the lotsfree level.

4.5.3 Examining Memory Usage of Processes

It's nice to know how much memory a specific process is consuming. Each process's address space, which is made up of many segments, can be measured in the following ways:

  • The total size of the process address space (represented by SZ or SIZE)

  • The resident size (the size of the part of the address space that's held in memory) of the process address space (RSS)

  • The total shared address space

  • The total private address space

There are many good tools for examining memory usage and estimating the amount of memory required for a certain process.

4.5.3.1 Solaris tools

One common tool is /usr/ucb/ps uax. Here's an example of the sort of data it gives:

% /usr/ucb/ps uax

USER       PID %CPU %MEM   SZ  RSS TT       S    START  TIME COMMAND

root     16755  0.1  1.0 1448 1208 pts/0    O 17:33:35  0:00 /usr/ucb/ps uax

root         3  0.1  0.0    0    0 ?        S   May 24  6:19 fsflush

root         1  0.1  0.6 2232  680 ?        S   May 24  3:10 /etc/init -

root       167  0.1  1.3 3288 1536 ?        S   May 24  1:04 /usr/sbin/syslogd

root         0  0.0  0.0    0    0 ?        T   May 24  0:16 sched

root         2  0.0  0.0    0    0 ?        S   May 24  0:00 pageout

gdm      14485  0.0  0.9 1424 1088 pts/0    S 16:17:57  0:00 -csh

Note that kernel daemons like fsflush, sched, and pageout report SZ and RSS values of zero; they run entirely in kernel space, and don't consume memory that could be used for running other applications. However, ps doesn't tell you anything about the amount of private and shared memory required by the process, which is what you really want to know.

The newer and much more informative way to determine a process's memory use under Solaris is to use /usr/proc/bin/pmap -x process-id. (In Solaris 8, this command moved to /usr/bin.) Continuing from the previous example, here's a csh process:

% pmap -x 14485

14485:  -csh

Address   Kbytes Resident Shared Private Permissions       Mapped File

00010000     144     144       8     136 read/exec         csh

00042000      16      16       -      16 read/write/exec   csh

00046000     136     112       -     112 read/write/exec    [ heap ]

FF200000     648     608     536      72 read/exec         libc.so.1

FF2B0000      40      40       -      40 read/write/exec   libc.so.1

FF300000      16      16      16       - read/exec         libc_psr.so.1

FF320000       8       8       -       8 read/exec         libmapmalloc.so.1

FF330000       8       8       -       8 read/write/exec   libmapmalloc.so.1

FF340000     168     136       -     136 read/exec         libcurses.so.1

FF378000      40      40       -      40 read/write/exec   libcurses.so.1

FF382000       8       -       -       - read/write/exec    [ anon ]

FF390000       8       8       8       - read/exec         libdl.so.1

FF3A0000       8       8       -       8 read/write/exec    [ anon ]

FF3B0000     120     120     120       - read/exec         ld.so.1

FF3DC000       8       8       -       8 read/write/exec   ld.so.1

FFBE4000      48      48       -      48 read/write         [ stack ]

--------  ------  ------  ------  ------

total Kb    1424    1320     688     632

This gives a breakdown of each segment, as well as the process totals.

4.5.4 Linux Tools

The ps command under Linux works essentially the same way it does under Solaris. Here's an example (I've added the header line for ease of identifying the individual columns):

% ps uax | grep gdm

USER       PID %CPU %MEM  SIZE   RSS TTY STAT START   TIME COMMAND

gdm      12329  0.0  0.7  1548   984  p3 S   18:18   0:00 -csh

gdm      13406  0.0  0.3   856   512  p3 R   18:37   0:00 ps uax

gdm      13407  0.0  0.2   844   344  p3 S   18:37   0:00 grep gdm

While ps is ubiquitous, it's not very informative. The Linux equivalent of Solaris's pmap are the entries in the /proc filesystem, which tell you very useful things.

One of the nicest features in Linux is the ability to work with the /proc filesystem directly. This access gives you good data on the memory use of a specific process, akin to pmap under Solaris. Every process has a directory under its process ID in the /proc filesystem; inside that directory is a file called status. Here's an example of what it contains:

% cat /proc/12329/status

Name:   csh

State:  S (sleeping)

Pid:    12329

PPid:   12327

Uid:    563     563     563     563

Gid:    538     538     538     538

Groups: 538 100

VmSize:     1548 kB

VmLck:         0 kB

VmRSS:       984 kB

VmData:      296 kB

VmStk:        40 kB

VmExe:       244 kB

VmLib:       744 kB

SigPnd: 0000000000000000

SigBlk: 0000000000000002

SigIgn: 0000000000384004

SigCgt: 0000000009812003

CapInh: 00000000fffffeff

CapPrm: 0000000000000000

CapEff: 0000000000000000

As you can see, this is a lot of data. Here's a quick Perl script to siphon out the most useful parts:

#!/usr/bin/perl

$pid = $ARGV[0];

@statusArray = split (/\s+/, `grep Vm /proc/$pid/status`);

print "Status of process $pid\n\r";

print "Process total size:\t$statusArray[1]\tKB\n\r";

print "            Locked:\t$statusArray[4]\tKB\n\r";

print "      Resident set:\t$statusArray[7]\tKB\n\r";

print "              Data:\t$statusArray[10]\tKB\n\r";

print "             Stack:\t$statusArray[13]\tKB\n\r";

print " Executable (text):\t$statusArray[16]\tKB\n\r";

print "  Shared libraries:\t$statusArray[19]\tKB\n\r";


4.6 Concluding Thoughts

There's a simple moral to this story: your memory requirements depend heavily on how many users there are and what they are doing. Large technical applications such as computational fluid dynamics or protein structure calculations can require incredible amounts of memory. It is easy to imagine a computer with 2 GB of memory that could support only one such job, whereas that same system could support several hundred interactive users quite comfortably.

One of the greatest improvements in computing in the last two decades has been the ability to assemble large amounts of memory for relatively little cost. Take advantage of it.


Chapter 5. Disks

I think Silicon Valley was misnamed. If you look back at the dollars shipped in products in the last decade, there has been more revenue from magnetic disks than from silicon. They ought to rename the place Iron Oxide Valley.

—Al Hoagland, 1982

Traditionally, I/O performance tuning has fallen behind less glamorous topics. This bias can be seen to the present day in the personal computer market: every computer manufacturer will advertise the clock rate of their CPUs, but it's quite a bit rarer that they announce the rotational speed or internal transfer rate of their hard drives. Often our initial thoughts of performance are ones of processor power or memory consumption, not of transfer rates to and from disk platters.

It's unfortunate, because the stakes in I/O tuning are exceedingly high; since disks are about six orders of magnitude slower than physical memory, avoiding trips to disk and making those necessary trips as fast as possible can have a huge impact on application performance.

Exacerbating the importance of I/O tuning is the fact that the most dramatic improvements in storage technology have been oriented at increasing the amount of data we can store, rather than how fast we can access it; rotational speed has gone up by about three times in the last ten years, but storage capacities have gone up by a factor of at least one hundred.

In this chapter, I describe this often neglected area of performance analysis and tuning. I focus initially on basic concepts surrounding disk drives: physical layout, access patterns, and the factors that govern disk performance in various situations. I also talk about the interfaces by which we communicate with the disks, such as IDE and SCSI, some common disk performance problems, the impact of various types of filesystems on performance, and some tools for disk performance monitoring and analysis.


5.1 Disk Architecture

A disk drive consists of a number of stacked metal disks coated with a magnetic recording medium; these disks are called platters. The platters rotate around a central spindle, generally at speeds between 5,400 and 15,000 rotations per minute. Between these platters are a set of movable arms, which have small sensors called read/write heads. In order for a disk operation to complete to a given part of the disk, the arms must move into place, and then the appropriate location on the platter must spin beneath the heads. With proper positioning thusly achieved, data can now be read from, or written to the disk.[1]

[1] The disk drive also stores some vital information, like the disk label, at the very beginning of the disk.

The usable storage area of the disk is divided up based on an address scheme.[2] The ring of data that is swept out by the heads on a given platter as the platter rotates is called a track; the group of tracks on all platters that share a given arm position is called a cylinder. Cylinders are numbered from the outside edge of the disk inward; that is, the lowest numbered cylinders are also the outermost. Each track is further broken up into sectors, which are generally 512 bytes[3] long. The number of sectors per track is dependent on the size of the platter and the recording density. Figure 5-1 is an illustration of disk architecture.

[2] Some space is reserved for the disk drive's internal use; it stores things like head alignment data and error correction data.

[3] This size is mandated by the SCSI standard.

Figure 5-1. Disk architecture

5.1.1 Zoned Bit Rate (ZBR) recording

If the number of sectors per track were constant, data would become less dense across tracks as you moved outward along the disk, wasting space as you move outward. To address this problem, all modern disk drives use a technique called zone bit rate (ZBR) recording, in which the disk is divided up into zones, and more sectors are stored on the outer tracks.[4] Lower-numbered zones have more sectors per track than higher-numbered zones. This results in a roughly constant data density, in terms of bits per inch; it also causes the data rate to vary with the cylinder number.

[4] Most modern disks have about fifteen zones.

The lowest numbered cylinder is the fastest, largest, and most dense (most bits/cylinder); the highest numbered cylinder is the slowest, smallest, and least dense (fewest bits/cylinder).

Unfortunately, this makes disk performance calculations somewhat complicated, since the transfer rate varies depending on which cylinder you're accessing.

5.1.1.1 Disk caches

SCSI and IDE disks typically have local caches associated with them. These caches are simply a small amount of memory, typically between 128 KB and 4 MB, associated with the disk's embedded controller. When data is read from the drive, a copy is stored in this cache. Because most of the delay in accessing a disk is either in the time spent transferring data from the platters or in the time spent moving the read/write heads into position and waiting for the appropriate sector to fly by, servicing a data request from the disk's local cache saves a lot of time, and for commonly accessed data, the local caches can be very effective. By default, most drives have read caching enabled.

What is usually not enabled is the ability to use this local cache memory to cache writes. Typically, a drive will not acknowledge a write as complete until the data is physically written to disk. If write caching is enabled, the disk will acknowledge the write as complete when the data is stored in this memory. This memory is volatile, which means that its contents vanish when power is lost: this is not a problem for caching reads, but a power failure at an inopportune moment could cause an acknowledged write to be silently discarded because the data hasn't been physically written to disk. There is absolutely no guarantee that anything written to the disk buffer cache will actually ever see the platters. Because of this reliability issue, these caches are not normally enabled. Enabling them can substantially increase performance; however, this performance improvement comes with the risk of data loss.

We'll discuss how to enable and check the status of these caches in Section 5.5.1 later in this chapter.

5.1.2 Access Patterns

Disk accesses tend to be either clustered together (sequential access) or spread out in no particular order (random access). Just because disk accesses are clustered together does not necessarily mean that they are sequential; most disk accesses that happen to be close together on the disk end up being sequential. Clustered accesses aren't necessarily, but often are, accesses to sequential sectors on the disk. Unfortunately, these terms also have relevance in other areas: it is possible for media to be "randomly accessible" in the sense that "any bit can be accessed about as fast as any other bit" (such as a hard drive), or "sequentially accessible" (such as a tape drive). When I use these terms in this chapter, I only refer to the pattern of accesses.

The performance of the two access patterns is given by two very different rules:

  • Sequential access is dominated by the actual transfer time from the disk platters (internal transfer speed).

  • Random access is dominated by the platter rotational speed and the seek time.

Increasing the transfer size or transforming random access patterns into sequential access patterns significantly increases disk subsystem performance. This is one of the most profitable areas to target for improvements in performance.

5.1.3 Reads

A disk read is normally synchronous; the requesting process blocks until the read is complete. This is not always the case, as an asynchronous read could have been issued by aioread(3). When the system reads the data, it is cached in case it is needed again. Caching uses the main memory area and is described in detail in Section 4.4.1. If you are debating expanding main memory or investing in a big disk controller in the interest of improving read performance, you are almost always better off expanding main memory; you will have an expanded filesystem cache that is accessible in less than a microsecond, and you will almost certainly reduce the number of reads issued, saving on bus loading and CPU time.

Generally, the goals in designing a read-oriented disk subsystem are to minimize the latency for the initial read of uncached data and to maximize throughput for sequential data.

5.1.4 Writes

Disk writes are a mix of synchronous and asynchronous events. Asynchronous events occur when a file is written to the local filesystem: the data is written to memory and the write is immediately acknowledged. This data is actually flushed out to disk later by means of the filesystem flush daemon (see Section 4.4.2). Applications can request that a file be written out immediately by calling fsync(3C) or by closing the file. Note that when a process exits, all of its open filehandles are closed and therefore flushed. This can take a while and cause a substantial amount of disk activity if many processes exit at the same time.

5.1.4.1 UFS write throttling

In order to prevent too much memory from being consumed by pending writes, Solaris implements a write throttle algorithm. This algorithm limits the amount of data pending to any given file; it is a per-process, per-file limitation. For each file, between 256 KB (the low-water mark, set by the ufs:ufs_LW variable) and 384 KB (the high-water mark, set by the ufs:ufs_HW variable) of data can be pending at any one time. When there is less data than the low-water mark pending to a file, flushing the data is left to the filesystem flush daemon. When the amount of pending data is between the two water marks, writes are scheduled to push the data out to disk. If more than the high-water mark data is pending, the application is not allowed to write any more data (that is, the write( ) system call will block) until the amount of pending data drops below the low-water mark again.

At high-data rates, asynchronous writes to disk become synchronous, and processes slow down. The writes are asynchronous because they are being cached in memory and immediately acknowledged, and the disk access happens later. When the process's filesystem write buffer is full for a specific file, writes become synchronous, because the limit on performance is now how fast data can be written to the disk, rather than how fast data can be cached in memory.

When you want to write quickly to a file and the underlying disk subsystem can cope with the throughput, it may still be impossible to drive the disk subsystem to maximum efficiency because the write throttle kicks in too early. The write throttle can also affect applications that perform large bursts of write activity at spaced intervals. The best diagnostic for this condition is the ufs_throttles kernel counter. It is incremented every time the write throttle kicks in, so a steadily increasing value means that you should bump up the write throttle.

Increasing the write throttle has the side effect of queuing a larger number of I/O requests for a file; this method can improve throughput, but it increases the average response time, since the queue is longer. A simple way to see the effect of changing the write throttle is to use ptime(1) to look at the time required to create a file with mkfile. Please note that you probably do not want to use /tmp for this test. The reason is described in Section 5.4.8 later in this chapter.

In practice, the watermarks are set far too low by default. The high-water mark should be set to many times the value of the maxphys tunable (for a description of this variable, see Section 4.4.4), and the low-water mark should be set to one-half to two-thirds of the high-water mark. In practice, I often set the low-water mark equal to 1/32nd of physical memory, and the high-water mark is set to 1/16th of physical memory. This means that if you have, say, one process writing full-out to sixteen separate files, you are in theory able to use all of the system's memory as a write cache. However, in practice, you would never exhaust memory this way, because the pagedaemon would kick in and start reaping cached pages and pushing them out to disk. For more information on the pagedaemon, see Section 4.3.

The entire write throttle mechanism is controlled by the ufs:ufs_WRITES kernel tunable; by default it is set to one, which corresponds to enabling the throttle. If you set it to zero, it is turned off. The potential side effect of this is very high memory consumption.

5.1.5 Performance Specifications

One of the problems in analyzing disk performance is the confusing way most vendors provide performance statistics for their devices.

5.1.5.1 One million bytes is a megabyte?

Disk vendors, as well as the SCSI specification, use the term megabyte to mean one million (1,000,000) bytes. The convention everywhere else, however, is to use megabyte to mean 1,024 kilobytes, or 220 (1,048,576) bytes. This five-percent difference applies to both performance specifications and storage specifications:

  • A disk drive rated to transfer 8.4 million bytes per second can only transfer 8.0 MB per second.

  • A disk drive with an advertised capacity of 9,100 million bytes will only store 8.67 GB.

This convention is adhered to by the entire industry, despite the confusion it causes.

5.1.5.2 Burst speed versus internal transfer speed

Disk vendors also like to quote the burst speed of the embedded controller. This specifies the rate that the embedded controller can transmit data on the bus. It does not account for command overhead or state changes. It is simply the speed that bits are transmitted, which can only be sustained if the disk can provide data at that speed and the target can deal with a flow that fast. Practically, this speed only happens when data is being transferred from the disk's local cache to the host system's memory. In most cases, the data does not reside in the disk's cache, and so the operation must be satisfied from the disk platters. For random I/O, the disk probably also needs to move the disk arm into position.

The internal transfer speed reflects the speed that bits are read from the disk platters to the read/write head, exclusive of all other factors (such as seeking). This measures only how fast data can be pulled off the platters; it doesn't measure the performance of any other part of the disk drive. The internal transfer speed is much lower than the burst speed, and can vary substantially depending on which cylinder is being read (see Section 5.1.1 earlier in this chapter). As a result, the internal transfer speed is a reasonably good predictor of sequential transfer performance.

The most important consequence of the difference between burst speed and internal transfer speed is that a single disk with a 40 MB per second burst speed is unlikely to be any faster than an otherwise identical disk with a 20 MB per second burst speed.

5.1.5.3 Internal transfer speed versus actual speed

Let's further complicate matters.

The internal transfer speed reflects the true performance abilities of a disk drive more than burst speed does. However, the internal transfer speed measures only the speed at which all data flies past the read/write heads, and this measurement includes a substantial amount of overhead. Much of this overhead is spent on error-correcting codes that prevent most disk errors from resulting in corrupted data. By applying the following formula, we can get a more accurate measure of a single disk's performance:

Here's an example, using the Seagate 18.2 GB Barracuda disk (ST118273W). By consulting the manufacturer's specification sheet to find the average number of sectors per track (237) and the rotational speed of the drive (7,200 rpm), we can perform the following computation:

This represents the speed that data can be transferred to/from the disk, assuming purely sequential access.

5.1.5.4 Average seek time

The average seek time is another number that is often quoted, but it is remarkably misleading. According to the disk vendors, the average seek time is the sum of all possible seeks over the number of possible seeks. While this is mathematically true, it turns out that the physics involved in moving the disk arm mean that short seeks take much less time than long seeks. In fact, the "average" seek does not at all resemble a "typical" seek -- it involves moving the disk arm much farther than in a typical seek.

The lesson here is that storing data in a way that minimizes seek time optimizes performance. Splitting a single disk into two slices, each of which is busy, turns out to be a very bad idea. In such a configuration, every time the disk fields an access for the slice it's not currently on, it will need to seek from the average position in one slice to the average position in the other slice, which is half the maximum seek time. This takes a long time. It would be a better idea to put the two slices on different disks, in order to minimize the seek time.

It is easy to disregard the effects of excessive seek distance. However, with maximum seek times in the neighborhood of 20 ms, every maximum-length seek incurs a delay equal to hundreds of millions of instructions.

5.1.5.5 Storage capacity and access capacity

For random-access activity, disk performance is dominated by the platter rotational speed and the seek time. Consequently, drives that spin at the same rate perform about the same, largely irrespective of capacity or physical size.[5] For random-access activity, the most effective way to increase access capacity is to buy as many fast-spinning drives as possible!

[5] 5.25" drives have slightly higher seek times, but this is not likely to have any practical impact.

Unfortunately, increases in access capacity (generally quantified as random access operations per second) have not kept pace with increases in storage capacity. To illustrate this principle, let's consider a 5,400 rpm 2.1 GB disk and a 7,200 rpm 9.1 GB disk. The larger disk has more than four times the storage capacity, but only about 30% greater access capacity. Therefore, the greatest random access I/O capacity is usually achieved by using several of the smallest feasible disks, rather than one larger disk with superior specifications! This is illustrated in Table 5-1.

Table 5-1. Access capacity comparisons for various 218 GB installations

Disks[6]

Number

Buses[7]

I/O ops per second per disk[8]

Aggregate I/O ops per second

Aggregatetransfer rate

4.3 GB

51

4

108

5,508

160 MB/sec

9.1 GB

24

2

85

2,040

80 MB/sec

18.2 GB

12

1

70

840

40 MB/sec

[6] All disk types rotate at the same speed.

[7] The minimum number of Ultra Wide (fast-40) SCSI buses required to support the drives.

[8] These numbers are empirical.

However, using smaller disks is usually more expensive, more complicated, and takes more space. It also necessitates the use of disk arrays, which we'll discuss later, to solve the problems brought about by increased subsystem failure rate that is observed when more components are used. In most environments, it is preferable to design mixtures of large and small capacity disks to ride the "sweet spot" of the disk cost/performance curve as much as possible.

Configuring disk subsystems based only on storage considerations is generally a serious mistake from a performance point of view.


5.2 Interfaces

An interface serves as the rules governing the communication between the host computer system and a set of devices. These rules are usually established through the auspices of a host adapter, which resides on one of the system's peripheral interconnects (see Section 3.5).

The modern disk market is largely driven by three disk protocols: IDE, SCSI, and Fibre Channel. We'll focus primarily on these protocols, as well as one older protocol, IPI, and some of the emerging storage interfaces (IEEE 1394/Firewire and USB). It is worth pointing out that there are no "native" IEEE 1394 or USB devices; they are all IDE drives with adapters attached, and as such have many of the same problems as a strictly IDE device does.

5.2.1 IDE

IDE, for Integrated Drive Electronics, or alternatively, ATA, for AT Attachment, have historically been used in low-cost systems, particularly Intel-based personal computers. However, IDE disks are starting to become common in low-end workstations, such as the Sun Ultra 5 and Ultra 10, in order to keep prices as low as possible. The IDE specification was intended to reduce interface costs by placing the controller hardware on the drive itself, as well as making firmware implementations as easy as possible.

IDE development is managed to some degree by a technical committee known as T13, which is part of the National Committee on Information Technology Standards (NCITS, pronounced "insights"). NCITS is chartered by and functions under rules that are approved by the American National Standards Institute (ANSI). These rules try to ensure that voluntary standards are developed by industry group consensus. NCITS develops Information Processing System standards, while ANSI approves the process under which they are developed and publishes them. The T13 Technical Committee web page is located at http://www.t13.org.

Following the success of the original ATA interface, the Small Form Factor Committee (an organization of drive manufacturers) created a backwards-compatible extension of the ATA interface called ATA-2 (alternatively Fast-ATA). This standard added faster PIO and DMA modes. Contrary to popular suspicion, the logical block addressing (LBA) scheme that was revamped in ATA-2 had nothing to do with surpassing the 504 MB barrier; even the original ATA specification had a capacity limit in excess of 100 GB.

Several years later, a further revision of the specification called ATA-3 had been released, which added more sophisticated power management, increased reliability, and added failure prediction. It did not define any faster modes. One of the major disadvantages of ATA is that it was designed for disks only; with the explosive popularity of CD-ROM drives, the interface was extended by means of the ATA Packet Interface (ATAPI), which facilitates adding such devices. ATAPI was formally integrated into ATA with the ATA-4 standard.

Enhanced IDE (EIDE) is a marketing term referring to ATA-2 and ATAPI implementations, whereas Fast-ATA is a competing marketing term that builds on ATA-2 only.

One of the consequences of the low-cost goals of the IDE device is that the host processor ends up taking a substantial role in processing I/O requests. IDE drives have essentially two ways to transfer data to the host processor: programmed I/O (PIO), and direct memory access, or DMA, which can be performed on a single-word or multi-word level. A PIO transfer mode causes the host processor to handle every transaction from disk into memory itself, whereas in a DMA transfer mode the IDE controller can write directly into the host's memory. The drive communicates its ready status to the controller by asserting the IRQ line of the drive, which can be done in two cases:

  • A read command has been issued to the drive; that command has been serviced and the requesting data is in the drive's buffer, ready to be taken up and transferred into memory.

  • A write command has been issued to the drive; that command has been transferred to the drive's buffer (if the drive does not have write caching enabled, the line will not be asserted until the data has been physically committed to disk).

During normal reads and writes, handling this interrupt can continuously bother the CPU and cause long delays in servicing the drive. The Read/Write Multiple commands are drive-level commands that attempt to address this problem by allowing up to 128 sectors to be transferred without incurring any intervening interrupts, substantially lessening the overhead on the host processor.

Single-word DMA performs very poorly relative to multi-word DMA; in fact, "DMA" usually carries with it the implicit assumption of multi-word transfers. Single-word DMA support was dropped in the ATA-3 specification.

As hard disks continued to get faster, the fastest multi-word DMA mode (at 16.7 MB/s) quickly became insufficient. Unfortunately, the interface was really designed for slow data transfer (~5 MB/s). Simply increasing the clock rate of the interface caused all sorts of signaling interference problems; as a result, a new type of DMA transfer mode (called UltraDMA) was introduced in a standard called Ultra-ATA, which was developed to bridge the gap between ATA-3 and the then-forthcoming ATA-4. Now that ATA-4 has been released, the Ultra-ATA standard is integrated into ATA-4.

The key technological advance introduced in UltraDMA was double-transition clocking: data is transferred on both the rising and falling edges of the clock signal. This is exactly the same technology used to improve transfer rates in DDR SDRAM (see Section 4.1). In order to improve data integrity, UltraDMA modules also use a cyclical redundancy checking (CRC) algorithm to compute an error-detection value for each block of data transferred, which is sent along with the block. This allows the recipient of the data to determine whether the data was corrupted in transit, in which case the data is retransmitted.

Despite the throughput increase given by double-transition clocking, transferring data faster than 33 MB/s finally exhausted the capabilities of the standard 40-conductor IDE cable. To use UltraDMA modes over 2, a special 80-conductor cable is required.[9] This cable uses the same 40 pins, but adds 40 ground lines between each of the original 40 signal lines in order to separate those lines and prevent interference.

[9] The cable was specified in the ATA-4 specification for UltraDMA modes 0, 1, and 2, but was optional.

These standards are reviewed in Table 5-2.

Table 5-2. IDE interfaces

Type

Mode

Cycle time (MHz)[10]

Data rate (MB/sec)

Supported in

PIO

0

1.67

3.3

ATA

1

2.61

5.2

ATA

2

4.17

8.3

ATA

3

5.56

11.1

ATA-2

4

8.33

16.6

ATA-2

5

11.11

22.2

Vaporware!

DMA (single-word)

0

1.04

2.1

ATA

(Removed in ATA-3)

1

2.08

4.2

ATA

(Removed in ATA-3)

2

4.17

8.3

ATA

(Removed in ATA-3)

DMA (multi-word)

0

2.08

4.2

ATA

1

6.67

13.3

ATA-2

2

8.33

16.6

ATA-2

UltraDMA

0

4.17

16.6

ATA-4

1

6.25

25.0

ATA-4

2

(UDMA/33)

8.33

33.3

ATA-4

3

11.11

44.4

ATA-5

4

(UDMA/66)

16.67

66.7

ATA-5

5

(UDMA/100)

25

100.0

ATA-6?

[10] ATA cycle times are usually represented in nanoseconds, but for the ease of comparison, they are shown here in MHz.

Table 5-3 provides a summary of the IDE implementations and the features it supports.

Table 5-3. Characteristics of IDE implementation

EIDE

Fast-ATA

Fast-ATA-2

Ultra-ATA

PIO Transfer Modes

Up to 3

Up to 3

Up to 4

All

DMA (Single-word) Modes

All

All

All

All

DMA (Multi-word) Modes

Up to 1

Up to 1

Up to 2

All

Read/Write Multiple

No

Yes

Yes

Yes

Data Checksumming

No

No

No

Yes

The implementations described here are supported in Linux kernels after 2.2.0, and support for the higher-performance Ultra-ATA can be patched into older kernels. The Linux IDE kernel driver, which was written by Mark Lord, has a particularly low overhead for setting up transactions; it is excellent for handing small data transfers.

5.2.1.1 Improving IDE performance in Linux

There are a few things the Linux kernel doesn't do by default that can increase IDE I/O performance, such as telling the IDE device driver to use 32-bit I/O, and ensuring that the disk is performing DMA transfers. In the 2.2.16 kernel, these are all off by default. This is best accomplished via hdparm (see Section 5.5.2.1 later in this chapter).

To see whether your disk is using 32-bit IDE transfers, use the -c switch:

# hdparm -c /dev/hda

/dev/hda:

I/O support = 0 (default 16-bit)

To enable 32-bit transfers, use -c 1:

# hdparm -c 1 /dev/hda

/dev/hda:

setting 32-bit I/O support flag to 1

I/O support = 1 (32-bit)

Similarly, you can check the status of the IDE DMA transfer mode via -d:

# hdparm -d /dev/hda

/dev/hda:

using_dma = 0 (off)

You can enable IDE DMA by specifying -d 1:

# hdparm -d 1 /dev/hda

/dev/hda:

setting using_dma to 1 (on)

using_dma = 1 (on)

You may also see some performance increase by tweaking the number of sectors transferred per interrupt (set by the -m switch to hdparm). As always, benchmark the changes to make sure that you're actually improving performance. When you've found settings you'd like to keep, run hdparm -k 1 /dev/hd X to save your settings across an IDE reset.

5.2.1.2 Limitations of IDE drives

IDE is sufficient for small applications that are not very performance-sensitive, particularly if they do not issue many concurrent disk I/O operations and the overall disk subsystem load is relatively light.

IDE is an attractive, low-cost option; at the time of this writing (mid-2001), commodity IDE drives are approximately half the price and twice the capacity of equivalent-performance SCSI drives. Unfortunately, IDE has several limitations; chief amongst these are its very limited capacity for expansion per channel and the fact that it is entirely single-threaded on a per channel basis (command overlap is not allowed, even with two disks on the channel). In comparison, SCSI offers flexible device attachment, support for almost any peripheral type, and command overlap. Higher-performance devices also tend to be available for SCSI well in advance of their availability for IDE.

5.2.2 IPI

When the SCSI standard was defined, a second mass-storage standard aimed at high-performance, high-cost installations was defined, known as the Intelligent Peripheral Interface (IPI). It is clear now that IPI has no future: it has been usurped by its little sibling SCSI as the dominant high-performance disk interface. We discuss it here for historical reasons, as well as for those who have a substantial installed base of IPI devices. IPI, in addition to Fibre Channel, falls under the guidance of the T11 Technical Committee (http://www.t11.org).

The IPI standard was designed to facilitate optimization as much as possible, resulting in the highest possible performance; to that end, global optimization was built into the protocol. The fully implemented IPI model consists of four levels:

  • Level 0 defines mechanical and electrical considerations such as cables and connectors.

  • Level 1 defines primitive transfer details such as bus protocols.

  • Level 2 defines disk string protocols, which describe device-specific commands, timing, and volume addressing.

  • Level 3 defines I/O channel protocols, which include logical commands, command-queuing rules, buffering systems, etc.

A fully implemented IPI system consists of a number of IPI-3 host adapters (also called facilities), which connect to the host system. A host adapter is a complete I/O computer that is responsible for the management of a number of disk strings. Each string consists of one to eight disk drives, which are interfaced to the host adapter by an IPI-2 string controller. A disk can be connected to multiple strings, a condition known as multiporting. Up to sixteen string controllers interact with the host adapter over a high-performance IPI-3 bus. Like the host adapter, the string controller is vested with substantial intelligence; it is responsible for optimizing accesses along its string. Both the IPI-3 bus and IPI-2 string can extend up to fifty meters.

All of the intelligence in IPI is innate to the host adapter and the string controller; even primitive operations like bad-block mapping are done by the string controller. The disk drives are completely without intelligence and usually lack even local caches. As a result, an IPI disk must have a variety of conditions satisfied before it can perform a transfer:

  • The requesting string must not be otherwise occupied.

  • The string controller must have a free buffer to store the data.

  • The read/write head must be correctly positioned.

  • The data must be about to fly under the head.

Because the bus is used for essentially any transfer and is a resource that is shared between every device on the string, bus contention in IPI is a serious problem. For this reason, no more than three disks should reside on a single IPI-2 string; if the full eight disks are configured, severe performance problems arise.

In order to give the string controller enough information to optimize the request queue, IPI disks provide their rotational position and the radial position of the read/write heads. Each string controller then sorts a series of requests into an optimized order. Sometimes the string controller may be faced with a situation where it needs to retrieve data from two disks that are about to become ready to transfer. In such a case, the IPI-3 channel may be able to transfer the request to another string if the disk is connected to multiple strings, and if the operating system supports such a thing.[11] Incidentally, most IPI implementations do not fulfill the entire specification. For example, Sun did not fully implement the IPI-3 facility, instead treating each string controller as paired to exactly one host adapter. IPI performs best under massive load, when long queues form for each I/O device: if the queues are short or nonexistent, the intelligence in the IPI chain is wasted.

[11] Solaris does not.

5.2.3 SCSI

The Small Computer System Interface (SCSI, pronounced "scuzzy")[12] specification has been the mid-range workstation disk standard of the last fifteen years, and is clearly the dominant I/O interconnect in the modern workstation market. It is presently controlled by the T10 Technical Committee; see http://www.t10.org.

[12] This is something of a common joke amongst system administrators who have been fighting with a SCSI chain that is not operating properly. ("That scuzzy interface is screwed up again.")

The SCSI standard defines a peripheral bus model, which connects all the physical devices and is completely distinct from any other host bus, except for one intersection point. This intersection point is the host adapter, often called the SCSI controller. I refer to it as the host adapter, however, for reasons that will become obvious shortly.

Every device on a SCSI bus has a unique address that consists of two parts: a target identifier and a logical unit number (LUN). One analogy for this is mailboxes. Most mailboxes have a single identifying number (the target identifier) but some mailboxes have subunits, which are distinguished by the LUN (for example, a common set of mailboxes for an apartment building would have a single identifier but possibly many discrete mailboxes included within). For instance, the address "123 Anywhere Street, Apartment 918" would be equivalent to a target identifer of "123 Anywhere Street" and a LUN of "918." A request's target is specified by asserting a signal on a single line; as a consequence, an 8-bit SCSI bus has eight possible target identifiers.

Since it originally targeted small systems, the SCSI protocol was designed with an eye towards providing optimum performance at a low cost. In contrast to designs such as IPI, where much of the intelligence is vested in the controller, SCSI places the intelligence in the peripheral device itself. This greatly simplifies peripheral communication. Generally, this is handled by a microprocessor located in each device, called an embedded controller. Since this scheme means that the peripheral device controls itself rather than having an external controller, it doesn't make much sense to call a host adapter a SCSI controller.

One of the places that peripheral intelligence has greatly simplified operational procedures is defect management. Disk drives are very complex electromechanical devices, and bad blocks are inevitable. With host-intelligent systems such as IPI, the host was required to recognize disk errors and manage the defect list for every physical disk. With SCSI, this monitoring is accomplished at the disk level. In addition, the disk can monitor certain internal detail -- for example, when the defect management code is about to run out of free blocks -- and signal an impending failure to the host. One side effect, however, of peripheral-intelligent designs is that the intimate knowledge about a device resides with that device, rather than with a host controller; as a result, optimizations must be completed within the embedded controller, which can be complicated. From the host adapter's point of view, the resulting degree of abstraction allows a fantastically wide array of devices to interact over SCSI, but complicates optimization matters in large configurations.

If two devices wish to speak on the bus at the same time, arbitration is required. SCSI uses a simple scheme in which the highest-numbered target that requests the bus is given preference. For this reason, host adapters are almost always target 7.[13] It has also been common practice to assign slow devices such as tapes and CD-ROM drives high target numbers, so that they are not starved for bandwidth by fast devices. This is not generally a concern in the real world, however.

[13] This is even true (for backwards-compatibility reasons) in Wide SCSI implementations, which have 16 target identifiers.

5.2.3.1 Multi-initiator SCSI

Although it is the standard, it is not necessary to have the host adapter configured at target 7. This leads to an interesting case called multi-initiator SCSI, where a SCSI bus is shared between several systems. This can be used to share a pool of disks between two systems, which is useful in high-availability implementations. Unfortunately, it has some drawbacks:

  • There is no provision within the SCSI standard for handling pending operations when a bus reset occurs (e.g., if one initiator decides to issue a bus reset, a system panic occurs, or a transient electrical fault). This means that the state of the disks becomes undefined and data could be corrupted.

  • Although the standard defines a way for host adapters to "own" devices (via the reserve and release SCSI commands), there is no standardized way to break a reservation.

It is also possible to use multi-initiator SCSI to enable fast system-to-system communication over short distances, which is much safer than the high-availability disk sharing that multi-initiator schemes are usually used for. This functionality is available in Linux as a kernel module.

5.2.3.2 Bus transactions

Operations on a SCSI bus are quite abstract. They are accomplished by means of several operations, all of which consume bandwidth. When a target[14] wants to initiate a request, it must arbitrate the bus. Once control is granted, the initiator selects the target, and then specifies the command. The target now has to decide if it can service the request immediately. If it can (for example, the request is for a read that can be satisfied from the disk's local cache), the data is transmitted back with a message-in or message-out. If the request can't be immediately serviced (for example, the request is for a read that must actually be read from the platters), the target disconnects. At some later point, the target arbitrates for the bus, reconnects to the old initiator, and transmits the data (if appropriate) via message-in or message-out. The target then transmits the completion with a status message, and then disconnects. This results in six to eight state changes on the bus, which is busy except for the time between the disconnect and reconnect. Generally, the nondata parts of the transaction take about 140 microseconds. By comparison, transferring 2 KB of data takes about 100 microseconds. The overhead is fairly high because only the data portion is transmitted at the full burst speed: commands and status are always transferred in the slower asynchronous mode.

[14] This is generally the host adapter or a device wishing to communicate with the host adapter, but this is not always true; some systems (mostly those intended for real-time use) allow intradevice operations.

5.2.3.3 Synchronous versus asynchronous transfers

The SCSI standard defines two means of transferring data: synchronous and asynchronous. This refers to the way that the host adapter and its targets coordinate data transfer. In asynchronous transfers, data can be transmitted at an arbitrary time; the receiver and the sender need not be clock-synchronized. This kind of transfer necessitates that each byte be acknowledged, which imposes a practical limit on transmission speed. In synchronous mode, transfers may only start at specific clock phases, but this alleviates the acknowledgement problem that asynchronous transfers suffer from. The host adapter tries to negotiate the best way to communicate with devices when the targets are set up. If for some reason a synchronous speed cannot be negotiated with a target, the host adapter falls back to asynchronous transfers. You can find the negotiated synchronous transfer speed in Solaris via prtconf -v and by looking for the SCSI device driver. The next example is from a Fast SCSI-2 drive connected to an Ultra 1 workstation:

% prtconf -v

...

            esp, instance #0

               Driver properties:

                   name <target0-sync-speed> length <4>

                       value <0x00002710>.

...

In this case, the negotiated synchronous transfer speed to the device was configured as target 0; attached to this controller, it is 0x00002710 KB/second. This number is reported in hexadecimal; representing it in decimal and dividing by a thousand gives us the negotiated speed, which is 10 MB/second.

This is often worth checking. Using cables that aren't of sufficiently high quality or have improper termination issues can often cause devices to fall back to a slower speed. If you aren't getting the performance out of your SCSI chain that you think you should be, take a look here first.

5.2.3.4 Termination

SCSI operates electrically via a straightforward bus mechanism. Traffic from one device is sent out on the bus; the receiving device is responsible for pulling traffic destined for it off the bus. However, data that goes past the last device on the SCSI chain can be "echoed" back down the bus. This reflection phenomenon causes severe problems that must be addressed by termination. In a nutshell, termination is the addition of a set of resistors to a device in order to prevent signal reflection by "absorbing" data that would otherwise echo.[15] The rules for termination are, in theory, quite simple: terminate the first and last devices on a SCSI bus. Many modern disks will automatically terminate themselves as necessary. Termination problems are generally characterized by extremely slow data rates or devices that don't appear reliably (if at all). Note that, because of their different electrical designs, differential and single-ended terminators are not compatible (to learn more about the difference, see Section 5.2.3.6 later in this chapter).

[15] Termination is a fairly complex topic, and a full discussion is beyond the scope of this text.

5.2.3.5 Command queuing

When the original SCSI specification was drafted, performance was a secondary concern: the market at which the standard was being aimed was more interested in low cost than high performance. By the time that the SCSI-2 revision was issued, performance had become a far more important issue. In order to handle the problem of request optimization (that is, in which order to service requests), something needed to be done.

However, since each SCSI device has a high degree of autonomy, the host adapter lacks sufficient information to make real optimization decisions. Performing global optimization isn't possible, as was customary (and implemented in other standards, such as IPI). The SCSI-1 standard made this worse by permitting only one command to be pending to any given target; if only one request can be outstanding, no optimization can be performed!

This problem was addressed in SCSI-2 by adding command queuing, in which the target is able to accept multiple requests. It then processes them in the most advantageous order. Each request is assigned a tag (hence the synonymous term tag queuing), which serves as an identifier so that each entity can associate a given communication with a request. This process improved performance by allowing the overlap of bus transactions with physical activity, which saves a substantial amount of time, especially in applications that involve a lot of small disk writes. Unfortunately, command queuing is most effective when the disk is very heavily loaded, and generally you will get more out of spreading out the disk usage to reduce utilization than you will from the command queuing optimization. It has a limited impact on throughput for single-threaded tasks common to workstations.

5.2.3.6 Differential signaling

Let's say that you and a friend are both standing on a strip of beach somewhere, and your friend wants to send you a message. He has a lantern, and he holds it near his feet if he wants to send a zero and near his head if he wants to send a one. This works well as long as you can clearly distinguish which is the "high" position and which is the "low" one. If he gets in a boat and sails a mile offshore, you will have a much harder time determining which lantern position is high and which is low. However, if he takes two lanterns with him, and holds them at the same height for a zero, and raises one and lowers the other for a one, you will have a much easier time. The first mode of signaling, with only one lantern, is called single-ended; the second is called differential.

Low-voltage differential (LVD) signaling is a new electrical standard, introduced with the SCSI-3 specification, that accomplishes essentially the same thing as differential signaling but at a lower cost.

Differential signaling arose to alleviate single-ended SCSI's limited transfer range: single-ended, 20 MB/sec implementations are limited to just a meter and a half. Differential implementations can send data as far as 25 meters. Since these distances refer to the entire signal path -- including the cables between units, most of the traces on the embedded controller, and about an extra foot for every connector[16] -- a given length of cable will not always go as far as you might think.

[16] This is due to impedance issues across the connectors.

Differential signaling does not provide any performance improvement in and of itself. If the only difference between two SCSI buses is that one uses single-ended and the other uses differential signaling, they will perform exactly the same.

5.2.3.7 Bus utilization

Traditional performance recommendations suggest that no disk string should ever have a utilization of more than about 40%. This is based on old mainframe disk structures, which generally lacked caching at the disk; therefore, for a transaction to complete, the read/write head must have access to the controller in order to perform any transfer at all. If the bus is busy when the data flies under the head, the system must wait for the next window to perform the I/O -- at least 8.3 milliseconds later on a 7200 rpm disk. This is known as a rotational position sense miss (RPS miss). Because SCSI targets always have local caches, however, much higher bus utilization can be obtained before performance starts to suffer. Remember that SCSI is a high overhead architecture; each request can take six to eight state changes. For small I/O-size operations, this overhead starts to dominate performance (approaching 60% of total time). The crucial issue then becomes how the high SCSI overhead shapes performance.

For sequential operation, seek and rotation times are not significant in determining performance -- bus utilization problems are the core issue. For 2 KB I/O operations on a 20 MB/second chain, performance plateaus at about 8 MB/second, no matter how many disks are configured. The bus is capable of far greater performance, and using 64 KB I/Os results in a 95% reduction in the number of commands issued and maximum performance of about 17 MB/second. Here is a general guideline for configuring SCSI disk subsystems for sequential access:

  • Configure 20 MB/second chains so that no more than 4-8 drives are active at any one time. Large (~64 KB) I/O sizes will saturate the bus with 4 drives; small (~2 KB) I/O sizes will take 8 drives. (Remember that you will see much higher throughput with 64 KB I/Os than with 2 KB I/Os: the amount of data the bus can transfer is the sum of both actual data and command overhead, and with 2 KB I/Os the majority of data bits on the bus are commands!)

A disk array consisting of a number of disk drives is equivalent to the same number of disk drives directly connected to the host adapter.

Expect sequential throughput to be substantially less than the rated bus speed, particularly for small I/O sizes.

In random-access operation, a completely different phenomenon is observed. If accesses are truly random, the disks must seek before every I/O operation. Under these circumstances, the throughput of the disks drops dramatically (to something on the order of 250 KB/second). As would be expected, then, you can configure as many disks as you want: throughput scales linearly. Of course, at 250 KB/second per disk and fifteeen disk drives, you are only sustaining about 3.7 MB/second, so it would be surprising if you saw any bus throughput limitations.

  • If data access is truly random, bus performance is not a limiting factor at all. You can safely configure ~80 devices on a single 20 MB/second bus.

  • For real-world random access conditions, configure 1 or 2 disk arrays (16-64 devices) per 20 MB/second bus.

5.2.3.8 Mixing different speed SCSI devices

One common misunderstanding about SCSI is how faster devices are affected by slower devices on the same bus. This question usually comes up when someone installs a new, fast SCSI disk on the same chain as an older, slower disk. The SCSI standard is carefully designed so that the management of an individual device is isolated from its environment as much as possible. The host adapter negotiates an appropriate burst transfer speed and mode with each device on the bus individually, and the bus is driven to the limits of each device. However, some complexity does arise; for example, the utilization of the bus varies with the speed of the transfer, particularly if asynchronous transfers are involved. Slower devices tend to hog the bus, since they cannot select fast enough burst transfer speeds to get off the bus expediently. This is almost never an issue in a workstation environment, but when many devices are present on a system, it is often a good idea to segregate low speed devices onto a separate bus to maintain throughput elsewhere.

5.2.3.9 SCSI implementations

In the spring of 1979, Shugart Associates began defining the Shugart Associates System Interface (SASI), which was subsequently released to the public domain in order to encourage disk controller companies to use the protocol. In 1980, SASI was placed before the ANSI Committee X3T9.3, in order to have it accepted in place of IPI (see Section 5.2.2 earlier in this chapter).

In the summer of 1981, NCR began exploring the possibility of using SASI instead of a proprietary interface that they had recently defined. This idea fell through due to fears that technical improvements would induce market confusion. In the early fall of 1981, however, Optimem (a Shugart subsidiary) requested the same technical improvements that NCR had. In a move to accept the inevitable, Shugart reapproached NCR, and a series of meetings were held to more fully document SASI and add several features drawn from NCR's proprietary interface. In December of 1981, NCR and Shugart jointly requested that X3T9.3 grant time at their next meeting in February of 1982 to consider the joint proposal. X3T9.3 responded by breaking the meeting into two parallel sessions, one for IPI and one for SASI. At the conclusion of that meeting, the sessions rejoined and decided that SASI should be assigned to an existing task group, X3T9.2, which had no current project. In April of 1982, X3T9.2 met and drafted a formal project proposal for the Small Computer System Interface (SCSI), which was to be based on SASI. Throughout 1982 and 1984, X3T9.2 met regularly, fully documenting SCSI with contributions from NCR (which proposed command sets for tapes, processors, and printers) and Optimem (which proposed an optical write-once, read-many, or WORM, command set). In April of 1983, NCR announced the NCR-5385, the first SCSI protocol chip; it was followed in April 1984 by the NCR-5380, which added significant hardware support for inexpensive SCSI implementations.[17] Also in April of 1984, X3T9.2 voted to forward SCSI to its parent committee in order to begin the lengthy ANSI standards approval process. Approval was gained in June of 1986, and SCSI was formally ANSI X3.131-1986.

[17] The NCR-5380 was the chip initially used by Apple Computer for the SCSI bus on the Macintosh Plus.

In July of 1985, a special working group began a series of meetings to define enhancements to the SCSI command set for disks, later known as the Common Command Set (CCS). In June of 1986, X3T9.2 began the SCSI-2 project, which endeavoured to add support for the CCS and other enhancements into the SCSI protocol. 1986 to 1989 was a period of intense work, and the SCSI standard grew from 200 pages to nearly 600 pages. SCSI-2 was forwarded for ANSI approval in February of 1989, and approved in September of 1990.

Further development work in SCSI, specifically SCSI-3, involved splitting the effort into multiple task groups, each focusing on specific parts of the protocol.

Although SCSI-2 is completely backwards-compatible with SCSI-1, SCSI-3 is not completely compatible with SCSI-2 due to some aggressive hardware-oriented extensions. However, software is not affected.

The SCSI-3 standard has been adopted piecemeal; many vendors have chosen to implement some of the more compatible parts already. For an overview of the SCSI implementations, please consult Table 5-4.

Table 5-4. SCSI implementation overview

SCSI-1

SCSI-2

SCSI-3

Media

Copper

Copper

Copper

fibre

Topologies

Tree

Tree

Multi-initiator tree

Tree

Multi-initiator tree

Arbitratrated loop

Fabric

Signalling

Single-ended

Single-ended

Differential

Single-ended

Differential

LV differential

Command queuing

None

Optional

Optional

Transfer modes

Synchronous

Asynchronous

Synchronous

Asynchronous

Synchronous

Asynchronous

Transfer clocking

5 MHz

5 MHz

10 MHz

5 MHz

10 MHz

20 MHz

Transfer width

8-bit

8-bit

16-bit

8-bit

16-bit

I find myself answering a lot of questions about exactly how fast a given standard is. A lot of people I know have ended up building tables that look remarkably like Table 5-5 and Table 5-6. I hope you find these useful.

Table 5-5. Deciphering SCSI variants

Standard

Clock (MHz)

Width (bits)

Max. rate (MB/sec)

Max. devices

Max. length[18]

SCSI-1

SCSI-1

5

8

5

8

6.0

Fast

SCSI-2

10

8

10

8

3.0

Wide

SCSI-2

5

16

10

16

3.0

Fast/Wide

SCSI-2

10

16

10

16

3.0

Ultra (fast-20)

SCSI-3

20

8

20

8

SE: 1.5 Diff: 12.5

Ultra Wide (fast-40)

SCSI-3

20

16

40

16

SE: 1.5 Diff: 12.5

Ultra2

SCSI-3

40

8

40

16

SE: 1.5 Diff: 12.5

Ultra2 Wide (fast-80)

SCSI-3

40

16

80

16

SE: 1.5 Diff: 12.5

Ultra160

SCSI-3

80

16

160

16

Diff: 12.5

[18] In meters, for synchronous operation. Unless specified, for single-ended (SE) configurations only.

Table 5-6. Deciphering SCSI connectors

Connector

Pins

Form factor

Used for

SCSI-1

50

D-shell connector 3 rows of pins

SCSI-1

DB-25

25

D-shell connector 2 rows of pins

SCSI-1, narrow SCSI-2

Primarily used on Macintosh

Centronix 50

50

Centronix connector

SCSI-1, narrow SCSI-2

Micro 50

50

Micro-type connector 2 rows of pins

Narrow SCSI-2, narrow SCSI-3

Micro 68

68

Micro-type connector 2 rows of pins

Wide SCSI-2, wide SCSI-3

SCA

80

Micro-type connector 2 rows of pins

SCSI-2, SCSI-3

Internal attachment only

"Universal" SCSI connector: integrated power and ID select

VHDC

68

Very High Density connector

Differential SCSI-3

HD-60

60

Micro-type connector with a bar instead of pins

Narrow SCSI-2

Old IBM SCSI adapters only

HD-68

68

Micro-type connector with a bar instead of pins

Wide SCSI-2

Old IBM SCSI adapters only

5.2.4 Fibre Channel

The Fibre Channel (FC) specification, defined by ANSI X3T9.3, uses SCSI-like commands, typically over fiberoptic cable rather than copper wires. Fibre Channel standards are the purview of the T11 Technical Committee (http://www.t11.org).

Fibre Channel is a full-duplex medium, meaning that it can support bidirectional data flow; it requires a fiber strand for each direction. Although it is defined by its own standard, it has been brought under the SCSI-3 umbrella. Fibre Channel has three very different topologies:

Point-to-point

One device connects to exactly one other device.

Arbitrated loop

Often called FC-AL, the devices and one or more hosts are connected in a ring using many point-to-point links.

Fabric

Switches and hubs are used to create a complex network in which there may be multiple paths from a given host to a given device. This is becoming known as storage-area networking, and is probably the direction enterprise-class storage will move in the future. Unfortunately, a discussion of the performance of storage area networks (SANS) is far too complicated for this text.

Fibre Channel devices are assigned a unique address, called the world wide name (WWN). Since devices can be connected into arbitrary fabrics, the common practice is to assign each device a completely unique WWN, much like Ethernet addresses are assigned. The Fibre Channel standard defines three classes of signaling, which are not interchangeable. They are usually described in terms of their data speed: 25 MB/second (FC-25), 50 MB/second (FC-50), or 100 MB/second (FC-100). The storage industry as a whole initially deferred acceptance of Fibre Channel until the arrival of FC-100 parts, but it is now in widespread use. In general, Fibre Channel disks should work just like SCSI disks. Be sure to follow any configuration rules provided by the vendor carefully.

Fibre Channel host adapters typically have two optical link modules, which enable the host adapter to connect to two independent Fibre Channel buses; the connection is managed by the serial-optical coupler, or SOC (typically called a GBIC). A typical SOC can handle up to 255 pending requests, which can occasionally become a limitation for single-connection interfaces and is usually a bottleneck for biported interfaces. In order to work around this restriction, it is vital that FC-interfaced devices be driven at large I/O sizes (8-64 KB). In light of this fact, the second port is probably most useful for attaching archival or other low-use devices.

One of Fibre Channel's most useful capabilities is the removal of distance restrictions imposed by the limitations of copper wires; the FC standard allows cable runs of up to ten kilometers. This allows the geographic dispersal of storage, which has remarkable implications for disaster recovery.

5.2.5 IEEE 1394 (FireWire)

Just like Fibre Channel, the IEEE 1394 specification has also been brought into the SCSI-3 fold. IEEE 1394, also known as FireWire, was originally developed by Apple Computer. It is rapidly entering widespread use in the midrange market for desktop systems.

By far the most important thing to consider in evaluating IEEE 1394 peripherals is that the bus speed is quoted in megabits per second -- not megabytes per second -- just like network interconnects. Currently, it is available in 100, 200, and 400 Mbits per second variants. IEEE 1394 supports the interconnection of up to 63 devices to a single chain, and is hot-pluggable.

IEEE 1394 supports isochronous data transfers, which differ from the more typical asynchronous data transfer type by guaranteeing the timely delivery of data. When an isochronous data transfer device such as a digital video camera is connected to the IEEE 1394 bus, that device grabs the specific, allocated portion of the bus that it requires. The bus also reserves 20% of the total available bandwidth for serial command overhead. Once the entire capacity of the bus has been allocated, the bus will not recognize additional devices -- regardless of whether those devices are actively transferring data. In addition, a condition known as multiple device overhead starts to exist when more than a handful of devices are introduced onto the bus; it then increases in impact as the number of attached devices increases. The self-configuration and hot-pluggability aspects of the technology increase latency in communicating with each device as the number of devices increases.

In general, IEEE 1394 is an excellent means of attaching devices to workstations, particularly if they are time sensitive (e.g., digital video production), but it is not ideally suited to even a medium-sized storage environment, due to the increasing drive overhead that accompanies multiple devices on the bus.

5.2.6 Universal Serial Bus (USB)

Universal Serial Bus, or USB, was developed to be a serial and parallel port replacement. When viewed in this context, it represents a large performance increase on the order of 250 Kbps to 12 Mbits/second, as well as ease-of-use improvements because it is hot-pluggable, has a robust power management, and is able to support up to 127 devices through a hub-based architecture.

Because of its relatively high peak data transfer rate (1.5 MB/second), USB became a reasonable connection method for relatively low-bandwidth devices such as printers, floppy drives, Zip drives, and small scanners. The inherently simple design that allows USB devices to be so affordable also forces them to be very host-centric; the microprocessor of the host system must initiate every transaction. As a result, USB does not scale well, and it is probably not a good idea to use it in anything beyond casual desktop peripheral connectivity.


5.3 Common Performance Problems

There are three common disk performance problems: I/O skew, disk overloading caused by paging, and unexplainably high service times on idle drives.

5.3.1 High I/O Skew

It is easy to overload a disk with too many accesses. A condition called high I/O skew exists when some disks have much higher loads than others. It is hard to overestimate the impact of excessive resource utilization. It is easy, if subjective, to diagnose high I/O skew: just use the iostat tool, discussed in greater detail in Section 5.5.4 later in this chapter, and look for a wide range in the sevice times and busy percentatges of the disks in the system.

The solution to this problem is very simple in theory: spread the workload over several disks. There are quite a few ways of accomplishing this:

  • Split the workload on functional grounds. For example, if the overloaded disk contains two large databases, move one to another disk. This assumes a stable workload and isn't often appropriate in the real world.

  • Split the workload by users. If the overloaded disk contains the home directories for both the engineering group and the documentation group, move one group to another disk. As with a functional split, this assumes a stable workload.

  • Allocate disk space on a round robin basis. This works well, but requires application-level support; the Solaris kernel uses this method to distribute accesses across multiple swap spaces.

  • Concatenate several disks. This method tends to overload a single "hot" disk while keeping the rest of the data relatively available for use.

  • Configure several disks into a stripe (a RAID 0 array; we'll discuss this in greater detail in Section 6.2.1). The interlace size will largely determine performance: too large of an interlace will not be sufficient to spread the data across multiple disks, and too small will cause small accesses to activate multiple disks. Stripes cannot be grown online, however, which limits future expansion, and they are vulnerable to disk failure.

  • Configure multiple disks into a combined mirror/stripe array. This is the fastest, safest storage configuration, but can be very expensive.

  • Configure a parity-protected array (typically RAID 5; see Section 6.2.6 in Chapter 6). This is cheap and safe, but may suffer substantial performance penalties.

There is no easy solution to this problem. It depends on the specifics of your installation and your budget.

5.3.2 Memory-Disk Interactions

The idea behind paging and swapping -- freeing memory by using disk space as a backing store -- in essence transforms an immediate memory problem into a possible disk problem. For more details on the mechanics of paging, see Section 4.3. Memory is far faster than disk, and in a situation where the system is forced to use paging space heavily, the relatively slow disk subsystem can create a serious bottleneck. Ideally, paging space should be located on an isolated controller and a fast disk that is not being used for anything else.

If you are using multiple swap areas, the Solaris kernel provides a simple, automatic, and effective round robin mechanism that distributes this I/O across multiple disks. The Linux kernel performs a similar optimization across swap areas of equal priority. This /etc/fstab file will result in a three-way stripe of swap space in Linux:

/dev/sda1       swap      swap     pri=1

/dev/sdb1       swap      swap     pri=1

/dev/sdc1       swap      swap     pri=1

Mirroring swap space may provide substantial reliability improvements, however. This reliability improvement does exact a toll on performance, in the form of more writes being issued.

5.3.3 High Service Times

Sometimes disks do very odd things. One of these is that lightly used disks often have extremely large service times (hundreds of milliseconds). The disk can't possibly be overloaded, so why does this happen? This situation has been conjectured to be caused by rounding errors at low activity levels, thermal recalibration, or problems with the filesystem layout. However, the problem was analyzed in some detail by Adrian Cockcroft of Sun Microsystems, and the root cause is the fsflush or bdflush process flushing memory to disk (see Section 4.4.2). Although it looks unusual, it is not a problem.


5.4 Filesystems

Modern filesystems come in two flavors: block-based and extent-based. An indirect block-based filesystem breaks all files into 8 KB data blocks and spreads them all over the disk; additional 8 KB indirect blocks keep track of where the data blocks reside. For files that are more than a few megabytes in size, doubly indirect blocks are used to store the location of the indirect blocks. When a file is created, only the minimum number of blocks are used. When a file is extended, extra blocks are taken from a map of unused blocks. The disk allocation system tries to optimize these spatial relationships so that the majority of file accesses are contiguous and don't require extensive seeking. Every time a block is assigned, however, this data must be written to disk. The metadata is always written synchronously, without the benefits of any caching. Metadata operations can have a substantial impact on I/O subsystem performance, precisely because they are synchronous.

An extent-based filesystem defines files in terms of extents, which contain a starting point and a size. If a 1 GB file is written, it is a single 1 GB extent. There are no indirect blocks: a sequential reads the extent, then all of the data. This means that there is very little metadata overhead in sequential file access. Unfortunately, extent-based filesystems tend to suffer from severe fragmentation problems. After many files have been created and deleted, it's hard to find the space to allocate a large file. As a result, all extent-based filesystems come with tools to defragment the filesystem. Note that filesystem defragmentation tends to be quite I/O intensive, for obvious reasons, and therefore, the timing of this event in relation to critical production tasks must be carefully decided. Figure 5-2 shows block-based versus extent-based filesystems.

Figure 5-2. Block-based versus extent-based filesystems

5.4.1 vnodes, inodes, and rnodes

Unix systems traditionally use inodes to keep track of information required to access a file (such as the file size, owner, permissions, where the data blocks are stored, etc.) System V systems, including Solaris, use a higher-level abstraction called a virtual node, or vnode. This simplifies implementing new filesystems: the kernel works in terms of vnodes, each of which contains some kind of inode that matches the underlying filesystem time. For UFS, these structures are still called inodes; for NFS, they are called rnodes.

5.4.1.1 The directory name lookup cache (DNLC)

Any time a file is opened, the directory name lookup cache, or DNLC, comes into play. This is a low level cache in kernel memory of what filenames are associated with which vnodes. When a file is opened, the DNLC figures out the right inode from the filename given; if the name is in the cache, the system does a fast lookup to avoid scanning directories. Each DNLC entry is a fixed size, so there is only space for a pathname component (e.g., /etc and password are distinct pathname components of the pathname /etc/passwd) of up to 30 characters, and longer filenames are not cached. Directories that have thousands of entries often take quite a while to search for a specific file, so a good DNLC hit rate is important if lots of files are being opened and the directories are quite large. As a result, you should strive to keep heavily used directory and symbolic link names down to less than 30 characters. In practice, this is not usually a problem.

You generally do not need to tune the DNLC, unless you are providing NFS service on a machine with less than 512 MB of memory. A good value in these situations is 8,000 or so. The maximum tested value is 34,906, which corresponds to a value of 2,048 for the maxusers parameter. The default size for a given system is determined by maxusers x 17) + 90, with a maximum value of 17,498 without explicit tuning. (We discuss parameters influenced by maxusers in Section 2.2.2.)

The vmstat -s command shows the DNLC hit rate since the system came up. In general, a hit rate of less than 90% indicates that some attention is required:

% vmstat -s |grep 'name lookups'

 34706339 total name lookups (cache hits 98%)

You can determine how many name lookups per second are being performed by inspecting the namei/s field reported by the sar -a command:

% sar -a 1 5

...

16:44:54  iget/s namei/s dirbk/s

16:44:55     111     603     203

There are two similar caches for inodes and rnodes. We will discuss these in the sections specific to those filesystems (Section 5.4.2.7 later in this chapter and Section 7.5.2.2).

5.4.2 The Unix Filesystem (UFS)

The Unix Filesystem (UFS) is the default Solaris filesystem; it is an indirect, block-based filesystem. Every file stored in the filesystem actually consists of several components:

  • The directory entry, which contains the name of the file, its permission mask, and creation, modification, and access dates.

  • The data blocks, which contain the actual data.

  • The access control list entries (optional), which provide a more robust mechanism for access control than the standard Unix permission model does.

When the filesystem is created, space is allocated for inodes, which store the directory entries and access control list entries. This number is static, and may not be increased without recreating the filesystem. You can find out how many free inodes are available by examining the output of df -e:

% df -e

Filesystem              ifree

/proc                     862

/dev/dsk/c0t0d0s0      294838

fd                          0

/dev/dsk/c0t0d0s4      246965

swap                     9796

/dev/dsk/c1t0d0s0     3256077
5.4.2.1 inode density

By default, the UFS filesystem allocates 1 inode for every 2-8 KB of usable disk space, as summarized in Table 5-7.

Table 5-7. Default UFS inode densities

Filesystem size

KB per inode

< 1 GB

2,048

< 2 GB

4,096

< 3 GB

6,144

> 3 GB

8,192

Each inode consumes a small amount (512 bytes) of disk space, but it adds up quickly. One inode per 16 KB of disk space is a much more reasonable default in most cases. In cases where the filesystem holds very large files, such as in a scientific workload, a much lower density is appropriate; as little as one inode per megabyte of disk space might be reasonable. Similarly, filesystems that are populated with a large number of very small files might do well with the default allocation. An excellent example is a filesystem that contains a Usenet news spool, which creates one file for every news article (typically very short).

The inode density can be configured when running newfs by using the -i bytes-per-inode switch.

5.4.2.2 Filesystem cluster size

When a file is being extended, the filesystem clusters data in 8 KB blocks whenever possible. This behavior permits the entire cluster to be written to disk in a single operation. It gives rise to two important advantanges:

  • Writing in a single operation is more efficient from the operating system's point of view, since I/O overhead is constant regardless of how much data is written.

  • When the file is read later, the entire cluster can be read in a single operation. This saves I/O overhead, but more importantly saves a disk seek between the retrieval of each block.

The number of contiguous filesystem blocks that will be stored on the disk for a single file is defined by the maxcontig filesystem parameter. In Solaris releases prior to 8, this parameter is set to seven disk blocks, or 56 KB, by default. This is the default because of a design limitation present in older Sun hardware.[19] Solaris 8 changed the default to 128 KB (16 blocks). Changing this parameter has the greatest impact in sequential operation, but should not severely impact randomly accessed filesystems, since such accesses generally do not read entire clusters. Even though a filesystem may have entirely random access patterns, keep in mind that the process of data backup is inherently sequential: configuring a large cluster size may accelerate backups substantially. The biggest gain in tuning maxcontig is achieved in RAID 5 devices, since with the default value each write almost ensures a very expensive read/modify/write operation (see Section 6.2.6).

[19] Specifically, systems based on the sun4 architecture are unable to transfer more than 64 KB in a single operation. The sun4c, sun4m, sun4d, and sun4u architectures do not have this problem.

The filesystem cluster size is specified by the -C blocks switch to newfs when the filesystem is being created, or it can be modified on existing filesystems by specifying the -a switch for tunefs.

Tuning maxcontig after the filesystem has been created does not accomplish quite the same thing as tuning maxcontig during filesystem creation; postcreation tuning does not reconfigure blocks that have already been written.

In general, maxcontig should be set to an integer multiple of the data width of the stripe on filesystems that reside on striped or RAID 5 devices.[20] For example, on a 6-disk RAID 5 volume with a 32 KB interlace, the stripe width is 160 KB (since one disk is used for parity computation), and a reasonable value of maxcontig is 160 or 320 KB (20 or 40 blocks, respectively). On filesystems that contain very large, sequentially accessed files, maxcontig should be set to a value equal to at least double the data width of the storage device.

[20] Remember that RAID 5 arrays have a data width of one disk less than the number of disks in the array.

It may also be necessary to tune the maximum I/O transfer size in order to derive the maximum benefit from tuning maxcontig to values above 128 KB. This can be done by setting the maxphys, md_maxphys, and vxio:vol_maxio kernel tunables in /etc/system. The first two are in units of bytes, and vxio:vol_maxio is in units of 512 bytes.

In random access workloads, different guidelines will apply. In summary, you should try and match the I/O size and the filesystem cluster size, then enable as large a filesystem cache as possible (if this parameter is tunable), and disable prefetching and read-ahead.

5.4.2.3 Minimum free space

When a filesystem is created, some fraction of the storage space is reserved in case of future emergencies. This fraction is governed by the minfree parameter, which is historically 10% of the filesystem size. This dates back to computing antiquity; in addition to giving a reasonable overflow amount, it provided a means of ensuring that the system could easily find a free disk block when the disk was nearly full. In modern environments, where 200 GB filesystems are quite common, a 10% free space reserve accounts for 20 GB of disk space! Since locating free blocks within even a 20 MB area is easy, the original performance concerns are moot as well.

In order to minimize the impact of this overhead, set minfree to be 1%. The minimum free space can be set when you are creating the filesystem by specifying -m percentage to newfs.

5.4.2.4 Rotational delay

The rotational delay parameter is a throwback to the days when disk controllers weren't able to handle data at the speed that the disk platter could provide it. In such cases, a strategy of storing blocks physically out-of-order was developed, in which two blocks were stored with enough radial position between them that the disk would rotate into position to read the second block just as the processing of the first block finished.

For example, consider a disk that rotates at 11 ms per revolution (about 5,400 rpm) but whose controller takes 2 ms to process each block. If data blocks are sequential, by the time the first one is read and processed, the second block has already passed under the heads, so we must wait for the platter to spin back into position; satisfying the read takes over 20 ms. If the blocks are physically located a quarter-revolution (2.75 ms) apart, however, the controller can handle the second block as soon as it is presented, and we can perform the whole operation in about 9 ms. These days disk controllers are fast enough to not be bothered by this problem, so rotdelay is best set to zero, which is the default.[21]

[21] If you are using magneto-optical disks, or another sort of unusually slow media, you may still want to set the rotdelay parameter.

The rotational delay can be set at filesystem creation by specifying the -d rotational-delay option to newfs.

5.4.2.5 fstyp and tunefs

You can determine information about a specific filesystem via the fstyp command. This command generates a lot of output, of which the first 18 lines are useful:

% fstyp -v /dev/dsk/c0t0d0s3 | head -18

ufs

magic   11954   format  dynamic time    Fri Feb 16 15:01:02 2001

sblkno  16      cblkno  24      iblkno  32      dblkno  456

sbsize  5120    cgsize  5120    cgoffset 72     cgmask  0xffffffe0

ncg     272     size    7790674 blocks  7670973

bsize   8192    shift   13      mask    0xffffe000

fsize   1024    shift   10      mask    0xfffffc00

frag    8       shift   3       fsbtodb 1

minfree 1%      maxbpg  2048    optim   time

maxcontig 16    rotdelay 0ms    rps     120

csaddr  456     cssize  5120    shift   9       mask    0xfffffe00

ntrak   27      nsect   133     spc     3591    ncyl    4339

cpg     16      bpg     3591    fpg     28728   ipg     3392

nindir  2048    inopb   64      nspf    2

nbfree  886360  ndir    2149    nifree  888732  nffree  6320

cgrotor 249     fmod    0       ronly   0       logbno  1216

fs_reclaim FS_RECLAIM

file system state is valid, fsclean is -3

The most important options are those that we've discussed previously: minfree, maxcontig, and rotdelay.

It's possible to tune a filesystem after it has been created by the means of the tunefs command.[22] In fact, all three UFS attributes we've focused on can be tuned: maxcontig via the -a switch, rotdelay via the -d switch, and minfree by -m. For example, to change the maximum number of contiguous blocks on a filesystem:

[22] "You can tune a filesystem, but you can't tune a fish." -- BSD tunefs manpage

# fstyp -v /dev/dsk/c0t0d0s3 | grep maxcontig

maxcontig 16    rotdelay 0ms    rps     120

# tunefs -a 32 /dev/dsk/c0t0d0s3

maximum contiguous block count changes from 16 to 32

# fstyp -v /dev/dsk/c0t0d0s3 | grep maxcontig

maxcontig 32    rotdelay 0ms    rps     120
5.4.2.6 Bypassing memory caching

If you are performing sequential I/O on very large files (in hundreds of megabytes to gigabytes range), or if you have an extremely random I/O pattern on small files, you may want to disable the buffering that is normally performed by the filesystem code. You may turn this buffering off for a UFS filesystem by using the directio option to mount_ufs(1M). However, UFS still suffers from some indirect block limitations, so using directio is not as fast as true raw access. Because direct I/O bypasses the filesystem cache, it also disables filesystem read-ahead; small reads and writes result in many separate I/O requests to the storage device, instead of being clustered into larger requests. Another side effect of direct I/O is that it does not put load on the Solaris memory subsystem, and removes the paging activity that is typically seen in pre-Solaris 8 systems without priority paging enabled (see Section 4.4.3).

5.4.2.7 The inode cache

The first of two interesting in-kernel caches relating to the UFS filesystem is the inode cache. Whenever an operation is performed on a file residing on a UFS filesystem, that inode must be read. The rate of inode reads is reported as iget/s by the sar -a command. The inode read from disk is cached in case it is needed again. When a file is not in use, its data is cached in memory by an inactive inode cache entry (each inode cache entry has the pages for the corresponding file attached to it); if an inactive entry that has pages attached to it is recycled for reuse, its associated pages are freed and placed on the free list. This is reported as %ups_ipf by sar -g. Any nonzero values reported here mean that the inode cache is too small.

The inode caching algorithm is implemented by keeping a "reuse list" of blank inodes. These are available instantly. The number of active inodes isn't limited, but the number of idle inodes (inactive, but cached) is kept between the value of ufs_inode and 75% of ufs_inode by a scavenging kernel thread. The only upper limit is the amount of kernel memory consumed by the inodes. The tested upper limit is 34,906, which corresponds to a maxusers value of 2,048. To get the raw statistics regarding the inode cache, run netstat -k inode_cache or kstat -n ufs:0:inode_cache:

% netstat -k inode_cache

inode_cache:

size 22207 maxsize 68156 hits 1044055 misses 23085 kmem allocs 22207 kmem frees0

maxsize reached 22218 puts at frontlist 95088 puts at backlist 544

queues to free 45284 scans 2558333385 thread idles 0 lookup idles 0 vget idles 0

cache allocs 23085 cache frees 880 pushes at close 0

If the maxsize_reached value is greater than the maxsize (which is equal to ufs_ninode), then the number of active inodes has exceeded the cache size at some point in the past, and you should probably increase ufs_ninode.

5.4.2.8 The buffer cache

UFS utilizes a second caching mechanism, called the buffer cache. In BSD Unix derivatives, the buffer cache is used to cache all disk I/O; in SVR4 Unices, such as Solaris, it is used to cache inode, indirect block, and cylinder-group related disk I/O only. The nbuf tunable controls how many page-sized buffers have been allocated, and a variable called p_nbuf controls how many buffers are allocated at once. By default, p_nbuf is set to 100. A variable called bufhwm, which is set by default to 2% of system memory, defines the maximum amount of memory that the buffer can use. You can see how big the buffer cache is on your system by /usr/bin/sysdef | grep bufhwm:

# /usr/sbin/sysdef | grep bufhwm

20914176        maximum memory allowed in buffer cache (bufhwm)

You can get statistics on the buffer cache by looking at kstat unix:0:biostats:[23]

[23] The kstat command is a recent addition to Solaris. Historically, there was an undocumented -k switch to netstat that would let you get these statistics. If your system doesn't have kstat, try netstat -k biostats.

% kstat unix:0:biostats

module: unix                            instance: 0

name:   biostats                        class:    misc

        buffer_cache_hits               40596836

        buffer_cache_lookups            40707195

        buffers_locked_by_someone       8221

        crtime                          49.729349302

        duplicate_buffers_found         0

        new_buffer_requests             0

        snaptime                        3630491.75512563

        waits_for_buffer_allocs         0

Comparing the buffer cache hits with the number of lookups indicates that the hit rate since the system came up is 99.981%, which is pretty reasonable. Contrary to some past writing in some Sun publications, you should not generally need to tune any of the buffer cache parameters.

If you see a hit rate less than 90% in the buffer cache, and your filesystem is more than 40,000 times the amount of physical memory (e.g., if you have 1 GB memory and your filesystem is ~40TB), then you should increase the buffer cache to some larger value.

5.4.3 Logging Filesystems

In order to implement the robustness required of modern systems, a filesystem must be able to provide reliable storage to its applications, and if a failure occurs, it needs to recover rapidly to a known state. This has not always been the case: system crashes have traditionally left the filesystem in an inconsistent state, and recovering that consistency took a long time.[24] One approach to overcoming these issues is to use a mechanism called logging (also known as journaling), in which we try to prevent the filesystem structure from damage by only applying changes through a rolling sequential log.

[24] On the scale of many hours, for a large filesystem with many files.

Logging allows us to maintain an accurate view of the filesystem state, so that despite a power failure or other catastrophe we know exactly what state the filesystem is in. Rather than scanning the entire filesystem (via fsck), we can check just the log and correct the disk against the last few entries if necessary. This can mean the difference between 20 seconds and 48 hours for checking filesystem integrity, during which the system is almost certainly not available for use. Unfortunately, logging does not come for free; it requires that a lot of writes be conducted synchronously, and metadata logging requires at least three writes for each file update. If we are concerned about performance alone -- damn the chances of data loss, full speed ahead! -- then logging isn't a good choice. But if we are building a high performance-clustered transaction server, reliable filesystems are probably worth the performance tradeoff.

There are three common ways of implementing logging:

Metadata logging

Only filesystem structure changes are logged; this is the most popular mechanism.

File and metadata logging

All changes to the filesystem are logged.

Log-structured

The entire filesystem is a log.

In a nonlogged filesystem, several disconnected, synchronous writes are used to implement a filesystem change. If an outage occurs in the middle of an operation, the state of the filesystem is unknown, and the entire filesystem must be checked for consistency. Metadata logging is based around a cyclic, append-only log area that it uses to record the state of each transaction. Before any on-disk structures are modified, an "intent-to-change data" record is written to the log. The directory structure is then changed, and a completion record is written to the log. Since every state change involved in processing every transaction is recorded in the log, we can check filesystem consistency by scanning the log rather than checking the entire filesystem. If we find an intent-to-change entry without a corresponding completion entry, we check the appropriate block, and adjust if necessary. Some mechanisms allow this log to reside on a disk separate from the filesystem, and some embed the log in the same partition as the data.

Several filesystems allow you to store data in the log, as well as the metadata. The data is written to the log, and then replayed into the filesystem. This ensures data integrity right up until the last block written, and helps performance for small writes. A small write in a nondatalogging filesystem incurs two seek and writes: the first one for the data itself and the second for the log write. Putting the data in the log reduces the number of steps to one.

Log-structured filesystems (also known as write-anywhere filesystems) implement the entire filesystem structure as a log. In such an environment, writing to the filesystem appends data blocks to the end of the log as data is written, marking earlier blocks invalid as it goes. This approach lets every file be written sequentially, regardless of block order, which makes writes very fast. Unfortunately, it also cripples read performance, since blocks are allocated in the order written -- which may mean that files are completely scattered across the disk. Log-structured filesystems also require a mechanism to scan through the filesystem and remove invalid blocks. It's also necessary to cache the location of the data blocks, since they can be in a random order, and this caching is very complex. These sorts of filesystems are efficient in metadata-intensive environments, but are not as efficient for data-intensive environments. In fact, there are currently no log-structured filesystems that I know of.

5.4.3.1 Solstice:DiskSuite

Sun's Solstice:DiskSuite product provides an enhancement known as the metatrans device, which adds a transaction logging mechanism to the existing filesystem structure. It provides metadata-only logging functionality, and allows the log to be split away from the filesystem. In an environment without any nonvolatile cache memory, this can provide a low-cost (albeit slower) equivalent.

5.4.3.2 Solaris

Starting with Solaris 7, there is a way to implement a logging mechanism without using the DiskSuite product. This mechanism is simple to use, and does not require any changes to the filesystem. Just mount the filesystem with the -o logging option to mount:

# mount -F ufs -o logging /dev/dsk/c1t0d0s0 /mnt

You can have this done at boot-time, as well:

# device          device     mount   fs       fsck  mount    mount

# to-mount        to fsck    point   type     pass  at boot  options

/dev/dsk/c1t0d0s0 /dev/rdsk/c1t0d0s0

                             /mnt    ufs      2     yes      logging

5.4.4 The Second Extended Filesystem (EXT2)

The Second Extended Filesystem, devised by Rmy Card, is used in all shipping Linux distributions as of this writing. It is an indirect block-based filesystem, quite similar to UFS.

You can find the number of free inodes on an ext2fs partition by using df -i:

% df -i

Filesystem            Inodes   IUsed   IFree IUse% Mounted on

/dev/hda1             513024  111953  401071   22% /

The single largest problem with ext2fs is that, by default, it writes metadata asynchronously. This means that power failures and other abrupt, unusual system halt operations often cause significant damage to the filesystem. Confounding this problem is that ext2fs does not support logging; for that, the use of IBM's JFS filesystem, the Third Extended Filesystem (ext3), or the Reiser filesystem, is required.

5.4.5 The Third Extended Filesystem (EXT3)

The Third Extended Filesystem (ext3fs) is a revision of ext2fs -- the most significant revision is support for journaling. It supports journaling of metadata only as well as journaling of file data. It also coexists peaceably with ext2: ext3 filesystems can be mounted as ext2 filesystems (although without journaling support). Furthermore, despite writing some data more than once, ext3 filesystems can exhibit higher throughput than ext2 filesystems, because the journaling process optimizes the motion of the hard drive read/write heads. As can be expected, I focus here on the performance tuning aspects of ext3.

The ext3 filesystem, by journaling all metadata changes, can magnify the effect of access time updates significantly. This can be disabled by mounting the filesystem with the noatime flag in order to disable access time updates.

5.4.5.1 Tuning the elevator algorithm

Most Linux block device drivers use a generic "elevator" algorithm for scheduling block I/O. This algorithm tries to minimize the amount of disk read/write head movement required. The /sbin/elvtune program can be used to trade throughput for latency, and vice versa. There are two key parameters to feed elvtune: the maximum read latency (-r value) and the maximum write latency (-w value). By default, these values are 8,192 and 16,384. For example, /sbin/elvtune -r 4096 -w 8192 /dev/sdc will change the elevator settings for the /dev/sdc device. In general, the write latency should be about twice the read latency. You will need to experiment with good values for these tunables; experience has shown that reducing these numbers by about half tends to be a good starting point for ext3 filesystems. When you've found values that work well for you, the calls to elvtune should be added at the end of the rc.local script; elevator algorithm tuning settings are not preserved across reboots.

In some cases, attempting to tune for maximum throughput at the expense of latency (that is, setting large read and write latencies via elvtune) can adversely affect actual throughput, while still increasing latency. One contributing reason is that the ext3 filesystem has writes scheduled every 5 seconds, as opposed to the 30-second delay associated with ext3. This factor keeps journal transactions from having a more measurable effect.

5.4.5.2 Choosing a journaling mode

The ext3 filesystem supports three different types of journaling. These are set via the data option to the mount call.

The data=journal option

Causes all data (both file data and metadata) to be journaled. This option provides the most assurances of data consistency, but also has the highest performance overhead.

The data=ordered option

Causes only metadata to be logged. File data is written before metadata, which means that all metadata must point at valid data. If the system crashes while appending data to a file located on a data=ordered ext3 filesystem, data blocks may have been written, but the metadata corresponding to the extension of the file will not have been, so the data blocks won't be part of any file. If a write is occurring in the middle of a file (which is much less common than writing at the end of a file; even overwriting a file is done by truncating the file and then overwriting it), however, it is possible for data corruption to occur.

The data=writeback option

Causes only metadata to be logged, with no special handling of file data; most other journaling filesystems work in this mode. The filesystem will be kept consistent, but old data may appear in files after an unclean shutdown. This journaling mode produces speed improvements in certain workloads because of its weakened data consistency assurances. The workloads tend to be dominated by heavy synchronous writes or creating and deleting many small files. If you switch from ext2 to ext3 and see a significant performance drop, the data=writeback option may provide a significant performance boost.

5.4.5.3 Transitioning from ext2 to ext3

Thankfully, it's easy to move from ext2 to ext3: no reformatting is required. Just run tune2fs -j /dev/hda2 (or whatever device should be moved) and change the appropriate 'ext2' line to 'ext3' in /etc/fstab. Transitioning the root filesystem is more difficult; consult the documentation accompanying your Linux distribution for the most current details.

5.4.6 The Reiser Filesystem (ReiserFS)

ReiserFS 3.6.x (which is included as a part of Linux 2.4.x kernels) is a journaling filesystem designed and developed by Hans Reiser and Namesys. The guiding principle behind ReiserFS development was to help create a single, shared environment (a "namespace") where applications could interact. That would allow users to interact with the filesystem directly, rather than build special-purpose layers to run on top of the filesystem (such as databases internal to applications).

Initially, ReiserFS focused on small file performance (less than a few thousand bytes), the area typically lacking in filesystems like ext2 and UFS. ReiserFS is about ten times as fast as ext2 when handling files under 1K in size. This speed is accomplished by the means of a balanced tree system for organizing filesystem metadata. This scheme relieves artificial restrictions on filesystem layout, as well having the pleasant side effect of only allocating inodes when they are needed, rather than creating a fixed set of inodes at the time of filesystem creation.

5.4.6.1 Tail packing

ReiserFS has a special feature called tail packing. ReiserFS calls a file that is smaller than a filesystem block (4 KB) a tail. One of the reasons ReiserFS has very good small file performance is because it is able to incorporate these tails directly into its organizational data structures, so they are close to their metadata. However, tails don't take up much disk space; because they don't fill up their entire filesystem block, they can waste disk space. For example, a 100 byte file will consume an entire 4 KB filesystem data block, for a storage efficiency of about 2.4%. To resolve this problem, ReiserFS does something called tail packing, which allows tails to be stored together in a filesystem block to improve storage efficiency. Unfortunately, tail packing incurs a significant performance hit. If you are willing to sacrifice storage capacity for performance, mounting your filesystem with the notail option will eliminate tail packing and provide a speedup.

ReiserFS filesystems start to slow down when they get more than about 90% full.

ReiserFS, like any new technology, has had some deployment and maturity issues; it historically has not cooperated well with NFS, for example. While ReiserFS is still gaining support, it looks to be a strong and robust filesystem, and is still under development. You can learn more about ReiserFS at http://www.namesys.com.

5.4.7 The Journaled Filesystem (JFS)

The Journaled Filesystem (JFS) is an extent-based design that was originally developed by IBM for use in their AIX operating system; it shares the increased reliability and fast recovery time characteristics with other logging filesystems.

JFS refers to a filesystem as an aggregate, which consists of an array of disk blocks containing a specific format. Each aggregate includes a superblock and an allocation map. The superblock identifies the partition as a JFS aggregate, and the allocation map describes the allocation state of each data block. An aggregate, along with the control structures necessary to define it, is called a fileset, which is mountable. Each aggregate maintains a log that records metadata change information.

IBM has ported JFS to Linux, with patches available for various kernels. The web site for JFS for Linux is http://www.ibm.com/developerworks/oss/jfs/.

5.4.8 The Temporary Filesystem (tmpfs)

Solaris supports a special filesystem type, called tmpfs, that preferentially uses memory rather than disk for storage, as long as physical memory is available. If memory becomes constrained, the files are written to swap space: as a result files on a tmpfs volume will not consume all the physical memory on the system. Unfortunately, this also means that files stored on a tmpfs volume will be lost during a reboot. The performance acceleration observed on such filesystems is not the result of caching -- equivalent caching is accomplished by the normal Solaris mechanism -- but rather from the fact that tmpfs files are never written to disk unless little free memory remains. Therefore, using tmpfs to try and accelerate an application that exhibits read-mostly behavior isn't wise, as the data is already being buffered in memory by the filesystem.

% df -k /tmp

Filesystem            kbytes    used   avail capacity  Mounted on

swap                 1707512  107472 1600040     7%    /tmp

By default, Solaris systems use tmpfs for /tmp, avoiding significant levels of I/O because /tmp is used for compiler temporary files, editor storage, etc. As a result, the traditional practice of creating a separate partition for /tmp tends to not be effective in Solaris. One nice side effect of this is that you can use df to see how much swap space is free.

5.4.9 Veritas VxFS

Veritas Software sells the extents-based VxFS filesystem, which has a number of features that make it well suited for handling large applications. One of the most significant is the ability to optionally use a direct path to I/O that bypasses the in-memory filesystem cache. This is more efficient than the UFS directio option. VxFS is also conducive to quite a few performance optimizations from the point of view of its internal organization. The filesystem also optionally uses a data-logging strategy, as opposed to the metadata-only logging strategy implemented in UFS+. However, in such an environment, the log device must have performance characteristics approaching that of the main storage subsystem, or the log disk will become a bottleneck. Unfortunately, VxFS is complicated and difficult to use correctly. VxFS reflects the complexity of modern storage.

You can access the VxFS documentation online via http://support.veritas.com.

5.4.10 Caching Filesystems (CacheFS)

Accessing a file from disk is less expensive in terms of performance and network traffic than accessing a file that resides on a network-resident volume (e.g., via NFS). CacheFS was introduced to Solaris with 2.3, and is available on many other platforms, including HP-UX and IRIX, for accelerating file accesses to NFS volumes. It requires no support on the NFS server: it is entirely client-based. CacheFS calls the cache directory the frontend, and the directory shared via NFS the backend. CacheFS uses this cache directory to maintain local copies of remotely stored files, in order to avoid incurring the overhead of consulting with the remote file server unless necessary.

The first step is to set up a cache directory with cfsadmin(1M). By default, these files are cached in 64 KB blocks, and only files of up to 3 MB are cached; larger files are left uncached.

cfsadmin is a monolithic application: it creates, deletes, lists, consistency-checks, and updates configuration parameters. Table 5-8 summarizes the available commands.

Table 5-8. Switches to cfsadmin

Switch

Use

-c cache-directory

Creates the specified cache directory

-d cache-id

Deletes the specified cache directory; the cache-id value for a given cache directory is shown in cfsadmin -l

-l cache-directory

Lists the filesystems cached in the specified cache directory

-s mount-point

Performs a consistency check on the cached filesystem if the filesystem was mounted with the demandconst flag

-u cache-directory

Increase parameters; if specified without -o options, resets the parameters to the default if possible

Starting a consistency check examines files as they are accessed, so a cfsadmin -s will not create a flood of consistency checks. The -c and -u switches also take options, specified by -o options, and summarized in Table 5-9.

Table 5-9. Options to cfsadmin

Option

Description

Default

maxblocks

The highest percentage of blocks in the cache directory that the specified cache may use.

90

minblocks

The percentage of blocks in the cache directory that the specified cache can use without limitation.

0

threshblocks

When block usage increases beyond this percentage, the cache cannot claim resources until the block usage has reached the level specified by minblocks.

85

maxfiles

The highest percentage of files in the cache directory that the specified cache may use.

90

minfiles

The percentage of inodes in the cache directory that the specified cache can use without limitation.

0

threshfiles

When inode usage increases beyond this percentage, the cache cannot claim resources until the inode usage has reached the level specified by minfiles.

85

maxfilesize

This option specifies the largest file that the cache is allowed to store, in megabytes. Files larger than this will not be cached.

3

Here are a few examples of how to use cfsadmin. Let's start out by creating a cache with the default parameters and mounting a filesystem against the cache:

# cfsadmin -c /cache

# mount -F cachefs -o ro,backfstype=nfs,cachedir=/cache \  docsun.cso.uiuc.edu:/services/patches /mnt

# df -k /mnt

Filesystem            kbytes    used   avail capacity  Mounted on

/cache/.cfs_mnt_points/docsun.cso.uiuc.edu:_services_patches

                     8187339 2529123 5658216    31%    /mnt

As you can see, there are some specialized options to mount for use with CacheFS. These options are summarized in Table 5-10.

Table 5-10. Options to mount for a CacheFS filesystem

Option

Description

Default

acdirmax

The time, in seconds, for which cached directory attributes are held before being purged from the cache.

30

acdirmin

The time, in seconds, for which cached directory attributes are guaranteed a space in the cache. After this many seconds, CacheFS checks to see if the directory modification time on the backend filesystem has changed; if so, the information is purged and a new copy is fetched.

30

acregmax

The time, in seconds, for which cached file attributes are held after the file is modified before being purged.

30

acregmin

The time, in seconds, for which cached file attributes are guaranteed a space in the cache after the file is modified. After this many seconds, if the file modification time on the backend has changed, the information is purged and a new copy fetched.

30

actimeo

A utility variable. Setting this to a value sets acdirmax, acdirmin, acregmax, and acregmin to that value.

30

backfstype

Specifies the type of the backend filesystem. Generally nfs.

None

backpath

Specifies where the backend filesystem is mounted. If this argument is not given, CacheFS picks a suitable value.

Auto

cachedir

The name of the cache directory.

None

cacheid

A string specifying a particular cache instance. If you do not give one, CacheFS will pick one for you.

Auto

demandconst

When this flag is set, cache consistency is verified only upon request, rather than automatically. This is useful for filesystems that change very rarely. Incompatible with noconst.

Toggle

(Off)

local-access

The front filesystem is used to interpret the permission bits, rather than having the backend verify this information. Setting this may be a security vulnerability.

Toggle

(Off)

noconst

Disables automatic cache consistency checking. Specify only when you know that the backend filesystem will not change. Incompatible with demandconst.

Toggle

(Off)

purge

Throw away any cached data.

No

ro

Mount as read-only. Incompatible with rw.

Toggle

(Off)

rw

Mount as read/write. This is the default behavior. Incompatible with ro.

Toggle

(On)

suid

Allow the execution of suid applications. This is the default behavior. Incompatible with nosuid.

Toggle

(On)

nosuid

Disallow the execution of suid applications. Incompatible with suid.

Toggle

(Off)

write-around

Causes writes to behave as they would under ordinary NFS: writes are made to the backend filesystem, and the cache entry is invalidated. This is the default behavior. Incompatible with non-shared.

Toggle

(On)

non-shared

Causes writes to be made to both the backend filesystem and the cache filesystem. Use this only when you are sure that no one else will be writing to the filesystem. Incompatible with write-around.

Toggle

(Off)

Let's take a look at the cache:

# cfsadmin -l /cache

cfsadmin: list cache FS information

   maxblocks     90%

   minblocks      0%

   threshblocks  85%

   maxfiles      90%

   minfiles       0%

   threshfiles   85%

   maxfilesize    3MB

  docsun.cso.uiuc.edu:_services_patches:_mnt

Looks good. In order to change some of our cache parameters without deleting and recreating the cache directory, we can use cfsadmin -u, once all cached filesystems have been unmounted:

# umount /mnt

# cfsadmin -u -o maxblocks=95,minblocks=65,threshblocks=90,\

  maxfiles=95,minfiles=80,threshfiles=85,maxfilesize=10 /cache

# cfsadmin -l /cache

cfsadmin: list cache FS information

   maxblocks     95%

   minblocks     65%

   threshblocks  90%

   maxfiles      95%

   minfiles      80%

   threshfiles   85%

   maxfilesize   10MB

  docsun.cso.uiuc.edu:_services_patches:_mnt

Now that we're done with our example, let's clean up after ourselves, and remove the mounted filesystem and its cache:

# umount /mnt

# cfsadmin -d docsun.cso.uiuc.edu:_services_patches:_mnt /cache

# cfsadmin -l /cache

To completely remove the filesystem cache, use cfsadmin -d all:

# cfsadmin -d all /cache

You can automatically mount a network filesystem while using CacheFS at boot by adding a line like this in /etc/vfstab:

# device          device     mount   fs       fsck  mount    mount

# to-mount        to fsck    point   type     pass  at boot  options

nfs-server:/apps  /cache     /apps   cachefs  3     yes

ro,backfstype=nfs,cachedir=/cache

When a new read is performed, the request is rounded up to a 64 KB chunk and passed to the NFS server. A copy of the returned data is written to the CacheFS store on local disk. A write to cached data causes an invalidation of the data in the cache; a subsequent read incurs the penalty of reloading the data from the server. This is not actually so bad, since in the short term reads are likely to be satisfied from the in-memory cache.

It is entirely possible that on a lightly loaded 100 Mbit network and a fast NFS server, a cache miss is faster than a cache hit to a heavily loaded local disk! If the local disk is also handling paging activity, it's possible that the activity from paging and from the cache is likely to be synchronized, especially on application startup.

The best filesystems to cache are read-only or read-mostly, and reside across busy, low-bandwidth networks. You can check the cache hit rate with the cachefsstat command, which is available in Solaris releases after 2.4:[25]

[25] If you are running Solaris 2.3 and 2.4, there is no way to obtain this information. An operating system upgrade may be in your future.

# cachefsstat

    /mnt

                 cache hit rate:    90% (54 hits, 6 misses)

             consistency checks:     31 (31 pass, 0 fail)

                       modifies:      0

             garbage collection:      0

This should give you an idea of how effective the caching is in your environment. I have observed hit rates substantially in excess of 95% for read-only application distribution systems.

5.4.10.1 Minimizing seek times by filesystem layout

One recurring theme in this chapter is the necessity to minimize seek time, especially for random access applications. It stands to reason that the more data a cylinder can hold, the less seeking is required. This is accomplished in striping by making logical cylinders that are larger than the physical cylinder size (which is, of course, fixed), but a similar effect can be accomplished even on a single disk. One of the effects of ZBR (see Section 5.1.1 earlier in this chapter) is that the outside cylinders are faster and hold more data. Because all the cylinders are larger, seeks on the outer cylinders are shorter than seeks in the inner cylinders! As a result, using the outside cylinders whenever possible results in fewer seeks.

The fact that long seeks take much longer than short seeks brings us to another important rule: configure disks with a single purpose. Placing two "hot" areas of data access on a disk is a sure way to cause the disk to spend much of its time seeking between them, rather than directly servicing requests.

For historical reasons, many system administrators partition a root drive into four slices, in this order: root, swap, /usr, and /home. This organization is far from optimal. The swap slice is likely to be accessed the most frequently, and /home is likely to be accessed more frequently than /usr. If a seek outside of the current partition occurs, it stands to reason that many times the disk arm will need to seek all the way from swap to /home, completely bypassing /usr. By simply reordering the partitions as root, swap, /home, and /usr, we have reduced the average observed seek time for the disk!

This seems to violate the first principle of performance tuning: we are getting something for free! However, in this case, the cost is intellectual rather than financial -- the expense of achieving this benefit is the effort spent in understanding how disk access works at a fundamental level, and then applying it to the real world.


5.5 Tools for Analysis

Now that we've discussed in some detail how disks operate, how they're connected to the system, and the filesystems that serve as an intermediate layer between those low-level components and us as users, we can move on to tools for analyzing and improving the performance of disk subsystems. In this section, I discuss how to enable disk caches; some tools to benchmark disk performance; a common pitfall in disk performance analysis, and how to avoid it; tools like sar and iostat that let us monitor the performance of our devices; and a low-level disk I/O tracing tool.

5.5.1 Enabling Disk Caches

We've discussed the nature of disk write caching; before you use this procedure, please be sure to read Section 5.1.1.1 earlier in this chapter. Blindly enabling write caches can have a profound affect on the reliability of your data. With that caveat, then, Example 5-1 shows how to invoke format in a mode that enables the changing of the cache policy on a per-disk basis.

Example 5-1. Enabling disk write caching on Solaris
# format -e

Searching for disks...done

AVAILABLE DISK SELECTIONS:

       0. c0t0d0 <SUN2.1G cyl 2733 alt 2 hd 19 sec 80>

          /sbus@1f,0/espdma@e,8400000/esp@e,8800000/sd@0,0

Specify disk (enter its number): 0

...

format> cache

CACHE MENU:

      write_cache - display or modify write cache settings

      read_cache  - display or modify read cache settings

      !<cmd>      - execute <cmd>, then return

      quit

cache> write_cache

WRITE_CACHE MENU:

        display   - display current setting of write cache

        enable    - enable write cache

        disable   - disable write cache

        !<cmd>    - execute <cmd>, then return

        quit

write_cache> display

Write Cache is disabled

write_cache> enable

write_cache> display

Write Cache is enabled

The best tool to use under Linux for changing a disk's cache policy is hdparm, which is available at ftp://metalab.unc.edu/pub/Linux/system/hardware/hdparm-4.1.tar.gz. Here's how to enable write caching for /dev/hda:

# hdparm -W1 /dev/hda

/dev/hda:

 setting drive write-caching to 1 (on)

If you'd rather not play so fast and loose with your data, here's how to turn it off:

# hdparm -W0 /dev/hda

/dev/hda:

 setting drive write-caching to 0 (off)

5.5.2 Disk Performance Benchmarking

After all the techniques that we have discussed for sizing disk subsystems to workloads and improving disk performance, tools to characterize the performance of an I/O subsystem would be most useful. Thankfully, there are several good ones available. Here, I cover hdparm, tiobench, and iozone.

5.5.2.1 hdparm

One nice, compact device for testing raw device performance in Linux is the hdparm command, which includes a few simple tests of sequential data transfer performance. By specifying the -Tt switches, it's possible to get a fairly good estimate of single-disk performance. This is very useful given the large number of widely varying disk drives available in the PC-compatible hardware arena.

# hdparm -Tt /dev/hda

/dev/hda:

 Timing buffer-cache reads:   128 MB in  3.94 seconds = 32.49 MB/sec

 Timing buffered disk reads:  64 MB in 15.31 seconds =  4.18 MB/sec
5.5.2.2 tiobench

tiobench is an open source, threaded disk performance analysis tool that has eclipsed the old bonnie benchmark. It presents a more complete range of data than hdparm, by including sequential and random-access patterns and reporting latency measurements.

Running the benchmark, especially under Linux, is very straightforward. This example was generated on a 300 MHz Celeron system running a 2.4 Linux kernel; the disk device is a 7,200 rpm IDE disk:

% tiotest

Tiotest latency results:

,-------------------------------------------------------------------------.

| Item         | Average latency | Maximum latency | % >2 sec | % >10 sec |

+--------------+-----------------+-----------------+----------+-----------+

| Write        |        1.827 ms |     1421.003 ms |  0.00000 |   0.00000 |

| Random Write |        0.453 ms |      409.878 ms |  0.00000 |   0.00000 |

| Read         |        0.241 ms |      262.779 ms |  0.00000 |   0.00000 |

| Random Read  |        0.087 ms |      107.313 ms |  0.00000 |   0.00000 |

|--------------+-----------------+-----------------+----------+-----------|

| Total        |        0.819 ms |     1421.003 ms |  0.00000 |   0.00000 |

`--------------+-----------------+-----------------+----------+-----------'

Tiotest results for 4 concurrent io threads:

,----------------------------------------------------------------------.

| Item                  | Time     | Rate         | Usr CPU  | Sys CPU |

+-----------------------+----------+--------------+----------+---------+

| Write          40 MBs |   14.1 s |   2.834 MB/s |   1.1 %  |  19.5 % |

| Random Write   16 MBs |   10.2 s |   1.533 MB/s |   0.2 %  |  15.3 % |

| Read           40 MBs |    1.1 s |  37.924 MB/s |   3.8 %  |  19.0 % |

| Random Read    16 MBs |    0.5 s |  32.338 MB/s |   4.1 %  |  14.5 % |

`----------------------------------------------------------------------'

Further options are available by perusing the output of tiotest -h. Of particular interest are the -S and -W flags. -S forces all writes to be done synchronously, and -W causes the writes to be done sequentially. tiobench also comes with a Perl script that allows you to automate the running of test suites.

As of this writing, tiobench is available at http://sourceforge.net/projects/tiobench/.

5.5.2.3 iozone

The iozone benchmark is quite possibly one of the most complete disk and filesystem benchmarks ever developed. Unfortunately, it is extremely complex (a side effect of its completeness), and beyond the scope of discussion here. Extensive documentation is available online, however. iozone is available as of this writing at http://www.iozone.org.

5.5.3 Second-Time-Through Improvements?

Let's look at a set of simple throughput benchmarks:

% mkfile -v 10m test-file

test-file 10485760 bytes

% ptime cat test-file > /tmp/glop

real         1.728

user         0.004

sys          0.273

% rm /tmp/glop

% ptime cat test-file > /tmp/glop

real         0.504

user         0.004

sys          0.211

The astute administrator might ask why the second run is so much faster than the first. The simple reason is that this is a completely bogus measurement. What occurred is the first copy legitimately copied the data from disk to temporary space, which on this system is a tmpfs filesystem.[26] However, the second run did not -- it copied the file from the in-memory filesystem cache to temporary space. When you run a benchmark, be careful that you are actually measuring disk performance, rather than in-memory filesystem cache performance. You can ensure the validity of your measurements by unmounting and remounting the filesystem in between trials.

[26] As such, the data was never written to disk. This benchmark measured the time required to read the file from disk and to store it into memory.

5.5.4 Using iostat

One of the most useful tools for monitoring ongoing disk performance is iostat. Unfortunately, iostat can be confusing; it presents output in many different forms, and in some cases it computes numbers strangely for historical reasons.

The most useful flags for the Solaris version of iostat are probably -x (for extended statistics), -n (to print names in the much more descriptive cXtXdXsX format),[27] and -P (for per-partition statistics). Other interesting flags are -C, which causes aggregation by controller ID; -m, which reports the filesystem mount points; and -z, which suppresses the output of any lines that are all zeroes (e.g., idle disks).

[27] The cXtXdXsX format lets you easily relate the pathname to a partition to how the partition is attached to the system. The number following c is the controller instance; the number following t is the device target number; the number following d is the device instance (LUN); and the number following s is the slice (partition) number. For example, c1t5d0s3 would be partition three (s3) on the disk with ID 5 (d5) attached to controller 1 (c1). IDE disks do not have a d value, and are represented as cXtXsX.

% iostat -mnPxz 30

...

                    extended device statistics

    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device

    0.0    0.2    0.0    0.8  0.0  0.0    0.0    9.5   0   0 c0t0d0s3 (/)

                    extended device statistics

    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device

    0.0    1.4    0.0   10.6  0.0  0.0    0.0   13.0   0   1 c0t0d0s3 (/)

                    extended device statistics

    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device

    0.0    2.6    0.0   22.4  0.0  0.0    0.0   12.1   0   1 c0t0d0s3 (/)

                    extended device statistics

    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device

    3.2    0.8   22.2    6.4  0.0  0.0    0.0    3.9   0   2 c0t0d0s3 (/)

                    extended device statistics

    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device

    0.2    0.2    1.4    1.6  0.0  0.0    0.0    9.1   0   0 c0t0d0s3 (/)

    0.0    2.2    0.0  206.4  0.0  0.0    0.0   21.7   0   1 c0t1d0s0 (/export/home)

The fields are as follows:

  • r/s and kr/s report the number of read operations per second and the number of kilobytes read per second. w/s and kw/s report the same information, but for writes.

  • wait is the average number of transactions waiting to be serviced.

  • actv is the number of requests currently being actively serviced.

  • wsvc_t is the average service time in the wait queue, in milliseconds.

  • asvc_t is the average service time for active transactions, in milliseconds.

  • %w represents the percent of time that there are transactions waiting to be serviced.

  • %b represents the percent of time that the disk was actively servicing transactions.

If you are using the Linux version of iostat, the flags to use are -d -x interval. This enables extended output and only displays disk information (rather than adding CPU statistics):

$ iostat -d -x 30

Linux 2.4.2-2 (aiua)      07/29/2001

Device: rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s avgrq-sz avgqu-sz  await svctm  %util

hde       2.88  1.32  52.90  2.23  99.06   28.50     2.31     0.65  11.82  8.34  4.60

hde1      0.00  0.00  49.60  0.00  49.61    0.00     1.00     0.28   5.64  5.64  2.80

hde2      2.88  1.32   3.30  2.23  49.44   28.50    14.10     0.37  67.26 33.29  1.84

hde5      0.00  0.00   0.00  0.00   0.01    0.00     8.00     0.00 100.00  100.00  0.00

hdg       5.89  1.61  34.27  3.27 233.97   39.41     7.28     5.17 137.62 31.69 11.90

hdg1      0.00  0.00  12.47  0.00  12.47    0.00     1.00     0.05   4.21  4.21  0.53

hdg2      5.89  1.61  21.80  3.27 221.50   39.41    10.40     5.11 203.98 45.35 11.37

You can optionally provide a specific device before the time interval to only see statistics for that device.

5.5.4.1 Historical limitations: iostat and queuing terminology

In the old days (when disk interfaces such as IPI dominated the marketplace), disk controllers really did directly control the disks. In such an environment, terms like utilization, service time, and wait time have clearly defined meanings, and some simple queuing theory mathematics can let you derive them from a handful of underlying measurements. iostat was originally developed in this era. Nowdays, the disk controller is not nearly as omnipresent in the disk subsystem: modern disks are extremely smart. The rather simple model implemented by iostat, and particularly its use of terminology, can be very confusing in modern environments.

In the old controller-centric environments, once the disk driver sent a request to the disk, it knew that the disk would single-mindedly service that request, and the length of time it required was the service time. With modern systems, the service time varies with the disk load and queue length, and there is no direct comparison to the older service time measurement. We can work around this by explicitly creating two queues: a wait queue, which refers to commands queued in the disk driver, and an active queue, which refers to commands queued in the disk itself. Unfortunately, iostat tries to report the new measurements with the old terminology; the "wait service time" is actually the time spent in the "wait" queue, which really doesn't make sense.

The number that iostat calls "service time" is defined as the queue length divided by the throughput. This is not really service time, which is more properly the amount of time it takes for the first command in line to be processed; what iostat calls "service time" would be better labeled formally as the "response time."

5.5.5 Using sar

The -d flag to sar causes it to report various per-partition disk activity statistics:

% sar -d 5 100

SunOS islington 5.8 Generic_108528-03 sun4u                     02/16/01

14:58:51   device        %busy   avque   r+w/s  blks/s  avwait  avserv

...

14:59:06   fd0               0     0.0       0       0     0.0     0.0

           nfs1              0     0.0       0       0     0.0     0.0

           sd0              93     2.6      81   17934     0.0    31.9

           sd0,a             0     0.0       0       0     0.0     0.0

           sd0,c             0     0.0       0       0     0.0     0.0

           sd0,d            93     2.6      81   17934     0.0    31.9

           sd1               0     0.0       0       0     0.0     0.0

           sd1,a             0     0.0       0       0     0.0     0.0

           sd1,c             0     0.0       0       0     0.0     0.0

           sd6               0     0.0       0       0     0.0     0.0

...

The first column, %busy, refers to the portion of the time that the drive was busy servicing a transfer request in the sampling interval; avque gives the average number of outstanding requests during the same period. The number of reads, writes, and bytes transferred (in 512-byte units) is given by read/s, write/s, and blks/s. avwait specifies the average wait time, in milliseconds; avserv refers to the average service time, also in milliseconds.

For more information on sar in general, please see Section 4.5.2.2.

5.5.6 I/O Tracing

It would be wonderful if we could break down iostat output even further -- for example, if we observe a high utilization rate on one partition, it would be very useful to know what process was responsible for this behavior. But to accomplish this, we must use a more complicated tool: the kernel trace facility.

Unix systems have had kernel tracing for a long time, but it was originally for development use only; enabling the facility in production kernels wasn't generally done because it provided only a small performance overhead. IBM decided to leave the kernel tracing facility enabled in their AIX kernel, and Sun has since followed suit.

The trace capability is rather robust and has some nice features; for example, the trace file is self-describing, so an external definitions file isn't required. It consists of three applications: prex(1), which controls probe execution; tnfxtract(1), which reads out the kernel trace buffer; and tnfdump(2), which displays the data in a human-readable format. However, it does have some problems. For instance, user-level probes can directly write to trace files, but the kernel probes are written to a ring buffer, which is implemented as a separate buffer per kernel thread to prevent having to worry about locking and contention issues on multiprocessor systems. The problem is that it's hard to size the buffer, and one highly active probe might refill its buffer repeatedly while others are just getting started. Thankfully, tnfxtract takes a snapshot of the buffer, so a second snapshot will include leftover data; tnfdump also is very good about sorting things into chronological order. While this facility is extremely powerful, we are only going to discuss its utility in regards to tracing I/O.

5.5.6.1 Using the kernel probes

When the prex command is invoked, it goes into an interactive command mode, where you can specify the details of the trace you wish to perform. Here, we invoke it with the -k switch, so that it attaches to the kernel's internal probes:

# prex -k

Type "help" for help ...

prex> buffer alloc 2m

Buffer of size 2097152 bytes allocated

prex> enable io

prex> trace io

prex> ktrace on

These commands allocate a 2 MB buffer for data storage[28] and enable the io probe set for tracing.[29] The ktrace on command flips the kernel probe "master switch," which controls when data is gathered.

[28] You do not need to explicitly declare the size of the buffer -- a simple buffer alloc will suffice.

[29] If you feel daring, you can enable all the kernel probes. This can be accomplished by issuing the command enable $all followed by trace $all. Be advised that this is very verbose, and probably not for the faint of heart.

At this point, wait for a little while until the behavior you're interested in occurs, and shut off the master switch:

prex> ktrace off

Then you can dump the trace records into a file and review them:

# mkdir /tmp/trace

# cd /tmp/trace

# tnfxtract iotrace.tnf

# tnfdump iotrace.tnf | more

The resulting output can be divided into two sections: a header and contents. Both are very wide, and have been edited unmercifully to fit on the page. The header is illustrated in Example 5-2.

Example 5-2. Header of tnfdump kernel trace output
probe tnf_name: "strategy" tnf_string: "... blockio;file .../driver.c;line 305;"

probe tnf_name: "pageout" tnf_string: "... pageio io;file .../vm_pvn.c;line 552;"

probe tnf_name: "biodone" tnf_string: "... blockio;file .../bio.c;line 1189;"

The output header details the probes that were enabled for the trace. The strategy probe records an I/O being issued, and biodone records an I/O being completed. The other mechanism that may cause I/O is the paging mechanism, recorded via the pageout probe, which is why it is included in the io probe set. The header also describes what probe sets a probe resides in (such as io and blockio), as well as the source code file and line where the probe actually resides.

The contents of the trace follow the format indicated in Example 5-3, again highly truncated.

Example 5-3. Contents of tnfdump kernel trace output
Elapsed  Delta  PID LWPID TID CPU Probe   Data / Description . . .

 0.000   0.000 2040     1 ... 1 strategy ...  buf: 0x707b45a0 flags: 4194313

16.911  16.911    0     0 ... 0 biodone  ...  buf: 0x707b45a0

17.027   0.116 2040     1 ... 1 strategy ...  buf: 0x706d8410 flags: 524569

26.101   9.074    0     0 ... 0 biodone  ...  buf: 0x706d8410

26.106   0.005    0     0 ... 0 biodone  ...  buf: 0x706d8410

26.158   0.051 2040     1 ... 1 pageout  ...

The next section, the trace data, shows exactly what I/O requests were handled, in chronological order. It does not show I/O requests that were serviced by means of the in-memory filesystem cache. The first column shows the time elapsed (in ms) since the first probe occured; the second represents the amount of time (in ms) spent performing the operation. The next three columns refer to the process ID, lightweight process ID, and thread ID of the generator of the I/O request. The CPU column describes what processor handled the request; this trace was taken on a dual-processor Sun Ultra 2, which explains why you see a mixture of CPU 0 and CPU 1. The final column describes the data generated by the probe. As you can imagine, tracking down exactly what a probe represents can be nontrivial. It helps to use tracing on a process that is being run through truss(1). I have found using truss -lDdaf command to be most useful.

Note that we have translated the flags field from binary to hexadecimal. The flags field is composed of logical additions of flags. You can get the canonical list from /usr/include/sys/buf.h; a reference list of some of the more useful ones is presented in Table 5-11.

Table 5-11. I/O tracing flags

Flag

Name

Description

0x000001

BUSY

Request busy (see WANTED).

0x000002

DONE

Transaction finished.

0x000004

ERROR

Transaction aborted.

0x000008

KERNBUF

Accessing a kernel buffer.

0x000010

PAGEIO

Do I/O to pages.

0x000040

READ

Request is a read.

0x000080

WANTED

Wake up when the BUSY flag goes off.

0x000100

WRITE

Request is a write.

0x000400

ASYNC

Asynchronous; don't wait for I/O completion.

0x002000

DONTNEED

Don't bother caching after a write.

0x008000

FREE

Free the page when done with it.

0x010000

INVAL

Request contains bogus data.

0x080000

NOCACHE

Don't cache the block when it's released.

0x400000

RETRYWRI

Retry the write until it succeeds.

In this case, we have observed all the I/O requests for a short-lived process. I'll rewrite the raw data shown in Example 5-3 into a little bit clearer format, as illustrated in Table 5-12.

Table 5-12. Actual operations visible in tntdump output

Buffer

Time (ms)

Operation

Flags field contents

0x707b45a0

0.000

I/O request issued.

0x400009

16.911

I/O request complete.

0x706d8410

17.027

I/O request issued.

0x080119

26.102

I/O request complete.

Examining the flags, the first I/O is an 8 KB request to a kernel buffer, which took 16.911 ms to complete, and the second is for a noncached 1 KB write to a kernel-buffered I/O page, which took 9.074 ms to complete. The first request is an update to the inode, and the second is a data write.

The method we've just described tracks all the I/O activity for the system. This can give you some interesting results, but it can also be a real pain. If you are interested in tracing the I/O done by a single process on a busy system, rather than looking at the behavior of the entire system, that can be done as well. In fact, there is a method to accomplish just that.

5.5.6.2 Using process filtering

prex implements a means of focusing tracing called process filtering. When process filtering is enabled, tracing in the kernel is limited to events initiated on behalf of the processes in a specific list. Process 0 is used to represent all the threads not associated with a process.

When it is run without arguments, the prex command pfilter displays the current process filtering mode (on or off) and the process ID list. The process ID list can be modified regardless of the state of process filtering.

A process must exist to be added to the process ID list; similarly, the list is automatically updated to delete any processes that no longer exist. Should the list become empty while process filtering is turned on, prex issues a warning. Process filtering mode is persistent between invocations of prex, so be sure to turn it off manually when you're done.

Let's work through an example, where we trace all the I/O generated by a vi session (process ID 19786):

# prex -k

Type "help" for help ...

prex> buffer alloc

Buffer of size 393216 bytes allocated

prex> pfilter add 19786

prex> pfilter on

prex> pfilter

Process filtering is on

Process filter set is {19786}

prex> trace io

prex> enable io

prex> ktrace on

...in a separate window, exit the  vi session ( :wq)...

prex> ktrace off

Warning: Process filtering on, but pid filter list is empty

prex> pfilter off

prex> quit

# tnfxtract /tmp/vi.trc

# tnfdump /tmp/vi.trc

...

Note that we received a warning from prex when the vi process exited and could no longer be traced.

5.5.6.3 Restarting prex

If you restart a kernel trace and it is only of short duration, you will see old information before you get to what you actually care about. This problem can be easily solved by unallocating and reallocating the buffer:

prex> buffer dealloc

prex> buffer alloc

These examples just scratch the surface of prex. It is a complex tool that serves many purposes. If you're interested in exploring further, there are other meaningful probe sets. One of these is vm, which contains all the virtual memory-related probes.


5.6 Concluding Thoughts

We've discussed most of the major issues surrounding I/O performance tuning: the basic concepts of disk drives, such as physical layout, access patterns, and the parameters that govern disk performance; the interfaces that communicate with those disks; various types of filesystems; and the tools that monitor the performance of our disks and I/O subsystems.

In many ways, disks occupy a unique pedestal in the field of computer architecture; they are the only portion of a system whose operation relies upon mechanical parameters and moving parts, and they are the last resort for online information storage. This realm is without the glamour of microprocessor design, where every cycle counts, and it is not one of the most fundamental parts of modern computer design, like memory. Perhaps for these reasons, it is one of the areas most commonly disregarded by systems administrators when approaching performance tuning problems.

Disk tuning is like a dark lake in the middle of a forest, filled with all manner of strange and wonderful things; it doesn't seem to hold much beauty, and it can even be a bit foreboding. But there are pearls of great performance worth just a few feet below the surface of that lake -- seize the chance to find a few.


Chapter 6. Disk Arrays

I/O certainly has been lagging in the last decade.

—Seymour Cray, 1976

Most of the improvements in disk technology have been made with the aim of increasing the capacity/price ratio. Although these changes have made mass storage much more affordable, they have brought about two fairly serious problems:

  • When a single disk can store tens of gigabytes of data, the reliability of an individual disk becomes a serious concern, as the failure of an individual disk results in the loss of large amounts of data.

  • Disk performance has lagged drastically behind capacity/price improvements.

To solve these problems, a great deal of effort has been put into designing methods of organizing sets of disks to enhance both reliability and performance. This has come to be known as RAID, which either stands for "Redundant Array of Inexpensive Disks" or "Redundant Array of Independent Disks," depending on who you listen to. There are seven levels of RAID; each takes a different approach to solving these problems. The types of RAID are summarized in Table 6-1.

Table 6-1. A summary of RAID levels

RAID level

Organization

Strengths

Weaknesses

RAID 0

Striping

Very fast, simple

Low reliability

RAID 1

Mirroring

Fast, simple

Expensive

RAID 2

Reliability viaHamming codes

Reliable

Inflexible

RAID 3

Parity

Fast sequential access, reliable

Slow random access, implementation difficult

RAID 4

Parity

Average performance, reliable

Bottlenecks on dedicated parity disk

RAID 5

Parity

Good performance, reliable, cheap

Slow writes, heavy cache requirements

RAID 10

Striped mirrors

Very fast, simple

Expensive

There is a fundamental tradeoff in configuring disk arrays -- in fact, it is the classical example of the one of the principles of performance tuning (see Section 1.2.2). The tradeoff is stated in the following note.

Fast, cheap, safe.

Of these three attributes, you may pick only two.

In this chapter, I discuss some of the fundamentals of assembling multiple disks into a single logical unit: the basic terminology used, the differences between software and hardware disk arrays, recipes for desiging disk arrays, and many other aspects of modern array implementation.


6.1 Terminology

I need to introduce some terminology before we can continue to discuss RAID:

  • Since a disk array involves multiple physical disks, I use the term logical volume to distinguish a volume composed of multiple disks.

  • The Mean Time Between Failure (MTBF) refers to the average time between failures of a component. For example, the MTBF of a given disk drive might be specified at 500,000 hours, meaning that the typical disk of that model will run for half a million hours before failing. Since this is nothing more than a statistical measure, an individual disk drive might well fail fifty (or one million) hours into service. These component-based measurements can be applied to the whole system; for example, a 9 TB array composed of one thousand 9 GB disks with a MTBF of 1.2 million hours each would be expected to have an aggregate MTBF of 1,200 hours (about fifty days).

  • The Mean Time To Data Loss (MTTDL) gives the amount of time that a system can operate before it suffers a failure sufficient to cause data loss. If all the components operate independently, the MTTDL is equal to the MTBF of the least reliable component; however, designers go to great lengths to ensure that the MTTDL is increased despite constant MTBF numbers for components. For example, a mirrored pair of disks won't suffer data loss unless both drives fail, which is much less likely than one disk failing. It is also critical to note that a high MTTDL does not imply continuous availability: a host controller failure might render a mirrored disk pair inaccessible, but the data could still be completely intact.

  • The Mean Time To Data Inaccessibility (MTTDI) is not formally defined; however, it is increasingly important to be able to always access your data, not just protect it from loss. The MTTDI gives the amount of time that a system can operate before it suffers a failure sufficient to make data unavailable.

  • While we won't discuss it here, the Mean Time To Repair (MTTR) is often an important metric, and certainly something that should be monitored. As we'll see, the loss of a disk with some disk array schemes can result in degraded performance and susceptibility to data loss; knowing how long it takes your system to recover from a failed disk (including the time spent to physically replace the failed disk) can be quite useful.


6.2 RAID Levels

RAID is defined in several levels, which describe different mechanisms for achieving increased performance and reliability. These are numbered starting at zero and going up to five (note that we do not count RAID 10; that is a combination of two already defined RAID levels). The first level of RAID, RAID 0, provides purely performance improvements, without increasing reliability at all, whereas RAID 1 mostly provides reliability improvement. RAID 2 uses a complex algebraic method to provide data reliability, and the other levels use parity-protected mechanisms to provide increased protection as well as performance improvements.

The heart of parity-protected RAID systems (levels 3, 4, and 5) is a mathematically simple, reversible parity computation. Every commercial implementation of parity-protected RAID uses some variation on this algorithm. It involves taking the data in an entire stripe width and applying the bitwise exclusive-or function (XOR) to generate a parity bit:

The XOR function has the properity of being reversible; if it is applied twice, the original data is regenerated. When any member disk in the system fails, the data can be computed by looking at the data on the surviving members:

These volumes can sustain the failure of any one disk, and the reliability of a parity-protected array with n member disks can be given as:

It is an extremely common misconception that parity-based RAID systems use a read-and-compare mechanism for reads. This is not generally the case.[1] There are robust error detection mechanisms at the disk and protocol level, and the overhead of such a read-and-compare system would be substantial.

[1] Some very paranoid fault-tolerant applications implement data checksumming at the device driver level.

Reliability can be increased by adding hot spare volumes in some RAID configurations, because in the event of one disk failing, a spare is immediately available (significantly reducing the MTTR). This reduces the likelihood, statistically, of data loss. Unfortunately, single-channel parity-protected arrays can suffer problems with controller failure, which appears to the host system as a failure of multiple disk drives. Disk arrays capable of being configured with redundant controllers are not subject to this problem.

6.2.1 RAID 0: Striping

In the first level of RAID, a number of physical disk drives are broken into stripes and addressed as a single logical unit. Each disk drive is divided into units called chunks, so that successive chunk-sized logical blocks are stored on consecutive disks. The chunk size is often called the interlace or stripe unit size. The stripe width refers to the chunk size multiplied by the number of disks in the array. Figure 6-1 shows how RAID 0 organizes data on a set of physical disks.

Figure 6-1. RAID 0 disk organization

This organizational scheme means that, in many cases, the request is serviced by accessing several disks. Since sequential performance is dominated by the time required to transfer data from the platters, performance is optimal for requests that are the same size as the stripe width. This has the added advantage of distributing disk load evenly across all disks, reducing the utilization of each device in the array and improving random-access performance; recall that random-access performance is a function of seek time, and the less seeking we have to do, the faster we can fulfill the next request. Clearly, RAID 0 arrays universally improve disk subsystem performance.

It is a common myth that striping only improves sequential performance. This is not true. Random access performance is dramatically improved because, in an array with N drives, each disk's utilization decreases by about 1/N. The lower utilization yields better I/O performance.

Given a four-disk array with a 32 KB interlace, a 128 KB data request will be satisfied by reading four consecutive blocks, which will each come from a separate physical disk. The sequential performance of striped logical volumes approaches the aggregate bandwidth of each of the member disks times the number of disks in the array, given that there are limitations for no operating system, SCSI host adapters, array controller, etc. In this example, if each drive has an internal transfer speed of 6 MB/second, the array should be able to sustain about 24 MB/second. Random access performance is not substantially improved in single-threaded operation, because the governing factors for performance are the rotational and seek times of the member disks. In multi-threaded sequential operation, the various requests arrive at the member disks in a random order, and so the array exhibits an essentially random-access pattern.

Performance generally does not improve linearly in RAID 0 arrays, since the member disks are not operated in spindle synchronization: each disk spins and seeks individually. We'll discuss spindle synchronization in more detail later, when we talk about RAID 3 arrays. In addition, RAID 0 devices do not improve performance when hot spots are accessed far more frequently than the rest of the volume, which causes an uneven I/O distribution on the member disks. One example is a named pipe that is written continuously, in which the near-continuous update to the last-modified time on the pipe causes a definite hot spot. One sneaky way around this problem is setting the sticky bit (accomplished by means of chmod +t filename) on the pipe, which causes the system to defer updates to the entry's directory entry. This is slightly risky, because in the event of a system crash, the last modify date may be invalid; however, neither the data sent over the pipe nor the integrity of the filesystem are put at risk.

The worst-case performance for a RAID 0 array occurs when the request size is very small and crosses a chunk boundary. Performance of a stripe will almost always be less than that of a single disk when the request size is 1 KB and crosses a chunk boundary; two drives are activated, and latency goes up while throughput is not improved.

Choosing the stripe unit size is a somewhat complicated decision, and one that can have a substantial impact on performance. In random-access environments, performance is optimized by involving as many disks as possible. Since only one or two disks are involved in any given request, disk utilization is minimized when the number of disks is maximized. In such a case, the chunk size should be rather large -- about 128 KB. This step sounds counterintuitive, but the intent is to have as many drives active as possible, each satisfying a different request. Performance in sequential environments is optimized when the size of the I/O requests is equal to the size of the stripe width, and the chunk size is large enough to make the I/O overhead worthwhile.

The performance benefits granted by RAID 0 do not come without cost. The striping across disks means that the integrity of the logical volume is dependent on every member of the disk. The instant a single disk fails, the entire volume becomes unusable, and nothing but replacing the failed component and restoring from backup media can regenerate the volume.

6.2.2 RAID 1: Mirroring

In order to combat the reliability problems associated with RAID 0 arrays and individual disks, RAID 1 was developed. This technique, often called mirroring or disk shadowing, relies on the fact that individual disks are quite reliable.[2] However, in a large installation, where a disk farm might contain 1,500 disks with a 1,000,000 hour MTBF for each disk, the system will suffer a disk failure about every 28 days. This is obviously not sufficient reliability for the datacenter. The core principle behind RAID 1 is to arrange information so that a single disk failure does not incur data loss. Under a mirroring scheme, at least one extra disk is reserved for each data disk; these disks are called submirrors. When data is written, it is written separately to the main data disk and every reserved disk; when data is read, it can be retrieved from any of the disks. If a member disk fails, the data is recovered from the surviving submirrors. Figure 6-2 illustrates RAID 1 disk organization.

[2] A typical MTBF for a modern disk is about 1,000,000 hours, or roughly 114 years, 56 days.

Figure 6-2. RAID 1 disk organization

Generally, two-way mirroring is sufficient, but three-way mirrors are useful in some cases, such as providing snapshot data backups without compromising the reliability of the on-line volume. While some implementations allow even more extensive mirroring, it is overkill. Mirroring makes a huge difference in reliability: a two-way mirror of disks with a 1,000,000 hour MTBF per disk has a MTTDL of 5.0 1011 hours![3]

[3] Assuming that disks fail independently. A fire in a disk cabinet could destroy all the drives, though, and a fire is significantly more likely than data loss from independent failures.

The performance of a mirrored device is mixed. Writes must be committed to all members, so even best-case write performance is slightly decreased in comparison to a single disk. Most mirroring mechanisms provide two policies for managing mirrored writes:

Parallel member write policy

Involves dispatching the write requests serially, but these requests are serviced in parallel. Since the dispatch process is much shorter than the physical write time, the writes to the mirror happen essentially in parallel. This tends to give rise to write performance about 80% of that of a single disk for a two-way mirror and 65% of that of a single disk for a three-way mirror; the degradation is due to seek and rotation delays.

Serial member write policy

Involves dispatching the first write requests, waiting for it to complete, then issuing the second write request. This causes writes to an N-way mirror to take N times as long when compared to single-disk performance.

Read performance of a mirror is substantially different. In terms of single-threaded performance (both random and sequential), the performance is about the same as that of a single member disk. An extremely paranoid implementation might read the data from all member disks and compare, but this is something that is only done in completely fault-tolerant systems. The best case for performance is random-access reads, where the system can pick the least busy disk to satisfy the request, and disk utilization for an N-way mirror is reduced by 1/N.

In the event of a mirrored disk failure, the disk must eventually be replaced. The process of bringing a replacement disk up to date is called resynchronization. The most straightforward way to accomplish this update is simply copying all the bits from the surviving disks to the replacement disk. This is exactly what must be done when a full resynchronization is required. Resychronizations are usually performed at a reduced speed (about 1 MB/second/disk) to minimize the impact of the resynchronization on system operation. However, a full resynchronization is not always required; sometimes a submirror is taken offline temporarily -- for example, to conduct a snapshot backup -- in which case only the changed blocks need to be copied. This process is usually accomplished through a dirty region log (DRL), in which a bitmap is provided for each disk. Each bit defines whether a disk region has been changed since the submirror was taken offline. A typical region is 32 KB in size; as a result, the DRL is quite small and its location has a minimal impact on performance.

6.2.3 RAID 2: Hamming Code Arrays

One common mechanism for implementing data corruption detection is a parity code, where the number of ones in a given length of data are counted; the parity bit is 1 if the number of ones is odd and 0 if it is even. A single parity bit can detect an error if any one bit is incorrect, but does not tell which data bit is incorrect. This 1-bit parity scheme is formally called a distance-2 code; if we look at the data bits, no 1-bit change will result in an identical parity bit being computed. Of course, if we change two bits (either any two data bits or one data bit and the parity bit), the parity matches the data and no error will be detected: there is a distance of two changes between legal combinations of parity and data. If we are interested in detecting more than one error, or in correcting an error, we need a distance-3 code, in which there is a distance of three changes between legal data/parity combinations.[4] This distance will let us correct errors of one bit, and detect but not correct errors of two bits. The number of bits required for a distance-3 code of a given length of data grows slowly compared to the number of bits of data. For example, 64 bits of data require seven parity bits, and 128 bits of data require eight parity bits. This type of encoding is called a Hamming code, after R. Hamming, who described a method of synthesizing this sort of code.

[4] This is the difference between "parity" and "error-correcting codes" or ECC. Parity protection lets you detect data corruption; ECC lets you repair it as well.

RAID 2 uses this algebraic method to generate the appropriate parity information that allows for data reconstruction in the event of disk failure. However, RAID 2 was never widely deployed due to the complexity of the Hamming code generation algorithms that needed to be implemented in hardware, coupled with restrictions that the algorithms placed on the number and organization of the disks in the array.

6.2.4 RAID 3: Parity-Protected Striping

RAID 3 attempts to avoid the problem of poor reliability associated with RAID 0 and the high cost associated with RAID 1. RAID 3 builds upon the idea of striping, but uses an extra disk drive to store a computed parity block. The system computes a parity bit for each bit in the stripe width, and stores the parity data on the corresponding location of the extra disk. It turns out that RAID 3 is extremely difficult to implement technically, because the interlace ranges in size from 1 to 32 bits, and commodity SCSI disks use 512-byte blocks. Most vendors claiming "RAID 3" functionality are actually using RAID 5 arrangements, but with all parity data stored in a single disk.

Because RAID 3 is based on the RAID 0 structure, it has a lot of similar benefits: very good performance with operations of sufficient size, and poor performance with small operations (those that only span two disks). While the parity computation itself does not impose a significant (>5%) performance penalty, all writes must involve the parity disk. In read activity, in which parity is not involved, the system performs about as well as RAID 0. The same is true in single-threaded write environments. However, multi-threaded writes start to cause serious problems, because the disk used for parity becomes overloaded; multi-threaded writes to a RAID 3 volume are very similar to that of a single disk. In some environments, this is acceptable. For most multipurpose servers, RAID 3 is not a good choice.

RAID 3 is aimed toward optimizing requests that involve all the devices in the array. Since each request to a RAID 3 volume must wait for all the physical subsidiary requests to complete, forcing all the member disks to be in exactly the same position reduces latency problems. This is called spindle synchronization, and works very well for optimizing sequential single-threaded requests. It is a nightmare for random-access performance: it behaves as one very large, very safe disk drive, with the random request rate for the volume the same as the request rate for a given member disk. For all these reasons, RAID 3 is essentially never used.

6.2.5 RAID 4: Parity-Protected Striping with Independent Disks

RAID 4 is an expansion on RAID 3; instead of using small chunk sizes, it picks sizes closer to those found in RAID 0 (between 2 and 128 KB). It also does not require the member drives to be spindle synchronized, making RAID 4 much easier to implement using commodity SCSI disks; virtually all devices advertised as "RAID 3" are actually implemented as RAID 4 devices for exactly this reason.

RAID 4 has essentially the same strengths and weaknesses as RAID 3; it excels at sequential throughput and reads, but write performance is as slow as a single member disk.

6.2.6 RAID 5: Distributed, Parity-Protected Striping

The weakness of the RAID 3 design philosophy is that it does not adequately solve the problems posed by multitasking environments. RAID 5 addresses this problem by intermixing parity data with actual data in the stripe width, rather than dedicating a single disk or pieces of multiple disks to parity data (see Figure 6-3).

Figure 6-3. RAID 5 array

One of the core truths in all parity-based RAID organizations is that they convert logical writes into multiple physical writes. In order to maintain data integrity, these writes must be committed atomically -- otherwise, the computed parity data and the actual data could get out of synchronization, corrupting data. The most common solution to this problem is implementing a two-phase commit process. When a write request is issued, a parity block is computed and written to a buffer area. A log is updated, indicating that a write is pending against the data. Once the data is safely placed in nonvolatile storage, the array management software writes the updated data and parity onto the member disks, and updates the log to reflect a clean status for those blocks. If any part of this process is interrupted, the array management software can restore the volume to a consistent state. This log is written to frequently, but pending writes tend to stay around for only a few milliseconds, so the log is very short.

RAID 5 performance is widely misunderstood, particularly for writes. Let's look at the simpler read case first. A single-threaded read to a RAID 5 array is almost as fast as a stripe with one fewer disk; for example, a six-disk RAID 5 array is as fast as a five-disk RAID 0 array. This is because one disk equivalent is held over for parity data in RAID 5. Maximum performance is achieved when the request is aligned on a stripe boundary and the request size is an integer multiple of the data width of the stripe. Multithreaded reads (that is, multiple concurrent reads) from RAID 5 volumes are faster than those on RAID 0 volumes; the additional disk involved in the RAID 5 array distributes the utilization over more drives. As with RAID 0 and RAID 3, performance is at a minimum when the request size is very small and spans two member disks -- this involves activating three disks (two data disk and a parity disk).

Unfortunately, write performance is complicated. The first problem is that the write typically involves a read-modify-write cycle. The rest of the data in the stripe (the parity context) must be read in order to compute the parity data. The new data is then inserted and the data and parity are written back to disk. This gives rise to three situations:

  • The write changes the entire stripe width. The old parity context isn't read, since we're about to write over it anyway. Due to the parity write, the percentage of performance relative to a single stripe is given by:

  • The write changes more than one data disk, but less than the entire stripe width. We must recover the parity context, which forces a read-write-modify cycle: we incur two reads, two data writes, and a parity write. Performance is about 25% of a single disk.

  • The write changes exactly one data disk. The parity context doesn't need to be recovered, since we can compute it just as if the disk had failed. The write involves one write and one parity write. Performance is about 50% of a single disk.

Unfortunately, this situation is even more complex in real life. Because of the necessity for safe two-phase writes, we incur three additional writes: two to update dirty region logs and one to write the updated data and parity. Even worse, the log and data writes must be written in a precise sequential order.

RAID 5 is complex, and deciding precisely how to configure such an array can be difficult. The most important strategy is to try and arrange for as many full-stripe writes as possible. If your application is dominated by a particular write transaction, design the array to fit those sizes. This is best accomplished by setting the data width of the volume to be equal to the typical I/O size. If the typical I/O size is 128 KB, and the RAID 5 array has six disks, the chunk size should be 26 KB:

RAID 5's real weakness is write performance, especially on small files. This becomes painfully apparent if you are using a RAID 5 volume that is incurring many writes over NFS V2, because of the small I/O sizes allowed by NFS V2. We'll talk about this in greater detail in Section 7.5.

When the array has suffered a failure, the performance characteristics change substantially. Under normal operating characteristics, not all disks need to be read, and no reads ever involve a parity computation. When a disk has failed, however, the only way to retrieve data is to read all the member disks and use the parity mechanism (described in Section 6.2 earlier in this chapter) to determine the missing piece of data. When a RAID 5 array is running in a degraded state, a single disk read requires 2 (n-1)/n reads, as opposed to just n reads in fully operational mode, and per-disk utilization approximately doubles. The good news is that write performance is not significantly affected.

6.2.7 RAID 10: Mirrored Striping

Simple RAID 1 arrays are now quite unusual, because storage requirements have outgrown single physical disk sizes; as a result, it is now commonplace to create mirrors whose submirrors are stripes. These are often called RAID 10, RAID 1+0, or RAID 0+1 arrays. Figure 6-4 shows RAID 1+0 organization.

Figure 6-4. RAID 1+0 organization

The read performance of a RAID 10 array benefits from both organizations, although random reads benefit only from mirroring when they are multi-threaded. Writes to RAID 10 arrays are typically about 30% slower than those to purely striped volumes, due to the overhead of writing two copies.

The order of operations in a RAID 10 array is actually quite important. There are two mechanisms of building such a device with four disks:

  • Construct two stripes of two drives each, then join them to construct a mirror, giving you one mirror whose submirrors happen to be stripes. This is a RAID 0+1 organization (see Figure 6-5).

  • Construct two two-way mirrors, then stripe them, giving you a stripe with components that happen to be mirrors. This is a RAID 1+0 organization (see Figure 6-6).

Figure 6-5. RAID 0+1
Figure 6-6. RAID 1+0

This may seem to be an academic consideration, but the two design philosophies have substantially different behavior when a disk fails. When a RAID 0+1 array suffers a disk failure, an entire submirror is taken offline. Any subsequent failure compromises the entire array. In the RAID 1+0 array, a single disk failure degrades only that component of the stripe: an N-device array can suffer at worst 2 failures, and at best N/2 failures, before data is lost. Unfortunately, vendors are often less than clear as to which sort of design they actually implement. You'll need to check with your vendor to be sure.

Solstice DiskSuite is notable in that it forces you to create stripes, then mirror the stripes (RAID 0+1); although this seems wrong, it does actually implement the correct (RAID 1+0) scheme.


6.3 Software Versus Hardware

RAID arrays are commonly implemented by host software or by a dedicated hardware array controller. Of course, the hardware RAID vendors will tell you that their hardware controllers are better, and software companies will tell you that their software products are better; although they accomplish the same function, they do have some substantial differences.

6.3.1 Software

In software-based array management, the underlying physical disks are visible to the administrator, and the RAID functions are implemented via a virtual device driver. The bonus is flexibility: the RAID functionality can use any available drive, regardless of packaging or location. Arbitrary location of disks can be useful if you are trying to build an array that is resistant to systematic faults (e.g., fire or flood). One of the common criticisms of such a design is that it is slower than controller-based RAID, especially for the parity computations involved in RAID 5. While it is true that software-based arrays tend to be slower than hardware-based arrays, it is not a function of parity computational performance; a typical array controller is equivalent to a 66 MHz Intel 486 processor, which can perform eight parity computations in about fifteen cycles. A modern high-performance workstation microprocessor can complete two parity computations in every clock cycle, and has a clock rate eight times higher! In addition, the load imposed on the system for the parity computation is trivial.

6.3.2 Hardware

In a hardware-based array, the array management is handled by a dedicated processor. This has a substantial effect in reducing host overhead, because a software array implementation has to manage two sets of I/O operations: the logical operation and the operations on the member disks. The member disk operations can be quite substantial, since each SCSI I/O requires a nontrivial amount of processor time. However, the most significant reason for using controller-based RAID implementations is that they have an architectural advantage for using nonvolatile memory to cache the RAID 5 write log. The controller can commit the logical write into memory and acknowledge it to the host before it is committed to disk, providing a huge latency advantage. They also perform internal optimizations that get around the slow parts of RAID 5, such as coalescing read-write-modify operations into full-stripe operations. This provides a substantial improvement in cases where the system is dominated by sequential RAID 5 writes.

The benefit of NVRAM caches in controller-based arrays is to accelerate writes. Don't bother attempting to accelerate reads by putting lots of NVRAM in a controller-based array.

If you are not using RAID 5, the benefit of controller-based NVRAM caching is to enable fast writes; the controller can write data into NVRAM, acknowledge the write, and complete the write at its leisure. For such an application, 16 MB of cache should be plenty. For RAID 5, you may need 64-128 MB in order to completely cache stripes effectively.

6.3.2.1 RAID overlap

While controller-based RAID implementations are superior for RAID 5 writes, they are less valuable for other RAID levels, especially since the host processor overhead for running RAID 0 or RAID 1 is very small. It is possible to combine host and controller-based arrays, to great effect. For example, taking a set of hardware RAID 5 arrays and striping them together with a software RAID 0 implementation gives good data reliability (from the RAID 5 level) and excellent read performance (from the RAID 0 overlap).


6.4 A Summary of Disk Array Design

Disk array design is fundamentally a matter of creating a list of the most critical attributes for the environment at hand. As a result, understanding the characteristics of the environment is absolutely necessary. The following questions drive our understanding:

  • Is the workload primarily read-driven, write-driven, or balanced?

  • What sort of access pattern is generated? Is it primarily random or primarily sequential?

  • What sort of resiliency to failure is required? How many disk failures must be tolerated? How sensitive is the environment to degraded performance in the event of a disk failure?

Think carefully about these issues as you design complex disk subsystems.

6.4.1 Choosing a RAID Level

The hallmark of RAID design is simple and important. Even though we've already said it, it bears restating.

Fast, cheap, safe.

Of these three attributes, you may pick only two.

One fairly common approach to picking a RAID level is to pick the two most important attributes in order. Then we can build the selection map shown in Figure 6-7.

Figure 6-7. RAID roadmap
    Previous Section < Free Open Study > Next Section


6.5 Software RAID Implementations

Many people snub their noses at software RAID implementations, citing concerns about performance and reliability. In general, I have found these considerations to be somewhat overstated. While software RAID solutions are often slightly idiosyncratic in their implementation and require some care in deployment, they are generally not as slow as their reputation indicates, particularly for non-parity-based disk arrays. Their chief shortcoming is probably their inability to provide caching at the device level. We'll briefly discuss the two most popular software array products, Solstice DiskSuite and the Linux md package.

6.5.1 Solaris: Solstice DiskSuite

Solstice DiskSuite revolves around the idea of metadevices. A metadevice is a group of physical slices that are addressed by the system as a single, logical device. These metadevices are treated just like physical devices. The metadevices begin with the letter d. The block metadevices reside in /dev/md/dsk (e.g., /dev/md/dsk/d0) and the raw metadevices reside in /dev/md/rdsk (e.g., /dev/md/rdsk/d0). By default, the system allows 128 metadevices, but the limit of the software is 1024.[5] One thing I have found to be helpful is to use the metadevice name to encode the physical device location. For example, under such a scheme, the disk with ID 4 on controller 1 would be metadevice d14. The component I/O requests to and from physical devices are handled by the metadisk driver. Because these metadevices appear as physical devices, you will still need to use newfs to create a filesystem after setting them up.

[5] Increasing the number of metadevices involves changing the nmd parameter in /kernel/drv/md.conf.

SDS has two different ways of accomplishing the same tasks. There is a graphical interface, invoked by metatool, and there are command-line utilities, which I focus on here. However, SDS is an extremely complex product: what we discuss here is only the essential heart of SDS configuration. Sun publishes two excellent technical references on this product, the Solstice [image] DiskSuite [image] 4.2 Reference Guide and the Solstice [image] DiskSuite [image] 4.2 User's Guide.

6.5.1.1 State databases

SDS stores its configuration information in a state database, also called a metainformation database. Because this information is integral to the proper functioning of the product, multiple copies (replicas) are stored on disk; when the data is written, each copy is updated in sequence. At least three, and at most fifty replicas must exist. By default, these replicas are 517 KB each.

In order to ensure data integrity, SDS implements a majority consensus algorithm for the replicas, in which a majority (half, rounded down, plus one) of the replicas must be available for the system to function. If this criteria is not met, the database is considered stale, and functionality fails:

  • The system will function normally if at least half of the replicas are active.

  • The system will panic if less than half of the replicas are available.

  • The system will not reboot without a majority of the replicas available.

As a result, the replicas should be spread across as many disks and controllers as possible, in order to limit the chances that a single component will fail and cause the entire database to become stale.

Replicas typically reside on either dedicated partitions or on slices that are parts of metadevices.[6] I have always put them on slice 3 or slice 7, but there is no technical reason for this. As a general rule, the total number of replicas should be determined by Table 6-2.

[6] If you place a replica on a slice that is being used for a metadevice, you must add the replica to the slice before you add the slice to the metadevice.

Table 6-2. Choosing how many replicas to configure

Number of disks

Number of replicas

Placement

1

3

All on a single slice

2-4

4-8

2 per disk

More than 5

5+

1 per disk

The state database replicas are managed through the metadb command. This command is rather complex, and its behavior is summarized in Table 6-3.

Table 6-3. Summary of metadb usage

Command

Description

metadb

Lists all configured replicas

metadb -i

Lists only inactive replicas

metadb -a -f slice

Creates new replicas

metadb -a slice

Creates additional replicas

metadb -d -f slice

Removes a failed replica

In addition, you may specify -c number-of-copies with both of the commands to add more than one replica to the slice. If the default size for each replica is too small, you may specify the -l size option to metadb to specify a larger size.

Using metadb without arguments reports the condition of all replicas:

# metadb

        flags           first blk       block count

     a m  pc luo        16              1034            /dev/dsk/c0t0d0s7

     a    pc luo        1050            1034            /dev/dsk/c0t0d0s7

     a    pc luo        16              1034            /dev/dsk/c0t8d0s7

     a    pc luo        1050            1034            /dev/dsk/c0t8d0s7

Flags that are in lowercase letters imply an "Okay" status; flags that are in uppercase letters are indicative of a problem. The possible flags are summed up in Table 6-4.

Table 6-4. metadb status flags

Flag

Description

o

Replica active prior to the last configuration change

u

Replica is up to date

l

Replica locator was read successfully

c

Replica's location was in /etc/opt/SUNWmd/mddb.cf

p

Replica's location was in /etc/system

a

Replica is active

m

Replica is the master replica

W

Replica has write errors

R

Replica has read errors

M

Replica has problems with master blocks

D

Replica has problems with data blocks

F

Replica has format problems

S

Replica is too small to hold the current state database

You can easily find inactive database replicas by using metadb -i. If the state database is stale (usually by means of disk failure), you must repair the problem. You can accomplish this by using metadb -i to report the inactive replicas, noting how many copies are on each slice, and then by using metadb -d -f replica-slice to remove the inactive replicas. Now, reboot the system and replace the failed disk; when the system comes back up, read the replicas with metadb -a -c number-of-copies replica-slice.

In order to initially create the state database, run metadb with the -a and -f switches. If you have less than five disks, you will want to create more than one copy per slice by specifying -c number-of-copies. This is done to maintain quorum in the event of a disk failure. You can't create a state database on a partition that has an existing filesystem.

Here's an example of initially creating state database replicas on a system with three disks, and two copies per slice:

# metadb -a -f -c 2 c0t0d0s7 c0t1d0s7 c0t2d0s7

# metadb

        flags           first blk       block count

     a m  pc luo        16              1034            /dev/dsk/c0t0d0s7

     a    pc luo        1050            1034            /dev/dsk/c0t0d0s7

     a    pc luo        16              1034            /dev/dsk/c0t1d0s7

     a    pc luo        1050            1034            /dev/dsk/c0t1d0s7

     a    pc luo        16              1034            /dev/dsk/c0t2d0s7

     a    pc luo        1050            1034            /dev/dsk/c0t2d0s7

You may want to add additional state database replicas after the initial creation, by leaving out the -f switch. The general format of the command is metadb -a -c number-of-copies replica-slice.

Once you have the state database set up, you can start working with metadevices.

6.5.1.2 RAID 0: stripes

You can only use a stripe to hold a filesystem that is not used at either boot time or during an upgrade/install process, e.g., /, swap, /usr, /var, or /opt. A prerequisite to creating a stripe is the existence of at least three stable state database replicas. Creating a stripe involves using the metainit command:

# metainit d0 1 2 c0t1d0s0 c0t2d0s0 -i 64k

d0: Concat/Stripe is setup

This creates a metadevice d0, which is a single (1) stripe of two (2) slices, namely c0t1d0s0 and c0t2d0s0. The optional -i flag sets the interlace size to 64 KB; by default, the interlace is set to 16 KB. Here's an example of creating a four-device stripe:

# metainit d1 1 4 c0t1t0s0 c0t2d0s0 c0t3d0s0 c0t4d0s0

d1: Concat/Stripe is setup

Because there was no specified interlace value, this array uses a 16 KB interlace.

One interesting use of a stripe is the creation of metadevices that consist of only one physical device. This is called encapsulation, and is accomplished by the following:

# metainit d2 1 1 c1t0d0s0

d1: Concat/Stripe is setup

Encapsulation is useful for mirroring, as we'll discuss later.

6.5.1.3 RAID 1: mirrors

A mirror consists of between one and three submirrors. A submirror consists of one or more striped metadevices -- thus the utility of encapsulations, since we can create a single-device stripe as a submirror. You can use a mirror (in DiskSuite lingo, a metamirror) for any filesystem, and, like a stripe, require at least three state database replicas to exist prior to creation. Unlike a stripe, you can use a mirror for any filesystem.

There are four cases in which you might want to create a mirror:

  • From scratch (unused slices)

  • From a filesystem that can be unmounted

  • From a filesystem that cannot be unmounted

  • From the root filesystem

Each involves a slightly different procedure.

If you are creating a mirror from scratch, you can follow this procedure, which has three high-level steps. First, create the two encapsulations with metainit; second, use metainit to configure the metamirror; third, use metattach to add the second submirror to the metamirror:

# metainit d10 1 1 c1t0d0s0

d10: Concat/Stripe is setup

# metainit d11 1 1 c1t1d0s0

d11: Concat/Stripe is setup

# metainit d50 -m d10

d50: Mirror is setup

# metattach d50 d11

d50: Submirror d11 is attached

When the metattach command is run, it attaches the d11 metadevice to the d50 metamirror, which creates a two-way mirror and causes a mirror resynchronization. Any data on the newly attached submirror is overwritten by the master submirror during this resynchronization. You can watch the progress of the resynchronization via metastat.

If you are creating a mirror for an existing filesystem (say, /data on c2t0d0s0) and do not wish to back up and restore the filesystem, you can perform this operation with a much reduced delay. The basic idea is to create two encapsulations, create the mirror, unmount the filesystem, cause the system to refer to the mirror device rather than the physical device, mount the filesystem, and attach the second submirror:

# metainit -f d20 1 1 c2t0d0s0

d20: Concat/Stripe is setup

# metainit d21 1 1 c2t1d0s0

d21: Concat/Stripe is setup

# metainit d51 -m d20

d51: Mirror is setup

# umount /data

(Edit /etc/vfstab to reflect /data being mounted as

/dev/md/dsk/md51 rather than /dev/dsk/c2t0d0s0)

# mount /data

# metattach d51 d21

d51: Submirror d21 is attached

The -f switch to metainit causes the metadevice to be created nondestructively over the existing mounted filesystem. When the metattach command is executed, the contents of the d21 metadevice are destroyed in the resulting resynchronization operation.

Sometimes a filesystem cannot be unmounted during regular system operation (for example, /usr on c3t0d0s0). The procedure for mirroring such a filesystem is essentially the same as mirroring an unmountable filesystem, except that we will reboot rather than unmount and remount the filesystem:

# metainit -f d30 1 1 c3t0d0s0

d30: Concat/Stripe is setup

# metainit d31 1 1 c3t1d0s0

d31: Concat/Stripe is setup

# metainit d52 -m d30

d52: Mirror is setup

 (Edit /etc/vfstab to reflect /data being mounted as

/dev/md/dsk/md51 rather than /dev/dsk/c2t0d0s0)

# reboot

...

# metattach d52 d31

d52: Submirror d31 is attached

Mirroring the root filesystem (on c4t0d0s0 in this example), incurs some particular complications. Specifically, we need to use the specialized metaroot command, as well as flushing all pending, logged transactions against all filesystems prior to reboot:[7]

[7] This procedure will only work on Solaris/SPARC systems. If you are running Solaris/i86pc, please consult the Sun documentation for a similar procedure.

# metainit -f d40 1 1 c4t0d0s0

d40: Concat/Stripe is setup

# metainit d41 1 1 c4t1d0s0

d41: Concat/Stripe is setup

# metainit d53 -m d40

d53: Mirror is setup

# metaroot d53

# lockfs -fa

# reboot

...

# metattach d53 d41

d53: Submirror d41 is attached
6.5.1.4 RAID 5 arrays

There are a few things to keep in mind when creating RAID 5 metadevices:

  • The metadevice must consist of at least three slices.

  • The interlace value is critical to good RAID 5 performance; see Section 6.2.6 earlier in this chapter for details.

  • The more slices a RAID 5 metadevice contains, the longer operations will take when a slice has failed.

  • RAID 5 metadevices cannot be striped or mirrored within SDS.

The good news is that it is very simple to create RAID 5 metadevices:

# metainit d60 -r c5t0d0s0 c5t1d0s0 c5t2d0s0 c5t3d0s0 c5t4d0s0

d60: RAID is setup

Because no interlace size was specified with the -i switch, this array has an interlace size of 16 KB. The metainit command then starts an initialization process, which can be monitored via metastat. You must wait for this to finish before using the new metadevice.

6.5.1.5 Hot spare pools

A hot spare pool is a collection of slices that are reserved for use as automatic substitutes in the event of a submirror or RAID 5 component failure. A hot spare device must consist of slices, not metadevices. Although a hot spare slice can be associated with one or more hot spare pools, and a hot spare pool can be shared between multiple submirrors and RAID 5 arrays, each submirror or RAID 5 device can only be associated with one hot spare pool. In addition, a hot spare slice cannot contain a state database replica.

Configuring a hot spare pool for use involves two steps: creating the hot spare pool and associating metadevices with the hot spare pool:

# metainit hsp001 c6t0d0s0 c6t1d0s0

hsp001: Hotspare pool is set up

Once the hot spare pool is created, you can associate a metadevice with it:

# metaparam -h hsp001 d10

# metaparam -h hsp001 d11

You may wish to add additional slices to a given hot spare pool after the intial configuration; this is accomplished by using the metahs command:

# metahs -a hsp001 c6t2d0s0

If you wish to add a slice to every hot spare pool in the system, specify -all instead of a hot spare pool name.

6.5.2 Linux: md

Linux supports software disk arrays by means of the multiple device (md) driver package. We are discussing the newer version of the RAID software, which became integrated into the 2.4 production kernels. If you are interested in using this RAID variant with 2.0 or 2.2 kernels, you should fetch a patch for your kernel, as those kernels do not have direct support for the RAID software I describe here. The Linux md software supports linear mode (or strict concatenation of disks), as well as RAID 0, 1, 5, and 0+1.

Before we start our discussion of the RAID levels supported and how to implement them, let's briefly talk about two underlying principles: persistent superblock and chunk size.

6.5.2.1 Persistent superblocks

In older versions of the Linux md support, the software RAID toolkit would read your /etc/raidtab configuration file and initialize the array. However, this requires that the filesystem with /etc/raidtab be mounted, which is not good if you intended to boot from disk. The solution to this problem is the persistent superblock. When an array is initialized with the persistent-superblock option in the /etc/raidtab file, a special superblock is written to the beginning of all the disks participating in the array. This lets the kernel read the RAID device configuration directly from the disks involved, instead of reading from a configuration file that might not be available.

6.5.2.2 Chunk size

The notion of a chunk size is analogous to the DiskSuite concept of the interlace size; it is the smallest "atomic" mass that can be written to a device. For example, a RAID 0 device with two disks and a chunk size of 4 KB that incurs a 16 KB data write will have the first and third 4 KB chunks written to the first disk in the array, and the second and fourth 4 KB chunks written to the second disk. The chunk size must be specified for all RAID levels, including linear mode, and is specified in kilobytes. Table 6-5 is a precise description of what the chunk size specifies for each type of disk array.

Table 6-5. Descriptions of chunk size effect on disk arrays

RAID level

Effect

Linear

None whatsoever.

RAID 0

Chunk-sized blocks are written to each disk, serially. For example, a write of 24 KB to an array of two disks with a chunk size of 8 KB will mean that 8 KB is written to both disks, in parallel, and then the remaining 8 KB is written to the first disk.

RAID 1

For writes, the chunk size is not important, since all data must be written to all disks. In this case, the chunk size specifies how much data will be read from the participating disks; since all active disks contain the same data, this lets us parallelize reads in a manner akin to a striped array.

RAID 5

RAID 5 writes must also update parity information. The chunk size is the size of the parity blocks: if one byte is written from a RAID 4 array, then chunk-size blocks will be read from all of the data disks, the parity information calculated, and a chunk-sized block of parity data written to the parity disk.

Good values for the chunk size vary from 32 to 128 KB, depending on the application.

All disk arrays are configured by means of entries in /etc/raidtab. Let's walk through some examples of how to write the configuration file entry for each RAID type, then discuss how to create the device itself.

6.5.2.3 Linear mode

A linear mode device is simply a concatenation of existing devices. It doesn't support any of the advanced RAID features we've discussed, but can be useful if your application calls for it. An example /etc/raidtab entry for a linear array is shown in Example 6-1.

Example 6-1. /etc/raidtab entry for a linear array
raiddev /dev/md0

   raid-level               linear

   nr-raid-disks            2

   chunk-size               32

   persistent-superblock    1

   device                   /dev/sdb1

   raid-disk                0

   device                   /dev/sdc1

   raid-disk                1

Since this is the first time we've seen a sample configuration, let's go through it step-by-step:

  • The first line defines the device we're creating, in this case /dev/md0.

  • The raid-level line describes the RAID level of this particular array device.

  • The nr-raid-disks line defines how many disks participate in the array.

  • The chunk-size parameter sets the chunk size, in KB.

  • The persistent-superblock line acts as a Boolean toggle for whether a persistent superblock should be created. It should always be set to 1.

  • The next four lines are device/RAID disk number pairs. They must always exist in pairs. The device line is a path to a disk partition, whereas raid-disk is the number of that disk in the array as a whole.

6.5.2.4 RAID 0: stripes

Stripes are configured essentially identically to linear-mode arrays:

raiddev /dev/md0

   raid-level               0

   nr-raid-disks            2

   chunk-size               32

   persistent-superblock    1

   device                   /dev/sdb1

   raid-disk                0

   device                   /dev/sdc1

   raid-disk                1
6.5.2.5 RAID 1: mirrors

The configuration file entry for a mirrored array is quite similar to that for a stripe or a linear mode array, but with a slight twist:

raiddev /dev/md0

   raid-level               1

   nr-raid-disks            2

   nr-spare-disks           1

   chunk-size               32

   persistent-superblock    1

   device                   /dev/sdb1

   raid-disk                0

   device                   /dev/sdc1

   raid-disk                1

   device                   /dev/sdd1

   spare-disk               0

There are two additional, closely related parameters here (I've highlighted the lines they're on):

  • The spare-disk line follows a device line, and specifies that the named disk is to be used as a hot spare. If one disk in the mirror fails, data will be reconstructed onto the spare disk.

  • The nr-spare-disks line specifies how many spare disks are attached to this array.

6.5.2.6 RAID 5 arrays

You must have at least three disks to build a RAID 5 array. The /etc/raidtab entry looks like this:

raiddev /dev/md0

   raid-level               5

   nr-raid-disks            7

   nr-spare-disks           0

   chunk-size               128

   persistent-superblock    1

   parity-algorithm         left-symmetric

   device                   /dev/sdb1

   raid-disk                0

   device                   /dev/sdc1

   raid-disk                1

   device                   /dev/sdd1

   raid-disk                2

   device                   /dev/sde1

   raid-disk                3

   device                   /dev/sdf1

   raid-disk                4

   device                   /dev/sdg1

   raid-disk                5

   device                   /dev/sdh1

   raid-disk                6

There is one new specification for a RAID 5 array, the parity-algorithm line. Leave it set to left-symmetric. You can add spare disks if you wish.

6.5.2.7 Creating the array

Once you've written the /etc/raidtab file, you can create the array by running mkraid device on the array device. For example, to create any of the arrays I have described, you would use mkraid /dev/md0. Disk arrays that require a complex intialization process (RAID 1 and 5) will start to resynchronize; this has no impact on the system's functionality whatsoever (you can mke2fs the new filesystem, work with it, etc.).You can get information on the state of any RAID device by means of the /proc/mdstat file. Once you have your array device working properly, you use the raidstop and raidstart commands to control it.

There are two important parameters to pass to mke2fs when you construct your filesystem. The first is the block size in bytes, specified by -b. This value should be at least 4,096 (4 KB). The second, which is only available for RAID 5 devices, is -R stride=N. This allows mke2fs to better place data structures on the device. For example, if the chunk size is 64 KB, it means that 64 KB of contiguous data will reside on one disk. If we want to build a filesystem with a 4 KB block size, we realize that there will be 16 filesystem blocks in 1 array chunk. We can then pass this information onto the mke2fs utility:

# mke2fs -b 4096 -R stride=16 /dev/md0

RAID 5 performance is heavily influenced by this option.

6.5.2.8 Autodetection

Autodetection allows the kernel to automatically detect and start up RAID devices on system boot. In order for autodetection to work, you must meet three criteria:

  1. Autodetection support must be compiled into the kernel.

  2. The RAID devices must have the persistent superblock enabled.

  3. The partition types of the RAID partitions must be set to 0xfd.

If all of these criteria are met, you can simply reboot the system -- autodetection should work normally (cat /proc/mdstat to be sure). Autostarted devices are automatically stopped at shutdown. You can now use /dev/md devices just as you would any other devices.

6.5.2.9 Booting from an array device

If you are interested in booting from an array device, there are a few things that you have to take care of. The first is that the kernel used to boot must have the RAID kernel module available to it. There are two ways that you can do this: either compile a new kernel that has the RAID modules linked in directly (not built as a loadable module), or instruct LILO to use a RAM disk that contains all the kernel modules necessary to mount the root partition. This is done by using the mkinitrd -- with=module ramdisk-name kernel command. For example, if your root partition resides on a RAID 5 device, you would be able to accomplish this by running mkinitrd -- with=raid5 raid5-ramdisk 2.4.5.

There is another problem; namely, that LILO 0.21 (the current distribution) does not understand how to deal with a RAID device. As a result, the kernel cannot be read from a RAID device, and your /boot filesystem will have to reside on a nonRAID device. A patch to LILO exists, included with Redhat 6.1, that can handle booting from a kernel located on a RAID 1 array only. This patch can be obtained from dist/redhat-6.1/SRPMS/SRPMS/lilo-0.21-10.src.rpm on any RedHat mirror. The patched version of LILO will accept boot=/dev/md0 in lilo.conf and will make each disk in the mirror bootable.

You must now set up a root filesystem on a RAID device. This method assumes that you will have a spare disk that you can install the system on, which isn't part of the RAID you will configure. Once you have the system installed, RAID support working, and the RAID device created via the normal procedures we've discussed and persisting properly through reboot, you can run mke2fs to create a filesystem on the new array device, and mount it (say to /newroot). You now have to copy the contents of your current root filesystem (on the spare disk) to the new root filesystem (the array). One good way to do this is via find and cpio:

# cd /

# find . -xdev | cpio -pm /newroot

Now edit the /newroot/etc/fstab file to supply the correct boot device.

The last step is to update LILO. Unmount the current /boot filesystem, and mount the boot device on /newroot/boot. Modify /newroot/etc/lilo.conf to point to the correct devices. Note that the boot device must still be a regular (nonRAID) device, but the root device should point to your new RAID. When you're finished, run lilo -r /newroot. You can then reboot your system, which will boot from the new RAID device.


6.6 RAID Recipes

I will provide four "disk array recipes" here, targeted toward the most common deployments. They are not meant to be silver bullets, but rather are intended to provide a starting point for your own efforts. We will then go over a case study to illustrate the principles we've discussed.

6.6.1 Attribute-Intensive Home Directories

Most general purpose home directory activity fits into this category.

This array pattern exhibits almost entirely random activity. When files are small, access to data is mostly determined by retrieval of directory entires, inode data, and the like; they require a significant amount of seeking. Generally, activity is almost entirely reads; writes are relatively rare.

Because of the random workload, bus utilization is probably not going to be a significant issue. Your goal is to provide as many disk drives as possible: the strategy should tend towards large numbers of small disks on a moderate number of controllers. Try for about 10 fully active disks per 40 MB/second SCSI chain.

If your application mix avoids high throughput situations (writes in particular), then RAID 5 is an excellent choice. If you require faster writes and can absorb the extra cost, RAID 0+1 is preferable.

Some filesystem tuning is going to be required. In general, adjust the free space to 1%, and configure the filesystem to use 16 KB per inode. Tuning the cluster size will also be necessary (see Section 5.4.2.2).

If the home directories are to be shared via NFS, write performance becomes more of an issue, and NVRAM or other nonvolatile storage is an extremely good idea. A ballpark figure is to configure three drives per concurrently active NFS client. Logging is also an excellent idea; if possible, use Veritas or Solstice DiskSuite's metatrans logging feature over the integrated filesystem logging mechanisms.

6.6.2 Data-Intensive Home Directories

Data-intensive home directories present a different problem; they are more likely to have a sequential workload that's not so heavily biased toward read operations.

In this case, the approach I've found to be fruitful is to stripe as many of the fastest drives (i.e., "highest internal transfer rate") as possible across a fairly large number of controllers. Try not to configure more than four disks per controller.

One significant problem is that multiple clients accessing a single disk can create a significant bottleneck. For this reason, it is probably a good idea to create multiple small disk arrays (say, one per two populated disk controllers -- use RAID 0+1 if possible, but RAID-5 is also acceptable if you have significant amounts of NVRAM for write caching) and concatenate these devices. Try to configure one RAID device for every three to five active clients.

Logging presents a more significant tradeoff here. If you do use log disks, try creating a small RAID 0+1 volume, and using that as the log device. If the workspaces are to be accessed via NFS, recall that each client connected via Fast Ethernet can demand up to 10 MB/second of activity.

6.6.3 High Performance Computing

The high performance computing environment is generally characterized by a roughly even mix of reads and writes,[8] sequential access patterns, and a desire for performance above all else -- the data can be easily regenerated.

[8] It's application-dependent, but most scientific applications I've worked with either are almost entirely read-based, or about even between writes and reads.

The core idea is to transfer I/O loads that require high performance from home directories to a separate "workspace" area. In general, this disk subsystem should be configured with a great number of fast disk controllers (40 MB/second), each with two or three of the fastest disks you can find. The key thing is to not overpopulate controllers with disks: the sequential workload can easily overload the bus.

Filesystem throttling due to the UFS write throttles (see Section 5.1.4.1) is something to watch out for, as is throttling due to the virtual memory subsystem. Upgrade to Solaris 8 if possible, and turn on priority paging if not. Much of this is not tunable in Linux, but try to keep the kernel revision current, in order to take advantage of ongoing virtual memory subsystem advances.

Logging may well present a significant performance problem. Since this data is likely to be regenerated quickly, and a reboot will necessitate restarting the computational job, the typical approach is to create as many RAID 0 volumes as possible, aiming for about six disks per array, and concatenate them.

6.6.4 Databases

Configuring disk arrays for databases opens up a whole new realm of specifications. Read operations may be done in small, random blocks, such as index lookups, or they may be done in larger, sequential chunks, such as when an entire table is scanned. Writes are normally small and synchronous. Oracle, for example, defaults to using a 2 KB block size. This keeps the disk service time low for small, random operations, but many blocks may be requested simultaneously for large table reads.

Configuring a fairly large amount of nonvolatile memory in the RAID controller is an excellent idea for two reasons; first, the writes are generally synchronous and in the "critical path" for user responsiveness; and second, large writes often occur as a long stream of small writes, which are amenable to coalescing.

Virtual memory contention can be a serious problem. Using the directio option for UFS filesystems (see Section 5.4.2.6) is a good idea, since most database software caches heavily internally. As usual, keeping up to date with Solaris and Linux core software releases is an excellent idea. Enabling priority paging or migrating to Solaris 8 will be very useful.

While data integrity is paramount, some parts of the database can be regenerated on the fly. For this reason, it may make sense to split the database into two parts:

  • Temporary tablespaces and indexes can reside on large, unsafe, fast stretches of disk, implemented as concatenated RAID 0 arrays. Aim for six to eight fast disks per 40 MB/second controller.

  • Areas where reliability is paramount are usually read-mostly. This can be safely done via RAID 5 volumes, or by RAID 0+1 if cost is a secondary concern to performance.

6.6.5 Case Study: Applications Doing Large I/O

Let's consider a transaction processing scenario. This application workload consists of a few processes taking in information over the network, performing some in-memory work on them, and then pushing out 16 KB blocks of data to disk. The users are complaining of poor throughput from the disk subsystem.

In this case, we would expect to see iostat -xtc 1 indicating sustained disk activity, and we know that the applications are doing large disk operations (in this case, 16 KB). In order to maximize disk throughput, we want to send as big a chunk of data as possible to each disk. Since a disk can typically sustain about 150 physical I/O operations per second and about 15 MB per second, as confirmed by some empirical observations, it follows that we should try to send each disk about 100 KB per operation. We can accomplish this by doing three things:

  1. When we are designing our volume, we want to specify the interlace size (recall that this is the contiguous chunk given to one disk before switching to the next one). We will pick something in the range of 64 KB-128 KB.

  2. When we build the filesystem, we need to place large chunks of the file on contiguous block numbers; this behavior is controlled by the maxcontig parameter (settable via newfs's -C switch when building a new filesystem, or tunefs -a on an existing filesystem). A good value for maxcontig here is 128 (set in 8 KB blocks), which will tell the filesystem to lay out files in blocks of 1 MB of contiguous data.

  3. Eventually, when we are using the filesystem, the kernel will need to flush data to disk. It will do this by looking for modified pages that are contiguously allocated in the filesystem, and push them out in one large, clustered I/O: the largest I/O it can issue is specified by maxphys, so we'll set that to 1,048,576.

We should probably also keep an eye on the UFS write throttle, in case that turns out to be a limiting factor in performance.


6.7 Concluding Thoughts

The technology behind disk arrays helps us solve one of the most limiting factors in performance and reliability: the rotation of platters coated with magnetic film. It's a remarkably simple idea that lets us, if not kill, at least stun two birds with one stone. Unfortunately, there are a great many myths and falsehoods surrounding how to design a disk array for optimal performance in any given environment, and it's often difficult to determine what the environment in question is really doing.

If I can give you one concluding thought to take away from this chapter, it is that disk array design is largely a pattern of trial and error. Define a workload that you can measure reproducibly, time it, use the material in this chapter to start on the right path, and then experiment -- change one parameter at a time, repeat your benchmark, and find the best solution to your problem. This isn't an area where one size fits all.


Chapter 7. Networks

One-two, one-two, and through and through The IMP-to-IMP went ACK and NACK, When the RFNM came, he said "I'm game," And sent the answer back,

Then hast thou joined the ARPANET? Oh come to me, my bankrupt boy! Quick, call the NIC! Send RFCs! He chortled in his joy.

—An excerpt from "ARPAWOCKY," D. L. Covill, May 1973 (RFC 527)

The gap between a systems administrator and a network administrator is often quite wide. Whereas a systems administrator is usually responsible for the hands-on management of a few hundred machines, perhaps in several buildings, a network administrator can easily be responsible for a network infrastructure that spans cities or states, from a high performance backbone down to the jacks at someone's desk. It is extremely rare to find someone who can play both positions well.

This chapter is aimed primarily at systems administrators who have little experience in networking, yet find themselves concerned that a performance problem is based in the network. This can be a quandary, particularly if there is no network engineer to be found! All of a sudden strange acronyms appear from the ether, and it can be hard to figure out what is going on. The foundation of performance tuning is understanding what the system is actually doing. This chapter provides an overview of network principles, encompassing the range from physical media all the way through to the TCP and UDP protocols. We'll then discuss the two most common file sharing mechanisms, NFS and CIFS. First, however, an important note about terminology.

The convention in the network world is to refer to data rates in megabits per second (Mb/s), rather than the megabytes per second (MB/s) common elsewhere in computing. In order to make a meaningful comparison, you must divide the number of Mb/s by eight. For example, a 10 Mb/s Ethernet connection transfers at most 1.25 MB/s.


7.1 Network Principles

A network is an entity designed to allow data sharing between devices by connecting them to a shared medium. This shared medium is known as the data link. In addition to this shared medium, there are defined rules for how the common medium is shared, in much the same way that roommates in an apartment have rules on who can take what from the refrigerator. This set of rules is called a media access control, or MAC. In addition, the devices on a network must have some unique identifier associated with them, so that they will not receive data not intended for them. This data is encapsulated within a sort of "binary envelope" called a frame, which specifies the destination and source address (the equivalent of a return address). The format of these addresses depends on the MAC protocol being used. Ethernet, the most common data link, shares a common format with Token Ring and FDDI. Ethernet relies upon a 48-bit number designed so that every network device on the planet is uniquely identifiable. The number, which is called the MAC address, hardware address, data link address, or physical address, serves the purpose of unique identification.[1]

[1] Although some network devices allow this value to be reconfigured, the point is that they are identifiers unique within a network.

As networks expand, additional components begin to be necessary. Each of these devices addresses a different problem of network growth. Foremost amongst these are repeaters, which address issues of signal degradation, and bridges and routers, which approach problems of scalability. Switches approach both problems.

As data is transmitted through any physical medium, the signal degrades. The degradation is worsened by distance. A repeater serves to regenerate the "noisy" signal into a new, clean copy; its name originates from this repeating of network traffic. Here, I use the term hub to refer to a type of repeater that has many ports; any traffic being sent on one port is repeated on all the other ports (note that "hub" can mean many other things: it can be a network center, a network closet, a point where many networks come together, etc.).

As networks become more crowded, congestion -- too many people trying to talk at once -- becomes an issue. One solution to this problem is to break one large network into several parts, which are then interconnected by a device called a bridge. Fundamentally, a bridge works by listening to all the traffic it can, and then building a bridging table that tells it which hosts live on which networks based on whom it sees sending data on its inputs. The bridge uses the table to pass traffic between its inputs as necessary to get the traffic to its destination (therefore, it won't pass traffic when the destination is on the same input as the receiver). This is known as a transparent bridge; there are many other sorts of bridges, however, and a full discussion of them is beyond the scope of this text.

Unfortunately, bridges do not allow indefinite scalability. Some frames cannot be localized into a single network; these broadcast frames must be delivered to all hosts on the network. As the number of hosts in a purely bridged network grows, broadcast traffic will eventually cause the network to be congested.

In order to solve this problem of scalability, at some point it becomes necessary to create populations of networks within a larger network, more properly known as an internetwork. Defining these boundaries is the province of the router, which has been known by many different names. In the pre-Internet days of ARPANET, routers were known as internet message processors (IMPs). More recently, they have been called gateways[2] and intermediate systems. A router serves to send information along a specific path between two networks: with modern routers, this often involves choosing the best path between several available options. The mechanism that a router uses to select the best route, as well as how to share information with other routers in the internetwork, is called a routing protocol. Because routers exist only to deliver data to a specific destination network, they typically do not pay attention to other nonrouter devices. When a router sees that a piece of incoming information is destined for its network, it acts like a host on that network and uses the data link identifier of the destination host to deliver the data. A router may route packets even when the target is not on a network directly connected to it by passing it to another router that is closer to the destination host.

[2] This is another example of sloppy word usage. A gateway is technically a system that translates protocols (e.g., LocalTalk to IP); a router doesn't do that. A gateway can be a router, but a router can't be a gateway.

On most networks, the majority of traffic is from one host to another (unicast) rather than broadcast. It is not necessary for the other hosts to see that data, since the frame is not addressed to them. In addition, repeaters in a workgroup environment have security issues; consider a case with three systems all connected to a hub. One machine, alice, opens a telnet session to another machine, bob. In a repeated network, the third data link leading to the third machine, spy, contains all the traffic between alice and bob. Most network interfaces can be put into promiscuous mode, where they listen to every packet on the medium, not just those addressed to themselves. While this is invaluable for debugging network problems, it also means that spy can read the contents of all traffic between alice and bob. A switch serves like a repeater in that it regenerates signals, but effectively creates "virtual networks" between hosts. Therefore, any traffic originating on one port is sent only to the destination port. This makes exploiting the security problem described more difficult, since spy can no longer directly see the traffic between alice and bob.[3] It also provides a potentially large performance improvement, since rather than every host sharing a 10 Mb/s medium, each host now has, effectively, a dedicated 10 Mb/s medium.

[3] However, for diagnostic reasons, most switches can be configured so all traffic from all interfaces is repeated to one port. For this reason, if the security of the switch is compromised, the switched network is still vulnerable to this sort of attack. There are also many other attacks that don't involve compromising the switch.

There are two types of switches: store-and-forward and cut-through. A store-and-forward switch buffers the entire inbound frame into memory, and then routes it to the appropriate destination, whereas a cut-through switch uses a "hot potato" approach. In the past, cut-through switches were favored because of the high cost of buffer memory. One advantage of cut-through switches is that they provide very little change in latency between packets, whereas a store-and-forward switch can have variable latency. Store-and-forward switches handle collisions gracefully (see Section 7.3.1.8 later in this chapter), whereas cut-through switches have serious problems handling collisions. Most modern switches use store-and-forward, while cut-through techniques have found a niche in some microprocessor scalable interconnects.

One of the classic texts on internetworking is Radia Perlman's Interconnections: Bridges and Routers, published by Addison-Wesley.

7.1.1 The OSI Model

Just like the levels of transformation described in our overview of computer architecture (see Chapter 1), there is an analogous formal description for network architectures. This is called the OSI model and consists of seven layers:

Physical layer

Describes the physical characteristics of the medium. This includes four specific categories: electrical/optical protocols, which describe the characteristics of the signals on the wire; mechanical protocols, which define the types of connectors used or the diameter of wire; functional protocols, which describe what specific signals mean (for example, what a specific pin encodes); and procedural protocols, which explain how something is done (for example, how much voltage is considered a binary digit).

Data link layer

Defines the protocols that control the behavior of the physical layer -- for instance, how devices are identified and how data is framed. An example of a data link protocol would be Ethernet.

Network layer

Describes how data is routed between logical paths. It also encompasses such details as how addresses are formatted.

Transport layer

Specifies the protocols that control the network layer, in much the same way that the data link protocols control the physical layer. Transport layer protocols control the traffic on a logical link, which is the end-to-end connection of devices that encompasses a series of data links.

Session layer, presentation layer, and application layer

Combine to describe how the user applications use the network.

In this chapter, we will approach networking from the ground up. That is, we'll start at the very lowest level of the OSI model -- the physical network -- and work upwards to the software that relies on the network infrastructure.


7.2 Physical Media

There are essentially two possibilities for the medium of a modern network: unshielded twisted-pair (UTP) or fiber optic cables. Fiber, in general, allows for higher data rates over longer distances and is substantially more secure, but is significantly more expensive.[4] There are three problems associated with transmitting data over a physical distance: attenuation, distortion, and interference.

[4] Although fiber itself is relatively cheap, its cost is driven up by the fact that the connectors are quite expensive, and it is expensive to work with. For example, it is very easy to splice two copper wires. It is substantially more difficult to splice two strands of glass. This is precisely why fiber is more secure, as well.

Attenuation occurs because it requires some amount of signal energy to "push" past the resistance of the wire. This results in a gradual degradation of the signal, because it is losing energy to the wire as it proceeds down the line. Distortion is a function of the way that signals travel through a medium: different frequencies are impeded in diferent ways, muddying the signal. Finally, interference occurs because of outside factors that introduce noise into the signal.

7.2.1 UTP

Unshielded twisted-pair cable is divided into several categories based on the characteristics of the cable. Higher categories are able to resist noise at higher frequencies, and therefore are more likely to resist external interference. The UTP category classifications are summarized in Table 7-1.

Table 7-1. UTP categories

Category

Maximum speed

Common use

1

56 Kbps

Voice communication

2

4 Mb/s

Integrated Services Digital Network (ISDN)

3

16 Mb/s

Data-grade media for local area networks: 10 Mb/s Ethernet, Token Ring

4

20 Mb/s

Extended distance local area networks

5

Over 100 Mb/s

Data-grade media: 100 Mb/s Ethernet, ATM and FDDI over UTP; short-distance 1 Gb/s Ethernet

There is a new, emerging cable standard, called Category 5E or Category 6. This cable relies on a special stiffening core in order to maintain the twist in the signaling lines. This standard has specific utility in very high-bandwidth applications (such as Gigabit Ethernet over copper media).

There are two types of connectors commonly used with UTP cabling: RJ-11 and RJ-45. RJ-11 is the standard four-pin connector used in telephone jacks, which is rarely found in modern network installations. RJ-45 actually defines a specific pin-out, but the term is commonly used to describe the standard eight-pin connector used almost universally in networking.

7.2.1.1 A note on terminology: plugs and jacks

A plug is the male connector on the end of a cable; a jack is the female connector in a patch panel or wall plate. When you are looking at a jack in a wall, with the clip facing down, pin 1 is leftmost. When you are looking "down" a cable with the clip on the plug facing down, pin 1 is rightmost.

7.2.2 Fiber

Fiberoptic cable consists of three parts: a glass or plastic core through which light propagates, surrounded by a cladding, which is also made of glass or plastic but has substantially different optical properties. The core and cladding are both surrounded by a jacket, which provides protection from environmental compromise. Light from a source enters the core: rays that enter at a shallow angle (almost straight down the core) are reflected by the cladding and propagate down the core, whereas rays that enter with a large angle are absorbed by the outer layers. A ray that enters the fiber is called a mode. It is for this reason that fiber is very sensitive to bend radius in the cable; if you bend a piece of cable in too tight of a loop, you will lose some of the transported light. This results in a phenomenon known as attenuation, and while it doesn't cause signal loss, it does reduce range. Figure 7-1 shows a cross-section of a fiberoptic cable.

Figure 7-1. A fiberoptic cable in cross-section

This type of propagation is called multimode (often abbreviated MMF, for multimode fiber), because there are many possible modes that can be sent through the fiber at once, and propagate through at differing rates. Multimode fiber generally has a core size of 50 or 62.5 microns. The light rays proceed through the fiber along different paths, which causes different rays to arrive at the receiving end of the fiber at different times. A smearing effect occurs that is proportional to distance. Eventually, the pulses are distorted enough to cause data integrity problems. This effect is tolerated because of the low cost of multimode fiber.

There are two varieties of multimode fiber: step-index and graded-index fiber. In step-index fiber, the refractive indices of the core and the cladding differ, but the refractive index is the same throughout the core. The result is a dispersion pattern like that described, where the lengths of the paths of light increase with distance. In graded-index fiber, the refractive index of the core varies across the cross section, reducing the difference in path lengths and correspondingly increasing the usable distance.

Single-mode fiber, or SMF, has a very narrow core, generally from two to ten microns. This means that only a single mode can propagate along the axis, which completely solves the internal reflection problems with multimode fiber. Single-mode fiber has a very high range (kilometers). Figure 7-2 illustrates multimode step-index, graded-index, and single-mode fiber.

Figure 7-2. Multimode step-index, graded index fiber, and single-mode fiber

There are many different common forms of connectors used in fiber installations, but there are three particularly common ones. FSD, or Fixed Shroud Device, is typically used in FDDI. ST is a common bayonet-style connector that uses one connector per strand. SC is the international standard: it is a clip-type connector that has the advantage over ST of being a single connector containing both the transmit and receive fibers. The common mnemonic for remembering which of these connectors is which is "stab-twist and stab-click." There have been recent pushes to create a new connector standard, of which the most likely contender is LC, which was designed at Lucent Technologies.


7.3 Network Interfaces

This section discusses the physical layers that interconnect network devices; we'll focus primarily on Ethernet.

7.3.1 Ethernet

The origins of Ethernet date back to 1973, when it was invented by Bob Metcalfe[5] and David Boggs, then of Xerox PARC (Palo Alto Research Center). The first implementation ran at 3 Mb/s. About 5,000 machines were built with this interface; most were deployed within Xerox, although some went to other corporations (particularly Boeing), government, and academia. The initial Ethernet standard was developed by a consortium, formed in 1979, of Xerox, Digital Equipment Corporation, and Intel. Their work was sent to a newly formed IEEE working group, which was later named IEEE Project 802. This working group was divided into three groups:

[5] Who later went on to found 3Com Corporation and evangelize Ethernet as a multivendor standard.

  • The High Level Interface (HILI) group, focusing on internetwork protocols and management (which later became 802.1).

  • The Logical Link Control (LLC) group, which emphasized end-to-end connectivity and the media-independent layers (which later became 802.2).

  • The Data Link and Medium Access Control (DLMAC) group, responsible for the medium access protocol itself.

In 1982, the DLMAC group further split into three committees: 802.3 for Ethernet, driven by Xerox, DEC, and Intel; 802.4 for Token Bus, championed by Burroughs, Honeywell, and Western Digital; and 802.5 for Token Ring, which was the domain of IBM. The resulting 802.3 Ethernet standards committee developed a series of specifications for 10 Mb/s Ethernet to support different kinds of physical media. Initially, Ethernet ran on thick coaxial cable only; later, thin coaxial cable, unshielded twisted pair, and fiber optic cables were supported.

There was little initial emphasis on performance in the development of Ethernet for two reasons. First, networks were generally small, and traffic was not approaching bandwidth limits, making sophisticated performance monitoring unnecessary. Second, it was technically difficult, since implementing advanced metering and control capabilities would add significant complexity and cost.

As it became apparent that the 10 Mb/s data rate was becoming too slow, an effort began to increase the data rate by an order of magnitude. However, due to political considerations, two groups formed with different approaches. The first group aimed to retain the existing protocol as much as possible, and make only minor interface changes, focusing on Category 5 UTP cable initially. This would leverage the "Ethernet" name as well as the large base of existing engineering expertise. This group, which received a huge marketing and public relations push, became formally known as the 802.3u Task Force. The other group wanted to define a new protocol, with the intent of building a more efficient transmission method, and focused on reusing existing Category 3 UTP cable. This group became the 802.12 Task Force. Both groups fought on the technical and marketing fronts during the standards development process, and eventually 802.3u and 802.12 were both passed as official standards by the IEEE Standards Board on June 13, 1995. Whatever advantage 802.12 may have held, the court of public opinion clearly favored 802.3u, and virtually all mainstream LAN vendors adopted the 802.3u standard.

It might be assumed that the adoption of 100 Mb/s networking products would have immediately killed the 10 Mb/s Ethernet marketplace. This was definitely not the case for several reasons: the higher bandwidth was not required in all cases, there was a massive installed base of 10 Mb/s devices, and there was a substantial initial cost premium associated with the newer devices.

Shortly after the adoption of the Fast Ethernet standard, people began to be interested in increasing Ethernet performance by another order of magnitude. This need was driven by the widespread deployment of 100BASE-T networks, which put an immense load on backbone infrastructures. While there were existing technologies, such as ATM, that were capable of much higher data rates (622 Mb/s at the time; currently the figure is closer to 2.4 Gb/s), deploying these in combination with the wildly popular Ethernet required the frame formats to change. This is a very expensive thing to do quickly, and thus an effort began to evaluate 1 Gb/s networking within the Ethernet model. The standard was finalized in June of 1998, and is beginning to see deployment as of 2000.

7.3.1.1 Fundamentals of Ethernet signaling

In 10 Mb/s Ethernet, the actual signals placed on the wire use a technique known as Manchester encoding, which allows the clock signal and the data to be transmitted in one logical parcel. One way of thinking about this process is that it is like differential signaling (see Section 5.2.3.6), but using one wire instead of two. This parcel, formally called a bit-symbol, includes the logical inverse of the encoded bit followed by the actual value of the encoded bit, so that there is always a signal transition in the middle of the bit-symbol. For example, the bit "0" would be encoded in Manchester as the bit-symbol "01." This seems silly, since it appears to double the amount of work required to send a bit of data, but just like differential signaling, it is useful in long-distance communications. Its biggest disadvantage is that it generates signal changes on the wire twice as fast as the data rate, which makes the transmission more susceptible to interference. As a result, 100 Mb/s Ethernet does not use Manchester encoding.

The data encapsulation for Ethernet is rather nontrivial. The data, which is a minimum of 46 bytes and a maximum of 1,500 bytes,[6] has a 23-byte header and a 4-byte trailing block attached. The contents of the header and trailer are described in Table 7-2.

[6] The MTU size of Ethernet (1,500 bytes) measures only the payload capacity of an Ethernet frame. The Ethernet headers are not counted against the 1,500-byte payload limit.

Table 7-2. Ethernet header

Bits

Content

0-55

Preamble

56-63

Start Frame Delimiter

64-111

Data address

112-159

Source address

160-175

Type

Data

Last 32

FCS

The preamble is a sequence of alternating ones and zeroes. This provides a single frequency on the network that the receiver can use to "lock on" to the incoming bit stream, and it is not passed through to the host system. It turns out that the preamble plays a critical role in collision detection; if two devices are transmitting at the same time, no matter how the waveforms line up, the interference can be detected, since the preamble always fits the alternating pattern. The preamble is followed by a start-of-frame marker, which is called the Start Frame Delimiter (SFD). The data address is the 48-bit hardware address of the destination (see Section 7.1 earlier in this chapter), and the source address is the 48-bit hardware address of the source. The type field describes the protocol the message is using.

7.3.1.2 Topologies

There are a set of defined Ethernet topologies, each described by a specific nomenclature. The original standard used thick coaxial cable, and was known as 10BASE5. The 10 encodes the network data rate in Mb/s, the "BASE" refers to the use of a signaling method known as baseband, and the 5 describes the maximum segment length in 100 meter increments. The coaxial cable serves as a linear bus to which all nodes are connected; the center conductor is used to transmit the signal and the shield is used as a ground reference. The coaxial cable used in 10BASE5 is quite thick (about a centimeter) and not very flexible. The actual connection between a given node is accomplished by an attachment unit interface (AUI) cable, which has a maximum length of 50 meters and uses the common 15-pin D-type connector. The AUI cable is connected to the medium attachment unit (MAU), which serves as the interface between the two. There are two methods for making this connection. The first method, which is known as an extrusive tap, involves the MAU clamping onto the thick coaxial cable and inserting a probe into it.[7] The second method is an intrusive tap, which involves cutting the coaxial cable and inserting the MAU in the middle. There may be no less than 2.5 meters between MAUs, and the distance must be a multiple of 2.5 meters. In today's environments, this topology is a nightmare to work with: minor changes can easily involve rerouting the cable, vampire tap mechanisms are notoriously difficult to get correct and tend to fail, and intrusive taps involve network downtime. Despite its limitations, this topology can still be found.

[7] This is also known as a vampire tap.

In order to alleviate many of the installation difficulties of 10BASE5, the 10BASE2 standard was defined to allow the use of thin (less than 5 mm) coaxial cable, commonly known as "Thinnet." The thin coaxial cable is more flexible and easy to work with, and the attachment is much simpler, requiring a simple BNC connector.[8] However, the use of the lower-cost cable reduced the maximum cable length to 185 meters. Due to difficulties in reliability, 10BASE2 was slowly eclipsed in the early 1990s by 10BASE-T.

[8] BNC is from Bayonet-Neill-Concelman.

7.3.1.3 10BASE-T

10BASE-T, more commonly known as "twisted-pair Ethernet," allows the use of standard, Category 3, two-pair (one for data transmission, one for reception -- a total of four active wires) cables. Because separate transmit and receive paths are implicit in such a cabling system, there must be some way for either end to make sure the link is physically connected. This is known as the Link Integrity test, and is accomplished by each end transmitting a link test pulse when there is no data to be sent. If a link test pulse is not received, the Ethernet interface will mark the line as link fail and (in theory) refuse to transmit data.

This topology, unlike 10BASE2 and 10BASE5, is not serial; instead, it relies on a star topology, where each host in the network is connected to a central hub, which acts as a repeater. Each link may be up to 100 meters (that is, the distance from any host to the hub must be no more than 100 meters).

7.3.1.4 100BASE-T4

The 100BASE-T4 protocol is the result of an effort to provide 100 Mb/s data rates over widely installed Category 3 wiring. It uses a star topology, and requires the continuous use of four pairs of wire, three of which are used to transmit data, and the remaining pair to listen for simultaneous activity from the other end of the link (thus inducing a collision). Because three out of the four pairs of wire are required to transmit data, 100BASE-T4 is a half-duplex medium only. One problem with this signaling scheme is a phenomenon known as pair skew. Pair skew means that each of the signals sent on each of the three pairs will spread apart slightly in time as they progress down the wire because of variations in the lengths of the cables. While the 100BASE-T4 hardware is designed to cope with this condition, it is important that all pairs for a particular cable reside in the same jacket. Although 100BASE-T4 is well supported, it is not widely used.

7.3.1.5 100BASE-TX

The overall topology of 100BASE-TX, which provides 100 Mb/s data rates over distances of up to 100 meters using Category 5 unshielded twisted-pair cable, is similar to that of the other twisted-pair standards. It differs only in the type of cable required and in some esoteric signaling mechanisms that we will not discuss here. 100BASE-TX has quickly become the dominant 100 Mb/s Ethernet technology in the marketplace.

7.3.1.6 Gigabit Ethernet topologies

There are currently four topology standards for Gigabit Ethernet: 1000BASE-SX, short-wavelength laser light over multi-mode fiber; 1000BASE-LX, long-wavelength laser light over multi-mode or single-mode fiber; 1000BASE-CX, short-haul copper; and 1000BASE-T, which uses Category 5 UTP cable. The difference between 1000BASE-SX and 1000BASE-LX is primarily one of distance, as illustrated in Table 7-3.

Table 7-3. Distance specifications for Gigabit Ethernet over fiber

Type

MMF (50 m)

MMF (62.5 m)

SMF (10 m)

1000BASE-SX

~500 m

~220 m

Not available

1000BASE-LX

~500 m

~500 m

~5 km

The intent of 1000BASE-CX is to cheaply enable interconnections between physically proximate devices, such as in a wiring closet. 1000BASE-CX has a range of approximately 25 meters. 1000BASE-T follows the same guidelines as 100BASE-T, but it suffers from technical complexity because of the huge impact of inteference when using copper-based cables at such high data rates.

7.3.1.7 The 5-4-3 rule

There is an IEEE rule that describes how repeated segments can be attached to each other. This rule defines the maximum size of an unrouted/unswitched network, and is called the 5-4-3 rule. It says that there can be no more than five repeated segments, nor more than four repeaters between any two Ethernet hosts, and of those five repeated segments, only three may be populated (that is, have devices other than bridges attached to them). These restrictions can be very limiting. Since switches regenerate and reframe packets, this rule does not apply to them.

7.3.1.8 Collisions

A collision occurs when two devices on the network sense that the network is idle and end up trying to send data at the same time. A collision may only be detected during the transmission of the preamble (see Section 7.3.1.1 earlier in this chapter). Since only one device can transmit at a time, both devices back off and try to retransmit. The retransmission algorithm mandates that each device wait a random amount of time, so the two are not likely to collide in their retransmission attempts. If a packet collides 16 consecutive times, it is aborted. Some collisions are normal on a repeated network, but excess collisions can cause serious performance problems. Unfortunately, there is no single good rule of thumb to say "this number of collisions is too many." As a general guideline, however, you can use Table 7-4.

Table 7-4. Collision rate guidelines

Network type

Cautionary levels

Severe levels

Unswitched

5-10%

15%

Switched

2-4%

5%

Resolving severe collision problems on unswitched networks is best done by moving to a switched design. The most likely cause for high collision rates on switched networks is duplex mismatches between the switch and hosts; that is, the switch thinks that the link is full-duplex, but the host is running at half-duplex, or vice versa.

A related problem is a late collision, which happens when a collision occurs but is not detected in time for the usual collision resolution protocol to occur. This is because the time required to send the data throughout the network is longer than the time to push the data onto the wire, so the two devices responsible for the collision never see the other data until they have put all of their own data on the network. Late collisions cause serious network performance problems, especially in NFS Version 2 environments. Late collisions are usually due to excessively long segments, faulty connectors, or defective devices.

The presence of late collisions on a network is a very strong indication of serious network problems. Normal collisions don't significantly impact performance until they exceed 10% on an unswitched network or 5% on a switched network.

7.3.1.9 Autonegotiation

When 100 Mb/s Ethernet was introduced, there was an obvious requirement for vendors to support the older 10 Mb/s standard in the new devices, especially in adapter cards and switches. 100BASE-T defines an auto-negotiation[9] feature that allows devices to interoperate with both 10 Mb/s and 100 Mb/s installations. Auto-negotiation also determines the capability of the device as half or full duplex.

[9] Sometimes called auto-sensing, which is misleading.

Duplex Modes

The duplex mode of a network link describes whether both parties can transmit concurrently. In a full-duplex link, both sides can transmit concurrently, effectively doubling the throughput on a well-balanced link. [a] In a half-duplex link, only one device can transmit at once.

[a] The ratio of transmitted data to received data is approximately 1.

So, auto-negotiation asks two questions, to each of which there can be two answers:

  • What speed can the interface run at? (10 or 100 Mb/s)

  • What duplex mode can the interface use? (Full or half)

Unfortunately, while the auto-negotiation mechanism looks great on paper, it can have some implementation issues. Although the final 100BASE-T standard defines a way of performing auto-negotiation, this was one of the last sections to be completed, and many vendors had shipping 100 Mb/s implementations before the standard was finished.

Let's look at an example where auto-negotiation works properly. Here, both the local network interface and the upstream network interface (e.g., on a workgroup switch) have auto-negotiation enabled. When the interfaces detect the link, each one transmits a list of the modes where it can operate. With this information, each side can independently pick the best common mode. The critical piece of information is that auto-negotiation is an active method of determining the mode for the link: each interface is expected to transmit the modes it can use in a specific format. If an interface does not receive this information, it assumes the other side is not an auto-negotiating device. The other important thing to understand is that auto-negotiation relies upon both devices picking the same choice from the list of common modes. This generally works, but it is not 100% reliable.

Many of the pre-standard interfaces use auto-sensing; they look at the incoming data and makes a guess about what the other end of the link is capable of. There is no explicit communication between the interfaces to negotiate this standard. Passively detecting the speed is easy. Each Ethernet packet starts with a sequence of bits that alternate between 1 and 0. To determine the link speed, the interface simply measures the time between these state changes. If an interface works from the fastest mode down, the other interface can pick the first speed it understands. Unfortunately, passive methods of reliably selecting a duplex mode are nearly impossible. This problem is compounded by the fact that an Ethernet link still functions if the duplex modes are mismatched (mostly).[10]

[10] Transmission of data in both directions is allowed in full-duplex mode, but this behavior is considered a collision in half-duplex mode.

In order to get around these problems, it is common practice to force an interface to a specific mode and not rely on any auto-negotiation or auto-sensing systems. This practice can cause problems, however; the classic example is when the upstream interface is forced into 100 Mb/s full-duplex operation, and the workstation configured at the other end of the link settles into 100 Mb/s half-duplex mode and starts to generate a high number of network errors. The reason for this is that most new interfaces use auto-negotiation; if auto-negotiation is turned off on the upstream side, the host on the other side of that link will send its own auto-negotiation information -- but the switch will not send any of its own, because its auto-negotiation functionality is disabled. The downstream host will then assume that the upstream interface is not capable of full-duplex mode, and use passive auto-sensing methods to select the transfer rate.

As a result, you must either trust auto-negotiation to succeed, or you must hand-configure both sides of the link. This is especially true if you intend to use full-duplex mode: if one side of the link has auto-negotiation disabled, the other side of the link will always find the right speed, but will also default to half-duplex mode.

7.3.1.10 Displaying and setting modes

On Solaris systems with the hme Ethernet driver,[11] you can view the current status of the Ethernet interface by using ndd -get device flag:

[11] hme is actually short for Happy Meal Ethernet. The predecessor 10/100 card, which was called be, is short for BigMac Ethernet (not Broken Ethernet, as is the common joke).

# ndd -get /dev/hme link_status

1

# ndd -get /dev/hme link_speed

1

# ndd -get /dev/hme link_mode

1

The meanings of the returned values are defined in Table 7-5.

Table 7-5. ndd return values

Flag

Meaning of "0"

Meaning of "1"

link_status

Link down

Link up

link_speed

10 Mb/s

100 Mb/s

link_mode

Half-duplex

Full-duplex

The easiest way to change the configuration parameters of an interface is a utility called hmeconfig, which a coworker of mine wrote when we were at the University of Illinois. It is available from http://arrakis.cso.uiuc.edu/jak/code/hmeconfig.html and is self-documenting.

You can also use ndd to force the interface into a specific mode of operation. This can be done at runtime, as described in the next example, but please note that the order of these commands is important. For example, this command sequence will force an hme interface to 100 Mb/s full-duplex operation:

# ndd -set /dev/hme instance 0

# ndd -set /dev/hme adv_100T4_cap 0

# ndd -set /dev/hme adv_100fdx_cap 1

# ndd -set /dev/hme adv_100hdx_cap 0

# ndd -set /dev/hme adv_10fdx_cap 0

# ndd -set /dev/hme adv_10hdx_cap 0

# ndd -set /dev/hme adv_autoneg_cap 0

If you want to preserve the changes across a reboot, you must change /etc/system:

set hme:hme_adv_autoneg_cap=0

set hme:hme_adv_100T4_cap=0

set hme:hme_adv_100fdx_cap=1

set hme:hme_adv_100hdx_cap=0

set hme:hme_adv_10fdx_cap=0

set hme:hme_adv_10hdx_cap=0

Each of the parameters has a clearly defined meaning, summarized in Table 7-6.

Table 7-6. hme network interface parameters

Variable

Meaning[12]

adv_autoneg_cap

Use auto-negotiation.

adv_100T4_cap

The device is capable of using the T4 cabling standard.

adv_100fdx_cap

The device is capable of running at 100 Mb/s full-duplex.

adv_100hdx_cap

The device is capable of running at 100 Mb/s half-duplex.

adv_10fdx_cap

The device is capable of running at 10 Mb/s full-duplex.

adv_10hdx_cap

The device is capable of running at 10 Mb/s half-duplex.

[12] For all of these, a value of 0 means "off" and a value of 1 means "on."

For example, forcing an interface into 10 Mb/s half-duplex operation is a matter of setting all these variables to zero except for adv_10hdx_cap, which should be set to 1.

You can also view the characteristics advertised by the other end of the connection by using the following commands:

# ndd -get /dev/hme lp_autoneg_cap

1

# ndd -get /dev/hme lp_100fdx_cap

1

# ndd -get /dev/hme lp_100hdx_cap

1

# ndd -get /dev/hme lp_10fdx_cap

1

# ndd -get /dev/hme lp_10hdx_cap

1

In this case, the upstream host is auto-negotiating and able to use all of the possible network modes.

Unfortunately, due to the vast array of different network interface cards supported by Linux, there is no generic method of forcing an interface. A careful reading of the documentation for your particular driver should elucidate how this can be done.

On multi-module Cisco switches running Version 5.5 of the Switching System, such as the Catalyst 5500 series, the show port card/number command can be used to display the status of a particular port:

Switch> (enable) show port status 2/6

Port  Name               Status     Vlan       Level  Duplex Speed Type

----- ------------------ ---------- ---------- ------ ------ ----- ------------

 2/6  Sample Connection  connected  trunk      normal full     100 100BaseTX

The duplex mode of a port can be switched by using set port duplex card/number value, where value is either "half" or "full." Similarly, you can set the speed or autonegotation value of a port by using set port speed card/number value, where value is either "100," "10," or "auto."[13] Here's an example where we set port 4 on card 3 to run at 10 Mb/s in half-duplex mode:

[13] An interface set to auto-negotiate speed will also auto-negotiate its duplex mode.

Switch> (enable) set port speed 4/3 10

Port 4/3 speed set to 10 Mbps.

Switch> (enable) set port duplex 4/3 half

Port 2/1 set to half-duplex.

Workgroup-class Cisco switches -- such as the Catalyst 2900 and 3900 series -- use a different configuration interface that is very much like Cisco's IOS operating system, which runs on most of their router products. If you have one of these switches, you can change a port by using the following command sequence:

Switch# configure terminal

configure# interface interface

configure# speed 10|100|auto

configure# duplex half|full|auto

configure# end

Changing these interface parameters usually involves the network interface being brought down and then up again. It can be dangerous to do this if you are connected to the system through the interface in question. . . .

7.3.2 FDDI

The Fiber Distributed Data Interface (FDDI) was developed by ANSI in the mid-1980s in response to growing pressures on the capabilities of existing bandwidth technologies. It had become clear that a new media was needed that could support the most recent, high performance workstations, as well as provide high levels of network reliability. These two characteristics -- performance and reliability -- shaped the new protocol.

FDDI is a 100 Mb/s architecture in which two continuous loops, or rings, connect to every host on the network. However, the rings are counter-rotating: traffic in each ring flows opposite to the other. One of these rings is dedicated as the primary, and during normal operation it carries all of the traffic, while the other ring remains idle. This provides a remarkable amount of fault tolerance.

If a station on both rings fails, or if the cable is damaged, the ring is automatically wrapped (that is, the "loose" ends on either side of the fault are "connected" by the FDDI hardware), which transforms it into a single ring. Network performance is absolutely not affected. However, FDDI is only tolerant of a single failure. If two or more failures occur, the ring will segment into multiple, independent rings that cannot communicate. The solution to this is the optical bypass switch, which is, in essence, a device that sits on the ring in place of the host. The optical bypass relays the signals on the FDDI ring to the host by means of mirrors; in the event of failure, the optical bypass will pass the light through itself and so maintain ring integrity.

However, a device need not be on both rings. One unique characteristic of FDDI is that there are multiple ways to connect devices. There are three defined devices: a concentrator, a single-attachment station (SAS), and a dual-attachment station (DAS). A dual-attachment station has two ports and is attached to both rings. In some cases, if a dual-attachment station is powered off, the ring incurs a fault, but modern FDDI interfaces have built-in optical bypass switches.

A concentrator, which is also called a dual-attachment concentrator (DAC), connects to both rings of a FDDI network and ensures that the failure of any attached stations does not compromise a ring. In essence, a concentrator is the equivalent of an Ethernet hub. A single-attachment station has a single port, and connects to only one ring through a concentrator: the primary advantage is that if the device is disconnected or powered off, the ring integrity is not compromised.

The FDDI frame format borrows heavily from the Token Ring frame format. A FDDI frame can be as large as 4,500 bytes. We won't discuss the protocol in detail, but the most important idea is that only one station (the one with the "token") can transmit at a time. When a station is done transmitting, it passes the token along to the next host. It also has very powerful network management protocols.

FDDI, as the name implies, is usually run over optical fiber. However, the FDDI protocol can also be deployed over copper cables, formally known as the Twisted Pair Distributed Data Interface (TP-DDI) but more commonly referred to as the Copper Distributed Data Interface (CDDI). CDDI supports a distance from the concentrator to a host of about 100 meters, whereas FDDI over multi-mode fiber is capable of 2 kilometers; using single-mode fiber gives an even larger distance.

FDDI was used extensively in backbones in the early 1990s because it was extremely fast for the time, capable of running over long distances, and fault-tolerant. With the appearance of ATM and Fast Ethernet at a lower price point, however, it has fallen out of favor.

7.3.3 ATM

Another common high-performance network design, Asynchronous Transfer Mode (ATM), offers an intriguing vision of a truly universally integrated services network, in which voice, data, and video can all flow over a single cost-effective infrastructure. Unfortunately, ATM is not without its downsides, and it has largely been overshadowed by Gigabit Ethernet, which offers faster data transfer rates and much less complex configuration.

ATM is a connection-oriented technology (it establishes explicit connections, called circuits, between endpoints) that uses a cell of 53 bytes to transfer information. This cell consists of 5 header bytes and 48 bytes of data.[14] Despite what is implied by its name, ATM does not transfer packets asynchronously. Cells are transmitted continuosly, even when no data is being sent. When there is no traffic, each cell is filled with a specific bit pattern that indicates that the cell is empty. The "asynchronous" part comes about because there is no set time when the first cell of a connection may start. The use of these small, fixed-length cells provides two definite advantages. It allows for simple, fast hardware switching, even at very high speeds; it also means that buffers can be allocated in exact segments, which prevents wasting of memory.

[14] Some readers might wonder why the data size is 48 bytes. Recall that ATM was designed to support both data and voice communications. Rumour has it that this number was settled upon because it was the average of what the voice engineers wanted (32 bytes) and the data engineers wanted (at least 64 bytes). The usual conclusion to this rumour is ". . . so everyone got screwed in the end."

ATM can be run over many sorts of physical medium at varying data rates, as summarized in Table 7-7.

Table 7-7. ATM physical media for various data rates

Data rate

Media

25 Mb/s

Category 3, 4, and 5 UTP

155 Mb/s

Category 5 UTP, multi-mode fiber

622 Mb/s

Multi-mode and single-mode fiber

One of the chief benefits of ATM technology, at least on paper, is that it has extensive quality of service (QoS) support. This allows applications to reserve specific amounts of bandwidth, and some applications to have priority handling of their cells.

There are three different approaches to transferring IP packets over ATM: classical IP over ATM, LAN Emulation (LANE), and Multiprotocol over ATM (MPOA). Classical IP maps IP addresses to ATM addresses, which allows ATM devices to generate IP traffic. Unfortunately, classical IP over ATM supports only IP proper (not IPX or AppleTalk or any other transport protocols), and it does not scale well. LANE takes a different approach, intended to accelerate the deployment of ATM, and attempts to make the ATM network invisible to legacy Ethernet LANs. The LANE interface looks like a standard Ethernet interface. Unfortunately, because LANE is so transparent, it loses the ability to use the ATM quality of service protocol.

7.3.4 Ethernet Versus ATM/FDDI

Before the widespread deployment of switched Fast Ethernet, ATM and FDDI had two big advantages: their performance decays more gracefully under load than did repeated Ethernet, and they are collision-free. In the modern world, however, switched Ethernet networks remove almost all of these advantages. As a consequence, FDDI has fallen almost completely out of use, and ATM has been relegated to a niche market. From a price/performance perspective, Fast or Gigabit Ethernet is hard to beat. ATM has specific technical advantages that may make it advantageous in certain environments, however.


7.4 Network Protocols

If we think of the media layers -- Ethernet, FDDI, and ATM -- as highways, then the network protocols are the rules of the road. It is not enough simply to have a strip of concrete; rather, we must paint lines to guide the traffic on its way. How can we get from San Francisco to Boston? When cars arrive simultaneously at a stop sign, who is allowed to pass first?[15] These rules are the domain of the two major network protocols: the Internet Protocol (IP) and the Transmission Control Protocol (TCP). Continuing the analogy of a highway, IP is analogous to the rules for laying out the interstate highway system and describing how to get from one city to the next, whereas TCP is a set of rules for how to travel on the roads.

[15] As we'll see later in TCP, the cars are allowed to smash into each other, and then new cars are dispatched. Wouldn't it be nice if automobile accidents were so easily managed?

7.4.1 IP

IP is a network-layer protocol (see Section 7.1.1 earlier in this chapter) conceived in the early 1970s by Vint Cerf and Bob Kahn. Other examples of network-layer protocols include IPX and AppleTalk, but we will not discuss them here.

IP specifies a header attached to the beginning of every data transferred. Together, the header and data form a packet. Table 7-8 describes the format of the IP header.

Table 7-8. IP header

Bits

Content

0-3

Version

4-7

Header length

8-15

Type of service

16-31

Total length

32-47

Identifier

48

Unused

49

Flag: don't fragment (DF)

50

Flag: more fragments (MF)

51-63

Fragmented offset

64-71

Time-to-live

72-79

Protocol

80-95

Header checksum

96-127

Source address

128-159

Destination address

160-191

Options and padding

The version field is usually set to 4, as the current version of IP in widespread use is Version 4 (IPv4). IP Version 6 (IPv6) is slowly gaining ground, but not yet in widespread deployment. Its performance impact has not been well-characterized, so we will not discuss it here. The header length specifies the length of the header field in 32-bit increments. The type of service field identifies special handling requirements for the packet as implemented by quality of service protocols, such as classification and priority handling. The total length field specifies the length of the entire packet, including the header, in bytes.[16]

[16] The maximum decimal number representable by 16 bits (the size of the total length field) is 65,535; as a result, the maximum possible size of an IP packet is 65,535 bytes.

7.4.1.1 Fragmentation

The identifier field controls fragmentation of a packet, in conjunction with the fragment offset field and the two flag bits. A packet must be fragmented into multiple smaller packets only if the original length exceeds the defined maximum transmission unit (MTU) size of the underlying data link. For example, a 6,500-byte packet travelling along an Ethernet network with a MTU of 1,500 bytes will be fragmented into a number of packets, each less than 1,500 bytes in size. When the "Don't fragment" flag is set, the packet cannot be fragmented, and will be dropped and an error message returned if fragmentation becomes necessary. When a router fragments a packet, it sets the "More fragments" bit to 1 in all but the last fragment, so that the receiver knows when the last fragment arrives. The fragment offset field encodes the information necessary to reassemble the fragments in the correct order.

7.4.1.2 Time-to-live

The time-to-live field is set with a certain fixed number (generally 64, but values such as 15 and 32 are often seen) when the packet is first generated. As the packet makes its way through the routers in an inter-network, this number is decremented. If the number reaches zero, the packet is discarded and an error message is returned to the source. This prevents "zombie" packets that are unable to find their destinations from roaming the inter-network and feasting on the packets of the living.

7.4.1.3 Protocols

The protocol field describes the type of data being transported. There are more than 100 assigned protocol numbers, but a few of the more common ones are noted in Table 7-9.

Table 7-9. Common IP protocol numbers

Protocol number

Description

1

Internet Control Message Protocol (ICMP)

6

Transmission Control Protocol (TCP)

17

User Datagram Protocol (UDP)

88

Internet Gateway Routing Protocol (IGRP)

89

Open Shortest Path First (OSPF, a routing protocol)

The header checksum contains the checksum for the header, obviously. However, this field must be recalculated at every router, because the time-to-live will have been decremented. The source and destination addresses are the 32-bit IP addresses.

7.4.1.4 IP addresses

The IP address is a 32-bit binary value that consists of a network portion, which uniquely identifies a network, and a host portion, which uniquely identifies a host on that network. Because it is cumbersome to write down 32-bit binary values, IP addresses are often represented in "dotted decimal" notation, in which each of the four 1-byte chunks are represented as a decimal number between 0 and 255. Dotted decimal notation is just a convienent way for humans to read IP addresses; all the network hardware ever sees is a 32-bit binary value.

One of the most interesting characteristics of an IP address is that the size of the network and host portions can vary depending on the size of the network. IP was designed to be very flexible, and can support huge networks with millions of hosts to small networks with only a few hosts. This makes IP addresses somewhat difficult to manage.

7.4.1.5 Classful addressing

The first scheme that developed to manage IP addresses involves dividing networks into three broad categories, or classes:

Class A network

Very large, with many hosts. There are few class A networks.

Class B network

Of moderate size, with a moderate number of hosts. Class B networks are much more common than class A networks.

Class C network

Very small (less than 250 attached hosts). Class C networks are common.

As a result, the network and host portions are split up according to Table 7-10.

Table 7-10. Network class summary

Class

Network portion (in bytes)

Available networks[17]

Host portion(in bytes)

Available hosts per network

A

1

127

3

16,777,216

B

2

32,767

2

65,536

C

3

8,388,607

1

256

[17] These are not quite what you would expect, because a portion of the class A address space is used for class B and C addresses.

An IP address in which the host portion is set to all zeros refers to the network as a whole, and is called the network address. Determining what network a host belongs to is the role of the address mask, a 32-bit binary number with the bytes that refer to the network portion all set to 1. Computing the bitwise AND of any IP address with its address mask will always return the network address. For an example, see Table 7-11.

Table 7-11. IP addresses, network masks, and network addresses on a class B network

Dotted decimal

1st byte

2nd byte

3rd byte

4th byte

Host IP address

176.16.146.38

10101100

00010000

10010010

00100110

Network mask

255.255.0.0

11111111

11111111

00000000

00000000

Network address(Bitwise AND)

176.16.0.0

10101100

00010000

00000000

00000000

Public Versus Private Address Space

Some IP addresses are reserved, per RFC 1918, for "private" use; that is, they may never appear on a public network. The reserved addresses are 10.0.0.0-10.255.255.255, 172.16.0.0-172.31.255.255, and 192.168.0.0-192.168.255.255.

Private addresses are often used on IP networks that don't participate in the Internet, such as point-to-point interconnects in a datacenter, or in networks that are "hidden" from the public Internet by means of a firewall or a Network Address Translation (NAT) mechanism.

7.4.1.6 Subnetting classful networks

Subnetworking can be a complex topic; it's something most people don't understand very well. Before we get started, then, a word of advice.

When working with subnets, it pays to work in binary. Dotted decimal notation exists merely for human convenience; the machines think in binary.

In order to improve the efficiency of assigning IP addresses, a concept called subnetting evolved. Subnetting involves breaking up a single large network into multiple smaller networks through extending the network portion by some amount less than a full byte. That is, the network portion intrudes into the host portion.

For example, let's say we wish to segment the 172.16.0.0 class B network. If we extend the network mask by 8 bits, that is, from 2 bytes to 3 bytes, we are now able to assign up to 256 subnets underneath the single class B address. Let's look at Chapter 7 to see how this looks in binary.

Table 7-12. Subnetting a class B network

Dotted decimal

1st byte

2nd byte

3rd byte

4th byte

Host IP address

176.16.146.38

10101100

00010000

10010010

00100110

Subnet mask

255.255.255.0

11111111

11111111

11111111

00000000

Subnet address(Bitwise AND)

176.16.146.0

10101100

00010000

10010010

00000000

We say that the extra eight bits in the subnet mask are the subnet portion, in complement to the host and network portions that we've already defined. Note that the network address is still 176.16.0.0!

Subnet masks are represented in three ways: the dotted decimal form, as is shown in Table 7-12; the hexadecimal form, which is simply the 32-bit binary value converted to a hexadecimal value (e.g., in our example in Table 7-12, the subnet mask in hexadecimal would be 0xFFFFFF00); and the bitcount form, which is the network address followed by a slash and the number of bits used for the subnet mask (e.g., continuing with the example given, 172.16.0.0/24).

If you need to design the addressing for a subnet, there are a few things you must keep in mind:

  • There are two "unusable" IP addresses in every subnet: the subnet address itself and the broadcast address.

  • The number of usable subnets in a network for a given subnet mask is given by 2ls - 2, where ls is the length of the subnet portion in bits.

  • The number of usable host addresses in a subnet is given by 2lh - 2, where lh is the length of the host portion in bits.

Our subnet example deals with a case where the subnet space falls on a byte boundary. This is not always the case -- nor even advantageous. Let's say you have the 192.168.144.0 class C network, and you need to support 6 departments, each with 25 systems. This is easily done, but only if the subnet portion is 3 bits long (23 - 2 gives us 6 possible subnets -- hope we don't add another department!). This leaves us with 5 bits for each subnetwork's host portion, or 30 hosts per network. How do we find the subnet mask for this division? Let's look at Table 7-13.

Table 7-13. Subnetting a class C network

Dotted decimal

1st byte

2nd byte

3rd byte

4th byte

Network address

192.168.144.0

11000000

10101000

10010000

00000000

Subnet mask

255.255.255.224

11111111

11111111

11111111

11100000

With three bits for the subnet portion, we can write down all the possible combinations: 000, 001, 010, 011, 100, 101, 110, and 111. We also know that the first and last addresses, 000 and 111, cannot be used. By calculating all the possible combinations for the five bits of the host portion of the address, we can work out the range of allowed host addresses for that subnet. For a summary of these calculations, see Table 7-14.

Table 7-14. Computing address ranges for a subnetted class C network

Subnet

Network address

Broadcast address

Host address range

001

192.168.144.32

192.168.144.63

192.168.144.33-192.168.144.62

010

192.168.144.64

192.168.144.95

192.168.144.65-192.168.144.94

011

192.168.144.96

192.168.144.127

192.168.144.97-192.168.144.126

100

192.168.144.128

192.168.144.159

192.168.144.129-192.168.144.158

101

192.168.144.160

192.168.144.191

192.168.144.161-192.168.144.190

110

192.168.144.192

192.168.144.223

192.168.144.193-192.168.144.222

The reason for the advice I gave when we started this adventure should now be (perhaps painfully) clear: when you are given an address, you have no idea just by looking at it whether it is a subnet address, a broadcast address, or a host address, and even knowing the subnet mask does not make things obvious. The key to working with subnets is to work in binary.

7.4.1.7 Moving to a classless world

By about 1990, the Internet was faced with two serious growth problems. As the Internet grew in popularity, there was a flood of new classful networks, and every one had to be included in the routing tables. As a result, the routers were starting to run out of memory, and spending far too much time doing address lookups. It had also become apparent that the pace of requests for new class B networks would soon exhaust the available supply. The Internet Engineering Task Force (IETF) devised a solution, which is known as classless interdomain routing (CIDR), simply classless routing, or supernetting.

CIDR is based on an extension of the principle of subnetting that we've already discussed. Supernetting involves moving the boundary between the host and network portion to the left, not just the right. Groups of neighboring classful networks can be combined into single routing table entries. Groups of class C networks can be assigned in batches of 2, 4, 8, or 16 to fill the needs of organizations that would otherwise have requested a class B network.

Instead of a subnet mask to distinguish between the network portion and the host portion of the address, classless IP addressing uses a length indication. This length indication, which consists of a slash and a number that follows the network address, is known as a variable length subnet mask, or VLSM. The following number specifies how many bits make up the network portion of the address. For example, 192.168.214.0/21 specifies 21 bits for the network portion. Using the standard subnet mask notation, this is equivalent to 255.255.248.0.

The big benefit of classless addressing is that it improves the efficiency of allocating IP addresses. For example, if an organization is assigned a class B address, it has 16,384 addresses available for use. If that organization has only 1,000 hosts, the remainder of the addresses are reserved but remain unused. In a classless world, the organization could simply be assigned one network with a VLSM of 22, which would provide it with 1024 addresses (the equivalent of 4 "aggregated" class C networks). The equivalent netmask would be 255.255.252.0.

7.4.1.8 Routing

One thing that we will not discuss here is the means by which routers discover and share information with each other, facilitating communications across IP-based internetworks. If you are interested in learning more about that, there is an excellent (if voluminous) reference, Routing TCP/IP by Jeff Doyle, published by Cisco Press.

7.4.2 TCP

The Transmission Control Protocol, or TCP, describes a connection-oriented, reliable communications protocol; it provides the appearance of a point-to-point connection. Such a connection has two characteristics: there is only one route from the source to the destination, and packets arrive in the same order that they are sent. However, because the underlying delivery mechanism (IP) is connectionless, TCP can only provide the appearance of a connection. Just like a telephone connection, TCP must establish a connection, transfer data, then disconnect when the data transmission is complete. TCP uses three mechanisms to accomplish this:

  • The first byte of each packet is labeled with a sequence number, so that the receiving service can reassemble out-of-order packets correctly.

  • There is a system of acknowledgments, checksums, and timers that provide reliability. For example, the receiving end can notify the sender that a packet has failed to arrive, and the sender will resend any packet that is not acknowledged by the receiving end within a certain amount of time.

  • There is a mechanism to control the flow of packets, called windowing. This decreases the chances of a packet being dropped because the received-packet buffer has overflowed.

TCP attaches a header to the data being sent, which contains fields for the source and destination address of the application (ports) as well as all the TCP control data. The data plus the TCP header is generally then encapsulated into an IP packet for delivery. The TCP header is rather complex, and is summarized in Table 7-15.

Table 7-15. Transmission Control Protocol header

Bits

Content

0-15

Source port

16-31

Destination port

32-63

Sequence number

64-95

Acknowledgment number

96-99

Header length

100-105

Reserved

106

Flag: Urgent (URG)

107

Flag: Acknowledgment (ACK)

108

Flag: Push (PSH)

109

Flag: Reset (RST)

110

Flag: Synchronize (SYN)

111

Flag: Final (FIN)

112-127

Window size

128-143

Checksum

144-159

Urgent pointer

160-191+

Options and padding

The sequence number identifies the position of the encapsulated data byte within the stream. For example, if a segment has a sequence number of 5,748 and contains 256 bytes of data, the next segment will have a sequence number of 6,004. This "expected next sequence number" value is reported in the acknowledgment number field, so that the sending host knows not only whether packets have been lost, but which ones as well.

The header length field specifies the length of the header in 32-bit increments. The reserved field is always set to six zeroes. The window size is used for flow control by specifying the number of bytes, starting with the byte indicated by the acknowledgment number that the sender of the packet will accept from the other end of the connection before requiring an acknowledgment.

The checksum is a 16-bit checksum of both the header and the encapsulated data for error correction. If the urgent flag is set, the urgent pointer field is used; when added to the sequence number, it specifies the end of the urgent data. The options field contains data specified by the application; the most common use of this field is to send the maximum segment size, which tells the receiver the largest segment size the sender is willing to accept. The remainder of the field is padded with zeroes to make an even 32-bit length.

Viewing and Setting TCP Tunables: ndd

ndd is a tool that allows you to view and set tunables on Solaris systems without rebooting. While it can be used on most devices that have a device file in /dev and a kernel module, we will use it primarily to tune TCP.

You can use ndd interactively:

# ndd /dev/tcp

name to get/set ? tcp_slow_start_initial

value ?

length ?

1

name to get/set ? ^D

You can also use it to display all available parameters:

# ndd /dev/icmp \?

?                             (read only)

icmp_wroff_extra              (read and write)

icmp_ipv4_ttl                 (read and write)

icmp_ipv6_hoplimit            (read and write)

icmp_bsd_compat               (read and write)

icmp_xmit_hiwat               (read and write)

icmp_xmit_lowat               (read and write)

icmp_recv_hiway               (read and write)

icmp_max_buf                  (read only)

icmp_status                   (read only)

You can also query many parameters at once:

# ndd -get /dev/hme link_status link_speed link_mode

1

0

0

And, finally, you can use it to set tunables:

# ndd -set /dev/tcp tcp_slow_start_initial 2

In Linux, the TCP tunables are generally adjusted permanently by means of editing the kernel source code and recompiling. They can also be adjusted at runtime by editing the files in /proc/sys/net/ipv4.

Please note that in our further discussions of TCP, I will provide details about the protocol's inner workings. Those of you with experience in this area will immediately realize that I am simplifying some things -- TCP is a complex protocol, and a full treatment is well beyond the scope of this text. The best reference for TCP is, by far, W. Richard Stevens's TCP/IP Illustrated, Volume 1 (Addison Wesley). It is simply outstanding in explaining how TCP works, and I strongly suggest you read it.

7.4.2.1 Connection initiation and SYN flooding

TCP defines a specific way of creating a new TCP connection, called the three-way handshake. There are many queues and tunable values that are used during this instantiation process. In order to tune effectively, you need to understand not only what is happening on the wire, but also what is happening in the system; namely, how listen( ) and accept( ) interact with the queues.

When the server calls listen( ), the kernel moves the socket from the TCP state CLOSED into the LISTEN state. Concurrently, the kernel creates and initalizes various data structures, including the socket buffers and two queues, the incomplete connection queue, and the completed connection queue.

When the first packet in a TCP connection is sent from the server (in which only the SYN bit is set), then an entry is added to the incomplete connection queue. The server sends a packet back to the client, in which the ACK bit is set (to signal acknowledgment of the client's SYN) as well as the SYN bit. The socket is now in the SYN_RCVD state. When the client's ACK of the server's SYN is received, the connection stays in this queue for one round-trip time, and then is moved to the completed connection queue.

The completed connection queue contains an entry for every active connection (one in which the three-way handshake has been successfully completed, but the user application has not yet accepted the connection). The sockets in the completed connection queue are in the ESTABLISHED state. Every call to accept( ) removes the front entry from the queue; if there are no entries, then the call to accept( ) usually blocks. There is a parameter, tcp_conn_req_min, which defines the minimum number of available connections in the completed connection queue in order for select( ) or poll( ) to return "readable" to a listening socket descriptor. You shouldn't ever need to touch this tunable.

Both of the queues I described are limited in size. When listen( ) is called, the server can specify the size of the second queue. Historically, this value has defined the number of entries for the sum of both queues. Solaris systems since 2.6 include a "fudge factor" (multiplying the argument by three-halves). The incomplete connection queue typically requires more entries than the completed connection queue; the only reason to specify a large backlog value is to enable the incomplete connection queue to grow as SYNs arrive from clients. Solaris has two kernel tunables that set the size of these lists: tcp_conn_req_max_q0 is the maximum number of connections with incomplete handshakes (the incomplete connection queue), whereas tcp_conn_req_max_q is the maximum number of completed connections awaiting to return from an accept( ) call (the completed connection queue).

The default values for tcp_conn_req_max_q0 and tcp_conn_req_max_q are 1,024 and 128, respectively. You can inspect the value of tcpListenDrop and tcpListenDropQ0 in the output of the netstat -sP tcp command. If you get many drops, you may need to increase these tunables. tcp_conn_req_max_q0 probably shouldn't be set above about 10,000; tcp_conn_req_max_q's reasonable upper bound is the current value of tcp_conn_req_max_q0.

If you're working under Linux, the equivalent to the tcp_conn_req_max_q0 parameter is tcp_max_syn_backlog, which can be found in /proc/sys/net/ipv4/.

Solaris versions prior to 2.5.1 (with patches 103630-09 and 103582-12 or above applied) have a single, unified connection queue that is governed by tcp_conn_req_max. While you can tune this parameter, the recommended course of action is to upgrade or apply the current version of the two TCP patches to your system.

A certain type of denial-of-service attack, called SYN flooding, involves sending a large number of SYN packets with nonexistent source addresses. Because the second SYN is never acknowledged, the listen queue fills up and new connections get through only as old ones time out and are discarded from the queue. Whenever a dubious connection is discarded, the tcpHalfOpenDrop counter is incremented; a high value indicates that a SYN flood was likely attempted. If you observe this behavior, you can improve your protection by increasing tcp_conn_req_max_q0.

7.4.2.2 Path MTU discovery and the maximum segment size

During the three-way handshake, the value for the maximum segment size (MSS) over that connection is determined. This value defines the largest amount of information that will be sent in a single TCP packet. The MSS is set to the minimum of the smallest MTU (message transfer unit, the largest amount of data that can be sent in one packet on a given interface) of an outgoing interface or the MSS announced by the other end of the session. If the remote peer does not announce an MSS, the value 536 will normally be assumed. If path MTU discovery is active, all outgoing packets have the IP option DF set (for "don't fragment").

All versions of Solaris set the default MSS to 536 bytes. This is not always what you want, and can be remedied by adjusting the tcp_mss_def_ipv4 parameter:[18]

[18] This parameter is called tcp_mss_def in pre-8 versions of Solaris.

# ndd -set /dev/tcp tcp_mss_def_ipv4 1460

The value to which the MSS should be set is determined by the MTU of the most commonly used outgoing interface, less 40 bytes for the TCP/IP headers. Since Ethernet has a MTU of 1,500 bytes, tcp_mss_def_ipv4 should be set to 1,460. Note that if the other side of the connection demands a lower MSS, that request will be honored. You shouldn't have to tune this under Linux.

If the TCP stack receives the ICMP error "Message fragmentation needed," it means a router on the way to the destination had to fragment the message, but was not allowed to do so (likely because the IP DF bit was set). As a result, the router had to throw the packet away, and sent back the ICMP error to notify the sender. Most newer routers actually transmit the needed MSS back to the host, but if the needed MSS isn't included, the correct MSS must be determined by trial and error.

RFC 1191 recommends rediscovering the path MTU of a TCP circuit every ten minutes. Unfortunately, early versions of Solaris (pre-2.5) try to rediscover the path MTU every 30 seconds. This is acceptable behaviour in LAN environments, but it is very rude for wide area networks, as it unnecessarily consumes the network capacity. The tunable is ip_ire_pathmtu_interval, which should be set to 600,000. You can control path MTU discovery in Solaris by means of the ip_path_mtu_discovery tunable, which you should leave set to 1. In Linux, it is set by the ip_no_pmtu_disc tunable in /proc/sys/net/ipv4/, which you should leave set to 0.

7.4.2.3 Buffers, watermarks, and windows

In order to understand the various buffers and watermarks in a modern TCP stack, we need to briefly discuss what happens when you send a piece of data through the various layers of the stack. This section provides a simplified explanation in order to ease understanding. An application can send almost any size of data to the socket buffers, which reside in the transport layer -- in this case, TCP. All of the sent application data will be copied to the socket buffer. If there is an insufficient amount of data, the application will be put to sleep. The TCP module then segments the contents of this socket buffer, with no segment exceeding the MSS. Data is only removed from the socket buffer when the data is acknowledged by the remote peer. These segments are then passed down to the IP layer, where they are fragmented if they are too large for the underlying physical link interface (larger than the MTU).

The kernel-wide tunable for the size of a specific protocol's receive buffer is the receive high watermark. In Solaris, these are given by the tcp_recv_hiwat and udp_recv_hiwat tunables, or by calling setsockopt( ) with the SO_RCVBUF parameter. By default, both the tcp_recv_hiwat and udp_recv_hiwat tunables are 8,192 before Solaris 8, and 24,576 in Solaris 8. I would set these to something in the range of 16 KB to 32 KB for most applications, but it may be necessary to experiment in your particular environment. Also, keep in mind that this parameter can strongly influence memory usage: if you have 512 concurrent connections, and both the reception and transmission high watermarks are set to 64 KB, you will need to have 64 MB of memory for your TCP socket buffers alone. A related parameter is tcp_recv_hiwat_minmss, which is the number of maximum-sized segments that must be able to fit into a receive buffer; it's not possible to set a receive buffer size that can't fit at least this many segments. By default, this parameter is 4, and should be left alone.

The transmit high watermark, tcp_xmit_hiwat, effectively determines the size of the send buffer.[19] This can be set on a per-socket basis by the socket option SO_SNDBUF. When the amount of data in the send buffer is greater than tcp_xmit_lowat but less than tcp_xmit_hiwat, it is reported as nonwriteable. The default for tcp_xmit_hiwat is 8,192, and you should set it to something in the neighborhood of 32 KB to 48 KB. Since the transmit buffer has to store all unacknowledged segments, it's reasonable to make it larger than the receive buffer.

[19] UDP effectively has no transmit buffering.

As a word of warning, setting this watermark near 65,535 may have the side effect of turning on the TCP window scaling option, because these values are rounded up to multiples of the MTU for each connection. In some cases, you may not want to use window scaling. In order to avoid accidentally using it, you should make sure that tcp_recv_hiwat and tcp_xmit_hiwat are set to at least 1 MTU below 65,535. For Ethernet interfaces, 64,000 is a good choice.

There are a set of low watermarks, tcp_xmit_lowat and udp_xmit_lowat, which you should not need to change. Applications can use the SO_SNDLOWAT and SO_RCVLOWAT to change these if necessary.

The tcp_max_buf and udp_max_buf parameters specify the maximum buffer size that can be requested by a setsockopt( ) call. By default, these values are 1,048,576 and 262,144, respectively. An attempt to specify a larger buffer size will fail with an EINVAL return code.

On pre-2.4 Linux systems, there are equivalent interfaces to change located in /proc/sys/net/core. These are rmem_max, rmem_default, wmem_max, and wmem_default. By default, they are all set to 65,536. Starting with 2.4, there is a new socket buffer autosizing algorithm, loosely controlled by a set of TCP-specific parameters, tcp_rmem and tcp_wmem, located in /proc/sys/net/ipv4/. Each of these files contains three values: the minimum, the default, and the maximum size allowed for each buffer.

  • The minimum size for each buffer is guaranteed to each TCP socket, even when the system is memory-constrained. By default, it is set to 8 KB.

  • The default size for each buffer overrides the rmem_default value in /proc/sys/net/core/, which is still used for non-TCP protocols. The default is 87,380 bytes, which results in a TCP window size of 65,535 bytes.

  • The maximum specified size of the receive buffers does not override the value set by rmem_max in /proc/sys/net/core/. Using setsockopt( ) with SO_RCVBUF does, however. The default is 174,760 bytes.

The TCP window is closely related to the socket buffer. A TCP window is the amount of outstanding data (unacknowledged by the recipient) that can be sent on a particular connection before the sending stack receives an acknowledgment from the receiver. For example, if two hosts are communicating over a TCP connection that has a window size of 64 KB, the sender can only send 64 KB of data; then it must stop and wait for an acknowledgment of some, or all, of the data. If the receiver acknowledges that all the data has been received, the sender is free to transmit another 64 KB. If the receiver acknowledges only the first 32 KB (for example, if the last 32 KB hasn't gotten there yet, or has been lost in transit) then the sender can only transmit another 32 KB. The primary intent of the window is for congestion control. The entire end-to-end network connection, which encompasses both hosts, all the intermediate routers, and all the intermediate connections (be they fiber, copper, satellite-based relay, etc.), can only handle data at some peak speed. If the transmitting occurs too fast, then the bottleneck is surpassed, and some data will be lost. The TCP window serves to "throttle down" the transmission speed to a level where congestion and the accompanying data loss don't occur. The TCP header specifies a 16-bit field to convey the window size. Therefore, the maximum window size is 216, or 65,536 bytes.

The fastest bulk data transfer rate can be found by dividing the window size your peer offers by the round-trip time. The optimum value for this parameter is called the bandwidth-delay product; that is, how fast your connection can run multiplied by the time it takes to send a data packet out and receive an acknowledgment back.[20] This makes intuitive sense: the lower the round trip time or the larger the window, the faster you can transmit. For example, if you are using a 10 Mb/s (~1.0 MB/s) Ethernet link and you observe an average latency of 6 ms to your most frequently visited hosts, then the product is 6 KB and the default value of 8 KB is sufficient. If the latency increases past 8 ms, however, increasing the buffer size may be helpful. This sort of tuning is particularly useful over slow links such as ISDN lines; with a latency of 40 ms and a 12 KB/s data rate, our buffers should be 48 KB! TCP window tuning really needs to be done on the client end, and is most effective on busy systems that take a long time to turn around a reply. Systems on high speed networks will probably also benefit.

[20] You can get a rough idea of the latency by using ping -s hostname.

In order to work around the performance limitations imposed by a maximum window size of 64 KB over very high latency links, RFC 1323 introduced a technique known as window scaling. Window scaling expands the definition of the TCP window to 32 bits, and uses a scaling factor to carry this 32-bit value in the original 16-bit field. The scaling factor is carried in a new TCP option, Window Scale. This option is sent only in a segment when the SYN bit is sent, meaning window scaling is fixed in each direction when a connection is opened. This three-byte option accomplishes two things:

  • Indicates that TCP is prepared to do both send and receive window scaling.

  • Specifies the scale factor that should be applied to its receive window. A TCP stack that is prepared to scale its windows should send this option, even if its own scale factor is 1. The scale factor is a logarithmically encoded power of 2. Both sides must send Window Scale options in their SYN segments in order to enable window scaling in either direction.

The window scaling behavior is controlled in Solaris by means of the tcp_wscale_always TCP tunable, which is zero by default. If tcp_wscale_always is set to a nonzero value, then the TCP window scaling option will always be negotiated during connection initiation. Otherwise, window scaling is only used if the buffer size is larger than 64 KB. The Linux equivalent is tcp_window_scaling in /proc/sys/net/ipv4/.

7.4.2.4 Retransmissions

Sometimes data is lost on the network. The reasons for data loss are many: perhaps a backhoe cut a buried network cable, a building service worker unplugged a router by accident, some piece of network hardware failed, etc. The exact reasons for data loss aren't important; what we will consider are the mechanisms TCP uses to control retransmission of data.

The most basic tunable for this behavior is tcp_rexmit_interval_initial, which specifies how long to wait, in milliseconds, before the last data sent but not acknowledged is retransmitted. The default value of this parameter since Solaris 2.5.1 has been 3,000, which is a reasonable length of time for general use; laboratory environments not connected to the Internet might do better with a shorter time (say, 400 ms).

After the initial retransmission, further retransmissions will start after tcp_rexmit_interval_min milliseconds have elapsed. Since Solaris 8, this value defaults to 400; in earlier releases, it was 200. It should probably be set to roughly half of the tcp_rexmit_interval_initial value.

Further retransmissions follow an exponential backoff algorithm, but the inter-retransmission delay is capped at tcp_rexmit_interval_max milliseconds. Since 2.6, the default has been 240,000 (which is compliant with RFC 1122); Solaris 8 reduced this time to 60,000.

Two related values are tcp_ip_abort_interval and tcp_ip_abort_cinterval. These specify the time, in milliseconds, that retransmissions should be attempted until the RST segment is set and the TCP connection is reset. The tcp_ip_abort_cinterval parameter is specific to connections that have not yet reached the ESTABLISHED state (that is, they are still in the process of handshaking). The defaults for these parameters are 480,000 and 180,000, respectively. tcp_ip_abort_interval should probably be increased to about 600,000.

7.4.2.5 Deferring acknowledgments

One very useful set of tunables on Solaris systems are those that govern deferring acknowledgments. These parameters interact to control when acknowledgments are sent by the receiver for data transmissions on a local area network. The first parameter is tcp_deferred_ack_interval, which is measured in milliseconds; the second is tcp_deferred_acks_max, which defines the maximum number of segments that can be received before an ACK is immediately sent.

In order to understand how these parameters interact, let's look at an example. When the first inbound packet is received, two things happen: the deferred acknowledgment timer is started and the packet count is increased. As more packets come in, there are two possible outcomes. One is that the packet count becomes equal to tcp_deferred_acks_max, causing an acknowledgment to be sent, the timer to be reset, and the packet count to be set to zero. The other outcome is that the deferred acknowledgment timer fires (when it is equal to tcp_deferred_ack_interval). This causes an acknowledgment to be sent, even if the number of accumulated packets is less than tcp_deferred_acks_max, and thereafter deferred acknowledgments will be disabled, and acknowledgments will be sent every other packet (which is how the TCP stack functions when talking to hosts farther away than the local area network).[21] By default, tcp_deferred_ack_interval is 100 milliseconds in Solaris 8; it was 50 milliseconds before that. The default for tcp_deferred_ack_max is 8 packets. This behavior is compliant with RFC 1122, which discusses the slow start algorithm.

[21] This is done to avoid causing problems with systems that are following the slow-start algorithm.

7.4.2.6 Window congestion and the slow start algorithm

When the connection is negotiated, the latency and bandwidth available to the connection are not clear. To avoid flooding a slow network, TCP implements a slow start algorithm, which limits the number of packets it sends until it sees an acknowledgment. This limit is called the congestion window, and the standard says that the initial congestion window should be one packet, and double each time an acknowledgment is received.[22] There are two defects in most TCP implementations that relate to the slow start algorithm:

[22] This does not work well for short connections because there are not enough packets sent to get the window size increased.

  • Most TCP implementations start with a default congestion window of 2. This is so widely done that changing the tcp_slow_start_initial variable from 1 to 2 in Solaris releases prior to 7 is a good idea; it (slightly) increases transmission speed on lightly loaded networks. Solaris releases after 7 have this configured by default.

  • A related problem occurs in the Microsoft TCP implementation: it does not immediately acknowledge receipt of a single packet, instead waiting for two or waiting for a short delay period. After receiving two packets, they acknowledge immediately. This causes higher response times than normal to be seen over high speed links; the latency in wide area networks or low speed links hides this problem. However, changing the tcp_slow_start_initial variable as described solves the problem. You can't tune this behavior in Linux.

7.4.2.7 TCP timers and intervals

There are three other interesting timers at play in TCP. These are the keepalive timer, the TIME_WAIT timer, and the FIN_WAIT_2 timer.

The keepalive timer is one of the more contentious issues in TCP stack tuning. The intent of the keepalive system is to free host resources by scavenging TCP connections in which the remote host is not responding. This comes at a cost in network utilization. The most common use of the keepalive timer system is in web servers. Recall that most TCP transactions follow a simple, four-step negotiation process:

  1. The HTTP client opens a connection to the HTTP server.

  2. The client forwards its query (requests a file).

  3. The server responds with the file (responds to the request).

  4. The server closes the connection.

If the client system or browser crashes, hangs, or is otherwise stopped prior to issuing its request, the server is forced to keep the connection open indefinitely. If you are running a busy web server, you should probably set this value to some reasonably small value, on the order of 5 to 10 minutes. It is controllable by the tcp_keepalive_interval parameter in Solaris. Linux actually implements three parameters to control this behavior, all of which reside in /proc/sys/net/ipv4. tcp_keepalive_time specifies how often TCP sends out keepalive messages when keepalive is enabled (by default, every two hours). The tcp_keepalive_probes tunable specifies how many probes TCP will send out until it decides that the connection is broken (by default, nine). Finally, tcp_keepalive_interval specifies how frequently the probes are dispatched; when multiplied by tcp_keepalive_probes, it describes how long a nonresponsive connection is allowed to linger before being killed. tcp_keepalive_interval has a default value of 75 seconds.

The TCP TIME_WAIT state has a negative reputation, and people often frantically try to avoid it. It is actually a very good thing. A TCP connection is assigned to the TIME_WAIT state after it is closed, but before it is reused. The maximum segment lifetime (MSL) is the maximum time that a TCP segment may survive in the network. Waiting twice this time ensures that there are no leftover segments coming in, and it is safe to reuse the socket resource. This is the reasoning behind the TIME_WAIT state. You can probably safely reduce the length of time a socket waits in the TIME_WAIT state. A good starting point is twice the round-trip time to an extremely remote site: 60,000 or so is reasonable. You can directly tune this in Solaris by means of the tcp_time_wait_interval tunable, which is in units of milliseconds (note that the tcp_time_wait_interval parameter used to be called tcp_close_wait_interval in releases of Solaris before 7). Linux systems cannot tune this parameter directly, as they implement a fast TCP TIME_WAIT recycling algorithm already (governed by the tcp_tw_recycle parameter, which you shouldn't change). However, Linux does implement a tcp_max_tw_buckets parameter, which is the maximum number of TIME_WAIT sockets held by the kernel at any one time. If this number is exceeded, enough TIME_WAIT sockets to get underneath the limit are immediately destroyed and a warning is printed. This is intended primarily as an attempt to prevent simple denial-of-service attacks; never lower this limit.

The FIN_WAIT_2 timer is used when a connection is being closed. FIN_WAIT_2 is a TCP state that occurs when a connection is actively closed, but the FIN from the other side hasn't arrived yet -- and it might not ever arrive. A crashed or misbehaving client may cause the connection to reside in FIN_WAIT_2 for a long time: until the FIN_WAIT_2 timer expires. If netstat -f inet (netstat inet in Linux) shows many connections in the FIN_WAIT_2 state, you should probably decrease the value for this timer; it is governed by the tcp_fin_wait_2_flush_interval TCP tunable in Solaris, or the /proc/sys/net/ipv4 Linux tunable tcp_fin_timeout. The Solaris default is 675,000 ms, and the Linux value as of the 2.4 kernels is 60 seconds (it was 180 seconds in 2.2). Solaris systems should probably tune the value down to 67,500 (one order of magnitude less).

7.4.2.8 The Nagle algorithm

The Nagle algorithm is a means of trading latency for throughput by aggregating small packets. It states that, under some circumstances, there will be a waiting period of 200 milliseconds before data is sent. This algorithm was published in RFC 896 in an attempt to solve a simple problem involving interactive connections such as telnet; every interactive keystroke across a telnet connection normally generates a single data packet. Furthermore, the remote telnet server is responsible for echoing the character back. This generates up to four segments (the transmission of the keystroke, the transmission of the echoed keystroke, and two acknowledgment packets). These small packets, called tinygrams, are not normally a problem on LANs, but they can contribute to severe congestion on WANs. In essence, the Nagle algorithm serves to collect small amounts of data and send them in a single segment. The algorithm has the further advantage of being self-clocking: the faster acknowledgments come back, the faster the data is sent.

The following variables are used to determine when data should be sent in the absence of an acknowledgment:

  • If there are no unacknowledged small packets, send this one immediately.

  • If the TCP window is not full and the packet size is equal to or larger than the maximum segment size, send an MTU-sized buffer immediately.

  • If the TCP window is not full, and either the interface is idle or the TCP_NODELAY flag is set, send the buffer immediately.

  • If there is less than half the TCP window in outstanding data, send the data immediately.

  • Otherwise, wait up to 200 milliseconds for more data to be sent before sending the buffer.

When an acknowledgment arrives, the Nagle buffer is flushed immediately (notice that we have now sent a number of MTU-sized packets and, perhaps, one small packet as well).

If the Nagle algorithm interacts very poorly with the acknowledgment timer (unless the small packet is following a full packet, in which case we could expect things to work properly), the Nagle algorithm won't be able to send more data. This causes severe performance problems.

For many (but certainly not all) workloads, the Nagle algorithm should probably be disabled, by setting the tcp_naglim_def variable to 1. Applications that want to disable the Nagle algorithm on a particular socket should use the NODELAY flag when setting up the socket.

Tuning TCP parameters carries with it the possibility of causing grief to everyone on your network, not just yourself. Please be careful.

7.4.3 UDP

The User Datagram Protocol, or UDP, is a connectionless, "best effort" protocol. In light of the high reliability afforded by TCP, it can be hard to imagine why anyone would wish to use UDP. The advantage is that rather than taking the effort to set up and tear down a connection required in TCP, the data is just sent in UDP. This results in very low latencies, and applications that send data in short bursts over reliable networks will see a performance advantage.

Another reason for UDP's better performance is that it uses a small header (only 64 bits). See Table 7-16.

Table 7-16. Universal Datagram Protocol header

Bits

Content

0-15

Source port

16-31

Destination port

32-47

UDP length

58-64

Checksum

The source and destination ports are identical to those discussed in the TCP header. The UDP length is the length of the entire segment in bytes, and the checksum is an optional field that contains the checksum for the entire segment.

As UDP is a much simpler protocol, it does not have nearly as many tunable parameters. The most important are the size of the transmission and reception buffers, and are discussed along with their TCP counterparts; see Section 7.4.2.3 earlier in this chapter.

7.4.4 TCP Versus UDP for Network Transport

One common conundrum is whether TCP or UDP should be used as the network transport for a specific application that offers a choice (e.g., NFS). In general, there are three key differences to take into consideration:

  • UDP provides higher performance, in theory.[23]

    [23] The truth is that most vendors have spent their time tuning their TCP implementations, so you will need to conduct experiments to measure the actual difference in your specific environment.

  • TCP has higher overhead, but also guarantees reliability, which UDP cannot.

  • TCP controls transmission costs in error-prone environments very well, e.g., over wide area networks.

In general, UDP can provide a significant performance increase over fast, well-maintained, switched local area networks (for example, in the context of a machine room), but this is highly dependent on your vendor's particular implementation, and the only way to know is to test.


7.5 NFS

The Network Filesystem (NFS), first developed by Sun Microsystems, has become a ubiquitous part of the computing landscape, in large part because of its simplicity and availability on most platforms. There are two flavors of NFS: Version 2, which has been in use since 1985, and Version 3, which was introduced in 1993. Most modern NFS implementations support both versions, with the newer version preferred if both the server and client are capable of using it. The most important concept in NFS is that the protocol is stateless; the functionality is defined such that the client and server need not maintain a continuous connection. The client simply submits a request, and the server processes it and responds. The server may have crashed or been shut down between the request and the response, and neither side need worry.

NFS works within the context of several other services, namely a file lock manager, a mechanism to start service (the mount protocol), and an automated name-lookup mounter. File locking poses an interesting problem for a stateless file sharing protocol, because locking involves state implicitly. To work around this, locking is implemented by the Network Lock Manager protocol (NLM), which relies on a related protocol implemented in rpc.statd. A list of all locked files is maintained on the server, which is also responsible for pushing the lock information out to its clients. Because NFS is stateless, the server and client need a mechanism to determine the other's state in order to know when to reacquire a lock (e.g., when the server is rebooted) and when to invalidate a lock (e.g., when the client unmounts the filesystem); this is the role played by statd.

The mount protocol itself is not very interesting, except to note that all negotiation is done at mount time; that is, the block size, the local attribute management, the version of the protocol to use, and the transport protocol are all determined when the remote filesystem is mounted.

The automounter is a client-side application overlaid on the mount protocol. Remote filesystems are assigned locations within the filesystem, and all references to those locations are trapped by the automounter. If the remote filesystem is not mounted, the automounter finds the remote filesystem and mounts it in the appropriate location. After a period of inactivity, the automounter unmounts the filesystem. When combined with universal naming services like NIS and NIS+, the automounter can help define a consistent filesystem across an enterprise. The automounter imposes no significant load on any component.

The Version 2 protocol is extremely simple, implementing only 18 operations. In 1993, due to customer pressures for technical features, the NFS protocol was revised to Version 3, making improvements in some key areas:

  • Write operations are much faster, using a two-stage commit protocol.

  • The number of packets actually crossing the network is reduced by permitting file attributes to be returned on every operation. Every operation that modifies a file's state returns the modified attributes.

  • The maximum file size was increased from 232 bytes to 264 bytes, and the maximum size of a data block was increased from 8 KB to 4 GB.[24]

    [24] The maximum size of the data block is usually determined by the underlying attributes of the network. For example, Solaris supports block sizes up to 64 KB, which is the largest block permitted by either the TCP or UDP implementation.

  • Much more sophisticated file access controls are supported (e.g., access control lists).

Although they are quite similar, the two versions of NFS are not directly interoperable. The client and server negotiate which protocol to use during the filesystem mounting process. Note that not all clients capable of Version 3 operation default like this, and some adjustments may be required on your platform.

The server reports the size of an exported filesystem only to allow the client to determine the amount of free space; this number is not relevant to manipulating files. Although the amount of free space may be reported as being very strange on the client (negative), file access will still work as expected. However, large files (bigger than 2 GB) are problematic on Version 2 servers because of an inherent addressability restriction in NFS Version 2.

You can select the version of the NFS protocol to use by specifying the vers=n option to the mount command, where n is either 2 or 3. In addition, either Version 2 or Version 3 may be run over TCP as well as UDP; this is specified by proto=tcp or proto=udp, respectively, as a switch to the mount command. A Version 2 mount will default to UDP, and a Version 3 mount will default to TCP.

7.5.1 Characterizing NFS Activity

In NFS Version 2, the bulk of operations consists of six functions: lookup, getattr, setattr, readlink, read, and write. These functions can be grouped into two categories: the first four manipulate the file's attributes (filename lookup, getting and retrieving attributes, and resolving symbolic links), and the last two involve reading and writing the contents of a file. These categories place very different loads on the server.

Attribute operations are lightweight. Because of their small size, most of the filesystem attributes will be cached in memory, and even if they are not, they are easily retrieved from disk. While the overhead of processing these requests is rather high (since the proportion of useful data bytes in the transmitted data is low), the overall small size of the packets leads to low bandwidth consumption.

However, data operations are a different story. In NFS Version 2, they are almost always the maximum size of 8 KB, and in Version 3 they can be much larger. In addition, while each file has only one set of attributes, it can have a great number of data blocks, most of which are not usually cached on the server. Because of the larger size of the operation, they also consume much more bandwidth.

NFS servers typically spend the vast majority of their time servicing attribute rather than data operations. When a client system wants to use a file on a remote server, it uses a series of lookup operations to find the file, a getattr operation to obtain the file's permissions mask, and finally issues a read to obtain the first block of data. For example, consider a read to a user's .forward file. It is common practice on systems with many users to sort home directories alphabetically, so for illustration's sake let's assume the file is located in /shared/home/u/us/user/.forward. To read the file, the client must be able to access each of the four directories exported to it. Verifying this means looking up each entry in the containing directory and obtaining the permissions mask to determine if the permissions are sufficient -- a total of four lookup operations.[25] The file itself must then be looked up, the permissions checked, and a single block read. We have caused six operations, only one of which was a data operation!

[25] The lookup operation causes attributes to be returned automatically for directories.

A scenario involving the access of many small files is attribute-intensive for reasons described previously. The classic example is a software development environment in which there are many small files, whether source code proper, header files, object files, or revision control files. Most home directory servers fit this pattern as well. Because NFS transactions block on the client, the performance of the system is dominated by the speed at which the server can handle the attribute requests. In a switched environment, 10 Mb/s Ethernet is usually acceptable for such a workload.

In a data-intensive environment where the average files are very large (e.g., image processing), the expense of transmitting data overcomes the attribute-processing time. When multiple clients are active on a repeated network, users begin to percieve sluggish performance at about 40% utilization; switched networks are immune to this phenomenon, although the uplink from the switch from the server can easily become very congested. Thankfully, NFS traffic is not generally constant; it is very burst-oriented. Although clients can make huge demands of servers and networks, it is done only on a sporadic basis, and there is little demand the rest of the time.

7.5.2 Tuning Clients

It is critical to realize that, on Unix systems, a remotely mounted filesystem is managed identically to a local disk subsystem. This means that the virtual memory subsystem is interposed between applications and the NFS client software; files stored on a remote mount are cached in memory just as local files are. This caching mechanism delays, and sometimes prevents, NFS activity.

Another approach to minimizing network activity is CacheFS, described in greater detail in Section 5.4.10. One of the best features of CacheFS is that it is completely client-side; the server is not aware of the client's internal caching. However, it is not a magic bullet. Because it makes copies of data blocks, the CacheFS subsystem must periodically scan the cached files, and causes any cached file that has been modified on the server to be purged and reloaded on the next access. Because most programs work on entire files (via read(2) and write(2)) rather than on specific data blocks (e.g., via mmap(2)), CacheFS usually ends up caching entire files. Filesystems that are frequently modified are, therefore, poor candidates for CacheFS; files will be continuously cached and then purged, which results in more network traffic than would be seen with an unaugmented NFS configuration. One excellent example of this is the /var/mail directory: a mail spool file of any substantial size will be flushed and reread every time a new mail message arrives, which will be very slow if new email is received frequently.

You can evaluate the performance of a CacheFS filesystem by using the cachefsstat(1m), cachefswssize(1m), and cachefslog(1m) commands. As a general rule, the hit rates for a given filesystem should be higher than 35%; lower hit rates mean that the cache is either too small, which is verifiable by comparing the size of the filesystem working set as reported by cachefswssize to the cache size, or the access pattern on the cached filesystem is very random. Rates of consistency failure in excess of 20% or so is a strong indication that the filesystem is being updated too quickly for CacheFS to be of real benefit.

In data-intensive NFS Version 3 environments, Solaris clients may wish to set the nfs:nfsv3_nra_kernel parameter to 6, which configures the client to request that the server read ahead six blocks of data. This value has shown good results in tests.

7.5.2.1 Obtaining statistics for an NFS-mounted filesystem

One way to determine the operational statistics for an NFS mount is to use iostat, as described in Section 5.5.4.

Another powerful tool is nfsstat -c, which displays some client-side NFS statistics. This example is taken from a lightly loaded workstation that mounts /home via NFS:

# nfsstat -c

...

Client rpc:

Connection oriented:

calls       badcalls    badxids     timeouts    newcreds    badverfs

164250      0           0           0           0           0

timers      cantconn    nomem       interrupts

0           0           0           0

...

Client nfs:

calls       badcalls    clgets      cltoomany

159786      0           159786      0

Version 3: (158872 calls)

null        getattr     setattr     lookup      access      readlink

0 0%        58283 36%   1863 1%     10003 6%    19134 12%   47 0%

read        write       create      mkdir       symlink     mknod

39926 25%   23395 14%   2209 1%     2 0%        12 0%       0 0%

remove      rmdir       rename      link        readdir     readdirplus

2276 1%     2 0%        487 0%      93 0%       480 0%      461 0%

fsstat      fsinfo      pathconf    commit

66 0%       1 0%        27 0%       105 0%

...

Version 3: (914 calls)

null        getacl      setacl

0 0%        914 100%    0 0%

...

Notice I have trimmed nonrelevant output. There are a few red flags to watch for:

  • If the badxids field is approximately equal to the timeouts field, and both are greater than 5% of the number of calls, look for bottlenecks on the NFS server.

  • If the badxids field is relatively small but the timeouts field is greater than 5% of the number of calls, look for network problems resulting in dropped packets.

The badxids field counts how many times a reply was received from a server that did not correspond to any outstanding request; the timeouts field corresponds to the number of times an NFS call timed out while waiting for a reply from the server.

If you are using a service mounted via UDP, nfsstat -m will return some statistics concerning the average time to execute certain commands:

% nfsstat -m

/home from cassandra:/workspaces

 Flags: vers=3,proto=tcp,sec=sys,hard,intr,link,symlink,acl,rsize=32768,

wsize=32768,retrans=5

/mnt from london:/services/patches

 Flags: vers=3,proto=udp,sec=sys,hard,intr,link,symlink,acl,rsize=32768,

wsize=32768,retrans=5

 Lookups:     srtt=7 (17ms), dev=3 (15ms), cur=2 (40ms)

 Reads:     srtt=18 (45ms), dev=11 (55ms), cur=7 (140ms)

As you can see, the first mount (/home) is mounted with NFS Version 3 over TCP, whereas the second (/mnt) is mounted with NFS Version 3 over UDP. The srtt (smoothed round-trip time) field is key: if it rises above 60 ms or so, the device will begin to feel "slow." The dev field represents the variability in the srtt field, and cur describes the current timeout level for retransmission.

Another way to watch the behavior of an NFS-mounted filesystem in Solaris is to use iostat -xn, as described in Section 5.5.4:

# iostat -xn 30

...

                              extended device statistics

  r/s  w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device

  8.2 20.4   68.2 1042.1  0.0  1.3    0.0   46.7   0  47 c0t0d0

  0.0  0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 cassandra:/workspaces

  2.8  0.0   78.1    0.0  0.2  0.3   56.9   89.9   1   8 london:/services/patches

...

This illustrates that there's more than one way to do something in performance tuning.

7.5.2.2 The rnode cache

The rnode cache corresponds to the UFS inode cache, but caches information about files accessed over NFS. This data is gathered as a result of getattr calls to the NFS server (which keeps this information in a vnode of its own). The default rnode cache size is twice the size of the DNLC, and shouldn't need to be tuned.

7.5.2.3 Tuning NFS clients for bursty transfers

This section is most relevant to applications that cycle between two stages that each make significant contributions to the application runtime. One of these stages should be I/O intensive, and the other should not be. Many high performance computing applications exhibit this sort of pattern.

NFS operations on a single file, even from a single process, are split up by the kernel onto a number of kernel threads equal to the nfs3_max_threads parameter, which is 8 by default. The client caches up to 64 KB of written data per thread before causing the process to block; therefore, increasing the number of NFS3 kernel threads may decrease or eliminate the time the application spends waiting for networked I/O rather than computing. nfs3_max_threads should be kept to a reasonable value (in the range of 6-8 per processor). What is actually accomplished here is parallelism between the data transfer over the wire, which is done by the kernel on the buffers held by the NFS kernel threads, and the application processing. This will not improve maximum sustained NFS performance.

In order for this to work, the application must keep the file descriptor open; if the file descriptor is closed, then you will also need to mount the remote filesystem with the undocumented -o nocto flag. By default, NFS clients will wait for their writes to complete to disk when a file is closed. This has the effect of ensuring consistency across other systems that may be able to read the file. The nocto flag has the effect of not guaranteeing such consistency, which can improve performance, but has the side effect of hiding changes to that file from other clients until some (unpredictable) later time.

7.5.2.4 Tuning NFS clients for sequential transfer

The nfs3_nra kernel parameter specifies the number of 32 KB blocks that an NFS client will try to read ahead in a given file. This can significantly improve NFS read throughput by amortizing the cost of an RPC call across more data. One upper bound on nfs3_nra is the number of NFS3 kernel threads (see Section 7.5.2.3 earlier in this chapter).

Another useful kernel parameter is clnt_max_conns. On a given client, all NFS requests that use TCP will share a pool of connections, governed by clnt_max_conns, which is equal to 1 by default. This has the side effect of distributing the load of processing across multiple CPUs. Tuning clnt_max_conns is most likely to buy you a performance increase on fast, multiprocessor machines connected via Gigabit Ethernet.

7.5.3 Tuning Servers

Tuning NFS servers is potentially a complicated issue. There are many questions to be answered:

  • Is the workload data or attribute intensive?

  • Are the clients able to cache the majority of their NFS-mounted data?

  • How many active clients must be supported?

  • How big are the filesystems to be shared?

  • Are the NFS requests directed at the same files repeatedly, or is the access pattern more random?

  • What is the underlying network configuration?

  • Is the underlying disk subsystem fast enough?

  • Is the workload adequately spread out over the available disks?

The single most important factor in tuning an NFS server is ensuring there is sufficient network bandwidth to the client, and the most important factor in determining how much bandwidth is "sufficient" is the type of activity that dominates the workload. Attribute-intensive NFS is easily handled with low-cost networking,[26] but data-intensive NFS demands a high-bandwidth infrastructure. While switched Ethernet is excellent for avoiding congestion in attribute-intensive environments, it does not address the primary problem with data-intensive NFS, which is insufficient bandwidth into the client. Full-duplex operation is considerably less important, because NFS operations tend to be unbalanced (the ratio of reads and writes is nowhere near 1:1). In most NFS installations, full-duplex Ethernet is indistinguishable from half-duplex.

[26] Of course, 100 Mb/s Ethernet is perfectly acceptable for this application. The primary advantage, however, is that more clients can be configured on a single network before overloading occurs.

It is important in data-intensive environments to use NFS Version 3 for the simple reason that the maximum block size in NFS Version 2 is 8 KB. This effectively limits network throughput on a 100 Mb/s connection to 4 MB/s. The advantage of high-speed networks in NFS Version 2 is that multiple full-speed conversations can occur without network degradation. NFS Version 3 improves substantially on the possible throughput, approaching the limits of the underlying network infrastructure, which is mainly due to the much larger data block sizes. This means the network strategy must be quite a bit different, relying instead on dedicated 100 Mb/s Ethernet or very low-density ATM networks.

It is also important to consider how many clients will be active at once. Very few applications present continuous NFS demands. Table 7-17 may be of use in approximating the number of supportable active clients.

Table 7-17. Estimating the number of supportable clients

Network media

Attribute-intensive clients/network

Data-intensive clients/network

10BASE-T

15-20

Not a good idea

100BASE-T (Repeated)

150-200

10-15

100BASE-T (Switched)

200-280

15-18

ATM/FDDI

200-300

15-20

Processor performance is usually not the limiting factor in NFS server performance. The processor is used primarily to process the network packets themselves and to undertake the work required by the protocol -- generally in that order -- as well as manage the interfaces for the disk and network devices. While IA-32-based systems usually seem to offer attractive price/performance in this arena, they are often hampered by relatively slow memory subsystems. In general, any fast, modern uniprocessor or mid-range multiprocessor system should be able to handle a large amount of attribute-intensive network traffic. A general rule of thumb for data-intensive traffic is that a single 440 MHz UltraSPARC-IIi processor can service about 170 Mb/s. On Solaris and Linux systems that use the in-kernel NFS daemon, scalability is between 75-90%, meaning that adding a second processor increases performance by 75-90%. This is nearly the worst case situation for SMP scalability, because the NFS service is handled entirely within kernel space, where locking is heavily stressed. Secondary caches improve performance by about 10%.

Disk performance is usually critical to NFS performance. After all this discussion of bandwidth, please note that this level of performance is achievable only if the filesystem being served resides on a disk subsystem fast enough to operate at these rates.

7.5.3.1 Designing disk subsystems for NFS servers

Attribute-intensive NFS is nearly all random-access. As a result, the disk read/write heads spend much more time seeking than they do actually retrieving data. The throughput of a disk in a random-access workload is relatively small. As a result, many disks can be configured on a single SCSI bus, and the goal in configuration should be to provide as many spindles as possible, since they are the limiting factor in peformance. For minimal latency, one rule of thumb is to configure one 7,200 rpm disk for every three clients. For attribute-intensive environments with a high amount of writes, using NVRAM in a disk array can substantially reduce disk utilization.

Data-intensive environments are much simpler. As a general guideline, if the network is not a limiting factor, a single active Version 2 client consumes 5.5 MB/s and a Version 3 client consumes 11 MB/s. This means that a stripe or other RAID configuration is essential, because even the fastest disk is unlikely to transfer more than 14 MB/s, and any concurrent use can substantially degrade throughput.

Using logging filesystems for the NFS server can be very useful (see Section 5.4.3), as it provides a way to safely and quickly commit updates to the filesystem and accelerates consistency checking at boot-time. In most cases, however, there is a performance penalty associated with using logging: the operation must first be logged, applied to the filesystem, then cleared from the log. In situations dominated by small writes (up to 16 KB), writing to a logging filesystem is about as fast as writing to a nonlogging filesystem. If many writes are being committed at once, logging is potentially much faster, because the seek distances inside the log are much smaller than the seek distances in the actual filesystem. For large writes, however, logging can impose a performance penalty of up to 20%.

7.5.3.2 NVRAM caching

NVRAM write acceleration substantially improves write throughput in NFS Version 2 environments because all writes in Version 2 are synchronous; they must be committed to nonvolatile storage before the operation can return.[27] This incurs at least three disk writes for a typical file: one to update the actual data, one to update the file's directory information, and an indirect block (and quite possibly a doubly indirect block). This means that an NFS Version 2 write can take 120 ms (3 synchronous writes at 20 ms apiece), or 6 times as long as the normal 20 ms or so for a local disk write. By committing writes to NVRAM instead of disk, write performance is greatly increased. Because this performance boost can be as much as 100%, NVRAM caching should almost always be configured. The only exception to this is in read-only NFS servers, because the NVRAM cache reduces the maximum throughput of the system by about 5% due to the overhead in managing the cache. This is nearly always a fair trade for the vast improvement in response time, if only because the maximum throughput of capacity of most systems is much greater than the typical load; it is a classic tradeoff between throughput and latency.

[27] Note that this is not the same as "to disk," and is intentional on the part of the designers of the protocol.

One of the design goals of NFS Version 3 was eliminating synchronous writes. Although NVRAM acceleration is still beneficial, the improvement is much smaller (on the order of 5%). There is no real reason to configure NVRAM caches for servers that only offer NFS Version 3 services.

Nonvolatile memory caching is still critically important for RAID 5 configurations because the disk array has no connection to the filesystem, and as a result it can accelerate more than just synchronous writes (see Section 6.2.6).

7.5.3.3 Memory requirements

The gut reaction of many systems administrators is to configure NFS servers with very large amounts of main memory to try and maximize the benefit of the caching implemented in the virtual memory subsystem. However counterintuitive it may seem, in most cases this is futile due to two factors. First, the size of the exported filesystem is much larger than the size of main memory; and second, most clients typically do not share most of their on-disk working set with other clients. The one exception to this rule is in temporary files where the client systems are short of memory.

There is a simple rule for sizing NFS server main memory: provide enough memory to cache any data that is likely to be referred to more than once every five minutes.[28] This can be roughly estimated at 128 MB of main memory per UltraSPARC-class microprocessor. It is worth taking into consideration that attribute-intensive environments benefit a little bit more from having large main memory subsystems than do data-intensive environments. If the NFS server provides temporary file space that is used heavily, configure main memory to be about 75% of the size of the active temporary files.[29] For application servers, main memory should be roughly the size of all heavily used binary files and libraries; this is not a typical use of NFS, and in fact it is very amenable to caching its active data. Because NFS service typically runs entirely within kernel space, the system requires essentially no swap space except for saving crash dumps in the event of a panic. In fact, you can configure a pure NFS server with no swap space at all!

[28] This "five-minute rule" actually originated in sizing main memory for database servers, and is based on an estimation of the cost per second of caching data.

[29] However, keep in mind that redirecting these temporary files to a filesystem local to the client is likely to provide much improved client performance as well as decreased network traffic.

Configuring NFS servers with large amounts of main memory is usually a waste of time.

7.5.3.4 The two basic types of NFS servers

Generally, there are two classes of NFS servers. Some run as user processes, much like sendmail; these are called user-space NFS servers. However, because of the performance improvements associated with having the NFS server tightly bound into the operating system, most modern NFS servers run entirely inside the kernel, like fsflush or bdflush (see Section 4.4.2.1 or Section 4.4.2.2). These are called in-kernel NFS servers. If you are at all concerned with NFS performance, your first step should be to use an in-kernel NFS server; the stock Solaris NFS server is of this type, as is the Linux knfsd server.

One side effect of running in kernel space is that all NFS services are handled at a higher priority than user processes. This means it is a bad idea to run other applications on a high-use NFS server, because the other processes will slow down.

Another side effect is that CPU time spent servicing NFS work is reported in the sys field of the CPU monitoring utilities (see Section 3.6.3). In general, CPU power on an NFS server should be increased if system time is consistently more than 50% of the overall processor utilization.

7.5.3.5 Tuning the number of NFS threads

Each NFS thread can handle a single NFS request at a time. Therefore, a larger pool of threads allows the server to deal with more concurrent NFS requests. The default setting, 16, is almost certainly too small. However, extra NFS threads are not problematic. To find an appropriate number, apply the following rules and take the largest value:

  • Use at least 2 threads for every active client process that is accessing an NFS resource.

  • Use 16 to 384 NFS threads per CPU. Slow systems like the SPARCstation 5 should use 16, and 450 MHz UltraSPARC-II processors should use 384.

  • Use 16 NFS threads for every 10 Mb/s of served networks.

You actually adjust this parameter by editing the line in /etc/init.d/nfs.server that starts nfsd. For example, to start 128 threads, use this line:

/usr/lib/nfs/nfsd -a 128

The number of NFS server threads should be increased when the number of active threads approaches the number that were started. You can find out approximately how many active NFS server threads there are by running this command:[30]

[30] This command actually counts the number of threads that are run through svc_run. There are other, non-NFS threads that do this, but NFS threads are typically the majority.

csh# echo '$<threadlist' | adb -k |& grep svc_run | grep -v grep | wc -l

       4
7.5.3.6 Adjusting the buffer cache

The buffer cache is used to cache disk I/O related to inode and indirect blocks only. The size of this cache is determined by the bufhwm variable, which is specified in KB. The default value is zero, which allows up to 2% of system memory to be used. This value can be increased to up to 20%. On a large system, this variable may need to be limited in order to prevent the server from exhausting the kernel address space.

You can monitor the activity of the buffer cache using sar -b:

% sar -b 5 10

...

01:22:48 bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s

01:23:03       0      71     100      32     103      69       0       0

This system is rather quiet. However, if a significant number of reads and writes occur per second (greater than 50), and the read hit rate (%rcache) and write hit rate (%wcache) fall below 90 and 65%, respectively, then the size of the buffer cache should be increased. You can read more about the buffer cache in Section 5.4.2.8.

7.5.3.7 The maxusers parameter

The maxusers parameter controls the size of various kernel tables, such as the process table. It is dynamically sized and equivalent to the amount of configured physical memory:[31] any system with more than 1 GB of RAM will have maxusers set to 1,024. If manually set, the minimum is 8 and the maximum is 2,048. The most important thing about this parameter is that it is used to compute the default size for the inode and directory name caches, which are controlled by ufs_ninode and ncsize. By default, these values are set to 7 x maxusers + 9C.

[31] This actually excludes the memory taken by the kernel at boot time.

7.5.3.8 The directory name lookup cache (DNLC)

The directory name lookup cache, or DNLC, caches directory lookups. A miss in this cache means that a disk I/O may be required to read the directory when walking the path name components, which is necessary to retrieve a file. The getattr, setattr, and lookup operations, which can represent more than 50% of the total number of NFS calls, all rely on the operation of this cache mechanism. You can find the hit rate of this cache by using vmstat -s:

% vmstat -s | grep name

  4300527 total name lookups (cache hits 92%)

In general, if the hit rate is below 90%, the ncsize variable should be tuned. This parameter controls the size of the DNLC in terms of the number of translations between names and locations on disk that can be cached. Each entry uses about 50 bytes, and the only limit on the size of the DNLC is the amount of available kernel memory. Because NFS servers generally do not need much physical memory, some tuning here is usually required. On a dedicated NFS server, doubling the amount specified by the default is reasonable (see Section 7.5.3.7 earlier in this chapter).

7.5.3.9 The inode cache

Every time an operation is performed on a filesystem, the inode read from disk is cached in case it is needed again. The number of idle inodes that are kept in the cache is governed by the ufs_ninode parameter. Each idle inode consumes about 320 bytes of kernel memory. Because every entry in the DNLC points to an entry in the inode cache, the two should be sized together.

Because ufs_ninode is simply a limit, you can change its value on a running system using adb. The tested upper limit corresponds to a value of 2,048 for maxusers, which is equivalent to a value of 34,906 for ncsize. Rather than directly tuning the inode cache, however, tune ncsize and let the system pick the ufs_ninode parameter.

7.5.3.10 Observing NFS server performance with nfsstat

One of the most useful utilities for picking up information on how an NFS server is performing is nfsstat -s, which displays some statistics on NFS service:

# nfsstat -s

Server rpc:

Connection oriented:

calls       badcalls    nullrecv    badlen      xdrcall     dupchecks

9035483     0           0           0           0           165117

dupreqs

0

...

Version 3: (8616387 calls)

null        getattr     setattr     lookup      access      readlink

3651 0%     3446452 39% 25 0%       1509014 17% 2317518 26% 1828 0%

read        write       create      mkdir       symlink     mknod

1037342 12% 0 0%        12 0%       1 0%        1 0%        0 0%

remove      rmdir       rename      link        readdir     readdirplus

69 0%       0 0%        8 0%        0 0%        102506 1%   165003 1%

fsstat      fsinfo      pathconf    commit

1263 0%     1248 0%     30446 0%    0 0%

...

There is a lot of data here, but there are a few important things to look out for.

  • A large number in the badcalls field indicates that there is a user in too many groups or there are many attempts to access a nonexported filesystem.

  • If readlink is greater than about 10%, users are using too many symbolic links.

  • If getattr is bigger than about 60%, attributes are probably not being cached successfully on the client. This is adjusted by the actimeo option to mount.

  • If null is greater than 1%, the automounter timeouts are probably too short.

  • If the number of writes exceeds 5%, it is probably an excellent idea to investigate some sort of nonvolatile caching or filesystem logging on the server.

7.5.4 Wide Area Networks and NFS

NFS clients and servers are often located on discontiguous networks joined by one or more routers. This brings up a problem that we have not discussed yet: the latency of network links. In local area networking, latency is not generally a problem because the distances are very short, minimizing media delays. Latency is of vital importance because of the request-response nature of most attribute-intensive workloads -- each request must receive a response before the next request can be issued. The latency of wide area networks can be high for three reasons:

  • Wide area networks are much more susceptible to transmission errors, which can cause significant retransmission of data. The time sent to transmit a single packet can be far higher than expected.

  • The physical media used to transfer data over long distances (particularly satellite links) can inject significant latency.

  • Routers take a finite amount of time to route packets from one network to another. Since there are usually several routers between networks, this can add up.

NFS over wide area networks is certainly possible, particularly in data-intensive environments where the ease of use of NFS overcomes the lessened bandwidth. The increased latency causes a substantial performance hit in attribute-intensive installations, but this does not necessarily destroy the idea. CacheFS can be of greater use in these environments, as can the use of TCP as a transport mechanism.


7.6 CIFS via Unix

Samba is a popular free software product for allowing Unix systems to provide file and print services to Windows systems via SMB and CIFS. Most of the system-level tuning that can be applied for NFS is also valid for Samba, so we will concentrate on a few salient Samba-specific tuning issues. Unfortunately, Samba is not widely deployed in performance-critical areas; you can almost always scale it horizontally (by adding more servers) rather than having to try to squeak out the last few percent out of the server. As a result of this limited necessity, it's hard to find good information about tuning Samba for high performance. With that caveat, here are a few of the things I've learned.

There are two things that you absolutely must avoid when tuning Samba for peak performance: do not leave the log level set to a high level unless you are actively debugging,[32] and realize that strict sync, strict locking, and sync always have huge repercussions for performance (strict locking less so than the others).

[32] In fact, leave it set to zero.

You can experiment with setting the TCP_NODELAY and IPTOS_LOWDELAY flags; they trade throughput for reduced latency, but may be of limited use. Tuning the send and receive buffers via SO_SNDBUF and SO_RCVBUF is usually not advantageous, so it's probably better to tune them at the system level (see Section 7.4.2.3 earlier in this chapter). You should turn on read raw and write raw, as they allow Samba to pull in big chunks of data from the network (up to 64 KB). If you turn off wide links, which allows you to follow a symbolic link out of your filesystem, turn on getwd cache, which caches directory paths. This prevents "walking" directory trees and can be a substantial performance boost.


7.7 Concluding Thoughts

From a historical point of view, network performance has lagged behind system performance. The network has traditionally been a bottleneck to performance; you could always build processors fast enough to fill network pipes, and disk arrays faster than networks. One of the most interesting developments of the last five years has been the vast increase in available network throughput: from 1991 to 2001, local area network bandwidth across Ethernet has increased from 10 megabits a second to an even 1 gigabit a second -- and work on a future 10 Gb standard is ongoing. As available capacity keeps scaling up, systems are having an increasingly difficult time processing data fast enough to keep up with the data flow.

In this chapter, we've gone over a lot of material that systems administrators often aren't exposed to: the principles behind network operations, the physical media used to convey information over distances, the underlying interfaces (Ethernet, FDDI, ATM) and protocols (particularly the ever-tunable TCP), finishing up with a discussion of the applications that let us share files across the office or across the world. The theories here are complex; this chapter may require several readings. If this work interests you, pick up a few of the texts we've mentioned in this chapter. They can only improve your understanding of the interface between the very different fields of systems and network engineering.


Chapter 8. Code Tuning

La perfection est atteinte non quand il ne reste rien à ajouter, mais quand il ne reste rien à enlever.(Perfection is attained not when there is nothing left to add, but when there is nothing more to take away.)

—Antoine de Saint-Exupery

The art of writing software tuned for maximum performance is a difficult one to learn, much less master; it could easily be the topic of a book all to itself. However, operating system tuning can only get you so far: poorly optimized software can be a bottleneck just as easily as a shortage of physical memory. In these cases, if you have access to the application's source code, some careful tuning can help you solve the performance problem. Generally, performance problems in software fall into two broad areas: poor algorithm selection and inefficient implementation.

In this chapter, I discuss the two most critical optimizations for system performance, some techniques for analyzing application performance (such as the Solaris trace normal form (TNF) probe facility and the GNU profiling tool gprof ), and some common patterns that lend themselves to optimization.


8.1 The Two Critical Approaches

There are two exceptionally valuable approaches for improving the performance of your applications. The first of these is to write refactored code. Refactoring is a process that originated out of the Smalltalk and academic programming communities, and that emphasizes clear, concise programming, as well as automated testing. When code has been refactored thoroughly, it is easy to understand and much easier to modify for future uses. Unfortunately, it requires a great deal of discipline on the part of the software developer. The definitive reference on this programming technique is Refactoring: Improving the Design of Existing Code by Martin Fowler et al (Addison Wesley).

The second approach is one of algorithmic choice. Although a thorough discussion of algorithmic analysis is well beyond the scope of this text, it's important to realize that there are many different approaches to the same problem, and some are much faster than others. As an example, we'll look at different approaches to searching a string for a specific pattern.

Typically, algorithmic replacement leads to the greatest speedups in application code. Writing clear, refactored code will make the process of finding bottlenecks and implementing algorithmic changes simpler.

You could say these two basic principles boil down to "write good code using fast algorithms." This is true, but my experiences have shown that people often write bad code using slow algorithms. There are a variety of reasons why this occurs:

  • Writing bad code is much faster than writing good code. Good software engineering practices often trade an increase in initial development time for a decrease in the time required to maintain the software by eliminating bugs in the design phase, or for a decrease in application runtime by integrating performance analysis into the development process. When faced with a deadline and irate management, it's often easier to just get it done, rather than do it well.

  • Implementing good algorithms is technically challenging. As we'll see in our string searching example, implementing a naive search algorithm is much easier than implementing a more complex system. When faced with management's desire for a product yesterday and the time required to implement such an algorithm (and, indeed, to design one if there is not one readily available), often the good algorithms lose out.

  • Good software engineers are difficult to find. The traits that make someone a "good" software engineer include but aren't limited to the traits that make someone good at writing source code. Software engineering involves architecture development, soliticing support from other development organizations, formal code review, ongoing documentation, version control, and typically a few well thought-out processes to hold the engineering work together. Concepts like pair-programming are just beginning to move from the academic world into the mainstream, and they aren't heavily practiced. Truly excellent software engineers tend to command very high salaries, and so it's hard to get a lot of them together.

  • It's quite rare that a group of engineers sit down to develop a piece of software from scratch. More likely, they're adding onto an existing framework, or developing the next version of the current product. The mistakes have already been made, and it's not time- or cost-effective to throw everything away and start over.

There are basically two kinds of performance improvements that come from modifying application source code: evolutionary and revolutionary. An evolutionary change is one that is accomplished without a significant alteration to the algorithms that provide a foundation for the application. For example, changing an multi-threaded application's locking strategies to relieve lock contention on a few key locks is typically an evolutionary change. Revolutionary change comes about from changing the underlying algorithm. The difference is that evolutionary changes tend to give you performance speedups of a few times at most -- it's very dependent on how the original application was written -- but revolutionary changes can improve application performance by orders of magnitude. Revolutionary changes also tend to be much harder to implement.

Clearly, algorithmic change has a huge potential for performance improvement. So, how does refactoring fit into the sketch we've drawn of software development? The basic idea here is whenever you write a line of code, it should be refactored. If you are rewriting pieces of legacy code, those should be refactored as well. Refactoring takes a lot of time and understanding at first, but it rapidly becomes second nature. As the code becomes more highly refactored, it becomes much easier to identify and implement performance improvements. Because the code is clear, compact, and easy to understand, interpreting profiling output becomes straightforward. Implementing algorithmic change becomes vastly easier and much faster. I can't overemphasize the importance of refactoring in application development and tuning. With that philosophy out of the way, then, let's move on to a discussion of string searching as an illustration of the impact of algorithmic selection.

8.1.1 String Searching Algorithms

Let's say that we have a string of text, perhaps "Here is a string of text which we wish to search." We want to find the word "wish" in this string. Henceforth, we'll call the string we are searching S, or the source string, and the string we are searching for T, or the target string. How can we approach this problem? Let's look at a few different approaches.

8.1.1.1 Algorithm 1: naive searching

As a first approach, let's look for T in S by comparing the first character in T to the first character in S, and then following these rules:

  1. We find a mismatch. This means there is no match given the current alignment of the sequences (e.g., where we first starting comparing). Move T one space further along in S, and start comparing again.[1]

    [1] This is actually a slightly improved variation of the "truly naive" search algorithm. In the truly naive algorithm, we would continue until we had run out of characters in T, regardless of whether we'd found a mismatch halfway through. This process of aborting as soon as we find a mismatch is called short-circuiting. Actual implementations of the naive algorithm almost always incorporate short-circuiting.

  2. We find a match. This is a potential match, so we need to keep searching; compare the next characters in T or S.

  3. We have used up all the characters in T. We have found a match!

  4. We have used up all the characters in S. No match exists in the string.

Computational Complexity and Big O Notation

One approach to measuring the complexity of an algorithm is to use what is called the asymptotic complexity measure. It means that we do not count the exact number of steps the algorithm requires, but rather consider how that number grows with the size of the inputs to the algorithm. This asymptotic complexity is written using big O notation (the "O" comes from "order").

For example, let's consider this very simple line of code:

a = 3 b * 5;

c = c + 1;

This piece of code takes a constant amount of time, which we write as O(1). In terms of actual instruction time, it may be difficult to say exactly how long it takes, but whatever it is, it should be the same whenever this code is executed.

Let's consider a simple loop structure:

for (i = 0; i <= n; i++) {

     j[i] = j[i]++;

}

This loop will run precisely n times, and because the inside of the loop takes a constant amount of time, the total running time is linearly proportional to n. We write it as O(n). The exact running time might be something like 9n nanoseconds; perhaps it's even 9n + 6 nanoseconds, because the loop needs some time to start up. The big O notation allows for both a multiplication factor (like 9) and an additive factor (like 6). As long as the running time is linearly proportional to n, the correct notation is O(n).

Let's look at one more loop, a more complex one:

for (i = 0; i <= n; i++) {

     for (j = 0; j <= n; j++) {

          k[i][j] = k[i][j]++;

     }

}

The outer loop executes n times, the inner loop executes n times for every time the outer loop executes, and the assignment in the inner part of the loop structure runs in constant time. Since the inner loop actually executes n2 times, the complexity of this loop is O(n2). This notation is widely used in algorithmic analysis, so we'll use it in the following discussions.

Repeat until either conditions 2 or 3 are true. For example, let's say we want to find the string "cad" (T) in the string "cabcadchad" (S). The progress of our algorithm would follow what's shown in Table 8-1.

Table 8-1. Naive search algorithm

Step

Char. in S

Char. in T

Rule

State of Search

1

c

c

2

S:   cabcadchad

T:   cad

2

a

a

2

S:   cabcadchad

T:   cad

3

b

d

1

S:   cabcadchad

T:   cad

4

a

c

2

S:   cabcadchad

T:    cad

5

b

a

1

S:   cabcadchad

T:    cad

6

c

b

1

S:   cabcadchad

T:     cad

7

c

c

2

S:   cabcadchad

T:      cad

8

a

a

2

S:   cabcadchad

T:      cad

9

b

b

2

S:   cabcadchad

T:      cad

10

3

In the worst case, this naive searching algorithm will require a

number of comparisons equal to:

length of pattern - number of possible start points = T x (S-T)

For example, using a 48-character string as S

and a 4-character string as T, we would require

4x(48-4)=176 comparisons in the worst case. This

gives the naive search algorithm a complexity of

O(st), where s is the length

of S and t is the length of

T. In our efforts to improve performance, we

think carefully about this method for searching, and decide we'd like

a more efficient one.

8.1.1.2 Algorithm 2: Knuth-Morris-Pratt searching

 If the naive algorithm fails to find

a match for a character, it advances the starting point forward one

character. Unfortunately, this throws away all the information

obtained by matching so far. For example, let's say

we have "abababcd" as

T and

"ababababc..." as the search string

(it's not important what's after c,

as long as it causes a mismatch). When the match fails at the

position after c, the naive algorithm would have us advance one

character in S and start checking again. Because

we know that the first six characters of T were

matched, advancing by one cannot work. However, shifting the pattern

by two gives us another possible match. We know that the first four

characters already match, so we can continue the search from the same

place in S. This means we never have to look at

characters in the source text more than once. The Knuth-Morris-Pratt

(KMP) algorithm has complexity

O(s+t).

In general, KMP is fast exactly where the naive algorithm is

slow -- when S and T

both contain repeated patterns of characters. It is particularly

effective when the alphabet is small (e.g., the genetic code, which

uses only four characters: G, C, A, and T, or in bit patterns).

8.1.1.3 Algorithm 3: Boyer-Moore searching

The last algorithm we come to in our

search for fast, simple string matching is the Boyer-Moore algorithm,

which was first published in 1977. It uses a different approach,

together with two techniques to improve the amount by which the

pattern is shifted in the event of a failed match, to provide much

better performance.

The change of approach is to attempt to match T

from right to left, rather than left to right, while still moving

T left to right across S.

For example, if T is

"wish" and S

is "dish," we would successfully

match "h,"

"s," and

"i" before failing on

"d" and

"w."

The first technique for improving pattern shifting in the event of a

mismatch is to move the pattern along so that a match is made with

the character in S that we are inspecting, if

possible. For example, let's consider the string of

text "Here is a string of text which we wish to

search." We want to find the word

"wish" in this string:

S:   here is a piece of text which we wish to search

        |

T:   wish

"H" does not match

"e," and no

"e" appears in

T, so no match can contain the

"e." Therefore, we can move the

pattern along past the "e" -- up

to four places, in fact, the width of T. Now we

have:

S:   here is a piece of text which we wish to search

            |

T:       wish

The space does not match anything in T, so we

can advance four characters again:

S:   here is a piece of text which we wish to search

                |

T:           wish

The match has failed again, but this time against an

"i," which occurs in the pattern.

We shift T so that the

"i" in S and

the "i" in T

are aligned:

S:   here is a piece of text which we wish to search

                |

T:             wish

There is no "c" in

T, so shift T over by 4.

Eventually, by repeating this algorithm, we will find the occurence

of T in S. This match

occurs at the 34th position in S, but we require

only 15 comparisons. Either the naive method or the KMP algorithm

would have required at least 37 comparisons -- a huge performance

benefit. In general, the longer the pattern, the fewer the

comparisons.

The second mechanism used by the Boyer-Moore algorithm is essentially

an adaptation of the "fail array"

technique used by KMP to decide how far to slide

T along S, but adapted to

the right-to-left search method. In general, the Boyer-Moore

algorithm provides exceedingly fast searching when compared to the

naive algorithm or the KMP algorithm, particuarly for long patterns

in a text with a large alphabet. Its complexity is

O(s/t).

8.1.2 Caveats in Optimization

There are

three pitfalls to watch for in code tuning: diminishing returns, the

development-to-use ratio, and functional equivalence. The pitfall of

diminishing returns is illustrated

by a program that exhibits the characteristics illustrated in Table 8-2.

Table 8-2. Timing characteristics of a sample program

Function

Fraction of time

main

8%

foo

14%

bar

33%

baz

45%

If we improve the performance of baz by 20%, the

runtime of the program will decrease by 9%, because

baz takes 45% of the total running time. If we

improve the performance of foo by the same

amount (20%), however, the runtime of the program only improves by

2.8%. Therefore, tuning efforts should be concentrated on areas where

the improvements will provide the most effect. This usually means

resolving bottlenecks in critical sections of code; fixing

bottlenecks in noncritical code sections is not likely to help

performance much.

Another thing to watch out for is the

development time to runtime ratio. In many fields, a program may be

so specialized as to be run only a few times. In these cases, the

time spent by developers in profiling, tuning, and testing the code

may be so great that it exceeds the improvements gained by

optimization.

The last major caveat of code optimization is functional equivalence.

Not all optimizations guarantee bitwise-exact results. For example,

instead of dividing two numbers, you can multiply one by the inverse

(reciprocal) of the other. This can be a much faster operation, but

in some cases, the results will not be identical. One instance where

this is true is when the divisor is extremely small.[2] When the reciprocal of this

very small number is computed, it will overflow, and produce an

infinity. You must be careful to check the results of your program

after each optimization using representative input sets to avoid

these sorts of errors.

[2] Specifically, it must be denormalized;

the number must be so small that the exponent part of the

floating-point representation is insufficient to represent all of the

exponent, so the leading bits of the fractional part are set to zero

to reduce the number's magnitude. For example, the

number 6.0 10 -324 would have the first

fifty-one bits of its fractional part set to zero, which would leave

only one significant fractional digit. Denormalization typically

occurs once a number is smaller than about 2.3 10

-308.


8.2 Techniques for Code Analysis

There are many techniques for analyzing an applications' performance. Broadly, they can be classed as probe-based or profiler-based. Probe-based analysis relies upon the programmer inserting pieces of code into the application; these code fragments provide timestamping and other meaningful data that can be analyzed later. Profiler-based analysis depends on the application being compiled with profiling support. This causes the program to generate information about its performance as it runs, which can then be analyzed via an external profiling application.

Before we can begin an analysis of what components are responsible for the time consumed, it is good to know exactly how much time is being spent! For this, we have a set of relatively coarse-grained timing tools: time, timex, and (in Solaris) ptime.

8.2.1 Application Timing: time, timex, and ptime

The most simple question we can ask about a program's speed is "How long does it take to run?" Rather than sitting in front of a computer with a stopwatch, some common tools can be used to measure program execution time.

8.2.1.1 time

The first of these tools is the time command, which is built into the csh and ksh shells (it also may exist as a separate command in /bin). At the end of the program, time presents a breakdown of how the running time of the application was spent:

  • The elapsed time, which is the amount of "wall clock" time that the program took to run.

  • The amount of time spent in user mode, which is also called user time (i.e., executing instructions that the compiler generated from your source code, in addition to time spent in library routines).

  • The amount of time spent in kernel mode, also called system time (i.e., time spent in system calls, issuing I/O requests, etc.).

In addition to these figures, the csh built-in time command provides other useful information. Let's look at an example:

% time sieve > /dev/null

6.490u 0.530s 0:16.74 41.9%     0+0k 0+0io 111pf+0w

The first, second, and third fields correspond to the seconds of user time, the seconds of system time, and the elapsed time for the process, respectively. In general, the user time should far exceed the system time. It is probably a sign that the program is doing something inefficient if the system time is disproportionately large: the application might be generating lots of exception conditions or using system calls inefficiently. Note, however, that "I/O wait time" (the time spent rewinding a tape, seeking the disk read/write heads, etc.) is not considered CPU time.

The fourth field refers to the percentage utilization, which is the ratio of elapsed time to CPU time (the sum of the user and system times). There are many reasons why the CPU time is not in the neighborhood of the elapsed time; after all, the system may have a significant number of other processes, the application in question may induce a lot of I/O (either directly or through exhausting physical memory and incurring paging activity), or the system may not be able to retrieve data to and from memory fast enough.

The fifth field represents the average memory utilization statistics. The first of these is the average shared-memory space, which defines how much memory is consumed by the application (this memory may be shared between multiple invocations of the same process, which is why it is called "shared memory"). The second figure represents the average unshared-memory space, which is used for storing data structures. These figures represent actual physical memory usage only. Also, these measurements are not supported on all platforms, so are sometimes reported as 0+0k.

The sixth field represents the amount of block input and output I/O operations, respectively. This field is also not supported on all implementations of the time command, so it is sometimes reported as 0+0io.

The seventh and last field describes the number of page faults and swaps incurred by the application. If the number of page faults is unusually high, it indicates that the process was probably choking on memory requirements. However, every program incurs some page faults (simply by virtue of how the page allocation mechanism works; see Section 4.2.5 in Chapter 4).

Note that there is another version of the time command, one which is not built into the shell:

# /bin/time catman -w

real       36.3

user       11.9

sys         3.5

In Linux, the time command is located in /usr/bin, and presents different output:

% /usr/bin/time sieve > /dev/null

5.55user 0.25system 0:05.95elapsed 97%CPU (0avgtext+0avgdata 0maxresident)k

0inputs+0outputs (112major+9778minor)pagefaults 0swaps

This output is comparable to the shell built-in time function.

8.2.1.2 timex

A slightly higher-resolution way of getting a program's runtime is the timex command. It is similar in syntax to the other timing commands we've discussed; it just provides higher accuracy:

# timex catman -w

real       12.78

user       11.71

sys         2.67

(Note that the runtime of the catman -w process is significantly decreased compared to the last time we ran it. This is because of filesystem caching; timex itself is not introducing the problem.)

timex has three options worth noting. If process accounting is installed, specifying the -o flag causes timex to report information on the number of blocks read and written. The -s flag reports the total system activity that occured during the process' runtime. The -p flag gives the process accounting record for the command.

8.2.1.3 ptime

Solaris offers a high-resolution method for timing application runtimes invoked via ptime. ptime has the advantage of being simple to run:

# ptime catman -w

real       14.455

user        0.005

sys         0.009

This example also nicely illustrates that the sum of system time and user time need not add up to the real (elapsed) time: catman -w, which rebuilds the windex database used by man -f, man -k, and apropos, is fairly disk-intensive. The fact that disk activity is not accounted for in either user time or system time goes a long way towards explaining this huge discrepancy.

On parallel systems, the user time can be larger than the elapsed time because the combined user time across all processors is reported.

8.2.1.4 Mechanisms of timing

It's worth taking a moment to examine how these commands work. The /bin/time and timex commands rely on sampling the state of the processor at the clock interrupt. This gives an accuracy of about 1/100th of a second (the frequency of the clock interrupt, by default).

ptime uses an altogether different mechanism. When the kernel suffers a state change, system call, page fault, or scheduler change, it logs a high-resolution timestamp. This is known as microstate accounting. It significantly improves accuracy but induces some overhead. You can also turn on microstate accounting programmatically.

8.2.2 Timing-Specific Code Sections

There are many techniques for obtaining timing information about specific sections of code. I present two of them here: gethrtime(), a function call that returns the current time in nanoseconds, and directly accessing the TICK register.

8.2.2.1 Timing via gethrtime

It's recommended that you enable microstate accounting before using the gethrtime() system call. There are two ways of doing this: enabling microstate accounting in the code, or by using ptime to measure the elapsed time of the program. I present an example of both in Example 8-1 and Example 8-2.

Example 8-1. gethrtime_timing.c (without enabling microstate accounting)
/* gethrtime_timing.c */

#include <stdio.h>

#include <sys/time.h>

#include <sys/types.h>

#include <unistd.h>

int

main(int argc, char **argv)

{

    hrtime_t timeStart, timeEnd;

    int i, j = 250000;

    timeStart = gethrtime(  );

    for (i = 0; i < j; i++) {

        /* The function being measured goes here */

    }

    timeEnd = gethrtime(  );

    printf ("Average time: %lld ns\n", (timeEnd - timeStart) / j);

    return 0;

}
Example 8-2. gethrtime_ustate_timing.c (enabling microstate accounting)
/* gethrtime_ustate_timing.c */

#include <errno.h>

#include <fcntl.h>

#include <stdio.h>

#include <stdlib.h>

#include <sys/stat.h>

#include <sys/time.h>

#define _STRUCTURED_PROC 1

#include <sys/procfs.h>

/* The init_micro_acct(  ) function is drawn from the file timing.c, which is

   included in the Forte Developer 6 release: look in

   /opt/SUNWspro/WS6/examples/analyzer/omptest/timing.c */

void

init_micro_acct(  )

{ /* SunOS 5.5 or higher */

    int    ctfld;

    long   ctl[2];

    char   procname[1024];

    sprintf(procname, "/proc/%d/ctl", getpid(  ));

    ctlfd = open(procname, O_WRONLY);

    if(ctlfd < 0) {

        fprintf(stderr, "open %s failed, errno = %d\n",

                              procname, errno);

    }

    ctl[0] = PCSET;

    ctl[1] = PR_MSACCT;

    if (write(ctlfd, ctl, 2*sizeof(long)) < 0) {

        fprintf(stderr, "write failed, errno = %d\n", errno);

    }

    close(ctlfd);

    printf("Enabling microstate accounting.\n");

    return;

}

int

main(int argc, char **argv)

{

    hrtime_t timeStart, timeEnd;

    int i, j = 250000;

    /* Start microstate accounting */

    init_micro_acct(  );

    timeStart = gethrtime(  );

    for (i = 0; i < j; i++) {

        /* The function being measured goes here */

    }

    timeEnd = gethrtime(  );

    printf ("Average time: %lld ns\n", (timeEnd - timeStart) / j);

    return 0;

}

I built these applications on an Sun Ultra 10 (300 MHz UltraSPARC-II processor) with the Solaris 8 system and the Forte Developer 6 update 1 compiler, specifying the getpid() system call as the function to be timed. I also increased the number of iterations (j) to 5 million. Example 8-3 illustrates the differences between running gethrtime_timing alone, running gethrtime_timing with microstate accounting turned on via ptime, and running gethrtime_ustate_timing.

Example 8-3. Differences in execution between three timing programs
% /opt/SUNWspro/bin/cc gethrtime_timing.c -o gethrtime_timing

% /opt/SUNWspro/bin/cc gethrtime_ustate_timing.c -o gethrtime_ustate_timing

% gethrtime_timing

Average time: 1714 ns

% ptime gethrtime_timing

Average time: 2708 ns

real       13.550

user        9.785

sys         3.620

% gethrtime_ustate_timing

Enabling microstate accounting.

Average time: 2648 ns

As you can see, the gethrtime_ustate_timing and the gethrtime_timing figures are comparable, whereas the gethrtime_timing figure is significantly lower (without microstate accounting). This is precisely what we'd expect: it sharply illustrates the overhead incurred during microstate accounting. The advantage of microstate accounting is improved resolution in monitoring.

8.2.2.2 Timing via the TICK register

The SPARC architecture specifies a TICK register, which provides an accurate, low-overhead way of measuring elapsed time in terms of processor cycles. Here, we'll present a way of accessing the TICK register directly from user programs that has much less overhead than the other timing mechanisms we've discussed: gethrtime() has an overhead of about 75 CPU cycles, whereas the TICK register can be accessed in about six cycles. The TICK register can be accessed in user space in Solaris 8, but not in earlier releases.[3]

[3] Accessing the TICK register when the nonprivileged trap (NPT) bit is set in the processor causes a trap, and usually the code will abort. This bit is disabled in Solaris 8, so nonprivileged user code can read the TICK register.

The most convenient way to access the TICK register is through an in-lined assembly template, as illustrated in Example 8-4 and Example 8-5.

Example 8-4. readtickreg.il
.inline readtickreg,1

rd      %tick, %o1

stx     %o1, [%o0]

.end
Example 8-5. gettick_timing.c
/* gettick_timing.c */

#include <math.h>

#include <stdio.h>

#include <stdlib.h>

#include <sys/types.h>

extern void readtickreg(void *);

#pragma no_side_effect(readtickreg);

int

main(int argc, char **argv)

{

    uint64_t c1=0, c2=0, overhead=0, total=0, cost;

    int i, j=1, *x;

    x = &j;

    for (i = 0; i < 10; i++) {

        readtickreg ((void *)&c1);

        readtickreg ((void *)&c2);

        overhead += c2 - c1;

    }

    overhead = floor (overhead * 1.0 / 10);

    for (i = 0; i < 134217728; i++) {

        readtickreg ((void *)&c1);

        /* The function being measured goes here: we will use

           *x += i; as an example */

        *x += i;

        readtickreg ((void *)&c2);

        total += c2 - c1;

    }

    cost = floor (total * 1.0 / 134217728 - overhead);

    printf ("Overhead of readtickreg: %lld\n", overhead);

    printf ("Cost in processor cycles: %lld\n", cost);

    return 0;

}

134217728 is an arbitrary number.

We can then build and test this code:

% /opt/SUNWspro/bin/cc -xarch=v8plusa gettick_timing.c -o gettick_timing readtickreg.il -lm

% gettick_timing

Overhead of readtickreg: 6

Cost in processor cycles: 7

So the overhead of the call to the TICK register is six cycles, whereas the cost in processor cycles for the assignment is seven cycles.

I replaced the test function with getpid(), and I found that it costs on average 495 cycles. Because this test was run on a 300 MHz Ultra 10, like the previous tests, we can then find out that one cycle is 3.33 nanoseconds (1 second/300,000,000 cycles is 0.000000000333 seconds, or 3.33 nanoseconds). This means the cost of a getpid() call is just about 1,650 nanoseconds, which is pretty close to what we measured using gethrtime().

8.2.3 Probe-Based Analysis: Solaris TNF

The Solaris trace normal form (TNF) probe functionality allows the detailed monitoring of specific code flows. TNF probing involves inserting specific macro calls into your application's code, and then analyzing the output that they generate. TNF probes are analyzed by means of the prex, tnfdump, and tnfxtract tools, which we originally described for monitoring probes in the Solaris kernel itself. For a quick summary on how to use these tools, see Section 5.5.6 in Chapter 5.

8.2.3.1 Inserting probes

Probes themselves are declared by a series of macros. These probes can be placed anywhere in C and C++ programs, including multi-threaded code, drivers, loadable kernel modules, and shared objects.

The general format for the probe is as follows:

TNF_PROBE_2 (name, keys, detail,

             argument_type_1, argument_name_1, argument_value_1,

             argument_type_2, argument_type_2, argument_value_2);

Let's look at this piece by piece.

  • The macros in general are named TNF_PROBE_N, where N is a number ranging from zero to five; this represents the number of "data points" (argument type/name/value triplets) recorded by the macro. If zero is specified, no data points are logged.[4] The macro name can also be suffixed with _DEBUG, in which case the probe will not be inserted unless the code is compiled with the -DTNF_DEBUG preprocessor flag set.

    [4] However, the probe is still marked by name; it's often useful for measuring simple entry and exit paths through a function, or confirming that a certain code path is being followed.

  • The first argument is the name of the probe. The use of the name declares it, so no other declaration is necessary; it also is a block-scope declaration, so it does not otherwise affect the namespace of the program.

    The convention is to name probes after their functions, and to suffix the first probe in a function with _start and the last with _end. For example, a function named foo might have probes named foo_start and foo_end.

  • The second argument is a list of keys. This is a list of space-separated keywords that specify the groups that the probe belongs to: semicolons, single quotation marks, and the equals character are not permitted. These keywords are most useful in conjunction with using prex to select a specific set of functions. For example, all the probes in a function bar might have "bar" specified in their keys fields; this makes it very easy to turn on all the probes in that function.

  • The third argument is a list of details. This string is made up of attribute-value pairs, where the attribute and value are separated by a space, and each pair is separated by a semicolon. The detail field is used for two things. The first is supply an attribute that can be selected on via prex. For example, if one attribute is shape, then prex can select probes on the basis of their shape value. The second utility is to annotate a probe with a string that is written to a trace file only once. Attributes should be named with the vendor stock ticker symbol, or some other fairly unique symbol first, followed by the percent sign (e.g., ORA%coverart). Some attributes are predefined, including name (the name of the probe), keys (the keys associated with the probe), file (what file the probe is in), line (what line the probe is on), slots (how many slots/data points the probe writes to the trace file), object (the executable or shared object that contains the probe), and debug (where a probe is debug-only or production).

The next three arguments act as a triplet that define a piece of information to be logged.

  • argument_type_n defines the type of the nth argument. Some predefined types exist, and are summarized in Table 8-3.

    Table 8-3. TNF argument types

    Type

    Associated C type

    tnf_int

    Int

    tnf_uint

    Unsigned int

    tnf_long

    Long

    tnf_ulong

    Unsigned long

    tnf_longlong

    Long long(only on 64-bit systems)

    tnf_ulonglong

    Unsigned long long(only on 64-bit systems)

    tnf_float

    Float

    tnf_double

    Double

    tnf_string

    Char *

    tnf_opaque

    Void *

    There is a documented interface for declaring a new TNF type; please consult the TNF_DECLARE_RECORD manpage.

  • argument_name_n defines the name of the nth argument. And quotation marks are not necessary.

  • argument_value_n defines the value of the nth argument. This is evaluated to yield a value that is included in the trace file; in a multi-threaded program, it is the user's requirement to place locks around the TNF probe if argument_value_n contains a variable that should be read-protected.

Any source code that uses the TNF probing macros must include the TNF header file, <tnf/probe.h>. The code must also be compiled against the TNF library (-ltnfprobe; note that if libtnfprobe is explicitly linked into the program, it must be before libthread). You can turn off all the probes by specifying -DNPROBE as a preprocessor option.

8.2.3.2 Caveats

The single biggest caveat when using this trace method is that TNF probes may have a significant impact on application performance. You will need to experiment carefully to see precisely what the effect on your code is. However, the utility is very useful in many cases; don't be scared away just by the concern of inducing some experimental error.

8.2.4 Profiler-Based Analysis: gprof

Profiling provides information about where a program spent its execution time and what functions called what other functions; the data is known as the profile and the call graph, respectively. This technique works particularly well for getting an overview of what a program is doing, particularly a large or complex application for which explicit instrumentation would be unreasonable (e.g., via TNF probes). However, one caveat is that the program must be run in a real-world setting: what the program does affects the information that is produced by the profiling software.

In general, profiling an application takes three steps:

  • Compile and link your program with profiling turned on.

  • Execute your application.

  • Run a profile analyzer, such as GNU's gprof, to produce human-readable results.

Before we work through an example of using profiling, let's see how profiling is implemented.

8.2.4.1 Implementing profiling

Profiling works by modifying, at the time of compilation, how every function is called. This modification causes some information to be stored about where it was called from,[5] which allows the profile analyzer to build the call graph. Profiling also periodically finds out which function is currently being executed. Typically, this is done at the rate of about 100 times per second, but it varies from system to system. There are two ways to implement this periodic sampling: via a system call or by telling the kernel to waken and deliver a profiling signal to the process.

[5] This is usually implemented by examining the stack frame to find both the address of the child and the return address in the parent function; because it is very architecture-dependent, this is typically done in assembler.

Some operating systems implement a profil( ) system call, which creates an array in the kernel's memory space in which each entry accounts for a few bytes (usually between 2 and 8) of the program's address space. Every time the system's clock ticks during the lifetime of the program, the program counter is inspected and the appropriate slot in the kernel memory structure is incremented. This typically imposes low overhead, because the kernel has to wake up to process the clock interrupt anyway.

However, other operating systems don't implement profil( ) or anything similar. In these cases, the kernel (by some mechanism) is made to periodically deliver a signal to the process, which then performs the same examine-and-increment procedure described. Since this method requires the kernel to wake up and transmit a signal into user space, it imposes much higher overhead.

8.2.4.2 Compiling with profiling

For the next examples, I'll use the following piece of code, which is a very simple implementation of the Sieve of Eratosthenes in C:[6]

[6] The exact means of functionality for this (inefficient) piece of code isn't particularly important. However, the idea behind the Sieve of Eratosthenes is to find all the prime numbers up to some upper bound by crossing out multiples of small primes. For example, no multiple of 2 can be prime, since 2 is a prime number, so we cross out all multiples of 2 in a list of integers. A famous theorem in number theory, long since proved, is that to determine if a number a is prime, you need only attempt to divide it by numbers up to the square root of a, which cuts down substantially on the amount of work to be done. There are still much faster methods of finding prime numbers.

/* sieve.c */

#include <stdio.h>

#include <math.h>

#define  UPPERBOUND    10000000

int SievedIntegers[UPPERBOUND];

void

init(  )

{

    int i;

    for (i = 0; i <= UPPERBOUND; i++) {

        SievedIntegers[i] = 1;

    }

}

void

printPrimes(  )

{

    int i;

    for (i = 2; i <= UPPERBOUND; i++) {

        if (SievedIntegers[i] == 1) {

            printf ("%d\n\r", i);

        }

    }

}

void

cancel (int p)

{

    int q = 0;

    for (q = 2 * p; q <= UPPERBOUND; q = q + p) {

        SievedIntegers[q] = 0;

    }

}

int

isPrime(int p)

{

    int q;

    int r = 1;

    for (q = 2; q <= sqrt(p); q++) {

        if ((p % q) == 0) {

            r = 0;

            break;

        }

    }

    return r;

}

int

main(int argc, char **argv)

{

    int i;

    init(  );

    for (i = 2; i <= sqrt(UPPERBOUND); i++) {

        if (isPrime(i)) {

            cancel(i);

        }

    }

    printPrimes(  );

    return 0;

}

The first step is to compile this code, for which I'll use gcc:

% gcc -g -Wall -o sieve sieve.c -pg -lm

Some flags here merit explanation. The -g flag specifies the inclusion of debugging symbols into the binary, which allows gprof to produce a line-by-line list of how many times each line of code was executed, while the -pg flag indicates that I wish to gather profiling data.

8.2.4.3 Execution with profiling

The executable is runnable just as any executable normally would be. When the program exits normally,[7] a file named gmon.out will be written to the program's current working directory at exit time. As a result, if your program calls chdir( ), the gmon.out file will be written in the directory the program last chdir( )'d to. This file contains all the profiling data for that invocation of the program.

[7] That is, by means of main( ) calling return( ) or calling exit( ).

8.2.4.4 Profile analysis

Now that data has been gathered, we can look at it via gprof executable. Here, I have specified the -b flag to gprof, which causes it to suppress detailed explanations of each field, for the sake of brevity. Let's take a look at the output, piece by piece:

% gprof -b sieve

Flat profile:

Each sample counts as 0.01 seconds.

  %   cumulative   self              self     total

 time   seconds   seconds    calls  us/call  us/call  name

 72.78      4.01     4.01      446  8991.03  8991.03  cancel

 15.06      4.84     0.83        1 830000.00 830000.00  init

 12.16      5.51     0.67        1 670000.00 670000.00  printPrimes

  0.00      5.51     0.00     3161     0.00     0.00  isPrime

...

This lists all the functions that were sampled. The time field represents the percentage of the total program execution time that was spent in that process; they should sum to 100%. The cumulative seconds field shows the amount of time spent executing a function, plus all the functions above it. The self seconds field is the amount of time associated with this function alone. Functions that have a self seconds value of 0.00 appeared in the call graph, but not in the histogram samples. Therefore, they were executed but never sampled in the random sampling effort.

The number of times the function was called is represented by the calls field. The self us/call value represents the average time, in microseconds, spent in this function per call, whereas the total ms/call field shows the average time, again in microseconds, spent in this function and in all functions above it. The total us/call field expresses the total number of microseconds spent in this function and its descendants per call.

The second part of the gprof output is the call graph:

...

                        Call graph

granularity: each sample hit covers 4 byte(s) for 0.18% of 5.51 seconds

index % time    self  children    called     name

                                                 <spontaneous>

[1]    100.0    0.00    5.51                 main [1]

                4.01    0.00     446/446         cancel [2]

                0.83    0.00       1/1           init [3]

                0.67    0.00       1/1           printPrimes [4]

                0.00    0.00    3161/3161        isPrime [5]

-----------------------------------------------

                4.01    0.00     446/446         main [1]

[2]     72.8    4.01    0.00     446         cancel [2]

-----------------------------------------------

                0.83    0.00       1/1           main [1]

[3]     15.1    0.83    0.00       1         init [3]

-----------------------------------------------

                0.67    0.00       1/1           main [1]

[4]     12.2    0.67    0.00       1         printPrimes [4]

-----------------------------------------------

                0.00    0.00    3161/3161        main [1]

[5]      0.0    0.00    0.00    3161         isPrime [5]

-----------------------------------------------

Index by function name

   [2] cancel                  [5] isPrime

   [3] init                    [4] printPrimes

The call graph diagrams how much time was spent in each function and its children. Each set of dashed lines divides the table into entries, one for each function. In each entry, the primary line is the one that starts with the index number in square brackets. This line indicates which function the entry describes, the preceding lines indicate the function's callers, and the following lines describe the function's subroutines (also called children in this context).

Each primary line specifies four important fields: % time, which gives the percentage of the total time that was spent in this function and its children; self, which specifies the amount of time spent in this function alone; children, which is the total amount of time spent in the subroutine calls made by this function; and called, which is the number of times the function was called. If the function called itself (exhibited recursion), there will be two numbers specified for the called field, separated by a + character. The first number counts nonrecursive calls, and the second counts recursive calls.

The other lines in an entry are relatively analogous to the primary line, with one exception. The called field for nonprimary lines contain two numbers, which represent the number of times the function was called by this function, followed by the total number of times it was called by all callers.

If the identity of a function's callers cannot be ascertained, a dummy caller line is created, which is named <spontaneous>. This happens sometimes with signal handlers, and is almost always represented as main( )'s caller.

We have only covered the highlights of using gprof. For more information, consult the manpage.

8.2.4.5 Caveats

There are some definite caveats to using application profiling.

The sampling mechanism takes samples at fixed intervals during the program's runtime. If the program is not running at the time the sampling mechanism runs, no data is gathered. For example, let's say a process generates a working set that is much larger than physical memory. Much of the system's time will be spent handling moving pages from disk to memory and vice versa; this is not considered program runtime and will not be reported. It can also be a benefit: profiling results should be quite similar regardless of the load on the system at the time the profiling data was gathered.

The sampling process itself is subject to statistical inaccuracies. If a function runs for a very short period of time, it is possible that the function will be "missed." If the function can be expected to be seen once, it's possible that it won't show up at all, or that it will be caught running several times. In contrast, the call graph and the counts of the number of times a function is called are derived by explicit counting, so they will be completely accurate.

One general rule of thumb is that a runtime figure is accurate if it is considerably bigger than the sampling period (at least two orders of magnitude).

Some of the call graph information is estimated; there is no direct information concerning them. The assumption is that the time spent in a function is not associated with who called that function. For example, if a function WriteBook( ) used a total of sixty seconds and was called ten times, with three of those calls originating from CallFromEditor( ), WriteBook( ) contributes eighteen seconds to CallFromEditor( )'s children time. This is not always a valid assumption. Say that another function, OnTrain( ), calls WriteBook( ) in all other cases, but in this case WriteBook( ) returns very quickly (it's hard to get much writing done on a train, after all). In this case, almost all the time spent in WriteBook( ) should be attributed to CallFromEditor( ), but gprof has no way of knowing this, so it blindly -- and quite incorrectly -- associates the forty-two seconds of WriteBook( )'s runtime to OnTrain( ).


8.3 Optimization Patterns

We now turn our attention toward patterns for optimization. The techniques presented here serve two purposes. The first is to explain how some of the most common optimizations work, so that you can understand whether a specific compiler optimization flag is likely to help performance. The second is to enable you to, in some cases, apply these optimizations by hand. This is sometimes necessary, when the compiler is not able to make assumptions about whether an optimization is safe.[8] This does not mean that you should apply these optimizations everywhere! The compilers are usually pretty reliable. Careful use of timing techniques in critical code flow sections will help you identify cases where the compiler could use some help.

[8] By default, compilers are pessimistic about whether certain optimizations are permissible. Humans are often able to look at a piece of code and determine, by visual inspection and a thorough understanding of the code flow, whether a given optimization is safe.

We cannot present every optimization, so we'll focus on the most common and important ones. Generally, these optimizations fall into a handful of categories: optimizations on arithmetic, on loops, and on string operations.

These optimizations will be presented in C. If you are interested in Java tuning, please consult Java Performance Tuning by Jack Shiraz (O'Reilly). IBM has written an excellent summary text on this topic, which is called the Optimization and Tuning Guide for Fortran, C, and C++.

8.3.1 Arithmetic

One of the most fundamental arithmetic optimizations is to replace a sequence of operations by a sequence of equivalent, less complex operations. For example, the compiler might replace a line of code like:

x = pow (y, 5);

with a line like:

x = y * y * y * y * y;

because in many cases repeated multiplication is less expensive than exponentiation. This is called strength reduction. It is by far the most common arithmatic optimization.

Another common example of strength reduction is replacing a division with a multiplication-by-inverse. For example, you could replace this line of code:

x = y / z;

with a line like:

x = y * (1 / z);

and see a performance increase simply because, on many platforms, inverting and multiplying is much faster than dividing.

Another good idea in mathematics-intensive codes is to link against your vendor's high-performance mathematics library, if one is available. This step can produce fantastic performance increases by calling highly tuned assembly-language routines for basic math functions.

8.3.2 Loops

Please note that all of these loop optimizations are performed automatically by a modern optimization compiler. However, compilers are required to be somewhat conservative (the emphasis is on generating "correct" code, rather than fast code); using some of these techniques manually can help the compiler find opportunities for other, more complex optimizations.

Loop fusion is the process of combining the statements inside several loops into a single loop.

Untuned

Tuned

for (i = 0; i <= 1024; i++) {

     x = x * a[i] * b[i];

}

for (j = 0; j <= 1024; j++) {

     y = y * a[j] * b[j];

}

for (i = 0; i <= 1024; i++) {

     x = x * a[i] * b[i];

     y = y * a[j] * b[j];

}

This technique not only halves the loop overhead, it also creates better opportunities for the compiler to overlap instructions to take advantage of parallelism in the underlying microprocessor. You must exercise care, however, as this optimization has the possibility of decreasing performance by increasing the cache miss rate (for example, when each loop accesses a significant portion of the data cache). It also can be complicated by problems of data dependence; you must be sure that the work done in the second loop is not reliant on the work finished by the first loop.

Floating invariant flow control moves flow control statements that do not change between iterations of a loop outside of the loop.

Untuned

Tuned

for (i = 0; i <= 1024; i++) {

     for (j = 0; j <= 1024; j++) {

          if (a[i] >= 64) {

               b[i] = -a[i];

          }

          x = x + a[j] - b[j];

     }

}

for (i = 0; i <= 1024; i++) {

     if (a[i] >= 64) {

          b[i] = -a[i];

     }

     for (j = 0; j <= 1024; j++) {

          x = x + a[j] - b[j];

     }

}

Most compilers do this automatically, but the compiler may not be able to if the loop contains calls to other procedures or in complex loops where the invariance cannot easily be determined.

Loop unrolling replicates the functional part of the loop while reducing the iteration count proportionally.

Untuned

Tuned

for (i = 0; i <= 1024; i++) {

     for (j = 0; j <= 4; j++) {

          a[i] = b[i, j];

     }

}

for (i = 0; i <= 1024; i++) {

     a[i] = b[i, 1];

     a[i] = b[i, 2];

     a[i] = b[i, 3];

     a[i] = b[i, 4];

}

This is primarily of use to generate more opportunities for scheduling parallelism in the compiler and microprocessor.

Stride refers to the relationship between the layout of an array's elements in memory and the order in which they are processed. For example, an array of 32-bit integers is accessed at stride 1 if they are accessed by monotonically increasing the value of the rightmost subscript.

Untuned

Tuned

for (i = 0; i <= 1024; i++) {

     for (j = 0; j <= 16; j++) {

          a[j] = b[j, i];

     }

}

for (j = 0; j <= 16; i++) {

     for (i = 0; i <= 1024; i++) {

          a[j] = b[j, i];

     }

}

The lower the stride, the higher performing the software, due to optimal use of the cache hierarchy.

8.3.3 Strings

The runtime of the standard C library functions is proportional to the lengths of their operands. It's quite common to loop over calls to these functions and consume a significant amount of CPU time. There a few simple optimizations you can take to reduce this effect:

  • Instead of calling strlen( ) in a loop involving the string itself, you can rewrite the loop so that you set a temporary value equal to strlen( ) before the loop, and then modify the loop as you add and remove characters.

  • If you are testing whether a string is empty, strlen( ) is probably not the fastest way to do it. If the strings are of substantial size, strlen() will check every character until it finds the terminator character. You can replace it with *s == '\0', which checks only the first character as well as saving a function call. Similarly, if you want to clear a string, *s = '\0' is much faster than strcpy(s, "").

  • strcat( ) scans the full length of the string on each call. If you've been keeping track of the length of the string as you go along (as described regarding strlen( ) previously), you have a ready-made index to the end of the string, and can use strcpy( ) or memcpy( ) from there.

  • You might be able to save a little bit of time with a strcmp( ) between two strings containing natural language by checking if the first characters of the strings are equal before doing the call to strcmp( ). Because of the statistically nonuniform distribution of letters in natural languages, the chances of a match are a bit better than would be otherwise expected. However, this optimization can be counterproductive, so be aware of the environment you're working in (e.g., if you are comparing adjacent strings in sorted output, this will not buy you anything).


8.4 Interacting with Compilers

Compilers have a well-deserved reputation for presenting their users with a bewildering array of options, none of which make particular sense. The truth is that very few of these options are really necessary to tune, but picking correct compiler settings can make a huge difference in application performance. Furthermore, you don't need to completely understand how the compiler works in order to properly optimize your applications.[9]

[9] However, if you are interested in learning about how, exactly, a compiler works, read Compilers: Principles, Techniques, and Tools by A. Aho, R. Sethi, and J. Ullman (Addison Wesley). This is commonly known as "the dragon book" and is one of the best textbooks on the theory behind compiler design.

There are two compilers in widespread use in the Sun and Linux worlds. The first is the Forte Developer compilers, which are sold by Sun as the follow-on to the now-obsolete Sun WorkShop compilers. More information, including a trial download, is available at http://www.sun.com/forte/. The second is the free GNU compiler, gcc. The chief advantages of gcc are that it is free and has been ported to a wide variety of platforms; it is almost always available, even for esoteric systems. In my opinion, the Forte Developer compilers generate significantly better code, and are much better documented and easier to work with. We will focus here primarily on options for the Forte compilers. No matter what, keep in mind that compilers are constantly being optimized and refined. It pays to have the most recent version of the compilers, as it's not unusual to see speedups in the 10 to 15% range with each new major compiler release.

There are two caveats about compiler flags. The first is that order is important: subsequent flags will override earlier flags. This is particularly a problem with -fast; specifying -fast -xO4 with the Forte Developer 6 compilers will result in a lower optimization level being used than just -fast alone. The second is more of a heads-up; namely, that not every compiler flag need be preceded with -x.

8.4.1 Typical Optimizations: -fast

The most important compiler option to know for Sun systems is -fast. This macro expands to different things depending on what language you are compiling and what version of the compiler you're using. For a summary, see Table 8-4.

Table 8-4. Expansions of the -fast macro

Compiler

C

Fortran 77

Fortran 90

Sun WorkShop 5.0

-xO4 -single -fns-fsimple=1-ftrap=%none -libmil-native

-xO4 -dalign -depend-fns -fsimple=1-ftrap=%none -libmil-native

-xO3 -dalign -fns

-ftrap=common -f

-native -xlibmopt

Forte Developer 6

-O5 -single

-xmemalign=8s -fns

-fsimple=2

-ftrap=%none -xlibmil

-native -xprefetch=no

-xvector=no

-O5 -dalign -depend

-xpad=local -fns

-fsimple=2

-ftrap=%none -xlibmil

-native -xlibmopt

-xvector=yes

-O5 -dalign -depend

-xpad=local -fns

-fsimple=2

-ftrap=common -f

-xlibmil -native

-xlibmopt

-xvector=yes

Forte Developer 6 Update 1

-O5 -single

-xmemalign=8s -fns

-fsimple=2

-ftrap=%none

-xalias_level=basic

-xbuiltin=%all

-xlibmil -native

-xprefetch=no

-xvector=no

-O5 -dalign -depend

-xprefetch

-xpad=local -fns

-fsimple=2

-ftrap=%none -xlibmil

-native -xlibmopt

-xvector=yes

-O5 -dalign -depend

-xprefetch

-xpad=local -fns

-fsimple=2

-ftrap=common -f

-xlibmil -native

-xlibmopt

-xvector=yes

For the most part, we won't discuss what these flags actually mean. Suffice it to say that they represent a good mix of basic optimizations.

8.4.2 Optimization Level: -xO

The most commonly tuned optimization flag is the global optimization control -xOn, where n specifies the optimization level (from 1 through 5, inclusive). The default optimization level is -xO2 for C programs and -xO3 for Fortran programs in the Forte Developer 6 compilers. Table 8-5 is a summary of what each optimization level does.

Table 8-5. Optimization level summary for Forte compilers

Level

Description

-xO1

Basic local optimizations only.

-xO2

Default for C. Level 1 plus global optimizations: algebraic simplification, subexpression elimination, optimized register allocation, eliminating dead code, propagation of constants, tail-call elimination.[10]

-xO3

Default for Fortran. Level 2 plus loop optimizations (unrolling and fusion). Software pipelining.

-xO4

Level 3 plus function inline and more aggressive global optimization.

-xO5

The highest level of optimization. It is most likely to perform well when combined with profile feedback (see Section 8.4.11 later in this chapter).

[10] Tail calls are not eliminated until -xO4 if -g is specified, to ease debugging.

gcc, the free GNU compiler, performs no optimization by default. It has three levels of optimization, which provide varying returns summarized in Table 8-6.

Table 8-6. Optimization level summary for gcc

Level

Description

-O1

Basic local optimization only.

-O2

Any optimization that doesn't involve a size-for-speed tradeoff; notably no loop unrolling or inlining. (Enabling loop unrolling is done via -funroll_loops.)

-O3

Level 2 optimizations plus function inlining.

8.4.3 Specifying Instruction Set Architecture: -xarch

The -xarch flag specifies what instruction set architecture the program should be built with. For SPARC platforms, there are nine choices, as summarized in Table 8-7.

Table 8-7. Valid -xarch values

Value

Instruction set

Bitness

Restrictions

v7

SPARC V7(no fsmuld, integer mul or integer div instructions)

32-bit

Any SPARC machine

v8a

SPARC V8(no fsmuld instruction)

32-bit

Any microSPARC-I or later machine

v8

SPARC V8

32-bit

Any SuperSPARC or later machine

v8plus

SPARC V9(no VIS)

32-bit

Any UltraSPARC machine

v8plusa

SPARC V9

32-bit

Any UltraSPARC machine

v8plusb

SPARC V9(with UltraSPARC-III extensions)

32-bit

UltraSPARC-III machines

v9

SPARC V9(no VIS)

64-bit

Any UltraSPARC machine running in 64-bit mode

v9a

SPARC V9

64-bit

Any UltraSPARC machine running in 64-bit mode

v9b

SPARC V9(with UltraSPARC-III extensions)

64-bit

UltraSPARC-III machines running in 64-bit mode

In general, the best performance of a 32-bit application on UltraSPARC-class processors is obtained with -xarch=v8plusa (or v8plusb on UltraSPARC-III systems). Specifying an early architecture here can significantly impede performance of your applications; pick the latest architecture possible while still supporting all your systems. It may be worth while to compile multiple versions of a program that particularly benefits from an optimization only available on certain platforms.

If you are compiling applications with -fast on an UltraSPARC platform and either don't specify an -xarch value or specify -xarch=native, it will assume you want to use v8plusa or v8plusb. This will generate code that won't run on pre-UltraSPARC machines, and generates an error message:

cc: Warning: -xarch=native has been explicitly specified, or implicitly specified by

a macro option, -xarch=native on this architecture implies -xarch=v8plusa which

generates code that does not run on pre-UltraSPARC processors

This is just a warning, and won't interfere with anything. If you'd like it to go away, just explicitly specify -xarch=v8plusa (or any value you like, except for the native values).

8.4.4 Specifying Processor Architecture: -xchip

Restricting the type of processor that will be used to run the application provides the compiler with a great deal of information. This information is mostly used to schedule instructions and handle branches in the optimal way; for example, the number of cycles of delay between loading a data value and using it in a subsequent computation (called the load-use delay) is very processor-dependent. Tuning this parameter can have big payoffs. There are thirteen options for SPARC systems, which are laid out in Table 8-8.[11]

[11] There are also options for Solaris on Intel systems; the most useful of these is pentium_pro.

Table 8-8. Valid -xchip values

Value

Architecture

old

Very old (pre-SuperSPARC) processors

super

SuperSPARC chips(any SuperSPARC chip slower than 60 MHz: SM61 or slower)

super2

SuperSPARC-II chips(any SuperSPARC chip faster than 75 MHz: SM71 or better)

micro

MicroSPARC-I chips(SPARCclassic, LX)

micro2

MicroSPARC-II chips(Voyager, SPARCstation 4, SPARCstation 5)

hyper

HyperSPARC-I chips

hyper2

HyperSPARC-II chips

ultra

UltraSPARC-I chips(All UltraSPARC-I processors are slower than 200 MHz)

ultra2

UltraSPARC-II chips

ultra2i

UltraSPARC-IIi chips(Ultra 5, Ultra 10, etc.)

ultra3

UltraSPARC-III chips

native

The current architecture (what is being used to compile), assuming a 32-bit environment

native64

The current architecture (what is being used to compile), assuming a 64-bit environment

8.4.5 Function Inlining: -xinlining and -xcrossfile

Function inlining is the process of including a function inside the function that called it. It eliminates some of the overhead of jumping to another location in memory, and provides more opportunities for the compiler to find parallel scheduling opportunities.

In gcc, inlining optimizations are turned on by -finline-functions.

It's a bad idea to turn on inlining when the optimization level is higher than -xO4, as that level of optimization already inlines functions. Specifying the inlining option twice in this fashion can cause performance slowdowns. However, specifying -xcrossfile is a good idea: it lets you inline functions that reside in separate source code files. Use this with -xO4 or higher levels of optimization for the best results.

8.4.6 Data Dependency Analysis: -xdepend

Part of the compiler's job is to analyze data dependencies within loops, and restructure the loops if necessary. This restructuring may give the compiler more opportunities to unroll or otherwise optimize the loop, which can improve performance. The compiler will also attempt to cache block.[12]

[12] Cache blocking is a technique wherein the computation is divided up so that the data accessed in the divided parts fits in the processor cache. This improves cache efficiency.

-xdepend requires that you specify -xO3 or higher, and it is set by default in -fast.

8.4.7 Vector Operations: -xvector

The -xvector flag tells the compiler to use the optimized vector math library, which can be significantly faster than its scalar counterpart. This flag has the most effect when the application repeatedly calls math library intrinsics, such as log, sin, and exp.

-xvector also enables -xdepend.

8.4.8 Default Floating Point Constant Size: -xsfpconst

By default, the Forte C compiler treats floating-point constants as doubles unless explicitly declared as floats. As a result, many extra conversion instructions are often required to transform double-precision floating-point constants into single-precision ones. This is a particularly big problem for codes that perform a large number of division or square root operations, which require almost twice as many cycles when working on double-precision variables. The -xsfpconst flag forces the compiler to treat floating-point constants as single-precision.

8.4.9 Data Prefetching: -xprefetch

Data prefetching is a technique that allows the processor to overlap executing instructions with fetching data from memory. This is particularly helpful on latency-bound applications on high-latency systems, such as applications that have a repeated regular access pattern (e.g., many large loops) on high-latency hardware. The -xprefetch flag tells the compiler to insert specific prefetching instructions into the program to facilitate this behavior. Because this is very architecturally dependent, you get the best results when this option is used in concert with the -xchip and -xtarget options.

Note that UltraSPARC-I processors allow the prefetch instruction, but don't actually do anything. Therefore, code compiled with -xprefetch will run on all UltraSPARC-based systems, but not do anything on UltraSPARC-I systems; potentially provide a significant improvement on UltraSPARC-II systems; and potentially provide a large improvement on UltraSPARC-III systems, because of their on-chip caches dedicated to prefetching.

8.4.10 Quick and Dirty Compiler Flags

This is a quick and dirty reference of compiler flags for specific sorts of applications. As always, be sure to test; they may not be ideal for your particular program. These flags all assume you are using an UltraSPARC-based system:

  • For applications that require strict floating-point behavior (IEEE 754), you should try -fast -xarch=v8plus -fsimple=0.

  • For C applications where pointer arguments to functions don't alias each other, try -fast -xarch=v8plus -xrestrict; if they follow the ISO C 1999 pointer dereferencing rules, also try specifying -xalias_level=std.

  • Fortran applications should use -stackvar, which forces the compiler to allocate local variables on the program's stack; try -fast -xarch-v8plus -stackvar.

  • For applications running on UltraSPARC-III systems, use -xprefetch to enable prefetching; try -fast -xdepend -xchip=ultra3 -xprefetch.

8.4.11 Profiling Feedback

The compiler has a built-in mechanism for targeting aggressive optimization on the most frequently run portions of a program. This scheme is called profile feedback. It uses runtime execution frequency data from the application to direct further optimizations on a subsequent compiling run.

The use profile feedback optimization, you must first build the application with the -xprofile=collect:name flag, where name is the name of the executable. A subsequent run of the application will generate a name.profile directory, which will contain the runtime data. This training run will take more time than the application normally would. Finally, rebuild the application with -xprofile=use:name; the compiler will use the data it gathered during the first application run to improve optimization.

Example 8-6 gives some source code that we can use to demonstrate the utility of profiling feedback.

Example 8-6. profiling.c
/* profiling.c */

#include <stdio.h>

int

main(int argc, char **argv)

{

        int i, n = 512, sum = 0;

        for (i = 0; i < 100000000; i++) {

                if (i > n) {

                        sum++;

                } else {

                        sum--;

                }

        }

        printf ("sum: %d\n", sum);

}

We then build profiling.c and use profile feedback to improve its performance, as shown in Example 8-7.

Example 8-7. Using profiling feedback to improve application performance
% /opt/SUNWspro/bin/cc -fast -xarch=v8plusa profiling.c -o profiling \-xprofile=collect:profiling

% ./profiling

sum: 99998974

% timex ./profiling

sum: 99998974

real        3.39

user        3.38

sys         0.01

% /opt/SUNWspro/bin/cc -fast -xarch=v8plusa profiling.c -o profiling \-xprofile=use:profiling

% timex ./profiling

sum: 99998974

real        0.69

user        0.67

sys         0.01

This is a remarkable improvement! Not all applications will improve this drastically, of course.


8.5 Concluding Thoughts

In this chapter, we've discussed techniques for developing fast code, analyzing algorithms, timing the runtime of our applications, using code profiling to find hot spots in our source code, and extracting the most from our compilers. In short, it's been a whirlwind introduction to the expansive field of code tuning. If you're interested in learning more, one excellent reference for optimizing applications is the book Techniques for Optimizing Applications: High Performance Computing by Rajat Garg and Ilya Sharapov (Prentice Hall).

If you only take away one piece of information or advice from this entire chapter, let it be this: write refactored code and take the time to implement fast algorithms. That sentence expresses the heart of application tuning.


Chapter 9. Instant Tuning

Trapped in time. . . surrounded by evil. . . low on gas.

—The tagline for the movie "Army of Darkness"

Sometimes, you don't have time to understand the fine details: users and management are breathing down the back of your neck, and you absolutely must fix a problem immediately. Perhaps you're the backup for a coworker who went out of town the day before ten users storm your shared office and demand that performance improve; perhaps you're a contracted systems administrator who needs a quick checklist for things that might possibly be wrong with a system; perhaps you've read this book cover-to-cover and are looking for a concise reference so you can throw away that legal pad full of notes.

In this chapter, we'll summarize procedures for approaching two common but not simple questions: where do I look in order to identify and solve common performance problems, and what tuning, in general, do I need to do on a fresh system in order to ensure performance is maximized?

Before we get into this material, however, a word of caution.

Warning! This chapter contains words of great peril![1] It illustrates solutions and approaches without explanation. As a result, it runs the risk of turning you into a Cargo Cult Performance Analyst (see the introduction in Chapter 2). Be careful!

[1] Klatuu, Barata, Nicto!


9.1 Top Five Tuning Tips

These are the five most basic questions to ask when you see a system that is performing poorly. Note that all of the threshold guidelines here assume that the data is being gathered over 30-second intervals, unless otherwise noted.

9.1.1 Where Is the Disk Bottleneck?

Almost every system that feels sluggish is under a heavy disk I/O load. Look for disks that have a "service time" greater than 50 milliseconds and a disk that is more than a few percents busy. The metric of service time actually should be called "response time": it measures the delay between a process issuing a read request and that request being completed. This is very often in the critical path for user applications. It is quite possible that a disk that is heavily overloaded will have response times measured in thousands of milliseconds (not a typographic error). You can get this information from iostat -xnP 30 in Solaris:

# iostat -xnP 30

...

                    extended device statistics

    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device

    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c0t0d0s0

    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c0t0d0s2

    0.7    2.7    5.3    9.5  0.0  0.1    0.0   16.1   0   2 c0t0d0s3

...

In Linux, install the optional package that contains iostat, and look at the await column:

# iostat -x -d 30

Linux 2.4.2-2 (aiua)      07/29/2001

...

Device: rrqm/s wrqm/s  r/s  w/s  rsec/s  wsec/s avgrq-sz avgqu-sz  await svctm  %util

hde       4.79  1.40  86.95 3.26 158.87  37.43     2.18     1.01   11.22   8.14  7.34

hde1      0.00  0.00  82.15 0.00  82.15   0.00     1.00     0.46    5.64   5.64  4.63

hde2      4.79  1.40   4.80 3.26  76.70  37.43    14.16     0.55   68.05  34.40  2.77

hde5      0.00  0.00   0.00 0.00   0.02   0.00     8.00     0.00  100.00 100.00  0.00

hdg       9.85  2.36  36.69 5.20 371.85  60.86    10.33     8.60  205.22  45.61 19.11

hdg1      0.00  0.00   0.06 0.00   0.07   0.00     1.03     0.00    2.94   2.94  0.00

hdg2      9.85  2.36  36.63 5.20 371.78  60.86    10.34     8.60  205.54  45.68 19.10

...

UFS filesystems that are otherwise idle will sometimes have high service times due to the action of fsflush; since the filesystem is basically idle, this isn't a problem. I've never seen this phenomenon on Linux systems. If this is a Solaris system, and it has more than 512 MB of memory, the inode cache is large enough. However, in small-memory systems, increasing that cache might help by reducing the number of disk I/Os needed for filesystem management.

There is basically one technique for approaching disk I/O problems, and that is spreading out the load as much as possible. This might be best accomplished by moving "hot" directories or files to another, less loaded, disk, or by investing in a disk array with a nonvolatile memory (NVRAM) cache for speeding up writes.

9.1.2 Do You Have Enough Memory?

In Solaris, there is one guiding light for whether you are short of memory, and that is the sr field of vmstat. Please remember that the first line of vmstat's output is worthless! The other fields are largely worthless in answering this question, especially in Solaris 7 and earlier. There are a lot of misconceptions about this, two of which are the most prevalent:

  • The size of the free list is not an indicator of a memory shortage, because Solaris will consume any unused memory for caching recently used files.

  • The number of page-ins or page-outs per second is a bad metric, because Solaris handles all filesystem I/O by means of the paging mechanism. Thousands of kilobytes paged in or out just means that the system is working.

Unfortunately, if you are used to looking at vmstat output from Solaris 7, Solaris 8 changes the ballgame. It now "properly" reports information: pages that are used for filesystem caching are reported as free (because they effectively are). So, now everyone who asked "Where did all my free memory go?" in Solaris 7 is asking "Where did all this free memory come from?" after upgrading to Solaris 8.

The way to tell if you are short of memory is still the rate at which the page scanner inspects pages to see if they can be freed. If you are legitimately short of memory in Solaris 7 or earlier systems, the threshold at which you should order more memory is about 250 pages per second. If you are looking at a Solaris 8 system, any page scanner activity at all means you are short of memory.

9.1.3 Are the Processors Overloaded?

The best metric for processor overloading is the length of the run queue. The way to find this out on both Solaris and Linux is to use the vmstat command and look at the values in the procs r column. If this number is greater than about four times the number of processors in the system, then processes are probably waiting too long for a slice of time.

If the number of mutex stalls (available in Solaris as the smtx field of mpstat) is significant (over about 250 times the number of processors), you should upgrade your CPUs to faster ones, rather than adding more.

9.1.4 Are Processes Blocked on Disk I/O?

If a process is blocked, it is a sign of a disk bottleneck. You can view the number of blocked processes by means of the vmstat command's procs b field. Whenever there are any blocked processes, the system reports all CPU idle time as wait-for-I/O time. If the number of blocked processes approaches or exceeds the number of processes in the run queue, there is a severe disk bottleneck, and you should start looking for it with iostat.

If you are administering a very batch-oriented environment, you will see your fair share of blocked processes. This isn't necessarily something to worry about, but it's an indicator that improving disk subsystem performance will improve the throughput of your batch jobs.

9.1.5 Does System Time Heavily Dominate User Time?

If any of the CPU consumption utilities (vmstat, mpstat, etc.) indicate that there is more CPU time being spent in the kernel (system time) rather than running processes (user time), you might have a problem. The Solaris NFS server runs entirely inside the kernel, as does the Linux knfsd optional package, in which case system time is expected to dominate.

If the system in question isn't an NFS server, tracking down the problem can be difficult. The best approach is to identify the top few processes in terms of processor-time utilization (by prstat, top, or ps) and then use a process-tracing utility like truss to see what system calls are being issued. If the system/user skew is accompanied by a high number of mutex stalls (mpstat's smtx column), start thinking about upgrading your processors to faster ones (but don't add more).


9.2 Instant Tuning Recipes

This section presents some recipes for performance tuning. Of course, these recommendations are for general scenarios, so feel free to modify them for your environment.

The best recipe for any machine, be it a server or a workstation, is to keep the installed operating system up-to-date, and be vigilant about maintaining patches. Performance patches are quite common and sometimes have dramatic effects, or enable you to use a newer algorithm (e.g., priority paging in Solaris) on older operating systems.

9.2.1 Single-User Development Workstations

One of the most common tuning scenarios is a computer used by a single developer. Such users often require high performance for compilation and other tasks that are both processor- and filesystem-intensive, but the users may not have the high speed disks that you would expect on a large server.

9.2.1.1 Filesystems

My preferred method of filesystem layout is to break the system disk into two partitions, one for swap space and the other for everything else. The swap partition should be the lower-numbered one on the disk; for an explanation of why, see Section 5.4.10.1. However, lots of people will disagree with me, and this topic borders on the religious. If you have more than one local disk, you should place a swap partition on each of them, in order to spread out the load a bit. On Solaris, all filesystems should be mounted with the logging option, to improve recovery time in the event of a power failure (see Section 5.4.3).

NFS-mounted filesystems containing essentially invariant files, such as application programs, should be mounted read-only in order to avoid NFS writes of file access times. You should also use CacheFS, if it's available for your platform, to cache commonly used files -- it is particularly useful for application directories. It shouldn't be used for mail directories, and it may be useful for home directories depending on what the read/write mixture of files is. Any application that regularly performs large writes to an NFS filesystem shouldn't use CacheFS. You can read more about implementing CacheFS on Section 5.4.10.

9.2.1.2 Swap space

Choosing the amount of swap space is fairly difficult. If you don't have a recommendation from your application vendor, configure at least 128 MB of virtual memory. Using swap files is less orderly than swap partitions, and disk space is cheap, so more never hurts. Most desktop machines will never need more than 512 MB of paging space.

9.2.1.3 Kernel tuning

Most operating systems are well configured for this sort of activity, so there's not much to do. The one exception is that if you are running Solaris 7 or earlier, you should turn on priority paging by adding the following lines to /etc/system:

*

* enable priority paging on pre-Solaris 8 systems

set priority_paging = 1

For a detailed explanation of priority paging, see Section 4.4.3.1.

9.2.2 Workgroup Servers

In this case, we'll consider a workgroup server as one that is performing a variety of tasks in support of a small department. These tasks might include email services, file storage, light web service, etc. In a Unix-only environment, directories will be exported via NFS; we'll have to support SMB/CIFS on Windows networks with Samba. The server may also be the first-line proxy cache for the department's web browsers, as well as booting any thin clients, X terminals, or other network-reliant computers. The system is probably connected to the network via a few Fast Ethernet connections.

The workload mixture is probably going to largely be NFS or Samba. The basic sendmail software supplied with Solaris is capable of supporting at least a few hundred users without much, if any tuning; if you need to support more than that, you are probably outside of the workgroup server realm, but software products like iPlanet Messaging Server can support millions of users, along with much more efficient methods of storing messages. The web server usage is also probably going to be quite light. A uniprocessor UltraSPARC-II or faster Pentium 3-class system is easily capable of these tasks.

9.2.2.1 Memory

This sort of workload is not particularly memory-intensive: the largest active user program will probably be sendmail, and the rest of main memory will be a cache for the filesystems. Since most files will only be requested once due to client-side caching, the filesystem cache doesn't need to be large. Anything that is going to be reread frequently (e.g., applications) is an excellent candidate for being mounted via CacheFS on the clients.

As a rough guideline, start with 128 MB of memory for the system, then add about 64 MB of memory for each attached Fast Ethernet interface. If you are supporting X terminals or SunRays, you will need to add about 128 MB of memory for each one, depending on the workload that the clients will be running (one rule of thumb is to add as much memory to the server as you would install in a desktop machine, less 32 MB per client).

9.2.2.2 Disks

The two most important things to do as far as disk configuration is concerned are to ensure the reliability of data and accelerate writes as much as possible. If you have a hardware RAID device (such as a Sun StorEdge A1000), configure it as a RAID 1+0 or RAID 5 device, depending on how important capacity is (see my recommendations in Section 6.6). If you are deploying software RAID, you will want to use the DiskSuite logging feature instead of the UFS logging option, as the latter decreases performance. DiskSuite-based logging filesystems should use a dedicated disk of at least 7,200 rpm.[2] Hardware-based disk arrays can be safely mounted with the logging option, as writes to the log will be cached in the array controller's NVRAM.

[2] This may seem like wasting a lot of space; really, it's not. We are interested in the performance of this drive, not its capacity.

You should mirror the system disk in this configuration.

9.2.2.3 Filesystems

I would create a single, large root partition, like in the workgroup server recipe. However, /var should reside on a separate partition; it will need to be write-accelerated. This doesn't have that much to do with log files; instead, we're trying to improve the performance of things like delivering mail to an inbox (a client rewriting the mail file on exit is a significant disk operation). /var should be fairly large; precisely how large depends on your usage patterns.

Create a separate filesystem for the user home directories.

9.2.2.4 Swap space

You should not need very much swap space for this sort of workload. A few hundred megabytes is more than sufficient.

9.2.2.5 Optimizing NFS

There are three important tuning changes to make for NFS servers: be sure you're using NFS Version 3, increase the number of NFS server threads, and, on Solaris systems, increase the TCP buffer sizes.

NFS Version 3 implements a few algorithmic changes that improve performance significantly, especially on file writes. All of your NFS clients should mount their filesystems and specify the vers=3 parameter. You can use nfsstat to find out whether any Version 2 calls are being made on your server. For more details about NFS Version 3, see Section 7.5.

Picking an appropriate number of NFS kernel threads is important. The out-of-the-box default is acceptable, but certainly not conducive to maximum performance. You actually adjust this parameter by editing the line in /etc/init.d/nfs.server that starts nfsd. 128 is a reasonable number to start with:

/usr/lib/nfs/nfsd -a 128

On Solaris systems, you'll probably need to increase the TCP receive and transmit buffers, which are tuned by the tcp_recv_hiwat and tcp_xmit_hiwat variables. You will have to experiment with reasonable values, but a good starting point is to set tcp_xmit_hiwat to 56 KB (57344) and tcp_recv_lowat to 32 KB (32768). You shouldn't set these values greater than 64,000 in order to avoid turning on TCP window scaling. Linux web servers have equivalent parameters, but their default values are high enough and don't need to be tuned. For more details, see Section 7.4.2.3.

9.2.2.6 Kernel tuning

You may want to increase the maxphys variable to enable the kernel to push more contiguous pieces of modified files out to disk. You can read more about maxphys in Section 4.4.4.

If your Solaris system is mostly performing NFS workloads and has less than 128 MB of memory, you should increase the directory name lookup cache and the inode cache. You can read more about these caches in Section 5.4.1.1 and Section 5.4.2.7 in Chapter 5. Increasing them to 8,000 entries each is sufficient:

*

* increase the dnlc and inode caches for low-memory servers

set ncsize = 8000

set ufs:ufs_ninode = 8000

Pre-Solaris 8 systems must turn on priority paging; see the previous recommendations for single-user workstations, or read Section 4.4.3.1.

9.2.3 Web Servers

There seems to be a definite trend to overbuild web servers to provide static content. A single UltraSPARC-II or fast Pentium-3 processor can saturate a 100-megabit Ethernet segment with static content easily, and scalability is effectively linear up to at least four processors. As the Web moves towards being a more dynamically generated medium, however, processor performance becomes more of a concern. Unfortunately, there are no good rules of thumb for sizing dynamic HTTP workloads simply because the workloads vary so much.

9.2.3.1 Memory

Web servers require quite a bit of memory to operate quickly, because they tend to perform best with large TCP buffers (which consume memory quickly when there are many connections) and with extensive filesystem caching. As a rough guideline, I would start with 256 MB of memory, and then add the size of the files that are responsible for about 90% of the requests (you can derive this information by inspecting the log files). This will give you enough memory to cache the "hottest" pages.

Pre-Solaris 8 systems should turn on priority paging. See Section 4.4.3.1.

9.2.3.2 Disks

For very large web sites or those providing web service for a large number of sites, disk storage becomes an issue. A single 7,200 rpm disk can sustain about 50 static HTTP operations per second, so you should work out how many peak operations per second you intend to sustain and stripe disks until you reach that limit. RAID 5 arrays are an excellent choice; they provide some fault resilience while maintaining good performance (largely because the workload is very read-heavy).

On busy web sites, it may be necessary to stripe and mirror the directory containing the HTTP logs. These will grow fast, especially during peak load.

9.2.3.3 Filesystems

I would still advise creating a single, large root partition, just like in the workgroup server recipe. However, as noted previously, you should be sure to create a dedicated file system for the HTTP logs, and take steps to improve write performance.

Whether or not you mirror the system disk is up to you. In large installations where the web servers are a group of Netra T1s, or other lightweight machines behind a load-balancing switch, or a round robin DNS entry, the failure of a machine isn't a big problem if your infrastructure is set up to work around it. You can just wait for the machine to be replaced, reinstall it via automated tools, and put it into service. This works best when your actual content is provided to the front-line web servers via an NFS backend.

9.2.3.4 Swap space

Again, web servers tend not to need a lot of swap space. 512 MB should be plenty, just to absorb anonymous memory requests that go unused. For details, see Section 4.3.2.1.

9.2.3.5 Networks

Network sizing is a little bit more tricky. We'll focus on static operations here; if your web server is performing a lot of dynamic content serving, you'll need to develop your own guidelines based on your particular workload. The intent of Table 9-1 is to provide you with a rough means of sizing your network capacity. For example, if you have a thousand users coming in via 56K dialup lines, you won't have to sustain more than 400 HTTP operations per second. Conversely, if you are trying to support 1,000 operations per second on a full-duplex Fast Ethernet link, you are probably doomed to fail.

Table 9-1. Network utilization for HTTP clients

Network connection

Bandwidth(per second)

Peak static HTTP operations/second

56K Dialup

56 Kbps

0.4

ISDN

128 Kbps

1

768K SDSL(Full-duplex)

768 Kbps

5

T1

1.5 Mb/sec

10

Cable modem

6

35

Ethernet

10

60

DS3

45 Mb/sec

300

Fast Ethernet(Full-duplex)

200 Mb/sec

600

Gigabit Ethernet(Full-duplex)

2,000 Mb/sec

5,000

There are two issues you should also be aware of regarding web server performance. The first is that SSL (also known as Secure HTTP) is an absolute performance killer. Divide the number of sustainable HTTP operations per processor by ten. There is a new generation of hardware SSL accelerator cards coming onto the market now, but I have no experience with them and cannot provide you with any good advice. The second issue to keep in mind is software products like Solaris's Network Cache Accelerator (NCA). These products run in kernel space and act as a caching mechanism for incoming (usually static) HTTP requests. This saves the trouble of copying the request from the kernel into user space, waking up the web server application, fetching the data from whatever cache (or, worse, disk) the page resides in, and then returning it to the kernel. The NCA product simply caches static web pages in a piece of kernel memory and transparently answers requests, but much faster. A newer generation of NCA will probably support hooks for dynamic content retrieval. Another product in this same vein is Tux, which is a very high-performance web server for Linux that runs significant portions of itself in kernel space. The caveat is that these products sometimes exhibit stability issues, so you'll need to test them carefully before deploying them.

9.2.3.6 Kernel tuning

You will probably want to increase the default values for the incomplete connection queue and the completed connection queue. In Solaris, these tunables are called tcp_conn_req_max_q0 and tcp_conn_req_max_q, respectively, and are 1,024 and 128 by default. The sign that you should increase these variables is large values of tcpListenDrop and tcpListenDropQ0, which are viewable via netstat -sP tcp. tcp_conn_req_max_q0 probably shouldn't be set above about 10,000, and tcp_conn_req_max_q shouldn't need to be set larger than tcp_conn_req_max_q0. In Linux, the equivalent to the tcp_conn_req_max_q0 parameter is tcp_max_syn_backlog, which can be found in /proc/sys/net/ipv4/. For more information on the connection queues, see Section 7.4.2.1.

You will also want to increase the TCP receive and send buffers. These are governed by the tcp_recv_hiwat and tcp_xmit_hiwat variables in Solaris. You will have to experiment with reasonable values, but I would start by increasing tcp_xmit_hiwat to 48 KB (49,152) and tcp_recv_lowat to 32 KB (32,768). You shouldn't set these values greater than 64,000 in order to avoid turning on TCP window scaling. Linux web servers have their equivalent values set to values more appropriate to high performance, but you can tune them if you like. For more details, see Section 7.4.2.3.

9.2.3.7 Special case: proxy servers

The big difference between web servers and proxy servers is that proxy servers tend to be much more write-intensive (as they write cached pages to disk). The ideal configuration for a proxy server therefore involves a disk array with nonvolatile memory to cache and accelerate writes. A similar effect can be obtained by using a dedicated disk to log transactions to the cache filesystem.

If you are configuring a fast uniprocessor proxy server with, for example, eight 9 GB disks for data, I would stripe six of the disks with a 64 KB interlace size with DiskSuite or some other software RAID software. The remaining two disks would have one 128 MB partition each, be mirrored, and be used for storing the log information. Do not put anything else on those log disks. They are fully busy from a service-time point of view, even if they have spare storage and throughput available.


Free Open Study

Colophon

Our look is the result of reader comments, our own experimentation, and feedback from distribution channels. Distinctive covers complement our distinctive approach to technical topics, breathing personality and life into potentially dry subjects.

The animal on the cover of System Performance Tuning, Second Edition is a swordfish, a marine fish with a long, sword-shaped snout. The swordfish is distinguished from the marlin by its flattened snout, high dorsal fin, toothless mouth, and lack of scales. The fastest swimmer of all fish, a swordfish can grow up to 14 feet in length and weigh a thousand pounds or more. This combination of speed and size gives it strength and momentum -- enough to drive its sword through the planking of a boat. The swordfish hunts by charging through schools of fish, striking to either side with its sword and then returning to collect its kill.

Colleen Gorman was the production editor and the copyeditor for System Performance Tuning, Second Edition. Mary Brady was the proofreader, and Linley Dolby provided quality control. Tom Dinse wrote the index.

Edie Freedman designed the cover of this book. The cover image is a 19th-century engraving from the Dover Pictorial Archive. Emma Colby produced the cover layout with QuarkXPress 4.1 using Adobe's ITC Garamond font.

Melanie Wang designed the interior layout, based on a series design by David Futato. Mihaela Maier converted the files using tools created by Mike Sierra. The text font is Linotype Birka; the heading font is Adobe Myriad Condensed; and the code font is LucasFont's TheSans Mono Condensed. The illustrations that appear in the book were produced by Robert Romano and Jessamyn Read using Macromedia FreeHand 9 and Adobe Photoshop 6. The tip and warning icons were drawn by Christopher Bing. This colophon was written by Colleen Gorman.

The online edition of this book was created by the Safari production group (John Chodacki, Becki Maisch, and Madeleine Newell) using a set of Frame-to-XML conversion and cleanup tools written and maintained by Erik Ray, Benn Salter, John Chodacki, and Jeff Liggett.


index

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]


index_SYMBOL

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

100BASE-T4 Ethernet
100BASE-TX Ethernet
10BASE-T Ethernet
10BASE2 Ethernet cabling
10BASE5 Ethernet cabling
5-4-3 rule (Ethernet)
64-bit architecture
applications
disadvantages
80x86 processors (Intel), memory page size


index_A

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

access patterns (disks)
acctcom command, reviewing process accounting records
ACK bit (TCP)
adb command, kernel variable tuning
address mask
addresses
frames (networks)
IP
duplicates
memory
address space layout
limitations
SCSI bus
world wide name (Fibre Channel)
algorithms
asymptotic complexity measure
big O notation
Boyer-Moore searching
CRC (cyclical redundancy checking)
elevator
inode cache (UFS)
Kesslers Best Bin
Knuth-Morris-Pratt searching
Linux, epochs
majority consensus (Solstice DiskSuite)
Nagles 2nd
naive
page coloring
problem solving and
selecting
slow start (TCP)
string searching
Alpha processors (DEC)
memory page size
ALU (arithmetic and logic unit)
application layer (OSI model)
applications [See also software]
64-bit architecture
algorithms, selecting
analyzing
compiling with profiling
executing with profiling
profiling
compiler flags
floating-point behavior
Fortran
function pointers
UltraSPARC-III applications
multiprocessor systems, design issues
performance
books about
MIPS
ptime command
time command
timex command
profiling considerations
refactored code
arbitrated loop Fibre Channel topology (FC-AL)
architecture
64-bit 2nd
books about
disks
FC (Fibre Channel)
filesystems
FireWire
hardware level
IDE
interfaces
IPI (Intelligent Peripheral Interface)
levels of transformation
MBus
memory address space layout
memory hierarchy and caching
multiprocessor systems
networks
OSI model
peripheral buses
processors
caching
clock rates
design basics
piplining
process scheduling
SCSI (Small Computer System Interface)
UFS (Unix Filesystem)
virtual memory
von Neumann model
XBus
arithmetic and logic unit (ALU)
arithmetic optimization
arrays (disks)
associative mapping
asymptotic complexity measure
asynchronous data transfers (SCSI)
Asynchronous Transfer Mode (ATM)
asynchronous writes (disks)
ATA (AT Attachment) [See IDE]
ATA Packet Interface (ATAPI)
ATA-2 interface
ATA-3 interface
ATAPI (ATA Packet Interface)
ATM (Asynchronous Transfer Mode)
atomic operations
attachment unit interface (AUI) cable
attenuation (fiberoptic cabling)
AUI (attachment unit interface) cable
auto-negotiation (Ethernet)
autodetection (RAID arrays)
automounter (NFS)
average memory utilization statistics
average seek time (disks)
average shared-memory space


index_B

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

bandwidth-delay product
barriers (threads)
base metric (SPECint_base95)
base time quantum
batch queuing programs
bdflush command, high disk service times and
benchmarking 2nd [See also testing]
books about
commercial workload
databases
decision support systems
disks
hdparm
iozone
multiple pass results
tiobench
e-commerce transactional web servers
Linpack benchmarks
memory
lmbench
STREAM
MFLOPS (megaflops)
MIPS
SPEC benchmarks
TCP (Transaction Processing Council) benchmarks
terminal driven database environments
user defined
automation considerations
problem set selection
rule creation
runtime selection
web performance
big O notation
bits (data paths)
blocked processes
books
application performance
benchmarking
computer architecture
Java code optimization
networking
routing
TCP
Unix
workload management
booting
processor failure and
RAID devices
bottlenecks
disks
blocked processes
locating
mutex locking
Boyer-Moore searching algorithms
branch delay slot (RISC processors)
bridges
broadcast frames
buffer cache
NFS
UFS
buffers
receive
send
burst speed
busses
CompactPCI
Fibre Channel connections
multiprocessor systems
PCI, architecture
peripheral interconnect
monitoring tools
SBus
architecture
burst transfer size
clock speed
interrupt processing
transfer mode
SCSI
addresses
command queuing
differential signaling
termination
transaction operations
utilization
busstat command, peripheral interconnect monitoring


index_C

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

cables
failures and error generation
IDE
cabling
10BASE2 Ethernet
10BASE5 Ethernet
AUI (attachment unit interface)
fiberoptic
UTP
CacheFS
cache parameters, changing
cache, removing
mount command options
NFS and
cachefsstat command, filesystem cache hit rates
caches
access times
associative mapping
cache-aligned block copy cache-busting
cache-busters
direct mapping
directories, creating
disks
enabling
DNLC (directory name lookup cache)
filesystem
bdflush command
cyclic caching
disk interaction constraints
fsflush command
memory consumption
priority paging
removing
hierarchy
L1
L2
lines
linked list cache-busting
locality
mapping
miss cost
multiprocessor systems
naming service, static performance issues
NFS
buffer
DNLC
inode
non-unit stride cache-busting
organization
parameters, changing
rnode (NFS)
size anomalies
UFS
buffer
disabling
inode
write-back behavior
write-through behavior
caching
capacity planning, defined
CDDI (Copper Distributed Data Interface)
cfsadmin command
cache directories, creating
options
characterization [See workflow characterization]
checksum (TCP headers)
chunks (RAID arrays)
Linux md package
CIDR (classless interdomain routing)
circuit switched protocols
circuits (ATM)
CISC (complex instruction set computing)
instruction set length
ckpacct command, process accounting
class A networks
class B networks
subnetting
class C networks
classes, IP addresses
classless interdomain routing (CIDR)
clients
NFS
bursty transfers
sequential transfers
TCP/IP
clock rates
critical path
processors
SBus
clusters, UFS (Unix Filesystem)
code [See programming code]
code morphing
collisions (Ethernet)
combined processor memory switch (CPMS)
command queuing (SCSI)
commercial workload benchmarks
communication
busses
crossbars
Fireplane
interconnectionless architectures
multiprocessor systems 2nd
CompactPCI
compilers
code optimization considerations
fast option
flags
floating-point behavior
Fortran applications
function pointers
UltraSPARC-III systems
profile feedback
xarch option
xchip option
xcrossfile option
xdepend option
xinlining option
xO option
xprefetch option
xsfpconst option
xvector option
compiling, profiling and
complex instruction set computing [See CISC]
computational grids
computer architecture [See architecture]
concentrator (FDDI network interface)
conditional branches
prediction
congestion window (TCP)
control units, von Neumann model
controller (SCSI)
Copper Distributed Data Interface (CDDI)
copy-on-write faults
CPI (cycles per instruction)
cpio command, RAID devices, copying root filesystem
CPMS (combined processor memory switch)
CPU [See processors]
cpustat command
cputrack command
CRC (cyclical redundancy checking) algorithm
critical path
critical sections
cron, performance analysis tool intervals and
crontab file
disk quotas, enabling
sar command, enabling
crossbars, multiprocessor systems
Crusoe processor (Transmeta) 2nd
cut-through switches
cycles per instruction [See CPI]
cyclic caching
cyclical redundancy checking (CRC) algorithm
cylinders


index_D

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

d-cache
DAC (dual-attachment concentrator)
data blocks (UFS)
data cache
data encapsulation (Ethernet)
data link (networks)
data link address
Data Link and Medium Access Control (DLMAC) group (Ethernet)
data link layer (OSI model)
data paths, bits
data transfer patterns (networks)
data transfer rates (TCP)
databases, benchmarks
DDR SDRAM modules
DEC Alpha processors
memory page size
decision support systems, benchmarking
defect management, disks
deferred acknowledgements (TCP)
demand paging
denial-of-service attacks, SYN flooding
design issues
applications, refactored code
NFS disk subsystems
processors
RAID arrays
subsystems
test design
development time to runtime ratio (code tuning)
differential signaling (SCSI)
diminishing returns (code optimization)
direct cache mapping
direct memory access (DMA)
Direct Rambus (RDRAM)
directory entry (UFS)
directory name lookup cache (DNLC) 2nd
disabling
disk arrays [See RAID]
disk quotas
disks
access patterns
architecture
arrays
asynchronous writes compared to synchronous
average seek time
benchmarking
hdparm
iozone
multiple pass results
tiobench
bottlenecks
blocked processes
locating
burst speed
caches
enabling
cluster size (UFS)
defect management
distributed parity-protected striping (RAID)
elevator algorithms
Fibre Channel
Hamming code arrays (RAID)
high I/O skew
high service time problems
internal transfer speed
compared to real speed
logical volumes
megabyte, size of
minfree parameter
mirrored striping (RAID)
mirroring
disk replacement
MTBF (Mean Time Between Failure)
MTTDI (Mean Time To Data Inaccessibility)
MTTDL (Mean Time To Data Loss)
MTTR (Mean Time To Repair)
NFS
paging and swapping problems
parity-protected striping (RAID)
parity-protected striping with independent disks (RAID)
partition statistics
performance monitoring tools, iostat
RAID levels
distributed parity-protected striping
Hamming code arrays
mirrored striping
mirroring
overview
parity-protected striping
parity-protected striping with independent disks
striping
random access and storage capacity issues
read/write latency
reads
rotational delay parameter
RPS misses
seek times, reducing
specifications
spindle synchronization 2nd
string utilization
striping (RAID)
web servers
workgroup servers and
write throttling
writes
ZBR (zone bit rate recording)
dispadmin command, process scheduling
dispatch queue, kernel threads
dispatch tables, tuning
distributed parity-protected striping
DMA (direct memory access)
DNLC (directory name lookup cache) 2nd
doubly indirect blocks (filesystems)
DRAM (dynamic random access memory)
dual-attachment concentrator (DAC)
duplex mode
dynamic performance analysis
dynamic priority
dynamic random access memory (DRAM)


index_E

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

e-commerce, benchmarking
E-series processors (Sun)
echo command, NFS threads
EDO (extended data output) memory
education
usage and performance agreements
users
EIDE (Enhanced IDE)
elvtune command, disk read/write latency
embedded controllers
environment
computational grids
importance of
terminal driven database, benchmarks
user limits on
epochs (Linux scheduling algorithm)
errors
kernel memory allocation
Message fragmentation needed
Ethernet
100BASE-T4
100BASE-TX
10BASE-T
5-4-3 rule
auto-negotiation
collisions
configuration parameters, changing
Gigabit
signal basics
status, displaying
topologies
executable data segment (memory)
executable text segment (memory)
execution, profiling and
ext2fs
ext3fs
extended data output (EDO) memory
extent-based filesystems


index_F

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

fabric (Fibre Channel topology)
fast instruction set computing (FISC)
fast-page mode (FPM) memory
FC (Fibre Channel), architecture
FC-AL (arbitrated loop Fibre Channel topology)
FDDI (Fiber Distributed Data Interface)
fiberoptic cabling
Fibre Channel (FC), architecture
file and metadata filesystem logging
file locking, NFS and
files
nsswitch.conf, static performance issues
remote swapfiles with NFS
resolve.conf, static performance issues
system, persisting maxpgio value changes
filesystems [See also NFS]
architecture
CacheFS
caches
hit rates, checking
removing
copying root to RAID device
DNLC (directory name lookup cache)
ext2fs
ext3fs
free space reservations
information about, determining
inodes
JFS (Journaled Filesystem)
logging
overview
Solaris
Solstice DiskSuite software
network, mounting
ReiserFS
tail packing
rotational delay parameter
seek times, reducing
Solaris, tmpfs
tunefs command
tuning, single-user workstations
VxFS
web servers
workgroup servers and
FIN_WAIT_2 timer (TCP)
find command (RAID devices), copying root filesystem
Fireplane (Sun)
FireWire, architecture
FISC (fast instruction set computing)
Fixed Shroud Device (FSD) connectors
floating-point behavior, compiler flags
Fortran applications, compiler flags
FPM (fast page mode) memory
fragmentation, IP packets
frames (networks)
free list memory management
FSD (Fixed Shroud Device) connectors
fsflush command 2nd
high disk service times and
fstyp command, filesystem information
full-duplex mode
function inlining
function pointers, compiler flags


index_G

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

gateways
gethrtime system call
Gigabit Ethernet
Gigaplane XB crossbar (Sun)
gprof (GNU)
application profile analysis
graded-index fiber
graphics, processor support
grep bufhwm command, UFS buffer cache


index_H

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

half-duplex mode
Hamming code arrays
hard real-time process scheduling
hardware
disks, architecture
failures and error generation
fiberoptic cabling
IDE cables
RAID array management
swap space and
UTP cabling
hardware address
hdparm command
disk benchmarking
disk caches, enabling
IDE performance, enhancing
header length field (TCP headers)
heap space segment (memory)
Heisenbergs Principle of Uncertainty
hertz
high I/O skew (disks)
High Level Interface (HILI) group (Ethernet)
hme driver, Ethernet status, displaying
hmeconfig, Ethernet parameters, changing
host adapters
IPI-3
SCSI
hot spare pools (RAID arrays)


index_I

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

i-cache (processor instructions)
I-series processors (Sun)
I/O (input/output) subsystem
PIO
process accounting
tuning, importance of
UFS cache, disabling
von Neumann model 2nd
I/O tracing
prex command
process filtering
I/O wait time (processor)
IBM PowerPC processors, memory page size
ICMP errors, Message fragmentation needed
IDE (Integrated Drive Electronics)
architecture
compared to SCSI
disk caches
limitations
Linux, enhancing performance
idle time (processor)
IEEE 1394 [See FireWire]
if statements, conditional branches (instruction sets)
IMPs (internet message processors)
in-kernel NFS servers
inbound traffic
indirect blocks (filesystems)
inode cache
NFS
UFS
inodes
UFS (Unix Filesystem)
input/output [See I/O subsystem]2nd [See I/O subsystem]
instruction register
Instruction Set Architecture (ISA) 2nd
instructions
branch prediction
conditional branches
i-cache
instruction set architecture, specifying
LDSTUB
NOP (no-operation)
stages of
variable length, effect on processor
Integrated Drive Electronics [See IDE]
Intel
80x86 processors, memory page size
MMX extensions
P6 processor
Intelligent Peripheral Interface (IPI)
interconnectionless architectures, multiprocessor systems
interfaces
architecture
ATA
ATAPI
ATM (Asynchronous Transfer Mode)
EIDE
Ethernet
signal basics
topologies
FC (Fibre Channel)
FDDI (Fiber Distributed Data Interface)
FireWire
IDE architecture
IPI (Intelligent Peripheral Interface)
networks, promiscuous mode
SASI (Shugart Associates System Interface)
Ultra-ATA
USB (Universal Serial Bus)
interleaving memory
intermediate systems (networks)
internal clock rate
internal transfer speed (disks)
compared to real speed
internet message processors (IMPs)
internetworking, books about
internetworks
interprocess communication (IPC)
interrupts
Linux
round robin distribution
Solaris
intimately shared memory
intimately shared memory (ISM)
inverse request-response patterns (networks)
iostat command
disk bottlenecks, locating
disk performance monitoring
NFS operational statistics
iozone disk benchmark
IP addresses
classes
classless
duplicates
private
subnetting
IP protocol
header fields
packets
fragmentation
time-to-live
protocol field (header)
IPC (interprocess communication)
IPI (Intelligent Peripheral Interface) architecture
IPI-2 string controllers
IPI-3 host adapters
irqtune application, Linux interrupt adjustments
ISA (Instruction Set Architecture) 2nd
isainfo, kernel information
ISM (intimately shared memory)
isochronous data transfers


index_J

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

jacks
JFS (Journaled Filesystem)
journaling
modes, setting
support


index_K

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

keepalive timer (TCP)
kernel
I/O tracing
prex command
process filtering
tuning variables
tuning, single-user workstations
virtual memory variables
web servers
workgroup servers and
write throttle variable
kernel memory allocation errors
kernel threads
categories
dispatch queue
quantum
sleep state
Kesslers Best Bin algorithm
Knuth-Morris-Pratt searching algorithms
kstat unix\:0\:biostats command, UFS buffer cache statistics
kswapd (Linux pagedaemon)
ktrace command, I/O tracing


index_L

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

L1 cache 2nd
L2 cache
late collisions
latency
wide area networks and NFS
LDSTUB (load-store-unsigned-byte) instructions
level 1 cache 2nd
level 2 cache
levels of transformation
lightweight process (LWP), kernel threads
limit command, user limitations
linear mode device (RAID arrays), Linux md package
lines (caches)
Link Integrity test
linked list cache-busting
Linpack benchmark
Linux
bdflush command
disk bottlenecks, locating
elevator algorithms
ext2fs
ext3fs
filesystems, ReiserFS 2nd
IDE performance, enhancing
interrupts
JFS (Journaled Filesystem)
kernel variable tuning
md package, RAID array management
memory page size, determining
process scheduling
processor load, vmstat command
ps command, process memory consumption
swap space, monitoring
TCP buffer settings
virtual memory management
lmbench memory benchmarking tool
load average (processors)
locking, NFS and
lockstat command, mutex information
log-structured filesystem logging
logging filesystems
overview
Solaris
Solstice DiskSuite software
Logical Link Control (LLC) group (Ethernet)
logical unit number (LUN)
logical volumes
loop optimizations
low watermarks (TCP)
LUN (logical unit number)
LWP (lightweight process), kernel threads


index_M

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

MAC (media access control)
MAC address
major page faults
majority consensus algorithm (Solstice DiskSuite)
Manchester encoding
mapping caches
MAR (memory address register)
master ports
MAU (medium attachment unit)
maxcontig filesystem parameter
maximum segment size (MSS), TCP packets
maximum transmission unit (MTU)
maxpgio, kernel variable tuning
maxusers parameter
kernel tables
MB (megabyte)
MBus architecture
md package (Linux RAID array management)
MDR (memory data register)
Mean Time Between Failure (MTBF)
Mean Time To Data Inaccessibility (MTTDI)
Mean Time To Data Loss (MTTDL)
Mean Time To Repair (MTTR)
measurements, rationale for
media access control (MAC)
medium attachment unit (MAU)
megabyte (MB)
megaflops (MFLOPS)
megahertz (MHz)
memory [See also virtual memory]
access time
address space layout
addressing limitations
associative mapping
benchmarking
Linpack benchmark
tools
cache access times
cache lines
caches and hierarchy
characteristics
consumption considerations 2nd
cycle time
data path bits
data transfer measurement methods
demand paging
direct cache mapping
filesystem caching
bdflush command
cyclic caching
disk interaction constraints
fsflush command
priority paging
free list
hardware-based RAID array management
interleaving
intimately shared memory
kernel memory allocation errors
lm benchmark
management, Linux
modules, types of
monitoring
memstat command
sar command
vmstat command
NFS requirements
NVRAM, NFS and
page coloring
page scanner
paging
paging compared to swapping
performance metrics
physical implementation
process accounting
process utilization, displaying
processes
monitoring consumption
ps command
uax command
processor caches
requirements
determining
estimating
speed compared to processor speed
STREAM benchmark
swap space
anonymous memory
monitoring
optimizing
sizing
swapping
effect on interactive performance
system performance, effect on
TLB (transaction lookaside buffer)
uncacheable areas
virtual address spaces
web servers
workgroup servers and
memory address register (MAR)
memory data register (MDR)
memory management unit (MMU)
memory system (von Neumann model)
memstat command, memory monitoring
Message fragmentation needed (ICMP error)
message passing patterns (networks)
metadata logging (filesystems)
metadb command, Solstice DiskSuite state databases
metadevices
metainformation database (Solstice DiskSuite)
metamirrors (Solaris DiskSuite)
metattach command, Solaris DiskSuite
metrics, memory performance
MFLOPS (megaflops)
MHz (megahertz)
microprocessors [See processors]
microSPARC processors, memory page size
microstate accounting
enabling
migrating ext2fs to ext3fs
minfree filesystem parameter
minor page faults
MIPS (millions of instructions per second)
MIPS processors, memory page size
mirrored striping
mirroring disks
disk replacement
Linux md package
Solaris DiskSuite
miss cost (memory cache)
mkinitrd command, RAID arrays, booting
mkraid device command, Linux md package
MMU (memory management unit)
MMX extensions (Intel)
monacct command, process accounting
Motorola PowerPC processors, memory page size
mount command
journaling mode, setting
logging option
options to CacheFS
mounting network filesystems
mounts, NFS (Network File System), nesting
mpstat command
mutex information
processor statistics
MSS (maximum segment size), TCP packets
MTBF (Mean Time Between Failure)
MTTDI (Mean Time To Data Inaccessibility)
MTTDL (Mean Time To Data Loss)
MTTR (Mean Time To Repair)
MTU (maximum transmission unit)
MTU (message transfer unit), TCP packets and
multi-initiator SCSI
multi-word DMA
multimode fiber
multiporting
multiprocessor systems [See also processors; SMP]2nd
application behavior
busses
caches
communication basics 2nd
crossbars
Fireplane (Sun)
interconnectionless architectures
load average statistic
optimum processor load considerations
process queues
processor statistics
mpstat command
thread synchronization
threads
multitasking
mutexes
lockstat command
mpstat command
processor load and
mutual exclusion locks (mutexes)


index_N

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

Nagles algorithm 2nd
naive search algorithm
naming services, static performance issues
National Committee on Information Technology Standards (NCITES)
NCA (Network Cache Accelerator)
NCITES (National Committee on Information Technology Standards)
ndd command
Ethernet, changing operational mode
TCP tuning
nesting NFS mounts
netstat, error rate monitoring
network address
network administrator, responsibilities compared to systems administrator
Network Cache Accelerator (NCA)
Network File System [See NFS]
network layer (OSI model)
Network Lock Manager (NLM)
networks
architecture
ATM (Asynchronous Transfer Mode)
books about
bridges
broadcast frames
CDIR (classless interdomain routing)
clients
collisions
data links
duplex modes
FDDI (Fiber Distributed Data Interface)
filesystems, mounting
frames
gateways
interfaces, promiscuous mode
intermediate systems
internetworks
IP
latency, wide area networks and NFS
MAC (media access control)
OSI model architecture
packet size distribution
packets
pattern analysis
payload-free packets
performance factors
physical layer components
cables and connectors
interfaces
repeaters
routers
routing
scalability
servers
sizing, web servers and
subnetting
switches
TCP
three-way handshake
transparent bridges
UDP
unicast traffic
newfs command
cluster size, specifying
free space parameter
rotational delay parameter
NFS (Network File System)
buffer cache
CacheFS
CacheFS and
caches
DNLC (directory name lookup cache)
inode
client behavior
clients
bursty transfers
sequential transfers
disk subsystems, designing
memory requirements
nesting mounts
NVRAM
operating principles
operational statistics
overview
remote swapfiles
rnode cache
Samba and
servers
monitoring with nfsstat
tuning
types of
threads and
version, selecting
wide area networks and latency issues
workgroup servers and
nfsstat command
NFS operational statistics
NFS servers
NLM (Network Lock Manager)
non-unit stride cache-busting
nonuniform memory access (NUMA)
NOP (no-operation) instructions
nsswitch.conf file, static performance issues
NUMA (nonuniform memory access)
NVRAM
NFS
RAID arrays and


index_O

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

OLTP (online transaction processing), benchmarks
operating system
kernel variables, tuning
real-time process scheduling
optical bypass switch (FDDI network interface)
optimization [See performance tuning]
OSI model network architecture
outbound traffic
overhead, process accounting


index_P

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

P6 processor (Intel)
packet-switched protocols
packets 2nd
collisions
flow control
fragmentation
MSS (maximum segment size)
payload-free
size distribution
time-to-live
page scanner
paging
compared to swapping
disk problems and
free list
page coloring
swap space
anonymous memory
sizing
virtual memory
parallel member write policies (disk mirroring)
parameters
caches, changing
Ethernet, changing
maxcontig
maxusers
kernel tables
minfree
pt_cnt
rotational delay
parity context (disk striping)
parity-protected striping
parity-protected striping with independent disks
pattern analysis, networks
data transfer patterns
inverse request-response
message passing patterns
packet size distribution
request-response
payload-free packets
PCI (Peripheral Component Interconnect) bus
architecture
CompactPCI
PCI Special Interest Group web site
PCI-to-PCI bridge
perf file, sar command, enabling
performance accounting
sar command
SE Toolkit software
performance analysis tools
performance instrumentation counters (PICs)
performance tuning
definition
dynamic performance analysis
environment, importance of
Heisenbergs Principle of Uncertainty
I/O subsystem, importance of
latency
problems with, overview of
proxy servers
resource utilization
single-user workstations
filesystems
kernel
swap space
static factors
swap space
test design
throughput and latency considerations
trade-offs and
web servers
disks
filesystems
kernel
memory
network sizing
swap space
workgroup servers
disks
filesystems
kernel
memory
NFS
swap space
Peripheral Component Interconnect bus [See PCI bus]
peripherals
interconnect performance monitoring tools
PCI bus architecture
SBus architecture
persistent superblocks (Linux md package)
physical address
physical layer (OSI model)
cables and connectors
interfaces
PICs (performance instrumentation counters)
PIO (programmed I/O)
pipes, sticky bit
piplining
conditional branches
platters
plugs
pmap command, estimating memory requirements with
point-to-point Fibre Channel topology
political considerations, workload management
PowerPC processors, memory page size
preempting threads
prefetching, application optimization and
presentation layer (OSI model)
prex command
I/O tracing
restarting
priocntl command, process priorities, adjusting
priority paging
priority, processes
adjusting 2nd
determining 2nd
private IP addresses
probe-based application analysis
Solaris TNF
problem solving
algorithms
hardware level architecture
levels of transformation
memory hierarchy and
programming languages and
/proc directory (Linux), kernel variable tuning
process accounting
enabling
overhead
records, reviewing
process filtering, I/O tracing
process monitoring
prstat command
top command
process queues, viewing state
process scheduling
classes
dispatch queues
dispatch tables, tuning
dynamic priority
Linux
priorities
adjusting 2nd
determining 2nd
real-time
scheduling classes, listing
SMP systems
Solaris
Solaris threading model
static priority
processes
blocked
free list memory management
memory
estimating requirements
monitoring consumption
ps command
uax command
memory usage, displaying
virtual address spaces
processors [See also mulitprocessor systems; SMP]
architecture, specifying with compiler options
cache hierarchy
cache mapping
cache organization
cache size anomalies
cache-busters
caching
CISC design
clock rates
conditional branches
CPI (cycles per instruction)
critical path
DEC Alpha
memory page size
design basics
dispatch queues
failure
FISC design
graphics support
I/O wait time
idle time
improvements in
instruction stages
Intel 80x86, memory page size
Intel P6
internal clock rate
load average
load, vmstat command
locality
metrics, MIPS
microSPARC, memory page size
MIPS, memory page size
MMX extensions
mpstat command
optimum load considerations
performance monitoring
performance, importance of
piplining
PowerPC, memory page size
process accounting
process queues
process scheduling
psrinfo command
RISC design, second generation
sets, creating with psrset command
SPARC
LDSTUB instructions
memory page size
speed compared to memory speed
spin locking
state, changing with psradm command
Sun E-series
Sun I-series
superpiplined
superscalar 2nd
SuperSPARC, memory page size
system time
system time compared to user time
time command operation
Transmeta Crusoe 2nd
UltraSPARC, memory page size
user time
variable length instructions
VIS extensions
vmstat command
von Neumann model
profile feedback (compilers)
profiler-based application analysis
GNU gprof
profiling
application analysis
compiling with
executing with
program counter
programmed I/O (PIO)
programming code [See also applications]
algorithms
Boyer-Moore searching
Knuth-Morris-Pratt searching
string searching
analyzing
ptime command
time command
timex command
arithmetic optimization
compiler optimization considerations
compilers, profile feedback
fast compiler option
linked list cache-busting
loop optimization
microstate accounting, enabling
non-unit stride cache-busting
pitfalls
poor, reasons for
string optimization
TICK register, enabling
timing information, gethrtime system call
xarch compiler option
xchip compiler option
xcrossfile compiler option
xdepend compiler option
xinlining compiler option
xO compiler option
xprefetch compiler option
xsfpconst compiler option
xvector compiler option
programming languages, problem solving and
promiscuous mode (network interfaces)
protocol field (IP headers)
protocols
circuit-switched
IP
NLM (Network Lock Manager)
packet-switched
routing
TCP
three-way handshake
UDP
proxy servers, tuning
prstat command, process monitoring
prtconf command, synchronous transfer rate (SCSI)
ps command
process memory consumption
process priorities, determining 2nd
psradm command, processor state, changing
psrinfo command
processor failure monitoring
processor information
psrset command, processor sets, creating
pt_cnt parameter
ptime command
application analysis
disk write throttle


index_Q

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

QoS (quality of service) (ATM)
quantum (kernel threads)


index_R

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

RAID
arrays
creating with Linux md package
design considerations
autodetection
booting
deployment models
attribute-intensive home directories
data-intensive home directories
database environment
high performance computing
hardware-based array management
hot spare pools
large I/O applications
level 5
Linux md package
Solaris DiskSuite and
levels
distributed parity-protected striping
Hamming code arrays
mirrored striping
mirroring
overlapping
overview
parity-protected striping
parity-protected striping with independent disks
selecting
striping
Linux md package
logical volumes
MTBF (Mean Time Between Failure)
MTTDI (Mean Time To Data Inaccessibility)
MTTDL (Mean Time To Data Loss)
MTTR (Mean Time To Repair)
NVRAM and
software-based array management
Solstice DiskSuite
strip unit size, selecting
raidstart command
raidstop command
random access (disks)
RAID disk striping
storage capacity considerations
rate metric (SPECfp_rate95)
RDRAM (Direct Rambus)
read latency rate
read/write heads
reads (disks)
real-time scheduling
dispatch table
receive buffer (TCP)
refactored code
register file
registers, PCI busses
ReiserFS
overview
tail packing
renice command, process priority, adjusting 2nd
repeaters
replicas (Solstice DiskSuite state databases)
request-response patterns (networks)
resolv.conf file, static performance issues
resource utilization
resources
books
application performance
benchmarking
computer architecture
Java code optimization
networking
routing
TCP
Unix
workload management
environment, limiting users
management of
utilization, process accounting
resynchronization (disk mirroring)
retransmission (TCP)
rings (FDDI network interface)
RISC (reduced instruction set computing)
branch delay slots
instruction set length
second generation design
RJ-11 connectors
RJ-45 connectors
rnode cache, NFS
rnodes (filesystems)
rotational delay filesystem parameter
rotational position sense (RPS), misses
round robin distribution (interrupts)
routers
routing
books about
protocols
RPS (rotational position sense), misses
runacct command, process accounting


index_S

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

Samba, NFS and
sar command
disk partition statistics
enabling
inode cache monitoring
memory monitoring
NFS buffer cache
records, reviewing
swap space monitoring
SASI (Shugart Associates System Interface)
SBus (Sun)
architecture
burst transfer size
clock speed
interrupt processing
transfer mode
SC connectors
scalability, networks
scheduling classes
listing
SCSI (Small Computer System Interface)
bus transactions
bus utilization
command queuing
compared to IDE
different speed devices, mixing
differential signaling
disk caches
implementations
multi-initiator
synchronous compared to asynchronous transfers
termination
SCSI-3
SDRAM (synchronous DRAM)
SE Toolkit software
Second Extended Filesystem (ext2fs)
sectors
seek time (disks), reducing
segments, virtual memory
send buffer (TCP)
sequence number (TCP headers)
sequential access (disks)
RAID disk striping
serial member write policies (disk mirroring)
serial-optical coupler (SOC), Fibre Channel bus connections
servers
NFS
monitoring with nfsstat
tuning
types of
proxy
TCP/IP
web
disks
filesystems
kernel
memory
network sizing
swap space
workgroup
disks and
filesystems and
kernel and
memory and
NFS and
swap space and
session layer (OSI model)
shells, user limits on resources
Shugart Associates System Interface (SASI)
signal degradation, repeaters and
single-mode fiber (SMF)
size of
slave ports
sleep state (threads)
slow start algorithm (TCP)
Small Computer System Interface [See SCSI]
SMF (single-mode fiber)
SMP (symmetric mulitprocessor) systems, process scheduling
SOC (serial-optical coupler), Fibre Channel bus connections
soft real-time process scheduling
software [See also applications]
irqtune, Linux interrupt adjustments
Linux md package
RAID array management
SE Toolkit
Solstice DiskSuite 2nd
Solaris
cyclic caching
disk bottlenecks, locating
disk quotas, enabling
filesystem logging
filesystems, tmpfs
fsflush command
interrupts
kernel variable tuning
memory addressing limitations
memory page size, determining
memory requirements, determining
page coloring algorithms
performance accounting software
priority paging
process accounting, enabling
process scheduling
processor load, vmstat command
remote swapfiles
swap space, displaying status
swapfiles, creating
threading model
TNF
argument types
inserting probes
uax command, process memory consumption
ufs write throttling
vnodes
Solstice DiskSuite
filesystem logging
RAID array management
SPARC processors
LDSTUB instructions
memory page size
SPARC V9 instruction set
spatial locality, caches
SPEC benchmarks
SPECweb99 benchmark
speed optimized RISC processors
speed, attitude and requirements in modern world
spin locking
spindle
spindle synchronization 2nd
SRAM (static random access memory)
ST connectors
stacks
stalling (piplining)
state database (Solstice DiskSuite)
static interrupt distribution (Solaris)
static performance tuning
hardware failures and
static priority
static random access memory (SRAM)
step-index multimode fiber
sticky bit
storage-area networking
store-and-forward switches
STREAM memory benchmarking tool
strength reduction (arithmetic optimization)
string controllers, IPI-2
string optimization
string searching algorithms
string utilization, SCSI
stripe unit size, selecting
stripe width (RAID arrays)
striping disks
Linux md package
Solaris DiskSuite
submirrors
submirrors (RAID arrays)
subnet masks
subnetting
alternatives to
Sun
E-series processors
Fireplane
Gigaplane XB crossbar
I-series processors
MBus architecture
microSPARC processors, memory page size
SBus
architecture
burst transfer size
clock speed
interrupt processing
transfer mode
Solstice DiskSuite software 2nd
SPARC processors, memory page size
SuperSPARC processors, memory page size
UltraSPARC processors, memory page size
UltraSPARC-III architecture
VIS extensions
XBus architecture
superpipelined processors
superscalar processors 2nd
SuperSPARC processors, memory page size
swapfiles, creating (Solaris)
swapping
compared to paging
disk problems and
swap space
anonymous memory
monitoring
optimizing
sizing
tuning, single-user workstations
web servers
workgroup servers and
Unix mechanism
switches (networks)
SYN bit (TCP)
SYN flooding (denial-of-service attacks)
synchronous data transfers, SCSI
synchronous DRAM (SDRAM)
synchronous writes (disks)
system file, persisting maxpgio value changes
system time (processor)
System V model (process scheduling)
systems administrator, responsibilities compared to network administrator


index_T

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

T10 Technical Committee web site
T11 Technical Committee web site 2nd
T13 Technical Committee web site
tags, cache mapping
tail packing (ReiserFS)
target identifier, SCSI addresses
TCP (Transmission Control Protocol)
books about
compared to UDP
congestion window
data transfer rates
deferred acknowledgements
denial-of-service attacks, SYN flooding
FIN_WAIT_2 timer
headers
keepalive timer
low watermarks
Nagles algorithm
receive buffers
retransmission
send buffer
slow start algorithm
three-way handshake
TIME_WAIT timer
transmit high watermark
tuning with ndd command
window scaling
windows
TCP/IP
clients
servers
temporal locality, caches
terminal-driven database benchmarks
termination, SCSI
testing
design considerations
kernel variable tuning, importance of
Third Extended Filesystem (ext3fs)
thrashing
threads
barriers
categories
critical sections
dispatch queue
mulitprocessor systems
mutexes
NFS and
preempting
quantum
sleep state
Solaris model
spin locking
synchronization, mulitprocessor systems
three-way handshake (TCP)
TICK register, enabling
time command, application analysis
time-to-live (IP packets)
TIME_WAIT timer (TCP)
timeshare class kernel threads
timex command, application analysis
tiobench disk benchmark
TLB (transaction lookaside buffer)
tmpfs filesystem
TNF (trace normal form)
argument types
inserting probes
Token Bus
Token Ring
tools, performance analysis
top command, process monitoring
TP-DDI (Twisted Pair Distributed Data Interface )
TPC (Transaction Processing Council), benchmarks
tracks
transaction lookaside buffer (TLB)
Transmeta Crusoe processor 2nd
Transmission Control Protocol [See TPC]
transmit high watermark (TCP)
transparent bridges
transport layer (OSI model)
tunefs command
tuning [See performance tuning]
Twisted Pair Distributed Data Interface (TP-DDI)


index_U

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

uax command, process memory consumption
UDP (User Datagram Protocol)
compared to TCP
UFS (Unix Filesystem)
buffer cache
caching
cluster size
inode cache
inodes
overview
ufs write throttling (disks)
ulimit command, user limits
Ultra-ATA
UltraDMA
UltraSPARC Port Architecture switched interconnect (UPA)
UltraSPARC processors, memory page size
UltraSPARC-III
architecture
compiler flags
UMA (uniform memory access)
unicast network traffic
unit stride, programming code and
Universal Serial Bus (USB)
Unix
books about
memory swapping
Unix Filesystem [See UFS]
unswitched networks, resolving collision problems
UPA (UltraSPARC Port Architecture switched interconnect)
uptime command, processor load average
usage and performance agreements
USB (Universal Serial Bus)
User Datagram Protocol (UDP)
user defined benchmarks
automation considerations
problem set selection
rule creation
runtime selection
user time (processor)
user-space NFS servers
users
disk quotas
educating
environmental limits
limitations on
supported, setting number of
usage and performance agreements
utilization, resources
UTP (unshielded twisted-pair) cabling


index_V

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

variable length subnet mask (VLSM)
VAX MIPS
very long instruction word (VLIW)
virtual address space, processes
virtual memory [See also memory]
architecture
kernel variables
management, Linux
pages
segments
VIS extensions (Sun)
VLIW (very long instruction word)
VLSM (variable length subnet mask)
vmstat command
DNLC (directory name lookup cache)
intervals for
memory monitoring
memory requirements, determining
process queues
processor load
processor statistics
vnodes
von Neumann model
VUPs (VAX units of processing)
VxFS filesystem


index_W

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

watermarks
disk write throttling
TCP
web performance, benchmarking
web servers
disks
filesystems
kernel
memory
network sizing
swap space
web sites
hdparm command
JFS
lmbench memory benchmark
PCI Special Interest Group
SPEC benchmarks
STREAM benchmark
T10 Technical Committee
T11 Technical Committee 2nd
T13 Technical Committee
tiobench
TPC (Transaction Processing Council)
VxFS filesystem
wide area networks, NFS and latency issues
window scaling (TCP)
windowing, TCP packet flow control
workflow characterization
performance analysis tools
process accounting
workload management
workgroup servers
disks
filesystems
kernel
memory and
NFS
swap space
workload management
compared to financial transaction management
guidelines
maxusers parameter
political considerations
usage and performance agreements
users
limitations on
limiting resource environment
workstations (single-user)
filesystem tuning
kernel tuning
swap space tuning
world wide name (WWN)
write latency rate
write throttling (disks)
write-anywhere filesystems
write-back behavior (caches)
write-through behavior (caches)
writes (disks)
WWN (world wide name)


index_X

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

XBus architecture


index_Z

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X] [Z]

ZBR (zone bit rate recording)
zone bit rate (ZBR) recording