A memory chip fails inside a running mainframe. In a typical server, this would be a crash: a blue screen, a kernel panic, an outage. On an IBM Z system, the error is detected, the affected memory is reconstructed from redundant data held elsewhere in the memory subsystem, and the repair is logged. The running workload continues without interruption. No operator is notified. No ticket is opened. The system fixed itself.
This is not exceptional behavior for a mainframe. It is the baseline. The IBM Z platform is engineered from the ground up around one requirement: that hardware failures, software errors, and maintenance activities must not cause unplanned outages. The engineering discipline behind this is called RAS: Reliability, Availability, and Serviceability.
IBM coined the term RAS to describe mainframe design principles in the 1960s. It has since become a general industry term for fault-tolerant system design, but nowhere is it more deeply embedded in hardware and software architecture than on IBM Z and z/OS. This post explains what each component of RAS means, how it is implemented at the hardware and software levels, and why the combination produces the availability numbers that mainframes are known for.
The Three Dimensions
Reliability is the probability that the system produces correct results consistently. A reliable system does not silently produce wrong answers. It detects errors before they propagate, corrects what it can correct, and escalates what it cannot. IBM Z achieves reliability through pervasive hardware error detection: ECC on every bus, parity checking on every latch, and RAIM across the entire memory subsystem. The goal is that no undetected data corruption ever reaches an application.
Availability is the proportion of time the system is operational. A highly available system continues to function when components fail. It does this through redundancy (so no single component is a single point of failure), through fast automatic recovery (so failures that do occur are healed before users notice), and through planned maintenance that requires no service interruption. IBM Z targets five-nines (99.999%) to seven-nines (99.9999999%) availability depending on the configuration, corresponding to seconds to milliseconds of unplanned downtime per year.
Serviceability is the ease with which a problem can be diagnosed and repaired. A serviceable system captures diagnostic information at the moment of failure, makes that information accessible to support teams, and allows repairs to be performed concurrently with normal operation. IBM Z's serviceability features include hardware predictive failure analysis, software dump capture, and the ability to replace almost every hardware component without powering off the system.
Reliability: Getting the Answer Right
RAIM: Redundant Array of Independent Memory
The most significant reliability feature in IBM Z hardware is RAIM (Redundant Array of Independent Memory). Standard servers use ECC (Error Correcting Code) memory, which can detect and correct single-bit errors and detect multi-bit errors. ECC cannot survive the failure of an entire DRAM chip or memory channel. RAIM can.
RAIM stripes data across multiple independent memory channels, similar to how RAID stripes data across disks. If an entire memory channel fails, the data can be reconstructed from the remaining channels. If a DRAM chip fails, it is marked and spared: IBM Z uses DRAM marking, which permanently records that a specific DRAM is faulty and routes around it on subsequent accesses. The data held by the failed DRAM is reconstructed from ECC across the remaining DRAMs in the same channel. None of this requires operating system involvement. It happens in firmware, transparently to z/OS and every workload running on the machine.
The IBM z16 supports up to 40 TB of RAIM-protected memory per system. Every byte of that memory is continuously protected at a level that standard ECC cannot match.
Pervasive Error Detection in Hardware
IBM Z processors implement parity checking on every latch in the chip: the small storage elements that hold values as instructions execute. If a cosmic ray or electrical noise flips a bit in a latch, the parity check detects the inconsistency. The processor logs the error, retries the operation (many transient faults do not recur on retry), and if the error persists, marks the affected core as failed and moves its workload to a spare or another available core. The application running on that core continues without any interruption.
ECC is applied not just to main memory but to every internal bus, every cache, and every I/O interface in the system. The principle is that no data should travel from one component to another without a check that it arrived correctly.
Software Reliability: ESTAE and Recovery Routines
At the z/OS software level, reliability is implemented through a structured recovery framework. When a z/OS component encounters an unexpected condition (an abend, a program check, an impossible state), it does not simply crash. The z/OS error recovery architecture provides every component with the ability to register an ESTAE (Extended Specify Task Abnormal Exit) routine: a program that gains control when a failure occurs, captures diagnostic information, attempts to clean up, and decides whether to retry, percolate the error upward, or terminate gracefully.
This is the mechanism that causes an `ASRA` abend in a CICS task to abend only that task, not the entire CICS region. CICS registers its own ESTAE routine at startup. When z/OS detects a program check in the CICS address space, it passes control to the CICS ESTAE rather than terminating the address space. CICS captures the error, backs out the transaction, and continues processing other tasks. The region keeps running. The user whose transaction abended sees an error message. The other 3,000 users sharing the region notice nothing.
Availability: Staying Operational
Hardware Redundancy
IBM Z hardware is built with no single points of failure at the component level. Every critical subsystem has redundant paths:
I/O paths: Every disk volume (DASD) attached to the mainframe is reachable via multiple independent FICON (Fibre Channel connectivity) paths through multiple channel adapters. If one adapter fails, one channel cable is severed, or one switch in the fabric fails, the I/O subsystem automatically routes through an alternate path. The application sees a slight delay on the I/O that was in flight. It does not see a failure.
Power: IBM Z systems have redundant power supplies and power distribution units. A single power supply failure does not cause an outage. Most large installations add external UPS (Uninterruptible Power Supply) systems and dual-feed power from separate utility circuits.
Cooling: IBM Z machines have redundant cooling systems. A fan or cooling module failure is detected and logged, and the remaining cooling capacity maintains temperature within safe limits while the failed unit is replaced.
Processor sparing: The IBM Z CPC has more physical processors than are configured for customer use. Spare processors sit idle and are activated automatically if a configured processor develops a permanent fault. The workload migrates to the spare without interruption.
Concurrent Maintenance
One of the most operationally significant availability features is concurrent maintenance: the ability to replace failed hardware components while the system continues running. On IBM Z, an engineer can replace a failed memory book, a failed I/O adapter, or a failed power supply without scheduling a maintenance window or interrupting the workload. The hardware is designed so that each replaceable component can be isolated, swapped, and reintegrated online.
This extends to microcode updates. IBM Z firmware can be updated while the system runs, using a rolling update process that updates one subsystem at a time while the others continue to handle workload. A firmware patch that would require a weekend maintenance window on a commodity server is applied during business hours on a mainframe, invisibly to the applications.
LPAR Isolation
As covered in the Architecture: Mainframe post, LPAR isolation is enforced by the PR/SM hypervisor at the firmware level. A software failure, including a complete operating system crash, in one LPAR cannot affect any other LPAR on the same physical machine. A z/OS LPAR can crash and restart while a Linux LPAR on the same machine continues to run without interruption.
This isolation is stronger than hypervisor isolation on commodity platforms because PR/SM runs below the operating system in firmware rather than as a privileged OS process. There is no code path by which an OS-level fault can bypass it.
Automatic Restart Manager
ARM (Automatic Restart Manager) is a z/OS component that monitors registered subsystems and restarts them automatically if they fail. CICS regions, DB2 subsystems, MQ queue managers, and other critical started tasks register with ARM at startup. If any of them terminates abnormally, ARM restarts them according to a defined policy, without operator intervention.
ARM policies specify:
- Which element (subsystem) to restart.
- The order of restarts when dependencies exist (RRS must restart before DB2, DB2 before CICS).
- How many restart attempts to make before escalating to an operator.
- Whether to restart on the same LPAR or on another member of the Parallel Sysplex if the original LPAR is unavailable.
The sysplex-aware restart is the critical feature. If an entire LPAR fails, ARM detects the failure via the XCF (Cross-system Coupling Facility) structure and begins restarting the affected subsystems on surviving LPARs. CICS TORs, AORs, and FORs, DB2 members, and MQ queue managers all restart on the surviving systems. Because CICS and DB2 maintain their shared state in the Coupling Facility rather than in LPAR-local memory, the restarted subsystems pick up where the failed ones left off.
Sysplex Failure Manager
SFM (Sysplex Failure Manager) manages the response when an entire sysplex member fails or becomes unresponsive. If a member stops communicating via XCF, SFM declares it failed and triggers the recovery sequence: ARM restarts its subsystems elsewhere in the sysplex, the Coupling Facility releases any retained locks held by the failed member, and DB2 performs its group restart to resolve in-doubt transactions.
SFM also handles the case where a member is still running but has lost connectivity to the CF (a "CF partitioning" scenario). In this case, SFM must decide which partition has the correct view of shared state and terminate the other, to prevent split-brain data corruption. This decision is guided by the SFM policy configured by the installation.
Serviceability: Finding and Fixing Problems Fast
Predictive Failure Analysis
IBM Z hardware continuously monitors its own health and compares current readings against historical patterns. Components that are degrading but have not yet failed produce characteristic signatures: elevated error rates on a memory channel, increased retry counts on an I/O adapter, anomalous temperature readings in a cooling module. IBM's predictive failure analysis algorithms detect these signatures and raise alerts before the component causes an unplanned outage.
When predictive failure analysis flags a component, IBM service engineers are notified automatically via the Hardware Management Console (HMC). They schedule a concurrent repair during normal business hours, replacing the component before it fails. The alternative, waiting for the component to fail unplanned, would trigger the hardware redundancy and recovery mechanisms described above. But predictive replacement eliminates even the brief disruption of an unplanned failover.
The Hardware Management Console
The HMC (Hardware Management Console) is the management interface for IBM Z hardware. It provides a graphical console for managing LPARs, monitoring hardware health, applying microcode updates, and viewing hardware error logs. Every hardware error detected by the machine is recorded in the HMC error log with a timestamp, affected component, error type, and recovery action taken.
The HMC also provides remote support access: IBM service engineers can connect to the HMC remotely (with customer authorization) to diagnose hardware problems and apply microcode fixes without a site visit. For a machine running in a data center in Frankfurt, IBM support in Poughkeepsie can diagnose and apply a microcode patch remotely within hours of a problem being reported.
Software Diagnostic Capture: SVC Dump and SDUMP
When a software failure occurs in z/OS, the operating system captures a dump: a snapshot of the failing address space's memory, registers, and control blocks at the moment of failure. There are two primary dump mechanisms:
An SVC dump (Supervisor Call dump) is requested by a failing component to capture its own state plus selected other address spaces. CICS requests an SVC dump when it terminates abnormally: the dump captures the CICS address space in full, including all task control areas, DSA storage, program storage, and the recent trace entries. This dump is written to a DASD dataset (a SYS1.DUMPxx dataset) and is later analyzed with IPCS (Interactive Problem Control System).
An SDUMP is a system dump that captures multiple address spaces simultaneously. It is appropriate for problems that span address space boundaries, such as an abend in the interface between CICS and DB2. The SDUMP captures both the CICS and DB2 address spaces, along with z/OS common storage areas, in a single atomic snapshot.
The combination of the dump content and the SMF records (covered in the previous post) gives IBM support engineers the full picture of what was happening at the moment of failure: the system state, the recent event history, and the resource consumption pattern that preceded the failure.
IPCS: Analyzing Dumps
IPCS (Interactive Problem Control System) is the z/OS tool for analyzing dumps. It runs as an ISPF application under TSO and provides a command interface for navigating dump contents. IBM provides IPCS subcommands specific to each major subsystem: `VERBX DFHPD` formats a CICS dump, showing the program link stack, task control areas, DSA usage, and recent trace. `VERBX DB2` formats a DB2 dump. Each subcommand produces structured output that maps directly to the subsystem's internal control blocks.
For cloud-native engineers used to structured logs and distributed tracing tools, IPCS feels archaic. But it provides a level of diagnostic depth that has no equivalent elsewhere: a CICS dump analyzed with IPCS can show exactly which line of COBOL code was executing, what was in every Working Storage field, what CICS commands had been issued in the preceding milliseconds, and what storage was allocated and to which task. This granularity is what makes production problem diagnosis on z/OS tractable even for failures that reproduce only sporadically.
The Availability Numbers in Context
IBM quotes up to 99.9999999% availability (seven nines) for a properly configured Parallel Sysplex. Converting this to downtime: seven nines is approximately 31 milliseconds of unplanned downtime per year. Five nines (99.999%) is about 5 minutes per year. These numbers require the full stack: RAIM-protected hardware, redundant I/O paths, LPAR isolation, a Parallel Sysplex with a correctly configured Coupling Facility, ARM policies for all critical subsystems, and SFM for sysplex-level failure management.
Achieving these numbers also requires operational discipline. The hardware and software provide the capability. Whether an installation achieves it depends on how the system is configured, maintained, and operated. A Parallel Sysplex with a single Coupling Facility is not as available as one with duplexed CF structures. An ARM policy that has never been tested under failure conditions may not work as expected when a real failure occurs. RAS is not a property that a mainframe has by default. It is a property that an installation builds and maintains.
RAS and the Application
Hardware and system-level RAS provides the platform, but applications must also be designed to participate. A COBOL program that does not handle error conditions, does not implement syncpoints, and does not register with ARM contributes to availability failures even on a perfect platform.
Application-level RAS on z/OS means: using CICS transaction backout to ensure data consistency on failure, designing jobs to be restartable at a known checkpoint rather than from the beginning, using DB2 with appropriate isolation levels and lock management to avoid long-running lock holders, and registering critical started tasks with ARM so that they recover without operator intervention.
The mainframe's RAS features do not compensate for badly designed applications. RAIM prevents memory corruption. ESTAE prevents a single task's failure from crashing the region. The Parallel Sysplex prevents a machine failure from causing an outage. But a CICS application that leaks shared storage, a batch job that holds exclusive locks on shared datasets for hours, or a DB2 application that repeatedly deadlocks will degrade availability regardless of the platform underneath it. RAS is a property of the whole stack, hardware through application, not just the infrastructure.
Summary
RAS is the engineering philosophy that makes mainframes different from every other computing platform. Reliability through pervasive error detection and correction: RAIM prevents memory failures from producing wrong results; ESTAE routines contain software failures to the component that experienced them. Availability through redundancy and automatic recovery: no single hardware component is a single point of failure; ARM restarts subsystems automatically; Parallel Sysplex ensures that an LPAR failure causes neither data loss nor outage. Serviceability through predictive analysis and deep diagnostic capture: failing components are identified and replaced before they cause outages; software failures produce rich diagnostic dumps that make root cause analysis tractable.
The nines of availability that mainframes achieve are not marketing claims. They are the measurable result of 60 years of systematic engineering, applied at every layer from DRAM chip design to Coupling Facility lock structures to z/OS ESTAE recovery routines. Understanding RAS is understanding why the mainframe is still the right platform for the workloads that cannot afford to be wrong or unavailable.
Part of the Mainframe Decoded series — IBM Z and z/OS, clearly explained for engineers.