Alarms and fault handling

Warnings, alarms and trips

Ask three engineers what an alarm is and you will get four answers. Settle the words before writing a line of code, because they decide what the program must do.

A warning tells the operator something is drifting: a filter differential pressure creeping up, a lubrication tank at 20 %. The machine keeps running and nobody has to press anything.

An alarm demands an operator response within a defined time. It stays on the alarm list until someone acknowledges it, even if the condition has already gone away. ISA-18.2 and EEMUA 191, the two documents most alarm philosophies are written from, are blunt about this: if nobody needs to act, it is not an alarm. It is an event, and it belongs in a log.

A trip (or fault) stops equipment. The program drives outputs to a safe state on its own, without waiting for anyone, and refuses to restart until a person has confirmed it is safe.

The same physical condition often produces all three. A gearbox oil temperature at 75 °C is a warning, at 85 °C an alarm, and at 95 °C a trip. That is three thresholds and three behaviours from one sensor, and three entries in the alarm philosophy that say who must do what at each.

Most alarm-handling failures are not bugs in a single line. The usual failure is a plant with 400 configured alarms, 60 of them active at any moment, and operators who acknowledge the horn without reading the screen. EEMUA 191 suggests that an operator can handle about one alarm every ten minutes in steady operation. Every alarm you add should be one somebody would be grateful for at 3 a.m.

The documents behind an alarm philosophy

ANSI/ISA-18.2, Management of Alarm Systems for the Process Industries, was first published in 2009, and IEC 62682 is its international counterpart. EEMUA 191 is older: the Engineering Equipment and Materials Users' Association first published it in the UK in 1999. The incident usually cited behind it is the 1994 explosion at the Milford Haven refinery, where the operators in the last minutes before it were facing far more alarms than anyone could read. All three were written for process plants with control rooms, but the ideas carry straight over to packaging lines, machine tools and water works.

The performance figures that most alarm philosophies quote from ISA-18.2 and EEMUA 191 are worth knowing, because a customer's specification may hold you to them:

The numbers are guidance, not hard limits, but they are useful in a design review. A filling line with twelve operators' worth of alarms on one panel fails them before it is built.

Detect: a condition and a delay

Raw conditions chatter. A float switch bounces as the surface ripples, a pressure reading dips for 200 ms when a valve opens elsewhere, a flow switch reads "no flow" for the first two seconds after a pump starts because the water has not got there yet. If you alarm on the raw condition, you get nuisance alarms, and nuisance alarms train operators to ignore real ones.

So detection has two parts. First, qualify the condition so it only counts when it means something: no-flow only matters while the pump is supposed to be running. Second, delay it with a TON so it must persist before it counts. Chapter 19 applied the same idea to analog alarms with deadbands; here the focus is on what happens after detection.

PROGRAM Main
VAR
    PumpRunning : BOOL;   (* input: contactor feedback *)
    FlowSwitch  : BOOL;   (* input: paddle flow switch, TRUE = flow *)
    NoFlowCond  : BOOL;
    FlowDelay   : TON;
    LowFlow     : BOOL;   (* detected alarm condition *)
END_VAR

(* Qualify: "no flow" only means something while the pump runs *)
NoFlowCond := PumpRunning AND NOT FlowSwitch;

(* Delay: 3 s covers pipe fill time after a start *)
FlowDelay(IN := NoFlowCond, PT := T#3s);
LowFlow := FlowDelay.Q;

END_PROGRAM

Switch PumpRunning on and leave FlowSwitch off. FlowDelay.ET climbs and LowFlow comes on after 3 s. Switch FlowSwitch on at 2 s instead and the timer resets: a normal start, no alarm. Choose the delay from the process, not from habit. Pipe fill time, valve stroke time and transmitter response are all measurable on commissioning; 3 s here is a typical figure for a few metres of DN50 pipe.

ISA-18.2 has names for both halves. Qualifying an alarm by plant state, as PumpRunning does here, is suppression by design: the alarm is deliberately not shown when the plant is in a state where it means nothing. The TON is an on-delay. There is a matching off-delay, which keeps an alarm active for a while after its condition clears, so a signal hovering at the threshold does not raise and clear the same alarm forty times an hour. The function block later in this chapter has both.

LowFlow is not yet an alarm. It follows the condition: when flow returns, it vanishes. If it disappeared in 4 seconds while the operator was on the other side of the line, nobody would ever know it happened. That is what latching is for.

Latch until acknowledged

An alarm has two independent pieces of state, and mixing them up is the root of most broken alarm code. One is whether the condition is still present, which detection tells you scan by scan. The other is whether the operator has seen it, which is memory, and only the operator's acknowledge clears it.

Together they give four states, and the alarm stays on the list in three of them:

The third row is the one beginners miss. A condition that came and went must stay visible until somebody acknowledges it, otherwise a 4-second overload at 2 a.m. leaves no trace. The second row means acknowledging does not make the alarm go away; it only says "I know". The alarm leaves the list only when the condition is gone and it has been acknowledged.

PROGRAM Main
VAR
    HighTemp : BOOL;    (* input: detected condition (already delayed) *)
    Ack      : BOOL;    (* input: acknowledge push button *)
    Shown    : BOOL;    (* on the alarm list *)
    Unacked  : BOOL;    (* waiting for the operator *)
    Horn     : BOOL;    (* output *)
    Lamp     : BOOL;    (* output: flashing = unacknowledged *)
    FlashT   : TON;
    Flash    : BOOL;
END_VAR

(* A new occurrence: show it and demand an acknowledge *)
IF HighTemp AND NOT Shown THEN
    Shown   := TRUE;
    Unacked := TRUE;
END_IF;

IF Ack THEN
    Unacked := FALSE;
END_IF;

(* Leaves the list only when gone AND acknowledged *)
IF NOT HighTemp AND NOT Unacked THEN
    Shown := FALSE;
END_IF;

(* 1 Hz flasher *)
FlashT(IN := NOT FlashT.Q, PT := T#500ms);
IF FlashT.Q THEN
    Flash := NOT Flash;
END_IF;

Horn := Unacked;
Lamp := Shown AND (Flash OR NOT Unacked);

END_PROGRAM

Try both orders. Switch HighTemp on, then Ack (and release it): the horn stops, the lamp goes steady, the alarm stays shown until you switch HighTemp off. Now switch HighTemp on and off again without acknowledging: the lamp keeps flashing on a condition that is already gone, until Ack clears it.

The flasher deserves a look. FlashT(IN := NOT FlashT.Q, …) is a timer that restarts itself: when Q comes on, the next call sees IN FALSE and resets, and the call after that starts timing again. Each time Q pulses, Flash toggles, so the lamp changes state roughly every half second. On a real controller you would usually take the flash from a system clock instead, so every lamp on the panel flashes in step: S7 CPUs can provide clock memory bits (a byte of bits that toggle at fixed frequencies, set in the CPU properties), and a single shared flasher in CODESYS or Logix does the same job. Lamps flashing out of step on one panel look like a fault to the operator.

Annunciator sequences

The behaviour in the table is older than PLCs. Hardwired annunciator panels, rows of backlit windows with a horn, implemented it with relays, and ISA-18.1 gave each variant a name. The one programmed above locks in: a condition that clears before acknowledgement keeps flashing until someone acknowledges it, and after acknowledgement it goes out by itself when the condition clears. Two variants are common enough to know. A manual-reset sequence keeps the window lit after the condition clears until a separate reset button is pressed. A ringback sequence signals the return to normal as well, with a slower flash and a different tone, so the operator knows the condition has gone and can confirm it. Ringback is worth its extra code on equipment where the operator must act when a fault clears, for example to restart a standby pump.

One alarm, one function block

Every alarm on a machine follows the same rules, so the rules belong in one function block (chapter 18) with one instance per alarm. The block below takes the qualified condition and the acknowledge button, applies an on-delay and an off-delay, takes the acknowledge edge itself, and reports the three things an HMI needs: is it active, is it on the list, is it waiting for the operator.

FUNCTION_BLOCK FB_Alarm
VAR_INPUT
    Cond     : BOOL;    (* qualified condition *)
    Ack      : BOOL;    (* acknowledge button; the edge is taken here *)
    OnDelay  : TIME;    (* condition must persist this long *)
    OffDelay : TIME;    (* and must be gone this long before it clears *)
END_VAR
VAR_OUTPUT
    Active  : BOOL;     (* condition after both delays *)
    Shown   : BOOL;     (* on the alarm list *)
    Unacked : BOOL;     (* waiting for the operator *)
END_VAR
VAR
    OnT     : TON;
    OffT    : TON;
    AckEdge : R_TRIG;
END_VAR

OnT(IN := Cond, PT := OnDelay);
OffT(IN := NOT Cond, PT := OffDelay);
IF OnT.Q THEN
    Active := TRUE;
ELSIF OffT.Q THEN
    Active := FALSE;
END_IF;

AckEdge(CLK := Ack);
IF Active AND NOT Shown THEN
    Shown   := TRUE;
    Unacked := TRUE;
END_IF;
IF AckEdge.Q THEN
    Unacked := FALSE;
END_IF;
IF NOT Active AND NOT Unacked THEN
    Shown := FALSE;
END_IF;
END_FUNCTION_BLOCK

PROGRAM Main
VAR
    BearingHot    : BOOL;   (* input: bearing temperature above limit *)
    FilterBlocked : BOOL;   (* input: filter differential pressure switch *)
    Ack           : BOOL;   (* input: acknowledge push button *)
    fbBearing     : FB_Alarm;
    fbFilter      : FB_Alarm;
    Horn          : BOOL;   (* output *)
END_VAR

fbBearing(Cond := BearingHot, Ack := Ack, OnDelay := T#1s, OffDelay := T#5s);
fbFilter(Cond := FilterBlocked, Ack := Ack, OnDelay := T#10s, OffDelay := T#2s);

Horn := fbBearing.Unacked OR fbFilter.Unacked;

END_PROGRAM

Switch BearingHot on: one second later the bearing alarm is active, shown and unacknowledged, and the horn sounds. Switch it off and on a few times inside five seconds and Active stays TRUE throughout, because the off-delay will not clear it until the condition has been gone for five seconds. That is the off-delay doing its job on a signal that hovers at the limit. Hold Ack down, then switch FilterBlocked on: after ten seconds the filter alarm sounds the horn even though Ack is still pressed, because the block only acknowledges on an edge.

The delays differ because the processes differ. A hot bearing is real after a second. A filter switch flickers whenever a pump starts, so it must stay made for ten seconds before anyone is called. Choosing those numbers is a process decision; the block only makes sure every alarm applies its numbers the same way.

The design point is that every alarm now behaves identically, and a fix lands everywhere at once. When the site decides that the horn should re-sound if an acknowledged alarm clears and comes back, you change FB_Alarm and every instance gets it. Add a Shelved input later and you have ISA-18.2 shelving, the controlled, time-limited removal of a nuisance alarm from the list, on every alarm in the plant.

Acknowledge is not reset

Two buttons sit on most operator panels, often next to each other, and they mean different things.

Acknowledge is about people. It says "I have seen this alarm". It silences the horn and stops the flashing. It never changes what the machine does.

Reset is about the machine. It says "the cause has been dealt with; you may leave the tripped state". It clears latched trips so the equipment can be started again, but only if the condition that caused the trip is gone. A reset pressed while the motor is still overheating does nothing, and the code must make sure of that rather than trusting the operator to check.

Some plants combine them into one button that acknowledges on the first press and resets on the second. That works as long as the program still keeps the two states separate internally. A reset that clears a trip whose cause is still present is never acceptable, because the next start command will run straight into the same fault, or start a pump against a closed valve that the trip was protecting.

ISA-18.2 calls an alarm that needs a reset as well as an acknowledge a latching alarm. Use them sparingly, for trips where a person must look at the equipment before it runs again. A plant where every alarm latches teaches operators to press reset as a reflex, which defeats the purpose.

Alarm systems in the vendor tools

On a real project you will usually not write the alarm list yourself. Every platform has an alarm system that stores, timestamps and displays alarms, and your PLC code feeds it. What the code still owns is the detection, the qualification and the trip reaction.

In Studio 5000, the digital alarm instruction ALMD runs on a tag of type ALARM_DIGITAL. It has the state machine from this chapter built in: In is the condition, ProgAck acknowledges and ProgReset resets on a false-to-true transition, and the tag's configuration says whether acknowledgement is required and whether the alarm latches. There is an analog counterpart, ALMA, with high-high, high, low and low-low limits and deadbands. In a Structured Text routine the call takes the tag first, then the operands in order:

// Studio 5000 ST, vendor syntax. Tags: BearingHot_ALM : ALARM_DIGITAL,
// BearingHot_Cond, HMI_Ack, HMI_Reset : BOOL
ALMD(BearingHot_ALM, BearingHot_Cond, HMI_Ack, HMI_Reset, 0, 0);
//   tag             In               ProgAck  ProgReset  ProgDisable  ProgEnable

In TIA Portal on an S7-1500, the Program_Alarm instruction, called inside an FB, raises a program alarm when its SIG input changes. The CPU itself timestamps the change and sends the alarm to every connected HMI, so the time on the screen is the controller's time, not the panel's. ProDiag, an option for S7-1500, goes further and generates supervision alarms on tags and FB interfaces without explicit code. On classic HMI configurations you will also meet discrete alarms in WinCC, where each bit of an alarm word in a data block triggers one message.

In TwinCAT 3, alarms are events of the EventLogger. You declare event classes in the type system, create an FB_TcAlarm instance for each alarm, and call its Raise, Clear and Confirm methods; the instance reports whether it is waiting for confirmation and when it was raised, cleared and confirmed. CODESYS has an Alarm Manager configured with alarm groups and classes, where the class decides whether an alarm needs acknowledgement.

All of these move the alarm list out of your code and into the product, which is the right place for it. None of them decides when a no-flow condition means something, or which outputs a trip must drive and in which direction. That stays in your ST.

First-out: which fault came first

When a machine trips, faults cascade. A compressor loses oil pressure, the motor trips on overload a fraction of a second later, discharge pressure falls, and three alarms appear on the screen within one second. The first one is the cause and the others are consequences. An engineer who chases the overload will replace a perfectly good motor starter.

A first-out record captures the cause. The rule is simple: when an alarm becomes active and no first-out is recorded yet, record this one. Leave the record alone until all alarms have cleared.

This is the natural place for an array of structures (chapter 15): one STRUCT per alarm holding its state, an array of them, a parallel array of delay timers, and one FOR loop that applies the same rules to all of them.

TYPE ST_Alarm :
STRUCT
    Present : BOOL;   (* condition, after its delay *)
    Shown   : BOOL;   (* on the alarm list *)
    Unacked : BOOL;   (* waiting for the operator *)
END_STRUCT
END_TYPE

PROGRAM Main
VAR CONSTANT
    N_ALARMS : INT := 3;
END_VAR
VAR
    LowOilPress     : BOOL;   (* input *)
    HighBearingTemp : BOOL;   (* input *)
    MotorOverload   : BOOL;   (* input *)
    Ack             : BOOL;   (* input *)
    Cond     : ARRAY[1..3] OF BOOL;
    Delay    : ARRAY[1..3] OF TON;
    Alarms   : ARRAY[1..3] OF ST_Alarm;
    AckEdge  : R_TRIG;
    FirstOut : INT;           (* 0 = none, else the alarm number *)
    AnyShown : BOOL;
    Horn     : BOOL;
    i        : INT;
END_VAR

Cond[1] := LowOilPress;
Cond[2] := HighBearingTemp;
Cond[3] := MotorOverload;
AckEdge(CLK := Ack);

AnyShown := FALSE;
Horn     := FALSE;
FOR i := 1 TO N_ALARMS DO
    Delay[i](IN := Cond[i], PT := T#300ms);
    Alarms[i].Present := Delay[i].Q;

    IF Alarms[i].Present AND NOT Alarms[i].Shown THEN
        Alarms[i].Shown   := TRUE;
        Alarms[i].Unacked := TRUE;
        IF FirstOut = 0 THEN
            FirstOut := i;
        END_IF;
    END_IF;
    IF AckEdge.Q THEN
        Alarms[i].Unacked := FALSE;
    END_IF;
    IF NOT Alarms[i].Present AND NOT Alarms[i].Unacked THEN
        Alarms[i].Shown := FALSE;
    END_IF;

    AnyShown := AnyShown OR Alarms[i].Shown;
    Horn     := Horn OR Alarms[i].Unacked;
END_FOR;

IF NOT AnyShown THEN
    FirstOut := 0;            (* list empty: ready for the next event *)
END_IF;

END_PROGRAM

Switch on HighBearingTemp, wait a moment, then MotorOverload. FirstOut shows 2 and stays 2. Acknowledge, switch both off, and once the list is empty FirstOut returns to 0. Adding a fourth alarm means one more line in the mapping section and a bigger array; the rules themselves are written once.

When the delays reorder the faults

The loop above gives every alarm the same 300 ms delay, and that hides a trap. On a real machine the delays differ, because each one is chosen for its own signal, and a first-out recorded after the delays can name the wrong fault.

The fix is to record when each raw condition came on, and to use those times to decide the first-out once a trip has been confirmed. The delays still filter out the dips; they no longer decide the order.

PROGRAM Main
VAR
    LowOilRaw   : BOOL;   (* input: oil pressure switch open *)
    OverloadRaw : BOOL;   (* input: motor protection relay *)
    Reset       : BOOL;   (* input: reset push button *)
    Scan        : DINT;   (* a scan counter stands in for a clock here *)
    OilSince    : DINT;   (* scan the raw condition came on; 0 = not present *)
    OvlSince    : DINT;
    OilDelay    : TON;
    OvlDelay    : TON;
    Tripped     : BOOL;
    FirstOut    : INT;    (* 0 = none, 1 = low oil, 2 = overload *)
END_VAR

Scan := Scan + 1;

(* Remember when each raw condition started *)
IF NOT LowOilRaw THEN
    OilSince := 0;
ELSIF OilSince = 0 THEN
    OilSince := Scan;
END_IF;
IF NOT OverloadRaw THEN
    OvlSince := 0;
ELSIF OvlSince = 0 THEN
    OvlSince := Scan;
END_IF;

(* The delays decide whether it is real *)
OilDelay(IN := LowOilRaw, PT := T#1s);
OvlDelay(IN := OverloadRaw, PT := T#200ms);

(* The raw times decide the order *)
IF (OilDelay.Q OR OvlDelay.Q) AND NOT Tripped THEN
    Tripped := TRUE;
    IF OilSince > 0 AND (OvlSince = 0 OR OilSince <= OvlSince) THEN
        FirstOut := 1;
    ELSE
        FirstOut := 2;
    END_IF;
END_IF;

IF Reset AND NOT LowOilRaw AND NOT OverloadRaw THEN
    Tripped  := FALSE;
    FirstOut := 0;
END_IF;

END_PROGRAM

Switch LowOilRaw on, then OverloadRaw a few tenths of a second later. The trip confirms when the overload's 200 ms delay runs out, and FirstOut is 1, because OilSince is the smaller number. Switch them on in the other order and it reads 2. A short dip on LowOilRaw alone still trips nothing, because its own delay never finishes. On a real controller, store a timestamp instead of a scan count: a millisecond system time, or the controller's wall clock if the first-out has to be lined up with an HMI log.

A safe state for every output

When a trip latches, the program must drive every affected output to its safe state. The beginner's version is "turn everything off". On many machines that is right. On many others it is dangerous.

Consider a 2,000-litre jacketed reactor running an exothermic batch:

The same thinking applies on discrete machines. A gripper holding a 30 kg part should hold, not open. A vertical axis should brake, not coast. A press should stop at the top of the stroke if it can. The safe state is a process decision, made with the people who know the process, written down per output, and implemented in one place: the command section, where each output has exactly one owner (chapter 21).

PROGRAM Main
VAR
    Running      : BOOL;   (* input: batch in progress *)
    Trip         : BOOL;   (* input: latched process trip *)
    FeedPump     : BOOL;   (* output *)
    SteamValve   : BOOL;   (* output, TRUE = open *)
    CoolingValve : BOOL;   (* output, TRUE = open *)
    Agitator     : BOOL;   (* output *)
END_VAR

(* Command section: the trip is applied once, per output, in its own direction *)
FeedPump     := Running AND NOT Trip;
SteamValve   := Running AND NOT Trip;
CoolingValve := Trip;     (* normal temperature control omitted here *)
Agitator     := Running OR Trip;

END_PROGRAM

Switch Running on, then Trip. Feed and steam stop, cooling opens fully, and the agitator keeps turning even if Running drops. Every output names its trip behaviour in its own line. Nobody has to hunt for a "fault routine" that overrides outputs from somewhere else.

How hard to stop

Not every trip should stop a machine the same way. IEC 60204-1, the standard for the electrical equipment of machines, defines three stop categories, and they give a useful vocabulary for the reaction to a process fault as well:

In a trip table, write the category next to each fault. A jammed capper wants category 2 (finish the cycle, hold, keep the torque on so nothing drops). A drive overtemperature wants category 1. Writing "stop" alone leaves the decision to whoever writes the code that night.

The categories also apply to safety stop functions, and there they must be done by the safety system, not by the standard program.

When the PLC itself stops

Your command section can only drive outputs while the program runs. When the CPU goes to STOP, faults, or loses its network connection to a remote I/O station, the output modules decide what happens, and they do it from their configuration, not from your code. Siemens output modules on ET 200SP and S7-1500 have a reaction to CPU STOP parameter (switch off, keep last value, or output a substitute value). ControlLogix output modules have separately configurable output states for Program mode and for a communication fault (off, on, or hold the last state). CODESYS and TwinCAT fieldbus devices have similar settings per device or per terminal.

Check those parameters on every project, output by output, against the safe-state table. The default is usually off, which is right for most outputs and wrong for the cooling valve in the table above. A reactor whose cooling closes because someone put the PLC in STOP to download a change is a failure the program cannot prevent, because the program is not running.

No automatic restart

When a trip clears, the machine must not start by itself. EN 60204-1 states it plainly for stop functions and power restoration: resetting must not on its own initiate a restart. The machine waits in a stopped state until an operator has reset the trip and then gives a fresh, deliberate start command.

The classic bug is a start latch that survives the trip. A conveyor's run latch is written Run := (Start OR Run) AND NOT Stop;, and the trip is applied only at the output: Motor := Run AND NOT Tripped;. The motor stops on a trip. Then the fitter clears the jam, somebody presses reset, and the conveyor lurches forward with the fitter's hand still on the belt, because Run never dropped.

Two rules prevent it:

1. The trip drops the run latch itself, not just the output. After a trip, the machine's memory says "stopped".
2. Start on an edge. A start button that is held, jammed or bridged by a stuck HMI bit while the reset is pressed must not count as a new start. Use an R_TRIG, and only accept the edge while the machine is not tripped. An edge that arrives while tripped is simply lost; the operator presses again.

Restart rules also cover power-up. After a power failure, retained variables come back with their old values. A VAR RETAIN run latch would restart a mixer the moment the 24 V supply returns. Run latches belong in non-retained memory, so the machine powers up stopped; counts, totals and recipe values are what you retain. Studio 5000 has a direct way to say this in ST: the non-retentive assignment Run [:=] … clears its target when the controller goes to Run, so a latch written that way cannot survive a mode change.

Should a trip survive a power cycle?

The run latch must not. The trip latch is a different question, and I would retain it on most machines.

The failure looks like this. A pump trips on high discharge pressure. The operator presses reset, nothing happens because the pressure switch is still made, and after the third try somebody opens the panel and switches the PLC supply off and on. With a non-retained trip latch, the PLC powers up with Tripped FALSE and the first-out cleared. If the pressure switch has meanwhile relaxed, the pump can be started straight back into the blockage, and the evidence of what happened has gone. With a retained trip latch and first-out, the machine powers up still tripped, showing the original cause, and someone has to deal with it.

Retaining the trip does not restart anything, because the run latch is still non-retained and a start still needs a fresh edge. Write the decision down in the alarm philosophy either way. "Power cycle clears trips" should be a choice somebody made on purpose, because operators will find it within a week.

Writing the alarm text

The code decides when an alarm appears; the text decides whether anyone can act on it. A message the operator can use says what has happened, where, and what to do, in that order, and fits on one line of the alarm banner:

Put the device tag from the drawing in every message, the same tag your variable names carry (chapter 21), so the operator, the technician and the code all use one word for one thing. Say what the program has already done ("inlet closed automatically"), so nobody runs to close a valve that is already shut. And write the texts with the operators, during commissioning, while they are standing at the machine. The design book has more on designing an alarm list that people trust.

From handling faults to finding them

Alarm handling the way plants expect it has a shape you can now recognise in anyone's code: a qualified, delayed condition; a latch that only the operator's acknowledge clears; acknowledge and reset kept apart; a first-out taken from the raw order of events; a deliberate safe state per output, including what the hardware does when the program stops; and no restart without a person's say-so.

Much of this logic only shows its bugs in a particular sequence: an acknowledge before the condition clears, a start held through a reset, two faults whose delays differ. Finding those sequences before the machine does is a skill of its own. Testing and debugging covers testing ST with plant cases, watching and forcing values online, and debugging from a hypothesis instead of from hope.