Network Loop & Broadcast Storm — Line Down

intermediate · 45 min

You will be able to tell a broadcast storm apart from a dead device or a multicast problem in under two minutes, find the physical loop from the switch instead of by pulling cables, and configure RSTP, ring redundancy, and storm control so it cannot happen again.

ℹ️ An IP packet carries a TTL (time-to-live) counter that kills it after a set number of hops. An Ethernet frame has no TTL field at all. That single missing field is the whole problem: a broadcast frame that finds a loop is never aged out by anything. It circulates and multiplies until a human breaks the loop physically. No switch reboot, no PLC restart, and no software fix will clear it.

What a loop actually does

A loop is two live paths between the same two switches. Someone patches a spare cable "to be safe", or plugs both ends of a drop into the same panel switch.

When a switch receives a broadcast frame — one addressed to every device — it copies it out of every port except the one it arrived on. That is called flooding. In a loop the frame comes back and gets flooded again. Now there are two copies. Then four. Because the Ethernet header has no TTL, nothing ages them out.

Two things then fail together:

1. Bandwidth disappears. Cyclic I/O frames queue behind the flood.
2. The MAC address table thrashes. A switch learns which device sits on which port from the source address of arriving frames. In a loop the same source address arrives on two ports repeatedly, so the switch rewrites that entry thousands of times a second. Ordinary traffic between two healthy devices now leaves by the wrong port.

Effect 2 is why the whole line drops, not just the branch holding the loop.

Diagnosis: tell the four causes apart first

"Everything offline at once" has four common causes. They look identical on the HMI. They do not look identical on the switch.

• Broadcast storm (loop). Link LEDs across many ports blink hard and in near-unison, including ports that should be nearly idle. Devices on unrelated branches drop together. Switch management may become slow or unreachable as the CPU saturates — but see the caveat below.
• Single device or cable failure. One node offline, its neighbours healthy. One port's LED dark or flickering. Error counters rise on that port only. Switch management responds normally.
• Multicast flooding (IGMP snooping wrong). Heavy traffic everywhere, but steady rather than growing. Common on EtherNet/IP, where implicit I/O is often multicast.
• Power or ground fault. Devices drop in a physical cluster — one panel, one feed, one tray.

Caveat on the management interface. Losing access to the switch's web page is supporting evidence of a storm. Retaining access is not evidence against one. Modern managed switches use control-plane policing to protect the CPU, and storm-control thresholds are often tuned deliberately so that an operator can still reach the switch and mitigate the storm by hand. A storm confined to one VLAN will also leave a separate management VLAN completely responsive. Confirm with port counters and the log, not with whether the web page loads.

What unmanaged switches can and cannot do

Managed switches agree on a loop-free topology by exchanging BPDUs (Bridge Protocol Data Units) — small control frames that decide which redundant port to block. An unmanaged switch does not generate BPDUs, does not act on them, and never blocks a port of its own accord.

What it does with other switches' BPDUs is the part that decides whether spanning tree can still save you, and it varies by product:

• Most unmanaged switches flood BPDUs transparently, because they do not implement the 802.1D rule that reserved multicast addresses (01:80:C2:00:00:00) are not forwarded. In that case the managed switches on either side can see each other, and spanning tree can converge and block the redundant port normally. Cisco STP will also block a port that receives back a BPDU it originated itself — which covers the case of a loop patched between two ports of the same unmanaged switch.
• Some unmanaged switches drop them. Then the managed switches are blind to each other through that path, and the loop is genuinely invisible to spanning tree.
• Per-VLAN BPDUs may not survive. Cisco PVST+ BPDUs on non-native VLANs can fail to traverse an unmanaged switch even when standard BPDUs pass.
• Detection is a race against the storm. Even where BPDUs do pass, the switch must process one within the hello interval. A storm that saturates the CPU first can prevent the port ever being blocked.

So do not assume spanning tree will catch it, and do not assume it cannot. Verify on the switch. The practical conclusion is unchanged: an unmanaged switch in a production panel removes your guarantees, so treat any loop involving one as something you must find physically.

RSTP must be planned, not just switched on

RSTP (Rapid Spanning Tree Protocol, IEEE 802.1w) blocks redundant ports so one path stays active. Enabling it without design decisions is where sites get hurt.

• Set the root bridge deliberately. Lowest bridge priority wins; the default is 32768 across essentially every vendor (range 0–61440 in steps of 4096), and the tiebreaker is the lowest MAC address. Leave defaults and the root becomes whichever switch happens to have the lowest MAC — often the cheapest box in a far corner, with all traffic bending toward it. Give the lowest priority to the switch nearest the PLC and the second-lowest to its backup.
• Declare device ports as edge ports (Cisco calls this PortFast) so they forward immediately and do not raise a topology change every time a drive powers up.
• Add BPDU guard to those edge ports. If a BPDU arrives where an end device should be, someone plugged in a switch, and the port disables itself before it can become a loop.
• Do not use RSTP for a fast I/O ring. RSTP is often sub-second on a small point-to-point topology, but its recovery time is not bounded — it stretches with topology size and has no guaranteed ceiling you can set a PROFINET or EtherNet/IP connection watchdog against. Use a purpose-built ring protocol with a deterministic figure instead: MRP (Media Redundancy Protocol, IEC 62439-2) on PROFINET, or DLR (Device Level Ring) on EtherNet/IP.
• MRP defines maximum recovery times of 500 ms, 200 ms and 30 ms for rings of up to 50 switches, and 10 ms for rings of up to 14 switches. Check your ring size before promising a number.
• MRP designates one media redundancy manager that blocks one of its ring ports. DLR designates one ring supervisor, which sends beacon frames out both ring ports simultaneously in opposite directions around the ring, and blocks normal data traffic on its secondary port while still allowing DLR protocol frames through. Beacons completing the loop are what confirm ring integrity.
• Each ring needs exactly one manager or supervisor. Zero leaves a live loop; two fight each other.
• Never run RSTP and a ring protocol on the same ports.

Storm control buys time, it does not fix loops

Storm control counts broadcast, multicast, or unicast frames arriving on a port and discards above a threshold. On Cisco it is measured over a one-second interval and set as a percentage of port bandwidth, a rate in bits per second, or a rate in packets per second; 100 percent means no limit and 0.0 blocks all traffic of that type. Some platforms can shut the port instead of filtering.

Set it carefully. Industrial protocols use broadcast and multicast legitimately — PROFINET DCP discovery is multicast, EtherNet/IP implicit I/O frequently is too. A threshold tight enough to kill a storm can also kill your commissioning tool and running I/O. Keep the broadcast limit tighter than the multicast limit, apply it to access ports rather than uplinks, and confirm device discovery still works before you leave.

Step-by-step

1. Declare it before you touch anything. A storm drops I/O comms and can stop production mid-cycle. Tell the line lead. Confirm guards are closed, nobody is in the cell, and stored energy is parked. A device that loses comms may hold last state or go to a fail-safe state depending on configuration — assume you do not know which.
2. Confirm it is a storm. Read the switch face plate: many LEDs pulsing hard together, versus one dark port for a single fault.
3. Get management access out of band. If the web interface will not load or responds too slowly to work with, use the console port — that path survives a saturated data plane. Siemens SCALANCE, Cisco IE and Rockwell Stratix all offer both a browser interface (WBM, WebUI, Device Manager) and a console/CLI; the console is the one to reach for when the network itself is the fault.
4. Read the log before the counters. On Cisco IOS, show logging. The loop signature is %SW_MATM-4-MACFLAP_NOTIF — it names the host MAC, the VLAN, and the two ports it is flapping between. Those two ports are the mouths of your loop.
5. Read the port counters. show interfaces counters on Cisco, or the port statistics page in SCALANCE WBM. Loop ports carry input and output rates far above every other port.
6. Use the switch's own loop finder — knowing what it will do. SCALANCE managed switches include a Loop Detection function in WBM that sends test frames and reports which ports see them return. Before enabling it on a live plant, note three things: the loop reaction is configurable and can be set to disable the port; a port it blocks must be re-enabled manually through WBM and will not recover by itself; and it cannot be used on ports configured as ring ports. Enable it deliberately on the suspects, with the reaction setting you actually want.
7. Isolate from the switch, not the field. Administratively shutdown one suspect port and watch traffic collapse to normal. Reversible, logged, and it takes seconds.
8. Remove the physical loop. Trace the cable on that port. Look for a patch lead with both ends in one switch, a spare drop landed in two panels, or a contractor's unmanaged switch. Photograph the panel, remove the cable, label it.
9. Prove it is clear. no shutdown the port. Confirm MAC flap messages stopped and counters are normal. In TIA Portal check the diagnostic buffer for PROFINET station failure entries; in Studio 5000 confirm no module shows fault 16#0203.
10. Close the hole. BPDU guard on every end-device port, loop detection where available, storm control on access ports, and write down the bridge priorities you set.

Never do this

Never walk the line pulling patch leads to find the loop. This is the expensive mistake people actually make. Every cable you pull drops a device that was running fine, turning a network fault into an unplanned production stop — and it destroys the evidence that would have named the port. Disable ports from the switch instead.

Never power-cycle switches to "clear" a storm. The loop is physical wiring; it returns the moment the switch forwards again. You only erase the log that named the flapping ports, and on a ring you may force a redundancy manager re-election.

Never enable RSTP across a running plant mid-shift as an emergency fix. Every switch starts electing a root, ports pass through a discarding state, and I/O that was limping stops completely. If the ring already runs MRP or DLR, you have just put two redundancy protocols on the same ports.

Never set storm control so tight that discovery breaks. Clamping multicast on a PROFINET or EtherNet/IP segment can leave you unable to find devices with a commissioning tool, and can drop live I/O.

Never conclude "not a storm" just because the switch web page still loads. Control-plane policing and a well-tuned storm-control threshold are both designed to keep management reachable during a storm. Use the port counters and the log to decide.

Never leave an unmanaged switch in a production panel because it "worked". It cannot participate in loop prevention, its handling of other switches' BPDUs is not guaranteed, and it removes the assurance that spanning tree will catch the next loop.