Segmenting a Plant Network Without Dropping the I/O

intermediate · 18 min

A flat factory network works until the day it doesn't — then a broadcast storm, a worm, or a mystery dropout takes the whole site down at once. Here is how to tell those three failures apart on a running line, where the Purdue levels actually get cut, why a controller and its I/O must stay together, and how to migrate an existing flat network cell by cell without stopping production more than once.

ℹ️ Segmentation is usually sold as a security project. Treat it as a troubleshooting project instead. The boundary you draw is the boundary you can search when a line stops — on a flat network every fault has the entire plant as its suspect list. And the one constraint that overrides every tidy drawing is not the org chart: it is layer 2. A controller and its I/O share a broadcast domain or they do not talk.

The Purdue stack drawn as real cabinet hardware — enterprise racks, the IDMZ firewall appliance, HMI panels, a DIN-rail S7-1500 and field instruments — with the overriding rule made visual: PLC and I/O inside one emerald broadcast domain, and a coral X through the router someone tried to put between them.

Tell the causes apart before you touch a switch

A broadcast domain is the set of devices a broadcast frame reaches. A flat network is one broadcast domain for the whole plant. Several very different faults all reach the operator as "the line stopped." Each has its own test, and the order matters — the cheapest test that can clear a hypothesis goes first.

On capturing. Set up a mirror port (SPAN) and capture before changing anything — but do not sell this as a free action. Configuring SPAN writes to the running switch configuration, consumes a physical port, and loads the switch CPU. Mirroring a busy port into a slower or already-loaded destination port drops frames at the destination. Confirm the destination port is genuinely spare, and treat SPAN setup as a planned change with a rollback.

Capture with a ring buffer and a filter, or you will lose the capture. A PROFINET segment at a 1–4 ms cycle produces tens of gigabytes a day:

dumpcap -i eth0 -b filesize:100000 -b files:200 -f "ether proto 0x8892 or udp port 2222 or tcp port 44818"

The levels, defined once

Purdue / ISA-95 gives you names, not rules. Level 0 field devices — sensors, valves, motors. Level 1 control — PLCs, safety controllers, drives. Level 2 supervisory — HMIs, line SCADA. Level 3 site operations — MES, plant historian, engineering workstations. Level 3.5 the IDMZ (industrial DMZ), a buffer zone added by practice, not in the original Purdue drawing. Levels 4/5 business IT and enterprise.

IEC 62443-3-2 supplies the vocabulary that survives an audit: you define a System under Consideration (SuC), partition it into zones (groups of assets with common security requirements) and conduits (the permitted communication paths between zones), and establish a target security level (SL-T) for each. Purdue tells you where to draw. 62443 makes you justify each line with a risk assessment. Note the standard also distinguishes SL-T (what you are aiming for) from SL-C (what the equipment can support) and SL-A (what you actually achieved) — an audit will ask for all three.

Why flat fails — three separate problems

Broadcast load. Every ARP, every DCP identify, every discovery beacon reaches every node. A single loop turns that into a storm that saturates links and starves cyclic traffic long before anyone notices a "network" problem.

One infection reaches everything. A laptop plugged in at the packing line has a clear layer-2 path to the reactor PLC. There is nothing to stop lateral movement because there is no boundary to stop it at.

No troubleshooting boundary. With one domain, "which device is flooding" has 800 candidates. With cell/area zones, it has twelve.

The rule that overrides tidy drawings

Keep a controller and its I/O in the same broadcast domain.

For PROFINET RT this is absolute. Cyclic I/O frames are raw Ethernet, EtherType 0x8892 — no IP header, so no router forwards them. DCP, the discovery protocol behind Accessible devices and PRONETA, is layer 2 too and does not cross a router. An RT-over-UDP variant exists (ports 34962/34963), but standard IO devices do not use it. The context manager on UDP 34964 (DCE/RPC) is routable, which is why a device can sometimes be reached for configuration across a subnet while its cyclic data cannot cross one.

For EtherNet/IP the popular claim that implicit traffic is layer 2 is wrong, but the correction that "so it routes fine" is also wrong. Implicit (cyclic I/O) messaging runs over UDP 2222; explicit messaging over TCP 44818. Both are IP. What actually happens:

• Multicast implicit I/O will not cross a router, full stop. Per the ODVA specification, multicast I/O packets are sent with TTL = 1. The first router decrements it to zero and discards the frame. Configuring PIM and IGMP on the router does not help. (Rockwell unicast uses TTL = 64.)
• Within one broadcast domain, multicast still needs every switch in the path to run IGMP snooping with an active querier — snooping without a querier ages out group memberships and silently kills I/O.
• Unicast implicit I/O does route. Unicast has been the default for Logix EtherNet/IP input connections since RSLogix 5000 v18, and v20 extended it to Safety I/O. Multicast persists for produced/consumed tags with multiple consumers, redundant scanners, rack-optimized connections, and older adapters — so check what your specific system is actually doing before assuming either.
• The unicast trade-off is bandwidth and CPU, not latency. Unicast does not add latency or jitter. The producer sends one copy per consumer, so producer CPU load and link bandwidth scale linearly with consumer count. That is the number to check before flipping a connection to unicast, and it is why unicast is not a licence to route I/O across a plant.

Rockwell's CPwE guidance keeps a controller and its I/O in one cell/area zone. Follow it.

Practical rule: the segment boundary goes above the controller, never between the controller and its I/O.

Vendor scope: the 0x8892 behaviour is PROFINET-specific; 2222/44818 are ODVA/CIP-specific. Do not generalise to Modbus TCP (TCP 502) or OPC UA (TCP 4840) — those are plain client/server and route fine.

VLAN or physical separation

A VLAN splits one physical switch into separate broadcast domains using 802.1Q tags. Cheap, reconfigurable, and adequate for cell/area zones. Use physical separation for safety networks, for zones where a shared switch failure is unacceptable, and wherever a compliance regime demands it. A VLAN misconfiguration is one command away from collapsing back into a flat network; a missing cable is not.

Priority-tagging gotcha — verify on your model and release. PROFINET RT frames are commonly priority-tagged: an 802.1Q tag with priority 6 but VLAN ID 0 — priority without VLAN membership. Some switch platforms discard VID 0 frames on an access port whose VLAN is not the port's native VLAN. Ping still works; cyclic I/O does not.

This is not uniform Cisco behaviour, and stating it as such will send you chasing the wrong thing on the wrong hardware:

• Cisco's VLAN 0 Priority Tagging feature is documented on classic-IOS industrial platforms — IE 2000, IE 2000U, IE 4000, IE 5000, CGS 2520 and the Ethernet Switch Module — where it is enabled by default.
• On the IOS XE industrial line (IE3x00 / IE3500) the feature arrived in Cisco IOS XE 17.18.1. Earlier IOS XE releases on those platforms do not have it.
• Other vendors handle VID 0 differently again. Check the release notes for the exact switch in front of you.

Fixes, in order of preference: make the PROFINET VLAN the native VLAN on the port; run a release with VLAN 0 priority-tagging support; or configure the IO controller to send untagged.

Ring redundancy

MRP (Media Redundancy Protocol, IEC 62439-2) is the standard ring protocol for PROFINET. IEC 62439-2 defines worst-case recovery of 500 ms, 200 ms or 30 ms for rings of up to 50 switches, and 10 ms for rings of up to 14 switches. RSTP typically reconverges in one to several seconds — fine for an office, not for a cyclic I/O connection with a 3× watchdog on a 4 ms update rate. Pick one protocol per physical ring and elect a manager.

Where the IT/OT firewall goes

Two boundaries, not one: Level 3 ↔ 3.5 and Level 3.5 ↔ Level 4. The governing rule, stated directly in the Rockwell/Cisco CPwE IDMZ design guides, is that all traffic from either side terminates in the IDMZ; no IACS traffic directly traverses it. Every flow lands on a broker, replica, or proxy inside the zone.

Must pass: historian replication (IDMZ ↔ Level 3), OPC UA terminating on an aggregating server in the IDMZ, a remote-desktop gateway, patch and antivirus mirrors, a file-drop host, and NTP/DNS in one direction. Must never pass end-to-end: SMB, direct RDP, and any controller protocol from Level 4 to Level 2 or below.

Step-by-step

1. Capture first. Mirror port plus a ring-buffered, filtered capture across a shift change. Schedule the SPAN setup as a change — it writes to the switch config, consumes a port, and loads the CPU.
2. Build the flow list from the capture: source, destination, protocol, port, direction. This becomes your conduit list.
3. Draw zones on the real asset list. One cell/area zone = one controller, its I/O, its drives, its local HMI.
4. Fix physical loops and duplicate IPs before anything else. Cheaper to fix flat than segmented.
5. Stand up the IDMZ hosts — historian replica, OPC UA aggregator, RDP gateway — while the flat network still runs.
6. Move one cell. ⚠️ Changing a device's VLAN drops its connection to the controller; outputs go to their configured fault state and the machine stops. Do it in a planned stop, on the least critical cell, with the OEM backup and a known-good switch config to hand.

• On PROFINET, the VLAN membership is the change that matters. Do not hand-edit device IPs: the IO controller assigns the IP by DCP from the station name at connection setup and will overwrite yours. Change the controller's configuration, not the device.
• On EtherNet/IP, re-IPing an adapter also invalidates the module's path in the controller project — update both, and download in the same stop.

1. Enable IGMP snooping and a querier together. ⚠️ Snooping without an active querier ages out group memberships and silently kills multicast I/O. ⚠️ If the VLAN already has a querier, adding a second triggers an election — if the winner later stops querying or is removed, memberships age out and you reproduce the exact failure you were preventing. Decide deliberately which device is the querier, give it an in-subnet source IP, and document it. Same planned stop.
2. Run the firewall in log-only mode for a full production cycle, then enforce.
3. Document each conduit with its business justification and its SL-T. That document is what makes the next change safe.

Never do this

Never put a controller on one VLAN and its remote I/O on another because the drawing looked cleaner. This is the expensive mistake. On PROFINET the RT frames are not routable at all, so the rack never comes up — and because ping works, the team spends two days blaming the device. On EtherNet/IP the failure mode depends on connection type: multicast I/O dies at the first router on TTL and never works; unicast I/O may establish, run for hours, then degrade when the routed path hiccups or the RPI budget is missed. Either way you are chasing a fault across a plant, and it costs weeks.

Never assume "it's IP, so it routes." EtherNet/IP multicast I/O carries TTL = 1 by specification. Multicast routing on the router does not change that.

Never cut over during production. Every VLAN, IP, or subnet-mask change on a controlled device drops its connection.

Never use a NAT box to paper over duplicate machine IPs and skip the zone design. Layer-2 NAT on Stratix switches and the 1783-NATR layer-3 NAT router are legitimate for integrating identical OEM skids, but they are an addressing tool, not a security boundary. NAT does not inspect, authenticate, or log.

Never leave one shared VLAN for "engineering access" that reaches every cell. It quietly rebuilds the flat network you just spent a shutdown removing.

Never build a ring without deciding the protocol. MRP and RSTP in the same physical loop, or a ring with no manager elected, gives you the storm you were trying to prevent.

Never conclude "VLAN boundary" on a PROFINET device before you have read its station name. It is the most common cause of the exact symptom, and the check is free.