Capturing and Reading Industrial Traffic Without Stopping the Line

intermediate · 25 min

A switch will not show your laptop other devices' traffic — that is what a switch is for. Read the port counters first, then set up a mirror session or a TAP without knocking a device off the network.

ℹ️ Most "network" faults are solved before Wireshark is ever opened. The switch already counted every corrupt frame, every discard and every late collision, per port, for free. A capture answers "what did the devices say to each other?" — it does not answer "is this cable any good?" Read the counters first; capture only when they come back clean.

A managed switch delivers a unicast frame to exactly one port, so a laptop on a spare port captures only broadcast, multicast and its own traffic — a SPAN mirror session is what puts the real traffic in front of Wireshark, at the cost of one port and switch CPU.

Start at the counters, not at Wireshark

Every managed switch counts bad frames per port. Reading them takes two minutes and no downtime. A capture takes an hour and, done carelessly, disturbs the line.

Cisco IOS: show interfaces counters errors. Siemens SCALANCE: Web Based Management, then Information, Ethernet Statistics, Packet Error. Hirschmann, Moxa and Phoenix Contact use different menu names.

Note the numbers, clear them (clear counters on Cisco, the Reset Counters button on SCALANCE), run the machine five minutes, read again. The rate of increase is the evidence. 40,000 CRC errors accumulated since commissioning in 2019 is not today's fault.

Read both ends of every suspect link. A link has two interfaces and they do not report the same symptoms. This matters most for duplex mismatch, below, where reading one end will actively mislead you.

Telling the causes apart from counters alone

A CRC (cyclic redundancy check, FCS on Cisco) is a checksum on every Ethernet frame. If it fails, the frame was corrupted on that cable.

The single most useful distinction, and the one most often missed: does the malformed frame carry a good CRC or a bad one? Bad CRC means the frame was damaged in transit — physical layer. Good CRC means the transmitter meant to send exactly that, and it was simply the wrong size — configuration. Siemens splits these into separate counters and Cisco largely does not, which is why SCALANCE is often faster to read.

Duplex mismatch: read both ends

A duplex mismatch does not light up one port with every error at once. The symptoms divide:

• The half-duplex end increments late collisions (a collision detected after the first 64 bytes / 512 bit times), plus alignment and FCS errors.
• The full-duplex end increments FCS errors and runts, and reports no collisions at all — a full-duplex interface cannot have one.

So if you see late collisions on a switched point-to-point link, the fix is at the far end as much as this one. Make speed and duplex agree at both ends; on PROFINET and EtherNet/IP device ports, leave both on autonegotiate unless the manual says otherwise. The only other cause worth ruling out is a cable run beyond spec length.

Output drops with clean error counters mean congestion, a speed step-down (1 Gbit uplink into a 100 Mbit device port), or a multicast storm — design, not cable.

Counters clean and the machine still faulting? Now a capture earns its time.

Why plugging into a spare port shows you little

A switch forwards a unicast frame only to the port where the destination MAC was learned. From a spare port your laptop sees broadcasts, multicasts, its own traffic, and unknown-unicast flooding — frames whose destination MAC is not currently in the switch's MAC table, which get flooded to every port in the VLAN. It does not see the established unicast conversation you came to look at.

Two things to check before concluding the port is useless: a spare port is often in a different VLAN, or administratively shut, in which case you see nothing whatsoever. And if you are seeing a lot of flooded unicast, that is itself a finding worth chasing.

SPAN (Switched Port ANalyzer, Cisco's name; other vendors say port mirroring) copies frames from a source port to a destination port. TAP (test access point) is hardware wired into the link that copies traffic to a monitor port.

Pick by symptom. Protocol or logic problem — missing update, wrong register, refused connection? SPAN is fine. Physical errors? SPAN is generally the wrong tool: a store-and-forward switch validates each frame on ingress and discards corrupt ones before the mirror copy is made, so the frames you want mostly never arrive. (Cut-through platforms are the exception and may mirror a frame that later proves bad.) Use a TAP, or trust the counters. Timing accuracy also wants a TAP.

SPAN is not free — what it can cost you

Two constraints, both documented by Cisco, both routinely ignored:

1. An oversubscribed destination drops mirrored frames silently, inventing gaps that never existed. Mirroring a 1 Gbit port into a 100 Mbit destination is the classic way to fabricate the fault you are hunting.
2. A congested destination can affect forwarding on the source port itself. Cisco states that if a destination port is oversubscribed it can become congested and that congestion can affect traffic forwarding on one or more source ports; packets can be dropped at ingress source ports, egress source ports or the destination. Mirroring is therefore low risk, not no risk — and on small embedded and industrial switches, where mirroring may be handled by the CPU rather than the switch fabric, the margin is thinner still.

Practical rule: mirror one direction if that answers the question, keep the destination at least as fast as the source, and take the session down as soon as you have the capture.

Step-by-step

1. Read and clear the port counters at both ends of the suspect link. If they are dirty, fix the physical layer first.
2. Identify the source port by MAC address (show mac address-table on Cisco), not by the panel label.
3. Choose a destination port that is empty and unused, and at least as fast as the source port. Mirroring 1 Gbit into a 100 Mbit port will invent gaps.
4. Safety: a SPAN destination stops being a normal port. On Cisco it drops all incoming packets by default, transmits nothing but mirrored traffic, and does not participate in Layer 2 protocols — STP, VTP, CDP, DTP or PAgP.
5. Configure it. Cisco IOS: monitor session 1 source interface Gi1/0/5 both, then monitor session 1 destination interface Gi1/0/24, then show monitor session 1 to verify. On SCALANCE it lives under Port Mirroring in Web Based Management.
6. Disable Wi-Fi, VPN and any second NIC so the capture holds one interface only.
7. Confirm frames are actually arriving — start Wireshark and watch the packet count climb before you touch the machine. A mis-set mirror produces a silent, empty capture and wastes the one reproduction you were allowed.
8. Capture to a ring buffer: dumpcap -i 4 -b filesize:100000 -b files:20 -w line3.pcapng (filesize is in kB, so this is twenty 100 MB files, about 2 GB).
9. Reproduce the fault and note the wall-clock time the HMI showed when it tripped.
10. Remove the session: no monitor session 1. A forgotten mirror burns bandwidth and strands a port.

Display filters that matter

Display filters go in Wireshark's top bar and hide frames without deleting them.

• pn_dcp — PROFINET naming and addressing. A steady stream of DCP Identify requests means a controller cannot find a device.
• pn_rt — all PROFINET real-time frames. The cyclic I/O data.
• pn_io — PROFINET connection setup and alarms. Where the disconnect reason lives.
• mbtcp — Modbus TCP on port 502: function code, transaction ID, register addresses.
• enip — EtherNet/IP encapsulation: TCP 44818 explicit, UDP 2222 implicit I/O.
• cip — service codes, class/instance/attribute, Forward_Open and Forward_Close.
• opcua — OPC UA binary on TCP 4840.

Do not substitute eth.type == 0x8892 for pn_rt. PROFINET RT is routinely VLAN-tagged with priority 6, and on a tagged frame Wireshark reports eth.type as 0x8100 — the tag indicator — while the inner EtherType appears as vlan.etype. The raw filter therefore hides most of your RT traffic and fabricates gaps in a stream that is actually healthy. Use pn_rt, or eth.type == 0x8892 || vlan.etype == 0x8892 if you need it written out.

Wireshark auto-decodes Modbus by default on TCP 502 (and 802 for Modbus/TLS). Elsewhere it looks like raw TCP; right-click a frame, choose Decode As, pick Modbus/TCP.

Reading cyclic traffic and spotting a missed update

Cyclic traffic is one frame repeating at a fixed interval, so a missed update is a gap in a flat rhythm.

Filter to a single conversation. Set View, Time Display Format, Seconds Since Previous Displayed Packet. Open Statistics, I/O Graph, set the interval to 100 ms, plot MAX of frame.time_delta_displayed. A flat line at your configured interval is healthy; every spike is a candidate.

Gap tolerance is protocol-specific, and you must know the real threshold before calling a spike a fault:

• PROFINET: the device drops the application relationship when cyclic data stops for longer than update time multiplied by the accepted-update-cycles (watchdog) factor. That factor commonly defaults to 3; confirm it in TIA Portal, and remember the reduction ratio feeds the update time.
• EtherNet/IP on Rockwell Logix: an implicit connection times out at a multiplier times the RPI, but the multiplier is not fixed at 4. The controller selects 4, 8, 16, 32 or 64 so that the resulting timeout is at least 100 ms. At a 20 ms RPI that is 4x = 80 ms, escalated to meet the floor; at a 2 ms RPI the multiplier is 64, giving 128 ms. Applying a flat 4x to a fast connection will have you chasing gaps an order of magnitude smaller than the ones that actually fault the device.

Where the protocol carries a sequence number, check it. A jump proves loss; a long delta only proves delay.

Capturing without disturbing production

Mirroring is the low-risk option, but not a zero-risk one — see the SPAN constraints above. Keep the destination at least as fast as the source and remove the session when you are done.

A TAP is different: the link drops while you break it, so the device falls off, the controller loses its I/O and outputs go to their fail state. Do that during a planned stop, never on a running line.

Never do this

Never point a mirror session at a port that already has something plugged into it. This is the mistake that stops lines. The destination stops forwarding normal traffic and leaves spanning tree, so the HMI, drive or I/O rack on that "free-looking" port goes dark instantly — and the outage looks nothing like the config change you just made.

Before you commit, confirm the port is genuinely spare, not merely idle. show interfaces status reading notconnect only proves nothing is linked at this instant — a powered-down drive, or a machine on a duty cycle, reads notconnect and comes back mid-shift. Cross-check the port description, the patch schedule, and whether the MAC table ever learned anything there.

Two more that cost real money. Do not run DCP Set from PRONETA or TIA Portal's Accessible Devices against a running PROFINET line; assigning a name or IP to a live device drops its connection to the controller. Read-only scanning is safe, writing is not. And never drop an unmanaged switch inline "just to see" — it mirrors nothing and breaks the LLDP neighbour topology a PROFINET controller expects for device replacement.