Intermittent Ethernet Dropouts — Prove It's the Physical Layer

intermediate · 45 min

You'll be able to tell a duplex mismatch from a damaged cable from VFD noise using switch port error counters alone, then fix the one you actually have — without swapping hardware you don't need to swap.

ℹ️ A network that fails the same way every time is a configuration problem. A network that works for six hours and then drops for two seconds is a physical layer problem — the copper, the connectors, the transceivers. That single distinction sorts most plant-floor dropouts before you open a single manual. Treat intermittent as a physical-layer word until the counters prove otherwise.

Words you need in the next five minutes

Physical layer — everything below the software: cable, connectors, and the transceivers in the switch port and the PLC comms card.
Duplex — whether a link can send and receive at the same time. Full duplex can. Half duplex takes turns and listens for collisions.
Auto-negotiation — the handshake where both ends advertise their speed and duplex and settle on the best they share. If one end is forced to a fixed setting, it stops advertising, and the auto end has nothing to agree with — so it falls back to half duplex.
Collision — two devices transmitting at once on a half-duplex link. Normal, if it happens early in the frame.
Late collision — a collision detected after the first 64 bytes (512 bit times) of a frame. IEEE 802.3 treats this as illegal. On a switched point-to-point link it has two credible causes: a duplex mismatch, or a copper run longer than the 100 m segment limit, where the signal takes longer to propagate than the transmitter takes to finish the frame. Duplex mismatch is far the more common of the two.
FCS / CRC error — the Frame Check Sequence is a 32-bit checksum carried in every Ethernet frame. If the receiver's own calculation disagrees, the frame arrived corrupted and gets thrown away.
Runt — a frame shorter than the 64-byte minimum. Usually a transmission that was aborted part-way.
Alignment error — a frame that doesn't end on a whole octet boundary and fails its CRC.
VFD — variable frequency drive, the power electronics that vary motor speed. Its IGBT output switches fast, and those fast voltage edges are what couple into your data cable.

Tell the causes apart before you fix anything

Duplex mismatch

Cable / connector

VFD noise or bad shield

When it fails

Under heavy two-way traffic

When the machine moves, vibrates, or heats up

When one specific drive accelerates or runs

Link LED

Stays solid

Blinks, drops, re-links

Stays solid

Counter signature

Late collisions + runts on one end, FCS/CRC on the other

FCS/CRC on both ends, plus link flaps

FCS/CRC on the receive side only, zero collisions

Proof test

Read duplex at both ends

Wiggle test, then TDR

Lock out that drive and re-run the line

The short version: errors with collisions means duplex or an over-length run. Errors with no collisions means noise or damaged copper. A link that drops and comes back means connector, cable, or a dying port.

Step-by-step

Pin the port. Trace the cable from the failing device to the switch. Read the port number off the label, or count from port 1. Write it down — every command below needs it.

Read the counters before you change anything. On Cisco IOS and Rockwell Stratix switches: show interfaces GigabitEthernet1/0/12 counters errors gives you Align-Err, FCS-Err, Runts, Giants, Single-Col, Multi-Col, Late-Col. show interfaces GigabitEthernet1/0/12 gives input errors, CRC, and the speed/duplex line. On a Rockwell EtherNet/IP module, open the module's web page and read the Ethernet media counters (Alignment Errors, FCS Errors, Late Collisions) — these come from the CIP Ethernet Link object. On Siemens, the S7-1500 CPU web server has a Communication page with Parameters, Statistics, Resources and Connections tabs, and a separate Diagnostics buffer page for the event history.

Rule duplex in or out — before you spend a production window on anything else. This is a sixty-second configuration read and it can end the investigation. Read the negotiated duplex at both ends: the switch port and the device. If they disagree, or if one end is forced and the other is auto, stop; you found it. Cisco switches running CDP also log %CDP-4-DUPLEX_MISMATCH when a neighbour reports a different duplex.

First check the negotiated speed. Auto-negotiation is mandatory for 1000BASE-T, so a true duplex mismatch can only exist on a link actually running at 10 or 100 Mbit/s. If both ends report 1 Gbit/s, duplex is not your fault — go to step 5. And if Late-Col is climbing but duplex is confirmed matched at both ends, measure the cable run: anything past 100 m of copper produces late collisions on its own.

Clear, run, re-read. clear counters GigabitEthernet1/0/12, run the machine hard for ten minutes, read again. A non-zero counter that is frozen is old history. A counter that climbs is your live fault. This step is what separates a diagnosis from a guess. Never declare anything from a counter you have not seen move.

Rule out the drive. If the errors are receive-side FCS/CRC with zero collisions, clear the counters, lock out the suspect VFD, and run the line. Zero errors with the drive off and errors with it on is proof, not opinion. Coordinate the stop with production, and follow your site's lockout/tagout procedure — this is qualified-person work.

Test the cable — with the machine safely stopped.

Wiggle test. Do this only with the machine in a safe, stopped state, or on a section of the run that is clear of all moving parts. Never put your hands on a cable entering a drag chain, or anywhere on a machine that is running. With the counters cleared, flex the cable by hand at the connectors, at every tie-wrap, and at the drag chain entry, then run the line again and re-read. Errors that correlate with a flexed section are a broken conductor or a bad crimp.

TDR. A time domain reflectometer sends a pulse and times the echo. On Cisco IOS switches: test cable-diagnostics tdr interface GigabitEthernet1/0/12, then show cable-diagnostics tdr interface GigabitEthernet1/0/12. It reports each pair (A, B, C, D) as Terminated, Open, Shorted, or ImpedanceMis, with a distance.

Run TDR only on a stopped line. Cisco warns that on a port whose link is up at 10 or 100 Mbit/s the test affects link state and traffic, and on older IOS releases it bounces the interface outright. On a running machine that drops the very I/O connection you are chasing, and can fault the station or trip the process. Three more constraints: TDR is available only on 10/100/1000 copper ports — not SFP or fibre — and support varies by switch platform and IOS version; on fixed 10/100 ports you must disable auto-MDIX at both ends first or the reading misleads you; and never start the test from both ends of the cable at once, which returns false results.

For a pair reported Open or Shorted, Cisco documents the distance to the fault as accurate to roughly ±2 m — so it points you at a section, not a millimetre. Distances reported for healthy pairs carry a much wider tolerance and are not a length certification. A certified tester such as a Fluke DSX-class unit is better if you have one — it checks wire map, length, return loss, and crosstalk against the cabling standard's limits.

Look at the route. Is the data cable in the same tray as motor leads? Coiled next to a drive in the panel? Crushed under a tie-wrap? Bent hard right at the plug?

Fix exactly one thing, then re-measure. Clear counters, run a full shift, re-read. Zero means fixed. Anything else means go back to step 4.

Fixing each cause

Duplex mismatch. Set both ends the same. Auto on both ends is the correct default. For 1000BASE-T (gigabit over copper), IEEE 802.3 makes auto-negotiation mandatory, so forcing it isn't a supported option anyway — and by the same token, if the link is genuinely running at gigabit, a duplex mismatch is not what you are looking at. Only hard-set both ends when a legacy device genuinely cannot negotiate — and then label the port so the next person knows.

Cable fault. Replace the run; do not splice it. Use stranded, flex-rated cable anywhere that moves and solid conductor only on fixed runs, and match the RJ45 plug to the conductor type. A solid conductor in a stranded-rated plug makes a contact that passes a bench test and fails under vibration. In drag chains and on robot arms, use cable with a stated bend-cycle rating. If the run is over 100 m of copper, it is out of spec regardless of what the tester says about continuity — re-route it, or put a switch or fibre segment in the middle.

Connector and strain. Every cable gets strain relief close to the connector. The plug must never carry the cable's weight or take side load. On machines, M12 connectors — D-coded for 100 Mbit/s, X-coded for gigabit — survive vibration and washdown far better than RJ45.

VFD coupling. Separate the data cable from motor cables; where they must meet, cross at 90 degrees. Your vendor's wiring guidelines give the actual spacing — Rockwell's Industrial Automation Wiring and Grounding Guidelines (publication 1770-4.1) sorts conductors into categories with routing rules, and PI publishes equivalent guidance for PROFINET. Look up the number for your installation rather than eyeballing it. Use the drive manufacturer's shielded motor cable, and terminate its shield 360 degrees at both the drive gland and the motor gland. A pigtail is an inductor at these frequencies and largely defeats the shield.

Shielding and bonding. PI's Functional Bonding and Shielding of PROFIBUS and PROFINET guideline calls for shields bonded at both ends onto a properly bonded equipotential system, using a 360-degree clamp onto bare metal — not a coated surface — with the cable strain-relieved separately. The shield is never the strain relief.

Never do this

Never force one end to 100/full to "stabilise" a flapping link. This is the expensive mistake. The link goes solid, the LED looks healthy, and you have just created or hidden a duplex mismatch that will corrupt frames under load for the next two years. If you touch duplex, you touch both ends, in the same visit, and you write it on the panel.

Never run a TDR test on a line that is running. It is a diagnostic that takes the link down. Schedule it.

Never put your hands on a cable while the machine is moving. The wiggle test is a stopped-machine test. No intermittent fault is worth a hand in a drag chain.

Two more that cost real money. Never replace the PLC comms module, the switch, or the drive because the controller reported a timeout. A Rockwell 16#0203 connection-timeout fault or a Siemens station-failure event that lands in OB86 tells you the connection died — it says nothing about why. Prove the physical layer first. And never lift a shield ground or cut a bonding conductor to make noise go away. It sometimes appears to work, it turns the cable into an antenna, and on a grounding conductor it can be a safety violation.

Finally: never declare it fixed on a cleared counter. Cleared counters are always zero. Run a shift, then read.

Key points

A network fault that is intermittent rather than consistent is a physical-layer problem — cable, connector, or transceiver — until switch port counters prove otherwise.
Late collisions on a modern switched point-to-point link are the fingerprint of a duplex mismatch, because IEEE 802.3 defines a late collision as one detected after the first 64 bytes of a frame and there is no legal reason for that on a full-duplex switched link.
FCS/CRC errors with zero collisions point at damaged copper or coupled electrical noise, not at duplex; the presence or absence of collisions is the single fastest way to split the causes.
The only honest measurement is a cleared counter plus a run under real production load — a frozen non-zero counter is history, and a counter that climbs is the live fault.
A duplex change must always be made at both ends of the link in the same visit; forcing one end to make a flapping link look stable creates the exact fault it appears to cure.

Codes and symptoms

Late-Col — Late collisions counter (Cisco IOS / Rockwell Stratix, show interfaces counters errors): A collision detected after the first 64 bytes (512 bit times) of a frame, which IEEE 802.3 defines as illegal. On a switched point-to-point link a climbing Late-Col count is a duplex mismatch until proven otherwise — but confirm the run is within the 100 m copper limit, because an over-length segment produces late collisions on its own. Note that a genuine duplex mismatch only exists at 10 or 100 Mbit/s; auto-negotiation is mandatory for 1000BASE-T.
FCS-Err — Frame Check Sequence errors: Correctly sized frames whose 32-bit CRC checksum failed on arrival, so the frame was discarded. Caused by corrupted data on the wire — damaged cable, a bad connector, coupled noise, or the full-duplex side of a duplex mismatch.
Align-Err — Alignment errors: Frames that do not end on a whole octet boundary and also fail their CRC. Usually a duplex mismatch or a physical problem such as cabling, a failing port, or a failing network interface.
Runts — Runt frames: Received frames shorter than the 64-byte Ethernet minimum, typically a transmission aborted by a collision. Seen on the half-duplex end of a duplex mismatch.
%CDP-4-DUPLEX_MISMATCH — Cisco IOS syslog: duplex mismatch discovered: Cisco Discovery Protocol saw a neighbour reporting a different duplex setting than the local port. It is a strong hint, not a proof — confirm by reading the negotiated duplex at both ends. It also depends on CDP being enabled at both ends, so its absence proves nothing.
%PM-4-ERR_DISABLE — Cisco IOS syslog: link-flap error, port placed in err-disable: The port went up and down more than the flap threshold — five times in ten seconds by default, as shown by 'show errdisable flap-values' — so the switch shut it down. Almost always a layer 1 cause: a bad cable, a bad transceiver, or a duplex mismatch.
Open / Shorted / ImpedanceMis — Cisco TDR pair status (show cable-diagnostics tdr interface): Per-pair result of the built-in time domain reflectometer. Terminated means healthy; Open is a break, Shorted is a short, ImpedanceMis is a mismatch that may sit in the remote device rather than the cable. For an Open or Shorted pair, Cisco documents the distance to the fault as accurate to roughly plus or minus 2 metres; distances reported for healthy pairs carry a far wider tolerance. Run this only on a stopped line — the test affects link state and traffic on a link that is up at 10 or 100 Mbit/s, and bounces the port outright on older IOS. Available only on 10/100/1000 copper ports, not SFP or fibre, and support varies by platform.
16#0203 — Rockwell Logix connection fault: connection timed out: A CIP Class 1 implicit-messaging connection that was already established stopped receiving data within its timeout. Distinct from 16#0204, which means the connection request itself timed out and the connection never opened. It confirms the connection died; it does not identify the cause. Check switch port counters and cabling before replacing hardware.
OB86 — Siemens rack/station failure organization block: The S7 CPU calls OB86 when a distributed PROFINET or PROFIBUS station stops responding to cyclic communication, and logs the event in the diagnostic buffer. What the CPU does when OB86 is absent from the program is platform-specific and configurable — check the manual for your CPU family rather than assuming, and read the diagnostic buffer for the actual event. Like 16#0203, it reports the loss, not the root cause.