3750 Unicast Flooding Issue

Since I ran into this issue and wasn’t really able to find anyone posting on this I thought I should put something together for anyone else that runs into it. I had an issue with a stack of 3750x switches where there was unicast flooding to all of the ports in the same VLAN. While doing research I came across suggestions of asymmetric l2 routes and timeout values for the arp tables and tcam table overruns. My issue turned out to be none of these, the arp timeout values where all increased and that didn’t solve the problem. My network if farily simple with a collapsed core and l2 asymmetric routing wasn’t the issue. The tcam tables were different not being overrun on this switches as it can handle 8K arp entries and I am no where near that.

So what did that leave me with? An issue where the ARP tables of all members of the stack were not getting updated in a timely manner. As seen below with the following command:

remote command all sh mac add count | i Total

Switch : 3 : (Master)

———————

Total Mac Addresses    : 152

Total Mac Addresses    : 585

Total Mac Addresses    : 39

Total Mac Addresses    : 381

Total Mac Addresses    : 384

Total Mac Addresses    : 22

Total Mac Addresses    : 28

Total Mac Addresses    : 178

Total Mac Addresses    : 0

Total Mac Address Space Available: 6402

Switch : 1 :

————

Total Mac Addresses    : 152

Total Mac Addresses    : 585

Total Mac Addresses    : 162

Total Mac Addresses    : 22

Total Mac Addresses    : 39

Total Mac Addresses    : 381

Total Mac Addresses    : 384

Total Mac Addresses    : 28

Total Mac Addresses    : 0

Total Mac Address Space Available: 6418

Switch : 2 :

————

Total Mac Addresses    : 152

Total Mac Addresses    : 585

Total Mac Addresses    : 165

Total Mac Addresses    : 22

Total Mac Addresses    : 39

Total Mac Addresses    : 381

Total Mac Addresses    : 384

Total Mac Addresses    : 28

Total Mac Addresses    : 0

Total Mac Address Space Available: 6415

Switch : 4 :

————

Total Mac Addresses    : 152

Total Mac Addresses    : 585

Total Mac Addresses    : 39

Total Mac Addresses    : 381

Total Mac Addresses    : 384

Total Mac Addresses    : 22

Total Mac Addresses    : 28

Total Mac Addresses    : 140

Total Mac Addresses    : 0

Total Mac Address Space Available: 6440

After many hours of troubleshooting with TAC, they finally came to the conclusion that we were hitting bug:

CSCut64281    Ports on Member of the stack takes long time to learn/age MAC addr

This was only evident in the 15.1x code train, this issue didn’t exist in 15.0 which is why some of my older switches weren’t seeing it. Only the brand new shiny ones I had installed last year. The fix was finally available in the last couple of months in 15.2.(2)E3. I finally finished testing the release on some slightly non prod switches and then decided to roll out to my campus, now I am seeing the following in the upgraded switches:

Switch : 3 : (Master)

———————

Total Mac Addresses    : 142

Total Mac Addresses    : 536

Total Mac Addresses    : 44

Total Mac Addresses    : 70

Total Mac Addresses    : 30

Total Mac Addresses    : 21

Total Mac Addresses    : 27

Total Mac Addresses    : 150

Total Mac Addresses    : 0

Total Mac Address Space Available: 7151

Switch : 1 :

————

Total Mac Addresses    : 142

Total Mac Addresses    : 535

Total Mac Addresses    : 44

Total Mac Addresses    : 70

Total Mac Addresses    : 30

Total Mac Addresses    : 21

Total Mac Addresses    : 27

Total Mac Addresses    : 151

Total Mac Addresses    : 0

Total Mac Address Space Available: 7151

Switch : 2 :

————

Total Mac Addresses    : 142

Total Mac Addresses    : 534

Total Mac Addresses    : 44

Total Mac Addresses    : 70

Total Mac Addresses    : 30

Total Mac Addresses    : 20

Total Mac Addresses    : 27

Total Mac Addresses    : 150

Total Mac Addresses    : 0

Total Mac Address Space Available: 7154

Switch : 4 :

————

Total Mac Addresses    : 142

Total Mac Addresses    : 535

Total Mac Addresses    : 44

Total Mac Addresses    : 70

Total Mac Addresses    : 30

Total Mac Addresses    : 21

Total Mac Addresses    : 27

Total Mac Addresses    : 146

Total Mac Addresses    : 0

Total Mac Address Space Available: 7156

While not perfect it definitely seems to be a lot better than the previous reports. I keep looking for the bug to be posted on Cisco’s site, but it is still private at this point.