I recently tried standing up a Cisco 3825 router attached to a Cisco 3750E switch which was in turn connected via vPC to a set of Nexus 7010 switches. I spent the better part of two days trying to get the BGP peers/neighbors to establish between the two Cisco Nexus 7010 switches and the Cisco 3825 router. It was really bizarre in that I was able to ping every interface involved so I had Layer 3 connectivity yet only one of the Nexus 7010 switches could establish a BGP neighbor with the 3825 router. The keepalive timer kept expiring on the second Nexus 7010 switch. After a few days I opened a case with Cisco and a week later I was informed that the configuration I was trying to implement was not supported (didn’t work).

Layer 3 and vPC Recommendations

I was provided a copy of the Nexus 7000 virtual Port-Channel Best Practices & Design Guidelines which clearly indicates on page 25 that routers should not be connected to a vPC link but should instead be connected via a Layer 3 switch port. Here are some bullet points;

• Use separate L3 links to hook up routers to a vPC domain is still standing.
• Don’t use L2 port channel to attach routers to a vPC domain unless you can statically route to HSRP address
• If both, routed and bridged traffic is required, use individual L3 links for routed traffic and L2 port-channel for bridged traffic

I was still currious to understand more of the inner-workings.. why didn’t it work or wasn’t it allowed? I only had to flip through the next few slides although I can really say that I completely understand just yet.

1. Packet arrives at R
2. R does lookup in routing table and sees 2 equal paths going north (to 7k1 & 7k2)
3. Assume it chooses 7k1 (ECMP decision)
4. R now has rewrite information to which router it needs to go (router MAC 7k1 or 7k2)
5. L2 lookup happens and outgoing interface is port-channel 1
6. Hashing determines which port-channel member is chosen (say to 7k2)
7. Packet is sent to 7k2
8. 7k2 sees that it needs to send it over the peer-link to 7k1 based on MAC address
9. 7k1 performs lookup and sees that it needs to send to S
10. 7k1 performs check if the frame came over peer link & is going out on a vPC.
11. Frame will only be forwarded if outgoing interface is NOT a vPC or if outgoing vPC doesn’t have active interface on other vPC peer (in our example 7k2)

I’m not embarrassed to say that I followed everything up until step 11. Why exactly is it that frames will only be forwarded if the outgoing interface is NOT a vPC or if the outgoing vPC doesn’t have an active interface on another vPC peer? Isthere anyone that can shed any additional light on this topic?

I’ve never experienced such a restriction in all my years of working with the Avaya (formerly Nortel) Ethernet Routing Switch 8600 and their Split Multilink Trunking (SMLT) technology. I actually have a Cisco 3825 router connected via a SMLT attached Ethernet Routing Switch 5520 (Layer 2) with the Cisco 3825 and the Avaya 8600s all running BGP.

Cheers!