It’s both loved and loathed in the network engineering community but BGP came through for us in the past 24 hours.
We utilize BGP to provide dynamic routing between the many Internet Service Providers we are peering with and at the many Data Centers and circuits over which we peer. This past weekend we had an issue with our primary Internet Service Provider (AT&T) but BGP did it’s job and dutifully detected the dead router and re-routed traffic to the remaining Internet Service Providers. The actual outage time was less then 60 seconds. Even though it occurred around 1:30AM EST in the morning we’re hosting websites that need to be accessible in every timezone around the world. While it was 1:30AM on the East coast it was only 10:30PM on the West coast where shoppers were still busy picking through the online goods and placing orders. And while it might have been a little too early for our friends in the UK (6:30AM GMT), we could have shoppers online from either France or Germany (7:30AM GMT +1).
Dec 14 2014 01:27:20.337: %BGP-5-ADJCHANGE: neighbor 12.251.xxx.xxx Down BGP Notification sent Dec 14 2014 01:27:20.337: %BGP-3-NOTIFICATION: sent to neighbor 12.251.xxx.xxx 4/0 (hold time expired) 0 bytes Dec 14 2014 01:27:22.650: %BGP_SESSION-5-ADJCHANGE: neighbor 12.251.xxx.xxx IPv4 Unicast topology base removed from session BGP Notification sent Dec 14 2014 01:33:25.052: %BGP-5-ADJCHANGE: neighbor 12.251.xxx.xxx Up
We also utilize BGP internally in combination with BFD (Bidirectional Forwarding Detection) to help reduce the failover time on the internal network. We’ve actually had BFD accidentally trip a number of times because it can be too sensitive which can create just as many issues having routes flapping back and forth between multiple paths.
As of this writing I have ~ 511,000 IP routes in my BGP routing tables.
Looking at a peering point on the East coast of the United States;
511499 network entries using 132989740 bytes of memory 2550193 path entries using 244818528 bytes of memory 836860/82048 BGP path/bestpath attribute entries using 187456640 bytes of memory 293120 BGP AS-PATH entries using 13504896 bytes of memory 12459 BGP community entries using 1489256 bytes of memory 51 BGP route-map cache entries using 3264 bytes of memory 0 BGP filter-list cache entries using 0 bytes of memory BGP using 580262324 total bytes of memory BGP activity 11331307/10819807 prefixes, 150321731/147771538 paths, scan interval 60 secs
Here’s a look at a peering point on the West coast of the United States;
511029 network entries using 132867540 bytes of memory 1021218 path entries using 98036928 bytes of memory 246998/81716 BGP path/bestpath attribute entries using 55327552 bytes of memory 145392 BGP AS-PATH entries using 6562258 bytes of memory 0 BGP route-map cache entries using 0 bytes of memory 0 BGP filter-list cache entries using 0 bytes of memory BGP using 292794278 total bytes of memory BGP activity 4607568/4096503 prefixes, 24245483/23224265 paths, scan interval 60 secs
The delta in path entries between the two is a result of the number of BGP peers I have on that specific router.
East = 2,550,193
West = 1,021,218
As you can guess I have a number of additional peers on the East coast than I have on the West coast – plans are in the works to resolve that next calendar year.
You can see the dramatic growth in the number of BGP routes being advertised over the Internet from http://bgp.potaroo.net/.
Note: This is a series of posts made under the Network Engineer in Retail 30 Days of Peak, this is post number 21 of 30. All the posts can be viewed from the 30in30 tag.