Internet Announcements: When a private ASN Shows In Your Path

BGP Shouldn’t Pass Extra AS #’s!

I ran into an interesting issue that might seem pretty pedestrian to someone who deals with Internet peering on a regular basis. This is the first time I’ve had to deal with this using a Cisco IOS-XR platform running multiple VRF’s, however. My Internet providers were not propagating my announcements. I realized a private ASN shows up under certain conditions in IOS-XR.

These routers are running our MPLS network, and I recently added a new VRF with no route bleeding for Internet connectivity. Part of my turning this new VRF up relates to the fact that part of my company was sold off, creating the need to separate network infrastructures. Of course this includes moving the sold part to its own Internet connectivity. I purchased a new AS # from ARIN and had them split our existing resources appropriately.

What Lead to this Mess?

The following changes were being made in order to begin splitting Internet connectivity between the two halves of our company:

  • Split our allocation from a /20 to a /21 (announcement prefix changes, etc)
  • Move our routes from AS “a” to AS “b”
  • Move our announcements to new Internet links

To keep things interesting, one of the ISP’s on the new links is the same ISP we used on one of the two circuits we were migrating from. While this didn’t cause any real issues, it caused some false assumptions about what was going on.

Swinging the Hammers…

A few weeks ago, we stood up new Internet circuits terminated on these MPLS routers, configured in our new Internet VRF. In the initial configuration, we selected an unused IP space, and advertised that block. I was concerned because I wasn’t seeing our new advertisement in any looking glass servers online, and I made a bad assumption at this point.

I concluded we must have an advertising loop because of advertising space that technically overlapped from two different AS numbers to the same ISP. I decided to schedule a new change control to have the old announcement stopped. When I did, the route fell out of the tables but never showed up from the new ASN on either ISP. At this point I knew it was something I was doing wrong, so I picked one ISP and decided I’d work with them to see why they were not announcing the network. If I could fix them, I could fix the other, I reasoned.

Would the Real Issue Please Step Forward?

Essentially the key to the issue came from the engineer at the ISP telling me the route was being dropped on their transport network because he was seeing a PRIVATE ASN in the AS Path. While I had configured a local-as and a remote-as on my BGP peer, for some reason I didn’t understand, IOS-XR was inserting an AS hop from my primary BGP process, outside the VRF.

It wasn’t until one of the other network engineers pointed out a different ‘show’ command on the router would actually display this anomaly. Until we found that, I was unable to tell if I was affecting the problem or not.

I was using the command “show bgp vrf internet neighbors x.x.x.x advertised-routes” which produces output something like:

RP/0/RSP0/CPU0:router-name#show bgp vrf internet neighbors x.x.x.x advertised-routes
Wed Jan 11 14:02:48.782 EDT
Network Next Hop From AS Path
Route Distinguisher: 1010:2 (default for vrf internet)
y.y.y.y/24 x.x.x.x Local i

Notice how that shows the AS Path as “i”, which looks pretty good to me. Jeff, another engineer here, pointed to a slightly different cantation of the same command, producing more usable output:

RP/0/RSP0/CPU0:router-name#show bgp vrf internet advertised neighbor x.x.x.x
Thu Jan 12 09:42:30.045 EDT
Route Distinguisher: 1010:2 (default for vrf internet)
y.y.y.y/24 is advertised to x.x.x.x
Path info:
neighbor: Local neighbor router id: a.a.a.a
valid local best import-candidate
Received Path ID 0, Local Path ID 1, version 192578
Attributes after inbound policy was applied:
next hop:
origin: IGP metric: 0
extended community: RT:64601:1
Attributes after outbound policy was applied:
next hop: y.y.y.z
origin: IGP metric: 0
aspath: 777777 64600
extended community:

Do you see what I see? See that “aspath 777777 64600”? Now I can see the private AS in the path!

Consider that my neighbor was configured with a “local-as” flag specifying my AS #. In the example, that would be 777777. Why was it appending the 64600 AS? That one is used as the main BGP process ID, and in the core MPLS VRF, but nowhere in this VRF. That’s either a bug or my own ignorance. I’m not sure which.

Now For a Fix

There was just no affecting the output of that command. Every reasonable effort failed miserably. First I added the “remove-private-as” flag to the neighbor. Nothing. Then I tried prepended my ASN three times to see if I could affect what was in the path. The path changed to “777777 64600 777777 777777 777777”. That isn’t a prepend! That’s a post-pend! And that’s just weird.

I dropped my prepending and started whacking with a machete.

I wound up playing with flags on the local-as directive. The “no-prepend” switch seemed like the thing I needed based on how it is documented, but it made no difference. I also tried “replace-as” which made no difference. And this is how bad it got… I tried both of those things together, so “local-as 777777 no-prepend replace-as”. That appears to have dropped the entire AS Path from the announcement, leaving the local-as as specified, which caused the route to display in route-reflectors.

Conclusion and Final Thoughts

My understanding of this problem is weak on the side of what happens prior to route-policy being applied to the neighbor. I am digging in further at the first possible moment. Our initial TAC case with Cisco yielded no clues from the networking giant. At some point if we upgrade code and this is a bug that gets fixed… I’m concerned our neighbors could be affected here. Also, what if I need to prepend one of our links? I’m not sure I can!

Anyway, I thought it was interesting for the two people who read this blog… and is worth keeping as a note to myself as well.

Leave a Reply