2547oDMVPN – MPLS over DMVPN

BGP/MPLS VPN was initially defined in RFC2547, which was later obsoleted by RFC4367 and updated by some other RFCs.

2547oDMVPN is also known as MPLS VPN over DMVPN. This solution extends MPLS VPN to the branches. The branches in this deployment are connected to the hub through a Layer 3 SP service, a tunneled model using GRE is needed to extend MPLS to the branches. DMVPN provides two key advantages for extending MPLS VPNs to the branches, bulk encryption and a scalable overlay model. DMVPN allows the hub to have a single multipoint GRE tunnel interface to support large numbers of spokes. Cisco recommends that to seamlessly extend the enterprise MPLS Layer3 VPN network to the remote branches(Spokes), the DMVPN hub should be a P device to label switching packets between the hub and the branches.

The MPLS network requires packets to be label switched all the way between source PEs and destination PEs. Running MPLS over DMVPN tunnels makes the remote branch router a full function PE router, which means label imposition is done in the branch router and label switching must be performed all the way between spokes. This requirement make the direct spoke-spoke communication impossible due to the lack of a label allocation mechanism on the dynamically created spoke-spoke tunnels. However, label switching between spoke PE routers can easily be done if spoke-hub-spoke switching path is implemented. With this approach, the hub router act as a MPLS P router, maintains the LDP neighbor relationship, and exchanges label allocation information with all spoke routers. The hub router label switches the packets in-and-out the mGRE interface between the spokes. So we have to use DMVPN Phase 1 design for this deployment, where spoke-to-spoke traffic traverse the hub.

While this solution breaks the benefit of dynamically building spoke-to-spoke tunnels, it provides an acceptable and often more deterministic path for spoke-to-spoke communications and meets the segmentation requirement. It is a very attractive solution when the large enterprise needs to extend their MPLS-segmented data center or large campus to remote branches.

We will use following network topology for this discussion.

Configuration

R1 - PE1 Router

hostname R1
!
vrf definition CUST-ABC
 rd 100:65001
 !
 address-family ipv4
  route-target export 100:65001
  route-target import 100:65001
 exit-address-family
!
vrf definition CUST-XYZ
 rd 100:65002
 !
 address-family ipv4
  route-target export 100:65002
  route-target import 100:65002
 exit-address-family
!
ip cef
!
interface Loopback0
 ip address 10.0.0.1 255.255.255.255
 ip ospf 1 area 0
!
interface FastEthernet0/0
 ip address 10.10.15.1 255.255.255.0
 ip ospf network point-to-point
 ip ospf 1 area 0
 mpls ip
!
interface FastEthernet0/1
 ip address 10.10.14.1 255.255.255.0
 ip ospf network point-to-point
 ip ospf 1 area 0
 mpls ip
!
interface FastEthernet1/0
 vrf forwarding CUST-ABC
 ip address 172.16.18.1 255.255.255.0
!
interface FastEthernet1/1
 vrf forwarding CUST-XYZ
 ip address 172.16.19.1 255.255.255.0
!
router ospf 1
 router-id 10.0.0.1
!
router bgp 100
 bgp router-id 10.0.0.1
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 10.0.0.4 remote-as 100
 neighbor 10.0.0.4 update-source Loopback0
 !
 address-family ipv4
 exit-address-family
 !
 address-family vpnv4
  neighbor 10.0.0.4 activate
  neighbor 10.0.0.4 send-community extended
 exit-address-family
 !
 address-family ipv4 vrf CUST-ABC
  redistribute connected
  neighbor 172.16.18.8 remote-as 65001
  neighbor 172.16.18.8 activate
  neighbor 172.16.18.8 as-override
 exit-address-family
 !
 address-family ipv4 vrf CUST-XYZ
  redistribute connected
  neighbor 172.16.19.9 remote-as 65002
  neighbor 172.16.19.9 activate
  neighbor 172.16.19.9 as-override
 exit-address-family
!

R2 - PE2 Router (DMVPN Spoke)

hostname R2
!
vrf definition CUST-ABC
 rd 100:65001
 !
 address-family ipv4
  route-target export 100:65001
  route-target import 100:65001
 exit-address-family
!
vrf definition CUST-XYZ
 rd 100:65002
 !
 address-family ipv4
  route-target export 100:65002
  route-target import 100:65002
 exit-address-family
!
ip cef
!
crypto isakmp policy 10
 encr aes
 authentication pre-share
 group 2
crypto isakmp key DmVpNpR3$h@r3dK3Y address 0.0.0.0        
!
crypto ipsec transform-set TRANSFORM-SET esp-aes esp-sha-hmac 
 mode transport
!
crypto ipsec profile PROTECT-GRE
 set transform-set TRANSFORM-SET 
!
interface Loopback0
 ip address 10.0.0.2 255.255.255.255
 ip ospf 1 area 0
!
interface Tunnel0
 bandwidth 4096
 ip address 10.10.236.2 255.255.255.0
 ip mtu 1400
 ip nhrp authentication NhRp@UtH
 ip nhrp map multicast 1.1.67.6
 ip nhrp map 10.10.236.6 1.1.67.6
 ip nhrp network-id 100
 ip nhrp nhs 10.10.236.6
 ip tcp adjust-mss 1360
 ip ospf 1 area 0
 mpls ip
 tunnel source FastEthernet0/0
 tunnel destination 1.1.67.6
 tunnel key 123
 tunnel protection ipsec profile PROTECT-GRE
!
interface FastEthernet0/0
 ip address 1.1.27.2 255.255.255.0
!
interface FastEthernet1/0
 vrf forwarding CUST-ABC
 ip address 172.16.210.2 255.255.255.0
!
interface FastEthernet1/1
 vrf forwarding CUST-XYZ
 ip address 172.16.211.2 255.255.255.0
!
router ospf 1
 router-id 10.0.0.2
!
router bgp 100
 bgp router-id 10.0.0.2
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 10.0.0.4 remote-as 100
 neighbor 10.0.0.4 update-source Loopback0
 !
 address-family ipv4
 exit-address-family
 !
 address-family vpnv4
  neighbor 10.0.0.4 activate
  neighbor 10.0.0.4 send-community extended
 exit-address-family
 !
 address-family ipv4 vrf CUST-ABC
  redistribute connected
  neighbor 172.16.210.10 remote-as 65001
  neighbor 172.16.210.10 activate
  neighbor 172.16.210.10 as-override
 exit-address-family
 !
 address-family ipv4 vrf CUST-XYZ
  redistribute connected
  neighbor 172.16.211.11 remote-as 65002
  neighbor 172.16.211.11 activate
  neighbor 172.16.211.11 as-override
 exit-address-family
!
ip route 0.0.0.0 0.0.0.0 1.1.27.7


R3 - PE3 Router (DMVPN Spoke)

hostname R3
!
vrf definition CUST-ABC
 rd 100:65001
 !
 address-family ipv4
  route-target export 100:65001
  route-target import 100:65001
 exit-address-family
!
vrf definition CUST-XYZ
 rd 100:65002
 !
 address-family ipv4
  route-target export 100:65002
  route-target import 100:65002
 exit-address-family
!
ip cef
!
crypto isakmp policy 10
 encr aes
 authentication pre-share
 group 2
crypto isakmp key DmVpNpR3$h@r3dK3Y address 0.0.0.0        
!
crypto ipsec transform-set TRANSFORM-SET esp-aes esp-sha-hmac 
 mode transport
!
crypto ipsec profile PROTECT-GRE
 set transform-set TRANSFORM-SET 
!
interface Loopback0
 ip address 10.0.0.3 255.255.255.255
 ip ospf 1 area 0
!
interface Tunnel0
 bandwidth 4096
 ip address 10.10.236.3 255.255.255.0
 ip mtu 1400
 ip nhrp authentication NhRp@UtH
 ip nhrp map multicast 1.1.67.6
 ip nhrp map 10.10.236.6 1.1.67.6
 ip nhrp network-id 100
 ip nhrp nhs 10.10.236.6
 ip tcp adjust-mss 1360
 ip ospf 1 area 0
 mpls ip
 tunnel source FastEthernet0/0
 tunnel destination 1.1.67.6
 tunnel key 123
 tunnel protection ipsec profile PROTECT-GRE
!
interface FastEthernet0/0
 ip address 1.1.37.3 255.255.255.0
!
interface FastEthernet1/0
 vrf forwarding CUST-ABC
 ip address 172.16.123.3 255.255.255.0
!
interface FastEthernet1/1
 vrf forwarding CUST-XYZ
 ip address 172.16.133.3 255.255.255.0
!
router ospf 1
 router-id 10.0.0.3
!
router bgp 100
 bgp router-id 10.0.0.3
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 10.0.0.4 remote-as 100
 neighbor 10.0.0.4 update-source Loopback0
 !
 address-family ipv4
 exit-address-family
 !
 address-family vpnv4
  neighbor 10.0.0.4 activate
  neighbor 10.0.0.4 send-community extended
 exit-address-family
 !
 address-family ipv4 vrf CUST-ABC
  redistribute connected
  neighbor 172.16.123.12 remote-as 65001
  neighbor 172.16.123.12 activate
  neighbor 172.16.123.12 as-override
 exit-address-family
 !
 address-family ipv4 vrf CUST-XYZ
  redistribute connected
  neighbor 172.16.133.13 remote-as 65002
  neighbor 172.16.133.13 activate
  neighbor 172.16.133.13 as-override
 exit-address-family
!
ip route 0.0.0.0 0.0.0.0 1.1.37.7

R4 - P Router / BGP VPNv4 Route Reflector

hostname R4
!
ip cef
!
interface Loopback0
 ip address 10.0.0.4 255.255.255.255
 ip ospf 1 area 0
!
interface FastEthernet0/0
 ip address 10.10.14.4 255.255.255.0
 ip ospf network point-to-point
 ip ospf 1 area 0
 mpls ip
!
interface FastEthernet0/1
 ip address 10.10.45.4 255.255.255.0
 ip ospf network point-to-point
 ip ospf 1 area 0
 mpls ip
!
router ospf 1
 router-id 10.0.0.4
!
router bgp 100
 bgp router-id 10.0.0.4
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor RR-CLIENTS peer-group
 neighbor RR-CLIENTS remote-as 100
 neighbor RR-CLIENTS update-source Loopback0
 neighbor 10.0.0.1 peer-group RR-CLIENTS
 neighbor 10.0.0.2 peer-group RR-CLIENTS
 neighbor 10.0.0.3 peer-group RR-CLIENTS
 !
 address-family ipv4
 exit-address-family
 !
 address-family vpnv4
  neighbor RR-CLIENTS send-community extended
  neighbor RR-CLIENTS route-reflector-client
  neighbor 10.0.0.1 activate
  neighbor 10.0.0.2 activate
  neighbor 10.0.0.3 activate
 exit-address-family
!

R5 - P Router

hostname R5
!
ip cef
!
interface Loopback0
 ip address 10.0.0.5 255.255.255.255
 ip ospf 1 area 0
!
interface FastEthernet0/0
 ip address 10.10.15.5 255.255.255.0
 ip ospf network point-to-point
 ip ospf 1 area 0
 mpls ip
!
interface FastEthernet0/1
 ip address 10.10.45.5 255.255.255.0
 ip ospf network point-to-point
 ip ospf 1 area 0
 mpls ip
!
interface FastEthernet1/0
 ip address 10.10.56.5 255.255.255.0
 ip ospf network point-to-point
 ip ospf 1 area 0
 mpls ip
!
router ospf 1
 router-id 10.0.0.5
!

R6 - P Router / DMVPN HUB

hostname R6
!
ip cef
!
crypto isakmp policy 10
 encr aes
 authentication pre-share
 group 2
crypto isakmp key DmVpNpR3$h@r3dK3Y address 0.0.0.0        
!
crypto ipsec transform-set TRANSFORM-SET esp-aes esp-sha-hmac 
 mode transport
!
crypto ipsec profile PROTECT-GRE
 set transform-set TRANSFORM-SET 
!
interface Loopback0
 ip address 10.0.0.6 255.255.255.255
 ip ospf 1 area 0
!
interface Tunnel0
 bandwidth 4096
 ip address 10.10.236.6 255.255.255.0
 no ip redirects
 ip mtu 1400
 ip nhrp authentication NhRp@UtH
 ip nhrp map multicast dynamic
 ip nhrp network-id 100
 ip tcp adjust-mss 1360
 ip ospf network point-to-multipoint
 ip ospf hello-interval 10
 ip ospf 1 area 0
 mpls ip
 tunnel source FastEthernet0/1
 tunnel mode gre multipoint
 tunnel key 123
 tunnel protection ipsec profile PROTECT-GRE
!
interface FastEthernet0/0
 ip address 10.10.56.6 255.255.255.0
 ip ospf network point-to-point
 ip ospf 1 area 0
 mpls ip
!
interface FastEthernet0/1
 description To INTERNET
 ip address 1.1.67.6 255.255.255.0
!
router ospf 1
 router-id 10.0.0.6
!
ip route 0.0.0.0 0.0.0.0 1.1.67.7


R7 - Internet Router

hostname R7
!
interface FastEthernet0/0
 ip address 1.1.27.7 255.255.255.0
!
interface FastEthernet0/1
 ip address 1.1.37.7 255.255.255.0
!
interface FastEthernet1/0
 ip address 1.1.67.7 255.255.255.0
!

R8 - Customer ABC Site-1

hostname R8
!
interface FastEthernet0/0
 ip address 172.16.18.8 255.255.255.0
!
interface FastEthernet0/1
 ip address 192.168.10.1 255.255.255.0
!
router bgp 65001
 no synchronization
 bgp router-id 172.16.18.8
 bgp log-neighbor-changes
 network 192.168.10.0
 neighbor 172.16.18.1 remote-as 100
 no auto-summary
!

R9 - Customer XYZ Site-1

hostname R9
!
interface FastEthernet0/0
 ip address 172.16.19.9 255.255.255.0
!
interface FastEthernet0/1
 ip address 192.168.10.1 255.255.255.0
!
router bgp 65002
 no synchronization
 bgp router-id 172.16.19.9
 bgp log-neighbor-changes
 network 192.168.10.0
 neighbor 172.16.19.1 remote-as 100
 no auto-summary
!
R10 - Customer ABC Site-2

hostname R10
!
interface FastEthernet0/0
 ip address 172.16.210.10 255.255.255.0
!
interface FastEthernet0/1
 ip address 192.168.20.1 255.255.255.0
!
router bgp 65001
 no synchronization
 bgp router-id 172.16.210.10
 bgp log-neighbor-changes
 network 192.168.20.0
 neighbor 172.16.210.2 remote-as 100
 no auto-summary
!

R11 - Customer XYZ Site-2

hostname R11
!
interface FastEthernet0/0
 ip address 172.16.211.11 255.255.255.0
!
interface FastEthernet0/1
 ip address 192.168.20.1 255.255.255.0
!
router bgp 65002
 no synchronization
 bgp router-id 172.16.211.11
 bgp log-neighbor-changes
 network 192.168.20.0
 neighbor 172.16.211.2 remote-as 100
 no auto-summary
!

R12 - Customer ABC Site-3

hostname R12
!
interface FastEthernet0/0
 ip address 172.16.123.12 255.255.255.0
!
interface FastEthernet0/1
 ip address 192.168.30.1 255.255.255.0
!
router bgp 65001
 no synchronization
 bgp router-id 172.16.123.12
 bgp log-neighbor-changes
 network 192.168.30.0
 neighbor 172.16.123.3 remote-as 100
 no auto-summary
!

R13 - Customer XYZ Site-3

hostname R13
!
interface FastEthernet0/0
 ip address 172.16.133.13 255.255.255.0
!
interface FastEthernet0/1
 ip address 192.168.30.1 255.255.255.0
!
router bgp 65002
 no synchronization
 bgp router-id 172.16.133.13
 bgp log-neighbor-changes
 network 192.168.30.0
 neighbor 172.16.133.3 remote-as 100
 no auto-summary
!

Verification

# Verify DMVPN

!! -- DMVPN HUB - P Router -- !!
R6#show dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
        N - NATed, L - Local, X - No Socket
        # Ent --> Number of NHRP entries with same NBMA peer
        NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
        UpDn Time --> Up or Down Time for a Tunnel
======================================================================
Interface: Tunnel0, IPv4 NHRP Details 
Type:Hub, NHRP Peers:2, 

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     1 1.1.27.2            10.10.236.2    UP 00:05:14     D
     1 1.1.37.3            10.10.236.3    UP 00:05:10     D

!! -- DMVPN Spoke - PE2 Router -- !!
R2#show dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
        N - NATed, L - Local, X - No Socket
        # Ent --> Number of NHRP entries with same NBMA peer
        NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
        UpDn Time --> Up or Down Time for a Tunnel
======================================================================
Interface: Tunnel0, IPv4 NHRP Details 
Type:Spoke, NHRP Peers:1, 

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     1 1.1.67.6            10.10.236.6    UP 00:09:22     S

!! -- DMVPN Spoke - R3 PE3 Router -- !!
R3#show dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
        N - NATed, L - Local, X - No Socket
        # Ent --> Number of NHRP entries with same NBMA peer
        NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
        UpDn Time --> Up or Down Time for a Tunnel
======================================================================
Interface: Tunnel0, IPv4 NHRP Details 
Type:Spoke, NHRP Peers:1, 

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     1 1.1.67.6            10.10.236.6    UP 00:10:28     S

# Verify BGP VPNv4 Peering 

!! -- BGP VPNv4 Route Reflector -- !!
R4#show bgp vpnv4 unicast all summary 
BGP router identifier 10.0.0.4, local AS number 100
BGP table version is 13, main routing table version 13
12 network entries using 2016 bytes of memory
12 path entries using 768 bytes of memory
4/4 BGP path/bestpath attribute entries using 576 bytes of memory
2 BGP AS-PATH entries using 48 bytes of memory
2 BGP extended community entries using 48 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 3456 total bytes of memory
BGP activity 12/0 prefixes, 12/0 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.0.0.1        4          100      19      27       13    0    0 00:11:49        4
10.0.0.2        4          100      19      27       13    0    0 00:11:36        4
10.0.0.3        4          100      19      27       13    0    0 00:11:33        4

# Verify network reachability b/w Customer ABC Sites

!! -- Customer ABC Site-1 -- !!
R8#show ip route | beg Gate
Gateway of last resort is not set

B    192.168.30.0/24 [20/0] via 172.16.18.1, 00:13:03
C    192.168.10.0/24 is directly connected, FastEthernet0/1
     172.16.0.0/24 is subnetted, 3 subnets
B       172.16.210.0 [20/0] via 172.16.18.1, 00:13:03
C       172.16.18.0 is directly connected, FastEthernet0/0
B       172.16.123.0 [20/0] via 172.16.18.1, 00:13:03
B    192.168.20.0/24 [20/0] via 172.16.18.1, 00:13:03

!! -- Customer ABC Site-2 -- !!
R10#show ip route | beg Gate
Gateway of last resort is not set

B    192.168.30.0/24 [20/0] via 172.16.210.2, 00:13:37
B    192.168.10.0/24 [20/0] via 172.16.210.2, 00:13:37
     172.16.0.0/24 is subnetted, 3 subnets
C       172.16.210.0 is directly connected, FastEthernet0/0
B       172.16.18.0 [20/0] via 172.16.210.2, 00:13:37
B       172.16.123.0 [20/0] via 172.16.210.2, 00:13:37
C    192.168.20.0/24 is directly connected, FastEthernet0/1

!! -- Customer ABC Site-3 -- !!
R12#show ip route | beg Gate
Gateway of last resort is not set

C    192.168.30.0/24 is directly connected, FastEthernet0/1
B    192.168.10.0/24 [20/0] via 172.16.123.3, 00:14:05
     172.16.0.0/24 is subnetted, 3 subnets
B       172.16.210.0 [20/0] via 172.16.123.3, 00:14:05
B       172.16.18.0 [20/0] via 172.16.123.3, 00:14:05
C       172.16.123.0 is directly connected, FastEthernet0/0
B    192.168.20.0/24 [20/0] via 172.16.123.3, 00:14:05

!! -- Traffic b/w Customer ABC Site 1 & Site 2 -- !!

R8#ping 192.168.20.1 source 192.168.10.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.20.1, timeout is 2 seconds:
Packet sent with a source address of 192.168.10.1 
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 220/264/296 ms

R8#traceroute 192.168.20.1 source 192.168.10.1
Type escape sequence to abort.
Tracing the route to 192.168.20.1

  1 172.16.18.1 64 msec 96 msec 64 msec
  2 10.10.15.5 [MPLS: Labels 22/27 Exp 0] 260 msec 292 msec 256 msec
  3 10.10.56.6 [MPLS: Labels 22/27 Exp 0] 248 msec 284 msec 276 msec
  4 172.16.210.2 [AS 100] [MPLS: Label 27 Exp 0] 300 msec 292 msec 260 msec
  5 172.16.210.10 [AS 100] 292 msec *  236 msec


!! -- Traffic b/w Customer ABC Site 1 & Site 3 -- !!

R8#ping 192.168.30.1 source 192.168.10.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.30.1, timeout is 2 seconds:
Packet sent with a source address of 192.168.10.1 
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 224/257/288 ms

R8#traceroute 192.168.30.1 source 192.168.10.1
Type escape sequence to abort.
Tracing the route to 192.168.30.1

  1 172.16.18.1 76 msec 80 msec 80 msec
  2 10.10.15.5 [MPLS: Labels 21/27 Exp 0] 256 msec 292 msec 328 msec
  3 10.10.56.6 [MPLS: Labels 23/27 Exp 0] 320 msec 316 msec 308 msec
  4 172.16.123.3 [AS 100] [MPLS: Label 27 Exp 0] 264 msec 380 msec 320 msec
  5 172.16.123.12 [AS 100] 244 msec *  260 msec


!! -- Traffic b/w Customer ABC Site 2 & Site 3 -- !!
!! -- This traffic is routed via MPLS over DMVPN --!!

R10#ping 192.168.30.1 source 192.168.20.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.30.1, timeout is 2 seconds:
Packet sent with a source address of 192.168.20.1 
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 268/320/356 ms

R10#traceroute 192.168.30.1 source 192.168.20.1
Type escape sequence to abort.
Tracing the route to 192.168.30.1

  1 172.16.210.2 72 msec 84 msec 40 msec
  2 10.10.236.6 [MPLS: Labels 23/27 Exp 0] 404 msec 372 msec 428 msec
  3 172.16.123.3 [AS 100] [MPLS: Label 27 Exp 0] 352 msec 340 msec 428 msec
  4 172.16.123.12 [AS 100] 396 msec *  412 msec

## How it is routed inside MPLS Network? 

- R2/PE router shows it is reachable via next-hop 10.0.0.3
- R2/PE router has received this update from Route Reflector 10.0.0.4
- MPLS VPN label is 27

R2#show ip route vrf CUST-ABC 192.168.30.0

Routing Table: CUST-ABC
Routing entry for 192.168.30.0/24
  Known via "bgp 100", distance 200, metric 0
  Tag 65001, type internal
  Last update from 10.0.0.3 00:22:01 ago
  Routing Descriptor Blocks:
  * 10.0.0.3 (default), from 10.0.0.4, 00:22:01 ago
      Route metric is 0, traffic share count is 1
      AS Hops 1
      Route tag 65001
      MPLS label: 27
      MPLS Flags: MPLS Required

- The next-hop IP 10.0.0.3 is reachable via Tunnel0 and MPLS label 23
- You can see this label in above traceroute output from R10 also.
- This traffic will go over DMVPN Tunnel to next-hop IP 10.10.236.6, which is DMVPN Hub Tunnel IP

R2#show ip route 10.0.0.3
Routing entry for 10.0.0.3/32
  Known via "ospf 1", distance 110, metric 49, type intra area
  Last update from 10.10.236.6 on Tunnel0, 00:33:19 ago
  Routing Descriptor Blocks:
  * 10.10.236.6, from 10.0.0.3, 00:33:19 ago, via Tunnel0
      Route metric is 49, traffic share count is 1

R2#show mpls forwarding-table 10.0.0.3        
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop    
Label      Label      or Tunnel Id     Switched      interface              
17         23         10.0.0.3/32      0             Tu0        point2point 

R2#show ip cef 10.0.0.3 detail 
10.0.0.3/32, epoch 0
  local label info: global/17
  1 RR source [no flags]
  nexthop 10.10.236.6 Tunnel0 label 23

- R6/P router don't have visibility to customer network.
- It just forward packets based on MPLS labels assigned to BGP source IPs of PE Routers
- R6 will "pop(remove)" the MPLS label and forward this packet to next-hop IP 10.10.236.3 (DMVPN Spoke R3 Tunnel IP).

R6#show ip route 192.168.30.0
% Network not in table

R6#show ip route 10.0.0.3
Routing entry for 10.0.0.3/32
  Known via "ospf 1", distance 110, metric 25, type intra area
  Last update from 10.10.236.3 on Tunnel0, 00:37:24 ago
  Routing Descriptor Blocks:
  * 10.10.236.3, from 10.0.0.3, 00:37:24 ago, via Tunnel0
      Route metric is 25, traffic share count is 1

R6#show mpls forwarding-table 10.0.0.3
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop    
Label      Label      or Tunnel Id     Switched      interface              
23         Pop Label  10.0.0.3/32      13312         Tu0        10.10.236.3 

R6#show ip cef 10.0.0.3 detail 
10.0.0.3/32, epoch 0
  local label info: global/23
  nexthop 10.10.236.3 Tunnel0

- R3/PE router shows this remote site network is reachable via BGP from 172.16.123.12 (Cust-ABC Site-3 R12 Router)

R3#show ip route vrf CUST-ABC 192.168.30.0

Routing Table: CUST-ABC
Routing entry for 192.168.30.0/24
  Known via "bgp 100", distance 20, metric 0
  Tag 65001, type external
  Last update from 172.16.123.12 00:42:02 ago
  Routing Descriptor Blocks:
  * 172.16.123.12, from 172.16.123.12, 00:42:02 ago
      Route metric is 0, traffic share count is 1
      AS Hops 1
      Route tag 65001
      MPLS label: none