The procedure described in this article can be used as part of the bootstrap process of Cisco IOS-XE devices. We start with a CSR1000V SD-WAN edge router in CLI mode, registered with vManage.
The topology has a single router connected to the Internet via an Ethernet interface. We configured the IP address on this interface and the default static route statically.
The configuration includes an out-of-band management interface, so we are not limited to console connectivity for troubleshooting and can use SSH over the Ethernet interface. This part of the configuration is optional, and you can skip the templates for out-of-band VPN 512.
Figure 1. SD-WAN templates sample topology
See the router’s basic CLI configuration in the listing below. We will substitute it with generated configuration from the feature and device templates while ensuring that the device is still able to connect to vManage.
We don’t use the service-side (LAN) configuration in this example. Additional templates can enable other settings after the device is switched to template-based management.
Base CLI Configuration
The basic configuration of the SD-WAN edge device includes the following details:
System parameters (lines 3-10)
Transport and tunnel interface configuration (lines 25-33, 41-45)
Transport VPN routing (line 35)
Out-Of-Band Management VPN and Interface (optional, lines 12-23)
config-transaction
!
system
system-ip 3.1.1.1
site-id 1
admin-tech-on-failure
organization-name SD-WAN-TESTLAB
vbond 100.1.1.51
!
hostname CSR01
!
vrf definition Mgmt-intf
rd 1:512
!
address-family ipv4
route-target export 1:512
route-target import 1:512
exit-address-family
exit
!
interface GigabitEthernet1
vrf forwarding Mgmt-intf
ip address 192.168.1.53 255.255.255.0
!
interface GigabitEthernet2
no shutdown
ip address 21.1.1.2 255.255.255.252
!
interface Tunnel0
no shutdown
ip unnumbered GigabitEthernet2
tunnel source GigabitEthernet2
tunnel mode sdwan
!
ip route 0.0.0.0 0.0.0.0 21.1.1.1
!
line vty 0 4
transport input ssh
!
sdwan
interface GigabitEthernet2
tunnel-interface
encapsulation ipsec
color default
exit
exit
!
commit
Feature templates
In this article, we will create all feature templates from scratch.
To add a feature template refer to the screenshot below.
Start by selecting Configuration > Templates in the side menu. Then click on the Feature tab and the “Add Template” button. Select CSR1000v as the device type. I highlighted the templates used in this article – System, VPN, and Ethernet.
Figure 2. Add new feature template
Step 1. System template configuration
This template defines System IP, Site ID, and Hostname details. We want to generate the following configuration with the template:
To add a System template select the “Cisco System” button in the “Add Template” window shown in Figure 2. Set the name and description of the template.
Default options have variables defined for the parameters we want to define. The only mandatory setting is the Console Baud Rate, and figure 3 shows the required details.
Figure 3. System template configuration
Step 2. Transport VPN
This template defines transport VPN, a global routing context (VRF 0) in Cisco IOS-XE. With this template, we aim to generate the static default route from the complete configuration:
ip route 0.0.0.0 0.0.0.0 21.1.1.1
To add it click on the “Cisco VPN” button on the add feature template page, shown in Figure 2. Add name and description of the template.
Figure 4. Transport VPN template configuration
Set VPN value of 0, which is reserved ID for transport VPN.
For our configuration, we need only specify a default route. To be able to re-use template, use a device variable for the next hop (vpn_next_hop_ip_address_0). This variable can be either the IP address of the next-hop or interface name for point-to-point connections.
Don’t forget to press Add button (#10), as the configuration window will not warn you that the route is not added, which will cause the device to lose connectivity to vManage. Not saving sub-configuration sections is a common issue, which can happen across many vManage configuration pages, as the Web user interface doesn’t force a user to save or discard when adding sub-elements.
Step 3. Transport interface
This template generates the configuration shown below:
interface GigabitEthernet2
no shutdown
ip address 21.1.1.2 255.255.255.252
!
interface Tunnel0
no shutdown
ip unnumbered GigabitEthernet2
tunnel source GigabitEthernet2
tunnel mode sdwan
!
sdwan
interface GigabitEthernet2
tunnel-interface
encapsulation ipsec
color default
exit
exit
Use procedure from the first section to get to the Feature template selection (Figure 2) and then select “Cisco VPN Interface Ethernet” as the type.
Specify the template name and description. Change the following values:
This configuration is similar to the previous two steps. The management VPN uses a reserved ID of 512. SD-WAN overlay doesn’t transport it over, which is why it is called out-of-band. You can only access it locally or expand it using a dedicated network.
In this example, we don’t set up any static routes in VPN 512, as we plan to connect to the device locally via port GigabitEthernet1. In your network, you can set up static routes without any risk to the transport or other VPNs, as each VPN is a VRF, which has its isolated routing table.
The interface configuration differs from the transport interface by not having the tunnel option enabled.
Figure 7. Management interface in VPN 512
Step 5. Configure device template
We have created five feature templates, as shown in Figure 8.
Figure 8. Feature templates list
Let’s create a device template using the steps shown in Figure 9. Select device type, then provide a name and description for the template. Change the following templates:
Cisco System
Cisco VPN 0
Cisco VPN Interface Ethernet
Cisco VPN 512
Cisco VPN Interface Ethernet (add it first)
Figure 9. Create a device template
Cisco AAA template defines user authentication for out-of-band management access. The pre-configured AAA template defines default credentials – user/password combination of admin/admin. Use them to SSH to the device via VPN 512.
You should change it in the production network as a security precaution by defining a custom Cisco AAA template. Save the device template.
Step 6. Apply device template
Now it’s time to apply the template to the device. Open the template configuration page select the device template created in the previous step. Click on the three dots column and then on Attach Devices.
Select CSR01, click on the right-pointing arrow, and then the Attach button.
Fill in values for the variables, which came from the templates defined earlier, and press Update.
List of variables and their values:
ge1_if_ipv4_address – 192.168.1.53/24
vpn_next_hop_ip_address_0 – 21.1.1.1
ge2_if_ipv4_address – 21.1.1.2/30
system_host_name – CSR01
system_system_ip – 3.1.1.1
system_site_id – 1
Figure 10. Apply the device template
The final step is to check the configuration changes in CLI, as shown in Figure 11. Check the transport interface and VPN 0 static route.
In this blog post, we want to show how to enable a zone-based firewall on the Cisco SD-WAN platform. The example continues on the topology in the Direct Internet Access article. We introduced an additional site to demonstrate that the configuration applied doesn’t affect inter-site traffic.
The diagram below shows the topology with a PC behind CSR1. It can access the Internet without any restrictions after enabling Direct Internet Access. If we want to restrict what is reachable from the service side, two options are available – access lists attached to the internal interface and zone-based firewall functionality. As on the traditional, non-SD-WAN router, access-lists are used in scenarios when no stateful inspection is required, for example, when you want to drop RFC 1918 traffic coming from your guest WIFI segment. Cisco zone-based firewall adds the ability to identify the application and stateful inspection, which allows return traffic if it’s part of the permitted session.
Figure 1. Sample Topology
Initial setup for DIA blocking all outbound traffic
The first section will create and associate a security policy that will block all traffic from the internal network to the Internet.
To apply the firewall rules, use a localized security policy. Navigate to Configuration > Policy. Click on Add Security Policy. Choose a custom policy from the list below, as this option shows all possible configuration elements.
Figure 2. Create a Security Policy
The first screen in the wizard is firewall configuration. Click on Add Firewall Policy, and then on Create New.
Figure 3. Add New Firewall Policy
Define source and destination zone pairs by clicking on Apply Zone-Pairs button. Select source and destination zones. You can create zones within the wizard by specifying which VPNs will be part of them.
Figure 4. Define Security Zones
Provide Name and Description for the firewall policy. We can add rules as required; however, we want to drop all the traffic first. The default action, which is applied when no rules match the packet, is Drop (you can change this behavior by selecting Pass in the dropdown menu).
Figure 5. Name and Description of Security Policy
Review created firewall policy and press Next.
Figure 6. Firewall Policy Review
We’ve clicked through all remaining pages of the wizard without adding any configuration. On the “Policy summary” page, provide Security Policy Name and its description.
Figure 7. Security Policy Review
After clicking on the Save Policy button, we can see the policy we’ve just created in the list.
Figure 8. List of Security Policies
Now it’s time to apply the policy to the device. The device template applies the policy to the device.
Figure 9. List of Device Templates
Choose ISR1-Security-Policy in the Security Policy dropdown. And press the Update button to push the configuration to the device.
Figure 10. Apply Security Policy to Template
The listing below shows the config lines are sent to the device based on the configuration we’ve made so far (you can check this via configuration difference preview before the configuration push). As we haven’t specified any specific rules, the policy uses only the class-default class with drop action. The ‘inspect’ firewall policy is defined and applied within the zone-pair configuration block.
parameter-map type inspect-global
alert on
log dropped-packets
multi-tenancy
vpn zone security
policy-map type inspect VPN1-to-VPN0
class class-default
drop
zone security VPN0
vpn 0
zone security VPN1
vpn 1
zone-pair security ZP_VPN1_VPN0_VPN1-to-VPN0 source VPN1 destination VPN0
service-policy type inspect VPN1-to-VPN0
The test shows that the ICMP traffic is blocked as soon as the policy is applied.
Figure 11. Traffic to the Internet is now blocked
To demonstrate that our policy didn’t affect traffic within VPN 1, let’s ping the PC behind CSR02 at another site.
Figure 12. Site-to-site traffic is not affected
Adjust policy to allow ICMP traffic
This section will allow ICMP traffic to the Internet by modifying the policy.
Return to the policy list via Configuration > Security. Choose the policy that we’ve created earlier and press Edit.
Figure 13. Edit existing security policy
Click on the firewall section, choose the “VPN1-to-VPN0” firewall rule and press Edit.
Figure 14. Edit existing firewall policy
Add the new rule by clicking on the “Add Rule” button.
Figure 15. Add a rule to the firewall policy
Specify ACL entry name and choose action to apply. Drop is self-descriptive. Inspect creates a stateful record of the session and automatically allows return traffic. Pass action – will not allow return traffic; for this action to work, you will be required to create another zone-pair for reverse direction and explicitly allow return traffic.
Define matching traffic and action. Protocol number 1 matches ICMP. Select one of the pre-defined protocols for TCP and UDP traffic. If unavailable or non-standard port numbers need to be specified, use ‘tcp’ or ‘udp’ as a protocol along with the “Destination Ports” condition.
Figure 16. Define the access-list entry
Review the rule, save it, and its parent firewall policy.
Figure 17. Firewall rule and policy summary
vManage sends the following commands to the device.
The first three commands are object groups that identify the source, destination, and protocol.
Then access list is defined using the object groups. The class map that follows uses the ACL as a “match” condition.
Finally, policy-map now has a custom class-map statement placed above the default. The action for this traffic is ‘inspect,’ so return packets are automatically allowed.
object-group network VPN1-to-VPN0-seq-Allow_ICMP-network-dstn-og_
host 15.15.15.10
object-group network VPN1-to-VPN0-seq-Allow_ICMP-network-src-og_
192.168.11.0 255.255.255.0
object-group service VPN1-to-VPN0-seq-Allow_ICMP-service-og_
icmp
ip access-list extended VPN1-to-VPN0-seq-Allow_ICMP-acl_
permit object-group VPN1-to-VPN0-seq-Allow_ICMP-service-og_ object-group VPN1-to-VPN0-seq-Allow_ICMP-network-src-og_ object-group VPN1-to-VPN0-seq-Allow_ICMP-network-dstn-og_
class-map type inspect match-all VPN1-to-VPN0-seq-1-cm_
match access-group name VPN1-to-VPN0-seq-Allow_ICMP-acl_
policy-map type inspect VPN1-to-VPN0
class type inspect VPN1-to-VPN0-seq-1-cm_
inspect
class class-default
drop
Figure 18. ICMP traffic to the Internet is now allowed
To monitor or troubleshoot firewall sessions, use the Real-Time menu in the device monitoring, as shown in the screenshot below. The following options provide relevant real-time information:
Policy Zone Pair Sessions – displays list of sessions and which zone pair and class map is matched for each session
Policy Zone Pair Statistics – shows statistics of packets and bytes matching per class-map
IP NAT Translation – displays NAT translations, which can be useful in DIA troubleshooting scenarios
Figure 19. Use vManage Web interface to view firewall sessions
To view active sessions using CLI as they are passing, use show policy-firewall sessions command:
CSR01#show policy-firewall sessions platform ?
all detailed information
destination-port Destination Port Number
detail detail on or off
icmp Protocol Type ICMP
imprecise imprecise information
session session information
source-port Source Port
source-vrf Source Vrf ID
standby standby information
tcp Protocol Type TCP
udp Protocol Type UDP
v4-destination-address IPv4 Desination Address
v4-source-address IPv4 Source Address
v6-destination-address IPv6 Desination Address
v6-source-address IPv6 Source Address
| Output modifiers
<cr> <cr>
It is possible to filter the output using one of the keywords above. We will display all sessions with 'all' keyword.
CSR01#show policy-firewall sessions platform all
--show platform hardware qfp active feature firewall datapath scb any any any any any all any --
[s=session i=imprecise channel c=control channel d=data channel A/D=appfw action allow/deny]
Session ID:0x00000001 192.168.11.10 8 15.15.15.10 1 proto 1 (1:1:1:1) (0x3:icmp) [sc]
To display detailed information on the session, which includes ingress and egress interfaces, translated addresses, and other information use detail keyword.
Packet capture provides a way of getting a copy of the packets traversing a router. This can be useful for troubleshooting purposes when you want to see if the packets are being received or sent by the router via the expected interface.
There are 2 ways to perform the packet capture – one is using the vManage user interface, and another one is using CLI directly on the router. In this article, we will explain how to use both of them.
Using vManage
Packet capture is reachable via the Device Troubleshooting page – Monitor > Network > Device name > Troubleshooting. By default, there is no Packet Capture option under the Traffic section, as shown in Figure 1.
Figure 1. Packet Capture in vManage before Data Streaming is enabled
The pop-up alert displays: “Data Stream is disabled. Go to the Settings page to enable Data Stream to use Packet Capture, Speed Test, and Debug Logs”. To run packet captures via vManage we must enable Data Stream.
Navigate to Administration > Settings. Then expand the Data Stream option, as shown in the screenshot below. Click on the “Enabled” radio box, type in the hostname, which is the name or IP address of vManage and VPN that the edge router should use to reach the controller.
Figure 2. Enable Data Stream Configuration in vManage
Note that for virtualized platforms, like the one we use for the lab, VPN 512 (out-of-band) cannot be used. To make this work, we are using the public IP of vManage, which is reachable via transport VPN 0. Our lab topology is shown in the figure below.
Figure 3. Data Streaming Topology
After enabling Data Streaming, the Packet Capture option is now visible in the Troubleshooting section. After clicking on this option, we can define packet capture parameters.
Figure 4. Packet Capture in vManage after Data Streaming is enabled
Packet capture screen requires VPN and Interface filter selection. You can optionally provide other filters, such as source and destination IPs and protocol information. Traffic is captured in both ingress and egress directions. Let’s change the filter to protocol 1 (ICMP) and start capture by pressing the Start button.
Figure 5. Packet Capture Parameters
By default, the capture will run for 5 minutes. You can stop the timer at any time to download packets captured so far. The file in pcap format will be available for download shortly.
Figure 6. Packet Capture Progress
The content of the file can be viewed in Wireshark, as shown below.
Figure 7. Display Captured Packets in Wireshark
CSR01#show monitor capture
Status Information for Capture 71f87e76_847e_4770_8289_56b5242ac115
Target Type:
Interface: GigabitEthernet3, Direction: BOTH
Status : Active
Filter Details:
IPv4
Source IP: any
Destination IP: any
Protocol: 1
Buffer Details:
Buffer Type: LINEAR (default)
Buffer Size (in MB): 5
Limit Details:
Number of Packets to capture: 0 (no limit)
Packet Capture duration: 300
Packet Size to capture: 0 (no limit)
Maximum number of packets to capture per second: 1000
Packet sampling rate: 0 (no sampling)
Using CLI on the router
If for some reason you can’t use vManage, you can use IOS-XE Embedded Packet Capture directly on the device (the previous process uses this feature on the backend). Use SSH to connect to the device either via client installed on your computer or via the tools menu in vManage.
The next configuration commands provide an example of running packet capture.
Embedded packet capture commands begin with monitor capture commands. They are available in exec mode, other operational commands, like “show” and “debug”.
CSR01#monitor capture ?
WORD Name of the Capture
clear Clear all Buffers
start Enable all capture points
stop Disable all capture points
Specify a name for the packet capture instance, in our example it is TEST_CAPTURE. The available command options are shown below.
CSR01#monitor capture TEST_CAPTURE ?
WORD Name of the Capture
access-list access-list to be attached
buffer Buffer options
class-map class name to attached
clear Clear Buffer
control-plane Control Plane
export Export Buffer
interface Interface
limit Limit Packets Captured
match Describe filters inline
start Enable Capture
stop Disable Capture
stop_export Disable Capture and Export Buffer
The next commands configure the same options we used in vManage:
GigabithEthernet3 as interface
ICMP packets only (IP protocol 1)
CSR01#monitor capture TEST_CAPTURE interface GigabitEthernet3 both
CSR01#monitor capture TEST_CAPTURE match ipv4 protocol 1 any any
Below are the available options for inline filters.
CSR01#monitor capture TEST_CAPTURE match ?
any all packets
ipv4 IPv4 packets only
ipv6 IPv6 packets only
mac MAC filter configuration
pktlen-range Packet length range to capture
CSR01#monitor capture TEST_CAPTURE match ipv4 ?
A.B.C.D/nn IPv4 source Prefix <network>/<length>, e.g., 192.168.0.0/16
any Any source prefix
host A single source host
protocol Protocols
CSR01#monitor capture TEST_CAPTURE match ipv4 protocol ?
<0-255> An IP protocol number
tcp Filter by TCP protocol
udp Filter by UDP protocol
CSR01#monitor capture TEST_CAPTURE match ipv4 protocol 1 ?
A.B.C.D/nn IPv4 source Prefix <network>/<length>, e.g., 192.168.0.0/16
any Any source prefix
host A single source host
CSR01#monitor capture TEST_CAPTURE match ipv4 protocol 1 any ?
A.B.C.D/nn IPv4 destination Prefix <network>/<length>, e.g., 192.168.0.0/16
any Any destination prefix
host A single destination host
To validate capture parameters run the command: show monitor capture TEST_CAPTURE. As shown in the listing below, by default, the capture will run till its buffer will reach 10MB.
CSR01#show monitor capture TEST_CAPTURE
Status Information for Capture TEST_CAPTURE
Target Type:
Interface: GigabitEthernet3, Direction: BOTH
Status : Inactive
Filter Details:
IPv4
Source IP: any
Destination IP: any
Protocol: 1
Buffer Details:
Buffer Type: LINEAR (default)
Buffer Size (in MB): 10
Limit Details:
Number of Packets to capture: 0 (no limit)
Packet Capture duration: 0 (no limit)
Packet Size to capture: 0 (no limit)
Maximum number of packets to capture per second: 1000
Packet sampling rate: 0 (no sampling)
Now we can activate the defined capture.
CSR01#monitor capture TEST_CAPTURE start
After running some pings from a test PC connected to the service side via GigabitEthernet3, we can validate that packets are being captured. The brief format is shown below. Detailed and dump options display truncated and full packet content.
Let’s stop packet capture with the following command.
CSR01#monitor capture TEST_CAPTURE stop
To analyze packet capture buffer offline, use export it using the command shown below:
CSR01#monitor capture TEST_CAPTURE export ?
bootflash: Location of the file
flash: Location of the file
ftp: Location of the file
http: Location of the file
https: Location of the file
pram: Location of the file
rcp: Location of the file
scp: Location of the file
sftp: Location of the file
tftp: Location of the file
CSR01#monitor capture TEST_CAPTURE export bootflash:test_capture.pcap
Exported Successfully
SD-WAN deployments use the Internet as the transport to replace WAN networks traditionally designed to leverage centralized Internet access via the data center. Direct Internet Access (DIA) refers to the configuration when Internet-facing traffic breaks out directly from the branch router.
Is Network Address Translation (NAT) required for DIA to operate? Yes, NAT maintains a translation table, that tracks outbound sessions from the service side VPNs (LAN), so the return traffic can be sent back without having to leak service VPN routes into VPN 0.
In the Cisco SD-WAN solution, transport-facing and user-facing interfaces belong to different VPNs or VRFs. VPN 0 contains transport (or underlay) network-facing interfaces, such as Internet and MPLS. Service-side VPNs contain user-facing interfaces.
The figure below shows logical VPN isolation within a router. In the routing table of VPN 0, there will be no entries for subnets where Host A and Host B are located. These subnets can even have the same IP addresses.
Figure 1. Transport and Service-Side VPNs and DIA
For DIA to work we need to allow traffic to flow between these virtual routers (or VPNs). To direct traffic from service-side VPNs we can use either static routes or a centralized data policy. NAT in transport VPN allows return traffic to be sent back.
Direct Internet Access on Cisco SD-WAN platforms is enabled in 2 steps. The first one is the NAT configuration on the transport interface. The second step directs traffic from service-side VPN using either a static route or centralized data policy.
Step 1: Enable NAT on the transport interface
Let’s start with a very basic topology, shown in Figure 2.
Figure 2. Sample DIA Topology
Edge router has a device template assigned, which references a basic set of feature templates required to provide connectivity.
The first step of both static route and policy-based configuration is to enable NAT on an interface in the transport VPN – GigabitEthernet2. This is done by adjusting the interface template.
Figure 3. Enable NAT on transport interface
The following commands are pushed to the device.
ip nat inside source list nat-dia-vpn-hop-access-list interface GigabitEthernet2 overload
ip nat translation tcp-timeout 3600
ip nat translation udp-timeout 60
interface GigabitEthernet2
ip nat outside
We couldn’t find a way to modify the nat-dia-vpn-hop-access-list used in ip nat insidecommand. This ACL is not visible in the running configuration or in the output of show ip access-lists. In IOS-XE this access list identifies traffic to be translated. In SD-WAN, however, to achieve this data policy needs to be configured.
Step 2: Direct traffic from service-side VPN
There are 2 ways to achieve this:
Static route in service VPN template
Centralized data policy
Step 2 (option 1): Static Route Configuration
Let’s configure a static default route under VPN 1 (Service VPN). Note that VPN is selected as a gateway option.
Figure 4. Configure a static default route
The static route command that is pushed to the device looks like this: ip nat route vrf 1 0.0.0.0 0.0.0.0 global
Note that the static route command has nat keyword. If we turn off NAT enabled in the previous step, the route will disappear. This essentially means that you have to do address translation for the configuration to work.
Let’s do the test from our test PC, confirming that the remote server sees the request as it’s coming from the router’s external IP address.
Figure 5. Test NAT configuration from Windows PC
To check the list of translations, we can run the following command on the router:
CSR01#show ip nat translations verbose
Pro Inside global Inside local Outside local Outside global
tcp 21.1.1.2:5062 192.168.11.10:49158 15.15.15.10:80 15.15.15.10:80
create: 11/14/21 23:16:21, use: 11/14/21 23:16:27, timeout: 00:00:56
RuleID : 1
Flags: timing-out
ALG Application Type: NA
WLAN-Flags: unknown
Mac-Address: 0000.0000.0000 Input-IDB:
VRF: 1, entry-id: 0xe9f7f840, use_count:1
In_pkts: 7 In_bytes: 978, Out_pkts: 7 Out_bytes: 982
Output-IDB: GigabitEthernet2
CSR01#show ip nat translation
Pro Inside global Inside local Outside local Outside global
tcp 21.1.1.2:5062 192.168.11.10:49158 15.15.15.10:80 15.15.15.10:80
Step 2 (Option 2): Centralized policy
We have removed the static route created in Step 2 (option 1), as the traffic will be directed by the centralized policy.
To implement DIA we will configure the traffic data section of the centralized policy that will match traffic coming from 192.168.11.0/24 (LAN segment) to 15.15.15.10/32 (the webserver).
Only one centralized policy can be activated globally at a time. The centralized policy contains multiple component policies. In this example, we will define a data policy, which then can be applied to a site list. In the following steps, we will create a new policy.
Create a centralized policy
Navigate to Configuration > Policies and click on Add Policy button.
Figure 6. Add Centralized Policy
The first step of creating a policy is called “Create Groups of Interest”. These are variables that we can later use in the policy. For our example, we will define:
Data Prefix – for source prefix 192.168.11.0/24, and we will use 15.15.15.10/32 directly in our traffic matching configuration without variable definition
Site List – we want apply only to a single site with Site ID of 1; it is recommended not to have the same site in multiple site lists to ensure that only 1 policy of each type is applied to that site
VPN List – service VPN 1
Figure 7. Configure Groups of Interest (variables)
After defining all required variables and pressing the Next button, we are moved to the Topology and VPN Membership section of the wizard (see the top part of the screenshot below). We don’t need to configure anything for our data policy, so we just press Next.
On the Configure Traffic Rules step of the wizard click on the “Traffic Data” section of the policy, click on Add Policy > Create New (refer to the bottom part of the next screenshot).
Figure 8. Configure Traffic Rules
In a new data policy window enter the name and description of the policy. Adjust the default action to Accept to ensure that the packets that don’t match our criteria for DIA will not be dropped. Finally, press + button to add a rule that will be matching DIA traffic and apply NAT to it.
Figure 9. Create Data New Policy
In the pop-up window select Custom policy. The other options are just a subset of the match and set conditions tailored for different scenarios, custom lists all of them.
Custom rule is added on top of the default action. If there are several rules, you can re-arrange them on the left panel. Ensure that the Match section is selected, add Source and Destination data prefixes to set the conditions for the rule.
Select the data prefix that we set up earlier in groups of interest as the source. Type-in destination as the actual address without the use of the variable. Both options lead to the same result, however, the use of variables allows you to use descriptive naming of the object plus adjusting the values outside of the policy configuration. Click on the Actions button.
Figure 10. Add a new custom rule to the policy and define match conditions
Select NAT VPN action, as shown in the following screenshot. The fallback option is useful when you want this traffic to follow the routing table when NAT cannot be used, for example, when the interface is down. Press the “Save data policy” button.
Figure 11. Set policy action to NAT VPN
The next screenshot list the data policy that we built in the previous step. Notice that the reference count is 0, as we haven’t yet applied it yet. Press Next.
Figure 12. Traffic data policies
The final step is to apply the policy. Click on the Traffic Data section, and then under data-policy-1 press “New Site List and VPN List”. In the pop-up window select “From Service” direction, site-1 as the site list, and VPN1 as the VPN list. Press the Save Policy button.
Figure 13. Apply data policy
The final step is to activate the policy.
Figure 14. Activate centralized policy
At this stage vManage will push the policy to vSmart as part of its running-config:
vSmart will use OMP to distribute the policy to edge routers. In contrast to vSmart, edge routers will not display the policy in the running configuration. Use show sdwan policy from-vsmart command instead.
Let’s check from the client machine that NAT works:
CSR01#show ip nat translation
Pro Inside global Inside local Outside local Outside global
tcp 21.1.1.2:5064 192.168.11.10:62121 15.15.15.10:80 15.15.15.10:80
Cisco SD-WAN devices can be either in vManage or CLI mode.
In vManage mode, the configuration is performed on vManage and then pushed to the device. Local configuration changes are not allowed. In CLI mode, changes are performed locally on the device. vManage mode is the preferred and recommended option for most SD-WAN implementations. However, you can occasionally switch devices into CLI mode to perform specific tasks.
Administrators can connect to the device using SSH or serial console to use various show and debug commands in both modes.
We will focus on edge devices in this article. However, controllers can also be in one of these modes. Cisco-hosted controllers are initially provisioned in CLI mode and then converted to vManage mode by a network administrator. Some features require controllers to be switched to vManage mode.
vManage mode
Consider a sample network shown in Figure 1. Our task is to prepare an edge router for a remote office. We’ve applied the required feature and device templates with dynamic IP assignment on the Internet-facing interface. After the router is shipped to the remote location and plugged in, it cannot establish connectivity to vManage. We discover that the ISP doesn’t have DHCP enabled, and a static IP configuration is required.
Figure 1. SD-WAN Edge Sample Scenario
Let’s assume that we can access the router via console cable attached to a laptop on-site.
Run the “show sdwan system status” command on the edge device to validate the device mode. In the sample output below, the CSR01 router is in vManage mode (vManaged: true).
CSR01#show sdwan system status
Viptela (tm) vEdge Operating System Software
Copyright (c) 2013-2021 by Viptela, Inc.
Controller Compatibility: 20.3
Version: 17.03.03.0.4762
<output omitted>
Personality: vEdge
Model name: CSR1000V
Services: None
vManaged: true
Commit pending: false
Configuration template: CSR01
<output omitted>
Firstly, we try to set a static IP address on the transport interface using CLI. We use the “config-t” command to enter the configuration mode. The “commit” command activates the configuration.
CSR01#config-t
CSR01(config)# interface GigabitEthernet 2
CSR01(config-if)# ip address 21.1.1.2 255.255.255.252
CSR01(config-if)# commit
The following warnings were generated:
'system is-vmanaged': This device is being managed by the vManage. Any configuration changes to this device will be overwritten by the vManage after the control connection to the vManage comes back up.
Proceed? [yes,no] yes
Commit complete.
CSR01#show run interface GigabitEthernet 2
Building configuration...
<output omitted>
interface GigabitEthernet2
description Transport Interface
ip address 21.1.1.2 255.255.255.252
In our example, we didn’t have connectivity to vManage, so the changes were committed. However, as the warning message above advises vManage will overwrite modifications done on the device (see the example in the following listing) once the control connection is up.
CSR01#show run interface GigabitEthernet 2
Building configuration...
<output omitted>
interface GigabitEthernet2
description Transport Interface
ip address dhcp
Rollback to the original configuration (i.e. setting dynamic IP on the transport interface) causes connectivity to vManage fail, which introduces the connectivity problem we tried to solve in the first place. When we have connectivity to vManage, the CLI commands are just rejected, as shown in the listing below. vManage mode doesn’t permit changes on the edge device.
CSR01#config-t
CSR01(config)# interface GigabitEthernet2
CSR01(config-if)# ip address 21.1.1.2 255.255.255.252
CSR01(config-if)# commit
Aborted: 'system is-vmanaged': This device is being managed by the vManage. Configuration through the CLI is not allowed.
CLI Mode
There are several solutions to the problem described above. The first is to switch the device into CLI mode beforehand before sending it to the remote location. We can then adjust the configuration and set static IP address, and move the device back to vManage mode.
The second solution is using hidden support commands, which is not a supported way of configuring the device unless directed by Cisco TAC. However, it works and can be helpful in lab environments or as a quick fix to get the branch running (with full router reset later).
Let’s see how both options work with the examples below.
Switching device to CLI mode via vManage
An administrator can switch the device to CLI mode (and back) via vManage. Navigate to Configuration > Devices. Click on the Change Mode drop-down menu and select CLI mode.
Figure 2. Switching device to CLI mode via vManage
In the dialog window, select and move to the right panel the router you want to switch into CLI mode and click on Upgrade to CLI Mode.
After the conversion, we can use “show sdwan system status” command to validate that the vManaged property is false.
CSR01#show sdwan system status
Viptela (tm) vEdge Operating System Software
Copyright (c) 2013-2021 by Viptela, Inc.
Controller Compatibility: 20.3
Version: 17.03.03.0.4762
Build: Not applicable
<output omitted>
Personality: vEdge
Model name: CSR1000V
Services: None
vManaged: false
Commit pending: false
Configuration template: None
<output omitted>
After the device comes back online, we can update templates to include static IP address configuration and switch it back to the vManage mode. To do this, we need to attach the device template to the vEdge router. If the device had a template attached before, the variable values would be populated automatically.
Figure 3. Switching device to vManage mode
Switching device to CLI mode using hidden commands
This procedure in this section uses hidden commands and should be used only as a last resort. Cisco TAC may not support it; use it at your own risk.
When the router hasn’t been switched to CLI mode beforehand and cannot establish a control connection to vManage, refer to the following example to switch to CLI mode.
After entering the configuration mode, let’s see available options under the system command:
CSR01#config-t
CSR01(config)# system ?
Possible completions:
admin-tech-on-failure Collect admin-tech before reboot due to daemon failure
allow-same-site-tunnels Allow tunnels to be formed between vEdges in the same site
console-baud-rate Console baud-rate
control-session-pps Control session policer rate, in packets per second
controller-group-list Controller group list
debug
description System description
device-groups List of vManage groups to which the device belongs
disable
domain-id Domain ID
environment
fnf
gps-location GPS latitude and longitude of the device
host-policer-pps Rate at which to police packets bound to the control plane (in pps) per QOS level
icmp-error-pps Rate at which to police ICMP error messages either generated or received (in pps).
idle-timeout Idle CLI timeout, in minutes
ignore
location Location description of the device
max-controllers (DEPRECATED) Set the maximum number of controllers to which the device can connect - Deprecated in 15.4
max-omp-sessions Set the maximum number of OMP sessions the device can have
mode-button
mtu
on-demand Set various configuration for On-demand tunnels
organization-name Organization name
overlay-id Overlay ID
port-hop Enable port hopping for all tlocs
port-offset Port offset (unique value; use only if multiple Viptela devices are behind the same NAT)
site-id Site ID
sp-organization-name Service Provider Organization name
system-ip System IP address
system-tunnel-mtu Control tunnel MTU
tcp-optimization-enabled Carve out a dedicated core to use for TCP optimization - applies after reboot
timer Set various timer timeouts
tls ssl-opt cert management config
track-default-gateway Enable/Disable default gateway tracking
track-interface-tag OMP Tag attached to routes based on interface tracking
track-transport Enable transport tracking
upgrade-confirm Configure software upgrade confirmation timeout
vbond Configure remote vBond or local IPv4 vbond address
<cr>
“unhide viptela_internal” command enables the display of “system” command hidden options. The example below shows these subcommands.
CSR01(config)# unhide viptela_internal
CSR01(config)# system ?
Possible completions:
admin-tech-on-failure Collect admin-tech before reboot due to daemon failure
allow-same-site-tunnels Allow tunnels to be formed between vEdges in the same site
allow-sw-vedge (HIDDEN) Allow non-release software vedges to operate without certificates
console-baud-rate Console baud-rate
control-session-pps Control session policer rate, in packets per second
controller-group-list Controller group list
daemon-reboot (HIDDEN) Reboot device if a non-restartable daemon fails
daemon-restart (HIDDEN) Restart restartable daemons if they fail
debug
description System description
device-groups List of vManage groups to which the device belongs
disable
dnsd-ttl config DNS reply TTL in secs
domain-id Domain ID
dpi-cache-expiry (HIDDEN) Cache expiry time in minutes
dpi-cache-size (HIDDEN) Cache size
dpi-disable-track-tx (HIDDEN) Enable/Disable DPI TRACK TX
dpi-enable (HIDDEN) Enable/Disable DPI
dpi-gc-time (HIDDEN) Garbage collect time in secs
dpi-multicore (HIDDEN) Enable multi-core for dpi
dpi-stat-time (HIDDEN) Stats collection time for dpi
environment
fnf
fp-buffer-check (HIDDEN) Enable fastpath buffer validity check
fp-qos-interval config fp qos interval
fp-qos-weight-percent-factor config fp qos weight percent factor
gps-location GPS latitude and longitude of the device
host-policer-pps Rate at which to police packets bound to the control plane (in pps) per QOS level
icmp-error-pps Rate at which to police ICMP error messages either generated or received (in pps).
idle-timeout Idle CLI timeout, in minutes
ignore
increase-bp-count (HIDDEN) Increase port backpressure threshold for all ports
is-vmanaged Device is managed by the vmanage
last-vmanage-transaction-id Used by vManage to maintain integrity of transactions initiated by it towards device
location Location description of the device
max-controllers (DEPRECATED) Set the maximum number of controllers to which the device can connect - Deprecated in 15.4
max-omp-sessions Set the maximum number of OMP sessions the device can have
mode-button
mtu
on-demand Set various configuration for On-demand tunnels
organization-name Organization name
overlay-id Overlay ID
patch-confirm (HIDDEN) Configure software patch confirmation timeout
port-bp-threshold configure port backpressure threshold
port-hop Enable port hopping for all tlocs
port-offset Port offset (unique value; use only if multiple Viptela devices are behind the same NAT)
pseudo-confirm-commit Only valid for vmanage ..
reboot-on-failure (HIDDEN) Reboot device if any daemon fails
simulated-color Simulated device's color
simulated-devices Additional number of simulated devices
simulated-wan-ip Starting IP address for the simulated interface
site-id Site ID
sp-organization-name Service Provider Organization name
system-ip System IP address
system-tunnel-mtu Control tunnel MTU
tcp-optimization-enabled Carve out a dedicated core to use for TCP optimization - applies after reboot
timer Set various timer timeouts
tls ssl-opt cert management config
track-default-gateway Enable/Disable default gateway tracking
track-interface-tag OMP Tag attached to routes based on interface tracking
track-transport Enable transport tracking
unpin-flows-with-reboot (HIDDEN) Enable with reboot OR Disable flow pinning to FP cores
upgrade-confirm Configure software upgrade confirmation timeout
vbond Configure remote vBond or local IPv4 vbond address
ztp-status ZTP status
<cr>
To switch the device to CLI mode, use is-vmanaged option.
CSR01(config)# interface GigabitEthernet 2
CSR01(config-if)# ip address 21.1.1.2 255.255.255.0
CSR01(config-if)# commit
Aborted: 'system is-vmanaged': This device is being managed by the vManage. Configuration through the CLI is not allowed.
CSR01(config-if)# exit
CSR01(config)# system is-vmanaged false
CSR01(config-system)# commit
Commit complete.
CSR01#show run int GigabitEthernet 2
Building configuration...
interface GigabitEthernet2
description Transport Interface
ip dhcp client default-router distance 1
ip address 21.1.1.2 255.255.255.0
The device will connect to vManage. However, the controller will not overwrite the configuration. At this stage, vManage’s view of the router configuration will not reflect the actual state. We need to remediate this as soon as possible, either by switching the device to CLI mode or applying a configuration template that brings consistency. Let’s update the device template to use static interface configuration to get to the consistent state. In the screenshot below, we see the template adjustments. The bottom part of the figure contains the snippet of the command difference. As we saw in the previous command listing, the device already had a static IP address configured, but vManage is not aware of that change, so from its perspective, the device still has DHCP settings configured.
Figure 4. Apply Device Template to WAN Edge
Changing WAN Edge System IP
In the final section of this blog post, let’s see another example when switching to CLI mode can be helpful. When done via template change, the change of System IP can cause the configuration push to be stuck and not apply correctly.
As a workaround, you can switch the device into CLI mode, change System IP, and move it to template configuration back. Follow the steps below.
Step 1. To switch the device to CLI mode, refer to the procedure described in the first section and Figure 2.
Step 2. Using CLI apply the configuration change. In the example below, we change System IP from 1.1.1.101 to 1.1.1.11.
Step 3. Move device to vManaged mode by applying template and ensuring that the System IP variable matches the new value.
This blog post describes configuring a site-to-site IPsec VPN tunnel from a Cisco SD-WAN IOS-XE-based router to a non-SD-WAN device.
How to enable configure Cisco SD-WAN IPsec Tunnels to a non-SD-WAN device? In Cisco SD-WAN template-based deployment, IPsec tunnels are configured via the Cisco VPN Interface IPsec feature template. This template is then applied to the Transport VPN (0) or one of the Service VPNs.
Cisco SD-WAN IPSec Tunnels Step-by-step
Figure 1. Configuration Map: Elements Diagram
The logical elements required to be configured shown in Figure 1. All pre-configured elements have check mark symbols next to them.
In our example, the edge device has a device template attached with basic configuration applied, such as system and transport interfaces, sufficient for the router to have control connections to the controllers.
The device template uses a service VPN, which is described by the Cisco VPN feature template. This type of template, despite its name, is not related to IPsec VPN settings. Cisco VPN feature template defines VRF settings and is a container for routing and participating interface information. In our example, the user-facing interface is assumed to be configured and associated with the VRF.
While it is possible to configure all child templates from within the device template, in our example, we will pre-configure child feature templates first and then select them in the device template.
Create and Configure Cisco VPN Interface IPsec Feature Template
The first two steps deal with configuration of IPsec feature template.
Select model of devices that this feature template will be applied
Select Cisco VPN Interface IPsec
Figure 3. Create new feature template in vManage
Step 2. Configure feature template
Customize IPSec tunnel parameters. There are 5 sections in IPsec template:
Basic configuration, such as name and IP address of the tunnel interface and its underlying source (local router) and destination (remote router)
Dead-peer detection settings
IKE or Phase 1 parameters
IPSEC or Phase 2 parameters
Advanced Settings
SD-WAN requires an IP-numbered interface (/30) and supports route-based tunnels known as VTI (Virtual Template Interface) in Cisco IOS documentation.
Instead of specifying interesting traffic using ACL known as policy-based tunnels, route-based tunnels use static or dynamic routing over a tunnel interface.
Figure 4. Configure feature template in vManage
As figure 4 shows, there are various options available for both IKE and IPSEC security parameters. These need to match between tunnel endpoints.
Adjust device template to use IPsec Feature Template
Step 3. Add feature template to the device template.
IPsec interface template can now be attached to the service VPNs. Figure 6 shows how to modify the existing device template.
Figure 6. Add IPsec template to service-side VPN.
Routing Configuration
Figure 7. Configuration Map: Static Routes over IPsec interface
Step 4. Set-up routing over the tunnel. This can be static or dynamic-routing protocol-based. In the screenshot below, the static route configuration is shown.
Figure 8. Configure routing over IPsec tunnels
Step 5. Test the tunnel.
As the tunnels are VTI-based and have Layer 3 addresses on both sides, the simplest test is to ping the remote side of the tunnel.
There is limited information available via real-time monitoring using vManage web interface. Native SD-WAN tunnels are also IPSec-based. These tunnels have centralized authentication and key management done by OMP instead of IKE/ISAKMP protocols used in non-SD-WAN tunnels. Real-time device options that contain string IKE in their name will be relevant to us in the context of this article.
Figure 9. Validate IKE tunnel status
Using SSH connection to the router these 2 commands can be used to check operational details of the tunnel:
show crypto isakmp sa / show crypto ikev2 sa
show crypto ipsec sa
We will demonstrate output of these commands in the practical example below.
Cisco SD-WAN IPSec Tunnels Example
Now it’s time for a practical example. We will establish an IPsec tunnel to a Cisco IOS-XE router configured to match VPN gateways settings in public clouds. For example, AWS provides sample configuration files for different platforms (see this URL). We will apply configuration from the Cisco IOS sample, and we can assume that if our router can work with it, it will work with a real AWS gateway. The configuration is slightly adjusted to use IKEv2 by replacing all isakmp commands with IKEv2-variants.
Figure 10 Cisco SD-WAN IPSec Tunnel Lab Diagram
External router configuration
Non SD-WAN router shown on the top in figure 10 has the following configuration:
interface GigabitEthernet2
ip address 5.5.5.10 255.255.255.0
ip route 0.0.0.0 0.0.0.0 5.5.5.1
crypto ikev2 keyring KEYRING-1
peer 21.1.1.2
address 21.1.1.2
pre-shared-key cisco
crypto ikev2 proposal IKE-PROPOSAL-1
encryption aes-cbc-256
integrity sha1
group 16
crypto ikev2 policy IKE-POLICY-1
match address local 5.5.5.10
proposal IKE-PROPOSAL-1
crypto ikev2 profile IKE-PROFILE-1
match address local interface GigabitEthernet2
match identity remote address 21.1.1.2 255.255.255.255
authentication remote pre-share
authentication local pre-share
keyring local KEYRING-1
crypto ipsec transform-set TRANSFORM-SET esp-256-aes esp-sha-hmac
mode tunnel
crypto ipsec profile IPSEC-PROFILE-1
set security-association lifetime kilobytes 102400000
set transform-set TRANSFORM-SET
set ikev2-profile IKE-PROFILE-1
interface Tunnel0
ip address 169.254.23.2 255.255.255.252
ip tcp adjust-mss 1400
tunnel source GigabitEthernet2
tunnel mode ipsec ipv4
tunnel destination 21.1.1.2
tunnel protection ipsec profile IPSEC-PROFILE-1
ip route 192.168.22.0 255.255.255.0 Tunnel0
SD-WAN configuration
We followed the same steps described in the first part of the article to configure vManage. To make it easier to follow, the majority of parameters are hardcoded into the template. In a real deployment, per-device variables can be used to allow for template re-use.
Then the feature template was added to the device template under VPN 1 section (see Figure 6 above) and route to 192.168.15.0/24 was added to VPN 1 feature template (see Figure 8).
Testing and Validation
Let’s assume that we have access only to the SD-WAN router, and testing will be done only from one side of the connection. We will use the router’s command-line interface via SSH from the vManage web console, as it gives access to information not available via the web interface.
The first test that we perform is checking if the remote side of the tunnel is reachable. The “vrf 1” parameter makes sure that the router uses the correct interface.
CSR01#ping vrf 1 169.254.23.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 169.254.23.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
CSR01#
If ping responses are not received, we can run “show crypto ikev2 sa” and “show crypto ipsec sa” commands. The first command displays if the IKEv2 security association is established, which is a prerequisite for IPSEC security associations. The troubleshooting should start here. If IKEv1 is used, the command is “show crypto isakmp sa.”
CSR01#show crypto ikev2 sa
IPv4 Crypto IKEv2 SA
Tunnel-id Local Remote fvrf/ivrf Status
1 21.1.1.2/500 5.5.5.10/500 none/1 READY
Encr: AES-CBC, keysize: 256, PRF: SHA1, Hash: SHA96, DH Grp:16, Auth sign: PSK, Auth verify: PSK
Life/Active Time: 86400/545 sec
IPv6 Crypto IKEv2 SA
We were running ping of the tunnel interface in our example, which is directly connected to both routers. This test might be successful; however, the connectivity between devices behind the tunnel gateways may still not work.
In this case, we can use the “show crypto ipsec sa” command. It displays a set of counters for the number of encrypted and decrypted packets.
If the encrypted packets count is not increasing, that usually suggests a local routing problem when traffic is not being sent out of the tunnel interface.
If the encrypted packets count does increase but decrypted doesn’t, it can mean that the remote router has routing misconfiguration.
There are some useful debug commands available, such as “debug crypto ikev2”. It can generate extensive output on a router with multiple tunnels, so be careful not to overload the production router. In the example below, we’ve changed the key on the other side of the tunnel to break the tunnel. Auth exchange failed message is logged, suggesting that we have mismatched keys and “show crypto ikev2” will not display any tunnels.
CSR01#debug crypto ikev2
Payload contents:
VID IDi AUTH SA TSi TSr NOTIFY(INITIAL_CONTACT) NOTIFY(SET_WINDOW_SIZE) NOTIFY(ESP_TFC_NO_SUPPORT) NOTIFY(NON_FIRST_FRAGS)
*Jul 10 00:46:36.630: IKEv2:(SESSION ID = 2,SA ID = 1):Sending Packet [To 5.5.5.10:500/From 21.1.1.2:500/VRF i0:f0]
Initiator SPI : EE7E2D729412F370 - Responder SPI : B37D8CA8BAB8C150 Message id: 1
IKEv2 IKE_AUTH Exchange REQUEST
Payload contents:
ENCR
*Jul 10 00:46:36.633: IKEv2:(SESSION ID = 2,SA ID = 1):Received Packet [From 5.5.5.10:500/To 21.1.1.2:500/VRF i0:f0]
Initiator SPI : EE7E2D729412F370 - Responder SPI : B37D8CA8BAB8C150 Message id: 1
IKEv2 IKE_AUTH Exchange RESPONSE
Payload contents:
NOTIFY(AUTHENTICATION_FAILED)
*Jul 10 00:46:36.633: IKEv2:(SESSION ID = 2,SA ID = 1):Process auth response notify
*Jul 10 00:46:36.633: IKEv2-ERROR:(SESSION ID = 2,SA ID = 1):
*Jul 10 00:46:36.633: IKEv2:(SESSION ID = 2,SA ID = 1):Auth exchange failed
*Jul 10 00:46:36.633: IKEv2-ERROR:(SESSION ID = 2,SA ID = 1):: Auth exchange failed
*Jul 10 00:46:36.633: IKEv2:(SESSION ID = 2,SA ID = 1):Abort exchange
*Jul 10 00:46:36.633: IKEv2:(SESSION ID = 2,SA ID = 1):Deleting SA
CSR01# show crypto ikev2 sa
CSR01#
And finally we can perform end to end test from the test machines using ping and tracert commands.
Cisco IP Service Level Agreements (SLAs) is a proprietary feature available on Cisco routers and switches, which actively generates monitoring traffic, processes replies, and measures network performance.
This feature can be used to perform continuous end-to-end connectivity testing with automated re-routing over failover links. It can also simulate different application behavior, such as voice and video to check if the network provides the expected level of service.
The examples and features described in this article are based on Cisco IOS-XE version 16.9.
Configuration Components
IP SLA configuration starts with defining an SLA operation and then scheduling it to run immediately or at a specific time.
SLA Operation Definition
To create or edit an SLA operation use the “ip sla <operation-id>” global configuration mode command. It places CLI into the IP SLA configuration sub-mode, where you can select one of the IP SLA types and provide corresponding configuration parameters.
One of the simplest types of IP SLA operations is icmp-echo. The router pings a specified IP address and records round-trip times if the remote side is reachable. In the example below, we will set the type of SLA entry as ICMP echo with the destination’s IP address of 10.0.3.1. The sample topology is shown in Figure 1.
Figure 1. IP SLA Test Topology
The configuration mode changes to IP SLA echo where you can adjust different optional parameters, such as request sending frequency and timeout for replies.
X(config)#ip sla 1
X(config-ip-sla)# icmp-echo 10.0.3.1 source-ip 10.0.0.1
X(config-ip-sla-echo)#?
IP SLAs Icmp Echo Configuration Commands:
data-pattern Data Pattern
default Set a command to its defaults
exit Exit operation configuration
frequency Frequency of an operation
history History and Distribution Data
no Negate a command or set its defaults
owner Owner of Entry
request-data-size Request data size
tag User defined tag
threshold Operation threshold in milliseconds
timeout Timeout of an operation
tos Type Of Service
verify-data Verify data
vrf Configure IP SLAs for a VRF
Scheduling IP SLA
Once SLA is defined and optional parameters are specified, start it by running the “ip sla schedule <id>” command. IP SLA can be configured to start at a specific time or as soon as the command is entered, which is shown in the example below.
X(config)#ip sla schedule 1 start-time now life forever
To check SLA operation’s details use the “show ip sla statistics <id> details” command.
X#show ip sla statistics 1 details
IPSLAs Latest Operation Statistics
IPSLA operation id: 1
Latest RTT: 1 milliseconds
Latest operation start time: 10:08:18 UTC Sat Jan 30 2021
Latest operation return code: OK
Over thresholds occurred: FALSE
Number of successes: 2
Number of failures: 0
Operation time to live: Forever
Operational state of entry: Active
Last time this entry was reset: Never
To see configuration details, including default values for various parameters, use the “show ip sla configuration” command.
X#show ip sla configuration
IP SLAs Infrastructure Engine-III
Entry number: 1
Owner:
Tag:
Operation timeout (milliseconds): 5000
Type of operation to perform: icmp-echo
Target address/Source address: 10.0.3.1/10.0.0.1
Type Of Service parameter: 0x0
Request size (ARR data portion): 28
Data pattern: 0xABCDABCD
Verify data: No
Vrf Name:
Schedule:
Operation frequency (seconds): 60 (not considered if randomly scheduled)
Next Scheduled Start Time: Start Time already passed
Group Scheduled : FALSE
Randomly Scheduled : FALSE
Life (seconds): Forever
Entry Ageout (seconds): never
Recurring (Starting Everyday): FALSE
Status of entry (SNMP RowStatus): Active
Threshold (milliseconds): 5000
Distribution Statistics:
Number of statistic hours kept: 2
Number of statistic distribution buckets kept: 1
Statistic distribution interval (milliseconds): 20
Enhanced History:
History Statistics:
Number of history Lives kept: 0
Number of history Buckets kept: 15
History Filter Type: None
Once SLA operation is scheduled, it cannot be modified. Instead, the old operation can be deleted, and then a new one created again using the same ID.
Different IP SLA Types
To view available types of SLA operations, use context-sensitive help as shown in the next example.
Jitter is a variation in the delay between packets. The smaller the jitter the better performance of time-sensitive applications such as voice. It also means that the network delivers packets with a predictable delay and doesn’t experience congestions causing intermittent delays along the paths.
There are 2 types of SLA operations performing Jitter measurements – ICMP and UDP-based.
ICMP Jitter
ICMP Jitter SLA operation is based on ICMP message types (Timestamp Request and Timestamp Reply). The destination can be any device that supports these ICMP messages. Not all devices support it, or it can be blocked by the firewalls. The previously shown ICMP Echo operation is based on more commonly used Echo and Reply message types.
The configuration in the example below demonstrates how to configure ICMP jitter operation and, that once launched, it reports the round-trip-time (RTT) and jitter statistics from the source to the destination, and vice versa.
X(config)# ip sla 2
X(config-ip-sla)# icmp-jitter 10.0.3.1 source-ip 10.0.0.1
X(config)# ip sla schedule 2 start-time now life forever
X#show ip sla statistics 2
IPSLAs Latest Operation Statistics
IPSLA operation id: 2
Type of operation: icmp-jitter
Latest RTT: 1 milliseconds
Latest operation start time: 00:20:39 UTC Sun Jan 31 2021
Latest operation return code: OK
RTT Values:
Number Of RTT: 10 RTT Min/Avg/Max: 1/1/1 milliseconds
Latency one-way time:
Number of Latency one-way Samples: 0
Source to Destination Latency one way Min/Avg/Max: 0/0/0 milliseconds
Destination to Source Latency one way Min/Avg/Max: 0/0/0 milliseconds
Jitter Time:
Number of SD Jitter Samples: 9
Number of DS Jitter Samples: 9
Source to Destination Jitter Min/Avg/Max: 0/1/1 milliseconds
Destination to Source Jitter Min/Avg/Max: 0/1/1 milliseconds
Over Threshold:
Number Of RTT Over Threshold: 0 (0%)
Packet Late Arrival: 0
Out Of Sequence: 0
Source to Destination: 0 Destination to Source 0
In both Directions: 0
Packet Skipped: 0 Packet Unprocessed: 0
Packet Loss: 0
Loss Periods Number: 0
Loss Period Length Min/Max: 0/0
Inter Loss Period Length Min/Max: 0/0
Number of successes: 4
Number of failures: 0
Operation time to live: Forever
To measure jitter, a source router sends a number of packets (10, by default) periodically. The time between these packets is called interval (20ms). The operation is repeated at a specified frequency (60 seconds by default).
The example below shows default configuration values – every 60 seconds the router will send 10 packets with 20 milliseconds interval between each packet.
X#show ip sla configuration 2
IP SLAs Infrastructure Engine-III
Entry number: 2
Owner:
Tag:
Operation timeout (milliseconds): 5000
Type of operation to perform: icmp-jitter
Target address/Source address: 10.0.3.1/10.0.0.1
Packet Interval (milliseconds)/Number of packets: 20/10
Type Of Service parameter: 0x0
Vrf Name:
Schedule:
Operation frequency (seconds): 60 (not considered if randomly scheduled)
<output is truncated>
UDP Jitter and IP SLA responder
UDP is used in voice and video communications. Using UDP traffic for measurements suits better to simulate such applications. In an IP SLA operation configuration codec type can be specified, and this will define packet size and enable various voice-specific metric calculations, such as Mean Opinion Score (MOS).
Figure 2. IP SLA Responder Configuration
To use the UDP Jitter destination device must also be a Cisco device with an IP SLA responder feature enabled on it. The responder’s basic configuration is completed with the single command on router Z:
Z(config)#ip sla responder
Back on the router X, IP SLA configuration is done per the example below.
X(config)#ip sla 3
X(config-ip-sla)#udp-jitter 10.0.3.1 2345 source-ip 10.0.0.1
X(config)#ip sla schedule 3 start-time now life forever
X#show ip sla statistics
IPSLAs Latest Operation Statistics
IPSLA operation id: 3
Type of operation: udp-jitter
Latest RTT: 1 milliseconds
Latest operation start time: 05:53:56 UTC Sun Jan 31 2021
Latest operation return code: OK
RTT Values:
Number Of RTT: 10 RTT Min/Avg/Max: 1/1/1 milliseconds
Latency one-way time:
Number of Latency one-way Samples: 0
Source to Destination Latency one way Min/Avg/Max: 0/0/0 milliseconds
Destination to Source Latency one way Min/Avg/Max: 0/0/0 milliseconds
Jitter Time:
Number of SD Jitter Samples: 9
Number of DS Jitter Samples: 9
Source to Destination Jitter Min/Avg/Max: 0/1/1 milliseconds
Destination to Source Jitter Min/Avg/Max: 0/1/1 milliseconds
Over Threshold:
Number Of RTT Over Threshold: 0 (0%)
Packet Loss Values:
Loss Source to Destination: 0
Source to Destination Loss Periods Number: 0
Source to Destination Loss Period Length Min/Max: 0/0
Source to Destination Inter Loss Period Length Min/Max: 0/0
Loss Destination to Source: 0
Destination to Source Loss Periods Number: 0
Destination to Source Loss Period Length Min/Max: 0/0
Destination to Source Inter Loss Period Length Min/Max: 0/0
Out Of Sequence: 0 Tail Drop: 0
Packet Late Arrival: 0 Packet Skipped: 0
Voice Score Values:
Calculated Planning Impairment Factor (ICPIF): 0
Mean Opinion Score (MOS): 0
Number of successes: 1
Number of failures: 0
Operation time to live: Forever
Path operations
When measuring latency between two hosts it is useful to know how much delay is contributed by each hop along the path. Path-type SLAs work by running traceroute first and then performing echo or jitter operation against each discovered hop.
X(config)#ip sla 4
X(config-ip-sla)#path-echo 10.0.3.1 source-ip 10.0.0.1
X(config)#ip sla schedule 4 start-time now life forever
The configuration is based on the same network topology with 2 paths available to the destination.
Figure 3. IP SLA Path Operations
The “show ip sla statistics aggregated 4 details” command will display round-trip time statistics for each hop. For example, the command output below shows statistics for the first hop in the path going via router B. The remaining hops statistics are omitted in the example below.
X#show ip sla statistics aggregated 4 details
IPSLAs aggregated statistics
Distribution Statistics:
Bucket Range: 0 to < 20 ms
Avg. Latency: 1 ms
Percent of Total Completions for this Range: 100 %
Number of Completions/Sum of Latency: 1/1
Sum of RTT squared low 32 Bits/Sum of RTT squared high 32 Bits: 1/0
Operations completed over threshold: 0
Start Time Index: *09:54:47.234 UTC Tue Feb 2 2021
Path Index: 3
Hop in Path Index: 1
Type of operation: path-echo
Number of successes: 18
Number of failures: 0
Number of over thresholds: 0
Failed Operations due to Disconnect/TimeOut/Busy/No Connection: 0/0/0/0
Failed Operations due to Internal/Sequence/Verify Error: 0/0/0
Failed Operations due to Control enable/Stats retrieve Error: 0/0
Target Address 172.16.100.6
<output is truncated>
Other operations
IP SLA can perform application-specific monitoring, for example, it can run HTTP or FTP operations against a remote server. For generic TCP applications, TCP operation can be used, which validates if the TCP connection can be established and how long it takes to connect to the server.
The other available operation types are DHCP (checks how long it takes to obtain an IP address) and DNS queries.
Reactions and Proactive Monitoring
In the previous sections, we’ve reviewed different types of SLA operations and how to check their operations using CLI. By using SNMP-based monitoring software, administrators can poll SLA data from routers periodically, which is a more scalable approach to monitoring. Cisco provides SNMP MIB called RTTMON that provides access to IP SLAs from monitoring software.
It is also possible for a router to send an SNMP trap or Syslog message if a certain event happens. IP SLA includes a feature called proactive threshold monitoring. To enable it IP SLA reaction must be configured.
The example below uses a previously configured IP SLA operation of icmp-echo type.
ip sla 1
icmp-echo 10.0.3.1 source-ip 10.0.0.1
ip sla schedule 1 life forever start-time now
The reaction configuration below references SLA operation 1 and selects timeout as a monitored element. Then we enable simple SNMP configuration to send a trap when operations times out.
The final section of this document addresses how a router can change its data plane operation in response to IP SLA operational status.
The feature responsible for translating the status of IP SLA operations to the data plane protocols is called Enhanced Object Tracking. First-Hop Redundancy Protocols, such as HSRP and VRRP use the tracking objects to take over or release data forwarding roles. Static and policy-based routing can also use the tracking objects to re-route around degraded paths.
EEM (Embedded Event Manager) scripts can monitor tracking object’s status changes. This makes it possible to run powerful EEM scripting in response to track status change.
A Track object can have 2 states – Up and Down. IP SLA operations can have several return codes. For the purpose of object tracking, we are interested in the 3 return codes discussed below. All other codes that are not covered in this article translate into the Down state of the Track object.
Let’s re-use the IP SLA ICMP Echo operation from the previous example and check the code of the operation. The second command in the listing shows how to check the default threshold (5000ms):
X#show ip sla statistics 1 | incl code
Latest operation return code: OK
X#show ip sla configuration 1 | include Threshold
Threshold (milliseconds): 5000
The operation’s return code is OK, which means that the ping operation is successful and its round-trip time is within the threshold (in our network it is 1ms). OK status always translates to the UP state of the corresponding track object.
If we will turn off the remote interface on router Z, the return code will change to Timeout, as per the listing below. This is the second SLA operation code, which always translates into the Down state of the Track object.
The third code that is relevant for the purpose of track operation is OverThreshold. To demonstrate it, we increased the request data size to 5000 bytes, so the operation takes longer than 1ms. We also decreased the threshold to 1ms, so the operation goes over the threshold.
The OverThreshold code translates differently to the track object state, which we will discuss in the next section.
X(config)#ip sla 2
X(config-ip-sla)#icmp-echo 10.0.3.1 source-ip 10.0.0.1
X(config-ip-sla-echo)# request-data-size 5000
X(config-ip-sla-echo)# threshold 1
X(config)#ip sla schedule 2 start-time now life forever
X#show ip sla statistics 2
IPSLAs Latest Operation Statistics
IPSLA operation id: 2
Latest RTT: 2 milliseconds
Latest operation start time: 09:44:48 UTC Sat Feb 6 2021
Latest operation return code: Over threshold
Number of successes: 2
Number of failures: 0
Operation time to live: Forever
IP SLA Track State vs. Reachability
“track <id> ip sla <sla-id>” command creates a track object that references SLA operation. There are 2 types of track objects for SLA – reachability, and state. Let’s configure both types against the previously configured SLA 2 to see the difference.
X(config)#track 1 ip sla 2 reachability
X(config)#track 2 ip sla 2 state
X#show track
Track 1
IP SLA 2 reachability
Reachability is Up
1 change, last change 00:00:29
Latest operation return code: Over threshold
Latest RTT (millisecs) 2
Track 2
IP SLA 2 state
State is Down
1 change, last change 00:00:17
Latest operation return code: Over threshold
Latest RTT (millisecs) 2
Track 1 is based on reachability and it is UP, even when the underlying IP SLA object is over the threshold. As the name suggests, we only interested in the fact that we can reach the remote side of the connection. Track 2 is state-based and it returns DOWN for the same IP SLA object. Track of the state type checks both reachability and that the IP SLA object is under the specified threshold.
Static Routes
The next listing demonstrates how to use the track objects with static routes:
X(config)#ip route 0.0.0.0 0.0.0.0 172.16.100.2 track 1 200
X(config)#ip route 0.0.0.0 0.0.0.0 172.16.100.6 track 2
X#sh ip route 0.0.0.0
Routing entry for 0.0.0.0/0, supernet
Known via "static", distance 200, metric 0, candidate default path
Routing Descriptor Blocks:
* 172.16.100.2
Route metric is 0, traffic share count is 1
The route via 172.16.100.6 is the preferred, as it has the default administrative distance of 1. The route via 172.16.100.2 is a backup route with an administrative distance of 200. However, the route table shows that the router prefers the route via 172.16.100.2.
For a route to be installed into the routing table the corresponding track must be UP. Because track object #2 is Down, the static route via 172.16.100.6 is withdrawn and the backup route is chosen and installed into the routing table.
EEM Scripts with IP SLA
The final example of this article is an EEM (Embedded Event Manager) script that adjusts the router configuration, generates a Syslog message in response to the Track state change.
The script is triggered when Track object #1 goes down, with the “event track <object-id> state down” command.
The access-list is being adjusted by the script in this example only for demonstration purposes. It can be adjusted to suit real-life requirements. For example, the script can reconfigure VPN tunnel endpoints or enable a backup interface. EEM can perform many other non-CLI actions as well.
To test the script, I’ve turned off the remote interface on router Z causing IP SLA and track to go down. The router generates a test Syslog message and creates an access-list, as shown in the listing below. “terminal monitor” command displays console messages into VTY lines, such as SSH or Telnet.
X#terminal monitor
*Feb 6 10:13:50.285: %TRACK-6-STATE: 1 ip sla 2 reachability Up -> Down
*Feb 6 10:13:50.530: %HA_EM-6-LOG: TRACK-1-DOWN: TEST Message
X#sh run | section ^ip access-list
ip access-list extended INTERNET-IN
permit ip host 1.2.3.4 any
In my previous articles, Cisco SD-WAN and Cisco router platforms were briefly introduced. This article is focused on Cisco routers that currently support the SD-WAN image. The blog post contains consolidated information about Cisco SD-WAN router portfolio, performance, feature licensing and various information that can be useful when selecting a correct platform for deployment.
In this blog post I will summarize available information on Cisco ISR and ASR performance. The following platforms will be covered: ISR G2, ISR 1100, ISR 4000, ASR 1000.
Update: check my new article on SD-WAN routers and platforms here.
ISR G2
Let’s start with ISR G2 performance numbers. ISR G2s are legacy products with Classic IOS, however, they are still around and it is important to know how they perform to properly size newer replacement routers.
Important: These are not real-world numbers. Please read further.
Model
Packets Per Second
Megabits Per Second
Cisco 860
25,000
197
Cisco 880
50,000
198
Cisco 890
100,000
1,400
Cisco 1921
290,000
2,770
Cisco 1941
330,000
2,932
Cisco 2901
330,000
3,114
Cisco 2911
352,000
3,371
Cisco 2921
479,000
3,502
Cisco 2951
579,000
5,136
Cisco 3925
833,000
6,903
Cisco 3925E
1,845,000
6,703
Cisco 3945
982,000
8,025
Cisco 3945E
2,924,000
8,675
Table 1. Cisco ISR G2 RFC 2544 Performance
The second column displays the number of packets per second that the platform can forward under maximum CPU utilization just before starting to drop the packets. For a router’s CPU it takes the same amount of effort to route the 64-byte packet as it would take for 1500-byte one. So it is usually a more reliable metric that removes packet size from the equation.
The third column displays the value in bytes per second (i.e. packet size in bytes x packets per second). As the results can differ more than 20x times based on the size of the packets selected, the specification must provide average packet size that was used during the test.
What is IMIX? The traffic doesn’t consist of packets of the same size, many tests are using packets of different sizes (called Internet Mix (IMIX)). For example, in a simple IMIX sample in every 12 packets transmitted – 7 will be 40 bytes long, 4 – 576, and 1 – 1500. The average packet size in this case will be 340 bytes.
Values provided in Table 1 are based only on IP packet routing without any additional processing, such as QoS, encryption, or NAT, so it is a maximum performance that a platform can deliver. The real-world number will be significantly smaller.
Another important thing to note is how a packet is counted, for example, it can be counted twice – as it enters an ingress interface and exits egress one. Cisco counts this is as a single packet, as it is seen by the forwarding engine. On the other hand, to select a router for a specific WAN interface bandwidth utilization in each direction must be added. For example, in the case of 10Mbps WAN with expected 9Mbps download and 3Mbps upload – calculation should be based on 12Mbps of the load.
For G2 platforms Cisco recommended WAN-link based sizing is as per the table below. Values are much smaller compared to normal IP forwarding. It is also expected that the router will not be running at 99% CPU and will be dropping packets.
Platform
WAN Link
860
4
880
8
890
15
1921
15
1941
25
2901
25
2911
35
2921
50
2951
75
3925
100
3945
150
3925E
250
3945E
350
Table 2. ISR G2 Recommended Sizing Based on WAN Link Speed
ISR 4000
ISR 4000s are running IOS-XE and have introduced performance-based licensing with 3 tiers:
Default
Performance (x2-3 of default throughput level)
Boost (removes shaping completely)
Cisco publishes the following statistics for basic IP routing without services with IMIX traffic (~330 bytes packets).
*- bottleneck was the physical interface speed, not forwarding CPU
As the routers are capable to forward significantly more traffic than default and performance license allows, the numbers in table 3 for these license tiers are close to real-life when services are getting added. It is safe to choose ISR 4000 with “factory default” and “performance” levels and in most cases lower models with a “performance” license if you plan to use multiple services.
Recently added boost license removes shaping completely. Table 3 displays PPS values for ISR 4000, however, most of the routers didn’t have high CPU utilization, as the bottleneck was at the interface clock speed. The calculation is based on an IMIX size of 330 Bytes.
The data provided should be used as an only approximation, as there are many variables that can affect actual device performance which also will not scale linearly with CPU load increase.
ISR 1100
ISR 1100 is a new branch office platform running IOS-XE and similar to Cisco 890 and 1921. Published performance numbers are listed in Table 4. IP forwarding of ISR 1100 is comparable to ISR 4221 with a boost license. Note that ISR 1100 doesn’t support voice features.
Platform
RFC-2544
(Mbps, IMIX)
RFC-2544
(pps, IMIX)
Encryption
(Mbps, AES 256, IMIX)
NAT (Mbps, IMIX)
ACL + NAT + HQoS (Mbps, IMIX)
C1100-4P
1,252
475,000
230
660
330
C1100-8P
1,750
660,000
335
960
510
Table 4. ISR 1100 Performance
ASR 1000
In the cases when you need more than 10Gbps of throughput provided by ISR 4461, ASR 1000 will be the platform of choice. All models in the ASR 1000 range have 2 dedicated hardware components – RP (Route Processor) and ESP (Embedded Service Processor). RP is responsible for control-plane operations and ESP for data forwarding.
Lower-end models, such as ASR1001-X and ASR1002-X have RP and ESP integrated into chassis. The throughput of the system depends on ESP, which runs Cisco-proprietary programmable ASICs called Quantum Flow Processor (QFP).
The performance of 3 integrated models is shown in Table 5. For the models presented in Table 5, an incremental throughput license is required.
Model
ESP Bandwidth (Mbps)
Throughput (pps)
ASR1001-X
20,000
19,000,000
ASR1002-X
30,000
36,000,000
ASR1002-HX
100,000
58,000,000
Table 5. ASR 1000 Performance (integrated ESP models)
Related Links
RFC-2544: Provides information on recommended way to perform testing
Cisco routers are one of the most widely deployed WAN devices. Traditionally they are individually managed and for the larger networks, administrators require additional tools to monitor, perform configuration backup, and to automate tasks.
Many newer Cisco technologies have some form of a central controller and managed data-plane devices. For example, ACI in the data center and SD-Access for the campus. In WAN space, the Cisco portfolio included IWAN (Intelligent WAN) technology and cloud-managed products from Meraki acquisition. In 2017 Cisco has acquired Viptela and its SD-WAN product line. This post contains an overview of this technology and some basic terminology.
Traditional WAN design
To understand the benefits of SD-WAN, let’s consider how most of the Wide Area Networks are designed. Multiple branch offices connect via an MPLS network to one or two data centers, which also provide centralized Internet access. It is secured by high-performance firewalls, intrusion protection, and web filtering platforms. Each branch or remote office has a single or pair of routers forwarding multiple types of traffic, such as:
Business applications (SAP, ERP)
Office 365 (Outlook, Sharepoint, etc)
Internet browsing
Video and IP telephony
Interactive applications, such as remote desktops
Management and Operational Issues
The device-centric approach has many challenges. For example, application performance troubleshooting requires an administrator to check every router in path hop-by-hop and takes a significant amount of time.
In many WAN environments, quality of service (QoS) configuration is static in nature, as a change in QoS design may take several maintenance windows to deploy across the network.
In a similar way, wireless deployments have transformed from autonomous to controller-based, as many tasks require a coordinated approach in management. For example, Radio Resource Management is one of such tasks, when the channel and transmit power selection is very difficult to maintain manually on every access point.
WAN links are also relatively expensive. In many networks, standby WAN links are required for high availability. Establishing these links takes time and service providers may require fixed-term commitment. In contrast, Internet links are affordable and have shorter lead times to provision.
With traditional design described earlier, traffic going to the workload and applications in a data center has to compete with the services reachable via the public Internet. It is cost-effective to offload Internet traffic to a branch local Internet link.
This interface can also be used as a secondary WAN link connecting sites over VPN connections. However, it is difficult to manage multiple tunnels as the number of routers goes up while providing consistent user experience and ensuring that the security is not compromised.
SD-WAN Design Approach
SD-WAN addresses these issues. A centralized set of controller devices provides a level of abstraction, so network administrators can spend more time on creating policies and configuration templates without having to touch every device on the network.
WAN is treated as a transport-agnostic fabric. Underlay network provides connectivity between tunnel endpoints and doesn’t need to have knowledge about reachability information behind these gateways. As a result, overlay tunnels can be created dynamically and networks can recognize application traffic and select the best path in real-time.
Components and Architecture
SD-WAN operations comprise of 4 planes, implemented by a set of controllers and gateways:
Management plane controller (vManage)
Orchestration plane controller (vBond)
Control plane controller (vSmart)
Data plane forwarding device (vEdge)
Controllers can be hosted and managed by Cisco as a subscription-based product or can be deployed on-premises. vManage, vBond, and vSmart are virtual machines available for download as OVA files. ESXi and KVM are the supported hypervisors.
Figure 1. SD-WAN Architecture
The first component to be configured in a new SD-WAN network is vManage, which can be deployed as a single appliance or cluster of at least 3 nodes. vManage implements a management plane and is the place where all configuration happens. It also performs fabric monitoring and can expose centralized API access for external applications to the SD-WAN network.
vBond is responsible for accepting registration and authenticating vSmart controllers and vEdges. Every device needs to be pointed to vBond during provisioning. It then ensures that all other elements are able to locate each other. vBond must have a public IP address and should be placed into DMZ, so it can be accessed over the Internet.
vSmart controls all overlay routing and secure tunnel establishment between vEdges. The control protocol between vSmart and vEdge elements is called OMP (Overlay Management Protocol). It is protected by DTLS and carries not only reachability information, but also security associations details for IPSec tunnels. vSmart performs policy propagation to the edge devices.
vEdge devices are gateways performing data forwarding over overlay networks. This can be Viptela appliances (vEdge Routers), or Cisco devices running SD-WAN image such as Cisco ISR 4000. There is an option of software vEdge Cloud routers hosted in the public cloud – AWS or Azure.
Cisco works on getting routers with SD-WAN image to have feature parity with Viptela appliances, so always check release notes, as there might be a feature not yet supported on Cisco ISRs.
The next few sections explain the most important terms and concepts of SD-WAN, such as VPNs, TLOCs, and OMP.
Figure 2. SD-WAN Terminology
VPNs
Viptela SD-WAN uses the concept of VPN which is a way to segregate networks. Each VPN has interface allocation and a routing table isolated from other VPNs. It is similar to the Cisco VRF (Virtual Routing and Forwarding) instance. VPN number is globally significant and must match for communication to happen. Encapsulated IP packets carry VPN tag, so egress gateway can determine which VPN packet belongs to.
There are 513 VPNs with the first and last reserved for fabric operations. VPN 0 is transport VPN and is similar to the global VRF context. Interfaces in VPN 0 are called tunnel interfaces and have IP addresses visible by transit networks and form underlay of the fabric. Communication between the network controllers of SD-WAN happens over VPN 0.
VPN 512 is used for Out-Of-Band-Management network.
All other VPNs 1-511 can be used to forward user data.
In Figure 2, VPN 100 and VPN 200 are created in the network. Subnets A, B, and E can communicate with each other within VPN 100. And subnets C and D can communicate with each other within VPN 200.
TLOCs (Transport LOCators)
One of the tasks of OMP is to distribute reachability information. Each destination can be reachable via a specific interface on one of the vEdges on the network. TLOC is a composite structure describing this interface and consists of:
System IP address of the OMP
Color of the link
Encapsulation of the tunnel (IPSec or GRE)
TLOC is similar in concept to the next hop in BGP. Color is a pre-defined tag that describes type of the WAN interface, for example mpls, 3g or biz-internet.
OMP (Overlay Management Protocol)
vSmart exchanges information with vEdges using OMP. This protocol covers all control-plane aspects required to transmit data on top of the overlays.
OMP is responsible for exchange of 3 types of routes:
vRoutes, reachability on the LAN side of the router. vEdge supports static routes, dynamic protocols – BGP and OSPF are supported. Information about a source routing protocol, its metric is carried along with these routes. VPN, the Site ID is another important information present in vRoutes as well.
Service Routes. The way to perform service chaining and insert a firewall or a load balancer
TLOC Routes. Carries information on how to reach specific TLOC such as IP addresses of the interface.
