The procedure described in this article can be used as part of the bootstrap process of Cisco IOS-XE devices. We start with a CSR1000V SD-WAN edge router in CLI mode, registered with vManage.
The topology has a single router connected to the Internet via an Ethernet interface. We configured the IP address on this interface and the default static route statically.
The configuration includes an out-of-band management interface, so we are not limited to console connectivity for troubleshooting and can use SSH over the Ethernet interface. This part of the configuration is optional, and you can skip the templates for out-of-band VPN 512.
Figure 1. SD-WAN templates sample topology
See the router’s basic CLI configuration in the listing below. We will substitute it with generated configuration from the feature and device templates while ensuring that the device is still able to connect to vManage.
We don’t use the service-side (LAN) configuration in this example. Additional templates can enable other settings after the device is switched to template-based management.
Base CLI Configuration
The basic configuration of the SD-WAN edge device includes the following details:
System parameters (lines 3-10)
Transport and tunnel interface configuration (lines 25-33, 41-45)
Transport VPN routing (line 35)
Out-Of-Band Management VPN and Interface (optional, lines 12-23)
config-transaction
!
system
system-ip 3.1.1.1
site-id 1
admin-tech-on-failure
organization-name SD-WAN-TESTLAB
vbond 100.1.1.51
!
hostname CSR01
!
vrf definition Mgmt-intf
rd 1:512
!
address-family ipv4
route-target export 1:512
route-target import 1:512
exit-address-family
exit
!
interface GigabitEthernet1
vrf forwarding Mgmt-intf
ip address 192.168.1.53 255.255.255.0
!
interface GigabitEthernet2
no shutdown
ip address 21.1.1.2 255.255.255.252
!
interface Tunnel0
no shutdown
ip unnumbered GigabitEthernet2
tunnel source GigabitEthernet2
tunnel mode sdwan
!
ip route 0.0.0.0 0.0.0.0 21.1.1.1
!
line vty 0 4
transport input ssh
!
sdwan
interface GigabitEthernet2
tunnel-interface
encapsulation ipsec
color default
exit
exit
!
commit
Feature templates
In this article, we will create all feature templates from scratch.
To add a feature template refer to the screenshot below.
Start by selecting Configuration > Templates in the side menu. Then click on the Feature tab and the “Add Template” button. Select CSR1000v as the device type. I highlighted the templates used in this article – System, VPN, and Ethernet.
Figure 2. Add new feature template
Step 1. System template configuration
This template defines System IP, Site ID, and Hostname details. We want to generate the following configuration with the template:
To add a System template select the “Cisco System” button in the “Add Template” window shown in Figure 2. Set the name and description of the template.
Default options have variables defined for the parameters we want to define. The only mandatory setting is the Console Baud Rate, and figure 3 shows the required details.
Figure 3. System template configuration
Step 2. Transport VPN
This template defines transport VPN, a global routing context (VRF 0) in Cisco IOS-XE. With this template, we aim to generate the static default route from the complete configuration:
ip route 0.0.0.0 0.0.0.0 21.1.1.1
To add it click on the “Cisco VPN” button on the add feature template page, shown in Figure 2. Add name and description of the template.
Figure 4. Transport VPN template configuration
Set VPN value of 0, which is reserved ID for transport VPN.
For our configuration, we need only specify a default route. To be able to re-use template, use a device variable for the next hop (vpn_next_hop_ip_address_0). This variable can be either the IP address of the next-hop or interface name for point-to-point connections.
Don’t forget to press Add button (#10), as the configuration window will not warn you that the route is not added, which will cause the device to lose connectivity to vManage. Not saving sub-configuration sections is a common issue, which can happen across many vManage configuration pages, as the Web user interface doesn’t force a user to save or discard when adding sub-elements.
Step 3. Transport interface
This template generates the configuration shown below:
interface GigabitEthernet2
no shutdown
ip address 21.1.1.2 255.255.255.252
!
interface Tunnel0
no shutdown
ip unnumbered GigabitEthernet2
tunnel source GigabitEthernet2
tunnel mode sdwan
!
sdwan
interface GigabitEthernet2
tunnel-interface
encapsulation ipsec
color default
exit
exit
Use procedure from the first section to get to the Feature template selection (Figure 2) and then select “Cisco VPN Interface Ethernet” as the type.
Specify the template name and description. Change the following values:
This configuration is similar to the previous two steps. The management VPN uses a reserved ID of 512. SD-WAN overlay doesn’t transport it over, which is why it is called out-of-band. You can only access it locally or expand it using a dedicated network.
In this example, we don’t set up any static routes in VPN 512, as we plan to connect to the device locally via port GigabitEthernet1. In your network, you can set up static routes without any risk to the transport or other VPNs, as each VPN is a VRF, which has its isolated routing table.
The interface configuration differs from the transport interface by not having the tunnel option enabled.
Figure 7. Management interface in VPN 512
Step 5. Configure device template
We have created five feature templates, as shown in Figure 8.
Figure 8. Feature templates list
Let’s create a device template using the steps shown in Figure 9. Select device type, then provide a name and description for the template. Change the following templates:
Cisco System
Cisco VPN 0
Cisco VPN Interface Ethernet
Cisco VPN 512
Cisco VPN Interface Ethernet (add it first)
Figure 9. Create a device template
Cisco AAA template defines user authentication for out-of-band management access. The pre-configured AAA template defines default credentials – user/password combination of admin/admin. Use them to SSH to the device via VPN 512.
You should change it in the production network as a security precaution by defining a custom Cisco AAA template. Save the device template.
Step 6. Apply device template
Now it’s time to apply the template to the device. Open the template configuration page select the device template created in the previous step. Click on the three dots column and then on Attach Devices.
Select CSR01, click on the right-pointing arrow, and then the Attach button.
Fill in values for the variables, which came from the templates defined earlier, and press Update.
List of variables and their values:
ge1_if_ipv4_address – 192.168.1.53/24
vpn_next_hop_ip_address_0 – 21.1.1.1
ge2_if_ipv4_address – 21.1.1.2/30
system_host_name – CSR01
system_system_ip – 3.1.1.1
system_site_id – 1
Figure 10. Apply the device template
The final step is to check the configuration changes in CLI, as shown in Figure 11. Check the transport interface and VPN 0 static route.
In this blog post, we want to show how to enable a zone-based firewall on the Cisco SD-WAN platform. The example continues on the topology in the Direct Internet Access article. We introduced an additional site to demonstrate that the configuration applied doesn’t affect inter-site traffic.
The diagram below shows the topology with a PC behind CSR1. It can access the Internet without any restrictions after enabling Direct Internet Access. If we want to restrict what is reachable from the service side, two options are available – access lists attached to the internal interface and zone-based firewall functionality. As on the traditional, non-SD-WAN router, access-lists are used in scenarios when no stateful inspection is required, for example, when you want to drop RFC 1918 traffic coming from your guest WIFI segment. Cisco zone-based firewall adds the ability to identify the application and stateful inspection, which allows return traffic if it’s part of the permitted session.
Figure 1. Sample Topology
Initial setup for DIA blocking all outbound traffic
The first section will create and associate a security policy that will block all traffic from the internal network to the Internet.
To apply the firewall rules, use a localized security policy. Navigate to Configuration > Policy. Click on Add Security Policy. Choose a custom policy from the list below, as this option shows all possible configuration elements.
Figure 2. Create a Security Policy
The first screen in the wizard is firewall configuration. Click on Add Firewall Policy, and then on Create New.
Figure 3. Add New Firewall Policy
Define source and destination zone pairs by clicking on Apply Zone-Pairs button. Select source and destination zones. You can create zones within the wizard by specifying which VPNs will be part of them.
Figure 4. Define Security Zones
Provide Name and Description for the firewall policy. We can add rules as required; however, we want to drop all the traffic first. The default action, which is applied when no rules match the packet, is Drop (you can change this behavior by selecting Pass in the dropdown menu).
Figure 5. Name and Description of Security Policy
Review created firewall policy and press Next.
Figure 6. Firewall Policy Review
We’ve clicked through all remaining pages of the wizard without adding any configuration. On the “Policy summary” page, provide Security Policy Name and its description.
Figure 7. Security Policy Review
After clicking on the Save Policy button, we can see the policy we’ve just created in the list.
Figure 8. List of Security Policies
Now it’s time to apply the policy to the device. The device template applies the policy to the device.
Figure 9. List of Device Templates
Choose ISR1-Security-Policy in the Security Policy dropdown. And press the Update button to push the configuration to the device.
Figure 10. Apply Security Policy to Template
The listing below shows the config lines are sent to the device based on the configuration we’ve made so far (you can check this via configuration difference preview before the configuration push). As we haven’t specified any specific rules, the policy uses only the class-default class with drop action. The ‘inspect’ firewall policy is defined and applied within the zone-pair configuration block.
parameter-map type inspect-global
alert on
log dropped-packets
multi-tenancy
vpn zone security
policy-map type inspect VPN1-to-VPN0
class class-default
drop
zone security VPN0
vpn 0
zone security VPN1
vpn 1
zone-pair security ZP_VPN1_VPN0_VPN1-to-VPN0 source VPN1 destination VPN0
service-policy type inspect VPN1-to-VPN0
The test shows that the ICMP traffic is blocked as soon as the policy is applied.
Figure 11. Traffic to the Internet is now blocked
To demonstrate that our policy didn’t affect traffic within VPN 1, let’s ping the PC behind CSR02 at another site.
Figure 12. Site-to-site traffic is not affected
Adjust policy to allow ICMP traffic
This section will allow ICMP traffic to the Internet by modifying the policy.
Return to the policy list via Configuration > Security. Choose the policy that we’ve created earlier and press Edit.
Figure 13. Edit existing security policy
Click on the firewall section, choose the “VPN1-to-VPN0” firewall rule and press Edit.
Figure 14. Edit existing firewall policy
Add the new rule by clicking on the “Add Rule” button.
Figure 15. Add a rule to the firewall policy
Specify ACL entry name and choose action to apply. Drop is self-descriptive. Inspect creates a stateful record of the session and automatically allows return traffic. Pass action – will not allow return traffic; for this action to work, you will be required to create another zone-pair for reverse direction and explicitly allow return traffic.
Define matching traffic and action. Protocol number 1 matches ICMP. Select one of the pre-defined protocols for TCP and UDP traffic. If unavailable or non-standard port numbers need to be specified, use ‘tcp’ or ‘udp’ as a protocol along with the “Destination Ports” condition.
Figure 16. Define the access-list entry
Review the rule, save it, and its parent firewall policy.
Figure 17. Firewall rule and policy summary
vManage sends the following commands to the device.
The first three commands are object groups that identify the source, destination, and protocol.
Then access list is defined using the object groups. The class map that follows uses the ACL as a “match” condition.
Finally, policy-map now has a custom class-map statement placed above the default. The action for this traffic is ‘inspect,’ so return packets are automatically allowed.
object-group network VPN1-to-VPN0-seq-Allow_ICMP-network-dstn-og_
host 15.15.15.10
object-group network VPN1-to-VPN0-seq-Allow_ICMP-network-src-og_
192.168.11.0 255.255.255.0
object-group service VPN1-to-VPN0-seq-Allow_ICMP-service-og_
icmp
ip access-list extended VPN1-to-VPN0-seq-Allow_ICMP-acl_
permit object-group VPN1-to-VPN0-seq-Allow_ICMP-service-og_ object-group VPN1-to-VPN0-seq-Allow_ICMP-network-src-og_ object-group VPN1-to-VPN0-seq-Allow_ICMP-network-dstn-og_
class-map type inspect match-all VPN1-to-VPN0-seq-1-cm_
match access-group name VPN1-to-VPN0-seq-Allow_ICMP-acl_
policy-map type inspect VPN1-to-VPN0
class type inspect VPN1-to-VPN0-seq-1-cm_
inspect
class class-default
drop
Figure 18. ICMP traffic to the Internet is now allowed
To monitor or troubleshoot firewall sessions, use the Real-Time menu in the device monitoring, as shown in the screenshot below. The following options provide relevant real-time information:
Policy Zone Pair Sessions – displays list of sessions and which zone pair and class map is matched for each session
Policy Zone Pair Statistics – shows statistics of packets and bytes matching per class-map
IP NAT Translation – displays NAT translations, which can be useful in DIA troubleshooting scenarios
Figure 19. Use vManage Web interface to view firewall sessions
To view active sessions using CLI as they are passing, use show policy-firewall sessions command:
CSR01#show policy-firewall sessions platform ?
all detailed information
destination-port Destination Port Number
detail detail on or off
icmp Protocol Type ICMP
imprecise imprecise information
session session information
source-port Source Port
source-vrf Source Vrf ID
standby standby information
tcp Protocol Type TCP
udp Protocol Type UDP
v4-destination-address IPv4 Desination Address
v4-source-address IPv4 Source Address
v6-destination-address IPv6 Desination Address
v6-source-address IPv6 Source Address
| Output modifiers
<cr> <cr>
It is possible to filter the output using one of the keywords above. We will display all sessions with 'all' keyword.
CSR01#show policy-firewall sessions platform all
--show platform hardware qfp active feature firewall datapath scb any any any any any all any --
[s=session i=imprecise channel c=control channel d=data channel A/D=appfw action allow/deny]
Session ID:0x00000001 192.168.11.10 8 15.15.15.10 1 proto 1 (1:1:1:1) (0x3:icmp) [sc]
To display detailed information on the session, which includes ingress and egress interfaces, translated addresses, and other information use detail keyword.
Packet capture provides a way of getting a copy of the packets traversing a router. This can be useful for troubleshooting purposes when you want to see if the packets are being received or sent by the router via the expected interface.
There are 2 ways to perform the packet capture – one is using the vManage user interface, and another one is using CLI directly on the router. In this article, we will explain how to use both of them.
Using vManage
Packet capture is reachable via the Device Troubleshooting page – Monitor > Network > Device name > Troubleshooting. By default, there is no Packet Capture option under the Traffic section, as shown in Figure 1.
Figure 1. Packet Capture in vManage before Data Streaming is enabled
The pop-up alert displays: “Data Stream is disabled. Go to the Settings page to enable Data Stream to use Packet Capture, Speed Test, and Debug Logs”. To run packet captures via vManage we must enable Data Stream.
Navigate to Administration > Settings. Then expand the Data Stream option, as shown in the screenshot below. Click on the “Enabled” radio box, type in the hostname, which is the name or IP address of vManage and VPN that the edge router should use to reach the controller.
Figure 2. Enable Data Stream Configuration in vManage
Note that for virtualized platforms, like the one we use for the lab, VPN 512 (out-of-band) cannot be used. To make this work, we are using the public IP of vManage, which is reachable via transport VPN 0. Our lab topology is shown in the figure below.
Figure 3. Data Streaming Topology
After enabling Data Streaming, the Packet Capture option is now visible in the Troubleshooting section. After clicking on this option, we can define packet capture parameters.
Figure 4. Packet Capture in vManage after Data Streaming is enabled
Packet capture screen requires VPN and Interface filter selection. You can optionally provide other filters, such as source and destination IPs and protocol information. Traffic is captured in both ingress and egress directions. Let’s change the filter to protocol 1 (ICMP) and start capture by pressing the Start button.
Figure 5. Packet Capture Parameters
By default, the capture will run for 5 minutes. You can stop the timer at any time to download packets captured so far. The file in pcap format will be available for download shortly.
Figure 6. Packet Capture Progress
The content of the file can be viewed in Wireshark, as shown below.
Figure 7. Display Captured Packets in Wireshark
CSR01#show monitor capture
Status Information for Capture 71f87e76_847e_4770_8289_56b5242ac115
Target Type:
Interface: GigabitEthernet3, Direction: BOTH
Status : Active
Filter Details:
IPv4
Source IP: any
Destination IP: any
Protocol: 1
Buffer Details:
Buffer Type: LINEAR (default)
Buffer Size (in MB): 5
Limit Details:
Number of Packets to capture: 0 (no limit)
Packet Capture duration: 300
Packet Size to capture: 0 (no limit)
Maximum number of packets to capture per second: 1000
Packet sampling rate: 0 (no sampling)
Using CLI on the router
If for some reason you can’t use vManage, you can use IOS-XE Embedded Packet Capture directly on the device (the previous process uses this feature on the backend). Use SSH to connect to the device either via client installed on your computer or via the tools menu in vManage.
The next configuration commands provide an example of running packet capture.
Embedded packet capture commands begin with monitor capture commands. They are available in exec mode, other operational commands, like “show” and “debug”.
CSR01#monitor capture ?
WORD Name of the Capture
clear Clear all Buffers
start Enable all capture points
stop Disable all capture points
Specify a name for the packet capture instance, in our example it is TEST_CAPTURE. The available command options are shown below.
CSR01#monitor capture TEST_CAPTURE ?
WORD Name of the Capture
access-list access-list to be attached
buffer Buffer options
class-map class name to attached
clear Clear Buffer
control-plane Control Plane
export Export Buffer
interface Interface
limit Limit Packets Captured
match Describe filters inline
start Enable Capture
stop Disable Capture
stop_export Disable Capture and Export Buffer
The next commands configure the same options we used in vManage:
GigabithEthernet3 as interface
ICMP packets only (IP protocol 1)
CSR01#monitor capture TEST_CAPTURE interface GigabitEthernet3 both
CSR01#monitor capture TEST_CAPTURE match ipv4 protocol 1 any any
Below are the available options for inline filters.
CSR01#monitor capture TEST_CAPTURE match ?
any all packets
ipv4 IPv4 packets only
ipv6 IPv6 packets only
mac MAC filter configuration
pktlen-range Packet length range to capture
CSR01#monitor capture TEST_CAPTURE match ipv4 ?
A.B.C.D/nn IPv4 source Prefix <network>/<length>, e.g., 192.168.0.0/16
any Any source prefix
host A single source host
protocol Protocols
CSR01#monitor capture TEST_CAPTURE match ipv4 protocol ?
<0-255> An IP protocol number
tcp Filter by TCP protocol
udp Filter by UDP protocol
CSR01#monitor capture TEST_CAPTURE match ipv4 protocol 1 ?
A.B.C.D/nn IPv4 source Prefix <network>/<length>, e.g., 192.168.0.0/16
any Any source prefix
host A single source host
CSR01#monitor capture TEST_CAPTURE match ipv4 protocol 1 any ?
A.B.C.D/nn IPv4 destination Prefix <network>/<length>, e.g., 192.168.0.0/16
any Any destination prefix
host A single destination host
To validate capture parameters run the command: show monitor capture TEST_CAPTURE. As shown in the listing below, by default, the capture will run till its buffer will reach 10MB.
CSR01#show monitor capture TEST_CAPTURE
Status Information for Capture TEST_CAPTURE
Target Type:
Interface: GigabitEthernet3, Direction: BOTH
Status : Inactive
Filter Details:
IPv4
Source IP: any
Destination IP: any
Protocol: 1
Buffer Details:
Buffer Type: LINEAR (default)
Buffer Size (in MB): 10
Limit Details:
Number of Packets to capture: 0 (no limit)
Packet Capture duration: 0 (no limit)
Packet Size to capture: 0 (no limit)
Maximum number of packets to capture per second: 1000
Packet sampling rate: 0 (no sampling)
Now we can activate the defined capture.
CSR01#monitor capture TEST_CAPTURE start
After running some pings from a test PC connected to the service side via GigabitEthernet3, we can validate that packets are being captured. The brief format is shown below. Detailed and dump options display truncated and full packet content.
Let’s stop packet capture with the following command.
CSR01#monitor capture TEST_CAPTURE stop
To analyze packet capture buffer offline, use export it using the command shown below:
CSR01#monitor capture TEST_CAPTURE export ?
bootflash: Location of the file
flash: Location of the file
ftp: Location of the file
http: Location of the file
https: Location of the file
pram: Location of the file
rcp: Location of the file
scp: Location of the file
sftp: Location of the file
tftp: Location of the file
CSR01#monitor capture TEST_CAPTURE export bootflash:test_capture.pcap
Exported Successfully
SD-WAN deployments use the Internet as the transport to replace WAN networks traditionally designed to leverage centralized Internet access via the data center. Direct Internet Access (DIA) refers to the configuration when Internet-facing traffic breaks out directly from the branch router.
Is Network Address Translation (NAT) required for DIA to operate? Yes, NAT maintains a translation table, that tracks outbound sessions from the service side VPNs (LAN), so the return traffic can be sent back without having to leak service VPN routes into VPN 0.
In the Cisco SD-WAN solution, transport-facing and user-facing interfaces belong to different VPNs or VRFs. VPN 0 contains transport (or underlay) network-facing interfaces, such as Internet and MPLS. Service-side VPNs contain user-facing interfaces.
The figure below shows logical VPN isolation within a router. In the routing table of VPN 0, there will be no entries for subnets where Host A and Host B are located. These subnets can even have the same IP addresses.
Figure 1. Transport and Service-Side VPNs and DIA
For DIA to work we need to allow traffic to flow between these virtual routers (or VPNs). To direct traffic from service-side VPNs we can use either static routes or a centralized data policy. NAT in transport VPN allows return traffic to be sent back.
Direct Internet Access on Cisco SD-WAN platforms is enabled in 2 steps. The first one is the NAT configuration on the transport interface. The second step directs traffic from service-side VPN using either a static route or centralized data policy.
Step 1: Enable NAT on the transport interface
Let’s start with a very basic topology, shown in Figure 2.
Figure 2. Sample DIA Topology
Edge router has a device template assigned, which references a basic set of feature templates required to provide connectivity.
The first step of both static route and policy-based configuration is to enable NAT on an interface in the transport VPN – GigabitEthernet2. This is done by adjusting the interface template.
Figure 3. Enable NAT on transport interface
The following commands are pushed to the device.
ip nat inside source list nat-dia-vpn-hop-access-list interface GigabitEthernet2 overload
ip nat translation tcp-timeout 3600
ip nat translation udp-timeout 60
interface GigabitEthernet2
ip nat outside
We couldn’t find a way to modify the nat-dia-vpn-hop-access-list used in ip nat insidecommand. This ACL is not visible in the running configuration or in the output of show ip access-lists. In IOS-XE this access list identifies traffic to be translated. In SD-WAN, however, to achieve this data policy needs to be configured.
Step 2: Direct traffic from service-side VPN
There are 2 ways to achieve this:
Static route in service VPN template
Centralized data policy
Step 2 (option 1): Static Route Configuration
Let’s configure a static default route under VPN 1 (Service VPN). Note that VPN is selected as a gateway option.
Figure 4. Configure a static default route
The static route command that is pushed to the device looks like this: ip nat route vrf 1 0.0.0.0 0.0.0.0 global
Note that the static route command has nat keyword. If we turn off NAT enabled in the previous step, the route will disappear. This essentially means that you have to do address translation for the configuration to work.
Let’s do the test from our test PC, confirming that the remote server sees the request as it’s coming from the router’s external IP address.
Figure 5. Test NAT configuration from Windows PC
To check the list of translations, we can run the following command on the router:
CSR01#show ip nat translations verbose
Pro Inside global Inside local Outside local Outside global
tcp 21.1.1.2:5062 192.168.11.10:49158 15.15.15.10:80 15.15.15.10:80
create: 11/14/21 23:16:21, use: 11/14/21 23:16:27, timeout: 00:00:56
RuleID : 1
Flags: timing-out
ALG Application Type: NA
WLAN-Flags: unknown
Mac-Address: 0000.0000.0000 Input-IDB:
VRF: 1, entry-id: 0xe9f7f840, use_count:1
In_pkts: 7 In_bytes: 978, Out_pkts: 7 Out_bytes: 982
Output-IDB: GigabitEthernet2
CSR01#show ip nat translation
Pro Inside global Inside local Outside local Outside global
tcp 21.1.1.2:5062 192.168.11.10:49158 15.15.15.10:80 15.15.15.10:80
Step 2 (Option 2): Centralized policy
We have removed the static route created in Step 2 (option 1), as the traffic will be directed by the centralized policy.
To implement DIA we will configure the traffic data section of the centralized policy that will match traffic coming from 192.168.11.0/24 (LAN segment) to 15.15.15.10/32 (the webserver).
Only one centralized policy can be activated globally at a time. The centralized policy contains multiple component policies. In this example, we will define a data policy, which then can be applied to a site list. In the following steps, we will create a new policy.
Create a centralized policy
Navigate to Configuration > Policies and click on Add Policy button.
Figure 6. Add Centralized Policy
The first step of creating a policy is called “Create Groups of Interest”. These are variables that we can later use in the policy. For our example, we will define:
Data Prefix – for source prefix 192.168.11.0/24, and we will use 15.15.15.10/32 directly in our traffic matching configuration without variable definition
Site List – we want apply only to a single site with Site ID of 1; it is recommended not to have the same site in multiple site lists to ensure that only 1 policy of each type is applied to that site
VPN List – service VPN 1
Figure 7. Configure Groups of Interest (variables)
After defining all required variables and pressing the Next button, we are moved to the Topology and VPN Membership section of the wizard (see the top part of the screenshot below). We don’t need to configure anything for our data policy, so we just press Next.
On the Configure Traffic Rules step of the wizard click on the “Traffic Data” section of the policy, click on Add Policy > Create New (refer to the bottom part of the next screenshot).
Figure 8. Configure Traffic Rules
In a new data policy window enter the name and description of the policy. Adjust the default action to Accept to ensure that the packets that don’t match our criteria for DIA will not be dropped. Finally, press + button to add a rule that will be matching DIA traffic and apply NAT to it.
Figure 9. Create Data New Policy
In the pop-up window select Custom policy. The other options are just a subset of the match and set conditions tailored for different scenarios, custom lists all of them.
Custom rule is added on top of the default action. If there are several rules, you can re-arrange them on the left panel. Ensure that the Match section is selected, add Source and Destination data prefixes to set the conditions for the rule.
Select the data prefix that we set up earlier in groups of interest as the source. Type-in destination as the actual address without the use of the variable. Both options lead to the same result, however, the use of variables allows you to use descriptive naming of the object plus adjusting the values outside of the policy configuration. Click on the Actions button.
Figure 10. Add a new custom rule to the policy and define match conditions
Select NAT VPN action, as shown in the following screenshot. The fallback option is useful when you want this traffic to follow the routing table when NAT cannot be used, for example, when the interface is down. Press the “Save data policy” button.
Figure 11. Set policy action to NAT VPN
The next screenshot list the data policy that we built in the previous step. Notice that the reference count is 0, as we haven’t yet applied it yet. Press Next.
Figure 12. Traffic data policies
The final step is to apply the policy. Click on the Traffic Data section, and then under data-policy-1 press “New Site List and VPN List”. In the pop-up window select “From Service” direction, site-1 as the site list, and VPN1 as the VPN list. Press the Save Policy button.
Figure 13. Apply data policy
The final step is to activate the policy.
Figure 14. Activate centralized policy
At this stage vManage will push the policy to vSmart as part of its running-config:
vSmart will use OMP to distribute the policy to edge routers. In contrast to vSmart, edge routers will not display the policy in the running configuration. Use show sdwan policy from-vsmart command instead.
Let’s check from the client machine that NAT works:
CSR01#show ip nat translation
Pro Inside global Inside local Outside local Outside global
tcp 21.1.1.2:5064 192.168.11.10:62121 15.15.15.10:80 15.15.15.10:80
Cisco SD-WAN devices can be either in vManage or CLI mode.
In vManage mode, the configuration is performed on vManage and then pushed to the device. Local configuration changes are not allowed. In CLI mode, changes are performed locally on the device. vManage mode is the preferred and recommended option for most SD-WAN implementations. However, you can occasionally switch devices into CLI mode to perform specific tasks.
Administrators can connect to the device using SSH or serial console to use various show and debug commands in both modes.
We will focus on edge devices in this article. However, controllers can also be in one of these modes. Cisco-hosted controllers are initially provisioned in CLI mode and then converted to vManage mode by a network administrator. Some features require controllers to be switched to vManage mode.
vManage mode
Consider a sample network shown in Figure 1. Our task is to prepare an edge router for a remote office. We’ve applied the required feature and device templates with dynamic IP assignment on the Internet-facing interface. After the router is shipped to the remote location and plugged in, it cannot establish connectivity to vManage. We discover that the ISP doesn’t have DHCP enabled, and a static IP configuration is required.
Figure 1. SD-WAN Edge Sample Scenario
Let’s assume that we can access the router via console cable attached to a laptop on-site.
Run the “show sdwan system status” command on the edge device to validate the device mode. In the sample output below, the CSR01 router is in vManage mode (vManaged: true).
CSR01#show sdwan system status
Viptela (tm) vEdge Operating System Software
Copyright (c) 2013-2021 by Viptela, Inc.
Controller Compatibility: 20.3
Version: 17.03.03.0.4762
<output omitted>
Personality: vEdge
Model name: CSR1000V
Services: None
vManaged: true
Commit pending: false
Configuration template: CSR01
<output omitted>
Firstly, we try to set a static IP address on the transport interface using CLI. We use the “config-t” command to enter the configuration mode. The “commit” command activates the configuration.
CSR01#config-t
CSR01(config)# interface GigabitEthernet 2
CSR01(config-if)# ip address 21.1.1.2 255.255.255.252
CSR01(config-if)# commit
The following warnings were generated:
'system is-vmanaged': This device is being managed by the vManage. Any configuration changes to this device will be overwritten by the vManage after the control connection to the vManage comes back up.
Proceed? [yes,no] yes
Commit complete.
CSR01#show run interface GigabitEthernet 2
Building configuration...
<output omitted>
interface GigabitEthernet2
description Transport Interface
ip address 21.1.1.2 255.255.255.252
In our example, we didn’t have connectivity to vManage, so the changes were committed. However, as the warning message above advises vManage will overwrite modifications done on the device (see the example in the following listing) once the control connection is up.
CSR01#show run interface GigabitEthernet 2
Building configuration...
<output omitted>
interface GigabitEthernet2
description Transport Interface
ip address dhcp
Rollback to the original configuration (i.e. setting dynamic IP on the transport interface) causes connectivity to vManage fail, which introduces the connectivity problem we tried to solve in the first place. When we have connectivity to vManage, the CLI commands are just rejected, as shown in the listing below. vManage mode doesn’t permit changes on the edge device.
CSR01#config-t
CSR01(config)# interface GigabitEthernet2
CSR01(config-if)# ip address 21.1.1.2 255.255.255.252
CSR01(config-if)# commit
Aborted: 'system is-vmanaged': This device is being managed by the vManage. Configuration through the CLI is not allowed.
CLI Mode
There are several solutions to the problem described above. The first is to switch the device into CLI mode beforehand before sending it to the remote location. We can then adjust the configuration and set static IP address, and move the device back to vManage mode.
The second solution is using hidden support commands, which is not a supported way of configuring the device unless directed by Cisco TAC. However, it works and can be helpful in lab environments or as a quick fix to get the branch running (with full router reset later).
Let’s see how both options work with the examples below.
Switching device to CLI mode via vManage
An administrator can switch the device to CLI mode (and back) via vManage. Navigate to Configuration > Devices. Click on the Change Mode drop-down menu and select CLI mode.
Figure 2. Switching device to CLI mode via vManage
In the dialog window, select and move to the right panel the router you want to switch into CLI mode and click on Upgrade to CLI Mode.
After the conversion, we can use “show sdwan system status” command to validate that the vManaged property is false.
CSR01#show sdwan system status
Viptela (tm) vEdge Operating System Software
Copyright (c) 2013-2021 by Viptela, Inc.
Controller Compatibility: 20.3
Version: 17.03.03.0.4762
Build: Not applicable
<output omitted>
Personality: vEdge
Model name: CSR1000V
Services: None
vManaged: false
Commit pending: false
Configuration template: None
<output omitted>
After the device comes back online, we can update templates to include static IP address configuration and switch it back to the vManage mode. To do this, we need to attach the device template to the vEdge router. If the device had a template attached before, the variable values would be populated automatically.
Figure 3. Switching device to vManage mode
Switching device to CLI mode using hidden commands
This procedure in this section uses hidden commands and should be used only as a last resort. Cisco TAC may not support it; use it at your own risk.
When the router hasn’t been switched to CLI mode beforehand and cannot establish a control connection to vManage, refer to the following example to switch to CLI mode.
After entering the configuration mode, let’s see available options under the system command:
CSR01#config-t
CSR01(config)# system ?
Possible completions:
admin-tech-on-failure Collect admin-tech before reboot due to daemon failure
allow-same-site-tunnels Allow tunnels to be formed between vEdges in the same site
console-baud-rate Console baud-rate
control-session-pps Control session policer rate, in packets per second
controller-group-list Controller group list
debug
description System description
device-groups List of vManage groups to which the device belongs
disable
domain-id Domain ID
environment
fnf
gps-location GPS latitude and longitude of the device
host-policer-pps Rate at which to police packets bound to the control plane (in pps) per QOS level
icmp-error-pps Rate at which to police ICMP error messages either generated or received (in pps).
idle-timeout Idle CLI timeout, in minutes
ignore
location Location description of the device
max-controllers (DEPRECATED) Set the maximum number of controllers to which the device can connect - Deprecated in 15.4
max-omp-sessions Set the maximum number of OMP sessions the device can have
mode-button
mtu
on-demand Set various configuration for On-demand tunnels
organization-name Organization name
overlay-id Overlay ID
port-hop Enable port hopping for all tlocs
port-offset Port offset (unique value; use only if multiple Viptela devices are behind the same NAT)
site-id Site ID
sp-organization-name Service Provider Organization name
system-ip System IP address
system-tunnel-mtu Control tunnel MTU
tcp-optimization-enabled Carve out a dedicated core to use for TCP optimization - applies after reboot
timer Set various timer timeouts
tls ssl-opt cert management config
track-default-gateway Enable/Disable default gateway tracking
track-interface-tag OMP Tag attached to routes based on interface tracking
track-transport Enable transport tracking
upgrade-confirm Configure software upgrade confirmation timeout
vbond Configure remote vBond or local IPv4 vbond address
<cr>
“unhide viptela_internal” command enables the display of “system” command hidden options. The example below shows these subcommands.
CSR01(config)# unhide viptela_internal
CSR01(config)# system ?
Possible completions:
admin-tech-on-failure Collect admin-tech before reboot due to daemon failure
allow-same-site-tunnels Allow tunnels to be formed between vEdges in the same site
allow-sw-vedge (HIDDEN) Allow non-release software vedges to operate without certificates
console-baud-rate Console baud-rate
control-session-pps Control session policer rate, in packets per second
controller-group-list Controller group list
daemon-reboot (HIDDEN) Reboot device if a non-restartable daemon fails
daemon-restart (HIDDEN) Restart restartable daemons if they fail
debug
description System description
device-groups List of vManage groups to which the device belongs
disable
dnsd-ttl config DNS reply TTL in secs
domain-id Domain ID
dpi-cache-expiry (HIDDEN) Cache expiry time in minutes
dpi-cache-size (HIDDEN) Cache size
dpi-disable-track-tx (HIDDEN) Enable/Disable DPI TRACK TX
dpi-enable (HIDDEN) Enable/Disable DPI
dpi-gc-time (HIDDEN) Garbage collect time in secs
dpi-multicore (HIDDEN) Enable multi-core for dpi
dpi-stat-time (HIDDEN) Stats collection time for dpi
environment
fnf
fp-buffer-check (HIDDEN) Enable fastpath buffer validity check
fp-qos-interval config fp qos interval
fp-qos-weight-percent-factor config fp qos weight percent factor
gps-location GPS latitude and longitude of the device
host-policer-pps Rate at which to police packets bound to the control plane (in pps) per QOS level
icmp-error-pps Rate at which to police ICMP error messages either generated or received (in pps).
idle-timeout Idle CLI timeout, in minutes
ignore
increase-bp-count (HIDDEN) Increase port backpressure threshold for all ports
is-vmanaged Device is managed by the vmanage
last-vmanage-transaction-id Used by vManage to maintain integrity of transactions initiated by it towards device
location Location description of the device
max-controllers (DEPRECATED) Set the maximum number of controllers to which the device can connect - Deprecated in 15.4
max-omp-sessions Set the maximum number of OMP sessions the device can have
mode-button
mtu
on-demand Set various configuration for On-demand tunnels
organization-name Organization name
overlay-id Overlay ID
patch-confirm (HIDDEN) Configure software patch confirmation timeout
port-bp-threshold configure port backpressure threshold
port-hop Enable port hopping for all tlocs
port-offset Port offset (unique value; use only if multiple Viptela devices are behind the same NAT)
pseudo-confirm-commit Only valid for vmanage ..
reboot-on-failure (HIDDEN) Reboot device if any daemon fails
simulated-color Simulated device's color
simulated-devices Additional number of simulated devices
simulated-wan-ip Starting IP address for the simulated interface
site-id Site ID
sp-organization-name Service Provider Organization name
system-ip System IP address
system-tunnel-mtu Control tunnel MTU
tcp-optimization-enabled Carve out a dedicated core to use for TCP optimization - applies after reboot
timer Set various timer timeouts
tls ssl-opt cert management config
track-default-gateway Enable/Disable default gateway tracking
track-interface-tag OMP Tag attached to routes based on interface tracking
track-transport Enable transport tracking
unpin-flows-with-reboot (HIDDEN) Enable with reboot OR Disable flow pinning to FP cores
upgrade-confirm Configure software upgrade confirmation timeout
vbond Configure remote vBond or local IPv4 vbond address
ztp-status ZTP status
<cr>
To switch the device to CLI mode, use is-vmanaged option.
CSR01(config)# interface GigabitEthernet 2
CSR01(config-if)# ip address 21.1.1.2 255.255.255.0
CSR01(config-if)# commit
Aborted: 'system is-vmanaged': This device is being managed by the vManage. Configuration through the CLI is not allowed.
CSR01(config-if)# exit
CSR01(config)# system is-vmanaged false
CSR01(config-system)# commit
Commit complete.
CSR01#show run int GigabitEthernet 2
Building configuration...
interface GigabitEthernet2
description Transport Interface
ip dhcp client default-router distance 1
ip address 21.1.1.2 255.255.255.0
The device will connect to vManage. However, the controller will not overwrite the configuration. At this stage, vManage’s view of the router configuration will not reflect the actual state. We need to remediate this as soon as possible, either by switching the device to CLI mode or applying a configuration template that brings consistency. Let’s update the device template to use static interface configuration to get to the consistent state. In the screenshot below, we see the template adjustments. The bottom part of the figure contains the snippet of the command difference. As we saw in the previous command listing, the device already had a static IP address configured, but vManage is not aware of that change, so from its perspective, the device still has DHCP settings configured.
Figure 4. Apply Device Template to WAN Edge
Changing WAN Edge System IP
In the final section of this blog post, let’s see another example when switching to CLI mode can be helpful. When done via template change, the change of System IP can cause the configuration push to be stuck and not apply correctly.
As a workaround, you can switch the device into CLI mode, change System IP, and move it to template configuration back. Follow the steps below.
Step 1. To switch the device to CLI mode, refer to the procedure described in the first section and Figure 2.
Step 2. Using CLI apply the configuration change. In the example below, we change System IP from 1.1.1.101 to 1.1.1.11.
Step 3. Move device to vManaged mode by applying template and ensuring that the System IP variable matches the new value.
This blog post describes configuring a site-to-site IPsec VPN tunnel from a Cisco SD-WAN IOS-XE-based router to a non-SD-WAN device.
How to enable configure Cisco SD-WAN IPsec Tunnels to a non-SD-WAN device? In Cisco SD-WAN template-based deployment, IPsec tunnels are configured via the Cisco VPN Interface IPsec feature template. This template is then applied to the Transport VPN (0) or one of the Service VPNs.
Cisco SD-WAN IPSec Tunnels Step-by-step
Figure 1. Configuration Map: Elements Diagram
The logical elements required to be configured shown in Figure 1. All pre-configured elements have check mark symbols next to them.
In our example, the edge device has a device template attached with basic configuration applied, such as system and transport interfaces, sufficient for the router to have control connections to the controllers.
The device template uses a service VPN, which is described by the Cisco VPN feature template. This type of template, despite its name, is not related to IPsec VPN settings. Cisco VPN feature template defines VRF settings and is a container for routing and participating interface information. In our example, the user-facing interface is assumed to be configured and associated with the VRF.
While it is possible to configure all child templates from within the device template, in our example, we will pre-configure child feature templates first and then select them in the device template.
Create and Configure Cisco VPN Interface IPsec Feature Template
The first two steps deal with configuration of IPsec feature template.
Select model of devices that this feature template will be applied
Select Cisco VPN Interface IPsec
Figure 3. Create new feature template in vManage
Step 2. Configure feature template
Customize IPSec tunnel parameters. There are 5 sections in IPsec template:
Basic configuration, such as name and IP address of the tunnel interface and its underlying source (local router) and destination (remote router)
Dead-peer detection settings
IKE or Phase 1 parameters
IPSEC or Phase 2 parameters
Advanced Settings
SD-WAN requires an IP-numbered interface (/30) and supports route-based tunnels known as VTI (Virtual Template Interface) in Cisco IOS documentation.
Instead of specifying interesting traffic using ACL known as policy-based tunnels, route-based tunnels use static or dynamic routing over a tunnel interface.
Figure 4. Configure feature template in vManage
As figure 4 shows, there are various options available for both IKE and IPSEC security parameters. These need to match between tunnel endpoints.
Adjust device template to use IPsec Feature Template
Step 3. Add feature template to the device template.
IPsec interface template can now be attached to the service VPNs. Figure 6 shows how to modify the existing device template.
Figure 6. Add IPsec template to service-side VPN.
Routing Configuration
Figure 7. Configuration Map: Static Routes over IPsec interface
Step 4. Set-up routing over the tunnel. This can be static or dynamic-routing protocol-based. In the screenshot below, the static route configuration is shown.
Figure 8. Configure routing over IPsec tunnels
Step 5. Test the tunnel.
As the tunnels are VTI-based and have Layer 3 addresses on both sides, the simplest test is to ping the remote side of the tunnel.
There is limited information available via real-time monitoring using vManage web interface. Native SD-WAN tunnels are also IPSec-based. These tunnels have centralized authentication and key management done by OMP instead of IKE/ISAKMP protocols used in non-SD-WAN tunnels. Real-time device options that contain string IKE in their name will be relevant to us in the context of this article.
Figure 9. Validate IKE tunnel status
Using SSH connection to the router these 2 commands can be used to check operational details of the tunnel:
show crypto isakmp sa / show crypto ikev2 sa
show crypto ipsec sa
We will demonstrate output of these commands in the practical example below.
Cisco SD-WAN IPSec Tunnels Example
Now it’s time for a practical example. We will establish an IPsec tunnel to a Cisco IOS-XE router configured to match VPN gateways settings in public clouds. For example, AWS provides sample configuration files for different platforms (see this URL). We will apply configuration from the Cisco IOS sample, and we can assume that if our router can work with it, it will work with a real AWS gateway. The configuration is slightly adjusted to use IKEv2 by replacing all isakmp commands with IKEv2-variants.
Figure 10 Cisco SD-WAN IPSec Tunnel Lab Diagram
External router configuration
Non SD-WAN router shown on the top in figure 10 has the following configuration:
interface GigabitEthernet2
ip address 5.5.5.10 255.255.255.0
ip route 0.0.0.0 0.0.0.0 5.5.5.1
crypto ikev2 keyring KEYRING-1
peer 21.1.1.2
address 21.1.1.2
pre-shared-key cisco
crypto ikev2 proposal IKE-PROPOSAL-1
encryption aes-cbc-256
integrity sha1
group 16
crypto ikev2 policy IKE-POLICY-1
match address local 5.5.5.10
proposal IKE-PROPOSAL-1
crypto ikev2 profile IKE-PROFILE-1
match address local interface GigabitEthernet2
match identity remote address 21.1.1.2 255.255.255.255
authentication remote pre-share
authentication local pre-share
keyring local KEYRING-1
crypto ipsec transform-set TRANSFORM-SET esp-256-aes esp-sha-hmac
mode tunnel
crypto ipsec profile IPSEC-PROFILE-1
set security-association lifetime kilobytes 102400000
set transform-set TRANSFORM-SET
set ikev2-profile IKE-PROFILE-1
interface Tunnel0
ip address 169.254.23.2 255.255.255.252
ip tcp adjust-mss 1400
tunnel source GigabitEthernet2
tunnel mode ipsec ipv4
tunnel destination 21.1.1.2
tunnel protection ipsec profile IPSEC-PROFILE-1
ip route 192.168.22.0 255.255.255.0 Tunnel0
SD-WAN configuration
We followed the same steps described in the first part of the article to configure vManage. To make it easier to follow, the majority of parameters are hardcoded into the template. In a real deployment, per-device variables can be used to allow for template re-use.
Then the feature template was added to the device template under VPN 1 section (see Figure 6 above) and route to 192.168.15.0/24 was added to VPN 1 feature template (see Figure 8).
Testing and Validation
Let’s assume that we have access only to the SD-WAN router, and testing will be done only from one side of the connection. We will use the router’s command-line interface via SSH from the vManage web console, as it gives access to information not available via the web interface.
The first test that we perform is checking if the remote side of the tunnel is reachable. The “vrf 1” parameter makes sure that the router uses the correct interface.
CSR01#ping vrf 1 169.254.23.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 169.254.23.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
CSR01#
If ping responses are not received, we can run “show crypto ikev2 sa” and “show crypto ipsec sa” commands. The first command displays if the IKEv2 security association is established, which is a prerequisite for IPSEC security associations. The troubleshooting should start here. If IKEv1 is used, the command is “show crypto isakmp sa.”
CSR01#show crypto ikev2 sa
IPv4 Crypto IKEv2 SA
Tunnel-id Local Remote fvrf/ivrf Status
1 21.1.1.2/500 5.5.5.10/500 none/1 READY
Encr: AES-CBC, keysize: 256, PRF: SHA1, Hash: SHA96, DH Grp:16, Auth sign: PSK, Auth verify: PSK
Life/Active Time: 86400/545 sec
IPv6 Crypto IKEv2 SA
We were running ping of the tunnel interface in our example, which is directly connected to both routers. This test might be successful; however, the connectivity between devices behind the tunnel gateways may still not work.
In this case, we can use the “show crypto ipsec sa” command. It displays a set of counters for the number of encrypted and decrypted packets.
If the encrypted packets count is not increasing, that usually suggests a local routing problem when traffic is not being sent out of the tunnel interface.
If the encrypted packets count does increase but decrypted doesn’t, it can mean that the remote router has routing misconfiguration.
There are some useful debug commands available, such as “debug crypto ikev2”. It can generate extensive output on a router with multiple tunnels, so be careful not to overload the production router. In the example below, we’ve changed the key on the other side of the tunnel to break the tunnel. Auth exchange failed message is logged, suggesting that we have mismatched keys and “show crypto ikev2” will not display any tunnels.
CSR01#debug crypto ikev2
Payload contents:
VID IDi AUTH SA TSi TSr NOTIFY(INITIAL_CONTACT) NOTIFY(SET_WINDOW_SIZE) NOTIFY(ESP_TFC_NO_SUPPORT) NOTIFY(NON_FIRST_FRAGS)
*Jul 10 00:46:36.630: IKEv2:(SESSION ID = 2,SA ID = 1):Sending Packet [To 5.5.5.10:500/From 21.1.1.2:500/VRF i0:f0]
Initiator SPI : EE7E2D729412F370 - Responder SPI : B37D8CA8BAB8C150 Message id: 1
IKEv2 IKE_AUTH Exchange REQUEST
Payload contents:
ENCR
*Jul 10 00:46:36.633: IKEv2:(SESSION ID = 2,SA ID = 1):Received Packet [From 5.5.5.10:500/To 21.1.1.2:500/VRF i0:f0]
Initiator SPI : EE7E2D729412F370 - Responder SPI : B37D8CA8BAB8C150 Message id: 1
IKEv2 IKE_AUTH Exchange RESPONSE
Payload contents:
NOTIFY(AUTHENTICATION_FAILED)
*Jul 10 00:46:36.633: IKEv2:(SESSION ID = 2,SA ID = 1):Process auth response notify
*Jul 10 00:46:36.633: IKEv2-ERROR:(SESSION ID = 2,SA ID = 1):
*Jul 10 00:46:36.633: IKEv2:(SESSION ID = 2,SA ID = 1):Auth exchange failed
*Jul 10 00:46:36.633: IKEv2-ERROR:(SESSION ID = 2,SA ID = 1):: Auth exchange failed
*Jul 10 00:46:36.633: IKEv2:(SESSION ID = 2,SA ID = 1):Abort exchange
*Jul 10 00:46:36.633: IKEv2:(SESSION ID = 2,SA ID = 1):Deleting SA
CSR01# show crypto ikev2 sa
CSR01#
And finally we can perform end to end test from the test machines using ping and tracert commands.
Cisco IP Service Level Agreements (SLAs) is a proprietary feature available on Cisco routers and switches, which actively generates monitoring traffic, processes replies, and measures network performance.
This feature can be used to perform continuous end-to-end connectivity testing with automated re-routing over failover links. It can also simulate different application behavior, such as voice and video to check if the network provides the expected level of service.
The examples and features described in this article are based on Cisco IOS-XE version 16.9.
Configuration Components
IP SLA configuration starts with defining an SLA operation and then scheduling it to run immediately or at a specific time.
SLA Operation Definition
To create or edit an SLA operation use the “ip sla <operation-id>” global configuration mode command. It places CLI into the IP SLA configuration sub-mode, where you can select one of the IP SLA types and provide corresponding configuration parameters.
One of the simplest types of IP SLA operations is icmp-echo. The router pings a specified IP address and records round-trip times if the remote side is reachable. In the example below, we will set the type of SLA entry as ICMP echo with the destination’s IP address of 10.0.3.1. The sample topology is shown in Figure 1.
Figure 1. IP SLA Test Topology
The configuration mode changes to IP SLA echo where you can adjust different optional parameters, such as request sending frequency and timeout for replies.
X(config)#ip sla 1
X(config-ip-sla)# icmp-echo 10.0.3.1 source-ip 10.0.0.1
X(config-ip-sla-echo)#?
IP SLAs Icmp Echo Configuration Commands:
data-pattern Data Pattern
default Set a command to its defaults
exit Exit operation configuration
frequency Frequency of an operation
history History and Distribution Data
no Negate a command or set its defaults
owner Owner of Entry
request-data-size Request data size
tag User defined tag
threshold Operation threshold in milliseconds
timeout Timeout of an operation
tos Type Of Service
verify-data Verify data
vrf Configure IP SLAs for a VRF
Scheduling IP SLA
Once SLA is defined and optional parameters are specified, start it by running the “ip sla schedule <id>” command. IP SLA can be configured to start at a specific time or as soon as the command is entered, which is shown in the example below.
X(config)#ip sla schedule 1 start-time now life forever
To check SLA operation’s details use the “show ip sla statistics <id> details” command.
X#show ip sla statistics 1 details
IPSLAs Latest Operation Statistics
IPSLA operation id: 1
Latest RTT: 1 milliseconds
Latest operation start time: 10:08:18 UTC Sat Jan 30 2021
Latest operation return code: OK
Over thresholds occurred: FALSE
Number of successes: 2
Number of failures: 0
Operation time to live: Forever
Operational state of entry: Active
Last time this entry was reset: Never
To see configuration details, including default values for various parameters, use the “show ip sla configuration” command.
X#show ip sla configuration
IP SLAs Infrastructure Engine-III
Entry number: 1
Owner:
Tag:
Operation timeout (milliseconds): 5000
Type of operation to perform: icmp-echo
Target address/Source address: 10.0.3.1/10.0.0.1
Type Of Service parameter: 0x0
Request size (ARR data portion): 28
Data pattern: 0xABCDABCD
Verify data: No
Vrf Name:
Schedule:
Operation frequency (seconds): 60 (not considered if randomly scheduled)
Next Scheduled Start Time: Start Time already passed
Group Scheduled : FALSE
Randomly Scheduled : FALSE
Life (seconds): Forever
Entry Ageout (seconds): never
Recurring (Starting Everyday): FALSE
Status of entry (SNMP RowStatus): Active
Threshold (milliseconds): 5000
Distribution Statistics:
Number of statistic hours kept: 2
Number of statistic distribution buckets kept: 1
Statistic distribution interval (milliseconds): 20
Enhanced History:
History Statistics:
Number of history Lives kept: 0
Number of history Buckets kept: 15
History Filter Type: None
Once SLA operation is scheduled, it cannot be modified. Instead, the old operation can be deleted, and then a new one created again using the same ID.
Different IP SLA Types
To view available types of SLA operations, use context-sensitive help as shown in the next example.
Jitter is a variation in the delay between packets. The smaller the jitter the better performance of time-sensitive applications such as voice. It also means that the network delivers packets with a predictable delay and doesn’t experience congestions causing intermittent delays along the paths.
There are 2 types of SLA operations performing Jitter measurements – ICMP and UDP-based.
ICMP Jitter
ICMP Jitter SLA operation is based on ICMP message types (Timestamp Request and Timestamp Reply). The destination can be any device that supports these ICMP messages. Not all devices support it, or it can be blocked by the firewalls. The previously shown ICMP Echo operation is based on more commonly used Echo and Reply message types.
The configuration in the example below demonstrates how to configure ICMP jitter operation and, that once launched, it reports the round-trip-time (RTT) and jitter statistics from the source to the destination, and vice versa.
X(config)# ip sla 2
X(config-ip-sla)# icmp-jitter 10.0.3.1 source-ip 10.0.0.1
X(config)# ip sla schedule 2 start-time now life forever
X#show ip sla statistics 2
IPSLAs Latest Operation Statistics
IPSLA operation id: 2
Type of operation: icmp-jitter
Latest RTT: 1 milliseconds
Latest operation start time: 00:20:39 UTC Sun Jan 31 2021
Latest operation return code: OK
RTT Values:
Number Of RTT: 10 RTT Min/Avg/Max: 1/1/1 milliseconds
Latency one-way time:
Number of Latency one-way Samples: 0
Source to Destination Latency one way Min/Avg/Max: 0/0/0 milliseconds
Destination to Source Latency one way Min/Avg/Max: 0/0/0 milliseconds
Jitter Time:
Number of SD Jitter Samples: 9
Number of DS Jitter Samples: 9
Source to Destination Jitter Min/Avg/Max: 0/1/1 milliseconds
Destination to Source Jitter Min/Avg/Max: 0/1/1 milliseconds
Over Threshold:
Number Of RTT Over Threshold: 0 (0%)
Packet Late Arrival: 0
Out Of Sequence: 0
Source to Destination: 0 Destination to Source 0
In both Directions: 0
Packet Skipped: 0 Packet Unprocessed: 0
Packet Loss: 0
Loss Periods Number: 0
Loss Period Length Min/Max: 0/0
Inter Loss Period Length Min/Max: 0/0
Number of successes: 4
Number of failures: 0
Operation time to live: Forever
To measure jitter, a source router sends a number of packets (10, by default) periodically. The time between these packets is called interval (20ms). The operation is repeated at a specified frequency (60 seconds by default).
The example below shows default configuration values – every 60 seconds the router will send 10 packets with 20 milliseconds interval between each packet.
X#show ip sla configuration 2
IP SLAs Infrastructure Engine-III
Entry number: 2
Owner:
Tag:
Operation timeout (milliseconds): 5000
Type of operation to perform: icmp-jitter
Target address/Source address: 10.0.3.1/10.0.0.1
Packet Interval (milliseconds)/Number of packets: 20/10
Type Of Service parameter: 0x0
Vrf Name:
Schedule:
Operation frequency (seconds): 60 (not considered if randomly scheduled)
<output is truncated>
UDP Jitter and IP SLA responder
UDP is used in voice and video communications. Using UDP traffic for measurements suits better to simulate such applications. In an IP SLA operation configuration codec type can be specified, and this will define packet size and enable various voice-specific metric calculations, such as Mean Opinion Score (MOS).
Figure 2. IP SLA Responder Configuration
To use the UDP Jitter destination device must also be a Cisco device with an IP SLA responder feature enabled on it. The responder’s basic configuration is completed with the single command on router Z:
Z(config)#ip sla responder
Back on the router X, IP SLA configuration is done per the example below.
X(config)#ip sla 3
X(config-ip-sla)#udp-jitter 10.0.3.1 2345 source-ip 10.0.0.1
X(config)#ip sla schedule 3 start-time now life forever
X#show ip sla statistics
IPSLAs Latest Operation Statistics
IPSLA operation id: 3
Type of operation: udp-jitter
Latest RTT: 1 milliseconds
Latest operation start time: 05:53:56 UTC Sun Jan 31 2021
Latest operation return code: OK
RTT Values:
Number Of RTT: 10 RTT Min/Avg/Max: 1/1/1 milliseconds
Latency one-way time:
Number of Latency one-way Samples: 0
Source to Destination Latency one way Min/Avg/Max: 0/0/0 milliseconds
Destination to Source Latency one way Min/Avg/Max: 0/0/0 milliseconds
Jitter Time:
Number of SD Jitter Samples: 9
Number of DS Jitter Samples: 9
Source to Destination Jitter Min/Avg/Max: 0/1/1 milliseconds
Destination to Source Jitter Min/Avg/Max: 0/1/1 milliseconds
Over Threshold:
Number Of RTT Over Threshold: 0 (0%)
Packet Loss Values:
Loss Source to Destination: 0
Source to Destination Loss Periods Number: 0
Source to Destination Loss Period Length Min/Max: 0/0
Source to Destination Inter Loss Period Length Min/Max: 0/0
Loss Destination to Source: 0
Destination to Source Loss Periods Number: 0
Destination to Source Loss Period Length Min/Max: 0/0
Destination to Source Inter Loss Period Length Min/Max: 0/0
Out Of Sequence: 0 Tail Drop: 0
Packet Late Arrival: 0 Packet Skipped: 0
Voice Score Values:
Calculated Planning Impairment Factor (ICPIF): 0
Mean Opinion Score (MOS): 0
Number of successes: 1
Number of failures: 0
Operation time to live: Forever
Path operations
When measuring latency between two hosts it is useful to know how much delay is contributed by each hop along the path. Path-type SLAs work by running traceroute first and then performing echo or jitter operation against each discovered hop.
X(config)#ip sla 4
X(config-ip-sla)#path-echo 10.0.3.1 source-ip 10.0.0.1
X(config)#ip sla schedule 4 start-time now life forever
The configuration is based on the same network topology with 2 paths available to the destination.
Figure 3. IP SLA Path Operations
The “show ip sla statistics aggregated 4 details” command will display round-trip time statistics for each hop. For example, the command output below shows statistics for the first hop in the path going via router B. The remaining hops statistics are omitted in the example below.
X#show ip sla statistics aggregated 4 details
IPSLAs aggregated statistics
Distribution Statistics:
Bucket Range: 0 to < 20 ms
Avg. Latency: 1 ms
Percent of Total Completions for this Range: 100 %
Number of Completions/Sum of Latency: 1/1
Sum of RTT squared low 32 Bits/Sum of RTT squared high 32 Bits: 1/0
Operations completed over threshold: 0
Start Time Index: *09:54:47.234 UTC Tue Feb 2 2021
Path Index: 3
Hop in Path Index: 1
Type of operation: path-echo
Number of successes: 18
Number of failures: 0
Number of over thresholds: 0
Failed Operations due to Disconnect/TimeOut/Busy/No Connection: 0/0/0/0
Failed Operations due to Internal/Sequence/Verify Error: 0/0/0
Failed Operations due to Control enable/Stats retrieve Error: 0/0
Target Address 172.16.100.6
<output is truncated>
Other operations
IP SLA can perform application-specific monitoring, for example, it can run HTTP or FTP operations against a remote server. For generic TCP applications, TCP operation can be used, which validates if the TCP connection can be established and how long it takes to connect to the server.
The other available operation types are DHCP (checks how long it takes to obtain an IP address) and DNS queries.
Reactions and Proactive Monitoring
In the previous sections, we’ve reviewed different types of SLA operations and how to check their operations using CLI. By using SNMP-based monitoring software, administrators can poll SLA data from routers periodically, which is a more scalable approach to monitoring. Cisco provides SNMP MIB called RTTMON that provides access to IP SLAs from monitoring software.
It is also possible for a router to send an SNMP trap or Syslog message if a certain event happens. IP SLA includes a feature called proactive threshold monitoring. To enable it IP SLA reaction must be configured.
The example below uses a previously configured IP SLA operation of icmp-echo type.
ip sla 1
icmp-echo 10.0.3.1 source-ip 10.0.0.1
ip sla schedule 1 life forever start-time now
The reaction configuration below references SLA operation 1 and selects timeout as a monitored element. Then we enable simple SNMP configuration to send a trap when operations times out.
The final section of this document addresses how a router can change its data plane operation in response to IP SLA operational status.
The feature responsible for translating the status of IP SLA operations to the data plane protocols is called Enhanced Object Tracking. First-Hop Redundancy Protocols, such as HSRP and VRRP use the tracking objects to take over or release data forwarding roles. Static and policy-based routing can also use the tracking objects to re-route around degraded paths.
EEM (Embedded Event Manager) scripts can monitor tracking object’s status changes. This makes it possible to run powerful EEM scripting in response to track status change.
A Track object can have 2 states – Up and Down. IP SLA operations can have several return codes. For the purpose of object tracking, we are interested in the 3 return codes discussed below. All other codes that are not covered in this article translate into the Down state of the Track object.
Let’s re-use the IP SLA ICMP Echo operation from the previous example and check the code of the operation. The second command in the listing shows how to check the default threshold (5000ms):
X#show ip sla statistics 1 | incl code
Latest operation return code: OK
X#show ip sla configuration 1 | include Threshold
Threshold (milliseconds): 5000
The operation’s return code is OK, which means that the ping operation is successful and its round-trip time is within the threshold (in our network it is 1ms). OK status always translates to the UP state of the corresponding track object.
If we will turn off the remote interface on router Z, the return code will change to Timeout, as per the listing below. This is the second SLA operation code, which always translates into the Down state of the Track object.
The third code that is relevant for the purpose of track operation is OverThreshold. To demonstrate it, we increased the request data size to 5000 bytes, so the operation takes longer than 1ms. We also decreased the threshold to 1ms, so the operation goes over the threshold.
The OverThreshold code translates differently to the track object state, which we will discuss in the next section.
X(config)#ip sla 2
X(config-ip-sla)#icmp-echo 10.0.3.1 source-ip 10.0.0.1
X(config-ip-sla-echo)# request-data-size 5000
X(config-ip-sla-echo)# threshold 1
X(config)#ip sla schedule 2 start-time now life forever
X#show ip sla statistics 2
IPSLAs Latest Operation Statistics
IPSLA operation id: 2
Latest RTT: 2 milliseconds
Latest operation start time: 09:44:48 UTC Sat Feb 6 2021
Latest operation return code: Over threshold
Number of successes: 2
Number of failures: 0
Operation time to live: Forever
IP SLA Track State vs. Reachability
“track <id> ip sla <sla-id>” command creates a track object that references SLA operation. There are 2 types of track objects for SLA – reachability, and state. Let’s configure both types against the previously configured SLA 2 to see the difference.
X(config)#track 1 ip sla 2 reachability
X(config)#track 2 ip sla 2 state
X#show track
Track 1
IP SLA 2 reachability
Reachability is Up
1 change, last change 00:00:29
Latest operation return code: Over threshold
Latest RTT (millisecs) 2
Track 2
IP SLA 2 state
State is Down
1 change, last change 00:00:17
Latest operation return code: Over threshold
Latest RTT (millisecs) 2
Track 1 is based on reachability and it is UP, even when the underlying IP SLA object is over the threshold. As the name suggests, we only interested in the fact that we can reach the remote side of the connection. Track 2 is state-based and it returns DOWN for the same IP SLA object. Track of the state type checks both reachability and that the IP SLA object is under the specified threshold.
Static Routes
The next listing demonstrates how to use the track objects with static routes:
X(config)#ip route 0.0.0.0 0.0.0.0 172.16.100.2 track 1 200
X(config)#ip route 0.0.0.0 0.0.0.0 172.16.100.6 track 2
X#sh ip route 0.0.0.0
Routing entry for 0.0.0.0/0, supernet
Known via "static", distance 200, metric 0, candidate default path
Routing Descriptor Blocks:
* 172.16.100.2
Route metric is 0, traffic share count is 1
The route via 172.16.100.6 is the preferred, as it has the default administrative distance of 1. The route via 172.16.100.2 is a backup route with an administrative distance of 200. However, the route table shows that the router prefers the route via 172.16.100.2.
For a route to be installed into the routing table the corresponding track must be UP. Because track object #2 is Down, the static route via 172.16.100.6 is withdrawn and the backup route is chosen and installed into the routing table.
EEM Scripts with IP SLA
The final example of this article is an EEM (Embedded Event Manager) script that adjusts the router configuration, generates a Syslog message in response to the Track state change.
The script is triggered when Track object #1 goes down, with the “event track <object-id> state down” command.
The access-list is being adjusted by the script in this example only for demonstration purposes. It can be adjusted to suit real-life requirements. For example, the script can reconfigure VPN tunnel endpoints or enable a backup interface. EEM can perform many other non-CLI actions as well.
To test the script, I’ve turned off the remote interface on router Z causing IP SLA and track to go down. The router generates a test Syslog message and creates an access-list, as shown in the listing below. “terminal monitor” command displays console messages into VTY lines, such as SSH or Telnet.
X#terminal monitor
*Feb 6 10:13:50.285: %TRACK-6-STATE: 1 ip sla 2 reachability Up -> Down
*Feb 6 10:13:50.530: %HA_EM-6-LOG: TRACK-1-DOWN: TEST Message
X#sh run | section ^ip access-list
ip access-list extended INTERNET-IN
permit ip host 1.2.3.4 any
Explain the role and function of network components: Controllers (Cisco DNA Center and WLC*)
Compare traditional campus device management with Cisco DNA Center enabled device management
* Wireless LAN Controllers functions were discussed in the blog post dedicated to wireless devices, however, this article will use WLCs along with other types of controllers for comparison.
DNA Center Overview
Cisco DNA Center is a management software and a controller for SD-Access. At the time of the writing, it is available only as a hardware appliance. DNA Center is positioned in Cisco’s product line as the replacement for Cisco APIC-EM. It can also replace Cisco Prime Infrastructure.
There are 3 available options of hardware appliances to choose from:
DN2-HW-APL (C220 M5, 44 cores) – up to 1000 devices
DN2-HW-APL-L (C220 M5, 56 cores) – up to 2000 devices
DN2-HW-APL-XL (C480 M5, 112 cores) – up to 5000 devices
Cisco supports a single node deployment or a cluster of 3 appliances for high availability.
DNA Center can operate with 2 types of networks:
Traditional campus networks
SD-Access fabric
Traditional Networks
DNA Center can work with non-SD-Access networks similar to traditional network management software. Policy-driven automation is available in this mode, but it is optional. Assurance and analytics functionality can be opened with even only read-only access to the network and provide a safe way to get familiar and to evaluate DNA Center’s features.
SD-Access Fabric
Cisco introduced a new paradigm for medium- to large- size enterprise networks called intent-based networking. In this architecture, an administrator communicates intent to the controller or requesting “what” he wants to achieve. He doesn’t need to specify device-specific instructions (“how?”) for the changes to be applied. A controller accepts instructions from an administrator via GUI or from an application via API and then applies configuration to the devices it controls.
Software-Defined Access or SD-Access is an implementation of this paradigm. We’ve published an article called SD-Access Components and it explains functions of DNA Center only briefly; this blog post is expanding its coverage. SD-Access has multiple underlying protocols to provide scalable and flexible, but relatively complex virtualized network infrastructure. DNA Center is the key that hides the complexity by providing a level of abstraction that allows network operators to focus their attention on more high-level configuration concepts, such as policies.
Role in the network
In SD-Access fabric, DNA Center plays a more essential role compared to one in the non-SDA network. With SD-Access fabric, while the underlay network can be built manually, the overlay networks are created and operated via DNA Center. In the traditional network, an administrator can decide which tasks should be performed by DNA Center and which ones are to be done directly on the device.
DNA Center has ability to perform many tasks in the network lifecycle:
Day 0. Onboarding and discovery. During this stage DNA Center can be used for zero-touch provisioning (ZTP) with Plug and Play (PnP) protocol
Day 1. Provisioning. Policy-based templates can be defined and applied to multiple devices grouped into a hierarchy of sites
Day 2, N. Operation via policy configuration, monitoring, troubleshooting, and software patching. DNA Center has multiple features simplifying network operations tasks, including, Software and Image Management, zero-touch RMA
Controller operations
Management plane
A network controller can use many protocols to interact with the network devices and this communication is referred to as Southbound connectivity. DNA Center can use multiple management protocols, for example, CLI, Netconf or SNMP. To provide some comparison with other controllers – Cisco SD-WAN uses Netconf to push configuration from vManage to vEdge devices and Cisco WLC uses CAPWAP protocol to communicate with access points.
End devices, such as switches and routers run normal IOS-XE software. Their configuration mode can be accessed locally and changes pushed from DNA Center can be observed via a device’s configuration files.
Control plane
Cisco DNA Center in both modes of operations is distributed across switches, access points and routers. It keeps the network devices in charge of their control plane operation. The network will continue to function if the DNA Center appliance is not reachable. For example, it doesn’t participate in dynamic route propagation or reflection, as vSmarts in SD-WAN.
Data plane
Cisco DNA Center doesn’t perform transit data forwarding functions. In comparison, Cisco Wireless LAN Controller can switch traffic in certain deployment modes.
Features and Functions
DNA Center functions and features can be divided into 4 groups, as shown in Figure 1.
Figure 1. DNA Center Features and Functions
Automation
This group of features is responsible for performing operational and provisioning tasks without applying the configuration manually to the devices. Some of the examples are available below.
Network Design and Profiles
Logical separation of the network into a hierarchy of regions and sites. The profiles, which include common parameters, such as DNS, DHCP server details, are then associated with this logical containers, so all sites under them inherit the settings.
Software Image Management (SWIM)
This feature ensures that consistent software image versions are deployed to devices in the network. DNA Center performs checks prior and post-installation. For example, free space on the flash memory is one of such checks.
Network Plug and Play (PnP)
A very useful feature when the number of deployed devices is high. A device just needs to be plugged in and receive an IP address to receive its configuration automatically. To locate DNA Center several discovery methods are supported, such as using a DHCP option and a DNS name.
QoS Configuration Automation
One of the challenging aspects of the day-to-day operation of the network is QoS policy implementation. Applications on the network can change or new ones are added. If the network is managed manually, keeping configuration up-to-date with properly classified traffic and its preferred treatment can consume a lot of time. Different hardware QoS implementation on different models of devices further increases the complexity.
DNA Center provides an intuitive user interface that allows the administrator to select one of the pre-defined application templates and to choose if its business-relevant or not. Then scope, or which devices should have this policy, is selected and configuration is applied to the network.
Assurance
Another function of a controller is to provide centralized monitoring. DNA Center component responsible for it is called DNA Assurance. It provides many unique features, such as the correlation of different types of information; focused 360 views for the network devices and clients; and retrospective view with Network time travel feature.
Dashboards
There are multiple dashboards available each focusing on different aspects of network health. Performance of business-relevant applications, clients and network devices is monitored and top issues are displayed.
Device 360, Client 360 and Network Time Travel
These features provide a device-centric view of a device or a client. It provides an administrator with ability to quickly access relevant to an endpoint or device information and its health score. For example, it simplifies troubleshooting when a user complains about application performance. By using the search function to quickly locate the user and his device, an administrator can identify if there are issues with the network reachability, such as poor RF signal or packet drops.
It also usually takes some time after an issue occurs and before an administrator starts working on the ticket. By that time, an alert can be cleared out making troubleshooting more difficult. Network Time Travel allows focusing the device view on a specific time in the past (up to 14 days) to see events and alerts that were active at that time.
Path Trace
Path Trace visually displays every device in the path between two IP addresses across the network. It can optionally include information about devices that can be blocking the traffic with access lists, as well as interface and QoS (Quality of Service) statistics.
AI Network Analytics
Analytics powered by Artificial Intelligence/Machine Learning algorithms helps to proactively identify issues. First, the network-specific baseline is gathered and learning occurs. This information is then used to evaluate anomalies to alert the administrator of a possible issue.
SD-Access
This group of functions is specific to SD-Access. Includes functions required to perform the SD-Access controller’s management features for fabric infrastructure and fabric wireless.
Fabric Assurance
DNA Center provides additional monitoring features for fabric, such as the correlation of fabric’s underlay and overlay, reachability between fabric edge, control and border nodes.
Group-based Policy Configuration
DNA Center integrates with Cisco ISE (Identity Service Engine) to enable the use of identity-based policies using Cisco TrustSec. Group-based Policy configuration allows an administrator to configure group and policy management from the user interface of the DNA Center, which then communicates with ISE and fabric.
The main purpose of this feature is to let network devices to infer user identity without relying on IP address or VLAN information mapping. For example, when a user or a device is authenticated with ISE, traffic from this device is marked with a special tag called SGT (Security/Scalable Group Tag). SD-Access places this tag into VXLAN header, so other devices can tell which user or group this datagram belongs to and use this information to apply security and QoS policies.
Platform
In the management plane section, we introduced the concept of Southbound protocols in Software Defined Networks, i.e. from the controller to end devices, such as switches and routers. Northbound protocols, as their name suggests are working in the opposite direction, and are responsible for communication from external services to the controller for 3rd party integration. Cisco DNA Center supports REST (Representational State Transfer) APIs for such integration.
Integration with Service Management Platforms
Cisco DNA Center can be integrated via API with Service Management Platforms. This integration provides the ability to interact with platforms such as ServiceNow. For example, the Software Imaging feature of the DNA Center can log a change request in ServiceNow and perform image push only once it is approved. Another use case can be automated ticket logging when DNA Center discovers an issue.
IPAM Integration
IP Address management provides centralized management of IP pool allocation. Integrating with such a system allows DNA Center to reserve pools for workflows.
Self-test questions
What are 2 types of networks that DNA Center can work with?
• Traditional networks as a management platform
• SD-Access fabric as a controller and a management platform
DNA Center supports Southbound and Northbound protocols. What are they?
• Southbound protocols are responsible for communication from a controller to managed devices
• Northbound protocols or APIs provide access to the controller from external systems
This is the third article about the roles and functions of different network components (clock on the links for the first and second parts). In this part we will discuss operations of devices deployed on the network edge – Routers, Firewalls and Intrusion Protection Systems (IPSs).
Network edge provides connectivity between the company’s branch offices, data centers, remote workers, business partners, and the Internet. These devices must support a variety of transport technologies and provide security services.
There are two main types of edge connectivity:
Internet
WAN
Internet connectivity due to its public nature needs higher security control and devices such as firewalls, IPSs are usually deployed to enable the perimeter protection. Wide Area Network (WAN) in many environments was traditionally considered as a trusted network with only routers installed at the edge of each campus and branch.
In general, WAN links are more expensive and may not be available at every location. WAN connectivity is usually backed by agreed service level contracts and has guaranteed bandwidth and predictable latency. On the other hand, Internet links are cheap and offer more bandwidth, but no end-to-end service performance guarantees.
VPN tunneling over the Internet is used as a backup WAN connectivity method in many networks. Newer technologies, such as Cisco SD-WAN often use several Internet connections – as primary and backup WAN transport. SD-WAN routers can actively monitor links performance end-to-end and re-route traffic automatically based on the configured policy.
With these trends, the dividing line between WAN and Internet becomes less clear. Implementing security services on SD-WAN routers or installing firewalls behind it can be a reasonable choice.
Routers
The main function of a router is to perform Layer 3 forwarding or in most networks route IP (or IPv6) packets. Routers can run dynamic routing protocols to find the best paths to remote networks.
As we discussed in part 1 of this series, a Layer 3 Switch performs a similar function.
What is the difference between a Layer 3 switch and a router?
Historically, switches performed only Layer 2 functions and routers were responsible for Layer 3 operations. Topologies, such as router on a stick for inter-VLAN routing, were often deployed. Figure 1 displays a sample network demonstrating how the traffic between VLAN 10 and VLAN 20 hairpins via a router.
Figure 1. Router on a stick
With the introduction of Layer 3 functionality in switches, inter-VLAN routing functionality was moved to them from routers. The term Layer 3 switching is used to describe fast, hardware-accelerated routing.
For example, aggregation-level router, such as Cisco ASR 1002-HX has a performance of up to 78Mpps, which is outperformed by the entry-level Catalyst C9200-24PXG access switch capable of switching up to 262 Mpps (million packets per second).
There is a balance between flexibility and performance. Routers are capable of providing more services and this is the reason they have lower layer 3 processing performance numbers. For example, routers support many types of WAN links (such as DSL variations and 4G LTE), can accept full Internet routing table, provide advanced QoS capability and application awareness, perform firewall zone-based services, establish VPN tunnels, act as voice gateways and many other features.
Cisco website has a tool called Feature Navigator that allows us to look up a feature that a specific platform or software version provides.
Cisco router product portfolio
Cisco routers can be grouped based on the type of location where they are typically deployed:
Service provider routers: ASR 1000, ASR 9000, Cisco 8000 Series
Cisco router software
Previously, all enterprise Cisco routers were running the Cisco IOS software. It is now mostly replaced by IOS-XE. ISR 900 models are the only devices in the current product line that use IOS.
IOS-XE is IOS’s successor and the majority of enterprise-level devices, including ISR 1000, ISR 4000, ASR 1000 and CSR 1000v, are running it. IOS-XE is based on Linux kernel with IOS being a process called IOSd. IOS-XE and IOS share command-line syntax.
Enterprise platforms running IOS-XE can also run the SD-WAN version of the software, which allows the router to be managed by SD-WAN controllers (more information about SD-WAN platforms is available here).
Service provider routers, such as NCS 5000/5500 and ASR 9000 are running IOS XR software.
Self-test question: What are the functions of a router?
• Layer 3 (in most cases IP or IPv6) traffic forwarding
• Maintains remote network reachability information via static configuration or dynamic exchange with other routers
• Supports a wide variety of interfaces, such as Ethernet, DSL, and LTE
• Has application visibility and ability to apply granular Quality of Service policies
• Provides different services, such as VPN, firewall and VoIP services
Firewalls and Intrusion Protection Systems (IPSs)
Let’s start this section by describing the logical functions of a firewall and an intrusion protection system.
Cisco firewall and IPS functions
A firewall evaluates traffic against configured ruleset and then allows or blocks it. A stateful firewall keeps track of allowed connections and can recognize return traffic, i.e. being part of an existing session, so it can be allowed too.
An Intrusion Protection System performs the security policy enforcement on transit traffic by either comparing its content to a set of patterns or by analyzing its behavior. These pre-defined patterns are called signatures and must be regularly updated.
Traditional firewalls and IPSs
Cisco product line used to have two different types of devices – one performing the firewall functions and another one was responsible for intrusion protection. Cisco PIX and its successor ASA (and routers with security feature set enabled) were performing traditional stateful firewall functions. They had some a limited IPS feature set too. However, for the full IDS/IPS functionality, Cisco IPS appliances and hardware modules were required. After SourceFire acquisition, its standalone IPS products were also added to Cisco’s product line.
Cisco provided integration options, however, they were based on two separately managed systems running in parallel. For example, ASA could accept an expansion module providing IPS functionality connected via ASA’s backplane. Later, hardware modules were replaced by virtual software processes using ASA as a host.
Next-Generation firewalls
Many security vendors took an approach of closely integrating both types of features in a single device, which became known as a Next-Generation Firewall (NGFW). Cisco also released a unified software platform that inherited ASA code as a stateful firewall engine and Snort IPS as intrusion protection system. This software is running on the current NGFW platforms and is called Firepower Threat Detection (or FTD).
In addition to stateful firewall and IPS functions, Next-generation firewalls can also provide remote and site-to-site VPN services, malware protection and URL filtering. The intelligence behind FTD NGFW services is provided by Cisco’s TALOS group that collects and analyzes threats to develop definition updates.
All FTD software platforms can be centrally controlled by Firepower Management Center. Smaller models can be configured locally with Web-based Firepower Device Management.
Cisco firewall and IPS product portfolio
Current Cisco’s firewall and IPS product portfolio includes:
Firepower 1000/2100 (ASA or FTD image; locally or centrally managed)
Firepower 4100/9000 (ASA or FTD image; only central management for FTD)
Cisco NGFWv (virtual FTD – hypervisors and public clouds)
Cisco ASAv (virtual ASA – hypervisors and public clouds)
Cisco NGIPSv (for VMware)
ASA 5500-X
Meraki MX
Firewall deployment modes
Firewalls and IPSs are typically deployed on the network boundary with external networks, such as the Internet. Cisco NGFWs support 2 deployment modes:
Routed
Transparent
Figure 2. Routed vs Transparent Deployment Modes
In routed mode, a firewall acts as a Layer 3 device, with each interface is assigned an IP address. Example in Figure 2 has NGFW in the routed mode option on the left. Notice that the workstation uses the INSIDE interface of the firewall as its default gateway.
On the right side, NGFW operates in transparent mode and performs the role of a Layer 2 device. It must be placed between the local network and the router, as there is no explicit configuration on the workstation, such as default gateway configuration to force the traffic to traverse the firewall.
The diagram shows that the transparent firewall is physically connected to the router ensuring that non-local traffic is not able to bypass the firewall.
In cases when such connectivity is not possible, so-called VLAN stitching can be used. To implement it, the connection between the router and the firewall external interface is allocated to different VLANs, which are stitched together by the firewall.
Self-test question: What are the functions of a Next-Gen Firewall?
• Enforce security policy by blocking or allowing packets
• Perform deep packet analysis with application awareness to provide intrusion protection
• Provide additional services, such as VPN, Malware protection and URL filtering
Self-test question: What are two deployment modes of a Next-Gen Firewall?
• Routed mode. In this mode, firewall operates similar to a router and has different IP addresses on interfaces
• Transparent mode. In this mode operates as a network switch and don’t have IP addresses assigned to data interfaces
This is the second part of the series of articles about the roles and functions of different network components (the first part is available here). In this part, we will discuss the operations of Cisco Wireless Access Points (APs) and Cisco WLAN Controllers (WLCs). The purpose of this blog post is to explain what a Cisco-based wireless network consists of and how these elements interact with each other.
Wireless Standards
IEEE 802.11 set of standards defines Layer 1 and Layer 2 operations of wireless networks. The latest standard that Cisco Access Points support at the time of writing is 802.11ax (Wi-Fi 6).
IETF’s RFC 5415 standardizes communication protocol between a WLC and an Access Point – Control And Provisioning of Wireless Access Points (CAPWAP).
Access Points (APs)
Wireless clients connect to an Access Point to communicate with each other and with the devices on the wired network that the AP is connected to. Single Access Point forms a BSS (Basic Service Set), which is identified by its MAC address.
Access Point advertises one or many wireless networks identified by an SSID (Service Set ID). A WLAN can be mapped to a VLAN on the wired side of an access point.
ESSID is the same wireless network, as identified by an SSID but advertised by multiple Access Points that are connected to the same wired network.
Models
The current portfolio of Cisco Access Points is represented by:
Wi-Fi 6 (802.11ax) models, such
as Catalyst 9115, 9117, 9120 and 9130
802.11ac Wave 2 models, such as
Aironet 1815, 2800, 3800 and 4800
Outdoor and Industrial, such as
Aironet 1540, 1552, 1560 and 1570
Meraki MR45 and MR55
Small Business 100, 300 and 500
series
Cisco website provides a selector tool that performs a side-to-side comparison of different AP and controller models. It can be accessed via this URL.
Autonomous vs Lightweight APs
Access Point’s mode of operations can be
either Autonomous or Controller-based. Let’s consider the difference between
management, control and data planes for Access Points operating in different
modes to understand their functions.
Management
Plane
The management plane deals with the static configuration of Access Points. APs in autonomous mode can be managed directly via Web interface or CLI. In contrast, controller-based APs don’t allow direct configuration changes and, instead, are managed by the controller, which provides a centralized interface for an administrator. The controller is not always a dedicated physical or virtual appliance, it can also be cloud-based service (Meraki) or even another access point (Mobility Express and Embedded WLC).
Control
Plane
The control plane is responsible for dynamic access point operations, such as radio parameters management and user authentication. Autonomous APs perform all these tasks on their own. Controller-based (or Lightweight) APs shift these tasks to the controller. For example, a controller can instruct access points to change a radio channel and decrease transmit power, as it can make more informed decisions based on data received from several adjacent access points in the network.
Data
Plane
The Data plane is responsible for moving data between wireless clients and the wired networks. An autonomous AP switches data directly to the wired network based on its SSID-to-VLAN mapping. Lightweight APs have different mode operations which define how they switch data:
Local or Split MAC mode. In this mode, all user data traffic is tunneled to WLC
FlexConnect – central switching mode. Data plane is similar to local mode, however, some traffic can be switched locally. When the controller is not reachable, AP operates as an autonomous AP
FlexConnect – local switching mode. Data plane is similar to autonomous AP, which switches traffic locally to wired network based on configured SSID-to-VLAN mapping using 802.1q tagging
SD-Access mode. In this mode, AP connects to the SD-Access Edge switch and transmits data via SD-Access fabric using VXLAN encapsulation (check this link for more information on SD-Access).
Self-test question: What are the functions of an Access Point?
• Advertises one or more wireless networks identified by SSIDs and allows wireless clients to connect to these networks
• Allows wireless clients to communicate with each other and access wired network
Self-test question: What are the two modes of Access Points operations and their difference?
• Autonomous. Standalone Access Point that operates independently and is individually managed
• Lightweight or controller-based. Requires a controller to perform management and control plane tasks. Data plane operations may be performed locally or tunneled to WLC
Wireless LAN Controllers (WLCs)
Managing a number of autonomous APs is
getting more difficult as device number grows, as the configuration must be
consistent across many devices. WLCs solve this problem by providing
centralized management of the wireless network.
Models
Current Cisco portfolio of controllers
consists of:
WLC 3504 (AireOS)
WLC 5520 (AireOS)
WLC 8540 (AireOS)
Mobility Express on APs
(AireOS)
Catalyst 9800 series (IOS-XE):
9800-L, 9800-40, 9840-80, 9800-CL (virtual)
Embedded WLC on APs and
Switches (IOS-XE)
The recently released versions of WLCs can be compared using the same tool shown in the Access Points section.
Software
Cisco Wireless LAN Controllers were traditionally running AireOS software. The Cisco controller-less solution with the WLC role performed by an 802.11ac Access Points is called Cisco Mobility Express.
Newer controllers are now IOS-XE software-based. New Catalyst 9100 Access Points can run the WLC role and this newer IOS-XE based solution is called Embedded Wireless Controller. Based on the fact that new controllers are IOS-XE based, AireOS most likely will be replaced by IOS-XE. A feature comparison of both platforms can be found here.
A controller can have multiple functions
depending on types of the deployment. The next sections discuss available
options.
Meraki cloud-based management
Meraki MR APs are first associated with their serial numbers with Meraki Cloud, which provides management access for the wireless LAN deployment. The AP-to-Controller communication is out-of-band and Meraki MR APs will continue to function when connectivity to Meraki Cloud is lost. During connectivity outages ability to perform configuration changes is not available.
No user data is being transferred through
Meraki Cloud infrastructure. Security operations, such as authentication are
performed by Meraki Access Point locally. For example, RADIUS authentication
requests for WPA2 Enterprise are being sent directly from an access point.
Split-MAC
This type of deployment is suitable for large campuses, where sufficient infrastructure exists for the controller to be deployed locally. In this scenario, controllers are actively participating in data forwarding. Access Points establish CAPWAP tunnels to the controller. One tunnel is used for the control plane and another carries encapsulated data payload.
From a wired network perspective, all wireless users traffic is originating from WLC’s LAN interface. This simplifies the configuration of switching infrastructure, as access point facing ports no longer require 802.1q trunk configuration and maintenance of allowed VLANs on that interface. Such ports can be configured as access ports. CAPWAP traffic is unicast UDP traffic between the AP and the WLC.
Figure 1 shows a simplified view of the traffic flow with the split MAC. If A sends a frame to C, AP will send it over the CAPWAP tunnel (in yellow) to WLC. AP and WLC can be in different VLANs, as CAPWAP is IP routed traffic. WLC will de-capsulate it and send it on its LAN interface connected to port 2 of the switch. The switch will learn the MAC address of A via port #2 (facing WLC).
Figure 1. Split MAC Traffic Flow
Split-MAC configuration usually offers faster roaming when the user moves from one access point to another. There are some associated drawbacks, such as the requirement to maintain a dedicated WLC, and bandwidth scaling limits imposed by the controller’s platform and increased dependency on the WLC, as Lightweight Access Points cannot operate without an active connection to it.
FlexConnect
The WLC and access points also support FlexConnect mode of operation. It allows Lightweight Access Point to locally switch some or all of the user traffic instead of sending it to the controller via the CAPWAP tunnel. This mode’s purpose is to decrease the amount of traffic that needs to be sent to a controller from the branch offices.
WLC appliances support 2 modes – Central
Switching and Local Switching.
When WLAN is configured to use Central Switching, traffic from an AP is still tunneled to WLC, however, local-site traffic can be enabled for local switching by configuring Split Tunneling. When there is an active connection between WLC and AP, it is in Connected Mode. When the connection is lost, AP moves into a standalone mode and performs switching locally.
AP in FlexConnect Local Switching mode switches all traffic locally, even when AP can reach WLC. It is similar to the operation of autonomous APs which also switches traffic locally by mapping SSIDs to VLANs. Access Points are still controlled by WLC retaining the benefits of centralized management.
Figure 2. FlexConnect Local Switching Traffic Flow
Embedded WLCs (and Mobility Express) rely on FlexConnect Local Switching operation, as there is no benefit in sending encapsulated data over the tunnel to another Access Point that performs the role of WLC.
SD-Access Mode
SD-Access fabric-integrated WLC actively participates in the fabric operation via the control plane integration. For example, a WLC can update host tracking databases of the edge switch when a client registers, so this information is then distributed via a fabric LISP-based control plane.
WLC controls fabric-integrated access
points perform the same functions as non-fabric WLCs, plus fabric specific
operations. For example, a WLC provides an Access Point with VXLAN information
(VNI) during client registration. By integrating with Cisco ISE, WLC can also
provide AP with security tags (SGTs), so the policy can be enforced upstream.
In fabric mode, a WLC doesn’t participate in the data plane operation and all data is encapsulated locally by the fabric access point.
Self-test question: What are the functions of a WLC?
• Provide centralized management of the wireless network
• In some modes of operation transmit user traffic received from APs via CAPWAP tunnel
Self-test question: What are two types of the Cisco WLCs software?
• AireOS. This is the traditional Cisco WLC platforms software
• IOS-XE. New controllers are based on this version of the software
Self-test question: What is Split-MAC WLC mode of operation?
• An Access Points sends all user traffic to WLC where it breaks out centrally
