IT Management Solutions

4 Dec

Isn’t it painful when someone messes up your routers and the whole network loses the default route to the Internet? But isn’t it more painful when you realise that this happened a few hours ago and you didn’t even notice? Been there right?

 

So….what can we do to detect these issues before the IT director comes to you with an angry face? Well, if you have SolarWinds a few options come to mind:

  • Use SNMP in NPM to monitor the routing table and examine the existence of the 0.0.0.0/0 route
  • Use NCM to schedule show ip route 0.0.0.0/0 commands to check if it exists
  • Use VNQM and create an IP SLA operation to 8.8.8.8 (or any other external route)

 

Yep, all of the above would work well, but there is one that really stands out: receive a Syslog message straight away when we lose the default route. But there is a problem here guys, out of the box, no device will notify you when a route is lost,  they will notify you when a routing neighbour goes down, when an interface goes down, but won’t do that on default routing changes.

 

But what would you say if I tell you that you can set up a customised Syslog message that is fired every time the default route is lost ( you can even define the content of the message and the priority)?  - Quoting Sheldon Cooper:  No, I’m not crazy! My mom had me tested!

 

The solution is called EVENT MANAGEMENT

Event management is a powerful and flexible feature available in some network devices that provides a programmatic method to control and perform on-board automation. It gives you the ability to adapt the behaviour of your network devices to align them with your business needs. There are several vendors that have an event management feature such as Cisco Embedded Event Manger (EEM) or Juniper Event Manager.

 

CAVEAT: The purpose of this blog is not to discuss the full potential of Event management, but rather to demonstrate how to make use of it within the SolarWinds Orion platform. If you want to know more about this cool feature, please visit the links above.

 

Let’s get our hands dirty

What we are going to do is to configure our devices to notify me when the default route is gone using EEM. I’m going to use a Cisco device for demonstration purposes, however, as we pointed out before, there are other platforms that support this feature.

 

Imagine the following topology where we have two devices with internet access distributing the default route into the network:

Topology

 

We are using OSPF with a single area in order to redistribute the default route from the WAN routers into our network (PROSWRTRTR01).

 

The following is the output of the Cisco command show ip route:


Gateway of last resort is 10.0.13.3 to network 0.0.0.0

 

O*E2  0.0.0.0/0 [110/1] via 10.0.13.3, 00:00:19, FastEthernet1/1

                [110/1] via 10.0.12.2, 00:00:38, FastEthernet1/0

      10.0.0.0/8 is variably subnetted, 9 subnets, 2 masks

C        10.0.0.1/32 is directly connected, Loopback0

O        10.0.0.2/32 [110/2] via 10.0.12.2, 00:00:38, FastEthernet1/0

O        10.0.0.3/32 [110/2] via 10.0.13.3, 00:00:19, FastEthernet1/1

C        10.0.12.0/24 is directly connected, FastEthernet1/0

L        10.0.12.1/32 is directly connected, FastEthernet1/0

C        10.0.13.0/24 is directly connected, FastEthernet1/1

L        10.0.13.1/32 is directly connected, FastEthernet1/1

C        10.0.14.0/24 is directly connected, GigabitEthernet0/0

L        10.0.14.1/32 is directly connected, GigabitEthernet0/0

      172.16.0.0/32 is subnetted, 1 subnets

O IA     172.16.0.1 [110/2] via 10.0.13.3, 00:00:19, FastEthernet1/1

      192.168.25.0/32 is subnetted, 1 subnets

O IA     192.168.25.1 [110/2] via 10.0.14.4, 00:00:04, GigabitEthernet0/0

      192.168.100.0/32 is subnetted, 1 subnets

O IA     192.168.100.1 [110/2] via 10.0.14.4, 00:00:04, GigabitEthernet0/0

 

Everything looking good so far.

 

Configuration time

First of all I need to monitor default route status on of our Cisco router. Using the track feature will do here:

 

PROSWRTRTR01 (config)# track 1 ip route 0.0.0.0 0.0.0.0 reachability

PROSWRTRTR01 (config-track)#exit

 

Nice… that was easy! Now let’s go ahead and configure Cisco EEM in order to trigger a Syslog message when we lose the default route:

 

PROSWRTRTR01 (config)#event manager applet DEFAULTROUTELOST

PROSWRTRTR01 (config-track)# event track 1 state down

PROSWRTRTR01 (config-track)#action 1.0 syslog priority errors msg “Default route 0.0.0.0/0 is lost”

 

Isn’t it great when configuring a cool feature is so easy? With just four commands we have already configured it.

 

NOTE: please make sure that your device is configured to send Syslog messages with your SolarWinds server’s IP address as the destination.

 

Now it’s testing time. On the scenario previously illustrated these two devices have internet access and distribute the default route into the network:

Topology2

 

We are using OSPF with a single area in order to redistribute the default route from the WAN routers into our network (PROSWRTRTR01).

 

What happens when the default route is lost from one of the devices?  Let’s say that PROSWRTWAN01 stops sharing the default route within the OSPF area.

 

<Router>#show ip route
Gateway of last resort is 10.0.13.3 to network 0.0.0.0

O*E2  0.0.0.0/0 [110/1] via 10.0.13.3, 00:02:40, FastEthernet1/1

      10.0.0.0/8 is variably subnetted, 9 subnets, 2 masks

C        10.0.0.1/32 is directly connected, Loopback0

(continues…)

 

As you can see, we can still reach the Internet as we get the default route from the other device. Let’s break PROSWRTWAN02 too then:

*Aug  3 21:08:53.947: %TRACKING-5-STATE: 1 ip route 0.0.0.0/0 reachability Up->Do wn

*Aug  3 21:08:54.035: %HA_EM-3-LOG: DEFAULTROUTELOST: Default route 0.0.0.0/0 is lost

 

This syslog message is the one we have configured and is telling us that the default route is gone as there is not any routing neighbour advertising a route to this subnet (in this case, the default route).
Let’s double check the routing table:

 

Gateway of last resort is not set

      10.0.0.0/8 is variably subnetted, 9 subnets, 2 masks

C        10.0.0.1/32 is directly connected, Loopback0

 

SolarWinds Configuration time!

So…. thus far we have configured our Cisco router(s) to send us Syslog messages if we lose the default route. So we can now go back to the SolarWinds web console and review the information the Syslog dashboard is showing. At this point I’m sure you guys are busy bees and have lots of stuff to do, and most likely don’t have time (or the will) to review that dashboard periodically looking for default route events. No worries, we can configure SolarWinds to forward this syslog message to our mailbox. Steps:

  • RDP the SolarWinds server
  • Open Syslog Viewer
  • Open Rules/Filters
  • Add a new Rule
  • Change Name to : Default route is lost
  • Go to Message tab
    • Syslog Message Pattern: *Default route 0.0.0.0/0 is lost*

Add New Syslog Rule

 

  • Go to Alert Actions tab
    • Add new action: email

Edit Syslog Rule

 

    • Complete email recipient and Reply address
    • Complete SMTP server

Edit Syslog Rule2

 

  • Finished!

NOTE: there are options that you can implement such as limiting the IP address range, or filtering by severity level, etc… For this particular scenario, the steps above will work for us.

 

To Recap

In this blog article, we have reviewed how we can use the Event Management feature of your network devices in order to get an immediate notification when any of our network devices loses the default route.

 

This is just an example though, the possibilities for Cisco EEM are endless. As an illustration: we all know Syslog is great but has many downsides: such as verbosity and volume (too many non-important messages being generated) or inconsistency (different devices will send different types of Syslog). This is something that we can solve by using Cisco EEM:

 

  1. We can modify the output of any existing Syslog message,
  2. We can modify the severity level (why is interface down only a notification level??!!)
  3. We can get alerted immediately instead of having to wait for the next SNMP poll. (you are polling your devices aren’t you?!)
  4. We can get notifications for new types of events (ie, default route gone) with customised output. We can even attach the output of a show command in a Syslog message!

 

>>>DOWNLOAD SCRIPT - DEFAULT ROUTE IS LOST<<<

 

Training Courses for SolarWinds Customers

Prosperon Networks are the UK's leading authority on SolarWinds IT Management Solutions. We run training courses that suit a number of roles in your organisation, these courses cater for engineers, helpdesk operators and management personnel who all use monitoring platforms differently. The SolarWinds products retain their simplicity and ease of use, however product training in some form is recommended to get the most out of the tools we use every day.