IT Defrag

Now, in part 2, let’s talk about those 3 challenges and what we can do to resolve them.

Issue Number 1

Let’s talk about high availability first. Typically, high availability on firewalls has been accomplished by installing two firewalls, joining them together and configuring one to be the primary and the other to be a hot standby. This has worked well, but if our firewall is now supposed to be our core router, two issues arise from this setup (in some sense they are the same issue, but I’ll break it out as two issues.)

The main problem is that doing HA in this way creates a single point of failure. You might ask: “well wait, if there are two units HA’d together, how is that a single point of failure?“ This is basically the same single point of failure when doing switch stacking or VSS that is, both firewalls share the same configuration and to some extent the same control plane (okay, they don’t really share the same control plane, but in my opinion the way the HA functions, I’d argue it kind of does).

The second part of the issue is since only one firewall is active, if there is an opposite switch failure downstream, loss of connectivity may occur if the firewall does not also failover.

The easy fix for these issues is to configure active/active high availability. Configuring the HA this allows both firewalls to do this independent of the other, thus greatly reducing the risk of the first item. It also allows for traffic to flow through either, which completely eliminate the second issue. There still is the risk of one firewall crashing both. The best solution would be to have both firewalls completely separated from a management and configuration standpoint, but still be able to push state data back and forth between them. That, however, would need to be implemented by firewall vendors and is unlikely to be implemented.

Issue Number 2

The second issue we can talk about is throughput. The process stack for features of a next-generation firewall is very CPU intensive. This has required high performance CPUs in order to accomplish the amount of packets per second throughput needed to saturate 10G, 25G, 40G, or even 100G. This powerful CPU requirement has always driven up cost on the hardware side of the firewall, so enterprises have always done studies to ensure they purchase the proper sized firewall. This study was typically just for your internet edge firewall or just your firewall enclave, which may not need to do high throughput. With the firewall becoming your core router, lateral throughput needs to be considered in this study. One thing to note, some engineers may take the HA solution I talked about above and go, well if both firewalls are active, I’ve doubled my throughput! This is a folly to say. What if a firewall crashes or you reboot one for an upgrade? You’ve now halved you throughput and you may not have enough bandwidth available. Each firewall needs to be able to provide full throughput for the entire network.

The solution is, and this might be hard, is just to get more money so that you buy bigger firewalls. A part of this solution also does land on firewall vendors as they should be trying to optimize their code so they can do more pps and throughput and same hardware, but in some cases there are just hard limits.

Issue Number 3

This last issue is going to eat up most of your time as a network engineer. The issue is figuring out proper ACLs for networks that never had ACLs. I’d suggest not doing things such as:

permit ip any any

I’d avoid this because if you’re going to get the money for firewalls, let’s at least also do the effort to go through and figure out what endpoints should have access to so that we can start to place boundary’s. There are tools and software that can help you generate PPS (ports, protocols and services) documentations so that you can easily generate ACLs. If needed, you can do the any any, but set target dates where you’re going to finish all your rules and then remove the any any. Be prepared for weeks of end users coming to you and asking for more access.

So that’s all I have. I really do feel like as network engineers we need to move to place firewalls as our core routers to help protect our network. There are issues that come with it, but I feel like they can be overcome and that the benefits outweigh the risks that come with it.

In the enterprise networking world the firewall has always been tool to allow the engineer to securely provide access to the internet or to segment off a section of the network. It has always been something that has been attached on the side, away from the core network infrastructure. It’s not to say it’s not core infrastructure, it definitely is, but it’s not center of network activity. The mantra has always been, let routers route, switches switch and firewalls firewall. It also been understandable why you would not want something like a firewall handling your BGP peering at your edge, it just doesn’t have the feature set nor the true capability of doing that… Well, at least old firewalls.

Enter the modern networking world where cyber security is quickly becoming the number one priority for enterprise networks. The cyber security threats can not only come from the internet, but even from within the enterprise network itself. A sobering example of this is the 2020 CVE, CVE-2020-10148, where the network monitoring software, Solarwinds Orion, was compromised. Malicious actors were able to gain access into the code base for Solarwinds Orion allowing them to insert highly sophisticated code that would detect what environment it was in, attempt to gather information, and then call home. This compromised software was then downloaded, unknowingly, by Solarwinds Orion customers and deployed in their network. This gave the malicious actors direct and open access to core networking infrastructure and key servers since the Server Solarwinds Orion was deployed on, needed to have access to the entire environment it was intended to monitor.

The Solarwinds attack highlights a key vulnerability in the historical network mantra, once the attack goes beyond or begins after your firewall, it will more or less have complete and open access to everything inside your network, or in other words, complete lateral movement in the network. As network engineers we need to move to secure this lateral movement within the network. This can only effectively be done by promoting the firewall from its sideline duties of firewalling internet access or firewalling small segmentation zones to being the core router in the network. No longer should we be using routers or layer 3 switches as our core routers. All core layer 3 functions should be handled by firewalls, no exception.

Promoting the firewall to your core router presents some challenges, namely:

1. Firewalls have historically had poor network throughput placing a cap on total throughput thru it
2. Traditional high availability firewall configurations don’t suite well as the firewall being the core router
3. It will be difficult to build secure and proper firewall rules since services on the network are most likely not known so engineers will not want to risk disrupting the network due to firewall blocking traffic flows (e.g. putting a permit ip any any on the bottom which defeats the entire purpose of the firewall.)

In my next post I’ll talk about the details associated with these 3 points and how we can overcome them.