Just Fixed the Issue of Multiple Default Route Entries in Linux CentOS Stream 9

This one is brief! Just completed setting up a brand-new virtual machine with CentOS Stream 9. Everything went smoothly till the networking phase. Despite the fact that this article was prepared with reference to CentOS 9, I would like to think that the same remedy should work for any other Linux distribution if this problem arises.

In the beginning, I configured the internal interface such that I could access the CLI from my LAN network. When I implemented my internet-facing interface, the issue emerged. I was unable to access my virtual machine from the internet (outside network) nor was i able to ping the internet from the VM.

Since CentOS utilizes firewallD by default, the firewall was the first thing I suspected and instantly disabled it, but the problem persisted. I then started looking into networking issues.

When I printed the routing table using “ip route” command, I quickly realized one thing which was off! The routing table had two default route entries as seen in the screen shot below:

What was also quick to notice is the metrics of the two default routes. A metric is a value that represents the distance or cost of reaching a destination network through a certain route. Static routing uses metrics to determine the best path when there are multiple routes to the same destination. The lower the metric, the higher the priority of the route. If you have multiple static routes to the same destination, the one with the lowest metric will be preferred. The first entry had metric 100, while the second entry had metric 101. Now this explains why I was able to access the server from with my LAN, but not from the outside network (internet).

My first thought of a quick fix was to delete the first default route entry, since as you can see I already had other static route entries that take care of my access within my LAN. Ideally I should have one default route to the internet and static entries for the LAN, because LAN subnets are more specific. So I went ahead and deleted the route:

# ip route del default via 172.26.16.129 dev ens33

Now the challenge with issuing a “route del” at the CLI, is that this state is not persistent, it will work in runtime, but the route will re-surface when the network service is restarted or server reboots. I went ahead and rebooted the server and for sure the second default route re-appeared! I also used the “nmcli” which is the network manager utility on CentOS 9 to check the networking on the server, and you can see both interfaces have default routes.

See below for “ens33” which is my LAN interface, has a default route with metric 100:

And see below for “ens35” which is my internet facing interface, has a default route with metric 101:

So the two default routes is what is causing my trouble right now, and this is a self inflicted trouble as I will explain later. There are two ways to resolve this problem but first, let me show you what I believe was a self inflicted trouble. At first, I configured my network interfaces using the CentOS GUI via the web console as seen below:

At the IPv4 settings, I assigned the IP address, mask and gateway. Assigning a Gateway here automatically creates a default route entry in the routing table. This wouldn’t be a problem, but adding the second interface and configuring it with a gateway caused a problem!

And this was my recipe for trouble. This automatically created the second default route. The metric for these automatically generated default routes is also automatically assigned. The interface that comes up first gets the lowest metric. And if you toggle (on/off) and interface which had a lower metric comes back up with a higher metric and makes the whole setup unpredictable. There are two ways around this:

A) You can choose to maintain the two default routes, but in this case you will have to play around with the metrics. But like I mentioned, the metrics change when you toggle the interface, but I believe there is a way to make them persistent via the CLI or some script file that am yet to discover.

B) Remove the “Gateway” entry from one of the interfaces and this is what I recommend you do. The Gateway definition should be done on ONLY one interface which is going to serve as the default gateway. Like I mentioned earlier, it makes a lot of sense to make the WAN interface your default gateway, and then add static routes for the LAN interface because the LAN networks are more specific. That is they can only fall within these subnets:

1. 10.0.0.0/8  (10.0.0.0 to 10.255.255.255)
2. 172.16.0.0/12 (172.16.0.0 to 172.31.255.255)
3. 192.168.0.0/16 (192.168.0.0 to 192.168.255.255)

Once you have done this, you need to toggle the interfaces (up/down), this will refresh the routing table and you should see ONLY one default route.  You can toggle the interfaces from the GUI if you have access to the web console:

Or you can use CLI and run the following command to restart the network service:

Alternatively, you can use “nmcli” as below:

You can also use nmcli to modify, reload, or activate specific network connections or devices. Check your routing table again, and you should have one default route entry and other static route entries that you enter manually.

This resolves the multiple-default route entry problem, and your server should be able to connect to both the LAN and the internet without issues.

Important to Note: Protect Your Server from Internet Attacks

If you intend to expose one of the interfaces to the internet, don’t forget to secure your server with a firewall against attacks (like DDoS). Here are some examples to help you quickly set up your firewall on CentOS 9, which by default utilizes firewallD.

Assign your interfaces to the correct zones. I recommend you put the LAN interface in zone “internal” and the WAN interface in zone “public”:

Whitelist the LAN subnets (trusted/private networks) to be able to access your server.

And if you have a requirement to allow an outside IP to access your server, this is how you would do it:

And this is how to save runtime configuration to be permanent, but before saving your configuration, you need to verify it.

And if case you have to remove/delete a firewall rule, this is how you would do it:

Reloading the firewall service for new changes to take effect:

I sincerely hope this article was useful. It began as a brief post to address the multiple-default-route issue, but it quickly expanded into important server networking and security settings. I’m hoping to have more time soon to publish in-depth articles regarding Linux security and firewalls. Use this website’s Q&A forum to ask any queries you may have about this content. Am highly active on this forum and ought to have speedy responses. And if you are an expert, kindly contribute to our question-and-answer forum by responding to some technical questions. Happy linuxing! 😊