I Almost Gave Up Using FirewallD on CentOS 9 Until I Figured This Out

I hope this article will help you save some time and energy by avoiding the things that I tried that didn’t work because it took me practically the entire day to get firewalld functioning on my fresh CentOS 9 installation. We shall start with the basics and later dive into the details.

What’s a firewall?

A firewall is a technique used to shield computers from harmful or unauthorized traffic coming from inside or outside networks. A firewall can either be a physical appliance (built on customized vendor hardware) or software defined (can run on COTS hardware). The firewall function enables a user to define a set of rules to regulate incoming network traffic on network hosts. These rules are used to categorize incoming traffic and either allow it through (ACCEPT) or stop it (DROP/REJECT).

This post is about “FirewallD” – a firewall daemon developed by Red Hat which provides a dynamically managed firewall with support for network/firewall zones that define the trust level of network connections or interfaces. It has support for IPv4, IPv6 firewall settings, ethernet bridges and IP sets. It also provides an interface for services or applications to add firewall rules directly.

Even though my deployment scenario was fairly straightforward and uncomplicated, trust me when I say that I almost gave up on FirewallD and resort to the good old iptables (There is technique to mask FirewallD and fallback to iptables if you are more comfortable with the later, but that’s a story for another day.). Just to give you a background of how I ended up here. Am staging a VM to be used for hosting a dockerized application and our development team prefer CentOS Stream 9 as the host OS. If you have been using CentOS, then you know that starting with CentOS 7, FirewallD has replaced iptables as the default firewall management tool.

To briefly describe my scenario, the VM has two interfaces; one for internal access via the LAN, and the second interface connected to the external/public network. The public facing interface is to allow the VM access to the internet for software updates and upgrades and also allow authorized SSH access for the development team sitting in a different geographic location. Just like in the network topology below:

The diagram above is to give you a high level topology of the setup, but what you actually see as a physical firewall is the FirewallD software service running directly on the VM OS (CentOS 9 Stream). The interface ens33 is connected to the internal network (zone: internal in firewalld), whereas interface ens35 is connected to the public internet (zone: public in firewalld). I was able to setup and config the above topology, add the interfaces to the zones, create rich-rules to allow internal traffic, create rich-rules to allow the remote user (public IP). The “rich-rules” is a way to construct firewall rules using the “rich language” syntax. The “rich language” uses keywords with values and is an abstract representation of iptables rules.

What then was the challenge?

The problem was that the SSH port 22 was accessible to everyone on the internet, not only the chosen remote public IP. I used the ping.eu tool to check port 22, and the result was “open!”

I tried everything to limit access to the ssh service to just internal users and the chosen remote IP addresses, but in vain. I’ll briefly go over everything I tried and how I finally figured it out just when I was about to give up. But first, let me go through some crucial commands and parameters for setting up firewalld. Keep in mind that I’m using CentOS 9, and i will be running commands as root user to avoid having to type sudo at every prompt.

Step 1: Starting/stopping firewalld

Step 2: Making sure firewalld starts at boot time

Step 3: Checking the status of the firewalld service

Step 4: Adding the interfaces to the respective zones. According to my topology ens33 is in zone “internal” and ens35 in zone public.

Step 5: Adding the rich-rules to allow traffic from the LAN subnets and from the chosen remote public IP, however there is some unexpected behaviour, as we will see later!

We are now prepared to put our rules to the test and see if they hold true! But first, let me give you a few crucial steps to check the configuration.

Step 6: We are ready to save our running config into the permanent configuration, reload the firewalld service and test our rules.

Step 7: Let’s test the rules again

At this point I connected back to ping.eu to run a port check, I expect that the rules will only allow the selected public IP, however this was not the case. The port was still returning “open”.

Step 8: Let’s tighten the rules.

Using the “target” parameter, you can change the default behavior of a zone. Here are some of the options:

• ACCEPT: Accepts all incoming packets except those disallowed by specific rules.
• REJECT: Rejects all incoming packets except those allowed by specific rules. When firewalld rejects packets, the source machine is informed about the rejection.
• DROP: Drops all incoming packets except those allowed by specific rules. When firewalld drops packets, the source machine is not informed about the packet drop.

It appears that the behavior we are looking for is Reject or Drop since we have rich-rules in that zone allowing traffic from specific a IP. So, let’s go ahead and use Drop.

At this point I thought I had nailed it, setting the target (default behavior) of the public zone to “DROP” and using rich-rules to permit a specific IP address should work! However, the result was still surprising, a port check from ping.eu tool still shows SSH port 22 is “open”!

Step 8: How about adding a “deny-any” at the bottom of the rich-rules to further restrict the rules?

For example, using priorities to arrange the rules and including an explicit deny-any at the bottom of the list (a lower priority value has higher precedence). Will this work? Let’s find out.

Again, i saved and reloaded the configuration but still didn’t work! SSH port 22 was still open to the public!

Step 9: So how exactly did I get this to work? We are about to find out

First off, let’s roll back and remove the reject rule, because clearly it’s not helping.

And now let’s look closely at our public zone settings, and pay more attention

It appears that enabling the “SSH” service in the zone configuration takes precedence over the zone’s default “target: DROP” behavior as well as over rich-rules! Surprised? Now I don’t know if this is by design or if it’s a software bug! but bottom line is that it’s very confusing. As long as the “services: ssh” is allowed in the zone settings, it does not matter how tight your rich-rules are or what you set for “target: DROP/REJECT”, the ssh service will still be open for any source IP trying to access the interfaces in that zone, which for my case was the internet facing interface ens35. So, to make this work, we have to remove the service “ssh” from the zone setting and leave the control to the rich-rules which are set to allow access to port=22 (ssh) from a specific remote public IP.

Step 10: Saving our configurations and testing again

Now let’s run a port check from ping.eu? Port is close and problem solved 😊

The last test is to attempt to connect to the server using the ssh service from a whitelisted IP (an IP that is permitted by the rich-rules). I’m glad mine worked, and I hope yours will too.

Bonus Step: Using GUI to manage the firewall

If have access to a remote web console or your server is connected to a screen and you prefer using a GUI to manage the firewall, you can install the graphical firewall configuration package.

I hope sharing my firewalld experience with you has been beneficial. Please use the Q&A part of our website if you have any questions about this post. I should be able to react right away because I participate in the Q&A forum rather frequently. Wishing you a joyful Linuxing!