Let's dive into lower-level firewall configuration at the OS level and go a little beyond the bare basics.
Even when you are hosting with cloud providers that spoil you with their ease of configuration for firewalls, some special requirements may make it necessary for you to understand the lower-level tools.
I have hit this with several different clients, it comes up when their VMs are running a combination of tools such as Docker, third-party security tooling and even CPanel. It gets especially tricky when these third-party tools don't play nice with each other out of the box and set up inconsistent rules. You can end up having to mediate these tools with custom configuration on your part.
In this article, I will run through a few common scenarios with iptables, one of the most popular tools for configuring the Linux kernel firewall. I will cover basic configuration, connection state, persistence, logging, and touch on interaction with Docker in particular.
Linux Firewalls
Linux is such a diverse ecosystem where you choose your tooling, and this extends to firewall configuration utilities. iptables has been around for a long time and is very common but there are others.
Newer versions of Debian ship with nftables, which is a younger cousin of iptables. It is an updated design but bears a lot of similarities.
Under the hood, both iptables and nftables are built around a programmable framework called netfilter. netfilter allows userspace applications to plug into different parts of the Linux kernel networking stack and to define callbacks. Outside of very niche use cases, it is uncommon to use netfilter directly.
These are the common tools, but you could find others or GUI wrappers around the above.
iptables
Set up iptables
In your favourite distribution, install iptables. For example Debian and Ubuntu:
sudo apt-get update
sudo apt-get install iptables
If you have firewalld installed, for this example please remove or disable it. It is usually not a good idea to mess with iptables directly when you have firewalld running over the top.
sudo systemctl stop firewalld
sudo systemctl disable firewalld
Defaults
Now list the rules.
sudo iptables -L
You should see no restrictions by default, which looks like this:
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
You see in the above there are three default chains:
INPUT– incoming traffic.FORWARD– incoming traffic that is not handled locally, but is passed on to another network (including local Docker networks)OUTPUT– outgoing traffic.
No rules are defined under any of the chains, and the default for all chains is ACCEPT – so any incoming or outgoing packet is accepted.
Example
Let's open port 80 on a host to the world, but open port 8080 only to local network private IP traffic on 10.128.0.0/16.
CAREFUL: Use a non-production host to experiment with OS-level firewalls. They are not as easy to undo as cloud-based firewall rules. If you get the rules wrong, you will lock yourself or your users out. If you do get stuck on your test host with the examples below, reboot the host and the rules will reset, more on that later.
Open web ports
# 1. Accept incoming TCP traffic to destination port 8080 from any source
sudo iptables -A INPUT -p tcp --dport 8080 -j ACCEPT
# 2. Accept TCP traffic to destination port 8080 from the local network CIDR range
sudo iptables -A INPUT -p tcp --dport 8081 -s 10.128.0.0/16 -j ACCEPT
# 3. Accept TCP traffic on port 22 from your your personal IP address
my_ip=# REPLACE THIS WITH YOUR LOCAL IP e.g. 123.45.67.89
sudo iptables -A INPUT -p tcp --dport 22 -s $my_ip/32 -j ACCEPT
Command breakdown
sudo iptables -A INPUT -p tcp --dport 8081 -s 10.128.0.0/16 -j ACCEPT command breakdown:
-A INPUT– add to theINPUTchain (refer to the original table above).-p tcp– the rule applies to the TCP protocol only--dport 8081– the rule applies to the destination port 8081 only-s 10.128.0.0/16– the rule applies to the source IP-j ACCEPT– jump target for the rule.ACCEPTmeans that the connection will be allowed through.
See man iptables (or here) for complete docs.
Resulting iptables
sudo iptables -L -n will that look like this:
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:8080
ACCEPT tcp -- 10.128.0.0/16 0.0.0.0/0 tcp dpt:8081
ACCEPT tcp -- 123.45.67.891 0.0.0.0/0 tcp dpt:22
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Evaluation
An incoming connection will be evaluated against the above rules line by line. Once the first condition is met, the action for that condition will be executed. In our case ACCEPT, which will allow the traffic through. No further rules will be evaluated after the first match.
If no rule matches the traffic, then the default policy for the chain will be used. Which in our example is also ACCEPT. So the above config still means "Accept all traffic".
Reject
Let's reject the traffic that does not match the rules. Again we can append another rule to the end, but this time with a REJECT action.
# 4. REJECT all packets that are not yet matched by other rules
sudo iptables -A INPUT -j REJECT
Note in all the examples above, iptables -A is used, which appends each rule to the end.
So now the last line of the INPUT table will be:
REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable
Test
To try this out, I like to spin up a one-liner python web server, and run this on your remote host:
nohup python3 -m http.server 8080 2>/dev/null &
nohup python3 -m http.server 8081 2>/dev/null &
Then on your local machine see if you can reach the web server. *nix:
my_host=<your remote host IP>
# should succeed:
curl $my_host:8080
# should fail immediately with "Connection refused":
curl $my_host:8081
Remember that 8080 is open to the world, so you should get a response immediately.
If 8080 is not working:
Check that your
iptables -Lmatches the output in the example above – look for the ACCEPT before the blanket REJECT.Check for any other firewalls between you and your remote host. For example, if you're running on AWS, for this example, open up the security group to allow your local IP on all ports.
Problem
There is one issue with the above setup. Go back into your remote host, and try any sort of outbound traffic.
ping google.com
curl https://google.com:443/
Both of the above hang and eventually time out. But what's going on? Your OUTPUT chain is still an ACCEPT-all policy. At a first glance, it might seem that this combination of rules should allow all traffic out of your host.
And it does but it doesn't. Your outbound packets will leave your host and reach the Google web server. But the response back from Google will hit your INPUT chain, not match any of the rules and get REJECTed.
You can try to fix this by whitelisting any IP you expect to contact, but that is usually not feasible, so you can make use of the connection state.
Connection State
Basic iptables rules are stateless. This means that no state about pre-existing connections is taken into consideration. With stateless rules, for traffic to come out of the host and get a response back you need ACCEPT rules in both the INPUT and OUTPUT tables.
The stateless rules use very few resources as it is just a matter of evaluating the rules one by one without any other context. This would also make them very performant.
You can have stateful rules with the conntrack matcher. When conntrack is used, each connection is tracked in memory and can be looked up during rule evaluation.
Fixing outbound traffic
To allow a response to come back for any outbound connection that you make, add this rule:
sudo iptables -I INPUT 1 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
This is saying:
-I INPUT 1Insert a rule at index 1 – we don't want this rule to go to the end, it needs to come before the explicitREJECT.-m conntrackUse theconntrackmatcher.--ctstate ESTABLISHED,RELATEDMatch connections that have a state ofESTABLISHEDorRELATED– existing connections only (i.e. created after an acceptedOUTPUTmessage)-j ACCEPTAccept them all.
Now you can check the rules:
sudo iptables -L -n
And see the output like this:
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:8080
ACCEPT tcp -- 10.128.0.0/16 0.0.0.0/0 tcp dpt:8081
ACCEPT tcp -- 123.45.67.89 0.0.0.0/0 tcp dpt:22
REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Try Google again and you will get a response:
ping google.com
curl https://google.com:443/
REJECT vs DROP
You can see in the examples above, the REJECT line ends with reject-with icmp-port-unreachable. Any connection that reaches this rule will immediately get an icmp-port-unreachable response from your server, killing the connection.
An alternative action is DROP. In that case, your server will silently drop the connection without a response – the client will wait until it times out.
REJECT provides faster feedback. But at other times DROP may be preferable. If someone is not where they should be – do you need to bother to send them a response at all?
Persistence
iptables out of the box does not have persistence built in. Your rules will stay in place until a reboot, but once you reboot your machine the rules will be wiped.
Persistence is managed outside of iptables themselves using external utilities.
iptables-persistent
For the simple setup above, you could make your rules persistent by installing this package (Debian/Ubuntu):
sudo apt-get install iptables-persistent
This will:
give you two new tools:
iptables-saveandiptables-restore.Save your current config to a file such as
/etc/iptables/rules.v4Add an
init.dscript/etc/init.d/iptables-persistent(or/etc/init.d/netfilter-persistent) to restore the saved rules on boot.
iptables-persistent – Manual operations
You can trigger them manually these by calling:
sudo iptables-save >/etc/iptables/rules.v4
sudo iptables-restore </etc/iptables/rules.v4
This creates a human-readable and editable file that you can tweak if needed.
But do note, that iptables-save and iptables-restore are very crude tools for basic use cases. Changes to rules are not automatically reflected in the persisted rules. You will need to manage the lifecycle yourself, by calling sudo iptables-save >/etc/iptables/rules.v4 after each subsequent change.
Logging
Logging for iptables is not enabled out of the box. If you do need to enable logging for debugging or auditing purposes, make sure you consider how much you want to log. Here is one technique that you can try:
Add a new Chain called
LOGGINGJump to this chain from wherever you want this log to come. For example to log all INPUT, add at an index 1 (
-I INPUT 1). Or if you want to log only under certain conditions insert it at a more appropriate place in the ordered list of rules.Log the message. If you're enabling this in prod. It is recommended that you do some throttling here, you don't want a DoS to flood your log and hence your disk space.
RETURNout of theLOGGINGtable
This can be accomplished using the example below. It will issue a log for all incoming traffic other than the traffic that matches the first rule (port 8080 in our previous example).
# 1.
sudo iptables -N LOGGING
# 2.
sudo iptables -I INPUT 2 -j LOGGING
# 3.
# -m limit --limit 10/s will throttle at 10 messages per second. Good for debugging purposes.
sudo iptables -A LOGGING -m limit --limit 5/s -j LOG --log-prefix "iptables-input: " --log-level info
# 4.
sudo iptables -A LOGGING -j RETURN
The logs will go to your default system log (e.g. /var/log/messages on Debian). You can configure this behaviour using your logging utility such as rsyslogd.
CAUTION because logging connections on a real production server can fill up your disk space quite fast, look into limiting what you log and into log rotation (logrotate for your system logs).
Clean up
If you are keeping the above host around, you might want to shut off the test web servers and clear the iptables.
# One-liner to kill processes with the words 'http.server'
ps x | grep http.server | awk '{print $1}' | head -n -1 | xargs kill
If you want to clean up iptables, you can try the -D <chain> <position> syntax to delete rules one by one (e.g. iptables -D INPUT 1 to delete first rule in INPUT). Or reboot the machine (if you haven't set up a restore on boot using iptables-restore).
Brief word on Docker
The Docker daemon has a built-in limited understanding of iptables (and the higher level wrapper around it that you may be using –firewalld). The daemon sets the networking up somewhat differently for host and bridge types of Docker networking.
By default, when you bind to a host port, the Docker daemon will add rules to your iptables to allow all incoming traffic for any ports that you listen to for host networking. This is where you need to be careful with your iptables setup and its persistence.
You can read more on what Docker does in their documentation on "Docker and iptables". Play around with it and see what it does. You will see that it creates new chains that get jumped to from the standard FORWARD chain.
Persistence with Docker
If you're going to modify the rules and you want to make use of persistence, make sure you set the JSON value "tables": false in /etc/docker/daemon.json and restart the Docker daemon. Otherwise, the daemon can override or conflict with the changes that you make in iptables yourself.
IPv6
This article only touched on IPv4. iptables does not apply to IPv6 traffic, but its companion tool ip6tables with the same CLI interface will let you define IPv6. Similarly, you can use ip6tables-restore and ip6tables-save.