Table of contents
Let's dive into lower-level firewall configuration at the OS level and go a little beyond the bare basics.
Even when you are hosting with cloud providers that spoil you with their ease of configuration for firewalls, some special requirements may make it necessary for you to understand the lower-level tools.
I have hit this with several different clients, it comes up when their VMs are running a combination of tools such as Docker, third-party security tooling and even CPanel. It gets especially tricky when these third-party tools don't play nice with each other out of the box and set up inconsistent rules. You can end up having to mediate these tools with custom configuration on your part.
In this article, I will run through a few common scenarios with iptables
, one of the most popular tools for configuring the Linux kernel firewall. I will cover basic configuration, connection state, persistence, logging, and touch on interaction with Docker in particular.
Linux Firewalls
Linux is such a diverse ecosystem where you choose your tooling, and this extends to firewall configuration utilities. iptables
has been around for a long time and is very common but there are others.
Newer versions of Debian ship with nftables
, which is a younger cousin of iptables
. It is an updated design but bears a lot of similarities.
Under the hood, both iptables
and nftables
are built around a programmable framework called netfilter
. netfilter
allows userspace applications to plug into different parts of the Linux kernel networking stack and to define callbacks. Outside of very niche use cases, it is uncommon to use netfilter
directly.
These are the common tools, but you could find others or GUI wrappers around the above.
iptables
Set up iptables
In your favourite distribution, install iptables
. For example Debian and Ubuntu:
sudo apt-get update
sudo apt-get install iptables
If you have firewalld
installed, for this example please remove or disable it. It is usually not a good idea to mess with iptables
directly when you have firewalld
running over the top.
sudo systemctl stop firewalld
sudo systemctl disable firewalld
Defaults
Now list the rules.
sudo iptables -L
You should see no restrictions by default, which looks like this:
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
You see in the above there are three default chains:
INPUT
– incoming traffic.FORWARD
– incoming traffic that is not handled locally, but is passed on to another network (including local Docker networks)OUTPUT
– outgoing traffic.
No rules are defined under any of the chains, and the default for all chains is ACCEPT
– so any incoming or outgoing packet is accepted.
Example
Let's open port 80 on a host to the world, but open port 8080 only to local network private IP traffic on 10.128.0.0/16
.
CAREFUL: Use a non-production host to experiment with OS-level firewalls. They are not as easy to undo as cloud-based firewall rules. If you get the rules wrong, you will lock yourself or your users out. If you do get stuck on your test host with the examples below, reboot the host and the rules will reset, more on that later.
Open web ports
# 1. Accept incoming TCP traffic to destination port 8080 from any source
sudo iptables -A INPUT -p tcp --dport 8080 -j ACCEPT
# 2. Accept TCP traffic to destination port 8080 from the local network CIDR range
sudo iptables -A INPUT -p tcp --dport 8081 -s 10.128.0.0/16 -j ACCEPT
# 3. Accept TCP traffic on port 22 from your your personal IP address
my_ip=# REPLACE THIS WITH YOUR LOCAL IP e.g. 123.45.67.89
sudo iptables -A INPUT -p tcp --dport 22 -s $my_ip/32 -j ACCEPT
Command breakdown
sudo iptables -A INPUT -p tcp --dport 8081 -s 10.128.0.0/16 -j ACCEPT
command breakdown:
-A INPUT
– add to theINPUT
chain (refer to the original table above).-p tcp
– the rule applies to the TCP protocol only--dport 8081
– the rule applies to the destination port 8081 only-s 10.128.0.0/16
– the rule applies to the source IP-j ACCEPT
– jump target for the rule.ACCEPT
means that the connection will be allowed through.
See man iptables
(or here) for complete docs.
Resulting iptables
sudo iptables -L -n
will that look like this:
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:8080
ACCEPT tcp -- 10.128.0.0/16 0.0.0.0/0 tcp dpt:8081
ACCEPT tcp -- 123.45.67.891 0.0.0.0/0 tcp dpt:22
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Evaluation
An incoming connection will be evaluated against the above rules line by line. Once the first condition is met, the action for that condition will be executed. In our case ACCEPT
, which will allow the traffic through. No further rules will be evaluated after the first match.
If no rule matches the traffic, then the default policy for the chain will be used. Which in our example is also ACCEPT
. So the above config still means "Accept all traffic".
Reject
Let's reject the traffic that does not match the rules. Again we can append another rule to the end, but this time with a REJECT
action.
# 4. REJECT all packets that are not yet matched by other rules
sudo iptables -A INPUT -j REJECT
Note in all the examples above, iptables -A
is used, which appends each rule to the end.
So now the last line of the INPUT
table will be:
REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable
Test
To try this out, I like to spin up a one-liner python web server, and run this on your remote host:
nohup python3 -m http.server 8080 2>/dev/null &
nohup python3 -m http.server 8081 2>/dev/null &
Then on your local machine see if you can reach the web server. *nix:
my_host=<your remote host IP>
# should succeed:
curl $my_host:8080
# should fail immediately with "Connection refused":
curl $my_host:8081
Remember that 8080 is open to the world, so you should get a response immediately.
If 8080 is not working:
Check that your
iptables -L
matches the output in the example above – look for the ACCEPT before the blanket REJECT.Check for any other firewalls between you and your remote host. For example, if you're running on AWS, for this example, open up the security group to allow your local IP on all ports.
Problem
There is one issue with the above setup. Go back into your remote host, and try any sort of outbound traffic.
ping google.com
curl https://google.com:443/
Both of the above hang and eventually time out. But what's going on? Your OUTPUT
chain is still an ACCEPT
-all policy. At a first glance, it might seem that this combination of rules should allow all traffic out of your host.
And it does but it doesn't. Your outbound packets will leave your host and reach the Google web server. But the response back from Google will hit your INPUT
chain, not match any of the rules and get REJECT
ed.
You can try to fix this by whitelisting any IP you expect to contact, but that is usually not feasible, so you can make use of the connection state.
Connection State
Basic iptables
rules are stateless. This means that no state about pre-existing connections is taken into consideration. With stateless rules, for traffic to come out of the host and get a response back you need ACCEPT
rules in both the INPUT
and OUTPUT
tables.
The stateless rules use very few resources as it is just a matter of evaluating the rules one by one without any other context. This would also make them very performant.
You can have stateful rules with the conntrack
matcher. When conntrack
is used, each connection is tracked in memory and can be looked up during rule evaluation.
Fixing outbound traffic
To allow a response to come back for any outbound connection that you make, add this rule:
sudo iptables -I INPUT 1 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
This is saying:
-I INPUT 1
Insert a rule at index 1 – we don't want this rule to go to the end, it needs to come before the explicitREJECT
.-m conntrack
Use theconntrack
matcher.--ctstate ESTABLISHED,RELATED
Match connections that have a state ofESTABLISHED
orRELATED
– existing connections only (i.e. created after an acceptedOUTPUT
message)-j ACCEPT
Accept them all.
Now you can check the rules:
sudo iptables -L -n
And see the output like this:
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:8080
ACCEPT tcp -- 10.128.0.0/16 0.0.0.0/0 tcp dpt:8081
ACCEPT tcp -- 123.45.67.89 0.0.0.0/0 tcp dpt:22
REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Try Google again and you will get a response:
ping google.com
curl https://google.com:443/
REJECT vs DROP
You can see in the examples above, the REJECT
line ends with reject-with icmp-port-unreachable
. Any connection that reaches this rule will immediately get an icmp-port-unreachable
response from your server, killing the connection.
An alternative action is DROP
. In that case, your server will silently drop the connection without a response – the client will wait until it times out.
REJECT
provides faster feedback. But at other times DROP
may be preferable. If someone is not where they should be – do you need to bother to send them a response at all?
Persistence
iptables
out of the box does not have persistence built in. Your rules will stay in place until a reboot, but once you reboot your machine the rules will be wiped.
Persistence is managed outside of iptables
themselves using external utilities.
iptables-persistent
For the simple setup above, you could make your rules persistent by installing this package (Debian/Ubuntu):
sudo apt-get install iptables-persistent
This will:
give you two new tools:
iptables-save
andiptables-restore
.Save your current config to a file such as
/etc/iptables/rules.v4
Add an
init.d
script/etc/init.d/iptables-persistent
(or/etc/init.d/netfilter-persistent
) to restore the saved rules on boot.
iptables-persistent – Manual operations
You can trigger them manually these by calling:
sudo iptables-save >/etc/iptables/rules.v4
sudo iptables-restore </etc/iptables/rules.v4
This creates a human-readable and editable file that you can tweak if needed.
But do note, that iptables-save
and iptables-restore
are very crude tools for basic use cases. Changes to rules are not automatically reflected in the persisted rules. You will need to manage the lifecycle yourself, by calling sudo iptables-save >/etc/iptables/rules.v4
after each subsequent change.
Logging
Logging for iptables
is not enabled out of the box. If you do need to enable logging for debugging or auditing purposes, make sure you consider how much you want to log. Here is one technique that you can try:
Add a new Chain called
LOGGING
Jump to this chain from wherever you want this log to come. For example to log all INPUT, add at an index 1 (
-I INPUT 1
). Or if you want to log only under certain conditions insert it at a more appropriate place in the ordered list of rules.Log the message. If you're enabling this in prod. It is recommended that you do some throttling here, you don't want a DoS to flood your log and hence your disk space.
RETURN
out of theLOGGING
table
This can be accomplished using the example below. It will issue a log for all incoming traffic other than the traffic that matches the first rule (port 8080
in our previous example).
# 1.
sudo iptables -N LOGGING
# 2.
sudo iptables -I INPUT 2 -j LOGGING
# 3.
# -m limit --limit 10/s will throttle at 10 messages per second. Good for debugging purposes.
sudo iptables -A LOGGING -m limit --limit 5/s -j LOG --log-prefix "iptables-input: " --log-level info
# 4.
sudo iptables -A LOGGING -j RETURN
The logs will go to your default system log (e.g. /var/log/messages
on Debian). You can configure this behaviour using your logging utility such as rsyslogd
.
CAUTION because logging connections on a real production server can fill up your disk space quite fast, look into limiting what you log and into log rotation (logrotate
for your system logs).
Clean up
If you are keeping the above host around, you might want to shut off the test web servers and clear the iptables
.
# One-liner to kill processes with the words 'http.server'
ps x | grep http.server | awk '{print $1}' | head -n -1 | xargs kill
If you want to clean up iptables
, you can try the -D <chain> <position>
syntax to delete rules one by one (e.g. iptables -D INPUT 1
to delete first rule in INPUT
). Or reboot the machine (if you haven't set up a restore on boot using iptables-restore
).
Brief word on Docker
The Docker daemon has a built-in limited understanding of iptables
(and the higher level wrapper around it that you may be using –firewalld
). The daemon sets the networking up somewhat differently for host and bridge types of Docker networking.
By default, when you bind to a host port, the Docker daemon will add rules to your iptables
to allow all incoming traffic for any ports that you listen to for host networking. This is where you need to be careful with your iptables
setup and its persistence.
You can read more on what Docker does in their documentation on "Docker and iptables". Play around with it and see what it does. You will see that it creates new chains that get jumped to from the standard FORWARD
chain.
Persistence with Docker
If you're going to modify the rules and you want to make use of persistence, make sure you set the JSON value "tables": false
in /etc/docker/daemon.json
and restart the Docker daemon. Otherwise, the daemon can override or conflict with the changes that you make in iptables
yourself.
IPv6
This article only touched on IPv4. iptables
does not apply to IPv6 traffic, but its companion tool ip6tables
with the same CLI interface will let you define IPv6. Similarly, you can use ip6tables-restore
and ip6tables-save
.