<![CDATA[Konst's Blog – Site Reliability Engineering and Technology]]>https://blog.konst.kiwiRSS for NodeSat, 18 Apr 2026 13:35:51 GMT60<![CDATA[Neovim with the TypeScript language server (LSP)]]>https://blog.konst.kiwi/neovim-with-the-typescript-language-server-lsphttps://blog.konst.kiwi/neovim-with-the-typescript-language-server-lspTue, 14 Oct 2025 09:45:16 GMTI’m trying a few new things.

I’ve got a project that I’ve been experimenting with and I plan to publish as open source at some point. For this project I decided to revisit my old friend TypeScript. It’s a language I’ve used in the past, but I haven’t touched it professionally.

I’ve also been using Neovim more and more as my IDE instead of my old default Visual Studio Code. Especially for the smaller hobby projects. Switching it up a bit.

If you’re unfamiliar, definitely check it out, it is Vim but easier to customise and extend with Lua scripting. One big benefit is the variety of third party plugins you can find for it. Someone will burn me for this comparison, but Neovim reminds me of Emacs and its extensibility with Elisp, which is something I dabbled with earlier on in my career.

One thing that has been tricky though is that, while popular, Neovim is fairly new and at the time of writing is on v0.11.x. Some things are changing quickly and some things don’t have a lot of examples online.

Today I figured out TypeScript LSP and I thought I’d share it.

Dependencies

You may be running npm that pulls in some dependencies for you. For example I have versioned and local typescript per package.json that I pull in. But because nvim is global, you should set up some global dependencies.

You’ll need to install npm install -g typescript typescript-language-server

The former you already know, the latter comes from https://github.com/typescript-language-server/typescript-language-server and is a standalone binary.

If you haven’t already, installing nvim I will leave it as an exercise for the reader. I also alias vi to nvim because I have a few decades of habit of typing vi to open files.

init.lua

nvim is configured by init.lua, which is not there by default if you haven’t already started configuring it.

Make sure ~/.config/nvim/ exists, create it if not.

nvim ~/.config/nvim/init.lua

And this is where all the docs I’ve found get out of date. From nvim v0.11.x, the LSP config is:

" Enable TypeScript via the Language Server Protocol (LSP)
vim.lsp.enable('tsserver')

" Set the TS config for the LSP
vim.lsp.config('tsserver', {
  " Make sure this is on your path
  cmd = {'typescript-language-server', '--stdio'},
  filetypes = { 'typescript' },
  " This is a hint to tell nvim to find your project root from a file within the tree
  root_dir = vim.fs.root(0, {'package.json', '.git'}),
  on_attach = on_attach,
  capabilities = capabilities,
  " optional settings = {...} go here, refer to language server code: https://github.com/typescript-language-server/typescript-language-server/blob/5c483349b7b4b6f79d523f8f4d854cbc5cec7ecd/src/ts-protocol.ts#L379
})

Then you can set some key bindings, this is what I’m starting with, I’ll adapt and add some more later. Note that these are not specific to the TypeScript LSP, you can find more examples online for LSP keybindings:

-- lsp keymaps
vim.keymap.set('n', '<leader>rn', vim.lsp.buf.rename, {})
vim.keymap.set('n', 'gd', vim.lsp.buf.definition, {})
vim.keymap.set('n', '<leader>f', function()
  vim.lsp.buf.format { async = true }
end, {})

The ’n’ above tells it to enable these keybindings in normal vim mode (not insert mode). And <leader> can be configured, defaults to \

Testing

Now open up a typescript file.

Keep in mind that as per the config above, we need a package.json or .git in your tree for nvim to see your full project. But it will work on any random .ts file in isolation.

My example:

nvim /tmp/test.ts

To check if LSP is configured successfully, run this command in nvim:

:checkhealth vim.lsp

And you should see the diagnostic like so. In particular look for Active Clients (what is configured for your current buffer) and Enabled Configuration (global available LSP settings):

vim.lsp:                                                                    ✅

- LSP log level : WARN
- Log path: /Users/kt/.local/state/nvim/lsp.log
- Log size: 11 KB

vim.lsp: Active Clients ~
- tsserver (id: 1)
  - Version: ? (no serverInfo.version response)
  - Root directory: nil
  - Command: { "typescript-language-server", "--stdio" }
  - Settings: {}
  - Attached buffers: 1

vim.lsp: Enabled Configurations ~
- tsserver:
  - cmd: { "typescript-language-server", "--stdio" }
  - filetypes: typescript


vim.lsp: File Watcher ~
- file watching "(workspace/didChangeWatchedFiles)" disabled on all clients

vim.lsp: Position Encodings ~
- No buffers contain mixed position encodings

Then go back to the typescript file and type

function testyTest(a: number) {
}

function testyTestEmpty() {
}

tes

And from here trigger your code completion. Default keybinding is C-n. And this is what I see, code completion comes up, and you can make a selection with TAB:

Image showing neovim with the above code snippet, tes, followed by a drop down offering the options testyTest or testyTestEmpty

Diagnostic hints and errors

After you’ve accepted that, you can try something like this:

function testyTest(a: number) {
}

function testyTestEmpty() {
}

testyTest()

Once you exit insert mode, you will see these marks:

image showing the above code snippet with an H next to the function defintion of testyTest and an E next to the invocation of testyTest

This is Neovim’s diagnostic marks - the H indicates a Hint, E indicates Error. The default keybindings are ]d / [d to jump backwards and forwards between them in the buffer, and <C-w>d to shows diagnostic at cursor like so:

image showing details of the diagnostic error on the testyTest() function invocation

See the docs for full diagnostic mark keybindings and of course you can remap them.

Going further

I’ve only just scratched the surface here. I found that there wasn’t a particular single guide to string everything together. So I wrote up what I found.

Check out all the other features available in LSP. Look up general guides on Neovim LSP, they don’t need to be TypeScript specific.

Also look at the excellent Neovim docs, either through their website or through :help lsp / :help diagnostics in-app, these are the parts I referenced here:

]]><![CDATA[Four for Feb]]>https://blog.konst.kiwi/four-feb24-linkshttps://blog.konst.kiwi/four-feb24-linksMon, 05 Feb 2024 08:49:44 GMTI remember the days before the Internet was for everyone. It was a nerdy thing that you'd disconnect your parents' phone line for. Social media, endless scroll feeds, and hourly news updates were not a thing. Twitter was not there, and hate groups were hidden in their own little fringe communities rather than being algorithmically rubbed into your face. You couldn't put the Internet in your pocket to take with you everywhere.

A few weird people, myself included, had personal blogs, and you'd do Trackbacks to your friend's blogs or to other Internet weirdos with similar interests. I didn't even think of putting in a hit counter or analytics.

A little before blogging, Geocities! Who needs a reminder? Who doesn't miss those colours, gradients and WordArt GIF banners?

Facebook started appearing later in this phase for me, for uni students only; I logged in once, sent a poke, and left it for several years. IRC, MSN or ICQ is where you'd truly talk to people. The choice of messaging system alone would tell you a bit about a person.

No, perhaps it wasn't all great and wonderful as I described. There is a bit of omission there. But those are some of the bits I remember fondly.

In that spirit, I recently read a few things that reminded me of those old days of wonder and hope. About relatively low tech, about blogging with interesting ideas – over viral content and likes or posturing.

  • I love Kelly Shortridge's blog, and especially her post on Amonia Plant Safety. Say whaat? Just read it if you're in tech, or not. I quoted it in a tech conference talk on incidents last year.
  • My System (1978) (alt text version here) - a short write-up by a man called Michael Holley that gives a glimpse into how you'd put a personal computer together in the 70s. I love the enthusiasm, and the pride in that photo.
  • A post about local exploration A single small map is enough for a lifetime. Coincidentally, I have been doing a lot more of this (not in the same manner as the author!) since going full remote at the end of last year, and it has brought me a lot of fascination for the things I've been missing within short biking distance from my house.
  • Saving the best for last. I really enjoyed Where Have All the Websites Gone by Jason. It gave me the idea for this post. Check out Jason's other content, instant follow.
]]><![CDATA[Rate limiting with iptables]]>https://blog.konst.kiwi/rate-limiting-with-iptableshttps://blog.konst.kiwi/rate-limiting-with-iptablesSat, 20 May 2023 10:07:42 GMTWhen running any site on the internet, it will be scanned regularly by various scripts. This is true for everything I've hosted, from a cat charity blog to financial software sites. It's not personal.

Rate limiting is a rudimentary line of defence against scanners, brute force attempts, SYN floods, and basic Denial of Service (DoS) attacks. Do not put all your faith into rate limiting, but it is a useful tool to whack on in most circumstances.

When to use iptables

There are lots of different places where you can rate limit. I've often run into a scenario where I need to run a single-instance site without putting extra infrastructure around it. There could be cost or other restrictions in play.

It could even be a third-party app you're running that you cannot easily modify. This is where knowledge of iptables is beneficial.

Background on iptables

I have done an introduction to iptables and connection tracking in an earlier post, and this post will assume a basic understanding of what iptables are.

Target state

I will use Docker for this example, as Docker port forwarding creates a few interesting steps. But you can use any web service with iptables.

If you're not on Docker, you will use the INPUT chain instead of FORWARD. The rest of the principles are the same.

Docker's default bridge networking is an extra layer inside your host. The outside traffic is forwarded to the internal Docker network. When the traffic wants to get out of Docker, it is forwarded out of this network.

What we want to get to in terms of iptables is illustrated in this diagram below:

Docker's default bridge networking will configure the FORWARD and DOCKER-USER chains for us. Then we will plug our rate limit solution into that.

FORWARD chain

The FORWARD chain is when your host works as a proxy, forwarding the traffic it has received to somewhere else. In the context of Docker, this is when the traffic is forwarded to the internal bridge network.

DOCKER-USER chain

The Docker daemon configures this as the first rule in the FORWARD chain. By default, it is empty. When you add your own rules into this chain, they get evaluated before traffic to the Docker network.

Set up

1. Create a RATE-LIMIT chain

Follow the basic set up steps for iptables if you don't already have them configured.

Add new chains called RATE-LIMIT and LOGGING (or any other name you think is most appropriate):

sudo iptables --new-chain RATE-LIMIT
sudo iptables --new-chain LOGGING
sudo iptables -L -n # view all the rules

This should give you a list of rules ending with the new empty chain:

Chain LOGGING (0 references)
target     prot opt source               destination

Chain RATE-LIMIT (0 references)
target     prot opt source               destination

2. Docker

If you don't already have Docker setup, follow the official instructions. Also, look at the post-install steps to give your user Docker access if you don't want to use sudo.

To run a hello-world with Docker, I use this:

docker run -d -p 8080:80 nginx:latest

# Test it
curl localhost:8080

The last step will print a welcome page HTML.

3. Set the ratelimit

The rate limit can be implemented using the hashlimit module. There is also a recent module that could be used to accomplish a similar result.

For this example, we set quite an aggressive limit to demonstrate this feature:

sudo iptables -A RATE-LIMIT --match hashlimit --hashlimit-mode srcip --hashlimit-upto 10/min --hashlimit-name per_ip_conn_rate_limit  --jump RETURN
sudo iptables -A RATE-LIMIT -j LOGGING
sudo iptables -A RATE-LIMIT -j DROP

Explanation:

  1. Adding a rule to the end of the RATE-LIMIT chain 2.--hashlimit-upto 10/min Match if the IP has made fewer than ten requests per minute 3.-hashlimit-name per_ip_conn_rate_limit use the IP to group connections together (other options include source / dest ports).

  2. If the number of requests is under the specified limit, RETURN to the previous chain 5. If the number of requests exceeds the specified limit, the rule does not match.

  3. LOG the offending request.

  4. After the log, we will DROP the offending request

Logging is optional but recommended – you want to see who is probing you and when.

4. Configure the log

sudo iptables -A LOGGING -m limit --limit 5/s -j LOG --log-prefix "iptables-throttled: " --log-level info
sudo iptables -A LOGGING -j RETURN

This will write to your system log, usually /var/log/messages or /var/log/system, with the prefix iptables-throttled. No more than five times per second to avoid swamping your storage.

More on the logic behind this in the earlier post

If you've used Docker, you'll link this to the FORWARD chain (see the example below). The FORWARD chain, when configured by Docker, will have the first rule that goes to DOCKER-USER like so:

Chain FORWARD (policy DROP)
target     prot opt source               destination         
DOCKER-USER  all  --  0.0.0.0/0            0.0.0.0/0    

...

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all  --  0.0.0.0/0            0.0.0.0/0

You can add your new chains to DOCKER-USER, and these will be evaluated before any other Docker-related rules. So add your rate limit here:

sudo iptables -I DOCKER-USER 1 --jump RATE-LIMIT

Your sudo iptables -L -n should now end with something like this:

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RATE-LIMIT  all  --  0.0.0.0/0            0.0.0.0/0           
RETURN     all  --  0.0.0.0/0            0.0.0.0/0           

Chain LOGGING (1 references)
target     prot opt source               destination         
LOG        all  --  0.0.0.0/0            0.0.0.0/0            limit: avg 5/sec burst 5 LOG flags 0 level 6 prefix "iptables-throttled: "
RETURN     all  --  0.0.0.0/0            0.0.0.0/0           

Chain RATE-LIMIT (1 references)
target     prot opt source               destination         
RETURN     all  --  0.0.0.0/0            0.0.0.0/0            limit: up to 5/sec burst 4 mode srcip htable-expire 60000
LOGGING    all  --  0.0.0.0/0            0.0.0.0/0
DROP       all  --  0.0.0.0/0            0.0.0.0/0

If you're not using Docker, then add this RATE-LIMIT chain to the top of your INPUT chain instead. I'll leave that as an exercise to experiment with, but it builds on all the same principles.

6. Extra: Whitelisting IPs / CIDR ranges

If you want some IPs or CIDR ranges to bypass the rate limit, add some RETURN rules to the top of your RATE-LIMIT chain.

Docker networks have IP ranges, and you would typically not want Docker containers that talk to each other to be rate limited like you limit external traffic.

You can get their IP ranges with docker network ls and then docker inspect <network_name>, and look under Subnet.

For example, if you're only using the default bridge network, it is called bridge, and you can get its IP range by calling docker inspect bridge | grep Subnet. If this is 172.17.0.0/16, then you can add the exception:

sudo iptables -I RATE-LIMIT 1 --source "172.17.0.0/16" -j RETURN

7. Test

Make sure that your host is open to remote connections on port 8080. Then run a script from a different machine on a loop hitting your open port and see if it gets limited.

IP="<replace with your host's URL or IP>"
i=1
while true; do
  curl -sS --connect-timeout 2 "$IP:8080" >/dev/null
  echo "Attempt $i got exit code $?"
   i=$((i+1))
done

You should see connection timeouts after a short time, and the remote host's system log (e.g. /var/log/messages) should contain your logs with the iptables-throttled prefix.

When you tail -f /var/log/messages, you should see lines that look like this:

May 20 08:29:57 helloworld kernel: [597429.154452] iptables-throttled: IN=ens5 OUT=docker0 MAC=12:34:56:78:9A:BC:12:34:56:78:9A:BC:08:00 SRC=123.456.789.1 DST=172.17.0.2 LEN=133 TOS=0x00 PREC=0x00 TTL=44 ID=0 DF PROTO=TCP SPT=54980 DPT=80 WINDOW=2052 RES=0x00 ACK PSH URGP=0

Note that the DST address is your Docker container's IP address on your internal network, not your host's IP address. Also, see the ACK PSH messages in the example above – these are parts of the TCP protocol.

The rate limit and log messages are based on the components of the TCP protocol, not on each HTTP request.

Extension: Persistence

As I described in detail in my first article on iptables – iptables do not persist across reboot by default.

The Docker daemon dockerd will recreate its own networking and rules when the container restarts, but your additions to DOCKER-USER will not stick.

To make them persist, there are several approaches you can take:

  1. Use the iptables-persistent module to save the state and load it on boot (see my first article on iptables).

    • With docker this gets more complicated. You must prevent the Docker daemon from adding further iptables rules. Otherwise, they can clash with your changes. You can do that by setting the key iptables to false in /etc/docker/daemon.json

    • Drawback is that you must persist all the tables generated by the Docker daemon and any other service on your host. Not just your additions.

  2. Add the above rules into a script that runs after the Docker daemon is started.

Option 2 is summarised below.

Option 2 – Persistence with systemd

You can follow this example if your Linux system uses systemd to manage services (default on most distros). We will wait for the Docker daemon service to start, create its own rules, and then attach our rules.

With sudo, create a file /opt/configure-iptables-ratelimit.sh and paste all of the commands you executed above into it.

#!/bin/sh
set -e

sudo chmod +x /opt/configure-iptables-ratelimit.sh

Create a file /etc/systemd/system/iptables-ratelimit.service, and paste the following into it (with sudo):

[Unit]
Description=Configure rate limiting with iptables
After=docker.service

[Service]
Type=oneshot
ExecStart=/opt/configure-iptables-ratelimit.sh

[Install]
WantedBy=multi-user.target

If you have other services that modify iptables, add them to the After= clause in your systemd config. If you're not using docker, do not add the After line.

Enable the service on boot: sudo systemctl enable iptables-ratelimit

Reboot to test the persistence. Remember to start the docker container again (or add a systemd config to start it).

Further reading

]]><![CDATA[Time pressure did not cause your outage]]>https://blog.konst.kiwi/time-pressure-did-not-cause-your-outagehttps://blog.konst.kiwi/time-pressure-did-not-cause-your-outageSun, 30 Apr 2023 19:00:39 GMTYou are crafting your beautiful application, paying extra attention to detail, but your boss is breathing down your neck saying: "Ship! Ship! Ship! 🚢 ".

Shortcuts are taken. Mistakes are made. Your bug goes into Prod, and now you have an outage.

Maybe you've failed to try out your new feature on real-world data, and a bad query is now hammering your database, which does not scale.

You didn't have time to cover all scenarios on realistic datasets.

You didn't have time to write the extra tests.

You didn't think of all the possible scenarios.

This is where you, as a developer, throw your arms up in despair and say: "time pressure!"

I admit it, I absolutely love a good "I told you so!". We warned you, Evilboss, and now look what happened!

Nelson from Simpsons, pointing his finger with his mouth open. The subtitle says: "I told you so"

While time pressure may be a significant contributing factor, I also firmly believe developers can do much better, even under the constraint of time. Focusing on this one issue is a distraction.

The Root Cause

"Find the Root Cause" is a common phrase. What happened? They ask. The answer is usually – a bug.

A hand-drawn word "bug" in all caps, with arrows pointing to it from all sides.

However, we must think beyond that to improve our software craft.

Typical flow

If your dev outfit is more than one or two developers, you likely have a release process of some sort. Typically it would be a variant of something like the diagram below.

Boxes with arrows between them. In this order: Code, PR, Build, Automated Tests, Manual Testing, Deploy, Release (Feature Flag On), Monitoring and Alerting

You may have more or fewer steps, but if you're entirely missing something like testing, that is a red flag that something is already not quite right.

You typically go through all these layers for something to be released, even in a rush. You can move through this flow quickly while minimising problems if this flow is set up well.

Failures in the flow

When something has failed, look at your version of your development flow above.

Assume that humans are fallible and will write bugs. You cannot prevent that.

Where could your issue have been caught, after the code has been typed but before it hits prod? It may be one or two steps or all the steps, or maybe you need an extra step.

Returning to our example, a bad API is written that is not tested with production data, causing chaos on the database in prod. These are questions you could ask:

  1. Code – a bug is written

    • Did the developer call a dependency that was not well documented?

    • Was it a junior working without enough support?

    • Was there a lack of sound patterns to follow?

    • Was there an unintended side effect from a complicated legacy method?

    • Is there fragile legacy code that needs a clean-up?

  2. Peer Review (PR) – bug is approved

    • Are your PRs just rubber stamps?

    • Was the peer reviewer not qualified?

    • Was there too much code in one PR?

  3. Build – build is kicked off and passes

    • Do you have unit tests as part of the build?

    • Do you have good patterns for unit tests to cover similar scenarios?

    • Do you have linting or other static code checking?

  4. Automated Tests – integration tests or end-to-end tests

    • Is your test dataset realistic?

    • Do you cover performance?

    • Are your tests flaky and are now just being ignored?

  5. Deployment and Release – code is deployed, then a feature flag is turned on

    • Do you have a rollback plan? Feature flag off?

    • Can you release a feature to 10% of users for the first few hours to test for errors and performance issues?

  6. Monitoring and Alerting – this can be an ambulance at the bottom of the cliff, but an important one.

    • Did your alerts tell you about the problem, or did your users?

    • How soon did your alerts go off after the problem was shipped? Can you narrow this time?

Critically look through all this, and then identify the most impactful improvements that can be made.

Idealism vs Reality

Not every company can have all the steps above; not everyone is open to change. We don't all live in idealistic blog posts or ride unicorns.

A silhouette of a unicorn, standing on its rear legs, in some water. A giant full moon in the background reflects on the water below.

This is where you use the failure to your advantage, show your bosses and coworkers that things need to improve, and then start working towards it bit by bit. You do not need permission to do this!

One test here. One good pattern to follow in the future. A couple of PR comments to your workmates to nudge them towards doing things a little better each time too.

I have seen dreadful codebases at companies with dreadful cultures improve through small iterations made by people who care to improve.

And your results at the end of it will most likely be appreciated. If not, take the experience from doing this to somewhere else where it will be recognised.

Back to Time Pressure

Yes, time pressure sucks and makes things worse. You should absolutely raise it as an issue, propose more reasonable alternatives and the like. But you will still have to deal with too much work to squeeze into too little time as part of life.

When time pressure is there, and it is not budging, embrace it as being another constraint, part of an elegant puzzle that needs solving.

What do you think? Leave a comment down below.

]]><![CDATA[Dev Retro 2022 – sudo reboot]]>https://blog.konst.kiwi/dev-retro-2022-sudo-reboothttps://blog.konst.kiwi/dev-retro-2022-sudo-rebootWed, 28 Dec 2022 00:16:17 GMTAs a software developer, I have turned my long-time hobby into a job, which is not always a good idea. Your job does become more than just a way to make money, which leads to trouble. This blog post is my retrospective on a big mindset reboot I have been going through in the past year or so.

Late 2021 was when I realised I need a big mindset shift, and 2022 was when I carried through that change and became a lot more intentional about seeking what I want for myself.

This is going to be opinionated. If the advice below goes against the grain for you, that is ok. There isn't a single mould to fit us all, and this is not intended as general-purpose advice for everyone. But I hope I can share my way of thinking with some people.

The Constant Hype

Software Development changes quickly around us, things that were trendy a few years ago will go stale or obsolete. Tools, techniques, and whole paradigms shift.

My father, a Fortran developer, retired this year and likely will not be replaced with another Fortran guru. Fortran was first created in 1957. Even Fortran has not been constant, if you look at it over 60 years, it has had a lot of evolution. But it is hard for me to imagine sticking to one main programming language for my entire career.

Things change fast, and there is always a lot to master. It is not just the programming language that I'm talking about. In the past few years, things that have piqued my interest may have looked like this:

  • Mobile apps!

  • Single Page (web) Apps! React.js, no Vue.js!

  • Blockchain!

  • Machine learning!

  • Computer Vision!

  • Go, Python, TypeScript, Bash, .NET Core, Rust, keep going.

  • All the clouds!

  • Docker, no Serverless, no go back to containers with Kubernetes!

  • Wait get away from cloud lock-in!

  • Now the AI tyrants are writing code and taking our jobs!

Woddy and Buzz Lightyear from Toystory. Buzz is potining into the distance with a smile, and says "Hype, hype everywhere!". Woody is looking on in horror.

Ok, "hype" is a strong word – a lot of new things are fads, but others are legitimate trends that are good to learn. Trouble is, it can be hard to separate, and you definitely cannot sit still for too long without major change. (Unless you are doing some super specialised Fortran-like archaeology, delivering to an industry that resists change.)

If you dive further into each item on that list above, you will find years of further digging you can do into each of those areas to hone your skills.

No to "Always Keep Growing"

This is where people tell you to "always keep growing"! I have heard this so many times. And honestly, this has been my motto.

While well-meaning, to me this phrase now comes with bad baggage from observing myself as well as others. Elitism. Burnout. Not valuing your workplace or personal relationships. Not valuing your time outside of work. Moving away from your passion into a "higher tier" position or to a "higher tier" company.

I fell into this trap so many times, and after much self-reflection, I am now careful not to chase that.

Adapt with purpose

Instead of "always keep growing". Think about what you want to do for yourself and how you can manoeuvre yourself to get there. Be intentional. Be explicit with yourself. Be explicit when you talk to your employer – "I want to move towards this-and-that. I do not want to keep doing so-and-so for long."

That move could be backward, sideways, or in any direction you want. Sometimes it helps to let go of the concept of "growing", and seek something that is fulfilling, even if it feels like more of a steady state.

Andre the giant from the film "Princess Bride". He's holding a boulder with a big smirk on his face. Quote on top of it says "I did that on purpose"

But, keep in mind, that this will include letting go and neglecting some skills that you no longer need on your journey. It is fine to be an expert in X one day and then forget about it later.

This can also mean taking another position or job that is at a less "fancy" company. Even if to others this may look like a demotion. Even if you end up taking less pay in exchange for having a sense of purpose and fulfilment, or a better work-life balance.

Don't chase the shiny titles

In the past year and something, I have done a very intentional move from Dev Management, being a leader of leads, to a position back to focus on more hands-on tech roles in the Site Reliability Engineering (SRE) and Cloud Architecture space.

It is not that I never want to do management or that I don't like leadership positions. I needed to find something where I can hone in on the technical tools a lot more. And moving into a higher-level leadership position was not helping that goal. Even if it has been something that I was intentionally after for a while.

It does not mean that I never want to pursue that path again. It just means I'm going to put that on hold for now, which is right for me at this point.

The same traps exist if you're pure only the "Individual Contributor" (IC) path, it's easy to get caught up in chasing titles. Senior. Staff. Principal. I was officially a "Master Engineer II" at a top company once (too weird, rephrased on my CV). What I find is that all of it can be somewhat arbitrary and up to the whim of a particular company or people manager. There is also nothing wrong with going from Principal in one company to losing that title in another, especially when you move to a different field.

Don't think about the supposed prestige or the titles of a role too much. They are not universally recognised and chasing those titles can be a distraction away from what you're truly after.

A new approach to side missions

“What’s the world’s greatest lie?” the boy asked, completely surprised.

“It’s this: that at a certain point in our lives, we lose control of what’s happening to us, and our lives become controlled by fate. That’s the world’s greatest lie”

– The Alchemist, Paulo Coelho

When taking on a side mission at work, or a home project, where I am somewhat free to look at something else, I used to pick something relevant to what I was doing at work at the time.

My change this year was to let go of that. I have enough experience and enough good reputation in the industry where honing in deeper on something turns into diminishing returns. There are only so many extra opportunities to be gained by going deeper into something I already know.

Instead, I welcome the distraction of doing something tangential, something radically different from what I normally do. Who knows where it will lead career-wise? Maybe nowhere? But I will enjoy the journey nonetheless! And that is more important than the destination to me today.

Future

There are a few things I want to focus on in 2023 that are outside of my hands-on technical skills:

  • Technical writing, which is why I'm here.

  • Public speaking.

  • Launching my personal side project using the tech I want to play with.

As I've said, this a is a personal retrospective and I hope it can help share my current mindset. The advice is not one side fits all, you always need to think about how things fit your own situation. Please do drop me a line if you have different oppionon in the comments below or find me over at Mastodon.

]]><![CDATA[Hands-on Linux firewall rules with iptables]]>https://blog.konst.kiwi/hands-on-linux-firewall-rules-with-iptableshttps://blog.konst.kiwi/hands-on-linux-firewall-rules-with-iptablesTue, 13 Dec 2022 10:51:04 GMTLet's dive into lower-level firewall configuration at the OS level and go a little beyond the bare basics.

Even when you are hosting with cloud providers that spoil you with their ease of configuration for firewalls, some special requirements may make it necessary for you to understand the lower-level tools.

I have hit this with several different clients, it comes up when their VMs are running a combination of tools such as Docker, third-party security tooling and even CPanel. It gets especially tricky when these third-party tools don't play nice with each other out of the box and set up inconsistent rules. You can end up having to mediate these tools with custom configuration on your part.

In this article, I will run through a few common scenarios with iptables, one of the most popular tools for configuring the Linux kernel firewall. I will cover basic configuration, connection state, persistence, logging, and touch on interaction with Docker in particular.

Linux Firewalls

Linux is such a diverse ecosystem where you choose your tooling, and this extends to firewall configuration utilities. iptables has been around for a long time and is very common but there are others.

Newer versions of Debian ship with nftables, which is a younger cousin of iptables. It is an updated design but bears a lot of similarities.

Under the hood, both iptables and nftables are built around a programmable framework called netfilter. netfilter allows userspace applications to plug into different parts of the Linux kernel networking stack and to define callbacks. Outside of very niche use cases, it is uncommon to use netfilter directly.

These are the common tools, but you could find others or GUI wrappers around the above.

iptables

Set up iptables

In your favourite distribution, install iptables. For example Debian and Ubuntu:

sudo apt-get update
sudo apt-get install iptables

If you have firewalld installed, for this example please remove or disable it. It is usually not a good idea to mess with iptables directly when you have firewalld running over the top.

sudo systemctl stop firewalld
sudo systemctl disable firewalld

Defaults

Now list the rules.

sudo iptables -L

You should see no restrictions by default, which looks like this:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

You see in the above there are three default chains:

  • INPUT – incoming traffic.

  • FORWARD – incoming traffic that is not handled locally, but is passed on to another network (including local Docker networks)

  • OUTPUT – outgoing traffic.

No rules are defined under any of the chains, and the default for all chains is ACCEPT – so any incoming or outgoing packet is accepted.

Example

Let's open port 80 on a host to the world, but open port 8080 only to local network private IP traffic on 10.128.0.0/16.

CAREFUL: Use a non-production host to experiment with OS-level firewalls. They are not as easy to undo as cloud-based firewall rules. If you get the rules wrong, you will lock yourself or your users out. If you do get stuck on your test host with the examples below, reboot the host and the rules will reset, more on that later.

Open web ports

# 1. Accept incoming TCP traffic to destination port 8080 from any source
sudo iptables  -A INPUT -p tcp --dport 8080 -j ACCEPT

# 2. Accept TCP traffic to destination port 8080 from the local network CIDR range
sudo iptables  -A INPUT -p tcp --dport 8081 -s 10.128.0.0/16 -j ACCEPT

# 3. Accept TCP traffic on port 22 from your your personal IP address 
my_ip=# REPLACE THIS WITH YOUR LOCAL IP e.g. 123.45.67.89 
sudo iptables  -A INPUT -p tcp --dport 22 -s $my_ip/32 -j ACCEPT

Command breakdown

sudo iptables -A INPUT -p tcp --dport 8081 -s 10.128.0.0/16 -j ACCEPT command breakdown:

  • -A INPUT – add to the INPUT chain (refer to the original table above).

  • -p tcp – the rule applies to the TCP protocol only

  • --dport 8081 – the rule applies to the destination port 8081 only

  • -s 10.128.0.0/16 – the rule applies to the source IP

  • -j ACCEPT – jump target for the rule. ACCEPT means that the connection will be allowed through.

See man iptables (or here) for complete docs.

Resulting iptables

sudo iptables -L -n will that look like this:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:8080
ACCEPT     tcp  --  10.128.0.0/16        0.0.0.0/0            tcp dpt:8081
ACCEPT     tcp  --  123.45.67.891         0.0.0.0/0            tcp dpt:22

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Evaluation

An incoming connection will be evaluated against the above rules line by line. Once the first condition is met, the action for that condition will be executed. In our case ACCEPT, which will allow the traffic through. No further rules will be evaluated after the first match.

If no rule matches the traffic, then the default policy for the chain will be used. Which in our example is also ACCEPT. So the above config still means "Accept all traffic".

Reject

Let's reject the traffic that does not match the rules. Again we can append another rule to the end, but this time with a REJECT action.

# 4. REJECT all packets that are not yet matched by other rules
sudo iptables -A INPUT -j REJECT

Note in all the examples above, iptables -A is used, which appends each rule to the end.

So now the last line of the INPUT table will be:

REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable

Test

To try this out, I like to spin up a one-liner python web server, and run this on your remote host:

nohup python3 -m http.server 8080 2>/dev/null &
nohup python3 -m http.server 8081 2>/dev/null &

Then on your local machine see if you can reach the web server. *nix:

my_host=<your remote host IP>
# should succeed:
curl $my_host:8080

# should fail immediately with "Connection refused":
curl $my_host:8081

Remember that 8080 is open to the world, so you should get a response immediately.

If 8080 is not working:

  • Check that your iptables -L matches the output in the example above – look for the ACCEPT before the blanket REJECT.

  • Check for any other firewalls between you and your remote host. For example, if you're running on AWS, for this example, open up the security group to allow your local IP on all ports.

Problem

There is one issue with the above setup. Go back into your remote host, and try any sort of outbound traffic.

ping google.com
curl https://google.com:443/

Both of the above hang and eventually time out. But what's going on? Your OUTPUT chain is still an ACCEPT-all policy. At a first glance, it might seem that this combination of rules should allow all traffic out of your host.

And it does but it doesn't. Your outbound packets will leave your host and reach the Google web server. But the response back from Google will hit your INPUT chain, not match any of the rules and get REJECTed.

You can try to fix this by whitelisting any IP you expect to contact, but that is usually not feasible, so you can make use of the connection state.

Connection State

Basic iptables rules are stateless. This means that no state about pre-existing connections is taken into consideration. With stateless rules, for traffic to come out of the host and get a response back you need ACCEPT rules in both the INPUT and OUTPUT tables.

The stateless rules use very few resources as it is just a matter of evaluating the rules one by one without any other context. This would also make them very performant.

You can have stateful rules with the conntrack matcher. When conntrack is used, each connection is tracked in memory and can be looked up during rule evaluation.

Fixing outbound traffic

To allow a response to come back for any outbound connection that you make, add this rule:

sudo iptables -I INPUT 1 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

This is saying:

  1. -I INPUT 1 Insert a rule at index 1 – we don't want this rule to go to the end, it needs to come before the explicit REJECT.

  2. -m conntrack Use the conntrack matcher.

  3. --ctstate ESTABLISHED,RELATED Match connections that have a state of ESTABLISHED or RELATED – existing connections only (i.e. created after an accepted OUTPUT message)

  4. -j ACCEPT Accept them all.

Now you can check the rules:

sudo iptables -L -n

And see the output like this:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:8080
ACCEPT     tcp  --  10.128.0.0/16        0.0.0.0/0            tcp dpt:8081
ACCEPT     tcp  --  123.45.67.89         0.0.0.0/0            tcp dpt:22
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Try Google again and you will get a response:

ping google.com
curl https://google.com:443/

REJECT vs DROP

You can see in the examples above, the REJECT line ends with reject-with icmp-port-unreachable. Any connection that reaches this rule will immediately get an icmp-port-unreachable response from your server, killing the connection.

An alternative action is DROP. In that case, your server will silently drop the connection without a response – the client will wait until it times out.

REJECT provides faster feedback. But at other times DROP may be preferable. If someone is not where they should be – do you need to bother to send them a response at all?

Persistence

iptables out of the box does not have persistence built in. Your rules will stay in place until a reboot, but once you reboot your machine the rules will be wiped.

Persistence is managed outside of iptables themselves using external utilities.

iptables-persistent

For the simple setup above, you could make your rules persistent by installing this package (Debian/Ubuntu):

sudo apt-get install iptables-persistent

This will:

  1. give you two new tools: iptables-save and iptables-restore.

  2. Save your current config to a file such as /etc/iptables/rules.v4

  3. Add an init.d script /etc/init.d/iptables-persistent (or /etc/init.d/netfilter-persistent) to restore the saved rules on boot.

iptables-persistent – Manual operations

You can trigger them manually these by calling:

sudo iptables-save >/etc/iptables/rules.v4
sudo iptables-restore </etc/iptables/rules.v4

This creates a human-readable and editable file that you can tweak if needed.

But do note, that iptables-save and iptables-restore are very crude tools for basic use cases. Changes to rules are not automatically reflected in the persisted rules. You will need to manage the lifecycle yourself, by calling sudo iptables-save >/etc/iptables/rules.v4 after each subsequent change.

Logging

Logging for iptables is not enabled out of the box. If you do need to enable logging for debugging or auditing purposes, make sure you consider how much you want to log. Here is one technique that you can try:

  1. Add a new Chain called LOGGING

  2. Jump to this chain from wherever you want this log to come. For example to log all INPUT, add at an index 1 (-I INPUT 1). Or if you want to log only under certain conditions insert it at a more appropriate place in the ordered list of rules.

  3. Log the message. If you're enabling this in prod. It is recommended that you do some throttling here, you don't want a DoS to flood your log and hence your disk space.

  4. RETURN out of the LOGGING table

This can be accomplished using the example below. It will issue a log for all incoming traffic other than the traffic that matches the first rule (port 8080 in our previous example).

# 1.
sudo iptables -N LOGGING
# 2.
sudo iptables -I INPUT 2 -j LOGGING
# 3.
# -m limit --limit 10/s will throttle at 10 messages per second. Good for debugging purposes.
sudo iptables -A LOGGING -m limit --limit 5/s -j LOG --log-prefix "iptables-input: " --log-level info
# 4.
sudo iptables -A LOGGING -j RETURN

The logs will go to your default system log (e.g. /var/log/messages on Debian). You can configure this behaviour using your logging utility such as rsyslogd.

CAUTION because logging connections on a real production server can fill up your disk space quite fast, look into limiting what you log and into log rotation (logrotate for your system logs).

Clean up

If you are keeping the above host around, you might want to shut off the test web servers and clear the iptables.

# One-liner to kill processes with the words 'http.server' 
ps x | grep http.server | awk '{print $1}' | head -n -1 | xargs kill

If you want to clean up iptables, you can try the -D <chain> <position> syntax to delete rules one by one (e.g. iptables -D INPUT 1 to delete first rule in INPUT). Or reboot the machine (if you haven't set up a restore on boot using iptables-restore).

Brief word on Docker

The Docker daemon has a built-in limited understanding of iptables (and the higher level wrapper around it that you may be using –firewalld). The daemon sets the networking up somewhat differently for host and bridge types of Docker networking.

By default, when you bind to a host port, the Docker daemon will add rules to your iptables to allow all incoming traffic for any ports that you listen to for host networking. This is where you need to be careful with your iptables setup and its persistence.

You can read more on what Docker does in their documentation on "Docker and iptables". Play around with it and see what it does. You will see that it creates new chains that get jumped to from the standard FORWARD chain.

Persistence with Docker

If you're going to modify the rules and you want to make use of persistence, make sure you set the JSON value "tables": false in /etc/docker/daemon.json and restart the Docker daemon. Otherwise, the daemon can override or conflict with the changes that you make in iptables yourself.

IPv6

This article only touched on IPv4. iptables does not apply to IPv6 traffic, but its companion tool ip6tables with the same CLI interface will let you define IPv6. Similarly, you can use ip6tables-restore and ip6tables-save.

Further Reading

]]>