Don't use full hostnames for logging

I’m going to explain why reverse DNS (rDNS) resolution to write hostnames/FQDNs in your logfiles is a bad idea. Even if you don’t read this article, consider doing this one thing:

Turn off reverse DNS resolution for logfiles. It introduces uncertainty into something that should be an accurate record, it reduces the effectiveness of some controls, and it can reduce your server/application performance.

Here’s a log snippet that demonstrates reverse (IP -> hostname) DNS resolution in a log entry:

Jan  4 21:08:57 server sshd[14619]: Failed password for invalid user root from host.naughty.com port 48156 ssh2`

What we see

Let’s work through a fictional scenario based on an attacker trying brute-force methods against SSH on a Linux system, with Fail2Ban installed. The fictional attacker is coming from 192.168.1.1.

The target system’s syslog contains:

Jan  7 15:48:02 server sshd[31063]: Failed password for invalid user root from nice.person.com port 41330 ssh2`
Jan  7 15:48:02 server sshd[31069]: Failed password for invalid user root from nice.person.com port 43432 ssh2`
Jan  7 15:48:03 server sshd[31063]: Failed password for invalid user root from nice.person.com port 41330 ssh2`

fail2ban.log contains:

2015-01-07 15:48:03,542 fail2ban.actions: WARNING [ssh-iptables] Ban 10.0.0.1`

The attacker is coming from 192.168.1.1, so how did that appear in the logs as nice.person.com and why did it ban a completely different IP address? Reverse DNS, of course. If the attacker has the ability to control their DNS servers, they can manipulate reverse DNS (rDNS) records to return false information.

Behind the scenes

When sshd (the SSH daemon) sees a login attempt for 192.168.1.1, it tries to resolve the address using recursive DNS. However, the attacker’s DNS server is configured to server inaccurate data, and returns a result that looks like:

1.1.168.192.in-addr.arpa.    IN    PTR    nice.person.com.`

sshd writes nice.person.com into its syslog message and moves on.

fail2ban continuously monitors syslog, and spots three failed logins from the same host in a short period, so needs to ban them. It extracts the hostname (nice.person.com) from the syslog message and performs a forward (hostname -> IP) DNS lookup to use in a firewall rule:

nice.person.com.    IN    A    10.0.0.1`

This returned IP address is not the same IP that the attacker is coming from. With a little tweaking, a malicious DNS server could be configured to serve different IP addresses for the same hostname, or even random addresses.

The outcome is not only that the attacker can still brute-force SSH logins with fail2ban ineffective, but also causes a denial of service for another client.

It’s also inefficient

When services, applications and daemons write log entries with hostnames, they convert each client IP address into a hostname. This can be slow, causing overhead. One address lookup can trigger a large number of DNS queries. Caching helps, but it’s not a solution. With very high traffic services, resolving hostnames can be a significant drain on your infrastructure’s resources in terms of system calls and IO, and can easily lead to capacity issues.

Over the years, I’ve seen:

• DNS servers struggle with the volume of requests purely from running production services, let alone malicious activity.
• DNS servers rate-limit high-volume clients, which causes timeouts, delays and errors.
• Excessive caching cause post-change outages.
• Active malicious activity using this to bypass controls.

Some services take specific action based on a hostname or domain which will require a rDNS lookup, but still log the client’s IP address, not hostname.

The solution

1. Turn off DNS resolution in logfiles. It’s as easy as that.
2. Keep your log information as accurate as possible. When your service is already working with a unique IP address, don’t introduce uncertainty by converting it to a hostname and back again.
3. Query the hostname when you analyse the data, if you must.