Linux logs, journald and auditd

Linux spreads its evidence across several places. Knowing which file or service owns the answer saves you from grepping the entire disk.

The main sources

• journald — the systemd journal, read with journalctl.
• /var/log/auth.log — authentication on Debian/Ubuntu (Red Hat uses /var/log/secure).
• /var/log/syslog or messages — general system messages.
• /var/log/nginx/ or apache2/ — web server access and error logs.
• auditd — kernel-level auditing of syscalls and file access.

journalctl -u ssh --since "1 hour ago"
journalctl -p err -b
grep "Failed password" /var/log/auth.log | tail -20
last -a | head

Reading authentication events

A failed SSH login looks like this:

Failed password for invalid user admin from 203.0.113.9 port 51234 ssh2

That single line is not an incident. A thousand of them in five minutes from one address is a brute-force attempt. A success immediately after a run of failures is much more interesting.

Check:

1. Was the account expected to log in?
2. Was the source address expected?
3. Was the authentication method a password, key or something else?
4. What happened next on the host?