Server security

Intro

I would argue, that one of the first things we should do when we install the new server, is security. Laying the secure foundation for future services will give us few security advantages, and, arguably, will teach us few good practices to do so.

Richard Clarke, a former special adviser to the president on cybersecurity, once said, “If you spend more on coffee than on IT security, then you will be hacked. What’s more, you deserve to be hacked.”

This course is not about security. Security is an extremely complex topic. We won’t claim our system becomes secure after following steps laid below, but we will take minimal necessary steps for us to protect our system, especially if your hosting provider doesn’t firewall built-in.

Security is the good first thing to focus on, because it gives us the ability to build everything on top of the secure system.

Also, we will learn basics of security audit in linux using auditd.

Ansible-hardening

ansible-hardening role uses industry-standard security hardening guides to secure Linux hosts.

It all starts with the Security Technical Implementation Guide (STIG) from the Defense Information Systems Agency (DISA), part of the United States Department of Defense.

Installation

To install ansible-hardening from Ansible Galaxy, use this command

$ ansible-galaxy install git+https://github.com/openstack/ansible-hardening

Let’s use very simple playbook, that will look like this:

- name: Harden all systems
  hosts: all
  become: yes
  vars:
    security_enable_firewalld: no
    security_rhel7_initialize_aide: yes
    security_enable_virus_scanner: yes
  roles:
    - ansible-hardening
  tasks:
    - cron:    
        name: aide
        cron_file: aide
        user: root
        special_time: daily
        state: absent

    - name: Add crontab to check aide
      become: true
      cron:
        minute="18"
        hour="18"
        job="/sbin/aide --check | /bin/mail -s \"$HOSTNAME - Daily aide integrity check run\" root"
        name="Aide check"

    - template:
        src: templates/cron_clamav.sh.j2
        dest: /usr/local/bin/cron_clamav.sh
        owner: root
        group: root
        mode: 0655

    - name: Add clamav to cron
      become: true
      cron:
        minute="18"
        hour="15"
        job="/usr/local/bin/cron_clamav.sh"
        name="Clamav"

    - template:
        src: templates/wireguard_fix.py.j2
        dest: /usr/local/bin/wireguard_fix.py
        owner: root
        group: root
        mode: 0655

    - name: Add wireguard network fix to cron
      cron:
        minute="*/15"
        job="/usr/local/bin/wireguard_fix.py"
        name="Wireguard fix"

    # https://bugzilla.redhat.com/show_bug.cgi?id=1263328#c19
    - template:
        src: templates/cron_policy.cil.j2
        dest: /root/cron_policy.cil
        owner: root
        group: root

    - command: semodule -i /root/cron_policy.cil

    - service:
        name: crond
        state: restarted

To run this role, run:

$ ansible-playbook -i hosts security.yml -e 'ansible_user=rick ansible_ssh_private_key_file=key_file'

Using this configuration, we are able to install fail2ban, auditd, aide, change ssh configuration and a lot more. You can grab security.yml file here.

Auditing using auditd

After running security roles above, we will have auditd daemon installed on our systems.

We won’t dive into auditd much in this tutorial, but let explore few of the basic features of this amazing software.

If you want to generate a summary report on all command executions on the server, run:

$ sudo aureport -x --summary

Will produce output like this

Executable Summary Report
=================================
total  file
=================================
8219  /usr/sbin/sshd
503  /usr/bin/sudo
78  /usr/sbin/crond
72  /usr/sbin/xtables-multi
52  /usr/lib/systemd/systemd
20  /usr/bin/kmod
18  /usr/sbin/ebtables-restore
9  /usr/sbin/groupadd
5  /usr/sbin/useradd
4  /usr/sbin/usermod
2  /usr/lib/systemd/systemd-update-utmp

It will give us the list of most popular commands executed locally. 1

Let take a look at the statistics of failed events:

$ sudo aureport --failed

Will produce output like this:

Failed Summary Report
======================
Range of time in logs: 03/09/2018 19:53:41.727 - 03/10/2018 09:20:23.014
Selected time for report: 03/09/2018 19:53:41 - 03/10/2018 09:20:23.014
Number of changes in configuration: 0
Number of changes to accounts, groups, or roles: 4
Number of logins: 0
Number of failed logins: 257
Number of authentications: 0
Number of failed authentications: 188
Number of users: 2
Number of terminals: 2
Number of host names: 58
Number of executables: 3
Number of commands: 0
Number of files: 0
Number of AVC's: 0
Number of MAC events: 0
Number of failed syscalls: 0
Number of anomaly events: 0
Number of responses to anomaly events: 0
Number of crypto events: 171
Number of integrity events: 0
Number of virt events: 0
Number of keys: 0
Number of process IDs: 301
Number of events: 631

Let’s take a look at authentication report for all the attempts made to your system:

$ sudo aureport -au -i

Will produce output similar to this:

Authentication Report
============================================
# date time acct host term exe success event
============================================
261. 03/10/2018 08:05:45 ? 4li54-1-88-174-4-30.fbx.proxad.net ssh /usr/sbin/sshd no 9338
262. 03/10/2018 08:05:47 (unknown) 88.174.4.30 ssh /usr/sbin/sshd no 9339
263. 03/10/2018 08:05:47 (unknown) 88.174.4.30 ssh /usr/sbin/sshd no 9346
264. 03/10/2018 08:38:24 ? 120.205.199.218 ssh /usr/sbin/sshd no 9394
265. 03/10/2018 08:38:25 (unknown) 120.205.199.218 ssh /usr/sbin/sshd no 9400
266. 03/10/2018 08:38:29 ? 120.205.199.218 ssh /usr/sbin/sshd no 9407
267. 03/10/2018 08:38:30 (unknown) 120.205.199.218 ssh /usr/sbin/sshd no 9411
268. 03/10/2018 08:38:37 ? 120.205.199.218 ssh /usr/sbin/sshd no 9420
269. 03/10/2018 08:38:39 (unknown) 120.205.199.218 ssh /usr/sbin/sshd no 9421
270. 03/10/2018 08:40:31 ? 120.205.199.218 ssh /usr/sbin/sshd no 9433
271. 03/10/2018 08:40:33 (unknown) 120.205.199.218 ssh /usr/sbin/sshd no 9434

As you can see, there are a lot of failed authentication attempts going on at any given time.

For example, you can search for all events (if any) touching the file /etc/ssh/sshd_config:

$ sudo ausearch -f /etc/ssh/sshd_config -i

Please refer to documentation if you want to learn more about auditd, ausearch, autrace or aureport.

  1. Please note that not all commands are logged by default. Only security-related ones are logged. 

Updated: