SSH hardening? ─┬─ Disable password auth ──→ key-only authentication
├─ Change default port ──→ obscurity (reduces bot noise, NOT real security)
├─ Limit root login ──→ PermitRootLogin no (use sudo)
├─ Use ed25519 keys ──→ faster, smaller, more secure than RSA
└─ 2FA ──→ Google Authenticator PAM for high-security servers
# /etc/ssh/sshd_config — Production Hardened
Port 2222 # Non-standard port (reduces noise)
PermitRootLogin no # No direct root login
PubkeyAuthentication yes # Key-based only
PasswordAuthentication no # Disable passwords
ChallengeResponseAuthentication no
KbdInteractiveAuthentication no
UsePAM yes
Key algorithms (modern, secure)
HostKey /etc/ssh/ssh_host_ed25519_key
HostKey /etc/ssh/ssh_host_rsa_key
KexAlgorithms curve25519-sha256,curve25519-sha256@libssh.org,diffie-hellman-group16-sha512
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com
Access control
AllowUsers deploy admin@10.0.1.0/24 # Restrict who and from where
MaxAuthTries 3
MaxSessions 5
LoginGraceTime 30
Session settings
ClientAliveInterval 300 # Check client every 5min
ClientAliveCountMax 2 # Disconnect after 2 missed checks
X11Forwarding no
AllowTcpForwarding no
PermitTunnel no
Logging
LogLevel VERBOSE
SyslogFacility AUTH
# Generate ed25519 key (modern, recommended)
ssh-keygen -t ed25519 -a 100 -C "deploy@myapp" -f ~/.ssh/id_ed25519
Generate RSA key (legacy compatibility, 4096-bit)
ssh-keygen -t rsa -b 4096 -a 100 -C "deploy@myapp" -f ~/.ssh/id_rsa
Copy key to server
ssh-copy-id -i ~/.ssh/id_ed25519.pub -p 2222 deploy@server
SSH config (~/.ssh/config)
Host myapp
HostName 10.0.1.1
Port 2222
User deploy
IdentityFile ~/.ssh/id_ed25519
ServerAliveInterval 60
ServerAliveCountMax 3
apt install fail2ban -y
systemctl enable fail2ban
/etc/fail2ban/jail.local (override defaults)
[DEFAULT]
bantime = 3600 # 1 hour ban
findtime = 600 # 10 minute window
maxretry = 3 # 3 failures = ban
destemail = admin@myapp.com
sender = fail2ban@myapp.com
action = %(action_mwl)s # Ban + send email with logs
Log path
logpath = /var/log/auth.log
[sshd]
enabled = true
port = 2222
filter = sshd
logpath = /var/log/auth.log
maxretry = 3
bantime = 3600
[sshd-ddos]
enabled = true
port = 2222
filter = sshd-ddos
logpath = /var/log/auth.log
maxretry = 5
bantime = 86400 # 24h ban for DDoS
[nginx-http-auth]
enabled = true
port = http,https
filter = nginx-http-auth
logpath = /var/log/nginx/error.log
maxretry = 3
[nginx-limit-req]
enabled = true
port = http,https
filter = nginx-limit-req
logpath = /var/log/nginx/error.log
maxretry = 5
bantime = 1800
[recidive]
enabled = true
filter = recidive
logpath = /var/log/fail2ban.log
bantime = 86400 # 24h ban for repeat offenders
maxretry = 3
# /etc/fail2ban/filter.d/nginx-limit-req.conf
[Definition]
failregex = ^.limiting requests by zone.client: .*$
ignoreregex =
# Check status
fail2ban-client status
fail2ban-client status sshd
Unban IP
fail2ban-client set sshd unbanip 1.2.3.4
Ban IP manually
fail2ban-client set sshd banip 5.6.7.8
View banned IPs
fail2ban-client get sshd banned
Check jail logs
zgrep 'Ban' /var/log/fail2ban.log*
# Reset to clean state
ufw --force reset
Default policies
ufw default deny incoming
ufw default allow outgoing
SSH (from specific IP ranges only)
ufw allow from 10.0.1.0/24 to any port 2222 proto tcp comment 'SSH from office'
ufw allow from 203.0.113.50 to any port 2222 proto tcp comment 'SSH from home'
Web
ufw allow 80/tcp comment 'HTTP'
ufw allow 443/tcp comment 'HTTPS'
Application ports (internal only)
ufw allow from 10.0.1.0/24 to any port 3306 proto tcp comment 'MySQL'
ufw allow from 10.0.1.0/24 to any port 5432 proto tcp comment 'PostgreSQL'
ufw allow from 10.0.1.0/24 to any port 6379 proto tcp comment 'Redis'
ufw allow from 10.0.1.0/24 to any port 9090 proto tcp comment 'Prometheus'
ufw allow from 10.0.1.0/24 to any port 3000 proto tcp comment 'Grafana'
Rate limit SSH
ufw limit 2222/tcp
Enable
ufw --force enable
ufw status numbered
# Flush existing rules
iptables -F
iptables -X
Default policies
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT
Allow loopback
iptables -A INPUT -i lo -j ACCEPT
iptables -A OUTPUT -o lo -j ACCEPT
Allow established connections
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
SSH rate limiting
iptables -A INPUT -p tcp --dport 2222 -m state --state NEW -m recent --set --name SSH
iptables -A INPUT -p tcp --dport 2222 -m state --state NEW -m recent --update --seconds 60 --hitcount 4 --rttl --name SSH -j DROP
iptables -A INPUT -p tcp --dport 2222 -m state --state NEW -j ACCEPT
Web
iptables -A INPUT -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -j ACCEPT
Internal services
iptables -A INPUT -s 10.0.1.0/24 -p tcp --dport 3306 -j ACCEPT
iptables -A INPUT -s 10.0.1.0/24 -p tcp --dport 5432 -j ACCEPT
Log dropped packets
iptables -A INPUT -j LOG --log-prefix "DROPPED: " --log-level 4
Save rules
apt install iptables-persistent
netfilter-persistent save
# Network hardening
net.ipv4.tcp_syncookies = 1 # SYN flood protection
net.ipv4.tcp_max_syn_backlog = 65536 # SYN queue size
net.ipv4.tcp_synack_retries = 2 # Reduce SYN-ACK retries
net.ipv4.ip_forward = 0 # Disable IP forwarding
net.ipv4.conf.all.send_redirects = 0 # No ICMP redirects
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.all.accept_redirects = 0 # No ICMP redirects accepted
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
TCP hardening
net.ipv4.tcp_rfc1337 = 1 # Protect against TIME_WAIT assassination
net.ipv4.tcp_fin_timeout = 15 # Reduce TIME_WAIT
net.ipv4.tcp_keepalive_time = 300 # 5min keepalive
net.ipv4.tcp_keepalive_intvl = 15
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_max_tw_buckets = 65536 # Max TIME_WAIT sockets
net.core.somaxconn = 65535 # Increase listen queue
net.core.netdev_max_backlog = 65536
net.ipv4.tcp_max_orphans = 65536
Connection tracking
net.netfilter.nf_conntrack_max = 1048576 # Track 1M connections
net.netfilter.nf_conntrack_tcp_timeout_established = 7200
Memory
vm.swappiness = 10 # Prefer RAM over swap
vm.overcommit_memory = 1 # Allow overcommit (for Redis)
vm.dirty_ratio = 10
vm.dirty_background_ratio = 5
File descriptors
fs.file-max = 2097152
fs.nr_open = 1048576
Security
kernel.exec-shield = 1 # No-execute stack (x86)
kernel.randomize_va_space = 2 # ASLR enabled
kernel.kptr_restrict = 1 # Restrict kernel symbols
Apply
sysctl --system
# Apply all sysctl settings
sysctl --system
Verify key settings
sysctl net.ipv4.tcp_syncookies
sysctl net.core.somaxconn
sysctl vm.swappiness
sysctl kernel.randomize_va_space
Persist across reboots
sysctl -p /etc/sysctl.d/99-hardening.conf
apt install auditd audispd-plugins -y
systemctl enable auditd
/etc/audit/auditd.conf
local_events = yes
write_logs = yes
log_file = /var/log/audit/audit.log
log_group = adm
log_format = RAW
flush = INCREMENTAL_ASYNC
freq = 50
max_log_file = 100
num_logs = 5
name_format = HOSTNAME
max_log_file_action = ROTATE
space_left = 75
space_left_action = SYSLOG
action_mail_acct = root
admin_space_left = 50
admin_space_left_action = SUSPEND
disk_full_action = SUSPEND
disk_error_action = SUSPEND
# /etc/audit/rules.d/audit.rules
Delete existing rules
-D
Buffer size
-b 8192
Monitor key system files
-w /etc/passwd -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/group -p wa -k identity
-w /etc/sudoers -p wa -k sudo
-w /etc/ssh/sshd_config -p wa -k sshd
Monitor privilege escalation
-w /usr/bin/sudo -p x -k sudo_exec
-w /usr/bin/su -p x -k su_exec
Monitor cron
-w /etc/crontab -p wa -k cron
-w /etc/cron.d/ -p wa -k cron
-w /var/spool/cron/ -p wa -k cron
Monitor system calls
-a always,exit -F arch=b64 -S unlink -S unlinkat -S rename -S renameat -F auid>=1000 -F auid!=4294967295 -k delete
-a always,exit -F arch=b64 -S chmod -S chown -S fchmod -S fchown -F auid>=1000 -F auid!=4294967295 -k perm_mod
Monitor network
-a exit,always -F arch=b64 -S bind -S listen -k network
Failed login attempts
-a always,exit -F arch=b64 -S connect -F auid>=1000 -F auid!=4294967295 -k network_connect
# Search for key events
ausearch -k identity # Changes to passwd/shadow/group
ausearch -k sudo_exec # Sudo executions
ausearch -k sshd # SSH config changes
ausearch -k delete # File deletions
ausearch -k network # Network binds/listens
Generate reports
aureport --auth # Authentication report
aureport --file # File access report
aureport --host # Host-based report
aureport --summary # Summary report
apt install aide -y
Initialize database
aideinit
Copy to reference location
cp /var/lib/aide/aide.db.new /var/lib/aide/aide.db
Check for changes
aide --check
Update database after legitimate changes
aide --update
cp /var/lib/aide/aide.db.new /var/lib/aide/aide.db
Cron: daily check
echo '0 3 * root /usr/bin/aide --check | mail -s "AIDE Daily Report" admin@myapp.com' > /etc/cron.d/aide
# Install rkhunter and chkrootkit
apt install rkhunter chkrootkit -y
Update rkhunter database
rkhunter --update
rkhunter --propupd
Scan
rkhunter --check --skip-keypress --report-stage
chkrootkit
Cron: daily check
echo '0 4 * root /usr/bin/rkhunter --check --skip-keypress --report-stage 2>&1 | mail -s "RKHunter Report" admin@myapp.com' > /etc/cron.d/rkhunter
Possible intrusion? ─┬─ Confirmed ──→ Isolate (iptables DROP all), preserve evidence, incident response
├─ Suspected ──→ Investigate (audit logs, aide --check, last -f /var/log/wtmp)
├─ File modified ──→ Compare with AIDE database, verify change is legitimate
├─ Unknown process ──→ Check /proc/PID, lsof, netstat, kill and investigate
└─ Brute force ──→ fail2ban status, ban IP, check auth.log for patterns
Server needs hardening? ─┬─ Internet-facing ──→ Priority 1: SSH keys + fail2ban + firewall + auditd
├─ Internal app ──→ Priority 2: Firewall rules + patches + sysctl
└─ Database server ──→ Priority 3: Local auth only + encrypted connections
Can't SSH? ─┬─ Connection refused ──→ Service running? (systemctl status sshd)
├─ Permission denied ──→ Key correct? (authorized_keys, 700/600 perms)
├─ Too many auth failures ──→ fail2ban banned? (fail2ban-client status)
├─ Host key mismatch ──→ Remove from ~/.ssh/known_hosts
└─ Connection timeout ──→ Firewall blocking? (ufw status, iptables -L)
# Lynis — system audit
apt install lynis -y
lynis audit system --quick
Key checks
lynis audit system --tests-from-group "authentication" "firewalls" "kernel" "ssh"
OpenVAS (Greenbone) — vulnerability scanner
(Docker-based)
docker run -d --name openvas \
-p 9392:9392 \
-e PASSWORD=admin \
greenbone/openvasd:latest
Rapid7 — external port scan
nmap -sV -sC -O --script vuln 10.0.1.1
Test SSH configuration
ssh-audit 10.0.1.1 # Install: pip install ssh-audit
Test SSL/TLS
testssl.sh -E https://api.myapp.com # Comprehensive SSL test
# 1. Port scan
nmap -sV -sC -p- 10.0.1.1
2. SSH configuration
ssh-audit 10.0.1.1
3. Check for default credentials
hydra -l root -P /usr/share/wordlists/rockyou.txt ssh://10.0.1.1:2222
4. Check for SUID binaries (potential privilege escalation)
find / -perm -4000 -type f 2>/dev/null
5. Check world-writable files
find / -xdev -type f -perm -0002 2>/dev/null
6. Check crontab for malicious entries
crontab -l
ls -la /etc/cron.d/
7. Check listening services
ss -tlnp
8. Check for unusual users
awk -F: '$3 == 0 {print $1}' /etc/passwd # Root-equivalent users
awk -F: '$2 == "" {print $1}' /etc/shadow # Users without passwords
# /etc/sysctl.d/99-performance.conf
Increase connection capacity
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65536
net.ipv4.tcp_max_tw_buckets = 65536
net.core.netdev_max_backlog = 65536
TCP window scaling for high-BDP networks
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
Fast connection recycling
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15
#!/bin/bash
/usr/local/bin/harden-server.sh — Run on new server provision
set -euo pipefail
echo "🔒 Starting server hardening..."
1. Update system
apt update && apt upgrade -y
apt install -y fail2ban ufw auditd aide lynis unattended-upgrades
2. Create deploy user
useradd -m -s /bin/bash deploy
mkdir -p /home/deploy/.ssh
cp /root/.ssh/authorized_keys /home/deploy/.ssh/
chown -R deploy:deploy /home/deploy/.ssh
chmod 700 /home/deploy/.ssh
chmod 600 /home/deploy/.ssh/authorized_keys
usermod -aG sudo deploy
3. Configure SSH
sed -i 's/#PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config
sed -i 's/#PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config
sed -i 's/#Port 22/Port 2222/' /etc/ssh/sshd_config
systemctl restart sshd
4. Configure firewall
ufw default deny incoming
ufw default allow outgoing
ufw allow from 10.0.1.0/24 to any port 2222 proto tcp
ufw allow 80/tcp
ufw allow 443/tcp
ufw --force enable
5. Configure fail2ban
systemctl enable fail2ban
systemctl start fail2ban
6. Initialize AIDE
aideinit
cp /var/lib/aide/aide.db.new /var/lib/aide/aide.db
7. Set sysctl
sysctl --system
8. Configure unattended upgrades
dpkg-reconfigure -plow unattended-upgrades
echo "✅ Server hardening complete!"
# /etc/systemd/system/myapp.service
[Unit]
Description=MyApp API Server
After=network.target postgresql.service redis.service
Wants=postgresql.service redis.service
[Service]
Type=notify
User=deploy
Group=deploy
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/bin/myapp --config /etc/myapp/config.yaml
Restart=on-failure
RestartSec=5
StartLimitBurst=3
StartLimitIntervalSec=60
Security hardening
NoNewPrivileges=yes # No sudo/suid
ProtectSystem=strict # Read-only filesystem
ProtectHome=true # No /home access
PrivateTmp=true # Private /tmp
ReadWritePaths=/var/log/myapp /var/run/myapp
ReadOnlyPaths=/etc/myapp
Network restrictions
IPAddressAllow=10.0.1.0/24 127.0.0.1
IPAddressDeny=any
Resource limits
LimitNOFILE=65536
MemoryMax=2G
CPUQuota=200%
[Install]
WantedBy=multi-user.target
Security incident detected? ─┬─ Active breach ──→ ISOLATE (iptables DROP all), preserve evidence
├─ Suspicious login ──→ Check auth.log, last -f, who, fail2ban status
├─ File modified ──→ aide --check, verify against git/tracked state
├─ Service crashed ──→ journalctl -u service, dmesg, check OOM
└─ Unknown traffic ──→ ss -tlnp, netstat, iptables -L -n -v
net.ipv4.tcp_congestion_control=bbr (better throughput)net.core.somaxconn=65535 (avoid dropped connections)fs.file-max=2097152 (avoid "too many open files")vm.swappiness=10 (prefer RAM, swap only under pressure)net.ipv4.tcp_tw_reuse=1 (recycle TIME_WAIT sockets)net.ipv4.tcp_max_syn_backlog=65536 (handle SYN floods)| # | Pitfall | Impact | Fix |
|---|---------|--------|-----|
| 1 | SSH with password auth | Brute force compromise | Disable PasswordAuthentication, use ed25519 keys |
| 2 | Default SSH port 22 | Constant bot scanning | Change to non-standard port (security through obscurity + fail2ban) |
| 3 | No fail2ban | Unlimited brute force attempts | 3 retries → 1h ban, recidive → 24h ban |
| 4 | UFW disabled | All ports exposed | ufw default deny incoming, allow only needed ports |
| 5 | World-writable files | Privilege escalation | find / -perm -0002, fix permissions |
| 6 | No file integrity monitoring | Undetected modifications | Install AIDE, daily cron checks |
| 7 | SUID binaries in unusual places | Privilege escalation | Audit SUID: find / -perm -4000 |
| 8 | Exposed database ports | Direct DB attacks | Bind to 127.0.0.1 or restrict with firewall |
| 9 | No audit logging | No accountability | Auditd rules for auth, sudo, file changes |
| 10 | Default sysctl values | SYN floods, connection exhaustion | Hardened sysctl.conf (syncookies, somaxconn, tw_reuse) |
| 11 | Root login enabled | Direct root compromise | PermitRootLogin no, use sudo |
| 12 | Missing security updates | Known exploits | unattended-upgrades for critical patches |
# SSH: regenerate host keys
ssh-keygen -A
SSH: test configuration
sshd -t
Firewall: show all rules with numbers
ufw status numbered
iptables -L -n -v --line-numbers
Auditd: search logs
ausearch -k sshd -ts today
aureport --auth -ts today
AIDE: check file integrity
aide --check
System: check open ports
ss -tlnp
lsof -i -P -n | grep LISTEN
System: check users with shell access
grep -v '/nologin\|/false' /etc/passwd
System: recent logins
last -20
lastb -20 # Failed logins
Security: quick check
lynis audit system --quick
Kernel: check sysctl values
sysctl -a | grep -E 'syncookies|somaxconn|swappiness'
Certificates: check expiry
for cert in /etc/letsencrypt/live/*/fullchain.pem; do
echo "$cert: $(openssl x509 -in $cert -noout -enddate | cut -d= -f2)"
done