bookmark_border4 Ways to Secure Your Linux Server

Every minute, every few seconds, your server is receiving a number of malicious connections; from an IP address in Moldava checking to see if a far out port is open, to someone in Iran trying to login into your server with a random username. It’s a bit unsettling but rest assured these connections are likely stemming from autonomous crawlers scanning the web and you are not being personally targeted.

The attacks seem to take on two different forms; 1) The port is accessed via SSH protocol (i.e. accessible through the use of specialized software or browser extensions), or 2) The port can be accessed via HTTP (i.e. accessible via a fresh browser install). Depending on your setup, you will likely spot attempts of type one in your system authentication log files. Attempts of type two should be visible on any firewall or web server logs you may have setup.

The rest of this post is written with a Debian based OS in mind, so commands may be slightly different depending on your distro. Below are a few basic items to begin securing your Linux server.

  1. SSH Keys
  2. Uncomplicated Firewall (UFW)
  3. Apache Web Server
  4. Fail2Ban

SSH Keys

You don’t need to use SSH keys to log into your server but it would make it a lot more secure than having a generic root account with a password.

SSH keys utilize public-key cryptography where a public key is used for encryption and a private key is used for decryption. Your public key can be shared openly without compromising your server’s security. However, you will need to make sure your private key is never disclosed.

Your private key is a file that you will use every time you need to make a connection to your server. This means in order to connect to your server the individual must physically (or would it be digitally) have this file. This communication occurs over port 22 and also known as Secure Shell (SSH).

This document over at Digital Ocean is extremely informative on how SSH works, how to setup SSH for your server and even walks you through disabling root login.

If you happen to have multiple servers, you will need to consider trade-offs between security and convenience. The issue is explained beautifully by tylerl and YaOzl at Stack Overflow and I highly recommend reading through the thread. In summary, you must make a choice between using the same key for all of your servers or generating different keys for each server (potentially, inconvenient). Moreover, since it is recommend you add a passphrase to your key-pair, you have the option of using the same passphrase or using a different one for each key-pair.

Last but definitely not least, make sure to back up your private keys in a secure location and produce an additional external backup in case of an emergency. Your private key is only as secure as you make it.

Uncomplicated Firewall (UFW)

While you could directly use iptables to manage your server’s firewall, a simpler alternative is to use Uncomplicated Firewall (UFW). UFW is a front-end to iptables that is easier to learn.

UFW is disabled by default so you will want to enable it as soon as possible. You will also want to make your first rule so that you can SSH back into your system. It is a good idea to set this rule to LIMIT to prevent brute force attacks on your server.

sudo ufw limit 22/tcp
sudo ufw enable
sudo ufw logging on 

If your home IP happens to be static (unlikely, but worth a look), you could limit your rule even further by specifying your home IP.

If you are using your server to serve a website you will need to write additional rules to allow connections to port 80. If your site makes use of SSL certificates you will also need to open port 443. Since you want to make your site available to everyone make sure to use ALLOW instead of LIMIT here.

sudo ufw allow 80/tcp
sudo ufw allow 443/tcp

If you are using multiple servers, you can be write even more specific rules for your firewall (i.e. having your web server’s port 80 only listen to your load balancer’s IP).

You can view your current UFW configuration with the command below.

sudo ufw status verbose

Apache Web Server

If you have installed Apache web server, it might be a good idea to turn off some of the default settings. When Apache runs into a problem it displays an error page that shows a little too much information for our visitors including our OS and web server version. To remove this information, head over to your Apache folder and into your conf-available folder. Locate the file security.conf and look for the following blocks to change your settings.

cat conf-available/security.conf | grep -in "ServerTokens"
cat conf-available/security.conf | grep -in "ServerSignature"

You will want to turn off your ServerSignature and change your ServerTokens value to what is most appropriate for you.

A second setting we may want to change is the directory listing that is enabled by default in the absence of an index.html file. This option can be overridden by a virtual host file so if the configuration below appears to have no effect consider looking into your virtual host configurations.

Lastly, if your website accepts uploads it is a good idea to limit your request size. By default, the request size is unlimited which can cause issues with your site or perhaps be abused in a malicious way.

Head over to your main Apache configuration file, apache2.conf and locate the directory tags near the bottom of the file.

<Directory /var/www/>
        Options -Indexes #Remove directory listing, note the -
        AllowOverride None
        Require all granted
        LimitRequestBody 512000 #Set request size in bytes
</Directory>

Make sure to restart your Apache web server to save these changes.

sudo systemctl restart apache2

Fail2ban

Consider installing Fail2ban to prevent brute force attacks. This will allow you to ban malicious IP addresses for a variable amount of time. The application comes with an SQLITE database so you can preserve long-term bans over server resets.

sudo apt-get install fail2ban

Fail2ban is ready to use as soon as you install it but I would recommend increasing the ban time and double checking the default settings correspond with your setup.

After installing Fail2ban, create a copy of the jail.conf file and name it jail.local. (Fail2ban is configured to read the settings off your .local file). Next, locate the default bantime variable and set it to something higher. This variable will be towards the top under default settings. You can configure different ban times for different Fail2ban “jails”. If you would like to permanently ban these IP’s enter a value of -1.

cd /etc/fail2ban
cp jail.conf jail.local
cat jail.local | grep -in "bantime"

Now, restart Fail2ban to save your changes.

sudo fail2ban-client reload

If you would like to further configure Fail2ban, I recommend this article which provides more background information and instructions.

Remember to routinely monitor your system for malicious activity. This is probably best done through specialized software or probably writing your own scripts!

bookmark_borderWordPress: 404 After Changing Permalinks

Whew! This took me way longer than it should have. Long story short, I was missing the rewrite module on Apache, my Apache configuration file needed an update and my Apache virtual host setup was messed up! I’m not positive at what point I broke my virtual host setup but I knew I had made a mistake when I ran the command below and saw a backup file under my virtual hosts. Sadly, I only came across the command below after a few hours of researching the issue. Let’s blame it on the fact I had not had a morsel of food for seven hours.

apache2ctl -S

First we must verify that the rewrite module in Apache is enabled. If the module is not enabled we will not be able to run the directives under our WordPress site’s .htaccess file. If the module is already enabled on your server, you will receive a message stating so. It will also be visible under /etc/apache2/mods-enabled as rewrite.load. Run the commands below to enable the module and restart Apache.

sudo a2enmod rewrite
sudo service apache2 restart

After verifying the rewrite module is enabled, check to see if there is still a 404 error on your pages. If there isn’t, great! Else, we continue, when you change the permalink format through your WordPress admin console it inserts rewrite rules to the hidden file .htaccess in your WordPress directory. .htaccess files contain overrides or additional information about your site configuration. Versions 2.3.9+ of Apache, by default do not allow .htaccess files to override the configuration specified under /etc/apache2/apache2.conf. Apache’s main configuration file also includes any configurations you may have written for any enabled virtual hosts. Below is a snippet from the default Apache configuration file.


# Sets the default security model of the Apache2 HTTPD server. It does
# not allow access to the root filesystem outside of /usr/share and /var/www.
# The former is used by web applications packaged in Debian,
# the latter may be used for local directories served by the web server. If
# your system is serving content from a sub-directory in /srv you must allow
# access here, or in any related virtual host.
...
..
<Directory /var/www/>
        Options Indexes FollowSymLinks
        AllowOverride None
        Require all granted
</Directory>

The AllowOverride None directive above is stating that for directory /var/www/ do not allow the server configuration to be overwritten through the use of .htaccess files. There are two reasons for this logic 1) performance gain and 2) security. WordPress’s default behavior is to edit the .htaccess file when you make changes to the permalink format so you may choose to keep .htaccess but keep in mind Apache does not recommend the use .htaccess if you have access to the main configuration file.

The AllowOverride directive accepts three different values; 1) None, 2) All, and 3) Directive Name. If you choose to keep using the .htaccess file, the third option allows for better security than simply entering All by allowing us to enter which directives can be overwritten. For the purposes of this particular issue we will be using the value FileInfo so we may make use of the rewrite module but visit the official Apache documentation for more information on the directive names available and how to use them.

Below is the code you want present in your Apache configuration file so WordPress’s .htaccess may be able to rewrite your URL’s and serve your pages.

       <Directory [YOUR_DIRECTORY_PATH]>
                Options Indexes FollowSymLinks
                AllowOverride FileInfo #Alternatively, use AllowOverride All
                Require all granted
        </Directory>

If you would like to remove the .htaccess file and instead include your rewrite logic in your configuration file you will need to include the rewrite directives within the Directory tags as shown below.

       <Directory [YOUR_DIRECTORY_PATH]>
                Options Indexes FollowSymLinks
                AllowOverride FileInfo #Alternatively, use AllowOverride All
                Require all granted
                RewriteEngine On
                RewriteBase /
                RewriteRule ^index\.php$ - [L]
                RewriteCond %{REQUEST_FILENAME} !-f
                RewriteCond %{REQUEST_FILENAME} !-d
                RewriteRule . /index.php [L]
        </Directory>

After editing your configuration files make sure to restart your Apache server to make our changes active. By this point, the 404 error on your WordPress pages should be gone.

If it is not and you are making use of virtual hosts, I urge you to verify that your Apache virtual host configuration is correct. I spent quite a while with this issue because I had a larger underlying issue with my virtual host configuration. I was able to identify the issue by not resorting to making changes to apache2.conf when I felt positive the changes on my virtual host configuration file were supposed to work. My particular issue was a file I had renamed as a backup was still linked as a virtual host and was occupying the same server name as the new file I thought was the active virtual host. Both my backup and newer file had the same server name specified. I removed my backup from the server entirely and it worked. My brain was toasted.

bookmark_borderHaProxy: Cannot Bind Socket

As a total noob in system administration, it took me a while to realize that my machine’s services might have been stopped/restarted after Digital Ocean reached out about performing an emergency droplet migration after detecting problems on the physical server where my load balancing droplet/VM was running.

This was confirmed by running the following command and noting that the last time HaProxy had been active was on Digital Ocean’s maintenance day. (Please note that I am running these commands on Ubuntu.)


systemctl status haproxy.service


With this newfound knowledge, I attempted to restart HaProxy as sudo but ran into an error. After checking the HaProxy log (located in /var/log/), I found the following error message.


"Jul 15 17:10:13 ab1 haproxy[32723]: [ALERT] 202/171013 (32723) : Starting []: cannot bind socket [000:000.000.000:80]:"


Something was already occupying/listening on port 80! After using the the netstat command and feeding its output to grep we identified the culprit…Apache! Of course…my droplet was restarted and the apache2 service is probably set to start on boot.

sudo netstat -ltnp | grep -i ":80"


Our site was up and running after stopping the Apache service and starting HaProxy.

sudo service apache2 stop
sudo service haproxy start


I was able to confirm the issue arises on booting by checking our boot log with the following command.


sudo journalctl -b


Jul 15 16:34:11 ab1 systemd[1]: Started The Apache HTTP Server.
Jul 15 16:34:11 ab1 haproxy[978]: [ALERT] 195/163411 (978) : Starting frontend http: cannot bind socket [000.000.000.000:80]
Jul 15 16:34:11 ab1 haproxy[978]: Proxy www-https started.
Jul 15 16:34:11 ab1 haproxy[978]: Proxy www-https started.
Jul 15 16:34:11 ab1 haproxy[978]: Proxy app-backend started.
Jul 15 16:34:11 ab1 haproxy[978]: Proxy app-backend started.
Jul 15 16:34:11 ab1 systemd[1]: haproxy.service: Main process exited, code=exited, status=1/FAILURE
Jul 15 16:34:11 ab1 systemd[1]: haproxy.service: Failed with result 'exit-code'.
Jul 15 16:34:11 ab1 systemd[1]: Failed to start HAProxy Load Balancer.


Now if we want to prevent this from happening in the future, we have to set which services we want to auto start on boot which will be the topic of an upcoming post.


Hasta la vista baby!