Software Checklist for a New Ubuntu Server

I recently upgraded my servers OS drive to an SSD, and what a world of difference that has made in speed over the old HDD. I decided that I might as well reinstall the entire OS while I was at it, so this is a list of things I think are useful to have on a new server.

Update #

The first thing I do of course is update everything and restart the server if needed:

$ sudo apt update
$ sudo apt upgrade

Docker #

I run pretty much everything in Docker. Because of this, I only needed to move my volumes over to my new SSD and docker-compose up -d everything. With this, I’d say about 95% of my server was back up and running as it was before. Minimal downtime!

For installation on Ubuntu, I just followed the official install instructions.

msmtp #

For any system mail that may be generated, I want it to be sent directly to my email. This can be accomplished with the packages msmtp, msmtp-mta, and bsd-mailx. Unfortunately, msmtp is too old of a version in Ubuntu 20.04 for some of the configuration options I want to use, so I need to pin it to a newer release. If you are already running Ubuntu 20.10, or don’t care to have a newer msmtp feel free to skip this next section.

Pin the newer msmtp #

Ubuntu 20.10 (groovy) has a new enough version for us to use. In order to get it without hosing our entire system, we set priorities for package sources and pin the newer version of msmtp.

First you will want to edit your /etc/apt/sources.list (or wherever you keep your sources list) and add this to the bottom:

# For msmtp
deb http://us.archive.ubuntu.com/ubuntu groovy universe
# deb-src http://us.archive.ubuntu.com/ubuntu groovy universe

Then, make a new file at /etc/apt/preferences.d/msmtp.pref with the contents of:

# If packages have different priorities, the one with the higher priority wins.
# Thus, no package from groovy will ever be picked (default 500 priority from focal > -10).
Package: *
Pin: release n=groovy
Pin-Priority: -10

# If the packages have the same priority, the package with a higher version number (most recent) wins.
# Thus, msmtp with focal and groovy priority of 500 will go with the newer (groovy) one.
Package: msmtp
Pin: release n=groovy
Pin-Priority: 500

Package: msmtp-mta
Pin: release n=groovy
Pin-Priority: 500

I have left my comments in the file for future reference. We are telling the package manager to never select any packages from groovy, except for msmtp, msmtp-mta, and their dependencies.

Running the following command should show that you have two pinned packages, and that the groovy/universe packages have a -10 priority.

apt-cache policy

Now do an update:

$ sudo apt update

Make sure that there are still no updates present. If there are, you somehow did the pinning incorrectly. There should be no updates, as we will never pull updates from groovy (except for msmtp, msmtp-mta, and their dependencies).

Install and set up msmtp #

Installation is simple:

$ sudo apt install msmtp msmtp-mta bsd-mailx

If you did the pinning correctly, you should see that it installed msmtp version 1.8.11 (or newer). Once you have msmtp and friends installed, you need to configure it.

I use SendGrid for my mailing needs. My example /etc/msmtprc will assume you are doing the same. If not, you will need to change the authentication:

account         default
host            smtp.sendgrid.net
port            587
auth            on
user            apikey
password        <the_generated_api_key>
tls             on
tls_trust_file  /etc/ssl/certs/ca-certificates.crt
tls_starttls    on
aliases         /etc/aliases
logfile         /var/log/msmtp
# New msmtp 1.8.11 config options
from            %[email protected]
set_from_header on

Be sure to change the “example.com” portion of the config to match whatever your hostname is. Additionally, you will need to create a log file for msmtp to use (helpful for debugging)

$ sudo touch /var/log/msmtp
$ sudo chown msmtp:adm /var/log/msmtp
$ sudo chmod 660 /var/log/msmtp

As well as an /etc/aliases file:

root: [email protected]
default: [email protected]

With this setup, any user on the system can send mail to any local user, as well as any normal email. The from envelope will be forced to the format <local_username>@example.com (from /etc/msmtprc), and any mail going to a local user will be routed based on the /etc/aliases file. In this case, mail being sent to the root user will go to [email protected], and mail going to any other user will use the default alias, [email protected].

Test msmtp #

There are a couple of ways to test this, but a simple one is to do the following:

mail -s "test" root < /dev/null
mail -s "test" [email protected] < /dev/null

You can test that the aliases are working, as well as sending to an actual email address. If there are any issues, you can check the logs at /var/log/msmtp.

SmartMonTools #

Now that we have working mail out of the system, we can set up SMART monitoring. Pretty simple to install:

$ sudo apt install smartmontools

It should be good out of the box for monitoring, but I have used the ArchWiki complete smartd.conf example for automated testing. Add this to your /etc/smartd.conf file, and be sure to comment out the existing DEVICESCAN line:

# From Arch Wiki: https://wiki.archlinux.org/index.php/S.M.A.R.T.#Complete_smartd.conf_example
# The word DEVICESCAN will cause any remaining lines in this
# configuration file to be ignored: it tells smartd to scan for all
# ATA and SCSI devices.
# -a            monitor all attributes
# -o on         enable automatic offline data collection
# -S on         enable automatic attribute autosave
# -n standby,q  do not run check if disk is in standby, and suppress log message to that effect so as not to cause a write to disk
# -s ...        schedule short and long self-tests
# -W ...        monitor temperature
# -m ...        mail alerts
# -M test       send a test email every time smartd starts
DEVICESCAN -a -o on -S on -n standby,q -s (S/../.././02|L/../../6/03) -W 4,35,40 -m root -M test

Once you have that added (and you commented out the existing DEVICESCAN line) go ahead and restart the service.

$ sudo systemctl restart smartd.service

If all goes well, you should get a test email for each drive that it found and is now monitoring. A short self-test will be started every day between 2-3am, and an extended self test will be started weekly on Saturdays between 3-4am. Any issues that may arise will be sent to the root user (and thus wherever your /etc/aliases file sends it). If you don’t want the test email sent whenever smartd starts, you can remove the -M test part of the config.

SSH #

In addition to using TMUX whenever I SSH into my server, I also have a few extra modifications I do to /etc/ssh/sshd_config. You might need to find and replace/uncomment them:

PermitRootLogin no
PasswordAuthentication no

The disallows root from ever logging in, and also does not allow password only authentication. Be sure you are logging in with an SSH key before you set PasswordAuthentication to no!

And More! #

I’m sure there’s plenty of other things that can be done for a new server. When I discover them, I’ll be sure to update this article! It’ll be helpful to look back on if I ever do this again.