In this guide we will be installing Debian 9 (aka stretch) on a physical server with 4 disks. The role of this machice is to be used as a Storage/NAS system. We will create a software RAID 10 setup, with LVM and LUKS full disk encryption. Our goals:

* Install a Debian 9 with RAID10/LVM/LUKS.
* Secure SSH.
* Enable Firewall (UFW).
* Setup bonding with the two network cards.
* Setup remote system unlock with *dropbear* and *initramfs*.
* Setup disk-monitoring with *smartmontools* and *mdadm*.
* Setup *kexec* for faster reboots.

## Computer specs

* Type: HP ProLiant MicroServer
* CPU: AMD Turion(tm) II Neo N40L Dual-Core Processor
* RAM: 2GB
* Disks: 4x3TB SATA
* Network:
* Broadcom Limited NetXtreme BCM5723 Gigabit Ethernet PCIe (on board)
* Intel Corporation 82574L Gigabit Network Connection (extra)

## Assumptions

* Server IP: 192.168.1.10
* Netmask: 255.255.255.0
* Gateway IP: 192.168.1.1
* DNS IP: 192.168.1.1
* Hostname: storage.example.com

## Install Debian stretch

### Basic Settings

It would probably be more clear if there were screenshots for each step, but this was an installation on a physical server and taking photos for each step, opposes my laziness :). Just follow the instructions and you will be fine.

* Choose: **Install**
* Language: **English**
* Country: **other**
* Europe: **Cyprus**
* Country to base default locale settings: **United States**
* Keymap to use: **American English**
* Primary network interface: **enp3s0: Broadcom Limeted NetXtream BCM5723 Gigabit Ethernet PCIe**
* Let it get an IP from DHCP
* Hostname: **storage**
* Domain name: **example.com**
* Root password: **SomethingBigAndUnpredictable**
* Re-enter password to verify: **SomethingBigAndUnpredictable**
* Full name: **Sysadmin**
* Username: **admin**
* Choose a password for the new user: **AlsoSomethingBigAndUnpredictable**
* Re-enter password to verify: **AlsoSomethingBigAndUnpredictable**
* Select your time zone: **Asia/Nicosia**
* Partitioning method: **Manual**

Feel free to adjust the above according to your own preferences.

### Partitioning

There are 4 disks of 3TB each (3.0 TB SATA):

* SCSI1 (0,0,0) (sda)
* SCSI2 (0,0,0) (sdb)
* SCSI3 (0,0,0) (sdc)
* SCSI4 (0,0,0) (sdd)

Not really SCSI but SATA in fact.

#### Partition the devices

Then create a raid partition for */boot*:

* Select the free space of sda and ‘Enter’
* Create a new partition
* New partition size: **512 MB**
* Location of new partition: **Beginning**
* Use as: **physical volume for RAID**
* Done setting up the partition

Lastly create the raid partition to be used by the encrypted volume:

* Select the free space of sda and ‘Enter’
* Create a new partition
* New partition size: **3.0 TB**
* Location of new partition: **Beginning**
* Use as: **physical volume for RAID**
* Done setting up the partition

Repeat the above steps for sdb, sdc and sdd.

#### Setup Software RAID 10

First select ‘Configure software RAID’ and follow these steps:

* Write the changes to the storage devices and configure RAID? **Yes**

The we create the software RAID (MD) devices. First we create device *md0* for /boot:

* Create MD Device
* RAID10
* Number of active devices inn the RAID10 array: **4**
* Number of spare devices inn the RAID10 array: **0**
* Active devices for the RAID10 array (use ‘Space bar’ to select)
* **/dev/sda2**
* **/dev/sdb2**
* **/dev/sdc2**
* **/dev/sdd2**
* Press ‘Continue’ when done.

Then we create the software RAID device to be used for the encrypted volume (*md1*):

* Create MD Device
* RAID10
* Number of active devices inn the RAID10 array: **4**
* Number of spare devices inn the RAID10 array: **0**
* Active devices for the RAID10 array (use the ‘Space bar’ to select)
* **/dev/sda3**
* **/dev/sdb3**
* **/dev/sdc3**
* **/dev/sdd3**
* Press ‘Continue’ when done.

* Press ‘Finish’ when done.

#### Create the /boot volume

When done press ‘Finish partitioning and write changes to disk’.`

When finished you will see a ‘RAID10 device #0 1GB Software RAID device’:

* Select: #1 1.0GB
* Use as: **Ext4 journaling file system**
* Mount point: **/boot**
* Done setting up the partition

#### Setup the encrypted volume

We will be using the software RAID */dev/md1* device for the encrypted volume.

Now select ‘Configure encrypted volumes’ and follow these steps:

* Write the changes to disk and configure encrypted volumes? **Yes**

* Create encrypted volumes
* Select: */dev/md1*
* Erase data: **yes** (this will take a long time)
* Done setting up the partition

* Write the changes to disk and configure encrypted volumes? **Yes**
* Finish
* Encryption passphrase: **MyVeryLongEncryptionPassphrase**
* Re-enter the passphrase to verify: **MyVeryLongEncryptionPassphrase**

#### Setup LVM

Next we select the ‘Configure the Logical Volume Manager’ option and follow these steps:

* Write the changes to disks and configure LVM? **Yes**

* Create volume group
* Volume group name: **VG00**
* Devices for the new volume group:
* /dev/mapper/md1_crypt

Then we create the Logical Volumes (LV). First let’s create a SWAP volume:

* Create logical volume
* Volume group: **VG00**
* Logical volume name: **SWAP**
* Logical volume size: **2048MB** (2G is more than enough for this system)

Lastly we create the system (ROOT) volume. On an enterprise installation we may want to use different volumes for /usr, /home, /var, etc but for a home installation we will be fine to use just one.

* Create logical volume
* Volume group: **VG00**
* Logical volume name: **ROOT**
* Logical volume size: **5996818MB** (All available space)

Press ‘Finish’ when done.

### Start the installation

After all the steps are completed these Logical Volumes will be present on the system:

* LVM VG VG00, LV ROOT 0 6.0 TB
* LVM VG VG00, LV SWAP 0 2.0 GB

#### Create the ROOT filesystem

Under the ‘LVM VG VG00, LV ROOT 0 6.0 TB’ line select the ‘#1 6.0TB’ option:

* Use as: **Ext4 journaling file system**
* Mount point: **/**
* Done setting up this partition

#### Create the SWAP space

Under the ‘LVM VG VG00, LV SWAP 0 2.0 GB’ line select the ‘#1 2.0GB’ option:

* Use as: **swap area**
* Done setting up this partition

Now we are ready to write the changes and start the installation. Press the ‘Finish partitioning and write changes to disk’ option to continue:

* Write the changes to disks? **Yes**

Wait for the base install to finish. Then select a country close to you. No debian mirrors in Cyprus so I use UK:

* Debian archive mirror country: **United Kingdom**
* Debian archive mirror: **ftp.uk.debian.org**
* HTTP proxy: (none)

Wait for the APT configuration to Finish.

* Participate in the package usage survey: **no**

* Choose software to install:
* SSH server
* standard system utilities

Wait while software is installing

* Install the GRUB boot loader to the master boot record.
* Device for boot loader installation:
* **/dev/sda**

Wait for the installation to finish and reboot. Remember to remove the USB during the reboot cycle.

## Post install steps

During start-up you will see the ‘Please unlock md1_crypt’ prompt. Type your LUKS passphrase to unlock the disk and continue.

### Update and Upgrade

Login as *root*:

“`
# apt update && apt -y dist-upgrade
“`

### Install essential packages

“`
# apt -y install vim htop multitail ntp byobu ufw unattended-upgrades downtimed
“`

### Secure ssh

You need to generate an SSH key pair on you PC, if you don’t have one (you should!):

“`
$ ssh-keygen -b 4096
“`

Copy the public key:

“`
$ cat ~/.ssh/id_rsa.pub
“`

Paste the public key at the end of the */root/.ssh/authorized_keys* file in your server and try to login from your PC:

“`
$ ssh root@192.168.1.10
“`

Some final adjustments on your *SSH* config (*/etc/ssh/sshd_config*). Change these values:

“`
Port 2233
PasswordAuthentication no
“`

Restart *SSH*:

“`
# systemctl restart ssh.service
“`

### Enable the UFW firewall

We are using port 2233 for *SSH* so we need to allow that and enable the firewall:

“`
# ufw allow 2233/tcp
# ufw enable
“`

### Setup bonding

Since we have two ethernet cards, we may take advantage of thr Linux bonding feature and join them as one. We will be using the *Adaptive load balancing* mode which provides load balancing of transmit, load balancing of receive for IPv4 and requires no configuration from the switch side.

First we need to install *ifenslave*:

“`
# apt -y install ifenslave
“`

Set up this in */etc/network/interfaces*:

“`
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# enp2s0 is manually configured, and slave to the “bond0″ bonded NIC
auto enp2s0
iface eth0 inet manual
bond-master bond0

# enp3s01 is also manually configured, thus creating a 2-link bond.
auto enp3s0
iface eth1 inet manual
bond-master bond0

# bond0 is the bonded NIC and can be used like any other normal NIC.
# bond0 is configured using static network information.
auto bond0
iface bond0 inet static
address 192.168.1.10
gateway 192.168.1.1
netmask 255.255.255.0

# bond0 uses adaptive load balancing
bond-mode 6
bond-miimon 100
bond-slaves enp2s0 enp3s0
“`

An `ifup bond0` should bring the bonded interface up. Or you can just `reboot`.

### Setup remote-unlock with dropbear

The server will be a headless system, located in a difficult to access location. So we need a way to unlock it when a power failure occurs. The most convenient way to do this is to use a [mandos server](https://wiki.recompile.se/wiki/Mandos) but convenience comes at a [cost](https://www.recompile.se/mandos/man/intro.8mandos). A safer and easier way is to use dropbear during boot (initrd). The weak point of this solution is that the server will be basically offline until the sysadmin manually unlocks it, to boot.

First we install *dropbear* for *initrd*:

“`
# apt -y install dropbear-initramfs
“`

Then we set a custom ssh port for *dropbear*. This better be different than the custom ssh port we used earlier. Change the dropbear port to 2244 in /etc/dropbear-initramfs/config:

“`
DROPBEAR_OPTIONS=”-p 2244″
“`

Add the static IP in the initramtools configuration (*/etc/initramfs-tools/initramfs.conf*):

“`
IP=192.168.1.10::192.168.1.1:255.255.255.0:storage:enp3s0:off
“`

Copy the *authorized_keys* file in */etc/dropbear-initramfs*:

“`
# cp /root/.ssh/authorized_keys /etc/dropbear-initramfs/
“`

Regenerate the initrd file:

“`
# update-initramfs -u
“`

Now reboot and ssh to it to test it:

“`
$ ssh -p 2244 root@192.168.1.10
“`

If your pubkeys are in place you will enter a busybox shell. Enter the `crypt-unlock` command, supply your unlock passphrase and the system will boot to the encrypted system.

### Setup a local MTA for notifications

We will be using our main mailserver as a smarthost for mail to go through.

Install the *postfix* MTA and the *mail* utility:

“`
# apt -y install postfix mailutils
“`

Answer these questions:

* General type of mail configuration: **Internet with smarthost**
* System mail name: **storage.example.com**
* SMTP relay host (blank for none): **smtp.example.com**

Test it:

“`
# echo ‘Testing #1’ | mail -s ‘Test #1’ user@example.com
“`

If you get a mail in your mailbox then everything is set. If not extra configuration may be needed on the smarthost. Contact the sysadmin of the smarthost, or check the logs if you access to it.

### Setup pro-active disk monitoring

Install *smartmontools*:

“`
# apt -y install smartmontools
“`

Enable S.M.A.R.T, offline testing, attribute autosave, short and long test on all 4 devices. Add these lines in */etc/smartd.conf*:

“`
/dev/sda -a -d sat -o on -S on -s (S/../.././02|L/../../6/03) -m user@example.com -M exec /usr/share/smartmontools/smartd-runner
/dev/sdb -a -d sat -o on -S on -s (S/../.././02|L/../../6/03) -m user@example.com -M exec /usr/share/smartmontools/smartd-runner
/dev/sdc -a -d sat -o on -S on -s (S/../.././02|L/../../6/03) -m user@example.com -M exec /usr/share/smartmontools/smartd-runner
/dev/sdd -a -d sat -o on -S on -s (S/../.././02|L/../../6/03) -m user@example.com -M exec /usr/share/smartmontools/smartd-runner
“`

Restart *smartmontools*:

“`
# systemctl restart smartmontools.service
“`

### Setup Software RAID10 monitoring

We also need to setup monitoring for the software raid. Add your email address in the */etc/mdadm/mdadm.conf* file:

“`
MAILADDR user@example.com
“`

Restart the *mdmonitor* service:

“`
# systemctl restart mdmonitor.service
“`

### Setup kexec for faster reboots

*Kexec* is a Linux kernel mechanism that can load a fresh kernel from a running system. This results in a “reboot” without in fact rebooting the computer. The system loads a new kernel, the system appears “rebooted” but skipping the BIOS?UEFI initialization, thus resulting in faster reboots.

Install *kexec-tools*:

“`
# apt -y install kexec-tools
“`

The ‘Should kexec-tools handle reboots (sysvinit only)?’ question is related only to *sysvinit* systems. Since we are using *systemd*, it has no effect in our case.

Now if you want to reboot instead of running `reboot` you can run `systemctl kexec`. The latter command will reboot the system without going though BIOS/UEFI, POST etc and your system downtime is minimized.

And we are done! Store your server in a protected location, add a UPS for power backup and you are ready.

References
———-
* https://wiki.debian.org/Bonding
* https://help.ubuntu.com/community/UbuntuBonding
* https://www.theo-andreou.org/?p=1579
* https://wiki.recompile.se/wiki/Mandos
* http://forums.ayksolutions.com/forum/documentation/knowledgebase/general-server-questions/641-proactively-monitoring-hard-drive-health-using-smartd

In this guide we will show how you can remotely decrypt a headless Debian or Ubuntu Linux system, that has been encrypted with [LUKS](https://en.wikipedia.org/wiki/LUKS).

### Prerequisites

* A LUKS encrypted Debian jessie or Ubuntu xenial system
* Keyboard and monitor for the initial system setup
* Allow SSH root access on the decrypted system using public key authentication
* Use a different port for ssh (assuming port 4422) on the decrypted system

**NOTE: using a different port than the standard SSH port (22) serves a double purpose. For once, you will not received the scary *WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!* message every time you will try to remotely unlock the system and as an additional bonus you will get less SSH attacks on the active system.**

### Installing *dropbear*

[Dropbear](https://en.wikipedia.org/wiki/Dropbear_(software)) is a lightweight SSH server especially suitable for initial ramdisk (initrd) environments and other lightweight systems.

Install *dropbear:*

“`
# apt -y install dropbear
“`

### Setup public key authentication for dropbear

Create the homedir for the root user and the SSH configuration directory:

“`
# mkdir -p /etc/initramfs-tools/root/.ssh
“`

Append your client SSH pubkey to *authorized_keys*:

“`
# cat ~/.ssh/id_rsa.pub | ssh -p 4422 root@encrypted-system “cat >> /etc/initramfs-tools/root/.ssh/authorized_keys”
“`

### Setup a static IP for the unlock environment

This step is optional but highly recommended if you are setting up a static, permanent service. If you skip this step DHCP will kick in, provided you have a DHCP Server in your environment.

Run this command to update the */etc/initramfs-tools/initramfs.conf* configuration file:

“`
echo IP=10.0.0.67::10.0.0.1:255.255.255.0:encrypted-system:eth0:off >> /etc/initramfs-tools/initramfs.conf
“`

Explanation of the different fields:
`[host_ip]::[gateway_ip]:[netmask]:[hostname]:[device]:[autoconf]`

**NOTE: there are two successive colons (::) after the *host_ip*.**

### Setup the unlock script

Copy the following text in */etc/initramfs-tools/hooks/crypt_unlock.sh*:

“`
#!/bin/sh
#
# By Stinky Parkia
# https://stinkyparkia.wordpress.com/2014/10/14/remote-unlocking-luks-encrypted-lvm-using-dropbear-ssh-in-ubuntu-server-14-04-1-with-static-ipst/

PREREQ=”dropbear”

prereqs() {
echo “$PREREQ”
}

case “$1” in
prereqs)
prereqs
exit 0
;;
esac

. “${CONFDIR}/initramfs.conf”
. /usr/share/initramfs-tools/hook-functions

if [ “${DROPBEAR}” != “n” ] && [ -r “/etc/crypttab” ] ; then
cat > “${DESTDIR}/bin/unlock” << EOF #!/bin/sh if PATH=/lib/unlock:/bin:/sbin /scripts/local-top/cryptroot; then kill \`ps | grep cryptroot | grep -v "grep" | awk '{print \$1}'\` # following line kill the remote shell right after the passphrase has # been entered. kill -9 \`ps | grep "\-sh" | grep -v "grep" | awk '{print \$1}'\` exit 0 fi exit 1 EOF chmod 755 "${DESTDIR}/bin/unlock" mkdir -p "${DESTDIR}/lib/unlock" cat > “${DESTDIR}/lib/unlock/plymouth” << EOF #!/bin/sh [ "\$1" == "--ping" ] && exit 1 /bin/plymouth "\$@" EOF chmod 755 "${DESTDIR}/lib/unlock/plymouth" echo To unlock root-partition run "unlock" >> ${DESTDIR}/etc/motd
fi
“`

Make the script executable:

“`
# chmod +x /etc/initramfs-tools/hooks/crypt_unlock.sh
“`

### Apply the configuration

Apply the changes in the initial ramdisk:

“`
# update-initramfs -u
“`

Reboot the system:

“`
# reboot
“`

### Remotely unlock the system

From your client, SSH into the initial ramdisk:

“`
ssh root@encrypted-system
“`

If everything is correct you will be greeted by this MOTD:

“`
To unlock root-partition run unlock

BusyBox v1.22.1 (Ubuntu 1:1.22.0-15ubuntu1) built-in shell (ash)
Enter ‘help’ for a list of built-in commands.
“`

Unlock the system and boot into it:

“`
# unlock
Please unlock disk sda3_crypt:
“`

You will get the following message and you will exit the remote shell if successful:

“`
cryptsetup: sda3_crypt set up successfully
Connection to 10.0.0.67 closed.
“`

You can now login to the active Linux system using the alternative port 4422:

“`
ssh -p 4422 root@encrypted-system
“`

If you can login successfully to your system you can remove the keyboard and monitor and hide your system somewhere where the Sun does not shine :).

Thanks to [Stinky Parkia](https://stinkyparkia.wordpress.com/) for the excellent guide and the brilliant unlock script.

References
———-
* https://stinkyparkia.wordpress.com/2014/10/14/remote-unlocking-luks-encrypted-lvm-using-dropbear-ssh-in-ubuntu-server-14-04-1-with-static-ipst/