prd

FreeBSD Home Server

March 27, 2019 · 11 minute read

Introduction

This is my home server, more or less. I use this page to remember what I’ve done and as a reference for less-often used commands, in case I need it later on.

Maybe some of it might be of help to others, so I published it.

I got tired of my Synology DS1812+ and went back to the glorious FreeBSD, ZFS and jails setup.

Upon first boot

SSH access

Copy public part of a pre-generated SSH key from your client machine for easy SSH administration.

From client machine:

$ ssh-copy-id username@ip.of.server.host
$ ssh username@ip.of.your.server

Remove password authentication and rely solely on pubkeys:

# vim /etc/ssh/sshd_config
ChallengeResponseAuthentication no
PasswordAuthentication no
PubkeyAuthentication yes

Make sure the SSHd is enabled on startup in /etc/rc.conf:

sshd_enable="YES"

Restart the SSH service:

# service sshd restart

Updating the OS

# freebsd-update fetch
# freebsd-update install

Useful tools

$ su -
# pkg install sudo vim-tiny tmux lsof rsync git

Run rehash if you are using csh to update $PATH.

Run visudo and uncomment “%wheel” line.

Vim config

This is my tiny vimrc for some sane defaults, including saving a keystroke for ex commands by switching the use of : and ;.

~/.vimrc:

set nocompatible hidden nobackup visualbell
filetype indent on

set modelines=0 nomodeline
set clipboard+=unnamed
set wildchar=<Tab> wildmenu wildmode=list:longest,full
set path+=**
set laststatus=2
set tabstop=4 softtabstop=4 shiftwidth=0 expandtab shiftround
set smartcase
set backspace=2

let mapleader = ","
nnoremap ; :
nnoremap : ;
nnoremap Q :.!$SHELL<CR>
nnoremap <silent> <leader>l :set list!<CR>
set listchars=tab:▸\ ,eol:¬,trail:·

tmux

I rarely bother with reconfiguring. I like using defaults. With that said, I am bothered by always trying tmux attach upon login and not finding any active session.

This little script in ~/bin checks if there is a running session already and then attaches to that, otherwise it will create one.

#!/bin/sh

tmux has
if [ $? == 1 ]; then
    tmux
else
    tmux attach
fi

I call it ta and chmod +x ~/bin/ta for an easy tmux jump start.

HDD monitoring

# pkg install smartmontools
# echo 'smartd_enable="YES"' >> /etc/rc.conf

Create and edit /usr/local/etc/smartd.conf as suggested.

Comment DEVICESCAN.
Uncomment/create:
/dev/ada0 -a -o on -S on -s (S/../.././02|L/../../6/03)
/dev/ada1 -a -o on -S on -s (S/../.././02|L/../../6/03) 
/dev/ada2 -a -o on -S on -s (S/../.././02|L/../../6/03)
/dev/ada3 -a -o on -S on -s (S/../.././02|L/../../6/03)
/dev/ada4 -a -o on -S on -s (S/../.././02|L/../../6/03)
/dev/ada5 -a -o on -S on -s (S/../.././02|L/../../6/03)
...

# service smartd start
# smartd -q onecheck
# smartctl -tshort /dev/ada0
...
# smartctl -a /dev/ada0

Message of the day

Get rid of all that welcome stuff, generate something nice with figlet and edit /etc/motd:

 _____              ____ ____  ____  
|  ___| __ ___  ___| __ ) ___||  _ \ 
| |_ | '__/ _ \/ _ \  _ \___ \| | | |
|  _|| | |  __/  __/ |_) |__) | |_| |
|_|  |_|  \___|\___|____/____/|____/ 

Ports

# portsnap fetch extract

For ports updates from time to time:

# portsnap fetch update

NTP

Set NTP to sync time on boot (if you only installed ntpd and not ntpdate):

# echo 'ntpd_sync_on_start="YES"' >> /etc/rc.conf

Edit /etc/ntp.conf if needed. NTP pools are pretty clever and try to use the closest one anyway…

Get rid of any leap second fetching errors:

# pkg install ca_root_nss && service ntpd onefetch

Disable syslogd network

I do not gather logs from other sources and I do not send my logs away. Disable network binding by starting with double s flags, as in -ss. Add to /etc/rc.conf:

syslogd_flags="-ss"

User/group administration

Show user:

# id <user>

Add group and add members:

# pw groupadd <groupname>
# pw groupmod <groupname> -m <user>

ZFS

Show status:

# zpool status
# zfs list
# df -h

Create a new pool and file systems:

# zpool create -f -m /data data raidz2 /dev/... /dev/...
# zpool status
# zfs create [options] poolname/whatever

…or import an existing pool from a previous installation:

# zpool import <poolname>

Replacing zfs-on-root mirror

The new disk MUST be of the same size or bigger than the one still in the pool.

# zpool status
# zpool detach zroot ada0p3 // failed drive/partition (first drive in this case)
// Replace physical drive
# gpart create -s GPT ada0
# gpart add -b 40 -l gptboot0 -s 512K -t freebsd-boot ada0 // get the -l (label) flag right
# gpart add -s 2G -l swap0 -t freebsd-swap ada0 // or whatever swap space used
# gpart add -t freebsd-zfs -l zfs0 ada0
# zpool status
# zpool attach zroot ada1p3 ada0p3 // zpool attach <pool> <existing> <new>

jails

Taken from https://clinta.github.io/freebsd-jails-the-hard-way/ for manual managing of jails, since a lot of third-party tools are badly maintained. And the magic ain’t that magical.

Base jail and a skeleton template

Create a zfs for all jails:

# zfs create -o mountpoint=/usr/local/jails zroot/jails

Set up a template base installation with ports inside:

# zfs create -p /usr/local/jails/templates/base-12.0-RELEASE
# fetch ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/12.0-RELEASE/base.txz -o /tmp/base.txz
# tar -xvf /tmp/base.txz -C /usr/local/jails/templates/base-12.0-RELEASE
# fetch ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/12.0-RELEASE/ports.txz -o /tmp/ports.txz
# tar -xvf /tmp/ports.txz -C /usr/local/jails/templates/base-12.0-RELEASE
# cp /etc/resolv.conf /usr/local/jails/templates/base-12.0-RELEASE/etc/resolv.conf
# cp /etc/localtime /usr/local/jails/templates/base-12.0-RELEASE/etc/localtime

Update template:

# env UNAME_r=12.0-RELEASE freebsd-update -b /usr/local/jails/templates/base-12.0-RELEASE fetch install
# env UNAME_r=12.0-RELEASE freebsd-update -b /usr/local/jails/templates/base-12.0-RELEASE IDS

Create a skeleton template that each jail will be copied from:

# zfs create -p zroot/jails/templates/skeleton-12.0-RELEASE

Copy variable skeleton content from base template:

# mkdir -p /usr/local/jails/templates/skeleton-12.0-RELEASE/usr/ports/distfiles /usr/local/jails/templates/skeleton-12.0-RELEASE/home /usr/local/jails/templates/skeleton-12.0-RELEASE/portsbuild
# mv /usr/local/jails/templates/base-12.0-RELEASE/etc /usr/local/jails/templates/skeleton-12.0-RELEASE/etc
# mv /usr/local/jails/templates/base-12.0-RELEASE/usr/local /usr/local/jails/templates/skeleton-12.0-RELEASE/usr/local
# mv /usr/local/jails/templates/base-12.0-RELEASE/tmp /usr/local/jails/templates/skeleton-12.0-RELEASE/tmp
# mv /usr/local/jails/templates/base-12.0-RELEASE/var /usr/local/jails/templates/skeleton-12.0-RELEASE/var
# mv /usr/local/jails/templates/base-12.0-RELEASE/root /usr/local/jails/templates/skeleton-12.0-RELEASE/root

Create symlinks in base template to the skeleton:

# cd /usr/local/jails/templates/base-12.0-RELEASE
# mkdir skeleton
# ln -s skeleton/etc etc
# ln -s skeleton/home home
# ln -s skeleton/root root
# ln -s skeleton/tmp tmp
# ln -s skeleton/var var
# cd usr/ports/
# ln -s ../../skeleton/usr/ports/distfiles distfiles
# cd ../../usr/
# ln -s ../skeleton/usr/local local

Edit make.conf so portbuild work dir is inside skeleton:

# echo "WRKDIRPREFIX?=  /skeleton/portbuild" >> /usr/local/jails/templates/skeleton-12.0-RELEASE/etc/make.conf

Snapshot of the skeleton and a place for the thin jails:

# zfs snapshot zroot/jails/templates/skeleton-12.0-RELEASE@skeleton
# zfs create zroot/jails/thinjails

Create /etc/jail.conf with global settings:

# /etc/jail.conf

# Global settings

interface = "em0";
host.hostname = "$name.vkbox";
path = "/usr/local/jails/$name";
ip4.addr = 10.0.1.$ip;
mount.fstab = "/usr/local/jails/$name.fstab";

exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.poststop = "sleep 2";
exec.clean;
mount.devfs;

Update all jails base installation and ports:

# env UNAME_r=12.0-RELEASE freebsd-update -b /usr/local/jails/templates/base-12.0-RELEASE fetch install
# portsnap -p /usr/local/jails/templates/base-12.0-RELEASE/usr/ports auto

Simply backup /usr/local/jails/thinjails/ to save the configuration for each jail.

Start jails on boot:

# echo 'jail_enable="YES"' >> /etc/rc.conf
# service jail start

Create a new jail

New jail (called httpd):

# zfs clone zroot/jails/templates/skeleton-12.0-RELEASE@skeleton zroot/jails/thinjails/httpd
# echo hostname=\"httpd\" > /usr/local/jails/thinjails/httpd/etc/rc.conf
# mkdir -p /usr/local/jails/httpd

Update and append /etc/jail.conf:

httpd {
    $ip = 91;
}

Create jail fstab /usr/local/jails/httpd.fstab:

/usr/local/jails/templates/base-12.0-RELEASE  /usr/local/jails/httpd/ nullfs   ro          0 0
/usr/local/jails/thinjails/httpd     /usr/local/jails/httpd/skeleton nullfs  rw  0 0

Start/stop jail:

# jail -c httpd
# jail -r httpd

Login:

# jexec httpd

Do whatever you want with your new jailed environment. Like, changing root password:

# passwd

Shell script for new jails

I have put the above commands in a shell script in ~/bin for easy creation of new jails:

#!/bin/sh

if [ `whoami` != "root" ]; then
    echo Must be root.
    exit 1
fi

_NAME=$1
_IP=$2

if [ -z "$_NAME" ] || [ -z "$_IP" ]; then
    echo "usage: $0 <name> <last octet in IP>"
    exit 1
fi

echo Cloning zfs skeleton.
zfs clone zroot/jails/templates/skeleton-12.0-RELEASE@skeleton zroot/jails/thinjails/$_NAME
echo Settings hostname in new jail.
echo hostname=\"$_NAME\" > /usr/local/jails/thinjails/$_NAME/etc/rc.conf
echo Creating a mountpoint for new jail.
mkdir -p /usr/local/jails/$_NAME

echo Updating /etc/jail.conf.
echo "
$_NAME {
    \$ip = $_IP;
}
" >> /etc/jail.conf

echo Creating a fstab for new jail.
echo "/usr/local/jails/templates/base-12.0-RELEASE  /usr/local/jails/$_NAME/ nullfs   ro          0 0
/usr/local/jails/thinjails/$_NAME     /usr/local/jails/$_NAME/skeleton nullfs  rw  0 0
" >> "/usr/local/jails/$_NAME.fstab"

echo Start the new jail with 'jail -c $_NAME'.
echo Do not forget to change root password inside jail.

I have named it ~/bin/newjail and run it as such:

$ sudo newjail httpd 91

List jails

See current active jails:

# jls

Remove a jail

Delete the entry in /etc/jail.conf and then remove the mountpoint, fstab and zfs dataset:

# rm -rf /usr/local/jails/<name>
# rm -rf /usr/local/jails/<name>.fstab
# zfs destroy zroot/jails/thinjails/<name>

UTF-8

/etc/login.conf:

    :setenv=... // append ",LC_COLLATE=C
    // At the end of the default list, append
    :charset=UTF-8:\
    :lang=en_US.UTF-8: // or whatever from `locale -a`

Update:

# cap_mkdb /etc/login.conf

Nginx

SSL

Using Letsencrypt, install the certbot “standalone” client:

# pkg install py36-certbot

Close down nginx, so certbot can bind to 80443 for obtaining the certs:

# service nginx stop

Get your certificates:

# certbot-3.6 certonly --standalone -d example.org [-d www.example.org]
# certbot-3.6 certonly --standalone -d another.com [-d www.another.com]

Certs and keys are stored in /usr/local/etc/letsencrypt/live/example.org.

Edit nginx config (/usr/local/etc/nginx/nginx.conf) to use the certs for your new domains:

server {
    listen 80;
    listen 443 ssl;
    server_name www.example.org example.org;
    ssl_certificate /usr/local/etc/letsencrypt/live/example.org/fullchain.pem;
    ssl_certificate_key /usr/local/etc/letsencrypt/live/example.org/privkey.pem;
    if ($scheme != "https") {
        return 301 https://$host$request_uri;
    }
    location / {
        root /mnt/example.org;
        index index.html index.htm;
    }
}

Start nginx and browse securely:

# service nginx start

To renew all certs automatically and non-interactive, quietly run certbot renew every night with hooks for shutting down the http server (and let certbot bind to the ports) and starting it once finished. The hooks will not run, unless there is a need for renewal.

# crontab -e
0 0 * * * /usr/local/bin/certbot-3.6 renew --quiet --pre-hook 'service nginx stop' --post-hook 'service nginx start'

Proxy

I use the nginx server to proxy requests to some services in other jails, instead of exposing more ports in my firewall.

This is a typical proxy in /usr/local/etc/nginx/nginx.conf:

server {
    listen 80;
    server_name service.example.com;
    location / {
        proxy_set_header   X-Real-IP        $remote_addr;
        proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
        proxy_set_header   Host             $host;

        proxy_pass http://10.0.1.94:3000/;

    }
}

See nginx documentation for fine tuning.

Plex in a FreeBSD jail

Kudos to https://nbari.com/post/plex-jail/.

For downloading metadata, Plex uses 127.0.0.1. Therefore the jail needs to have a loopback interface.

Create a lo1 interface. Add to rc.conf:

cloned_interfaces="lo1"

Add a made-up loopback IP in /etc/jail.conf for the appropriate jail:

plex {
    ip4.addr  = lo1|127.0.92.1;
    ip4.addr += em0|10.0.1.92;
}

This will add 127.0.92.1 as the main IP. This works as a specific setting for this jail, even though there is a lot of global stuff for the other jails.

Inside a jail, access to the loopback address 127.0.0.1 is redirected to the first IP address assigned to the jail. To make the jail loopback correspond with the new lo1 interface, that interface must be specified first in the list of interfaces and IP addresses given when creating a new jail.

Finally, edit /etc/hosts inside the jail so that localhost is mapped correctly:

127.0.92.1       localhost  localhost.your.domain

Profit.

Transmission torrent server

Create a new jail as described above.

Mount external folder into the jail by appending to /usr/local/jails/transmission.fstab:

/data/torrents     /usr/local/jails/transmission/mnt nullfs  rw  0 0  

Install transmission:

# pkg install transmission-daemon transmission-web

Take note of the group ID given to the group name transmission by the installation. In my case, it was 921.

Add the same group name with the same ID in the host machine for managing your data:

# pw groupadd transmission -g 921
# chgrp -R transmission /data/torrents

Start the transmission daemon upon jail startup (back inside the jail):

# echo 'transmission_enable="YES"' >> /etc/rc.conf
# service transmission start

Shut it down and configure transmission-daemon to allow incoming connections from other than localhost, by requiring RPC authentication and disable RPC whitelist. Change the umask from 18 to 2 (which will give rw to owner and group).

# service transmission stop

/usr/local/etc/transmission/home/settings.json:

"download-dir": "/mnt",
"rpc-authentication-required": true,
"rpc-bind-address": "0.0.0.0",
"rpc-password": "XXXXXXXXXXXXXXXXXXX",
"rpc-port": 9091,
"rpc-url": "/",
"rpc-username": "prd",
"rpc-whitelist-enabled": false,
"umask": 2,

To properly set the download-dir, add it as a flag to the transmission-daemon:

/etc/rc.conf:

transmission_download_dir="/mnt"

Start:

# service transmission start

Browse to http://a.b.c.d/web/ (remember the trailing slash) and start downloading/seeding!

mstream

A streaming music player written in node, https://mstream.io.

Create a new jail. Mount any music directories you would like inside, by editing /usr/local/jails/<jail>.fstab. Start the jail and enter it.

Install mstream using git and node package manager:

# pkg install git npm
# cd /usr/local/
# git clone https://github.com/IrosTheBeggar/mStream.git
# cd mStream
# npm install
# npm link 

Run the setup wizard:

# mstream --wizard

Start the server and try it out:

# mstream -j /usr/local/mStream/save/default.json

Move the config file to /usr/local/etc/ for consistency:

# mv /usr/local/mStream/save/default.json /usr/local/etc/mstream.conf

Create a new service shell script, pointing to your configuration.

/usr/local/etc/rc.d/mstream:

#!/bin/sh
#
# PROVIDE: mstream
# REQUIRE: LOGIN FILESYSTEMS
# KEYWORD: shutdown

. /etc/rc.subr

name=mstream
rcvar=mstream_enable

command="/usr/local/bin/mstream"

load_rc_config $name

# Set PATH so mstream can find node
PATH=$PATH:/usr/local/bin

#
# DO NOT CHANGE THESE DEFAULT VALUES HERE
# SET THEM IN THE /etc/rc.conf FILE
#
mstream_enable=${mstream_enable-"NO"}
pidfile=${mstream_pidfile-"/var/run/mstream.pid"}

run_rc_command "$1"

Make executable:

# chmod +x /usr/local/etc/rc.d/mstream

Enable in /etc/rc.conf and set the configuration flag:

mstream_enable="YES"
mstream_flags="-j /usr/local/etc/mstream.conf"

Reboot the jail. Since mstream doesn’t seem to have a real background daemon setting, it will eat your terminal if you service start mstream. Do it to verify that everything is working, but reboot if you would like it running in the background. It will eat your terminal if you jail -c mstream as well. I just close my tmux pane and carry on with my life…

This is something of an ugly hack, but it works for now…