miniNAS initial setup

master
EmaMaker 2021-12-26 16:23:16 +01:00
commit cb3a5ff7a9
8 changed files with 548 additions and 0 deletions

88
README.md Normal file
View File

@ -0,0 +1,88 @@
# miniNAS Setup
Files for my miniNAS Server, a self-hosted server for different services.<br> Built from computers parts I had laying around, and a custom-built wooden case built from scrap wood.<br>
Currently running an AMD A10-5700 4/4 APU @3.4GHz, 2x4GB DDR3 RAM, 1TB and 500GB HDDs and a 120GB Kingston SSD<br>
miniNAS ships different services: nextcloud, gitea, searx and whatever I decide to add to it.<br>
This repo contains the file for configuring the server and an explanation of how they work
## Contents of the repo
* docker_compose: docker compose files for the different services. They use environment variables for volume directories and other things ([more info on environment variables](#env-vars)).
Plase those files wherever you want, but keep in mind that systemd entries will need to be changed accordingly. The directory where they are placed should follow this scheme: *compose_dir*/*service_name*/docker-compose.yml. For example gitea.yml should be placed in *compose_dir/gitea/ and renamed to docker-compose.yml
* systemd: systemd unit entries for starting the docker containers. Those need to be placed in /etc/systemd/system, then reload the daemon with
sudo systemctl daemon-reload
Now you can enable/start/disable/stop them. Use
sudo systemctl enable docker-compose@service_name --now
to enable and start the service on the spot
Also note that systemd entries use docker installed from snap (stored in /var/snap/docker). If using another distro, you'll have to change this. I used this as an example
https://gist.github.com/mosquito/b23e1c1e5723a7fd9e6568e5cf91180f
* gitea: contains app.ini config file for gitea. Place it in $GITEA_DATADIR/gitea/conf/
* nginx: contains nginx template. $NGINX_TMPL needs to point to it.
* ddclient: DDClient config script, location depends on the distro. Activate its systemd entry. DDclient updates the dns record with the public ip of the machine. I use google domains so it's configured for that, stripped down of the creds of course. Examples for different services can be found on the ddclient wiki.
## Backup strategy
I recently saw a [video about backups by Jeff Geerling](https://www.youtube.com/watch?v=S0KZ5iXTkzg) and I decided to apply his 3-2-1 rule, although mine looks a bit more scuffed. I keep all of my important files on my nextcloud instance, and use my gitea as my git service.
1) Nextcloud data lives on the 500GB HDD of miniNAS. All of it gets compressed and copied to the 1TB HDD on a daily basis. This way I have two copies on two different medias of the same data, and it can be easily restored by just unpacking the data and starting up a new instance
2) I use Nextcloud's sync client on my machines (mostly my desktop and laptop), which downloads the data locally on the machine it's used. I tend to sync all of my nextcloud data, although most times I sync just part of it, so most of my data as a couple more copies on two different machines.
3) I keep my projects on Nextcloud too, and every project has its own git repo which is then pushed to my own gitea. Gitea data lives on the 1TB drive of miniNAS, this means that I have two copies of my Gitea data on two different medias. I tend to sync my Projects on both my desktop and laptop, this means that I have a bit more redudancy for my projects too (see point 2)
4) My Gitea data gets mirrored to my GitHub periodically and automatically
5) Both my compressed nextcloud backups and my gitea data gets compressed and copied to an external 750GB drive periodically.
I'm still looking for an offsite backup solution for nextcloud that is cost-effective. After a friend's suggestion, I started looking into [IPFS](https://ipfs.io/).
## Notes on services:
### Nginx
* Taken from: https://github.com/gilyes/docker-nginx-letsencrypt-sample/blob/master/docker-compose.yml
and adapted to my needs.
* Ports: Each port needed by a service must be listed under "ports" in the docker-compose. The router needs to port forward the ports needed to the machine. Ports 3000 and 2222 are needed by gitea, ports 80 and 443 are needed for letsencrypt and https traffic and need to be always open. This service also takes care of creating new certificates for new subdomains when needed and renewing existing certificates. Ports needed by each service will need to be exposed in the corresponding docker-compose.yml
## Environment variables {#env-vars}
Environment variables can be placed wherever on the system, in a single text file. Then change *EnvironmentFile* in *docker-compose@.service* to point to the absolute path of the file you have created. <br>
These are the available environment variables:
### Docker
* **SERVER_DOCKER_NETWORKNAME**: name of the network. Create it with
docker network create my_network
### Letsencrypt (https/ssl certificates)
* **LETSENCRYPT_EMAIL**: email to send letsencrypt messages to
* **LETSENCRYPT_DOMAIN**: domain to request certificates for
### Certs
* **CERTS_DIR**: directory where letsencrypt certificates are stored, with owner UID 1000 and GID 1000. Generate them with
certbot certonly
then follow the on-screen instructions and copy them to CERTS_DIR
### Gitea
* **GITEA_DATADIR**: directory where gitea data will be stored
* **GITEA_CONFIGDIR**: directory where gutea config will be stored. Copy you app.ini here
### Nginx
* **NGINX_TMPL**: nginx template
### Nextcloud
* **NEXTCLOUD_DATADIR**: directory where nextcloud data is stored (files, apps, etc.)
* **NEXTCLOUD_DB_DATADIR**: directory where nextcloud database data is stored. Database is needed when restoring a backup
#### Autoconfiguration for nextcloud. More about it on [Nextcloud's official docker repo](https://github.com/nextcloud/docker)
* **NEXTCLOUD_ADMIN_USER**
* **NEXTCLOUD_ADMIN_PWD**
* **NEXTCLOUD_MYSQL_ROOTPWD**
* **NEXTCLOUD_MYSQL_PWD**

7
ddclient/ddclient.conf Normal file
View File

@ -0,0 +1,7 @@
ssl=yes
run_daemon=true
use=web, web='https://domains.google.com/checkip'
protocol=googledomains
login=''
password=''
emamaker.com #to be changed on the need

26
docker_compose/gitea.yml Normal file
View File

@ -0,0 +1,26 @@
version: "3.3"
services:
server:
image: gitea/gitea:latest
restart: always
volumes:
- $GITEA_DATADIR:/data
- $GITEA_CONFIGDIR:/etc/gitea
- /etc/timezone:/etc/timezone:ro
- /etc/localtime:/etc/localtime:ro
- $CERTS_DIR/fullchain.pem:/var/lib/gitea/cert.pem
- $CERTS_DIR/privkey.pem:/var/lib/gitea/privkey.pem
expose:
- "3000"
- "2222"
environment:
- VIRTUAL_HOST=gitea.$LETSENCRYPT_DOMAIN
- VIRTUAL_PORT=3000
- LETSENCRYPT_HOST=gitea.$LETSENCRYPT_DOMAIN
- LETSENCRYPT_EMAIL=$LETSENCRYPT_EMAIL
networks:
default:
external:
name: $SERVER_DOCKER_NETWORKNAME

View File

@ -0,0 +1,41 @@
version: '2'
services:
db:
image: mariadb:10.5
restart: always
command: --transaction-isolation=READ-COMMITTED --binlog-format=ROW
volumes:
- $NEXTCLOUD_DB_DATADIR:/var/lib/mysql
environment:
- MYSQL_ROOT_PASSWORD=$NEXTCLOUD_MYSQL_ROOTPWD
- MYSQL_PASSWORD=$NEXTCLOUD_MYSQL_PWD
- MYSQL_DATABASE=nextcloud
- MYSQL_USER=nextcloud
app:
image: nextcloud:latest
restart: always
expose:
- "80"
- "443"
links:
- db
volumes:
- $NEXTCLOUD_DATADIR:/var/www/html
environment:
# - NEXTCLOUD_ADMIN_USER=testuser #this is just for initial setup, and the user can be delete afterwards
# - NEXTCLOUD_ADMIN_PASSWORD=test123
- NEXTCLOUD_TRUSTED_DOMAINS=nextcloud.emamaker.com emamaker.com
- MYSQL_PASSWORD=$NEXTCLOUD_MYSQL_PWD
- MYSQL_DATABASE=nextcloud
- MYSQL_USER=nextcloud
- MYSQL_HOST=db
- VIRTUAL_HOST=nextcloud.$LETSENCRYPT_DOMAIN
- LETSENCRYPT_HOST=nextcloud.$LETSENCRYPT_DOMAIN
- LETSENCRYPT_EMAIL=$LETSENCRYPT_EMAIL
networks:
default:
external:
name: $SERVER_DOCKER_NETWORKNAME

45
docker_compose/nginx.yml Normal file
View File

@ -0,0 +1,45 @@
version: "2"
services:
nginx:
restart: always
image: nginx
container_name: nginx
ports:
- "80:80"
- "443:443"
- "3000:3000"
- "2222:2222"
volumes:
- "/etc/nginx/conf.d"
- "/etc/nginx/vhost.d"
- "/usr/share/nginx/html"
- $CERTS_DIR:/etc/nginx/certs:ro"
nginx-gen:
restart: always
image: jwilder/docker-gen
container_name: nginx-gen
volumes:
- "/var/run/docker.sock:/tmp/docker.sock:ro"
- "$NGINX_TMPL:/etc/docker-gen/templates/nginx.tmpl:ro"
volumes_from:
- nginx
entrypoint: /usr/local/bin/docker-gen -notify-sighup nginx -watch -wait 5s:30s /etc/docker-gen/templates/nginx.tmpl /etc/nginx/conf.d/default.conf
letsencrypt-nginx-proxy-companion:
restart: always
image: jrcs/letsencrypt-nginx-proxy-companion
container_name: letsencrypt-nginx-proxy-companion
volumes_from:
- nginx
volumes:
- "/var/run/docker.sock:/var/run/docker.sock:ro"
- "$CERTS_DIR:/etc/nginx/certs:rw"
environment:
- NGINX_DOCKER_GEN_CONTAINER=nginx-gen
networks:
default:
external:
name: $SERVER_DOCKER_NETWORKNAME

88
gitea/app.ini Normal file
View File

@ -0,0 +1,88 @@
APP_NAME = EmaMaker's Git
RUN_MODE = prod
RUN_USER = git
[repository]
ROOT = /data/git/repositories
[repository.local]
LOCAL_COPY_PATH = /data/gitea/tmp/local-repo
[repository.upload]
TEMP_PATH = /data/gitea/uploads
[server]
APP_DATA_PATH = /data/gitea
DOMAIN = gitea.emamaker.com
SSH_DOMAIN = gitea.emamaker.com
HTTP_PORT = 3000
ROOT_URL = https://gitea.emamaker.com/
DISABLE_SSH = false
SSH_PORT = 2222
SSH_LISTEN_PORT = 22
LFS_START_SERVER = true
LFS_CONTENT_PATH = /data/git/lfs
LFS_JWT_SECRET = 2qhrh0N_CRBXt6acnm_axI1ovLcQ7EgM6ZhfX_p30OI
OFFLINE_MODE = false
[database]
PATH = /data/gitea/gitea.db
DB_TYPE = sqlite3
HOST = localhost:3306
NAME = gitea
USER = root
PASSWD =
LOG_SQL = false
SCHEMA =
SSL_MODE = disable
CHARSET = utf8
[indexer]
ISSUE_INDEXER_PATH = /data/gitea/indexers/issues.bleve
[session]
PROVIDER_CONFIG = /data/gitea/sessions
PROVIDER = file
[picture]
AVATAR_UPLOAD_PATH = /data/gitea/avatars
REPOSITORY_AVATAR_UPLOAD_PATH = /data/gitea/repo-avatars
DISABLE_GRAVATAR = false
ENABLE_FEDERATED_AVATAR = true
[attachment]
PATH = /data/gitea/attachments
[log]
MODE = console
LEVEL = info
ROUTER = console
ROOT_PATH = /data/gitea/log
[security]
INSTALL_LOCK = true
SECRET_KEY = wJMrysZaDkpvOwdhPXGxYKGJiHykVD81hUn0WZeqRIZOYz0nl2d9taYBYseVHqBz
REVERSE_PROXY_LIMIT = 1
REVERSE_PROXY_TRUSTED_PROXIES = *
INTERNAL_TOKEN = eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJuYmYiOjE2NDA1MzA2NDV9.T_HmouqM-ogLMTOF8rZCTu0oBfI5MKav94lKgNJ-vlc
PASSWORD_HASH_ALGO = pbkdf2
[service]
DISABLE_REGISTRATION = true
REQUIRE_SIGNIN_VIEW = false
REGISTER_EMAIL_CONFIRM = false
ENABLE_NOTIFY_MAIL = false
ALLOW_ONLY_EXTERNAL_REGISTRATION = false
ENABLE_CAPTCHA = false
DEFAULT_KEEP_EMAIL_PRIVATE = false
DEFAULT_ALLOW_CREATE_ORGANIZATION = true
DEFAULT_ENABLE_TIMETRACKING = true
NO_REPLY_ADDRESS = noreply.localhost
[mailer]
ENABLED = false
[openid]
ENABLE_OPENID_SIGNIN = true
ENABLE_OPENID_SIGNUP = false

239
nginx/nginx.tmpl Normal file
View File

@ -0,0 +1,239 @@
{{ $CurrentContainer := where $ "ID" .Docker.CurrentContainerID | first }}
{{ define "upstream" }}
{{ if .Address }}
{{/* If we got the containers from swarm and this container's port is published to host, use host IP:PORT */}}
{{ if and .Container.Node.ID .Address.HostPort }}
# {{ .Container.Node.Name }}/{{ .Container.Name }}
server {{ .Container.Node.Address.IP }}:{{ .Address.HostPort }};
{{/* If there is no swarm node or the port is not published on host, use container's IP:PORT */}}
{{ else if .Network }}
# {{ .Container.Name }}
server {{ .Network.IP }}:{{ .Address.Port }};
{{ end }}
{{ else if .Network }}
# {{ .Container.Name }}
server {{ .Network.IP }} down;
{{ end }}
{{ end }}
# If we receive X-Forwarded-Proto, pass it through; otherwise, pass along the
# scheme used to connect to this server
map $http_x_forwarded_proto $proxy_x_forwarded_proto {
default $http_x_forwarded_proto;
'' $scheme;
}
# If we receive Upgrade, set Connection to "upgrade"; otherwise, delete any
# Connection header that may have been passed to this server
map $http_upgrade $proxy_connection {
default upgrade;
'' close;
}
gzip_types text/plain text/css application/javascript application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
log_format vhost '$host $remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent"';
access_log off;
{{ if (exists "/etc/nginx/proxy.conf") }}
include /etc/nginx/proxy.conf;
{{ else }}
# HTTP 1.1 support
proxy_http_version 1.1;
proxy_buffering off;
proxy_set_header Host $http_host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $proxy_connection;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $proxy_x_forwarded_proto;
client_max_body_size 10G;
# Mitigate httpoxy attack (see README for details)
proxy_set_header Proxy "";
{{ end }}
server {
server_name _; # This is just an invalid value which will never trigger on a real hostname.
listen 80;
access_log /var/log/nginx/access.log vhost;
return 503;
}
{{ if (and (exists "/etc/nginx/certs/default.crt") (exists "/etc/nginx/certs/default.key")) }}
server {
server_name _; # This is just an invalid value which will never trigger on a real hostname.
listen 443 ssl http2;
access_log /var/log/nginx/access.log vhost;
return 503;
ssl_session_tickets off;
ssl_certificate /etc/nginx/certs/default.crt;
ssl_certificate_key /etc/nginx/certs/default.key;
}
{{ end }}
{{ range $host, $containers := groupByMulti $ "Env.VIRTUAL_HOST" "," }}
upstream {{ $host }} {
{{ range $container := $containers }}
{{ $addrLen := len $container.Addresses }}
{{ range $knownNetwork := $CurrentContainer.Networks }}
{{ range $containerNetwork := $container.Networks }}
{{ if eq $knownNetwork.Name $containerNetwork.Name }}
## Can be connect with "{{ $containerNetwork.Name }}" network
{{/* If only 1 port exposed, use that */}}
{{ if eq $addrLen 1 }}
{{ $address := index $container.Addresses 0 }}
{{ template "upstream" (dict "Container" $container "Address" $address "Network" $containerNetwork) }}
{{/* If more than one port exposed, use the one matching VIRTUAL_PORT env var, falling back to standard web port 80 */}}
{{ else }}
{{ $port := coalesce $container.Env.VIRTUAL_PORT "80" }}
{{ $address := where $container.Addresses "Port" $port | first }}
{{ template "upstream" (dict "Container" $container "Address" $address "Network" $containerNetwork) }}
{{ end }}
{{ end }}
{{ end }}
{{ end }}
{{ end }}
}
{{ $default_host := or ($.Env.DEFAULT_HOST) "" }}
{{ $default_server := index (dict $host "" $default_host "default_server") $host }}
{{/* Get the VIRTUAL_PROTO defined by containers w/ the same vhost, falling back to "http" */}}
{{ $proto := or (first (groupByKeys $containers "Env.VIRTUAL_PROTO")) "http" }}
{{/* Get the HTTPS_METHOD defined by containers w/ the same vhost, falling back to "redirect" */}}
{{ $https_method := or (first (groupByKeys $containers "Env.HTTPS_METHOD")) "redirect" }}
{{/* Get the first cert name defined by containers w/ the same vhost */}}
{{ $certName := (first (groupByKeys $containers "Env.CERT_NAME")) }}
{{/* Get the best matching cert by name for the vhost. */}}
{{ $vhostCert := (closest (dir "/etc/nginx/certs") (printf "%s.crt" $host))}}
{{/* vhostCert is actually a filename so remove any suffixes since they are added later */}}
{{ $vhostCert := trimSuffix ".crt" $vhostCert }}
{{ $vhostCert := trimSuffix ".key" $vhostCert }}
{{/* Use the cert specified on the container or fallback to the best vhost match */}}
{{ $cert := (coalesce $certName $vhostCert) }}
{{ $is_https := (and (ne $cert "") (exists (printf "/etc/nginx/certs/%s.crt" $cert)) (exists (printf "/etc/nginx/certs/%s.key" $cert))) }}
{{ if $is_https }}
{{ if eq $https_method "redirect" }}
server {
server_name {{ $host }};
listen 80 {{ $default_server }};
access_log /var/log/nginx/access.log vhost;
return 301 https://$host$request_uri;
}
{{ end }}
server {
server_name {{ $host }};
listen 443 ssl http2 {{ $default_server }};
access_log /var/log/nginx/access.log vhost;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers 'ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS';
ssl_prefer_server_ciphers on;
ssl_session_timeout 5m;
ssl_session_cache shared:SSL:50m;
ssl_session_tickets off;
ssl_certificate /etc/nginx/certs/{{ (printf "%s.crt" $cert) }};
ssl_certificate_key /etc/nginx/certs/{{ (printf "%s.key" $cert) }};
{{ if (exists (printf "/etc/nginx/certs/%s.dhparam.pem" $cert)) }}
ssl_dhparam {{ printf "/etc/nginx/certs/%s.dhparam.pem" $cert }};
{{ end }}
{{ if (ne $https_method "noredirect") }}
add_header Strict-Transport-Security "max-age=31536000";
{{ end }}
{{ if (exists (printf "/etc/nginx/vhost.d/%s" $host)) }}
include {{ printf "/etc/nginx/vhost.d/%s" $host }};
{{ else if (exists "/etc/nginx/vhost.d/default") }}
include /etc/nginx/vhost.d/default;
{{ end }}
location / {
{{ if eq $proto "uwsgi" }}
include uwsgi_params;
uwsgi_pass {{ trim $proto }}://{{ trim $host }};
{{ else }}
proxy_pass {{ trim $proto }}://{{ trim $host }};
{{ end }}
{{ if (exists (printf "/etc/nginx/htpasswd/%s" $host)) }}
auth_basic "Restricted {{ $host }}";
auth_basic_user_file {{ (printf "/etc/nginx/htpasswd/%s" $host) }};
{{ end }}
{{ if (exists (printf "/etc/nginx/vhost.d/%s_location" $host)) }}
include {{ printf "/etc/nginx/vhost.d/%s_location" $host}};
{{ else if (exists "/etc/nginx/vhost.d/default_location") }}
include /etc/nginx/vhost.d/default_location;
{{ end }}
}
}
{{ end }}
{{ if or (not $is_https) (eq $https_method "noredirect") }}
server {
server_name {{ $host }};
listen 80 {{ $default_server }};
access_log /var/log/nginx/access.log vhost;
{{ if (exists (printf "/etc/nginx/vhost.d/%s" $host)) }}
include {{ printf "/etc/nginx/vhost.d/%s" $host }};
{{ else if (exists "/etc/nginx/vhost.d/default") }}
include /etc/nginx/vhost.d/default;
{{ end }}
location / {
{{ if eq $proto "uwsgi" }}
include uwsgi_params;
uwsgi_pass {{ trim $proto }}://{{ trim $host }};
{{ else }}
proxy_pass {{ trim $proto }}://{{ trim $host }};
{{ end }}
{{ if (exists (printf "/etc/nginx/htpasswd/%s" $host)) }}
auth_basic "Restricted {{ $host }}";
auth_basic_user_file {{ (printf "/etc/nginx/htpasswd/%s" $host) }};
{{ end }}
{{ if (exists (printf "/etc/nginx/vhost.d/%s_location" $host)) }}
include {{ printf "/etc/nginx/vhost.d/%s_location" $host}};
{{ else if (exists "/etc/nginx/vhost.d/default_location") }}
include /etc/nginx/vhost.d/default_location;
{{ end }}
}
}
{{ if (and (not $is_https) (exists "/etc/nginx/certs/default.crt") (exists "/etc/nginx/certs/default.key")) }}
server {
server_name {{ $host }};
listen 443 ssl http2 {{ $default_server }};
access_log /var/log/nginx/access.log vhost;
return 500;
ssl_certificate /etc/nginx/certs/default.crt;
ssl_certificate_key /etc/nginx/certs/default.key;
}
{{ end }}
{{ end }}
{{ end }}

View File

@ -0,0 +1,14 @@
[Unit]
Description=%i service with docker compose
Requires=snap.docker.dockerd.service
After=snap.docker.dockerd.service
[Service]
Type=oneshot
RemainAfterExit=true
WorkingDirectory=/etc/compose/%i
ExecStart=/snap/bin/docker-compose up -d
ExecStop=/snap/bin/docker-compose down
[Install]
WantedBy=multi-user.target