Varnish, Nginx with SSL install under Ubuntu 16.04 / 18.04

Install Varnish 5.1 6 on Ubuntu 16.04 sound´s like easy. WTF? Not realy

Here is my configuration of Varnish with Nginx ssl on Ubuntu 16.04/18.04
With my configuration you don´t need adjust varnish port config. Feel free to adjust your Varnish config, increase your memory pool etc.

Update april 2018

*Please install directly Varnish 6 and no older Version *

Update July 2018

Notice for Ubuntu 18.04 it gives no deb package at the moment

you must use the xenial packages, see github comment

https://github.com/varnishcache/pkg-varnish-cache/issues/100

 

 

 


Install Varnish

First install Varnish  4.1 or 5.1 6

You can use Ubuntu repository

apt-get install varnish

Now you have Varnish 4.1 on the System

Or use these instructions to install Version 6

https://varnish-cache.org/releases/install_debian.html

So you will get Varnish 5.1 6

start and enable Varnish as service

sudo systemctl start varnish.service

sudo systemctl enable varnish.service

 


Trouble with systemd and varnish

Look at here when you in trouble with systemd and Varnish or below, its easy to solve.

THX to Mattia for this great blog post or the new solution from varnish below. Varnish solution testet with version 6.

http://deshack.net/how-to-varnish-listen-port-80-systemd/ (Link is no longer valid)

This Snippet is from Mattia´s blog post
The real problem

The official tutorial is a little bit outdated. Or, better, doesn’t take into account the testing version of Debian, which uses systemd instead of init.d as init system. And this makes a huge difference, explained in a bug report. Basically, /etc/default/varnish is only read by the /etc/init.d/varnish script, not by the systemd init script (/lib/systemd/system/varnish.service).

Now that we know this little detail not reported in the documentation, it’s easy for us to solve the problem.

The easy solution

All we have to do is override the systemd init script of varnish and change something.

# cp /lib/systemd/system/varnish.service /etc/systemd/system/
# nano /etc/systemd/system/varnish.service

We come up with something like this:

[Unit]
Description=Varnish HTTP accelerator

[Service]
Type=forking
LimitNOFILE=131072
LimitMEMLOCK=82000
ExecStartPre=/usr/sbin/varnishd -C -f /etc/varnish/default.vcl
ExecStart=/usr/sbin/varnishd -a :6081 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,256m
ExecReload=/usr/share/varnish/reload-vcl

[Install]
WantedBy=multi-user.target

Which is similar to what we saw before. We already now that we have to change the port passed as a value to the -a flag:

ExecStart=/usr/sbin/varnishd -a :80 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,256m

Why don’t just let this script read the /etc/default/varnish file? Because we don’t know, for now, what systemd or other init systems are going to do with default files. They could ignore them in the future, for example. Therefore, the safest solution for us is writing the full command in the script itself.

Problems: reloading Varnish

After discovering what explained above, we could consider the varnish default file completely unuseful. That’s not right. In any moment we could want to reload the Varnish configuration, this way:

# systemctl reload varnish.service

And here come other problems. That command runs the /usr/share/varnish/reload-vcl script, which reads the /etc/default/varnish file. This implies that we have to update both /etc/systemd/system/varnish.service and /etc/default/varnish in order to make Varnish work properly.


Solution from Varnish

Customize Varnish default config

https://varnish-cache.org/docs/trunk/tutorial/putting_varnish_on_port_80.html

/etc/systemd/system/varnish.service.d/customexec.conf

cd /etc/systemd/system/

mkdir varnish.service.d

vim customexec.conf

[Service]
ExecStart=
ExecStart=/usr/sbin/varnishd -a :80 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s default,1g

run systemctl daemon-reload and systemctl restart varnish

don´t edit your /etc/default/varnish changes there will not be affected to your varnish config. Edit your systemd conf file /etc/systemd/system/varnish.service.d/customexec.conf


Adjust some Varnish conf files.

/etc/varnish/default.vcl

adjust the port on the default.vcl or more when you need it. With Varnish 6 is default 8080.

Change it to 8080


#
# This is an example VCL file for Varnish.
#
# It does not do anything by default, delegating control to the
# builtin VCL. The builtin VCL is called when there is no explicit
# return statement.
#
# See the VCL chapters in the Users Guide at https://www.varnish-cache.org/docs/
# and https://www.varnish-cache.org/trac/wiki/VCLExamples for more examples.

# Marker to tell the VCL compiler that this VCL has been adapted to the
# new 4.0 format.
vcl 4.0;

# Default backend definition. Set this to point to your content server.
backend default {
.host = "127.0.0.1";
.port = "8080";
}

sub vcl_recv {
# Happens before we check if we have this in cache already.
#
# Typically you clean up the request here, removing cookies you don't need,
# rewriting the request, etc.
}

sub vcl_backend_response {
# Happens after we have read the response headers from the backend.
#
# Here you clean the response headers, removing silly Set-Cookie headers
# and other mistakes your backend does.
}

sub vcl_deliver {
# Happens when we have all the pieces we need, and are about to send the
# response to the client.
#
# You can do accounting or modifying the final object here.
}

look with properties Varnish is running

ps aux | grep varn
varnish 18342 0.0 5.3 118936 84584 ? SLs 08:53 0:00 /usr/sbin/varnishd -a :6081 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,256m
vcache 18350 0.0 5.5 339568 87252 ? Sl 08:53 0:01 /usr/sbin/varnishd -a :6081 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,256m
root 31137 0.0 0.0 11228 876 pts/6 S+ 09:47 0:00 grep –color=auto varn
root 31600 0.1 0.2 181600 3560 pts/4 Sl+ 09:09 0:04 varnishhist

Varnish is running you can test it with

your ip:6081

You will see a screen with a backend error, lets fix us this.


Install a webserver

Now we need the nginx web server or apache

apt-get install nginx

We need 3 config files

The first config file are redirect from port 80 to 443 and add www to your domain.

vim /etc/nginx/sites-available/redirect


server {
listen 80;
server_name example.com;
return 301 https://www.example.com$request_uri;
}

We need our main config file that will be run our website. I will serve a static html page for the moment.

vim /etc/nginx/sites-available/roundtrip


server {
listen 8080 default_server;
root /var/www/html;

server_name _;

location / {
# First attempt to serve request as file, then
# as directory, then fall back to displaying a 404.
try_files $uri $uri/ =404;
}

}

The last config file our ssl termination with nginx. I´m using let´s encrypt certs. You can use self signed cert´s or buy something.

vim /etc/nginx/sites-available/ssl


server {
listen 443 http2 ssl;

server_name www.exmaple.com exmaple.com;
ssl_certificate /etc/letsencrypt/live/www.exmaple.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/www.exmaple.com/privkey.pem;
location / {
proxy_pass http://127.0.0.1:6081;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-Port 443;
proxy_set_header Host $host;
}

}

lets activate our new nginx config


cd /etc/nginx/sites-enabled/

ln -s /etc/nginx/sites-available/redirect redirect
ln -s /etc/nginx/sites-available/roundtrip roundtrip
ln -s /etc/nginx/sites-available/ssl ssl

check the config

nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

service nginx restart

That´s it!

quic webserver and client

You don´t know what is quic?

 

Please read these excellent post

Google’s QUIC protocol: moving the web from TCP to UDP

You are using Chrome ? Open a new Tab and type  chrome://net-internals/#quic

Please Check your Firewall Port UDP/443 so that Traffic is Outgoing.

No time for read the blog post?

Podcast about quic and some other things

http://podcast.sysca.st/podcast/4-curl-libcurl-future-web-daniel-stenberg/

More Details for Chrome or Iron User

https://www.chromium.org/quic

You want run on your own Server?

the only webserver that can do that

https://github.com/mholt/caddy/wiki/QUIC

http://devsisters.github.io/goquic/

thx to Mattias Geniar 

 

 

NGINX traffic monitoring

Ich habe zwei Tools gefunden

ngxtop ein Python Tool , was relativ schnell Übersicht bringt

Monitor Nginx Real-time Metrics on Linux/Ubuntu in FREE

 

Sowie goaccess das auch mit fast allen Logfiles umgehen kann

http://vujade.co/monitoring-nginx/

https://goaccess.io/features

https://github.com/allinurl/goaccess

 

Subsonic und SSL NGINX

So jetzt habe ich es endlich hinbekommen. Ich habe mich mit dem Nginx Webserver doch etwas mehr rum ärgern dürfen.

wie man Subsonic unter Ubuntu installiert, steht hier beschrieben

This installation option applies to Debian Linux, including Ubuntu.

First, install Java: sudo apt-get install openjdk-6-jre
Download the Subsonic .deb package and install it: sudo dpkg -i subsonic-x.x.deb

Notice that the installer configures your system to start Subsonic automatically when booting. After installing, open the Subsonic web page on http://localhost:4040.

Change user

By default the Subsonic process is run as the root user. To improve security it is strongly advised that you run it as a dedicated user with less powerful privileges. You can specify this user by setting the SUBSONIC_USER variable in /etc/default/subsonic.

Note that non-root users are by default not allowed to use ports below 1024. Also make sure to grant the user write permissions in the music directories, otherwise changing album art and tags will fail.

Advanced configuration

To change the port number, Java memory settings or other startup parameters, edit the SUBSONIC_ARGS variable in /etc/default/subsonic. To restart Subsonic, do sudo service subsonic restart

Troubleshooting

Check the logs in /var/subsonic.

Damit ist der Subsonic Server installiert. Aktuell läuft er mit Root rechten. Das sollte schnellstens geändert werden.

Also legen wir noch einen neuen Benutzer unter Ubuntu an.

sudo useradd -r -s /bin/false USERNAME

Unter diesem Benutzer lassen wir dann Subsonic laufen.

Das tragen wir dann in der Subsonic Config Datei ein

sudo nano /etc/default/subsonic

# The user which should run the Subsonic process. Default “root”.
# Note that non-root users are by default not allowed to use ports
# below 1024. Also make sure to grant the user write permissions in
# the music directories, otherwise changing album art and tags will fail.

SUBSONIC_USER=UserName

Wenn wir schon in der Config Datei unterwegs sind, fügen wir auch noch gleich den SSL Support hinzu sowie etwas mehr RAM.

SUBSONIC_ARGS=”–https-port=8443 –max-memory=512″

Danach sollte dieser Benutzer noch Rechte im Verzeichnis geben wo die Musik liegt Beispiel /var/musik
Sonst klappt das Indizieren nicht und viele weitere Dinge nicht.

Jetzt geht es dran mit Port 8443 ist nicht so schön nach außen zu kommunizieren.
Ihr habt nun verschiedene Möglichkeiten.

IPTABLES
Wenn der Server daheim hinter einem Router steht Port von außen 443 und innen nach 8443 weiterleiten
NGINX SSL Proxy

Ich verwende letzte Variante

sudo apt-get install nginx

Danach ist der Webserver installiert.

Jetzt erstellt man unter /etc/nginx/sites-available/subsonic.conf eine Datei mit folgendem Inhalt.

server{
listen 443;

ssl_certificate /etc/nginx/ssl/server.crt;
ssl_certificate_key /etc/nginx/ssl/server.key;
ssl on;
server_name server.local;
root /usr/share/nginx/html;
index index.html index.htm;

location / {
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass https://localhost:8443;
}
}

Natürlich mit einem symlinc nach sudo ln -s /etc/nginx/sites-available/subsonic.conf /etc/nginx/sites-enabled/subsonic.conf und einem restart von nginx ist die Config nun scharf.

Hier nochmal an Rancor einen Dank. Da ich fast abgedreht hätte an diesem kleinen Config Schnippel. Ich hatte so einen Zusammen nur hatte ich da einen kleinen Fehler drin.

Damit ist der Subsonic Server unter https://server.local mit einer SSL Verbindung zu erreichen. Ohne Angaben irgendwelcher Port Nr. etc. Damit ist der Zugriff von fast allen Netzen in denen der Port 443 freigeben ist, zu erreichen.

Ich habe in der Config zwar ein Cert hinterlegt aber er nimmt doch das Standard Cert von Subsonic, was ich nicht will. Daher hinterlegen wir im Subsonic Server noch ein eigenes Cert.

Dazu erzeugen wir ein eigenes Cert.

sudo keytool -genkey -alias sitename -keyalg RSA -keystore subsonic.jks -keysize 4096 -validity 35600

Ihr könnt natürlich hier die Werte ändern wie ihr gerne möchtet, also nicht so lange gültig ist etc.

Wenn das Cert erstellt ist kopieren wir es nach
sudo cp subsonic.jks /etc/ssl/certs/

Dann editieren wir noch folgende Datei

sudo nano /usr/bin/subsonic

In Nano die alt + C Taste drücken, damit seht ihr die Zeilen
Ihr müsst in Zeile 113 springen und dort zwei Zeilen hinzufügen. Diese sind nicht vorhanden.

-Dsubsonic.ssl.keystore=/etc/ssl/certs/subsonic.jks
-Dsubsonic.ssl.password=123456

Danach noch den Subsonic neustarten.

sudo /etc/init.d/subsonic restart

Danach liefert Subsonic die SSL Verbindung mit eurem neuen SSL Cert aus.

Dazu gibt es noch einen alten Forums Eintrag wo es ähnlich beschrieben ist.

http://forum.subsonic.org/forum/viewtopic.php?f=2&t=7548