Re: Preserving Source IP using SSL Preread + OpenVPN

i managed to do this, but not with nginx alone:

My setup:

Single Server with OpenVPN and HTTPS both on Port 443.
Additionally: since many firewalls with deep-packet-inspection (DPI) will block the OpenVPN Traffic even when on Port 443 i wanted a solution for OpenVPN where all packets _can_ be layered within regular SSL Traffic, which will look like Plain HTTPS to a DPI-Firewall. This special scenario requires that the client-device will layer all packets within SSL, this can be done using "stunnel" or "ssldroid" but is not covered here.


So my Port 443 will go by these rules:

Input on Port 443 -> is it OpenVPN Traffic? -> Keep real remote IP -> Forward to Port 1194 [OpenVPN Daemon]
Input on Port 443 -> is it non OpenVPN Traffic? -> is the requested Domain "sslvpn.mydomain.com"? -> Unpack (remove SSL-Layer) Traffic -> Forward to Port 1194 [OpenVPN Daemon]
Input on Port 443 -> is it non OpenVPN Traffic? -> is the requested Domain NOT "sslvpn.mydomain.com"? -> Keep real remote IP -> Forward to Port 4443 [Nginx Vhosts]

I managed to keep the real remote IPs in all cases but the one where i need the layering of OpenVPN-Packets within SSL. This is the only flaw, but i can live with it since usually i only use OpenVPN on Port 443 directly and need my Webroots up and running.

Tools i used:
OpenVPN 2.4.6
Nginx 1.12.2
sslh 1.18

While Nginx has the ability of forwarding the realip, this is only true for the proxy_protocol, which many applications don't support.
sslh can transparently forward traffic while keeping the realip using netfilter.

Important: in the version i used, sslh has to be configured on IP-Addresses bound to Interfaces other than localhost. Do not use 127.0.0.1, this will not work! In my setup, the server has the IP 192.168.1.251.

sslh is started with these params:
# sslh -p 192.168.1.251:443 --openvpn 192.168.1.251:1194 --anyprot 192.168.1.251:9443 --transparent

This means, sslh will listen on port 443 and redirect openvpn traffic to 1194 (OpenVPN Daemon) and everything else to 9443 (one of the nginx listening-ports).
The transparent parameter will initiate the netfilter/caps module which is important.

For this to work we need some rules for iptables:

# iptables -t mangle -N SSLH
# iptables -t mangle -A OUTPUT --protocol tcp --out-interface eth0 --sport 1194 --jump SSLH
# iptables -t mangle -A OUTPUT --protocol tcp --out-interface eth0 --sport 9443 --jump SSLH
# iptables -t mangle -A SSLH --jump MARK --set-mark 0x1
# iptables -t mangle -A SSLH --jump ACCEPT
# ip rule add fwmark 0x1 lookup 100
# ip route add local 0.0.0.0/0 dev lo table 100

Replace "eth0" with whatever your interface is called.

The OpenVPN-Server config is not special, it's only required that it is running on Port 1194 (or you have to adjust the config above) and that it is listeing in TCP-Mode.

Now to the nginx part (snippets):

stream {
map $ssl_preread_server_name $name {
sslvpn.mydomain.com sslvpn_backend;
default https_backend;
}

server {
listen 9443;
proxy_pass $name;
proxy_connect_timeout 300s;
proxy_timeout 300s;
proxy_protocol on;

ssl_preread on;
}

upstream sslvpn_backend {
server 127.0.0.1:8443;
}

server {
listen 127.0.0.1:8443 ssl proxy_protocol so_keepalive=on;

ssl_certificate /etc/letsencrypt/live/sslvpn.mydomain.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/sslvpn.mydomain.com/privkey.pem;

proxy_protocol off;
proxy_connect_timeout 300s;
proxy_timeout 300s;
proxy_pass openvpn-ssltunnel-inet-only;
}

upstream openvpn-ssltunnel-inet-only {
# route traffic back to openvpn
server 127.0.0.1:1194;
}

upstream https_backend {
server 127.0.0.1:4443;
}
}


The first block will initiate listening on port 9443. There i turn on the proxy_protocol (important to keep the real-ip) and also start "ssl_preread", to inspect the requested domain.
The mapping is quite easy: if the domain was "sslvpn.mydomain.com" use the "sslvpn_backend", in all other cases use the default "https_backend".

The sslvpn_backend just redirects the traffic to port 8443 where the server is a simple stream server with ssl-layering, but also disables the proxy protocol. afterwards the traffic is routed back to port 1194, where the OpenVPN-Daemon can now parse the unpacked traffic.

The "https_backend" is just routing all traffic to the destination 4443 where all "real" https-vhosts of nginx reside.

A typical vhost looks like this:

http {
server {
listen 127.0.0.1:4443 ssl http2 proxy_protocol;

set_real_ip_from 127.0.0.1;
real_ip_recursive on;
real_ip_header proxy_protocol;

include includes/ssl.conf;
ssl_certificate /etc/letsencrypt/live/www.mydomain.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/www.mydomain.com/privkey.pem;

server_name www.mydomain.com;

access_log /var/log/nginx/www_mydomain_com.access_log main;
error_log /var/log/nginx/www_mydomain_com.error_log info;

root /var/www/mydomain/htdocs;

try_files $uri $uri/ /index.php?$args;

include includes/expires.conf;

include includes/php-fpm.conf;
}

}

The vhost will listen on 4443, activate the submodules ssl and proxy_protocol.
Within the vhost the realip will be decoded (available via proxy_protocol).
That way, within the logs (access and error) the real-remote-ip will be visible and even within PHP in the $_SERVER['REMOTE_ADDR''] variable.

I hope this setup can help someone like me.

Best regards,
David