DNS over HTTPS on Ubiquity USG

If you wish to run your DNS queries on  over HTTPS to Cloudflare’s using Ubiquity USG,  then read on.

The first step is to compile and install cloudflared. I used a mac and and docker to compile the binary from github.

docker run --rm -v "$PWD":/usr/src/myapp -w /usr/src/myapp -e GOOS=linux -e GOARCH=mips golang bash -c "go get -v github.com/cloudflare/cloudflared/cmd/cloudflared; GOOS=linux GOARCH=mips go build -v -x github.com/cloudflare/cloudflared/cmd/cloudflared"

After compiling – you’ll end up with a file called cloudflared on your local machine. Copy this over to the USG in the /confi/scripts/ folder. You could use SCP, or a GUI based client for Mac like Transmit.

Login to the USG via shell, then change sudo to su . Change permission to make the file executable.

sudo su
chmod +x /config/scripts/cloudflared

Create an auto startup script and place in config/scripts/post-config.d

vi /config/scripts/post-config.d/cloudflare-dns.sh



# start DNS proxy to Cloud Flare

/usr/bin/pkill cloudflared

nohup /config/scripts/cloudflared --no-autoupdate --proxy-dns --proxy-dns-port 5053 &>/var/log/cloudflared.log &

Change permission to make the file executable.

chmod +x /config/scripts/post-config.d/cloudflare-dns.sh

Create a new file using a text editor such as TextEdit or Atom on your mac and create the following json file. The structure of a json file is just as important as the words themselves. Incorrect placement of brackets, indentations, line breaks or any other structural element will make the json file invalid. Save it by naming it config.gateway.json


        "service": {
                "dns": {
                        "forwarding": {
                                "options": [

Now copy this file to your Controller under the following folder (You could use SCP, or a GUI based client for Mac like Transmit. )  – On Cloud Key install the path for the .json file is: /srv/unifi/data/sites/[site name/default]/

Read here for more information on the config.gateway.json file -https://help.ubnt.com/hc/en-us/articles/215458888-UniFi-USG-Advanced-Configuration

Once done, head over to your controller dashboard , select devices, select USG, select config, select manage device, select force provision.

Once the provisioning completes, validate that the DNS on your machine is pointing to the USG IP. Test your DNS by visiting the following IP on your mac.



Screenshot 2018-10-22 at 9.11.24 PM

If doesn’t work delete the 2 lines in red in config.gateway.json and force reprovision. 


So apparently, after a reboot of the USG, I started seeing errors in the logs stating that a secure connection to cannot be made. I first thought the ISP is bloking the connection but later realized that my WAN IP perisited after a reboot.  I tried mutiple combinations, but the following worked

<Kill the process>
ps -ef| grep cloudflared
kill -9 <Process ID of cloudflared from command above>

<remove startup script>
rm -rf /config/scripts/post-config.d/cloudflare-dns.sh

<reboot USG>
<start cloudflared>
nohup /config/scripts/cloudflared --no-autoupdate --proxy-dns --proxy-dns-port 5053 &>/var/log/cloudflared.log &
<validate no errors in /var/log/cloudflared>

<Reset contents of config.gateway.json to the one above - which means if you deleted the text in RED put it back>
<Force provision USG again>
<Test - https://www.cloudflare.com/ssl/encrypted-sni/>
<If it still doenst work, delete 2 RED lines in config.gateway.json>






DIY Home Firewall – Part 1 Sophos XG

I have been a  ASUS fan for a long time. Had been running AC-87U for a while at my house. I had the Merlin firmware on it with a Openvpn client to my VPN provider. Served my needs for a few years before I realized that  I wasn’t  able to run a secure VPN server on it/ behind it.  I used a Synology NAS behind the router to host a VPN server and connect to it when I was outside the network. And in no time the NAS was hacked 🙂 – how did I know ? Well the light’s wont stop blinking even when I’d shut down all packages. I Ran a packet capture  and sure enough it was running a crypto mining malware. Anyways more on that later.

I looked around and a few options presented themselves for a home firewall

  1. Sophos XG UTM – An Enterprise grade true UTM with features like IPS, AV, Firewall, Anti-Spam etc. A very nice GUI and a forever Free Home License. (https://demo.sophos.com/webconsole/webpages/login.jsp – user:demo pass:demo)
  2. Pfsense – An open source firewall with ‘packages’ that you can configure to get near to a UTM performance. Widely supported and a range of tutorials available on the web.
  3. Opnsense – A fork from Pfsense  with a slicker GUI and some improvements/enhancements from Pfsense
  4. Untangle – Marketed towards SME’s with extensive reporting features, a home license is 50$ a year. You could also get a community version without a charge. (http://demo.untangle.com/admin/index.do)
  5. Clear OS – While not exactly a firewall, ClearOS is an operating system for servers, gateways and firewalls. It delivers applications through a marketplace to enable SMB’s to consume services without the need for multiple hardware. They offer an online demo here – https://www.clearos.com/clearos-live-demo
  6. Ubiquiti USG – An appliance based firewall, delivers excellent value for money and can be controlled via an intutive portal. Offers fancy dashboard for reporting based on Deep Packet Inspection (DPI)

My internet connection at home is a 1G Fiber broadband. The fiber terminates on an ISP provided ONT box, which acts as a layer 2 device. The ONT gives a RJ-45 handoff.

So with my needs clear, 1G throughput and a couple of RJ45 ports (one for LAN and one for WAN), SSLVPN server – i started searching.

I eliminated the Ubiquiti USG because it doesn’t offer an inbuilt SSL VPN server and the people on the forums reported throughput issues on the smaller USG-3P device. The 4P device looked too big for my house and so i skipped it.

Untangle was easy to eliminate as I wasn’t too fond of the GUI (personal choice) . ClearOS was also eliminated as it looked too heavy for my requirements. So choice was down to Sophos XG, Pfsense and Opnsense.

I ended up selecting Sophos XG as the UI looked polished and components looked easy to setup (this would change later, as I moved to Opnsense ultimately). Part of it was also because I wanted somethign that would just work, without having to spend a great deal of time on it (trying to be lazy).

For the hardware, i zeroed it down to a Qotom 335G4 box from China. It has an i5 processor with 4 cores , upto 8 GB RAM, support for AES-NI and most importantly all Intel NIC’s (apparently all platforms XG, pfsense and opnsense reported issues with RealTek NIC’s). I ordered it with 8GB RAM and a 60GB SSD (came installed as a msata drive) . It also has a SATA connecter – so you can hook up a SATA SSD to the box later.

Ordered the box from Alibaba and it was with me in a couple of days.

Q300G4 Ports 800

The install was a breeze.I got the ISO from the Sophos XG site after registering. I used Etcher to burn the ISO to a USB drive. Connected the box to power and an HDMI monitor. To boot from the USB, hit DEL during startup (when you see the boot logo). Continue with the onscreen instructions, till you device reboots from the inbuilt storage.

The only confusing part is that XG doest not recognise the ports in order (same for OPNsense while PfSense recognised them in order)

For XG (default config) - The LAN Port is the one on the extreme left and WAN is the 3rd port from left.
For OPNsense - The LAN Port is the one one extreme left and WAN is the one next to it (or second from the left).
For pfswense - The first port from left is WAN and the one next to it is LAN (or the second from left)

So you would want to connect your PC to the extreme left port and wait for the DHCP to assign you an IP. Once connected the default IP for the web admin is You can change this after you run the initial setup. The initial setup is nothing too complicated and will get you up and running in about 5 minutes. The initial config also has a default rule allowing all outbound LAN communication and blocking all inbound.

I reconfigured the ASUS router under AP mode and in no time, I had a functional UTM on. I suppose the entire thing took about 20 minutes and I was back on the internet – sweet !.

Once you are on and connected, here are few things you can do to complete your setup

  1. Define DHCP pool and create static MAC to DHCP bindings
  2. Create Users or clienteles users
  3. Enable user authentication ( I do this only for the guest network). All other devices have a static MAC to IP binding.
  4. Enable SSL VPN
  5. Enable IPS policies and AV
  6. Create custom firewall rules
  7. Create schedules for internet access
  8. Enable content filtering
  9. Create a VLAN for separating IOT traffic
  10. Add more firewall policies for IOT to WAN communication; LAN to IOT communication
  11. Enable multicast broadcast across VLAN’s for DLNA access from one VLAN to the other.

Sophos has some pretty good documentation on this and the forums are very helpful ( the guys are awesome !)

With IPS and and content filtering on – I get about 300Mbps on the wireless network (more than enough for my needs). Sophos uses snort which is single threaded which explains that it doesn’t go beyond 300Mbps on a single connection. If you have a few machines running in parallel or if you iperf3 to a public perf server with multi-thread you can go upto 700Mbps in my case. Under normal loads with about 20 devices the CPU stays under 5%.

To conclude , if you are looking at a home firewall solution to replace your router and you want things to just work (limited customization) SophosXG offers a very compelling UTM in a box solution – worth a try !

End – for now

P:S – I’ll probably do a Sophos vs Pfsense vs Opnsesne post later.

Raspberry PI VPN Gateway with ExpressVPN

Although I run a OPNsense box as a firewall, I still use a raspberrypi connected to my LAN as my VPN gateway for my devices. The PI is a low cost hardware and delivers decent VPN performance for my LAN (~30Mbps).  Alternatively I could have the OPNsense box run a VPN client but I decided against it keeping my requirements in mind. Part of it was also because ExpressVPN provides a very easy PI package to connect to VPN servers. If you are out to configure the PI to act as a VPN client for you LAN devices read on.

Assuming you already have the PI setup with an IP address and connected to the network (I have PI connected to the local network through a wire – eth0). You would do the following configuration.

Part 1 – Setup PI to Forward traffic to VPN.

Enable IP Forwarding for the PI to forward packets

nano /etc/sysctl.conf
net.ipv4.ip_forward = 1

Enable NAT for packets received on LAN to the tunnel.

sudo iptables -t nat -A POSTROUTING -o tun0 -j MASQUERADE
sudo iptables -A FORWARD -i tun0 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
sudo iptables -A FORWARD -i eth0 -o tun0 -j ACCEPT

To make the rules persistent use

sudo apt-get install iptables-persistent

Apply the rules at startup

sudo apt-get install iptables-persistent

Part 2 – Setup Express VPN

Download the ExpressVPN installer file to pi. Install and activate using the command

sudo dpkg -i expressvpn_1.2.0_armhf.deb
expressvpn activate

You get the activation code from the Downloads page on ExpressVPN when you download the ExpressVPN installer for PI.

List servers to connect to

expressvpn list

Connect to a particular server and to check the status

expressvpn connect ussc
expressvpn status

Disconnect from server

expressvpn disconnect

Connect to the last connected server

expressvpn connect

At this point, you should have the PI connected to the VPN and ready to accept connections from your LAN devices. All that is left to do is to use the PI’s IP as the default gateway for devices that need a VPN connection (in my case, I point the Apple TV to use the VPN)

Lan is -
Firewall -
PI -
Apple TV -; Default GW

You could also set the PI to provide your the ExpressVPN DNS to your LAN devices by

sudo apt-get install dnsmasq

This ensures your DNS queries do not go leaking to your ISP. Don’t forget to set the DNS on your devices that need a VPN connection to the PI IP.

End – for now 🙂

The Leaking DNS in Ubuntu 18.04 and OpenVPN

So  finally i got to installing Ubuntu LTS 18.04 on my work PC. Have been toying with Linux for a number of years but never took the plunge.  The place where i work forces us to use Vmware Horizon as a VDI solution to access to our work machines, so the operating system doesn’t really matter.  Gives me a perfect stage !

Anyways the short story is – the install went smooth and I managed to get Vmware Horizon working on my machine without any major issues. I routinely use a lot of SSL VPN to my home to access my files and also to tunnel my traffic back home when i am on a public WiFi so i went about setting that up.

Install OpenVPN

sudo apt install openvpn

I already had the .ovpn file from the VPN server. It reads

cipher AES-256-CBC
auth SDHA512
reneg-sec 0
remote abc.com 1194 udp
lport 0
verify-x509-name "SSLVPN CA"name
remote-cert-tls server
comp-Izo adaptive

To connect to the VPN simply run the following.

sudo openvpn –config <path to the ovpn file> –dev tun0

sudo openvpn --config profile.ovpn --dev tun0

If you get an error , double check your ovpn profile and the error message. Sometimes the profile is not in the format expected by ovpn (check aginst my profile above)

You can also use the GUI to import the profile and connect to the VPN. Install the GUI by issuing the following command.

sudo apt-get install openvpn network-manager-openvpn network-manager-openvpn-gnome

To connect to the VPN – click on the network icon on top right of the menu bar and then goto VPN. Import the ovpn profile and reboot the pc.

Happy that everything works as intended, i went on to verify that all my traffic was routed via the tunnel and that i was able to access the home network. To verify that my DNS queries were also sent via the tunnel i went to http://dnsleak.com and behold – my DNS was leaking.

What does that mean ?That means that my DNS requests were being sent to the the sites through the local DNS server and not through the VPN.

 To check what was happening, I went to read the contents of /etc/resolv.conf. There i was directed to see systemd-resolve status to find out what DNS servers were in use. systemd-resolved is a system service that provides network name resolution to local applications.

systemd-resolve --status

I could see that my home DNS server was listed as the DNS resolver under the tunnel interface and the local DNS server was listed as a resolver under the physical interface (WiFi). Reading around a bit i found that by default, systemd-resolved queries all interfaces for DNS resolutions. If you want to avoid dns leak, it seems you need to force it to only use the link created by Openvpn.

To do that, we make use of a helper script called openvpn-systemd-resolved.

sudo apt install openvpn-systemd-resolved

Now add the following lines to the OpenVPN configuration file (.opvn from before)

script-security 2
up /etc/openvpn/update-systemd-resolved
down /etc/openvpn/update-systemd-resolved
dhcp-option DNSSEC allow-downgrade
dhcp-option DOMAIN-ROUTE .

<the ‘.’ above is very important. If you miss it, the DNS queries will continue to leak>

  1. script-security 2 – this directive offers policy-level control over OpenVPN’s usage of external programs and scripts. Script security level 2 allows calling of built-in executables and user-defined scripts.
  2. up and down directives specify the commands to run after successful TUN/TAP device open and close, in this case, which are /etc/openvpn/update-resolv-conf and /etc/openvpn/update-resolv-conf correspondingly. These scripts align the resolv.conf with the DNS address supplied by OpenVPN server, returning it to its original state when the connection is closed.
  3. dhcp-option DOMAIN-ROUTE means for querying a particular domain use this DNS server. Using a . dot/period instead of a specific domain name matches all domain names and sends all queries out to the VPN.

Restart openvpn and Validate by going to http://dnsleak.com. Now my systemd-resolve –status reads.

Link 16 (tun0)
Current Scopes: DNS
LLMNR setting: yes
MulticastDNS setting: no
DNSSEC setting: allow-downgrade
DNSSEC supported: yes
DNS Servers:
DNS Domain: ~.

Link 2 (wlp2s0)
Current Scopes: DNS
LLMNR setting: yes
MulticastDNS setting: no
DNSSEC setting: no
DNSSEC supported: no
DNS Servers:

This seems to only work if i launch the VPN from the command line. If I use the network manager it doesn’t work.

The End — For Now 🙂