Highly Available OpenVPN connection between two offices

When you connect two (or more) offices over internet using VPN,
you depend heavily both on your router’s and service provider’s reliability.
In perfect world, routers never break and provider links never go down.
I wish we all lived in a perfect world, but we all live in a slightly different place.
To create a solid infrastructure, which will not betray you in a decisive moment,
you must always remember that every single component is potentially unreliable,
and you must avoid SPOFs (single points of failure) at any cost.

In this scope typical SPOFs are your router and your internet provider’s link.
There are also other SPOFs, like switches and UPSes, but they are far more trivial,
and will be only briefly discussed at the end of this article.

Before starting a main part I would like to thank Cartman,
who not only motivated me, but also done major part of the job
by shaping my style and providing me with real-world working examples.
This is more his story than mine, nevertheless he generously allowed me
to publish it under my name.

Preface
There is only one way to learn how to create highly-available networks:
you must create one yourself and test it, test it, test it…

Imagine we have a company SnakeVenom INC with 2 imaginary offices,
one office is HQ in Canada, and another is Venom Research Lab in Zimbabwe.
For convenience lets call HQ OfficeA and Venom Lab OfficeB.
To achieve real high availability we must have two routers and two ISPs links
in each office and use following [software] technologies to gain an advantage
from redundant hardware:

• VRRP – while one router dies switch to another.
• WAN LoadBalancing – balance load between ISP links.
• OpenVPN – create encrypted tunnels between offices.
• OSPF – dynamically build routes between office private networks (over VPN).

Take a look at the illustration above:

In OfficeA, Canada, we have two routers, oa-rt-01 and oa-rt-02,
each has links from two [imaginary] providers, BigBoy Systems and FatCat Telecom.
In OfficeB, Zimbabwe, we also have two routers: ob-rt-01 and ob-rt-02,
each also has two links from two providers, Mugabe Broadband and Zulu Networks.

Each router has 3 ethernet interfaces: eth0 and eth1 for WAN, eth2 for LAN.

BigBoy Systems has network 1.1.1.0/24 and gateway 1.1.1.101.
oa-rt-01 is connected to BigBoy with eth0 1.1.1.201, oa-rt-02 with eth0 1.1.1.202.
FatCat Telecom has network 2.2.2.0/24 and gateway 2.2.2.101.
oa-rt-01 is connected to FatCat with eth1 2.2.2.201, oa-rt-02 with eth1 2.2.2.202.
Mugabe Broadband has network 3.3.3.0/24 and gateway 3.3.3.101.
ob-rt-01 is connected to Mugabe with eth0 3.3.3.201, ob-rt-02 with eth0 3.3.3.202.
Zulu Networks has network 4.4.4.0/24 and gateway 4.4.4.101.
ob-rt-01 is connected to Zulu with eth1 4.4.4.201, ob-rt-02 with eth1 4.4.4.202.

oa-rt-01 and oa-rt-02 are connected to OfficeA local network 10.0.100.0/24
with eth2 having shared virtual VRRP address 10.0.100.254.
ob-rt-01 and ob-rt-02 are connected to OfficeB local network 10.0.200.0/24
with eth2 having shared virtual VRRP address 10.0.200.254.

I think explanations is enough and there is no need to write more empty text.
Comrades, fasten your seat belts, we are going to make some real action here!

Initial setup of the routers
Configuration of office router includes many things like DHCP server, Firewall, QoS, etc…
I will not include all these needed but trivial things, rather concentrate on article specifics:
VRRP, WAN LoadBalancing, OpenVPN and OSPF.
Lets load pre-created config.boot files from GitHub:
oa-rt-01: Download!
oa-rt-02: Download!
ob-rt-01: Download!
ob-rt-02: Download!
These files contain all needed stuff, except specific things we will set up ourselves now.
We must download them to /opt/vyatta/etc/config/config.boot on respective routers
and than reboot each router afterwards.
NB!
If you will follow the steps of this scenario, you must have your interface and network setup
similar to mine, or you must change configuration settings according to your own scenario!

VRRP
oa-rt-01
configure
set interfaces ethernet eth2 vrrp vrrp-group 10 priority 150
set interfaces ethernet eth2 vrrp vrrp-group 10 virtual-address 10.0.100.254
set interfaces ethernet eth2 vrrp vrrp-group 10 description "LAN VRRP group A"
set interfaces ethernet eth2 vrrp vrrp-group 10 advertise-interval 1
set interfaces ethernet eth2 vrrp vrrp-group 10 authentication type ah
set interfaces ethernet eth2 vrrp vrrp-group 10 authentication password s0mEpa5s
commit
save

oa-rt-02
configure
set interfaces ethernet eth2 vrrp vrrp-group 10 priority 100
set interfaces ethernet eth2 vrrp vrrp-group 10 virtual-address 10.0.100.254
set interfaces ethernet eth2 vrrp vrrp-group 10 description "LAN VRRP group A"
set interfaces ethernet eth2 vrrp vrrp-group 10 advertise-interval 1
set interfaces ethernet eth2 vrrp vrrp-group 10 authentication type ah
set interfaces ethernet eth2 vrrp vrrp-group 10 authentication password s0mEpa5s
commit
save

ob-rt-01
configure
set interfaces ethernet eth2 vrrp vrrp-group 20 priority 150
set interfaces ethernet eth2 vrrp vrrp-group 20 virtual-address 10.0.200.254
set interfaces ethernet eth2 vrrp vrrp-group 20 description "LAN VRRP group B"
set interfaces ethernet eth2 vrrp vrrp-group 20 advertise-interval 1
set interfaces ethernet eth2 vrrp vrrp-group 20 authentication type ah
set interfaces ethernet eth2 vrrp vrrp-group 20 authentication password 50mepAs5
commit
save

ob-rt-02
configure
set interfaces ethernet eth2 vrrp vrrp-group 20 priority 100
set interfaces ethernet eth2 vrrp vrrp-group 20 virtual-address 10.0.200.254
set interfaces ethernet eth2 vrrp vrrp-group 20 description "LAN VRRP group B"
set interfaces ethernet eth2 vrrp vrrp-group 20 advertise-interval 1
set interfaces ethernet eth2 vrrp vrrp-group 20 authentication type ah
set interfaces ethernet eth2 vrrp vrrp-group 20 authentication password 50mepAs5
commit
save

WAN LoadBalancing and static routes
oa-rt-01 and oa-rt-02
configure
set protocols static route 0.0.0.0/0 next-hop 1.1.1.101 distance 10
set protocols static route 0.0.0.0/0 next-hop 2.2.2.101 distance 20
set protocols static route 3.3.3.201/32 next-hop 1.1.1.101
set protocols static route 4.4.4.202/32 next-hop 1.1.1.101
set protocols static route 3.3.3.202/32 next-hop 2.2.2.101
set protocols static route 4.4.4.201/32 next-hop 2.2.2.101

set load-balancing wan interface-health eth0 failure-count 3
set load-balancing wan interface-health eth0 nexthop 1.1.1.101
set load-balancing wan interface-health eth0 success-count 2
set load-balancing wan interface-health eth0 test 10 resp-time 5
set load-balancing wan interface-health eth0 test 10 target 3.3.3.201
set load-balancing wan interface-health eth0 test 10 ttl-limit 1
set load-balancing wan interface-health eth0 test 10 type ping
set load-balancing wan interface-health eth0 test 20 resp-time 5
set load-balancing wan interface-health eth0 test 20 target 4.4.4.202
set load-balancing wan interface-health eth0 test 20 ttl-limit 1
set load-balancing wan interface-health eth0 test 20 type ping

set load-balancing wan interface-health eth1 failure-count 3
set load-balancing wan interface-health eth1 nexthop 2.2.2.101
set load-balancing wan interface-health eth1 success-count 2
set load-balancing wan interface-health eth1 test 10 resp-time 5
set load-balancing wan interface-health eth1 test 10 target 3.3.3.202
set load-balancing wan interface-health eth1 test 10 ttl-limit 1
set load-balancing wan interface-health eth1 test 10 type ping
set load-balancing wan interface-health eth1 test 20 resp-time 5
set load-balancing wan interface-health eth1 test 20 target 4.4.4.201
set load-balancing wan interface-health eth1 test 20 ttl-limit 1
set load-balancing wan interface-health eth1 test 20 type ping

set load-balancing wan rule 100 description "Exclude Class A private networks"
set load-balancing wan rule 100 destination address 10.0.0.0/8
set load-balancing wan rule 100 exclude
set load-balancing wan rule 100 inbound-interface eth2
set load-balancing wan rule 100 protocol all

set load-balancing wan rule 110 description "Exclude Class B private networks"
set load-balancing wan rule 110 destination address 172.16.0.0/12
set load-balancing wan rule 110 exclude
set load-balancing wan rule 110 inbound-interface eth2
set load-balancing wan rule 110 protocol all

set load-balancing wan rule 120 description "Exclude Class C private networks"
set load-balancing wan rule 120 destination address 192.168.0.0/16
set load-balancing wan rule 120 exclude
set load-balancing wan rule 120 inbound-interface eth2
set load-balancing wan rule 120 protocol all

set load-balancing wan rule 1000 description "Default Load Balancer"
set load-balancing wan rule 1000 destination address 0.0.0.0/0
set load-balancing wan rule 1000 inbound-interface eth2
set load-balancing wan rule 1000 interface eth0 weight 2
set load-balancing wan rule 1000 interface eth1 weight 1
set load-balancing wan rule 1000 protocol all
set load-balancing wan rule 1000 source address 10.0.100.0/24
commit
save

ob-rt-01 and ob-rt-02
configure
set protocols static route 0.0.0.0/0 next-hop 3.3.3.101 distance 10
set protocols static route 0.0.0.0/0 next-hop 4.4.4.101 distance 10
set protocols static route 1.1.1.201/32 next-hop 3.3.3.101
set protocols static route 2.2.2.202/32 next-hop 3.3.3.101
set protocols static route 1.1.1.202/32 next-hop 4.4.4.101
set protocols static route 2.2.2.201/32 next-hop 4.4.4.101

set load-balancing wan interface-health eth0 failure-count 3
set load-balancing wan interface-health eth0 nexthop 3.3.3.101
set load-balancing wan interface-health eth0 success-count 2
set load-balancing wan interface-health eth0 test 10 resp-time 5
set load-balancing wan interface-health eth0 test 10 target 1.1.1.201
set load-balancing wan interface-health eth0 test 10 ttl-limit 1
set load-balancing wan interface-health eth0 test 10 type ping
set load-balancing wan interface-health eth0 test 20 resp-time 5
set load-balancing wan interface-health eth0 test 20 target 2.2.2.202
set load-balancing wan interface-health eth0 test 20 ttl-limit 1
set load-balancing wan interface-health eth0 test 20 type ping

set load-balancing wan interface-health eth1 failure-count 3
set load-balancing wan interface-health eth1 nexthop 4.4.4.101
set load-balancing wan interface-health eth1 success-count 2
set load-balancing wan interface-health eth1 test 10 resp-time 5
set load-balancing wan interface-health eth1 test 10 target 1.1.1.202
set load-balancing wan interface-health eth1 test 10 ttl-limit 1
set load-balancing wan interface-health eth1 test 10 type ping
set load-balancing wan interface-health eth1 test 20 resp-time 5
set load-balancing wan interface-health eth1 test 20 target 2.2.2.201
set load-balancing wan interface-health eth1 test 20 ttl-limit 1
set load-balancing wan interface-health eth1 test 20 type ping

set load-balancing wan rule 100 description "Exclude Class A private networks"
set load-balancing wan rule 100 destination address 10.0.0.0/8
set load-balancing wan rule 100 exclude
set load-balancing wan rule 100 inbound-interface eth2
set load-balancing wan rule 100 protocol all

set load-balancing wan rule 110 description "Exclude Class B private networks"
set load-balancing wan rule 110 destination address 172.16.0.0/12
set load-balancing wan rule 110 exclude
set load-balancing wan rule 110 inbound-interface eth2
set load-balancing wan rule 110 protocol all

set load-balancing wan rule 120 description "Exclude Class C private networks"
set load-balancing wan rule 120 destination address 192.168.0.0/16
set load-balancing wan rule 120 exclude
set load-balancing wan rule 120 inbound-interface eth2
set load-balancing wan rule 120 protocol all

set load-balancing wan rule 1000 description "Default Load Balancer"
set load-balancing wan rule 1000 destination address 0.0.0.0/0
set load-balancing wan rule 1000 inbound-interface eth2
set load-balancing wan rule 1000 interface eth0 weight 1
set load-balancing wan rule 1000 interface eth1 weight 1
set load-balancing wan rule 1000 protocol all
set load-balancing wan rule 1000 source address 10.0.200.0/24
commit
save

OpenVPN
Following illustration will help to understand our OpenVPN setup:

Generate static keys somewhere:
sudo openvpn --genkey --secret a1-b1.key
sudo openvpn --genkey --secret a1-b2.key
sudo openvpn --genkey --secret a2-b1.key
sudo openvpn --genkey --secret a2-b2.key
Distribute keys across routers:
oa-rt-01: /root/a1-b1.key /root/a1-b2.key
oa-rt-02: /root/a2-b1.key /root/a2-b2.key
ob-rt-01: /root/a1-b1.key /root/a2-b1.key
ob-rt-02: /root/a1-b2.key /root/a2-b2.key
Place everything in /root directory and chmod 0600.

Enable OpenVPN ports:
configure
set firewall name WAN_local rule 400 action accept
set firewall name WAN_local rule 400 description "Accept OpenVPN from fellows"
set firewall name WAN_local rule 400 protocol udp
set firewall name WAN_local rule 400 destination port 1011,1012,1021,1022
commit
save
You may also limit access for fellow router IPs only to achieve paranoid security.

oa-rt-01
configure
set interfaces openvpn vtun0 description "A1-B1"
set interfaces openvpn vtun0 local-port 1011
set interfaces openvpn vtun0 remote-host 3.3.3.201
set interfaces openvpn vtun0 remote-host 4.4.4.201
set interfaces openvpn vtun0 remote-port 1011
set interfaces openvpn vtun0 local-address 172.30.12.11
set interfaces openvpn vtun0 remote-address 172.30.34.11
set interfaces openvpn vtun0 mode site-to-site
set interfaces openvpn vtun0 encryption aes128
set interfaces openvpn vtun0 protocol udp
set interfaces openvpn vtun0 shared-secret-key-file /root/a1-b1.key
set interfaces openvpn vtun0 openvpn-option "--float"

set interfaces openvpn vtun1 description "A1-B2"
set interfaces openvpn vtun1 local-port 1012
set interfaces openvpn vtun1 remote-host 3.3.3.202
set interfaces openvpn vtun1 remote-host 4.4.4.202
set interfaces openvpn vtun1 remote-port 1012
set interfaces openvpn vtun1 local-address 172.30.12.12
set interfaces openvpn vtun1 remote-address 172.30.34.21
set interfaces openvpn vtun1 mode site-to-site
set interfaces openvpn vtun1 encryption aes128
set interfaces openvpn vtun1 protocol udp
set interfaces openvpn vtun1 shared-secret-key-file /root/a1-b2.key
set interfaces openvpn vtun1 openvpn-option "--float"
commit
save

oa-rt-02
configure
set interfaces openvpn vtun0 description "A2-B1"
set interfaces openvpn vtun0 local-port 1021
set interfaces openvpn vtun0 remote-host 3.3.3.201
set interfaces openvpn vtun0 remote-host 4.4.4.201
set interfaces openvpn vtun0 remote-port 1021
set interfaces openvpn vtun0 local-address 172.30.12.21
set interfaces openvpn vtun0 remote-address 172.30.34.12
set interfaces openvpn vtun0 mode site-to-site
set interfaces openvpn vtun0 encryption aes128
set interfaces openvpn vtun0 protocol udp
set interfaces openvpn vtun0 shared-secret-key-file /root/a2-b1.key
set interfaces openvpn vtun0 openvpn-option "--float"

set interfaces openvpn vtun1 description "A2-B2"
set interfaces openvpn vtun1 local-port 1022
set interfaces openvpn vtun1 remote-host 3.3.3.202
set interfaces openvpn vtun1 remote-host 4.4.4.202
set interfaces openvpn vtun1 remote-port 1022
set interfaces openvpn vtun1 local-address 172.30.12.22
set interfaces openvpn vtun1 remote-address 172.30.34.22
set interfaces openvpn vtun1 mode site-to-site
set interfaces openvpn vtun1 encryption aes128
set interfaces openvpn vtun1 protocol udp
set interfaces openvpn vtun1 shared-secret-key-file /root/a2-b2.key
set interfaces openvpn vtun1 openvpn-option "--float"
commit
save

ob-rt-01
configure
set interfaces openvpn vtun0 description "B1-A1"
set interfaces openvpn vtun0 local-port 1011
set interfaces openvpn vtun0 remote-host 1.1.1.201
set interfaces openvpn vtun0 remote-host 2.2.2.201
set interfaces openvpn vtun0 remote-port 1011
set interfaces openvpn vtun0 local-address 172.30.34.11
set interfaces openvpn vtun0 remote-address 172.30.12.11
set interfaces openvpn vtun0 mode site-to-site
set interfaces openvpn vtun0 encryption aes128
set interfaces openvpn vtun0 protocol udp
set interfaces openvpn vtun0 shared-secret-key-file /root/a1-b1.key
set interfaces openvpn vtun0 openvpn-option "--float"

set interfaces openvpn vtun1 description "B1-A2"
set interfaces openvpn vtun1 local-port 1021
set interfaces openvpn vtun1 remote-host 1.1.1.202
set interfaces openvpn vtun1 remote-host 2.2.2.202
set interfaces openvpn vtun1 remote-port 1021
set interfaces openvpn vtun1 local-address 172.30.34.12
set interfaces openvpn vtun1 remote-address 172.30.12.21
set interfaces openvpn vtun1 mode site-to-site
set interfaces openvpn vtun1 encryption aes128
set interfaces openvpn vtun1 protocol udp
set interfaces openvpn vtun1 shared-secret-key-file /root/a2-b1.key
set interfaces openvpn vtun1 openvpn-option "--float"
commit
save

ob-rt-02
configure
set interfaces openvpn vtun0 description "B2-A1"
set interfaces openvpn vtun0 local-port 1012
set interfaces openvpn vtun0 remote-host 1.1.1.201
set interfaces openvpn vtun0 remote-host 2.2.2.201
set interfaces openvpn vtun0 remote-port 1012
set interfaces openvpn vtun0 local-address 172.30.34.21
set interfaces openvpn vtun0 remote-address 172.30.12.12
set interfaces openvpn vtun0 mode site-to-site
set interfaces openvpn vtun0 encryption aes128
set interfaces openvpn vtun0 protocol udp
set interfaces openvpn vtun0 shared-secret-key-file /root/a1-b2.key
set interfaces openvpn vtun0 openvpn-option "--float"

set interfaces openvpn vtun1 description "B2-A2"
set interfaces openvpn vtun1 local-port 1022
set interfaces openvpn vtun1 remote-host 1.1.1.202
set interfaces openvpn vtun1 remote-host 2.2.2.202
set interfaces openvpn vtun1 remote-port 1022
set interfaces openvpn vtun1 local-address 172.30.34.22
set interfaces openvpn vtun1 remote-address 172.30.12.22
set interfaces openvpn vtun1 mode site-to-site
set interfaces openvpn vtun1 encryption aes128
set interfaces openvpn vtun1 protocol udp
set interfaces openvpn vtun1 shared-secret-key-file /root/a2-b2.key
set interfaces openvpn vtun1 openvpn-option "--float"
commit
save

OSPF
oa-rt-01
configure
set protocols ospf parameters router-id 10.0.0.11
set protocols ospf area 0.0.0.0 network 10.0.100.0/24
set protocols ospf area 11.11.11.11 network 172.30.34.11/32
set protocols ospf area 12.12.12.12 network 172.30.34.21/32
set protocols ospf passive-interface eth2
commit
save

oa-rt-02
configure
set protocols ospf parameters router-id 10.0.0.12
set protocols ospf area 0.0.0.0 network 10.0.100.0/24
set protocols ospf area 21.21.21.21 network 172.30.34.12/32
set protocols ospf area 22.22.22.22 network 172.30.34.22/32
set protocols ospf passive-interface eth2
commit
save

ob-rt-01
configure
set protocols ospf parameters router-id 10.0.0.21
set protocols ospf area 0.0.0.0 network 10.0.200.0/24
set protocols ospf area 11.11.11.11 network 172.30.12.11/32
set protocols ospf area 21.21.21.21 network 172.30.12.21/32
set protocols ospf passive-interface eth2
commit
save

ob-rt-02
configure
set protocols ospf parameters router-id 10.0.0.22
set protocols ospf area 0.0.0.0 network 10.0.200.0/24
set protocols ospf area 12.12.12.12 network 172.30.12.12/32
set protocols ospf area 22.22.22.22 network 172.30.12.21/32
set protocols ospf passive-interface eth2
commit
save

Grand Finale!
That’s ALL! You now got your router/ISP setup fully redundant and highly available.
Other HA options such as ethernet bonding, spare power supplies,
power generators, big UPSes and satellite internet links may be added to harden your setup,
the core functionality is already built and you can just easily extend it.

However, before buying a bottle of champagne and celebrating you must test your setup
from every angle, put down routers and providers, see what happens afterwards.
Anyway remember, there is no thing more important than testing in our business!

If you got any suggestions or you’ve just found a bug in my setup,
you are welcome to comment! Thanks for your attention!