How to interconnect AWS VPCs with VyOS

Interconnecting AWS VPCs with VyOS

Introduction

TL;DR: We run a number of VPCs in multiple regions, and so far only partial interconnectivity existed. After recently adding a set of new VPCs for services that require a stable and performant interconnection a new solution was required. We used VyOS and the AWS VPNs to pull it off.

I’d like to tell you a joke about our VPN setup but it’s virtually private.

How did we do it?

In hindsight, creating a VPN mesh would have been better for increased reliability and latency reduction. And perhaps a full deploy of BGP would have been even saner. While I know both, BGP is one of the more complicated concepts of networking and a full mesh increases complexity in maintenance and extendability. Therefore a centralized router presented itself as a more favorable solution.

In this context “centralized” breaks down to a single EC2 instance as a router and x (in our case x=3) VPN tunnels from AWS VPCs to this EC2. You’ll find a rough outline of the setup in the picture below.
A note here: Read this post to the end if you want to avoid the pitfalls I stepped into. But let me start at the beginning.

Create the EC2 instance

VyOS has its own AMI in the AWS Marketplace. wasted a whole day fiddling with this AMI as the version it ships with (1.1.0) has a particularly nasty bug in how IPsec is internally routed (SPOILER: it is not). Do yourself a favor: spin up an instance with this AMI, upgrade the running image and create your own AMI from there on.

$ ssh vyos@x.x.x.x -i key.pem
$ add system image http://mirror.vyos.net/iso/release/1.1.6/vyos-1.1.6-amd64.iso
  # Get through the (annoying) input requests.
  # Confirm successful addition of your new boot image: show system image
$ reboot

Let’s be smart here and avoid future headaches by associating an Elastic IP address with your ENI (Elastic Network Interface). This has a couple of advantages. First and foremost, you don’t have to alter the AWS route tables of the EC2’s VPC any more, you route traffic to the ENI. Whenever you rebuild the EC2, you simply spin it up with the ENI specified. Second, the EIP is sticky to the ENI and your VPNs (see next step) will always point to a legit Customer Gateway. Third, your VyOS config now has less parts to be changed.
With these prerequisites, spinning up a new VyOS box is less hassle.

Create the VPN tunnels in your VPCs

This is as straight-forward as it can be:

(if necessary) create a Virtual Private Gateway.
create a Customer Gateway pointing to the EIP you allocated earlier.
create the VPN (in our case with static routing).
Make sure to add ALL CIDRs on the other side of the tunnel to the static routing.
download (generic) config for PSKs, endpoint IPs and Link-local addresses.
NOTE:
- If you want to use BGP you can download a VPN config for Vyatta and therefore for VyOS. Don’t forget to enable route propagation in AWS!

Security groups and route tables.

Security groups and route tables are easy, but you have to keep your mind sharp as they are hell to debug.

Route tables are the easier of the two as you’re literally just pointing the CIDRs on the other end of the equation to either the ENI of your VyOS box or to the VPG you just created (and this is only necessary if you’re trying to avoid route propagation).

Security Groups are more tricky.
In the VPC of your VyOS box, make sure to allow incoming traffic from the VPG addresses given to you by AWS for the incoming VPN connections. Additionally, whitelist the CIDRs in your connected VPCs.
In your “satellite” VPCs, make sure to whitelist the CIDRs in all other VPCs. I tend to add the EIP of the VyOS-box for troubleshooting purposes.

Create your VyOS config

The VyOS config, as mentioned, is based on a JSON-like file stored at /config/config.boot and can be loaded by running load in configure mode in the CLI. SCP became my best friend during this project. The default config is a good starting point. There’ll be a glimpse in our config further down. As a best practice, edit your config locally, scp’ it to your VyOS-box, load it, run a commit-confirm, and if all went well, run a commit and save to finalize the change.

Pitfalls to avoid.

Build your own, up-to-date AMI as a first thing. Believe me.
Use commit-confirm rather than simple commit. It’s your parachute!
Propagate your routes.
Double check your security groups and route tables.
You will forget that your VPN has its own route table. If for some reason you don’t want to propagate, doublecheck.

The promised glimpse into our config.

I skipped parts of our config irrelevant to this project, and obviously redacted all security relevant parts. The parts you will have to edit are the following:

Create VTIs for each tunnel (as the AWS VPN are actually two tunnels) and assign them the link-local addresses given from AWS.
Add a next-hop-interface route for each tunnel and point it to the corresponding VTIs
Add the ESP-group and IKE-group settings to your config, they can be reused by every new tunnel to AWS (and potentially elsewhere, too).
Finally, add the site-to-site peers, two for each AWS VPN connection, according to the VGW IPs given from AWS and your own EIP.
The local-address is the private IP of your eth0.
Make sure to bind the correct VTI to your connection.
“Tunnel 1” is not a good description.

interfaces {
    ethernet eth0 {
        ...
    }
    vti vti0 {
        address 169.254.x.x/30
        description "Tunnel 1"
        ip {
            source-validation disable
        }
        mtu 1436
    }
    vti vti1 {
        address 169.254.y.y/30
        description "Tunnel 2"
        ip {
            source-validation disable
        }
        mtu 1436
    }
    ...
}
protocols {
    static {
        interface-route <<Remote CIDR>> {
            next-hop-interface vti0 {
                distance 1
            }
            next-hop-interface vti1 {
                distance 2
            }
        }
        ...
    }
}
vpn {
    ipsec {
        esp-group AWS {
            compression disable
            lifetime 3600
            mode tunnel
            pfs enable
            proposal 1 {
                encryption aes128
                hash sha1
            }
        }
        ike-group AWS {
            dead-peer-detection {
                action restart
                interval 15
                timeout 30
            }
        ikev2-reauth no
        key-exchange ikev1
        lifetime 28800
        proposal 1 {
        dh-group 2
        encryption aes128
        hash sha1
        }
    }
    ipsec-interfaces {
        interface eth0
    }
    site-to-site {
        peer <<VPG-IP1>> {
            authentication {
                id <<EIP>>
                mode pre-shared-secret
                pre-shared-secret thiswouldnotbesosecretnowwouldit
                remote-id <<VPG-IP1>>
            }
            connection-type initiate
            description "Tunnel 1"
            ike-group AWS
            ikev2-reauth inherit
            local-address <<Local address of your ENI>>
            vti {
                bind vti0
                esp-group AWS
            }
        }
        peer <<VPG-IP2>> {
            authentication {
                id <<EIP>>
                mode pre-shared-secret
                pre-shared-secret imeanreallythiswouldnotbesecretanymore
                remote-id <<VPG-IP2>>
            }
            connection-type initiate
            description "Tunnel 2"
            ike-group AWS
            ikev2-reauth inherit
            local-address <<Local address of your ENI>>
            vti {
                bind vti1
                esp-group AWS
                }
            }
        }
    }
}