Please, we are begging you. Adopt IPv6 and help everyone lower costs. CGNAT servers, NAT64 servers, regular NAT on a home router, IPv4 address space exhaustion. There's just so much going on with IPv4 where adding dual stacked IPv6 would help reduce cost.

Looking at our NAT64 server(we are not a residential ISP and so don't have CGNAT, nor do we use NAT much internally other than a few places(like our internal VPN)) we can see that we do see some traffic flowing through that. It's not too much, since we don't sell virtual servers directly, and the clients buying VMs are going through another seller who does use the NAT64 server as well, but we do make use of it internally. A lot of our internal stuff runs on IPv6 only machines to save us on an IPv4 address. For us, one of the biggest issue is Github not support IPv6, and so all that traffic has to go through the NAT64 server. We've only got a 1G link between the server and our switch. Thankfully traffic on that server isn't too high or else we'd need to look into upgrading it quicker, but every once in a while it gets to a point that makes us think if we should upgrade to a 10G link. But a 10G link would have costs associated with it. We don't purely run NAT64 on the server, it's running more stuff as well, and taking it down to install a 10G adapter, and bring it back up will have some downtime associated with it, plus the hardware cost and labor cost.

Github is one of the services that we utilize which still doesn't support IPv6 and has us considering alternatives. In fact, as of writing this post, github still doesn't support cloning repositories over IPv6. It causes thoughts like should we switch the git platform that we are on in order to reduce our costs and reduce the amount of git cloning traffic that goes on via the NAT64 server?

IPv4 exhaustion is another big issue. The RIRs are out of IPv4 addresses to give away and we have to resort to either getting on the waiting list(which is a multi year process) or buy the IP space, which can cost about $10k per /24 of IPv4(which is only 256 addresses). There's a reason that AWS and other providers(including us) are raising the cost for IPv4 addresses and it's because it's not cheap to acquire them.

Good news​

The good news is that we are seeing signs towards progress here. Last month, BackBlaze announced support for IPv6 for their S3 compatible object storage api. That's a service that we do make use of, and can see less traffic flow through our NAT64 server. Github does seem like they are making progress towards it as well, just not quick enough. AWS is announcing more and more support for IPv6 in their products.

What can you do?​

You can help out in this too! Anytime you see IPv6 missing on an ISP connection, or a service you use or website you visit, ask them if they have plans to support IPv6 anytime soon. Show that there is desire for IPv6 from users out there. Financially support services that support IPv6 over those who don't. For us, that might mean moving from Github to Gitlab, and while the change over might take a little bit of getting used to(I personally use both platforms, and always takes me a second to figure out where the buttons are again when context switching), in the long run, it's going to show the need for IPv6 to retain and get new customers. Give the issue on github/gitlab/hosting platform for IPv6 a thumbs up to show that you want support for it.

Akvorado is a very powerful tool in network engineer's toolbox. We use it at Ryamer to help us identify how traffic comes into and out of our network and optimize that by seeing if we could shift any of that traffic over to the IX that we are part of. But today I'm not here to talk about how we've reduced costs at Ryamer, but instead on another larger network, who we are going to call X(not the one formerly known as Twitter) for now.

What is Akvorado? It's a flow analyzer that takes in Netflow/Sflow data and makes nice and pretty charts that humans can use in order to get an eye at their network. With some work, you can configure Akvorado to tell you what port the traffic came in and out of. Now with that comes a great feature, we can mark ports as transit or peering, and almost always peering ports are cheaper than transit ports.

On network X, after setting up Akvorado, we were able to see something. There was a lot of incoming traffic from AS32934 aka Facebook aka Meta, and both network X and AS32934 were connected to a mutual internet exchange, AMS-IX. But the traffic wasn't coming in from the AMS-IX port, instead it was coming from the transit port. That means that Facebook most likely doesn't participate in the route servers at AMS-IX. With that knowledge, we were able to simply request peering with AS32934. After a few days, the sessions were up and all of that traffic which used to be coming in via the transit port got moved over to the peering port.

Facebook was the largest source of incoming traffic for said network. Getting that off of transit and onto the peering port cut the usage of the transit port by a huge amount, and in fact after doing this with a few more networks(looking at both inbound and outbound traffic) we were able to get this to a point where the IX port handled a larger part of the traffic than transit. Of course it does help that AMS-IX is such a large and well connected exchange where a lot of large networks meet at.

Another thing that Akvorado does is reveal the 95th percentile of traffic. If you know you are getting close to exceeding your commit and will start getting charged for overages, you can shift traffic over to another transit port in order to not get hit by those fees.

But anyone can use the same tricks at any other point. Akvorado is free to run and it's very easy to configure. Half an hour of work and you too can get great insight into your network. At Ryamer, we are actively trying to improve our network connectivity and reduce latency by peering with as many networks as it makes sense to. Akvorado has helped us see what networks we talk to a lot and plan out capacity increases.

And it's done. Our Network site is no longer a simple HTML/CSS/JS site, but instead a more powerful site powered by Docusaurus. It was a surprisingly quick change. Less than three hours total, with a lot of the time being dedicated to figuring out how to set this thing up. Moving over the old stuff didn't take too long either, since most of it wasn't anything fancy. In under two and a half hours, I had everything but this blog post written.

And in a total of with deploying and this blog post it's going to be a 3 hour journey. Hope to have most content for you guys to read here soon!