Finding correct OpenVPN MTU and MSSFIX settings can really drive you batty. No really. It’s ridiculous how much trouble MTU settings can cause and how often people don’t realize that MTU is actually what’s causing the problem!
My Experience with OpenVPN MTU Problems
As many of you are aware, I run a ton of VPNs. I use a combination of OpenVPN and/or WireGuard depending on what I am doing with it. Generally I use pfSense as my firewall (although sometimes a Ubiquiti Dream Machine Pro).
Mostly, I run a VPN Between Friends and Family and I run a VPN to our new building, and all of these locations have AXIS security cameras that are recording to a combination of an onsite and remote Synology for video surveillance. This means there is a lot of UDP IP camera traffic traversing the VPN.
In the beginning with only one or two cameras I noticed that there were frame drops, but it wasn’t really enough to matter or cause a significant degradation in the video feed. But after I started to get 5, 6, or 7 cameras up I noticed things would degrade very quickly. UDP packets would get dropped like crazy, but TCP traffic would work perfectly.
So what gives?
TCP vs. UDP for OpenVPN
Before we get to fixing our OpenVPN MTU issues, let’s first take a minute to understand the difference between UDP and TCP. Trust me, this is worth understanding and it can have major impacts on your VPN’s overall performance.
TCP is an acronym for Transmission Control Protocol. UDP is an acronym for User Datagram Protocol. While we’re not going to go into all of the gory detail of differences between these protocols there are a few things that are very important to understand.
First, TCP creates a session between two devices and checks to make sure the packet was delivered. If the a reply is not received within the limits of the defined TTL (time-to-live) the data will be retransmitted. This guarantees delivery even over unstable connections.
UDP on the other hand is session-less. It simply fires a packet down the wire and hopes for the best. If the other end never receives the data nothing happens, it is simply lost.
In the case of transmitting a file such as a word document, if a some of the data doesn’t arrive, the document will be corrupted and unreadable. In the case of streaming music, you’ll just get some degraded sound quality for a second. This is why most file transfers used TCP and most streaming uses UDP.
OpenVPN Defaults to UDP for the Tunnel
Now it might come as a huge shocker to learn that OpenVPN (and most VPNs) use UDP for their tunnel! That’s crazy talk right? What do you mean my VPN can lose data? Madness I say!
Well it actually makes perfect sense. If an application needs TCP, that TCP session will be tunneled inside the OpenVPN UDP transmission. And if the application detects a packet didn’t make it… it will retransmit just like normal. The VPN doesn’t need to do it There is no reason to tunnel TCP over a TCP tunnel! In fact doing so adds lots of overhead, can cause jitter and latency problems and a lot of other undesirable behavior.
In the case of UDP, if it was going to get lost outside the tunnel it will still get lost inside the tunnel. If you’re having problems with lost packets, maybe the application should be using TCP. Or maybe you have an OpenVPN MTU problem (we’re getting there).
You should never use TCP for the OpenVPN tunnel itself unless you have a specific technical challenge to overcome and you understand the greater impacts.
MTU is an acronym for Maximum Transmission Unit. Simply put, it defines the maximum size of a packet traversing the network. Anything bigger than this number must be broken into multiple packets. The default MTU for Ethernet is 1500 bytes. For two devices to properly communicate they need to know this number. If they transmit packets larger than 1500 bytes the packets will be discarded by one of the network devices.
In the case of OpenVPN MTU settings, its important to realize that we’re tunneling data and that the VPN overhead is going to take away some of our MTU!
So here’s the problem I here all of the time: “My OpenVPN works fine for TCP traffic, but UDP traffic is very buggy and only works sometimes.” Why is this? Because TCP has many mechanisms to deal with networks and it will lower its Maximum Segment Size (MSS) and try again (retransmit). UDP being fire and forget will never know there was a problem and just keep firing those oversize packets into oblivion to be dropped by the VPN tunnel!
In my case this translated to SSH and web sessions working perfectly, but video streams and VoIP running over the tunnel would fail miserably, resulting is stuttering and choppy video and audio.
Fixing OpenVPN MTU Issues
The first thing you need to do to fix your OpenVPN MTU problem is to figure out what your largest MTU actually is. You can do this using the ping command. “ping -f” tells ping not to fragment the packet under any circumstances. “ping -l” tells ping the packet size to use.
ping -f <IP of Device on other end of VPN> -l <MTU to test> ping -f 192.168.100.1 -l 1500
And you can see in our case the pings failed because they were too large to pass through our VPN tunnel. Keep trying the command over and over. Each time lower the packet size (-l) by 10 until the pings work.
You can see that in our case, we had to drop down to a 1450 byte packet to get data to cross the tunnel.
On both our client and server and server size I made the following Open VPN MTU change: –tun-mtu 1450;
You can make this change in pfSense’s OpenVPN in the Advanced Configuration portion as follows:
What about MSSFIX for OpenVPN?
A lot of people will suggest setting MSSFIX to 40 bytes smaller and also including it in your statement, as some devices don’t properly learn MTU settings. MSSFIX is a workaround used when MTU path discovery is broken for some reason.
My advice, try the tun-mtu 1450 (or whatever your tunnel needs to be set to) without MSSFIX first and see if it resolves your issue first. Only add MSSFIX if absolutely necessary. Keep it simple.
If you do need MSSFIX, you can add it as follows: –tun-mtu 1450; –mssfix 1410; or in pfSense as follows: