[BUG] matchmaking, nat and general networking bug with a possible workaround

[SunBlade_ger]

bug description:
the port you set in the settings menu is completely ignored by warframe (client AND SERVERS).
the client will always send and receive on UDP port 4950&4955. [minor bug]
the server tries to infer your forwarded ports from IP traffic, which is impossible. and succeeds only under special circumstances. [critical bug]

does this even affect me?
if you have frequent matchmaking/invite problems, the chances of you being affected are high.
if warframe gives you the "Strict-NAT" message, but port forwarding works with all other programs, then you are definitely affected.
even without the "Strict-NAT" message, if your modem, router, firewall or whatever can connect multiple devices on your home network to the internet, you are probably affected by this bug.
unfortunately there is no easy way, at least i haven't found any, to check whether you are affected by this or not.

Edit: thanks to a co-worker i may have found a better workaround. your router might be able to redirect outgoing connections to fixed ports:
https://forums.warframe.com/topic/1195761-bug-matchmaking-nat-and-general-networking-bug-with-a-possible-workaround/?do=findComment&comment=11608874

Edit 2: (2020-06-11) the port setting in the game client got fixed. your hosting capabilities still do not get tested.



since this workaround is pretty technical you should only read the rest if you are interested or can configure your firewall to an unusual degree.

first, the terminology i will use throughout this post:
a Host is basically a pc connected to a network.
a Local (host) is connected to your own network (LAN).
a Remote (host) is connected to a network outside.
a Router connects IP-segments.
an IP is, depending on the context, the Internet Protocol v4 or more likely the IP address of a host.
UDP (User Datagram Protocol) is a port based IP protocol.
NAT (Network Address Translation) is a broad term, of which i will use only the IP masquerading aspect.
a Connection is a uni-directional virtual UDP communication line between two hosts, allowing packets to travel back and forth.
a Packet transports data from a source to a destination.
a Source is the IP and, in case of UDP, port of the sending host, written on every packet.
a Destination is the IP and, in case of UDP, port of the receiving host, written on every packet.
a Reply is a packet, in response to a received packed, where source and destination are switched.
a Server is a program or computer offering services. (deliberately ambiguous)
a Client is a program or computer using those services.

and now, how your router works:
everyone has another name for the system i am about to describe, Wikipedia for example calls it "port-restricted cone NAT", i like to call it "connection based NAT".
no matter what you call it, your router uses this system to connect you to the internet.
and in order to allow others to connect to you, your router uses a technique commonly called "port forwarding" or "port mapping".

instead of actual port numbers i use ABC, LOC, OPN, SRV, RND and XYZ, where LOC, SRV and RND (yellow) are usually unknown random dynamic ports.

https://www.dropbox.com/sh/ub6ioim7ywm2j5e/AAAzfdbYmudDBfWG473cl6xta/Warframe/nat_diagramm.png?dl=0

packet traversal for outgoing connections (green arrows) is initiated by outbound traffic:
1. local opens UDP port LOC and sends a packet to remote on port XYZ.
2. router replaces source IP with public IP, and source port with a random port RND before forwarding the packet
3. router saves this connection details (local IP&port, public IP&port and remote IP&port) in his NAT table
4. remote receives the packet from public IP and port RND, oblivious to local IP and port LOC.
inbound traffic then can be classified as reply by:
1. remote sends a reply from his IP and port XYZ to public IP and port RND.
2. router looks up in his NAT table the local IP&port by matching remote IP&port and public IP&port in his table.
3. router replaces destination IP&port with the looked up local IP&port before forwarding the packet.
4. local receives reply on port LOC from remote IP and port XYZ, oblivious to public IP and port RND.

packet traversal for incoming connections (blue arrows) is initiated by inbound traffic on port OPN:
1. remote opens UDP port SRV and sends a packet to public on port OPN.
2. router looks up local IP&port by matching port OPN against his static port mapping rules.
3. router replaces destination IP&port with the looked up local IP&port before forwarding the packet.
4. router saves this connection details (local IP&port, public IP&port and remote IP&port) in his NAT table
5. local receives the packet on port ABC, oblivious to public IP and port OPN.
outbound traffic then can be classified as reply by:
1. local sends a reply from his IP and port ABC to remote IP and port SRV.
2. router looks up in his NAT table the public IP&port by matching remote IP&port and local IP&port in his table.
3. router replaces source IP&port with the looked up public IP&port before forwarding the packet.
4. remote recieves reply on port SRV from public IP and port OPN, oblivious to local IP and port ABC.

for conveniences sake, most programmers, user and admins, assume ABC and OPN to be the same.
while connections are only uni-directional, your router ensures bi-directional packet traffic.

WF however uses ABC for LOC and expects OPN and RND to also be ABC, which most routers will not do, because that would break their NAT table when multiple locals with the same port want to connect to the same remote&port.
WF also assumes RND to stay the same when LOC does not change, which is utterly impracticable on bigger networks, for the afore mentioned reason. and for the same reason routers normally put any RND far away from any OPN.

my current workaround (red arrows):
since WF servers assume OPN to be RND, i force my router to use OPN instead of RND for connections from ABC. and then, static port forwarding works like a charm.

how to do this workaround yourself:
this only works if your router allows direct access to the nat and forwarding chains. i know on Sophos and MikroTik routers this can be done, but not on AVM Fritz!Box or Telekom Speedport.

i am going to use my MikroTik as an example.

in the following commands, replace {my ip} with you PCs LAN IP, and {public port 1&2} with the ports you want to use for port forwarding.
if you want to do this for multiple PCs, {public port 1&2} needs to be unique. that means when you have three PCs you need to dedicate six unique ports.
also, if you changed the default interface lists you need to change WAN.

Quote

/ip firewall nat
add disabled=no chain=srcnat out-interface-list=WAN protocol=udp src-address={my ip} src-port=4950 action=masquerade to-ports={public port 1}
add disabled=no chain=srcnat out-interface-list=WAN protocol=udp src-address={my ip} src-port=4955 action=masquerade to-ports={public port 2}
add disabled=no chain=dstnat in-interface-list=WAN protocol=udp dst-port={public port 1} action=dst-nat to-addresses={my ip} to-ports=4950
add disabled=no chain=dstnat in-interface-list=WAN protocol=udp dst-port={public port 2} action=dst-nat to-addresses={my ip} to-ports=4955
move [:pick [find] ([:len [find]] -1)] 0
move [:pick [find] ([:len [find]] -1)] 0
move [:pick [find] ([:len [find]] -1)] 0
move [:pick [find] ([:len [find]] -1)] 0

that creates four nat rules, and moves them to the top.
now, whenever WF tries to connect anywhere, your router will always send from {public port 1&2}. and incoming connections on {public port 1&2} get redirected to WF.
if you are however the only one on your LAN playing WF, i strongly suggest using 4950&4955 for {public port 1&2}.

and since the WF client is currently also bugged and ignores what ports you have set in the settings menu. this forcible rewire of local ports to public ports, enables multiplayer for people on the same LAN.

i hope this helps a few people.

[SunBlade_ger]

what if you don't have a MikroTik?
well, all good firewalls are based upon linux and its netfilter. so, if you where able to configure IP masquerading on your router, you may also be able to understand and/or use netfilter commands.


ok then, get your nerd goggles and let's dissect my MikroTik commands:

/ip firewall nat
     every firewall has multiple chains. this moves your MikroTik's focus to the nat chains, so that all subsequent commands operate on those chains.

add
     pretty self-explanatory. we want to add a new rule.
disabled=no
     just to be sure we want that rule to be active.
chain=srcnat
     the rule should be added to the source nat chain, which runs on packets leaving your router.
out-interface-list=WAN
     limit this rule to packets leaving through the public interface.
protocol=udp
     only work on UDP traffic.
src-address={my ip} src-port=4950
     and match only if source is {my ip} and port 4950.
action=masquerade to-ports={public port 1}
     now we replace source with the public ip and force port to be {public port 1} possibly breaking already established connections on that port.

since warframe inexplicably listens on two ports we need to repeat that for src-port=4955 and to-ports={public port 2}

add disabled=no
     now we need a matching port forwarding rule
chain=dstnat
     the destination nat chain, runs on packets entering your router.
in-interface-list=WAN protocol=udp dst-port={public port 1}
     we want to match only incoming connections on your public interface and UDP port {public port 1}
action=dst-nat to-addresses={my ip} to-ports=4950
     here we redirect the connection to {my ip} and port 4950

we also need repeat this for dst-port={public port 2} and to-ports=4955

move [:pick [find] ([:len [find]] -1)] 0
     this moves the bottom rule to the top. repeat that three more times to be sure our new rules match before any other.



now we can translate that into netfilter commands for linux-boxes:

iptables
     that is the command to edit netfilter rules.
-t nat
     the table we want to edit is called nat. (this is not the "NAT table")
-I POSTROUTING 1
     insert the rule in the postrouting chain at 1st place. (no need to move it afterwards)
-o ppp0
     limit this rule to packets leaving through the public interface. (you may need to look up what your public interface is. ppp0 is commonly the first dial up interface.)
-p udp
     only work on UDP traffic.
-s {my ip} --sport 4950
     and match only if source is {my ip} and port 4950.
-j MASQUERADE --to-ports {public port 1}
     now we replace source with the public ip and force port to be {public port 1} possibly breaking already established connections on that port.

and as above we need to repeat that for --sport 4955 and --to-ports {public port 2}.

iptables -t nat
     now we need a matching port forwarding rule
-I PREROUTING 1
     we insert this rule at the top of the prerouting chain.
-i ppp0 -p udp --dport {public port 1}
     we want to match only incoming connections on your public interface and UDP port {public port 1}
-j DNAT --to-destination {my ip}:4950
     here we redirect the connection to {my ip} and port 4950

and again we need to repeat that for --dport {public port 2} and --to-destination {my ip}:4955.

so, our netfilter commands should look like this:

Quote

iptables -t nat -I POSTROUTING 1 -o ppp0 -p udp -s {my ip} --sport 4950 -j MASQUERADE --to-ports {public port 1}
iptables -t nat -I POSTROUTING 2 -o ppp0 -p udp -s {my ip} --sport 4955 -j MASQUERADE --to-ports {public port 2}
iptables -t nat -I PREROUTING 1 -i ppp0 -p udp --dport {public port 1} -j DNAT --to-destination {my ip}:4950
iptables -t nat -I PREROUTING 2 -i ppp0 -p udp --dport {public port 2} -j DNAT --to-destination {my ip}:4955

this should be more helpful than my MikroTik specific commands.

[SunBlade_ger]

what does DE need to change to make WF compatible with all routers?

first and foremost DE needs to understand that connections are a virtual constructs, an idea humans have invented for themselves.
and second: UDP may be a stateless protocol, but since it is port based, the principles of connections still apply.
while a connection is only uni-directional, it ensures bi-directional packet traffic.

actual implementation:
the game client currently ignores which ports you have set in the settings menu, and always listens only on the default ports. this needs to be fixed first.

UPnP is an important part of your networking set up, because not everyone can configure static port forwarding.
so, you should add extra failsaves, like ...
if the ports are already forwarded to another ip, you should ask the user if he is the only one playing WF, and whether you should overwrite the forwarding rules or change the ports in the settings menu.
and always delete those rules when the game client exits, so the user doesn't get annoyed, if his router's DHCP server assigns rotating IP addresses.

then the WF servers need to be notified which ports the user has configured in the settings menu.
since those two ports are (in theory) secrets, i suggest using the login connections for this since they are TLS encrypted https streams.

then you need to check if and how the client accepts connections.
you should use a dedicated test server/ip which is never contacted by the game client, to test if you get a connection to the user configured ports.

if not: you can test if the router is vulnerable to punch through attacks.
first you need to know the which ports the router assigns, by letting the game client establish UDP connections to your login server/ip.
then if your dedicated test server/ip can connect to the observed ports, you can use them instead of the configured ports.

lastly, users need to be informed about the outcome of these tests, when they hit Analyse Network.
the message box could say something like this:

https://www.dropbox.com/sh/ub6ioim7ywm2j5e/AAB2OxqSPJWgCci5oiyXn279a/Warframe/nat_messages.png?dl=0

[SunBlade_ger]

a co-worker had a brilliant idea and i already could verify his theory on a friend's Fritz!Box.

his theory: a port forwarding rule shifting only a single UDP port may also trigger on outgoing connections.

if you want to do this on your router, https://portforward.com/router.htm has good guides for almost any router.
the only differences:
- your "public" or "external" port needs to differ from your "internal" port (if this works you can try keeping the internal port number for the external port)
- only UDP
- only a single port (you need to create two rules per IP)

for the external port i suggest putting it in the range of 10000-19999, and spread them at least 5 apart.
the internal ports need to be 4950 and 4955, since the game client still ignores port settings.
the internal ports need to match the ports you set in the settings menu.
the external ports need to be unique, each device needs a new set of them.
i also suggest disabling UPnP in WF, because it might interfere with the static port forwarding.

after setting up this port redirect for 14950->4950 and 14955->4955, my friend could host and join games.
and his public ports where indeed 14950&14955 for incoming and outgoing connections.
here are his settings:

Spoiler

https://www.dropbox.com/sh/ub6ioim7ywm2j5e/AAC3LzlGfbXp_lSzmLv--3WEa/Warframe/fbOverview.jpg?dl=0
https://www.dropbox.com/sh/ub6ioim7ywm2j5e/AAABdxrUIRfyGW_PRct01qG1a/Warframe/fbPorts.jpg?dl=0

https://www.dropbox.com/sh/ub6ioim7ywm2j5e/AAA4mrhA_qQzgs8DKrAR0cWla/Warframe/fbRule.jpg?dl=0

i hope this helps. also, i am curious to know which router can reproduce this behaviour, and which can not.

[Mohmi]

wow... I am very impressed, good work!

[Kyzarkit]

So, is this something I should expect DE to patch to honor the in-game settings, or should I set up port forwarding?

There are 3 of us that play in the same house, the matchmaking works ~50% of the time, but we hit Strict NAT issues eventually.  

I really don’t want to have to play around with router settings for one game.

 

[SunBlade_ger]

On 2020-06-06 at 11:28 PM, Kyzarkit said:

So, is this something I should expect DE to patch to honor the in-game settings, or should I set up port forwarding?

Since DEs bug fixing record is awful at best, you should wait for Deadlock Protocol. if DE doesn't manage to fix their game client till then, you may want to fiddle with your router.

the game client should be easy to fix. i assume they simply forgot a variable assignment or something. since having this option in the settings menu doesn't make sense when your networking code does not support that.

[maciejs]

On 2020-06-08 at 3:28 AM, SunBlade_ger said:

Since DEs bug fixing record is awful at best, you should wait for Deadlock Protocol. if DE doesn't manage to fix their game client till then, you may want to fiddle with your router.

the game client should be easy to fix. i assume they simply forgot a variable assignment or something. since having this option in the settings menu doesn't make sense when your networking code does not support that.

Uh, thank you for your faith, I guess ;)... There is indeed a bug with preserving port numbers, which will be fixed ASAP.

[SunBlade_ger]

On 2020-06-09 at 7:53 PM, maciejs said:

Uh, thank you for your faith, I guess ;)... There is indeed a bug with preserving port numbers, which will be fixed ASAP.

thanks and good work. i can verify that the game client adheres to the port settings since Deadlock Protocol. up to five people can now play simultaneous on one LAN without port collisions.

the other problems still persist:
- which ports i have configured for port forwarding is still not transmitted to the login/matchmaking servers.
- "Analyze Network" still only tests outgoing connections and does not check whether i can host via port forwarding or nat penetration.
- the game client still initiates connections to unconfigured ports when joining a running game.

[maciejs]

Could you elaborate on these a bit? We do not typically transmit ports, rather we rely on the port we're receiving the information from. Are you still taking about multiple players behind same NAT?

[SunBlade_ger]

23 hours ago, maciejs said:

We do not typically transmit ports, rather we rely on the port we're receiving the information from.

this is exactly the problem. your system relies on the penetration vulnerability of most routers. if the router changes the port of the outgoing connection for whatever reason (penetration hardening, already established connection, paranoid admin, etc.), incoming connections on that port might not get redirected to the player.
this is the reason you set up port mapping/forwarding rules either manually or by using upnp. to have fixed ports which accept incoming connections. and these port mappings/forwardings can NOT be inferred by IP traffic, because the port of outgoing connections can deviate for whatever reasons.

 

On 2020-06-12 at 8:29 PM, maciejs said:

Are you still taking about multiple players behind same NAT?

This affects all players behind a NAT. your system relies on edge cases, which are true most of the time, but once this is no longer the case, your matchmaking breaks.

 

On 2020-06-12 at 8:29 PM, maciejs said:

Could you elaborate on these a bit?

well, there are several interconnected issues here. the ones i have found up till now:

1. currently your system relies on NAT penetration, which is unreliable. you should focus primarily on port mapping/forwarding by either upnp or the user. as a backup plan you may want to try NAT penetration. and if everything fails you can try a timed double NAT penetration.
2. "Analyze Network" is currently completely useless. it does not check if my port mapping/forwarding works and, if that fails, it does not check if my router is vulnerable to penetration attacks.
   in short: it does NOT check if i can host a game session.
3. when joining a game session, my game client tries to connect to three consecutive ports, where only the first is the one discovered for NAT penetration.
   this can lead to serious confusion, because the other two ports may be redirecting to other instances of the game client.
   and this can be dangerous, because other services may be listening on those ports. and you might break those services by sending them "random" data.

all these issues seem to originate from a bug in your mindset. i don't know which detail you got wrong, i can only speculate. for example:
https://forums.warframe.com/topic/13981-strict-nat/
glen's explanation of strict nat seems to assume that port mapping/forwarding for incoming connections also enforces port preservation for outgoing connections, which is false. port preservation is an edge case of NAT, and can not be guaranteed.
he also seems to assume that port mapping/forwarding follows a port shift, which is also false. port mappings/forwardings are static and independent of outgoing connections.

all these evidences suggest that there is a fundamental flaw in your understanding of routers. i can not rectify that. all i can do is test uncommon edge cases and point out where you might have gone wrong.

PS.: with "you" i mean whoever feels responsible. and with "your system" i mean your entire networking code, not just the game client.

[War-.]

I set up my network like you said. it's amazing. I have no problem hosting games anymore. I hope the DE will do something about it!

@SunBlade_ger thanks a lot!

[SunBlade_ger]

5 hours ago, warmmuug said:

I set up my network like you said. it's amazing. I have no problem hosting games anymore. I hope the DE will do something about it!

glad i could help. since maciejs turned up here, i am confident this will be fixed someday.

 

On 2020-06-12 at 8:29 PM, maciejs said:

Could you elaborate on these a bit?

to elaborate even further i did some research. what i call nat penetration, the system you rely on for over seven years now, is more commonly called "NAT Punch-through" (link) or "UDP hole punching" (link). and as you can see that is more guesswork than certainty.
and most manufacturer are lazy, so their consumer grade routers loosely adhere to the peculiarities of a "Full cone NAT". this is again guessing and no guarantee.
since you use upnp and nat-pmp, you already have a way better master plan. but you do not use it.
a working port forwarding is rock solid and penetrates even the strictest "Symmetric NAT". you should capitalize on that.


proving a working UDP port forwarding is very complicated, since any UDP packet the client sends may already "taint" the routers nat table. i suggest transmitting the configured ports on login via TCP, after you set up the port forwarding using upnp. and if you want to make absolutely sure port forwarding works, the client sends the port setting only to the login server which in turn sends that info the the other servers. once your servers receive the port setting they can try to send packets to the configured port.
if that succeeds and the client receives those packets you have proven a rock solid port forwarding till the user logs out. no need to recheck or change anything after this.
if the client does not receive any UDP packets for several seconds after login it is safe to assume port forwarding has failed and the client can start sending UDP packets.


and i think the clients connection attempts on three consecutive ports does more harm than good. because you already observe port shifts, you don't need to take chances and accidentally break something.

[Luke_eX]

So when do you think that this bug will get fixed, kinda causing me problem in trading right now....

[maciejs]

What bug specifically? I need more details 🙂 Client ports should work fine as of Deadlock Protocol.

[SunBlade_ger]

On 2020-06-29 at 4:17 PM, maciejs said:

What bug specifically? I need more details 🙂 Client ports should work fine as of Deadlock Protocol.

well, if i had to guess, he probably means number 1.

On 2020-06-13 at 11:29 PM, SunBlade_ger said:

1. currently your system relies on NAT penetration, which is unreliable. you should focus primarily on port mapping/forwarding by either upnp or the user.

i mean, the game client already creates port mappings/forwardings via UPnP. i think, after more than seven years now, your systems should finally start using them, instead of wildly guessing ports for UDP punch-through.


so, the question is: when will you change your infrastructure to honour the port setting on all levels?
do we need to wait for the next big update? or can you do that even without a hotfix?

PS.: after watching this forum for a few weeks, i am pretty confident that many of the other networking problems are caused by this bug.

[maciejs]

I doubt it's that, it has nothing to do with trading.

[SunBlade_ger]

On 2020-07-02 at 8:45 PM, maciejs said:

I doubt it's that, it has nothing to do with trading.

well, looking through the forum. most people who have problems trading, simply can not invite or be invited. which i can reliably reproduce by randomizing ports for outgoing connections. which in turn means most of your player base has this bug. that means both players are unable to connect, since WF seems to initiate connections both ways.

[maciejs]

Invites are relayed though, it all goes through our central server, so if you can hit that, you would get an invite. I will wait from more info from the OP. I do not think most of our players randomize their ports, tbh, our data is definitely not showing that.

Can you send me a log from a game session where you think game is not using the port it should be using?

 

[SunBlade_ger]

 

On 2020-07-07 at 4:48 PM, maciejs said:

I do not think most of our players randomize their ports

true, randomizing ports is extremely rare, but it is a very good simulation of a busy router.

On 2020-07-07 at 4:48 PM, maciejs said:

Can you send me a log from a game session where you think game is not using the port it should be using?

i can do that. but testing uncommon nat edge cases by yourself, should be a lot easier and faster. well, here a test i just made.

 

The Setup:

here is an unusual but by no means impossible edge case:

1. i have configured ports 4990&4995 in Warframe for incoming connections.
2. my router has static port mapping/forwarding rules for UDP ports 4990&4995.
3. another person on my network, already uses ports 4000 - 4999 for outgoing connections.

 

Expected Behaviour:

when i host a game, other people will connect to me on ports 4990&4995.

 

What happens instead:

when i host a game, other people do not connect to me.
my router deflects connection attempts on ports 12310&12320 because they have no port mapping/forwarding attached.
other people can join my game as long as i can establish outgoing connections to them.
the static port mapping/forwarding rules i have created, do not get hit a single time.

 

The resulting problem:

i can not play with someone who has a similar setup, despite having ports mapped/forwarded.

 

How to reproduce:

get two router which allow edge case testing. randomize ports for outgoing connections. and disallow incoming connections except for the static port mappings/forwardings.

[SunBlade_ger]

well, do you intent to fix that?

or did you find an error in my deductions or tests?

[maciejs]

I probably will look into this in more detail at some point, but it's rare enough that it's not really a high priority task.

[SunBlade_ger]

i know that mindset, i too make that mistake on a regular basis. every time my code bloats up with the handling of special cases, i should get suspicious about my basics and assumptions. and when i am finished and go through my code once more, i find some errors which ultimately eliminate most of my code when fixed.

well, at least now you know that there are giant sandpits under your castles.

[_Vasur_]

Hi all
just wanted to drop by since my friend and i had a host problem. But since the latest update we were finally able to play again. Since my friend got in February a new router we had a connect issue and only he was able to host and not me but since my friend is quite shy i would normally invite and symbolise to play together. With this bug it was quite demoralising to see that he was unable to connect to my session and after searching online for this problem i came accross this thread. 

Thank you for your investigation. 😄

Cheers