Netcode Question

[xethier]

In the last year the bandwidth requirements for warframe have been steadily increasing.

When i started the average match required 100k/sec or so for the host to be able to peak to. (averaging around 65-75k/sec)

 

now that's almost double. peaking at 160k and being able to sustain 90k-110k.

 

In the last week i've been dropped by the host at 35m in two t3s.

and i can't count how many draco / stephano runs i wind up back at the liset around halfway through wave 4.

 

now. my question.

 

how much longer are we going to be suffering continued netcode bloating and problems before these issues are addressed?

 

I couldn't host this game when i started. (due to limited upload bandwidth and absolutely no way for me to upgrade due to geographical location (ISP offers nothing better than what i have in this area)) -- now, given the number of players and popularity of the game, good hosts are a rarity.

 

i've been a staunch supporter of DE over the last year, but this is starting to push me away from the game. i don't mind the grind, and the effort required to do these types of missions. (i like that) but what i don't like is having HOURS of my time WASTED due to network problems.

Edited January 17, 2015 by xethier

[dandeloreon]

I agree that something needs to be done to the network code to fix the issues, and I can see a few cases that need addressed. All of these cases need to be addressed with various variables taken into account. Also for all cases I think there should be a cool off for reconnecting that is either 15 minutes ( for survival), 10 waves ( for defense ), and a similar case for other longer running missions.

 

Case 1 - Player crashes mid game... for tower missions, they should be able to rejoin as long as a certain time frame limits are meet. I would suggest 30 seconds to 5 minutes. The time range given should be based on the size of patches released since the person ended the game, and how long the session was.

 

Case 2 - Host crashes mid game - This is honestly the worst case, since everyone looses everything. I think that something needs to be done to allow a chance to continue the tower mission with the rewards intact. This is especially important on missions where you get to the 30+ minute mark ( survival ) or the 30+ wave mark ( defense missions ).

 

Case 3 - Random disconnect - This has happened to me a few times, and can be extremely upsetting when you just get something good at that point.

 

insane case - Random session move - This may sound dumb, but i have seen various cases where i go from maximum health and shields to dead because of a host migration. That's right... I died without seeing whatever killed me. This is another major issue.

[xethier]

Those would be useful band-aids for when the netcode decides to fall over.

What I am frustrated about is more fundamental. I want to see the bandwidth requirements reduced and netcode optimized so that it doesn't crap out so often... so those band-aids aren't needed as frequently and missions aren't failed or simply vanish, and more players can host the game without fear of a host-migration.

[RorschachRev]

Central relaying of data via a dedicated SOCKS5 server would be better than P2P in terms of overall bandwidth management required to host. Uploading 1 copy of the data is far better than uploading 4 copies with latency. I linked source code for client patches, and software for server. I'm tempted to just call with a business proposal if they don't reply by the time I finish this project for pay.

 

The most optimized 3D data position code I recall was 1 byte per object, plus the object description. This was heralded as the "theoretical ideal" but I haven't done the math myself. I might take a look at their network data to see how bad it is depending on how bored I get. 

 

For a  "dedicated server" that focuses on data routing only, a quick optimization of network data is finished. Case 1 isn't dealt with, but perhaps the Host could allow a "join" of a reconnected player with a better grace period (a grace period at least as long as it takes me to restart the game after a crash - usually the Evolution Engine is still trying to close the memory leak by the time the grace period is over.) Case 2 is a mix of host migration and data replay, the reasons for the host crashing seem to vary from memory, CPU, network, engine crash, weird bugs, etc. They just need more stable code in general in order to fix some of these issues. The tradeoff tends to be "content or stability" and a middle ground is hard to find. Case 3 not dealt with either, but the grace period may help. Case 4 "insane" is probably due to the new host being unsuitable to handle the requirements.

 

For a server that keeps track of the Scene Graph and loot tables: network code is almost exclusively download, Case 1 is grace period but with easier fix (after doing a lot of ground work and infrastructure), Case 2 shouldn't happen any more (depending on stability of the server), Case 3 is grace period, Case 4 shouldn't happen any more (depending on performance of the server). 

[xethier]

Central relaying of data via a dedicated SOCKS5 server would be better than P2P in terms of overall bandwidth management required to host. Uploading 1 copy of the data is far better than uploading 4 copies with latency. I linked source code for client patches, and software for server. I'm tempted to just call with a business proposal if they don't reply by the time I finish this project for pay.

 

The most optimized 3D data position code I recall was 1 byte per object, plus the object description. This was heralded as the "theoretical ideal" but I haven't done the math myself. I might take a look at their network data to see how bad it is depending on how bored I get. 

 

For a  "dedicated server" that focuses on data routing only, a quick optimization of network data is finished. Case 1 isn't dealt with, but perhaps the Host could allow a "join" of a reconnected player with a better grace period (a grace period at least as long as it takes me to restart the game after a crash - usually the Evolution Engine is still trying to close the memory leak by the time the grace period is over.) Case 2 is a mix of host migration and data replay, the reasons for the host crashing seem to vary from memory, CPU, network, engine crash, weird bugs, etc. They just need more stable code in general in order to fix some of these issues. The tradeoff tends to be "content or stability" and a middle ground is hard to find. Case 3 not dealt with either, but the grace period may help. Case 4 "insane" is probably due to the new host being unsuitable to handle the requirements.

 

For a server that keeps track of the Scene Graph and loot tables: network code is almost exclusively download, Case 1 is grace period but with easier fix (after doing a lot of ground work and infrastructure), Case 2 shouldn't happen any more (depending on stability of the server), Case 3 is grace period, Case 4 shouldn't happen any more (depending on performance of the server). 

 

the problem with dedicated servers from the perspective of DE is overall latency rather than bandwidth utilization. for that model to function properly they would have to have to have colo spots worldwide for the playerbase. that's an expense that (while probably minor compared to the number of users) isn't negligible to their profits.

 

3d data position for some objects in the game i can see as being reasonable. (mobs) but some have no business being tracked between server and host. (ammo, mods, credits) -- (you can see them "lag" when opening a chest while running on a host that's lagging or reaching his peak bandwidth.) -- these can just as easily be served by allowing the client to decide where these objects exist within the game engine and the server simply decides what is spawned, not where. 

 

case one actually does exist within the game, but is invalidated by a host migration. when you restart the game client it will ask you if you wish to reconnect. 

 

case two usually results in a failed host migration and an immediate trip back to the liset with all progress lost. and it happens frequently. mainly due to the increased requirements by the game, usually this is a result of a lower-end pc hosting and not necessarily bandwidth. -- or it's caused by a host stuck with a very low "peak" upload bandwidth limit. especially DSL given the lowered MTU (1492) and packet fragmentation and retransmission when exceeding 80% or so of rated maximum upload. -- this will usually fail over gracefully enough to allow for a host migration, but even a successful host migration can lead to a failed mission for various game-play reasons.

 

case three :: see case one. sometimes you can reconnect by restarting the game client, however, it's not bulletproof. there should be a manual "reconnect to lost squad" button somewhere for players that have lost connection but the client didn't restart.

 

case four this happens less frequently to players, but quite frequently during defense / mobile defense missions as the defense targets will occasionally continue to take damage throughout the migration period. 

 

overall, cases 1, 3, and 4 are the easiest to solve and happen less frequently than case 2.

 

case 2 is my biggest gripe because

a) it results from bad netcode design (host controlling 3d location between itself and clients, compounded by the update frequency)

b) can be optimized.

c) can be mitigated by putting in a one-per-engine-launch upload bandwidth test and invalidating a user without sufficient upload bandwidth for full-sized-squad hosting.

d) is heavily affected by PC performance. (you're gating 3 players by the 4th's system performance) -- they've tried to put in checks for this in host election but without checking users network performance, only ping times & pc speeds... insufficient for number of variables that can cause this issue.

Edited January 19, 2015 by xethier