Network Compression And Congestion Control

[AMACOMX]

this was so informative :D :D but i have a question though !!

 

the time it takes  this 100 MB file to be devided into small chunks ... compress each chunk send to client then decompress then recollect them into 100 MB file ...wouldn't it be faster if u just send smaller chunks from the very beginning ? i mean saves the time and processing of the compress/decomress/devide/recollect 

[hoeskioeh]

MOAR!

 

i really love these semi-technical posts! <3

 

You guys are awesome!(*)

 

--

(*) post-u14 gui designers excluded :-P

[maciejs]

So, this should help people with slow upload speed? I have 0.3 Mbps upload speed and in times I host, people usually complain that they're lagging. My download speed is about 3 Mbps so that shouldn't represent a problem, right?

It helps both with download & upload, since there's less data being transferred, but in practice, download bandwidth is rarely a problem, most conneciton offer more downstream than upstream and also client is receiving data only from the host (and maybe some VOIP), while the host needs to send information to N clients. As mentioned above, 0.3Mbps is a little bit low for the most frantic scenarios, but with new update there should be a visible improvement.

 

this was so informative :D :D but i have a question though !!

 

the time it takes  this 100 MB file to be devided into small chunks ... compress each chunk send to client then decompress then recollect them into 100 MB file ...wouldn't it be faster if u just send smaller chunks from the very beginning ? i mean saves the time and processing of the compress/decomress/devide/recollect 

Hah, that'd be easy if we just knew upfront we just have to send a 100MB file. It's not how games work, though. We generate information on-the-fly and send small updates every frame (or whenever it differs enough).

[IronRushBR]

I am proud for support this game so many time since the first month, really. We are improving every month.

[Aqualuna]

Do you update monsters players etc at the same rate? (in your example 30 updates/s)
Do compression algorithms add latency? That is, instead of being able to send the gamedata immediatly after it has been generated you would wait for more data to be available to compress it, with the side effect of adding latency.

 

[maciejs]

 

Do you update monsters players etc at the same rate? (in your example 30 updates/s)

Do compression algorithms add latency? That is, instead of being able to send the gamedata immediatly after it has been generated you would wait for more data to be available to compress it, with the side effect of adding latency.

 

 

Update frequency depends on perceived importance (for this particular client). Importance depends on many different factors (distance, state, are we fighting this guy, is he moving fast and so on). Compression algorithm doesn't add extra latency, one of the reasons we chose this library is that it deals very well with fairly small buffers.

[Evil-Ninja]

I am a new Warframe player and so far I am impressed by Warframe's game performance, a few bumps once in a while when new players are dropping into an instance and a couple of network drops (apparently unrelated to Warframe, modem issues) but no real complaints with the game play. In fact compared to what I have been playing recently it is a fabulously smooth gaming experience.

 

Like someone else here said, I am both excited and concerned with performance "enhancements" at this point, however if it performs just as good or as better than before you can expect that I will continue to play and financially support Warframe either way.

 

PS: Now if I could just get along better with Steam...

Edited January 29, 2015 by Dyremaker

[Seffard]

All of this means that more people with more limited bandwith can hopefully afford to be a good host.

[ZeyadEbeed]

هاي

[-TSA-TimSin]

I really appreciate those insights in your work. Thanks for sharing. :)

[Y3H0]

Thanks for the comprehensive explanation but just being curious, did you guyz tried LZ4 ?

[maciejs]

Thanks for the comprehensive explanation but just being curious, did you guyz tried LZ4 ?

We have. It's a great algorithm, but again - more suitable for bigger chunks of data. I make sure to keep track of what Yann's doing, though, tried his latest creation as well (zstd).

[Aqualuna]

I was wondering if you are training the compressor to generate dictionaries for different game modes? (def, surv, etc). Or wouldn't it make any noticeable difference? Also when you train the compressor are you training it to reduce data in the high throughput situations or the total data transferred during a game? I would think the RAD tool is sophisticated enough to allow you to chose what you want to compress more...

[Loswaith]

Thanks for the info, its very informative and I personally find this kind of thing quite interesting (being a programmer myself).

 

...

- re: pickups (BrazilianJoe) - most of the 'heavy lifting' is done client-side so that we don't have to spam the network with constant updates (it has some similarities to approach #1 on your list). Spawning a bunch of pickups will result in a small spike, but unless we're talking crazy numbers (hundreds at the same time), this shouldn't significantly affect your bandwidth. Once they come to rest they're pretty much free as far as network is concerned

...

With reguards to pick-ups:

 

While you mention that the client does a lot of the heavy lifting in this reguard.  With more significant lag you can notice that loot actually will stutter its way through a drop trajectory and/or can take considerable time to pick up (given you can still kill enemies while waiting I'd presume pickups are given low priority).  From this it seems that the locations on these bits of loot are waiting on responses from the host rather than letting the client handle all of this aspect themselves (ie trajectories, position locations and pickup handlers for that client).

 

For the majority of drops (mods and specials excluded due to position markers) it seems that aside from the initial spawning of the loot having the client handle all other aspects from then on out, without host conformations would be the better way to go and reduce a reasonable chunk of network traffic being used for this (presuming this is actually the case).  Given the vast majority of loot is ammunition, resources, health and energy orbs, which is instanced to individual players already, moving it off to the client would see little difference for the gameplay overall but make for a smoother visual and responsiveness while freeing up some network headroom for more important data.

 

Obviously for mods and other loot with the visual markers you would want that handled by the host so all players know the rough locations, but even now that appears to be an aproximate area rather than specific point, thus clients could handle some of the drop animation traversals as well.

 

I have also noticed that for reloading a weapon with lag that it appears to be waiting on reponse from the host before actually doing the reload.  Where simply sending it to a client/host without waiting for a response for the animation could make for more responsive result as it would matter little if players are slightly out of sync on the reload animations from their squadmates (and they could even potentially be completly droped altogeather if congestion gets high, as for the most part you dont really see other player avatars reloading in the heat of combat anyway.  Though I've no idea if this would cause other issues from the engine).

Edited February 1, 2015 by Loswaith

[xethier]

I've been begging for these types of updates for over a year. I'm very pleased that these types of issues are being worked on and improved. Thank you DE!

[maciejs]

I was wondering if you are training the compressor to generate dictionaries for different game modes? (def, surv, etc). Or wouldn't it make any noticeable difference? Also when you train the compressor are you training it to reduce data in the high throughput situations or the total data transferred during a game? I would think the RAD tool is sophisticated enough to allow you to chose what you want to compress more...

There's a global dictionary for the time being. Our traffic characteristcs are not that much different between game modes. I've given it training data collected from variety of missions, with the most demanding types being slightly overrepresented (but that's natural, since single session of survival generates way more data than single extermination for example).

 

Loswaith (re: pickups) - it's data driven, but for most pickup types, we actually do almost what you describe, ie. send the creation message + spawn pos + initial velocity and let the client handle the rest himself. It's a little bit more complicated as there's a system that tries to make sure they don't drift apart too far, but that's the gist of it. IIRC not all pickups use that system, but vast majority does. Typical game session uses maybe 1% of total traffic for pickups (we have a net profiler that shows us exactly where the bandwidth is going). Tbh I don't know too many details about how we handle reloads, I'll take a look/talk to a programmer in charge on Monday, see if we can improve it.

[Aqualuna]

There's a global dictionary for the time being. Our traffic characteristcs are not that much different between game modes. I've given it training data collected from variety of missions, with the most demanding types being slightly overrepresented (but that's natural, since single session of survival generates way more data than single extermination for example).

 

Loswaith (re: pickups) - it's data driven, but for most pickup types, we actually do almost what you describe, ie. send the creation message + spawn pos + initial velocity and let the client handle the rest himself. It's a little bit more complicated as there's a system that tries to make sure they don't drift apart too far, but that's the gist of it. IIRC not all pickups use that system, but vast majority does. Typical game session uses maybe 1% of total traffic for pickups (we have a net profiler that shows us exactly where the bandwidth is going). Tbh I don't know too many details about how we handle reloads, I'll take a look/talk to a programmer in charge on Monday, see if we can improve it.

 

Thanks for the detailed response. Do you think we could see some network stats in the UI (latency, packet loss, throughput, etc) in a near future? Many times players are downloading stuff or the LAN is being used by family members or friends, or the ISP is just sucking at some time of the day, it would be great if you could give us tools that could help us improve the game experience.

[Loswaith]

There's a global dictionary for the time being. Our traffic characteristcs are not that much different between game modes. I've given it training data collected from variety of missions, with the most demanding types being slightly overrepresented (but that's natural, since single session of survival generates way more data than single extermination for example).

 

Loswaith (re: pickups) - it's data driven, but for most pickup types, we actually do almost what you describe, ie. send the creation message + spawn pos + initial velocity and let the client handle the rest himself. It's a little bit more complicated as there's a system that tries to make sure they don't drift apart too far, but that's the gist of it. IIRC not all pickups use that system, but vast majority does. Typical game session uses maybe 1% of total traffic for pickups (we have a net profiler that shows us exactly where the bandwidth is going). Tbh I don't know too many details about how we handle reloads, I'll take a look/talk to a programmer in charge on Monday, see if we can improve it.

Thanks for the info, good to hear more about whats going on :)

 

It does seem odd then that loot can excessively slow down on their trajectory (ie. it can take 3-5s for loot to hit the ground, like it is traveling with slow frame rate) with some lag and it can also take up to 60s at times standing ontop of pickups to actually pick them up (dispite happly killing enemies immediatly, and the game running mostly smoothly).

It makes the game feel rather unpolished (which is why I presumed it was being host handled, with a low priority), prehaps this is caused by other issues in the code that in future may need to be looked at (as it isnt an immediate issue, more an issue of the game appearing to be polished).

 

Thanks for the work on the code/game and sharing the information of this processes.

[maciejs]

For the actual pickup action, it still needs to send an information to the host, so there might be some latency, but 60s sounds crazy. I'm still tweaking the congestion control a little bit, but once I'm done, I'll do another iteration on pickups.

[LisRestall]

I just stopped by to say thanks for doing a great job. I've done some ODS runs earlier using Vauban and noticed that I'm downloading way lesser packets from the host unlike before.

[BrazilianJoe]

For the actual pickup action, it still needs to send an information to the host, so there might be some latency, but 60s sounds crazy. I'm still tweaking the congestion control a little bit, but once I'm done, I'll do another iteration on pickups.

 

 

Loot drop angles are a rather inconsequential event which is not significant physics-wise. Maybe they can be further simplified?

 

To further optimize the drop logic & network comm, the drop angle randomization could be originated from a PRNG. This way, you only transmit the seed, and only once per drop source, all the drops picking up sequential numbers from the PRNG on the client machines after that. 

 

http://en.wikipedia.org/wiki/Pseudorandom_number_generator

 

PRNGs could be used for a lot of inconsequential events. 

 

A long-period, high quality PRNG could even be used for more significant stuff, such as angle variation on bullet trajectory in the accuracy calculation. As long as the seed is in sync, every machine shoots the bullets the same way.

Edited February 3, 2015 by BrazilianJoe

[xethier]

For the actual pickup action, it still needs to send an information to the host, so there might be some latency, but 60s sounds crazy. I'm still tweaking the congestion control a little bit, but once I'm done, I'll do another iteration on pickups.

 

this is great to hear. frequently i see very long delays for mod and resource pickups. i'm not overly concerned about it specifically but it gives the appearance that it's "slowing down the works". by your explanation... it isn't. 

 

this push to optimize the netcode and performance by those of us with limited upload bandwidth is a HUGE plus. this is the single biggest gripe i've had with warframe since i started playing. I'm ecstatic that you folks are pushing into this.

[BrazilianJoe]

Since the new netcode, I have been experiencing much more of: 

 

 

Which is precisely the kind of gameplay disruption the new code aims to avoid. 

 

I experience enemies and warframes running in place, enemy and player "teleport", making for a very janky experience. It does not happen every time though. 

 

I am on a good network connection (35Mbps down - 3Mbps up) with not a lot of traffic going on, i.e. no torrents. There is the occasional Minecraft/Netflix/Youtube/Facebook going on because of the other computers, but that's not nearly enough to choke the bandwidth. 

 

I am available to join a group of fast + slow connections to make test cases as host and client. I live in Brazil, so it might be a good one to make long distance team tests because of the lag as well. 

Edited February 6, 2015 by BrazilianJoe

[JujuHex]

A bit late to reply to this, but the net-code update has improved my experience incredibly!  Better load times, less latency/lag in game, less connections dropped. Great work maciejs, I am impressed. As I side-note, I have no problems when playing with people on the other side of the world, but often when playing with real life friends or others in my country, someone tends to drop out with alarming frequency.  I'm sure this is not a problem on your side, and more an issue of our IP's, as close as they are, not liking to talk to each other and having poor infrastructure in general.  Again, great work, and thanks for the in-depth explanation of what happens behind the scenes, I actually learned something!

[maciejs]

As far as disconnections go, if you could send me logs from a session when this happened, ideally both from the host and client who disconnnected, I could verify and make sure it's not us at fault. If anything, my recent changes should make connection a little bit more stable, but there might be some bugs still lurking.