Network Compression And Congestion Control

[Senketsu_]

Yay

[xyLoneZ]

So much research and work.

 

Thank you for enlightening us with a comprehensive post and your constant efforts for our playing experience! It's really appreciated. Can't wait to test the results with the next update.

This.

 

Also, thanks for releasing internal data just to explain it to us. Not many company, if at all, does that to please their customers.

[XTechnomancer]

Hard work = good things = more fun = DE get more money and praise = DE employees are happy = more hard work = AWESOME NOT AS LAGGY T4 SURVIVAL 60 MINUTE CHAOS FOR ME!!! Thanks for hard work! (It pays off)

Is this constructive or me getting too excited about future improvements?

Edited January 27, 2015 by XTechnomancer

[Wandering_Cyberdragoness]

Will this mean less game-killing lag spikes/forced disconnects?

[intrand]

It's posts/information like this that lends credibility to Warframe still being in "Beta" mode -- but for serious. Without getting too far off topic, it's easy to lose sight of the game and focus on short term financial goals, which I'm sure DE is certainly hitting with the success of Nova Prime Access. I digress.

 

It's great to see some improvements that aren't bandaids.

 

Good work, DE (those of you who pushed for this to be tested and finally implemented).

 

Now don't screw it up :)

[GNAAHH]

Coming from someone with third world internet and having experienced disconnecting from our host yesterday just after completing a T4 void survival where it didn't save any progress...looking forward to these changes. :)

Having the same third world internet, I'm also looking forward to these (and the Strict NAT improvements as well)

[Thaumatos]

    I realize what’s on your mind right now is: “OK, nice graph, but why should I care, will it play better?”. I’m actually both excited and nervous for the next update - yes, the compression and congestion changes are coming on PC this week. Console players will be receiving this down the road.  Compression is easy to quantify and I can guarantee it works as expected. Congestion control is much less predictable so it’s harder to measure the impact. I don’t expect it to be perfect, nothing ever is, I do expect it to be significantly better than what we have right now. It doesn’t affect lag directly, but better congestion control means lower latency as well. We can’t make the data go faster (unless we break the speed of light barrier, working on that next), but we do try hard to make sure it’s not us slowing it down.

 

Will this do anything to the issue of getting your @ss handed to you as a host vs clients being able to sleep through missions?  Will it get better/worse?

[Othergrunty]

Improvements are always nice.

 

Too bad i understand so little of these things so the first idea going through my mind is always if there is any danger involved for me.

 

I guess not.

So yay, for subtile changes.

[-Krux-]

ugh.. i wna work with you guys.

 

you hiring?

[ZizWing]

This sounds promising, let's hope it all works!

 

Also... I finally know the difference between TCP and UDP now :D (needed it for port-forwarding a couple times)

[DeusInsania]

Indeed a great read. Can't wait to test out this stuff too :o

Hopefully now tho, you guys can fix some of the older bugs that are still around :'c and/or the newer ones from u15.

 

Keep up the good fight tho DE

[Jax_Cavalera]

Really appreciated this tech post thanks.  rare that I bother to read these things but this one.. it was really well presented :)

[TheOriginalChaosBlade]

Nice! As a developer (but of boring .NET apps) i like seeing all the details, as little or as more given, But these challenges are always interesting to read and learn about. Much appreciated.

[Mujaki]

You mentioned that Warframe uses network code from "The Darkness 2."

Is there any possibility that these improvements will be implemented in that game as well?

[Fubukin]

Thank you very much for this very informative opening post!

 

I've noticed the game running a bit better lately. I have my PC's LAN connection open on the second monitor (together with MSI Afterburner for all the other interesting stats), as that is currently the only sure-fire way to know whether or not I was chosen to host after a host migration. This is important because I am cursed with just 450kbit up for the foreseeable future.

 

I am very very much looking forward to the better compression, and especially with the upcoming 8 player raids, I guess it it direly needed.

 

After all, it is a sad fact that especially here in Europe, there are countless areas where ISPs still offer contracts at or even below 1mbit/s upload, and most of those either don't have an upgrade option or are hopelessly overpriced. Paying twice as much as you would normally have to for 'gaming internet' is usually not a viable option for a budget.

 

I'd have loved to hear you explain more about CPU load, especially the fps hit that you get when hosting, but I guess that wasn't your focus this time. Bandwidth is, definitely, the more scarce resource for a lot of people.

 

Well, that just leaves the still pretty messy host migration system to be looked over, right? And maybe we could get a little marker somewhere that shows which player is currently hosting. Also, how do you know what someone's upload cap is? Would it be a help if the player can set that in their installation settings (not account settings), so that little number gets requested by the matchmaking to decide who hosts?

 

I mean, I can host just fine for 1-2 players depending on mission type. Soon I can hopefully host stuff like Ext for a whole team without the last joiner having to suffer, right?

 

Really looking forward to that, great work!

[HAYABU5A]

excellent post ! very informative and comprehensive

looking forward to any noticeable improvements however small or subtle they may be 

even 1% better makes it all worth it and at least the netcode is improved going into the future 

[Valador]

Just another reason Warframe is my favorite game, constant Ideas go into it and each one bring it closer to perfection.

[DrCrawford]

great post very informative, if possible i think we would love to have this type of post be more of a regular blog (or semi regular). It gives us the players an understanding of the challenges you face making the game we love run smoother, and honestly it's fascinating to see how the game is going to improve in it's technical aspects.

[RorschachRev]

20 bytes?!?! 

For 3D position and orientation, some games use as little as 2 bytes total. I don't see why you need 5 32 bit numbers in order to store this information. 

1 byte for ID allows for 256 unique object references, I'd make it a cap of 128 by using a 7 bit number

1 byte for rotationallows for 360 degrees, with a 2 degree precision. I'd make it 512 degrees with a 9 bit number (use bit shifting on both pieces of data for fast extraction to a 32 bit number if you need one.)

 

If the positions are tracked by full meter increments, then 1 byte allows for 1 meter resolution up to 256 meters away. I would use a map ceiling, probably 128 meters high, allowing for a lower limit set vertically. 2 bytes per direction would allow for 256 meter maps, with 0.004 meter resolution.

 

That's a total of:

1 byte ID

1 byte rotation

2*3 bytes for location.

8 bytes total.

 

How do you end up with 20 bytes? I can probably just look at the network traffic and figure it out, but I'm just shocked.

[maciejs]

You mentioned that Warframe uses network code from "The Darkness 2."

Is there any possibility that these improvements will be implemented in that game as well?

This ship has sailed, I'm afraid. However, as mentioned, D2 was substantially less demanding than Warframe, so it wouldn't benefit as much from these changes.

 

Thank you very much for this very informative opening post!

 

I've noticed the game running a bit better lately. I have my PC's LAN connection open on the second monitor (together with MSI Afterburner for all the other interesting stats), as that is currently the only sure-fire way to know whether or not I was chosen to host after a host migration. This is important because I am cursed with just 450kbit up for the foreseeable future.

 

I am very very much looking forward to the better compression, and especially with the upcoming 8 player raids, I guess it it direly needed.

 

After all, it is a sad fact that especially here in Europe, there are countless areas where ISPs still offer contracts at or even below 1mbit/s upload, and most of those either don't have an upgrade option or are hopelessly overpriced. Paying twice as much as you would normally have to for 'gaming internet' is usually not a viable option for a budget.

 

I'd have loved to hear you explain more about CPU load, especially the fps hit that you get when hosting, but I guess that wasn't your focus this time. Bandwidth is, definitely, the more scarce resource for a lot of people.

 

Well, that just leaves the still pretty messy host migration system to be looked over, right? And maybe we could get a little marker somewhere that shows which player is currently hosting. Also, how do you know what someone's upload cap is? Would it be a help if the player can set that in their installation settings (not account settings), so that little number gets requested by the matchmaking to decide who hosts?

 

I mean, I can host just fine for 1-2 players depending on mission type. Soon I can hopefully host stuff like Ext for a whole team without the last joiner having to suffer, right?

 

Really looking forward to that, great work!

450 kbps up might be a little bit tight with 3 clients (that's less than 20 KB/s per client), but only in most demanding scenarios (survival). I'm from Europe myself and to be honest in most countries Internet is much better than here in NA :)

 

Host is punished a little bit when it comes to CPU load, that's true. Most of the AI simulation runs on the server and network replication is not free either. If you're playing on a multi-core machine, most of the server networking code should run in multiple threads/background, so impact is smaller than one could expect, but it's definitely not free. We keep trying to improve other areas (AI mostly).

 

Could you be more specific on the 'messy host migration' issue? Can't help if we don't know what's wrong.

We did consider exposing bandwidth settings, but in the end decided against it. Connection quality is determined by many factors and depends on both host & client, so relying on automatic detection seemed like a safer bet.

[Grimini]

I have no idea about this thing is happening! Thank for the enlightenment into such thing which is complicated but yet interesting, i even share with my friends for this partial knowledge of online gaming world~ Thank for all the effort and i will continuing support warframe :D

[Shados]

Nice to hear you guys are finally attempting some significant netcode improvements; living in a country where most people are on extremely dodgy ADSL1/2 connections and have maybe 0.5Mbps upstream bandwidth I can tell you the current situation is crap.

 

That said, it sounds like you're testing your congestion control algorithm on a local network with artificial limits. I understand why you're doing this, but it is a very bad idea to rely on this kind of testing when developing a congestion control mechanism because you're never going to really hit the biggest problem: bufferbloat. You'll only really see this issue in the wild where you have potentially dozens of buffers between localhost and your final hop, and it will S#&$ all over any hope you had of having consistent latencies if you don't take it into account.

Edited January 29, 2015 by Shados

[Verouzer]

Wow. Just wow. Thanks for explaining the details of the situation. We really appreciate your hardwork.

[maciejs]

20 bytes?!?! 

For 3D position and orientation, some games use as little as 2 bytes total. I don't see why you need 5 32 bit numbers in order to store this information. 

1 byte for ID allows for 256 unique object references, I'd make it a cap of 128 by using a 7 bit number

1 byte for rotationallows for 360 degrees, with a 2 degree precision. I'd make it 512 degrees with a 9 bit number (use bit shifting on both pieces of data for fast extraction to a 32 bit number if you need one.)

 

If the positions are tracked by full meter increments, then 1 byte allows for 1 meter resolution up to 256 meters away. I would use a map ceiling, probably 128 meters high, allowing for a lower limit set vertically. 2 bytes per direction would allow for 256 meter maps, with 0.004 meter resolution.

 

That's a total of:

1 byte ID

1 byte rotation

2*3 bytes for location.

8 bytes total.

 

How do you end up with 20 bytes? I can probably just look at the network traffic and figure it out, but I'm just shocked.

Try to give us a benefit of the doubt, guys. We've been doing this 'game programming' thing for a while, we have some idea of what we're doing. I never said we were using 20 bytes for position/rotation (see the 'we don’t use full precision for everything' part). You make a lot of assumptions which are not true in our case. Warframe running under these assumptions would be a game about bunch of avatars walking around in a very small world. Not terribly exciting.To give a better picture:

- we have way more than 256 objects, try thousands,

- our maps are an order of magnitude bigger than 256 meters,

- not all rotations can be restricted to single axis (aim direction for example needs at least 2 angles),

- characters need to replicate way more than only position/rotation. Off the top of my head: animation information, damage information, health, armor, inventory changes, target information, posture. 20 bytes really is not much (and it was an arbitrary number, really, sometimes it's less, in many cases it's more than this). You could maybe squeeze a bit or two here and there, but I've taken this route few times before (and still audit it periodically), low hanging fruit is long gone.

 

Nice to hear you guys are finally attempting some significant netcode improvements; living in a country where most people are on extremely dodgy ADSL1/2 connections and have maybe 0.5Mbps upstream bandwidth I can tell you the current situation is crap.

 

That said, it sounds like you're testing your congestion control algorithm on a local network with artificial limits. I understand why you're doing this, but it is a very bad idea to rely on this kind of testing when developing a congestion control mechanism because you're never going to really hit the biggest problem: bufferbloat. You'll only really see this issue in the wild where you have potentially dozens of buffers between localhost and your final hop, and it will S#&$ all over any hope you had of having consistent latencies if you don't take it into account.

Our system takes more than packet loss into account, we do react on latency changes as well. There are tools that let you simulate bloat to some extent, but I agree it's impossible to emulate a real-world scenarios with perfect accuracy. We'll try to tweak the system once it goes live. In any case, I expect improvements over our current implementation.

[Mathstudent]

So, this should help people with slow upload speed? I have 0.3 Mbps upload speed and in times I host, people usually complain that they're lagging. My download speed is about 3 Mbps so that shouldn't represent a problem, right?