Lets Fix Warframe's Performance - Heres How

[Fourdee]

U11, great update.

Love the implementation of "Multithreaded rendering".

However, not sure why you would do that, considering Warframes performance lock is due to code loop update, not the render. Always has been since <U5.

 

Heres my Setup:

- Lowest graphics settings (everything off). 720p, HD7770, FX-6300

 

Heres my FPS after 30mins in ODS:

- 2 fps.

 

How to gain massive performance increase on item drops:

- Remove them from constant network location sends on the host (Why have you coded this, WHY?!). Let the client take over.

- Host will only send the ID of the item, ONCE.

- Host then sends Quat/Vector/Float (velocity) of the item ONCE

- Clients now update the items, locally, once recieved from server.

- No more constant item Vector/Quat/Float updates required at host.

- No more laggy items

- Increased performance

- Reduces network bandwidth (i'd expect at least 5x reduction, considering the current, constant item sends)

 

"yeah but wont the items be in different places on the host/clients?"

As long as the new items are sent/recieved, and, processed the same(which should be the case regardless of client/host). Client/Host physx will match.

You could always send a "confirmation" vector packet, from host, on a tickrate basis, or, once idle to confirm location.

 

More performance gains on Items:

- Add max timers to the drops. As Nekro, i see over 1000+ health orbs, with no timeout. A simple timer, even const int ItemTimerMax = 21600; //frames

- For christ sake, add a render limit on the imposters. So simple, yet, you havnt done this. So more performance to be gained from this.

I actually want to cry when i see 2000+ items on screen, regardless of distance, regardless of FPS.

 

 

Multithreading:

- Offload some of the code loop updates into another thread, which runs constantly. Fix the main FPS lock issue.

- Send some "process intensive" updates to that thread. Order your update loop to accommodate the offload.

- eg:

 

unsigned __stdcall Thread_2_Update(void* Dummy)

{

//-------------------------------------------------------------

while(gThread_2_IsActive)

{

gThreadSleepCount = 0;

 

//Raknet Wrapper

Raknet_Update();

 

//Network_Send Update

while(cGame_Network.bThreadNetSendActive)

{

cGame_Network.Update_Send();

cGame_Network.bThreadNetSendActive = false;

}

 

//Culling Update

while(cGame_Culling_System.bThread_CullingActive)

{

cGame_Culling_System.Update();

//cGame_Mesh.Update();

cGame_Culling_System.bThread_CullingActive = false;

//End of thread, sleep for 2ms

gThreadSleepCount = 2;

}

 

Sleep(gThreadSleepCount);

 

}

//_endthreadex(0);

return 0;

//-------------------------------------------------------------

}

 

I offloaded the entire network send loop into another thread, in my game. Performance of code loop is now sublime.

Edited November 27, 2013 by Fourdee

[xethier]

you, sir, are my goddamn hero.

 

i've been screaming for exactly this since i started playing the game, however you seem to have FAR more detailed information that what i've been able to find.

 

not only have you defined the problem (better than i've been able to)

but you've provided solutions.

 

all of my +1's 

 

the current usage of network resources by client is between 15 - 40k/sec depending on load.

this equates to 120k/sec under heavy usage.

for 3 players.

 

talk about lazy coding, good lord.

i was astounded to see how much bandwidth this game uses (and still manages to glitch out)

[silversponge]

If this is true and it seems like it is then it will allow currently the lowest PCs that are able to run warframe (like me) a lot of leeway and may even lead to not only performance improvements but allow me to also increase graphics settings without a massive fps drop :)

Edited November 27, 2013 by silversponge

[NetSlayer]

Regarding particle systems, it is often a cluster of somewhat overlapping particles close together that causes graphics cards to struggle.

It is better to generate fewer particles at birth then have those particles spawn off a child particle.

Hard to explain >.< 

[SpFinX]

Really, this shouldn't die in depths of forum. Let's keep it pinned and before DE eyes! Performance issues on def/survival are really annoying.

[xethier]

i'm not as concerned about graphical performance as the incredibly lack-luster network performance.

if DE wants to host dedicated servers with this bloated network code, fine.

but as someone with limited network resources (and due to regional monopolies) totally unable to improve it...

the network code here is terrible.

 

the OP brings up a very poignant issue in that the server is constantly updating items. why?

let the client handle that. it's irrelevant to the larger game.

i understand mob facing, movement and even the ballistics being tracked. but item drops? constantly? really?

when i saw items spawning from containers and lagging on their way to the floor, i realized what was going on and almost quit playing right there. (glad i didn't) but seriously. this is idiotic coding.

[HeelSouls]

Regarding particle systems, it is often a cluster of somewhat overlapping particles close together that causes graphics cards to struggle.

It is better to generate fewer particles at birth then have those particles spawn off a child particle.

Hard to explain >.< 

 

Does this explain it?

[Morcant]

Wish I could upvote more than once. It's nice to get a coder's perspective. Should PM this to DE_Steve.

[[DE]Rebecca]

Should PM this to DE_Steve.

Will forward to DE_Steve for review, thanks.

[NetSlayer]

 

Does this explain it?

Nice almost there!

alright, the orange child particles inherit the same physics as the birth particles. essentially it would look similiar to the particles spread out on the left. but with fewer birth particles overlapping. Essentially the further away from birth the more leniency there is for overlap as it's less likely to occur.

[[DE]Steve]

"Love the implementation of "Multithreaded rendering.

However, not sure why you would do that, considering Warframes performance lock is due to code loop update, not the render. Always has been since <U5."

But the renderer was in the 'code loop update'. So we took the renderer/visibility out of the core loop which is absolutely why we did that and why there is now half the work to be done on the game thread.

Rough list of things that run threaded outside of the 'main loop':

-Rendering

-Visibility

-Particles

-Physics

-Collision

-Raycasts

-Networking

-Voip

-I/O & Decompression

-Texture uploads

The timers do exist, though are conservative (you have to look away for some of those types) but I agree in principle and will revisit. The dead-reckoning is tricky but it is used for things that don't affect gameplay (corpses, death effects).

The one point I can agree with is our networking code needs love. It is being worked on (see Presence 2.0) with the finite time & people we have. Generally we send network IDs and positions only if they change (duh, of course, since 2004) but new quantization modes are generating dirty flags when they shouldn't so you can get crappy oscillation. For the vast majority of data the server ISN'T constantly updating things that are changing, not sure why anyone thinks that.

Thanks for the time and thought you put into this! Hopefully you see the improvements you're after in a coming hotfix.

[xethier]

"Love the implementation of "Multithreaded rendering.

However, not sure why you would do that, considering Warframes performance lock is due to code loop update, not the render. Always has been since <U5."

But the renderer was in the 'code loop update'. So we took the renderer/visibility out of the core loop which is absolutely why we did that and why there is now half the work to be done on the game thread.

Rough list of things that run threaded outside of the 'main loop':

-Rendering

-Visibility

-Particles

-Physics

-Collision

-Raycasts

-Networking

-Voip

-I/O & Decompression

-Texture uploads

The timers do exist, though are conservative (you have to look away for some of those types) but I agree in principle and will revisit. The dead-reckoning is tricky but it is used for things that don't affect gameplay (corpses, death effects).

The one point I can agree with is our networking code needs love. It is being worked on (see Presence 2.0) with the finite time & people we have. Generally we send network IDs and positions only if they change (duh, of course, since 2004) but new quantization modes are generating dirty flags when they shouldn't so you can get crappy oscillation. For the vast majority of data the server ISN'T constantly updating things that are changing, not sure why anyone thinks that.

Thanks for the time and thought you put into this! Hopefully you see the improvements you're after in a coming hotfix.

 

wow, thanks DE_Steve. 

I'm glad to know that someone is working on the network side of things and I'm sure we're all appreciative of the response.

All my conversations here about network code have been anecdotal as I don't have access to the code and given my fairly limited understanding of it, wouldn't make much sense to me without spending a gross amount of time digging through it. (Not at all a software developer)

 

That said, hopefully Presence 2.0 will show some improvements. Thanks for the heads-up.

 

edit:

 

um... presence 2.0? is there additional information somewhere that's available to non-council members?

I couldn't find anything in the regular forums in regards to it. (and google failed me as well)

Edited November 27, 2013 by xethier

[JustJesse]

wow, thanks DE_Steve. 

I'm glad to know that someone is working on the network side of things and I'm sure we're all appreciative of the response.

All my conversations here about network code have been anecdotal as I don't have access to the code and given my fairly limited understanding of it, wouldn't make much sense to me without spending a gross amount of time digging through it. (Not at all a software developer)

 

That said, hopefully Presence 2.0 will show some improvements. Thanks for the heads-up.

 

edit:

 

um... presence 2.0? is there additional information somewhere that's available to non-council members?

I couldn't find anything in the regular forums in regards to it. (and google failed me as well)

I just looked through the Design Council forums, and there doesn't seem to be any mention of Presence 2.0 there either. Seems that we've all got something to look forward to.

[[DE]Steve]

Ah sorry - this was our internal name for this part of U11: "The contact-list network protocol has been completely rewritten to work-around Strict NAT and network problems caused by having a large number of contacts (the new system still being tuned, please be patient)"

[xethier]

Ah, gotcha. Makes sense now.

I'm running into a problem with the new system though. 

 

I run a custom-built router as part of my effort to speed up my local connection. (i'm on a terrible DSL due to regional monopoly issues)

I can't get something better. 

 

My dsl is 85k/up and 550k/down. I run a very robust QOS setup etc.

The new system (due to the fact that my ping-times are very low partly due to my lightning fast router) still considers me a good choice for host.

Even though under load i can't sustain enough bandwidth for 3 clients.

 

The end result is that my sessions online are still godawful for those unfortunate enough to connect to me. (see: void missions (especially survival and defense))

 

I'm hoping to see the bandwidth upload requirements for this game reduced out of the stratosphere enough so that I can feasibly host.

 

Maybe I'm just going to have to wait until 2025 when my local isp decides to stop being greedy... 

But I'd love to see some optimization done to the netcode to reduce bandwidth down to 50-60k/sec max.

[Fourdee]

Will forward to DE_Steve for review, thanks.

Thanks Rebecca, appreciate it.

 

 

Reply

Interesting, thanks for the detailed insight/information. Feeds the geek inside of me (sad i know lol).

 

As i don't have access to your source code, i can only guess from my experience how you guys are implementing stuff.

 

 

The timers do exist, though are conservative (you have to look away for some of those types) but I agree in principle and will revisit.

Brilliant, good to hear!

 

 

The one point I can agree with is our networking code needs love. It is being worked on (see Presence 2.0) with the finite time & people we have. Generally we send network IDs and positions only if they change (duh, of course, since 2004)

Again, as above, great to hear.

The networking code of warframe is the backbone of the players experience. Glad to see its not being ignored :)

 

 

but new quantization modes are generating dirty flags when they shouldn't so you can get crappy oscillation.

I have noticed these on infested Ai in laggy situations (when the mesh scales on one axis etc). Seems the quat becomes errored and does not appear normalized.

 

 

For the vast majority of data the server ISN'T constantly updating things that are changing, not sure why anyone thinks that.

As above, without source code, the players can only guess from what they see, and, previous experience.

And example is when opening a locker connected to a laggy host. The items slow down (controlled by host). Whether thats due to a High resolution timer multiplier on the position updates, controlled by host performance, iam not sure.

However, you do get the feeling that each position update is being sent from the host, over network, because of the visual slowdown.

 

 

Thanks for the time and thought you put into this! Hopefully you see the improvements you're after in a coming hotfix.

We might moan/cry alot, but, the players are here because they enjoy/care about this great game.

This game is gold dust, and all that hard work is paying off. Just hope you guys can find the time to improve its backbone.

Looking forward to it Steve, and, thanks again for the detailed replied. 

 

 

ps.

A few things i forgot to mention in my OP:

 

I also forgot to mention Vortex ;) :

- Any chance you guys will remove items being effected by the Vortex ability? 

I think it would be worth the performance result. If gameplay impact is too much, just disable ammo items from vortex physx, might help?

 

Auto host assignment:

So glad to see this being implemented. However, i think it needs a touch more love:

- Doesnt appear to work for Voids/Orikin? Seems the key holder = host. Would be nice to see this automatically adjusted for best host.

- Enable a option in a match to vote for a new host, then, if successful, do a host migration to the "best" host. No end of times i've joined matches (250ms ping limit) and had to leave because its unplayable.

 

Anyway, thanks again for your time and info Steve.

Looking forward to seeing improvements in the future.

Looking more forward to being able to break the 30min survival 2fps performance issue and get 1hour + ;)

Edited November 28, 2013 by Fourdee

[Towik]

Hello 

 

I would like to point out that , I no longer can play Warframe

 

after the 11.0.1 or 11.0.2 Hotfix , my gpu is being capped to 30% , doesn't matter what options i choose it's always capped ever since one of those Hotfixes . I mean U11 update , gave me better performance but after one of those hotfixes that i mention ,everything is going nuts .

 

I though with the PS4 release , coming soon , the PC users would get a huge improvements but like I said all went crazy after 11.0.1 or 11.0.2 .

 

I am not sure.

 

I'm planning to put my 550 FPB as Dedicated Physx card and buy a 7870 or 7850 as main GPU , the thing is would anyone know how to do that , I mean from the side of drivers and so on.

 

Take care.

Edited November 28, 2013 by Towik

[Garompa]

Yes! I remember! I KNOW WHAT THE PROBLEM IS, thanks OP!

 

Before all this lag existed. I remember mods, and resources where client side! now they're host sided...

How do I know this? well because my friends and I would get completely different mods from the same drop! Only resources were the same but the quantity varied.

 

Bring it back, client side of things.... this lag is unplayable

[Vercinaigh]

Yes! I remember! I KNOW WHAT THE PROBLEM IS, thanks OP!

 

Before all this lag existed. I remember mods, and resources where client side! now they're host sided...

How do I know this? well because my friends and I would get completely different mods from the same drop! Only resources were the same but the quantity varied.

 

Bring it back, client side of things.... this lag is unplayable

That isn't how it worked before, or ever. The only thing that changed is some mods drop the same for all people, Void is still random rather than set for a team, wish it was all set like all getting the same mod.