Jump to content

DirectX 12 Debugging


Recommended Posts

Seasons Greetings, Tenno!

Warframe calls for aid!

First of all I want to shout-out to those of you already testing our beta DirectX 12 support! In the first week since launch we’ve turned around fixes for almost all of the crash-bugs thanks to your help!

At DE we have a leaderboard that helps us prioritize the most frequent and serious bugs. Right now, despite about only 1% of players using the beta driver, the top of the leaderboard is currently a DirectX 12 crash:

Bugs.png

The trouble is, unlike crashes in our code, these bug reports give us very little information because the code that’s crashing is actually in the graphics driver. All we know is that we’ve probably done something wrong and the driver crashed.

The bugs come in two main flavors: a common one that happens randomly, and a rare one that happens before you can even login. Although we want to fix both we’re most interested in the rare one because if it happens every time we can set your game into a special diagnostic mode that might tell us what we’re doing wrong.

We’ve also integrated NVIDIA Nsight Aftermath into the game to try to catch these problems in the act -- in theory we should be able to look at the GPU crash dump in a tool at our end and figure out what went wrong. All we need now is someone with the magic touch!

To help all you need is an NVidia GeForce 1050 or better (it doesn’t handle crashes for AMD drivers). If you qualify and accept this challenge:

1) Close Warframe: any edits to your config will be overwritten when the game shuts down so make sure it isn’t running.

2) Find your config: open up File Explorer and paste %LOCALAPPDATA%\Warframe in the address bar at the top. In that folder you should find a file called EE.cfg that you can edit in Notepad. At the bottom there should be a Windows_Config section where you can add Graphics.DebugDriver=3 in there like this:

CFG.png

3) Boot the Warframe Launcher and click on the gear icon in the upper-right corner. Make sure DirectX 12 is selected like this:

Launcher.png

4) Play the game until it crashes.

If you catch a driver crash:

1) Please run through the crash reporter and set aside the WAR-Number.
2) After it submits, go back to the %LOCALAPPDATA%\Warframe folder you had open before and there should be a Crashes folder inside there now.
3) If Aftermath caught a driver-crash there should be a file called GPU.bin there; we have no idea how big these files are -- they might be huge -- so right-click on it and under “Send to” select “Compressed (Zipped) Folder”
4) Go to http://support.warframe.com/ and open a ticket and title it ‘DX12 CRASH FOR DEVS’; attach the compressed GPU file and include the WAR-Number if possible.

Final notes:

- Graphics.DebugDriver=3 only works for DirectX 12
- When enabled it may have a small impact on framerate (again, only in DirectX 12)

Thanks in advance! Once we can ensure DirectX 12 is as stable as DirectX 11 is we can move on to making it even faster and smoother!

  • Like 30
Link to post
Share on other sites
  • [DE]Megan featured and pinned this topic

Too bad that nsight aftermath requires pascal (I have a 970) otherwise I would've helped xD

 

The shader conversion in realtime is insanely stuttery though, it needs a temporary pre-caching option until you guys figure out something more seamless.

Edited by HunterDigi
  • Like 1
Link to post
Share on other sites

Hi @[DE]Glen

Would this work to help isolate the cause of this behavior I reported?

 

On 2020-11-28 at 9:29 AM, Jarriaga said:

No crashes, but this is a reproducible problem:

 

TYPE: In game
DESCRIPTION: VSync "caps" GPU utilization at 55%, severely limiting performance. DX11 uses as much of my 1070 as it can to hit 60FPS, and then triggers dynamic resolution. DX12 doesn't go beyond 55% GPU utilization and locks me down to 30FPS whenever it can't sustain 60FPS. Dynamic resolution doesn't seem to be triggering.  
 

VISUAL: Format -- GPU / Temp (Celsius)  / Utilization % / Memory usage %

1/3 -- DX11 Vsync ON 3840x2160 (71/98/46)

waDvL5M.jpg

2/3 -- DX12 Vsync ON 3840x2160 (65/51/39)

DlANL2Z.jpg

3/3 -- DX12 Vsync OFF 3840x2160 (70/99/47)

dpaS5hO.jpg


REPRODUCTION: Enable Vsync when DX12 mode is active at 3840x2160 native full screen (Refresh rate 60Hz). Other settings used: Nvidia control panel VSYNC mode: FAST, triple buffering is ON, Enhanced graphics engine is active, dynamic resolution is active (Auto).
EXPECTED RESULT: DX12 VSYNC should not cap GPU utilization.
OBSERVED RESULT: Severe performance degradation due to GPU utilization cap when DX12 Vsync is enabled. 
REPRODUCTION RATE: 100% reproduction rate. Disabling Vsync in DX12 or switching back to DX11 corrects the issue without further changes; turning it back on with DX12 locks me down to 30FPS unless I move my camera to spot where nothing is seen. Issue is seen with Vsync modes "On" and "Auto".

I can reproduce it 100% of the time.

Edited by Jarriaga
  • Like 2
Link to post
Share on other sites

That's some interesting data Jarriaga. I know for me I'm only running a 60hz monitor but it does feel somewhat more sluggish somehow. I usually run Vsync off, but cap at 72 fps, which usually feels a little bit snappier on Warframe, most games I wouldn't mind just leaving it at 60. I'm running a GTX 1060 still, and am on DX12 on the deferred engine. Knock on wood, I haven't actually had any crashes yet though.

Something does feel a bit off on DX12 somehow though. I've also had issues seeing GPU utilization myself, it's rather odd, I've noticed Microsoft's regular old generic task manger to check GPU doesn't seem to work properly with DX12. It usually reports 1% or less utilization at any time when normally I'd be getting anywhere between 30-80% on DX11. (Not sure if this does this on other games almost nothing I play has dx12 support.) It's definitely being used though, the temperature goes up when the game runs, and I don't have onboard video at all on this Ryzen 7.

Edited by Drex83
  • Like 1
Link to post
Share on other sites

I've got a 1060 so I can give it a shot... Haven't had any crashes thus far. Any ways I could possibly try to make it crash?

  • Make a lot spawn in Cambion Drift?
  • Toss as many vortex grenades down as I can to create an artificial sun?
  • Organize a full Mirage team spamming Simulor?

I usually play it safe.

Link to post
Share on other sites
12 minutes ago, Debbysheen said:

noice! i've tried DX12 some days ago and the only thing was that it is too choppy, maybe my pc wasn't strong enough, today i got a 3070, im gonna test it out this weekend and see what i can find.

will still be choppy no matter what series you have.

Link to post
Share on other sites
8 hours ago, Sahysa said:

"Too bad that nsight aftermath requires pascal (I have a 980) otherwise I would've helped xD"

Doubly so for me because I have the the crash on DX12 startup. 

 

If only you'd kept the WAR#

Link to post
Share on other sites
6 hours ago, Debbysheen said:

noice! i've tried DX12 some days ago and the only thing was that it is too choppy, maybe my pc wasn't strong enough, today i got a 3070, im gonna test it out this weekend and see what i can find.

This is WIP -- it smooths out the longer you play. We can optimize that but right now we're focusing on making it stable.

  • Like 3
Link to post
Share on other sites
34 minutes ago, [DE]Glen said:

This is WIP -- it smooths out the longer you play. We can optimize that but right now we're focusing on making it stable.

Exactly what i experienced this morning . Choppy at first and after that it got better and in the end almost no difference.

Link to post
Share on other sites
9 hours ago, Arcerinex said:

I've got a 1060 so I can give it a shot... Haven't had any crashes thus far. Any ways I could possibly try to make it crash?

  • Make a lot spawn in Cambion Drift?
  • Toss as many vortex grenades down as I can to create an artificial sun?
  • Organize a full Mirage team spamming Simulor?

I usually play it safe.

We've tried a lot of things to stress test the game to make it crash. Combat stress-test levels, levels with 182 random loadouts, etc -- I don't think the bug is something like that since we've tried quite a lot already.

Link to post
Share on other sites

So far we've had some excellent help!

@R4ndom reproduced three graphics crashes and submitted the GPU dumps -- it's not clear what was wrong but he's going to try some more things for us!

@Srovy discovered a nasty VRAM corruption bug that could occur if you changed certain graphics settings in DirectX 12 -- a fix for that will be coming in the next hotfix.

@CrispLV managed to reproduce some strange rendering artifacts on landscape missions -- huge swathes of the land turning jet-black. Nobody here can reproduce and we suspect it might be specific to GTX 10x0 series cards in a certain config. If you see this while testing we're interested in your specs and settings!

 

  • Like 3
Link to post
Share on other sites
12 hours ago, [DE]Glen said:
21 hours ago, Sahysa said:

"Too bad that nsight aftermath requires pascal (I have a 980) otherwise I would've helped xD"

Doubly so for me because I have the the crash on DX12 startup. 

 

If only you'd kept the WAR#

WAR-2977222 (fresh)

WAR-2975492

There is a partial dxdiag in there.

Edited by Sahysa
Link to post
Share on other sites
On 2020-12-02 at 8:53 PM, Drex83 said:

That's some interesting data Jarriaga. I know for me I'm only running a 60hz monitor but it does feel somewhat more sluggish somehow. I usually run Vsync off, but cap at 72 fps, which usually feels a little bit snappier on Warframe, most games I wouldn't mind just leaving it at 60. I'm running a GTX 1060 still, and am on DX12 on the deferred engine. Knock on wood, I haven't actually had any crashes yet though.

Something does feel a bit off on DX12 somehow though. I've also had issues seeing GPU utilization myself, it's rather odd, I've noticed Microsoft's regular old generic task manger to check GPU doesn't seem to work properly with DX12. It usually reports 1% or less utilization at any time when normally I'd be getting anywhere between 30-80% on DX11. (Not sure if this does this on other games almost nothing I play has dx12 support.) It's definitely being used though, the temperature goes up when the game runs, and I don't have onboard video at all on this Ryzen 7.

I've done some further testing, and the actual trigger is the combination of DX12 + Vsync + Enhanced engine. I thought it was Vsync alone because I have been using the enhanced engine for months and didn't try turning it off until now.

Thus, my options are:

- DX11 Enhanced Engine Vsync = Working as expected.

- DX12 classic engine Vsync = Working as expected.

I will be switching back to DX11 for the time being, as no screen tearing is a non-negotiable must for me as I play on a TV (Hence the reason why I use Vsync) and the enhanced engine is too good to go back to classic. 

Edited by Jarriaga
Link to post
Share on other sites
13 hours ago, Sahysa said:

WAR-2977222 (fresh)

This is very interesting thank you! This is one of our rare startup crashes!

It does smell driver related -- I know you're on very new drivers but there's a few things going wrong here:

- we're asking the driver to create a basic resource and it's saying "No such interface supported"

- your driver is also claiming to not support depth-bounds test and we know your card supports it

When's the last time you let Windows Update do its thing?

  • Like 3
Link to post
Share on other sites

I'm not sure if this helps, but I am having repeated crashes on specifically fissure missions. Particularly, when a fissure spawns (whether on-screen or off-screen). The crash does not happen on every fissure though, only on fissures that happen at least 2 minutes into the mission. I am am on Intel HD Graphics though.

No crashes in Dx11 (not at first at least)

WAR-2980080
WAR-2981088 (latest)

Link to post
Share on other sites
2 minutes ago, Aadi880 said:

I'm not sure if this helps, but I am having repeated crashes on specifically fissure missions. Particularly, when a fissure spawns (whether on-screen or off-screen). The crash does not happen on every fissure though, only on fissures that happen at least 2 minutes into the mission. I am am on Intel HD Graphics though.

No crashes in Dx11 (not at first at least)

WAR-2980080
WAR-2981088 (latest)

We have a fix for that second one coming in the next patch.

  • Like 2
Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...