The Great Ensmallening

[Kontrollo]

On 2020-09-09 at 9:55 PM, [DE]Glen said:

In the second update we will apply this same technology to the rest of the textures in the game and we expect we’ll see similar if not more savings. Again, we spent time looking carefully at the quality tradeoff and again we’re extremely pleased. Here’s a particularly challenging case: a normal-map:

Before (19 MB)

After (8 MB)

In this case the compression reduced the footprint from approximately 19MB to 8MB and the differences are extremely difficult to discern.

That looks more like a bump map, unless that's already the rendered result, just not on the final geometry? But technicalities aside, this is actually very interesting, although I don't know a lot about image compression theory, these formats or the render pipeline.

Now the real question: won't you have to inspect every single one of those bump/normal (and other non-RGB) textures rendered with the game engine on their actual geometry? AFAIU it's not enough to just visually compare these in texture form, e.g. they don't have gamma. And normal maps "don't really make sense, visually".

Sounds like a lot of menial work. That said, I guess you might just pick a safe, low "pre-compression" value (what they call lambda) for these kinds of "riskier" textures and get the high savings from RGB textures, especially where they have low impact (i.e. not Warframes, weapons, cosmetics, etc).

[YUNoJump]

Read the Dev Workshop on it, has all the information you'd need. TL:DR they're doing part of it next week and other parts later on so that people don't have to redownload the entire game all at once. If you need to redownload the full thing anyway then you might as well just do it now and get the updates, because it'll be at least 3 weeks before the changes are fully complete.

[DeMonkey]

29 minutes ago, ThorienKELL said:

Supposedly change should get implemented "next week"

Wait, if you know it's next week, why ask if it's happened yet?

[ThorienKELL]

9 minutes ago, DeMonkey said:

Wait, if you know it's next week, why ask if it's happened yet?

Because I've read it few days back?

[OneYenShort]

9 minutes ago, DeMonkey said:

Wait, if you know it's next week, why ask if it's happened yet?

Probably because things change and sometimes things have come out early.

[DeMonkey]

4 minutes ago, ThorienKELL said:

Because I've read it few days back?

You say that as if it's a valid and obvious answer. That's not how time works.

5 minutes ago, OneYenShort said:

Probably because things change and sometimes things have come out early.

Such a change and subsequent release would likely be detailed in the dev workshop and follow up patch notes.

[(XBOX)Father Fro3091]

Xbox needs a reduction we are still at 67.8 gb.

play station and pc have both gotten reductions already. Is this a Microsoft thing? 

[ThorienKELL]

9 minutes ago, DeMonkey said:

You say that as if it's a valid and obvious answer. That's not how time works.

Don't you have anything ...better to do? (*checks the number of forum posts) - bah you clearly don't. 
I've read it few days back, I have no way of knowing how old was the article in the first place, I wanted to get more info.
Does this satisfy you? Can we end this pointless interaction now, pretty please? 

[DatDarkOne]

18 hours ago, TheLexiConArtist said:

I've had a lot of problems cropping up over the years, things started going downhill some time after the Tridolon update. Lots of it lately involves Warframe absolutely hammering my disk IO.

Hopefully these ensmallened textures will take some of that load off. If not, I really need to profile the damn thing and find out what it's hitting so relentlessly.

Have you tried locking your pagefile size?  I do that with all my Windows installs to help avoid some of that.  

[quxier]

5 hours ago, DatDarkOne said:

Have you tried locking your pagefile size?  I do that with all my Windows installs to help avoid some of that.  

But if OS uses pagefile it means that there is "not enough RAM" (it might be for example 500MB free). Wouldn't it just break the game, especially in the open worlds?

[DrivaMain]

On 2020-09-12 at 6:02 AM, (XB1)KSI Father Fro said:

Xbox needs a reduction we are still at 67.8 gb.

play station and pc have both gotten reductions already. Is this a Microsoft thing? 

Good god, 67.8 GB? Have you tried verify game cache? That’s a lot of junk files right there.

[DatDarkOne]

21 hours ago, quxier said:

But if OS uses pagefile it means that there is "not enough RAM" (it might be for example 500MB free). Wouldn't it just break the game, especially in the open worlds?

I said to lock the pagefile size not delete it.  Meaning to set the minimum and maximum size to be the same.  It can still be as large or as small as you want.  It just won't be using processor and drive write times to try and recalculate the size on the fly.  

This has more performance effect under load on HDD than SSD/M.2

edit: as for the breaking the game part, that shouldn't happen unless you have a low amount of ram.  Like 4gigs or lower.  Having 8 gigs or more and you can pretty much safely go without a pagefile at all in most situations.  

[Aadi880]

On 2020-09-12 at 1:35 AM, Vlada91 said:

but i just don't understand why they doing this...its not like COD warzone with 500gb and that they need to do this.it's ridiculous 30gb.in that spend time they need to work on something more important future update and not this nonsense

To save space?

The fact that it isn't like COD warzone is exactly why they are doing this. Warframe caters to an extremely large group of players, players that may be on budget, players that play on old specs, or people that run on external hard drives/SSD, and ALSO nintendo switch. NOT doing this will just come back to bite in the future, where loading times becomes significant enough for high end users to notice.

The non-functional requirements of a software is just as important as the functional ones. There is zero benefit in making fixes and new content if nobody can play it due to file size.

[taiiat]

 

On 2020-09-11 at 2:31 AM, Javrendei said:

- If "client optimization" main purpose is to clear outdated resources from .cache files, why not optimize those files one at a time and break the big files (notably F.TextureDx9) to a few smaller ones?

the reason why the main Assets are generally blocked together as one File in games, is that helps optimize Seek Times on Hard Disks (and even SSD's to a minor degree!).
having buckets the size of oceans isn't that convenient to work with, but it benefits the end user so smart games try and do it.

On 2020-09-11 at 7:55 AM, Sokdenumeros said:

1 - I assume after decompression the texture will use the same ammount of vram as before right?

should be essentially the same, yeah.

On 2020-09-11 at 3:35 PM, Vlada91 said:

but i just don't understand why they doing this...its not like COD warzone

because Digital Extremes has some staff that care about being efficient. tidying up your home once or twice a month means a lot less total work than tidying up once every decade or something, hehe.
same deal for the games, if you clean up on a somewhat regular basis things just stay a lot more organized overall.
do a little sometimes to save a lot later and such, if you get my drift.

On 2020-09-11 at 6:28 PM, Zahnny said:

Forum post specified 6.6gb, then I saw an article saying 15gb and now 25gb

in total, the game will be about half the size it is now, give or take. but it's being deployed in 3 batches. 

On 2020-09-12 at 3:25 AM, quxier said:

But if OS uses pagefile it means that there is "not enough RAM" (it might be for example 500MB free). Wouldn't it just break the game, especially in the open worlds?

sometimes yes, sometimes no. Applications may intentionally store some stuff in PageFile if it's so low priority that there's no sense in keeping it in RAM. Windows also does some dumb stuff with data sometimes but that's a different matter.
this is why Technicians would never recommend disabling your PageFile even if you had 64GByte of RAM - it still has value and Applications may try to use it and if you don't have it that screws with things.

you're not wrong though, if the System has no more free Memory (actually free, Applications that request Memory but don't store anything in it is the same thing as being used since it's unavailable but in a Memory usage chart it doesn't show up since technically it's empty but other Applications can't use it so there's no difference), then it will put stuff in PageFile in an emergency measure. but the PageFile gets used for more than strictly that purpose.

8 hours ago, DrivaMain said:

Good god, 67.8 GB? Have you tried verify game cache? That’s a lot of junk files right there.

it's not out of the ordinary for Consoles, the rules that MS and Sony put onto games forces them to store data less efficiently.

[Javrendei]

4 hours ago, taiiat said:

the reason why the main Assets are generally blocked together as one File in games, is that helps optimize Seek Times on Hard Disks (and even SSD's to a minor degree!).
having buckets the size of oceans isn't that convenient to work with, but it benefits the end user so smart games try and do it.

It wont be much difference in Seek times between one single 30 GB (it is usually fragmented anyway) and five 6GB files, but it could eliminate that "You need to have 30 GB free space on your drive in order to start optimization" issue. Make files smaller, optimize them in order one by one.

[ChonkyTotoro]

thanks alot! It can help while im working for a new better pc

 

[taiiat]

6 hours ago, Javrendei said:

It wont be much difference in Seek times between one single 30 GB (it is usually fragmented anyway) and five 6GB files

it actually can make a big difference. you lose a lot of time opening and closing Files, and when data is split across Files there is no way for an Application to be able to ensure that data is as contiguous as possible.
furthermore having a large block of data allows you to have a lookup table for your own data so that you can find things faster than the Operating System would be able to since you already know where all of the data inside of that block is.

 

i know, it comes with some downsides, and they can be a bear to deal with - but it has many upsides and one of the big upsides is helping the reasonable Loading time performance of Hard Disks.

[NoSpax]

Am 12.9.2020 um 01:02 schrieb (XB1)KSI Father Fro:

Xbox needs a reduction we are still at 67.8 gb.

play station and pc have both gotten reductions already. Is this a Microsoft thing? 

Yes. You basically have a warframe folder the size of two clients.

Microsoft has been doing malpractice on their own store, which caused players to download 40 GB again and again and again. Just google "PSO 2 NA release disaster" or so. This revealed a massive design flaw on the store, because PSO2 has a flaw on non-JP OS which literally destroyed the DRM from the MS store from the inside out and also clogged the harddrive of wasted space in the 100+-ish GB.

I have that odd hunch, the Xbox treats the initial download as a ROM and moves patches to a "patch" folder, where the recent version actually resides. Warframe updates REPLACE the base files, xbox seems to add them on top AND replace them there. 

TL;DR: What happens, if you uninstall the game and re-download it? Will it get cut in half?

[Xarsis01]

Awesome! Smaller file sizes are great and those images where so similar from my (admittedly) quick glances I couldn't even tell a difference!

[NoSpax]

vor 3 Stunden schrieb taiiat:

furthermore having a large block of data allows you to have a lookup table for your own data so that you can find things faster than the Operating System would be able to since you already know where all of the data inside of that block is.

Warframe already uses a TOC system like a wierd indexed TAR file or ZIP or whatever binary archive. Even with this, you have loadtimes up to a minute on HDD. This causes players to get dropped from a group. You can spot HDD players easily on open-world scenarios. 

The warframe gamefilea are already split up into "categories" (I have yet to get a sense wtf all those prefixes H,F,E of each fileclusters mean). A few more wouldn't hurt, SSD/M.2 already give a f**k how many files you have to read.

If you load a corpus level, why load stuff from a massive 29 GB file, if you can just take what you need from a file of 6 GB? You are using filehandlers anyway, HDD need to seek anyway. IF want something to drink, do you buy just a bottle or the whole box?

When you are on Fortuna, you can even keep that filehandler for the Corpus Textures open, when a player goes back and forth on Fortuna to OV.

If the player goes back to the orbiter, flush all that and load the Main files. 

Transitioning from the huge Texture file to seperate files would need a while during update cycles, so you can mark Textures in the old file as blank and just put them into a new file. The blank space will be optimized to freespace anyway. 

This is what DE could do, just download the new Textures into a new file and blank them in the other. This way, the old file would get smaller and smaller until it completely vanishes to 5KB or so and the "F.TextureDX9" filename can be dropped. Need a new faction, just add a file for them.

Warframe seems to be modular, why not go along that idea?

Here is another pro: If the master texture file gets corrupted due to a bad HDD/filesystem, you need to redownload 30GB just to fix a damage of 32KB. Unless there is a detection for the internal EE filesystem I am not aware of.

if only Grineer textures were damaged, just that 4-6GB.

[taiiat]

5 minutes ago, NoSpax said:

Warframe already uses a TOC system like a wierd indexed TAR file or ZIP or whatever binary archive. Even with this, you have loadtimes up to a minute on HDD. This causes players to get dropped from a group. You can spot HDD players easily on open-world scenarios. 

If you load a corpus level, why load stuff from a massive 29 GB file, if you can just take what you need from a file of 6 GB? You are using filehandlers anyway, HDD need to seek anyway.

IF want something to drink, do you buy just a bottle or the whole box?

 

corruption

yes, Landscapes request a lot of data at once and so Hard Disks perform poorly in those regions. the strategy is very effective elsewhere though, when the system is not already under load and then trying to load additional data ASAP.

the size of the File is not an issue for Loading, because the system isn't parsing the entire File at launch or anything. the entire point is to not do that. self managing your data means that a big File that is more likely to be contiguous doesn't come with any performance drawbacks while offering the potential Latency gains.

buying a single unit or a Case is not a metaphor that is applicable to this scenario.

 

corruption is a downside of having very large banks of data, yes. though you could remedy that by each section in the bank of data having its own Checksum, then if one part gets corrupted it wouldn't affect the rest. simpler without  losing some of the benefits that are currently present.

[OneYenShort]

18 hours ago, taiiat said:

you're not wrong though, if the System has no more free Memory (actually free, Applications that request Memory but don't store anything in it is the same thing as being used since it's unavailable but in a Memory usage chart it doesn't show up since technically it's empty but other Applications can't use it so there's no difference), then it will put stuff in PageFile in an emergency measure. but the PageFile gets used for more than strictly that purpose.

There is one slight gotcha in your statement.  When programs request memory, it is expecting contagious blocks of memory.  The OS doesn't report free vs. largest contiguous free.  The memory control aspects of the OS does very little RAM defragging.  This is why sometime you'll still get "out of memory" issues, even though it seems like you have plenty.  Pagefile might help some aspects of this, but not all.  (I run into this issue with my work.)

Now, when it comes to the graphic driver calls to use onboard vram, that aspect I don't know anything about.  I've avoided that area of low level coding so far.

[DatDarkOne]

44 minutes ago, OneYenShort said:

There is one slight gotcha in your statement.  When programs request memory, it is expecting contagious blocks of memory.  The OS doesn't report free vs. largest contiguous free.  The memory control aspects of the OS does very little RAM defragging.  This is why sometime you'll still get "out of memory" issues, even though it seems like you have plenty.  Pagefile might help some aspects of this, but not all.  (I run into this issue with my work.)

Ha.  First time I ran into an aspect of this was back in the 32 to 64bit OS transition.  Experimenting I installed 32 bit versions of Windows XP and Vista on system with 8gigs of ram just to see what happen.  The results were very interesting.  Hehe, also found a way around the issue using msconfig.  😀 

[(PSN)crashteddy03]

On 2020-09-13 at 4:30 PM, DatDarkOne said:

edit: as for the breaking the game part, that shouldn't happen unless you have a low amount of ram.  Like 4gigs or lower.  Having 8 gigs or more and you can pretty much safely go without a pagefile at all in most situations.  

Previously I would have agreed with that statement and I did run without a page file for many years on my previous laptop (8GB RAM, discrete graphics, Windows 7). On my slightly newer laptop running Windows 10 I discovered that I couldn't get into the open worlds (PoE or OV) with the page file disabled (8GB RAM, integrated graphics <gah!>).

I had read somewhere that having a page file with an SSD creates a lot of extra wear on the drive. Has anyone else heard that before or have any insight into it?

[taiiat]

2 hours ago, OneYenShort said:

When programs request memory, it is expecting contagious blocks of memory.  The OS doesn't report free vs. largest contiguous free.  The memory control aspects of the OS does very little RAM defragging.  This is why sometime you'll still get "out of memory" issues, even though it seems like you have plenty. 

this influences things too, yes. generally the Memory that is lost to 'dead zones' is on the smaller side, but that can definitely cause problems sometimes.

in my world, which consists of a lot of 3D Applications, the Memory issue is generally far more of what i mentioned that they will request Memory "just in case" but then never use it.
for example, i have audited both Division 2 and MH World and had each consuming ~20GByte of Memory, though only actually using half of that or less.
in the case of Div2 they have some broad covering Anti-Cheat that prevents me from reading their Memory stack, which i respected because i didn't want to get banned. but also left me unable to diagnose the ridiculous Memory waste.
for MH World though, i was able to deduce that the game would save Level data for each region forever after Loading it when you go to play in that area. however... it would never unload it. and you might think  "well they're just Caching it a little overzealously to make subsequent Loads faster". i thought that too before i started investigating, but no, the game still loads from Disk if you leave and return an area you have already been!!!!!! so it doesn't even get any benefit out of it atleast, it's just literally pissing Memory into space.

 

ok that's enough ranting for now.

27 minutes ago, (PS4)crashteddy03 said:

I had read somewhere that having a page file with an SSD creates a lot of extra wear on the drive. Has anyone else heard that before or have any insight into it?

technically it's extra writes that you on paper could otherwise not have if you didn't have PageFile enabled - in practice Applications will generally not be storing much in there as long as you have a reasonable amount of Memory, so not much actually gets written there.
and furthermore the average SSD is going to have a minimum effective lifespan of around 500TBW (TeraBytes Written). that isn't 100% Cell failure either, that's reaching a point where the majority of the Cells are dead and performance has dropped significantly, or something else went wrong and the Drive went dead just of its own accord.
i bring that up because realistically you're probably not going to burn out an SSD that you're even using every day before.... you get bored with the Capacity it has versus new ones, or it stops being fast enough for your tastes, or you've just had it so long that you would end up replacing it anyways (since Consumers do tend to replace things just out of habit even if it's still fine).

in short, you don't really need to worry about it unless your System is pretty low on Memory. in modern times where people are often running like a dozen Browser Tabs and 5 Game Launchers and a bunch of VOIP Applications and blah blah all on top of playing a game, 16GByte+ should generally ensure that you don't need to store things in PageFile, but Applications may still elect to store small amounts of data there. with things trending towards 32GByte in order to hold this position.

okay that wasn't very short so in actually short, it doesn't matter what types of Data Writing happens to an SSD, it's strictly the amount of total Data that wears Cells out. and by far the biggest contributors to SSD's wearing out is installing Video Games, Updating Windows, Et Cetera. that's where large amounts of Writes are happening.