Stop new feature development and fix Host Migration issues once and for all

[Nekomian]

1 hour ago, CephalonCarnage said:

That will be an artifical situation caused by the natiure of circuit - people drop in, see the optiosn and quit. Deliberately, not network drops. It may be that they take longer to populate their state in the lobby because its possibly built with the same lazy-loading code used in open world lobby areas, not game missions. These are deliberately designed to host oats of players on a DE server, so you can form the squad. Not P2P at all.

I'm a bit confused by this, because all of the options for the Circuit rotation are displayed in your orbiter - there's no need to connect, see what is present, and disconnect from a mission. The players also have to physically go to the weapons and see them, unless they can identify them by appearance that well from far away to know exactly what they have - I don't think most casual players can do that. In my situation I think it was solely a technical issue and not a deliberate one (especially if their model had not even loaded in yet, they physically could not see on their screen what the options were).

To add to the second part - unlike normal missions where you only have one set of equipment, the game has to load in all potential choices from your disk, so it's expected to take longer. It also spot loads when your allies make new choices, because it needs to load in all of their equipment choices too (not all their potential ones, but whatever one they apply for the mission). You can see the batch and spot loads of resources from the EE log in a session where it occurs, if you're curious. It's also why the game prevents you from swapping equipment too much when someone is loading in, because it's increasing their load time as they have yet another resource to queue up.

[Tiltskillet]

3 minutes ago, Naroxas44 said:

I'm a bit confused by this, because all of the options for the Circuit rotation are displayed in your orbiter - there's no need to connect, see what is present, and disconnect from a mission.

Decrees can't be seen from the Orbiter, and it's possible some players still go to the Cave and ditch if they don't like the options.  There are also a couple of bugs with the previews that might make some people distrustful of relying on them.

But I think what is more common is that they're checking out who else is there,  how strong their selections seem to be, and ditching if they don't like how promising these seem.

Or it could be I'm projecting, because I do some of that myself.  (Although Decrees I either scout solo or don't fret over.)

[CephalonCarnage]

13 minutes ago, Naroxas44 said:

I'm a bit confused by this, because all of the options for the Circuit rotation are displayed in your orbiter - there's no need to connect, see what is present, and disconnect from a mission. The players also have to physically go to the weapons and see them, unless they can identify them by appearance that well from far away to know exactly what they have - I don't think most casual players can do that. In my situation I think it was solely a technical issue and not a deliberate one (especially if their model had not even loaded in yet, they physically could not see on their screen what the options were)

You shouldn't be surprised. So many kids with the impulse control of a gnat will load in, look and quit. The same ones who will quit if the relic isn;t right or the mission is not going quite their way. So although DE added the view, I'm sure many players don't even know its there. Old habits and all that.

 

15 minutes ago, Naroxas44 said:

To add to the second part - unlike normal missions where you only have one set of equipment, the game has to load in all potential choices from your disk, so it's expected to take longer. It also spot loads when your allies make new choices

In relays, and open world lobbies you see many players slowly loading in - all those ghost excaliburs are common. They are obviously using a server (you choose it when entering the world) and Teshin's cave won't be any different. So the characters are loaded slowly and lazily as you look at them. This is nothing new. I doubt the game loads in all potential choices and also spot loads, I think it loads in the character you have, and loads in again when you swap loadout. The problem is really underpowered servers slowly pushing all the myriad assets from all the players present to players. DE knows this, that's why popular events have a hundred servers allocated to them.

[taiiat]

4 hours ago, SDGDen said:

i don't think DE even has *that* many relay servers, they probably just have some they rent out rather than having their own hardware in a datacenter somewhere. 

yeah, it's not that many perse, it's like ballpark 1-2 Dozen for the entire Planet. they're intended more as a backup system than a primary route.
just uh, a backup route that gets utilized more often than one might hope.

[Nekomian]

3 hours ago, Tiltskillet said:

Decrees can't be seen from the Orbiter, and it's possible some players still go to the Cave and ditch if they don't like the options

I guess I could see that, but the game throws so many decrees at you that it's never been a reason for me personally. It seems a little silly to me to waste 20s loading in, immediately leaving, and then wasting 20s more to load into another Circuit run. I could understand a bit more with steel path and wanting a team with a certain composition for that, but most equipment can be viable with decree stacking even at a solo level, and there's also recruiting chats if someone wants a specific team to just power through to tier 10 for the week.

3 hours ago, CephalonCarnage said:

In relays, and open world lobbies you see many players slowly loading in - all those ghost excaliburs are common. They are obviously using a server (you choose it when entering the world) and Teshin's cave won't be any different. So the characters are loaded slowly and lazily as you look at them.

So I'm aware it loads resources based on what relay servers identify the players in the hub area have, but it has to do that because players can come and go (and aren't in a squad with you) and background asset streaming is necessary in this case. I don't think Teshin's cave does this though, as it doesn't have those same properties (player equipment changes are "instant") and the logs I've seen do not indicate it's doing this - when teammates choose different equipment it micro-stutters and spot loads, but it does not do this for any of your own equipment (which I do see it loaded in logs as well). That makes me think it batch loads all equipment for the player from disk, so they can transition between it seamlessly in the cave, while spot-loading whatever squadmates have, to avoid having to load all potential loadouts for them too because that takes a lot longer and more resources (and on systems with limited resources, like the Switch or lower end PCs, this is probably significant enough to warrant doing it this way).

I would be surprised if they were using servers to handle this load since the P2P connection (host to client in this case) can just communicate it to all squad mates if there's an equipment change, and there's no reason it needs to go to anywhere else other than the squad players. I'd honestly need an actual dev response to confirm it, but this would be my best assumption based on my own experiences and logs.

[Tiltskillet]

55 minutes ago, Naroxas44 said:

I guess I could see that, but the game throws so many decrees at you that it's never been a reason for me personally. It seems a little silly to me to waste 20s loading in, immediately leaving, and then wasting 20s more to load into another Circuit run. I could understand a bit more with steel path and wanting a team with a certain composition for that, but most equipment can be viable with decree stacking even at a solo level, and there's also recruiting chats if someone wants a specific team to just power through to tier 10 for the week.

I can't speak for anybody else's motivation.  For me it's probably not worth the time overall, but occasionally can make a difference in the build I pick.  Or even make me configure a build around a decree.   And if nothing else it's nice to have one less thing to dither over in the Cave.  I like not having the squad waiting on me, or keeping it to the minimum if I'm still the slow one.

  * * *

Unrelated to the above but I've had an odd bug in the Circuit I wanted to tell you about since it seems like something you might understand:  Occasionally I'll lag behind most or all of the squad when loading into each new area.  Like, a lot of lag, maybe 20 seconds or so, because sometimes they'll have picked up all the decrees by the time I get there.  The interesting thing is it -only- has happened when I'm hosting.  Not every time I host, maybe 20% of the time.  But when it happens on a run it will happen on every load in, getting neither better nor worse that a can tell.  I've never had this happen with Onslaught that I can recall, which seems the most similar to Circuit.

No issues with gameplay once I get there.  Ping is always in the normal range and AFAICT it's not associated with other internet issues on my end.

 

[Nekomian]

1 hour ago, Tiltskillet said:

Occasionally I'll lag behind most or all of the squad when loading into each new area.  Like, a lot of lag, maybe 20 seconds or so, because sometimes they'll have picked up all the decrees by the time I get there.  The interesting thing is it -only- has happened when I'm hosting.  Not every time I host, maybe 20% of the time.  But when it happens on a run it will happen on every load in, getting neither better nor worse that a can tell.

Ah, I know exactly what you're talking about - it's been some weird bug since Duviri release, where sometimes a few members of the squad get "stuck" in the previous round's area while others move ahead. I've not had 20s of delay, but I have had about 10s before; it's hard to search for because there's a lot of reports of progression stops on release that come up when looking for it, which aren't quite the same thing.

There isn't really anything to indicate specifically why the delay is happening from logs (it doesn't appear to be network or disk load related), but my best guess would be that the routine doing checks for moving to the next area gets delayed somehow for players under specific circumstances (maybe if they go into the portal vs teleport in from Teshin's Cave), or that the clients aren't syncing with the host for the move properly, so some go ahead while others don't. I've noticed it both as host and client, so I'm just as puzzled by any sort of trigger or condition for it too and it's a bit difficult to test.

It seems likely that it's something that will have to have a coding fix to resolve, maybe additional checks or timings for the synchronization to happen consistently, since it's so varied and doesn't seem to be a network or performance issue or anything from the player's end that could resolve it.

[CephalonCarnage]

12 hours ago, Naroxas44 said:

I don't think Teshin's cave does this though, as it doesn't have those same properties

doesn't need the same properties, just needs to reuse the same code as used in open world lobbies. That's how developers roll. Its not the same as normal missions that do not have a waiting area, so its certain the same code is being used, just with a 4-player limit per instance. Same with Zariman that is not an open world but shares the same lobby as one.

[Nekomian]

2 hours ago, CephalonCarnage said:

so its certain the same code is being used

While I understand reusing code in places for the sake of time efficiency reasons, I don't think you can be absolutely certain that it's the exact same or the implementation of it is the same either. I don't think the servers are creating session instances with 4-player limits for the cave either like how Relays function, as again there's no reason to use this server space when there's a mission host right there - the cost of maintaining that many "instances" to track all players is also expensive at scale, so I doubt they'd want to add more to the already existing number.

Zariman hub is treated like a relay and reflects that, because it's not in mission with a squad. Any time a mission is started, I assume they offload the computation for the session to that host. The logs I have indicate it's doing this too, with messages like "Local player X sending request for Y‚ loadout", net info of pings / ACKed packets / SEQ data (to and from host) appearing before (and delaying) spot loading of loadouts, "Game [Info]: UpgradeManager::Clear()" and "Sys [Info]: Creating loader for /Temp/OtherDuviriPlayer1_Y loadout..." immediately after one another when a player changes equipment, among other indications.

Meanwhile, relays are only directly communicating with their own IPs (the one I just tested was from DigitalOcean, which is a cloud storage provider), and it directly states "Net [Info]: Sending loadout: 2788 bytes" indicating it's directly sending to the server, as well as other info that teshin's cave does not have (i.e. the resloader directly building loadouts with no indication of spot loads for it, like "Sys [Info]: Resloader 0x000002 (/Temp/HubPlayers/Z) starting" & "Sys [Info]: Resource load completed 0x000002 (/Temp/HubPlayers/Z) in one pass and 0.5s (I/O ~= 6.7%, inherited 6,818 of 9,078)).

That is to say, even if parts of the resource loading are similar, the way the data is loaded between these two things and the logging behind it leads me to believe they're not the same, with the hubs preferring the "lazy load" method of just asynchronously handling the loadout since it's not a mission and is not required instantly while places where it's instantly needed, like Teshin's Cave, opt for spot loads on equipment changes with triggers to indicate when to update the loadout for the player. Unless you have some additional info I'm unaware of, we're probably not going to reach any definitive conclusion without dev insight, since this is all just conjecture anyways.

[CephalonCarnage]

2 hours ago, Naroxas44 said:

I don't think you can be absolutely certain that it's the exact same or the implementation of it is the same either. I don't think the servers are creating session instances with 4-player limits for the cave either like how Relays function, as again there's no reason to use this server space when there's a mission host right there - the cost of maintaining that many "instances" to track all players is also expensive at scale, so I doubt they'd want to add more to the already existing number.

The reason is that no 4-man squad mission uses a lobby. only the oipen worlds. And Zariman is not an open world but also uses the lobby system. So reusing the code for open worlds for circuit fits perfectly, you cannot use the normal "join mission" code because there is no lobby mechanism there. Doesn't matter if you only have 4 man squads and 1 is chosen as host, there is no other mission type where that occurs. Those cases, you join the squad in progress. The Teshin cave starts and you're not in a squad, you just happen to be in an open relay with just 3 other players who end up forming the squad because there's nobody else allowed in there. But they could if they wanted to - I'm surprised the cave doesn't have tens of players browsing the options and whoever goes through the gate forms a squad, just like Fortuna or Cetus or Deimos. They even have the same tunnel that allows the host to be chosen or set up, there's no other reason to have that middle bit between cave and circuit map if the host was already chosen like joining a exterminate mission on Mars.

Don't forget Teshin's cave doesn't allow player loadouts, so not surprising the code is tweaked by more than just removing the arsenal UI.

[Nekomian]

1 hour ago, CephalonCarnage said:

The Teshin cave starts and you're not in a squad, you just happen to be in an open relay with just 3 other players who end up forming the squad because there's nobody else allowed in there.

I still don't know where you're getting "this place is a closed relay basically" when the devs themselves have stated it's classified as "mission" (https://www.twitch.tv/videos/1805952072?t=0h44m36s & https://www.twitch.tv/videos/1805952072?t=0h47m12s ) and there's no indication in logs of it creating any sort of connection like other hub areas do (i.e. Cetus / Fortuna / Necralisk / Zariman). It looks like they just have a routine running during the cave portion of the mission that listens for events and resets the player's loadout on them to avoid desync, spot loading in the resources as necessary when they change. I've provided plenty of evidence of this too.

Given you can change your arsenal in hubs but cannot use weapons, yet have the exact opposite in actual missions, and the fact that your "host" connection never changes (it does not host migrate or re-establish a host during the loading tunnel to Duviri / Circuit from cave) in the cave, I really don't think they're implementing it how you're describing. Just saying "there can be no other way because there's no other mission type where that occurs" is confusing when they have an in-house engine and teams that can work with it and create new frameworks to adapt to new demands without repurposing something that was designed not to work at scale for missions themselves (i.e. opening thousands of instances just to relay traffic, costing them a lot more money when they have a P2P host right there for the player as a centralized communication for the squad).

[Tiltskillet]

On 2023-10-08 at 4:18 PM, Tiltskillet said:

On 2023-10-08 at 4:04 PM, Naroxas44 said:

I'm a bit confused by this, because all of the options for the Circuit rotation are displayed in your orbiter - there's no need to connect, see what is present, and disconnect from a mission.

Decrees can't be seen from the Orbiter, and it's possible some players still go to the Cave and ditch if they don't like the options.  There are also a couple of bugs with the previews that might make some people distrustful of relying on them.

But I think what is more common is that they're checking out who else is there,  how strong their selections seem to be, and ditching if they don't like how promising these seem.

Or it could be I'm projecting, because I do some of that myself.  (Although Decrees I either scout solo or don't fret over.)

I forgot that there's one more reason I'll do it: if I'm not hosting I might ditch, particularly if I have really fun rolls and have a good amount of time to play.  Obviously that doesn't put people through host migration.  But conceivably there might be people who don't want to host, who ditch if they're selected.  I could see doing this myself if I might get interrupted and need to bail.  Better to have someone else hosting in that case.

[(XBOX)Hellsteeth30]

Just fix the "reset" nature of migrations, some frames have a bit of setup to get going and having that setup nuked by hosts clearing off can get annoying.

[CephalonCarnage]

On 2023-10-10 at 1:03 AM, Naroxas44 said:

I still don't know where you're getting "this place is a closed relay basically" when the devs themselves have stated it's classified as "mission" (https://www.twitch.tv/videos/1805952072?t=0h44m36s & https://www.twitch.tv/videos/1805952072?t=0h47m12s ) and there's no indication in logs of it creating any sort of connection like other hub areas do (i.e. Cetus / Fortuna / Necralisk / Zariman). It looks like they just have a routine running during the cave portion of the mission that listens for events and resets the player's loadout on them to avoid desync, spot loading in the resources as necessary when they change. I've provided plenty of evidence of this too.

If it looks like a duck, walks like a duck and quacks like a duck...

you've provided no evidence at all. All you say is that its somehow different to a closed relay, despite everything about it being exactly like relays, and then saying its a mission despite no mission having anything like the loading lobby. So maybe they have placed the relay code using the first player as a server instance or creating a new one on demand without the usual 'choose server' startup. That's not so strange, but that's massively easier than changing the whole mission startup to be like open relays.

[(XBOX)OneOmniverse]

On 2023-10-08 at 10:51 AM, Silligoose said:

It would be great to see a feature that saves progress as a mission progresses (in terms of pick-ups and other progression elements), but a recent experience indicates to me DE doesn't really care much about players losing progress during missions: I was playing SP circuit for an hour or so, encountered a bug that prohibited further progress due to objectives not initializing and knew due to this, I'd have to exit the mission and lose rewards and progression of many rounds. While still in mission, I contacted Support with evidence of my progress and despite that, they couldn't help me in retaining rewards or progression. I'm not mad at Support - they did what they could and that was appreciated, but they are following the guidelines as laid out by DE and apparently those guidelines include not helping players despite proof. That's just the way it is.

Aside from my anecdotal proof, looking at the state of content releases with the myriad of game-breaking bugs, the state of asset releases that are incomplete, them avoiding to even acknowledge the lack of care regarding these releases, even the way they "celebrate" 10 years with their community by way of FOMO skin packages inflated in price with premium currency, it seems evident the player experience is not very high on the priority list at DE.

It is a shame to hear you've had to play solo due to losing progress or rewards.

Since being purchased by Tencent back I'm 2020 it's not a surprise they have gotten greedy with prices and FOMO, companies when bought by someone huge often like to say, they will stay independent or no changes will come of it,

A lot of the times it's true, for a while, eventually it doesn't end well and I suspect war frame will be just another one of them eventually sad but that's what happens when you let yourself be bought by huge conglomerate, let alone a foreign one, who is known to gobble up western gaming companies. It was good while it lasted though. 

[Nekomian]

15 hours ago, CephalonCarnage said:

you've provided no evidence at all

This is gonna be my last post regarding this because it's 1. derailing the topic about host migrations and 2. it feels like you're convinced it works this way despite logs stating otherwise, and outside of quite literally running Wireshark logging or something I don't know how to demonstrate to you that it's not behaving in the way you're describing.

15 hours ago, CephalonCarnage said:

saying its a mission despite no mission having anything like the loading lobby

It's not a loading lobby. It reloads the user's loadout when they swap equipment, that's it. Nothing else is similar to a lobby / hub area - you can't trade, you have full parkour and use of abilities and weapons (like in missions), etc.

Which would be easier and more efficient here: copy pasting huge chunks of code that interface with relay networking (and rewriting all of it to fit with missions), generating hub areas for every single duviri mission (1000s of them daily) on servers and tracking all changes that occur (incurring costs from those server cycles), then migrating the host once the "mission" starts OR simply write a script that listens for an event, and send that event to the P2P host to communicate with clients in mission to reload a player's loadout when called (i.e. equipment is picked up or swapped). I don't develop for this game, but I know which one makes more sense in regards to scalability given the existing architecture / platform.