State of the TennoNet 2017

[AdamantSpartan]

And here I thought the way to improve a connection was to download more RAM for the servers.

 

But in all seriousness, keep up the good work.

[Fylas]

so long story short,  95 out of 100 network issues players had where providers being asshats that completely ignored all connection protocol rules, changed your ports, changed your address or flat out terminated your idling connection without telling anyone making the connection a corpse. Which means you practically have to integrate countermeasures in client and server to recognize that kind of behavior by the ISP and compensate by sending notes to the server that the connection needs to be renewed since the client is no longer at the spot on the internet where it used to be. -_- not your fault but your problem indeed O_o the crap you guys have to put up with T_T 

[Cortanis]

On 1/30/2017 at 11:31 AM, [DE]Glen said:

As we dug into the numbers we found even more horrifying statistics: when your network wants to ruin your day it doesn't just do it once, it does it all the time: if you're one of these unlucky Tenno your network port will change on average 140 times per day and your address on average 5 times!

This part just made me cringe. I'd hate to be one of those poor tenno. At that point, I start looking for a new isp.

[UnknownCoffeeRebel]

Thanks a whole lot for your efforts....after reading it all,It's all clear to me,and I'm amazed even more through the hell you went through,only so that we can enjoy this amazing game...respect is weak word compared to what I feel towards you and the whole DE team...keep the great job,we love y'all :)

 

Kisses,hugs and coffee mugs!

Also  "It's not my fault but it's my problem,"  truly is showing how much you lads care...and for this,I can't help but admire you!

[(XBOX)JestersShade]

Hi there.

Very nice and instructive post.

Since you're talking about network issues, a friend of mine has huge connection problems since last Warframe update (we're on Xbox One)

The issue is not on the first screen (the hotfix check), but on the second one, where you log in.

Some days, he tries during 15 to 20 minutes to get connected! The connection seems to time out each time (and it takes a really long time to time out each time...)

I don't know if it's related but since he doesn't have a good internet connection, it may be.

And since we're on console, we can't give you logs either... :/ (and we get through Xbox Live)

So if it's not on the radar, please add it and thanks very much for your network tracking ;)

[(XBOX)F L Lion2112]

Juicy IT Nerd data! Love it.

[taiiat]

On 1/31/2017 at 10:35 AM, Krhymez said:

From trying to "fix" issues i was having... i found that these settings were the issue. Even a few people in game that i have spoken to, have or had the same issues. In your statistics Nat-pmp was used very little. If that is the case, why is it enabled by default? Does it cause any issues to have it enabled? 

these methods are handy tools for routing Networking data, and generally are preferred as they do a better job than more 'classic' methods - but Consumer Networking Hardware is made with little in the ways of standards and if you pick up two Routers / Modems, chances are they handle the same type of instructions and data differently than each other.

even though there are standards to conform to, the manufacturer doesn't. (the same here applies to ISP's)
so Warframe gives you the option to prevent Warframe from attempting to use these tools, if you need it.

 

similar to how there's compatibility issues with some things over USB - the design specifications are completely compatible, but the manufacturers don't all abide to these specifications.

Edited February 1, 2017 by taiiat

[-CM-Darkvamper]

Interesting read, although the part about simply changing adresses every 5 minutes kind of blew my mind.

Keep up the great work and thanks for keeping the game online during the holidays, it was much appreciated!

[Maziani]

I appreciate full disclosure and read the entire first post.

[Kidkilla]

Thanks for the info. I do have a question - these network attacks, how frequent and how bad? 

[Koldraxon-732]

2 hours ago, Kidkilla said:

Thanks for the info. I do have a question - these network attacks, how frequent and how bad? 

I have reason to believe there's a few reasons why said attacks are still happening:

Spoiler

 

1: Hema cost hasn't changed or isn't changing; that covers the Mutagen Mass/Sample price and acquisition locations and methods.

2: Baro bringing nothing new; weapons or otherwise.

3: No Umbra yet. U20 should have Umbra according to a Devstream.

4: 'Personal reasons'.

5: 'Other reasons' (supporting opposing games, like Destiny, in a malicious manner).

6: Attack on DE's public image via network attacks.

7:  s.

 

 

Edited February 2, 2017 by Koldraxon-732

[Dukinie]

Wow - thanks for the info! It's always interesting to see stats like this, as not many companies like to share, but even just the fun little thing about the spike in traffic when the Baro is around is really interesting! (or I think so anyway!)

On the whole NAT level, we used to be under a strict NAT when we changed from and old copper ISP to one of the "newer" fiber providers as they didnt supply us with a public IP, but instead provided us with an IP from their NAT, calling them up and asking them to swap us over solved all problems on that front instantly (We were still able to play while on the strict NAT, but had some problems grouping if I recall correctly). 

 

Again, thanks for the info! I'd love to see more random stats as I find them really interesting.

Even just random snips and weird stuff, like the 20x extra traffic, lowest points of the day, what worldpart has the highest % of players, best connections etc. that'd be fun to read!

[Stratego89]

I'm impressed you guys actually managed to figure out what was causing this problem for people, I really am. I can't even begin to imagine just how much you all had to push yourselves when the only information you could get to solve it was from players...

 

I wasn't one of those affected, but I want to give you a thank you anyways for being so dedicated.

Edited February 3, 2017 by Stratego89

[notlamprey]

What's really boiling my noodles right now is that I never used to have issues with connection stability in the past, and now it seems that I do.

It's becoming normal for me to see the dreaded "Network Not Responding" multiple times in 24 hours. On occasion I'm either unable to launch the game (due to ERROR_INTERNET_CANNOT_CONNECT or the launcher getting stuck "Checking for updates..."), or I find myself disconnected to Login at least once per session.

I have a housemate who is currently a bit more active than I am, and she has reported occasional connection instability as well. As far as I can tell, this is only an issue during evening hours (GMT -6).

I'm not really equipped to make any good guesses as to why this might be happening, as nothing has changed for me on the ISP side of things - to my knowledge. I just thought it was an odd new problem to have, and I'd be interested in anyone's guess as to why this might be the case.

[taiiat]

On 2/2/2017 at 9:56 PM, notlamprey said:

as nothing has changed for me on the ISP side of things - to my knowledge.

i'm not pointing fingers at anyone but i wouldn't be that sure of nothing changing, an ISP not telling Customers what's going on isn't much of a surprise.

as just an example, a few years ago my ISP decided to move their entire service network over to a completely new system, replacing everything on their end with more up to date everything.
which is good, except they didn't tell anyone. so on some day pretty much everything just stopped working, unable to reliably resolve connections, DNS fetch being incredibly slow, the works. to support what was necessary to be supported on the new service network however, it ended up costing ~$750 worth of replacing my entire network. 
the service is objectively better since then, but still a large disruption to a service that is critically important for me and everyone else located here (and for quite a lot of people/things nowadays). and not functioning without warning for a few days (can't really get like 30 minute shipping so) isn't exactly a smooth transition.

again, just an example.

[Maziani]

My current employer replaced only the core router in the network Feb 2015. Subsequently, the increase rate of packet transmission resulted in the firewall between subnets being drowned (apparently the new router had some additional packets that it sent as part of automatic network configuration behavior, which contributed to an increase in total bandwidth consumed).

The firewall dropped packets on ingress-- the firewall CPU simply could not keep up with the new transmission rate.

The entire network went up/down for 48 hours until networking finally admitted to seeing the CPU pegging at 100% periodically. They then bypassed the Firewall and the issues went away.

 

Effectively, the router upgrade resulted in the equivalent of a denial of service attack against the firewall-- and the firewall couldn't handle the new load and denied way too much (way more than it should have.)

[Loswaith]

Thanks for the info, and especially the huge effort the team has put in to get to the bottom of the issue. 

[(PSN)O1goodeninja]

Was getting errors all the time now just 1 in the last week ty d.e.

[Shabranighdo]

That's why I love working as a software developer.

If you think you can configure everything on your server to solve any kind of network problem be sure there's already an underpayed and overworked network administration out there with a complicated but cheap as hell network infrastructure he knows nothing about to crush your hopes.

 

Keep up the good work :)

[helioth137]

even though you didn't cook the pie (charts) they were delicious...

ummm, yeah, thanks for the hard work, not gonna lie i didn't read the whole post but I appreciate your hard work none the less :)

[arch111]

Just read this and I really feel for you guys who are basically acting net-detectives. It can't be easy work.

I am allso one of those whose provider keep changing my IP. I can't manually tell it to keep a certain adress since the router handle that for all my devices. I have bugged them about it but to no gain.

Keep up the good work!

[(XBOX)JestersShade]

On 01/02/2017 at 11:39 AM, (Xbox One)JestersShade said:

Hi there.

Very nice and instructive post.

Since you're talking about network issues, a friend of mine has huge connection problems since last Warframe update (we're on Xbox One)

The issue is not on the first screen (the hotfix check), but on the second one, where you log in.

Some days, he tries during 15 to 20 minutes to get connected! The connection seems to time out each time (and it takes a really long time to time out each time...)

I don't know if it's related but since he doesn't have a good internet connection, it may be.

And since we're on console, we can't give you logs either... :/ (and we get through Xbox Live)

So if it's not on the radar, please add it and thanks very much for your network tracking ;)

 

Hi again.

Just to let you know that even with the latest build (19.8.0 on Xbox One) my friend still has some connection issues so it's still there for low connection.

I don't know what details / info / help we can provide but needing 10 to 20 minutes to get connected is obviously way too long, as you sure understand.

Thanks for your work.

[TenGaugeBoson]

MMMM charts and Graphs . Oh so lovely charts and graphs...

[(XBOX)JestersShade]

Hi there.

I wanted to bump this thread, because the TennoNet seems to be getting worse with each update, on Xbox One at least.

1 ) I already mentionned my friend issue but I'll say it again: since the Glast Gambit (19th january on xbox One), if you have a bad connection (3 to 4 Mb bandwidth) you may not succeed in logging to the game. My friend needs around 15 to 20 minutes every day, with several tryies ending in "check your info" message.

I have the same bandwidth but my connection may be more stable, and I have only managed to get the same result as he is, if I have a download in the background.

(we didn't have any issue prior to the Glast Gambit, so our connections quality certainly don't help, but are not the core of the issue).

 

2 ) Since last update (The Pacifism Defect (19.13.1) released the 20th of march on Xbox One), we get the "Network Not Responding" message really too often.

The problem is that if it happens while getting out of a mission, we end up being in our ship in first person view, with nothing being loaded.

The only workaround I know is to us the menu to go in our clan dojo, then leaving it, and so far, it did the trick

 

So, @[DE]Glen if you could investigate the difference between how the old network used to work compared to the new, it would be really nice because the first problem is really going to drive my friend (and maybe others ?) away from Warframe (15-20 min to connect to do a 30 minute sortie every day ...).

 

Thank you.

Edited April 3, 2017 by (XB1)JestersShade
typo

[Death_Master_]

@[DE]Glen Hello! as you are looking at network architecture i have some things to tell.

1) False strict NAT warning when having 4 ISP links(all TCP/UDP connects are working in round-robin).

2) No easy failover for such case(which can result in half of squad migrating to separate session).

3) No way to suppress strict NAT warnings.

What i can offer for this:

1) Check all external IPs and if they all work - no warning needed.

2) Implement 1-to-many and many-to-1 architecture for sessions(code may be tricky there but I'm sure it is possible).

3) Add to settings button to suppress such warnings.