Jump to content

Update 8 Login Fiasco


[DE]Glen
 Share

Recommended Posts

A lot of people had difficulty logging in after Update 8 was released and I wanted to explain what changed, why it was changed, what went wrong, and what is going to happen. In short, we’ve fixed a lot of problems, one fix has been delayed and should be released today, and we are aware that certain anti-virus programs cause problems that we are continuing to investigate.

What Changed

Warframe has always used encrypted channels to communicate with our servers but despite this hackers were exploiting a security vulnerability to forge rewards and leaderboard statistics. When we received reports that one hacker was selling his cheating services for real money we decided that just banning hacked accounts was not enough anymore and we had to act. Update 8 included increased security for all connections to the game servers which should not only make it harder for hackers to cheat but also makes it a lot harder for someone to eavesdrop on your connection to the game. Unfortunately it turned out that a lot of regular users were caught in the cross-fire between us and the hackers.

What Went Wrong

After the update went out we started getting scattered reports of login problems but we could not tell what the problem was; at midnight our time (only a few hours after the update) I added diagnostics to the process that were necessary to understand what was going wrong. Unfortunately at that hour we did not have the QA resources available to test the build so we deferred the update until the next day (Friday). Our intention was release a hotfix update early the next morning but as the day went on more and more fixes got piled into the patch and it wasn’t until sometime after dinner that we finally released a more thoroughly tested update with the required diagnostics.

As soon as we started getting logs from people who were still having problems after the update I began isolating permutations of a common problem. A few hours after the first hotfix I pushed out another one that got most stranded users back into the game. For the rest of the evening I sorted through hundreds of support tickets, checking logs and looking for other problems.

On Saturday morning I woke up early and started wading through the tickets again – there were a number of strange cases that we didn’t understand and it wasn’t until Saturday evening that we had enough diagnostics to put together a clear picture of the remaining problems.

Saturday night I added support for another permutation of the same cases I’d fixed the night before and tried to push out a hotfix; sadly we had no QA working on that particular Saturday and a pretty nasty bug had made its way into the hotfix undetected – within minutes of it being deployed we yanked it back because it was causing widespread crashes.

This was a real nightmare for us – we had a quick fix we needed to get out but we were stuck because we needed to resolve a separate issue first. I spent the rest of the weekend isolating and fixing the secondary issue and had it ready for QA just before midnight on Sunday. Not wanting to repeat the mistake of Saturday-night I wanted to get some QA testing done before this update went out.

On Monday things got worse because almost all of our QA staff was on holiday; we had made a deal with them to try to get as much testing done for Update 8 – they gave up their long-weekend holiday before Update 8 in return for getting it the week after instead. Once again this left us in a bind – I had fixed a pretty major bug (and had to touch a lot of code to do so) and we had only a skeleton crew to test it.

What Is Going to Happen

Right now we’re trying to get enough testing done for this update so that we can get it out as soon as possible. In lieu of most QA being on leave, staff members from other departments have volunteered to help test in order to try to make sure we don’t have a repeat of Saturday night.

There are other known login failure issues that we will still investigating and most caused by anti-virus software intercepting with the connection (Warframe cannot currently distinguish friendly anti-virus interception and malicious eavesdropping). While we investigate there have been workarounds posted for ESET and NOD32 anti-virus software packages and we will try to collect workarounds for other packages as we get details in to support.

In conclusion I just want to say that I’m very sorry that I broke the game for you and your friends; rest assured that we are working as fast as possible to get a fix out there.

Link to comment
Share on other sites

It is understandable that you all have been busy more than enough, not to mention the full focus on Update 8, bug fixes and overload of tickets. Here was I complaining by myself about my 3000 platinum not being returned after the Optional reset for over a week. My ticket is still unanswered but now I finally realise there are more important things that needs to be put on the front line. In my opinion, you should not be sorry at all for most of us are aware that you are doing what you can to fix it for us. Stay strong, DE and keep those hackers off your feet!

Link to comment
Share on other sites

Not a good weekend for you guys was it, hm ? =/

We are thankful for your work here in an case, especially since you worked through the weekend to get these things fixed for us.

 

Thank you, Glen, and for the Volunteers that help you now getting the next fix pushed out as soon and safe as possible.

Link to comment
Share on other sites

DE_Glen, thank you very much for this information, I thought I was a isolated case of some random ban for some other reason while I was desesperately waiting for someone to awser my support ticket, but now that DEMark looked at my ticked and you posted this info, I can rest in peace knowing that you were really working hard for everyone and for the people like me who got caught in the crossfire betwen the you and the hackers.

 

Digital Extremes, you are awersome.

Link to comment
Share on other sites

Have you guys considered making a public test client? I know this is kind of already in test phase, (testception?) but I mean, I'm sure many of us would be more than happy to do the pilot testing as well. We are one big pile of resource generator just waiting to be fed, after all :)

Anyway, keep up the good work. You deserve some coffee! And a cold beer.

Link to comment
Share on other sites

In conclusion I just want to say that I’m very sorry that I broke the game for you and your friends; rest assured that we are working as fast as possible to get a fix out there.

That's unimportant. I'm just very glad that you broke the game for cheaters and hackers =)

Link to comment
Share on other sites

I didn't even know there were hackers in this "Co-op" game. Why in the world do you need to hack a game like this anyway? People need to get some skills and stop this nonsense and work for your rank and gear like everybody else. I hope you catch these hackers and drop the ball on them hard. IP termination and all, not just accounts. In fact, just send the FBI on them, seriously, you can do that. The community will also gladly help you guys in any way we can. Also, a hacker selling services may possibly be from within the DE group. I'd start there and work out. Regarldless, this hacking needs to stop.

Link to comment
Share on other sites

Thanks for the info! The hearts of the honest players are with you, DE! You have made a great, great game so far, and you're being great with the community, you really listen to us, and support us, and we should support you back. Don't blame yourself, Glen, mistakes happens and we know you were trying to fix everything for us!

Link to comment
Share on other sites

Thanks for the update on the update Glen, :)

 

Make sure you get sleep. It is easy to get lost in a big pile of work and forget things like eating, sleeping, using the bathroom... Been there, done that. Made a BIG mess...

 

Keep up the good work, but take care of yourself too.

Link to comment
Share on other sites

Shame, and I just realized off-topic, there was an oppurtunity to get a Braton Vandal. I missed my chance. I still want the gun, since I've beta tested, but I doubt the admins would be kind enough to even accept platnium to put the thing on my account. :| I sent a ticket, but never got a reply back.

Link to comment
Share on other sites

Shame, and I just realized off-topic, there was an oppurtunity to get a Braton Vandal. I missed my chance. I still want the gun, since I've beta tested, but I doubt the admins would be kind enough to even accept platnium to put the thing on my account. :| I sent a ticket, but never got a reply back.

Dude The Braton Vandal only for Grand Master Founders.you can not buy with Platin etc.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...