[DE]Glen Posted May 27, 2013 Share Posted May 27, 2013 A lot of people had difficulty logging in after Update 8 was released and I wanted to explain what changed, why it was changed, what went wrong, and what is going to happen. In short, we’ve fixed a lot of problems, one fix has been delayed and should be released today, and we are aware that certain anti-virus programs cause problems that we are continuing to investigate. What Changed Warframe has always used encrypted channels to communicate with our servers but despite this hackers were exploiting a security vulnerability to forge rewards and leaderboard statistics. When we received reports that one hacker was selling his cheating services for real money we decided that just banning hacked accounts was not enough anymore and we had to act. Update 8 included increased security for all connections to the game servers which should not only make it harder for hackers to cheat but also makes it a lot harder for someone to eavesdrop on your connection to the game. Unfortunately it turned out that a lot of regular users were caught in the cross-fire between us and the hackers. What Went Wrong After the update went out we started getting scattered reports of login problems but we could not tell what the problem was; at midnight our time (only a few hours after the update) I added diagnostics to the process that were necessary to understand what was going wrong. Unfortunately at that hour we did not have the QA resources available to test the build so we deferred the update until the next day (Friday). Our intention was release a hotfix update early the next morning but as the day went on more and more fixes got piled into the patch and it wasn’t until sometime after dinner that we finally released a more thoroughly tested update with the required diagnostics. As soon as we started getting logs from people who were still having problems after the update I began isolating permutations of a common problem. A few hours after the first hotfix I pushed out another one that got most stranded users back into the game. For the rest of the evening I sorted through hundreds of support tickets, checking logs and looking for other problems. On Saturday morning I woke up early and started wading through the tickets again – there were a number of strange cases that we didn’t understand and it wasn’t until Saturday evening that we had enough diagnostics to put together a clear picture of the remaining problems. Saturday night I added support for another permutation of the same cases I’d fixed the night before and tried to push out a hotfix; sadly we had no QA working on that particular Saturday and a pretty nasty bug had made its way into the hotfix undetected – within minutes of it being deployed we yanked it back because it was causing widespread crashes. This was a real nightmare for us – we had a quick fix we needed to get out but we were stuck because we needed to resolve a separate issue first. I spent the rest of the weekend isolating and fixing the secondary issue and had it ready for QA just before midnight on Sunday. Not wanting to repeat the mistake of Saturday-night I wanted to get some QA testing done before this update went out. On Monday things got worse because almost all of our QA staff was on holiday; we had made a deal with them to try to get as much testing done for Update 8 – they gave up their long-weekend holiday before Update 8 in return for getting it the week after instead. Once again this left us in a bind – I had fixed a pretty major bug (and had to touch a lot of code to do so) and we had only a skeleton crew to test it. What Is Going to Happen Right now we’re trying to get enough testing done for this update so that we can get it out as soon as possible. In lieu of most QA being on leave, staff members from other departments have volunteered to help test in order to try to make sure we don’t have a repeat of Saturday night. There are other known login failure issues that we will still investigating and most caused by anti-virus software intercepting with the connection (Warframe cannot currently distinguish friendly anti-virus interception and malicious eavesdropping). While we investigate there have been workarounds posted for ESET and NOD32 anti-virus software packages and we will try to collect workarounds for other packages as we get details in to support. In conclusion I just want to say that I’m very sorry that I broke the game for you and your friends; rest assured that we are working as fast as possible to get a fix out there. Link to comment Share on other sites More sharing options...
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!Register a new account
Already have an account? Sign in here.Sign In Now