Last Wednesday, we released our brand new chat system. With its 1-on-1 and custom group chats abilities, it has the potential to significantly improve in-game communication. But it is new, with a new interface and new way of doing things, so it will take time getting used to. But once you experience the power of a custom battle group, or cross-clan admin chat, I hope you will love it!
The upgraded chat uses new technology, so it was being tested extensively for many months - on desktop since November via the Throne Room, then on mobile devices since mid-January with our mobile Throne Room release. It was not possible to release the new and old chat side by side, so we gathered as much data as we could, made many improvements and changes, all in order to ensure that the full release went smoothly. Alas, it was not to be so!
The day after the release, we've noticed a 100% or more increase in average run times across all our functions. Here is a screen shot of our health status tool:
That set of alarm bells and our 72 hour marathon started. The problems were intermittent - no issues for 55 minutes out of an hour, then 5 minutes of serious slow down (these are just for illustration, not actual numbers). Worse kind of a problem! You work really hard and fast for 5 minutes, then analyze your often incomplete data for next 55, hoping you will be ready to catch more data next slow down.
Over time, the slow downs occurred more frequently and lasted longer. The next day revealed even worse performance degradation:
By that point we were on track, though. We identified bugs in the chat login code. Small bugs, so insignificant they would not even be noticeable to you guys, but there was a tiny horde of them! We fixed them off by one by on Thursday and Friday. Performance improved and we were almost 100% back to normal by Friday afternoon, but we still did not know what was the real cause for the slow down.
That mystery was finally unveiled on Saturday morning as we investigated why some maintenance jobs were running for hours, rather than minutes. Oh, the horror that was revealed! In our over-zealous effort to ensure the chat code release was smooth, we had created a number of performance measurement hooks. But we made one little tiny mistake. Errors had a lot of info, and I mean a lot, 100MB worth of error info per error! That was the real cause of intermittent problems. Our error management and categorization system was not designed to deal with so many monsters and started to chug! Yup, we lost at tower defense...
You see, we were prepared for the worst! If an error occurred, we knew exactly when and how... and.... that killed us.... (Editor's Note: The lesson here is to simply live in ignorance, Greg.)
Ashamed as I write this, my only consolation is that I can report that we are back to normal now:
If anyone has issues with chat not loading, or anything else, please do message support and we will help you. The in-game contact form is ideal, but if you cannot get to it, email Support@RealmOfEmpires.com
9 comments:
so, your telling me the reason for the lag in attacks being able to be sent in a timely fashion was error codes? not buying that. stupid windmill spinning around I need to send troops not wait for tab to get done sending the last one.
Every new log in to a different realm..my chat might show..1 out of 10 times..doesn't seem to work for all.
The game is still taking ridiculously long to load every little thing... despite you reporting that everything is back to normal. I hope you guys get this sorted out soon.
If anyone is still having performance issues since we applied the fix this weekend, PLEASE contact support@realmofempires.com
Specific individuals can still be having problems even if the rest of the realm is all-green and enjoying smooth play, but we won't know (and can't help you) unless you write in to our support team.
Thank you for all your hard work! Came from ag, but having a wonderful time spending too much of my day and effort to make sure I don't get killed in some virtual world that doesn't have any consequences irl... I LOVE THIS GAME 😱
Careful Devs when you mess with something that isn't broke you are in danger of screwing up a once great game.Oh it's obvious this new chat is to make people use in-game chat rather than apps like Line.One thing you are forgetting can't take screenshots and post on new chat.You can on line that's why it's necessary new game chat sucks really bad.I spent a lot of money on this game but with these changes I will not be spending any longer.
the only problem that we r having (realm 74) is that we have sent 100's of emails asking to end realm, but never get a response. Can someone plz end the realm, hand out rewards so we can move on!!
So end realm 74 already!
Great, Realm of Empires is down on Kongregate for several hours and there is no way to tell you guys, because bug report, chat, private message etc. all don't work...
Please fix this asap, because now players from kongregate can't even login, while other players can still attack them and everything!
Post a Comment