Emergency server maintenance on July20th, UTC 02:50 ~ UTC 03:45 [Cause] - A DDoS attack that started on July 20th, UTC 01:12 affected one of our servers that lead to players being unable to load the client starting UTC 01:55 - This server was migrated to a new cloud datacenter during the maintenance on July 16th for better performance and to account for increased load during WSOP, but the tech team made a mistake of not shielding this server with our DDoS protection service after the migration. [Actions Taken] - This configuration server was added to the DDoS protection service (CloudFlare) - All other servers were checked and added to the DDoS protection service - A second layer of DDoS protection will be added for increased protection
Posted Jul 20, 2020 - 03:44 UTC
Monitoring
DDoS Protection has been set up.
Posted Jul 20, 2020 - 03:40 UTC
Update
DDoS attack has been detected and identified. We're preparing the protection from DDoS attack.
Posted Jul 20, 2020 - 02:41 UTC
Identified
Official Statement Regarding Incidents on July 19th 1) Server Delays (Client loading, Login, Cash Game hand dealing, Spin & Gold table loading, Tournament registration and tournament table loading) - UTC 13:53 July 19th ~ UTC 17:11 July 19th [Cause] - Increased load on account service due to an influx of players and observers resulting in service bottlenecks [Actions Taken] - Issued a temporary fix during the server-downtime of July 20th to alleviate the issue. - Will move services that were causing the bottlenecks to other servers thereby distributing the load (by emergency maintenance on July 24th 2) Two incidents of widespread client disconnections - UTC 17:48 July 19th and UTC 18:17 July19th [Cause] - While performing tournament table configuration changes (increase of late registration time to account for delays in 1), there was a case of server-client protocol mismatch between the tournament service and game client that lead to certain clients being disconnected [Actions Taken] - Fixed all protocol mismatches on July 20th and issued client hotfix 3) WSOP Events #32, #33 not resuming after the break [Cause] - Unforseen bug (multithread timing issue) caused by high load during peak times [Actions Taken] - Issued a temporary fix during the server-downtime of July 20th to alleviate the issue. - Will issue more comprehensive fixes by the emergency maintenance on July 24th to fix the root cause. - At the worst case, even if these incidents were to happen again, we will add code to manually re-start the tournaments affected by July 24th
Posted Jul 19, 2020 - 13:45 UTC
This incident affected: Account Service, Lobby, Cash Games, and Tournaments.