Emergency Maintenance for DDoS attack prevention
Incident Report for GGNetwork
Resolved
This incident of DDoS has been resolved.

Emergency server maintenance on July20th, UTC 02:50 ~ UTC 03:45
[Cause]
- A DDoS attack that started on July 20th, UTC 01:12 affected one of our servers that lead to players being unable to load the client starting UTC 01:55
- This server was migrated to a new cloud datacenter during the maintenance on July 16th for better performance and to account for increased load during WSOP, but the tech team made a mistake of not shielding this server with our DDoS protection service after the migration.
[Actions Taken]
- This configuration server was added to the DDoS protection service (CloudFlare)
- All other servers were checked and added to the DDoS protection service
- A second layer of DDoS protection will be added for increased protection
Posted Jul 20, 2020 - 03:44 UTC
Monitoring
DDoS Protection has been set up.
Posted Jul 20, 2020 - 03:40 UTC
Update
DDoS attack has been detected and identified. We're preparing the protection from DDoS attack.
Posted Jul 20, 2020 - 02:41 UTC
Identified
Official Statement Regarding Incidents on July 19th
1) Server Delays (Client loading, Login, Cash Game hand dealing, Spin & Gold table loading, Tournament registration and tournament table loading)
- UTC 13:53 July 19th ~ UTC 17:11 July 19th
[Cause]
- Increased load on account service due to an influx of players and observers resulting in service bottlenecks
[Actions Taken]
- Issued a temporary fix during the server-downtime of July 20th to alleviate the issue.
- Will move services that were causing the bottlenecks to other servers thereby distributing the load (by emergency maintenance on July 24th
2) Two incidents of widespread client disconnections
- UTC 17:48 July 19th and UTC 18:17 July19th
[Cause]
- While performing tournament table configuration changes (increase of late registration time to account for delays in 1), there was a case of server-client protocol mismatch between the tournament service and game client that lead to certain clients being disconnected
[Actions Taken]
- Fixed all protocol mismatches on July 20th and issued client hotfix
3) WSOP Events #32, #33 not resuming after the break
[Cause]
- Unforseen bug (multithread timing issue) caused by high load during peak times
[Actions Taken]
- Issued a temporary fix during the server-downtime of July 20th to alleviate the issue.
- Will issue more comprehensive fixes by the emergency maintenance on July 24th to fix the root cause.
- At the worst case, even if these incidents were to happen again, we will add code to manually re-start the tournaments affected by July 24th
Posted Jul 19, 2020 - 13:45 UTC
This incident affected: Login & Account Services, Lobby, Cash Games, and Tournaments.