Playing AI Dungeon
Partial Coreweave Outage (@January 17, 2023)
Coreweave (the provider we host Griffin on) is having some performance issues they recently alerted us about. We are monitoring and will update when performance returns to full. Until then some players may have technical issues for some generations.
Heroku Slowdowns (@December 14, 2022)
We are currently looking into some model slow downs. We are reaching out to Heroku which seems to be the cause of the issues.
Heroku Outage (@December 8, 2022)
December 8, 2022 7:18pm MST
Heroku service has been restored and AI Dungeon should be back to online status.
December 8, 2022 5:51pm MST
Heroku has issued an update on the outage. They have identified the cause of the outage and are working on a fix. They said they will provide an update within 30min.
December 8, 2022 5:28pm MST
The outage seems to be caused by a Heroku outage. We use Heroku to host portions of AI Dungeon. We’ll update the community when service is restored.
December 8, 2022 4:46pm MST
We’re experiencing a 90% outage across the app. Our team is aware of the issue and is diagnosing the problem.
Heroku Outage (@November 30, 2022)
11/30/22 4:11 pm MT We were alerted that AI Dungeon was down for 80% of players for about 20 minutes due to an upstream issue with one of our tech providers. The issue is recovering on its own and should be resolved shortly. 11/30/22 4:16 pm MT It appears all systems are once again operating at full capacity.
Description of the incident.
Database Partial Outage (@November 13, 2022)
11/13/22 4:29 pm MT We are looking into a partial outage on AI Dungeon. Currently diagnosing increased 500 errors in the Latitude API Update 4:37 pm MT We got the server and database back to good health and are diagnosing the cause of the hiccups. We will continue to monitor.
Heroku Intermittent Outage with AI Dungeon API (@September 16, 2022 - @September 22, 2022 )
The Heroku instance running our AI Dungeon API has had intermittent issues since last Friday that keep recurring. The team is digging into why this is happening and working to mitigate. We have reached out to Heroku for more information about the behavior we’re seeing.
Healthy moving forward, but previously have had intermittent issues with the servers running our AI Dungeon API @September 20, 2022: The Heroku instance running our AI Dungeon API has had intermittent issues since last Friday that keep recurring. The team is digging into why this is happening and working to mitigate. We have reached out to Heroku for more information about the behavior we’re seeing. Update @September 21, 2022: We've identified the intermittent issues this past week as a Heroku open connections limit we have been hitting (even using their largest plan). AI Dungeon is now healthy given Heroku allowing us to bypass their normal limits. I will share more details tomorrow as we finalize the fix.
Action Counts Off + Griffin Outage (@August 25, 2022)
Description of the incident.
And we had another outage (a combination of database and then Coreweave instances going down). Action counts were off for a bit and ads were behaving oddly. We're awarding 200 actions to any impacted players.
Actions moving forward include a retro with the team and open conversation with Coreweave about how we build more resilience into the pod cluster, even with high traffic volumes.
Griffin Outage (Coreweave) (@August 25, 2022)
Our Griffin Pods had issues that didn’t recover even with a restart. Coreweave was able to help us successfully get the pods back online. We'll be working with them to figure out what caused this and how to avoid similar outages in the future.
Database Performance Issues (@August 25, 2022)
9:00 am: Some users are experiencing lag. We are currently digging in. 9:11 am: We've identified the issue related to database performance and are working on a fix. 9:58 am: A fix has been pushed by rolling back a change we made related to the upcoming gold system and scales changes. We will be adjusting given the performance issues before we push this again.
Heroku Outage (@August 23, 2022)
We had intermittent network issues due to an outage with one of our core providers, Heroku.
AI Art temporary outage (@August 16, 2022)
Pixray, Disco Diffusion, and VQgan were temporarily unavailable due to a service outage.
Heroku Outage (@August 15, 2022)
Our hosting provider had an outage that cause about 30 minutes of downtime and 20 minutes of degraded performance.
Coreweave Outage (@August 12, 2022)
One of our AI infrastructure partners, Coreweave, had an outage today that impacted us and other AI experiences. Griffin was unable to generate for 20 minutes because of this outage. We contacted the company and they quickly resolved the issue.
Partial Android App Outage (@August 7, 2022 )
Version 153 of the Android App is installing but the icon isn’t showing for some Android devices. If you haven’t upgraded we invite you not to. We have a new build already waiting on Google to review that a developer worked on late last night.
This was caused by a relatively small package upgrade that worked in local testing but caused an issue with certain devices in production deployment for Google. Frankly we were surprised by this one since Google review is supposed to catch stuff like this which we can’t test without pushing live. We’ve contacted them to expedite this review and looked into any options for rolling back.
Apologies. We will update here once the new version is live. Android players experiencing this issue can play using their mobile browser as an interim solution.
© Latitude 2023