Status

Status

Status

Healthy moving forward, but previously have had intermittent issues with the servers running our AI Dungeon API @Last Tuesday: The Heroku instance running our AI Dungeon API has had intermittent issues since last Friday that keep recurring. The team is digging into why this is happening and working to mitigate. We have reached out to Heroku for more information about the behavior we’re seeing. Update @Last Wednesday: We've identified the intermittent issues this past week as a Heroku open connections limit we have been hitting (even using their largest plan). AI Dungeon is now healthy given Heroku allowing us to bypass their normal limits. I will share more details tomorrow as we finalize the fix.

AI Dungeon

AI Dungeon Web
Android App
iOS App

Models

Griffin
Wyvern
Dragon

Voyage

Voyage
Voyage Studio
AI Art

APIs

AI Dungeon API (intermittent issues)
Latitude API
Voyage API

Details

Intermittent Outage with AI Dungeon API (@September 16, 2022 - @Last Thursday )

The Heroku instance running our AI Dungeon API has had intermittent issues since last Friday that keep recurring. The team is digging into why this is happening and working to mitigate. We have reached out to Heroku for more information about the behavior we’re seeing.

Past Incidents

Action Counts Off + Griffin Outage (@August 25, 2022)

Description of the incident.

And we had another outage (a combination of database and then Coreweave instances going down). Action counts were off for a bit and ads were behaving oddly. We're awarding 200 actions to any impacted players.

Actions moving forward include a retro with the team and open conversation with Coreweave about how we build more resilience into the pod cluster, even with high traffic volumes.

Griffin Outage (Coreweave) (@August 25, 2022)

Our Griffin Pods had issues that didn’t recover even with a restart. Coreweave was able to help us successfully get the pods back online. We'll be working with them to figure out what caused this and how to avoid similar outages in the future.

Performance Issues (@August 25, 2022)

9:00 am: Some users are experiencing lag. We are currently digging in. 9:11 am: We've identified the issue related to database performance and are working on a fix. 9:58 am: A fix has been pushed by rolling back a change we made related to the upcoming gold system and scales changes. We will be adjusting given the performance issues before we push this again.

Heroku Outage (@August 23, 2022)

We had intermittent network issues due to an outage with one of our core providers, Heroku.

AI Art temporary outage (@August 16, 2022)

Pixray, Disco Diffusion, and VQgan were temporarily unavailable due to a service outage.

Heroku Outage (@August 15, 2022)

Our hosting provider had an outage that cause about 30 minutes of downtime and 20 minutes of degraded performance.

Coreweave Outage (@August 12, 2022)

One of our AI infrastructure partners, Coreweave, had an outage today that impacted us and other AI experiences. Griffin was unable to generate for 20 minutes because of this outage. We contacted the company and they quickly resolved the issue.

Partial Android App Outage (@August 7, 2022 )

Version 153 of the Android App is installing but the icon isn’t showing for some Android devices. If you haven’t upgraded we invite you not to. We have a new build already waiting on Google to review that a developer worked on late last night.

This was caused by a relatively small package upgrade that worked in local testing but caused an issue with certain devices in production deployment for Google. Frankly we were surprised by this one since Google review is supposed to catch stuff like this which we can’t test without pushing live. We’ve contacted them to expedite this review and looked into any options for rolling back.

Apologies. We will update here once the new version is live. Android players experiencing this issue can play using their mobile browser as an interim solution.

💠
image

© Latitude 2022