tag:www.narratorstatus.com,2005:/historyNarrator Status - Incident History2024-03-28T08:51:48-04:00Narratortag:www.narratorstatus.com,2005:Incident/185504762023-09-20T16:06:12-04:002023-09-20T18:04:03-04:00Narrator is experiencing some request failures<p><small>Sep <var data-var='date'>20</var>, <var data-var='time'>16:06</var> EDT</small><br><strong>Resolved</strong> - The issue has been resolved and post mortem will be written and shared</p><p><small>Sep <var data-var='date'>20</var>, <var data-var='time'>13:53</var> EDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Sep <var data-var='date'>20</var>, <var data-var='time'>12:52</var> EDT</small><br><strong>Update</strong> - We have noticed the issue to not be solved by our initial solution, we are currently testing another solution now.</p><p><small>Sep <var data-var='date'>19</var>, <var data-var='time'>23:10</var> EDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we will continue to monitor for the next couple of days</p><p><small>Sep <var data-var='date'>19</var>, <var data-var='time'>20:43</var> EDT</small><br><strong>Identified</strong> - We have identified the issue and are working to scale up our systems to ensure reliability</p><p><small>Sep <var data-var='date'>19</var>, <var data-var='time'>19:56</var> EDT</small><br><strong>Investigating</strong> - Narrator servers are experiencing intermittent failures, we are diving into it now.</p>tag:www.narratorstatus.com,2005:Incident/177318802023-06-30T15:00:21-04:002023-06-30T15:00:21-04:00Internal Issue with our database<p><small>Jun <var data-var='date'>30</var>, <var data-var='time'>15:00</var> EDT</small><br><strong>Resolved</strong> - We have found and fixed the issue</p><p><small>Jun <var data-var='date'>30</var>, <var data-var='time'>14:20</var> EDT</small><br><strong>Investigating</strong> - We are currently experiencing an issue with our database causing our Auth to not work. You will see this error "Error: User is not authorized to access this resource with an explicit deny"<br /><br />We are actively working on it.</p>tag:www.narratorstatus.com,2005:Incident/175627482023-06-13T17:04:35-04:002023-06-13T17:04:35-04:00AWS servers seem to be having issues<p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>17:04</var> EDT</small><br><strong>Resolved</strong> - AWS is back online</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>15:08</var> EDT</small><br><strong>Investigating</strong> - AWS Lambda servers seem to be having issues. We are currently trying to contact them but the AWS portal is also down.</p>tag:www.narratorstatus.com,2005:Incident/161266712023-02-14T03:00:29-05:002023-02-14T03:00:29-05:00Planned Upgrades<p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>03:00</var> EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>00:00</var> EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>13</var>, <var data-var='time'>14:13</var> EST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:www.narratorstatus.com,2005:Incident/110900602022-09-09T14:23:56-04:002022-09-09T14:23:56-04:00Issues with AWS<p><small>Sep <var data-var='date'> 9</var>, <var data-var='time'>14:23</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'> 9</var>, <var data-var='time'>14:15</var> EDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Sep <var data-var='date'> 9</var>, <var data-var='time'>13:55</var> EDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Sep <var data-var='date'> 9</var>, <var data-var='time'>11:58</var> EDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:www.narratorstatus.com,2005:Incident/98732282022-04-29T01:01:22-04:002022-04-29T01:01:22-04:00Planned Upgrades<p><small>Apr <var data-var='date'>29</var>, <var data-var='time'>01:01</var> EDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Apr <var data-var='date'>29</var>, <var data-var='time'>00:01</var> EDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr <var data-var='date'>28</var>, <var data-var='time'>12:43</var> EDT</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:www.narratorstatus.com,2005:Incident/90738132022-01-13T11:12:41-05:002022-01-13T11:12:42-05:00Narrator's query engine unable to connect<p><small>Jan <var data-var='date'>13</var>, <var data-var='time'>11:12</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>13</var>, <var data-var='time'>11:01</var> EST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jan <var data-var='date'>13</var>, <var data-var='time'>11:01</var> EST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jan <var data-var='date'>13</var>, <var data-var='time'>10:58</var> EST</small><br><strong>Investigating</strong> - We've noticed issues with queries being unable to connect to the underlying data source. Engineering team is looking into it.</p>tag:www.narratorstatus.com,2005:Incident/88669412021-12-18T01:01:44-05:002021-12-18T01:01:44-05:00Planned Upgrades<p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>01:01</var> EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>00:22</var> EST</small><br><strong>Update</strong> - Scheduled maintenance is still in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>00:09</var> EST</small><br><strong>Update</strong> - Scheduled maintenance is still in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>00:01</var> EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'>16</var>, <var data-var='time'>12:22</var> EST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:www.narratorstatus.com,2005:Incident/87959072021-12-10T01:51:27-05:002021-12-10T01:51:27-05:00Planned Upgrades<p><small>Dec <var data-var='date'>10</var>, <var data-var='time'>01:51</var> EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Dec <var data-var='date'>10</var>, <var data-var='time'>01:37</var> EST</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Dec <var data-var='date'>10</var>, <var data-var='time'>00:01</var> EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>16:48</var> EST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:www.narratorstatus.com,2005:Incident/87825592021-12-07T21:04:32-05:002021-12-07T21:04:32-05:00Issues with AWS<p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>21:04</var> EST</small><br><strong>Resolved</strong> - AWS has come back online. We'll continue to monitor any further issues.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>13:17</var> EST</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>12:24</var> EST</small><br><strong>Monitoring</strong> - AWS is reporting incidents in us-east-1</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>11:47</var> EST</small><br><strong>Investigating</strong> - We're currently experiencing issues with 3rd party services. You may see some interruptions.</p>