tag:status.short.io,2005:/historyShort.io Status - Incident History2024-04-25T05:41:35ZShort.iotag:status.short.io,2005:Incident/205357512024-04-15T09:23:49Z2024-04-15T09:23:49ZHigh rate of 500 errors<p><small>Apr <var data-var='date'>15</var>, <var data-var='time'>09:23</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>15</var>, <var data-var='time'>09:04</var> UTC</small><br><strong>Identified</strong> - DDoS attack</p>tag:status.short.io,2005:Incident/205353882024-04-15T08:57:04Z2024-04-15T11:04:21ZHigh rate of 500 errors during redirects<p><small>Apr <var data-var='date'>15</var>, <var data-var='time'>08:57</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>15</var>, <var data-var='time'>08:37</var> UTC</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Apr <var data-var='date'>15</var>, <var data-var='time'>08:33</var> UTC</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Apr <var data-var='date'>15</var>, <var data-var='time'>08:13</var> UTC</small><br><strong>Identified</strong> - We are currently dealing with new DDoS attack, solution is being deployed</p>tag:status.short.io,2005:Incident/205026542024-04-11T05:17:05Z2024-04-11T05:17:05ZDDoS Attack, affected redirects<p><small>Apr <var data-var='date'>11</var>, <var data-var='time'>05:17</var> UTC</small><br><strong>Resolved</strong> - The incident has been resolved. A fix has been applied to prevent similar types of DDoS attacks.</p><p><small>Apr <var data-var='date'>11</var>, <var data-var='time'>05:13</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results</p><p><small>Apr <var data-var='date'>11</var>, <var data-var='time'>05:12</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating the redirection issue, the fix is in deployment process.</p>tag:status.short.io,2005:Incident/204816422024-04-08T22:31:17Z2024-04-15T19:53:36ZDDoS Attack, affected redirects<p><small>Apr <var data-var='date'> 8</var>, <var data-var='time'>22:31</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'> 8</var>, <var data-var='time'>22:30</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Apr <var data-var='date'> 8</var>, <var data-var='time'>22:29</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Apr <var data-var='date'> 8</var>, <var data-var='time'>22:15</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.short.io,2005:Incident/204641012024-04-06T09:50:31Z2024-04-06T09:50:31ZDDoS attack on Short.io<p><small>Apr <var data-var='date'> 6</var>, <var data-var='time'>09:50</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'> 6</var>, <var data-var='time'>09:44</var> UTC</small><br><strong>Identified</strong> - We are getting a new version of DDoS attack. The fix is applied</p>tag:status.short.io,2005:Incident/204497492024-04-04T16:56:35Z2024-04-04T16:56:35ZDDoS Attack, affected redirects<p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>16:56</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>16:50</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>16:05</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>16:00</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>15:59</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.short.io,2005:Incident/200964472024-02-28T12:37:08Z2024-02-28T12:37:08ZStatistics module malfunction<p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>12:37</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>12:01</var> UTC</small><br><strong>Identified</strong> - The issue has been identified</p><p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>11:19</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.short.io,2005:Incident/199488302024-02-09T16:02:23Z2024-02-09T16:02:23ZDDoS attack, affected redirects<p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>16:02</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>15:48</var> UTC</small><br><strong>Investigating</strong> - We've got a large unexpected DDoS attack, and currently investigating this issue</p>tag:status.short.io,2005:Incident/199214872024-02-06T13:37:24Z2024-02-06T13:37:24ZIncreased rate of 5xx errors during DDoS attack<p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>13:37</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>13:24</var> UTC</small><br><strong>Monitoring</strong> - There was a short spike of 500 errors during DDoS attack for few minutes while our system scaled out to handle this DDoS attack. We are investigate the reason of this accident, but the service is already operational</p>tag:status.short.io,2005:Incident/198939982024-02-02T14:30:00Z2024-02-02T15:15:52ZIncreased rate of 500 errors in redirects<p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>14:30</var> UTC</small><br><strong>Resolved</strong> - Between 14:30 and 14:35 there were increased 500 errors rate caused by unexpected consequences of our primary database optimization</p>tag:status.short.io,2005:Incident/195376612023-12-26T22:18:18Z2023-12-27T08:16:48ZRedirector outage<p><small>Dec <var data-var='date'>26</var>, <var data-var='time'>22:18</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>26</var>, <var data-var='time'>20:52</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Dec <var data-var='date'>26</var>, <var data-var='time'>20:36</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Dec <var data-var='date'>26</var>, <var data-var='time'>20:21</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.short.io,2005:Incident/194196122023-12-12T18:22:14Z2023-12-12T18:22:14ZRedirector partial outage<p><small>Dec <var data-var='date'>12</var>, <var data-var='time'>18:22</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>12</var>, <var data-var='time'>18:22</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Dec <var data-var='date'>12</var>, <var data-var='time'>17:59</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Dec <var data-var='date'>12</var>, <var data-var='time'>17:44</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.short.io,2005:Incident/192903252023-12-03T01:00:00Z2023-12-03T07:58:49ZDashboard displays empty screen<p><small>Dec <var data-var='date'> 3</var>, <var data-var='time'>01:00</var> UTC</small><br><strong>Resolved</strong> - A fix has been implemented and we are monitoring the results.</p>tag:status.short.io,2005:Incident/191882582023-11-21T14:00:00Z2023-11-21T15:55:45ZRedirector downtime<p><small>Nov <var data-var='date'>21</var>, <var data-var='time'>14:00</var> UTC</small><br><strong>Resolved</strong> - We've handled DDoS attack. Like previous one it was done with precise timing, so we did not have time to scale automatically. Like previous time our cache did not handle load and we passed data to upstream. This time we updated cache error management and working on faster reaction on DDoS attacks</p>tag:status.short.io,2005:Incident/189770772023-10-31T13:13:03Z2023-10-31T13:13:03ZIncreased statistics delay in US region<p><small>Oct <var data-var='date'>31</var>, <var data-var='time'>13:13</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>31</var>, <var data-var='time'>09:07</var> UTC</small><br><strong>Investigating</strong> - Our customers are not able to see clicks/data from last 12 hours. Should be fixed by 18:00 UTC</p>tag:status.short.io,2005:Incident/189465392023-10-27T15:48:58Z2023-10-27T15:48:58ZRedirector Degraded Perfomance<p><small>Oct <var data-var='date'>27</var>, <var data-var='time'>15:48</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>27</var>, <var data-var='time'>15:48</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Oct <var data-var='date'>27</var>, <var data-var='time'>15:31</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.short.io,2005:Incident/189426932023-10-27T08:26:20Z2023-10-27T08:26:20ZIncreased rate of 502 errors<p><small>Oct <var data-var='date'>27</var>, <var data-var='time'>08:26</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>27</var>, <var data-var='time'>07:52</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.short.io,2005:Incident/189390672023-10-26T22:58:37Z2023-11-03T23:13:02ZIncreased rate of 502 errors during redirect<p><small>Oct <var data-var='date'>26</var>, <var data-var='time'>22:58</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>26</var>, <var data-var='time'>21:44</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.short.io,2005:Incident/186459912023-10-08T13:00:37Z2023-10-08T13:00:37ZDatabase engine upgrade<p><small>Oct <var data-var='date'> 8</var>, <var data-var='time'>13:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct <var data-var='date'> 8</var>, <var data-var='time'>12:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>21:13</var> UTC</small><br><strong>Scheduled</strong> - We'll upgrade our metadata database from MySQL 5 to MySQL 8. In order to do this we'll need to switch it to the readonly mode for around 30 minutes. It won't affect any operations with short links (they are stored elsewhere), but you won't be able to update domain and user settings during that time</p>tag:status.short.io,2005:Incident/186796042023-10-03T16:37:47Z2023-10-03T16:37:47ZRedirector Partial Outage<p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>16:37</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>16:37</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>16:34</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>16:32</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.short.io,2005:Incident/182021872023-08-20T20:46:25Z2023-08-20T20:46:25ZOur service is under DDoS attack, we are working on a solution<p><small>Aug <var data-var='date'>20</var>, <var data-var='time'>20:46</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>20</var>, <var data-var='time'>15:08</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Aug <var data-var='date'>20</var>, <var data-var='time'>14:13</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p>tag:status.short.io,2005:Incident/181763612023-08-17T18:52:30Z2023-08-17T18:52:30ZRedirector outage<p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>18:52</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>18:51</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>18:51</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>18:27</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>18:26</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.short.io,2005:Incident/181584192023-08-16T02:48:46Z2023-08-16T02:48:46ZDDoS attack, affected redirects<p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>02:48</var> UTC</small><br><strong>Resolved</strong> - We've got a large and unexpected DDoS attack, caused 20 minutes of downtime of short links. We've stopped the attack and working on a solution to prevent impact from such attacks in future</p>tag:status.short.io,2005:Incident/178905192023-07-23T09:15:34Z2023-07-23T09:15:34ZMigration to the new CDN software (part 2)<p><small>Jul <var data-var='date'>23</var>, <var data-var='time'>09:15</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jul <var data-var='date'>23</var>, <var data-var='time'>09:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jul <var data-var='date'>18</var>, <var data-var='time'>15:55</var> UTC</small><br><strong>Scheduled</strong> - Due to limitations of AWS Network Load Balancers we'll need to disconnect elastic IP address from the old network load balancer and connect it to the new one. Some links will return connection refused during this time (around 5 minutes). This time we'll update US region<br /><br />Domains affected: domains, connected to Short.io via US IP address only<br />Domains not affected: Domains, configured via cname.short.io CNAME record, domains, purchased via Short.io, *.short.gy free subdomains</p>tag:status.short.io,2005:Incident/177891802023-07-09T08:15:17Z2023-07-09T08:15:17ZMigration to new CDN software<p><small>Jul <var data-var='date'> 9</var>, <var data-var='time'>08:15</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jul <var data-var='date'> 9</var>, <var data-var='time'>08:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jul <var data-var='date'> 7</var>, <var data-var='time'>11:00</var> UTC</small><br><strong>Scheduled</strong> - Due to limitations of AWS Network Load Balancers we'll need to disconnect elastic IP address from old network load balancer and connect it to the new one. Some links will return connection refused during this time (around 5 minutes)<br /><br />Domains affected: domains, connected to Short.io via EU IP address only<br />Domains not affected: Domains, configured via cname.short.io CNAME record, domains, purchased via Short.io, *.short.gy free subdomains</p>