tag:status.short.io,2005:/history Short.io Status - Incident History 2024-04-25T05:41:35Z Short.io tag:status.short.io,2005:Incident/20535751 2024-04-15T09:23:49Z 2024-04-15T09:23:49Z High rate of 500 errors <p><small>Apr <var data-var='date'>15</var>, <var data-var='time'>09:23</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>15</var>, <var data-var='time'>09:04</var> UTC</small><br><strong>Identified</strong> - DDoS attack</p> tag:status.short.io,2005:Incident/20535388 2024-04-15T08:57:04Z 2024-04-15T11:04:21Z High rate of 500 errors during redirects <p><small>Apr <var data-var='date'>15</var>, <var data-var='time'>08:57</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>15</var>, <var data-var='time'>08:37</var> UTC</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Apr <var data-var='date'>15</var>, <var data-var='time'>08:33</var> UTC</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Apr <var data-var='date'>15</var>, <var data-var='time'>08:13</var> UTC</small><br><strong>Identified</strong> - We are currently dealing with new DDoS attack, solution is being deployed</p> tag:status.short.io,2005:Incident/20502654 2024-04-11T05:17:05Z 2024-04-11T05:17:05Z DDoS Attack, affected redirects <p><small>Apr <var data-var='date'>11</var>, <var data-var='time'>05:17</var> UTC</small><br><strong>Resolved</strong> - The incident has been resolved. A fix has been applied to prevent similar types of DDoS attacks.</p><p><small>Apr <var data-var='date'>11</var>, <var data-var='time'>05:13</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results</p><p><small>Apr <var data-var='date'>11</var>, <var data-var='time'>05:12</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating the redirection issue, the fix is in deployment process.</p> tag:status.short.io,2005:Incident/20481642 2024-04-08T22:31:17Z 2024-04-15T19:53:36Z DDoS Attack, affected redirects <p><small>Apr <var data-var='date'> 8</var>, <var data-var='time'>22:31</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'> 8</var>, <var data-var='time'>22:30</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Apr <var data-var='date'> 8</var>, <var data-var='time'>22:29</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Apr <var data-var='date'> 8</var>, <var data-var='time'>22:15</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p> tag:status.short.io,2005:Incident/20464101 2024-04-06T09:50:31Z 2024-04-06T09:50:31Z DDoS attack on Short.io <p><small>Apr <var data-var='date'> 6</var>, <var data-var='time'>09:50</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'> 6</var>, <var data-var='time'>09:44</var> UTC</small><br><strong>Identified</strong> - We are getting a new version of DDoS attack. The fix is applied</p> tag:status.short.io,2005:Incident/20449749 2024-04-04T16:56:35Z 2024-04-04T16:56:35Z DDoS Attack, affected redirects <p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>16:56</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>16:50</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>16:05</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>16:00</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>15:59</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p> tag:status.short.io,2005:Incident/20096447 2024-02-28T12:37:08Z 2024-02-28T12:37:08Z Statistics module malfunction <p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>12:37</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>12:01</var> UTC</small><br><strong>Identified</strong> - The issue has been identified</p><p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>11:19</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p> tag:status.short.io,2005:Incident/19948830 2024-02-09T16:02:23Z 2024-02-09T16:02:23Z DDoS attack, affected redirects <p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>16:02</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>15:48</var> UTC</small><br><strong>Investigating</strong> - We've got a large unexpected DDoS attack, and currently investigating this issue</p> tag:status.short.io,2005:Incident/19921487 2024-02-06T13:37:24Z 2024-02-06T13:37:24Z Increased rate of 5xx errors during DDoS attack <p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>13:37</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>13:24</var> UTC</small><br><strong>Monitoring</strong> - There was a short spike of 500 errors during DDoS attack for few minutes while our system scaled out to handle this DDoS attack. We are investigate the reason of this accident, but the service is already operational</p> tag:status.short.io,2005:Incident/19893998 2024-02-02T14:30:00Z 2024-02-02T15:15:52Z Increased rate of 500 errors in redirects <p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>14:30</var> UTC</small><br><strong>Resolved</strong> - Between 14:30 and 14:35 there were increased 500 errors rate caused by unexpected consequences of our primary database optimization</p> tag:status.short.io,2005:Incident/19537661 2023-12-26T22:18:18Z 2023-12-27T08:16:48Z Redirector outage <p><small>Dec <var data-var='date'>26</var>, <var data-var='time'>22:18</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>26</var>, <var data-var='time'>20:52</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Dec <var data-var='date'>26</var>, <var data-var='time'>20:36</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Dec <var data-var='date'>26</var>, <var data-var='time'>20:21</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p> tag:status.short.io,2005:Incident/19419612 2023-12-12T18:22:14Z 2023-12-12T18:22:14Z Redirector partial outage <p><small>Dec <var data-var='date'>12</var>, <var data-var='time'>18:22</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>12</var>, <var data-var='time'>18:22</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Dec <var data-var='date'>12</var>, <var data-var='time'>17:59</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Dec <var data-var='date'>12</var>, <var data-var='time'>17:44</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p> tag:status.short.io,2005:Incident/19290325 2023-12-03T01:00:00Z 2023-12-03T07:58:49Z Dashboard displays empty screen <p><small>Dec <var data-var='date'> 3</var>, <var data-var='time'>01:00</var> UTC</small><br><strong>Resolved</strong> - A fix has been implemented and we are monitoring the results.</p> tag:status.short.io,2005:Incident/19188258 2023-11-21T14:00:00Z 2023-11-21T15:55:45Z Redirector downtime <p><small>Nov <var data-var='date'>21</var>, <var data-var='time'>14:00</var> UTC</small><br><strong>Resolved</strong> - We've handled DDoS attack. Like previous one it was done with precise timing, so we did not have time to scale automatically. Like previous time our cache did not handle load and we passed data to upstream. This time we updated cache error management and working on faster reaction on DDoS attacks</p> tag:status.short.io,2005:Incident/18977077 2023-10-31T13:13:03Z 2023-10-31T13:13:03Z Increased statistics delay in US region <p><small>Oct <var data-var='date'>31</var>, <var data-var='time'>13:13</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>31</var>, <var data-var='time'>09:07</var> UTC</small><br><strong>Investigating</strong> - Our customers are not able to see clicks/data from last 12 hours. Should be fixed by 18:00 UTC</p> tag:status.short.io,2005:Incident/18946539 2023-10-27T15:48:58Z 2023-10-27T15:48:58Z Redirector Degraded Perfomance <p><small>Oct <var data-var='date'>27</var>, <var data-var='time'>15:48</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>27</var>, <var data-var='time'>15:48</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Oct <var data-var='date'>27</var>, <var data-var='time'>15:31</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p> tag:status.short.io,2005:Incident/18942693 2023-10-27T08:26:20Z 2023-10-27T08:26:20Z Increased rate of 502 errors <p><small>Oct <var data-var='date'>27</var>, <var data-var='time'>08:26</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>27</var>, <var data-var='time'>07:52</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p> tag:status.short.io,2005:Incident/18939067 2023-10-26T22:58:37Z 2023-11-03T23:13:02Z Increased rate of 502 errors during redirect <p><small>Oct <var data-var='date'>26</var>, <var data-var='time'>22:58</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>26</var>, <var data-var='time'>21:44</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p> tag:status.short.io,2005:Incident/18645991 2023-10-08T13:00:37Z 2023-10-08T13:00:37Z Database engine upgrade <p><small>Oct <var data-var='date'> 8</var>, <var data-var='time'>13:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct <var data-var='date'> 8</var>, <var data-var='time'>12:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>21:13</var> UTC</small><br><strong>Scheduled</strong> - We'll upgrade our metadata database from MySQL 5 to MySQL 8. In order to do this we'll need to switch it to the readonly mode for around 30 minutes. It won't affect any operations with short links (they are stored elsewhere), but you won't be able to update domain and user settings during that time</p> tag:status.short.io,2005:Incident/18679604 2023-10-03T16:37:47Z 2023-10-03T16:37:47Z Redirector Partial Outage <p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>16:37</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>16:37</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>16:34</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>16:32</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p> tag:status.short.io,2005:Incident/18202187 2023-08-20T20:46:25Z 2023-08-20T20:46:25Z Our service is under DDoS attack, we are working on a solution <p><small>Aug <var data-var='date'>20</var>, <var data-var='time'>20:46</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>20</var>, <var data-var='time'>15:08</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Aug <var data-var='date'>20</var>, <var data-var='time'>14:13</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p> tag:status.short.io,2005:Incident/18176361 2023-08-17T18:52:30Z 2023-08-17T18:52:30Z Redirector outage <p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>18:52</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>18:51</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>18:51</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>18:27</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>18:26</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p> tag:status.short.io,2005:Incident/18158419 2023-08-16T02:48:46Z 2023-08-16T02:48:46Z DDoS attack, affected redirects <p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>02:48</var> UTC</small><br><strong>Resolved</strong> - We've got a large and unexpected DDoS attack, caused 20 minutes of downtime of short links. We've stopped the attack and working on a solution to prevent impact from such attacks in future</p> tag:status.short.io,2005:Incident/17890519 2023-07-23T09:15:34Z 2023-07-23T09:15:34Z Migration to the new CDN software (part 2) <p><small>Jul <var data-var='date'>23</var>, <var data-var='time'>09:15</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jul <var data-var='date'>23</var>, <var data-var='time'>09:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jul <var data-var='date'>18</var>, <var data-var='time'>15:55</var> UTC</small><br><strong>Scheduled</strong> - Due to limitations of AWS Network Load Balancers we'll need to disconnect elastic IP address from the old network load balancer and connect it to the new one. Some links will return connection refused during this time (around 5 minutes). This time we'll update US region<br /><br />Domains affected: domains, connected to Short.io via US IP address only<br />Domains not affected: Domains, configured via cname.short.io CNAME record, domains, purchased via Short.io, *.short.gy free subdomains</p> tag:status.short.io,2005:Incident/17789180 2023-07-09T08:15:17Z 2023-07-09T08:15:17Z Migration to new CDN software <p><small>Jul <var data-var='date'> 9</var>, <var data-var='time'>08:15</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jul <var data-var='date'> 9</var>, <var data-var='time'>08:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jul <var data-var='date'> 7</var>, <var data-var='time'>11:00</var> UTC</small><br><strong>Scheduled</strong> - Due to limitations of AWS Network Load Balancers we'll need to disconnect elastic IP address from old network load balancer and connect it to the new one. Some links will return connection refused during this time (around 5 minutes)<br /><br />Domains affected: domains, connected to Short.io via EU IP address only<br />Domains not affected: Domains, configured via cname.short.io CNAME record, domains, purchased via Short.io, *.short.gy free subdomains</p>