Downtime Report - April 1st

[Veritas]

Hi all,

Today, the site was offline for about 50 minutes between about 5:30 PM and 6:30 PM. This was the result of a router crash in the network of our server host, which has now been remedied. This was not the result of a DDoS or our DDoS filtering; it was an unrelated issue. We do not expect any further downtime at this time.

As always, if you are unable to access the site, or have any other issues, please don't hesitate to contact me.

The full RFO (reason for outage) from our host is below.

Veritas

[spoiler=BuyVM/Frantech RFO, 1 April 2014]First off, we'd like to apologize for any issue this network outage in New Jersey may have caused you.

At 1:56 PM on April 1st 2014, we were notified of packetloss in our New Jersey deployment.
Initial investigation looked to be a DDoS attack pending nullroute, but what we quickly realized it was to be a software fault.

Our initial debugging couldn't spot the cause of the issue so a quick router reboot was performed to clear any possible stuck kernel threads.
Unfortunately, this didn't help and only escalated the issue to the point that the router refused all connections.

At this point we placed an emergency ticket with our datacenter to provide us a KVM-over-IP unit as well as another reboot to the router which allowed Francisco to spot a problem with our E1000 driver's interrupt handling.
We have updated the driver and have contacted Intel for additional feedback on the issue at hand.

Again, we apologize for any inconvenience this issue has caused, if you have any questions or concerns, please feel free to let us know.
[/spoiler]