September 16, 2020 12:53AM EDT September 16, 2020 4:53AM UTC
[Investigating] The dal3 node is inaccessible and we're currently looking into the issue. We're sorry for the inconvenience caused.
September 16, 2020 2:31AM EDT September 16, 2020 6:31AM UTC
[Monitoring] Yesterday, we did a live migration of some VMs to load balance the node, the tool we used completed the migration but there were zombie VMs that were left on this node which was going to cause troubles if left like that and there was no other way but reboot the node. What we didn't saw coming was that there were also memory dumps generated by the tool that were filling up hundreds of gigabytes of disk space which led to this incident. This should now be resolved and we're still looking at the things to ensure this won't happen anymore. We're deeply sorry for the inconvenience caused.