At approximately 9:30 PM the RAID went offline due to a bad sector on the second bad disk (we are only able to replace one bad disk at a time until the rebuild completes) which forced us to start the rebuild all over again.
Instead of risking additional downtime by performing this rebuild while the node is online a second time we have opted to get the rebuild done fast with the node offline. The estimated down time (which started at 9:30 PM) is 4-6 hours, once this rebuild completes we will finally be able to replace the second bad disk and bring everyone back online.
We apologize and are working quickly to get this node back to normal operation.
Update 1:
Francisco is working hard to get LV-Storage02 back online. Another bad sector has forced the array rebuild to start over (4-5 hours unless there is another bad sector) while we are not giving up on this issue we are offering the following options:
1: We give you a replacement on a completely different node without issue and lose your existing data.
2: You continue to wait while we rebuild the array and clean up this mess.
For more information on these options please do not hessitate to open a ticket.
Update 2:
The node is now back online, for those who have not ready requested a replacement, you are free to continue using your existing service. Being completely honest with you it is more than likely your service will be unusable without reinstall. You can still attempt to run a file system check (fsck) to recover data but the odds are you will need to use a LiveCD such as Ubuntu Desktop or Knoppix to get a network environment online to transfer any data you wish to keep.
With that said the two best options at this point are to either request your disk image be destroyed and recreated prior to reinstalling. If you wish to go this route please confirm there is no data you wish to keep when opening a ticket. Otherwise we are still offering to reprovision any clients on another node while there are spots for them, again please confirm there is no data you wish to keep when requestion to be reprovisioned.
Failure to confirm there is no data you wish to keep with either option will result in delays as we will require confirmation of such.