Over the past 2 weeks people may have noticed quite a bit of random activity/instability within their hosting environments. Our Citrix XenServer beast (dual quad core intel e5140 8 cores 2.33ghz with 24MB cache and 16gb RAM) was having noticeable issues. 6/08/09 we began experiencing abnormal I/O hard drive errors. We attempted to reboot and run a fsck to correct these issues. At this time our 2nd drive became inaccessible. The current configuration was setup with 2×500GB drives with an extended partition to see the space as a whole (1TB total). When the 2nd drive died it destroyed the LVM partitions and killed the file system losing all data. We have had all 5 BitTraffic technicians working on this issue. We have also had help from several of the data center techs. The 2nd hard drive has simply failed. This type of situation does unfortunately occur as we all know. We are now going to implement a 2 socket RAID controller setup which will allow us to have 2 1TB hard drives with exact copies of each other. We already have the brand new 1TB HD in the system and have began rebuilding. We should have everyones accounts recreated and accessible very soon. We apologize for this inconvenience and know how stressful this situation can be. We are investing big $ in our new virtualization platforms and RAID setups. We hope to have cloud computing/redundancy/failovers all setup near the end of the month. This was terrible timing since we were already taking steps towards this type of setup. We cant explain how much we regret not already having a RAID setup. Only reason we have not is due to high expense. We will be implementing this technology regardless of cost for our clients and are on the move to using the best technology available. Please bare with us through this tough time. We will have all access restored asap!
Sincerely thank you,
BitTraffic Administrators