Portal Home > Knowledgebase > Industry Announcements > Web Hosting Main Forums > Providers and Network Outages and Updates > Precision Effect - downtime for nearly 48 hours now.
Precision Effect - downtime for nearly 48 hours now.
| Posted by ihatepe, 02-18-2010, 10:45 PM |
I never do this, but I created an account just to let off some steam.
I've been with PE for a couple of years now, and never had any significant problems with them. I understand they just underwent a change of management. Well, the new management are a bunch of incompetent imbiciles.
My site has now been down for about 48 hours. I have lost big $$$ in revenue, and my Google ratings have probably been destroyed. It is a blow from which my site may never recover. They've sent a few emails over the last 48 hours promising that the solution is 'just around the corner', and even been so kind as to give me 3 months free hosting as compensation.
Bunch of morons.
Avoid this company at all costs - they have likely destroyed one of my online businesses, and if I was located in the USA and had the time, I would sue the **** out of them. |
| Posted by Quartz, 02-18-2010, 10:48 PM |
I do know that PE has undergone management changes recently, as you also noted, however to be down this long is strange.
Have they contacted you with reasoning as to why this has occurred?
Also, I know you cannot post any domain(s) and would recommend not to go around the filters, so I would click the Report button on your post so that to send your domain and confirm it is at least hosted off PE's services. Also this would allow admins to also see if the site loads or does not load for them as well. |
| Posted by ihatepe, 02-18-2010, 10:51 PM |
I have no problem doing that - I will send the admins the site right now.
This was sent to me 11 hours ago:
Dear Client,
What happened was completely unexpected and a true catastrophe. Our whole team stayed after and worked to get everything back up and running as quickly as possible. The issue was caused by the datacenter corrupting the SAN that we were on. We've put measures into place that will prevent this from ever occuring in the future, including on-the-fly local data storage. We understand that your uptime is paramount and over the last 24 hours, we failed. Your business means everything to us, and your uptime means everything to you. I want to let you know that we are committed to prevent things like this, or even any down time atall, from occuring again. That being said, we have provided your account with 3 months of free service and a free upgrade to the next tiered account. We fully understand if you wish to leave our services due to the downtime incurred. For those of you that do not, we truly appreciate your loyalty, and we promise you that we will not let this happen again.
We expect connectivity by noon today. |
| Posted by ihatepe, 02-18-2010, 10:54 PM |
And this one, 20 hours ago:
Hi xxxxxxxx,
We have had sporadic connectivity for quite some time, sometimes its on sometimes it off.
We found out that the problem was not with us but with our Data Center, we here at precision effect have talked to the CEO of our data center personally and he told he is personally setting us up on a new fail over cloud system. We are clients in this matter just as you are a client of us, we can only sit and wait for them to fix the issue.
We still are unsure of the estimated time it will take for this to be done, but I will update you further as we know more.
We are absolutely furious about this downtime as much as you are. |
| Posted by jk079, 02-18-2010, 11:00 PM |
I've been a PE customer since Feb. 2008, and I have also been extremely disappointed by their handling of the downtime. They did not bother to contact me at all about the downtime until I submitted a support ticket, and their updates mostly concentrate on blaming other people.
I was already wary of the new management, given that I didn't even learn of it until a new name appeared on my credit card charges, and this is not inspiring further confidence.
I run a small personal site with very modest storage and bandwidth needs, but given that it is part of my professional public face, I need it to be up. That is why I chose PE in the first place. I am obviously now researching alternatives. |
| Posted by anon-e-mouse, 02-18-2010, 11:05 PM |
Quote:
|
Originally Posted by ihatepe
I have no problem doing that - I will send the admins the site right now.
|
Thanks and confirmed.
Moved to outages. |
| Posted by ihatepe, 02-18-2010, 11:06 PM |
Same here - they only bothered to mention the downtime after I submitted a support ticket, and since then I have heard nothing but excuses. I too heard nothing about the change of management.
And there is no 'sporadic connectivity' as they mentioned in their email - it is downtime, plain and simple.
If precision effect is relying on another company for their data servers, what exactly am I paying them for? |
| Posted by RSkeens, 02-18-2010, 11:16 PM |
Quote:
|
Originally Posted by jk079
I've been a PE customer since Feb. 2008
|
Domain is confirmed to be currently with Precision Effect, thank you for submitting. |
| Posted by ihatepe, 02-18-2010, 11:16 PM |
|
It's really too bad this was moved out of the general section, because most people won't see it. Is there any way to do a post on PE that will stay in the general section? |
| Posted by TimothyH, 02-18-2010, 11:58 PM |
Quote:
|
Originally Posted by ihatepe
It's really too bad this was moved out of the general section, because most people won't see it. Is there any way to do a post on PE that will stay in the general section?
|
Post a review of your experience with PE, it will then remain in the general section. |
| Posted by G-Force, 02-19-2010, 12:07 AM |
I've also been frustrated with PE with the new change in management. Everytime they upgrade the software on their servers, my Gallery 2 gets hosed. And to top it off all this downtime and potentially lost email.
I too am searching for alternatives. |
| Posted by katrayun, 02-19-2010, 03:43 AM |
Still down. It's the potentially lost email that's killing us. I'm starting to doubt we'll see progress made before the weekend.
We didn't know about the change in management until this occurrence. The knowledge will make it easy to abandon ship if necessary after this, sad but true. Precision Effect was an excellent host for the two years we've been with them, and if they make it through this period of instability, the new owners may prove themselves as well. But we cannot risk our business on seeing them through it.
Edit: Go figure, as soon as I post here things come back up. I hope this lasts longer than the hour or two of uptime we had this morning! |
| Posted by anon-e-mouse, 02-19-2010, 04:22 AM |
Quote:
|
Originally Posted by ihatepe
It's really too bad this was moved out of the general section, because most people won't see it. Is there any way to do a post on PE that will stay in the general section?
|
If people are looking for an outage thread they *should* look in here. A search will find it too  |
| Posted by RSkeens, 02-19-2010, 06:01 AM |
Quote:
|
Originally Posted by katrayun
Edit: Go figure, as soon as I post here things come back up.
|
Domain is confirmed to be currently with Precision Effect, thank you for submitting. |
| Posted by routingcast, 02-19-2010, 10:15 AM |
|
Is anyone able to access their FTP or cpanel? |
| Posted by rickjhh, 02-19-2010, 11:22 AM |
My website is up and my blog is up. Website came up late yesterday. Blog sometime during the night. My email is still not up. I was able to access a different control panel -- they used to call it "precision panel", I think. It's different now. No email information is out there, and squirrell mail doesn't work either.
I agree. I used PE for several years and they were absolutely great. I was, and remain somewhat of a neophyte, and Dave Garbus was always great with answering what were admittedly simple questions. I had a similar, but much shorter lived problem a few weeks ago when they were apparently migrating to a new data center or something like that.
I agree with the previous poster -- I should have been wary when I saw we were paying somebody new. Hopefully they get it fixed and I can breathe again, then figure out where to move things. |
| Posted by routingcast, 02-19-2010, 11:30 AM |
Can you tell me how you accessed it? I used to be able to go "pe1 . precisioneffect . com / panel" but that no longer works. I changed my nameservers and I need access to my files!
Quote:
|
Originally Posted by rickjhh
I was able to access a different control panel -- they used to call it "precision panel", I think.
|
|
| Posted by Muldoon, 02-19-2010, 11:40 AM |
Could someone from Precision Effect please contact me via the ticket I have submitted. I have now been ignored for over 17 hours and my website is still down, and I would appreciate a response knowing that you are working to resolve this.
Thank you. |
| Posted by jk079, 02-19-2010, 11:48 AM |
My site is back up. The panel looks different, my email forwarders were deleted, and my ftp accounts were deleted. I had to set them up again to get them to work.
Also, they apparently restored twice--first with an older backup and then with a newer backup. Of course, they overwrote database changes (new comments) that happened between the two restores. Those are now completely lost, since I didn't get comment notifications...because my email forwarders were nuked.
As for me, I'm moving to Hawk Host. |
| Posted by rickjhh, 02-19-2010, 11:54 AM |
Quote:
|
Originally Posted by routingcast
Can you tell me how you accessed it? I used to be able to go "pe1 . precisioneffect . com / panel" but that no longer works. I changed my nameservers and I need access to my files!
|
I used a bookmark that I have in Internet Explorer:
https://pe1.precisioneffect.com:2222/
It tells me there is a certificate error - hopefully no big deal, and then I have to log in as before.
The panel is called "Direct Admin" |
| Posted by rickjhh, 02-19-2010, 11:58 AM |
So does anybody have email that has come back? Do we have to start over with email? If we do start over does that cause any problems?
It would be great if they had the simple courtesy and decency to let us know what is happenning. If they have lost our stuff, at least we know what we need to do. |
| Posted by Muldoon, 02-19-2010, 12:29 PM |
|
Thank you for the link. I just made a backup and I'm downloading it now. Should download in 50 minutes if I don't lose the connection. However, I don't even know if what they restored is up-to-date. I guess I'll just hold on to this backup for now and hopefully not need it...I just wish I could get a response and get my site back up. |
| Posted by routingcast, 02-19-2010, 12:33 PM |
Thanks for this information! I was able to download a backup copy of my website and DB and transfer them over to my new host.
In regards to email, I can log onto SquirrelMail using my main account, but not with any of my email addresses.
Quote:
|
Originally Posted by rickjhh
I used a bookmark that I have in Internet Explorer:
It tells me there is a certificate error - hopefully no big deal, and then I have to log in as before.
The panel is called "Direct Admin"
|
|
| Posted by rickjhh, 02-19-2010, 12:33 PM |
I got this response at 11:25 Eastern
Dear Clients,
Our servers were completely restored last night onto SAS systems. There may still be some individual issues
to iron out. We ask that any clients with specific issues please open a new ticket with the issues so we can
quickly and effectively resolve them. I am closing the server down tickets in order to focus on the current
issues at hand.
I would like to thank everyone for their continued patience and loyalty during this catastrophic downtime.
We felt the move to a redundant infrastructure w/ iSCSI SAN was the best move for our clients. However, this
all came tumbling down when the datacenter we are in reported massive SAN input/output issues. Hewlett-
Packard Storage Works believe the issue was isolated to the storage nodes we were located on in the SAN.
It seems as if we've been cursed these past few weeks, and I really have no words to express how sorry I am
to all of our clients. We have really let you down, and I hold myself completely responsible for the
decision to move to this cloud.
Moving forth, we are on 15K SA-SCSI systems. These drives are ultra fast, and some of you might already be
experiencing near instant page loads. Most providers use SATA drives, simply because they are 2% cheaper.
However, they spin at 7200RPMs (as opposed to 15,000), have a higher failure rate, and are not capable of
multiple input/ouput operations at the same time.
Any clients wishing to stay with us through this fiasco will be provided with 3 months of free service so
that we can earn your confidence back. Please send in a ticket to Billing for this credit.
Thank you so much.
Ronny Fang, CTO
PrecisionEffect |
| Posted by Muldoon, 02-19-2010, 07:05 PM |
|
Well, they were finally able to get me back online around 4 1/2 hours ago or so. My site ended up being down for 48 hours +/- 30 minutes. |
| Posted by ieee488, 02-19-2010, 10:00 PM |
I can finally log into the control panel, but I still can't FTP in.
The control panel shows 1 FTP account. Shouldn't that FTP account be the main username+password? Yet, I get a login error.
FTP management shows no account.
Also, are all emails lost? |
| Posted by PE-Phil, 02-21-2010, 02:45 AM |
Hi everyone, this is a message we sent to all of our clients. I believe this should clear up many of the questions you have in this thread. We are so sorry that this has happened, and we're thankful that all data was able to be fully recovered.
What a week.. I cannot begin to express my sorrow at what transpired this past week.
At 12PM Wednesday afternoon we began to experience high load issues on our Cloud Infrastructure which hosted 2 of our servers, AlphaCentauri and UrsaMinor. By 1PM sites were going offline due to the extreme iowait times. Iowait is basically a bottleneck in data transmission from the storage device, the SAN.
In essence, data was being delivered so slowly to the server that no web pages were capable of being served. We diagnosed the cause of this problem to lie with the clustered 40TB iSCSI SAN provided by our datacenter (using Hewlett-Packard StorageWorks). The datacenter spent precious hours troubleshooting it to no avail. By midnight that night, they determined it was an issue with our cloud, that we were sending mass amounts of data to their system.
We knew this was not true... we host regular clients with dyanmic yet normal content. By this time the downtime was already unacceptable. They claimed that we were the only client having issues. I spoke with the CEO of our datacenter and convinced him to move us to another cloud.
By morning, the migration was complete yet the problem persisted. We were forced to immediately switch all of our customers to traditional boxes running 15K RPM SA-SCSI Drives. The sites were restored from backups that were a day (by that time 2 days) old. However our sync of the latest, fresh data from the SAN was being transferred at crawling speed due to the iowait issues. By this time the datacenter techs informed us other customers were having similar issues. Friday morning Hewlett-Packard technicians were in the datacenter troubleshooting and diagnosing their SAN.
As of this moment we still have no had an explanation from our DC what the issue was, or if it is fixed. The last we heard was that there may have been faulty drives in part of the SAN - coincidentally the part of the SAN that we were stored on. This has been an arduous journey. Many members of our staff stayed after and even stayed for double shifts, without even asking for overtime, during the past few days. It makes me happy to know that we have staff who care so much. Phil and I have not left the office since Wednesday night. We are completely invested in this company, and it breaks our heart to see so many of our clients websites go down.
We know how important your websites are, and how stressful this entire situation has been. I know that we have lost the trust of many of our clients, but we will continue to be here and build back your trust, one day at a time. For those of you who inevitably decide to seek an alternate provider, please send in a ticket for a full refund of February. You should not need to pay for a month when service was not met. We accept full responsibility for what happened, and we'll also hold your account for as long as you need to move data off of it. For those clients who decide to stay with us, we thank you and appreciate you from the bottom of our hearts. We promise we will not let you down again, ever.
The clients who have decided to stay with us, please don't forget to send in a ticket to Billing to have the hosting charges for the next 3 months waived.
If you have any questions, concerns, or comments, please feel free to email us, call us, or send in a support ticket.
Best Regards,
Ronny Fang, CTO
615-500-0317
PrecisionEffect.com
on Behalf of Our Entire Staff
Philip, Alby, Angela, Aaron, Kris, and Roshan. |
Add to Favourites 
Print this Article
Also Read
Shared Hosting
Shared Hosting
