isn't backing up standard prior to making a major config change?

Portal Home > Knowledgebase > Articles Database > isn't backing up standard prior to making a major config change?

Posted by hD-Scott, 12-15-2009, 10:18 PM
We have a server management company administering one of our servers. We're not going to mention any names, do please don't ask. This thread is not meant to flame them; however, before we begin to shop around for another server admin company, I want to ask this question: Isn't it standard procedure to backup any config files (or any files for that matter) prior to making any changes to the server? I ask this because this is the second time that this server mgmt company has make a major change to the server and didn't back up any files. The first time they totally uninstalled CSF (my preferred firewall), deleted all the config files including previous backups and installed APF and didn't even notify me. When I was getting all kinds of errors (whmcs couldn't connect to my servers), my clients where having issues and tickets were rolling in, I asked if they had made any changes and that's when the notified me that they uninstalled CSF and installed APF. When I told them to put it back, they did, but my previously customized config of CSF with all my ports and other settings was lost forever. It took me two days to get it running back as it was as I had to work through each glitch that would occur. Now, I've asked them to get spamassassin running again. It was running fine and then all of a sudden, it was no longer scanning email. I figure it is some change that they made and didn't tell me about. Anyway, for the past 2 days they have been working on this (and it still doesn't work). I asked for an update as my server is receiving lots of spam and they told me that they reinstalled exim and spamassassin, but yet it's still not working right. When I looked at the exim.conf file, it was one dated from 2 years (almost 3 years) ago. I had updated the exim.conf file yesterday and it took over an hour to configure it properly for the server. When I replaced my old one with the new one, I backed it up with a date (my general habit), but I didn't back up the new one as it wasn't necessary because I wasn't replacing it. But they destroyed the one I put in just yesterday and didn't back it up. So my question is, isn't this standard practice to backup files before changing them? Is this just me being picky? And shouldn't I expect a full report on any changes they make to MY server and preferably BEFORE they make them? In all my other dealings with hosting accounts, VPS, hybrids, and dedis, support has always ASKED prior to making any changes and then reported exactly what they did (usually including the command) without my asking. Am I asking too much?
Posted by smajer, 12-15-2009, 11:20 PM
As you explained in this case, they should have made backups of the configuration. Is this a managed server? You may have to pay extra for daily backups. It is good practice, yes. But you should also mention it before hand, as it is your server and whatever request you make is taken as that. But as stated, in this scenerio excpecting a backup was normal.
Posted by oliviakitty, 12-15-2009, 11:24 PM
As far as I know, It is standard practice to backup files before changing them. (This seems especially necessary when you are performing actions without the knowledge, much less consent, of the client.) Even with trivial software/scripts, there are warnings to back up existing files or an mechanism in place that does so. Restore points are vital.
Posted by hD-Scott, 12-15-2009, 11:26 PM
It's not a managed server, it an extra service that I pay for from a server management company. If I knew they were going to make these changes I would have requested backups (not of the server, but of the config files (ex: exim.conf.bak.15dec), but I did not know until I found an error that these changes had been made. I would say I have X issue and they assume without asking me that it's a firewall issue and instead of troubleshooting the firewall, they uninstall the current one and install one of their own choosing and wipe out all of the config files. In the latter case, all of a sudden spamassassin stopped working and floods of spam were getting through the server. I asked them to look into it. They insisted that spamassassin was installed (yeah, I know, I installed it and it WAS working), and they said it was currently working). I showed them that it was not and that's when they decided to reinstall exim and spamassassin, basically destroying all of my config files for both (months of work). Again, they told me this AFTER the fact and I've already discussed the not backing up of files previously with the first incident.
Posted by hD-Scott, 12-15-2009, 11:28 PM
oliviakitty That's what I thought. I'm certainly no Linux expert, but I can do most things and know when to hire someone who knows more. But this seems to be their MO and their business is server administration. I'm so fed up because it took months of trial and error to get my exim/spamassassin configs just right and working and now they've destroyed both.
Posted by Steven, 12-16-2009, 01:02 AM
Let me give you some perspective into how I do management. It depends on the type of work being performed. Generally speaking if CSF is installed already I work with it, if APF is installed I work with that. Unless its a new install I don't replace anything like that because either one does its job without problems. Take the exim.conf for example, I would backup the file and then perform the changes if it was going to be extensive. Customers generally come to me when they don't want to handle of the management. When I take them on I ask them if there's anything I need to be aware of. If there is, I take a note of it and don't distrupt the function UNLESS it is insecure, after which I would advise, and we would work out a solution. If it's a pretty basic setup I don't backup files, because I know they originally were setup. One thing I always backup before changing is my.cnf so I can look back at old settings if the server doesn't perform as well as I expect. From my experience, it is rare you will find a company which backups files before changes.
Posted by hD-Scott, 12-16-2009, 01:10 AM
Steven So you're saying that if you're going to reinstall both spamassassin and exim, you wouldn't backup the clients config files first, just in case something would go wrong? Because that is what happened. They reinstalled both apps replacing our config files and the only backups that remained are old ones that I myself had replaced and backed up. In the case of removing CSF, they deleted the directory which destroyed all backups that we had of config files. And all of my config files had custom settings in them after months of trial and error to get things working just right and in the case of the exim.conf, they replaced a recent RC with a config file that was over 2 years, almost 3 years old. I wouldn't feel comfortable with a management company who didn't make backups and then said, oops, sorry, we deleted the files and we can't get them back. My bad. That's why we're leaving our current company and looking for a replacement. I know backups are important and that's why I make them, but I don't have a backup of the entire server config because I have backups of the config files on the server. I didn't think (my bad) to download them as well, because I don't make a change without making a backup and only in the case of an entire server failure, would I have to completely start from scratch, but since I'd be building a whole new server, it wouldn't be that big of a deal and I would reharden and retweak again. All of my client's sites are backed up off-site, so it wouldn't be a disaster. I just find it hard to believe that someone would make major changes to config files of a server that wasn't theirs and didn't back them up and then said OOPS! Our bad. Sorry. Remember, I am paying these people for a service.
Posted by hD-Scott, 12-16-2009, 01:11 AM
Steven So you're saying that if you're going to reinstall both spamassassin and exim, you wouldn't backup the clients config files first, just in case something would go wrong? Because that is what happened. They reinstalled both apps replacing our config files and the only backups that remained are old ones that I myself had replaced and backed up. In the case of removing CSF, they deleted the directory which destroyed all backups that we had of config files. And all of my config files had custom settings in them after months of trial and error to get things working just right and in the case of the exim.conf, they replaced a recent RC with a config file that was over 2 years, almost 3 years old. I wouldn't feel comfortable with a management company who didn't make backups and then said, oops, sorry, we deleted the files and we can't get them back. My bad. That's why we're leaving our current company and looking for a replacement. I know backups are important and that's why I make them, but I don't have a backup of the entire server config because I have backups of the config files on the server. I didn't think (my bad) to download them as well, because I don't make a change without making a backup and only in the case of an entire server failure, would I have to completely start from scratch, but since I'd be building a whole new server, it wouldn't be that big of a deal and I would reharden and retweak again. All of my client's sites are backed up off-site, so it wouldn't be a disaster. I just find it hard to believe that someone would make major changes to config files of a server that wasn't theirs and didn't back them up and then said OOPS! Our bad. Sorry. Remember, I am paying these people for a service.
Posted by Steven, 12-16-2009, 01:14 AM
Reinstalling spam assassin wouldn't overwrite the spam assassin config files. Reinstalling exim would likely overwrite the files, however it depends on how it was modified to begin with. If it's a cpanel box for example there are specific ways it needs to be modified or you run the risk of cpanel wiping it out anyway. Side note: If it's cpanel you can configure cpbackup to backup config files which would include the exim.conf
Posted by Steven, 12-16-2009, 01:16 AM
Further more, I wouldn't just blindly reinstall spamassassin and exim to resolve the issue. That is lazy.
Posted by hD-Scott, 12-16-2009, 01:18 AM
It's directadmin and so directadmin doesn't override the config file. When they reinstalled spam assassin they wiped the directory and started from scratch which removed the config files. And yes, when you reinstall exim on directadmin, it puts back the default exim config file and a very old exim.pl file. The exim.conif file is from 2007 and is version 2, where we're halfway through version 3 for exim.conf files for DA now put out by nobaloney. I keep the exim.conf file up to date as new ones come out and I back up the old one, but there are dozens of places where you have to edit the file to fit your server and that takes a while to read the comments and make the appropriate changes and then to recreate your blacklists, whitelists, etc.
Posted by Steven, 12-16-2009, 01:20 AM
Sigh.. You don't have to wipe the configs to reinstall spamassassin, that served no purpose. Wonder why they did that. I am well aware of the exim config issue with directadmin, and know you have to backup the config files (however you build exim from an rpm, you can stick your config files in it so when you reinstall it, it shoves the right config files in)
Posted by hD-Scott, 12-16-2009, 01:21 AM
I wouldn't have either. The issue was a month ago, Spamassassin was working just fine. It tagged spam with the "*****SPAM*******, you could see it in the exim mainlog that it was filtering, and you could see it in the email headers that it was being filtered. However, since this server company took over, they have made many changes behind the scenes that we were not made aware of and whatever they did, has caused spamassassin to no longer filter the email. It is running as spamd and it is configured correctly in exim ( I did it again myself yesterday before they destroyed it), but yet no mail is getting filtered and the clients are receiving upwards of 200 spams per day and some email has gone missing and nowhere to be found. Their solution, after insisting that it was working (yes it was running, but not working), was to reinstall everything hoping that would fix the issue.
Posted by Steven, 12-16-2009, 01:24 AM
Just a note, you mention exim mainlog. Have you checked /var/log/maillog, spam assassin logs there.
Posted by hD-Scott, 12-16-2009, 01:26 AM
Probably for the same reason the uninstalled a perfectly running CSF firewall with all my tweaks and configs done to replace with the one they preferred to work with APF. They did this without telling me and when a client said they were blocked by the firewall, logged in trying to unblock them only to find out that CSF was gone along with all my config files. They thought that another issue was caused by the firewall, so instead of troubleshooting the firewall, they wiped it and put their own in, not even taking head of the ports that were open (so that whmcs could communicate with cpanel and my other DA servers) or the whitelisted IP addresses of the other servers so that backups could be run, etc, etc, etc. I've only used them a couple of months and this is my last month. I'm actively seeking out a new server admin. It's not like our needs are heavy. We submit one or two tickets per month (and ran this server ourselves for 6 months without issue before seeking assistance). And the only reason we have been requesting more support for this server is because when they fix one issue, they break another.
Posted by hD-Scott, 12-16-2009, 01:26 AM
Yes, we have. It's hasn't been filtering for about 3 weeks.
Posted by Steven, 12-16-2009, 01:27 AM
Just a test to run (assuming you already have, but throwing it out there just in case): spamassassin --lint come's back clean?
Posted by hD-Scott, 12-16-2009, 01:32 AM
Yes, it comes back clean. And I do know that the process logs in mailog, but we have exim setup (or we had it set up) so that it would also log in the mainlog spamassassin so you would see if an email had been rejected by spamassassin or that it was moved to the spam folder. But now nothing. Again, the process is running, but it's just not filtering the mail. Once we find a new server admin, we will then ask them to fix what the previous company broke.
Posted by hD-Scott, 12-16-2009, 01:35 AM
The problem is, not knowing all of the changes they have made, makes it a pain to troubleshoot where the issue arose. If you caused the issue, you at least know what changes you've made, what installs you've done, so you can backtrack, but here, as they don't give me any info, even when I ask, it's going to be a pain, and probably very costly to fix. Anyway, off to bed to worry about this tomorrow.
Posted by plumsauce, 12-16-2009, 02:25 AM
Not being able to back out of changes on a production server is inexcusable. Full stop. An admin working alone on his own server might not keep a work log, but in every other situation a work log ought to be kept. No change that has not been explicitly discussed should ever have been made. And, once they were made should have been reported. When working on client accounts, I discuss any changes, with full reasoning before making changes. I will only make changes that have been *explicitly* approved by the account holder. Then, after the changes, the actual changes are reported *again* to the account holder. All of this is to ensure that everyone is on the same page.
Posted by Cape Dave, 12-16-2009, 03:44 AM
They did this without telling me and when a client said they were blocked by the firewall, logged in trying to unblock them only to find out that CSF was gone along with all my config files." That alone would make me totally crazy!
Posted by oliviakitty, 12-16-2009, 07:34 AM
Wow, can any other server management companies weigh in? It's interesting to see who will and won't take good care of your server.
Posted by hD-Scott, 12-16-2009, 08:39 AM
That's why I created this thread. I want to know what the standard is and who will work with me and not against me when dealing with my server. Since I hired these guys, I've had more issues with my server than I did when I self-managed and it hurts my reputation as providing a great, reliable, hosting experience when the server is being inundated with spam, good email gets blocked before it gets through the server, so the clients don't even know they're missing it, and when other things happen, like squirrelmail getting all messed up since they took over. As I've said, I'm no linux expert, but I'm not a newbie. I can fix a lot of things and I was able to keep the server running quite well by myself without issue. I needed a php issue fixed that was beyond my experience so I hired this company and they did fix that issue, but in the end, I've had a lot more issues since then.

Add to Favourites Print this Article

Knowledgebase

isn't backing up standard prior to making a major config change?

Our Services

Client Menu

Legal