Shopping for a back-up program

In my latest blogs I have been describing how I have been exploring a way to back up my over 50,000 files to an off-site service so they will be secure in case my laptop and my local external backups are destroyed.  I explained why I chose Online Storage Solution (OLS-CS) to use for this off-site storage of my backed-up files.  I also explained why I needed a back-up program to handle the backups.  Copying files to the OLS-CS servers wasn't a satisfactory solution because later changes, additions, and deletions needed to be tracked and managed.  For that, you need a back-up program.

I had tried FoldersSynchronizer by softoBe and Personal Backup by Intego, but was having problems because when the connection was lost between my computer and the Network Drive on the OLS-CS servers, the backup just stopped, and I had to start the backup from the beginning over and over.

So I went out on the internet to see if there were any back-up programs that advertised their ability to maintain a connection or resume the backup after a connection is restored.

There are A LOT of backup programs out there.  Here is a list of a few that I looked at and a brief comment on each.

SmartBackup - $15 - Advertises itself as a fast and lightweight backup application for OSX, ready for Mountain Lion.  It appeared to be too simple for what I needed.  The program assumes it knows what you want when you are setting up the back-up parameters, leaving fewer things that you can change.

File Synchronization - starts at $15 - Appears designed to do synchronization only, not backups.

xTwin - $99.99 - Though expensive, this looks like a very good program.  It has gotten a couple really bad reviews, but those were for earlier versions.

Carbon Copy Cloner - $30 - Good reputation.

IBackup - Monthly fee based on amount of storage - This is a total on-line back-up service which includes storage and sharing of files. Not what I was looking for.

Mathusalem by softonic - No longer supported

SilverKeeper by LaCie - No longer supported

Of all the programs I looked at, the name I was most familiar with was Carbon Copy Cloner.  It has received glowing reviews for the past several years.  Being a contrarian, I had resisted getting it, but I decided that it was time to see if it lived up to the hype.  There is a 30-day free trial period.  After that it costs $39.95.



I thought I would start with backing up my Desktop folder.  There are only about 5500 files in that folder, and I thought it would be a good test of Carbon Copy Cloner (CCC).

This is a very comprehensive program with lots of options.  During the trial period, a window opens when you launch the program that reminds you of when the trial expires and gives you the option of continuing to use the trial or purchasing the program.

The main window allows you to select the source and the destination.  You can either choose to copy an entire disk to the destination or a particular folder.  The destination can either be a disk or a folder on a disk.  Since I was copying the Desktop folder, which resides inside the "bgmason" folder, which resides inside "Users" folder on my hard drive, I have the same folder structure created on the Network Drive.  I created a "Users" folder on the Network Drive, and then inside that I created a "bgmason" folder.  At this point, because of the previous work I had done making a backup of my files (see previous blogs), I had a "Desktop" folder and a "Documents" folder inside the "bgmason" folder on the Network Drive.

Once you make those selections, you have the ability to deselect files and folders that you don't want copied.  Even though they are small files, it doesn't make a lot of sense to copy all of the .DS_Store files, for example.  By deselecting them, there are that many fewer files to copy and it will somewhat reduce the time involved in making the backup.

On the destination side, you let CCC know what kind of media your backup will be made to.  You also are given options about whether or not to delete files on the destination that don't exist on the source or to save those files in an archive.

Finally, there is a button for further customizing the settings concerning data that already exists on the destination.  This includes a button called "Advanced settings..."  This is where I ran into trouble.  One of the selections is for removing files "on the destination that I have excluded from the backup".  It goes on to explain, "If an item exists on the destination (e.g. from a previous backup), and you have deselected that item from the list of items to be copied, that item will be removed from the destination."  Since I was deselecting some files, including the .DS_Store files, I thought I should select this.  That way, even though they currently existed on the destination because of my previous efforts to copy my files using FTP and the Finder, they would be removed.

All of these settings are remembered by the program so that the next time you select the same source and destination, you will not have to remember your previous settings.

So I started CCC off on its project to copy all the files and folders in my Desktop folder to the Network Drive.  This was taking some time, as I expected it would, and I proceeded to do other things on my laptop as the backup was proceeding.  But I glanced up at one point, and noticed that all of the files and folders in my Documents folder on the Network Drive were disappearing!  All that copying it had taken me days to accomplish using FTP and the Finder was being flushed down the drain!  I stopped CCC and assessed the damage.  I still had all the folders inside the Documents folder whose names started with A-C.  But the folders with names starting with D-Z were gone from the Network Drive.

In analyzing why this happened, I finally realized that when I said to remove files on the destination that I excluded from the backup, that this did not refer to the files and folders inside the "Desktop" folder, but rather the files and folders inside the "bgmason" folder.  Since I hadn't told CCC to copy the "Documents" folder, it commenced deleting it.

Needless to say, I was very discouraged at this point at decided to quit for the day. I would reconsider my approach, and then tackle the problem after I'd collected my thoughts.

Continuing saga of trying to back up files off-site

If you have read my previous blogs you know that I have over 50,000 files totaling over 30 GB that I want to back up to an off-site location. Because of the large number of files, I decided that most of the back-up services offered out there would not suit my situation. Those services are good for handling collaborative projects or for backing up particular "important" files and folders. But they are not set up to work with large numbers of files. I learned that what I needed was a "file storage" service.

After settling on Online Storage Solution (OLS-CS) to be my file storage service, my next task was to get my files copied to their servers.

I found that I could use the Finder and drag and drop files and folders that are located on my computer's hard drive to the Network Drive opened when connecting to OLS-CS. Or I could use an FTP client software to copy the files. This took a long time to move all the files in my Documents folder. Working on and off, it took about 5 days.

But once I had done that I realized that whenever I made a change to a file, or added or deleted a file, I would have to remember what I had done so I could copy those files to the OLS-CS servers. That would require a lot of effort and would be prone to errors. I needed a back-up program that would identify those incremental changes to my files and folders and copy the changed items to the Network Drive.

If you read my blog of July 23, you know that I had originally tried to use the back-up programs I own to get the files copied over, and that I ended up using my FTP software, Fetch, and the Finder instead. But I now knew that I had to be able to use a back-up program. So I tried once again.

The backup program that I have been using the longest is FoldersSynchronizer by softoBe. It has been around a very long time and is very easy to use. When I tried to use it, however, I found that the connection to the Network Drive gets dropped every once in awhile. It may appear to be connected when you look at the Network Drive in the Finder. The Finder wouldn't change appearances if the drop was only momentary. FoldersSynchronizer, however, stops the backup. It apparently has no way of resuming the upload once the connection is broken. In my research on the Internet, apparently the technology exists for resuming downloads, but it is much more difficult to resume an upload.

Another thing I began to realize as I started trying to use a back-up program is that in order for the program to do an incremental backup, it has to keep track of what existed before it begins doing the backup. So the total time it will take to do the incremental backup depends on how long it takes to scan the files to see which ones were changed, which ones are new, and which ones are no longer there.

When I saw that my old stand-by, FoldersSynchronizer, wasn't going to work, I decided to try a program I bought more recently, Intego's Personal Backup. I tried using it for several days. It seemed to make progress, but then it would quit. Each time it scanned my files to build a database of the files it would need to change. That process took three hours. Then it would start copying files to the Network Drive until the connection was broken, at which time it would simply stop copying. There is no error message. You just have to look at the information being provided and notice that nothing is changing. Each time you restart, there are fewer total files to copy because in the meantime the vast majority of them have not changed. Gradually you will get it down to the point where everything has been copied, and the only thing that will need to be updated are the files that you actually added, modified, or deleted during the period since the last backup.

Since I had already copied all my files to the Network Drive using FTP and the Finder, I couldn't figure out why FoldersSynchronizer and Personal Backup were finding that the files had changed and they had to copy the files all over again despite my previous effort. I eventually noticed that the creation, modification, and last modified dates were all being changed! When these programs copied the files to the Network Drive, the old dates were not being retained! Instead all the files were being given the date they were copied to the Network Drive.

I fired off a quick email to Support for FoldersSynchronizer, but never received a reply. My research on the internet tells me that this is a common problem and it is pretty much a mystery as to why this happens.

In my next blog I decide to go shopping for a backup program that won't fail when the connection to the Network Drive breaks. Stay tuned.

My Documents Are Backed up Off-site

Backing up files

After setting up the off-site backup service I chose (see previous blog), and testing it a bit, I decided it was time to get serious.  If this solution was going to work for me, it had to at least handle all the files I had in my Documents folder.


When I did a "Get Info" (Command-I) on my Documents folder, it said that I had a little over 30 GB in the folder.


FoldersSynchronizer

My favorite program for backing up files, outside of the scheduled Time Machine backup, is FoldersSynchronizer by softoBe.  I have used this program for years whenever I wanted to back up an entire folder of files to an external location.  I never have used it for trying to synchronize files.  


As indicated in the previous blog, there are two major ways of getting to your files on the off-site location.  One is to access them through an FTP program.  The other is to get to them by opening the Network Drive using the Finder.


So I started by trying to use FolderSynchronizer to copy all the files in my Documents folder to a Documents folder which I created in my area on  Online Storage Solution (OLS-CS).  I went to the "Go" menu of the Finder, then "Connect to Server . . .", then entered the server address.  Once I was connected, the network server appears as a disk drive on the desktop and as a shared device in the left panel of a Finder window.  From this point you can treat the device just as any other hard drive connected to your computer, creating folders, dragging files from one folder to another, etc.


With FoldersSynchronizer, you drag the source folder to the top of the Copy window and the destination folder to the bottom of the Copy window.  When you click the "Copy" button, the program starts inventorying the files it will have to copy.  Then it starts copying the files from the source to the destination.  I got an error message almost immediately which said that I didn't have permission to access the folder on the Network drive.  I have no idea why that error was received.  That is a mystery that can wait for another day.  Instead, I tried using Personal Backup, part of the Intego suite of products, which I also own.

This worked, after a fashion.  By the time it had taken an inventory of what it was going to copy, it was predicting it would take over 3 days to finish.  I let it run for an hour.  In that time it had copied about 500 MB.  Since I had been able to copy 2 GB in a half hour the day before, I decided to quit.


Doing things manually

I decided to do things manually using FTP, or simply dragging and dropping using the Finder.  If you have a lot to do, FTP is better because if you use drag and drop and encounter an error, everything just stops and you have to figure out what was copied and what wasn't and start over from there. FTP does some error handling. 


Because I had stopped the backup that had started, I had to determine how far the backup had gotten.  Then I started dragging and dropping groups of files from that point using the Finder.  I wanted to get to a point where I could drag and drop whole folders using FTP.


I probably would have been better off just letting the backup run using Personal Backup.  It ended up taking me 6 days to complete the backup.  It actually might have been better that I did end up doing it more or less manually because some things happened along the way.


In the first place, the OLS-CS server was down for a few hours two days in a row.  The first time (on 7/18) it was down for about 3 hours.  The second time (on 7/19), it was down for about 2 hours.  After the two hours, it appeared to be up.  I could get to my files.  However, I couldn't copy files to my folders.  Later in the evening when I tried, I had full functionality back.  Hopefully this service won't experience too many outages like that.  I tried submitting a help ticket, but this time I never got a reply.


Secondly, I ran across my Virtual Windows XP machine in my Documents folder.  That one file is over 8 GB.  It is the file that is used by a program called "Parallels" to run Microsoft Windows on my Mac.  My FTP program,  Fetch, was copying it as a text file. (I had Fetch set up to automatically select the correct format--text, binary, etc., when copying files.)  Fetch was uploading the 8GB file at 34 KB/s.  It was reporting that it would take 59 hours.  Well, none of that made any sense, so I just decided to be satisfied with on-site backups for that file.  I stopped the copying of the file and deleted it off the Network drive.


One thing I observed was that the time it takes to do a backup will depend on the upload speed you are able to get out of your connection.  When I started working on Friday, the upload speed started out at between 122 & 124 KB/s.  At one point on Saturday, I was only getting 73 KB/s.  In the evening Saturday, I saw 265 KB/s.  That didn't last long.  It was soon down to the 160's range.  The speed will change while the backup is in progress.

Job completed (for now)


I finished backing up the Documents folder on Saturday.  I think I still need to back up my photos and music off-site.  But that may not be true anymore now that Apple has made iCloud available.  But I may wait a couple weeks before I attempt to figure that out.  I want to continue testing this new service I've contracted with for a bit longer before I take these additional steps.