If you have read my previous blogs you know that I have over 50,000 files totaling over 30 GB that I want to back up to an off-site location. Because of the large number of files, I decided that most of the back-up services offered out there would not suit my situation. Those services are good for handling collaborative projects or for backing up particular "important" files and folders. But they are not set up to work with large numbers of files. I learned that what I needed was a "file storage" service.
After settling on Online Storage Solution (OLS-CS) to be my file storage service, my next task was to get my files copied to their servers.
I found that I could use the Finder and drag and drop files and folders that are located on my computer's hard drive to the Network Drive opened when connecting to OLS-CS. Or I could use an FTP client software to copy the files. This took a long time to move all the files in my Documents folder. Working on and off, it took about 5 days.
But once I had done that I realized that whenever I made a change to a file, or added or deleted a file, I would have to remember what I had done so I could copy those files to the OLS-CS servers. That would require a lot of effort and would be prone to errors. I needed a back-up program that would identify those incremental changes to my files and folders and copy the changed items to the Network Drive.
If you read my blog of July 23, you know that I had originally tried to use the back-up programs I own to get the files copied over, and that I ended up using my FTP software, Fetch, and the Finder instead. But I now knew that I had to be able to use a back-up program. So I tried once again.
The backup program that I have been using the longest is FoldersSynchronizer by softoBe. It has been around a very long time and is very easy to use. When I tried to use it, however, I found that the connection to the Network Drive gets dropped every once in awhile. It may appear to be connected when you look at the Network Drive in the Finder. The Finder wouldn't change appearances if the drop was only momentary. FoldersSynchronizer, however, stops the backup. It apparently has no way of resuming the upload once the connection is broken. In my research on the Internet, apparently the technology exists for resuming downloads, but it is much more difficult to resume an upload.
Another thing I began to realize as I started trying to use a back-up program is that in order for the program to do an incremental backup, it has to keep track of what existed before it begins doing the backup. So the total time it will take to do the incremental backup depends on how long it takes to scan the files to see which ones were changed, which ones are new, and which ones are no longer there.
When I saw that my old stand-by, FoldersSynchronizer, wasn't going to work, I decided to try a program I bought more recently, Intego's Personal Backup. I tried using it for several days. It seemed to make progress, but then it would quit. Each time it scanned my files to build a database of the files it would need to change. That process took three hours. Then it would start copying files to the Network Drive until the connection was broken, at which time it would simply stop copying. There is no error message. You just have to look at the information being provided and notice that nothing is changing. Each time you restart, there are fewer total files to copy because in the meantime the vast majority of them have not changed. Gradually you will get it down to the point where everything has been copied, and the only thing that will need to be updated are the files that you actually added, modified, or deleted during the period since the last backup.
Since I had already copied all my files to the Network Drive using FTP and the Finder, I couldn't figure out why FoldersSynchronizer and Personal Backup were finding that the files had changed and they had to copy the files all over again despite my previous effort. I eventually noticed that the creation, modification, and last modified dates were all being changed! When these programs copied the files to the Network Drive, the old dates were not being retained! Instead all the files were being given the date they were copied to the Network Drive.
I fired off a quick email to Support for FoldersSynchronizer, but never received a reply. My research on the internet tells me that this is a common problem and it is pretty much a mystery as to why this happens.
In my next blog I decide to go shopping for a backup program that won't fail when the connection to the Network Drive breaks. Stay tuned.
Backing up files
After setting up the off-site backup service I chose (see previous blog), and testing it a bit, I decided it was time to get serious. If this solution was going to work for me, it had to at least handle all the files I had in my Documents folder.
When I did a "Get Info" (Command-I) on my Documents folder, it said that I had a little over 30 GB in the folder.
FoldersSynchronizer
My favorite program for backing up files, outside of the scheduled Time Machine backup, is FoldersSynchronizer by softoBe. I have used this program for years whenever I wanted to back up an entire folder of files to an external location. I never have used it for trying to synchronize files.
As indicated in the previous blog, there are two major ways of getting to your files on the off-site location. One is to access them through an FTP program. The other is to get to them by opening the Network Drive using the Finder.
So I started by trying to use FolderSynchronizer to copy all the files in my Documents folder to a Documents folder which I created in my area on Online Storage Solution (OLS-CS). I went to the "Go" menu of the Finder, then "Connect to Server . . .", then entered the server address. Once I was connected, the network server appears as a disk drive on the desktop and as a shared device in the left panel of a Finder window. From this point you can treat the device just as any other hard drive connected to your computer, creating folders, dragging files from one folder to another, etc.
With FoldersSynchronizer, you drag the source folder to the top of the Copy window and the destination folder to the bottom of the Copy window. When you click the "Copy" button, the program starts inventorying the files it will have to copy. Then it starts copying the files from the source to the destination. I got an error message almost immediately which said that I didn't have permission to access the folder on the Network drive. I have no idea why that error was received. That is a mystery that can wait for another day. Instead, I tried using Personal Backup, part of the Intego suite of products, which I also own.
This worked, after a fashion. By the time it had taken an inventory of what it was going to copy, it was predicting it would take over 3 days to finish. I let it run for an hour. In that time it had copied about 500 MB. Since I had been able to copy 2 GB in a half hour the day before, I decided to quit.
Doing things manually
I decided to do things manually using FTP, or simply dragging and dropping using the Finder. If you have a lot to do, FTP is better because if you use drag and drop and encounter an error, everything just stops and you have to figure out what was copied and what wasn't and start over from there. FTP does some error handling.
Because I had stopped the backup that had started, I had to determine how far the backup had gotten. Then I started dragging and dropping groups of files from that point using the Finder. I wanted to get to a point where I could drag and drop whole folders using FTP.
I probably would have been better off just letting the backup run using Personal Backup. It ended up taking me 6 days to complete the backup. It actually might have been better that I did end up doing it more or less manually because some things happened along the way.
In the first place, the OLS-CS server was down for a few hours two days in a row. The first time (on 7/18) it was down for about 3 hours. The second time (on 7/19), it was down for about 2 hours. After the two hours, it appeared to be up. I could get to my files. However, I couldn't copy files to my folders. Later in the evening when I tried, I had full functionality back. Hopefully this service won't experience too many outages like that. I tried submitting a help ticket, but this time I never got a reply.
Secondly, I ran across my Virtual Windows XP machine in my Documents folder. That one file is over 8 GB. It is the file that is used by a program called "Parallels" to run Microsoft Windows on my Mac. My FTP program, Fetch, was copying it as a text file. (I had Fetch set up to automatically select the correct format--text, binary, etc., when copying files.) Fetch was uploading the 8GB file at 34 KB/s. It was reporting that it would take 59 hours. Well, none of that made any sense, so I just decided to be satisfied with on-site backups for that file. I stopped the copying of the file and deleted it off the Network drive.
One thing I observed was that the time it takes to do a backup will depend on the upload speed you are able to get out of your connection. When I started working on Friday, the upload speed started out at between 122 & 124 KB/s. At one point on Saturday, I was only getting 73 KB/s. In the evening Saturday, I saw 265 KB/s. That didn't last long. It was soon down to the 160's range. The speed will change while the backup is in progress.
Job completed (for now)
I finished backing up the Documents folder on Saturday. I think I still need to back up my photos and music off-site. But that may not be true anymore now that Apple has made iCloud available. But I may wait a couple weeks before I attempt to figure that out. I want to continue testing this new service I've contracted with for a bit longer before I take these additional steps.
