I have been devoting much of this blog to my efforts to backup the files on my laptop to "the cloud". This is in addition to the Time Machine backup that I have on an external drive connected to my laptop. I have two reasons for wanting to do this. One is to provide a backup which is off-site so that if something disastrous happens to my laptop and/or my home, I will still have all of my files. The second is so I can get to my files using my iPad no matter where I am.
As I have noted previously, many companies provide partial solutions, especially those who advertise the ability to collaborate with others. They allow you limited storage of your "important" files. I wanted a complete backup solution.
The solution I had come up with was to use Online Storage Solution (OLS-CS) for my offsite file storage. The back-up program I chose was xTwin because they provided a good, stable, and secure uploading capability. The files created by xTwin are in a special format which required xTwin or another program to reconstitute them as regular files. This meant that in order to get to my files from my iPad, I would need to use GoToMyPC. That required keeping my laptop open and running at all times, but it did allow me to get to my files using my iPad. I therefore had a complete solution.
The solution was not the solution
However things turned out not to work as I hoped. When I went to the OLS-CS storage service and tried to recover a file, it turned out I could not recover it. To make a long story short, after many attempts to get information from the OLS-CS storage service, they could not provide a satisfactory solution or an answer as to why I could not recover the files that I had uploaded. In fact, at one point, all of my back-up files disappeared! So I had to abandon them and try to find another solution for my off-site storage. OLS-CS did provide me a refund of the $39 I had given them for a year's worth of service. Fortunately I finally have a good solution.
I went back to the spreadsheet I put together when I was shopping for a backup program in August. I also did some more research to see if there were other programs out there that I missed.
Based on the information I was looking at, I found five backup solutions that met my requirements. Some services only allow backups of documents and photos, for example, and do not allow applications to be backed up. These five services all said that you are not restricted as to the types of files you can store. They all provide for file sharing, automatic backups, and access to the files using mobile devices. They all provide web access to your files and access using a desktop application. All five of the services provide a free trial period except for OpenDrive.
OpenDrive provides 5 GB of free storage, and up to 100TB of paid storage. They have a daily upload limit of 5 GB. You can tailor your plan based on your needs. They provide a tool on their website which you can use to calculate what your anticipated costs will be based on how much storage you think you will need.
LiveDrive provides up 5 TB of paid storage. The cost is $79.95 per year for limited file types. If you want to store other file types, the cost is an additional $96 per year.
MyPCBackup provides unlimited storage. The cost is $95.40 per year for limited file types. If you want to store other file types, the cost is an additional $100 per year.
ZipCloud and JustCloud both provide unlimited storage. The cost is $83.40 per year for limited file types. If you want to store other file types, the cost is an additional $100 per year.
LiveDrive
I decided to go with LiveDrive because they were slightly less expensive than the other services and I didn't think I would need more than 5 TB of storage since my laptop has a 1 TB hard drive. They only charged me the $79.95 per year fee. I have not been charged extra for backing up unusual file types. Apparently the information I got while doing my research was incorrect. They will charge you an extra $17.95 per year for backing up additional computers.
Installing LiveDrive and getting it up and running was very simple. You just go to their website, http://www.livedrive.com. There are four packages to choose from, "Backup", "Briefcase", "Pro Suite", and "Business". For my purposes, I chose "Backup". The other solutions are for businesses and people with more requirements than I have. You fill out a web form to sign up and create an account. You then have 14 days to give it a trial run.
You will want to download the application which runs on your computer (both Windows and Mac OS are supported). Once you are up and running, you also have the ability to work from within your web browser. I also downloaded the app for my iPad.
When you run the application on your computer, you will see the LiveDrive icon in your system menu bar. One of the choices on the drop down menu is "Manage Backups". Here you see in the left pane of the window that opens all of the folders on your computer. By clicking on the arrows to the left, you will see the subfolders contained within the folder. You simply check the folders you want backed up, and LiveDrive immediately begins backing up the folders and files within that folder to your account on their servers.
For my trial run, I checked just the "Desktop" folder. It contains about 5500 items. The backup went very smoothly with no problems. After it was done, I was able to see all the files using the web browser after logging in to my account on the LiveDrive site. I was also able to see all the files using the LiveDrive application on my iPad.
I can not tell you how thrilled and excited I was. After all the struggles I had gone through to find a solution for backing up my files off-site (see previous blogs), I finally found a solution that worked and with absolutely no hassle.
The next test was to see if I was going to have any problems backing up my Documents folder, which contains about 55,000 files. Backing up that folder took a bit longer. And I did notice that LiveDrive seemed to have a problem with one of my files. It kept trying over and over to backup the file, but didn't appear to be succeeding. I could see this when I looked at the status and clicked on "Detailed Status".
It turns out this was an old Apple IIgs spreadsheet file that had a forward slash in its name. LiveDrive converted the forward slash to a colon, but it was still having problems. So I went into that folder of spreadsheets and manually changed all the names, converting the forward slashes to colons. After I did that, the backup proceeded with no further problems. It was so satisfying to see "Your Backup is up-to-date" in the status window.
It is so cool to be able to go to my iPad and get to all of the documents on my laptop. Of course, as far as I can tell the files can't be opened on the iPad. You wouldn't want to go changing a backup file. Trying to then keep the files in sync would be a whole different ballgame.
This effort to find a solution for backing up my hard drive in the Cloud has proven to be much more difficult than I even anticipated. At least it gives me material for my blog.
In this installment of my quest, I try to use Carbon Copy Cloner to back up my files to the Network Drive in the Cloud, but run into more problems. So I go looking for a different back-up program.
In my last blog I described how Carbon Copy Cloner (CCC) had deleted most of the backup I had made by using the Finder and Fetch to copy the files from my hard drive to the off-site file storage service I had selected ( Online Storage Solution (OLS-CS) ). Nevertheless, I knew that the fault was mine for checking off an option in the program that caused CCC to do what it did. So I decided after a day of gathering my composure to continue on with Carbon Copy Cloner. I figured I had learned my lesson and hopefully would not make a mistake like that again.
So I restarted the backup of the Desktop folder using CCC. First CCC made a tally of changes that had been made since the last time it did a backup of the folder. The scanning of the 5500 files took less than a minute. Then it proceeded to backup the files. It finished the backup of the Desktop folder in three hours and forty minutes, but it reported a lot of errors. I guessed that this was due to connection problems with the Network Drive.
So I immediately reran the backup. This time the backup took over five hours (I’m not sure exactly how long because I was in bed asleep when it finished.) I assume that the time involved largely has to do with the connection speed I was able to get over the course of the backup. Again there were a lot of errors, but there were about half as many as the time before. This was promising. That told me that once the file was copied successfully, the next time I ran the backup, that file wasn’t touched. Where there was an error during copying however, the files were replaced on the next backup with a good copy of the file.
So I reran the backup again. This time it took three hours, and there were only 19 errors. Again, I reran it. Again it took three hours, and this time there were only 8 errors.
The next time I reran it, I came back to it after five hours. In that time it had only copied 243 MB, maybe ⅛ of the files it indicated it was going to copy during this run. In other words, it had stalled. There were no indications of errors or problems. I guess the connection to the Network Drive had been dropped, and CCC had been sitting there spinning its wheels ever since.
So one more time, I restarted the CCC backup. It ran overnight and finished at 10 in the morning. It had copied about 2 GB and finished with two errors. I guess I can live with that.
But now I got back to thinking about my Documents folder, with the 55,000 files totaling about 30GB. If I had had that much of a struggle to get 5,500 files copied from my Desktop folder, what was it going to be like to get ten times as many files copied?
What to try next?
I went back to the research I had done on the back-up programs that are out there.
The one program that promised to work with a Network Drive was xTwin. Even though it costs USD$99, it does have a seven-day trial. So I thought I’d give it a shot.
Defining the source, the files and folders to be copied, was not that much different than any of the other backup programs. But when I went to define the destination, there was immediately a big difference compared to any of the other programs. Once I let the program know that I wanted to back up to a Network Drive, there were all kinds of questions I needed to answer about how the connection was to be made. I had to choose between seven different protocols (AFP, SMB, HTTP, HTTPS, FTP, FTPS, or SFTP), and enter the required login data.
Connecting with OLS-CS, you use the HTTPS (Secure WebDAV) protocol.
Once I had let the program know what I wanted to do, I clicked on the “Start backup” icon. It reported that it would be backing up 33,279 files or 29.43 GB. It’s estimated time to complete the task: 22 - 24 hours. The console window shows your progress as it completes its task, updating the estimated time to complete as it goes. It also gives the speed it is able to accomplish as it copies the files. I was seeing anywhere from 50 to 425 kiloBytes per second.
The Downside
When I looked at the Finder window to see the progress that was being made copying the files to the Network Drive (and to make sure the files were being copied to the right place this time and that no files were being deleted), I came to fully understand how xTwin works. To quote from their FAQ page:
The backup data stored by xTwin is compatible with open standard technologies (CPIO or CPGZ archives, split files, Open SSL encryption…), and does not use proprietary formats, so you can still access your backup data without xTwin if you ever needed to.
In another part of the FAQ they say:
Keep in mind that xTwin is a backup solution, not a file-sharing solution. However, it is very easy to access the contents of a backup on another computer than the one it was generated from, as long as you have a copy of xTwin also installed on this computer.
What xTwin does is break up the files on your hard drive into chunks. You can set the size of the chunks, or let xTwin set the size. If a file is smaller than the chunk size selected, the file may be combined with other files to create a file of the set chunk size. Then those chunks are sent over the internet to the Network Drive. You can even set up xTwin to send more than one chunk at a time to speed up copy speed.
You then need a program (like xTwin) if you ever want to retrieve one of your files. Or, if you know how, you can use the Terminal to open your file.
So, what I learned that this indeed was the most reliable and efficient way to copy my files to my off-site storage. However, once they were copied (and xTwin can do incremental backups), I could not get to them on my iDevices. I could only get to them by running a program like xTwin that would be able to restore them to their original form.
So to summarize:
Using the Finder and FTP is slow and cumbersome and doesn’t lend itself to incremental backups.
Using FoldersSynchronizer, Personal Backup, and Carbon Copy Cloner, while each has pluses and minuses, is slow and cumbersome and has lots of problems working with an off-site Network Drive because of dropped connections.
Using xTwin is very reliable, but doesn’t leave the files in a form that allows for access from my iDevices.
So now I’m at the point where I need to decide what way to go from here. There is still the option of copying everything to an external drive and putting it in a safe deposit box. Or I could just forego having an off-site backup altogether. That, however, is not the best solution, in my opinion. I still want off-site backup, and I still want to be able to get to my files from my iDevices. Stay tuned.
If you have read my previous blogs you know that I have over 50,000 files totaling over 30 GB that I want to back up to an off-site location. Because of the large number of files, I decided that most of the back-up services offered out there would not suit my situation. Those services are good for handling collaborative projects or for backing up particular "important" files and folders. But they are not set up to work with large numbers of files. I learned that what I needed was a "file storage" service.
After settling on Online Storage Solution (OLS-CS) to be my file storage service, my next task was to get my files copied to their servers.
I found that I could use the Finder and drag and drop files and folders that are located on my computer's hard drive to the Network Drive opened when connecting to OLS-CS. Or I could use an FTP client software to copy the files. This took a long time to move all the files in my Documents folder. Working on and off, it took about 5 days.
But once I had done that I realized that whenever I made a change to a file, or added or deleted a file, I would have to remember what I had done so I could copy those files to the OLS-CS servers. That would require a lot of effort and would be prone to errors. I needed a back-up program that would identify those incremental changes to my files and folders and copy the changed items to the Network Drive.
If you read my blog of July 23, you know that I had originally tried to use the back-up programs I own to get the files copied over, and that I ended up using my FTP software, Fetch, and the Finder instead. But I now knew that I had to be able to use a back-up program. So I tried once again.
The backup program that I have been using the longest is FoldersSynchronizer by softoBe. It has been around a very long time and is very easy to use. When I tried to use it, however, I found that the connection to the Network Drive gets dropped every once in awhile. It may appear to be connected when you look at the Network Drive in the Finder. The Finder wouldn't change appearances if the drop was only momentary. FoldersSynchronizer, however, stops the backup. It apparently has no way of resuming the upload once the connection is broken. In my research on the Internet, apparently the technology exists for resuming downloads, but it is much more difficult to resume an upload.
Another thing I began to realize as I started trying to use a back-up program is that in order for the program to do an incremental backup, it has to keep track of what existed before it begins doing the backup. So the total time it will take to do the incremental backup depends on how long it takes to scan the files to see which ones were changed, which ones are new, and which ones are no longer there.
When I saw that my old stand-by, FoldersSynchronizer, wasn't going to work, I decided to try a program I bought more recently, Intego's Personal Backup. I tried using it for several days. It seemed to make progress, but then it would quit. Each time it scanned my files to build a database of the files it would need to change. That process took three hours. Then it would start copying files to the Network Drive until the connection was broken, at which time it would simply stop copying. There is no error message. You just have to look at the information being provided and notice that nothing is changing. Each time you restart, there are fewer total files to copy because in the meantime the vast majority of them have not changed. Gradually you will get it down to the point where everything has been copied, and the only thing that will need to be updated are the files that you actually added, modified, or deleted during the period since the last backup.
Since I had already copied all my files to the Network Drive using FTP and the Finder, I couldn't figure out why FoldersSynchronizer and Personal Backup were finding that the files had changed and they had to copy the files all over again despite my previous effort. I eventually noticed that the creation, modification, and last modified dates were all being changed! When these programs copied the files to the Network Drive, the old dates were not being retained! Instead all the files were being given the date they were copied to the Network Drive.
I fired off a quick email to Support for FoldersSynchronizer, but never received a reply. My research on the internet tells me that this is a common problem and it is pretty much a mystery as to why this happens.
In my next blog I decide to go shopping for a backup program that won't fail when the connection to the Network Drive breaks. Stay tuned.
