Continuing saga of trying to back up files off-site

If you have read my previous blogs you know that I have over 50,000 files totaling over 30 GB that I want to back up to an off-site location. Because of the large number of files, I decided that most of the back-up services offered out there would not suit my situation. Those services are good for handling collaborative projects or for backing up particular "important" files and folders. But they are not set up to work with large numbers of files. I learned that what I needed was a "file storage" service.

After settling on Online Storage Solution (OLS-CS) to be my file storage service, my next task was to get my files copied to their servers.

I found that I could use the Finder and drag and drop files and folders that are located on my computer's hard drive to the Network Drive opened when connecting to OLS-CS. Or I could use an FTP client software to copy the files. This took a long time to move all the files in my Documents folder. Working on and off, it took about 5 days.

But once I had done that I realized that whenever I made a change to a file, or added or deleted a file, I would have to remember what I had done so I could copy those files to the OLS-CS servers. That would require a lot of effort and would be prone to errors. I needed a back-up program that would identify those incremental changes to my files and folders and copy the changed items to the Network Drive.

If you read my blog of July 23, you know that I had originally tried to use the back-up programs I own to get the files copied over, and that I ended up using my FTP software, Fetch, and the Finder instead. But I now knew that I had to be able to use a back-up program. So I tried once again.

The backup program that I have been using the longest is FoldersSynchronizer by softoBe. It has been around a very long time and is very easy to use. When I tried to use it, however, I found that the connection to the Network Drive gets dropped every once in awhile. It may appear to be connected when you look at the Network Drive in the Finder. The Finder wouldn't change appearances if the drop was only momentary. FoldersSynchronizer, however, stops the backup. It apparently has no way of resuming the upload once the connection is broken. In my research on the Internet, apparently the technology exists for resuming downloads, but it is much more difficult to resume an upload.

Another thing I began to realize as I started trying to use a back-up program is that in order for the program to do an incremental backup, it has to keep track of what existed before it begins doing the backup. So the total time it will take to do the incremental backup depends on how long it takes to scan the files to see which ones were changed, which ones are new, and which ones are no longer there.

When I saw that my old stand-by, FoldersSynchronizer, wasn't going to work, I decided to try a program I bought more recently, Intego's Personal Backup. I tried using it for several days. It seemed to make progress, but then it would quit. Each time it scanned my files to build a database of the files it would need to change. That process took three hours. Then it would start copying files to the Network Drive until the connection was broken, at which time it would simply stop copying. There is no error message. You just have to look at the information being provided and notice that nothing is changing. Each time you restart, there are fewer total files to copy because in the meantime the vast majority of them have not changed. Gradually you will get it down to the point where everything has been copied, and the only thing that will need to be updated are the files that you actually added, modified, or deleted during the period since the last backup.

Since I had already copied all my files to the Network Drive using FTP and the Finder, I couldn't figure out why FoldersSynchronizer and Personal Backup were finding that the files had changed and they had to copy the files all over again despite my previous effort. I eventually noticed that the creation, modification, and last modified dates were all being changed! When these programs copied the files to the Network Drive, the old dates were not being retained! Instead all the files were being given the date they were copied to the Network Drive.

I fired off a quick email to Support for FoldersSynchronizer, but never received a reply. My research on the internet tells me that this is a common problem and it is pretty much a mystery as to why this happens.

In my next blog I decide to go shopping for a backup program that won't fail when the connection to the Network Drive breaks. Stay tuned.

My Documents Are Backed up Off-site

Backing up files

After setting up the off-site backup service I chose (see previous blog), and testing it a bit, I decided it was time to get serious.  If this solution was going to work for me, it had to at least handle all the files I had in my Documents folder.


When I did a "Get Info" (Command-I) on my Documents folder, it said that I had a little over 30 GB in the folder.


FoldersSynchronizer

My favorite program for backing up files, outside of the scheduled Time Machine backup, is FoldersSynchronizer by softoBe.  I have used this program for years whenever I wanted to back up an entire folder of files to an external location.  I never have used it for trying to synchronize files.  


As indicated in the previous blog, there are two major ways of getting to your files on the off-site location.  One is to access them through an FTP program.  The other is to get to them by opening the Network Drive using the Finder.


So I started by trying to use FolderSynchronizer to copy all the files in my Documents folder to a Documents folder which I created in my area on  Online Storage Solution (OLS-CS).  I went to the "Go" menu of the Finder, then "Connect to Server . . .", then entered the server address.  Once I was connected, the network server appears as a disk drive on the desktop and as a shared device in the left panel of a Finder window.  From this point you can treat the device just as any other hard drive connected to your computer, creating folders, dragging files from one folder to another, etc.


With FoldersSynchronizer, you drag the source folder to the top of the Copy window and the destination folder to the bottom of the Copy window.  When you click the "Copy" button, the program starts inventorying the files it will have to copy.  Then it starts copying the files from the source to the destination.  I got an error message almost immediately which said that I didn't have permission to access the folder on the Network drive.  I have no idea why that error was received.  That is a mystery that can wait for another day.  Instead, I tried using Personal Backup, part of the Intego suite of products, which I also own.

This worked, after a fashion.  By the time it had taken an inventory of what it was going to copy, it was predicting it would take over 3 days to finish.  I let it run for an hour.  In that time it had copied about 500 MB.  Since I had been able to copy 2 GB in a half hour the day before, I decided to quit.


Doing things manually

I decided to do things manually using FTP, or simply dragging and dropping using the Finder.  If you have a lot to do, FTP is better because if you use drag and drop and encounter an error, everything just stops and you have to figure out what was copied and what wasn't and start over from there. FTP does some error handling. 


Because I had stopped the backup that had started, I had to determine how far the backup had gotten.  Then I started dragging and dropping groups of files from that point using the Finder.  I wanted to get to a point where I could drag and drop whole folders using FTP.


I probably would have been better off just letting the backup run using Personal Backup.  It ended up taking me 6 days to complete the backup.  It actually might have been better that I did end up doing it more or less manually because some things happened along the way.


In the first place, the OLS-CS server was down for a few hours two days in a row.  The first time (on 7/18) it was down for about 3 hours.  The second time (on 7/19), it was down for about 2 hours.  After the two hours, it appeared to be up.  I could get to my files.  However, I couldn't copy files to my folders.  Later in the evening when I tried, I had full functionality back.  Hopefully this service won't experience too many outages like that.  I tried submitting a help ticket, but this time I never got a reply.


Secondly, I ran across my Virtual Windows XP machine in my Documents folder.  That one file is over 8 GB.  It is the file that is used by a program called "Parallels" to run Microsoft Windows on my Mac.  My FTP program,  Fetch, was copying it as a text file. (I had Fetch set up to automatically select the correct format--text, binary, etc., when copying files.)  Fetch was uploading the 8GB file at 34 KB/s.  It was reporting that it would take 59 hours.  Well, none of that made any sense, so I just decided to be satisfied with on-site backups for that file.  I stopped the copying of the file and deleted it off the Network drive.


One thing I observed was that the time it takes to do a backup will depend on the upload speed you are able to get out of your connection.  When I started working on Friday, the upload speed started out at between 122 & 124 KB/s.  At one point on Saturday, I was only getting 73 KB/s.  In the evening Saturday, I saw 265 KB/s.  That didn't last long.  It was soon down to the 160's range.  The speed will change while the backup is in progress.

Job completed (for now)


I finished backing up the Documents folder on Saturday.  I think I still need to back up my photos and music off-site.  But that may not be true anymore now that Apple has made iCloud available.  But I may wait a couple weeks before I attempt to figure that out.  I want to continue testing this new service I've contracted with for a bit longer before I take these additional steps.

Off-site back up solution

I may have finally found the solution I've been looking for which will provide an off-site location for backups of my laptop.

Backup, backup, backup

We are always told to backup, backup, backup. The more backups the better.

I started using computers in the olden days of the 1980's. In those days I was rescued many times by backups when a floppy disk failed or a hard drive crashed for whatever reason. Though things don't fail as often as they used to, it instilled in me a kind of obsession to make sure everything is backed up.

Time Machine, of course, is an amazing solution because you just have to set it up, and then basically, you can forget it (with some caveats, of course.)

The problem with Time Machine is that it only backs up files. If your boot disk goes belly-up, you can't get it back by using Time Machine. You have to create a boot disk from an original installation disk or disk image, and then you can restore your files from Time Machine. But I'm pretty sure you will have lost all of your settings, and those will all have to be restored.

So about once a month I try to create a bootable backup on an external hard drive. This is basically a clone of my laptop's hard drive. If my hard drive were to fail, I will be able to boot from the external backup and restore everything to a new laptop hard drive, if that is what is called for.

NomaDesk

But we are also told to make sure we have a back up far from our house in case the house is destroyed and all the computers and disk drives with it. Since 2009 I have been using a service called NomaDesk. I got in the door early and the cost was very reasonable. And, though they emphasized the ability to share files with other people, I was able to use the service to back up my files. As the years passed, however, they seemed to become more restrictive on the types of files they would allow on their system. The first thing I noticed was they couldn't handle "packages", which is how Apple bundles programs. As time went on, I was calling their support people more and more for help. Things would work for awhile, but then I was back with the support folks.

I must say, their support was excellent--very responsive and willing to try to fix the problem.

After the last go-around, though, I finally gave up on NomaDesk. It just wasn't working for me. So I went shopping.

Comparison shopping

One of the best sites I found for seeing what is out there is onlinefilestorage.com . This site specializes in the field of on-line backup and storage and is an excellent place to learn about on-line backup and storage and to comparison shop. I spent a day on the site doing just that and came up with a spreadsheet which you might find helpful. You can download it from here: online backup.xls . NomaDesk is not one of those they review.

Backup vs. storage

The first thing I learned after all of that research was that I didn't want an on-line backup solution. What I really wanted was on-line file storage. Most of the reviews on the onlinefilestorage.com web site were of companies that provide on-line backup. They provide software that monitors your local hard drive, and whenever you make changes, it backs-up the new file, much as Time Machine does. Or the software lets you schedule back-ups at regular intervals. But all of the services have certain limitations. Like NomaDesk, most won't back up your programs or applications. Some won't back up your computer's settings. A few limit the size of the file you can back up.

Off-site file storage does not have all these bells and whistles. Automatic or scheduled backups and the ability to sync may be exactly what you want. But I'm more of a hands-on person when it comes to things like this. Off-site file storage is simply a way to store your files on someone else's server. You manage the files yourself including adding, moving, and deleting.

Online Storage Solution (OLS-CS)

It didn't take me long to find Online Storage Solution (OLS-CS). I am very excited about this service. In my testing this morning everything worked perfectly, just as I wanted it to or expected it to. Here is an image of their Dashboard:

 

There is no free trial. You have to sign up before you can use it or even test it. The cost is only $48 for 2 years for UNLIMITED storage! If you don't like it, you can get a full refund within 45 days. If you find out you don't like it after the 45 days have passed, they will give you a pro-rated refund.

 

There are no restrictions on file types or file size you copy to their service. You can use your own FTP program or backup program to copy the files. You can access their server as a network drive from the Finder. Once the files have been copied to the off-site location, you have complete control over them just as if they were on your own computer. You can set up sharing, you can define permissions. You can even play or "stream" your mp3 songs by double-clicking on them, just as if they were on your local hard drive. But only realize that there is no syncing going on here between your files on their server and your files on your computer. You have to take care of that yourself. Which is exactly the way I want things to work for my purposes.

You can use an application called AjaXplorer on your iOS or Android devices to access your files. The iOS version costs $0.99. This means that because you have access to all the files you copied from your computer to the OLS-CS servers from within AjaXplorer, that if you have an app on your mobile device that can open a particular kind of file, you can not only read it, but work on it as well. Then it is up to that application as to how you get it back on your computer's hard drive, whether you have to email it to yourself or use some other method provided by the application.

Besides file storage, OLS-CS provides web site hosting complete with unlimited domains, email, and database access. They have a few projects in beta, as you can see from the screenshot.

Results

So I used Fetch, the FTP program I've been using for years, to copy some of my files to OLS-CS today. Copying about 2 GB took about a half hour. The FTP copying got hung up when two files were found with backslashes ( / ) in the file name. Other times the backslash was automatically converted to a colon ( : ). That was a minor glitch.

When I tried to Go to the OLS-CS service and see my files by connecting to their servers as a Network Drive, I had to contact their support for help. The only method they provide is by filling out a web form. They then reply by email. The response was received within a half hour. It turns out that there was a typo in their instructions. The instructions say to enter https://dav/ols18.com/username. The correct entry shoul be https://dav.ols18.com/username, using a dot rather than a backslash.

It will take some time to copy everything to OLS-CS. But so far, everything is working and I think I have found a solution that will finally be what I've wanted all along.


 

Posted with Blogsy