These plans already have been foiled: Amazon Drive failure to launch

Having recently reformatted my desktop to document my setup in my dotfiles and hackintosh Git repositories, I needed to reinitiate my backup process. It became clear Backblaze was not going to work for me any longer due to the fragility of their architecture. After admittedly minimal research (I have pursued far too many tangents lately!), I am ditching Backblaze and trying Arq+Amazon Cloud Drive instead. Backblaze was at least kind enough to give a refund for my unused time.

Backblaze Inherit Backup State / Backblaze failed to make this computer inherit your backup state. / ERR_error_unknown

Since I have many terabytes of data I plan to backup, it remains to be seen if I will really be able to upload it all to my new unlimited storage Amazon Drive. If it works, I will be paying about $60 per year to store 5 to 10 terabytes of data, plus the one time purchase of Arq for $50. If there is an issue with Amazon, I will at least be able to easily configure Arq to upload all or part of my data to another service. This is about the same cost as Backblaze, but is far more configurable and should hopefully also help me avoid needing to upload all my data repeatedly due to system changes. I also hope Arq is not designed such that it requires keeping more than a gigabyte of information in memory at all times due to my having selected several million files for backup.

For the record, below is my correspondence with Backblaze. The first truly helpful reply, I have to say, also made my mind up about quitting, as it made it sound like they were actively working on the problem, but I found that response posted verbatim in a forum years ago!

Failed inherit (Request #268142)

Sent April 25, 2017 18:06:

Is there anything you can tell me about my account or that I can try before I start from scratch again uploading 5 terabytes of data?

• Screen Shot 2017-04-25 at 18.01.34.png (30 KB)
• Screen Shot 2017-04-25 at 18.05.52.png (20 KB)

Received April 26, 2017 10:03:

Hello,

Thanks for writing in.

Sometimes Backblaze’s background processes can make an Inherit Backup State fail.

Here’s what I’d like you to do:

• Open the Backblaze system preference / control panel
• Click Settings -> Schedule
• Change the schedule to Backup on Click
• Disconnect all external drives
• Reboot your computer
• Attempt the Inherit Backup State again
• If it’s successful, connect your external drives, open the Backblaze settings again and select the externals you’d like backed up

Regards,
 Ryan

Sent April 26, 2017 14:17:

Thanks, but unfortunately that made no difference. I also tried uninstalling and reinstalling. There isn’t a log file somewhere I can look at?

I am disappointed I have somehow had to start from scratch about annually due to the transfer process never working. I excused it when you guys were new, but at this point I have spent a huge amount of time exposed to data loss because I chose to stick with Backblaze for all these years. If I am going to have to upload 5TB and growing every time I make an upgrade or configuration change to my computer (and pay Comcast hundreds of dollars for data overage fees), the very least Backblaze could do is not delete my previous backup until however many months it takes me to upload all the data again! It is also funny I actually have about 12TB of data I would like to back up, but I have never been able to get through my more critical files!

Is there any hope this madness will end? It seems I would be far better off simply running rsync to Amazon Glacier if I have to start over. Then I would never need to “transfer state” again…

Received April 28, 2017 00:19:

Hello,

I think I know what is going on, but you’ll have to help fill in some details. Something has gone slightly haywire on your system that we are seeing in very few customers backups related to the bzfileids.dat file.

There is a very specific file on your disk that is part of Backblaze which has bloated up to be too large. It is called bzfileids.dat and is found here:

Mac: /Library/Backblaze.bzpkg/bzdata/bzbackup/bzfileids.dat
Windows Vista/7: C:\ProgramData\Backblaze\bzdata\bzbackup\bzfileids.dat
Windows XP: C:\Documents and Settings\All users.windows\Application data\backblazebzdata\bzbackup\bzfileids.dat

This is a very simple file, it is a mapping from your filenames to a totally unique integer ID that is anonymous that we use to identify your files in the Backblaze datacenter. This means we never know any of your file names, or file contents.

For some reason, your computer wants to backup a few million files and your bzfileids has grown very large (yours is over 1 GB). When bztransmit (the process that runs once per hour) starts up, it reads this bzfileids.dat file into RAM. On a normal machine, this is about 20 MB, but on your machine something has gone haywire and bzfileids has grown far too large.

Now, there are several things that contribute to this file being large, so you can think about how this happened and let us know. We’re trying to understand this situation better:

A) If you have ever renamed a Time Machine folder at the top of your hard drive, Backblaze will bloat up trying to back it up. It is absolutely not supported to “back up a back up” and Backblaze can only function properly backing up the originals.

B) Lots of files. If you knew of a folder with hundreds of thousands of small files that didn’t change much you could back them up differently or exclude them from Backblaze backups.

C) Renaming top level folders with a lot of files. For example, if your top level folder name is /my_music and it contains 100,000 file names in it, then when you rename it /my_great_music Backblaze needs to add all of those filenames to that bzfileids.dat file which bloats it up. So the best thing you can do is keep your enormous folders the same over a long period.

D) Shorter Path Names. It would be best if your hundreds of thousands of files are on a disk called d instead of disk_that_contains_files and the top level folder is called f instead of folder_for_lots_of_files. Etc. The shorter the paths, the smaller the bzfileids.dat file is.

It’s possible to shrink the bzfileids.dat in case they’ve been temporarily bloated by one of the above situations, however it requires reuploading all data to the Backblaze servers. You can follow these steps to do that:

1. Visit https://secure.backblaze.com/user_signin.htm and sign in to your Backblaze account with your email address and password.
2. Click on the “Preferences” link in the upper left hand corner
3. Select your “old” computer from the list of computers.
4. Click the “Delete Computer” link next to it. This will delete the backed up data, the bloated bzfileids.dat and free up the paid license.
5. Click on “Overview”
6. Click the download link for your operating system in the bottom right corner.
7. Install Backblaze.

Here’s the problem. If you cannot reduce the number of files or path names significantly, then you absolutely are going to encounter this issue again. We are working on a fix for it, but it is proving to be very difficult and currently our engineering team has no ETA on a fix. If you uninstall and reinstall without changing anything, then your backup might start working again, but you will reencounter this problem a little ways down the road.

Regards,
 Ryan

Failed inherit (Request #270170)

Sent May 04, 2017 15:31:

This is a follow-up to your previous request #268142 “Failed inherit”

Thank you for the more detailed information. It was very helpful. Unfortunately I have determined the number of files I need to backup and other factors precludes me from being able to continue using BackBlaze. I wish I knew about this before I just paid for 2 years a few months ago. Is there a way I can get a refund?

Thank you

Received May 05, 2017 23:28:

Hi Charlie,

Normally we do not issue refunds outside of our 30 day policy. However, I’m prepared to offer a prorated refund in this case. In order to do so, the backup and license will need to be deleted using these steps:

1. Visit https://secure.backblaze.com/user_signin.htm and sign in to your Backblaze account with your email address and password.
2. Click on the “Preferences” link on the left hand navigation.
3. Locate the computer in the list of computers (there may only be one).
4. Click the “Delete Computer” link next to it and proceed with the deletion. It will remove that computer’s backup.
5. Click on “Overview” link in the upper left hand navigation.
6. In the Unused License area, click the Delete link and proceed with the deletion.

Please respond to this email when you’ve deleted that computer and license so we can process the prorated refund.

Regards,
 Ryan

Sent May 10, 2017 18:11:

Hello. I have completed the steps. I really appreciate the refund. If you guys need any more information or make changes that could help use cases like mine, feel free to reach out. Thank you!

Received May 11, 2017 16:05:

Hi Charlie,

The prorated refund of $72.72 has been processed, and should reflect on your bank statement within 3-4 business days. Let us know if you have any further questions.

Regards,
 Ryan