Cloud Services Like BittorrentSync Do Not Play Well With TrueCrypt: Experiments

Imagine yourself preparing for a business trip on which you will need a large number of sensitive files when you arrive at your destination. Being a sensible person, you're not comfortable having unencrypted sensitive files on your laptop, so you store the files in a large TrueCrypt container with state-of-the-art encryption.

When you open your laptop to use the sensitive files at your destination, you find that all the needed files are missing – none of them were properly synchronized to the laptop. Worse yet, you get home and find out that all the files created for the trip on your home desktop had been permanently destroyed!

Road warrior, you are screwed!

I very narrowly avoided having that happen to me during preparations for an upcoming trip. This post will alert you to things I learned that might help you avoid having such things happen to you. I discuss:

1. Encryption tools
2. Synchronization tools and issues
3. Major problems with existing synchronization tools – BitTorrentSync-Beta and other cloud services

Encryption Tools

There are dozens of ways to effectively protect sensitive files on computers that you use at home or
for travel. Disk encryption, file encryption, vaults, modern ZIP files, you name it. Each has its own set of costs and benefits and all of them can do the job if you use them properly. For our purposes, a major cost of each method is user convenience. If the user hates using the software properly, sooner or later protection will break down. Thus it's important to find an encryption system that feels comfortable to you

In my case, TrueCrypt vaults easily fit into my workflow, so I use TrueCrypt almost exclusively.

Sometimes the vault files get really large – 10 to 20 to even 50 GB. That's not a problem as long as the file is being used on one computer

File size does become an issue when you work on different computers and you want each computer to have its own current version of the large vault. Constantly copying 10 or 20 or 50 GB files from one computer to another is tedious and prone to error-making

That's why it's nice to have a system where the copying occurs virtually automatically: Click one shortcut on any online computer in your system and you can be confident that all the files you want to share across computers will be selected properly, nothing will be overlooked or forgotten and the process will be completed relatively quickly.

You need synchronization software

FreeFileSync has long been my favorite program for synchronizing data to different drives on one computer or to different computers that are operating on my network. It's frugal (i.e., free), dependable, user-friendly – I could go on and on about its merits. (Just be sure you download it from the SourceForge link above. See reviews at CNET about recent problems.)

The only thing about FreeFileSync that sometimes causes my eyes to roll is that it works by copying whole files from one location to another. That means it can be a little slow in synchronizing folders containing large files like TrueCrypt files. When there is a disparity between two file versions, FreeFileSync copies the entire current file from one place to the other rather than only the parts of the file that need to be updated. Programs like Dropbox, for example, owe some of their speed to the fact that they only exchange pieces of a file (Block-Level Synchronization) during synchronization rather than exchanging whole files

This relatively minor dissatisfaction over synchronization speed for large files left me constantly on the prowl for a faster but equally dependable way to synchronize my data trees.

That's how I developed an interest in BitTorrentSync-Beta.

BitTorrentSync  is now being widely touted as a formidable candidate for replacing Dropbox as a service for synchronizing your data between computers without having to store your files on potentially insecure servers. In April, 2013 a beta version of the program was released to the public and by November, 2013 a million people were regularly using the program (Wikipedia). By December of that year, the user base had increased to 2 million.

At first glance, the program seems to be an excellent frugal solution to the problem of data synchronization.

• Currently, there is no charge for using the service no matter how much data you synchronize. Even if they do begin charging for the program, the cost will likely be much lower than you would have to pay for similar services that require you to store your files on company servers for synchronization to occur.
• The program synchronizes files very, very quickly. Its method of peer to peer Block-Level Synchronization is far faster than that used by most cloud services.
• Your data never sits on third-party servers with all their well-recognized privacy and security issues.
• The amount of data you can synchronize is limited only by the size of your hard drives – terabytes of synchronized data are attainable.
• Compared with the existing cloud services, cost factors are negligible. Dropbox, for example, gives you only 2 GB of free storage. If you move to their paid plan, it will cost you a dollar per gigabyte per year to keep your data on the cloud. Yeah, you can shop around for better deals and earn a limited amount of extra storage through referrals etc, but have a life!
• Once you began experimenting with the program, you can see it has versatility that other such services lack. You can synchronize any folders on your computer to corresponding folders on other computers. If you want to use the program in Dropbox-like fashion, you can create one giant folder into which you throw your stuff and it will appear in a corresponding giant folder elsewhere.
• No piece of software is perfect, but there’s a responsive support unit and user community.

So, when I began preparing for my trip, I installed BitTorrentSync-Beta on my laptop and looked forward to quickly synchronizing files between my desktop and the laptop.

After playing with the program, I deployed it to synchronize some files and folders between computers on my network.

The results were delightful. Thousands of files effortlessly synchronized across my network without any apparent data loss

For the most part, these folders coincidentally all contained small to moderate size files – spreadsheets, text documents, etc.  None of my initial tests involved large TrueCrypt files, a factor I later learned to recognize as being of mission-critical importance.

Eventually, the stuff hit the fan when I synced my TrueCrypt files.

I saw that BitTorrentSync-Beta and TrueCrypt do not play nicely with one another. The same can be said about other cloud services I tested with TrueCryptFiles. It’s even possible that these services have similar issues synchronizing large, frequently updated data files created by programs besides TrueCrypt. Who knows?

This problem deserves much more attention than it is getting in current exuberant web discourse about cloud software and synchronization of data for the average consumer

BitTorrentSync-Beta Does Not Work with Large TrueCrypt Files: The Experiments

Before leaving on my trip, I fortunately conducted over a dozen rigorous experiments involving BitTorrentSync-Beta and large TrueCrypt files. The highlights of my method are as follows:

1. Create a 10 GB TrueCrypt file on the desktop/server computer.
2. Create additional “SyncMonitoringFiles” in the same folder as the TrueCrypt file. These will be used to verify that the program correctly synced small files no matter how it handled large files.
3. Copy the entire folder to another computer. Now identical versions of the files exist on each computer.
4. Modify the file contents on one computer. Do this inside the TrueCrypt as well as in the “SyncMonitoringFiles”.
5. Synchronize the files by starting BitTorrentSync-Beta on both computers.
6. Compare versions: use software to compare MD5’s or mount the volumes and inspect contents.
7. Repeat 1 through 6 to test one-way and two-way synchronization.

This is the same method that I've used to test cloud services over the last two or three years.

Sadly, my tests of BitTorrentSync-Beta yielded the same results that I obtained with the cloud services I examined. That’s why I mention the cloud service tests in the following paragraphs.

In total, the test results serve to warn users: You must display considerable vigilance before deploying cloud services, including BitTorrentSync, for everyday use. Very closely evaluate the performance of the synchronization programs when they handle TrueCrypt files.

300 MB or less? No problemo! At least I never found one.

• All the programs reliably performed one way and two-way synchronization of common small and medium-sized files.
• For reasons I do not understand, even small TrueCrypt files synchronized properly. It seems logical to suppose that if TrueCrypt files don’t play nicely with synchronization programs, they should get corrupted no matter how big the file is, but I am not aware of any reports of people having problems syncing the TrueCrypt files with DropBox, a service which limits the size of uploaded files.

As file size increased, synchronization programs created more and more errors as they handled TrueCrypt files.

• At 5 GB, a popular cloud service began displaying erratic behavior that made the service unreliable as a synchronization device. Once this service was deployed, a user who wanted peace of mind after the sync operation would always have to verify the accuracy of the sync by comparing MD5’ or manually inspecting vault content. Who wants to be bothered with that?
• At 10 GB, BitTorrentSync-Beta broke down weirdly. Before the synchronization process started, there was about a half hour difference between timestamps. The desktop version was clearly labelled as the newer version. At the end of the operation, the timestamp on the desktop version was identical to the timestamp of the earlier file version on the laptop! At first, I thought this would have no practical implications, but I double-checked everything and found major issues.
• The post-sync versions of the files had identical MD5’s. This suggested that the operation might create survivable errors during daily use.
• Then I mounted each of the TrueCrypt volumes and inspected their contents. Inspection of the mounted Truecrypts revealed that test files created in the TrueCrypt on one computer were not replicated in the volume on the other computer. To the contrary, all the new test files within the TrueCrypt volumes had disappeared on both computers!
• Here’s where the “SyncMonitoringFiles”  became very important. Those control files all synchronized perfectly in one-way and two-way syncs, even when BitTorrentSync-Beta was failing to handle the TrueCrypt files properly.

Let’s distill matters even further.

The core issue is one of knowing what direction a synchronization operation will take when a program like BitTorrentSync-Beta encounters a TrueCrypt file.

The results of these experiments indicate that BitTorrentSync-Beta and some other cloud services cannot properly detect which of two TrueCrypt files is the one the user wants to keep.

By itself, the timestamp difference between the files does not work consistently to signal “keep this one” or “keep that one”. Hence the program might properly perform a correct two-way sync on all the other files in a folder only to improperly reverse the desired direction of a sync when a TrueCrypt file is encountered.

For unknown reasons, the problem appears to be much more serious with larger TrueCrypt files than with small ones, so it’s very important to use large TrueCrypt files when you evaluate how well a synchronization program might work when it is deployed on your system.

The Takeaways:

The results destroyed my confidence in the reliability of existing cloud synchronization services for dealing with large TrueCrypt volumes. Had I been relying on BitTorrentSync-Beta to properly update my laptop for travel:

1. At my destination, I would’ve discovered that none of my recent needed files were in the crypt.
2. Returning home, I would've found that all the additional files I had put into the desktop version of the crypt were also lost.
3. Had I not backed up the original TrueCrypt file to a safe location before starting the synchronization program, my most recent working files would have been permanently lost on both computers.

From all this, I learned I should not rely on currently available cloud services to correctly synchronize TrueCrypt volumes. Well, Duh-Uh!

In my own experience, only two methods dependably synchronized contents of TrueCrypt files existing on different computers.

• Manually copy the entire current file to another computer and overwrite the outdated version  – a time-consuming process that has to be repeated every time synchronization is needed.
• Use FreeFileSync to automate the process. This program has always synchronized TrueCrypt files perfectly for me. (You might have a similar folder synchronization program that also works perfectly for you because it uses the same Windows copying routines that FreeFileSync uses.)

Folder synchronization programs typically copy entire files rather than exchanging parts of them, so the operation can be slow, especially when you are copying to a slow USB drive and very large TrueCrypt files are in play. In such cases, I generally work inside the TrueCrypt files rather than in their containing folder to increase speed of the operation

1. Wherever it is, mount each TrueCrypt file so that it is functioning as a drive. If it is on a different computer, set its drive as a network shareable device.
2. Start FreeFileSync or a similar program to carry out one-way or two-way synchronization of the files in each mounted drive – a tedious process, but it takes less time overall then moving scores of GB of data.

Finally, please remember that lots of people depend on TrueCrypt to keep their data secure. Publicize good or bad experiences you’ve had when synchronizing TrueCrypt files!

Oh, yeah! Keep me in your loop?

Updates:

SugarSync gives lame solution to TrueCrypt sync problem