066 - Files in the Cloud

Technical Difficulties

Mennonite Church USA

Gabe and Erik take a look at a number of different online sync and storage options for moving your files to the cloud.

Gabe’s Latest Crazy Online File Syncing Scheme

Listen to this section on SoundCloud: 0:00

Gabe begins the discussion by describing a scheme to use a separate Dropbox account running on his WebFaction web hosting, because he can sftp from within Sublime Text even though he’s unable to install any online syncing services.

Easy.

The cloud is always easy when you have IT autonomy…or can find a loophole.

Being able to sftp-out does not provide the same benefit of a proper syncing service, but there are some tools that try to help:

• Backup Box
• FTP Box
• Wappwolf

Generally, these tools work by offloading sync to a web host. Sure, you can achieve this benefit by sftping into a computer at home, but then you face issues such as DNS intermediaries if you don’t have a static external IP address, opening ports into your home network, or just having to leave your home computer on all the time. You may want to consider having someone else worry about your uptime.

Again with the Synology

This episode would have been about five minutes shorter if Gabe could have figured out how to install Dropbox on his beloved Synology. His loss is our gain.

Online Storage and Sync Options

Listen to this section on SoundCloud: 4:10

In a embarrassment of tech riches, there are many great sync systems available for free or a modest fee:

• Cloudfuse
• Dropbox
• Box
• Windows Azure
• Google Drive
• Plex Cloud Sync

Dropbox offers a very refined experience, both through its web interface and its seamless filesystem integration, but your Dropbox folder will probably require some pruning to stay below your subscription’s space limit. Hazel is a big help here, as it is with so many other menial system tasks.

Many of us remember when cool new services like IFTTT only supported Dropbox, but these days most web services start off with integration for virtually all of the usual hosting suspects. On the other hand, these systems aren’t much use if they can’t sync to mobile, and right now Dropbox is most widely supported.

“You almost need a Dropbox account at this point just to use iOS”

Steve Jobs purportedly called Dropbox a service rather than a product, but since so many developers have settled on it for sync, things may not have been that different in effect. A little bit of a shakeup in the developer community could be nice, because there are some interesting alternatives:

• BitTorrent Sync
• Transporter

When syncing needs go beyond the usual, there are other niche options available. Whether you feel the need to control your own data or you want more granularity in administering permissions, you can probably find a service that will meet your needs. While Transporter offers an elegant ready-to-go personal Dropbox, BitTorrent Sync offers a distributed sync setup which you may be able to construct with old equipment already on had.

Welcome @modernscientist

Because I knew that @modernscientist has been using BitTorrent Sync, I asked her to add to the disussion here.

BT Sync

I have recently started using BitTorrent Sync to manage the synchronization of approximately 200 GBs of data between two Macs used for my research. While the end result is similar to traditional cloud storage options such as Dropbox, the BitTorrent protocol does not utilize a centralized server to which all clients upload and download files. Instead, every client can upload and download to all other clients. This decentralized, peer-to-peer model requires the user to provide the devices that will serve as synchronization points (peers), but that requirement also has many advantages. In my situation, it is more cost efficient to provide my own (existing) hardware than pay the monthly fee for increased cloud storage. The BitTorrent protocol also allows users to avoid storing data on servers belonging to someone else, potentially making BitTorrent Sync more private.

BitTorrent Sync and Dropbox have many similar features, though there are some subtle differences. Much like Dropbox, BitTorrent Sync uses a folder paradigm for determining which files are synchronized. However, folders that are synchronized by BitTorrent Sync can be located anywhere accessible to the peer, including on an attached hard drive or local network. BitTorrent Sync has options that are analogous to Dropbox’s selective synchronization and version control features; however BitTorent Sync’s selective synchronization recognizes basic file globbing, making it more powerful than Dropbox’s implementation. Files can also be shared with others using BitTorrent Sync, and write access can optionally be restricted for these shares. Unfortunately, a share must include the entire contents of a torrent, and right now there’s no option for downloading shared files through a web browser.

Perhaps the most important–and very subtle–difference is that BitTorrent Sync does not support extended attributes. This isn’t an issue for me, as most of my files do not have extended attributes and those that do have them don’t seem to actually use them. Usage scenarios differ, so this limitation should be tested carefully. For example, devout users of file tagging in Mavericks, which relies on extended attributes, may find that BitTorrent Sync does not suit their needs.

BitTorrent Sync is available on the standard desktop platforms: Mac, Windows, Linux. Though my primary need is the synchronization of files between two Macs, there are advantages to additional peers. A minimum of one updated peer is required for synchronization, so more peers adds robustness to the system. Similarly, additional peers provide more download sources during synchronization, which can make this process faster. For these reasons, I am also running BitTorrent Sync on my Drobo 5N and on a Backupsy server. BitTorrent Sync runs as a command line application on these peers and, thus, setup is a little more complicated than on a Mac.

iOS and Android versions of BitTorrent Sync are also available. These applications are similar to their Dropbox counterparts in that they only download files on demand.

More info

@modernscientist blogged about this on her own site in more detail, so head there if you’d link to know more. Also, the Backupsy link provided above will discount the service as much as 40%, while getting her some extra storage space. Win/win.

The Challenges of Sharing and Syncing

Listen to this section on SoundCloud: 11:03

• Dropbox for Business
• Box

When you fold in additional users of your files in the cloud, you also fold in a lot of complexity. Syncing solutions designed for business can make this complexity more manageable, but that is more capability (and cost) than most individuals, families, and small businesses need. Leaving these specialized solutions aside, managing your shares can be a bit of a challenge.

Forgotten

It was me. He was trying to share with me.

Clicking on the “Sharing” icon in Dropbox’s web interface does provide a snapshot of your shares.

Dropbox Sharing

Google Drive also provides this information by showing what is shared with you on its web-view sidebar, and you can view the “visibility ” of all your documents by clicking the drop-down in the search field.

Shared with You Shared by You

One of the challenges of app integration is a cluttered Dropbox root folder. As Gabe alluded to, permission polices have had the unexpected benefit of causing more apps to use the app folder. Some more finicky (cranky?) users think apps unnecessarily restrict themselves.

Apps shouldn’t be able to claim Dropbox support unless they let the user specify which folder to use.

— potatowire (@potatowire) January 4, 2014

Regardless of whether these policies (or your own) locate folders in inconvenient or inappropriate locations, you can always symlink to whichever location you prefer.

Syncing apps are not always the best stewards of your internet connection, however, and while Drive will allow you pause sync, you can’t specifically throttle its bandwidth. Dropbox gives you both of these throttling options, but it sure does work the CPU over.

“It’s reticulating splines”

SFTP and Git: Less Friendly, More Configurable

Listen to this section on SoundCloud: 21:32

• Sublime SFTP
• Transmit
• Github
• Bitbucket
• Sourcetree

We may be living in a Post-PC world, but plenty of us would rather have explicit control over our data. Distributed Version Control with Git or Mercurial has long been a staple in programming circles, but if you take the time to learn the language, you’ll find that repositories make good homes for prose (or verse) too.

The Next Big Round of Innovation

Listen to this section on SoundCloud: 26:00

When something like sync is designed to fade into the background, the whole thing can seem a bit stale. The next round of innovation might involve more abstraction to the cloud, like Evernote, but hopefully some of it will be finding new convenient ways to access our data from anywhere, like TextDrop .

Until next week

Well, that’s it for this week. If you have anything that you’d like to add to or correct in the show notes you can find me on Twitter @potatowire or feel free to send an email to me at potatowire dot com.