teh cloud

shawn

shawn

running dog lackey of the oppressor class
Administrator
Joined
Mar 13, 2005
Location
Raleigh, NC
Anybody storing their data in Amazon S3 or anything like that? I found out that our backup server was moved into a datacenter next door to our primary server... which doesn't leave much leeway in case of natural disasters and things like that.

S3 is dirt cheap, but looks like it's a PITA to use with rsync.

I need about 20-30GB. The cheaper the better.
 
At work we use Dropbox. I'm not sure of the price but I know it's cheap or we wouldn't use it.

It's more of a "sync" but it is still backed up in a cloud. We have 4 engineers including myself. Each engineer has a desktop at work and a work laptop at home and we needed a good way to share files. Everyone has "dropbox" installed on their C drive. Inside dropbox each engineer has a folder. We do all our work in our folder. As long as you are synced, you have live update. It also has an event log so file recovery is a snap.
 
Shawn are you talking for hosting data or what?
I don't believe you can use Dropbox for data that is for hosting a site. Correct me if I am wrong. DB is like a de.licio.us for your files they go where you go. Shawn you looking for web site storage in the cloud?

I would def lean to Amazon or Azure and possibly look at G-Apps marketplace if you can utilize any of their services. Not sure what you are totally trying to do though. Have a look at rackspace cloud, and I think Mediatemple now has cloud too if my memory is correct. Dreamhost also has committed to OpenStack just recently.

I haven't delt with any Cloud vendors but I can tell you about CDN.
You know where to find me.
L8r
 
Yeah, this is to backup the site. Basically, we've got internal backups on the server. That stuff gets rsync'd offsite nightly. I need cheap, always-on storage with a fat enough upstream pipe to restore everything in a reasonable timeframe. I download occasional backups to a server here at the house, but it would literally take me days to upload that stuff if I needed to restore everything.

So yeah, this is just disaster recovery. A plane crashes into the data center, our hosting company quits paying the bills and gets locked out, the RAID card goes bonkers and corrupts all the discs, an earthquake lops SoCal off the map, etc. Or actually, the most likely scenario... some script kiddie gets root and rm -rf everything.

The more I looked into AWS and S3 specifically, the more it looked like it sucked. You could use S3FS to mount the cloud storage and copy to it, but because rsync thought it was local storage, you couldn't use SSH and all the inode stuff went across as an individual query for each file, so it was slow. You didn't get any multiplexing.
 
One of the things about AWS is that it sits slightly below what we usually expect - above the network, but just under a typical app. They've improved over time, with more app functionality in their mgt console, but the true power of AWS is still in their API's. Things like S3FS, Jungle Disk, etc. just use those API's to provide you with more of a friendly UI.
.
Best example I've seen of getting S3 to work with rsync is http://freewisdom.org/photos/2008/09/17/backup_with_rsync. Similar is http://notes.minty.org/cgi-bin/wiki.pl?Offsite_Rsync_Backups_Using_Amazon_S3, although may be an earlier description of the same method.
.
Other tools built on the API's out there too http://www.s3rsync.com/index.html#services. And example of using it, http://blog.eberly.org/2006/10/09/how-automate-your-backup-to-amazon-s3-using-s3sync/.
.
Not trying to sell AWS, but it's taken me a long time to understand what it is. S3 is just one piece of a big set of pieces that make up a sometimes confusing looking puzzle!
 
Back
Top