I'm migrating my server data again - this time to the cloud.
I haven't followed stats on my site for years now, and didn't realise how popular atmayogi.com, planetiskcon.com, and kirtanaustralia.com had become.
For the past week the sites have been doing the equivalent of crashing on a mate's couch. Only these guys are so hungry, that they've eaten him out of house and home.
During Maha Kirtan 1 in Canberra Dave sms'd me to tell me that the server was pulling a solid gig an hour streaming the kirtan to listeners. What's a GB in bandwidth terms? Well, the hosting plan the machine is on is 50GB a month. We've put through 60GB in five days.
I've found another server in the States to host the kirtan stream on. It's a Godaddy Virtual Private Server with 2,000GB per month bandwidth. That should be enough. That is going to cost US$100 per month. We'll see how the speed for streaming is with them. They tell me the machine will be available in 72 hours, so Kirtan Australia Internet Radio is down until then.
I'm trying to move as much as possible into the cloud. It makes it easier when I have to move things, and it lets other people take care of the hassle of configuration and administration. I'm getting a bit old for that kind of stuff now. It was useful when I was learning it, but I'd rather focus higher up the stack these days.
I'm hosting my email domain (worldsankirtan.net) in Google's cloud. They give me 25 GB of storage and handle my domain MX and smtp for US$50 a year.
The other significant, as in 12GB per day traffic, piece, are the mp3s on atmayogi.com and kirtanaustralia.com.
These I am moving into Amazon's cloud. The files themselves will live in Amazon S3, and they will be delivered via Cloudfront, Amazon's content distribution network. I upload the mp3s to S3 and they are stored in Singapore. This has a cost for storage, bandwidth, and requests. I then publish the files via a Cloudfront url. When a browser requests a file, Cloudfront will serve it from the nearest server, rather than from Singapore. So this should lead to optimal download speeds all over the world. There is another cost for this, based on bandwidth and requests. The costs for both are quite nominal, although I have yet to see what it looks like with the scale of traffic I'm putting through.
I will be putting a paypal button on the three main sites and publishing a report each month on income and expenses. If you think you are getting some value from atmayogi.com, planetiskcon.com, and kirtanaustralia.com, feel free to show your appreciation with some cold, hard, digital cash. :-D
It's 1.28am and i have to wake the Deities here in a couple of hours. I've been up all night working on this script, which moves mp3s from my local server to Amazon's cloud.
It's kind of time critical, since the cost of keeping the sites up right now is being measured in dollars per hour, rather than dollars per month.
The script I've written will find all the mp3s under the directory it is run in, upload them to S3 with public visibility, move them out of the local directory to a backup directory so no-one can download them from the server any more, and generate Apache redirects to put into the apache config so that the mp3 download urls stay the same, but requests are now served from the cloud.
To run it you need a linux / unix machine (I'm running it on an Ubuntu server) with s3cmd, a linux command line utility for S3, installed.
The script is not extensively tested - I've run it once successfully.
#!/bin/bash
SAVEIFS=$IFS
IFS=$(echo -en "\n\b")
# this script will find all mp3s in the directory it is run in and below. I run it in the webcontent root directory
# it will produce a redirects.txt file and logfile.txt file in the directory it is run. redirects.txt has the Apache config that you can cut and paste for the redirects. Logfile has some information about what files were moved around
# these files need to be manually scrubbed between each run at the moment. It would be good if the script rotated existing ones or created one with the time of launch as part of the filename
# also it should fail if it cannot create the redirects.txt file
# the script is not very safe, there is no transactional isolation - files could be moved to the backup directory without making it to S3 first, for example. Use at your own risk, and feel free to let me know of any improvements
for i in $( find . -name \*.mp3 ); do
filename=$i
escapedshortname=`echo $filename | cut -c3- | sed 's/\([, ]\)/\\\&/g'`
escapedname=`echo $i | sed 's/\([, ]\)/\\\&/g'`
shortname=`echo $i | cut -c3-`
directory=`dirname $i`
shortdirectory=`echo $directory | cut -c2-`
# you need to configure s3cmd before running this script; and obviously you need to set up an Amazon S3 account
s3bucketname=YOUR_S3_BUCKET_NAME
# backupdir is where the script will move the local copies of your mp3s. No trailing space
backupdir=PATH_TO_BACKUP_DIR
# This sets up the redirect for Cloudfront. You can figure it out for S3 by using s3cmd to push some files up and see the format of the url it returns
cloudfrontcname=YOUR_CLOUDFRONT_CNAME
# Move the mp3 to Amazon S3
echo s3cmd put $escapedshortname s3://$s3bucketname/$escapedshortname
s3cmd put --acl-public $shortname s3://$s3bucketname/$shortname
# Create an Apache Redirect
echo Redirect \"/files/$shortname\" \"http://$cloudfrontcname/$shortname\" >> redirects.txt
# Move the local file to a back up directory
# Create the correct backup directory under $backupdir if it doesn't already exist
if [ ! -d $backupdir/"$directory" ]; then
mkdir -p ./../oldfiles$shortdirectory
echo Create Directory ./../oldfiles$shortdirectory >> logfile.txt
fi
mv $shortname $backupdir$shortdirectory/
echo Moved $escapedshortname to $backupdir$shortdirectory/ >> logfile.txt
#Uncomment the line below for debugging - this will allow you to run the process one file at a time and see what is happening
#read -n 1 -p "Press any key to continue..."
done
IFS=$SAVEIFS
Recent comments
2 days 23 hours ago
4 days 7 hours ago
4 days 23 hours ago
2 weeks 2 days ago
3 weeks 3 days ago
4 weeks 2 days ago
4 weeks 2 days ago
4 weeks 6 days ago
4 weeks 6 days ago
4 weeks 6 days ago