Tuesday, 18 September 2012

Backup your log files to Amazon S3

As somewhat of a follow-up to my post earlier on Auto Scaling your AWS fleet with Cloud Watch, if any of your instances log to local storage, once said instances have been scaled down, getting that log data back will be difficult, or impossible if you trash your EBS volumes too.

One solution is to have a small script create a bucket within S3 as soon as your instance powers up. Next, have another script that copies your local log directory to the S3 bucket every few minutes. That way, once your EC2 instance is destroyed you still have access to all the log data ... just in case. This example refers to AWS instances but could easily be used on any instance, cloud or otherwise.

The tools of choice here are few:
  1. CloudBerry Amazon S3 Explorer Freeware - we will use this for both bucket management as well as the built in PowerShell snap-in.
  2. Windows Server Task Scheduler

Install and configure CloudBerry Amazon S3 Explorer

To get started, download and install CloudBerry Amazon S3 Explorer Freeware. Next, register the PowerShell snap-in by dropping to the command line (run as administrator) and launch the following.

C:\Windows\Microsoft.NET\Framework64\v2.0.50727\InstallUtil.exe "C:\Program Files\CloudBerryLab\CloudBerry Explorer for Amazon S3\CloudBerryLab.Explorer.PSSnapIn.dll"

Once successful, you are ready to go with the scripts. Overall, they are very lightweight and easy to read regardless of your PowerShell knowledge. 

First things first, set the PowerShell execution policy to unrestricted by executing the following PowerShell command: set-executionpolicy unrestricted. The Set-ExecutionPolicy cmdlet enables you to determine which Windows PowerShell scripts (if any) will be allowed to run on your computer. If you like you can read more about that cmdlet here.

S3 Folder creation via PowerShell & S3 Explorer

Create the first script that will add a folder to your S3 bucket. If you don't have one yet, use S3 Explorer or the AWS Console to create one. You can copy/paste the following into an empty PowerShell script.


#declare variables
$key = "xxxxx"  #AWSAccessKeyId
$secret = "xxxxx" #AWSSecretKey

#load CloudBerryLab PSSnapin
Add-PSSnapin CloudBerryLab.Explorer.PSSnapIn

$hname = hostname
$s3 = Get-CloudS3Connection -Key $key -Secret $secret
$destination = $s3 | Select-CloudFolder -path "mylogs" | Add-CloudFolder $hname

What this will does is sets up your AWS authentication, loads the CloudBerryLab  PowerShell snap-in, sets a variable $hname equal to the hostname of your instance. It then uses that variable to create a new folder in your mylogs S3 bucket.

Set this up as a basic task to Run whether user is logged on or not and have it set to trigger At startup. Configure the task action to Start a program, of which, powershell.exe is your program of choice here. Then add the following arguments:  -command "D:\tmp\logscopu.ps1"

Reboot the server to confirm - you should see a new folder in your S3 bucket.


S3 File copy via PowerShell & S3 Explorer

The second little script is just about as easy to read. 



#declare variables
$key = "xxxxx"  #AWSAccessKeyId
$secret = "xxxxx" #AWSSecretKey

#load CloudBerryLab PSSnapin
Add-PSSnapin CloudBerryLab.Explorer.PSSnapIn


#determine and set instance hostname
$hname = hostname

$s3 = Get-CloudS3Connection -Key $key -Secret $secret
$destination = $s3 | Select-CloudFolder -path "mylogs/$hname"
$src = Get-CloudFilesystemConnection | Select-CloudFolder -path "D:\Logs"
$src | Copy-CloudItem $destination -filter "*"

The PowerShell snap-in bits are a authenticating your account, setting up the destination bucket with a foldername that matches your instance name. And finally copying the local content of D:\logs\* to mylogs/$hname.

Set this up on a basic task as well - but make it fire every 5 minutes. You should now see your logs populating S3 about every 5 minutes. 

CloudBerry has many examples here that you can use and tweak to get just what you need.

No comments:

Post a Comment