Setting up my Solaris server as a centralized backup server

After some months of work I now have the time to set-up my server properly so that it can backup all my computers without a hassle. Since I wanted to let the server control, when the backups should be made I wrote a Ruby script which runs every hour and backs up all the available hosts (which are of course Macs ;-). The script should not run at the same time and produce a decent logfile.

Set up environment

First I had to make sure, that the server had the correct time. By default the ntp daemon did not run, so I configured it using the description at the grey blog. I did not use the European ntp server though instead I used de.pool.ntp.org.

To install the current version 1.8.7 of Ruby I entered as root:

# pfexec pkg install SUNWruby18

Then I created a ssh key for my Solaris root user:

# ssh-keygen -t dsa
Enter file in which to save the key (/root/.ssh/id_dsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_dsa.
Your public key has been saved in /root/.ssh/id_dsa.pub.
The key fingerprint is:
*DISCLOSED*

To get a ssh connection which is needed for rsync to each of my hosts I then copied the contents of /root/.ssh/id_dsa.pub to the to the file authorized_keys e.g. user1@host1:.ssh/authorized_keys

Now whenever I enter “ssh user1@host1” no password is needed to get a remote shell.

The Ruby Script

#!/usr/bin/ruby
# This script will fetch the current files from a couple of hosts via rsync
# and stores them locally
require 'ping'
require 'tempfile'
require 'open3'
require 'logger'
require 'fileutils'

ZFS_POOL="your_zfs_pool_here"
LOCAL_BACKUP_PATH="/#{ZFS_POOL}"

# Parse the commandline parameters
ARGV.each do |arg|
  case
  when arg == '--stdout'
    LOG_STDOUT = true
  when arg == '--dry-run'
    DRY_RUN = true
  end
end
LOG_STDOUT = (LOG_STDOUT rescue false) # Set default value to false
DRY_RUN = (DRY_RUN rescue false)       # dto.

# Check which output should be used for logging
if LOG_STDOUT
  # Log to stdout
  $LOG = Logger.new($stdout)
  $LOG.datetime_format = '%H:%M:%S'
else
  # Logfile will not exceed 1 MB
  $LOG = Logger.new('/var/log/backup_rb.log', 0, 1 * 1024 * 1024)
  $LOG.datetime_format = '%d.%m.%y %H:%M:%S'
end

# Kill older processes of this script
pids_to_kill = []
`ps -Al -o pid -o args|grep -e ruby|grep -e #{__FILE__}`.split("\n").each do |line|
  other_pid = line.split(" ")[0].to_i
  if other_pid != $$
    pids_to_kill << other_pid
    `ps -Al -o pid,ppid=MOM -o args|grep "1 rsync"|grep -v grep`.split("\n").each do |child_line|
      child_pid = child_line.split(" ")[0].to_i
      pids_to_kill << child_pid
    end
  end
end

if pids_to_kill.length > 0
  $LOG.info "****** Cleaning up... *******"
  $LOG.info "Killing old backup processes #{pids_to_kill.join(",")}"
  `kill -9 #{pids_to_kill.join(" ")}`
end

# Execute a command and store its output
class ExecCmd
  attr_reader :output,:error_output,:cmd,:exec_time

  def initialize(cmd,cmd_id)
    @output = ""
    @error_output = ""
    @exec_time = 0
    @cmd = cmd
    @cmd_id = cmd_id
  end

  def run
    start_time = Time.now
    begin
      $LOG.info "[#{@cmd_id}] Starting command: #{@cmd}..."
      Open3.popen3(@cmd) do |stdin, stdout, stderr|
        @output = stdout.read
        @error_output = stderr.read
      end
    rescue Exception => e
      @error_output += e.to_s
    ensure
      @exec_time = Time.now - start_time
      $LOG.info "[#{@cmd_id}] Command completed in #{@exec_time} seconds."
    end
  end

  # Log the stdio and stderr outputs
  def log_results
    $LOG.info "[#{@cmd_id}] #{@cmd}:"
    if @error_output.length > 0
      @error_output.split("\n").each { |line| $LOG.error "[#{@cmd_id}]  #{line}" }
    end
    if @output.length > 0
      @output.split("\n").each { |line| $LOG.info "[#{@cmd_id}]  #{line}" }
    end
  end

  # Returns false if the command hasn't been executed yet
  def run?
    return @exec_time > 0
  end

  # Returns true if the command was successful.
  def success?
    return @error_output.length == 0
  end
end

# Define for each host which user accounts are being backed up and which files should be excluded
default_excludes = ['.Trash', 'Downloads', 'Desktop', 'Music/iTunes/iTunes Music/Podcasts',
                    'Library/Caches', 'Library/Logs']
# format: hostname => { username => [excluded_files] }
HOSTS={ 'host1' => { 'user1' => default_excludes },
        'host2' => { 'user2' => default_excludes, 'user1' => default_excludes },
        'host3' => { 'user2' => default_excludes + ['Music/iTunes']},
        'host4' => { 'user3' => default_excludes }
      }

$LOG.info "****** Backup started... *******"

# Make a ZFS snapshot
snapshot_name = "#{ZFS_POOL}@backup-#{Time.now.strftime('%y-%m-%d_%H:%M')}"
$LOG.info "Creating ZFS snapshot #{snapshot_name}"
`zfs snapshot #{snapshot_name}`

pending_commands = {}
HOSTS.each do |hostname,user_data|
  $LOG.info "Calling #{hostname} ..."
  if Ping.pingecho(hostname)
    user_data.each do |user,excluded_files|
      exclude_file = Tempfile.new("tempfile")
      excluded_files.each { |filepath| exclude_file << filepath << "\n" }
      exclude_file.close
      user_hostname = "#{user}@#{hostname}"
      $LOG.info "Backing up #{user_hostname} ..."
      local_backup_path = "#{LOCAL_BACKUP_PATH}/#{hostname}/#{user}"
      FileUtils.mkdir(local_backup_path) unless File.exists? local_backup_path
      command = "rsync -#{DRY_RUN ? 'n' : ''}avz --delete --partial --exclude-from=#{exclude_file.path} #{user_hostname}: #{local_backup_path}/"
      rsync = ExecCmd.new(command, user_hostname)
      pending_commands[user_hostname] = rsync
      Thread.new do
        rsync.run
      end
    end
  else
    $LOG.warn "#{hostname} does not respond!"
  end
end

# Wait for the backup processes to complete
while pending_commands.length > 0
  pending_commands.each do |user_hostname, exec_cmd|
    if exec_cmd.run?
      exec_cmd.log_results
      pending_commands.delete(user_hostname)
    end
  end

  if pending_commands.length > 0
    $LOG.info "Still #{pending_commands.length} tasks backing up #{pending_commands.keys.join(', ')}"
    sleep 60
  end
end

$LOG.info "****** Backup complete. *******\n"

What it does

You can use the command line argumens –dry-run and –stdout. The first one will call rsync with the –dry-run option and the second one will write to stdout instead of a logfile.

On start the script looks for other instances of itself and will kill them and all orphaned rsync child processes.

It will create a ZFS snapshot of the target pool with the current time and date as a label.

Then it will ping all the hosts defined in HOSTS and will construct the rsync command with all the excluded files and the defined users and start a separate thread in which the command will be executed.

Then it will loop until all the rsync tasks have been completed.

The logfile is /var/log/backup_rb.log and looks like this:

I, [15.10.09 11:44:11#7343]  INFO -- : ****** Cleaning up... *******
I, [15.10.09 11:44:11#7343]  INFO -- : Killing old backup processes 7316,7325,7332,7328,7334
I, [15.10.09 11:44:11#7343]  INFO -- : ****** Backup started... *******
I, [15.10.09 11:44:11#7343]  INFO -- : Creating ZFS snapshot daten@backup-09-10-15_11:44
I, [15.10.09 11:44:11#7343]  INFO -- : Calling host1 ...
I, [15.10.09 11:44:11#7343]  INFO -- : Backing up user2@host1 ...
I, [15.10.09 11:44:11#7343]  INFO -- : [user2@host1] Starting command: rsync -avz --delete --partial --exclude-from=/tmp/tempfile20091015-7343-1hpu6rm-0 user2@host1: /daten/host1/user2/...
I, [15.10.09 11:44:11#7343]  INFO -- : Calling host2 ...
I, [15.10.09 11:44:11#7343]  INFO -- : Backing up user1@host2 ...
I, [15.10.09 11:44:11#7343]  INFO -- : [user1@host2] Starting command: rsync -avz --delete --partial --exclude-from=/tmp/tempfile20091015-7343-1f9jzvu-0 user1@host2: /daten/host2/user1/...
I, [15.10.09 11:44:11#7343]  INFO -- : Calling host3 ...
W, [15.10.09 11:44:16#7343]  WARN -- : host3 does not respond!
I, [15.10.09 11:44:16#7343]  INFO -- : Calling host4 ...
I, [15.10.09 11:44:16#7343]  INFO -- : Backing up user2@host4 ...
I, [15.10.09 11:44:17#7343]  INFO -- : [user2@host4] Starting command: rsync -avz --delete --partial --exclude-from=/tmp/tempfile20091015-7343-yv60ta-0 user2@host4: /daten/host4/user2/...
I, [15.10.09 11:44:17#7343]  INFO -- : Backing up user1@host4 ...
I, [15.10.09 11:44:17#7343]  INFO -- : [user1@host4] Starting command: rsync -avz --delete --partial --exclude-from=/tmp/tempfile20091015-7343-644sq6-0 user1@host4: /daten/host4/user1/...
I, [15.10.09 11:44:17#7343]  INFO -- : Still 4 tasks backing up user1@host2, user1@host4, user2@host4, user2@host1
I, [15.10.09 11:45:17#7343]  INFO -- : Still 4 tasks backing up user1@host2, user1@host4, user2@host4, user2@host1
I, [15.10.09 11:45:41#7343]  INFO -- : [user2@host4] Command completed in 84.087238
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] rsync -avz --delete --partial --exclude-from=/tmp/tempfile20091015-7343-yv60ta-0 user2@host4: /daten/host4/user2/:
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] receiving file list ... done
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] Dropbox/
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] Library/Application Support/SyncServices/Local/
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] Library/Application Support/SyncServices/Local/admin.syncdb
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] Library/Application Support/SyncServices/Local/TFSM/com.apple.Calendars/
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] Library/Application Support/SyncServices/Local/clientdata/120c2b27e9ab530b442181ced8799e35b30c85cb/
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] Library/Application Support/SyncServices/Local/conflicts/
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] Library/Calendars/
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] Library/Calendars/Calendar Cache
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] Library/Calendars/Calendar Sync Changes/
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] Library/Calendars/FE3DF9D9-8D76-4F44-973A-525E02717BFE.calendar/Info.plist
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] Library/Logs/Sync/syncservices.log
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] Library/Mail/
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] Library/Preferences/
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] Library/Preferences/iCalExternalSync.plist
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4]
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] sent 13772 bytes  received 1001023 bytes  13440.99 bytes/sec
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] total size is 64974563109  speedup is 64027.28
I, [15.10.09 11:46:17#7343]  INFO -- : Still 3 tasks backing up user1@host2, user1@host4, user2@host1
I, [15.10.09 11:47:17#7343]  INFO -- : Still 3 tasks backing up user1@host2, user1@host4, user2@host1

Run in crontab

Finally I added a new entry of the root user crontab with “crontab -e” which will start the script every hour.

0 * * * * /usr/bin/ruby /root/backup.rb
Advertisements

One thought on “Setting up my Solaris server as a centralized backup server

  1. Very nice script, but I’d be really happy if there was an option to specify also included directories, for example when using the script to backup the / filesystem.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s