Setting up my Solaris server as a centralized backup server

After some months of work I now have the time to set-up my server properly so that it can backup all my computers without a hassle. Since I wanted to let the server control, when the backups should be made I wrote a Ruby script which runs every hour and backs up all the available hosts (which are of course Macs ;-). The script should not run at the same time and produce a decent logfile.

Set up environment

First I had to make sure, that the server had the correct time. By default the ntp daemon did not run, so I configured it using the description at the grey blog. I did not use the European ntp server though instead I used de.pool.ntp.org.

To install the current version 1.8.7 of Ruby I entered as root:

# pfexec pkg install SUNWruby18

Then I created a ssh key for my Solaris root user:

# ssh-keygen -t dsa
Enter file in which to save the key (/root/.ssh/id_dsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_dsa.
Your public key has been saved in /root/.ssh/id_dsa.pub.
The key fingerprint is:
*DISCLOSED*

To get a ssh connection which is needed for rsync to each of my hosts I then copied the contents of /root/.ssh/id_dsa.pub to the to the file authorized_keys e.g. user1@host1:.ssh/authorized_keys

Now whenever I enter “ssh user1@host1” no password is needed to get a remote shell.

The Ruby Script

#!/usr/bin/ruby
# This script will fetch the current files from a couple of hosts via rsync
# and stores them locally
require 'ping'
require 'tempfile'
require 'open3'
require 'logger'
require 'fileutils'

ZFS_POOL="your_zfs_pool_here"
LOCAL_BACKUP_PATH="/#{ZFS_POOL}"

# Parse the commandline parameters
ARGV.each do |arg|
  case
  when arg == '--stdout'
    LOG_STDOUT = true
  when arg == '--dry-run'
    DRY_RUN = true
  end
end
LOG_STDOUT = (LOG_STDOUT rescue false) # Set default value to false
DRY_RUN = (DRY_RUN rescue false)       # dto.

# Check which output should be used for logging
if LOG_STDOUT
  # Log to stdout
  $LOG = Logger.new($stdout)
  $LOG.datetime_format = '%H:%M:%S'
else
  # Logfile will not exceed 1 MB
  $LOG = Logger.new('/var/log/backup_rb.log', 0, 1 * 1024 * 1024)
  $LOG.datetime_format = '%d.%m.%y %H:%M:%S'
end

# Kill older processes of this script
pids_to_kill = []
`ps -Al -o pid -o args|grep -e ruby|grep -e #{__FILE__}`.split("\n").each do |line|
  other_pid = line.split(" ")[0].to_i
  if other_pid != $$
    pids_to_kill << other_pid
    `ps -Al -o pid,ppid=MOM -o args|grep "1 rsync"|grep -v grep`.split("\n").each do |child_line|
      child_pid = child_line.split(" ")[0].to_i
      pids_to_kill << child_pid
    end
  end
end

if pids_to_kill.length > 0
  $LOG.info "****** Cleaning up... *******"
  $LOG.info "Killing old backup processes #{pids_to_kill.join(",")}"
  `kill -9 #{pids_to_kill.join(" ")}`
end

# Execute a command and store its output
class ExecCmd
  attr_reader :output,:error_output,:cmd,:exec_time

  def initialize(cmd,cmd_id)
    @output = ""
    @error_output = ""
    @exec_time = 0
    @cmd = cmd
    @cmd_id = cmd_id
  end

  def run
    start_time = Time.now
    begin
      $LOG.info "[#{@cmd_id}] Starting command: #{@cmd}..."
      Open3.popen3(@cmd) do |stdin, stdout, stderr|
        @output = stdout.read
        @error_output = stderr.read
      end
    rescue Exception => e
      @error_output += e.to_s
    ensure
      @exec_time = Time.now - start_time
      $LOG.info "[#{@cmd_id}] Command completed in #{@exec_time} seconds."
    end
  end

  # Log the stdio and stderr outputs
  def log_results
    $LOG.info "[#{@cmd_id}] #{@cmd}:"
    if @error_output.length > 0
      @error_output.split("\n").each { |line| $LOG.error "[#{@cmd_id}]  #{line}" }
    end
    if @output.length > 0
      @output.split("\n").each { |line| $LOG.info "[#{@cmd_id}]  #{line}" }
    end
  end

  # Returns false if the command hasn't been executed yet
  def run?
    return @exec_time > 0
  end

  # Returns true if the command was successful.
  def success?
    return @error_output.length == 0
  end
end

# Define for each host which user accounts are being backed up and which files should be excluded
default_excludes = ['.Trash', 'Downloads', 'Desktop', 'Music/iTunes/iTunes Music/Podcasts',
                    'Library/Caches', 'Library/Logs']
# format: hostname => { username => [excluded_files] }
HOSTS={ 'host1' => { 'user1' => default_excludes },
        'host2' => { 'user2' => default_excludes, 'user1' => default_excludes },
        'host3' => { 'user2' => default_excludes + ['Music/iTunes']},
        'host4' => { 'user3' => default_excludes }
      }

$LOG.info "****** Backup started... *******"

# Make a ZFS snapshot
snapshot_name = "#{ZFS_POOL}@backup-#{Time.now.strftime('%y-%m-%d_%H:%M')}"
$LOG.info "Creating ZFS snapshot #{snapshot_name}"
`zfs snapshot #{snapshot_name}`

pending_commands = {}
HOSTS.each do |hostname,user_data|
  $LOG.info "Calling #{hostname} ..."
  if Ping.pingecho(hostname)
    user_data.each do |user,excluded_files|
      exclude_file = Tempfile.new("tempfile")
      excluded_files.each { |filepath| exclude_file << filepath << "\n" }
      exclude_file.close
      user_hostname = "#{user}@#{hostname}"
      $LOG.info "Backing up #{user_hostname} ..."
      local_backup_path = "#{LOCAL_BACKUP_PATH}/#{hostname}/#{user}"
      FileUtils.mkdir(local_backup_path) unless File.exists? local_backup_path
      command = "rsync -#{DRY_RUN ? 'n' : ''}avz --delete --partial --exclude-from=#{exclude_file.path} #{user_hostname}: #{local_backup_path}/"
      rsync = ExecCmd.new(command, user_hostname)
      pending_commands[user_hostname] = rsync
      Thread.new do
        rsync.run
      end
    end
  else
    $LOG.warn "#{hostname} does not respond!"
  end
end

# Wait for the backup processes to complete
while pending_commands.length > 0
  pending_commands.each do |user_hostname, exec_cmd|
    if exec_cmd.run?
      exec_cmd.log_results
      pending_commands.delete(user_hostname)
    end
  end

  if pending_commands.length > 0
    $LOG.info "Still #{pending_commands.length} tasks backing up #{pending_commands.keys.join(', ')}"
    sleep 60
  end
end

$LOG.info "****** Backup complete. *******\n"

What it does

You can use the command line argumens –dry-run and –stdout. The first one will call rsync with the –dry-run option and the second one will write to stdout instead of a logfile.

On start the script looks for other instances of itself and will kill them and all orphaned rsync child processes.

It will create a ZFS snapshot of the target pool with the current time and date as a label.

Then it will ping all the hosts defined in HOSTS and will construct the rsync command with all the excluded files and the defined users and start a separate thread in which the command will be executed.

Then it will loop until all the rsync tasks have been completed.

The logfile is /var/log/backup_rb.log and looks like this:

I, [15.10.09 11:44:11#7343]  INFO -- : ****** Cleaning up... *******
I, [15.10.09 11:44:11#7343]  INFO -- : Killing old backup processes 7316,7325,7332,7328,7334
I, [15.10.09 11:44:11#7343]  INFO -- : ****** Backup started... *******
I, [15.10.09 11:44:11#7343]  INFO -- : Creating ZFS snapshot daten@backup-09-10-15_11:44
I, [15.10.09 11:44:11#7343]  INFO -- : Calling host1 ...
I, [15.10.09 11:44:11#7343]  INFO -- : Backing up user2@host1 ...
I, [15.10.09 11:44:11#7343]  INFO -- : [user2@host1] Starting command: rsync -avz --delete --partial --exclude-from=/tmp/tempfile20091015-7343-1hpu6rm-0 user2@host1: /daten/host1/user2/...
I, [15.10.09 11:44:11#7343]  INFO -- : Calling host2 ...
I, [15.10.09 11:44:11#7343]  INFO -- : Backing up user1@host2 ...
I, [15.10.09 11:44:11#7343]  INFO -- : [user1@host2] Starting command: rsync -avz --delete --partial --exclude-from=/tmp/tempfile20091015-7343-1f9jzvu-0 user1@host2: /daten/host2/user1/...
I, [15.10.09 11:44:11#7343]  INFO -- : Calling host3 ...
W, [15.10.09 11:44:16#7343]  WARN -- : host3 does not respond!
I, [15.10.09 11:44:16#7343]  INFO -- : Calling host4 ...
I, [15.10.09 11:44:16#7343]  INFO -- : Backing up user2@host4 ...
I, [15.10.09 11:44:17#7343]  INFO -- : [user2@host4] Starting command: rsync -avz --delete --partial --exclude-from=/tmp/tempfile20091015-7343-yv60ta-0 user2@host4: /daten/host4/user2/...
I, [15.10.09 11:44:17#7343]  INFO -- : Backing up user1@host4 ...
I, [15.10.09 11:44:17#7343]  INFO -- : [user1@host4] Starting command: rsync -avz --delete --partial --exclude-from=/tmp/tempfile20091015-7343-644sq6-0 user1@host4: /daten/host4/user1/...
I, [15.10.09 11:44:17#7343]  INFO -- : Still 4 tasks backing up user1@host2, user1@host4, user2@host4, user2@host1
I, [15.10.09 11:45:17#7343]  INFO -- : Still 4 tasks backing up user1@host2, user1@host4, user2@host4, user2@host1
I, [15.10.09 11:45:41#7343]  INFO -- : [user2@host4] Command completed in 84.087238
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] rsync -avz --delete --partial --exclude-from=/tmp/tempfile20091015-7343-yv60ta-0 user2@host4: /daten/host4/user2/:
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] receiving file list ... done
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] Dropbox/
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] Library/Application Support/SyncServices/Local/
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] Library/Application Support/SyncServices/Local/admin.syncdb
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] Library/Application Support/SyncServices/Local/TFSM/com.apple.Calendars/
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] Library/Application Support/SyncServices/Local/clientdata/120c2b27e9ab530b442181ced8799e35b30c85cb/
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] Library/Application Support/SyncServices/Local/conflicts/
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] Library/Calendars/
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] Library/Calendars/Calendar Cache
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] Library/Calendars/Calendar Sync Changes/
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] Library/Calendars/FE3DF9D9-8D76-4F44-973A-525E02717BFE.calendar/Info.plist
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] Library/Logs/Sync/syncservices.log
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] Library/Mail/
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] Library/Preferences/
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] Library/Preferences/iCalExternalSync.plist
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4]
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] sent 13772 bytes  received 1001023 bytes  13440.99 bytes/sec
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] total size is 64974563109  speedup is 64027.28
I, [15.10.09 11:46:17#7343]  INFO -- : Still 3 tasks backing up user1@host2, user1@host4, user2@host1
I, [15.10.09 11:47:17#7343]  INFO -- : Still 3 tasks backing up user1@host2, user1@host4, user2@host1

Run in crontab

Finally I added a new entry of the root user crontab with “crontab -e” which will start the script every hour.

0 * * * * /usr/bin/ruby /root/backup.rb