Setting up my Solaris server as a centralized backup server

After some months of work I now have the time to set-up my server properly so that it can backup all my computers without a hassle. Since I wanted to let the server control, when the backups should be made I wrote a Ruby script which runs every hour and backs up all the available hosts (which are of course Macs ;-). The script should not run at the same time and produce a decent logfile.

Set up environment

First I had to make sure, that the server had the correct time. By default the ntp daemon did not run, so I configured it using the description at the grey blog. I did not use the European ntp server though instead I used de.pool.ntp.org.

To install the current version 1.8.7 of Ruby I entered as root:

# pfexec pkg install SUNWruby18

Then I created a ssh key for my Solaris root user:

# ssh-keygen -t dsa
Enter file in which to save the key (/root/.ssh/id_dsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_dsa.
Your public key has been saved in /root/.ssh/id_dsa.pub.
The key fingerprint is:
*DISCLOSED*

To get a ssh connection which is needed for rsync to each of my hosts I then copied the contents of /root/.ssh/id_dsa.pub to the to the file authorized_keys e.g. user1@host1:.ssh/authorized_keys

Now whenever I enter “ssh user1@host1” no password is needed to get a remote shell.

The Ruby Script

#!/usr/bin/ruby
# This script will fetch the current files from a couple of hosts via rsync
# and stores them locally
require 'ping'
require 'tempfile'
require 'open3'
require 'logger'
require 'fileutils'

ZFS_POOL="your_zfs_pool_here"
LOCAL_BACKUP_PATH="/#{ZFS_POOL}"

# Parse the commandline parameters
ARGV.each do |arg|
  case
  when arg == '--stdout'
    LOG_STDOUT = true
  when arg == '--dry-run'
    DRY_RUN = true
  end
end
LOG_STDOUT = (LOG_STDOUT rescue false) # Set default value to false
DRY_RUN = (DRY_RUN rescue false)       # dto.

# Check which output should be used for logging
if LOG_STDOUT
  # Log to stdout
  $LOG = Logger.new($stdout)
  $LOG.datetime_format = '%H:%M:%S'
else
  # Logfile will not exceed 1 MB
  $LOG = Logger.new('/var/log/backup_rb.log', 0, 1 * 1024 * 1024)
  $LOG.datetime_format = '%d.%m.%y %H:%M:%S'
end

# Kill older processes of this script
pids_to_kill = []
`ps -Al -o pid -o args|grep -e ruby|grep -e #{__FILE__}`.split("\n").each do |line|
  other_pid = line.split(" ")[0].to_i
  if other_pid != $$
    pids_to_kill << other_pid
    `ps -Al -o pid,ppid=MOM -o args|grep "1 rsync"|grep -v grep`.split("\n").each do |child_line|
      child_pid = child_line.split(" ")[0].to_i
      pids_to_kill << child_pid
    end
  end
end

if pids_to_kill.length > 0
  $LOG.info "****** Cleaning up... *******"
  $LOG.info "Killing old backup processes #{pids_to_kill.join(",")}"
  `kill -9 #{pids_to_kill.join(" ")}`
end

# Execute a command and store its output
class ExecCmd
  attr_reader :output,:error_output,:cmd,:exec_time

  def initialize(cmd,cmd_id)
    @output = ""
    @error_output = ""
    @exec_time = 0
    @cmd = cmd
    @cmd_id = cmd_id
  end

  def run
    start_time = Time.now
    begin
      $LOG.info "[#{@cmd_id}] Starting command: #{@cmd}..."
      Open3.popen3(@cmd) do |stdin, stdout, stderr|
        @output = stdout.read
        @error_output = stderr.read
      end
    rescue Exception => e
      @error_output += e.to_s
    ensure
      @exec_time = Time.now - start_time
      $LOG.info "[#{@cmd_id}] Command completed in #{@exec_time} seconds."
    end
  end

  # Log the stdio and stderr outputs
  def log_results
    $LOG.info "[#{@cmd_id}] #{@cmd}:"
    if @error_output.length > 0
      @error_output.split("\n").each { |line| $LOG.error "[#{@cmd_id}]  #{line}" }
    end
    if @output.length > 0
      @output.split("\n").each { |line| $LOG.info "[#{@cmd_id}]  #{line}" }
    end
  end

  # Returns false if the command hasn't been executed yet
  def run?
    return @exec_time > 0
  end

  # Returns true if the command was successful.
  def success?
    return @error_output.length == 0
  end
end

# Define for each host which user accounts are being backed up and which files should be excluded
default_excludes = ['.Trash', 'Downloads', 'Desktop', 'Music/iTunes/iTunes Music/Podcasts',
                    'Library/Caches', 'Library/Logs']
# format: hostname => { username => [excluded_files] }
HOSTS={ 'host1' => { 'user1' => default_excludes },
        'host2' => { 'user2' => default_excludes, 'user1' => default_excludes },
        'host3' => { 'user2' => default_excludes + ['Music/iTunes']},
        'host4' => { 'user3' => default_excludes }
      }

$LOG.info "****** Backup started... *******"

# Make a ZFS snapshot
snapshot_name = "#{ZFS_POOL}@backup-#{Time.now.strftime('%y-%m-%d_%H:%M')}"
$LOG.info "Creating ZFS snapshot #{snapshot_name}"
`zfs snapshot #{snapshot_name}`

pending_commands = {}
HOSTS.each do |hostname,user_data|
  $LOG.info "Calling #{hostname} ..."
  if Ping.pingecho(hostname)
    user_data.each do |user,excluded_files|
      exclude_file = Tempfile.new("tempfile")
      excluded_files.each { |filepath| exclude_file << filepath << "\n" }
      exclude_file.close
      user_hostname = "#{user}@#{hostname}"
      $LOG.info "Backing up #{user_hostname} ..."
      local_backup_path = "#{LOCAL_BACKUP_PATH}/#{hostname}/#{user}"
      FileUtils.mkdir(local_backup_path) unless File.exists? local_backup_path
      command = "rsync -#{DRY_RUN ? 'n' : ''}avz --delete --partial --exclude-from=#{exclude_file.path} #{user_hostname}: #{local_backup_path}/"
      rsync = ExecCmd.new(command, user_hostname)
      pending_commands[user_hostname] = rsync
      Thread.new do
        rsync.run
      end
    end
  else
    $LOG.warn "#{hostname} does not respond!"
  end
end

# Wait for the backup processes to complete
while pending_commands.length > 0
  pending_commands.each do |user_hostname, exec_cmd|
    if exec_cmd.run?
      exec_cmd.log_results
      pending_commands.delete(user_hostname)
    end
  end

  if pending_commands.length > 0
    $LOG.info "Still #{pending_commands.length} tasks backing up #{pending_commands.keys.join(', ')}"
    sleep 60
  end
end

$LOG.info "****** Backup complete. *******\n"

What it does

You can use the command line argumens –dry-run and –stdout. The first one will call rsync with the –dry-run option and the second one will write to stdout instead of a logfile.

On start the script looks for other instances of itself and will kill them and all orphaned rsync child processes.

It will create a ZFS snapshot of the target pool with the current time and date as a label.

Then it will ping all the hosts defined in HOSTS and will construct the rsync command with all the excluded files and the defined users and start a separate thread in which the command will be executed.

Then it will loop until all the rsync tasks have been completed.

The logfile is /var/log/backup_rb.log and looks like this:

I, [15.10.09 11:44:11#7343]  INFO -- : ****** Cleaning up... *******
I, [15.10.09 11:44:11#7343]  INFO -- : Killing old backup processes 7316,7325,7332,7328,7334
I, [15.10.09 11:44:11#7343]  INFO -- : ****** Backup started... *******
I, [15.10.09 11:44:11#7343]  INFO -- : Creating ZFS snapshot daten@backup-09-10-15_11:44
I, [15.10.09 11:44:11#7343]  INFO -- : Calling host1 ...
I, [15.10.09 11:44:11#7343]  INFO -- : Backing up user2@host1 ...
I, [15.10.09 11:44:11#7343]  INFO -- : [user2@host1] Starting command: rsync -avz --delete --partial --exclude-from=/tmp/tempfile20091015-7343-1hpu6rm-0 user2@host1: /daten/host1/user2/...
I, [15.10.09 11:44:11#7343]  INFO -- : Calling host2 ...
I, [15.10.09 11:44:11#7343]  INFO -- : Backing up user1@host2 ...
I, [15.10.09 11:44:11#7343]  INFO -- : [user1@host2] Starting command: rsync -avz --delete --partial --exclude-from=/tmp/tempfile20091015-7343-1f9jzvu-0 user1@host2: /daten/host2/user1/...
I, [15.10.09 11:44:11#7343]  INFO -- : Calling host3 ...
W, [15.10.09 11:44:16#7343]  WARN -- : host3 does not respond!
I, [15.10.09 11:44:16#7343]  INFO -- : Calling host4 ...
I, [15.10.09 11:44:16#7343]  INFO -- : Backing up user2@host4 ...
I, [15.10.09 11:44:17#7343]  INFO -- : [user2@host4] Starting command: rsync -avz --delete --partial --exclude-from=/tmp/tempfile20091015-7343-yv60ta-0 user2@host4: /daten/host4/user2/...
I, [15.10.09 11:44:17#7343]  INFO -- : Backing up user1@host4 ...
I, [15.10.09 11:44:17#7343]  INFO -- : [user1@host4] Starting command: rsync -avz --delete --partial --exclude-from=/tmp/tempfile20091015-7343-644sq6-0 user1@host4: /daten/host4/user1/...
I, [15.10.09 11:44:17#7343]  INFO -- : Still 4 tasks backing up user1@host2, user1@host4, user2@host4, user2@host1
I, [15.10.09 11:45:17#7343]  INFO -- : Still 4 tasks backing up user1@host2, user1@host4, user2@host4, user2@host1
I, [15.10.09 11:45:41#7343]  INFO -- : [user2@host4] Command completed in 84.087238
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] rsync -avz --delete --partial --exclude-from=/tmp/tempfile20091015-7343-yv60ta-0 user2@host4: /daten/host4/user2/:
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] receiving file list ... done
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] Dropbox/
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] Library/Application Support/SyncServices/Local/
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] Library/Application Support/SyncServices/Local/admin.syncdb
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] Library/Application Support/SyncServices/Local/TFSM/com.apple.Calendars/
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] Library/Application Support/SyncServices/Local/clientdata/120c2b27e9ab530b442181ced8799e35b30c85cb/
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] Library/Application Support/SyncServices/Local/conflicts/
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] Library/Calendars/
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] Library/Calendars/Calendar Cache
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] Library/Calendars/Calendar Sync Changes/
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] Library/Calendars/FE3DF9D9-8D76-4F44-973A-525E02717BFE.calendar/Info.plist
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] Library/Logs/Sync/syncservices.log
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] Library/Mail/
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] Library/Preferences/
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] Library/Preferences/iCalExternalSync.plist
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4]
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] sent 13772 bytes  received 1001023 bytes  13440.99 bytes/sec
I, [15.10.09 11:46:17#7343]  INFO -- : [user2@host4] total size is 64974563109  speedup is 64027.28
I, [15.10.09 11:46:17#7343]  INFO -- : Still 3 tasks backing up user1@host2, user1@host4, user2@host1
I, [15.10.09 11:47:17#7343]  INFO -- : Still 3 tasks backing up user1@host2, user1@host4, user2@host1

Run in crontab

Finally I added a new entry of the root user crontab with “crontab -e” which will start the script every hour.

0 * * * * /usr/bin/ruby /root/backup.rb
Advertisements

I’m free!

After almost 21 years as an employee I decided to take the chance, quit my job as a Senior Software iPhone Developer in Berlin and now I am self-employed.

Feels very good to make my own decisions. I am planning to concentrate my contract work on iPhone and Mac software development, Adobe Flex and AIR and finally Ruby and Ruby on Rails. The contract situation looks very promising at the moment, so if everything works fine I can also work half of the time on my own iPhone application idea which pinched me for some time now.

As a result of being my own boss I already booked the ticket, flight and hotel for the upcoming Adobe MAX conference in L.A. in October. Since I visited the MAX conference in 2007 in Barcelona and have to think about my findings there all the time I thought it would be a good idea to get new insights into Adobe plans for the future.

On the other hand I already ordered tons of new hardware 🙂

I am typing this article on my new MacBook Pro 13″ with a 2.53 GHz processor. It arrived yesterday after I changed my order from the 250 GB HDD to a 320 GB one. They seem to have trouble with the huge success of this model. I’m planning to exchange the HDD with an OCZ Vertex 120 GB version and replace the superdrive with a OptiBay and a 500 GB HDD for in-time Time Machine backups and storing bigger files like movies. The setting is complete with a shiny new 24″ Cinema LED display and the bluetooth Mighty Mouse and keyboard.

I installed Snow Leopard from the DVD I got at the WWDC and it works very great although I have to live without 1password at the moment and Flex Builder wants to install Rosetta during install which I prefer not to do.

Time will tell if I will switch back to regular Leopard before I have full tool support.

Low power-consuming fileserver barebone

A couple of days ago Mr. Toto mentioned this barebone on Twitter. It has an Intel Atom 330 Dualcore CPU with hyperthreading and 5 HDD slots built-in:

linux300

Equiped with 1 GB RAM it will cost £372 (appr. 394,- €) including VAT and shipping to Germany. That’s almost double the price of the hand-built system and I don’t know if Solaris will run perfectly on it…

Shopping cart

Maybe someone else has already tried this out?

Building my ZFS-Fileserver

After a good amount of time I have come up with my preferred hardware setup for my fileserver. I have a couple of old harddrives laying around so I won’t need them. I also don’t want to build in a CD-ROM drive permanently. I will use an external one for setup.

The German PC Builder site of Alternate is very good and has some nice checks built-in to prevent an incompatible setup. I looked always for the cheapest components.

CPU

AMD Sempron64 LE-1250 Boxed, OPGA, “Sparta”

with bundled cooling system. Although it’s single-core it can run in 64-Bit mode and should not get too warm.

30,99 €

Power

Zalman ZM360B-APS

With 4 S-ATA power connectors.

49,99 €

Case

Sharkoon Rebel9 Economy-Edition

Plenty of space for 9 external 5.25″ drives in a midi sized tower(200 mm x 435 mm x 486 mm).

41,99 €

Case cooling

Arctic-Cooling AF8025 PWM

(3,99 x 2) 7,98 €

Mainboard

Asrock N61P-S

It has 4 S-ATA connectors on a NVIDIA nForce6. The build-in LAN is only 100-MBit, but I want to connect the fileserver via PowerLAN so that should be enough. Although it is not listed on Sun’s Solaris hardware compatibility list I try my luck with it.

36,49 €

RAM

Crucial DIMM 2 GB DDR2-667

The single-core CPU can only handle this type of RAM, but I guess that will be sufficient.

19,79 €

Complete

That’s it for the bare system.

187,14 €

From the site:

fileserver

Adobe Max Barcelona: Round-up Day 1

I attended the following sessions on my first day.

Working with Persistent Data in AIR

Although I already used the File API and the SQL API very intensively this session gave some new insights to the new synchronous SQL API and the new Encrypted Local Store which comes in handy.

Flex Roadmap

Some cool new features which might appear in the next version of Flex: primitive graphics. Another cool feature: the components are being split up into MVC parts so especially the View can be altered very extensively.

Local Database Access with AIR and Data Synchronisation Strategies

My second introduction to AIR and the SQL API 😦 Oh my, the presenter used most of the time telling the audience what it means that the SQL API can be used synchronously and asynchronously. Then 3 minutes before the end he remembers that there was another topic: Data synchronisation. That was the part that interested me. His only comment: “Well, data synchronisation is difficult”. That was very disappointing!

Developer or Programmer, tough question!

I have just read an article differentiating between the two job titles. I totally agree with the author although the title “Programmer” expresses much of my main work I am doing all kind of things besides that. However I would never call myself an architect since the only “Architects” I know are no real role-models for me.