Sign in to follow this  
Followers 0
lattera

ZFS + Rsync

3 posts in this topic

My employer's flagship product generates thousands of PDFs. We have three different copies of our product: a development copy for Quality Assurance testing of recently-written code, a staging copy to test pushing a new versions of our product, and a production copy that our users utilize. Each copy requires 42GB worth of PDFs. New PDFs are generated every day.

To maintain a sane development environment, we pull fresh copies of the PDFs every month from production. We pull from production to staging, then from production to development. The PDFs are stored on two separate servers that both run NTFS. We use Microsoft SyncToy to sync the PDFs across environments. The process can take several hours for each environment. The network load is high due to the PDFs being stored on multiple servers.

I recently had an idea. What if we store the PDFs on our ZFS NAS? We could use ZFS snapshotting and rsync to refresh the environments. We can do that on a regular basis via a cron job. ZFS snapshots take a few seconds and rsync is a really efficient tool. No network traffic will be involved since the synchronization is taking place all on the same server.

Here are the commands we would run:


DATE=`date '+%F_%T'`
zfs snapshot tank/site_data/prod/PDFs@$DATE
rsync -a /tank/site_data/prod/PDFs/.zfs/snapshot/$DATE/ /tank/site_data/dev/PDFs/

I really like this solution. Right now, we have to jump through a lot of hoops to sync up these PDFs. This will save us time, space, and internal bandwidth.

This article originally posted on my tech blog.

0

Share this post


Link to post
Share on other sites

rsync would have been my first choice with a *NIX system. I use it to keep files synced between my home server/home desktop and my laptop.

0

Share this post


Link to post
Share on other sites

Here is the final shell script:


#!/usr/bin/bash

function snapshot {
echo [*] Snapshotting $1 @ $2
echo zfs snapshot tank/shares/cifs/CompanyData/$1@$2
echo " [+] Snapshot of $1 @ $2 done"
}

function sync {
echo [*] Syncing $1 with snapshot...
echo rsync -aA /tank/shares/cifs/CompanyData/$2/.zfs/snapshot/$3/ /tank/shares/cifs/CompanyData/$1/
echo " [+] Syncing $1 done!"
}

if [ $# -eq 0 ]; then
DATE=`date '+%F_%T'`

snapshot Prod $DATE
sync Dev Prod $DATE
sync Alpha Prod $DATE
elif [ $# -eq 1 ]; then
DATE=`date '+%F_%T'`

snapshot Prod $DATE
sync $1 Prod $DATE
elif [ $# -eq 2 ]; then
DATE=`date '+%F_%T'`

snapshot $2 $DATE
sync $1 $2 $DATE
fi

0

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0