2 replies to this topic

[lattera]

My employer's flagship product generates thousands of PDFs. We have three different copies of our product: a development copy for Quality Assurance testing of recently-written code, a staging copy to test pushing a new versions of our product, and a production copy that our users utilize. Each copy requires 42GB worth of PDFs. New PDFs are generated every day.

To maintain a sane development environment, we pull fresh copies of the PDFs every month from production. We pull from production to staging, then from production to development. The PDFs are stored on two separate servers that both run NTFS. We use Microsoft SyncToy to sync the PDFs across environments. The process can take several hours for each environment. The network load is high due to the PDFs being stored on multiple servers.

I recently had an idea. What if we store the PDFs on our ZFS NAS? We could use ZFS snapshotting and rsync to refresh the environments. We can do that on a regular basis via a cron job. ZFS snapshots take a few seconds and rsync is a really efficient tool. No network traffic will be involved since the synchronization is taking place all on the same server.

Here are the commands we would run:

DATE=`date '+%F_%T'`
zfs snapshot tank/site_data/prod/PDFs@$DATE
rsync -a /tank/site_data/prod/PDFs/.zfs/snapshot/$DATE/ /tank/site_data/dev/PDFs/

I really like this solution. Right now, we have to jump through a lot of hoops to sync up these PDFs. This will save us time, space, and internal bandwidth.

This article originally posted on my tech blog.

[systems_glitch]

rsync would have been my first choice with a *NIX system. I use it to keep files synced between my home server/home desktop and my laptop.

[lattera]

Here is the final shell script:

#!/usr/bin/bash

function snapshot {
        echo [*] Snapshotting $1 @ $2
        echo zfs snapshot tank/shares/cifs/CompanyData/$1@$2
        echo "    [+] Snapshot of $1 @ $2 done"
}

function sync {
        echo [*] Syncing $1 with snapshot...
        echo rsync -aA /tank/shares/cifs/CompanyData/$2/.zfs/snapshot/$3/ /tank/shares/cifs/CompanyData/$1/
        echo "    [+] Syncing $1 done!"
}

if [ $# -eq 0 ]; then
        DATE=`date '+%F_%T'`

        snapshot Prod $DATE
        sync Dev Prod $DATE
        sync Alpha Prod $DATE
elif [ $# -eq 1 ]; then
        DATE=`date '+%F_%T'`

        snapshot Prod $DATE
        sync $1 Prod $DATE
elif [ $# -eq 2 ]; then
        DATE=`date '+%F_%T'`

        snapshot $2 $DATE
        sync $1 $2 $DATE
fi