Archiware P5 and Synology NAS.

Update: As of version 5.4.3 there is an official P5 add-on package for Synology NAS

Archiware P5 available for Synology

Note: The P5 app for Synology NAS first debuted with P5 v.5.3.3

On the Archiware P5 new-features page there’s a blurb about the Synology NAS integration:

From Version 5.3.3, Archiware P5 supports Synology NAS devices without restrictions.  

Synology NAS can serve as a data source or target for P5 Synchronize, P5 Backup and P5 Archive. The Archiware P5 application can now be installed on the Synology NAS itself.

Thanks to the snapshot capability of the DSM platform, powerful enterprise Synology NAS devices can also be used as repository for Backup2Go. This setup opens the possibility of introducing a professional data security solution at an affordable price point.

Let’s look in closer detail how to install Archiware P5 on a new Synology NAS.

For this post I have a new Synology 1515+ NAS, installed with five 6TB hard drives (It is very easy to install hard drives. No tools required). Note: I’ve purchased the NAS with my own money and was not paid to write this article.

At the time of this blog post the latest Synology DSM release is 6.1 and Archiware P5 is at version 5.4.2.

Step 1. Download Synology package from Archiware.com/download

Download Archiware P5 for Synology

 

awpst542spk

Requirements are DSM 5.2+ and Intel x86 64-Bit CPU only. (i.e. Atom but not Marvell).

Step 2. Find and Log into your NAS

Find your new NAS with the Synology Assistant app or use this handy website link:

Find your NAS

I had no luck with the app (it found my existing NAS, but not the new one). Using the website I was able to quickly locate the new NAS that I need to log into and setup. Very nice feature.

synology-1515-setup-welcome2crop

Step 3. Install the new DSM

Install or update new software. You will be prompted to go through the initial setup to prepare your new NAS.

synology-1515-install-diskstation-manager2

Step 4. Set up a new volume

Chose the Btrfs or ext4 filesystem. Btrfs supports snapshots, replication, and much more.

synology-1515-btrfs-setup

Step 5. Monitor the volume setup

Verifying the hard disks will take a moment. Take a break here.

synology-1515-storage-manager

Step 6. Open Package Center

packagecenter

Step 7. Install manually

Install Archiware P5 by selecting the “install manually” option to upload the awpst542.spk downloaded file from archiware.com

synology-1515-archiware-p5-package-center-upload

Step 8. Agree to continue.

Load the Synology P5 installer by agreeing to continue with this “unknown” publisher.

synology-1515-archiware-p5-package-center-unknown

Sep 9. Agree to trust the installer

synology-1515-archiware-p5-package-center-license

Step 10. Confirm the Install

synology-1515-archiware-p5-package-center-confirm-install

Step 11. P5 is now running on the Synology NAS.

Hooray! P5 is now installed. Select the app to examine the details.

synology-1515-archiware-p5-package-center-installed

synology-1515-archiware-p5-package-center2

 

Step 12. Examine the option to stop or uninstall the P5 application

synology-1515-archiware-p5-package-center-stop-uninstall

Step 13. Login to the P5 server running on NAS

To login to P5 open a new tab. Pay attention to the port number: “20,000” (vs 8000 on other platforms such as Solaris, Linux, OSX etc).

synology-1515-archiware-p5-port

 

Step 14. Set up your NAS as a client on another Server

To test the new Synology 1515+ NAS I then set up the NAS as a client on another P5 server, and set up a P5 Sync job to copy data from server with a ZFS based filesystem to the Synology NAS with a btrfs volume.

Testing: Set up the new client in P5 with a name and IP address, then set up a new sync job with source and destination. Start now. Watch the bits fly through the ether. Be happy.

Step 15. Other things to configure

To make your new NAS is working smoothly don’t forget to set up the email notifications, and set up some AFP, SMB, or NFS shares as required.

Take some time to explore the Package Center app and see what other great applications are offered on the Synology NAS.

Synology makes a great low-cost NAS appliance. For SMB or production setups I would recommend two or more (for redundancy, hot or cold spares, disaster recovery, offsite backups/replication). With P5 installed you can Sync your server data to a NAS for onsite or offsite backups, backup your NAS to tape, or use the NAS for your client workstation backups using Backup2Go. Using the new Btrfs filesystem provides many of the same advances as ZFS, including snapshots and replication, over traditional filesystems such as ext4 and hfs which sadly lack these features.

Conclusion:

The Synology NAS is a great experience. Adding Archiware P5 is a recommended way to include this NAS as part of any good backup, archive or DR (disaster recovery) scenario. Two thumbs up. Way up.

References:

Archiware P5 new features

Synology DSM

Camera Archives

For editing clients with a proper SAN this is the setup I like to use a watch folder on the SAN that sends to tape the camera archives automatically on a timed interval.

This requires

  1. proper SAN
  2. a watch folder setup with Archiware P5 archive
  3. camera archives, created in FCP X from the camera cards

Note: you can also use Adobe Prelude, Shotput or Resolve to create verified copies of camera cards. Use what you trust and works for you. The idea is not to copy by hand and avoid the perils of corrupt files.

Bonus: multiple drives to enable multi-streaming and parallelizing of your data. Why not makes cloned tapes copies and stream lots of data to all four (yes, four !) drives. See the illustration below.

 

 

Screen Shot 2016-08-03 at 10.35.40 AM

Watchman Monitoring + Archiware P5

I’ve been a little busy lately. I’m working on some scripts for Watchman Monitoring that alert when Archiware P5 needs attention. It’s really much more exciting than it sounds. 🙂

WatchmanArchiwareP5

Archiware P5 plugin (included with Watchman Client 6.6.0)

UPDATE: The Archiware P5 plugin is now included with the Watchman Monitoring client version 6.6.0

Use the link above to read up about Watchman Monitoring and the Archiware P5 plugin.

This plugin is now part of Watchman Monitoring thanks to Allen and his team! Of course, big thanks to a lot of help from Python magician and MacDevOps:YVR colleague, Wade Robson. I couldn’t have finished this plugin without his help. Merci, mon ami. (Early help to get started with this project is thanks to Scott Neal, automation expert and programming wizard. Thank you so much Scott, and thanks for the tasty Portland beer!).

Watchman Monitoring is a group of plugins that will warn when drives are failing, computers have restarted unexpectedly or backups are not running. All reporting goes to a beautiful web interface in the cloud which can keep a history of plugin issues. Watchman allows for integration with ticket systems and multiple users including clients and IT staff that can keep track of what’s up with their workstations, and servers.

Watchman Monitoring helps me keep tabs of major issues at all my clients before they become disasters. I even use it in discovery for new clients to see what issues exist but are ignored or unknown.

Since I set up a lot of SAN storage for my clients, and I use Archiware P5 for backups and archives I realized I needed to write a plugin for Watchman Monitoring that alerts me to issues. Instead of remoting in with VPN to each and every client every day to check on backups the only alternative is to automate it. These scripts watch the LTO tape drives and emails when they need cleaning, or warns when running jobs need tapes, if workstations haven’t backed up in a while or if tape pools need more tapes. And in Beta 2 we’ve added a check to see if the P5 maintenance support needs to be renewed to give you time to renew it before it expires. As well as better alerts for issues with running jobs, and lots of bug fixes.

We have it working on Mac servers running Archiware P5 and the next step is Linux, and the Unix family. Later on, Watchman will port it to Windows. The scripts are written in Python which is great for portability (except to Windows. Ha ha). And the P5 Watchman plugins should eventually run everywhere that Archiware P5 runs (OS X, Linux, FreeBSD, Solaris and Windows).

The best part of writing plugins for Watchman Monitoring is the great help that Allen and the whole team at Watchman have given us been throughout our development of these Archiware P5 plugins. And of course everyone at Archiware and Mike at PVT have been super helpful in explaining the use of the nsdchat cli for Archiware P5, even going so far as to add some features we needed to nsdchat when we explained how useful they’d be for this project. Mille mercis. Vielen danke.

Using GitHub to check code in, document business logic, write code, build a wiki and then track issues that need bug fixes or enhancement requests has been an adventure. It all starts with an problem that you want to be alerted for. It’s easy enough to add custom plugins to Watchman Monitoring you just need some ideas, a programmer (or two) and some time for testing, debugging, more testing and time. Did I mention you need lots of time? Ha ha

And now for a sneak peak of the Archiware P5 beta 2 plugins for Watchman Monitoring.

  1. Watchman nicely lists the new warnings and expirations for quickly getting to the issues you need to see.             Watchman Monitoring Archiware P5 warnings expiration X
  2. Expirations are tracked with Watchman. In this case we note the date when the maintenance for Archiware P5 needs to be renewed. Don’t want to miss that! Watchman Monitoring Archiware P5 Expirations plugin Xpng
  3. Server info is good to know. Uptime, port used, and what exactly is licensed.         Watchman Monitoring Archiware P5 Info plugin X
  4. The LTO tape drive is the heart of any tape library, and alerting when it needs cleaning is very important.                                               Watchman Monitoring Archiware P5 Devices plugin X
  5. Other plugins watch the tape pools, running and completed jobs, as well as Backup2Go (workstation backup).

Watchman Monitoring Archiware P5 B2Go plugin X

Watchman Monitoring Archiware P5 Pools plugin X

Watchman Monitoring Archiware P5 Jobs plugin X

Automatic Archive in Archiware P5

Let’s say you work in the media and entertainment industry, perhaps in Post production, and maybe, just maybe, you shoot a lot of digital film (R3D, Arri RAW, XDCAM, etc), and just maybe you have a SAN. And maybe you’re lucky and you set up a nice backup system using Archiware P5, for example. What about archive? What about finished projects, what do you do about that? Even more importantly, what do you do about the camera archives? That’s the digital film footage that comes in and gets copied to the SAN before creative work begins… it seems like important stuff. Very important to backup, and even better, to archive! How can we set up an automatic archive in Archiware P5?

DEFINITION: What is the difference between backup and archive? In the media and entertainment industry I would define backup as a continuous data protection of live data on a production volume, while archive is a copy of a finished project or original media that will be removed from the production volume and must kept safe for future retrieval. Backups will roll over (a new full backup every week, or month) and if that is all we had, then footage or projects that are done and gone off the SAN would be then lost. Archive separates out the finished projects or source material as needing an independent safekeeping. P5 Archive also has the option of creating a mini-MAM type database of proxy files that can be easily viewed in a web browser for quick identification of files to be restored.

So, how do we set up an automatic archive in Archiware P5? We want it to be automatic so we don’t have to think about it, since manually archiving like backups can be forgotten. If we set up a watch folder on the SAN then we just have to instruct everyone to drop their camera masters and other source files into the folder when they copy to the SAN and P5 will automatically archive them to an incremental tape archive. Wow. That’s awesome.

Tips and Tricks: If you edit with Final Cut Pro X then I recommend using it to make Camera Archives (a verified copy of the footage from the original card or drive) then placing this in the watch folder. If you’re using Adobe’s Premiere workflow, then Prelude can make a verified copy as well, but not in the same way exactly. This will be the subject of another post. Stay tuned.

What does an automatic archive look like?

P5 Archive watch folder

P5 Archive watch folder

Requirements:

  1. Pool (designated tapes, or a disk, for archive)
  2. Index (could be the archive default index, or a unique new index)
  3. Plan (an archive plan specifies the pool and index used as well as the what and how)

P5 Archive General Setup Screen Shot 2014-05-13 at 1.46.50 PM

Tips and Tricks: If you’re generating QT previews then you’ll want to move your index off of the default, or else you might fill up your system hard drive with movie files and then your backup server won’t boot. No, I never done this. OK, trust me it will happen 🙂

Tips and Tricks: Make sure to set up a backup job of the Archive index. This is a safety measure. The archive index is not saved to the tape in the same way backup jobs information. You need to create a backup job specifically to save your archive index. But you’re already running a backup job to backup your Archiware main index, right? Uh huh, thought so.

Tips and Tricks: When creating the archive pool set it at 512KB media block size for faster archiving of big video files.

Archive Index:

Use the advanced options to create a new Archive Index, select its location and optionally create new additional fields to help searching for archived projects.

P5 Archive Db config Screen Shot 2014-05-13 at 1.45.53 PM

Archive options:

Incremental or full? QT previews, yes or no? For an automatic archiving set up I suggest incremental archive and no QT previews. Using previews is up to you (and your disk space available), so plan accordingly.

P5 Archive Options Screen Shot 2014-05-13 at 1.46.59 PM

P5 Archive Preview Gen Screen Shot 2014-05-13 at 1.47.02 PM

If you have any questions or need help setting this up please do not hesitate to contact me. In the meantime, I’ll work on a nice PDF doc to summarize the setup. Download a demo of Archiware P5 and give it a whirl. You’ll see that it’s super easy to create an automatic archive location on your SAN and your editors will thank you when you can restore their files that they need (when they need it!).