Updating the P5 client on the Jellyfish

You’ve successfully installed Archiware’s P5 backup and archiving software on your backup server following my previous blog posts and after it has run smoothly for a while you decided to upgrade the version of P5 on your server, but how do you do this on the Lumaforge Jellyfish storage? I’m glad you asked.

There are a couple of ways to update your P5 agent, and I will show you the built in way in Archiware’s P5 software. Surprisingly after many years of using P5 I have never used this method before. I’ve been using Munki for years to upgrade all software on my Mac clients including P5 and on Linux and Solaris servers I’ve just done it by hand. Install over top of the previous version and voila upgrade! But what if you didn’t want to ssh in as root and just install over top, what if there was a better way? I present to you the official “Update client” dialog box. It’s nice.

Update-p5-jellyish-1

Updating client software assumes you’ve set up clients in the P5 server clients section, This is needed when you want to use these server agents to designate their attached storage as a backup, archive or sync source. And also, this assumes you’ve updated your server.

P5 client update Jellyfish 2 Screen Shot 2018-08-06 at 4.40.03 PM

During the update process there are some nice dialog boxes to let you know what is happening.

P5 client update Jellyfish 4 Screen Shot 2018-08-06 at 4.44.34 PM

And afterwards you can test your client with a Ping test.

P5 client update Jellyfish 3 Screen Shot 2018-08-06 at 4.44.27 PM

Success! Looks like we’ve updated our client successfully. How wonderful. And no need to mess about in Terminal with a root shell. No telling what kind of trouble we could get into with those elevated privileges…. much safer this way.

Thanks Archiware for making this great software. I depend on it every day.

 

 

P5 on the Jellyfish: Archiving Gotchas

TL;DR

Using Archiware P5 to Archive files to tapes is awesome, but watch out for little things you might miss, such as the path to the files and backing up your Archive Db.

P5 Archive on the Jellyfish

Using P5 Archive with the Lumaforge Jellyfish is a great way to preserve your digital archives. See this post for how to set up P5 on the Jellyfish

Using Archiware P5 for archiving makes sense. You want your completed projects and original camera footage on LTO tape. But how do you do archives? There are several different ways, and there be gotchas.

P5 Archive vs P5 Archive app

Using P5 Archive to manually archive completed projects to LTO tape is a process of logging into the server via a web browser and selecting the the project folder you want to archive to tape.

The completed project folder could be on the storage visible to the server or it could be storage the client sees. And that can make a difference. Where the storage is mounted is different on a Mac vs Linux. Its’ the difference between “/Volumes” and “/mnt”.

The same Jellyfish storage, either SMB or NFS, when seen on a Mac is mounted by default at “/Volumes” (this can be changed but for most people leave it at the default). But when archiving the storage via a Jellyfish client you will get “/mnt” path.

p5-smb-test2.png

Using the P5 Archive app, which is a Mac only companion application to P5 Archive, to run the archives you will see the storage archived as “/Volumes”.

This first Archiving gotcha is if you’re archiving the Jellyfish storage with the web application of P5 Archive you will have to find your footage and restore from the “/mnt” path vs if you’re archiving from the P5 Archive app which is running from a Mac and will see and store the footage using the “/Volumes” path.

All this to say that using both ways to archive may double up your footage in your archive which may be unintended. And from a restore in the web browser finding your footage may be confusing if you’re used to seeing it mounted in “/Volumes” and you actually find it under “/mnt”.

Note: the reason to use the P5 Archive app is because of the simplicity of right-clicking files in the finder which are on your storage and telling them to archive right then and there. Files are copied to tape then the original files on the storage are replaced with stub files. Right-click again to restore. Simple.

p5-archive-app-job-monitor.png

Backup your Archive!

Don’t forget to backup your archives. Or rather, your archive Db. A more recent addition is the ability to automate the backups on the Archive index, so don’t forget to enable it.

In the managed index section, choose your Archive index.

Set the target client where the backups are going and the backup directory. Choose a time and don’t forget to enable it (check the checkbox and hit apply before closing the windows).

Note: Repeat this setup for each Archive index you want to backup.

Archive Backup db setup3.png

Monitoring your Archive!

Don’t forget to enable email notifications for your P5 server to get your inbox full of status notifications and errors and other important stuff. But if you want to cut down on email notifications or you have multiple P5 servers (many different clients, perhaps), then you might want to check out Watchman Monitoring and the P5 plugin that is built-in). Find out easily when your tape pools are getting low, the tape drives needs to be cleaned, the support maintenance needs renewing etc. All in one dashboard. How convenient!

Maybe everything is going well…

Watchman-P5-info.png

Or maybe not!

Archiware-P5-Jobs-Watchman-tapes-required.png

 

Install P5 on the Jellyfish

TL;DR

You can easily install Archiware P5 backup and archive software on a Lumaforge Jellyfish storage server. Once you’ve done that you can backup to tape or disk or the cloud directly or through another P5 server. Backups are good. Archive are good. Restores are better.

P5 install on the Jellyfish (Linux) How-To:

Note: Thank you to Lumaforge’s CTO Eric Altman who gave me some basic instructions to get me going.

Step One: Download the latest Linux P5 rpm file 

http://p5.archiware.com/download

p5-Linux-rpm.png

Copy the downloaded rpm file to the root folder of your SMB or NFS file share.

 

Step Two: Install the rpm file

Open Terminal and ssh into your Jellyfish. Login as root or as another appropriate user.

yum localinstall /mnt/Primary/ShareSMB/awpst554.rpm

 

Step Three: Browse to server on port 8000 to test that the server is up

e.g. https://jellyfish:8000

Or in Terminal and ssh into your Jellyfish and ping your P5 server

cd /usr/local/aw 

./ping-server

Pinging PresStore application servers...

  lexxsrv pid: 4840 (server is running)

  lexxsrv url: http://127.0.1.1:8000/login 

Pinged 1 from 1 application servers.

 

Step Four: Decide if the Jellyfish storage will be a P5 client or a server.

Note: If configuring the Jellyfish storage as the main P5 server you may wish to set up a user that only has access to the shared volumes.

For my set up the Jellyfish storage is going to act as a P5 client to a main P5 server on a Mac mini (yes, they are useful for something). The Mac mini is this case is the P5 server and is attached to theOverland tape library via a Promise SANlink2 Thunderbolt Fibre Channel adapter.

NEOs-T24-large-new.jpg

macmini-ports.png

 

Step Five: Set up the Jellyfish storage as a P5 client

Log into your P5 server and add the Jellyfish by the IP known to the P5 server. In this case the P5 server is connected via 1GB to the Jellyfish in Port 1.

P5 clients jellyfish setup1.png

Note: You could also choose to plug into the Jellyfish via a 10GB port, but in my setup these 10GB ports are reserved for the edit stations. You should choose what’s appropriate for your setup.

P5 clients jellyfish setup2.png

Resource utilization of P5 on the server is low, topping off generally at 1GB of RAM at peak usage. While this does technically take resources from ZFS caching, the impact should be super minimal.

In my observations the CPU never spiked too high while both serving NFS and SMB mount points to multiple Final Cut Pro X workstations even with backups or archive jobs going to tape at the same time.

jellyfish-cpu-resources-graph.png

More Jellyfish P5

See the follow up post on Archiving gotchas with the Jellyfish here

 

Camera Archives

For editing clients with a proper SAN this is the setup I like to use a watch folder on the SAN that sends to tape the camera archives automatically on a timed interval.

This requires

  1. proper SAN
  2. a watch folder setup with Archiware P5 archive
  3. camera archives, created in FCP X from the camera cards

Note: you can also use Adobe Prelude, Shotput or Resolve to create verified copies of camera cards. Use what you trust and works for you. The idea is not to copy by hand and avoid the perils of corrupt files.

Bonus: multiple drives to enable multi-streaming and parallelizing of your data. Why not makes cloned tapes copies and stream lots of data to all four (yes, four !) drives. See the illustration below.

 

 

Screen Shot 2016-08-03 at 10.35.40 AM

Automatic Archive in Archiware P5

Let’s say you work in the media and entertainment industry, perhaps in Post production, and maybe, just maybe, you shoot a lot of digital film (R3D, Arri RAW, XDCAM, etc), and just maybe you have a SAN. And maybe you’re lucky and you set up a nice backup system using Archiware P5, for example. What about archive? What about finished projects, what do you do about that? Even more importantly, what do you do about the camera archives? That’s the digital film footage that comes in and gets copied to the SAN before creative work begins… it seems like important stuff. Very important to backup, and even better, to archive! How can we set up an automatic archive in Archiware P5?

DEFINITION: What is the difference between backup and archive? In the media and entertainment industry I would define backup as a continuous data protection of live data on a production volume, while archive is a copy of a finished project or original media that will be removed from the production volume and must kept safe for future retrieval. Backups will roll over (a new full backup every week, or month) and if that is all we had, then footage or projects that are done and gone off the SAN would be then lost. Archive separates out the finished projects or source material as needing an independent safekeeping. P5 Archive also has the option of creating a mini-MAM type database of proxy files that can be easily viewed in a web browser for quick identification of files to be restored.

So, how do we set up an automatic archive in Archiware P5? We want it to be automatic so we don’t have to think about it, since manually archiving like backups can be forgotten. If we set up a watch folder on the SAN then we just have to instruct everyone to drop their camera masters and other source files into the folder when they copy to the SAN and P5 will automatically archive them to an incremental tape archive. Wow. That’s awesome.

Tips and Tricks: If you edit with Final Cut Pro X then I recommend using it to make Camera Archives (a verified copy of the footage from the original card or drive) then placing this in the watch folder. If you’re using Adobe’s Premiere workflow, then Prelude can make a verified copy as well, but not in the same way exactly. This will be the subject of another post. Stay tuned.

What does an automatic archive look like?

P5 Archive watch folder

P5 Archive watch folder

Requirements:

  1. Pool (designated tapes, or a disk, for archive)
  2. Index (could be the archive default index, or a unique new index)
  3. Plan (an archive plan specifies the pool and index used as well as the what and how)

P5 Archive General Setup Screen Shot 2014-05-13 at 1.46.50 PM

Tips and Tricks: If you’re generating QT previews then you’ll want to move your index off of the default, or else you might fill up your system hard drive with movie files and then your backup server won’t boot. No, I never done this. OK, trust me it will happen 🙂

Tips and Tricks: Make sure to set up a backup job of the Archive index. This is a safety measure. The archive index is not saved to the tape in the same way backup jobs information. You need to create a backup job specifically to save your archive index. But you’re already running a backup job to backup your Archiware main index, right? Uh huh, thought so.

Tips and Tricks: When creating the archive pool set it at 512KB media block size for faster archiving of big video files.

Archive Index:

Use the advanced options to create a new Archive Index, select its location and optionally create new additional fields to help searching for archived projects.

P5 Archive Db config Screen Shot 2014-05-13 at 1.45.53 PM

Archive options:

Incremental or full? QT previews, yes or no? For an automatic archiving set up I suggest incremental archive and no QT previews. Using previews is up to you (and your disk space available), so plan accordingly.

P5 Archive Options Screen Shot 2014-05-13 at 1.46.59 PM

P5 Archive Preview Gen Screen Shot 2014-05-13 at 1.47.02 PM

If you have any questions or need help setting this up please do not hesitate to contact me. In the meantime, I’ll work on a nice PDF doc to summarize the setup. Download a demo of Archiware P5 and give it a whirl. You’ll see that it’s super easy to create an automatic archive location on your SAN and your editors will thank you when you can restore their files that they need (when they need it!).