Category Archives: Production

Hello Big Sur! See ya later Monterey

Posted on by
Reply

I am so happy to install macOS Big Sur 11.5.1, now that it is a ready for production. Have fun with macOS Monterey those of you on the bleeding edge. For media professionals using Xsan in production storage environments August is a great month to update to the soon to be yesterday’s bad boy Mr. Big Sur.

Server.app v.5.10 in macOS Catalina 10.15.7

Upgrading to a new major version of macOS can be fraught with peril for a fleet of mac devices but it is potentially fatal for a production SAN environment. That is why we wait. We want a nice stable storage system for our Final Cut Pro editors and other media creatives so it is safe to be one version behind. Less drama that way. We prefer our dramas to be on AppleTV+

Watch TV Upgrade Xsan

It is not boring to watch AppleTV+ while upgrading Xsan

The Xsan upgrade to Big Sur was pretty much not exciting except for one funny roadblock that I had set up myself last as a kind of booby trap for “future me”. More about that later. First the boring stuff. The last few weeks have been very busy updating and re-writing documentation in Pages.app and running multiple redundant full and incremental LTO backups with Archiware P5, syncing to nearline archives, and archiving finalized projects to the LTO shelf in paradise (sounds more exciting when you put it that way don’t you think?). Updating and re-writing documentation can sound like a waste of time but “future you” will appreciate what “past you” was doing today. And today I had fun updating Xsan to macOS Big Sur. Now I must write down all my thoughts before I each too much vegan vanilla ice cream and slip into a food coma.

“Planning for disasters, while hoping for none” is the IT mantra. We planned hard and we were ready to restore Xsan from Time Machine, if we had to. Not a joke. The server is backed up by Time Machine. The data is backed up to LTO, nearline archives racked and stacked in a server room and on redundant thunderbolt RAIDs which are parked on electric trucks ready to blast off at the earliest sign of danger. Well, everything except for the last part. Would be nice. And cloud backups for those clients that want them. Plan for the worst, pay for what you can to keep your business operational and lessen the impact of mechanical failures, human oopsies, or ransomware. Sysadmins are indistinguishable from malware sometimes, but we mean well. More seriously, humans makes mistakes and break things (that, me!) but ransomware is real and my elaborate backup and archive planning has saved a few customers this year.

Ok, now for the fun part. How to upgrade an Xsan to macOS Big Sur (11.5.1). Maybe go read last year’s blog post on upgrading the Xsan to macOS Catalina 10.15.6 which was detailed and thorough. Or read Apple’s new Xsan Management Guide. It’s got all the fundamentals explained.

Xsan volumes are typically made of up fibre channel RAID arrays. Nice icon!

Preparation is key. Be prepared. Get ready. Psych yourself up. I used Greg Neagle’s installinstallmacos.py to download macOS Big Sur as a disk image and had that and the App Store’s Server.app downloaded beforehand and not be dependent on internet access (production SANs are not always internet accessible). It is both true and not true that you can setup Xsan in Big Sur with the Server.app. It is true you need the Server.app for an upgrade from macOS Catalina 10.15.7 but if you’re starting from scratch in macOS 11 you will be building your Xsan in Terminal. Have fun! (We will cover this in a future post).

Download macOS Big Sur and the Server.app. Keep old copies zipped up. Cvlabel is nice too

Server.app manages only three (3) services for an Xsan upgrade: Profile Manager, Open Directory and Xsan. In macOS Big Sur new setups of Server.app Xsan is gone. Why they haven’t taken out Profile Manager and not kept Xsan instead made me scratch my head. No one in their right mind is using Profile Manager to install or manage profiles, they’re using commercial MDM vendors. But Xsan in macOS Big Sur (11) is not only production ready storage SAN awesome it has been upgraded to be compatible with Quantum’s Stornext 7 (previously it was only v.5)

Profile Manager does not belong here. Long Live Xsan!!

Installing macOS 11 Big Sur and upgrading Xsan to v7 is compatible (in my testing) with macOS 10.14 Mojave, 10.15 Catalina and of course macOS 11 Big Sur. If you don’t believe me check out this not updated in forever Apple’s compatibility chart.

Ok, by this time you get the idea I’m an expert, right? I’m ready to upgrade. But I run into my first real road block. And I have only myself to blame. I can’t launch the macOS Big Sur install app. It is blocked. “Contact your administrator”?! I am the sysadmin. Oh, ok. That’s me. What have I done now? I installed Hannes Juutilainen’s Big Sur Blocker last year, that’s what.

Of course I installed that. With Munki. On all my Mac clients that were upgraded to macOS Catalina. (And of course my Xsan controller has Munki!). But no worries, let me read up on my last year’s blog post about it to figure out how I installed it, there must be a launch daemon or something.

this is not how I expected it to go

Hmm, no didn’t mention there. And where is that pesky launch daemon that I can unload and get to this Big Sur install. Oh? It’s a launch agent. Unloaded. Hmm, still no. Ok, delete the app from /usr/local/bin, hmm, nope. ok kill the app process. Ok, now we can install macOS Big Sur. Sorry for the delay. I had told Munki to uninstall the bigsurblocker app and it did for every other Mac, I swear, really. It did.

Please proceed with the macOS Big Sur install

So ready for macOS Big Sur. Oh wait, we noticed that you’re running Server.app and well, we don’t do a lot of the same things anymore in the new Server.app so maybe this is a warning.

Warning. We noticed that you’re running Server.app and we don’t do those fun things anymore.

So a lot of progress bars and stuff. See my last upgrade blog post and it’s the same as installing macOS Big Sur on any Mac, except this Mac Mini is running an Xsan production SAN environment with a lot of RAID arrays in a server rack or two. Ok, yeah, just run the installer.

We noticed that Server app is no longer server app.

After macOS Big Sur is installed zip up your older server.app and drag in your new one (or use that fancy App Store app to do it for you if you’re lazy). Click a bunch of buttons (see all my old blog posts) and launch the new Server.app.

Profile Manager is updating. No one cares.

So we have to wait while the bag of scripts that is Profile Manager gets updated but no one uses it but it’s the most important app in Server.app now, no I am not bitter why do you ask. Xsan is awesome.

Xsan is off. Don’t panic.

Xsan is off. Don’t panic. Where’s my towel? Panic now!

Time to restore from your old Xsan configuration. Wheee…..

Xsan restore configuration.

Activate your Xsan and carry on upgrading all your Mac clients. Note: I did test macOS Mojave 10.14, macOS 10.15 Catalina and of course macOS 11.5.1 Big Sur Xsan clients. All worked.

Xsan on. Power up.

Upgrading Xsan with macOS Big Sur is easy if you’re going from macOS Catalina. Starting from scratch is another story to be covered in another blog post. Also not covered is certificate issues from self-signed certs breaking when I upgraded my Munki and MunkiReport server. That’s definitely another blog post. It’s just a webserver. Just. A. Web. Server. What is so hard? haha

Technical Errata:

With more than one Xsan controller it used to be recommended to upgrade the secondary before the primary but it is now best practise to upgrade the primary first to maintain the sanity of the OD data.

Xsan Upgrade Step by Step:

Clone the controllers. (+ Time Machine backups)
Turn off the clients.
Stop the Xsan Volume.
Run cvfsck on the volume.
**Upgrade the primary.
Confirm the secondary can see the primary.
*Upgrade the secondary.
Confirm the secondary can see the primary.
Check SAN access on both controllers.

Upgrade the clients as desired.

From Camera to the Clouds: the very real story of Hedge and Postlab

Posted on by
3

Update: This proxy workflow applies to Final Cut Pro 10.4.8 and earlier. For the new proxy workflow with FCPX 10.4.9 and Final Cut Pro 10.5 see my new blog post

Note: I want to explain how our current workflow for editing remotely. I am always testing new tools and methods, so workflows change all the time. This is a snapshot in time of what we are trying now. So far it works.

Hedge

We use Hedge to copy camera cards to multiple drives on set (or after a shoot if on location) and then we use Hedge once more to copy one of these drives to the office shared storage (Apple’s Xsan).

Why use Hedge? A nice simple app which hides its complexity well. Hedge has an easy interface to copy multiple sources (camera cards, usually) to multiple destinations (two external drives, or two SAN locations etc), and it does it well. It verifies, and double checks its work and leaves receipts. What was copied when. This is very nice and very useful for troubleshooting. It also has an API which made it easy to build an app that configures Hedge for its current task, and AppleScript support for extending automations after specified actions.

Kyno and Postlab

We are using two other tools in our remote ingest workflow currently: Kyno from Lesspain software for rewrapping and converting camera footage and Postlab, the remote collaboration tool for Final Cut Pro (and Premiere Pro). Testing with other tools is always ongoing and during a recent test of the workflow we also tried EditReady from Divergent Media.

The Workflow (so far)

While we are exploring various workflow automations we are currently doing the following steps manually.

  1. Hedge to copy camera cards two external drives on set, and then Hedge copy the drive to Xsan
  2. Making re-wrapped in MOV files from the original camera MXF files using Kyno and then
  3. Making H264 MOV 4K proxies in Kyno
  4. Uploading finished proxies to Postlab drive using Hedge
  5. Set up FCPX production and new FCPX library from template connected to proxies in Postlab drive

Hedge and Postlab

Hedge is super useful. Two times good. Hedge and Postlab are best friends. And the UI on both shows the simple aesthetic shared by the developers. Three panes. Source / Start to Destination / Projects. Whether you are copying Proxies to Postlab Drive or accessing your editing projects in Postlab the apps will guide you through.

Copying the Proxies with Hedge to Postlab Drive.

Details. Rewrap and Proxies

Workflows will depend on your goals, and your available tools. In this case we are using a Canon camera and ingesting MXF files. In order to edit with small Proxies in FCPX but also be able relink to original (and larger) files easily we need to in our case re-wrap the original MXF to QuickTime MOV container.

Right click on a clip in Kyno to rewrap to Mov.

Originals. Not Proxies.

And to be clear we are treating these in FCPX as “new” originals not as actual FCPX proxies. With the rewrapped MOV files we make transcoded H264 files which are swapped 1 for 1 with the original. When we need to export a final 4K version we can relink to the original 4K source and export easily.

Proxies. Not originals

The transocded H264 4K proxies we made in Kyno were 15x smaller than the original re-wrapped Mov files. We had almost 600GB in originals and 37GB for the 4K H264 proxies!!

Postlab Pro Tips

Working with Postlab pro tip #1 –> keep those FCPX libraries light. Keep all media and cache files out of the library. We knew that and we had Storage Locations set to outside of the library but one new issue came up when the libraries grew really big and we realized the editors were making multiple sequences, not backups, but versions. Now we are trying to work around this habit with Postlab itself. You can check in a version of the library and duplicate library for an alternate version. Modifications of old habits are always tough but technical reasons may force a change in habits here. We will see. Postlab pro tip #2 –> Keep your cache large and fast. By default the Postlab cache is your local drive and only 20GB. If you have a fast SSD or an external drive then move that cache and increase the size. It will help. Trust me.

Kyno vs EditReady

Another small issue we encountered in testing was that we could make the rewrapped Mov files in Kyno or in EditReady and both were fine. The only objection the editors had was that in Kyno we could keep the folder structure of the original camera cards and they felt that this lent some confidence to being able to track the files to the camera card folders if any media was missing or misplaced. The EditReady files kept the original names but they were all in one folder. As the tech I see no issue with FCPX handling these files since we’d be ingesting all the finished proxy files and all the files were named by the camera. Editors should be able to tell which reel the clips were from by the clip name and that’s all you need technically, but you can’t win every argument with an editor. As the tech you need to test alternative tools and methods and see what works technically but also see what can be accepted to work in the way the editors want to work. Changes to workflow are some of the hardest to make, making a system that is used, actually used, by the editors is the goal.

Errata

Errors. If you get them, how do you know? This was one area where I could comment on both Kyno and EditReady. I am spoiled by Hedge and it’s nice reports when it is done copying. And Postlab which has a Help menu :collect logs for support button, very nice. If your software tool is going to process a lot of files (rewrapping then transcoding) I want to know if there were errors. EditReady popped up a window to what had succeeded or failed and Divergent Media support told me to look in the logs for any issues encountered. Not great. While Kyno has a separate jobs window which shows jobs done or failed. But still no report. I would like a receipt or report or log at the end with files converted or failed to convert. It would help troubleshooting any issues when they arise. Tech support for both companies is great and responsive. Thanks again. And I’ll keep sending in feature requests.

Testing. More Testing. And Teamwork.

We are testing this workflow in production with a real project and getting feedback from the team. So far the proxies have proven to be easy to make, quick to upload to Postlab drive, simple to use in FCPX in Postlab. Assembling the cut and editing are going well. We will find out about the colour process when we get to that stage and relink to the originals. Stay tuned.

Thanks!

Thanks to Felipe Baez / cr8ivebeast for his assistance on this part of the workflow. We were having trouble relinking to the original MXF and he gave us the excellent tip to rewrap then in Kyno then make the smaller proxies. Works like a charm. Thank you Felipe! Here’s a link to a video Felipe made showing a similar procedure using Compressor to transcode and then relink in FCPX and it goes to show you that there are lot of ways to do things and to keep trying, and experimenting. You might learn a thing or two.

Compressor Tips and Tricks

Posted on by
Reply

 

Issue: Stuck job in Apple’s Compressor app.

Resolution: Remove the historical jobs in your local home folder.

~/Library/Application Support/Compressor/History/V4

Compressor-History2

Note: to get to your home folder hold down the OPTION key and select the Go menu in the Finder.

Compressor is the best sidekick to Apple’s Final Cut Pro X and it gets used a lot. But occasionally something goes awry. It’s software running on a computer. So we troubleshoot. What looked like a stuck running job was mostly leftover evidence of an old job. The Apple support document I found didn’t mention this tip but instead talked about zipping up your settings folder which has all your custom compressor settings for things like YouTube outputs or anything custom. Didn’t seem useful to me to remove but this historical stuff, don’t need it and POOF this solved the issues. It’s not always this easy but something you just take the win and go with it.

Reference:

Resolve an issue in Compressor: Learn how to isolate, troubleshoot, and fix issues in Compressor.

https://support.apple.com/en-ca/HT203476

To install macOS Mojave, or not to?

Posted on by
8

InstallMojave

Just the other day macOS Mojave was released and now the armies of Macs armed only with the AppStore are silently downloading the installer and ready to upgrade. You can’t hurry too fast to be on the bleeding edge, hurry faster!

Just in case you don’t want everyone to install macOS 10.14.0 (dot zero!) in the first week of its release here’s a way to slow down the upgrade hordes using Erik Berglund’s AppBlocker script. Erik Berglund is also the author of ProfileCreator (for creating profiles) and the author of many other great scripts.

Note: for true binary whitelisting check out Google’s Santa project and Upvote (and Moroz and Zentral, two other Santa sync servers).

Step 1. Get it

Clone or download the AppBlocker project from GitHub

AppleBlockerProject.png

Step 2. Do it

Edit the AppBlocker.py script with the Bundle Identifier of your app to block, in this case for the Mojave installer from the AppStore it is:

com.apple.InstallAssistant.Mojave

You can also edit the alert message, and the icon that is shown, as well as decide if the blocked app should be deleted or not. The script is easy to edit in BBEdit, or nano (in Terminal). Use whatever your favorite text editor is to make the necessary changes.

# List of all blocked bundle identifiers. Can use regexes.
blockedBundleIdentifiers = ['com.apple.InstallAssistant.Mojave']

# Whether the blocked application should be deleted if launched
deleteBlockedApplication = False

# Whether the user should be alerted that the launched applicaion was blocked
alertUser = True

# Message displayed to the user when application is blocked
alertMessage = "The application \"{appname}\" has been blocked by IT"
alertInformativeText = "Contact your administrator for more information"

# Use a custom Icon for the alert. If none is defined here, the Python rocketship will be shown.
alertIconPath = "/System/Library/CoreServices/CoreTypes.bundle/Contents/Resources/Actions.icns"

UPDATED NOTE:

To determine the Bundle identifier of other applications you can use osascript

osascript -e 'id of app "iTunes"'
com.apple.iTunes

If you want to block more than one app use a comma separated list in the AppBlocker.py script:

['com.apple.InstallAssistant.Mojave','com.apple.iTunes']

 

Step 3. Run it

Put the script where you want to run it. The default location as defined in the launchd plist included with the app is “/usr/local/bin”. Put the launchd.plist in “/Library/LaunchDaemons/” and start up your launchd to block your apps!

launchctl load /Library/LaunchDaemons/com.github.erikberglund.AppBlocker.plist

Step 4. Automate it

For bonus points we automate! Bundle it all up in a package with munkipkg, then distribute it with Munki to all your clients.

Using munkipkg is easy. Create the folder using munkipkg

./munkipkg --create AppBlocker

munkipkg: Created new package project at AppBlocker

Then you fill the payload folders with those items you downloaded from the AppBlocker project. LauchD plist in the LaunchDaemons folder and AppBlocker.py in the “usr local bin” (create each nested folder).

AppBlocker-Munkipkg3.png

And finally create a post install script (no “.sh”) with the launchctl action to start your plist.

AppBlocker-Munkipkg4.png

Last but not least add this package to your Munki repo as an unattended managed install  that everyone gets. Of course, only do this after testing your package locally somewhere to verify that it works properly. Remember the saying: “You may not test very often, but when you do it’s always in production.” Be very careful with your testing but always automate all the things.

Updated after the initial blog post to explain how to add more than one app to block, and how to use osascript to determine the bundle identifier.

 

 

 

I don’t get High — Sierra!

Posted on by
Reply

Friends don’t let friends install macOS High Sierra in production. Don’t get High, Sierra.

macOS 10.13 was released on Sep 25, 2017, and almost two months later with only one point release update, it’s still too new for production. Download it on a test machine or two or more, test it with your apps and systems, file bug reports and radars, but for the love of all that is Python and Monty! don’t run it on your production Xsan. Well, at least not yet. Wait until next year. Or as long as you can. Or until the new iMac Pro is released with 10.13 pre-installed or wait until they ship the new Final Cut Pro X 10.4 that may or may not require macOS High Sierra.

With that out of the way, I’ve just upgraded the production Xsan to … macOS Sierra. Yes, macOS 10.12.6 is stable and it’s a good time to install last year’s macOS release. Time to say good bye to macOS el Capitan 10.11.6, we hardly knew ya. Besides guaranteed security updates, stability and the annoying newness of a changed macOS, what else is there? In Xsan v5 they introduced a new “ignore permissions” checkbox for your Xsan volumes. Looking forward to that feature in production. No more Munki onDemand nopkg scripts to run chmod. No more tech support requests for folders, files, FCP X projects that won’t open because someone else used it, owns it, touched it. We’ll see how that pans out. I’ll let you know.

Upgrading Xsan to v5

Step 1. Back up your data

You’re doing this, right? I’m using Archiware P5 Backup to backup the current projects to LTO tape. I’m using Archiware P5 sync to sync the current Xsan volumes to Thunderbolt RAIDs, and using Archiware P5 Archive (and Archive app) to archive completed projects to the LTO project archive. That’s all I need to do, right?

Step 2. Back up your servers

Don’t forget the servers running your SAN! I use Apple’s Time Machine to backup my Mac Mini Xsan controllers. External USB3 drive. I also use another Mac Mini in target disk mode with Carbon Copy Cloner to clone the server nightly. (Hat tip to Alex Narvey, a real Canadian hero). And of course I grab the Xsan config with hdiutil and all the logs with cvgather. Because, why not?! For Archiware P5 backup server I also have a python scripts to backup everything, another scripts to export a readable list of tapes, and BackupMinder to rotate the backups. Add some rsync scripts and you’re golden.

 

Step 3. Upgrade the OS

Unmount the Xsan volume on your clients or shut them down, disconnect the fibre channel. Do something like that. Stop your volume. Download the macOS Sierra installer from the App Store. Double click upgrade. Wait. Or use Munki. I loaded in the macOS 10.12.6 installer app into Munki and set it up as an optional install to make this portion of the upgrade much quicker and cleaner.

In my case after the OS was upgraded I checked the App Store app for any Apple updates (you can also use Munki’s Managed Software Center to check) and of course there were some security updates. In this case the security upgrade hung on a slow network connection and the server crashed. Server down! I had to restore from Time Machine backup to the point where I just upgraded the server. It took some extra time  but it worked (can’t wait for next year’s mature APFS / Time Machine and restoring from snapshots instead).

Step 4. Upgrade Server

After macOS is upgraded you’ll need to upgrade the Server.app or just upgrade the services used by Server (even those not used by Server get upgraded).

Step 5. Upgrade the Xsan

Bur first we have to restore the Xsan config. Don’t panic! It may invoke bad memories of data loss and restoring from backups. Xsan PTSD is real.

Restore-previous-Xsan.png

Step 6. Upgrade the rest

Next you have to upgrade the Xsan volumes.

Xsan-volume-needs-upgrade

New version of Xsan, ch-ch-changes! Ignore permissions check box will remount the xsan with the “no-owners” flag. Let’s test this out.

 

Upgrade the OS and Server app on the backup controller. Upgrade the OS on the clients using Munki or App Store if you like doing it the hard way. Ha Ha.

Step 7. Enjoy

Plug those Thunderbolt to Fibre adapters back in, mount those Xsan volumes and be happy.

Step 8. Wait for the complaints

The next day the editors walked in and went straight to work with Final Cut Pro X. No one noticed anything. Xsan upgraded. Workstation macOS upgraded. Everything appeared to be the same and just worked. Thankless task but well worth it.

 

Reference: Apple’s iBook guide here

 

 

Archiware P5 and Synology NAS.

Posted on by
Reply

Update: As of version 5.4.3 there is an official P5 add-on package for Synology NAS

Archiware P5 available for Synology

Note: The P5 app for Synology NAS first debuted with P5 v.5.3.3

On the Archiware P5 new-features page there’s a blurb about the Synology NAS integration:

From Version 5.3.3, Archiware P5 supports Synology NAS devices without restrictions.  

Synology NAS can serve as a data source or target for P5 Synchronize, P5 Backup and P5 Archive. The Archiware P5 application can now be installed on the Synology NAS itself.

Thanks to the snapshot capability of the DSM platform, powerful enterprise Synology NAS devices can also be used as repository for Backup2Go. This setup opens the possibility of introducing a professional data security solution at an affordable price point.

Let’s look in closer detail how to install Archiware P5 on a new Synology NAS.

For this post I have a new Synology 1515+ NAS, installed with five 6TB hard drives (It is very easy to install hard drives. No tools required). Note: I’ve purchased the NAS with my own money and was not paid to write this article.

At the time of this blog post the latest Synology DSM release is 6.1 and Archiware P5 is at version 5.4.2.

Step 1. Download Synology package from Archiware.com/download

Download Archiware P5 for Synology

 

awpst542spk

Requirements are DSM 5.2+ and Intel x86 64-Bit CPU only. (i.e. Atom but not Marvell).

Step 2. Find and Log into your NAS

Find your new NAS with the Synology Assistant app or use this handy website link:

Find your NAS

I had no luck with the app (it found my existing NAS, but not the new one). Using the website I was able to quickly locate the new NAS that I need to log into and setup. Very nice feature.

synology-1515-setup-welcome2crop

Step 3. Install the new DSM

Install or update new software. You will be prompted to go through the initial setup to prepare your new NAS.

synology-1515-install-diskstation-manager2

Step 4. Set up a new volume

Chose the Btrfs or ext4 filesystem. Btrfs supports snapshots, replication, and much more.

synology-1515-btrfs-setup

Step 5. Monitor the volume setup

Verifying the hard disks will take a moment. Take a break here.

synology-1515-storage-manager

Step 6. Open Package Center

packagecenter

Step 7. Install manually

Install Archiware P5 by selecting the “install manually” option to upload the awpst542.spk downloaded file from archiware.com

synology-1515-archiware-p5-package-center-upload

Step 8. Agree to continue.

Load the Synology P5 installer by agreeing to continue with this “unknown” publisher.

synology-1515-archiware-p5-package-center-unknown

Sep 9. Agree to trust the installer

synology-1515-archiware-p5-package-center-license

Step 10. Confirm the Install

synology-1515-archiware-p5-package-center-confirm-install

Step 11. P5 is now running on the Synology NAS.

Hooray! P5 is now installed. Select the app to examine the details.

synology-1515-archiware-p5-package-center-installed

synology-1515-archiware-p5-package-center2

 

Step 12. Examine the option to stop or uninstall the P5 application

synology-1515-archiware-p5-package-center-stop-uninstall

Step 13. Login to the P5 server running on NAS

To login to P5 open a new tab. Pay attention to the port number: “20,000” (vs 8000 on other platforms such as Solaris, Linux, OSX etc).

synology-1515-archiware-p5-port

 

Step 14. Set up your NAS as a client on another Server

To test the new Synology 1515+ NAS I then set up the NAS as a client on another P5 server, and set up a P5 Sync job to copy data from server with a ZFS based filesystem to the Synology NAS with a btrfs volume.

Testing: Set up the new client in P5 with a name and IP address, then set up a new sync job with source and destination. Start now. Watch the bits fly through the ether. Be happy.

Step 15. Other things to configure

To make your new NAS is working smoothly don’t forget to set up the email notifications, and set up some AFP, SMB, or NFS shares as required.

Take some time to explore the Package Center app and see what other great applications are offered on the Synology NAS.

Synology makes a great low-cost NAS appliance. For SMB or production setups I would recommend two or more (for redundancy, hot or cold spares, disaster recovery, offsite backups/replication). With P5 installed you can Sync your server data to a NAS for onsite or offsite backups, backup your NAS to tape, or use the NAS for your client workstation backups using Backup2Go. Using the new Btrfs filesystem provides many of the same advances as ZFS, including snapshots and replication, over traditional filesystems such as ext4 and hfs which sadly lack these features.

Conclusion:

The Synology NAS is a great experience. Adding Archiware P5 is a recommended way to include this NAS as part of any good backup, archive or DR (disaster recovery) scenario. Two thumbs up. Way up.

References:

Archiware P5 new features

Synology DSM

Best of 2015: VidiXplore

Posted on by
Reply

This is another great product of 2015 and  when I found it, I thought VidiXplore proves that Media Asset Management could be done better and simpler. At the end of December 2015 they released version 1.0 with new some tricks, including some changes to make a migration from Final Cut Server a reality. Time to move some clients!!

To sum up VidiXplore, I’ll quote my tweet from Dec 21, 2015:

Finally the perfect solution for simple asset management! Keep proxies in the cloud, originals local. Search + share!

Working with video editors, animation and visual effects studios, I’ve come to realize that media asset management (MAM) systems can be complicated and painful. Changing workflow, oh no! Building a better pipeline is not easy, nor is it always welcome. Well, hello from the other side, we found the solution, or at least part of it.

With VidiXplore you have 3 steps:

Step 1. Manage your videos by keeping all the originals local. Use your own storage. Use your folder structure. Use your vids as you would normally. Don’t pay for cloud storage.

Step 2. Proxies (thumbnail vids) go into the cloud to be viewed by you and your team.

Step 3. Organize your videos and photos into collections, batch edit by adding tags to add metadata, search for particular assets and share them with colleagues and external clients.

That’s it. You’re already ahead of the game. We skipped right over step 4 which was “have a lot of meetings to debate proper metadata” and step 5 which was “convince everyone at the company to adopt a different workflow.”

With VidiXplore you switch to a monthly payment model, that is true. You don’t own the cloud platform, but what you gain is that you don’t pay a lot of money upfront to set up a large server (or many large server), nor do you need render farm for video transcoding nor for the databases you need to keep track of it all. Pay monthly. That’s the way for a lot of smaller companies. Lower up front starting cost. No extreme capital outlay in the beginning.

Honestly, VidiXplore is a refreshing and easy way for so many people to use asset management now, so why not try it? Harder to say that with a large system setup that costs a lot of money to set up, only to find that no one wants to use it. That’s not what anyone wants.

And now for something completely different…

Let’s take a quick look at VidiXplore. If you’ve installed the VidiXploreAgent-1.0 agent then you’ll have a nice “V” icon in your menu bar (Mac) or system tray (Windows). Use this to open the VidiXplore agent.

Vx Menu agent Open

In the VidiXplore menu you can access the settings where can you set whether certain file types get a Cloud Copy uploaded by default or how many concurrent jobs can run at once.

Vx agent prefs settings cloud copy

When the agent is open you’ll see folders you’ve configured for media, and an option to go the website of your particular instance of VidiXplore.

Agent

For the first web login you’ll see an intro screen which allows you to upload additional media to VidiXplore (that is, in addition to any particular media folders you’ve configured in the agent) and the option to connect cloud storage such as S3, Azure or Dropbox. Lastly, there are also the installers for the agent.

Welcome Screen Shot 2016-01-29 at 11.43.20 AM

Once logged into your website you see a basic layout with options to sort by files or collections, to specify all files or particular types, sort by location and bulk actions.

Collections Screen Shot

If we want we can sort the results and see only collections:

Sort collections

Or we can sort by files which have a cloud copy:

Sort cloud copy images

So many options to sort, search and find what we’re looking for. Of course, we want to also edit the metadata but not so much that we require weekly meetings to decide on the 500 important and required metadata fields. Just use tags. Of course, meetings are good, and so is process, but it is so quick to select bulk actions and add a tag to a group of objects. Done!

 

add tags bulk action MacDevOps

“Finally the perfect solution for simple asset management! Keep proxies in the cloud, originals local. Search + share!” 

There’s so much more you can do with VidiXplore, and I’ll go into more detail in another blog post, but this was just a highlight for my best of 2015.

Check out their website for more information:

VidiXplore.com

Best of 2015: Archiware P5 Archive app

Posted on by
Reply

Announced late in 2015 the Archiware P5 Archive app is a revolution for editors who want to control the archive and restore process. No longer the job of the IT Admin, editors can select files or folders on their SAN volume (or anywhere) and send them to the tape archive.

The Archive app is a brilliantly simple app that allows the right-click services action in OS X, or in another words a it’s a GUI app that presents a contextual menu that knows to how to the talk to your P5 Archive server. When the files are safely on tape the original files on the filesystems are replaced with stub files that can be used to start the restore process.

Requirements: Archiware P5 server with the Archive module setup with an Archive plan. Add to that the P5 Archive App which is installed on the clients.

Note: At the moment all archiving goes over the LAN by default, so if you have a fast SAN then you set up the P5 Archive app client settings as “localhost” instead of their actual client name. That means that when it goes to archive the file, the server knows that the files exist on the SAN at a known path (which is the same on the client and the server).

And now for some detailed steps and screenshots.

  1. Archiving completed projects

Choose the completed project folder and right-click. Select “Archive to P5”.

Note 1: If you want to restore files choose the folder that was archived and right-click. Choose “Restore from P5”.

Note 2: Restoring individual files that have been archived is possible by double-clicking the files with the “.p5a” extension, but it will be much faster to select an entire folder to restore than many individual files.

Note 3: For either archive or restore to work the P5 Archive app needs to be installed.

Note 4: To avoid having a services sub menu keep the contextual-menu items to four.

Right-click folder to archive

2. Archiving app status

When you are archiving or restoring files the Archive app will show you the status of your request. It will also show you the status of other jobs running on the P5 server. This is to let you know why perhaps your archive or restore is taking a long time (it’s possibly waiting for access to the tape drive and it currently busy backing up or archiving something else).

P5 Archive app Running jobs status

The P5 Archive app offers you three operations “cancel job”, “list items” and “get report”. The last two are great when you want to examine a completed job, for example. If you want to find out what files were archived in the particular job choose “list items”.

3. Restoring files

Archived files will have either one of or both of, 1) a”.p5a” file extension and 2) a P5 Archive app icon.

Folders and FCP X project bundles (which are folders) do not get the “.p5a” extension, but FCP X projects have the the icon.

p5a-icon.png

Note 1: Files can also be restored by the admin through the P5 web interface. They can be restored in place or to any other location that is required.

Note 2: On the P5 server jobs that are sent to archive or restored from tape show up as “cli job” with the tapes in use.  Actual files or folders involved need to be noted from the P5 Archive app not the P5 web admin console. Otherwise checking the P5 web restore tab will files actually archived (that can be restored).

That’s enough for the quick overview of this great new app. One of the best things in 2015.

For more information on Archiware’s new P5 Archive app check out their website:

P5 Archive app

2015 in review

Posted on by
Reply

The WordPress.com stats helper monkeys prepared a 2015 annual report for this blog, but nobody wants to read no stinkin’ reports so let me just sum it all up: Xsan, Munki, Thunderbolt, Archives. Or is that all one word? Thunderbolt Xsan Munki Archives! That’s better.

Here’s an excerpt from the report that no one will read:

The concert hall at the Sydney Opera House holds 2,700 people. This blog was viewed about 20,000 times in 2015. If it were a concert at Sydney Opera House, it would take about 7 sold-out performances for that many people to see it.

Surprisingly, or not, that opera would be about Xsan. Yes, Apple’s Xsan is still alive, and Apple even added new features with OS X 10.11 El Capitan. I’m still building Xsan shared storage SANs and upgrading old ones to new versions. That was one of the good news stories of 2015 for me.

You can build an Xsan with one or two Mac Minis and add your storage of choice. That used to mean more often than not the fibre channel storage from Promise. A great choice for larger deployments, the x30 Vtraks are solid.

But the real shocker for me in 2015 was stumbling upon the Accusys Thunderbolt SAN RAID, the A16T2-Share. For more than half off the price of a similar fibre channel storage RAID here’s a magical box powered by unicorns that has four (4) Thunderbolt connections. Plug one Thunderbolt cable into that Mac Mini, format the raid, setup Apple’s Xsan, and then plug the other three (3) Thunderbolt cables into iMacs, Mac Pro, MacBook Pros or any Xsan clients. Wow. Awesome.

Suddenly we have a game changer. An affordable SAN storage RAID for real block-level storage. Now more than ever we can afford to have true collaborative workflows for video editors and anyone in the creative. If you need to work together with fast connections to a shared pool then building an Xsan got much more attractive.

Disclaimer: I got a chance to test the Accusys A16T2-Share. And I would be crazy to recommend something without testing it thoroughly. This was used for several weeks by video editors in production. It was much faster than our 4GB fibre channel storage, of course, but it was also faster than our 8GB FC storage. Speed tests showed we got close to 1GB/sec, and even when it was 97% full we got 700MB/sec. Sa-weet.

I look forward to seeing what Accusys bring to NAB in 2016. What new box will they show up with? I hope for more than 4 client ports and faster Thunderbolt 3. Only 82 more sleeps till we all find out.

Apple’s Xsan and Accusys Thunderbolt storage A16T2-Share were big stars of 2015, but what else stood out? The two other bright shiny lights were Archiware’s new P5 Archive app, and Vidispine’s VidiXplore cloud based MAM. More on those in posts to follow. Both of these products have transformed workflows for editors. Stay tuned!

Move over El Capitan, hello Yosemite!

Posted on by
1

With all this talk about El Capitan, Apple’s as of yet unreleased version 10.11 of OS X, and its wondrous new features in Xsan, I think it might be time to upgrade to last year’s breakthrough version of OS X, Yosemite. Sure, you might be excited by the press releases for the built-in DLC in El Capitan but seriously sane folks stay 1-year behind the bleeding nose upgrades provided by Apple. So if OS X 10.11 is all the rage before its released it must be time to seriously consider upgrading that working Xsan running OS X 10.8 or OS X 10.9.

In my case, I upgraded a working Xsan running on Mac Minis and OS X 10.8.5. Here are some screenshots from the process. As always think worked better than I could have expected, and it is a much easier process that one expects. But stay sharp kids, danger lurks when you wake the dreamer…. Upgrading a SAN is serious business and doing anything like this without proper backups is taking your life in your own hands. In my case, full disk backups on Promise Pegasus RAIDs and full tape backups using Archiware P5.

Download the Yosemite installer form the App Store. Install. Download the new Server.app from the App Store. Install. Now upgrade your Xsan. That’s it. You’re done. No surprises, aren’t you happy? Ha ha. I’m kidding. The fun is just getting started.

If you’re actually following along, this isn’t a step by step recipe. Go to Apple’s site and read this Kbase and check out the migration guide.

Restore Xsan

Restore Xsan

Step 1 is to launch the new Server.app, find Xsan Admin. Just kidding, it isn’t there. Enable Xsan, and choose to Restore a previous SAN configuration. That wasn’t hard. High five! Actually, we’re not done yet. Set up OD now. Go!

Step 2. Set up your Xsan controller as an Open Directory (OD) master. Does’t matter if it’s joined to another domain, Xsan keeps itself organized in OD, so you need it.

Set up OD

Set up OD

Step 3. Admire your upgraded SAN, “how lovely the flowers do smell…. life is good.”

XSAN LIST

Xsan list

Step 4. Where did my Xsan admin go? Where do I add clients? Where are my clients? Huh? What? Why did I upgrade a perfectly working SAN to this version? Ha ha.

Take it all in, take a good look at what you’ve done to your Xsan. What? Just so the editors could have the latest version of Final Cut Pro (v.10.2.1) which is only compatible with OS X 10.10.4. I see what you’ve done Apple, very clever indeed. Hmm…

Click on the “Save configuration profile” button and download the profile somewhere. Use this to set up the SAN on your clients. Distribute via Profile Manager or install it manually. Up to you. I haven’t gotten it to work with Munki quite yet. Installing it requires the admin password for the Xsan controller. How convenient.

When you client is configured you’ll see a Profile in System Preferences. Remove it and your client is un-configured. No more Xsan.prefpane to list volumes and mount or unmount them. Nope. That would be too easy. Learn to love “xsanctl”, as in “xsanctl mount Xsan”. Read some xsanctl tips in this Kbase

Step 5. Set up a backup Xsan controller. You have one of those, right? In my case, I had a client which I wanted to promote to be a controller.  But first what to do about its status a client of the Xsan?

backup cannot be client

backup cannot be client

Open Server.app, enable Xsan, join current Xsan as a backup controller and set up a replica OD. Confirm, confirm, confirm. Think about what you’re doing, then do it!

confirm OD replica

confirm

Apple wizards are the best wizards, uh, i mean Setup Assistants. No wizards here…. So, you’ve setup a backup Xsan controller, and OD replica, and now look in Server.app. How amazing is that… wait, what? Where’d my Xsan volumes go? Huh? Where are the controllers? Weird. Very strange. Not comforting at all.

Xsan 4 no SAN list crop 122815

The Xsan window eventually shows the volumes and controllers, bur geez, almost gave me a heart attack. It’s not like I never seen Xsan go bad before. Xsan 1 nightmare still haunt me. They do. Backups. Need more backups. Archiware P5 Backups, do it now!

OK, you’ve survived the uncertainty of Xsan upgrades…. But wait more minute… cat the fsnameservers (no, it’s not the name of a band, it’s a command). Check it out. Holy smokes, batman. Xsan 4 by default will set your metatadata network to the public LAN, something that’d would be laughed at years ago, but they do it now by default. Of course, upgrading our SAN kept out metadata network the same. But strangely the Xsan backup controller is set to use the public for metadata when the primary controller is not. WTF.

Change your metadata network. Read the Kbase, and once again wield xsanctl like a boss.