About macvfx

DAM SAN Backups

Backup and Archive setups for media professionals

I recently did an online presentation on my Archiware P5 backups and archive workflows for media professionals.  

I use Archiware P5 to backup and archive media productions to tape. In this workflow I describe how original material, work in progress and conpleted projects are backed up daily and archived as needed.

This presentation goes into some detail of what two different storage setups look like and what similarities exist in my Xsan (fibre channel) and Jellyfish (10GbE) backup and archive workflows.

Despite the storage differences, the common workflow is that they all use the P5 Archive App to allow the Final Cut Pro X editors to archive completed projects. The simple way to do this is by right-clicking in the Finder and they will go to tape and get replaced by stub files (small files with the same name and negligible file size). When the project or any of the footage or assets needs to be restored it is another right click to restore. No IT people need to be involved. It’s magic. Almost.

Useful resources:

– Case Study: WorkSafe BC

Using P5 and the P5 Archive App

https://p5.archiware.com/solutions/worksafe-bc-hs-video-production

– The P5 Desktop Edition

Aimed at small teams and single users: P5 Archive and P5 Backup to a single LTO tape drive.

https://p5.archiware.com/desktop-edition

– P5 and Xsan

https://p5.archiware.com/solutions/xsan

– P5 and Lumaforge Jellyfish

https://p5.archiware.com/solutions/lumaforge

– Watchman Monitoring

https://www.watchmanmonitoring.com

– MunkiReport

https://github.com/munkireport/

– MacAdmins Slack:

https://macadmins.herokuapp.com

– MacDevOps:YVR Conference

https://mdoyvr.com

Best of 2018: the conferences

Part of a series of blog post on the “Best of 2018”

Part 2: the conferences

There was NAB in April and the FCPX Creative Summit in November (see more in my previous best of 2018 post here)

New conferences were the theme this year and I’ll start with one I couldn’t attend but really wanted to:

Query conf

⁃ all about osquery

⁃ Great group of people got together in San Francisco to discuss security with open source project originally from Facebook and now a verifiable industry (Kolide, uptycs etc)

⁃ Videos of the talks were posted on their site.

⁃ I had to miss this conference last year because it was just before MacDevOps: YVR and I could not be I two places at once.

Objective by the Sea

⁃ Awesome location. Maui. First time conference and my first time in Hawaii and it was spectacularly beautiful.

⁃ Great people. What a wonderfully diverse group of IT, Security experts and vendors

⁃ Patrick Wardle love fest. Everyone love Patrick and his Objective See free security tools. Look forward to his future projects with Digita Security

⁃ Looking forward to seeing how this conference evolves in the future. Especially if they end up doing more than one a year in different locations. Mahalo to the organizers for putting this together. As a fellow conference organizer I know it is not easy. It takes lots of love and Patrick and his team have lots of love to give. Thanks everyone!

MacDevOps:YVR

Disclaimer: I am the organizer

In 2018 we had our fourth annual conference and it was an amazing group of speakers and attendees. Many people took up my Quick Talk challenge. It is my firm belief that everyone has solved a problem and has knowledge they can share. I love it when people step up, literally step up on stage, and present a story, a solution to a problem, a tech problem they solved. We cheer them on.

What is MacDevOps:YVR?

Just the facts:

⁃ June 2019 will be the 5th annual conference

⁃ Inspired by DevOpsDays held everywhere all over the world this is an inspirational conference to bring together the creators of open source Mac projects and those in IT that use them.

⁃ Bringing a diverse group from around the world to learn about participating in software projects to manage Mac, the aim is to dispel the fear around version control (git), cloud (AWS, GCP, Azure) and various programming languages (Python, Swift, PHP etc)

⁃ Learn about what’s new in various open source projects we depend on: Munki, MunkiReport, Crypt, etc

⁃ Be inspired to share your our own solutions to problems with Quick Talks

⁃ 2019 will be partly security focused with a diverse group of security talks

⁃ Diversity and Inclusion will be front and centre of our IT panel. We are bringing in an amazing group to discuss.

⁃ Hack night. Working on gathering a team of MunkiReport contributors together to help organize some collaborative hacking and programming.

– Workshop. Learn how to make munkireport plugins. MunkiReport server is in PHP but the plugins are written in anything : bash, Python etc

Speakers for MacDevOps:YVR 2019 are on the MDO website.

June 12-14, 2018 join us in Vancouver, Canada for the firth annual MacDevOps:YVR conference.

Many thanks to the crew that that helps me organize this every year and makes the live event as good as it is. Without you I could not make it happen. Also many thanks to my awesome sponsors for helping us pay for bringing in speakers from around the world and paying for the event. Last but not least, thank you for all those that have attended and spoken at this event in the past. I love you all. You are amazing!

Resources

Use Git/GitHub to contribute to these community resources:

⁃ List of all conference videos Conference videos

⁃ MacAdmins Podcast community calendar GitHub repo

– Charles Edge has compiled a list of conferences and it is a great long list. Thanks Charles!

Best of 2018: FCPX and iMac Pro

Part of a series of blog post on the “Best of 2018”

Part 1: the iMac Pro and FCPX

The year started off with the new iMac Pro and Final Cut Pro X 10.4. Both new hardware and software were released in December 2017. New awesome hardware and software to start of 2018.

FCPX and the iMac Pro have proven themselves to be a great combination that has been amazing for FCPX editors everywhere. The new colour grading tools and other enhancements were warmly received in FCP X 10.4. The power of the iMac Pros was not exaggerated. Excellent pro hardware.

FCPX works great on a MacBook Pro and internal storage, with Apple’s Xsan and fibre channel or with Lumaforge Jellyfish 10GbE over NFS. I worked with all different setups in 2018 and happy to report that editors kept editing and left the storage and backup worries to me (and I didn’t worry since I’ve got Archiware P5 watching my back).

Working with the Jellyfish I installed the P5 Linux agent to backup and archive to tape. Getting the Jellyfish to back up to my P5 server running on a Mac Mini couldn’t have been easier. Through the year I worked with Archiware to make improvements in the P5 Archive app so that my editor clients can archive and restore more easily on their own. Works well and look forward to working more closely with both companies to help make awesome setups for FCPX editors and creative professionals everywhere.

NAB and FCPX

The week before NAB 2018, Apple announced a new version of Final Cut Pro X with support for closed captions, and the brand new ProRes RAW codec.

NAB in April is always a busy month with announcements from all companies in the media production and media asset management world and Apple’s public talk at NAB showing off new features so soon after their last major release was unexpected but very warmly received.

Of course there was one more major event in the 2018, in November there was the FCPX Creative Summit.

I attended this year and it was awesome. Apple released a brand new version with 3rd party integration in the form of extensions. This is huge. This will be amazing for FCPX editors who want to stay in FCPX and do their editing work but integrate with other apps.

What was the FCPX creative summit?

⁃ rendez-vous in Cupertino with Final Cut Pro editors, studio owners, plugin authors, creative apps vendors

⁃ Visit to Apple HQ. With Apple Pro Apps engineers, QA, managers and everyone involved.

⁃ In depth discussion of the next version of FCPX extensions which allow third party integration deep into the app for example: Frame IO for review and approve or Keyflow Pro or Cat DV media asset management apps.

⁃ Great team of people organizing. This event had multiple tracks and lots of great sessions for everyone. Well done. Enjoyed it immensely. Everyone using Final Cut Pro or involved in this creative universe should be there.

2018 was great year for pro hardware and software. The iMac Pro and the constant stream of FCPX updates kept us grinning from ear to ear. Great stuff. Awesome year.

Next up: best conferences of 2018

Reset Printer Queue

TIL (thing I learned)

Had a user upgrade to macOS 10.14.1 and no printers showed up anymore.

So using my Google fu I found some posts (see one below) which described a novel way to reset the print queue on macOS. An old trick apparently. Learn something new everyday.

A quick trip to a terminal and it worked! The existing printers returned to System Preferences and printing resumed.

$ cancel -a

Reference

Final Cut Pro X 10.4.4 update

Apple released a new update for Final Cut Pro X, v.10.4.4 and this adds many new features. Many were on the wish lists of Final Cut Pro editors. Also some surprises were included with the update: the inclusion of third party extensions which allow integrations unlike we’ve seen before. Excited to see what’s going to develop in that area.

Editors from around the world are gathered in Cupertino at the FCPX Creative Summit for news about the updates and to share ideas, workflows and learn from one another.

To find out more about the recent updates check out these blog posts and videos made by members of the Final Cut Pro X community. Enjoy.

Ripple Training what’s new

FCP.co blog post

Apple’s FCPX documentation 

Apple’s Compressor documentation

Blocking minor major macOS upgrades

Continuing our theme of welcoming our new macOS overlords, uh, I mean, blocking major macOS upgrades such as macOS 10.14 Mojave with AppBlock we shall examine some other methods of stopping the freight train known as Apple upgrades.

1) A smart person on the MacAdmins Slack posted a useful command to tell macOS not to download major upgrades.

In their testing, running:

`software update –ignore macOSInstallerNotification_GM`

blocks the installation of the Mojave notification package (at /Library/Bundles/OSXNotification.bundle).

However if it already installed, then it’s too late. They pushed out this command prior to that package being distributed by Apple, and they could subsequently see in install.log that the update is being found by softwareupdated but not being installed.

2) If you missed the chance to tell the Mac not to download major macOS upgrades then Rick Heil on his blog has detailed a way using munki to delete the bundle that triggers the macOS upgrade installer.

3) App Block

If your users are intent or their computers are all hell bent on downloading the install app then block it with App block detailed in my previously mentioned blog post

4) Warning

In an effort to get an early warning when users are about to upgrade I use Watchman Monitoring to send me an alert email when a Mac starts downloading the Install macOS app. Sometimes it’s enough of a warning to send an email to a user to ask them whether it is a good idea to upgrade at this time. If storage or software needed for production or backups aren’t qualified or tested thoroughly beforehand then upgrading in the early waves can be less than ideal and frought with peril.

In other interesting and related news, Victor (MicroMDM) was spelunking into the MDM Protocol for what prompts Macs like iOS devices to download major updates. Great post here

If you have any better ways to block macOS upgrades or want to contribute some great solutions let me know. Cheers

 

 

 

 

To install macOS Mojave, or not to?

InstallMojave

Just the other day macOS Mojave was released and now the armies of Macs armed only with the AppStore are silently downloading the installer and ready to upgrade. You can’t hurry too fast to be on the bleeding edge, hurry faster!

Just in case you don’t want everyone to install macOS 10.14.0 (dot zero!) in the first week of its release here’s a way to slow down the upgrade hordes using Erik Berglund’s AppBlocker script. Erik Berglund is also the author of ProfileCreator (for creating profiles) and the author of many other great scripts.

Note: for true binary whitelisting check out Google’s Santa project and Upvote (and Moroz and Zentral, two other Santa sync servers).

Step 1. Get it

Clone or download the AppBlocker project from GitHub

AppleBlockerProject.png

Step 2. Do it

Edit the AppBlocker.py script with the Bundle Identifier of your app to block, in this case for the Mojave installer from the AppStore it is:

com.apple.InstallAssistant.Mojave

You can also edit the alert message, and the icon that is shown, as well as decide if the blocked app should be deleted or not. The script is easy to edit in BBEdit, or nano (in Terminal). Use whatever your favorite text editor is to make the necessary changes.

# List of all blocked bundle identifiers. Can use regexes.
blockedBundleIdentifiers = ['com.apple.InstallAssistant.Mojave']

# Whether the blocked application should be deleted if launched
deleteBlockedApplication = False

# Whether the user should be alerted that the launched applicaion was blocked
alertUser = True

# Message displayed to the user when application is blocked
alertMessage = "The application \"{appname}\" has been blocked by IT"
alertInformativeText = "Contact your administrator for more information"

# Use a custom Icon for the alert. If none is defined here, the Python rocketship will be shown.
alertIconPath = "/System/Library/CoreServices/CoreTypes.bundle/Contents/Resources/Actions.icns"

UPDATED NOTE:

To determine the Bundle identifier of other applications you can use osascript

osascript -e 'id of app "iTunes"'
com.apple.iTunes

If you want to block more than one app use a comma separated list in the AppBlocker.py script:

['com.apple.InstallAssistant.Mojave','com.apple.iTunes']

 

Step 3. Run it

Put the script where you want to run it. The default location as defined in the launchd plist included with the app is “/usr/local/bin”. Put the launchd.plist in “/Library/LaunchDaemons/” and start up your launchd to block your apps!

launchctl load /Library/LaunchDaemons/com.github.erikberglund.AppBlocker.plist

Step 4. Automate it

For bonus points we automate! Bundle it all up in a package with munkipkg, then distribute it with Munki to all your clients.

Using munkipkg is easy. Create the folder using munkipkg

./munkipkg --create AppBlocker

munkipkg: Created new package project at AppBlocker

Then you fill the payload folders with those items you downloaded from the AppBlocker project. LauchD plist in the LaunchDaemons folder and AppBlocker.py in the “usr local bin” (create each nested folder).

AppBlocker-Munkipkg3.png

And finally create a post install script (no “.sh”) with the launchctl action to start your plist.

AppBlocker-Munkipkg4.png

Last but not least add this package to your Munki repo as an unattended managed install  that everyone gets. Of course, only do this after testing your package locally somewhere to verify that it works properly. Remember the saying: “You may not test very often, but when you do it’s always in production.” Be very careful with your testing but always automate all the things.

Updated after the initial blog post to explain how to add more than one app to block, and how to use osascript to determine the bundle identifier.

 

 

 

PostLab: FCP X + GitLab

Final Cut Pro X and Shared Projects: FINALLY !!

I’ve been playing with PostLab the last few days. It’s a free and open source app that lets you use GitLab with Final Cut Pro X to do version control of editing projects. Yes, this is very cool. Shared Projects, Read only versions of projects. Versions. Of. Projects. Commented. Makes it awesome to work on projects together.

Of course, like any workflow app it can be annoying to those who don’t want to play along. But I like the price and the simplicity of it. Using GitLab means you can have free private repos for shared project sharing. You can use their website on the internet to act as your gateway or you can setup your own internal GitLab server. For Free.

PostLab is pretty awesome with its Final Cut Pro X project sharing and it’s not $100K app that is expensive to setup and everyone hates it. It’s free and some people might not use it, but it could allow for effective remote workflows and nice finely grained version control for projects that need it even in an internal on site production SAN environments.

It’s worth checking out.

https://www.postlab.app/

Install PostLab, and the Xcode cli tools. Then launch PostLab, agree to the license, authorize accessibility for PostLab to enable it to launch FCPX. And you’re on your way.

All that’s left is to configure a GitLab account. Set u a group and a project. Configure token in GitLab to Enable PostLab with GitLab account access. Then start sharing projects. Enjoy.

Lots of cool set up videos on the PostLab website. Robot narrator says Jit-Lab instead of “Git” Lab, but it’s still worth watching. Do it now.

PostLab-FCPX-added-fx

 

Updating the P5 client on the Jellyfish

You’ve successfully installed Archiware’s P5 backup and archiving software on your backup server following my previous blog posts and after it has run smoothly for a while you decided to upgrade the version of P5 on your server, but how do you do this on the Lumaforge Jellyfish storage? I’m glad you asked.

There are a couple of ways to update your P5 agent, and I will show you the built in way in Archiware’s P5 software. Surprisingly after many years of using P5 I have never used this method before. I’ve been using Munki for years to upgrade all software on my Mac clients including P5 and on Linux and Solaris servers I’ve just done it by hand. Install over top of the previous version and voila upgrade! But what if you didn’t want to ssh in as root and just install over top, what if there was a better way? I present to you the official “Update client” dialog box. It’s nice.

Update-p5-jellyish-1

Updating client software assumes you’ve set up clients in the P5 server clients section, This is needed when you want to use these server agents to designate their attached storage as a backup, archive or sync source. And also, this assumes you’ve updated your server.

P5 client update Jellyfish 2 Screen Shot 2018-08-06 at 4.40.03 PM

During the update process there are some nice dialog boxes to let you know what is happening.

P5 client update Jellyfish 4 Screen Shot 2018-08-06 at 4.44.34 PM

And afterwards you can test your client with a Ping test.

P5 client update Jellyfish 3 Screen Shot 2018-08-06 at 4.44.27 PM

Success! Looks like we’ve updated our client successfully. How wonderful. And no need to mess about in Terminal with a root shell. No telling what kind of trouble we could get into with those elevated privileges…. much safer this way.

Thanks Archiware for making this great software. I depend on it every day.

 

 

P5 on the Jellyfish: Archiving Gotchas

TL;DR

Using Archiware P5 to Archive files to tapes is awesome, but watch out for little things you might miss, such as the path to the files and backing up your Archive Db.

P5 Archive on the Jellyfish

Using P5 Archive with the Lumaforge Jellyfish is a great way to preserve your digital archives. See this post for how to set up P5 on the Jellyfish

Using Archiware P5 for archiving makes sense. You want your completed projects and original camera footage on LTO tape. But how do you do archives? There are several different ways, and there be gotchas.

P5 Archive vs P5 Archive app

Using P5 Archive to manually archive completed projects to LTO tape is a process of logging into the server via a web browser and selecting the the project folder you want to archive to tape.

The completed project folder could be on the storage visible to the server or it could be storage the client sees. And that can make a difference. Where the storage is mounted is different on a Mac vs Linux. Its’ the difference between “/Volumes” and “/mnt”.

The same Jellyfish storage, either SMB or NFS, when seen on a Mac is mounted by default at “/Volumes” (this can be changed but for most people leave it at the default). But when archiving the storage via a Jellyfish client you will get “/mnt” path.

p5-smb-test2.png

Using the P5 Archive app, which is a Mac only companion application to P5 Archive, to run the archives you will see the storage archived as “/Volumes”.

This first Archiving gotcha is if you’re archiving the Jellyfish storage with the web application of P5 Archive you will have to find your footage and restore from the “/mnt” path vs if you’re archiving from the P5 Archive app which is running from a Mac and will see and store the footage using the “/Volumes” path.

All this to say that using both ways to archive may double up your footage in your archive which may be unintended. And from a restore in the web browser finding your footage may be confusing if you’re used to seeing it mounted in “/Volumes” and you actually find it under “/mnt”.

Note: the reason to use the P5 Archive app is because of the simplicity of right-clicking files in the finder which are on your storage and telling them to archive right then and there. Files are copied to tape then the original files on the storage are replaced with stub files. Right-click again to restore. Simple.

p5-archive-app-job-monitor.png

Backup your Archive!

Don’t forget to backup your archives. Or rather, your archive Db. A more recent addition is the ability to automate the backups on the Archive index, so don’t forget to enable it.

In the managed index section, choose your Archive index.

Set the target client where the backups are going and the backup directory. Choose a time and don’t forget to enable it (check the checkbox and hit apply before closing the windows).

Note: Repeat this setup for each Archive index you want to backup.

Archive Backup db setup3.png

Monitoring your Archive!

Don’t forget to enable email notifications for your P5 server to get your inbox full of status notifications and errors and other important stuff. But if you want to cut down on email notifications or you have multiple P5 servers (many different clients, perhaps), then you might want to check out Watchman Monitoring and the P5 plugin that is built-in). Find out easily when your tape pools are getting low, the tape drives needs to be cleaned, the support maintenance needs renewing etc. All in one dashboard. How convenient!

Maybe everything is going well…

Watchman-P5-info.png

Or maybe not!

Archiware-P5-Jobs-Watchman-tapes-required.png