Apple’s Final Cut Pro has a new proxy workflow. It’s even easier than before. Make proxies on import, or transcode afterwards. Create a new proxy library or copy events with only proxies, so many options to fit the workflow you need. It’s quick to upload smaller proxies to the cloud and work remotely with your team. Re-connect to the original footage for outputs, colour grading and archiving your project when you’re done.
Final Cut Pro and the Proxy Workflow
“Take your creativity anywhere. Maximize portability and performance by creating proxy copies of your media — as low as 1/8 size — in ProRes Proxy or H.264. The latest proxy engine allows you to create a proxy-only copy of your library to share locally or via the cloud and displays original media if proxies aren’t available. Third party tools such as review and approval app frame.io can also generate and deliver proxies to a Final Cut Pro library.” (Apple.com)
I’ll go over the basic workflow for making proxies and getting your library ready for use with Postlab or other similar cloud collaboration tools…. Seriously, there are no other similar tools! But we’ll go over how to keep your library small and light.
Part One – Final Cut Pro
Final Cut Pro 10.5 is the newest version of Final Cut Pro (which drops the “X”). Ready for Apple Silicon Macs and backwards compatible with macOS 10.15.6 (Catalina).
This new proxy workflow is compatible with Final Cut Pro X v10.4.9 and 10.4.10 as well the newest version 10.5. There were extra bug fixes (LUT for proxies) and new methods (copy new library with proxies) in 10.5 but the addition of the automatic proxy creation on import started with 10.4.9.
First step. Check your import preferences. Final Cut will refer to these when importing. The most important thing to check is that “leave files in place” is selected. This helps us keep the library light and portable. Especially important for editing with Final Cut Pro and Postlab. Keep all media and cache files outside of the library. The second this to check is to choose your proxy format (Pro Res Proxy or H264) at the size you want.
Choose how small or how large you want your proxies to be. Smaller proxies are faster to transfer and take up less storage but may not be ideal for editing your specific camera footage. Try to find a format that works best for your edit workflow.
You also have the option of creating proxies form footage that exists already in the library. Choose “Transcode Media” and select your options.
Part Deux – Editing in the Cloud with Postlab
Once you launch Postlab and login you’ll want to create a production and a library to edit. You have the option of importing an existing library or create a new one. Remember, only import your library if it is super light weight and the media is stored outside (not inside) the library.
Importing a lightweight Final Cut Pro library involves creating a name, writing a description and choosing the media location. If editing off centrally shared storage (on premise) or in the cloud (i.e. Postlab drive) then use “Shared” option. If everyone is using their own storage (external hard drives, NAS, SAN, etc) then choose “Individual”.
If you are creating a new empty library in Postlab then be sure to check the Postlab preferences – Templates tab to select what version of Final Cut Pro for the default empty library and if you want to use a Final Cut Pro template you’ve created already. This is a powerful option for keeping a team working with standard set of tools.
Now we start editing. Click “Start Editing” in Postlab. Final Cut Pro will open with your new library.
When you’ve made changes and want to check your Final Cut Pro project back into Postlab switch applications back to Postlab from Final Cut Pro and add a comment.
Once you’ve checked your project in a few times you’ll notice the list of comments you or your team have made with each check in. These will help you decide what project to revert to, if you need to. The icons (on the right) will allow you to revert, open a copy or export out the version you select.
Lastly, there is the status menu which you can use to mark the progress of the project.
I hope this helps you get started with the Final Cut Proxy workflow and ready to use Postlab too.
Setting up your very own Xsan at home… What could be more exciting? Nothing like SAN storage to cure those stacks of hard drive blues. Don’t have a spare fibre channel switch or fibre channel storage at home? No problem Grab some thunderbolt storage from Accusys and join the fun.
I am testing the A12T3-Share 12-drive desktop Thunderbolt RAID solution to build my Xsan. Accusys also have a 16 drive rack mounted raid storage box if you want to install a nice pro set up in the server room you have tucked neatly in your home office. Ha ha. Seriously, the 12 drive unit is whisper quiet and would be a great addition to any home lab or production storage setup. I mean, aren’t we all doing video production at home these days? And even if we are doing a proxy workflow in the clouds, we still need to store the original footage somewhere before it goes to LTO tape, or backed up in the clouds (hopefully another cloud). A few years ago I tested the Accusys 16 drive Thunderbolt 2 unit and it worked perfectly with my fibre channel storage but this time I am testing the newest Thunderbolt 3 unit. Home office test lab is GO!
It is a pretty straight forward setup but I ran into some minor issues that anyone could run into and so I want to mention them and save you all the frustration by learning from my mistakes. Always be learning. That’s my motto. Or “break things at home not in production”, but if your home is production now, then break things fast and learn very quickly.
First step is to download the software for the RAID and you’ll find it on the Accusys website.
(I found the support downloads well organized but still a bit confusing as to what i needed)
The installer is not signed which in our security conscious age is a little concerning, but examining the package with Suspicious package should allay any concerns.
The installer installs the RAIDGuard X app which you will need to configure the RAID.
Of course, RAIDGuard X needs a Java Runtime Environment to run. Why is this still a thing? Hmm…
RAIDGuardX will allow you to configure your connected Thunderbolt hardware.
Configure the array as you like. I only had four drives to test with. Just enough for RAID5.
Choose your favourite RAID level. I picked RAID5 for my 4 drives.
The first gotcha that got me was this surprisingly simple and easy to overlook section. “Assign LUN automatically” asks you to choose which port that LUN (the configured RAID) will be assigned to. If you don’t check anything like I didn’t in my first run through then you configure a RAID5 array that you’ll never see on your connected Mac. Fun, right? Ha ha.
Xsan requires a sacrifice…. I mean, a LUN (available RAID array). Check your Fibre Channel in System Information. Yes, this is from the thunderbolt storage. Hard to believe, but it’s true!
Setting up enterprise grade SAN storage requires a trip to the Mac App Store. Server.app
Open Server.app, enable Xsan, create a new volume and add your LUN from the Accusys Thunderbolt array. Set the usage to “any” (metadata and data) since this is a one LUN test setup.
Pro tip: connect your Xsan controller to your Open Directory server. Ok, just kidding. You don’t have an OD server in your home office? Hmm… Create an entry in /etc/hosts instead.
If you’ve set up your SAN volume then you will see it listed in the Finder.
Easy shareable SAN storage is possible with thunderbolt RAID arrays from Accusys. No more Fibre channel switches needed. Small SAN setups are possible for creative teams without a server room. This setup was a quiet 12 drive RAID and a Mac mini. Add some Thunderbolt cables. There are four thunderbolt 3 connections and you can add more with an additional RAID. Up to 8 connections with one of them for the Mac Mini running the SAN. Not bad at all. And Xsan is free. Add a Server app from the App Store, but the Xsan client is free and built-in (Xsan has been included with macOS since 10.7 so many years ago). Fibre channel protocol (even through Thunderbolt) is faster than network protocols and great for video production. Fast and shareable storage at home. Or in your office. Thunderbolt Xsan. T-SAN.
Note: I want to explain how our current workflow for editing remotely. I am always testing new tools and methods, so workflows change all the time. This is a snapshot in time of what we are trying now. So far it works.
We use Hedge to copy camera cards to multiple drives on set (or after a shoot if on location) and then we use Hedge once more to copy one of these drives to the office shared storage (Apple’s Xsan).
Why use Hedge? A nice simple app which hides its complexity well. Hedge has an easy interface to copy multiple sources (camera cards, usually) to multiple destinations (two external drives, or two SAN locations etc), and it does it well. It verifies, and double checks its work and leaves receipts. What was copied when. This is very nice and very useful for troubleshooting. It also has an API which made it easy to build an app that configures Hedge for its current task, and AppleScript support for extending automations after specified actions.
Kyno and Postlab
We are using two other tools in our remote ingest workflow currently: Kyno from Lesspain software for rewrapping and converting camera footage and Postlab, the remote collaboration tool for Final Cut Pro (and Premiere Pro). Testing with other tools is always ongoing and during a recent test of the workflow we also tried EditReady from Divergent Media.
The Workflow (so far)
While we are exploring various workflow automations we are currently doing the following steps manually.
Hedge to copy camera cards two external drives on set, and then Hedge copy the drive to Xsan
Making re-wrapped in MOV files from the original camera MXF files using Kyno and then
Making H264 MOV 4K proxies in Kyno
Uploading finished proxies to Postlab drive using Hedge
Set up FCPX production and new FCPX library from template connected to proxies in Postlab drive
Hedge and Postlab
Hedge is super useful. Two times good. Hedge and Postlab are best friends. And the UI on both shows the simple aesthetic shared by the developers. Three panes. Source / Start to Destination / Projects. Whether you are copying Proxies to Postlab Drive or accessing your editing projects in Postlab the apps will guide you through.
Details. Rewrap and Proxies
Workflows will depend on your goals, and your available tools. In this case we are using a Canon camera and ingesting MXF files. In order to edit with small Proxies in FCPX but also be able relink to original (and larger) files easily we need to in our case re-wrap the original MXF to QuickTime MOV container.
Originals. Not Proxies.
And to be clear we are treating these in FCPX as “new” originals not as actual FCPX proxies. With the rewrapped MOV files we make transcoded H264 files which are swapped 1 for 1 with the original. When we need to export a final 4K version we can relink to the original 4K source and export easily.
Proxies. Not originals
The transocded H264 4K proxies we made in Kyno were 15x smaller than the original re-wrapped Mov files. We had almost 600GB in originals and 37GB for the 4K H264 proxies!!
Postlab Pro Tips
Working with Postlab pro tip#1 –> keep those FCPX libraries light. Keep all media and cache files out of the library. We knew that and we had Storage Locations set to outside of the library but one new issue came up when the libraries grew really big and we realized the editors were making multiple sequences, not backups, but versions. Now we are trying to work around this habit with Postlab itself. You can check in a version of the library and duplicate library for an alternate version. Modifications of old habits are always tough but technical reasons may force a change in habits here. We will see. Postlab pro tip #2 –> Keep your cache large and fast. By default the Postlab cache is your local drive and only 20GB. If you have a fast SSD or an external drive then move that cache and increase the size. It will help. Trust me.
Kyno vs EditReady
Another small issue we encountered in testing was that we could make the rewrapped Mov files in Kyno or in EditReady and both were fine. The only objection the editors had was that in Kyno we could keep the folder structure of the original camera cards and they felt that this lent some confidence to being able to track the files to the camera card folders if any media was missing or misplaced. The EditReady files kept the original names but they were all in one folder. As the tech I see no issue with FCPX handling these files since we’d be ingesting all the finished proxy files and all the files were named by the camera. Editors should be able to tell which reel the clips were from by the clip name and that’s all you need technically, but you can’t win every argument with an editor. As the tech you need to test alternative tools and methods and see what works technically but also see what can be accepted to work in the way the editors want to work. Changes to workflow are some of the hardest to make, making a system that is used, actually used, by the editors is the goal.
Errors. If you get them, how do you know? This was one area where I could comment on both Kyno and EditReady. I am spoiled by Hedge and it’s nice reports when it is done copying. And Postlab which has a Help menu :collect logs for support button, very nice. If your software tool is going to process a lot of files (rewrapping then transcoding) I want to know if there were errors. EditReady popped up a window to what had succeeded or failed and Divergent Media support told me to look in the logs for any issues encountered. Not great. While Kyno has a separate jobs window which shows jobs done or failed. But still no report. I would like a receipt or report or log at the end with files converted or failed to convert. It would help troubleshooting any issues when they arise. Tech support for both companies is great and responsive. Thanks again. And I’ll keep sending in feature requests.
Testing. More Testing. And Teamwork.
We are testing this workflow in production with a real project and getting feedback from the team. So far the proxies have proven to be easy to make, quick to upload to Postlab drive, simple to use in FCPX in Postlab. Assembling the cut and editing are going well. We will find out about the colour process when we get to that stage and relink to the originals. Stay tuned.
Summer time or anytime is a good time to run a minecraft server. And when I am not troubleshooting IT networks, planning SAN storage upgrades, running a DevOps for Dummies bookclub and the MDOYVR podcast then I like to upgrade my minecraft server.
Every time there is an update to the java client there is demand from my users (uh, I mean, my kids) to immediately stop all other work (hey kids, I’m working here! let Dad work) and upgrade the minecraft server.
Like all other IT domains where there are variety of solutions and software fixes to problems, it would seem that Minecraft has official server downloads as well as the unofficial artisanal craft versions. I’ve tried a few, and some out of desperation… there was an incident with netherite blocks and the server wouldn’t start anymore but the Ppaer minecraft server fixed the issues!
The normal routine is that when an official release comes out the other versions may not be up to date as quick, so it’s back to the official versions.
If only one person needs an application then I think about using Munki to deploy that app. If more than one person should have it then Munki is definitely the way to automate app deployment. And really, if you’re going to take the time to download an app from a website, mount a disk image or un-pack a ZIP archive, run an installer, type an admin password, close that installer … then for the love of all that is good just put the app into your Munki repo and be done with it. Automate it.
Using Munki to solve problems makes sense. Automation helps everyone in this case. But if you’re putting in one off applications into your Munki repo more often than you need to, you need to get those apps into Autopkg. Using Autopkg recipes to download the latest apps and put them into your Munki repo automatically is an automation love fest, but if your apps don’t have recipes what are you going to do? Manually add your apps to Munki? No way. We need a robot 🤖❤️. Recipe robot, that is.
Using Recipe Robot we can build Autopkg recipes for most apps then add the recipes to the Autopkg community to enjoy. Everyone wins.
I recently created recipes for two important apps in my media workflow: Kyno and Hedge. I’ll show an example of this workflow using Recipe Robot and Munki Admin to demonstrate the workflow.
Step 1. Feed the robot.
Drag and and drop the app you want to create your Autopkg recipes.
Step 2. Watch the robot do it’s work
Step 3. Robot is done. Recipes made.
Various type of recipes can be made. I chose download and munki because those are what I am using to automate adding apps to my Munki repo. But there are other options: jss, Filewave, or “install” for example.
Step 4. Run those Recipes
You can use your recipes locally with Autopkg. Run them in Terminal or use Autopkgr , a very nice GUI app for automating the collection and scheduling of recipes. Note: Autopkg and Munki can all be run via cli (command line interface) but for this demo we are showing the GUI apps that are there provided by outstanding members of the community. Many Thanks to them and the contributors to their projects.
Autopkgr app can send notifications in macOS, emails, or post to your Slack group.
Step 5. See the recipes, Use them wisely
Here is an example of newly imported Kyno and Hedge apps in our Munki repo (via Munki Admin GUI).
Add a display name, choose which catalogs the apps will reside in, and check that the description will help explain what the app is.
Note: to get to your home folder hold down the OPTION key and select the Go menu in the Finder.
Compressor is the best sidekick to Apple’s Final Cut Pro X and it gets used a lot. But occasionally something goes awry. It’s software running on a computer. So we troubleshoot. What looked like a stuck running job was mostly leftover evidence of an old job. The Apple support document I found didn’t mention this tip but instead talked about zipping up your settings folder which has all your custom compressor settings for things like YouTube outputs or anything custom. Didn’t seem useful to me to remove but this historical stuff, don’t need it and POOF this solved the issues. It’s not always this easy but something you just take the win and go with it.
Resolve an issue in Compressor: Learn how to isolate, troubleshoot, and fix issues in Compressor.
Recently I was asked to look at a 4TB drive that was only showing less than 2TB available…. No problem, I said, this is easy to fix. Famous last words.
Just open up Disk Utility and resize the partition, or reformat the disk, right? Easy Peasey. Well, it took some troubleshooting to time to figure out and a trip to Terminal was required to solve this weird case, plus I learned a new command along the way. Fun.
Buying a 4TB hard drive then putting it into your external drive case for backups should be simple, but what if instead you got a nasty surprise and it showed up as less than 2TB?
Troubleshooting the issue:
4TB drives were presented to me and when I loaded them into an external SATA dock then showed as 4TB drives with a partitioned volume of less than 2TB.
I tried to delete the phantom partition, and I tried resize the volume to use the empty space in Disk Utility.app but it refused to budge. This needed a trip to Terminal.
Using “man” or “info” commands you can find out more about almost any particular command. Maybe some useful options or arguments would be listed or at least some examples would help.
NAMEdiskutil -- modify, verify and repair local disks
SYNOPSISdiskutil [quiet] verb [options]
DESCRIPTIONdiskutil manipulates the structure of local disks.
To find out more about what we’re faced with let’s ask diskutil what it sees:
Looking through the man page the “resizeVolume” command caught my eye. Also the “limits” option seemed interesting. How
diskutil resizeVolume disk2s2 limits
Resize limits for partition disk2s2 Backup:
Current partition size on map: 1.8 TB (1801419800576 Bytes)
Minimum (constrained by file usage): 846.4 MB (846426112 Bytes)
Recommended minimum (if used for macOS):26.8 GB (26843545600 Bytes)
Maximum (constrained by map space):4.0 TB (4000442028032 Bytes)
Reading through the man page revealed that the best way, and new to me, was to resize the partition to use all available space with “R”. Of course, so intuitive.
sudo diskutil resizeVolume disk2s2 R
I did get some errors. But repairing the disk fixed those issues. And I was able to resize the disk in Terminal with diskutil where Disk Utility.app had failed.
sudo diskutil resizeVolume disk2s2 R
Resizing to full size (fit to fill)
Started partitioning on disk2s2 Backup
Verifying the disk
Verifying file system
Volume was successfully unmounted
Performing fsck_hfs -fn -x /dev/rdisk2s2
Checking Journaled HFS Plus volume
Checking extents overflow file
Checking catalog file
Checking multi-linked files
Checking catalog hierarchy
Checking extended attributes file
Checking volume bitmap
Checking volume information
The volume Backup appears to be OK
File system check exit code is 0
Restoring the original state found as mounted
Modifying partition map
Growing file system
Finished partitioning on disk2s2 Backup
/dev/disk2 (external, physical):
#: TYPE NAMESIZE IDENTIFIER
0:GUID_partition_scheme*4.0 TB disk2
1:EFI 209.7 MB disk2s1
2:Apple_HFS Backup4.0 TB disk2s2
And lastly, the issue may have been caused by the old drive dock which refused to see the 4TB volumes even when correctly resized. A newer drive dock was required.
Use munki-pkg to package up stuff and make your life easier when managing Macs using munki. Here is an example of installing a screensaver.
Why use munki-pkg? How else do you install stuff using munki, run scripts, and version your testing buildings all in one easy to use application? This is all possible with munki-pkg.
Munki-pkg makes package (PKG) installers, Munki likes pkg installers. Munki will also install apps, run scripts, install profiles, and do many things but packages are useful because we can put files in specific places, such as the main computer level screensaver folder, then run a script to set it as a default.
Download munki-pkg and create a working project folder.
Create the folders you need and place your files (payloads) in the right places.
Create your post install script if you need one. Example: setting the screensaver you just installed as the default.
It’s 2019, and NetBoot is almost dead. All new Macs have T2 chips. Sent from the future to protect us from …. ourselves? No more NetBoot, no problem!!
When NetBoot first appeared and I was able to boot entire labs of Macs across the network I was amazed and overjoyed. It was awesome. Spinning globe, spinning…
But in the years since I’ve moved on to no-imaging. Using Munki to manage software means no more imaging, just install Munki and a small config change to point to the Munki server, thereafter the software that should be there goes on, and what’s not supposed to be there goes away. Simple. Just install one package, well, maybe two, then you’re good.
Well, what if you want to streamline or automate these things? What if these are new Macs which don’t have users configured? What if we could do all this from recovery mode? Hmm… Enter bootstrappr and installr!!
This awesome project allows to add packages to install in one step while booted in recovery mode. Plug in a USB stick with the bootstrapr script to run the package install magic or mount a disk image over http. Create a DMG with the included script make_dmg.sh. And now this is the best part: in recovery mode open the Terminal app from Utilities and type:
hdiutil mount http://server/yourDMG.dmg
When it’s done you can Reboot the Mac and you’ll have a set up customized to your liking with Munki installed and configured with custom settings.
The installr script works in the same way but adds the macOS installer to the party. You can also mount the DMG over http and re-image a Mac and then add your custom packages. It’s awesome. Truly amazing.
One note: Added packages in Installr must be in a special format. From the installr site: startosinstall requires that all additional packages be Distribution-style packages (typically built with productbuild) and not component-style packages (typically built with pkgbuild)
In one of my first tests with installr and pycreateuserpkg I was caught up by this, even though it is properly mentioned in the read me. Packages that work in Bootstrappr or munki directly don’t necessarily work when called by the macOS installer (startoinstall). Armin Briegel was helpful in the MacAdmins Slack and reminded me of this. Thanks Armin and thanks everyone on the MacAdmins Slack.
Many Thanks to Greg Neagle for creating these tools and Munki. Looking forward to hearing him speak at the next MacDevOps:YVR conference June 12-14, 2019. Greg will be speaking about his efforts to port some parts of Munki from Python to Swift. More info on the conference and speakers here: https://mdoyvr.com/speakers/
Also a shout out to Graham Gilbert who has worked on Imagr(MDOYVR talk), over the years, an imaging and automation tool which was also an inspiration (along with bootstrappr and installr) to Tim Perfit and his MDS project.
Update: corrected the names of installr and bootstrappr in the title because… autocorrect.