Apple’s Final Cut Pro has a new proxy workflow. It’s even easier than before. Make proxies on import, or transcode afterwards. Create a new proxy library or copy events with only proxies, so many options to fit the workflow you need. It’s quick to upload smaller proxies to the cloud and work remotely with your team. Re-connect to the original footage for outputs, colour grading and archiving your project when you’re done.
Final Cut Pro and the Proxy Workflow
“Take your creativity anywhere. Maximize portability and performance by creating proxy copies of your media — as low as 1/8 size — in ProRes Proxy or H.264. The latest proxy engine allows you to create a proxy-only copy of your library to share locally or via the cloud and displays original media if proxies aren’t available. Third party tools such as review and approval app frame.io can also generate and deliver proxies to a Final Cut Pro library.” (Apple.com)
I’ll go over the basic workflow for making proxies and getting your library ready for use with Postlab or other similar cloud collaboration tools…. Seriously, there are no other similar tools! But we’ll go over how to keep your library small and light.
Part One – Final Cut Pro
Final Cut Pro 10.5 is the newest version of Final Cut Pro (which drops the “X”). Ready for Apple Silicon Macs and backwards compatible with macOS 10.15.6 (Catalina).
This new proxy workflow is compatible with Final Cut Pro X v10.4.9 and 10.4.10 as well the newest version 10.5. There were extra bug fixes (LUT for proxies) and new methods (copy new library with proxies) in 10.5 but the addition of the automatic proxy creation on import started with 10.4.9.
First step. Check your import preferences. Final Cut will refer to these when importing. The most important thing to check is that “leave files in place” is selected. This helps us keep the library light and portable. Especially important for editing with Final Cut Pro and Postlab. Keep all media and cache files outside of the library. The second this to check is to choose your proxy format (Pro Res Proxy or H264) at the size you want.
Choose how small or how large you want your proxies to be. Smaller proxies are faster to transfer and take up less storage but may not be ideal for editing your specific camera footage. Try to find a format that works best for your edit workflow.
You also have the option of creating proxies form footage that exists already in the library. Choose “Transcode Media” and select your options.
Part Deux – Editing in the Cloud with Postlab
Once you launch Postlab and login you’ll want to create a production and a library to edit. You have the option of importing an existing library or create a new one. Remember, only import your library if it is super light weight and the media is stored outside (not inside) the library.
Importing a lightweight Final Cut Pro library involves creating a name, writing a description and choosing the media location. If editing off centrally shared storage (on premise) or in the cloud (i.e. Postlab drive) then use “Shared” option. If everyone is using their own storage (external hard drives, NAS, SAN, etc) then choose “Individual”.
If you are creating a new empty library in Postlab then be sure to check the Postlab preferences – Templates tab to select what version of Final Cut Pro for the default empty library and if you want to use a Final Cut Pro template you’ve created already. This is a powerful option for keeping a team working with standard set of tools.
Now we start editing. Click “Start Editing” in Postlab. Final Cut Pro will open with your new library.
When you’ve made changes and want to check your Final Cut Pro project back into Postlab switch applications back to Postlab from Final Cut Pro and add a comment.
Once you’ve checked your project in a few times you’ll notice the list of comments you or your team have made with each check in. These will help you decide what project to revert to, if you need to. The icons (on the right) will allow you to revert, open a copy or export out the version you select.
Lastly, there is the status menu which you can use to mark the progress of the project.
I hope this helps you get started with the Final Cut Proxy workflow and ready to use Postlab too.
Note: I want to explain how our current workflow for editing remotely. I am always testing new tools and methods, so workflows change all the time. This is a snapshot in time of what we are trying now. So far it works.
We use Hedge to copy camera cards to multiple drives on set (or after a shoot if on location) and then we use Hedge once more to copy one of these drives to the office shared storage (Apple’s Xsan).
Why use Hedge? A nice simple app which hides its complexity well. Hedge has an easy interface to copy multiple sources (camera cards, usually) to multiple destinations (two external drives, or two SAN locations etc), and it does it well. It verifies, and double checks its work and leaves receipts. What was copied when. This is very nice and very useful for troubleshooting. It also has an API which made it easy to build an app that configures Hedge for its current task, and AppleScript support for extending automations after specified actions.
Kyno and Postlab
We are using two other tools in our remote ingest workflow currently: Kyno from Lesspain software for rewrapping and converting camera footage and Postlab, the remote collaboration tool for Final Cut Pro (and Premiere Pro). Testing with other tools is always ongoing and during a recent test of the workflow we also tried EditReady from Divergent Media.
The Workflow (so far)
While we are exploring various workflow automations we are currently doing the following steps manually.
Hedge to copy camera cards two external drives on set, and then Hedge copy the drive to Xsan
Making re-wrapped in MOV files from the original camera MXF files using Kyno and then
Making H264 MOV 4K proxies in Kyno
Uploading finished proxies to Postlab drive using Hedge
Set up FCPX production and new FCPX library from template connected to proxies in Postlab drive
Hedge and Postlab
Hedge is super useful. Two times good. Hedge and Postlab are best friends. And the UI on both shows the simple aesthetic shared by the developers. Three panes. Source / Start to Destination / Projects. Whether you are copying Proxies to Postlab Drive or accessing your editing projects in Postlab the apps will guide you through.
Details. Rewrap and Proxies
Workflows will depend on your goals, and your available tools. In this case we are using a Canon camera and ingesting MXF files. In order to edit with small Proxies in FCPX but also be able relink to original (and larger) files easily we need to in our case re-wrap the original MXF to QuickTime MOV container.
Originals. Not Proxies.
And to be clear we are treating these in FCPX as “new” originals not as actual FCPX proxies. With the rewrapped MOV files we make transcoded H264 files which are swapped 1 for 1 with the original. When we need to export a final 4K version we can relink to the original 4K source and export easily.
Proxies. Not originals
The transocded H264 4K proxies we made in Kyno were 15x smaller than the original re-wrapped Mov files. We had almost 600GB in originals and 37GB for the 4K H264 proxies!!
Postlab Pro Tips
Working with Postlab pro tip#1 –> keep those FCPX libraries light. Keep all media and cache files out of the library. We knew that and we had Storage Locations set to outside of the library but one new issue came up when the libraries grew really big and we realized the editors were making multiple sequences, not backups, but versions. Now we are trying to work around this habit with Postlab itself. You can check in a version of the library and duplicate library for an alternate version. Modifications of old habits are always tough but technical reasons may force a change in habits here. We will see. Postlab pro tip #2 –> Keep your cache large and fast. By default the Postlab cache is your local drive and only 20GB. If you have a fast SSD or an external drive then move that cache and increase the size. It will help. Trust me.
Kyno vs EditReady
Another small issue we encountered in testing was that we could make the rewrapped Mov files in Kyno or in EditReady and both were fine. The only objection the editors had was that in Kyno we could keep the folder structure of the original camera cards and they felt that this lent some confidence to being able to track the files to the camera card folders if any media was missing or misplaced. The EditReady files kept the original names but they were all in one folder. As the tech I see no issue with FCPX handling these files since we’d be ingesting all the finished proxy files and all the files were named by the camera. Editors should be able to tell which reel the clips were from by the clip name and that’s all you need technically, but you can’t win every argument with an editor. As the tech you need to test alternative tools and methods and see what works technically but also see what can be accepted to work in the way the editors want to work. Changes to workflow are some of the hardest to make, making a system that is used, actually used, by the editors is the goal.
Errors. If you get them, how do you know? This was one area where I could comment on both Kyno and EditReady. I am spoiled by Hedge and it’s nice reports when it is done copying. And Postlab which has a Help menu :collect logs for support button, very nice. If your software tool is going to process a lot of files (rewrapping then transcoding) I want to know if there were errors. EditReady popped up a window to what had succeeded or failed and Divergent Media support told me to look in the logs for any issues encountered. Not great. While Kyno has a separate jobs window which shows jobs done or failed. But still no report. I would like a receipt or report or log at the end with files converted or failed to convert. It would help troubleshooting any issues when they arise. Tech support for both companies is great and responsive. Thanks again. And I’ll keep sending in feature requests.
Testing. More Testing. And Teamwork.
We are testing this workflow in production with a real project and getting feedback from the team. So far the proxies have proven to be easy to make, quick to upload to Postlab drive, simple to use in FCPX in Postlab. Assembling the cut and editing are going well. We will find out about the colour process when we get to that stage and relink to the originals. Stay tuned.
Great news everyone. Hedge has acquired PostLab and with this news the main developer Jasper Siegers joins the Hedge team. Great things will come of this collaboration. The brilliant idea of using version control for FCPX project sharing simplifies the whole process that has usually been left to huge media asset management systems that control data and projects. This is project sharing simplified and it’s about to get a lot more awesome with more developer time and a company to support it.
I am equally excited about Jasper coming to MacDevOps:YVR in June to talk about how and why he developed this app using GitLab and Docker and how version control for editors is useful and necessary.
Hedge is a small company in the Netherlands that makes the Hedge software to securely copy camera cards. I’ve incorporated this software with my Final Cut Pro X clients to secure their original footage on location as well as back in the office. Workflow post explains more.
Quick post to talk about some fun I’ve been having with the new Hedge API.
Background: Hedge is an awesome app for securely copying Camera Cards to multiple destinations to ensure data integrity and safety of the original footage. Hedge is one part of a workflow I build for my clients. Hedge is the first step in ensuring an easy and convenient transition from the cameras to the SAN to the tape archive powered by Archiware P5.
Using AppleScript and Automator I have built some apps to quickly set the file naming and data integrity preferences as we want them to be. And also quickly change them to something else depending on the needed workflow.
API or Clickety click click
Copying camera cards copies is what we use Hedge for. Certain preferences like logs or receipts are great to have to ensure the copy succeeded. Also file naming conventions are good to have. Set and forget, right? But what if you did forget? Or if you’re new and don’t know the convention or you don’t read documentation. What do we do? One way to solve this is build an app that launches Hedge and sets the correct preferences. And if we want to copy USB sticks or something else then we can launch another app that prepares Hedge with a different set of preferences. For extra points we just ask the user what they want with a nice dialog box and just do that.
Automator is awesome. Create workflows, apps, or services amongst many other things. For more info on Automator check out Apple’s official docs or this unofficial website of resources.
For this quick testing I used two AppleScripts with different preferences and settings defined. One for camera card copying and another for USB sticks that need different preferences set. The fancy automator app just calls the needed AppleScript. Make two apps and you have two different workflows without having to explain to users which prefs get set for what, or how the file naming should go to be consistent.
Example 1: Cam Card script (snippet)
Example 2: USB card copy (snippet)
Automator can do many things. Call AppleScripts, Run shell scripts, pop-up dialog boxes etc and this is just a simple example of building single purpose apps to set Hedge via its new API. Very cool and so many possibilities.
What if we could just build one app which asked the user what they wanted to do? We can do that!
Choose “Cam” and the appropriate preferences are set and file naming conventions applied.
Choose “USB” and a different set of preferences are set. Magic.
How do we do this? This piece of AppleScript chains an action to a response or button choice. Run a script or choose an action. The possibilities are endless. And thanks to the Hedge API we can set preferences on or off, and set destinations or many other things. We can do them programmatically and we can ensure they are set correctly. Fun times!
AppleScript vs POSIX:
I updated my AppleScript code with the POSIX path of the scripts it wants to load. It’s a major improvement! I had packaged up my scripts and my Hedge Setup app with munkipkg then deployed through Munki but when I demoed it — nothing worked…. because the start up disk drive was named something else. The fix: set a variable to be the POSIX path (Unix path in AppleScript friendly format).
Scripting and App Building
I hope that helped. We can do a lot of the same things with Python and in my testing I was working with a script written in python3 but since that’s not shipping on Macs in the current version of macOS by default (not yet!) then AppleScript was the quickest way to get this done. This is not restricted to AppleScript. Using Automator and your favourite scripting language you can build apps for your clients, co-workers, friends and family.
A note about the Hedge API:
There are two major calls I use in my scripts “setDestination” and “setPreferences”
The “setDestination” call looks like this:
Note: the token is generated for you when you have a Pro license.
The “the setPreferences” call uses plist keys.
Note: I’ll have more say about using the actual Hedge API after it is officially announced.
Part of a series of blog post on the “Best of 2018”
Part 1: the iMac Pro and FCPX
The year started off with the new iMac Pro and Final Cut Pro X 10.4. Both new hardware and software were released in December 2017. New awesome hardware and software to start of 2018.
FCPX and the iMac Pro have proven themselves to be a great combination that has been amazing for FCPX editors everywhere. The new colour grading tools and other enhancements were warmly received in FCP X 10.4. The power of the iMac Pros was not exaggerated. Excellent pro hardware.
FCPX works great on a MacBook Pro and internal storage, with Apple’s Xsan and fibre channel or with Lumaforge Jellyfish 10GbE over NFS. I worked with all different setups in 2018 and happy to report that editors kept editing and left the storage and backup worries to me (and I didn’t worry since I’ve got Archiware P5 watching my back).
Working with the Jellyfish I installed the P5 Linux agent to backup and archive to tape. Getting the Jellyfish to back up to my P5 server running on a Mac Mini couldn’t have been easier. Through the year I worked with Archiware to make improvements in the P5 Archive app so that my editor clients can archive and restore more easily on their own. Works well and look forward to working more closely with both companies to help make awesome setups for FCPX editors and creative professionals everywhere.
NAB and FCPX
The week before NAB 2018, Apple announced a new version of Final Cut Pro X with support for closed captions, and the brand new ProRes RAW codec.
NAB in April is always a busy month with announcements from all companies in the media production and media asset management world and Apple’s public talk at NAB showing off new features so soon after their last major release was unexpected but very warmly received.
Of course there was one more major event in the 2018, in November there was the FCPX Creative Summit.
I attended this year and it was awesome. Apple released a brand new version with 3rd party integration in the form of extensions. This is huge. This will be amazing for FCPX editors who want to stay in FCPX and do their editing work but integrate with other apps.
What was the FCPX creative summit?
⁃ rendez-vous in Cupertino with Final Cut Pro editors, studio owners, plugin authors, creative apps vendors
⁃ Visit to Apple HQ. With Apple Pro Apps engineers, QA, managers and everyone involved.
⁃ In depth discussion of the next version of FCPX extensions which allow third party integration deep into the app for example: Frame IO for review and approve or Keyflow Pro or Cat DV media asset management apps.
⁃ Great team of people organizing. This event had multiple tracks and lots of great sessions for everyone. Well done. Enjoyed it immensely. Everyone using Final Cut Pro or involved in this creative universe should be there.
2018 was great year for pro hardware and software. The iMac Pro and the constant stream of FCPX updates kept us grinning from ear to ear. Great stuff. Awesome year.
I’ve been playing with PostLab the last few days. It’s a free and open source app that lets you use GitLab with Final Cut Pro X to do version control of editing projects. Yes, this is very cool. Shared Projects, Read only versions of projects. Versions. Of. Projects. Commented. Makes it awesome to work on projects together.
Of course, like any workflow app it can be annoying to those who don’t want to play along. But I like the price and the simplicity of it. Using GitLab means you can have free private repos for shared project sharing. You can use their website on the internet to act as your gateway or you can setup your own internal GitLab server. For Free.
PostLab is pretty awesome with its Final Cut Pro X project sharing and it’s not $100K app that is expensive to setup and everyone hates it. It’s free and some people might not use it, but it could allow for effective remote workflows and nice finely grained version control for projects that need it even in an internal on site production SAN environments.
For editing clients with a proper SAN this is the setup I like to use a watch folder on the SAN that sends to tape the camera archives automatically on a timed interval.
a watch folder setup with Archiware P5 archive
camera archives, created in FCP X from the camera cards
Note: you can also use Adobe Prelude, Shotput or Resolve to create verified copies of camera cards. Use what you trust and works for you. The idea is not to copy by hand and avoid the perils of corrupt files.
Bonus: multiple drives to enable multi-streaming and parallelizing of your data. Why not makes cloned tapes copies and stream lots of data to all four (yes, four !) drives. See the illustration below.
Let’s say you work in the media and entertainment industry, perhaps in Post production, and maybe, just maybe, you shoot a lot of digital film (R3D, Arri RAW, XDCAM, etc), and just maybe you have a SAN. And maybe you’re lucky and you set up a nice backup system using Archiware P5, for example. What about archive? What about finished projects, what do you do about that? Even more importantly, what do you do about the camera archives? That’s the digital film footage that comes in and gets copied to the SAN before creative work begins… it seems like important stuff. Very important to backup, and even better, to archive! How can we set up an automatic archive in Archiware P5?
DEFINITION: What is the difference between backup and archive? In the media and entertainment industry I would define backup as a continuous data protection of live data on a production volume, while archive is a copy of a finished project or original media that will be removed from the production volume and must kept safe for future retrieval. Backups will roll over (a new full backup every week, or month) and if that is all we had, then footage or projects that are done and gone off the SAN would be then lost. Archive separates out the finished projects or source material as needing an independent safekeeping. P5 Archive also has the option of creating a mini-MAM type database of proxy files that can be easily viewed in a web browser for quick identification of files to be restored.
So, how do we set up an automatic archive in Archiware P5? We want it to be automatic so we don’t have to think about it, since manually archiving like backups can be forgotten. If we set up a watch folder on the SAN then we just have to instruct everyone to drop their camera masters and other source files into the folder when they copy to the SAN and P5 will automatically archive them to an incremental tape archive. Wow. That’s awesome.
Tips and Tricks:If you edit with Final Cut Pro X then I recommend using it to make Camera Archives (a verified copy of the footage from the original card or drive) then placing this in the watch folder. If you’re using Adobe’s Premiere workflow, then Prelude can make a verified copy as well, but not in the same way exactly. This will be the subject of another post. Stay tuned.
What does an automatic archive look like?
P5 Archive watch folder
Pool (designated tapes, or a disk, for archive)
Index (could be the archive default index, or a unique new index)
Plan (an archive plan specifies the pool and index used as well as the what and how)
Tips and Tricks:If you’re generating QT previews then you’ll want to move your index off of the default, or else you might fill up your system hard drive with movie files and then your backup server won’t boot. No, I never done this. OK, trust me it will happen 🙂
Tips and Tricks:Make sure to set up a backup job of the Archive index. This is a safety measure. The archive index is not saved to the tape in the same way backup jobs information. You need to create a backup job specifically to save your archive index. But you’re already running a backup job to backup your Archiware main index, right? Uh huh, thought so.
Tips and Tricks:When creating the archive pool set it at 512KB media block size for faster archiving of big video files.
Use the advanced options to create a new Archive Index, select its location and optionally create new additional fields to help searching for archived projects.
Incremental or full? QT previews, yes or no? For an automatic archiving set up I suggest incremental archive and no QT previews. Using previews is up to you (and your disk space available), so plan accordingly.
If you have any questions or need help setting this up please do not hesitate to contact me. In the meantime, I’ll work on a nice PDF doc to summarize the setup. Download a demo of Archiware P5 and give it a whirl. You’ll see that it’s super easy to create an automatic archive location on your SAN and your editors will thank you when you can restore their files that they need (when they need it!).