Category Archives: MAM

From Camera to the Clouds: the very real story of Hedge and Postlab

Posted on by
3

Update: This proxy workflow applies to Final Cut Pro 10.4.8 and earlier. For the new proxy workflow with FCPX 10.4.9 and Final Cut Pro 10.5 see my new blog post

Note: I want to explain how our current workflow for editing remotely. I am always testing new tools and methods, so workflows change all the time. This is a snapshot in time of what we are trying now. So far it works.

Hedge

We use Hedge to copy camera cards to multiple drives on set (or after a shoot if on location) and then we use Hedge once more to copy one of these drives to the office shared storage (Apple’s Xsan).

Why use Hedge? A nice simple app which hides its complexity well. Hedge has an easy interface to copy multiple sources (camera cards, usually) to multiple destinations (two external drives, or two SAN locations etc), and it does it well. It verifies, and double checks its work and leaves receipts. What was copied when. This is very nice and very useful for troubleshooting. It also has an API which made it easy to build an app that configures Hedge for its current task, and AppleScript support for extending automations after specified actions.

Kyno and Postlab

We are using two other tools in our remote ingest workflow currently: Kyno from Lesspain software for rewrapping and converting camera footage and Postlab, the remote collaboration tool for Final Cut Pro (and Premiere Pro). Testing with other tools is always ongoing and during a recent test of the workflow we also tried EditReady from Divergent Media.

The Workflow (so far)

While we are exploring various workflow automations we are currently doing the following steps manually.

  1. Hedge to copy camera cards two external drives on set, and then Hedge copy the drive to Xsan
  2. Making re-wrapped in MOV files from the original camera MXF files using Kyno and then
  3. Making H264 MOV 4K proxies in Kyno
  4. Uploading finished proxies to Postlab drive using Hedge
  5. Set up FCPX production and new FCPX library from template connected to proxies in Postlab drive

Hedge and Postlab

Hedge is super useful. Two times good. Hedge and Postlab are best friends. And the UI on both shows the simple aesthetic shared by the developers. Three panes. Source / Start to Destination / Projects. Whether you are copying Proxies to Postlab Drive or accessing your editing projects in Postlab the apps will guide you through.

Copying the Proxies with Hedge to Postlab Drive.

Details. Rewrap and Proxies

Workflows will depend on your goals, and your available tools. In this case we are using a Canon camera and ingesting MXF files. In order to edit with small Proxies in FCPX but also be able relink to original (and larger) files easily we need to in our case re-wrap the original MXF to QuickTime MOV container.

Right click on a clip in Kyno to rewrap to Mov.

Originals. Not Proxies.

And to be clear we are treating these in FCPX as “new” originals not as actual FCPX proxies. With the rewrapped MOV files we make transcoded H264 files which are swapped 1 for 1 with the original. When we need to export a final 4K version we can relink to the original 4K source and export easily.

Proxies. Not originals

The transocded H264 4K proxies we made in Kyno were 15x smaller than the original re-wrapped Mov files. We had almost 600GB in originals and 37GB for the 4K H264 proxies!!

Postlab Pro Tips

Working with Postlab pro tip #1 –> keep those FCPX libraries light. Keep all media and cache files out of the library. We knew that and we had Storage Locations set to outside of the library but one new issue came up when the libraries grew really big and we realized the editors were making multiple sequences, not backups, but versions. Now we are trying to work around this habit with Postlab itself. You can check in a version of the library and duplicate library for an alternate version. Modifications of old habits are always tough but technical reasons may force a change in habits here. We will see. Postlab pro tip #2 –> Keep your cache large and fast. By default the Postlab cache is your local drive and only 20GB. If you have a fast SSD or an external drive then move that cache and increase the size. It will help. Trust me.

Kyno vs EditReady

Another small issue we encountered in testing was that we could make the rewrapped Mov files in Kyno or in EditReady and both were fine. The only objection the editors had was that in Kyno we could keep the folder structure of the original camera cards and they felt that this lent some confidence to being able to track the files to the camera card folders if any media was missing or misplaced. The EditReady files kept the original names but they were all in one folder. As the tech I see no issue with FCPX handling these files since we’d be ingesting all the finished proxy files and all the files were named by the camera. Editors should be able to tell which reel the clips were from by the clip name and that’s all you need technically, but you can’t win every argument with an editor. As the tech you need to test alternative tools and methods and see what works technically but also see what can be accepted to work in the way the editors want to work. Changes to workflow are some of the hardest to make, making a system that is used, actually used, by the editors is the goal.

Errata

Errors. If you get them, how do you know? This was one area where I could comment on both Kyno and EditReady. I am spoiled by Hedge and it’s nice reports when it is done copying. And Postlab which has a Help menu :collect logs for support button, very nice. If your software tool is going to process a lot of files (rewrapping then transcoding) I want to know if there were errors. EditReady popped up a window to what had succeeded or failed and Divergent Media support told me to look in the logs for any issues encountered. Not great. While Kyno has a separate jobs window which shows jobs done or failed. But still no report. I would like a receipt or report or log at the end with files converted or failed to convert. It would help troubleshooting any issues when they arise. Tech support for both companies is great and responsive. Thanks again. And I’ll keep sending in feature requests.

Testing. More Testing. And Teamwork.

We are testing this workflow in production with a real project and getting feedback from the team. So far the proxies have proven to be easy to make, quick to upload to Postlab drive, simple to use in FCPX in Postlab. Assembling the cut and editing are going well. We will find out about the colour process when we get to that stage and relink to the originals. Stay tuned.

Thanks!

Thanks to Felipe Baez / cr8ivebeast for his assistance on this part of the workflow. We were having trouble relinking to the original MXF and he gave us the excellent tip to rewrap then in Kyno then make the smaller proxies. Works like a charm. Thank you Felipe! Here’s a link to a video Felipe made showing a similar procedure using Compressor to transcode and then relink in FCPX and it goes to show you that there are lot of ways to do things and to keep trying, and experimenting. You might learn a thing or two.

Automate it! Hedge API example apps

Posted on by
2

Quick post to talk about some fun I’ve been having with the new Hedge API.

Background: Hedge is an awesome app for securely copying Camera Cards to multiple destinations to ensure data integrity and safety of the original footage. Hedge is one part of a workflow I build for my clients. Hedge is the first step in ensuring an easy and convenient transition from the cameras to the SAN to the tape archive powered by Archiware P5.

TL;DR

Using AppleScript and Automator I have built some apps to quickly set the file naming and data integrity preferences as we want them to be.  And also quickly change them to something else depending on the needed workflow.

API or Clickety click click

Copying camera cards copies is what we use Hedge for. Certain preferences like logs or receipts are great to have to ensure the copy succeeded. Also file naming conventions are good to have. Set and forget, right? But what if you did forget? Or if you’re new and don’t know the convention or you don’t read documentation. What do we do? One way to solve this is build an app that launches Hedge and sets the correct preferences. And if we want to copy USB sticks or something else then we can launch another app that prepares Hedge with a different set of preferences.  For extra points we just ask the user what they want with a nice dialog box and just do that.

hedge

Automator

Automator is awesome. Create workflows, apps, or services amongst many other things. For more info on Automator check out Apple’s official docs or this unofficial website of resources.

Automator.png

For this quick testing I used two AppleScripts with different preferences and settings defined. One for camera card copying and another for USB sticks that need different preferences set. The fancy automator app just calls the needed AppleScript. Make two apps and you have two different workflows without having to explain to users which prefs get set for what, or how the file naming should go to be consistent.

Example 1: Cam Card script (snippet)Hedge-Automator.png

Example 2: USB card copy (snippet)

Hedge-ApplesriptUSB.png

 

Automator can do many things. Call AppleScripts, Run shell scripts, pop-up dialog boxes etc and this is just a simple example of building single purpose apps to set Hedge via its new API. Very cool and so many possibilities.

AppleScript

What if we could just build one app which asked the user what they wanted to do? We can do that!

AppleScript-DialogQuit.png

Choose “Cam” and the appropriate preferences are set and file naming conventions applied.

Hedge-CamCard-Prefs.png

Hedge-CamCard-FileNaming.png

Choose “USB” and a different set of preferences are set. Magic.

Hedge-USBCard-Prefs.png

Hedge-USBCard-FileNaming.png

How do we do this? This piece of AppleScript chains an action to a response or button choice. Run a script or choose an action. The possibilities are endless. And thanks to the Hedge API we can set preferences on or off, and set destinations or many other things. We can do them programmatically and we can ensure they are set correctly. Fun times!

Hedge-API-Script-Quit.png

AppleScript vs POSIX:

I updated my AppleScript code with the POSIX path of the scripts it wants to load. It’s a major improvement! I had packaged up my scripts and my Hedge Setup app with munkipkg then deployed through Munki but when I demoed it — nothing worked…. because the start up disk drive was named something else. The fix: set a variable to be the POSIX path (Unix path in AppleScript friendly format).

Scripting and App Building

I hope that helped. We can do a lot of the same things with Python and in my testing I was working with a script written in python3 but since that’s not shipping on Macs in the current version of macOS by default (not yet!) then AppleScript was the quickest way to get this done. This is not restricted to AppleScript. Using Automator and your favourite scripting language you can build apps for your clients, co-workers, friends and family.

A note about the Hedge API:

There are two major calls I use in my scripts “setDestination” and “setPreferences”

The “setDestination” call looks like this:

open ‘hedge://actions?json=[{“setDestination”:{“path”:”/Volumes/LaCie/Testing/Test1″}},{“token”:”1234567890123345555″}]’

Note: the token is generated for you when you have a Pro license.

The “the setPreferences” call uses plist keys.

Note: I’ll have more say about using the actual Hedge API after it is officially announced.

PostLab: FCP X + GitLab

Posted on by
2

Final Cut Pro X and Shared Projects: FINALLY !!

I’ve been playing with PostLab the last few days. It’s a free and open source app that lets you use GitLab with Final Cut Pro X to do version control of editing projects. Yes, this is very cool. Shared Projects, Read only versions of projects. Versions. Of. Projects. Commented. Makes it awesome to work on projects together.

Of course, like any workflow app it can be annoying to those who don’t want to play along. But I like the price and the simplicity of it. Using GitLab means you can have free private repos for shared project sharing. You can use their website on the internet to act as your gateway or you can setup your own internal GitLab server. For Free.

PostLab is pretty awesome with its Final Cut Pro X project sharing and it’s not $100K app that is expensive to setup and everyone hates it. It’s free and some people might not use it, but it could allow for effective remote workflows and nice finely grained version control for projects that need it even in an internal on site production SAN environments.

It’s worth checking out.

https://www.postlab.app/

Install PostLab, and the Xcode cli tools. Then launch PostLab, agree to the license, authorize accessibility for PostLab to enable it to launch FCPX. And you’re on your way.

All that’s left is to configure a GitLab account. Set u a group and a project. Configure token in GitLab to Enable PostLab with GitLab account access. Then start sharing projects. Enjoy.

Lots of cool set up videos on the PostLab website. Robot narrator says Jit-Lab instead of “Git” Lab, but it’s still worth watching. Do it now.

PostLab-FCPX-added-fx

 

Best of 2015: VidiXplore

Posted on by
Reply

This is another great product of 2015 and  when I found it, I thought VidiXplore proves that Media Asset Management could be done better and simpler. At the end of December 2015 they released version 1.0 with new some tricks, including some changes to make a migration from Final Cut Server a reality. Time to move some clients!!

To sum up VidiXplore, I’ll quote my tweet from Dec 21, 2015:

Finally the perfect solution for simple asset management! Keep proxies in the cloud, originals local. Search + share!

Working with video editors, animation and visual effects studios, I’ve come to realize that media asset management (MAM) systems can be complicated and painful. Changing workflow, oh no! Building a better pipeline is not easy, nor is it always welcome. Well, hello from the other side, we found the solution, or at least part of it.

With VidiXplore you have 3 steps:

Step 1. Manage your videos by keeping all the originals local. Use your own storage. Use your folder structure. Use your vids as you would normally. Don’t pay for cloud storage.

Step 2. Proxies (thumbnail vids) go into the cloud to be viewed by you and your team.

Step 3. Organize your videos and photos into collections, batch edit by adding tags to add metadata, search for particular assets and share them with colleagues and external clients.

That’s it. You’re already ahead of the game. We skipped right over step 4 which was “have a lot of meetings to debate proper metadata” and step 5 which was “convince everyone at the company to adopt a different workflow.”

With VidiXplore you switch to a monthly payment model, that is true. You don’t own the cloud platform, but what you gain is that you don’t pay a lot of money upfront to set up a large server (or many large server), nor do you need render farm for video transcoding nor for the databases you need to keep track of it all. Pay monthly. That’s the way for a lot of smaller companies. Lower up front starting cost. No extreme capital outlay in the beginning.

Honestly, VidiXplore is a refreshing and easy way for so many people to use asset management now, so why not try it? Harder to say that with a large system setup that costs a lot of money to set up, only to find that no one wants to use it. That’s not what anyone wants.

And now for something completely different…

Let’s take a quick look at VidiXplore. If you’ve installed the VidiXploreAgent-1.0 agent then you’ll have a nice “V” icon in your menu bar (Mac) or system tray (Windows). Use this to open the VidiXplore agent.

Vx Menu agent Open

In the VidiXplore menu you can access the settings where can you set whether certain file types get a Cloud Copy uploaded by default or how many concurrent jobs can run at once.

Vx agent prefs settings cloud copy

When the agent is open you’ll see folders you’ve configured for media, and an option to go the website of your particular instance of VidiXplore.

Agent

For the first web login you’ll see an intro screen which allows you to upload additional media to VidiXplore (that is, in addition to any particular media folders you’ve configured in the agent) and the option to connect cloud storage such as S3, Azure or Dropbox. Lastly, there are also the installers for the agent.

Welcome Screen Shot 2016-01-29 at 11.43.20 AM

Once logged into your website you see a basic layout with options to sort by files or collections, to specify all files or particular types, sort by location and bulk actions.

Collections Screen Shot

If we want we can sort the results and see only collections:

Sort collections

Or we can sort by files which have a cloud copy:

Sort cloud copy images

So many options to sort, search and find what we’re looking for. Of course, we want to also edit the metadata but not so much that we require weekly meetings to decide on the 500 important and required metadata fields. Just use tags. Of course, meetings are good, and so is process, but it is so quick to select bulk actions and add a tag to a group of objects. Done!

 

add tags bulk action MacDevOps

“Finally the perfect solution for simple asset management! Keep proxies in the cloud, originals local. Search + share!” 

There’s so much more you can do with VidiXplore, and I’ll go into more detail in another blog post, but this was just a highlight for my best of 2015.

Check out their website for more information:

VidiXplore.com

2015 in review

Posted on by
Reply

The WordPress.com stats helper monkeys prepared a 2015 annual report for this blog, but nobody wants to read no stinkin’ reports so let me just sum it all up: Xsan, Munki, Thunderbolt, Archives. Or is that all one word? Thunderbolt Xsan Munki Archives! That’s better.

Here’s an excerpt from the report that no one will read:

The concert hall at the Sydney Opera House holds 2,700 people. This blog was viewed about 20,000 times in 2015. If it were a concert at Sydney Opera House, it would take about 7 sold-out performances for that many people to see it.

Surprisingly, or not, that opera would be about Xsan. Yes, Apple’s Xsan is still alive, and Apple even added new features with OS X 10.11 El Capitan. I’m still building Xsan shared storage SANs and upgrading old ones to new versions. That was one of the good news stories of 2015 for me.

You can build an Xsan with one or two Mac Minis and add your storage of choice. That used to mean more often than not the fibre channel storage from Promise. A great choice for larger deployments, the x30 Vtraks are solid.

But the real shocker for me in 2015 was stumbling upon the Accusys Thunderbolt SAN RAID, the A16T2-Share. For more than half off the price of a similar fibre channel storage RAID here’s a magical box powered by unicorns that has four (4) Thunderbolt connections. Plug one Thunderbolt cable into that Mac Mini, format the raid, setup Apple’s Xsan, and then plug the other three (3) Thunderbolt cables into iMacs, Mac Pro, MacBook Pros or any Xsan clients. Wow. Awesome.

Suddenly we have a game changer. An affordable SAN storage RAID for real block-level storage. Now more than ever we can afford to have true collaborative workflows for video editors and anyone in the creative. If you need to work together with fast connections to a shared pool then building an Xsan got much more attractive.

Disclaimer: I got a chance to test the Accusys A16T2-Share. And I would be crazy to recommend something without testing it thoroughly. This was used for several weeks by video editors in production. It was much faster than our 4GB fibre channel storage, of course, but it was also faster than our 8GB FC storage. Speed tests showed we got close to 1GB/sec, and even when it was 97% full we got 700MB/sec. Sa-weet.

I look forward to seeing what Accusys bring to NAB in 2016. What new box will they show up with? I hope for more than 4 client ports and faster Thunderbolt 3. Only 82 more sleeps till we all find out.

Apple’s Xsan and Accusys Thunderbolt storage A16T2-Share were big stars of 2015, but what else stood out? The two other bright shiny lights were Archiware’s new P5 Archive app, and Vidispine’s VidiXplore cloud based MAM. More on those in posts to follow. Both of these products have transformed workflows for editors. Stay tuned!