A blog about archives, storage and everything

Category: Tricks And Tips

  • From Camera to the Clouds: the very real story of Hedge and Postlab

    Update: This proxy workflow applies to Final Cut Pro 10.4.8 and earlier. For the new proxy workflow with FCPX 10.4.9 and Final Cut Pro 10.5 see my new blog post

    Note: I want to explain how our current workflow for editing remotely. I am always testing new tools and methods, so workflows change all the time. This is a snapshot in time of what we are trying now. So far it works.

    Hedge

    We use Hedge to copy camera cards to multiple drives on set (or after a shoot if on location) and then we use Hedge once more to copy one of these drives to the office shared storage (Apple’s Xsan).

    Why use Hedge? A nice simple app which hides its complexity well. Hedge has an easy interface to copy multiple sources (camera cards, usually) to multiple destinations (two external drives, or two SAN locations etc), and it does it well. It verifies, and double checks its work and leaves receipts. What was copied when. This is very nice and very useful for troubleshooting. It also has an API which made it easy to build an app that configures Hedge for its current task, and AppleScript support for extending automations after specified actions.

    Kyno and Postlab

    We are using two other tools in our remote ingest workflow currently: Kyno from Lesspain software for rewrapping and converting camera footage and Postlab, the remote collaboration tool for Final Cut Pro (and Premiere Pro). Testing with other tools is always ongoing and during a recent test of the workflow we also tried EditReady from Divergent Media.

    The Workflow (so far)

    While we are exploring various workflow automations we are currently doing the following steps manually.

    1. Hedge to copy camera cards two external drives on set, and then Hedge copy the drive to Xsan
    2. Making re-wrapped in MOV files from the original camera MXF files using Kyno and then
    3. Making H264 MOV 4K proxies in Kyno
    4. Uploading finished proxies to Postlab drive using Hedge
    5. Set up FCPX production and new FCPX library from template connected to proxies in Postlab drive

    Hedge and Postlab

    Hedge is super useful. Two times good. Hedge and Postlab are best friends. And the UI on both shows the simple aesthetic shared by the developers. Three panes. Source / Start to Destination / Projects. Whether you are copying Proxies to Postlab Drive or accessing your editing projects in Postlab the apps will guide you through.

    Copying the Proxies with Hedge to Postlab Drive.

    Details. Rewrap and Proxies

    Workflows will depend on your goals, and your available tools. In this case we are using a Canon camera and ingesting MXF files. In order to edit with small Proxies in FCPX but also be able relink to original (and larger) files easily we need to in our case re-wrap the original MXF to QuickTime MOV container.

    Right click on a clip in Kyno to rewrap to Mov.

    Originals. Not Proxies.

    And to be clear we are treating these in FCPX as “new” originals not as actual FCPX proxies. With the rewrapped MOV files we make transcoded H264 files which are swapped 1 for 1 with the original. When we need to export a final 4K version we can relink to the original 4K source and export easily.

    Proxies. Not originals

    The transocded H264 4K proxies we made in Kyno were 15x smaller than the original re-wrapped Mov files. We had almost 600GB in originals and 37GB for the 4K H264 proxies!!

    Postlab Pro Tips

    Working with Postlab pro tip #1 –> keep those FCPX libraries light. Keep all media and cache files out of the library. We knew that and we had Storage Locations set to outside of the library but one new issue came up when the libraries grew really big and we realized the editors were making multiple sequences, not backups, but versions. Now we are trying to work around this habit with Postlab itself. You can check in a version of the library and duplicate library for an alternate version. Modifications of old habits are always tough but technical reasons may force a change in habits here. We will see. Postlab pro tip #2 –> Keep your cache large and fast. By default the Postlab cache is your local drive and only 20GB. If you have a fast SSD or an external drive then move that cache and increase the size. It will help. Trust me.

    Kyno vs EditReady

    Another small issue we encountered in testing was that we could make the rewrapped Mov files in Kyno or in EditReady and both were fine. The only objection the editors had was that in Kyno we could keep the folder structure of the original camera cards and they felt that this lent some confidence to being able to track the files to the camera card folders if any media was missing or misplaced. The EditReady files kept the original names but they were all in one folder. As the tech I see no issue with FCPX handling these files since we’d be ingesting all the finished proxy files and all the files were named by the camera. Editors should be able to tell which reel the clips were from by the clip name and that’s all you need technically, but you can’t win every argument with an editor. As the tech you need to test alternative tools and methods and see what works technically but also see what can be accepted to work in the way the editors want to work. Changes to workflow are some of the hardest to make, making a system that is used, actually used, by the editors is the goal.

    Errata

    Errors. If you get them, how do you know? This was one area where I could comment on both Kyno and EditReady. I am spoiled by Hedge and it’s nice reports when it is done copying. And Postlab which has a Help menu :collect logs for support button, very nice. If your software tool is going to process a lot of files (rewrapping then transcoding) I want to know if there were errors. EditReady popped up a window to what had succeeded or failed and Divergent Media support told me to look in the logs for any issues encountered. Not great. While Kyno has a separate jobs window which shows jobs done or failed. But still no report. I would like a receipt or report or log at the end with files converted or failed to convert. It would help troubleshooting any issues when they arise. Tech support for both companies is great and responsive. Thanks again. And I’ll keep sending in feature requests.

    Testing. More Testing. And Teamwork.

    We are testing this workflow in production with a real project and getting feedback from the team. So far the proxies have proven to be easy to make, quick to upload to Postlab drive, simple to use in FCPX in Postlab. Assembling the cut and editing are going well. We will find out about the colour process when we get to that stage and relink to the originals. Stay tuned.

    Thanks!

    Thanks to Felipe Baez / cr8ivebeast for his assistance on this part of the workflow. We were having trouble relinking to the original MXF and he gave us the excellent tip to rewrap then in Kyno then make the smaller proxies. Works like a charm. Thank you Felipe! Here’s a link to a video Felipe made showing a similar procedure using Compressor to transcode and then relink in FCPX and it goes to show you that there are lot of ways to do things and to keep trying, and experimenting. You might learn a thing or two.

  • Minecraft Server for My Kids and My Sanity

    Summer time or anytime is a good time to run a minecraft server. And when I am not troubleshooting IT networks, planning SAN storage upgrades, running a DevOps for Dummies bookclub and the MDOYVR podcast then I like to upgrade my minecraft server.

    Every time there is an update to the java client there is demand from my users (uh, I mean, my kids) to immediately stop all other work (hey kids, I’m working here! let Dad work) and upgrade the minecraft server.

    Like all other IT domains where there are variety of solutions and software fixes to problems, it would seem that Minecraft has official server downloads as well as the unofficial artisanal craft versions. I’ve tried a few, and some out of desperation… there was an incident with netherite blocks and the server wouldn’t start anymore but the Ppaer minecraft server fixed the issues!

    The normal routine is that when an official release comes out the other versions may not be up to date as quick, so it’s back to the official versions.

    Download the official Minecraft Server

    Or try the Paper Minecraft Server

    See also Michael Lynn’s two part family harmony blog series which started me on this road to keep the kids happy and maintain family happiness.

  • Automate those apps. Get some robot love 🤖 ❤️!

    If only one person needs an application then I think about using Munki to deploy that app. If more than one person should have it then Munki is definitely the way to automate app deployment. And really, if you’re going to take the time to download an app from a website, mount a disk image or un-pack a ZIP archive, run an installer, type an admin password, close that installer … then for the love of all that is good just put the app into your Munki repo and be done with it. Automate it.

    Using Munki to solve problems makes sense. Automation helps everyone in this case. But if you’re putting in one off applications into your Munki repo more often than you need to, you need to get those apps into Autopkg. Using Autopkg recipes to download the latest apps and put them into your Munki repo automatically is an automation love fest, but if your apps don’t have recipes what are you going to do? Manually add your apps to Munki? No way. We need a robot 🤖❤️. Recipe robot, that is.

    Using Recipe Robot we can build Autopkg recipes for most apps then add the recipes to the Autopkg community to enjoy. Everyone wins.

    I recently created recipes for two important apps in my media workflow: Kyno and Hedge. I’ll show an example of this workflow using Recipe Robot and Munki Admin to demonstrate the workflow.

    Step 1. Feed the robot.

    Drag and and drop the app you want to create your Autopkg recipes.

    RecipeRobot-FeedMe

    Step 2. Watch the robot do it’s work

    RecipeRobot-start

    Step 3. Robot is done. Recipes made.

    RecipeRobot-Done

    Various type of recipes can be made. I chose download and munki because those are what I am using to automate adding apps to my Munki repo. But there are other options: jss, Filewave, or “install” for example.

    reciperobot-options.jpg

    Step 4. Run those Recipes

    You can use your recipes locally with Autopkg. Run them in Terminal or use Autopkgr , a very nice GUI app for automating the collection and scheduling of recipes. Note: Autopkg and Munki can all be run via cli (command line interface) but for this demo we are showing the GUI apps that are there provided by outstanding members of the community. Many Thanks to them and the contributors to their projects.

    Autopkgr-notification

    Autopkgr app can send notifications in macOS, emails, or post to your Slack group.

    Step 5. See the recipes, Use them wisely

    MunkiAdmin-Recently ChangedPKGS

    Here is an example of newly imported Kyno and Hedge apps in our Munki repo (via Munki Admin GUI).

    MunkiAdmin-Description

    Add a display name, choose which catalogs the apps will reside in, and check that the description will help explain what the app is.

    References:

    Elliot Jordan – Autopkg talk at MacDevOps:YVR

    https://youtu.be/Q_cvgGtJ71M

    Elliot Jordan – Recipe Robot talk at MacDevOps:YVR

    https://youtu.be/DgjO1mfMHtI

     

  • Compressor Tips and Tricks

     

    Issue: Stuck job in Apple’s Compressor app.

    Resolution: Remove the historical jobs in your local home folder.

    ~/Library/Application Support/Compressor/History/V4

    Compressor-History2

    Note: to get to your home folder hold down the OPTION key and select the Go menu in the Finder.

    Compressor is the best sidekick to Apple’s Final Cut Pro X and it gets used a lot. But occasionally something goes awry. It’s software running on a computer. So we troubleshoot. What looked like a stuck running job was mostly leftover evidence of an old job. The Apple support document I found didn’t mention this tip but instead talked about zipping up your settings folder which has all your custom compressor settings for things like YouTube outputs or anything custom. Didn’t seem useful to me to remove but this historical stuff, don’t need it and POOF this solved the issues. It’s not always this easy but something you just take the win and go with it.

    Reference:

    Resolve an issue in Compressor: Learn how to isolate, troubleshoot, and fix issues in Compressor.

    https://support.apple.com/en-ca/HT203476

  • The case of the strange disappearing drive space

    Recently I was asked to look at a 4TB drive that was only showing less than 2TB available…. No problem, I said, this is easy to fix. Famous last words.

    Just open up Disk Utility and resize the partition, or reformat the disk, right? Easy Peasey. Well, it took some troubleshooting to time to figure out and a trip to Terminal was required to solve this weird case, plus I learned a new command along the way. Fun.

    The Problem:

    Buying a 4TB hard drive then putting it into your external drive case for backups should be simple,  but what if instead you got a nasty surprise and it showed up as less than 2TB?

    Troubleshooting the issue:

    4TB drives were presented to me and when I loaded them into an external SATA dock then showed as 4TB drives with a partitioned volume of less than 2TB.

    I tried to delete the phantom partition, and I tried resize the volume to use the empty space in Disk Utility.app but it refused to budge. This needed a trip to Terminal.

    man diskutil

    Using “man” or “info” commands you can find out more about almost any particular command. Maybe some useful options or arguments would be listed or at least some examples would help.

    NAME

     diskutil -- modify, verify and repair local disks

SYNOPSIS

     diskutil [quiet] verb [options]

DESCRIPTION

     diskutil manipulates the structure of local disks. 

     

    To find out more about what we’re faced with let’s ask diskutil what it sees:

    diskutil list
    /dev/disk2 (external, physical):

   #:                       TYPE NAME           SIZE       IDENTIFIER

   0:      GUID_partition_scheme               *4.0 TB     disk2

   1:                    EFI                   209.7 MB   disk2s1

   2:        Apple_HFS Backup                  1.8 TB     disk2s2

    Looking through the man page the “resizeVolume” command caught my eye. Also the “limits” option seemed interesting. How

    diskutil resizeVolume disk2s2 limits

Resize limits for partition disk2s2 Backup:

  Current partition size on map:         1.8 TB (1801419800576 Bytes)

  Minimum (constrained by file usage):   846.4 MB (846426112 Bytes)

  Recommended minimum (if used for macOS):26.8 GB (26843545600 Bytes)

  Maximum (constrained by map space):   4.0 TB (4000442028032 Bytes)

    The Answer:

    Reading through the man page revealed that the best way, and new to me, was to resize the partition to use all available space with “R”. Of course, so intuitive.

    sudo diskutil resizeVolume disk2s2 R

    I did get some errors. But repairing the disk fixed those issues. And I was able to resize the disk in Terminal with diskutil where Disk Utility.app had failed.

    sudo diskutil resizeVolume disk2s2 R

Resizing to full size (fit to fill)

Started partitioning on disk2s2 Backup

Verifying the disk

Verifying file system

Volume was successfully unmounted

Performing fsck_hfs -fn -x /dev/rdisk2s2

Checking Journaled HFS Plus volume

Checking extents overflow file

Checking catalog file

Checking multi-linked files

Checking catalog hierarchy

Checking extended attributes file

Checking volume bitmap

Checking volume information

The volume Backup appears to be OK

File system check exit code is 0

Restoring the original state found as mounted

Resizing

Modifying partition map

Growing file system

Finished partitioning on disk2s2 Backup

/dev/disk2 (external, physical):

   #:                       TYPE NAME          SIZE       IDENTIFIER

   0:      GUID_partition_scheme              *4.0 TB     disk2

   1:                        EFI             209.7 MB   disk2s1

   2:                  Apple_HFS Backup      4.0 TB     disk2s2

    And lastly, the issue may have been caused by the old drive dock which refused to see the 4TB volumes even when correctly resized. A newer drive dock was required.

  • Use Munki to install a screensaver

    Use munki-pkg to package up stuff and make your life easier when managing Macs using munki. Here is an example of installing a screensaver.

    Why use munki-pkg? How else do you install stuff using munki, run scripts, and version your testing buildings all in one easy to use application? This is all possible with munki-pkg.

    Munki-pkg makes package (PKG) installers, Munki likes pkg installers. Munki will also install apps, run scripts, install profiles, and do many things but packages are useful because we can put files in specific places, such as the main computer level screensaver folder, then run a script to set it as a default.

    Download munki-pkg and create a working project folder.

    Step 1.

    Create the folders you need and place your files (payloads) in the right places.

    munkipkg-payload.png

    Step 2.

    Create your post install script if you need one. Example: setting the screensaver you just installed as the default.

    #!/bin/sh

defaults -currentHost write com.apple.screensaver moduleDict -dict moduleName Brooklyn path /System/Library/Screen\ Savers/Brooklyn.saver/ type 0

     

    munkipkg-postinstall.png

    Step 3. Build your package

    Run munki-pkg on the command line and build your package. If you make changes then version up in your build-info.plist and build again.

    munkipkg-build.png

     

     

  • No NetBoot, No problem: installr and bootstrappr

    It’s 2019, and NetBoot is almost dead. All new Macs have T2 chips. Sent from the future to protect us from …. ourselves? No more NetBoot, no problem!!

    When NetBoot first appeared and I was able to boot entire labs of Macs across the network I was amazed and overjoyed. It was awesome. Spinning globe, spinning…

    Netboot-GlobeSpin.jpg

    But in the years since I’ve moved on to no-imaging. Using Munki to manage software means no more imaging, just install Munki and a small config change to point to the Munki server, thereafter the software that should be there goes on, and what’s not supposed to be there goes away. Simple. Just install one package, well, maybe two, then you’re good.

    Well, what if you want to streamline or automate these things? What if these are new Macs which don’t have users configured? What if we could do all this from recovery mode? Hmm… Enter bootstrappr and installr!!

    bootstrappr

    This awesome project allows to add packages to install in one step while booted in recovery mode. Plug in a USB stick with the bootstrapr script to run the package install magic or mount a disk image over http. Create a DMG with the included script make_dmg.sh. And now this is the best part: in recovery mode open the Terminal app from Utilities and type:

    hdiutil mount http://server/yourDMG.dmg

    Then:

    /Volumes/bootstrap/run

    When it’s done you can Reboot the Mac and you’ll have a set up customized to your liking with Munki installed and configured with custom settings.

    installr

    The installr script works in the same way but adds the macOS installer to the party. You can also mount the DMG over http and re-image a Mac and then add your custom packages. It’s awesome. Truly amazing.

    One note: Added packages in Installr must be in a special format. From the installr site: startosinstall requires that all additional packages be Distribution-style packages (typically built with productbuild) and not component-style packages (typically built with pkgbuild)

    productbuild --package component.pkg --version x.y --identifier com.example.component distribution.pkg

    In one of my first tests with installr and pycreateuserpkg I was caught up by this, even though it is properly mentioned in the read me. Packages that work in Bootstrappr or munki directly don’t necessarily work when called by the macOS installer (startoinstall). Armin Briegel was helpful in the MacAdmins Slack and reminded me of this. Thanks Armin and thanks everyone on the MacAdmins Slack.

    Many Thanks to Greg Neagle for creating these tools and Munki. Looking forward to hearing him speak at the next MacDevOps:YVR conference June 12-14, 2019. Greg will be speaking about his efforts to port some parts of Munki from Python to Swift. More info on the conference and speakers here: https://mdoyvr.com/speakers/

    Also a shout out to Graham Gilbert who has worked on Imagr (MDOYVR talk), over the years, an imaging and automation tool which was also an inspiration (along with bootstrappr and installr) to Tim Perfit and his MDS project.

    Update: corrected the names of installr and bootstrappr in the title because… autocorrect.

     

  • Reset Printer Queue

    TIL (thing I learned)

    Had a user upgrade to macOS 10.14.1 and no printers showed up anymore.

    So using my Google fu I found some posts (see one below) which described a novel way to reset the print queue on macOS. An old trick apparently. Learn something new everyday.

    A quick trip to a terminal and it worked! The existing printers returned to System Preferences and printing resumed.

    $ cancel -a

    Reference

  • Blocking minor major macOS upgrades

    Continuing our theme of welcoming our new macOS overlords, uh, I mean, blocking major macOS upgrades such as macOS 10.14 Mojave with AppBlock we shall examine some other methods of stopping the freight train known as Apple upgrades.

    1) A smart person on the MacAdmins Slack posted a useful command to tell macOS not to download major upgrades.

    In their testing, running:

    `software update –ignore macOSInstallerNotification_GM`

    blocks the installation of the Mojave notification package (at /Library/Bundles/OSXNotification.bundle).

    However if it already installed, then it’s too late. They pushed out this command prior to that package being distributed by Apple, and they could subsequently see in install.log that the update is being found by softwareupdated but not being installed.

    2) If you missed the chance to tell the Mac not to download major macOS upgrades then Rick Heil on his blog has detailed a way using munki to delete the bundle that triggers the macOS upgrade installer.

    3) App Block

    If your users are intent or their computers are all hell bent on downloading the install app then block it with App block detailed in my previously mentioned blog post

    4) Warning

    In an effort to get an early warning when users are about to upgrade I use Watchman Monitoring to send me an alert email when a Mac starts downloading the Install macOS app. Sometimes it’s enough of a warning to send an email to a user to ask them whether it is a good idea to upgrade at this time. If storage or software needed for production or backups aren’t qualified or tested thoroughly beforehand then upgrading in the early waves can be less than ideal and frought with peril.

    In other interesting and related news, Victor (MicroMDM) was spelunking into the MDM Protocol for what prompts Macs like iOS devices to download major updates. Great post here

    If you have any better ways to block macOS upgrades or want to contribute some great solutions let me know. Cheers

     

     

     

     

  • To install macOS Mojave, or not to?

    InstallMojave

    Just the other day macOS Mojave was released and now the armies of Macs armed only with the AppStore are silently downloading the installer and ready to upgrade. You can’t hurry too fast to be on the bleeding edge, hurry faster!

    Just in case you don’t want everyone to install macOS 10.14.0 (dot zero!) in the first week of its release here’s a way to slow down the upgrade hordes using Erik Berglund’s AppBlocker script. Erik Berglund is also the author of ProfileCreator (for creating profiles) and the author of many other great scripts.

    Note: for true binary whitelisting check out Google’s Santa project and Upvote (and Moroz and Zentral, two other Santa sync servers).

    Step 1. Get it

    Clone or download the AppBlocker project from GitHub

    AppleBlockerProject.png

    Step 2. Do it

    Edit the AppBlocker.py script with the Bundle Identifier of your app to block, in this case for the Mojave installer from the AppStore it is:

    com.apple.InstallAssistant.Mojave

    You can also edit the alert message, and the icon that is shown, as well as decide if the blocked app should be deleted or not. The script is easy to edit in BBEdit, or nano (in Terminal). Use whatever your favorite text editor is to make the necessary changes.

    # List of all blocked bundle identifiers. Can use regexes.
blockedBundleIdentifiers = ['com.apple.InstallAssistant.Mojave']

# Whether the blocked application should be deleted if launched
deleteBlockedApplication = False

# Whether the user should be alerted that the launched applicaion was blocked
alertUser = True

# Message displayed to the user when application is blocked
alertMessage = "The application \"{appname}\" has been blocked by IT"
alertInformativeText = "Contact your administrator for more information"

# Use a custom Icon for the alert. If none is defined here, the Python rocketship will be shown.
alertIconPath = "/System/Library/CoreServices/CoreTypes.bundle/Contents/Resources/Actions.icns"

    UPDATED NOTE:

    To determine the Bundle identifier of other applications you can use osascript

    osascript -e 'id of app "iTunes"'
    com.apple.iTunes

    If you want to block more than one app use a comma separated list in the AppBlocker.py script:

    ['com.apple.InstallAssistant.Mojave','com.apple.iTunes']

     

    Step 3. Run it

    Put the script where you want to run it. The default location as defined in the launchd plist included with the app is “/usr/local/bin”. Put the launchd.plist in “/Library/LaunchDaemons/” and start up your launchd to block your apps!

    launchctl load /Library/LaunchDaemons/com.github.erikberglund.AppBlocker.plist

    Step 4. Automate it

    For bonus points we automate! Bundle it all up in a package with munkipkg, then distribute it with Munki to all your clients.

    Using munkipkg is easy. Create the folder using munkipkg

    ./munkipkg --create AppBlocker

munkipkg: Created new package project at AppBlocker

    Then you fill the payload folders with those items you downloaded from the AppBlocker project. LauchD plist in the LaunchDaemons folder and AppBlocker.py in the “usr local bin” (create each nested folder).

    AppBlocker-Munkipkg3.png

    And finally create a post install script (no “.sh”) with the launchctl action to start your plist.

    AppBlocker-Munkipkg4.png

    Last but not least add this package to your Munki repo as an unattended managed install  that everyone gets. Of course, only do this after testing your package locally somewhere to verify that it works properly. Remember the saying: “You may not test very often, but when you do it’s always in production.” Be very careful with your testing but always automate all the things.

    Updated after the initial blog post to explain how to add more than one app to block, and how to use osascript to determine the bundle identifier.