Thunderbolt Storage in the field: Part 1 – The Gamma Carry test

I’m testing the new Accusys Gamma Carry thunderbolt storage in a series of blog posts with real life situations. Filming on location and editing with a remote team requires a combination of good workflow, great apps, and excellent storage. The Gamma Carry is small sturdy box meant for on set and on location editing and camera off loading. It also survives going back and forth between your office and your home office, and wherever you need to go. Disclaimer: I was not paid to write this review and this blog post is an ongoing field test with storage required by every day editors making films using local thunderbolt storage and the cloud. This is their story.

In the beginning….

The start of every creative editorial project is choosing the tools needed for the job, and working as team to make it happen. Making a film requires the hard work of everyone, not the least the IT / Tech who supports the crew and prepares the gear. On set the DIT (digital imaging technician) copies camera cards to multiple hard drives and backup devices before going back to the office to copy them to LTO (archive tape storage), but before the DIT can do their job the IT / Tech has to set up the RAID and design the best and safest backup and archive workflow.

Gamma Carry Overview

The Gamma Carry is an 8 drive external RAID box and it really small and sturdily built. There is a solid metal handle built into the top case and the drives carriers where you put the drives are metal and very solid. This entire unit is built to be solid and protect the drives it contains. The drive tray have a pin to lock the track in place when inserted as well as a thumb screw to lock them in. All drives must be attached with provided screws which is not as convenient as some others with quick plastic tray mounts, but this metal cage for your drive is solid and feels safe for a RAID that will probably be transported everywhere. Keep the data safe!

There are other little touches which are well thought out. The blue red green blinky lights are perfect for any Christmas holiday party and tell you important information about the state of your RAID at a glance but you with one button push turn off all the blinky lights. And keep working in the dark editorial suite not bothered by anything blinking. There’s a mute button and an extra port on the back with 60W to charge your laptop via the RAID. Very useful.

Gamma Carry The Setup

The set up of the Gamma Carry software wise is identical to the Accusys A12 T-Share which I set up recently and once you have the Accusys Mac installer then you have the RaidGuardX app. And the usual IT caveats apply, the software is not signed so you will have to right click on it to open it up. Hopefully they will sign this and notarize their software to make this easier for end users. Also there is a requirement for JRE to run RaidGuardX which means downloading and installed Java on your Mac. Also not optimal but only necessary for the RAID setup.

RaifGuardX detects the Gamma Carry

The set up of the Gamma Carry is the same as the Accusys A12 T-Share and a raid array you built with RaidGuardX will be recognized in the Gamma Carry. That could be a good thing, or not. I did have some excitement when it recognized the array I had built previously (I used the same drives form the Xsan setup). And this was the next step to resolve, because tp create a new array, or delete the array then create a new one I have to do one more thing. Since they were used with an Xsan I had a LUN label that identified them as such and had to remove this LUN label before proceeding. Occasionally we see this issue when re-using drives that had once made up a RAID which was part of an Xsan.

Delete the existing RAID array…. button grayed out.

Xsan to the resuce.

To see the RAID arrays avaialable when building an Xsan you can use the cvlabel command to list them. You can also use it to remove this label. WARNING: Do not do this when connect to an Xsan or Stornext storage network. Unless you know what you are doing. You are warned. This is dangerous. Removing a LUN label can bring down the entire SAN. That’s it. Now you know.

sudo cvlabel -l                                      

/dev/rdisk3 [ACCUSYS Gamma Carry      366] acfs-EFI "accusys"Sectors: 46881814495. Sector Size: 512.  Maximum sectors: 46881814495. Stripebreadth: 0.


sudo cvlabel -u "accusys"           

*WARNING* This program will remove the volume label from the
          device specified (accusys).

          After execution, the devices will not be usable by the
          Xsan. You will have to relabel the
          device to use it on the Xsan.


Do you want to proceed? (Y / N) -> y
Requesting disk rescan .% 
                                                                                     sudo cvlabel -l                          

/dev/rdisk3 [ACCUSYS Gamma Carry      366] unknown  Sectors: 46881814495. Sector Size: 512.  Stripebreadth: 0.

So all is good again now we can create new RAID arrays now that the Xsan LUN label was removed. Back to work! Once the drive is set up in Disk Utility as a new volume then you’re ready to go. In this case I added one more drive and created a 5 drive RAID5 set and formatted as HFS+. In my testing this was fast enough and would be faster if all filled with drives or SSDs. There are variations of this hardware with SSD ports instead of 1 or 2 drive bays to allow quicker ingest of SSDs which have camera footage on them.

Speed test of a 5 drive RAID5. Fast enough!

I then set up 48 hours of drive copies via Hedge for some testing of the RAID hardware. Thanks to Hedge Connect I get notified when the large copy finished. In this I was copying Thunderbolt 2 and 3 external hard drives (thunderbolt but hard drive and not that fast) to the Gamma Carry to be copied. The source drives are slow. Copying from SSDs would be way faster.

And of course Hedge found some minor warnings with these old drives. People don’t like LTO archival tape and want to have stacks of hard drives, but that data on those hard drives won’t last forever. Keep your important data in three places on two different kinds of media (tape, cloud, drives etc). This test of mine was to copy off old drives and use the Gamma to re-organize and re-sort for a new edit project. Now to back up to the cloud and set up Postlab projects with postlab drive for the proxies. In the mean time the editors can get the original footage when I carry over the Gamma Carry thunderbolt RAID.

This is part 1 and after some more testing I will publish some real world tests and experiences.

TCC troubleshooting

Download Howard Oakley’s Taccy app

Read Howard Oakley’s blog post on Catalina and privacy protection

Read Apple’s profile reference doc with respect to Privacy Preferences Policy Control payload

Read Rich Trouton’s guide to creating privacy pref policy profiles

This snippet (from MacAdmins slack) shows tcc in the logs if that is the issue:

log stream –debug –predicate ‘subsystem == “com.apple.TCC” AND eventMessage BEGINSWITH “AttributionChain”‘

Drop it! MunkiReport Db hacking

SQLite3 Db hacking for MunkiReport

Making modules for MunkiReport is easier than ever. Seriously.

please make:module

It’s easier than ever to make modules for MunkiReport (check out the recent MDOYVR MunkiReport workshop) and since the heavy lifting is done you can concentrate on the business logic (what makes sense) and the commands to execute or the scripts to run (python, shellI, etc). Worry about actionable data and less about the tables and views.

If you testing in production (which you should never do, always test is a testing environment) then you may happen to change a module (tables and fields etc) but keep the module name the same. This will confuse your database and you will need to erase it from the db to continue. In SQL speak this is “Drop table”. (You could also delete the munkireport Db and start again, but this is for those crazy enough to test in production and may want to keep the other data).

Sqlite commands for Munkireport

  1. Maintenance mode (current way)
sudo ./please down
Application is now in maintenance mode.

Old way –> sudo touch /Users/Shared/MunkiReport/munkireport-php/storage/framework/down

 2. Edit MunkiReport db

sudo /usr/bin/sqlite3 /path/to/MR/app/db/db.sqlite

 3. Exit maintenance mode (current way):

sudo ./please up
MunkiReport is now live.

Old way –> sudo rm /Users/Shared/MunkiReport/munkireport-php/storage/framework/down

Note: if you forget to get out of maintenance mode then clients can’t check in

“ERROR: Server error: MunkiReport is in maintenance mode, try again later.”

4. Migrate Db

sudo ./please migrate

Please use please migrate (or migrate db in web admin) if making changes to a module or else. That is, if you’re crazy to do this in production.

Server An error occurred while processing: \fancy_module_processor
Server Error: SQLSTATE[HY000]: General error: 1 no such table: fancy_module (SQL: select * from "fancy_module" where ("serial_number" = D09TP1QLH1K3) limit 1)

5. SQL hacking:

If you’re testing in prod and change a module’s fields but keep the same module name this will confuse your database and you will need to erase the entire db and start again or just erase this module from the db to continue. In SQL speak this is called “Drop table”.

A. List tables

For tables, the type field will always be ‘table’ and the name field will be the name of the table. So to get a list of all tables in the database, use the following SELECT command:

SELECT name FROM sqlite_master
WHERE type=’table’
ORDER BY name;

B. Drop table

Go through the list of tables and confirm the one you want to drop. Then do it. You’re backed up anyway, right? I mean, the data will come back when the clients check in again. So don’t worry.

Drop table to remove “fancy module” table from Db

DROP TABLE fancy_module;

3. Exit

.quit

REFERNCE:

MunkiReport WIKI – https://github.com/munkireport/munkireport-php/wiki

Jon Crain’s module making blog series – https://joncra.in/2018/11/30/creating-munkireport-modules.html

SQLite FAQ – https://www.sqlite.org/faq.html#q5

Notarize it!

Apple’s notarization service allows Apple to verify apps distributed outside of the App Store system. If you make your own apps to distribute to customers, clients, family or friends then you will have to notarize them by submitting them to Apple. This avoids painful dialog boxes in macOS 10.15 Catalina that prevent your app from launching by default.

NotarizeYourApps-Apple-Oct2019

Notarization The Hard Way

I’d been putting off notarizing my apps created for my clients for three reasons,

1)  it isn’t a strict necessity because most users are on macOS 10.14 Mojave,

2) I use Munki to distribute and install software which bypasses the requirements, and

3) I’m lazy

But it is only a matter of time before this would be a strict requirement and necessity. Also the relaxed requirements for notarization of apps was about to change again in February 2020 and I said this is the moment to do something. What now? Check with Rich Trouton and his blog Der Flounder.

codesign –force –options runtime –deep –sign “Developer ID Application: Name (#H7373736)” “/Applications/Cool-App.app/”

Rich Trouton is the modern major general of documentation and a super awesome dude. His blog Der Flounder has documented this process and now it was time to revisit this. Step by step recipes well explained with comments. What’s not to love?! Well, I didn’t get far because I missed some ingredients. Signing the app failed. I couldn’t notarize it without signing it. Hmm…

error: The specified item could not be found in the keychain.

It didn’t work the hard way, so let’s try it another way.

Notarization Made Easy

A very awesome app from Late Night Software called SD Notary can help make this process go smoothly. Their app detects if you have the right cert to run this process. Something which I thought I had, but did not.

Certificates, Identifiers & Profiles

It’s no accident I got tripped up in the same place with the cli and with the SD Notary app to notarize my app. I was missing the correct certificate. When I tried to codesign as a first step that’s when I got an error that I puzzled over for a minute.

The SD Notary app stopped me also at the first step because it said it couldn’t find a Developer Signing ID. And that with the command line error finally made me realize I’d missed something. And here I thought creating the app specific password was the hard part. (It wasn’t hard, but you have to look in the right place!).

A quick run back to the Apple developer site and a trip to the “certs identifier and profiles” section to create a new “Developer ID Application” cert (I had the installer one previously) solved that. I also had some trouble creating an app specific password, mostly because I was looking in the wrong place (in my dev account, not my apple ID account) but that got sorted.

Notarize-Apple-CreateNewCertificate

The “Developer ID Application” is what I needed.. Of course to get this I need to generate a cert signing request. There’s always a few steps. But once these are done then you’re good to go.

Once the proper Application type cert is in place, and the app specific password then you’re able to notarize via cli or an app like SD Notary. I tested this in Terminal:

xcrun altool --notarize-app --primary-bundle-id "com.apple.automator.Cool-App" --username "memyself@email.com" --password "really-cool-passw0rd" --file "/Applications/Cool-App.app.zip" 

No errors uploading '/Applications/Cool-App.app.zip'.
RequestUUID = 12345f-567e-476f-a229-6789cef906b

And in less than 3 minutes I received an email declaring it done. “Your Mac software was successfully notarized.

Then I went back to SD Notary and tried again. It was also successful and after selecting the app the entire process of signing, zipping, submitting to Apple, then stapling was done seamlessly.

SDNotary-Stapling

Hope that makes sense to someone. And the next time I notarize an app I will be able to do it seamlessly thanks to the help of everyone who has provided documentation and cool apps. Cheers.

References:

SD Notary app — Notarizing made easy

Rich Trouton’s Der Flounder blog

Apple dev docs

Howard Oakley’s Eclectic blog

FCP7 to FCPX

If you used classic Final Cut Pro 7 for years then eventually moved to FCPX now what do you do when you want to restore an old project? Read on…

FCP7 to FCPX

In the beginning we set up an older iMac that had been sitting around and already had macOS 10.12 and Final Cut Pro 7 (and even an early version of FCPX). We used this iMac to open up old FCP7 projects from our projects archive which were restored from LTO tape archive created by Archiware P5.

This process of restoring from tape archive back to the SAN then copying to an external drive to attach to this older iMac to convert worked but was cumbersome and not convenient. Opening old projects in FCP7 and then exporting out the XML was easy. Using SendToX to convert to FCPX XML was also easy. But getting the project to this old Mac off the main network was a drag.

Retroactive app

Use Retroactive app to install Final Cut Pro 7

Then one day I heard of this project that allowed to install iTunes on macOS 10.15 (Catalina) which only had the new Music app. Weird flex, but OK. Reading further it also allowed FCP7 to be installed on macOS 10.14 (Mojave)! Now this was a useful revelation. The app is called Retroactive and it would be very useful to us. Now FCP7 could be installed on the same Mac as FCPX. It would then have access to the network and the SAN where do all our editing and where we restore archive from LTO archive. Awesome.

The best part was that we moved from a dedicated old iMac running macOS 10.12 to a newer iMac Pro with macOS 10.14 on the Xsan and can run FCP7 thanks to that new app that makes it work. And then FCP7 to XML to sendtoX to FCPX is not too bad.

We also used Kyno to drill down into all the restored projects to identify en masse all the restored footage that it incompatible with FCPX. Renamed and then reconverted. All is well again for now. Archive restored, FCP7 projects converted to FCPX. Yeah, happy times.

Kyno batch rename dialog box

Kyno batch rename

Kyno FCPX incompatible files reanaming converting

Kyno convert and transcode

We had one minor snag in the process. Some of the restored projects didn’t use FCP7 they used early versions of FCPX with their events and projects folders separated (not the current library structure). Latest version of FCPX 10.4 did not know what to do with these projects that were also some times stored on sparse disk images (oh how the Xsan did not like these projects at the time). There was a menu for a while to convert these projects but it was now gone. What to do?

Back to the old iMac and we used FCPX 10.2.3 to convert these projects from 10.0 version to 10.2.3 library which can then be converted to the latest FCPX 10.4 format. Almost easier to convert FCP7 to FCPX in one shot but it worked and we were happily editing old projects in the latest version of FCPX.

fcpx 10.2.3 dialog box to update projects

fcpx 10.23. update projects and events dialog

Editing old projects in new FCPX

We have a way to restore old projects from LTO tape thanks to Archiware P5, a way to identify, rename and bulk convert old footage in an easy fashion thanks to Kyno and now also a way to convert FCP7 with SendToX and Retroactive to make it more seamless.

Hope this helps anyone else if want to do the same thing good luck.

Backup and Archive setups for media professionals

I recently did an online presentation on my Archiware P5 backups and archive workflows for media professionals.  

I use Archiware P5 to backup and archive media productions to tape. In this workflow I describe how original material, work in progress and conpleted projects are backed up daily and archived as needed.

This presentation goes into some detail of what two different storage setups look like and what similarities exist in my Xsan (fibre channel) and Jellyfish (10GbE) backup and archive workflows.

Despite the storage differences, the common workflow is that they all use the P5 Archive App to allow the Final Cut Pro X editors to archive completed projects. The simple way to do this is by right-clicking in the Finder and they will go to tape and get replaced by stub files (small files with the same name and negligible file size). When the project or any of the footage or assets needs to be restored it is another right click to restore. No IT people need to be involved. It’s magic. Almost.

Useful resources:

– Case Study: WorkSafe BC

Using P5 and the P5 Archive App

https://p5.archiware.com/solutions/worksafe-bc-hs-video-production

– The P5 Desktop Edition

Aimed at small teams and single users: P5 Archive and P5 Backup to a single LTO tape drive.

https://p5.archiware.com/desktop-edition

– P5 and Xsan

https://p5.archiware.com/solutions/xsan

– P5 and Lumaforge Jellyfish

https://p5.archiware.com/solutions/lumaforge

– Watchman Monitoring

https://www.watchmanmonitoring.com

– MunkiReport

https://github.com/munkireport/

– MacAdmins Slack:

https://macadmins.herokuapp.com

– MacDevOps:YVR Conference

https://mdoyvr.com