About macvfx

DAM SAN Backups

Raspberry Pi for Christmas

I finally got a chance to open up my Christmas present to myself a few months late. I was excited all the same because it was a Raspberry Pi 400.

This is an incredible form factor for the raspberry Pi. It’s an all in one unit that fits into a keyboard. The keyboard is the computer. Just incredible.

Once you plug in power, the mouse and an HDMI monitor you have a working Linux computer running Raspbian. What to do next? Install Tailscale of course !!

Tailscale is a mesh VPN and allows you to link up all your devices in a private network no matter where they are. I’ve blogged about it here. So far I’ve linked up macOS, iOS, windows, Linux centos as well as Synology and QNAP NAS so now let’s add a raspberry Pi.

First things first let’s update the raspberry Pi because it’s been sitting in its box for a while. I didn’t have much luck with the add / remove software gui app (maybe because it was still getting on wifi) but after getting on the network I fired up apt to update all the things.

sudo apt list --upgradeable

Using apt you can get a list of what is upgradeable. This was my first step the gui app didn’t list any software that had updates. A few months in a box and there should be a lot of updates. This is Linux we are talking about.

 sudo apt update

Then it is just a matter of upgrading everything.

The next step is to install Tailscale. Read the instructions for Raspberry Pi on the Tailscale website. Before installing Tailscale we have to install other needed components and we have to tell Linux where to find the software. Similar to my adventures with CentOS Linux and yum you have to tell Raspbian Linux what to do with apt.

First we fetch the signing key and tell it where the repo is. Note: Always be mindful when using curl to download and install items or scripts.

   curl -fsSL https://pkgs.tailscale.com/stable/raspbian/buster.gpg | sudo apt-key add -

curl -fsSL https://pkgs.tailscale.com/stable/raspbian/buster.list | sudo tee /etc/apt/sources.list.d/tailscale.list

Just a few more steps now. Install the needed https components. Check with apt for updates then we can finally install Tailscale.

sudo apt-get install apt-transport-https
sudo apt-get update
sudo apt-get install tailscale

The next step is to authorize Tailscale which is usually done with a login to our account we created the Tailscale network with. But this time I wanted to try a pregenerated keys from the Tailscale admin panel.

sudo tailscale up --authkey tskey-gh37374737292a4847382

Now to test the new secure Tailscale mesh VPN set up I wanted to login to my MunkiReport web server.

Using the Tailscale cli to list all devices on my private network I found the IP.

tailscale status

Find the ip of the server or device from the status list and connect.

Next I outputted the raspberry Pi cli history to a file and sent it to my iPhone where I edited this blog post with the WordPress app.

history > raspberry-history.txt

tailscale file cp raspberry-history.txt iPhone:

I am looking forward to working on many fun projects with my raspberry Pi. After Tailscale is on there then it is easy to connect from my iPhone or from anywhere Tailscale is installed.

In the example below I used ssh shellfish app on my iPhone to connect.

Tailscale has great documentation and despite this I had a small misstep when I went to look at the install instructions. It defaulted to Ubuntu or misdetected my raspbian linux so I wondered for a second why it wasn’t working. I soon realized I’d copy pasted the wrong thing. Checked the drop down install menu for the raspberry Pi instructions and voila happy times.

A Mesh VPN For Everyone

How to use Tailscale (wireguard based) mesh VPN to connect everything

What is Tailscale? It’s a mesh VPN based on the wireguard open source project. It’s a secure network to connect your own devices no matter where they are.

Tailscale is free to use with one account and up to 100 devices, which is enough to see how well this can work to connect up servers, storage and desktops. They have paid plans for teams and enterprise.

Tailscale macOS app icon

macOS and iOS

To start, download Tailscale on your Mac or iPhone then find your IP address. Once you are signed in and have your IP address you can connect easily between devices. For example, on your iPhone open the Tailscale app and see your installed devices. Click on your Mac and the IP will be copied into the clip board. Use this to connect with app such as Secure shellfish for SSH or VNC viewer for remote login. When you’re in the same network it’s impressive, but when you’re on a different network, separated far away, It’s magic.

The real test for me was to install Tailscale on some backup servers I manage to make it more secure and more convenient to access them. I had used a variety of remote control for business services and well, Tailscale is easier, quicker and much more awesome. All the other software I tried was much less awesome.

After using Tailscale mainly for remote control, I tested Tailscale to securely connect my remote Macs to my own MunkiReport server. I use Munki and MunkiReport to manage Macs and having Tailscale allows me to securely connect endpoints to the server without opening up ports on my router. MunkiReport allows me to detect malware (with DetectX plugin) or check on backup jobs with Archiware P5 backup software (using a module I wrote) or a multitude of other diagnostics such as disk space free, apps installed, and all kinds of great hardware and software metrics. So much reporting. And MunkiReport doesn’t need Munki specifically, so if Tailscale is installed for remote control why not report on everything else.

DSM Package Center: Tailscale (and Archiware P5) app on Synology NAS

Synology DSM

Having Tailscale installed in all the Synology NAS I manage in various physical locations allows me to securely connect to all of these NAS from anywhere. With remote work using a NAS is a great way to sync data between locations. Synology has a lot of great built-in tools to make this happen and a very robust quick connect feature combined with ddns, and let’s encrypt certificates to support it. After setting up a few to sync one location to another I was constantly getting notifications of IPs being blocked on my firewall. I had to open a port on my firewall to let in the ssh / rsync traffic through and despite a strong set of firewall rules with a geo block there were still connection failures and password attempts. Using Tailscale I can now have a P5 server set up on one Synology NAS connecting to the Tailscale IP of other remote units and it can easily backup, sync or archive the data from the various locations.

To install the Synology Tailscale package check out this GitHub page. Download the app then side load it (manual install in package center). To enable it you will have to have ssh on, a user with permission to use it, and one command to type.

sudo tailscale up

In one case I didn’t have SSH enabled on the remote unit so I remoted into a Mac on the same network, enabled an admin user, turned on ssh with a time limit on the account, and then logged in. Once the above command is run you will get a link to a website to authenticate the device with your account.

Linux (CentOS)

I have also installed and tested Tailscale on a Linux (CentOS) storage server. In my case a Jellyfish which has a ZFS volume shared over direct 10GbE for Final Cut Pro video editors using nfs or smb. the Jellyfish works well on premise, but wouldn’t it be nice to capture camera cards to the remote storage server via Tailscale? Oh yes it would. And what about playing back some of the video files via VLC on your iPhone! Or Files.app! Yes, to all the above. All made possible with Tailscale. And a huge shout out for their great documentation. Installing Tailscale on CentOS was super simple. Add a yum repo, install, tailscale, and then bring the service up. Couldn’t be easier.

sudo yum-config-manager --add-repo https://pkgs.tailscale.com/stable/centos/7/tailscale.repo

sudo yum install tailscale

Shared Devices

A small, but very exciting, feature was added part way through my testing of Tailscale which made it infinitely more awesome, shared devices. The concept is you are authenticated to your devices and can see in the Tailscale app all the IPs to connect to, but what if you could share one device (computer, server, NAS) with another person? Well, now you can. In the Tailscale admin console choose a device and send a share link to someone, they then will see this devices in their device list as shared. Home users can set up Tailscale to access all their own devices, but now can also choose to share access with a device in particular. For example, if you create an account, open a service (file sharing) and send a share link then the other person will login with the account you create and access the one thing you want them to. Maybe it’s a smb share to drop files. Works great for video collaboration and other kinds of teams.

There’s a whole lot more you can do with Tailscale (and wireguard) mesh VPN but I hope this gives you all some ideas to start with.

2021: Thunderbolt Shared Storage Report

It’s 2021 and what is the state of Thunderbolt Shared Storage?

Thunderbolt Shared Storage is a RAID which you connect to with Thunderbolt (and Ethernet) which can be shared with other workstations. It’s a Thunderbolt SAN. Shared by Thunderbolt.

I’ve long been a fan of Apple’s Xsan and other SAN products that use Fibre Channel (or iSCSI) to connect clients to super fast block storage. It acts like a fast direct attached RAID but you can share it with others. The sharing part is crucial to collaborative workflows. We used XSAN when I was in VFX and I now use XSAN for post production workflows. Editors like to edit, have large video files, and often work in teams. All those camera files aren’t getting any smaller. So you need a lot of storage that can be shared by a team of editors, colourists, motion graphic artists, producers, etc

What you don’t often want is a complicated network infrastructure or a server room with fibre channel switches and fibre channel RAIDs and assorted other equipment. You don’t want that. You’d have to call me and pay me to set up your storage, backup and archive workflow. While it is always recommended to work with a trusted contractor it can be expensive. For small teams a shared Thunderbolt storage SAN can be quick to set up, doesn’t take much room and can easily connect 4-8 editors. Thunderbolt 3 passive copper cables extend up to 2m but optical Thunderbolt cables are available up to 60m.

I want to review one such example of shared Thunderbolt storage that stands out, the Symply Workspace. It’s a RAID, but it’s a SAN too. It is storage you plug into with Thunderbolt, but with an extra ethernet cable you make it a SAN. Inside the storage it has Quantum StorNext which runs most of the world’s expensive enterprise SANs, but it’s in the Symply Workspace and it works with Apple’s Xsan client software (included free with macOS since macOS Lion 10.7 !!). So basically, it is enterprise storage in a Thunderbolt box. One more thing, add a 5 seat license to a simple but powerful media asset management (MAM) tool axle.ai to organize your assets, add keywords and access from anywhere. Almost too good to be true, so I had to test it and see.

I received a Symply Workspace to test with. I like testing storage. My clients always need more storage. I keep telling them to not fill it up, but they do. That’s why we have LTO, nearline and cloud archive with Archiware P5 but that’s another story. For now, let’s test this storage: how easy is it to setup, how awesome is it to edit with, and what is a MAM good for anyway? Let’s find out.

It starts with a box. You open the box. You take the drives out of the box. Two trays of drives. Then there’s the RAID itself. Put the drives in the RAID. No screwdriver or tools necessary, just slide and click. Next step read all documentation online, ha ha, just kidding, don’t read anything keep going. Ok, kidding a bit. There’s an info card in the box with a website link to help explain the setup. But I didn’t read it all, uh, I know Xsan, I can do this, ok, what’s the link? now back to building.

The tricky part is plugging in a few cables, which will be improved in the shipping production version with stenciled labels of where to plug in what. There’s three cables after the power cable. One Thunderbolt to a Mac, that’s easy. One ethernet to your local network, no problem. And lastly one more ethernet to your new SAN production network (aka metadata). Ok, what’s that? Add a 5-port switch and plug in all clients and the storage to this network for SAN private metadata. It really is a SAN. Like Apple’s Xsan or Quantum StorNext, you need a data pipe (40GbE Thunderbolt 3 in this case) to transfer the data, and a metadata network, to talk about the data (1GbE ethernet to our extra switch). The public network will be used to talk to Axle MAM or for re-sharing out the volume to non-Thunderbolt connected Macs.

Once the Symply Workspace is wired up and powered on you’ll be able to reach the unit via a local bonjour name in your web browser (http://symplyworkspace.local:8088) and from there a simple web interface allows you to monitor the status of the unit, download drivers to configure a Mac or Windows client, restart or shutdown, contact support or start troubleshooting if needed. It’s a great tool to do the few things you need to.

So with everyone working remotely how do we connect to our shared storage? I’m glad you asked. There are a lot of good options. My favourite is Tailscale a mesh VPN (based on the open source Wireguard project) which you would install on a locally connected Mac then you can remote in and share a screen or the storage. There’s also the Axle 5-seat license included with the Symply Workspace which can scan your storage, make proxies and serve it all via a website which can also be accessed from anywhere.

To be continued…

Update: Added a clarification that there is an included 5 seat license to Axle media asset management in the Symply Workspace

Do you know where your files are?

Trying to solve the problem of finding production media files across many storage platforms.

Map of storage locations created with Scaple https://www.literatureandlatte.com/scapple/overview

In media production environments you work with high speed and high capacity storage. It can be network based NAS, fibre channel SAN or Thunderbolt DAS. There’s always some backup RAIDs, individual source footage drives, file servers and even network appliances.

Glossary of Storage:

  • SAN – Storage Area Network (typically with Fibre Channel, also with Thunderbolt or iSCSI)
  • NAS – Network Attached Storage (popular vendors include Synology and QNAP)
  • DAS – Direct Attached Storage (hardware or software raid directly attached to a client or server)
  • LTO – Linear Tape Open (tape standard for backups/archives. Current gen8 holds 12TB per tape)
  • Cloud – Other people’s servers and storage. Hosted in a data center (AWS, GCP, Azure, & more)

The variety of file types is astounding: original camera footage, Final Cut Pro projects, stock footage purchased, b-roll shot from other projects, sound effects, music etc. How do you find anything? Files can be in many places, across many different kinds of storage. The question is how do you know where they are?

In the before times…

Before the great pandemic of 2020 I only heard one complaint with finding files on Apple Xsan storage: “Why does spotlight not work on my Mac?” Searching the Xsan volumes was hit and miss. When Spotlight worked it was fast and immediate. And when it didn’t, well…. Not so much. To help finding files we started using one called EasyFind from DevonTechnologies. It was free, and easy to use, but it was not fast enough for ad hoc video searches. Nor could it search across all storage areas at once. A new year, and new solutions required. 

EasyFind. Freeware for finding files. Easy to use but not fast.

One major issue with EasyFind was not being able to refine the search easily for audio and video assets. It seems skewed toward developers and while it allows you to include or exclude some file types globally in settings but it does not allow you to refine the results while searching. This makes it difficult to find what you need when you need it.

Searching for sound effects in EasyFind reveals “.h” source files. Only one file is relevant in these results.

It’s 2021 and a few things have changed over the last year: you may have a lot more places to search (SAN and NAS) and more importantly everyone is working remotely. It has become a lot more challenging to see your files, let alone search through all the storage locations and find what you need. FoxTrot Search to the rescue.

FoxTrot Search Pro. Not freeware but worth every penny. Indexes all locations so that search is fast!!

Editors and other creatives need something that will search through all their different storage places and quickly tell them where a certain file might be. It might be stock footage, a drone shot, sound effect or an old logo. But where exactly is it? What projects was it used in? What was it called?

For those who like organizing their files into logical folders, then the filesystem is your friend. Using The Finder has been the way to find things. But now there’s a lot of folders and a lot of file types in those folders. Choosing what to index in these folders help enormously. You can exclude certain subfolders on a designated storage or kinds of files that are not needed. Do not index mail or chat messages or even source code files if you know the files you want are movies and audio files. Narrow the scope where you can. Find files fast.

Choose what to index in FoxTrot Search and what not to by kind (file type) or ignore by subfolder.

You might think I’d suggest a full enterprise Media Asset Management (MAM) system at this point. And well, in the old days I would have. Apple’s old Final Cut Server worked perfectly well with classic Final Cut Pro (v6 and v7) but required rigid workflows and ingest habits. Same issue with a lot of more expensive bespoke enterprise systems. They catalog assets but at a great cost. Some clients avoid these for the cost, and others for the workflow restrictions. I’ll talk about some newer options later but for now editors just want to find the files in their own folder structure on their production storage. Is that too much to ask?

Using FoxTrot Search Pro and editor can easily search across multiple indicies (each index is a unique storage or separate folder location) than an admin has already set up and created. No waiting for indexing in real time, search now, with results instantly show in app. Narrow down and refine your search results easily. Don’t want to see Mail messages, or images, only audio files? Easy. Then further define only the file type you want. It works.

FoxTrot Search can refine searches by kind and extension to find the relevant files you actually want.

One of the recommendations to my clients with large storage (SAN or NAS) is to have secondary nearline archive as well as backups. Either a Thunderbolt RAID array like the Accusys Gamma Carry or a desktop QNAP or rack mounted Synology NAS. Using Synology is great for many reasons, for example, it includes a nice web login, and it can search for files. But to be completely honest, the built-in search only works sometimes. A common issue with search on the Synology is the corruption of the search index. It seems to happen some times for some storage locations, but all the time somewhere. Rebuilding this does not seem to fix it for long if at all. This was the major reason for switching to FoxTrot Search. I want something that works every time. Across all storage volumes: SAN, and NAS.

Synology NAS corrupt search index.

To be fair, I had to rebuild an index or two in FoxTrot Search as well, but it worked. Every. Single. Time. Rebuilding the Synology search indexes never seemed to fix the issues. With FoxTrot Search it’d warn me there was a lot (A LOT!) of items to index and it will take time but that’s why I’d do it over an evening and rebuild automatically them after hours or every weekend. FoxTrot Search would also warn me of problem files that took too long to index and could then be added to exclusion lists automatically.

When Spotlight stops working, your Synology search index is corrupted again or just want a great search tool for active storage locations, I recommend FoxTrot Search. It searches across all storage and provides access to the files where possible. And it’s fast. Really fast. Did I mention that yet?

FoxTrot Search Admin and Server apps

One word about the setup and the various FoxTrot Search apps. They have a personal edition for searching your local workstation and storage. The major difference with the Professional version is having multiple indices. They also have a server and a per seat license. I tested the Pro version with a few indices first and then set up the server to share these generated search index files with editors. I ended up making more indices after testing. Which so many storage locations I initially tried to make fewer indices but with so many thousands or in some cases millions of files it was better to make an index for each separate storage location (and per specific folder in some cases). It also helps when narrowing down a search, the editor can specify specific locations easily (as well specifying file type, language or file extension).

FoxTrot Search Server app. Define set port to connect to the server.

Once we tested FoxTrot on the local network we needed to make it work with the VPN. And now! The server version of FoxTrot Search allows you to set a fixed port for the server which we could open up on a firewall. In my initial testing I could connect to the server but not to each index of the various storage locations. This was frustrating. So close…. And I will admit here one criticism of FoxTrot Search is the documentation. It doesn’t really exist. They have a user forum and release notes but in my initial setup of the server I didn’t understand exactly what was needed. This could be solved with better documentation. I did resolve this issue with a few emails to the developers who explained to me that each index required its own port and therefore I needed to open more ports on the firewall. Ok, good to know. To confirm that these ports were or were not accessible in my testing I used “nc” in Terminal to scan open ports.

Scanning for specific open ports with nc binary. Port scanning is only one its many useful skills. Read the man pages for more exciting stuff it can do.

So FoxTrot Search is great for search across all active storage and is super helpful for seeing previews of video and audio files you may to use as an editor, but it can help find so much more. If you keep production documents, spreadsheets, PDFs, text files it will search through all them too. So your script or production run sheet is available to help you find what you need.

LTO (tape archives)

In my “where are my files” graphic at the top I show various storage locations including LTO and the cloud. The completed projects always go to LTO (tape) archives. How do we search those with FoxTrot Search? I use Archiware P5 which has a web server that is very easy to search with and restore any files via web login but to make things more fun why not have FoxTrot Search index the archive inventory? Of course it can. There’s a cli command in P5 to export the inventory of every file archived and this is a searchable tab separated (TSV) file. I’ve spent a lot of time with this because I’ve been working on a separate tool to analyse these inventory files (with sometimes millions of items in them) to see historical patterns and predict future trends. More on this specific tool later.

Xsan is archived to LTO (tape) with P5 and to the cloud. Postlab Drive with proxies (smaller version of original media) don’t need to be backed up but other creative production files do. The cycle continues.

The Cloud

New to many is the cloud. Proxies for editors in my recommended workflows get stored on a cloud drive like Postlab, FoxTrot Search can search that too. So many places for originals or backup copies to be. FoxTrot Search should be indexing all of them for you.

In a future blog post I will discuss new media asset management (MAM) systems and what’s changed over the years, but for now if you need a tool search across all active storage then take a look at FoxTrot Search.

Postlab Merge – Final Cut Pro workflow extension

Postlab Merge – Final Cut Pro workflow extension

Final Cut Pro (renamed recently from Final Cut Pro X) has “workflow extension” which allow third party apps to offer access to their interface from within Final Cut Pro.

In the top left corner of the Final Cut Pro windows is the “extension” icon. Click it for access to available extension.

Today you’ll see “Postlab Merge” which was a successful merger of MergeX software and Postlab. Install the latest version of Postlab (v20.4 as of this post) which includes the extension.

Use the Postlab Merge workflow extension to transfer metadata between FCP libraries.

Final Cut Pro Proxy Workflow

Apple’s Final Cut Pro has a new proxy workflow. It’s even easier than before. Make proxies on import, or transcode afterwards. Create a new proxy library or copy events with only proxies, so many options to fit the workflow you need. It’s quick to upload smaller proxies to the cloud and work remotely with your team. Re-connect to the original footage for outputs, colour grading and archiving your project when you’re done.

Final Cut Pro and the Proxy Workflow

“Take your creativity anywhere. Maximize portability and performance by creating proxy copies of your media — as low as 1/8 size — in ProRes Proxy or H.264. The latest proxy engine allows you to create a proxy-only copy of your library to share locally or via the cloud and displays original media if proxies aren’t available. Third party tools such as review and approval app frame.io can also generate and deliver proxies to a Final Cut Pro library.” (Apple.com)

I’ve written about another kind of proxy workflow before, but we will refer to that as the replace-originals-with-smaller-versions workflow and now we have the built-in easy proxy workflow. This new way is much easier. And it’s built-in.

I’ll go over the basic workflow for making proxies and getting your library ready for use with Postlab or other similar cloud collaboration tools…. Seriously, there are no other similar tools! But we’ll go over how to keep your library small and light.

Part One – Final Cut Pro

Final Cut Pro 10.5 is the newest version of Final Cut Pro (which drops the “X”). Ready for Apple Silicon Macs and backwards compatible with macOS 10.15.6 (Catalina).

This new proxy workflow is compatible with Final Cut Pro X v10.4.9 and 10.4.10 as well the newest version 10.5. There were extra bug fixes (LUT for proxies) and new methods (copy new library with proxies) in 10.5 but the addition of the automatic proxy creation on import started with 10.4.9.

Final Cut Pro version 10.5

Import Preferences

First step. Check your import preferences. Final Cut will refer to these when importing. The most important thing to check is that “leave files in place” is selected. This helps us keep the library light and portable. Especially important for editing with Final Cut Pro and Postlab. Keep all media and cache files outside of the library. The second this to check is to choose your proxy format (Pro Res Proxy or H264) at the size you want.

Final Cut Pro Import preferences window.

Choose how small or how large you want your proxies to be. Smaller proxies are faster to transfer and take up less storage but may not be ideal for editing your specific camera footage. Try to find a format that works best for your edit workflow.

Final Cut Pro – Proxy Frame sizes

You also have the option of creating proxies form footage that exists already in the library. Choose “Transcode Media” and select your options.

Final Cut Pro – Transcode Media (menu option)

Part Deux – Editing in the Cloud with Postlab

Once you launch Postlab and login you’ll want to create a production and a library to edit. You have the option of importing an existing library or create a new one. Remember, only import your library if it is super light weight and the media is stored outside (not inside) the library.

Importing a lightweight Final Cut Pro library involves creating a name, writing a description and choosing the media location. If editing off centrally shared storage (on premise) or in the cloud (i.e. Postlab drive) then use “Shared” option. If everyone is using their own storage (external hard drives, NAS, SAN, etc) then choose “Individual”.

If you are creating a new empty library in Postlab then be sure to check the Postlab preferences – Templates tab to select what version of Final Cut Pro for the default empty library and if you want to use a Final Cut Pro template you’ve created already. This is a powerful option for keeping a team working with standard set of tools.

Postlab Template Preferences

Now we start editing. Click “Start Editing” in Postlab. Final Cut Pro will open with your new library.

When you’ve made changes and want to check your Final Cut Pro project back into Postlab switch applications back to Postlab from Final Cut Pro and add a comment.

Postlab check-in (write a comment and upload your work)

Once you’ve checked your project in a few times you’ll notice the list of comments you or your team have made with each check in. These will help you decide what project to revert to, if you need to. The icons (on the right) will allow you to revert, open a copy or export out the version you select.

Postlab – List of comments

Lastly, there is the status menu which you can use to mark the progress of the project.

I hope this helps you get started with the Final Cut Proxy workflow and ready to use Postlab too.

Thunderbolt Storage in the field: Part 1 – The Gamma Carry test

I’m testing the new Accusys Gamma Carry thunderbolt storage in a series of blog posts with real life situations. Filming on location and editing with a remote team requires a combination of good workflow, great apps, and excellent storage. The Gamma Carry is small sturdy box meant for on set and on location editing and camera off loading. It also survives going back and forth between your office and your home office, and wherever you need to go. Disclaimer: I was not paid to write this review and this blog post is an ongoing field test with storage required by every day editors making films using local thunderbolt storage and the cloud. This is their story.

In the beginning….

The start of every creative editorial project is choosing the tools needed for the job, and working as team to make it happen. Making a film requires the hard work of everyone, not the least the IT / Tech who supports the crew and prepares the gear. On set the DIT (digital imaging technician) copies camera cards to multiple hard drives and backup devices before going back to the office to copy them to LTO (archive tape storage), but before the DIT can do their job the IT / Tech has to set up the RAID and design the best and safest backup and archive workflow.

Gamma Carry Overview

The Gamma Carry is an 8 drive external RAID box and it really small and sturdily built. There is a solid metal handle built into the top case and the drives carriers where you put the drives are metal and very solid. This entire unit is built to be solid and protect the drives it contains. The drive tray have a pin to lock the track in place when inserted as well as a thumb screw to lock them in. All drives must be attached with provided screws which is not as convenient as some others with quick plastic tray mounts, but this metal cage for your drive is solid and feels safe for a RAID that will probably be transported everywhere. Keep the data safe!

There are other little touches which are well thought out. The blue red green blinky lights are perfect for any Christmas holiday party and tell you important information about the state of your RAID at a glance but you with one button push turn off all the blinky lights. And keep working in the dark editorial suite not bothered by anything blinking. There’s a mute button and an extra port on the back with 60W to charge your laptop via the RAID. Very useful.

Gamma Carry The Setup

The set up of the Gamma Carry software wise is identical to the Accusys A12 T-Share which I set up recently and once you have the Accusys Mac installer then you have the RaidGuardX app. And the usual IT caveats apply, the software is not signed so you will have to right click on it to open it up. Hopefully they will sign this and notarize their software to make this easier for end users. Also there is a requirement for JRE to run RaidGuardX which means downloading and installed Java on your Mac. Also not optimal but only necessary for the RAID setup.

RaifGuardX detects the Gamma Carry

The set up of the Gamma Carry is the same as the Accusys A12 T-Share and a raid array you built with RaidGuardX will be recognized in the Gamma Carry. That could be a good thing, or not. I did have some excitement when it recognized the array I had built previously (I used the same drives form the Xsan setup). And this was the next step to resolve, because tp create a new array, or delete the array then create a new one I have to do one more thing. Since they were used with an Xsan I had a LUN label that identified them as such and had to remove this LUN label before proceeding. Occasionally we see this issue when re-using drives that had once made up a RAID which was part of an Xsan.

Delete the existing RAID array…. button grayed out.

Xsan to the resuce.

To see the RAID arrays avaialable when building an Xsan you can use the cvlabel command to list them. You can also use it to remove this label. WARNING: Do not do this when connect to an Xsan or Stornext storage network. Unless you know what you are doing. You are warned. This is dangerous. Removing a LUN label can bring down the entire SAN. That’s it. Now you know.

sudo cvlabel -l                                      

/dev/rdisk3 [ACCUSYS Gamma Carry      366] acfs-EFI "accusys"Sectors: 46881814495. Sector Size: 512.  Maximum sectors: 46881814495. Stripebreadth: 0.


sudo cvlabel -u "accusys"           

*WARNING* This program will remove the volume label from the
          device specified (accusys).

          After execution, the devices will not be usable by the
          Xsan. You will have to relabel the
          device to use it on the Xsan.


Do you want to proceed? (Y / N) -> y
Requesting disk rescan .% 
                                                                                     sudo cvlabel -l                          

/dev/rdisk3 [ACCUSYS Gamma Carry      366] unknown  Sectors: 46881814495. Sector Size: 512.  Stripebreadth: 0.

So all is good again now we can create new RAID arrays now that the Xsan LUN label was removed. Back to work! Once the drive is set up in Disk Utility as a new volume then you’re ready to go. In this case I added one more drive and created a 5 drive RAID5 set and formatted as HFS+. In my testing this was fast enough and would be faster if all filled with drives or SSDs. There are variations of this hardware with SSD ports instead of 1 or 2 drive bays to allow quicker ingest of SSDs which have camera footage on them.

Speed test of a 5 drive RAID5. Fast enough!

I then set up 48 hours of drive copies via Hedge for some testing of the RAID hardware. Thanks to Hedge Connect I get notified when the large copy finished. In this I was copying Thunderbolt 2 and 3 external hard drives (thunderbolt but hard drive and not that fast) to the Gamma Carry to be copied. The source drives are slow. Copying from SSDs would be way faster.

And of course Hedge found some minor warnings with these old drives. People don’t like LTO archival tape and want to have stacks of hard drives, but that data on those hard drives won’t last forever. Keep your important data in three places on two different kinds of media (tape, cloud, drives etc). This test of mine was to copy off old drives and use the Gamma to re-organize and re-sort for a new edit project. Now to back up to the cloud and set up Postlab projects with postlab drive for the proxies. In the mean time the editors can get the original footage when I carry over the Gamma Carry thunderbolt RAID.

This is part 1 and after some more testing I will publish some real world tests and experiences.

Thunderbolt Xsan: Set up a T-SAN

Setting up your very own Xsan at home… What could be more exciting? Nothing like SAN storage to cure those stacks of hard drive blues. Don’t have a spare fibre channel switch or fibre channel storage at home? No problem Grab some thunderbolt storage from Accusys and join the fun.

I am testing the A12T3-Share 12-drive desktop Thunderbolt RAID solution to build my Xsan. Accusys also have a 16 drive rack mounted raid storage box if you want to install a nice pro set up in the server room you have tucked neatly in your home office. Ha ha. Seriously, the 12 drive unit is whisper quiet and would be a great addition to any home lab or production storage setup. I mean, aren’t we all doing video production at home these days? And even if we are doing a proxy workflow in the clouds, we still need to store the original footage somewhere before it goes to LTO tape, or backed up in the clouds (hopefully another cloud). A few years ago I tested the Accusys 16 drive Thunderbolt 2 unit and it worked perfectly with my fibre channel storage but this time I am testing the newest Thunderbolt 3 unit. Home office test lab is GO!

It is a pretty straight forward setup but I ran into some minor issues that anyone could run into and so I want to mention them and save you all the frustration by learning from my mistakes. Always be learning. That’s my motto. Or “break things at home not in production”, but if your home is production now, then break things fast and learn very quickly.

First step is to download the software for the RAID and you’ll find it on the Accusys website.

(I found the support downloads well organized but still a bit confusing as to what i needed)

The installer is not signed which in our security conscious age is a little concerning, but examining the package with Suspicious package should allay any concerns.

The installer installs the RAIDGuard X app which you will need to configure the RAID.

Of course, RAIDGuard X needs a Java Runtime Environment to run. Why is this still a thing? Hmm…

RAIDGuardX will allow you to configure your connected Thunderbolt hardware.

Configure the array as you like. I only had four drives to test with. Just enough for RAID5.

Choose your favourite RAID level. I picked RAID5 for my 4 drives.

The first gotcha that got me was this surprisingly simple and easy to overlook section. “Assign LUN automatically” asks you to choose which port that LUN (the configured RAID) will be assigned to. If you don’t check anything like I didn’t in my first run through then you configure a RAID5 array that you’ll never see on your connected Mac. Fun, right? Ha ha.

Xsan requires a sacrifice…. I mean, a LUN (available RAID array). Check your Fibre Channel in System Information. Yes, this is from the thunderbolt storage. Hard to believe, but it’s true!

Setting up enterprise grade SAN storage requires a trip to the Mac App Store. Server.app

Open Server.app, enable Xsan, create a new volume and add your LUN from the Accusys Thunderbolt array. Set the usage to “any” (metadata and data) since this is a one LUN test setup.

Pro tip: connect your Xsan controller to your Open Directory server. Ok, just kidding. You don’t have an OD server in your home office? Hmm… Create an entry in /etc/hosts instead.

If you’ve set up your SAN volume then you will see it listed in the Finder.

Easy shareable SAN storage is possible with thunderbolt RAID arrays from Accusys. No more Fibre channel switches needed. Small SAN setups are possible for creative teams without a server room. This setup was a quiet 12 drive RAID and a Mac mini. Add some Thunderbolt cables. There are four thunderbolt 3 connections and you can add more with an additional RAID. Up to 8 connections with one of them for the Mac Mini running the SAN. Not bad at all. And Xsan is free. Add a Server app from the App Store, but the Xsan client is free and built-in (Xsan has been included with macOS since 10.7 so many years ago). Fibre channel protocol (even through Thunderbolt) is faster than network protocols and great for video production. Fast and shareable storage at home. Or in your office. Thunderbolt Xsan. T-SAN.

From Camera to the Clouds: the very real story of Hedge and Postlab

Update: This proxy workflow applies to Final Cut Pro 10.4.8 and earlier. For the new proxy workflow with FCPX 10.4.9 and Final Cut Pro 10.5 see my new blog post

Note: I want to explain how our current workflow for editing remotely. I am always testing new tools and methods, so workflows change all the time. This is a snapshot in time of what we are trying now. So far it works.

Hedge

We use Hedge to copy camera cards to multiple drives on set (or after a shoot if on location) and then we use Hedge once more to copy one of these drives to the office shared storage (Apple’s Xsan).

Why use Hedge? A nice simple app which hides its complexity well. Hedge has an easy interface to copy multiple sources (camera cards, usually) to multiple destinations (two external drives, or two SAN locations etc), and it does it well. It verifies, and double checks its work and leaves receipts. What was copied when. This is very nice and very useful for troubleshooting. It also has an API which made it easy to build an app that configures Hedge for its current task, and AppleScript support for extending automations after specified actions.

Kyno and Postlab

We are using two other tools in our remote ingest workflow currently: Kyno from Lesspain software for rewrapping and converting camera footage and Postlab, the remote collaboration tool for Final Cut Pro (and Premiere Pro). Testing with other tools is always ongoing and during a recent test of the workflow we also tried EditReady from Divergent Media.

The Workflow (so far)

While we are exploring various workflow automations we are currently doing the following steps manually.

  1. Hedge to copy camera cards two external drives on set, and then Hedge copy the drive to Xsan
  2. Making re-wrapped in MOV files from the original camera MXF files using Kyno and then
  3. Making H264 MOV 4K proxies in Kyno
  4. Uploading finished proxies to Postlab drive using Hedge
  5. Set up FCPX production and new FCPX library from template connected to proxies in Postlab drive

Hedge and Postlab

Hedge is super useful. Two times good. Hedge and Postlab are best friends. And the UI on both shows the simple aesthetic shared by the developers. Three panes. Source / Start to Destination / Projects. Whether you are copying Proxies to Postlab Drive or accessing your editing projects in Postlab the apps will guide you through.

Copying the Proxies with Hedge to Postlab Drive.

Details. Rewrap and Proxies

Workflows will depend on your goals, and your available tools. In this case we are using a Canon camera and ingesting MXF files. In order to edit with small Proxies in FCPX but also be able relink to original (and larger) files easily we need to in our case re-wrap the original MXF to QuickTime MOV container.

Right click on a clip in Kyno to rewrap to Mov.

Originals. Not Proxies.

And to be clear we are treating these in FCPX as “new” originals not as actual FCPX proxies. With the rewrapped MOV files we make transcoded H264 files which are swapped 1 for 1 with the original. When we need to export a final 4K version we can relink to the original 4K source and export easily.

Proxies. Not originals

The transocded H264 4K proxies we made in Kyno were 15x smaller than the original re-wrapped Mov files. We had almost 600GB in originals and 37GB for the 4K H264 proxies!!

Postlab Pro Tips

Working with Postlab pro tip #1 –> keep those FCPX libraries light. Keep all media and cache files out of the library. We knew that and we had Storage Locations set to outside of the library but one new issue came up when the libraries grew really big and we realized the editors were making multiple sequences, not backups, but versions. Now we are trying to work around this habit with Postlab itself. You can check in a version of the library and duplicate library for an alternate version. Modifications of old habits are always tough but technical reasons may force a change in habits here. We will see. Postlab pro tip #2 –> Keep your cache large and fast. By default the Postlab cache is your local drive and only 20GB. If you have a fast SSD or an external drive then move that cache and increase the size. It will help. Trust me.

Kyno vs EditReady

Another small issue we encountered in testing was that we could make the rewrapped Mov files in Kyno or in EditReady and both were fine. The only objection the editors had was that in Kyno we could keep the folder structure of the original camera cards and they felt that this lent some confidence to being able to track the files to the camera card folders if any media was missing or misplaced. The EditReady files kept the original names but they were all in one folder. As the tech I see no issue with FCPX handling these files since we’d be ingesting all the finished proxy files and all the files were named by the camera. Editors should be able to tell which reel the clips were from by the clip name and that’s all you need technically, but you can’t win every argument with an editor. As the tech you need to test alternative tools and methods and see what works technically but also see what can be accepted to work in the way the editors want to work. Changes to workflow are some of the hardest to make, making a system that is used, actually used, by the editors is the goal.

Errata

Errors. If you get them, how do you know? This was one area where I could comment on both Kyno and EditReady. I am spoiled by Hedge and it’s nice reports when it is done copying. And Postlab which has a Help menu :collect logs for support button, very nice. If your software tool is going to process a lot of files (rewrapping then transcoding) I want to know if there were errors. EditReady popped up a window to what had succeeded or failed and Divergent Media support told me to look in the logs for any issues encountered. Not great. While Kyno has a separate jobs window which shows jobs done or failed. But still no report. I would like a receipt or report or log at the end with files converted or failed to convert. It would help troubleshooting any issues when they arise. Tech support for both companies is great and responsive. Thanks again. And I’ll keep sending in feature requests.

Testing. More Testing. And Teamwork.

We are testing this workflow in production with a real project and getting feedback from the team. So far the proxies have proven to be easy to make, quick to upload to Postlab drive, simple to use in FCPX in Postlab. Assembling the cut and editing are going well. We will find out about the colour process when we get to that stage and relink to the originals. Stay tuned.

Thanks!

Thanks to Felipe Baez / cr8ivebeast for his assistance on this part of the workflow. We were having trouble relinking to the original MXF and he gave us the excellent tip to rewrap then in Kyno then make the smaller proxies. Works like a charm. Thank you Felipe! Here’s a link to a video Felipe made showing a similar procedure using Compressor to transcode and then relink in FCPX and it goes to show you that there are lot of ways to do things and to keep trying, and experimenting. You might learn a thing or two.

Minecraft Server for My Kids and My Sanity

Summer time or anytime is a good time to run a minecraft server. And when I am not troubleshooting IT networks, planning SAN storage upgrades, running a DevOps for Dummies bookclub and the MDOYVR podcast then I like to upgrade my minecraft server.

Every time there is an update to the java client there is demand from my users (uh, I mean, my kids) to immediately stop all other work (hey kids, I’m working here! let Dad work) and upgrade the minecraft server.

Like all other IT domains where there are variety of solutions and software fixes to problems, it would seem that Minecraft has official server downloads as well as the unofficial artisanal craft versions. I’ve tried a few, and some out of desperation… there was an incident with netherite blocks and the server wouldn’t start anymore but the Ppaer minecraft server fixed the issues!

The normal routine is that when an official release comes out the other versions may not be up to date as quick, so it’s back to the official versions.

Download the official Minecraft Server

Or try the Paper Minecraft Server

See also Michael Lynn’s two part family harmony blog series which started me on this road to keep the kids happy and maintain family happiness.