Backup Fast, Restore Quicker

Backing up is nice, restoring is better. Slow backups, mean slow restores. Make good decisions, and backup only the files you want to keep to the fastest storage you have.

When working with a fast fibre channel or Thunderbolt SAN your first choice for fastest backup destinations is a Thunderbolt RAID. I recommend to have this onsite with an off site LTO and/or cloud disaster recovery setup (a replicated SAN or shared storage system is nice to have too).

A built-in option to copy Xsan files is cvcp (cv stands for centravision).

cvcp -vxy /Volumes/TSAN/folder /Volumes/GammaRAID/backups

cvcp is fast. Really fast. And cli commands are scriptable. A very smart person (Jasper Siegers) wrote a script called cvcpSync which combined the power of rsync and cvcp. It was awesome. But there are limits to the best of scripts. For my clients I use Archiware P5 with large SAN and other shared storage to simplify the number of things which need to be monitored. One dashboard to monitor tape or cloud backups, tape archives and sync to nearline RAIDs or NAS.

With a recently Thunderbolt SAN deployment with Accusys T-Share I set up the Accusys Gamma Carry as a backup destination. I set up Archiware P5 to do the backup. It was fast. How fast? Over 1Gb/s. Fast backups are also fast restores. With the Gamma Carry I can run a backup then carry it off site. It’s an option as part of a complete backup strategy.

Archiware P5 backup 1.6TB in 53 minutes

(Luckily I have almost 2 TB of video from my Cycliq bike cameras to test backups. Sadly, after my last bike vs car incident I felt obliged to buy bike cameras for my safety. I edit small fun rides when I can. Sometimes traffic near-accidents too. Please be kind, don’t kill cyclists.)

Archiware P5 backup of a Thunderbolt SAN to a Thunderbolt Gamma Carry RAID

Note: In my tests I tested backup to a nearline RAID. I also like to use tape drives. LTO tape is another recommended option for backups or archives. Cloud or other offsite replication is also recommended if possible but is the slowest of all the options. Good to have slow and fast options, offsite and on premise, though any practical solution should be affordable and useful to help decision makers take the steps to preserve data and ultimately their own business.

LTO vs Cloud backup comparison: For LTO backups to one LTO7 drive I normally see 1TB in under 2 hours versus some recent cloud backups I did using rclone which took 9 hours for 1TB. Remember: restore times will equal your backup times. Want to restore 100TB? Got a spare 900 hours? 38 days for cloud restore vs 8 days with one LTO7 drive (much faster if you have more than one drive). Even faster if you restore from a Thunderbolt RAID. Only 2.5 days. Think about it.

Testing equipment:

Hardware: T-Share SAN, Gamma Carry Thunderbolt RAID

Software: Archiware P5

Do you want to build a Thunder SAN?

Thunderbolt Xsan in a box. I’ve written about the Accusys T-share in 2020 (and in 2015 when I first found this cool tech). What’s different now? New year, new macOS. And a new challenge: can we build Xsan only using Terminal? No apps. It’s the journey that counts, right? One nerd’s journey to make an Xsan with macOS 11 Big Sur cli. Destination adventure with family fun, next stop a blinking cursor on a command line prompt.

make Xsan

make —Xsan —-bigger

reboot

Sudo make me an Xsan sandwich. I wish it were that easy! Stick around for the two or three commands you do need.

Xsan goes Terminal

Important commands for using Xsan have always been cvadmin and cvlabel (cv is short for centravision the original creators) but more recently xsanctl and slapconfig are important for creating the SAN and the OD (Open Directory) environment. Read the man pages, search the web, read some help documents. This blog is for entertainment and occasional learnings.

Xsan Commands: where are they?

  /System/Library/Filesystems/acfs.fs/Contents/bin
	cvlabel			sncfgremove
cvaffinity cvmkdir sncfgtemplate
cvcp
cvmkfile sncfgtransform
cvdb
cvmkfs sncfgvalidate
cvdbset cvupdatefs sndiskmove
cverror cvversions snfsdefrag
cvfsck fsm snlatency
cvfsck_compat fsmpm snlicense
cvfsdb has_snfs_label snprodalert35chk
cvfsid mount_acfs snquota
cvgather sncfgconvert wingather
cvgather_fsm sncfgedit xsanctl
cvgather_multipath sncfginstall xsand
cvgather_sum sncfgquery xsandaily

Lots of interesting cv (CentraVision) and sn (StorNext) commands in macOS (this list is from 10.15 Catalina). Besides binaries, what else is there? Examples. A ton of example files:

/System/Library/Filesystems/acfs.fs/Contents/examples/

cvlabels.example fsnameservers.example rasexec.example
cvpaths.example fsports.example rvio.example
fsmlist.example nss_cctl.example snfs_metadata_network_filter.json.example

Just the facts. The Xsan basics

If you don’t have a fibre channel switch and fibre channel hardware RAIDs do not worry. You can build a useful Thunderbolt based Xsan with a little bit of effort. Just a little bit of peril It’s not too perilous, don’t worry.

Apple includes Xsan for free in macOS. Xsan is Apple’s fork Quantum’s StorNext SAN software. Want large fast storage made for Final Cut Pro editors, just add Xsan. Download Server.app from the Mac App Store and make your Xsan. Easy peasey. Right?

Why? Why are we doing this? Nothing beats fibre channel or Thunderbolt SAN speed for editing. Network attached storage (NAS) at 1GbE is barely usable. NAS at 10GbE is much better but still has road blocks for editors. Fibre channel or Thunderbolt with a big enough raid behind your SAN then life is great. Xsan can be shared by a small or media sized team of editors, producers and assistants.

Oh, ok. There is one problem. Apple did a major upgrade of Xsan (now version 7!) in macOS 11 Big Sur but apparently they took out the Xsan config in Server.app. (Note: This is what I was told early on and what seemed to be confirmed by Apple’s recent Xsan cli guide. It turns out that Xsan’s disappearance in Server.app to not be totally correct). Xsan is there in Server.app if you upgrade to macOS Big Sur but when you install Server on a clean macOS there is no Xsan visible in the app. Hmm. What do we do? Apple published a very nice handy guide about how to build Xsan in Terminal. So let’s get started. This is fun.

Accusys T-Share is a Thunderbolt SAN. Connect Macs with Thunderbolt cable.

What do we need? 1) Hardware raid. Ok check I have an Accusys T-Share. It’s a raid with Thunderbolt switch built in. 2) Mac. Ok I have a Mac Mini. 3) A network. Some cables, a switch and a DNS server. Ok I have a new raspberry Pi. That’s perfect.

Raspberry Pi 400 (the amazing linux computer shaped like a keyboard).

Step 1. Hardware raid. With the Accusys T-Share I just have to plug in some clients with a Thunderbolt 3 cable. Let’s fill the RAID with drives. I picked two different sizes. One group of larger disks for a data LUN (main production storage) and two smaller disks for a raid mirror to be used as metadata storage.

Step 2. A Mac running macOS Big Sur 11.5.2. Download the Accusys Mac installer on your Intel Mac (M1 is not supported with the T-Share yet as of this blog post).

Step 3. The network. Ok. This is the fun part. Let’s set up a DNS server. Ok, how do we do that? Remember that raspberry Pi you bought yourself for Christmas but never opened because you have been so busy and well you know life. Ok just me? Well, that one. Let’s use a raspberry Pi. A small inexpensive Linux computer. Install dns masq. It’s perfect for this.

The raid. Not only a great movie it’s the central part of this production media network for creatives. Once the drives are in the raid we have to make raid sets which become LUNs for Xsan. RAID5/6 for the data LUN and RAID1 (mirror) for the metadata LUN.

Read the label. Using Xsan cvlabel

Normally after we create RAID sets in the hardware raid utility we would open up Server.app and label the LUNs for Xsan use. But since we are now hardcore SAN architects we can use Terminal and the cvlabel the command to do this the hard way. Well, it’s not that hard but it can be intimidating the first few times. It’s much easier to label new LUNs than stare at a broken production SAN that has lost its labels. StorNext fun times. More about in another blog post.

Whether using Server.app in the good old days or cvlabel to label your LUNs now you should all be familiar with the command to list available LUNs. For larger SANs that won’t mount the first thing I’d check is see if the LUNs are all there. You don’t want a SAN to mount if it’s missing an important piece of itself.

cvlabel -l

This command lists available LUNs. It’s handy to know. Do this before trouble arises and you will be a cool dude when trouble happens. It does that occasionally. Prepare for the worst, hope for the best, IT motto.

To create labels for newly created RAID arrays use cvlabel to output a text file of the unlabelled LUNs, make some minor changes then label those LUNs. Create the template files first:

cvlabel -c

Edit the file. I like nano. Maybe you like vim. Or BBEdit. Or text edit. Change the name of LUNs from CVFS_unknown to whatever you like. I like to name LUNs based on the hardware they originate from so that I can find them, remove them, fix them or whatever I need to do for troubleshooting. Trust me. It’s a good idea.

cvlabel ~/Desktop/cvlabel
*WARNING* This program will over-write volume labels on the devices specified in the file "/Users/xavier/Desktop/cvlabel". After execution, the devices will only be usable by the Xsan. You will have to re-partition the devices to use them on a different file system.
Do you want to proceed? (Y / N) ->
Requesting disk rescan .

Congratulations this is the hardest part. You’ve labeled the RAID arrays as usable LUNs for Xsan. Ok, just kidding that’s not the hardest part. Have you ever heard of Open Directory? Do you fear LDAP and DNS? Well, maybe you should. It’s always DNS. Just saying.

DNS (domain name system) is just a fancy word for a list of IP addresses and host names. Using the raspberry Pi with dns masq installed we can populate the list of hosts for the Xsan and then we are golden. Hopefully if we did it right. Turns out we can make mistakes here too. Don’t use “.local” domain names. I did. It was late. I blame being tired. Changing them to “.lan” worked better.

Next up we finally create an Xsan in terminal. Or do we? let’s check the hostname first. It’s always DNS.

scutil —get HostName 

CrazyMac.local

scutil --set HostName XsanMac.lan.

And now we make very big Xsan using the Xsan guide example

xsanctl createSan 'VIDEOSAN' --account localadmin --pass 72DERjx1 --user localadmin --cert-auth-name videocert --cert-admin-email administrator@example.com

It was at this point that it started falling apart. It was late. I had messed up my DNS with “.local” and the Xsan wouldn’t go past this basic OD setup. I did what I always do and reach out to my Xsan colleagues and I got some curious feedback. “What do you mean Xsan isn’t in macOS Big Sur Server.app?” Hmm. I don’t see it on a fresh install. On an upgrade from 10.15 Catalina I do. So, uh, Where is it? And then it was revealed. In the View menu. Advanced. Ugh. It’s right there. Almost staring right at me. When I opened the app it said it couldn’t create an Xsan with my “.local”. That was helpful. Fixed that and Xsan with my pre-labeled LUNs was super quick to set up.

Xsan configuration in Server.app. “Ignore ownership” is the best thing ever for creatives. Trust me,

I’ll have to play with the cli set up again soon. Because there were some strange formatting it recommended to me when I tried some variations of the xsanctl createSan. I’ll dig into another day when I have more sleep. Ha ha.

There’s a lot of useful commands in macOS Big Sur Xsan which was upgraded to v7. You can check which version of Xsan you have in macOS with the cvversions command.

In Catalina (macOS 10.15.7)

File System Server:
Server Revision 5.3.1 Build 589[63493] Branch Head BuildId D
Built for Darwin 19.0 x86_64
Created on Tue Jun 22 21:08:03 PDT 2021
Built in /AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/XsanFS/XsanFS-630.120.1/buildinfo

In Big Sur (macOS 11.5.2)

File System Server:
Server Revision 7.0.1 Build 589[96634] Branch Head BuildId D
Built for Darwin 20.0 x86_64
Created on Wed Jun 23 00:32:35 PDT 2021
Built in /System/Volumes/Data/SWE/macOS/BuildRoots/d7e177bcf5/Library/Caches/com.apple.xbs/Sources/XsanFS/XsanFS-678.120.3/buildinfo

There’s a lot of cool new binaries in Xsan v7. We will dig into those next post. For now enjoy this and go forth make some Xsan volumes with Thunderbolt or fibre channel storage. It’s fun.

Hello Big Sur! See ya later Monterey

I am so happy to install macOS Big Sur 11.5.1, now that it is a ready for production. Have fun with macOS Monterey those of you on the bleeding edge. For media professionals using Xsan in production storage environments August is a great month to update to the soon to be yesterday’s bad boy Mr. Big Sur.

Server.app v.5.10 in macOS Catalina 10.15.7

Upgrading to a new major version of macOS can be fraught with peril for a fleet of mac devices but it is potentially fatal for a production SAN environment. That is why we wait. We want a nice stable storage system for our Final Cut Pro editors and other media creatives so it is safe to be one version behind. Less drama that way. We prefer our dramas to be on AppleTV+

Watch TV Upgrade Xsan

It is not boring to watch AppleTV+ while upgrading Xsan

The Xsan upgrade to Big Sur was pretty much not exciting except for one funny roadblock that I had set up myself last as a kind of booby trap for “future me”. More about that later. First the boring stuff. The last few weeks have been very busy updating and re-writing documentation in Pages.app and running multiple redundant full and incremental LTO backups with Archiware P5, syncing to nearline archives, and archiving finalized projects to the LTO shelf in paradise (sounds more exciting when you put it that way don’t you think?). Updating and re-writing documentation can sound like a waste of time but “future you” will appreciate what “past you” was doing today. And today I had fun updating Xsan to macOS Big Sur. Now I must write down all my thoughts before I each too much vegan vanilla ice cream and slip into a food coma.

“Planning for disasters, while hoping for none” is the IT mantra. We planned hard and we were ready to restore Xsan from Time Machine, if we had to. Not a joke. The server is backed up by Time Machine. The data is backed up to LTO, nearline archives racked and stacked in a server room and on redundant thunderbolt RAIDs which are parked on electric trucks ready to blast off at the earliest sign of danger. Well, everything except for the last part. Would be nice. And cloud backups for those clients that want them. Plan for the worst, pay for what you can to keep your business operational and lessen the impact of mechanical failures, human oopsies, or ransomware. Sysadmins are indistinguishable from malware sometimes, but we mean well. More seriously, humans makes mistakes and break things (that, me!) but ransomware is real and my elaborate backup and archive planning has saved a few customers this year.

Ok, now for the fun part. How to upgrade an Xsan to macOS Big Sur (11.5.1). Maybe go read last year’s blog post on upgrading the Xsan to macOS Catalina 10.15.6 which was detailed and thorough. Or read Apple’s new Xsan Management Guide. It’s got all the fundamentals explained.

Xsan volumes are typically made of up fibre channel RAID arrays. Nice icon!

Preparation is key. Be prepared. Get ready. Psych yourself up. I used Greg Neagle’s installinstallmacos.py to download macOS Big Sur as a disk image and had that and the App Store’s Server.app downloaded beforehand and not be dependent on internet access (production SANs are not always internet accessible). It is both true and not true that you can setup Xsan in Big Sur with the Server.app. It is true you need the Server.app for an upgrade from macOS Catalina 10.15.7 but if you’re starting from scratch in macOS 11 you will be building your Xsan in Terminal. Have fun! (We will cover this in a future post).

Download macOS Big Sur and the Server.app. Keep old copies zipped up. Cvlabel is nice too

Server.app manages only three (3) services for an Xsan upgrade: Profile Manager, Open Directory and Xsan. In macOS Big Sur new setups of Server.app Xsan is gone. Why they haven’t taken out Profile Manager and not kept Xsan instead made me scratch my head. No one in their right mind is using Profile Manager to install or manage profiles, they’re using commercial MDM vendors. But Xsan in macOS Big Sur (11) is not only production ready storage SAN awesome it has been upgraded to be compatible with Quantum’s Stornext 7 (previously it was only v.5)

Profile Manager does not belong here. Long Live Xsan!!

Installing macOS 11 Big Sur and upgrading Xsan to v7 is compatible (in my testing) with macOS 10.14 Mojave, 10.15 Catalina and of course macOS 11 Big Sur. If you don’t believe me check out this not updated in forever Apple’s compatibility chart.

Ok, by this time you get the idea I’m an expert, right? I’m ready to upgrade. But I run into my first real road block. And I have only myself to blame. I can’t launch the macOS Big Sur install app. It is blocked. “Contact your administrator”?! I am the sysadmin. Oh, ok. That’s me. What have I done now? I installed Hannes Juutilainen’s Big Sur Blocker last year, that’s what.

Of course I installed that. With Munki. On all my Mac clients that were upgraded to macOS Catalina. (And of course my Xsan controller has Munki!). But no worries, let me read up on my last year’s blog post about it to figure out how I installed it, there must be a launch daemon or something.

this is not how I expected it to go

Hmm, no didn’t mention there. And where is that pesky launch daemon that I can unload and get to this Big Sur install. Oh? It’s a launch agent. Unloaded. Hmm, still no. Ok, delete the app from /usr/local/bin, hmm, nope. ok kill the app process. Ok, now we can install macOS Big Sur. Sorry for the delay. I had told Munki to uninstall the bigsurblocker app and it did for every other Mac, I swear, really. It did.

Please proceed with the macOS Big Sur install

So ready for macOS Big Sur. Oh wait, we noticed that you’re running Server.app and well, we don’t do a lot of the same things anymore in the new Server.app so maybe this is a warning.

Warning. We noticed that you’re running Server.app and we don’t do those fun things anymore.

So a lot of progress bars and stuff. See my last upgrade blog post and it’s the same as installing macOS Big Sur on any Mac, except this Mac Mini is running an Xsan production SAN environment with a lot of RAID arrays in a server rack or two. Ok, yeah, just run the installer.

We noticed that Server app is no longer server app.

After macOS Big Sur is installed zip up your older server.app and drag in your new one (or use that fancy App Store app to do it for you if you’re lazy). Click a bunch of buttons (see all my old blog posts) and launch the new Server.app.

Profile Manager is updating. No one cares.

So we have to wait while the bag of scripts that is Profile Manager gets updated but no one uses it but it’s the most important app in Server.app now, no I am not bitter why do you ask. Xsan is awesome.

Xsan is off. Don’t panic.

Xsan is off. Don’t panic. Where’s my towel? Panic now!

Time to restore from your old Xsan configuration. Wheee…..

Xsan restore configuration.

Activate your Xsan and carry on upgrading all your Mac clients. Note: I did test macOS Mojave 10.14, macOS 10.15 Catalina and of course macOS 11.5.1 Big Sur Xsan clients. All worked.

Xsan on. Power up.

Upgrading Xsan with macOS Big Sur is easy if you’re going from macOS Catalina. Starting from scratch is another story to be covered in another blog post. Also not covered is certificate issues from self-signed certs breaking when I upgraded my Munki and MunkiReport server. That’s definitely another blog post. It’s just a webserver. Just. A. Web. Server. What is so hard? haha

Technical Errata:

With more than one Xsan controller it used to be recommended to upgrade the secondary before the primary but it is now best practise to upgrade the primary first to maintain the sanity of the OD data.

Xsan Upgrade Step by Step:

Clone the controllers. (+ Time Machine backups)
Turn off the clients.
Stop the Xsan Volume.
Run cvfsck on the volume.
**Upgrade the primary.
Confirm the secondary can see the primary.
*Upgrade the secondary.
Confirm the secondary can see the primary.
Check SAN access on both controllers.

Upgrade the clients as desired.

Raspberry Pi for Christmas

I finally got a chance to open up my Christmas present to myself a few months late. I was excited all the same because it was a Raspberry Pi 400.

This is an incredible form factor for the raspberry Pi. It’s an all in one unit that fits into a keyboard. The keyboard is the computer. Just incredible.

Once you plug in power, the mouse and an HDMI monitor you have a working Linux computer running Raspbian. What to do next? Install Tailscale of course !!

Tailscale is a mesh VPN and allows you to link up all your devices in a private network no matter where they are. I’ve blogged about it here. So far I’ve linked up macOS, iOS, windows, Linux centos as well as Synology and QNAP NAS so now let’s add a raspberry Pi.

First things first let’s update the raspberry Pi because it’s been sitting in its box for a while. I didn’t have much luck with the add / remove software gui app (maybe because it was still getting on wifi) but after getting on the network I fired up apt to update all the things.

sudo apt list --upgradeable

Using apt you can get a list of what is upgradeable. This was my first step the gui app didn’t list any software that had updates. A few months in a box and there should be a lot of updates. This is Linux we are talking about.

 sudo apt update

Then it is just a matter of upgrading everything.

The next step is to install Tailscale. Read the instructions for Raspberry Pi on the Tailscale website. Before installing Tailscale we have to install other needed components and we have to tell Linux where to find the software. Similar to my adventures with CentOS Linux and yum you have to tell Raspbian Linux what to do with apt.

First we fetch the signing key and tell it where the repo is. Note: Always be mindful when using curl to download and install items or scripts.

   curl -fsSL https://pkgs.tailscale.com/stable/raspbian/buster.gpg | sudo apt-key add -

curl -fsSL https://pkgs.tailscale.com/stable/raspbian/buster.list | sudo tee /etc/apt/sources.list.d/tailscale.list

Just a few more steps now. Install the needed https components. Check with apt for updates then we can finally install Tailscale.

sudo apt-get install apt-transport-https
sudo apt-get update
sudo apt-get install tailscale

The next step is to authorize Tailscale which is usually done with a login to our account we created the Tailscale network with. But this time I wanted to try a pregenerated keys from the Tailscale admin panel.

sudo tailscale up --authkey tskey-gh37374737292a4847382

Now to test the new secure Tailscale mesh VPN set up I wanted to login to my MunkiReport web server.

Using the Tailscale cli to list all devices on my private network I found the IP.

tailscale status

Find the ip of the server or device from the status list and connect.

Next I outputted the raspberry Pi cli history to a file and sent it to my iPhone where I edited this blog post with the WordPress app.

history > raspberry-history.txt

tailscale file cp raspberry-history.txt iPhone:

I am looking forward to working on many fun projects with my raspberry Pi. After Tailscale is on there then it is easy to connect from my iPhone or from anywhere Tailscale is installed.

In the example below I used ssh shellfish app on my iPhone to connect.

Tailscale has great documentation and despite this I had a small misstep when I went to look at the install instructions. It defaulted to Ubuntu or misdetected my raspbian linux so I wondered for a second why it wasn’t working. I soon realized I’d copy pasted the wrong thing. Checked the drop down install menu for the raspberry Pi instructions and voila happy times.

A Mesh VPN For Everyone

How to use Tailscale (wireguard based) mesh VPN to connect everything

What is Tailscale? It’s a mesh VPN based on the wireguard open source project. It’s a secure network to connect your own devices no matter where they are.

Tailscale is free to use with one account and up to 100 devices, which is enough to see how well this can work to connect up servers, storage and desktops. They have paid plans for teams and enterprise.

Tailscale macOS app icon

macOS and iOS

To start, download Tailscale on your Mac or iPhone then find your IP address. Once you are signed in and have your IP address you can connect easily between devices. For example, on your iPhone open the Tailscale app and see your installed devices. Click on your Mac and the IP will be copied into the clip board. Use this to connect with app such as Secure shellfish for SSH or VNC viewer for remote login. When you’re in the same network it’s impressive, but when you’re on a different network, separated far away, It’s magic.

The real test for me was to install Tailscale on some backup servers I manage to make it more secure and more convenient to access them. I had used a variety of remote control for business services and well, Tailscale is easier, quicker and much more awesome. All the other software I tried was much less awesome.

After using Tailscale mainly for remote control, I tested Tailscale to securely connect my remote Macs to my own MunkiReport server. I use Munki and MunkiReport to manage Macs and having Tailscale allows me to securely connect endpoints to the server without opening up ports on my router. MunkiReport allows me to detect malware (with DetectX plugin) or check on backup jobs with Archiware P5 backup software (using a module I wrote) or a multitude of other diagnostics such as disk space free, apps installed, and all kinds of great hardware and software metrics. So much reporting. And MunkiReport doesn’t need Munki specifically, so if Tailscale is installed for remote control why not report on everything else.

DSM Package Center: Tailscale (and Archiware P5) app on Synology NAS

Synology DSM

Having Tailscale installed in all the Synology NAS I manage in various physical locations allows me to securely connect to all of these NAS from anywhere. With remote work using a NAS is a great way to sync data between locations. Synology has a lot of great built-in tools to make this happen and a very robust quick connect feature combined with ddns, and let’s encrypt certificates to support it. After setting up a few to sync one location to another I was constantly getting notifications of IPs being blocked on my firewall. I had to open a port on my firewall to let in the ssh / rsync traffic through and despite a strong set of firewall rules with a geo block there were still connection failures and password attempts. Using Tailscale I can now have a P5 server set up on one Synology NAS connecting to the Tailscale IP of other remote units and it can easily backup, sync or archive the data from the various locations.

To install the Synology Tailscale package check out this GitHub page. Download the app then side load it (manual install in package center). To enable it you will have to have ssh on, a user with permission to use it, and one command to type.

sudo tailscale up

In one case I didn’t have SSH enabled on the remote unit so I remoted into a Mac on the same network, enabled an admin user, turned on ssh with a time limit on the account, and then logged in. Once the above command is run you will get a link to a website to authenticate the device with your account.

Linux (CentOS)

I have also installed and tested Tailscale on a Linux (CentOS) storage server. In my case a Jellyfish which has a ZFS volume shared over direct 10GbE for Final Cut Pro video editors using nfs or smb. the Jellyfish works well on premise, but wouldn’t it be nice to capture camera cards to the remote storage server via Tailscale? Oh yes it would. And what about playing back some of the video files via VLC on your iPhone! Or Files.app! Yes, to all the above. All made possible with Tailscale. And a huge shout out for their great documentation. Installing Tailscale on CentOS was super simple. Add a yum repo, install, tailscale, and then bring the service up. Couldn’t be easier.

sudo yum-config-manager --add-repo https://pkgs.tailscale.com/stable/centos/7/tailscale.repo

sudo yum install tailscale

Shared Devices

A small, but very exciting, feature was added part way through my testing of Tailscale which made it infinitely more awesome, shared devices. The concept is you are authenticated to your devices and can see in the Tailscale app all the IPs to connect to, but what if you could share one device (computer, server, NAS) with another person? Well, now you can. In the Tailscale admin console choose a device and send a share link to someone, they then will see this devices in their device list as shared. Home users can set up Tailscale to access all their own devices, but now can also choose to share access with a device in particular. For example, if you create an account, open a service (file sharing) and send a share link then the other person will login with the account you create and access the one thing you want them to. Maybe it’s a smb share to drop files. Works great for video collaboration and other kinds of teams.

There’s a whole lot more you can do with Tailscale (and wireguard) mesh VPN but I hope this gives you all some ideas to start with.

2021: Thunderbolt Shared Storage Report

It’s 2021 and what is the state of Thunderbolt Shared Storage?

Thunderbolt Shared Storage is a RAID which you connect to with Thunderbolt (and Ethernet) which can be shared with other workstations. It’s a Thunderbolt SAN. Shared by Thunderbolt.

I’ve long been a fan of Apple’s Xsan and other SAN products that use Fibre Channel (or iSCSI) to connect clients to super fast block storage. It acts like a fast direct attached RAID but you can share it with others. The sharing part is crucial to collaborative workflows. We used XSAN when I was in VFX and I now use XSAN for post production workflows. Editors like to edit, have large video files, and often work in teams. All those camera files aren’t getting any smaller. So you need a lot of storage that can be shared by a team of editors, colourists, motion graphic artists, producers, etc

What you don’t often want is a complicated network infrastructure or a server room with fibre channel switches and fibre channel RAIDs and assorted other equipment. You don’t want that. You’d have to call me and pay me to set up your storage, backup and archive workflow. While it is always recommended to work with a trusted contractor it can be expensive. For small teams a shared Thunderbolt storage SAN can be quick to set up, doesn’t take much room and can easily connect 4-8 editors. Thunderbolt 3 passive copper cables extend up to 2m but optical Thunderbolt cables are available up to 60m.

I want to review one such example of shared Thunderbolt storage that stands out, the Symply Workspace. It’s a RAID, but it’s a SAN too. It is storage you plug into with Thunderbolt, but with an extra ethernet cable you make it a SAN. Inside the storage it has Quantum StorNext which runs most of the world’s expensive enterprise SANs, but it’s in the Symply Workspace and it works with Apple’s Xsan client software (included free with macOS since macOS Lion 10.7 !!). So basically, it is enterprise storage in a Thunderbolt box. One more thing, add a 5 seat license to a simple but powerful media asset management (MAM) tool axle.ai to organize your assets, add keywords and access from anywhere. Almost too good to be true, so I had to test it and see.

I received a Symply Workspace to test with. I like testing storage. My clients always need more storage. I keep telling them to not fill it up, but they do. That’s why we have LTO, nearline and cloud archive with Archiware P5 but that’s another story. For now, let’s test this storage: how easy is it to setup, how awesome is it to edit with, and what is a MAM good for anyway? Let’s find out.

It starts with a box. You open the box. You take the drives out of the box. Two trays of drives. Then there’s the RAID itself. Put the drives in the RAID. No screwdriver or tools necessary, just slide and click. Next step read all documentation online, ha ha, just kidding, don’t read anything keep going. Ok, kidding a bit. There’s an info card in the box with a website link to help explain the setup. But I didn’t read it all, uh, I know Xsan, I can do this, ok, what’s the link? now back to building.

The tricky part is plugging in a few cables, which will be improved in the shipping production version with stenciled labels of where to plug in what. There’s three cables after the power cable. One Thunderbolt to a Mac, that’s easy. One ethernet to your local network, no problem. And lastly one more ethernet to your new SAN production network (aka metadata). Ok, what’s that? Add a 5-port switch and plug in all clients and the storage to this network for SAN private metadata. It really is a SAN. Like Apple’s Xsan or Quantum StorNext, you need a data pipe (40GbE Thunderbolt 3 in this case) to transfer the data, and a metadata network, to talk about the data (1GbE ethernet to our extra switch). The public network will be used to talk to Axle MAM or for re-sharing out the volume to non-Thunderbolt connected Macs.

Once the Symply Workspace is wired up and powered on you’ll be able to reach the unit via a local bonjour name in your web browser (http://symplyworkspace.local:8088) and from there a simple web interface allows you to monitor the status of the unit, download drivers to configure a Mac or Windows client, restart or shutdown, contact support or start troubleshooting if needed. It’s a great tool to do the few things you need to.

So with everyone working remotely how do we connect to our shared storage? I’m glad you asked. There are a lot of good options. My favourite is Tailscale a mesh VPN (based on the open source Wireguard project) which you would install on a locally connected Mac then you can remote in and share a screen or the storage. There’s also the Axle 5-seat license included with the Symply Workspace which can scan your storage, make proxies and serve it all via a website which can also be accessed from anywhere.

To be continued…

Update: Added a clarification that there is an included 5 seat license to Axle media asset management in the Symply Workspace

Do you know where your files are?

Trying to solve the problem of finding production media files across many storage platforms.

Map of storage locations created with Scaple https://www.literatureandlatte.com/scapple/overview

In media production environments you work with high speed and high capacity storage. It can be network based NAS, fibre channel SAN or Thunderbolt DAS. There’s always some backup RAIDs, individual source footage drives, file servers and even network appliances.

Glossary of Storage:

  • SAN – Storage Area Network (typically with Fibre Channel, also with Thunderbolt or iSCSI)
  • NAS – Network Attached Storage (popular vendors include Synology and QNAP)
  • DAS – Direct Attached Storage (hardware or software raid directly attached to a client or server)
  • LTO – Linear Tape Open (tape standard for backups/archives. Current gen8 holds 12TB per tape)
  • Cloud – Other people’s servers and storage. Hosted in a data center (AWS, GCP, Azure, & more)

The variety of file types is astounding: original camera footage, Final Cut Pro projects, stock footage purchased, b-roll shot from other projects, sound effects, music etc. How do you find anything? Files can be in many places, across many different kinds of storage. The question is how do you know where they are?

In the before times…

Before the great pandemic of 2020 I only heard one complaint with finding files on Apple Xsan storage: “Why does spotlight not work on my Mac?” Searching the Xsan volumes was hit and miss. When Spotlight worked it was fast and immediate. And when it didn’t, well…. Not so much. To help finding files we started using one called EasyFind from DevonTechnologies. It was free, and easy to use, but it was not fast enough for ad hoc video searches. Nor could it search across all storage areas at once. A new year, and new solutions required. 

EasyFind. Freeware for finding files. Easy to use but not fast.

One major issue with EasyFind was not being able to refine the search easily for audio and video assets. It seems skewed toward developers and while it allows you to include or exclude some file types globally in settings but it does not allow you to refine the results while searching. This makes it difficult to find what you need when you need it.

Searching for sound effects in EasyFind reveals “.h” source files. Only one file is relevant in these results.

It’s 2021 and a few things have changed over the last year: you may have a lot more places to search (SAN and NAS) and more importantly everyone is working remotely. It has become a lot more challenging to see your files, let alone search through all the storage locations and find what you need. FoxTrot Search to the rescue.

FoxTrot Search Pro. Not freeware but worth every penny. Indexes all locations so that search is fast!!

Editors and other creatives need something that will search through all their different storage places and quickly tell them where a certain file might be. It might be stock footage, a drone shot, sound effect or an old logo. But where exactly is it? What projects was it used in? What was it called?

For those who like organizing their files into logical folders, then the filesystem is your friend. Using The Finder has been the way to find things. But now there’s a lot of folders and a lot of file types in those folders. Choosing what to index in these folders help enormously. You can exclude certain subfolders on a designated storage or kinds of files that are not needed. Do not index mail or chat messages or even source code files if you know the files you want are movies and audio files. Narrow the scope where you can. Find files fast.

Choose what to index in FoxTrot Search and what not to by kind (file type) or ignore by subfolder.

You might think I’d suggest a full enterprise Media Asset Management (MAM) system at this point. And well, in the old days I would have. Apple’s old Final Cut Server worked perfectly well with classic Final Cut Pro (v6 and v7) but required rigid workflows and ingest habits. Same issue with a lot of more expensive bespoke enterprise systems. They catalog assets but at a great cost. Some clients avoid these for the cost, and others for the workflow restrictions. I’ll talk about some newer options later but for now editors just want to find the files in their own folder structure on their production storage. Is that too much to ask?

Using FoxTrot Search Pro and editor can easily search across multiple indicies (each index is a unique storage or separate folder location) than an admin has already set up and created. No waiting for indexing in real time, search now, with results instantly show in app. Narrow down and refine your search results easily. Don’t want to see Mail messages, or images, only audio files? Easy. Then further define only the file type you want. It works.

FoxTrot Search can refine searches by kind and extension to find the relevant files you actually want.

One of the recommendations to my clients with large storage (SAN or NAS) is to have secondary nearline archive as well as backups. Either a Thunderbolt RAID array like the Accusys Gamma Carry or a desktop QNAP or rack mounted Synology NAS. Using Synology is great for many reasons, for example, it includes a nice web login, and it can search for files. But to be completely honest, the built-in search only works sometimes. A common issue with search on the Synology is the corruption of the search index. It seems to happen some times for some storage locations, but all the time somewhere. Rebuilding this does not seem to fix it for long if at all. This was the major reason for switching to FoxTrot Search. I want something that works every time. Across all storage volumes: SAN, and NAS.

Synology NAS corrupt search index.

To be fair, I had to rebuild an index or two in FoxTrot Search as well, but it worked. Every. Single. Time. Rebuilding the Synology search indexes never seemed to fix the issues. With FoxTrot Search it’d warn me there was a lot (A LOT!) of items to index and it will take time but that’s why I’d do it over an evening and rebuild automatically them after hours or every weekend. FoxTrot Search would also warn me of problem files that took too long to index and could then be added to exclusion lists automatically.

When Spotlight stops working, your Synology search index is corrupted again or just want a great search tool for active storage locations, I recommend FoxTrot Search. It searches across all storage and provides access to the files where possible. And it’s fast. Really fast. Did I mention that yet?

FoxTrot Search Admin and Server apps

One word about the setup and the various FoxTrot Search apps. They have a personal edition for searching your local workstation and storage. The major difference with the Professional version is having multiple indices. They also have a server and a per seat license. I tested the Pro version with a few indices first and then set up the server to share these generated search index files with editors. I ended up making more indices after testing. Which so many storage locations I initially tried to make fewer indices but with so many thousands or in some cases millions of files it was better to make an index for each separate storage location (and per specific folder in some cases). It also helps when narrowing down a search, the editor can specify specific locations easily (as well specifying file type, language or file extension).

FoxTrot Search Server app. Define set port to connect to the server.

Once we tested FoxTrot on the local network we needed to make it work with the VPN. And now! The server version of FoxTrot Search allows you to set a fixed port for the server which we could open up on a firewall. In my initial testing I could connect to the server but not to each index of the various storage locations. This was frustrating. So close…. And I will admit here one criticism of FoxTrot Search is the documentation. It doesn’t really exist. They have a user forum and release notes but in my initial setup of the server I didn’t understand exactly what was needed. This could be solved with better documentation. I did resolve this issue with a few emails to the developers who explained to me that each index required its own port and therefore I needed to open more ports on the firewall. Ok, good to know. To confirm that these ports were or were not accessible in my testing I used “nc” in Terminal to scan open ports.

Scanning for specific open ports with nc binary. Port scanning is only one its many useful skills. Read the man pages for more exciting stuff it can do.

So FoxTrot Search is great for search across all active storage and is super helpful for seeing previews of video and audio files you may to use as an editor, but it can help find so much more. If you keep production documents, spreadsheets, PDFs, text files it will search through all them too. So your script or production run sheet is available to help you find what you need.

LTO (tape archives)

In my “where are my files” graphic at the top I show various storage locations including LTO and the cloud. The completed projects always go to LTO (tape) archives. How do we search those with FoxTrot Search? I use Archiware P5 which has a web server that is very easy to search with and restore any files via web login but to make things more fun why not have FoxTrot Search index the archive inventory? Of course it can. There’s a cli command in P5 to export the inventory of every file archived and this is a searchable tab separated (TSV) file. I’ve spent a lot of time with this because I’ve been working on a separate tool to analyse these inventory files (with sometimes millions of items in them) to see historical patterns and predict future trends. More on this specific tool later.

Xsan is archived to LTO (tape) with P5 and to the cloud. Postlab Drive with proxies (smaller version of original media) don’t need to be backed up but other creative production files do. The cycle continues.

The Cloud

New to many is the cloud. Proxies for editors in my recommended workflows get stored on a cloud drive like Postlab, FoxTrot Search can search that too. So many places for originals or backup copies to be. FoxTrot Search should be indexing all of them for you.

In a future blog post I will discuss new media asset management (MAM) systems and what’s changed over the years, but for now if you need a tool search across all active storage then take a look at FoxTrot Search.

Postlab Merge – Final Cut Pro workflow extension

Postlab Merge – Final Cut Pro workflow extension

Final Cut Pro (renamed recently from Final Cut Pro X) has “workflow extension” which allow third party apps to offer access to their interface from within Final Cut Pro.

In the top left corner of the Final Cut Pro windows is the “extension” icon. Click it for access to available extension.

Today you’ll see “Postlab Merge” which was a successful merger of MergeX software and Postlab. Install the latest version of Postlab (v20.4 as of this post) which includes the extension.

Use the Postlab Merge workflow extension to transfer metadata between FCP libraries.

Final Cut Pro Proxy Workflow

Apple’s Final Cut Pro has a new proxy workflow. It’s even easier than before. Make proxies on import, or transcode afterwards. Create a new proxy library or copy events with only proxies, so many options to fit the workflow you need. It’s quick to upload smaller proxies to the cloud and work remotely with your team. Re-connect to the original footage for outputs, colour grading and archiving your project when you’re done.

Final Cut Pro and the Proxy Workflow

“Take your creativity anywhere. Maximize portability and performance by creating proxy copies of your media — as low as 1/8 size — in ProRes Proxy or H.264. The latest proxy engine allows you to create a proxy-only copy of your library to share locally or via the cloud and displays original media if proxies aren’t available. Third party tools such as review and approval app frame.io can also generate and deliver proxies to a Final Cut Pro library.” (Apple.com)

I’ve written about another kind of proxy workflow before, but we will refer to that as the replace-originals-with-smaller-versions workflow and now we have the built-in easy proxy workflow. This new way is much easier. And it’s built-in.

I’ll go over the basic workflow for making proxies and getting your library ready for use with Postlab or other similar cloud collaboration tools…. Seriously, there are no other similar tools! But we’ll go over how to keep your library small and light.

Part One – Final Cut Pro

Final Cut Pro 10.5 is the newest version of Final Cut Pro (which drops the “X”). Ready for Apple Silicon Macs and backwards compatible with macOS 10.15.6 (Catalina).

This new proxy workflow is compatible with Final Cut Pro X v10.4.9 and 10.4.10 as well the newest version 10.5. There were extra bug fixes (LUT for proxies) and new methods (copy new library with proxies) in 10.5 but the addition of the automatic proxy creation on import started with 10.4.9.

Final Cut Pro version 10.5

Import Preferences

First step. Check your import preferences. Final Cut will refer to these when importing. The most important thing to check is that “leave files in place” is selected. This helps us keep the library light and portable. Especially important for editing with Final Cut Pro and Postlab. Keep all media and cache files outside of the library. The second this to check is to choose your proxy format (Pro Res Proxy or H264) at the size you want.

Final Cut Pro Import preferences window.

Choose how small or how large you want your proxies to be. Smaller proxies are faster to transfer and take up less storage but may not be ideal for editing your specific camera footage. Try to find a format that works best for your edit workflow.

Final Cut Pro – Proxy Frame sizes

You also have the option of creating proxies form footage that exists already in the library. Choose “Transcode Media” and select your options.

Final Cut Pro – Transcode Media (menu option)

Part Deux – Editing in the Cloud with Postlab

Once you launch Postlab and login you’ll want to create a production and a library to edit. You have the option of importing an existing library or create a new one. Remember, only import your library if it is super light weight and the media is stored outside (not inside) the library.

Importing a lightweight Final Cut Pro library involves creating a name, writing a description and choosing the media location. If editing off centrally shared storage (on premise) or in the cloud (i.e. Postlab drive) then use “Shared” option. If everyone is using their own storage (external hard drives, NAS, SAN, etc) then choose “Individual”.

If you are creating a new empty library in Postlab then be sure to check the Postlab preferences – Templates tab to select what version of Final Cut Pro for the default empty library and if you want to use a Final Cut Pro template you’ve created already. This is a powerful option for keeping a team working with standard set of tools.

Postlab Template Preferences

Now we start editing. Click “Start Editing” in Postlab. Final Cut Pro will open with your new library.

When you’ve made changes and want to check your Final Cut Pro project back into Postlab switch applications back to Postlab from Final Cut Pro and add a comment.

Postlab check-in (write a comment and upload your work)

Once you’ve checked your project in a few times you’ll notice the list of comments you or your team have made with each check in. These will help you decide what project to revert to, if you need to. The icons (on the right) will allow you to revert, open a copy or export out the version you select.

Postlab – List of comments

Lastly, there is the status menu which you can use to mark the progress of the project.

I hope this helps you get started with the Final Cut Proxy workflow and ready to use Postlab too.

Thunderbolt Storage in the field: Part 1 – The Gamma Carry test

I’m testing the new Accusys Gamma Carry thunderbolt storage in a series of blog posts with real life situations. Filming on location and editing with a remote team requires a combination of good workflow, great apps, and excellent storage. The Gamma Carry is small sturdy box meant for on set and on location editing and camera off loading. It also survives going back and forth between your office and your home office, and wherever you need to go. Disclaimer: I was not paid to write this review and this blog post is an ongoing field test with storage required by every day editors making films using local thunderbolt storage and the cloud. This is their story.

In the beginning….

The start of every creative editorial project is choosing the tools needed for the job, and working as team to make it happen. Making a film requires the hard work of everyone, not the least the IT / Tech who supports the crew and prepares the gear. On set the DIT (digital imaging technician) copies camera cards to multiple hard drives and backup devices before going back to the office to copy them to LTO (archive tape storage), but before the DIT can do their job the IT / Tech has to set up the RAID and design the best and safest backup and archive workflow.

Gamma Carry Overview

The Gamma Carry is an 8 drive external RAID box and it really small and sturdily built. There is a solid metal handle built into the top case and the drives carriers where you put the drives are metal and very solid. This entire unit is built to be solid and protect the drives it contains. The drive tray have a pin to lock the track in place when inserted as well as a thumb screw to lock them in. All drives must be attached with provided screws which is not as convenient as some others with quick plastic tray mounts, but this metal cage for your drive is solid and feels safe for a RAID that will probably be transported everywhere. Keep the data safe!

There are other little touches which are well thought out. The blue red green blinky lights are perfect for any Christmas holiday party and tell you important information about the state of your RAID at a glance but you with one button push turn off all the blinky lights. And keep working in the dark editorial suite not bothered by anything blinking. There’s a mute button and an extra port on the back with 60W to charge your laptop via the RAID. Very useful.

Gamma Carry The Setup

The set up of the Gamma Carry software wise is identical to the Accusys A12 T-Share which I set up recently and once you have the Accusys Mac installer then you have the RaidGuardX app. And the usual IT caveats apply, the software is not signed so you will have to right click on it to open it up. Hopefully they will sign this and notarize their software to make this easier for end users. Also there is a requirement for JRE to run RaidGuardX which means downloading and installed Java on your Mac. Also not optimal but only necessary for the RAID setup.

RaifGuardX detects the Gamma Carry

The set up of the Gamma Carry is the same as the Accusys A12 T-Share and a raid array you built with RaidGuardX will be recognized in the Gamma Carry. That could be a good thing, or not. I did have some excitement when it recognized the array I had built previously (I used the same drives form the Xsan setup). And this was the next step to resolve, because tp create a new array, or delete the array then create a new one I have to do one more thing. Since they were used with an Xsan I had a LUN label that identified them as such and had to remove this LUN label before proceeding. Occasionally we see this issue when re-using drives that had once made up a RAID which was part of an Xsan.

Delete the existing RAID array…. button grayed out.

Xsan to the resuce.

To see the RAID arrays avaialable when building an Xsan you can use the cvlabel command to list them. You can also use it to remove this label. WARNING: Do not do this when connect to an Xsan or Stornext storage network. Unless you know what you are doing. You are warned. This is dangerous. Removing a LUN label can bring down the entire SAN. That’s it. Now you know.

sudo cvlabel -l                                      

/dev/rdisk3 [ACCUSYS Gamma Carry      366] acfs-EFI "accusys"Sectors: 46881814495. Sector Size: 512.  Maximum sectors: 46881814495. Stripebreadth: 0.


sudo cvlabel -u "accusys"           

*WARNING* This program will remove the volume label from the
          device specified (accusys).

          After execution, the devices will not be usable by the
          Xsan. You will have to relabel the
          device to use it on the Xsan.


Do you want to proceed? (Y / N) -> y
Requesting disk rescan .% 
                                                                                     sudo cvlabel -l                          

/dev/rdisk3 [ACCUSYS Gamma Carry      366] unknown  Sectors: 46881814495. Sector Size: 512.  Stripebreadth: 0.

So all is good again now we can create new RAID arrays now that the Xsan LUN label was removed. Back to work! Once the drive is set up in Disk Utility as a new volume then you’re ready to go. In this case I added one more drive and created a 5 drive RAID5 set and formatted as HFS+. In my testing this was fast enough and would be faster if all filled with drives or SSDs. There are variations of this hardware with SSD ports instead of 1 or 2 drive bays to allow quicker ingest of SSDs which have camera footage on them.

Speed test of a 5 drive RAID5. Fast enough!

I then set up 48 hours of drive copies via Hedge for some testing of the RAID hardware. Thanks to Hedge Connect I get notified when the large copy finished. In this I was copying Thunderbolt 2 and 3 external hard drives (thunderbolt but hard drive and not that fast) to the Gamma Carry to be copied. The source drives are slow. Copying from SSDs would be way faster.

And of course Hedge found some minor warnings with these old drives. People don’t like LTO archival tape and want to have stacks of hard drives, but that data on those hard drives won’t last forever. Keep your important data in three places on two different kinds of media (tape, cloud, drives etc). This test of mine was to copy off old drives and use the Gamma to re-organize and re-sort for a new edit project. Now to back up to the cloud and set up Postlab projects with postlab drive for the proxies. In the mean time the editors can get the original footage when I carry over the Gamma Carry thunderbolt RAID.

This is part 1 and after some more testing I will publish some real world tests and experiences.