I was chatting with Per Olofsson on a recent episode of the MacDevOps podcast about some recent fixes with relocatable Python he did for MunkiReport version 5.7.0 and I happened to mention how much I love my MR dashboards with custom hot keys. He is a long time user of MunkiReport but hadn’t heard that you could make custom dashboards and I couldn’t remember where I had heard of it or even how I made them. Pretty typical of tech these days I think. You learn something, you make something and then you move to the next thing and forget what you were doing or how you did it. Well, thanks to documentation we can share the knowledge and spread the love.
Custom Dashboards
The MunkiReport wiki actually has a short entry which explains how to make a custom Dashboards. Basically, add some YAML files in the dashboards folders and you’re done. Follow the Read Me file for instructions. Pro Tip: Use the Widget Gallery in MR to find useful pieces to build into your dashboards. Note: I’ve added these custom dashboards to my local folder which is set in my “.env” to be outside of the main munkireport folder so it easier to update across version upgrades.
In the old days we used to forward ports. On your router the traffic for a server or service went to a port (where a number represents a service, some which are defined, but can be arbitrary) and to a destination IP address. Well, wouldn’t you know it, if ssh is port 22 or web traffic is on port 80 then everyone and their port scanner comes knocking. So then your firewall is tested, and then auto-ban and geo-block and emails go out. What if we could avoid that and not open (or forward) any port to make services work across the internet?
Tailscale is a mesh-VPN which uses WireGuard to securely establish a mesh (point to point) VPN of your devices. Suddenly your iPhone can securely send files to your Mac or raspberry Pi across the world. How cool is that? In today’s advanced lesson: you can backup and sync your Synology NAS using Archiware P5.
Step 1: Setting up Tailscale on Synology
It honestly used to be harder than this, these days you can simply add the Tailscale package via the Synology package center app and you’re done. Almost. There’s one more step.
Step 2: Set up Outgoing VPN access via Tailscale which requires editing some files (which necessitate Terminal and remote login access). This only has to be done once but future updates may require fixes. This was tested in DSM 7. Pro tip: only allow remote access to a restricted and time limited account so you don’t leave it on accidentally.
Step 3. Install Archiware P5 on Synology NAS
Using Archiware P5 to Backup and Sync your NAS is a good thing if you’re already using Archiware P5 to backup and sync all the other things, then at least you have only one dashboard to look at. I use P5 with my clients to backup their shared storage to LTO and it makes sense to backup all the things no matter where they are with P5 also. With Synology NAS package center it’s a simple one-click install for P5. Add your P5 clients to your P5 server via Tailscale and you’ve got a secure setup.
This post is just a quick overview of using Tailscale to set up your P5 clients (which is your Synology NAS in this case).
How to deploy applications using munki and simplemdm
You want to deploy apps to Macs but you also want to keep them up to date, fear not, we have a way. If you are using SimpleMDM for Mac management but hate the way MDMs deploy applications then listen up it’s easy(*) to set up Munki and use the power Autopkg to deploy and update all your apps. Note: SimpleMDM also offers a short list of curated apps to deploy without any extra setup but these instructions are for those who want to choose the apps they want to deploy. If that’s you then read on.
Managed Software Centre is the AppStore for all your apps you want your Macs to have
SimpleMDM: The basics
Macs are enrolled into SimpleMDM, then assigned to Groups. Groups have profiles assigned to them to enforce and escrow FileVault or set other policies. Simple enough, right?
Ok, what about apps?
SimpleMDM Category setting for a Munki’s Managed Software Centre
When you have apps in your Catalog you can assign a Munki category to the applications to make it show up in a nice group using Managed Software Centre (the client facing app).
With Apps in your Catalog you can manage them with Assignment Groups which are created as Munki (or not-Munki aka Standard). Next select Managed or Self-Serve, two concepts which make sense to Munki admins. One set of apps is required and will be installed without asking, and the other group is presented to the end user to choose as needed (they’re optional).
API key options. Allow Munki plugin access
API key
How do we get applications we want into SimpleMDM? Two ways exist. Import them manually and deploy via MDM or setup up Autopkg. For this we need the API key. Note: Only the munki plugin permissions are needed. Put the key into the Autopkgr.app SimpleMDM integration or set them as an environment variable and use autopkg in Terminal.
Autopkgr app choose autopkg recipes to use
Select recipes using Autopkgr (Linde Group) from the curated list of recipes created by IT Admins around the world or create your own recipes. What used to be a painstakingly difficult process by hand is now much easier with Recipe Robot by Elliot Jordan to help fish out the AppCast / Sparkle / Download URLs and transform into a nice autopkg recipe to be used by Munki (and ingested into SimpleMDM).
autopkg run -v Postlab.munki.recipe -k MUNKI_REPO_PLUGIN="SimpleMDMRepo" -k MUNKI_REPO="" -k extract_icon=True
MunkiImporter
Using API key provided by environment variable.
MunkiImporter: Using repo lib: MunkiLib
MunkiImporter: plugin: SimpleMDMRepo
Managed Software Centre notification
Managed Software Centre
Once Macs are enrolled and added to a SimpleMDM Group with the Munki assignment then the Managed Software Centre app will allow users to use the Self-Serve portal to install optional apps. Managed apps will install invisibly in the background.
The beauty of this integration is that Munki is awesome and works well. It is battle tested by many companies and organizations around the world. Using autopkg and its recipes to check for updates allows for a seamless automation of new apps into your catalog and then onto your fleet. Updated Macs are happy Macs.
Backing up is nice, restoring is better. Slow backups, mean slow restores. Make good decisions, and backup only the files you want to keep to the fastest storage you have.
When working with a fast fibre channel or Thunderbolt SAN your first choice for fastest backup destinations is a Thunderbolt RAID. I recommend to have this onsite with an off site LTO and/or cloud disaster recovery setup (a replicated SAN or shared storage system is nice to have too).
A built-in option to copy Xsan files is cvcp (cv stands for centravision).
cvcp is fast. Really fast. And cli commands are scriptable. A very smart person (Jasper Siegers) wrote a script called cvcpSync which combined the power of rsync and cvcp. It was awesome. But there are limits to the best of scripts. For my clients I use Archiware P5 with large SAN and other shared storage to simplify the number of things which need to be monitored. One dashboard to monitor tape or cloud backups, tape archives and sync to nearline RAIDs or NAS.
With a recently Thunderbolt SAN deployment with Accusys T-Share I set up the Accusys Gamma Carry as a backup destination. I set up Archiware P5 to do the backup. It was fast. How fast? Over 1Gb/s. Fast backups are also fast restores. With the Gamma Carry I can run a backup then carry it off site. It’s an option as part of a complete backup strategy.
Archiware P5 backup 1.6TB in 53 minutes
(Luckily I have almost 2 TB of video from my Cycliq bike cameras to test backups. Sadly, after my last bike vs car incident I felt obliged to buy bike cameras for my safety. I edit small fun rides when I can. Sometimes traffic near-accidents too. Please be kind, don’t kill cyclists.)
Archiware P5 backup of a Thunderbolt SAN to a Thunderbolt Gamma Carry RAID
Note: In my tests I tested backup to a nearline RAID. I also like to use tape drives. LTO tape is another recommended option for backups or archives. Cloud or other offsite replication is also recommended if possible but is the slowest of all the options. Good to have slow and fast options, offsite and on premise, though any practical solution should be affordable and useful to help decision makers take the steps to preserve data and ultimately their own business.
LTO vs Cloud backup comparison: For LTO backups to one LTO7 drive I normally see 1TB in under 2 hours versus some recent cloud backups I did using rclone which took 9 hours for 1TB. Remember: restore times will equal your backup times. Want to restore 100TB? Got a spare 900 hours? 38 days for cloud restore vs 8 days with one LTO7 drive (much faster if you have more than one drive). Even faster if you restore from a Thunderbolt RAID. Only 2.5 days. Think about it.
Thunderbolt Xsan in a box. I’ve written about the Accusys T-share in 2020 (and in 2015 when I first found this cool tech). What’s different now? New year, new macOS. And a new challenge: can we build Xsan only using Terminal? No apps. It’s the journey that counts, right? One nerd’s journey to make an Xsan with macOS 11 Big Sur cli. Destination adventure with family fun, next stop a blinking cursor on a command line prompt.
make Xsan
make —Xsan —-bigger
reboot
Sudo make me an Xsan sandwich. I wish it were that easy! Stick around for the two or three commands you do need.
Xsan goes Terminal
Important commands for using Xsan have always been cvadmin and cvlabel (cv is short for centravision the original creators) but more recently xsanctl and slapconfig are important for creating the SAN and the OD (Open Directory) environment. Read the man pages, search the web, read some help documents. This blog is for entertainment and occasional learnings.
Lots of interesting cv (CentraVision) and sn (StorNext) commands in macOS (this list is from 10.15 Catalina). Besides binaries, what else is there? Examples. A ton of example files:
If you don’t have a fibre channel switch and fibre channel hardware RAIDs do not worry. You can build a useful Thunderbolt based Xsan with a little bit of effort. Just a little bit of peril It’s not too perilous, don’t worry.
Apple includes Xsan for free in macOS. Xsan is Apple’s fork Quantum’s StorNext SAN software. Want large fast storage made for Final Cut Pro editors, just add Xsan. Download Server.app from the Mac App Store and make your Xsan. Easy peasey. Right?
Why? Why are we doing this? Nothing beats fibre channel or Thunderbolt SAN speed for editing. Network attached storage (NAS) at 1GbE is barely usable. NAS at 10GbE is much better but still has road blocks for editors. Fibre channel or Thunderbolt with a big enough raid behind your SAN then life is great. Xsan can be shared by a small or media sized team of editors, producers and assistants.
Oh, ok. There is one problem. Apple did a major upgrade of Xsan (now version 7!) in macOS 11 Big Sur but apparently they took out the Xsan config in Server.app. (Note: This is what I was told early on and what seemed to be confirmed by Apple’s recent Xsan cli guide. It turns out that Xsan’s disappearance in Server.app to not be totally correct). Xsan is there in Server.app if you upgrade to macOS Big Sur but when you install Server on a clean macOS there is no Xsan visible in the app. Hmm. What do we do? Apple published a very nice handy guide about how to build Xsan in Terminal. So let’s get started. This is fun.
Accusys T-Share is a Thunderbolt SAN. Connect Macs with Thunderbolt cable.
What do we need? 1) Hardware raid. Ok check I have an Accusys T-Share. It’s a raid with Thunderbolt switch built in. 2) Mac. Ok I have a Mac Mini. 3) A network. Some cables, a switch and a DNS server. Ok I have a new raspberry Pi. That’s perfect.
Raspberry Pi 400 (the amazing linux computer shaped like a keyboard).
Step 1. Hardware raid. With the Accusys T-Share I just have to plug in some clients with a Thunderbolt 3 cable. Let’s fill the RAID with drives. I picked two different sizes. One group of larger disks for a data LUN (main production storage) and two smaller disks for a raid mirror to be used as metadata storage.
Step 2. A Mac running macOS Big Sur 11.5.2. Download the Accusys Mac installer on your Intel Mac (M1 is not supported with the T-Share yet as of this blog post).
Step 3. The network. Ok. This is the fun part. Let’s set up a DNS server. Ok, how do we do that? Remember that raspberry Pi you bought yourself for Christmas but never opened because you have been so busy and well you know life. Ok just me? Well, that one. Let’s use a raspberry Pi. A small inexpensive Linux computer. Install dns masq. It’s perfect for this.
The raid. Not only a great movie it’s the central part of this production media network for creatives. Once the drives are in the raid we have to make raid sets which become LUNs for Xsan. RAID5/6 for the data LUN and RAID1 (mirror) for the metadata LUN.
Read the label. Using Xsan cvlabel
Normally after we create RAID sets in the hardware raid utility we would open up Server.app and label the LUNs for Xsan use. But since we are now hardcore SAN architects we can use Terminal and the cvlabel the command to do this the hard way. Well, it’s not that hard but it can be intimidating the first few times. It’s much easier to label new LUNs than stare at a broken production SAN that has lost its labels. StorNext fun times. More about in another blog post.
Whether using Server.app in the good old days or cvlabel to label your LUNs now you should all be familiar with the command to list available LUNs. For larger SANs that won’t mount the first thing I’d check is see if the LUNs are all there. You don’t want a SAN to mount if it’s missing an important piece of itself.
cvlabel -l
This command lists available LUNs. It’s handy to know. Do this before trouble arises and you will be a cool dude when trouble happens. It does that occasionally. Prepare for the worst, hope for the best, IT motto.
To create labels for newly created RAID arrays use cvlabel to output a text file of the unlabelled LUNs, make some minor changes then label those LUNs. Create the template files first:
cvlabel -c
Edit the file. I like nano. Maybe you like vim. Or BBEdit. Or text edit. Change the name of LUNs from CVFS_unknown to whatever you like. I like to name LUNs based on the hardware they originate from so that I can find them, remove them, fix them or whatever I need to do for troubleshooting. Trust me. It’s a good idea.
cvlabel ~/Desktop/cvlabel
*WARNING* This program will over-write volume labels on the devices specified in the file "/Users/xavier/Desktop/cvlabel". After execution, the devices will only be usable by the Xsan. You will have to re-partition the devices to use them on a different file system.
Do you want to proceed? (Y / N) ->
Requesting disk rescan .
Congratulations this is the hardest part. You’ve labeled the RAID arrays as usable LUNs for Xsan. Ok, just kidding that’s not the hardest part. Have you ever heard of Open Directory? Do you fear LDAP and DNS? Well, maybe you should. It’s always DNS. Just saying.
DNS (domain name system) is just a fancy word for a list of IP addresses and host names. Using the raspberry Pi with dns masq installed we can populate the list of hosts for the Xsan and then we are golden. Hopefully if we did it right. Turns out we can make mistakes here too. Don’t use “.local” domain names. I did. It was late. I blame being tired. Changing them to “.lan” worked better.
Next up we finally create an Xsan in terminal. Or do we? let’s check the hostname first. It’s always DNS.
scutil —get HostName
CrazyMac.local
scutil --set HostName XsanMac.lan.
And now we make very big Xsan using the Xsan guide example
It was at this point that it started falling apart. It was late. I had messed up my DNS with “.local” and the Xsan wouldn’t go past this basic OD setup. I did what I always do and reach out to my Xsan colleagues and I got some curious feedback. “What do you mean Xsan isn’t in macOS Big Sur Server.app?” Hmm. I don’t see it on a fresh install. On an upgrade from 10.15 Catalina I do. So, uh, Where is it? And then it was revealed. In the View menu. Advanced. Ugh. It’s right there. Almost staring right at me. When I opened the app it said it couldn’t create an Xsan with my “.local”. That was helpful. Fixed that and Xsan with my pre-labeled LUNs was super quick to set up.
Xsan configuration in Server.app. “Ignore ownership” is the best thing ever for creatives. Trust me,
I’ll have to play with the cli set up again soon. Because there were some strange formatting it recommended to me when I tried some variations of the xsanctl createSan. I’ll dig into another day when I have more sleep. Ha ha.
There’s a lot of useful commands in macOS Big Sur Xsan which was upgraded to v7. You can check which version of Xsan you have in macOS with the cvversions command.
In Catalina (macOS 10.15.7)
File System Server: Server Revision 5.3.1 Build 589[63493] Branch Head BuildId D Built for Darwin 19.0 x86_64 Created on Tue Jun 22 21:08:03 PDT 2021 Built in /AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/XsanFS/XsanFS-630.120.1/buildinfo
In Big Sur (macOS 11.5.2)
File System Server: Server Revision 7.0.1 Build 589[96634] Branch Head BuildId D Built for Darwin 20.0 x86_64 Created on Wed Jun 23 00:32:35 PDT 2021 Built in /System/Volumes/Data/SWE/macOS/BuildRoots/d7e177bcf5/Library/Caches/com.apple.xbs/Sources/XsanFS/XsanFS-678.120.3/buildinfo
There’s a lot of cool new binaries in Xsan v7. We will dig into those next post. For now enjoy this and go forth make some Xsan volumes with Thunderbolt or fibre channel storage. It’s fun.
I am so happy to install macOS Big Sur 11.5.1, now that it is a ready for production. Have fun with macOS Monterey those of you on the bleeding edge. For media professionals using Xsan in production storage environments August is a great month to update to the soon to be yesterday’s bad boy Mr. Big Sur.
Server.app v.5.10 in macOS Catalina 10.15.7
Upgrading to a new major version of macOS can be fraught with peril for a fleet of mac devices but it is potentially fatal for a production SAN environment. That is why we wait. We want a nice stable storage system for our Final Cut Pro editors and other media creatives so it is safe to be one version behind. Less drama that way. We prefer our dramas to be on AppleTV+
Watch TV Upgrade Xsan
It is not boring to watch AppleTV+ while upgrading Xsan
The Xsan upgrade to Big Sur was pretty much not exciting except for one funny roadblock that I had set up myself last as a kind of booby trap for “future me”. More about that later. First the boring stuff. The last few weeks have been very busy updating and re-writing documentation in Pages.app and running multiple redundant full and incremental LTO backups with Archiware P5, syncing to nearline archives, and archiving finalized projects to the LTO shelf in paradise (sounds more exciting when you put it that way don’t you think?). Updating and re-writing documentation can sound like a waste of time but “future you” will appreciate what “past you” was doing today. And today I had fun updating Xsan to macOS Big Sur. Now I must write down all my thoughts before I each too much vegan vanilla ice cream and slip into a food coma.
“Planning for disasters, while hoping for none” is the IT mantra. We planned hard and we were ready to restore Xsan from Time Machine, if we had to. Not a joke. The server is backed up by Time Machine. The data is backed up to LTO, nearline archives racked and stacked in a server room and on redundant thunderbolt RAIDs which are parked on electric trucks ready to blast off at the earliest sign of danger. Well, everything except for the last part. Would be nice. And cloud backups for those clients that want them. Plan for the worst, pay for what you can to keep your business operational and lessen the impact of mechanical failures, human oopsies, or ransomware. Sysadmins are indistinguishable from malware sometimes, but we mean well. More seriously, humans makes mistakes and break things (that, me!) but ransomware is real and my elaborate backup and archive planning has saved a few customers this year.
Xsan volumes are typically made of up fibre channel RAID arrays. Nice icon!
Preparation is key. Be prepared. Get ready. Psych yourself up. I used Greg Neagle’s installinstallmacos.py to download macOS Big Sur as a disk image and had that and the App Store’s Server.app downloaded beforehand and not be dependent on internet access (production SANs are not always internet accessible). It is both true and not true that you can setup Xsan in Big Sur with the Server.app. It is true you need the Server.app for an upgrade from macOS Catalina 10.15.7 but if you’re starting from scratch in macOS 11 you will be building your Xsan in Terminal. Have fun! (We will cover this in a future post).
Download macOS Big Sur and the Server.app. Keep old copies zipped up. Cvlabel is nice too
Server.app manages only three (3) services for an Xsan upgrade: Profile Manager, Open Directory and Xsan. In macOS Big Sur new setups of Server.app Xsan is gone. Why they haven’t taken out Profile Manager and not kept Xsan instead made me scratch my head. No one in their right mind is using Profile Manager to install or manage profiles, they’re using commercial MDM vendors. But Xsan in macOS Big Sur (11) is not only production ready storage SAN awesome it has been upgraded to be compatible with Quantum’s Stornext 7 (previously it was only v.5)
Profile Manager does not belong here. Long Live Xsan!!
Installing macOS 11 Big Sur and upgrading Xsan to v7 is compatible (in my testing) with macOS 10.14 Mojave, 10.15 Catalina and of course macOS 11 Big Sur. If you don’t believe me check out this not updated in forever Apple’s compatibility chart.
Ok, by this time you get the idea I’m an expert, right? I’m ready to upgrade. But I run into my first real road block. And I have only myself to blame. I can’t launch the macOS Big Sur install app. It is blocked. “Contact your administrator”?! I am the sysadmin. Oh, ok. That’s me. What have I done now? I installed Hannes Juutilainen’s Big Sur Blocker last year, that’s what.
Of course I installed that. With Munki. On all my Mac clients that were upgraded to macOS Catalina. (And of course my Xsan controller has Munki!). But no worries, let me read up on my last year’s blogpost about it to figure out how I installed it, there must be a launch daemon or something.
this is not how I expected it to go
Hmm, no didn’t mention there. And where is that pesky launch daemon that I can unload and get to this Big Sur install. Oh? It’s a launch agent. Unloaded. Hmm, still no. Ok, delete the app from /usr/local/bin, hmm, nope. ok kill the app process. Ok, now we can install macOS Big Sur. Sorry for the delay. I had told Munki to uninstall the bigsurblocker app and it did for every other Mac, I swear, really. It did.
Please proceed with the macOS Big Sur install
So ready for macOS Big Sur. Oh wait, we noticed that you’re running Server.app and well, we don’t do a lot of the same things anymore in the new Server.app so maybe this is a warning.
Warning. We noticed that you’re running Server.app and we don’t do those fun things anymore.
So a lot of progress bars and stuff. See my last upgrade blog post and it’s the same as installing macOS Big Sur on any Mac, except this Mac Mini is running an Xsan production SAN environment with a lot of RAID arrays in a server rack or two. Ok, yeah, just run the installer.
We noticed that Server app is no longer server app.
After macOS Big Sur is installed zip up your older server.app and drag in your new one (or use that fancy App Store app to do it for you if you’re lazy). Click a bunch of buttons (see all my old blog posts) and launch the new Server.app.
Profile Manager is updating. No one cares.
So we have to wait while the bag of scripts that is Profile Manager gets updated but no one uses it but it’s the most important app in Server.app now, no I am not bitter why do you ask. Xsan is awesome.
Time to restore from your old Xsan configuration. Wheee…..
Xsan restore configuration.
Activate your Xsan and carry on upgrading all your Mac clients. Note: I did test macOS Mojave 10.14, macOS 10.15 Catalina and of course macOS 11.5.1 Big Sur Xsan clients. All worked.
Xsan on. Power up.
Upgrading Xsan with macOS Big Sur is easy if you’re going from macOS Catalina. Starting from scratch is another story to be covered in another blog post. Also not covered is certificate issues from self-signed certs breaking when I upgraded my Munki and MunkiReport server. That’s definitely another blog post. It’s just a webserver. Just. A. Web. Server. What is so hard? haha
Technical Errata:
With more than one Xsan controller it used to be recommended to upgrade the secondary before the primary but it is now best practise to upgrade the primary first to maintain the sanity of the OD data.
Xsan Upgrade Step by Step:
Clone the controllers. (+ Time Machine backups) Turn off the clients. Stop the Xsan Volume. Run cvfsck on the volume. **Upgrade the primary. Confirm the secondary can see the primary. *Upgrade the secondary. Confirm the secondary can see the primary. Check SAN access on both controllers.
I finally got a chance to open up my Christmas present to myself a few months late. I was excited all the same because it was a Raspberry Pi 400.
This is an incredible form factor for the raspberry Pi. It’s an all in one unit that fits into a keyboard. The keyboard is the computer. Just incredible.
Once you plug in power, the mouse and an HDMI monitor you have a working Linux computer running Raspbian. What to do next? Install Tailscale of course !!
Tailscale is a mesh VPN and allows you to link up all your devices in a private network no matter where they are. I’ve blogged about it here. So far I’ve linked up macOS, iOS, windows, Linux centos as well as Synology and QNAP NAS so now let’s add a raspberry Pi.
First things first let’s update the raspberry Pi because it’s been sitting in its box for a while. I didn’t have much luck with the add / remove software gui app (maybe because it was still getting on wifi) but after getting on the network I fired up apt to update all the things.
sudo apt list --upgradeable
Using apt you can get a list of what is upgradeable. This was my first step the gui app didn’t list any software that had updates. A few months in a box and there should be a lot of updates. This is Linux we are talking about.
sudo apt update
Then it is just a matter of upgrading everything.
The next step is to install Tailscale. Read the instructions for Raspberry Pi on the Tailscale website. Before installing Tailscale we have to install other needed components and we have to tell Linux where to find the software. Similar to my adventures with CentOS Linux and yum you have to tell Raspbian Linux what to do with apt.
First we fetch the signing key and tell it where the repo is. Note: Always be mindful when using curl to download and install items or scripts.
The next step is to authorize Tailscale which is usually done with a login to our account we created the Tailscale network with. But this time I wanted to try a pregenerated keys from the Tailscale admin panel.
sudo tailscale up --authkey tskey-gh37374737292a4847382
Now to test the new secure Tailscale mesh VPN set up I wanted to login to my MunkiReport web server.
Using the Tailscale cli to list all devices on my private network I found the IP.
tailscale status
Find the ip of the server or device from the status list and connect.
Next I outputted the raspberry Pi cli history to a file and sent it to my iPhone where I edited this blog post with the WordPress app.
history > raspberry-history.txt
tailscale file cp raspberry-history.txt iPhone:
I am looking forward to working on many fun projects with my raspberry Pi. After Tailscale is on there then it is easy to connect from my iPhone or from anywhere Tailscale is installed.
In the example below I used ssh shellfish app on my iPhone to connect.
Tailscale has great documentation and despite this I had a small misstep when I went to look at the install instructions. It defaulted to Ubuntu or misdetected my raspbian linux so I wondered for a second why it wasn’t working. I soon realized I’d copy pasted the wrong thing. Checked the drop down install menu for the raspberry Pi instructions and voila happy times.
How to use Tailscale (wireguard based) mesh VPN to connect everything
What is Tailscale? It’s a mesh VPN based on the wireguard open source project. It’s a secure network to connect your own devices no matter where they are.
Tailscale is free to use with one account and up to 100 devices, which is enough to see how well this can work to connect up servers, storage and desktops. They have paid plans for teams and enterprise.
Tailscale macOS app icon
macOS and iOS
To start, download Tailscale on your Mac or iPhone then find your IP address. Once you are signed in and have your IP address you can connect easily between devices. For example, on your iPhone open the Tailscale app and see your installed devices. Click on your Mac and the IP will be copied into the clip board. Use this to connect with app such as Secure shellfish for SSH or VNC viewer for remote login. When you’re in the same network it’s impressive, but when you’re on a different network, separated far away, It’s magic.
The real test for me was to install Tailscale on some backup servers I manage to make it more secure and more convenient to access them. I had used a variety of remote control for business services and well, Tailscale is easier, quicker and much more awesome. All the other software I tried was much less awesome.
After using Tailscale mainly for remote control, I tested Tailscale to securely connect my remote Macs to my own MunkiReport server. I use Munki and MunkiReport to manage Macs and having Tailscale allows me to securely connect endpoints to the server without opening up ports on my router. MunkiReport allows me to detect malware (with DetectX plugin) or check on backup jobs with Archiware P5 backup software (using a module I wrote) or a multitude of other diagnostics such as disk space free, apps installed, and all kinds of great hardware and software metrics. So much reporting. And MunkiReport doesn’t need Munki specifically, so if Tailscale is installed for remote control why not report on everything else.
DSM Package Center: Tailscale (and Archiware P5) app on Synology NAS
Synology DSM
Having Tailscale installed in all the Synology NAS I manage in various physical locations allows me to securely connect to all of these NAS from anywhere. With remote work using a NAS is a great way to sync data between locations. Synology has a lot of great built-in tools to make this happen and a very robust quick connect feature combined with ddns, and let’s encrypt certificates to support it. After setting up a few to sync one location to another I was constantly getting notifications of IPs being blocked on my firewall. I had to open a port on my firewall to let in the ssh / rsync traffic through and despite a strong set of firewall rules with a geo block there were still connection failures and password attempts. Using Tailscale I can now have a P5 server set up on one Synology NAS connecting to the Tailscale IP of other remote units and it can easily backup, sync or archive the data from the various locations.
To install the Synology Tailscale package check out this GitHub page. Download the app then side load it (manual install in package center). To enable it you will have to have ssh on, a user with permission to use it, and one command to type.
sudo tailscale up
In one case I didn’t have SSH enabled on the remote unit so I remoted into a Mac on the same network, enabled an admin user, turned on ssh with a time limit on the account, and then logged in. Once the above command is run you will get a link to a website to authenticate the device with your account.
Linux (CentOS)
I have also installed and tested Tailscale on a Linux (CentOS) storage server. In my case a Jellyfish which has a ZFS volume shared over direct 10GbE for Final Cut Pro video editors using nfs or smb. the Jellyfish works well on premise, but wouldn’t it be nice to capture camera cards to the remote storage server via Tailscale? Oh yes it would. And what about playing back some of the video files via VLC on your iPhone! Or Files.app! Yes, to all the above. All made possible with Tailscale. And a huge shout out for their great documentation. Installing Tailscale on CentOS was super simple. Add a yum repo, install, tailscale, and then bring the service up. Couldn’t be easier.
A small, but very exciting, feature was added part way through my testing of Tailscale which made it infinitely more awesome, shared devices. The concept is you are authenticated to your devices and can see in the Tailscale app all the IPs to connect to, but what if you could share one device (computer, server, NAS) with another person? Well, now you can. In the Tailscale admin console choose a device and send a share link to someone, they then will see this devices in their device list as shared. Home users can set up Tailscale to access all their own devices, but now can also choose to share access with a device in particular. For example, if you create an account, open a service (file sharing) and send a share link then the other person will login with the account you create and access the one thing you want them to. Maybe it’s a smb share to drop files. Works great for video collaboration and other kinds of teams.
There’s a whole lot more you can do with Tailscale (and wireguard) mesh VPN but I hope this gives you all some ideas to start with.
It’s 2021 and what is the state of Thunderbolt Shared Storage?
Thunderbolt Shared Storage is a RAID which you connect to with Thunderbolt (and Ethernet) which can be shared with other workstations. It’s a Thunderbolt SAN. Shared by Thunderbolt.
I’ve long been a fan of Apple’s Xsan and other SAN products that use Fibre Channel (or iSCSI) to connect clients to super fast block storage. It acts like a fast direct attached RAID but you can share it with others. The sharing part is crucial to collaborative workflows. We used XSAN when I was in VFX and I now use XSAN for post production workflows. Editors like to edit, have large video files, and often work in teams. All those camera files aren’t getting any smaller. So you need a lot of storage that can be shared by a team of editors, colourists, motion graphic artists, producers, etc
What you don’t often want is a complicated network infrastructure or a server room with fibre channel switches and fibre channel RAIDs and assorted other equipment. You don’t want that. You’d have to call me and pay me to set up your storage, backup and archive workflow. While it is always recommended to work with a trusted contractor it can be expensive. For small teams a shared Thunderbolt storage SAN can be quick to set up, doesn’t take much room and can easily connect 4-8 editors. Thunderbolt 3 passive copper cables extend up to 2m but optical Thunderbolt cables are available up to 60m.
I want to review one such example of shared Thunderbolt storage that stands out, the Symply Workspace. It’s a RAID, but it’s a SAN too. It is storage you plug into with Thunderbolt, but with an extra ethernet cable you make it a SAN. Inside the storage it has Quantum StorNext which runs most of the world’s expensive enterprise SANs, but it’s in the Symply Workspace and it works with Apple’s Xsan client software (included free with macOS since macOS Lion 10.7 !!). So basically, it is enterprise storage in a Thunderbolt box. One more thing, add a 5 seat license to a simple but powerful media asset management (MAM) tool axle.ai to organize your assets, add keywords and access from anywhere. Almost too good to be true, so I had to test it and see.
I received a Symply Workspace to test with. I like testing storage. My clients always need more storage. I keep telling them to not fill it up, but they do. That’s why we have LTO, nearline and cloud archive with Archiware P5 but that’s another story. For now, let’s test this storage: how easy is it to setup, how awesome is it to edit with, and what is a MAM good for anyway? Let’s find out.
It starts with a box. You open the box. You take the drives out of the box. Two trays of drives. Then there’s the RAID itself. Put the drives in the RAID. No screwdriver or tools necessary, just slide and click. Next step read all documentation online, ha ha, just kidding, don’t read anything keep going. Ok, kidding a bit. There’s an info card in the box with a website link to help explain the setup. But I didn’t read it all, uh, I know Xsan, I can do this, ok, what’s the link? now back to building.
The tricky part is plugging in a few cables, which will be improved in the shipping production version with stenciled labels of where to plug in what. There’s three cables after the power cable. One Thunderbolt to a Mac, that’s easy. One ethernet to your local network, no problem. And lastly one more ethernet to your new SAN production network (aka metadata). Ok, what’s that? Add a 5-port switch and plug in all clients and the storage to this network for SAN private metadata. It really is a SAN. Like Apple’s Xsan or Quantum StorNext, you need a data pipe (40GbE Thunderbolt 3 in this case) to transfer the data, and a metadata network, to talk about the data (1GbE ethernet to our extra switch). The public network will be used to talk to Axle MAM or for re-sharing out the volume to non-Thunderbolt connected Macs.
Once the Symply Workspace is wired up and powered on you’ll be able to reach the unit via a local bonjour name in your web browser (http://symplyworkspace.local:8088) and from there a simple web interface allows you to monitor the status of the unit, download drivers to configure a Mac or Windows client, restart or shutdown, contact support or start troubleshooting if needed. It’s a great tool to do the few things you need to.
So with everyone working remotely how do we connect to our shared storage? I’m glad you asked. There are a lot of good options. My favourite is Tailscale a mesh VPN (based on the open source Wireguard project) which you would install on a locally connected Mac then you can remote in and share a screen or the storage. There’s also the Axle 5-seat license included with the Symply Workspace which can scan your storage, make proxies and serve it all via a website which can also be accessed from anywhere.
To be continued…
Update:Added a clarification that there is an included 5 seat license to Axle media asset management in the Symply Workspace
In media production environments you work with high speed and high capacity storage. It can be network based NAS, fibre channel SAN or Thunderbolt DAS. There’s always some backup RAIDs, individual source footage drives, file servers and even network appliances.
Glossary of Storage:
SAN – Storage Area Network (typically with Fibre Channel, also with Thunderbolt or iSCSI)
NAS – Network Attached Storage (popular vendors include Synology and QNAP)
DAS – Direct Attached Storage (hardware or software raid directly attached to a client or server)
LTO – Linear Tape Open (tape standard for backups/archives. Current gen8 holds 12TB per tape)
Cloud – Other people’s servers and storage. Hosted in a data center (AWS, GCP, Azure, & more)
The variety of file types is astounding: original camera footage, Final Cut Pro projects, stock footage purchased, b-roll shot from other projects, sound effects, music etc. How do you find anything? Files can be in many places, across many different kinds of storage. The question is how do you know where they are?
In the before times…
Before the great pandemic of 2020 I only heard one complaint with finding files on Apple Xsan storage: “Why does spotlight not work on my Mac?” Searching the Xsan volumes was hit and miss. When Spotlight worked it was fast and immediate. And when it didn’t, well…. Not so much. To help finding files we started using one called EasyFind from DevonTechnologies. It was free, and easy to use, but it was not fast enough for ad hoc video searches. Nor could it search across all storage areas at once. A new year, and new solutions required.
EasyFind. Freeware for finding files. Easy to use but not fast.
One major issue with EasyFind was not being able to refine the search easily for audio and video assets. It seems skewed toward developers and while it allows you to include or exclude some file types globally in settings but it does not allow you to refine the results while searching. This makes it difficult to find what you need when you need it.
Searching for sound effects in EasyFind reveals “.h” source files. Only one file is relevant in these results.
It’s 2021 and a few things have changed over the last year: you may have a lot more places to search (SAN and NAS) and more importantly everyone is working remotely. It has become a lot more challenging to see your files, let alone search through all the storage locations and find what you need. FoxTrot Search to the rescue.
FoxTrot Search Pro. Not freeware but worth every penny. Indexes all locations so that search is fast!!
Editors and other creatives need something that will search through all their different storage places and quickly tell them where a certain file might be. It might be stock footage, a drone shot, sound effect or an old logo. But where exactly is it? What projects was it used in? What was it called?
For those who like organizing their files into logical folders, then the filesystem is your friend. Using The Finder has been the way to find things. But now there’s a lot of folders and a lot of file types in those folders. Choosing what to index in these folders help enormously. You can exclude certain subfolders on a designated storage or kinds of files that are not needed. Do not index mail or chat messages or even source code files if you know the files you want are movies and audio files. Narrow the scope where you can. Find files fast.
Choose what to index in FoxTrot Search and what not to by kind (file type) or ignore by subfolder.
You might think I’d suggest a full enterprise Media Asset Management (MAM) system at this point. And well, in the old days I would have. Apple’s old Final Cut Server worked perfectly well with classic Final Cut Pro (v6 and v7) but required rigid workflows and ingest habits. Same issue with a lot of more expensive bespoke enterprise systems. They catalog assets but at a great cost. Some clients avoid these for the cost, and others for the workflow restrictions. I’ll talk about some newer options later but for now editors just want to find the files in their own folder structure on their production storage. Is that too much to ask?
Using FoxTrot Search Pro and editor can easily search across multiple indicies (each index is a unique storage or separate folder location) than an admin has already set up and created. No waiting for indexing in real time, search now, with results instantly show in app. Narrow down and refine your search results easily. Don’t want to see Mail messages, or images, only audio files? Easy. Then further define only the file type you want. It works.
FoxTrot Search can refine searches by kind and extension to find the relevant files you actually want.
One of the recommendations to my clients with large storage (SAN or NAS) is to have secondary nearline archive as well as backups. Either a Thunderbolt RAID array like the Accusys Gamma Carry or a desktop QNAP or rack mounted Synology NAS. Using Synology is great for many reasons, for example, it includes a nice web login, and it can search for files. But to be completely honest, the built-in search only works sometimes. A common issue with search on the Synology is the corruption of the search index. It seems to happen some times for some storage locations, but all the time somewhere. Rebuilding this does not seem to fix it for long if at all. This was the major reason for switching to FoxTrot Search. I want something that works every time. Across all storage volumes: SAN, and NAS.
Synology NAS corrupt search index.
To be fair, I had to rebuild an index or two in FoxTrot Search as well, but it worked. Every. Single. Time. Rebuilding the Synology search indexes never seemed to fix the issues. With FoxTrot Search it’d warn me there was a lot (A LOT!) of items to index and it will take time but that’s why I’d do it over an evening and rebuild automatically them after hours or every weekend. FoxTrot Search would also warn me of problem files that took too long to index and could then be added to exclusion lists automatically.
When Spotlight stops working, your Synology search index is corrupted again or just want a great search tool for active storage locations, I recommend FoxTrot Search. It searches across all storage and provides access to the files where possible. And it’s fast. Really fast. Did I mention that yet?
FoxTrot Search Admin and Server apps
One word about the setup and the various FoxTrot Search apps. They have a personal edition for searching your local workstation and storage. The major difference with the Professional version is having multiple indices. They also have a server and a per seat license. I tested the Pro version with a few indices first and then set up the server to share these generated search index files with editors. I ended up making more indices after testing. Which so many storage locations I initially tried to make fewer indices but with so many thousands or in some cases millions of files it was better to make an index for each separate storage location (and per specific folder in some cases). It also helps when narrowing down a search, the editor can specify specific locations easily (as well specifying file type, language or file extension).
FoxTrot Search Server app. Define set port to connect to the server.
Once we tested FoxTrot on the local network we needed to make it work with the VPN. And now! The server version of FoxTrot Search allows you to set a fixed port for the server which we could open up on a firewall. In my initial testing I could connect to the server but not to each index of the various storage locations. This was frustrating. So close…. And I will admit here one criticism of FoxTrot Search is the documentation. It doesn’t really exist. They have a user forum and release notes but in my initial setup of the server I didn’t understand exactly what was needed. This could be solved with better documentation. I did resolve this issue with a few emails to the developers who explained to me that each index required its own port and therefore I needed to open more ports on the firewall. Ok, good to know. To confirm that these ports were or were not accessible in my testing I used “nc” in Terminal to scan open ports.
Scanning for specific open ports with nc binary. Port scanning is only one its many useful skills. Read the man pages for more exciting stuff it can do.
So FoxTrot Search is great for search across all active storage and is super helpful for seeing previews of video and audio files you may to use as an editor, but it can help find so much more. If you keep production documents, spreadsheets, PDFs, text files it will search through all them too. So your script or production run sheet is available to help you find what you need.
LTO (tape archives)
In my “where are my files” graphic at the top I show various storage locations including LTO and the cloud. The completed projects always go to LTO (tape) archives. How do we search those with FoxTrot Search? I use Archiware P5 which has a web server that is very easy to search with and restore any files via web login but to make things more fun why not have FoxTrot Search index the archive inventory? Of course it can. There’s a cli command in P5 to export the inventory of every file archived and this is a searchable tab separated (TSV) file. I’ve spent a lot of time with this because I’ve been working on a separate tool to analyse these inventory files (with sometimes millions of items in them) to see historical patterns and predict future trends. More on this specific tool later.
Xsan is archived to LTO (tape) with P5 and to the cloud. Postlab Drive with proxies (smaller version of original media) don’t need to be backed up but other creative production files do. The cycle continues.
The Cloud
New to many is the cloud. Proxies for editors in my recommended workflows get stored on a cloud drive like Postlab, FoxTrot Search can search that too. So many places for originals or backup copies to be. FoxTrot Search should be indexing all of them for you.
In a future blog post I will discuss new media asset management (MAM) systems and what’s changed over the years, but for now if you need a tool search across all active storage then take a look at FoxTrot Search.
DAM SAN 