A blog about archives, storage and everything

Tag: Terminal

  • Backup the Backups

    Tales of migrating an archive system

    Over the years I’ve trusted Archiware P5 to protect large storage systems by copying to tape archives. It really is the best system for my clients. It works. I’ve used it to restore entire failed storage. Backups are great. Restores are better!

    And Archiware have been great to entertain my feature requests and ideas for making P5 do what I wanted. Mostly extending the cli (and API) to help monitor the system with MunkiReport and a variety of shell scripts. (GitHub)

    software versus hardware

    Eventually after a long happy service you may find yourself needing to migrate the physical server to a new host when it is too old to get macOS updates.

    Yes, Linux VM(and virtual private servers or VPS) are more common today than « Mac servers » but back in the day when Apple ruled the VFX world (remember Shake?!) and everyone ran Xsan fibre channel storage on Xserve raids, we also ran Mac servers with Apple’s Xserve hardware. Well that’s no longer the state of the Mac universe but we have these lovely mac minis for those who build small post-production studios and a running a Mac “server” is easier than learning Windows or Linux or even docker OS which you have to maintain. (Mac mini DevOps, it’s a thing. Ask Alex!)

    Imagine physical servers on premise watching over and backing up on site storage using a tape library. Yes! It’s true, LTO tape systems to backup and archive post production storage is very standard. This is the way.

    So there you are, one day you find a Mac that’s a bit old and needs to be upgraded. Maybe you’re smart and careful so you know it needs macOS updates. Yes it’s working but how many years of not being recent enough to get security updates are you going to risk running a “a perfectly working system”?or maybe you want 10gbe or something faster. Upgrade all the things. And while buying a new Mac might be easy (not with all the ram you want maybe but still) now what? Software Migration time.

    Migration of the backup and archive server is easy right? Well easy enough. Move the databases over to the new Mac, make sure your license is correct for the new hardware and off you go.

    documentation for the nation

    Since I wanted to double check my fuzzy memory I checked with Archiware’s website and found a link in their support to my most excellent colleague David Fox’s JPY website with migration instructions. Excellent. Good start.

    In my case I wanted to tweak the instructions for mac server migration versus Linux and I wanted to call out making sure all the archive indices were backed up properly so I revised his excellent list to be more specific to my scenario.

    Warning: this is the point when you realize that I’m going to mention another app I built. The inflection point in the blog. No, it was wasn’t vibe coded. It was seriously hand crafted and mostly artisanal. In this case, my P5 Export app has existed for a while and it emerged as a way to easily schedule SQL export and analysis of historical archive jobs.

    I had a shell script and a bundle of sql statements. And it worked but I wanted an easy way for my clients to use it so that’s where that app started. Well, even before that I had a lot of TSV exported inventory files and no real idea of what was in them. So there was a whole adventure with Jupyter notebooks and Python and that was fun and cool and worked briefly but it was so bespoke that it could not be used anywhere.

    why? What are we doing this for?

    My sql tools transformed the data into trends and reporting to see what and how much was archived. It was great. But it was a shell script and could only be wielded by the magician. So then it became an app. And a menu bar app too. That was my favourite part. An app. In the menu bar. It’s awesome.

    more more more

    Eventually I added my other export scripts which extracted useful info from my P5 archives: how many tapes I had and what were they, and what was in each one. Useful scripts made their way into my app. Report all the things.

    All this info is in Archiware P5 but not always the way you want to see it

    Recently Archiware have added dashboards to P5 and it’s a great addition and it hopefully means that I won’t have to support any hacks using bash and sql for much longer.

    Ultimately my app feature update to P5 Export today came down to a planned migration and some cli instructions for backing up the backup and archive databases. Why do it once manually when you can automate and make it work for everyone better. It’s a classic IT mantra. So of course it made perfect sense to include backups of the backups in my app too.

    why not regularly backup and export useful data intelligence from your archives. Make them work for you. You need P5 Export if you use Archiware P5, and you should, then check out the rest of my P5 helper tools. For making the most of your archives. Code.matx.ca

  • Community Projects: Part 1

    SetDefaultAppsX – A Community-Driven Evolution

    From Enterprise Lock-In to Universal macOS Tool

    When Scott Kendall released his SetDefaultApps script in December 2025, it solved a real problem: giving users a friendly GUI with swiftDialog to set their default applications using scriptingOSX’s utiluti for file types, URLs, and protocols on macOS. It worked beautifully—but only if you had Jamf Pro.

    That’s where the community stepped in.

    The MDM-specific Problem

    Scott’s original script was tightly coupled to Jamf Pro’s infrastructure. It relied on policy triggers for installing dependencies:


    jamf policy -trigger install_SwiftDialog
    jamf policy -trigger install_SymFiles
    jamf policy -trigger install_utiluti

    For enterprise Mac administrators already using Jamf, this was perfect. For everyone else—small businesses, education labs, home users, or shops using different MDM solutions—it was a non-starter.

    The X Factor: SetDefaultAppsX

    I took Scott’s excellent foundation and worked to make it a truly standalone tool. The “X” represents both the removal of dependencies and the cross-platform (MDM-agnostic) nature of the new version.

    Major Transformations

    1. Self-Contained Installation

    Instead of calling out to Jamf policies, SetDefaultAppsX now downloads swiftDialog directly from GitHub, verifies the package signature against Bart Reardon’s Team ID, and installs it automatically:


    expectedDialogTeamID="PWA5E9TQ59"
    LOCATION=$(curl -s https://api.github.com/repos/bartreardon/swiftDialog/releases/latest | awk -F '"' '/browser_download_url/ {print $4}')
    curl -L "$LOCATION" -o /tmp/swiftDialog.pkg
    # Verify signature before installation
    teamID=$(/usr/sbin/spctl -a -vv -t install "/tmp/swiftDialog.pkg" 2>&1 | awk '/origin=/ {print $NF}' | tr -d '()')

    No MDM required. No manual downloads. Just works.

    (Real time update: Writing this blog post made me add a feature to do the same with ScriptingOSX’s utiluti. Now the script checks and downloads both.)

    2. Hardware Detection That Actually Works

    Scott’s original used system_profiler, which sounds reasonable—until you run it in certain contexts where it returns “Unknown” due to , uh, issues. System_profiler cli is powerful, but it can time out and not always return the values I wanted. It could also be some script shenanigans but in any case.

    For my scripting needs the fix was simple but crucial: switch to sysctl queries that always work:


    # CPU Detection - Direct kernel query
    MAC_CPU=$(/usr/sbin/sysctl -n machdep.cpu.brand_string 2>/dev/null)

    # RAM Detection – Also via kernel
    MAC_RAM=$(/usr/sbin/sysctl -n hw.memsize 2>/dev/null | awk ‘{printf “%.0f GB”, $1/1024/1024/1024}’)

    The result? Instead of seeing “Unknown” or generic “chip_type”, I would see “Apple M3” on my MacBook Air or the full Intel CPU model for an Intel Mac. And it’s 20-30x faster.

    3. No Sudo Required

    The original required administrators to create directories in `/Library/Application Support` before users could run the script. Either management tools created this beforehand or users ran a script with sudo which was not ideal or workable for non-admins. So, SetDefaultAppsX includes automatic fallback:


    if [[ ! -d "${SUPPORT_DIR}" ]]; then
    echo "WARNING: Application Support directory not found"
    echo "Falling back to /Users/Shared/SetDefaultAppsX"
    SUPPORT_DIR="/Users/Shared/SetDefaultAppsX"
    # Automatically create writable directories
    /bin/mkdir -p "${SUPPORT_DIR}"
    fi

    Users can run the script immediately without any preparation. For enterprise deployments, there’s an optional `PrepareSetDefaultAppsX.sh` that sets up system-wide directories, but it’s truly optional.

    4. Modern Icon System

    Again, in my hacking of Scott’s perfectly working script, I ran into some required banner images and so instead of relying on file system icon resources that might not exist, SetDefaultAppsX uses SF Symbols:


    OVERLAY_ICON="SF=xmark.circle,weight=medium,colour1=#000000,colour2=#ffffff"

    Always available, always renders perfectly, and customizable.

    The X2 Portable Edition: Platypus-Ready

    But I didn’t stop there. I had an idea that users could run an app easily so SetDefaultAppsX2 takes portability even further—it’s designed for packaging as a standalone application using Platypus.

    The key difference is local binary detection:


    # Get the directory where the script is located
    SCRIPT_DIR="${0:a:h}"

    # Check for binaries in script directory first
    if [[ -x “${SCRIPT_DIR}/utiluti” ]]; then
    UTI_COMMAND=”${SCRIPT_DIR}/utiluti”
    elif [[ -x “/usr/local/bin/utiluti” ]]; then
    UTI_COMMAND=”/usr/local/bin/utiluti”
    fi

    This means you can package the script along with the `dialog` and `utiluti` binaries into a single app bundle with Platypus. Users double-click the app, and everything just works—no installation, no command line, no dependencies.

    Perfect for:

    • Quick distribution to non-technical users
    • Testing environments
    • Portable USB installations
    • Labs where users can’t install software system-wide

    Community Contributions Flow Both Ways

    The best part? Scott has been incorporating some of these improvements back into his Jamf-specific version. The hardware detection fixes and error handling enhancements benefit both the enterprise and standalone versions.

    This is open-source collaboration at its finest: Scott provided the excellent foundation and deep integration expertise, the community contributed cross-platform portability, and both versions improve together.

    The Technical Wins

    Let’s talk numbers:

    • Performance: 3-4x faster startup (sysctl vs system_profiler)
    • Reliability: 100% success rate for hardware detection (up from ~60%)
    • Portability: Works on any Mac, any MDM, or no MDM
    • Security: Package signature verification via Team ID
    • User Experience: No sudo required, automatic fallback, clear error messages

    What You Get

    Three versions for different needs:

    1. SetDefaultApps.sh – Scott’s original Jamf-integrated version
    2. SetDefaultAppsX.sh – MDM-agnostic standalone version
    3. SetDefaultAppsX2.sh – Portable version ready for Platypus packaging

    All three share the same excellent user interface powered by swiftDialog, the same UTI handling via utiluti, and the same goal: make setting default apps friendly and accessible.

    Getting Started

    The simplest possible workflow:

    # 1. Run the script
    ./SetDefaultAppsX.sh

    That’s it. The script downloads swiftDialog if needed, creates directories automatically, and presents users with a beautiful interface to set their default apps.

    For Platypus app building with X2:

    – Include dialog and utiluti binaries in your app bundle
    – Point Platypus to SetDefaultAppsX2.sh
    – Users get a double-clickable app with zero dependencies

    Credits Where Due

    – **Scott Kendall**: Original script author, Jamf integration expert
    – **Bart Reardon**: swiftDialog creator (the UI magic behind it all)
    – **scriptingOSX**: utiluti tool for UTI management
    – **The Community**: Testing, feedback, and collaborative improvements

    The Open Source Philosophy

    This is what makes the Mac admin community special. Scott could have kept his script locked down or enterprise-only. Instead, he shared it, accepted community modifications, and even pulled improvements back into his version.

    The result? Better tools for everyone—whether you’re managing 10,000 Macs with Jamf or helping your family set up their MacBooks.

    What’s Next?

    The scripts are stable and production-ready, but there’s always room for improvement:

    – Auto-installation of utiluti from GitHub releases (Done!)
    – Built-in default banner images
    – Dark mode support
    – Multi-language interface
    – Configuration file support for organizations

    But the foundation is solid: a truly portable, MDM-agnostic tool for one of macOS’s most user-requested features.

    —

    Try it yourself: The full source code, documentation, and evolution guide are available in the project repository. Whether you need the Jamf version, the standalone version, or the portable Platypus version, there’s a SetDefaultApps that fits your workflow.

    Because good tools should be accessible to everyone, not just those with enterprise MDM budgets.

    —

    *Special thanks to Scott Kendall for creating the original script and being open to community contributions, ScriptingOSX (Armin Briegel) for utiliti and to Bart Reardon for swiftDialog —the best thing to happen to Mac admin UIs in years. *

  • Don’t stop!

    Using a REST API to code a bunch of useful apps

    Background: I released some scripts to help users manage Archiware P5 servers on code.matx.ca with a first blog post as a background to the scripting approaches of cli vs REST API and then I discussed my cute Platypus built apps and my first swift/swiftUI Mac apps… so of course now I’m going to discuss two new Mac apps that use a REST API to explore a P5 Archive.

    I’m building tools to help me and my clients work with Archiware’s amazing and awesome P5 Archive product. It’s great. It archives, it restores, it has a great web UI, and it’s always getting better. So why build apps? Sometimes you want something different, in my case my clients wanted spreadsheets. Yup. Data in a sheet. To look at. So I poked at P5’s databases via cli (see P5 Archive Manager) and with the REST API. Here are the results, two new apps P5 Archive Overview and P5 Archive Search.

    https://code.matx.ca/ is code on GitHub + Mac apps that help manage data in Archiware P5

    Why API? And not cli

    Usually the cli (command line interface) is perfect for working in Terminal and in shell scripts or other programming languages. Using an API or application programming interface, allows different software applications to communicate and share data with each other. Instead of cli commands and arguments programs make requests using specific methods like GET or POST to retrieve or send data. See: API on Wikipedia

    Using an API for my Mac apps means they could use a protocol like HTTP, and make a request using GET (to retrieve data).

    The magic parts of an API

    • API Client: The application making the request.
    • Endpoints: Specific URLs where the API can be accessed.

    For the P5 Archive Overview app we need to use the specific API Endpoint for archive overview detailed by Archiware here which lucky for us is a simple call for data which our app can display, save as json, format as csv (for the spreadsheet!!) and stash in a SQLite database for historical searches.

    However, the P5 Archive Search app has to make many calls to walk through the index tree which is an inventory by path of the files archived. So we ask the user to name the storage they want to search and we make a breadcrumb through the storage, storing everything in SQLite as well as saving json and csv snippets of what we find. A lot more API calls but perfect for an app to do in the background.

    The P5 Archive Search app queries the P5 Archive Index Inventory REST API:

    “`

    GET http://{server}:{port}/rest/v1/archive/indexes/{archive-index}/inventory/{path}

    “`

    Example:

    “`

    http://192.168.1.100:8000/rest/v1/archive/indexes/Default-Archive/inventory/Volumes/BigStorageSMB

    Made for Macs, macOS 14 and up. Bring your own jq

    When running macOS 15 and up jq is installed for free and can help make the csv files form the json, but if you’re running macOs 14 then you need to install it with homebrew, MacPorts, or on your own.

    One funny story during the early testing, I had to fix the jq detection in my first version because it totally missed reading the unix path correctly and yeah, thanks to a friend on the MacAdmins Slack who had an issue when trying it out. I was able to go back and fix that. Friends help friends make better code.

    I’m not done making apps, and I’ll keep tweaking these three I have so far and making new ones from the scripts I have. The goal is to manage data, get a better understanding of the data in the archives and let the clients and owners of the data to know what they have.

    Stay tuned.

  • Add header here

    Or remove it, up to you,

    Had some fun creating a longer script to add a text header to some shell scripts, then because I wrote the wrong thing to all my shell scripts I had some more fun tweaking my script to find and remove this header. I’ve added it to my GitHub repo with a couple of other scripts based on the find command, one of my favourite unix tools since it is so handy.

    The script that should be a Unix one-liner: add (or remove) a header

    Some of the other example scripts based on find might be of interest to some, such as the

    File Name Check

    Especially important with certain filesystems (certain encrypted filesystems) with file name “length limits”. So why not check for these files and zip them up and put them aside for safe keeping. In practise, the only files which push this limit are downloaded (purchased) from stock photos sites and write the file name with every keyword. Nice, but why can’t we have standard metadata handling these days? (I can dream!)

    Archiware P5

    The last two scripts I made with Archiware P5 in mind, as I manage many servers for clients with P5 Archive and I really do love this software and the team. More Archiware P5 inspired scripts are in other repos here or on my main P5 code site

    Find A Trailing Space

    In this case, besides it just being nice to clean up folder names with invisible trailing spaces before the archive job it was also necessary when using the P5 Companion (desktop) app which will not archive a top-level folder with a space at the end of the name.

    Make (Only) One Thumbnail

    This script makes only one image thumbnail per RDC folder, as they normally have a lot of R3D files which are part of the same shot. Also, I don’t want a lot of duplicates and only one is enough.

    And while yes technically P5 Archive can make thumbnails and proxy videos when it is archiving (and I do use this feature) making proxies of RED files is an intensive process for older computers which means taking a long time, so pre-processing these R3D files ahead of time on faster computers can make the final archive job quicker. As part of some pre-processing before archiving to LTO (or wherever) is making sure some formats like R3D (aka RED) files have a thumbnail which will then end up in the P5 Archive created Archive index.

  • Steal This Idea

    check all your Macs at once with SOFA feed

    Note: this blog post relates to the previous one where I introduce the scripts to check SimpleMDM devices and compare with latest version info in the SOFA feed here: Use the SOFA feed to check if SimpleMDM devices needs updates

    Ok, please steal this idea. The idea? To check all your Macs at one time, instead of each device, on device, one at a time.

    What do I mean? Well, when I first heard about the SOFA feed which contained all the latest versions I didn’t know what to do with it honestly but soon after I realized that my clever script for checking XProtect version and which I made into a custom attribute in SimpleMDM and added to the dashboard was an incomplete idea.

    Ok, I’m smart, I got the XProtect version on each Mac by running a script and then I got SimpleMDM to display it in a dashboard. But what’s missing? Context. Is it the latest version or not? So I added a SOFA check to the script then made SimpleMDM display both the local version and the latest version so I’d know if it was the latest or not. Great, right? Well, maybe.

    The problem, I realized is that I wanted to do this for the macOS version too because I wanted to share info with a client/manager etc and realized the list of devices and info about macOS versions for example, lacked the context of whether it was the latest, and should we take action or not. That’s the point, right? collect info then do something about it, if action is required. Update your macOS now.

    And then I wondered why I’m getting every Mac to ask itself what is its macOS or XProtect version, etc, when SimpleMDM was asking a lot of those questions already and putting it in a dashboard, accessible via API….

    Then it happened, the idea that should be stolen by SimpleMDM and all other management tools. Don’t just display info about a Mac’s macOS version, show the latest version next to it, because I want to know if it should be updated. And also what is the latest that Mac can upgrade to. Maybe it’s running macOS 13.6, is that the latest or is 13.7.7, no wait it changed again, it’s 13.7.8. And by the way the latest compatible upgrade is 15.6.1, now that’s useful info.

    product_nameos_versionlatest_major_osneeds_updatelatest_compatible_oslatest_compatible_os_version
    Mac13,114.7.414.7.8yesSequoia 1515.6.1
    MacBookPro17,114.6.114.7.8yesSequoia 1515.6.1
    Mac13,215.615.6.1yesSequoia 1515.6.1
    iMac21,115.515.6.1yesSequoia 1515.6.1
    MacBookPro17,113.613.7.8yesSequoia 1515.6.1

    References:

    Check SimpleMDM device list and compare macOS version vs SOFA feed latest

    XProtect check version compared to latest SOFA

  • Use the SOFA feed to check if SimpleMDM devices needs updates

    I wrote a “simple” bash script to check SimpleMDM device list by API and check if any devices need updates and/or are compatible with the latest macOS. Of course, it will output some CSVs for fun and profit. Send to clients, managers, security professionals and be well.

    Note: It was a quick hack and for reasons I made 3 output CSVs for testing various presentations of the data that combines the full SimpleMDM device list and matches the macOS with available updates and max supported versions. There may be errors or omissions. Please test. Use and modify. I know I will. This is a test. Just a test.

    The script is in my GitHub repo

    Fetching SimpleMDM device list...
Downloading SOFA feed...
✅ Exported:
  → Full device CSV: /Users/Shared/simplemdm_devices_full_2025-07-30.csv
  → Outdated devices CSV: /Users/Shared/simplemdm_devices_needing_update_2025-07-30.csv
  → Supported macOS per model: /Users/Shared/simplemdm_supported_macos_models_2025-07-30.csv
✅ Export complete.

    References:

    SOFA MacAdmins Getting Started

    https://sofa.macadmins.io/getting-started.html

    https://github.com/macadmins/sofa/tree/main/tool-scripts

    SimpleMDM API docs

    https://api.simplemdm.com/v1#retrieve-one-dep-device

    squirke1977 / simpleMDM_API

    https://github.com/squirke1977/simpleMDM_API/blob/master/device_details.py

  • Firewall ch-ch-changes in macOS 15 Sequoia

    Knock knock

    “Who’s there?”

    macOS 15 Sequoia. Check your firewall checking scripts please

    If anyone is following along with my attempt to re-create MunkiReport in SimpleMDM then you’ll be happy to know the space madness is still strong and macOS 15 has made one tiny thing break, my firewall checking script.

    My firewall checking script began life as a simple check of the status in the alf pref file but that file no longer exists in macOS 15.

    See this Knowledge base article which lists in bug fixes that the file no longer exists and that the socketfilterfw binary be used instead, except that doesn’t work when Macs are managed.

    Application Firewall settings are no longer contained in a property list. If your app or workflow relies on changing Application Firewall settings by modifying /Library/Preferences/com.apple.alf.plist, then you need to make changes to use the socketfilterfw command line tool instead.

    Yes, my Macs are managed with MDM and yes I have a profile to enable the firewall but no I don’t trust it so can I check please with another method. Trust but verify.

    So thanks to some friends in the MacAdmins Slack I stole the idea from tuxudo to check firewall in macOS 15 using system profiler, because he had re-written the MunkiReport module already and so there I go again, stealing from MunkiReport and all the hard work they do.

    After playing with the output of system_profiler a bit I looked at the “Mode” 

    /usr/sbin/system_profiler SPFirewallDataType -detailLevel basic |grep Mode 
      Mode: Allow all incoming connections
      Stealth Mode: No

    Of course I could write some nice code to clean this up or instead I switched to searching for “Limit” and if there’s no hit on that there’s no limit (translated: firewall is not enabled“)

    /usr/sbin/system_profiler SPFirewallDataType -detailLevel basic |grep Limit

    And if there is a limit then the firewall is enabled. 

    Mode: Limit incoming connections to specific services and applications

    Simple. Good enough to add to my SimpleMDM script to run and populate the value to the custom attribute and update my dashboard. And my crazy mission to build everything into SimpleMDM dashboard is still… madness …. but also quite fun.

  • Backup Fast, Restore Quicker

    Backing up is nice, restoring is better. Slow backups, mean slow restores. Make good decisions, and backup only the files you want to keep to the fastest storage you have.

    When working with a fast fibre channel or Thunderbolt SAN your first choice for fastest backup destinations is a Thunderbolt RAID. I recommend to have this onsite with an off site LTO and/or cloud disaster recovery setup (a replicated SAN or shared storage system is nice to have too).

    A built-in option to copy Xsan files is cvcp (cv stands for centravision).

    cvcp -vxy /Volumes/TSAN/folder /Volumes/GammaRAID/backups

    cvcp is fast. Really fast. And cli commands are scriptable. A very smart person (Jasper Siegers) wrote a script called cvcpSync which combined the power of rsync and cvcp. It was awesome. But there are limits to the best of scripts. For my clients I use Archiware P5 with large SAN and other shared storage to simplify the number of things which need to be monitored. One dashboard to monitor tape or cloud backups, tape archives and sync to nearline RAIDs or NAS.

    With a recently Thunderbolt SAN deployment with Accusys T-Share I set up the Accusys Gamma Carry as a backup destination. I set up Archiware P5 to do the backup. It was fast. How fast? Over 1Gb/s. Fast backups are also fast restores. With the Gamma Carry I can run a backup then carry it off site. It’s an option as part of a complete backup strategy.

    Archiware P5 backup 1.6TB in 53 minutes

    (Luckily I have almost 2 TB of video from my Cycliq bike cameras to test backups. Sadly, after my last bike vs car incident I felt obliged to buy bike cameras for my safety. I edit small fun rides when I can. Sometimes traffic near-accidents too. Please be kind, don’t kill cyclists.)

    Archiware P5 backup of a Thunderbolt SAN to a Thunderbolt Gamma Carry RAID

    Note: In my tests I tested backup to a nearline RAID. I also like to use tape drives. LTO tape is another recommended option for backups or archives. Cloud or other offsite replication is also recommended if possible but is the slowest of all the options. Good to have slow and fast options, offsite and on premise, though any practical solution should be affordable and useful to help decision makers take the steps to preserve data and ultimately their own business.

    LTO vs Cloud backup comparison: For LTO backups to one LTO7 drive I normally see 1TB in under 2 hours versus some recent cloud backups I did using rclone which took 9 hours for 1TB. Remember: restore times will equal your backup times. Want to restore 100TB? Got a spare 900 hours? 38 days for cloud restore vs 8 days with one LTO7 drive (much faster if you have more than one drive). Even faster if you restore from a Thunderbolt RAID. Only 2.5 days. Think about it.

    Testing equipment:

    Hardware: T-Share SAN, Gamma Carry Thunderbolt RAID

    Software: Archiware P5

  • Do you want to build a Thunder SAN?

    Thunderbolt Xsan in a box. I’ve written about the Accusys T-share in 2020 (and in 2015 when I first found this cool tech). What’s different now? New year, new macOS. And a new challenge: can we build Xsan only using Terminal? No apps. It’s the journey that counts, right? One nerd’s journey to make an Xsan with macOS 11 Big Sur cli. Destination adventure with family fun, next stop a blinking cursor on a command line prompt.

    make Xsan

    make —Xsan —-bigger

    reboot

    Sudo make me an Xsan sandwich. I wish it were that easy! Stick around for the two or three commands you do need.

    Xsan goes Terminal

    Important commands for using Xsan have always been cvadmin and cvlabel (cv is short for centravision the original creators) but more recently xsanctl and slapconfig are important for creating the SAN and the OD (Open Directory) environment. Read the man pages, search the web, read some help documents. This blog is for entertainment and occasional learnings.

    Xsan Commands: where are they?

      /System/Library/Filesystems/acfs.fs/Contents/bin
    	cvlabel			sncfgremove
    cvaffinity		cvmkdir			sncfgtemplate
    cvcp			
    cvmkfile		sncfgtransform
    cvdb			
    cvmkfs			sncfgvalidate
    cvdbset			cvupdatefs		sndiskmove
    cverror			cvversions		snfsdefrag
    cvfsck			fsm			snlatency
    cvfsck_compat		fsmpm			snlicense
    cvfsdb			has_snfs_label		snprodalert35chk
    cvfsid			mount_acfs		snquota
    cvgather		sncfgconvert		wingather
    cvgather_fsm		sncfgedit		xsanctl
    cvgather_multipath	sncfginstall		xsand
    cvgather_sum		sncfgquery		xsandaily

    Lots of interesting cv (CentraVision) and sn (StorNext) commands in macOS (this list is from 10.15 Catalina). Besides binaries, what else is there? Examples. A ton of example files:

    /System/Library/Filesystems/acfs.fs/Contents/examples/

    cvlabels.example                           fsnameservers.example                      rasexec.example                            
    cvpaths.example                            fsports.example                            rvio.example                               
    fsmlist.example                            nss_cctl.example                           snfs_metadata_network_filter.json.example

    Just the facts. The Xsan basics

    If you don’t have a fibre channel switch and fibre channel hardware RAIDs do not worry. You can build a useful Thunderbolt based Xsan with a little bit of effort. Just a little bit of peril It’s not too perilous, don’t worry.

    Apple includes Xsan for free in macOS. Xsan is Apple’s fork Quantum’s StorNext SAN software. Want large fast storage made for Final Cut Pro editors, just add Xsan. Download Server.app from the Mac App Store and make your Xsan. Easy peasey. Right?

    Why? Why are we doing this? Nothing beats fibre channel or Thunderbolt SAN speed for editing. Network attached storage (NAS) at 1GbE is barely usable. NAS at 10GbE is much better but still has road blocks for editors. Fibre channel or Thunderbolt with a big enough raid behind your SAN then life is great. Xsan can be shared by a small or media sized team of editors, producers and assistants.

    Oh, ok. There is one problem. Apple did a major upgrade of Xsan (now version 7!) in macOS 11 Big Sur but apparently they took out the Xsan config in Server.app. (Note: This is what I was told early on and what seemed to be confirmed by Apple’s recent Xsan cli guide. It turns out that Xsan’s disappearance in Server.app to not be totally correct). Xsan is there in Server.app if you upgrade to macOS Big Sur but when you install Server on a clean macOS there is no Xsan visible in the app. Hmm. What do we do? Apple published a very nice handy guide about how to build Xsan in Terminal. So let’s get started. This is fun.

    Accusys T-Share is a Thunderbolt SAN. Connect Macs with Thunderbolt cable.

    What do we need? 1) Hardware raid. Ok check I have an Accusys T-Share. It’s a raid with Thunderbolt switch built in. 2) Mac. Ok I have a Mac Mini. 3) A network. Some cables, a switch and a DNS server. Ok I have a new raspberry Pi. That’s perfect.

    Raspberry Pi 400 (the amazing linux computer shaped like a keyboard).

    Step 1. Hardware raid. With the Accusys T-Share I just have to plug in some clients with a Thunderbolt 3 cable. Let’s fill the RAID with drives. I picked two different sizes. One group of larger disks for a data LUN (main production storage) and two smaller disks for a raid mirror to be used as metadata storage.

    Step 2. A Mac running macOS Big Sur 11.5.2. Download the Accusys Mac installer on your Intel Mac (M1 is not supported with the T-Share yet as of this blog post).

    Step 3. The network. Ok. This is the fun part. Let’s set up a DNS server. Ok, how do we do that? Remember that raspberry Pi you bought yourself for Christmas but never opened because you have been so busy and well you know life. Ok just me? Well, that one. Let’s use a raspberry Pi. A small inexpensive Linux computer. Install dns masq. It’s perfect for this.

    The raid. Not only a great movie it’s the central part of this production media network for creatives. Once the drives are in the raid we have to make raid sets which become LUNs for Xsan. RAID5/6 for the data LUN and RAID1 (mirror) for the metadata LUN.

    Read the label. Using Xsan cvlabel

    Normally after we create RAID sets in the hardware raid utility we would open up Server.app and label the LUNs for Xsan use. But since we are now hardcore SAN architects we can use Terminal and the cvlabel the command to do this the hard way. Well, it’s not that hard but it can be intimidating the first few times. It’s much easier to label new LUNs than stare at a broken production SAN that has lost its labels. StorNext fun times. More about in another blog post.

    Whether using Server.app in the good old days or cvlabel to label your LUNs now you should all be familiar with the command to list available LUNs. For larger SANs that won’t mount the first thing I’d check is see if the LUNs are all there. You don’t want a SAN to mount if it’s missing an important piece of itself. 

    cvlabel -l

    This command lists available LUNs. It’s handy to know. Do this before trouble arises and you will be a cool dude when trouble happens. It does that occasionally. Prepare for the worst, hope for the best, IT motto.

    To create labels for newly created RAID arrays use cvlabel to output a text file of the unlabelled LUNs, make some minor changes then label those LUNs. Create the template files first:

    cvlabel -c

    Edit the file. I like nano. Maybe you like vim. Or BBEdit. Or text edit. Change the name of LUNs from CVFS_unknown to whatever you like. I like to name LUNs based on the hardware they originate from so that I can find them, remove them, fix them or whatever I need to do for troubleshooting. Trust me. It’s a good idea. 

    cvlabel ~/Desktop/cvlabel
    *WARNING* This program will over-write volume labels on the devices specified in the file "/Users/xavier/Desktop/cvlabel". After execution, the devices will only be usable by the Xsan. You will have to re-partition the devices to use them on a different file system.
    Do you want to proceed? (Y / N) ->
    Requesting disk rescan .

    Congratulations this is the hardest part. You’ve labeled the RAID arrays as usable LUNs for Xsan. Ok, just kidding that’s not the hardest part. Have you ever heard of Open Directory? Do you fear LDAP and DNS? Well, maybe you should. It’s always DNS. Just saying.

    DNS (domain name system) is just a fancy word for a list of IP addresses and host names. Using the raspberry Pi with dns masq installed we can populate the list of hosts for the Xsan and then we are golden. Hopefully if we did it right. Turns out we can make mistakes here too. Don’t use “.local” domain names. I did. It was late. I blame being tired. Changing them to “.lan” worked better.

    Next up we finally create an Xsan in terminal. Or do we? let’s check the hostname first. It’s always DNS. 

    scutil —get HostName 

    CrazyMac.local

    scutil --set HostName XsanMac.lan.

    And now we make very big Xsan using the Xsan guide example 

    xsanctl createSan 'VIDEOSAN' --account localadmin --pass 72DERjx1 --user localadmin --cert-auth-name videocert --cert-admin-email administrator@example.com

    It was at this point that it started falling apart. It was late. I had messed up my DNS with “.local” and the Xsan wouldn’t go past this basic OD setup. I did what I always do and reach out to my Xsan colleagues and I got some curious feedback. “What do you mean Xsan isn’t in macOS Big Sur Server.app?” Hmm. I don’t see it on a fresh install. On an upgrade from 10.15 Catalina I do. So, uh, Where is it? And then it was revealed. In the View menu. Advanced. Ugh. It’s right there. Almost staring right at me. When I opened the app it said it couldn’t create an Xsan with my “.local”. That was helpful. Fixed that and Xsan with my pre-labeled LUNs was super quick to set up.

    Xsan configuration in Server.app. “Ignore ownership” is the best thing ever for creatives. Trust me,

    I’ll have to play with the cli set up again soon. Because there were some strange formatting it recommended to me when I tried some variations of the xsanctl createSan. I’ll dig into another day when I have more sleep. Ha ha.

    There’s a lot of useful commands in macOS Big Sur Xsan which was upgraded to v7. You can check which version of Xsan you have in macOS with the cvversions command.

    In Catalina (macOS 10.15.7)

    File System Server:
      Server  Revision 5.3.1 Build 589[63493] Branch Head BuildId D
       Built for Darwin 19.0 x86_64
       Created on Tue Jun 22 21:08:03 PDT 2021
       Built in /AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/XsanFS/XsanFS-630.120.1/buildinfo

    In Big Sur (macOS 11.5.2)

    File System Server:
      Server  Revision 7.0.1 Build 589[96634] Branch Head BuildId D
       Built for Darwin 20.0 x86_64
       Created on Wed Jun 23 00:32:35 PDT 2021
       Built in /System/Volumes/Data/SWE/macOS/BuildRoots/d7e177bcf5/Library/Caches/com.apple.xbs/Sources/XsanFS/XsanFS-678.120.3/buildinfo

    There’s a lot of cool new binaries in Xsan v7. We will dig into those next post. For now enjoy this and go forth make some Xsan volumes with Thunderbolt or fibre channel storage. It’s fun.

  • Xsan Upgrade and Big Sur Prep. Hello Catalina!

    Big Sur summer testing time!

    Summer time is beta testing time. A new macOS beta cycle with Big Sur is upon us. Test early, and test often. With all the excitement of Big Sur in the air, it’s time to look at Catalina.

    Our day to day production Xsan systems do not run beta software, not even the latest version of macOS, they only run tested and safe versions of macOS. I always recommend being a revision behind the latest. Until now that meant macOS 10.14 (Mojave). With the imminent release of macOS Big Sur (is it 10.16 or macOS 11?) then it’s time to move from 10.14.6 Mojave to 10.15.6 Catalina. It must be safe now, right? 

    Background

    Xsan is Apple’s based Storage Area Network (SAN) software licensed from Quantum (see StorNext), and since macOS 10.7 aka Lion it has been included with macOS for free (it was $1,000 per client previously!).

    Ethernet vs Fibre Channel vs Thunderbolt

    A SAN is not the same as a NAS (Network attached storage) or DAS (direct attached storage). A NAS or other network based storage is often 10GbE and can be quite fast and capable. I will often use Synology NAS with 10GbE for a nearline archive (a second copy of tape archive) but can also use it as a primary storage with enough cache. Lumaforge’s Jellyfish is another example of network based storage.

    Xsan storage is usually fibre channel based and even old 4GB storage is fast because … fibre channel protocol (FCP) is fast and the data frames are sent in order unlike TCP. It is more common to see 8GB or 16Gb fibre channel storage these days (though 32GB is starting to appear). And while fibre channel is typically what you use for Xsan you can also use shared Thunderbolt based storage like the Accusys A16T3-Share. I have tested a Thunderbolt 2 version of this hardware with Xsan and it works very well. I’m hoping to test a newer Thunderbolt 3 version soon. Stay tuned.

    Xsan vs macOS Versions

    We’ve discussed all the things that the Xsan is not and now what is it? Xsan is often created from multiple fibre channel RAID storage units but the data is entirely dependent on the Xsan controller that creates the volume. The Xsan controller is typically a Mac Mini but can be any Mac with Server.app (from Apple’s App Store). The existence of any defined Xsan volumes depends on the sanity of its SAN metadata controllers. If the SAN controllers die and the configuration files go with it then your data is gone.  POOF! I’ve always said that Xsan is a shared hallucination, and all the dreamers should dream the same dream. To make sure of this we always recommend running the same version of macOS on the Mac clients as well as the servers (the Xsan controllers). And while the Xsan controllers should be the same or at a higher macOS version level it can sometimes be the opposite in practise. To be sure what versions of macOS are interoperable we can check with Apple’s Xsan controllers and clients compatibility chart and Xsan versions included in macOS for the rules and exceptions. Check the included version of Xsan on your Mac with the cvversions command

    File System Server:
  Server  Revision 5.3.1 Build 589[63493] Branch Head BuildId D
   Built for Darwin 17.0 x86_64
   Created on Sun Dec  1 19:58:57 PST 2019
   Built in /BuildRoot/Library/Caches/com.apple.xbs/Sources/XsanFS/XsanFS-613.50.3/buildinfo

    This is from a Mac running macOS 10.13

    Host OS Version:
 Darwin 17.7.0 Darwin Kernel Version 17.7.0: Sun Dec  1 19:19:56 PST 2019; root:xnu-4570.71.63~1/RELEASE_X86_64 x86_64

    We see similar results from a newer build below:

    File System Server:
  Server  Revision 5.3.1 Build 589[63493] Branch Head BuildId D
   Built for Darwin 19.0 x86_64
   Created on Sun Jul  5 02:42:52 PDT 2020
   Built in /AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/XsanFS/XsanFS-630.120.1/buildinfo

    This is from a Mac running macOS 10.15.

    Host OS Version:
 Darwin 19.6.0 Darwin Kernel Version 19.6.0: Sun Jul  5 00:43:10 PDT 2020; root:xnu-6153.141.1~9/RELEASE_X86_64 x86_64

    Which tells us that the same version of Xsan are included with macOS 10.13 and 10.15 (and indeed is the same from 10.12 to 10.15). So we have situations with Xsan controllers running 10.13 and clients running 10.14 are possible even though macOS versions are a mismatch, the Xsan versions are the same. There are other reasons for keeping things the macOS versions the same: troubleshooting, security, management tools, etc  To be safe check with Apple and other members of the Xsan community (on MacAdmins Slack).

    Backups are important

    Do not run Xsan or any kind of storage in production without backups. Do not do it. If your Xsan controllers die then your storage is gone. Early versions of Xsan (v1 especially) were unstable and the backups lesson can be a hard one to learn. All later versions of Xsan are much better but we still recommend backups if you like your data. Or your clients. (Clients are the people that make that data and pay your bills). I use Archiware P5 to make tape backups, tape archives, nearline copies as well as workstation backups. Archiware is a great company and P5 is a great product. It has saved my life (backups are boring, restores are awesome!).

    P5-Restore-FCPX.png

    Xsan Upgrade Preparation

    When you upgrade macOS it will warn you that you have Server.app installed and you might have problems. After the macOS upgrade you’ll need to download and install a new version of Server.app. In my recent upgrades from macOS 10.13 to macOS 10.15 via 10.14 detour I started with Server.app 5.6, then install 5.8 and finally version 5.10.

    After the macOS upgrade I would zip up the old Server.app application and put in place the new version which I had already downloaded elsewhere. Of course you get a warning about removing the Server app

     

    Xsan-ServerApp-ZipRemovalDetected.png

    Install the new Server app then really start your Xsan upgrade adventure.

    Serverapp-setup.png

    Restore your previous Xsan setup.

    If everything goes well then you have Xsan setup and working on macOS 10.15.6 Catalina

    Xsan-Catalina-Upgrade-Success