A blog about archives, storage and everything

Category: Tricks And Tips

  • Archiware P5 scripts

    Background: I manage a few backup and archive servers for clients and these run the Archiware P5 suite of software (archive, backup and sync). To help manage these servers over the years I’ve written some P5 monitoring tools for Watchman and MunkiReport as well worked on helper scripts using the cli tools or more recently the API.

    In an effort to share some simple examples of what is possible I have organized a few samples from my GitHub repos on the code.matx.ca page with some useful descriptions and text about usage and purpose of the scripts. They are hosted on in repo here:

    https://github.com/macvfx/Archiware

    I have more scripts of general and specific interest in my repo to P5 users or anyone managing files. See this post on find scripts.

    The P5 code toolbox

    The following P5 scripts are just a few examples and I have more to share, if there’s interest. I created most of these simple tools initially to run on the P5 server directly but I have since created, for my clients, versions which run from anywhere. Also, in some cases, a few scripts have now been built into easy to use Mac applications where it makes sense. If you want some help and you want to hire me to help with these things please reach out.

    The scripts are in three categories: 

    1) P5 archive intelligence (all archive jobs from Db exported as a spreadsheet, or get recent archive jobs via REST API),

    2) P5 housekeeping (make all full tapes read only, show all appendable archive tapes), and 

    3) P5 info backup (export all volumes into one csv, and export all volumes inventory as TSV with barcode as the name)

    P5 Archive Intelligence

    What do I mean by “archive intelligence”? Simply, I want to know about everything I’ve archived. One should consider the P5 Archive server as the ultimate source for all things archived but in some cases my clients don’t use the P5 server directly, or they want the information organized differently, like in a spreadsheet. And while the Archiware P5 suite of software is ever evolving, growing and adding features (even some lovely visual dashboards in v.7.4) I have been attempting to solve the perennial question of “what do we have archived?” in better and more useful ways.

    P5 Information Backup

    Related to archive intelligence is knowing what is in my P5 archive system entirely. I modified a provided shell script from the Archiware cli manual to output a csv of a list of all P5 volumes in the tape library (aka jukebox) so that I might know what is in the system at all times, and even if my P5 server is not running I have a record of every tape. This is one of the scripts I run with my periodic and backup workflows but more on my own special P5 backups (backups of the P5 Db and other metadata) in another post. The Archiware provided P5 volume list script inspired my own script to list full and appendable archive tapes which I have as a one-click desktop app for my clients. When they want to restore something P5 will tell them what to put in the tape library but if you have a lot of tapes maybe you want to know what to remove and so I have a list of candidates (ie take out the full tapes, and leave the appendable archive tapes). Helpful, yes.

    P5 Inventory

    There is a P5 cli command to export out the complete inventory and depending on how long you’ve been archiving and how much is in your archive this tool can take a long time to export a list of every file ever. And because of my mostly non-advanced super skills some times I’d find this process would time out. (There are ways around this but that’s another post). Basically, Too much archive! When it didn’t error out I had a big file… so one day a friend of mine suggested we use Jupyter notebook and yes, use Python!, to do some data analysis. A really fun project, great tool, but this is a hard problem to solve. We made a thing, it worked for a while then I wanted to find a better way. People liked my bar graphs and total amount archived but they also wanted spreadsheets. So let’s give them what they want.

    Two (or three) approaches:

    • cli
    • api
    • db

    I like lower case letters which are acronyms but let’s explore further.

    cli

    Using the cli (command line interface) usually in a shell script (but also in clever Mac apps) typically requires running the script with the Archiware P5 cli (nsdchat) locally on the P5 server and certainly this is what I did when I was testing scripts and various tools. It makes sense if you’re administering a server and you remote in (ssh or screenshare) and that’s where you start. After I wile I discovered the trick to make these cli based scripts run from anywhere which was handy if I wanted to connect to all my p5 servers at once in a script or have my client use an app on their desktop which talks to the P5 server. More on awsock in another post.

    An example of a cli script using nsdchat I have a script for taking the inventory contents of each archive tape (LTO) and writing its contents to a TSV file (tab separated values) which is like a CSV (comma separated values).

    nsdchat Volume names

    Give me a list of p5 volumes (ie tapes)and tell me which one are archive tapes and that are readonly and what is the barcode.

    api

    Ok instead of a cli command dependent on the nsdchat binary installed somewhere we are doing http (web) magic with the API (application programming interface) — a set of known commands in a path based on GET, PUT, POST, DELETE. The API has a different way of doing things than the cli but you can ask a lot of the same basic questions.

    In my api-archive-overview script I am sending one command then using jq to select elements to organize the info into a csv (spreadsheet). This example is set to run locally but this is easily modifiable to run from anywhere. For one client I have a script that talks to every P5 server, each in a different city and asks them all what they’ve been archiving then organizes it all into one spreadsheet. It’s fun, and it’s helpful.

    db

    The best for last. I mentioned above my attempts to use the inventory command which itself goes to all the relavant databases and gathers all the requested data about every archive file, its size and when it was archived etc. Yeah, that’s one way to do it. I’ve shown two examples above for the cli and api but a third is to just talk to the database directly. This is an advanced technique and should only be attempted by an expert. Ok, I’m kidding. As long as you’re not writing to the database and only reading from it, this is pretty safe. What you do with the info is another magic trick and one which I’ve been working on. Two db examples are dump all jobs in a csv file and the second, dump only archive jobs. I’ve got a more advanced script which takes the data and uses sql commands to organize into a csv of how much data archived per day per week per month per year and totals, which is nice for some people who like spreadsheets and want to know about everything ever done. Caution: once you look into the Db you’ll see a lot of things, and sorting through it takes time. I found when making more advanced and selective scripts that the cli jobs used by the very old P5 Archive app (by Andre Aulich) for example showed up as system jobs not archive jobs, so you have to be careful if you want to include those. Have fun.

    P5 Housekeeping

    Finally, some housekeeping scripts are included in the example repo, like the script to make all archive tapes that are “full” to be marked “read only” which is handy is you also have a script to only export the contents of each tapes from archive tapes marked readonly so many little scripts to do little things.

    P5 Archive Prep scripts

    There’s another category of scripts which I haven’t elaborated on but I do have a few examples in my repo. I do have scripts to prepare or examine files and folders going to be archived to LTO with P5 Archive. These scripts do various things like check for trailing space in the name or check file name length but maybe the most important ones I have are scripts which take the path of the archived projects and create html maps, file size directory listings and spreadsheets (again!) of the exif data of all files to keep for future. Clients do refer to the archive stub files (p5c) but they also find it handy to see the directory map and the file size of archived items without going into the p5 server. I’m not trying to replicate the P5 server, or replace it, but this falls into p5 housekeeping and p5 information backup.

    That’s enough for now. If you’ve been reading and following along then let me know if you any questions or want any help with a P5 or other related projects. If you have better ways to do these things feel free to share. My scripts are always evolving and I love to learn.

    Reference:

    For more info on Archiware P5 scripting and building code to interact with it I’d recommend checking out the main P5 manual, as well as the CLI (command line interface) manual, and the API documentation, knowledge base (support). As well as the sample scripts and the Archiware blog, and the video series generally.

  • Add header here

    Or remove it, up to you,

    Had some fun creating a longer script to add a text header to some shell scripts, then because I wrote the wrong thing to all my shell scripts I had some more fun tweaking my script to find and remove this header. I’ve added it to my GitHub repo with a couple of other scripts based on the find command, one of my favourite unix tools since it is so handy.

    The script that should be a Unix one-liner: add (or remove) a header

    Some of the other example scripts based on find might be of interest to some, such as the

    File Name Check

    Especially important with certain filesystems (certain encrypted filesystems) with file name “length limits”. So why not check for these files and zip them up and put them aside for safe keeping. In practise, the only files which push this limit are downloaded (purchased) from stock photos sites and write the file name with every keyword. Nice, but why can’t we have standard metadata handling these days? (I can dream!)

    Archiware P5

    The last two scripts I made with Archiware P5 in mind, as I manage many servers for clients with P5 Archive and I really do love this software and the team. More Archiware P5 inspired scripts are in other repos here or on my main P5 code site

    Find A Trailing Space

    In this case, besides it just being nice to clean up folder names with invisible trailing spaces before the archive job it was also necessary when using the P5 Companion (desktop) app which will not archive a top-level folder with a space at the end of the name.

    Make (Only) One Thumbnail

    This script makes only one image thumbnail per RDC folder, as they normally have a lot of R3D files which are part of the same shot. Also, I don’t want a lot of duplicates and only one is enough.

    And while yes technically P5 Archive can make thumbnails and proxy videos when it is archiving (and I do use this feature) making proxies of RED files is an intensive process for older computers which means taking a long time, so pre-processing these R3D files ahead of time on faster computers can make the final archive job quicker. As part of some pre-processing before archiving to LTO (or wherever) is making sure some formats like R3D (aka RED) files have a thumbnail which will then end up in the P5 Archive created Archive index.

  • Steal This Idea

    check all your Macs at once with SOFA feed

    Note: this blog post relates to the previous one where I introduce the scripts to check SimpleMDM devices and compare with latest version info in the SOFA feed here: Use the SOFA feed to check if SimpleMDM devices needs updates

    Ok, please steal this idea. The idea? To check all your Macs at one time, instead of each device, on device, one at a time.

    What do I mean? Well, when I first heard about the SOFA feed which contained all the latest versions I didn’t know what to do with it honestly but soon after I realized that my clever script for checking XProtect version and which I made into a custom attribute in SimpleMDM and added to the dashboard was an incomplete idea.

    Ok, I’m smart, I got the XProtect version on each Mac by running a script and then I got SimpleMDM to display it in a dashboard. But what’s missing? Context. Is it the latest version or not? So I added a SOFA check to the script then made SimpleMDM display both the local version and the latest version so I’d know if it was the latest or not. Great, right? Well, maybe.

    The problem, I realized is that I wanted to do this for the macOS version too because I wanted to share info with a client/manager etc and realized the list of devices and info about macOS versions for example, lacked the context of whether it was the latest, and should we take action or not. That’s the point, right? collect info then do something about it, if action is required. Update your macOS now.

    And then I wondered why I’m getting every Mac to ask itself what is its macOS or XProtect version, etc, when SimpleMDM was asking a lot of those questions already and putting it in a dashboard, accessible via API….

    Then it happened, the idea that should be stolen by SimpleMDM and all other management tools. Don’t just display info about a Mac’s macOS version, show the latest version next to it, because I want to know if it should be updated. And also what is the latest that Mac can upgrade to. Maybe it’s running macOS 13.6, is that the latest or is 13.7.7, no wait it changed again, it’s 13.7.8. And by the way the latest compatible upgrade is 15.6.1, now that’s useful info.

    product_nameos_versionlatest_major_osneeds_updatelatest_compatible_oslatest_compatible_os_version
    Mac13,114.7.414.7.8yesSequoia 1515.6.1
    MacBookPro17,114.6.114.7.8yesSequoia 1515.6.1
    Mac13,215.615.6.1yesSequoia 1515.6.1
    iMac21,115.515.6.1yesSequoia 1515.6.1
    MacBookPro17,113.613.7.8yesSequoia 1515.6.1

    References:

    Check SimpleMDM device list and compare macOS version vs SOFA feed latest

    XProtect check version compared to latest SOFA

  • Use the SOFA feed to check if SimpleMDM devices needs updates

    I wrote a “simple” bash script to check SimpleMDM device list by API and check if any devices need updates and/or are compatible with the latest macOS. Of course, it will output some CSVs for fun and profit. Send to clients, managers, security professionals and be well.

    Note: It was a quick hack and for reasons I made 3 output CSVs for testing various presentations of the data that combines the full SimpleMDM device list and matches the macOS with available updates and max supported versions. There may be errors or omissions. Please test. Use and modify. I know I will. This is a test. Just a test.

    The script is in my GitHub repo

    Fetching SimpleMDM device list...
Downloading SOFA feed...
✅ Exported:
  → Full device CSV: /Users/Shared/simplemdm_devices_full_2025-07-30.csv
  → Outdated devices CSV: /Users/Shared/simplemdm_devices_needing_update_2025-07-30.csv
  → Supported macOS per model: /Users/Shared/simplemdm_supported_macos_models_2025-07-30.csv
✅ Export complete.

    References:

    SOFA MacAdmins Getting Started

    https://sofa.macadmins.io/getting-started.html

    https://github.com/macadmins/sofa/tree/main/tool-scripts

    SimpleMDM API docs

    https://api.simplemdm.com/v1#retrieve-one-dep-device

    squirke1977 / simpleMDM_API

    https://github.com/squirke1977/simpleMDM_API/blob/master/device_details.py

  • Dynamic Groups – SimpleMDM tricks and tips part2

    When we last left our hero the big news was the discovery custom attributes and running scripts to test for certain conditions in SimpleMDM, like “is the firewall on” to post in the main dashboard was all the excitement, this year we present “dynamic groups” which in combination with custom attributes or by itself ups the game to the next level. Keep up!

    What if we wanted to know what is the current version of XProtect across the Mac fleet? and what if this wasn’t collected by default by MDM tool, in my case, SimpleMDM. Well, I can write a script to collect this info, for my purposes I’ve chosen to use silnite from Howard Oakley of eclectic light co fame and write the version number to a custom attribute. The next step is use SimpleMDM’s new dynamic groups (in preview, at the time of this blog post), and then I can watch the result filter in with a special group watching for “is matching this version” or the opposite “is not this version”. Just depends on what you want to act on or how you want to see the information. The new dynamic groups is the exciting part. I’m sooo excited.

    The custom attribute

    Screenshot

    Setting up a custom attribute of “XProtectV: and a default value of “Version Unknown” should be done before the script runs. If I get the default result then the script didn’t run or some other reason.

    The code

    #!/bin/bash
LOG_DIR="/Users/Shared"
DATE=$(date +"%Y-%m-%d_%H-%M-%S")
LOG_FILE="$LOG_DIR/silcheck-log-$DATE.txt"
/usr/local/bin/silnite aj > "/Users/Shared/silnite-xprotectv-$DATE.json"
XPROTECTV=$(/usr/bin/plutil -extract XProtectV raw "/Users/Shared/silnite-xprotectv-$DATE.json")
echo "$XPROTECTV" | tee -a "$LOG_FILE"

    The simple script writes a log into /Users/Shared just because I want to and uses the silnite binary to write out the XProtect info and plutil to extract the info from the json Note: you could also use jq in latest macOS 15 but this way is more compatible across macOS versions for now. The XProtect version is saved as an attribute which SimpleMDM picks up and reports back to base.

    The dynamic group

    Screenshot

    The filter headings are a little cut off in the screenshot but it basically says choose from all devices, refer to the custom attribute I set of XprotectV and makes sure the value equals the latest (at blog post writing) 5297 and further filter results for devices last seen in the last day. If I had switched it to the not equal to version 5297 I would see all the devices not up to date. And it’s easy to change on the fly. Easier than refreshing the main device dashboard page to see these results as I was trying to do previously and that method made it hard to further filter.

    The exciting part

    Yes the best part is to set up a job in SimpleMDM that runs the scripts on the devices to refresh the value of XProtect (I have it set to recurring as well) and then watch the results roll into a dynamic group which has its members populate as the scripts runs and results report back. Easey peasy.

    Screenshot

    Addendum:

    Adding an example screenshot to show how you can change the filter from matches an exact value of XProtect, in this example, to “not equal to” to see all the devices that haven’t upgraded yet. It’s as easy as changing the filter and clicking on “staging filter changes” button. Et voilà !

    Updated: May 16, 2025 – 19h00 local time

  • SimpleMDM tricks and tips – part 1

    Custom Attributes

    Custom Attributes in SimpleMDM are a way to assign values in a few different cases. I will show one use case, scripting, and one example: checking the firewall.

    Note: for more fun use cases see Steve Quirke’s blog post, or the talk “Making SimpleMDM complicated” by Lucas Hall at MacDevOps:YVR in 2021 or even the official documentation.

    The goal: Checking the firewall

    I wanted to see the status of the macOS firewall in the device view dashboard. That’s so simple, right? Well, I wanted to see it at a glance for every device, and not have to go into each device entry to see if the firewall was enabled.

    Write a script:

    #!/bin/bash

# Check firewall status
firewall=$(defaults read /Library/Preferences/com.apple.alf globalstate)

if [ "$firewall" = "1" ]; then
    echo "Firewall is enabled"
elif [ "$firewall" = "0" ]; then
    # Set firewall status
    #defaults write /Library/Preferences/com.apple.alf globalstate -int 0
    echo "Firewall is NOT enabled"
else
    echo "Unable to determine firewall status"
fi

    Note: This is my script. This seems to work. If you have other working examples let me know.

    Add it to SimpleMDM scripts

    Add your script to the scripts section. Check the “Enable attribute support” check box.

    Add a custom attribute

    Set up a custom attribute that your script will populate with its variable later. I set up one for the firewall.

    Create a job

    Your script will need to run (once, or scheduled) to populate the value into the variable and into the custom attribute. Choose what script runs where on what Macs. And choose the custom attribute.

    And choose the custom attribute.

    Note: The cancel job if not started is helpful if your devices are not responding. And is a premonition to issues you may have with this feature and might give some flashbacks to the ancient way of using scripts in ARD (Apple Remote Desktop) to try to make changes, back in the days before MDM or good configuration management tools ie. munki puppet chef salt etc

    Dashboard Devices

    Add your custom attribute to the viewable columns in the Devices dashboard and your life will be full of joy. Seeing at a glance your scripts output variable as a custom attribute.

    And now I just have to recreate everything in MunkiReport as a custom attribute and then I’ll be good.

    Script debugging.

    Running scripts is all well and good until your devices don’t check in and don’t run the scripts for whatever reason. Rebooting the Mac helps. Refreshing the inventory in SimpleMDM helps (maybe) and well, you’ll see it’s like the old ARD scripts run ad hoc and you’ll wish for better tools like fully functional DDM (declarative device management) which is like configuration management of the days of old. Incorporate MunkiReport and Fleet’s osquery tools and save me the trouble of doing piecemeal.

    Enjoy the script output in the custom attributes for now and send me your awesome ideas for what to script next.

  • Customizing MunkiReport: Dashboards

    I was chatting with Per Olofsson on a recent episode of the MacDevOps podcast about some recent fixes with relocatable Python he did for MunkiReport version 5.7.0 and I happened to mention how much I love my MR dashboards with custom hot keys. He is a long time user of MunkiReport but hadn’t heard that you could make custom dashboards and I couldn’t remember where I had heard of it or even how I made them. Pretty typical of tech these days I think. You learn something, you make something and then you move to the next thing and forget what you were doing or how you did it. Well, thanks to documentation we can share the knowledge and spread the love.

    Custom Dashboards

    The MunkiReport wiki actually has a short entry which explains how to make a custom Dashboards. Basically, add some YAML files in the dashboards folders and you’re done. Follow the Read Me file for instructions. Pro Tip: Use the Widget Gallery in MR to find useful pieces to build into your dashboards. Note: I’ve added these custom dashboards to my local folder which is set in my “.env” to be outside of the main munkireport folder so it easier to update across version upgrades.

    Here are four examples of MunkiReport dashboards:

    Security

    Munki

    Archiware P5

    The Archiware P5 dashboard references widgets from my custom P5 module. It’s easy to make modules for MunkiReport. Check the wiki for more info.

  • How To Securely Sync Your Synology NAS with P5

    Use Tailscale Mesh-VPN with P5 Backup and Sync

    In the old days we used to forward ports. On your router the traffic for a server or service went to a port (where a number represents a service, some which are defined, but can be arbitrary) and to a destination IP address. Well, wouldn’t you know it, if ssh is port 22 or web traffic is on port 80 then everyone and their port scanner comes knocking. So then your firewall is tested, and then auto-ban and geo-block and emails go out. What if we could avoid that and not open (or forward) any port to make services work across the internet?

    Tailscale is a mesh-VPN which uses WireGuard to securely establish a mesh (point to point) VPN of your devices. Suddenly your iPhone can securely send files to your Mac or raspberry Pi across the world. How cool is that? In today’s advanced lesson: you can backup and sync your Synology NAS using Archiware P5.

    Step 1: Setting up Tailscale on Synology

    It honestly used to be harder than this, these days you can simply add the Tailscale package via the Synology package center app and you’re done. Almost. There’s one more step.

    Step 2: Set up Outgoing VPN access via Tailscale which requires editing some files (which necessitate Terminal and remote login access). This only has to be done once but future updates may require fixes. This was tested in DSM 7. Pro tip: only allow remote access to a restricted and time limited account so you don’t leave it on accidentally.

    Step 3. Install Archiware P5 on Synology NAS

    Using Archiware P5 to Backup and Sync your NAS is a good thing if you’re already using Archiware P5 to backup and sync all the other things, then at least you have only one dashboard to look at. I use P5 with my clients to backup their shared storage to LTO and it makes sense to backup all the things no matter where they are with P5 also. With Synology NAS package center it’s a simple one-click install for P5. Add your P5 clients to your P5 server via Tailscale and you’ve got a secure setup.

    This post is just a quick overview of using Tailscale to set up your P5 clients (which is your Synology NAS in this case).

  • Munki makes MDM manageable

    How to deploy applications using munki and simplemdm

    You want to deploy apps to Macs but you also want to keep them up to date, fear not, we have a way. If you are using SimpleMDM for Mac management but hate the way MDMs deploy applications then listen up it’s easy(*) to set up Munki and use the power Autopkg to deploy and update all your apps. Note: SimpleMDM also offers a short list of curated apps to deploy without any extra setup but these instructions are for those who want to choose the apps they want to deploy. If that’s you then read on.

    Managed Software Centre is the AppStore for all your apps you want your Macs to have

    SimpleMDM: The basics

    Macs are enrolled into SimpleMDM, then assigned to Groups. Groups have profiles assigned to them to enforce and escrow FileVault or set other policies. Simple enough, right?

    Ok, what about apps?

    SimpleMDM Category setting for a Munki’s Managed Software Centre

    When you have apps in your Catalog you can assign a Munki category to the applications to make it show up in a nice group using Managed Software Centre (the client facing app).

    With Apps in your Catalog you can manage them with Assignment Groups which are created as Munki (or not-Munki aka Standard). Next select Managed or Self-Serve, two concepts which make sense to Munki admins. One set of apps is required and will be installed without asking, and the other group is presented to the end user to choose as needed (they’re optional).

    API key options. Allow Munki plugin access

    API key

    How do we get applications we want into SimpleMDM? Two ways exist. Import them manually and deploy via MDM or setup up Autopkg. For this we need the API key. Note: Only the munki plugin permissions are needed. Put the key into the Autopkgr.app SimpleMDM integration or set them as an environment variable and use autopkg in Terminal.

    Autopkgr app choose autopkg recipes to use

    Select recipes using Autopkgr (Linde Group) from the curated list of recipes created by IT Admins around the world or create your own recipes. What used to be a painstakingly difficult process by hand is now much easier with Recipe Robot by Elliot Jordan to help fish out the AppCast / Sparkle / Download URLs and transform into a nice autopkg recipe to be used by Munki (and ingested into SimpleMDM). 

    autopkg run -v Postlab.munki.recipe  -k MUNKI_REPO_PLUGIN="SimpleMDMRepo" -k MUNKI_REPO="" -k extract_icon=True
    MunkiImporter
Using API key provided by environment variable.
MunkiImporter: Using repo lib: MunkiLib
MunkiImporter:         plugin: SimpleMDMRepo
    Managed Software Centre notification

    Managed Software Centre

    Once Macs are enrolled and added to a SimpleMDM Group with the Munki assignment then the Managed Software Centre app will allow users to use the Self-Serve portal to install optional apps. Managed apps will install invisibly in the background.

    The beauty of this integration is that Munki is awesome and works well. It is battle tested by many companies and organizations around the world. Using autopkg and its recipes to check for updates allows for a seamless automation of new apps into your catalog and then onto your fleet. Updated Macs are happy Macs.

    Reference:

    SimpleMDM Munki integration blog post

  • Hello Big Sur! See ya later Monterey

    I am so happy to install macOS Big Sur 11.5.1, now that it is a ready for production. Have fun with macOS Monterey those of you on the bleeding edge. For media professionals using Xsan in production storage environments August is a great month to update to the soon to be yesterday’s bad boy Mr. Big Sur.

    Server.app v.5.10 in macOS Catalina 10.15.7

    Upgrading to a new major version of macOS can be fraught with peril for a fleet of mac devices but it is potentially fatal for a production SAN environment. That is why we wait. We want a nice stable storage system for our Final Cut Pro editors and other media creatives so it is safe to be one version behind. Less drama that way. We prefer our dramas to be on AppleTV+

    Watch TV Upgrade Xsan

    It is not boring to watch AppleTV+ while upgrading Xsan

    The Xsan upgrade to Big Sur was pretty much not exciting except for one funny roadblock that I had set up myself last as a kind of booby trap for “future me”. More about that later. First the boring stuff. The last few weeks have been very busy updating and re-writing documentation in Pages.app and running multiple redundant full and incremental LTO backups with Archiware P5, syncing to nearline archives, and archiving finalized projects to the LTO shelf in paradise (sounds more exciting when you put it that way don’t you think?). Updating and re-writing documentation can sound like a waste of time but “future you” will appreciate what “past you” was doing today. And today I had fun updating Xsan to macOS Big Sur. Now I must write down all my thoughts before I each too much vegan vanilla ice cream and slip into a food coma.

    “Planning for disasters, while hoping for none” is the IT mantra. We planned hard and we were ready to restore Xsan from Time Machine, if we had to. Not a joke. The server is backed up by Time Machine. The data is backed up to LTO, nearline archives racked and stacked in a server room and on redundant thunderbolt RAIDs which are parked on electric trucks ready to blast off at the earliest sign of danger. Well, everything except for the last part. Would be nice. And cloud backups for those clients that want them. Plan for the worst, pay for what you can to keep your business operational and lessen the impact of mechanical failures, human oopsies, or ransomware. Sysadmins are indistinguishable from malware sometimes, but we mean well. More seriously, humans makes mistakes and break things (that, me!) but ransomware is real and my elaborate backup and archive planning has saved a few customers this year.

    Ok, now for the fun part. How to upgrade an Xsan to macOS Big Sur (11.5.1). Maybe go read last year’s blog post on upgrading the Xsan to macOS Catalina 10.15.6 which was detailed and thorough. Or read Apple’s new Xsan Management Guide. It’s got all the fundamentals explained.

    Xsan volumes are typically made of up fibre channel RAID arrays. Nice icon!

    Preparation is key. Be prepared. Get ready. Psych yourself up. I used Greg Neagle’s installinstallmacos.py to download macOS Big Sur as a disk image and had that and the App Store’s Server.app downloaded beforehand and not be dependent on internet access (production SANs are not always internet accessible). It is both true and not true that you can setup Xsan in Big Sur with the Server.app. It is true you need the Server.app for an upgrade from macOS Catalina 10.15.7 but if you’re starting from scratch in macOS 11 you will be building your Xsan in Terminal. Have fun! (We will cover this in a future post).

    Download macOS Big Sur and the Server.app. Keep old copies zipped up. Cvlabel is nice too

    Server.app manages only three (3) services for an Xsan upgrade: Profile Manager, Open Directory and Xsan. In macOS Big Sur new setups of Server.app Xsan is gone. Why they haven’t taken out Profile Manager and not kept Xsan instead made me scratch my head. No one in their right mind is using Profile Manager to install or manage profiles, they’re using commercial MDM vendors. But Xsan in macOS Big Sur (11) is not only production ready storage SAN awesome it has been upgraded to be compatible with Quantum’s Stornext 7 (previously it was only v.5)

    Profile Manager does not belong here. Long Live Xsan!!

    Installing macOS 11 Big Sur and upgrading Xsan to v7 is compatible (in my testing) with macOS 10.14 Mojave, 10.15 Catalina and of course macOS 11 Big Sur. If you don’t believe me check out this not updated in forever Apple’s compatibility chart.

    Ok, by this time you get the idea I’m an expert, right? I’m ready to upgrade. But I run into my first real road block. And I have only myself to blame. I can’t launch the macOS Big Sur install app. It is blocked. “Contact your administrator”?! I am the sysadmin. Oh, ok. That’s me. What have I done now? I installed Hannes Juutilainen’s Big Sur Blocker last year, that’s what.

    Of course I installed that. With Munki. On all my Mac clients that were upgraded to macOS Catalina. (And of course my Xsan controller has Munki!). But no worries, let me read up on my last year’s blog post about it to figure out how I installed it, there must be a launch daemon or something.

    this is not how I expected it to go

    Hmm, no didn’t mention there. And where is that pesky launch daemon that I can unload and get to this Big Sur install. Oh? It’s a launch agent. Unloaded. Hmm, still no. Ok, delete the app from /usr/local/bin, hmm, nope. ok kill the app process. Ok, now we can install macOS Big Sur. Sorry for the delay. I had told Munki to uninstall the bigsurblocker app and it did for every other Mac, I swear, really. It did.

    Please proceed with the macOS Big Sur install

    So ready for macOS Big Sur. Oh wait, we noticed that you’re running Server.app and well, we don’t do a lot of the same things anymore in the new Server.app so maybe this is a warning.

    Warning. We noticed that you’re running Server.app and we don’t do those fun things anymore.

    So a lot of progress bars and stuff. See my last upgrade blog post and it’s the same as installing macOS Big Sur on any Mac, except this Mac Mini is running an Xsan production SAN environment with a lot of RAID arrays in a server rack or two. Ok, yeah, just run the installer.

    We noticed that Server app is no longer server app.

    After macOS Big Sur is installed zip up your older server.app and drag in your new one (or use that fancy App Store app to do it for you if you’re lazy). Click a bunch of buttons (see all my old blog posts) and launch the new Server.app.

    Profile Manager is updating. No one cares.

    So we have to wait while the bag of scripts that is Profile Manager gets updated but no one uses it but it’s the most important app in Server.app now, no I am not bitter why do you ask. Xsan is awesome.

    Xsan is off. Don’t panic.

    Xsan is off. Don’t panic. Where’s my towel? Panic now!

    Time to restore from your old Xsan configuration. Wheee…..

    Xsan restore configuration.

    Activate your Xsan and carry on upgrading all your Mac clients. Note: I did test macOS Mojave 10.14, macOS 10.15 Catalina and of course macOS 11.5.1 Big Sur Xsan clients. All worked.

    Xsan on. Power up.

    Upgrading Xsan with macOS Big Sur is easy if you’re going from macOS Catalina. Starting from scratch is another story to be covered in another blog post. Also not covered is certificate issues from self-signed certs breaking when I upgraded my Munki and MunkiReport server. That’s definitely another blog post. It’s just a webserver. Just. A. Web. Server. What is so hard? haha

    Technical Errata:

    With more than one Xsan controller it used to be recommended to upgrade the secondary before the primary but it is now best practise to upgrade the primary first to maintain the sanity of the OD data.

    Xsan Upgrade Step by Step:

    Clone the controllers. (+ Time Machine backups)
    Turn off the clients.
    Stop the Xsan Volume.
    Run cvfsck on the volume.
    **Upgrade the primary.
    Confirm the secondary can see the primary.
    *Upgrade the secondary.
    Confirm the secondary can see the primary.
    Check SAN access on both controllers.

    Upgrade the clients as desired.