I was chatting with Per Olofsson on a recent episode of the MacDevOps podcast about some recent fixes with relocatable Python he did for MunkiReport version 5.7.0 and I happened to mention how much I love my MR dashboards with custom hot keys. He is a long time user of MunkiReport but hadn’t heard that you could make custom dashboards and I couldn’t remember where I had heard of it or even how I made them. Pretty typical of tech these days I think. You learn something, you make something and then you move to the next thing and forget what you were doing or how you did it. Well, thanks to documentation we can share the knowledge and spread the love.
Custom Dashboards
The MunkiReport wiki actually has a short entry which explains how to make a custom Dashboards. Basically, add some YAML files in the dashboards folders and you’re done. Follow the Read Me file for instructions. Pro Tip: Use the Widget Gallery in MR to find useful pieces to build into your dashboards. Note: I’ve added these custom dashboards to my local folder which is set in my “.env” to be outside of the main munkireport folder so it easier to update across version upgrades.
I am so happy to install macOS Big Sur 11.5.1, now that it is a ready for production. Have fun with macOS Monterey those of you on the bleeding edge. For media professionals using Xsan in production storage environments August is a great month to update to the soon to be yesterday’s bad boy Mr. Big Sur.
Server.app v.5.10 in macOS Catalina 10.15.7
Upgrading to a new major version of macOS can be fraught with peril for a fleet of mac devices but it is potentially fatal for a production SAN environment. That is why we wait. We want a nice stable storage system for our Final Cut Pro editors and other media creatives so it is safe to be one version behind. Less drama that way. We prefer our dramas to be on AppleTV+
Watch TV Upgrade Xsan
It is not boring to watch AppleTV+ while upgrading Xsan
The Xsan upgrade to Big Sur was pretty much not exciting except for one funny roadblock that I had set up myself last as a kind of booby trap for “future me”. More about that later. First the boring stuff. The last few weeks have been very busy updating and re-writing documentation in Pages.app and running multiple redundant full and incremental LTO backups with Archiware P5, syncing to nearline archives, and archiving finalized projects to the LTO shelf in paradise (sounds more exciting when you put it that way don’t you think?). Updating and re-writing documentation can sound like a waste of time but “future you” will appreciate what “past you” was doing today. And today I had fun updating Xsan to macOS Big Sur. Now I must write down all my thoughts before I each too much vegan vanilla ice cream and slip into a food coma.
“Planning for disasters, while hoping for none” is the IT mantra. We planned hard and we were ready to restore Xsan from Time Machine, if we had to. Not a joke. The server is backed up by Time Machine. The data is backed up to LTO, nearline archives racked and stacked in a server room and on redundant thunderbolt RAIDs which are parked on electric trucks ready to blast off at the earliest sign of danger. Well, everything except for the last part. Would be nice. And cloud backups for those clients that want them. Plan for the worst, pay for what you can to keep your business operational and lessen the impact of mechanical failures, human oopsies, or ransomware. Sysadmins are indistinguishable from malware sometimes, but we mean well. More seriously, humans makes mistakes and break things (that, me!) but ransomware is real and my elaborate backup and archive planning has saved a few customers this year.
Xsan volumes are typically made of up fibre channel RAID arrays. Nice icon!
Preparation is key. Be prepared. Get ready. Psych yourself up. I used Greg Neagle’s installinstallmacos.py to download macOS Big Sur as a disk image and had that and the App Store’s Server.app downloaded beforehand and not be dependent on internet access (production SANs are not always internet accessible). It is both true and not true that you can setup Xsan in Big Sur with the Server.app. It is true you need the Server.app for an upgrade from macOS Catalina 10.15.7 but if you’re starting from scratch in macOS 11 you will be building your Xsan in Terminal. Have fun! (We will cover this in a future post).
Download macOS Big Sur and the Server.app. Keep old copies zipped up. Cvlabel is nice too
Server.app manages only three (3) services for an Xsan upgrade: Profile Manager, Open Directory and Xsan. In macOS Big Sur new setups of Server.app Xsan is gone. Why they haven’t taken out Profile Manager and not kept Xsan instead made me scratch my head. No one in their right mind is using Profile Manager to install or manage profiles, they’re using commercial MDM vendors. But Xsan in macOS Big Sur (11) is not only production ready storage SAN awesome it has been upgraded to be compatible with Quantum’s Stornext 7 (previously it was only v.5)
Profile Manager does not belong here. Long Live Xsan!!
Installing macOS 11 Big Sur and upgrading Xsan to v7 is compatible (in my testing) with macOS 10.14 Mojave, 10.15 Catalina and of course macOS 11 Big Sur. If you don’t believe me check out this not updated in forever Apple’s compatibility chart.
Ok, by this time you get the idea I’m an expert, right? I’m ready to upgrade. But I run into my first real road block. And I have only myself to blame. I can’t launch the macOS Big Sur install app. It is blocked. “Contact your administrator”?! I am the sysadmin. Oh, ok. That’s me. What have I done now? I installed Hannes Juutilainen’s Big Sur Blocker last year, that’s what.
Of course I installed that. With Munki. On all my Mac clients that were upgraded to macOS Catalina. (And of course my Xsan controller has Munki!). But no worries, let me read up on my last year’s blogpost about it to figure out how I installed it, there must be a launch daemon or something.
this is not how I expected it to go
Hmm, no didn’t mention there. And where is that pesky launch daemon that I can unload and get to this Big Sur install. Oh? It’s a launch agent. Unloaded. Hmm, still no. Ok, delete the app from /usr/local/bin, hmm, nope. ok kill the app process. Ok, now we can install macOS Big Sur. Sorry for the delay. I had told Munki to uninstall the bigsurblocker app and it did for every other Mac, I swear, really. It did.
Please proceed with the macOS Big Sur install
So ready for macOS Big Sur. Oh wait, we noticed that you’re running Server.app and well, we don’t do a lot of the same things anymore in the new Server.app so maybe this is a warning.
Warning. We noticed that you’re running Server.app and we don’t do those fun things anymore.
So a lot of progress bars and stuff. See my last upgrade blog post and it’s the same as installing macOS Big Sur on any Mac, except this Mac Mini is running an Xsan production SAN environment with a lot of RAID arrays in a server rack or two. Ok, yeah, just run the installer.
We noticed that Server app is no longer server app.
After macOS Big Sur is installed zip up your older server.app and drag in your new one (or use that fancy App Store app to do it for you if you’re lazy). Click a bunch of buttons (see all my old blog posts) and launch the new Server.app.
Profile Manager is updating. No one cares.
So we have to wait while the bag of scripts that is Profile Manager gets updated but no one uses it but it’s the most important app in Server.app now, no I am not bitter why do you ask. Xsan is awesome.
Time to restore from your old Xsan configuration. Wheee…..
Xsan restore configuration.
Activate your Xsan and carry on upgrading all your Mac clients. Note: I did test macOS Mojave 10.14, macOS 10.15 Catalina and of course macOS 11.5.1 Big Sur Xsan clients. All worked.
Xsan on. Power up.
Upgrading Xsan with macOS Big Sur is easy if you’re going from macOS Catalina. Starting from scratch is another story to be covered in another blog post. Also not covered is certificate issues from self-signed certs breaking when I upgraded my Munki and MunkiReport server. That’s definitely another blog post. It’s just a webserver. Just. A. Web. Server. What is so hard? haha
Technical Errata:
With more than one Xsan controller it used to be recommended to upgrade the secondary before the primary but it is now best practise to upgrade the primary first to maintain the sanity of the OD data.
Xsan Upgrade Step by Step:
Clone the controllers. (+ Time Machine backups) Turn off the clients. Stop the Xsan Volume. Run cvfsck on the volume. **Upgrade the primary. Confirm the secondary can see the primary. *Upgrade the secondary. Confirm the secondary can see the primary. Check SAN access on both controllers.
People are afraid of documentation… But mostly people just hate it. They don’t like it. They don’t want it. It shouldn’t exist. Fingers in ears. I can’t hear you.
This is about primal fear. And hate. I hate hate. But these are real emotions. Let’s deal with it. What is the reality? Why is documentation is ignored, abandoned, or resisted at all?
As a Sysadmin perhaps you don’t care about documentation, that is, sharing information with others (co workers / bosses), you want to keep it to yourself. But you care very much about building systems. But there’s perhaps no attempt to explain any of this to anyone else. Who else is there really? No one cares. No one is around that would understand if you explained it.
Lesson # 1 – Document for yourself.
Paranoia makes us set up redundant systems for backups. Layers upon layers. Custom scripts and disparate apps. Where was this explained? Documented? Nowhere. Bin dir. Maybe.
If you could replace all that now with one app that did it all then you would. Time is valuable. Easier to monitor. Easier for someone else to monitor and take over.
Lesson # 2 – document for your replacement (job change, bus hit)
Do it continuously. Automate. Or set up systems that work automatically.
Lesson # 3. DevOps.
Integrate systems. IT systems manage computer but maybe they also built Inventory. Automatically. Alert Systems report continuously. Living systems report on the state of everything. Documentation is easier when it is current and relevant.
Lesson # 4. Sustainability
Commercial vs OpenSource. Support vs excellent team, talent retention and documentation. Pro/Con. If your custom solution is not well documented that can be a big problem. If you code is not shared, peer-reviewed, or supported by anyone that could be an issue. If it makes sense to switch to commercial software that is supported then do it. If an OpenSource project or code is supported by a larger community perhaps that makes sense.
Lesson # 5. Improve. Grow. Get better.
Discovery and Documentation lead to suggestions for improvement. Make changes. Code and disparate systems that struggle to be documented make us think about how to replace them or better balance the risks vs cost.
Lesson # 6. Human problems don’t always tech solutions.
Code doesn’t fix broken workflows. Meetings are with people. Talking through systems helps people understand pain points. Don’t forget people want to do their job, meet deadlines, do stuff.
I was explaining Munki (and autopkg) to some colleagues when I hit on the idea of the MacDevOps manifesto.
Munki and friends (apps used to augment and extend Munki) are helpful automation tools. Setting up automation systems take time and must be maintained and grown but they pay big dividends. Freeing us to do Dev work or other tasks they automate and iterate and repeat and build our systems in the way we want.
No more 100 machines built in a hundred different ways (unless we want to). But now we can check at a glance in MunkiReport to verify that indeed the latest Adobe Flash patch is installed. That may make our lives better. Especially if we need to satisfy corporate IT or our bosses that we are up to date and patched as required.
The MacDevOps Manifesto Part 1: Munki and friends
Munki is at its core free software created by Greg Neagle at Disney Animation and used worldwide in many different ways but essentially to distribute apps and run scripts on client workstations. There are many ways to customize it and if fits many different workflows. The MacDevOps:YVR conference I ran last June turned out to be a Munki love-in and showed me the many awesome and varied ways organizations are using it.
With AutoPkg, another free Mac open source project, Munki can get the latest updates to any software that it has recipes for and by extension install them on clients immediately. This fits the workflow of having Flash, Java and web browsers (Chrome or FireFox) updated as soon as possible for security patches. Exploits on the Mac are coming from these entry points and if you need to use these apps or plugins then having the latest versions helps. For this feature alone I use Munki. In a few months you will see that Munki with AutoPkg has downloaded dozens of versions of each app and keeping up with this takes time away from other tasks. Automation of simple tasks frees up our time so we can focus on other things. That is MacDevOps.
I also use Munki for installation of any app that is needed everywhere. If I have to download or install one app for one client workstation I put it in Munki and it is ready for installation anywhere with a simple click by the user in a self service portal or automatically by choosing managed installs. Of course if there is an app you don’t want installed (flash or Skype or messenger, etc) add it to Munki and mark it as managed uninstall. Done.
Scripts and files and config Profiles (replacement for mcx, managed preference settings for OS X) can be imported and used to configure workstations to make deployment easy and flexible. Put everything in Munki and then you don’t have to use golden master builds anymore. Buy a new Mac and install the Munki client. Done.
Add to this Munki Report which gives an excellent dashboard for what is installed and a total inventory of your client Macs. Very useful info which will let you know if you 15 different versions of flash or Photoshop or any app you choose to look for.
Last but least I always install Watchman Monitoring which reports to a secure cloud (web portal) to automatically monitor for bad drives, Ram, backups not running etc. It’s a great 50ft overview of all your installs and it can alert you immediately when a machine is having issues that you need to deal with (drives 90% full or Xsan volume not mounted, etc).
I find this combination of Munki and Watchman great for helping me manage my clients and I want to share these ideas about MacDevOps inspired ways of automating systems with everyone. Jump in and get involved with all these projects. You’ll be writing recipes for AutoPkg and sharing cool Munki tips and tricks with all your friends. And maybe like me you will start writing plugins for Watchman to monitor your favourite apps (I’m working on Archiware P5 backup and archive monitoring scripts).
Good luck to everyone and hope to see you at the next MacDevOps:YVR conference in June 2016. If you can’t make it go to your nearest Mac Dev / IT conference or start your own meet up somewhere local.
DAM SAN 