Big Sur summer testing time!
Summer time is beta testing time. A new macOS beta cycle with Big Sur is upon us. Test early, and test often. With all the excitement of Big Sur in the air, it’s time to look at Catalina.
Our day to day production Xsan systems do not run beta software, not even the latest version of macOS, they only run tested and safe versions of macOS. I always recommend being a revision behind the latest. Until now that meant macOS 10.14 (Mojave). With the imminent release of macOS Big Sur (is it 10.16 or macOS 11?) then it’s time to move from 10.14.6 Mojave to 10.15.6 Catalina. It must be safe now, right?
Background
Xsan is Apple’s based Storage Area Network (SAN) software licensed from Quantum (see StorNext), and since macOS 10.7 aka Lion it has been included with macOS for free (it was $1,000 per client previously!).
Ethernet vs Fibre Channel vs Thunderbolt
A SAN is not the same as a NAS (Network attached storage) or DAS (direct attached storage). A NAS or other network based storage is often 10GbE and can be quite fast and capable. I will often use Synology NAS with 10GbE for a nearline archive (a second copy of tape archive) but can also use it as a primary storage with enough cache. Lumaforge’s Jellyfish is another example of network based storage.
Xsan storage is usually fibre channel based and even old 4GB storage is fast because … fibre channel protocol (FCP) is fast and the data frames are sent in order unlike TCP. It is more common to see 8GB or 16Gb fibre channel storage these days (though 32GB is starting to appear). And while fibre channel is typically what you use for Xsan you can also use shared Thunderbolt based storage like the Accusys A16T3-Share. I have tested a Thunderbolt 2 version of this hardware with Xsan and it works very well. I’m hoping to test a newer Thunderbolt 3 version soon. Stay tuned.
Xsan vs macOS Versions
We’ve discussed all the things that the Xsan is not and now what is it? Xsan is often created from multiple fibre channel RAID storage units but the data is entirely dependent on the Xsan controller that creates the volume. The Xsan controller is typically a Mac Mini but can be any Mac with Server.app (from Apple’s App Store). The existence of any defined Xsan volumes depends on the sanity of its SAN metadata controllers. If the SAN controllers die and the configuration files go with it then your data is gone. POOF! I’ve always said that Xsan is a shared hallucination, and all the dreamers should dream the same dream. To make sure of this we always recommend running the same version of macOS on the Mac clients as well as the servers (the Xsan controllers). And while the Xsan controllers should be the same or at a higher macOS version level it can sometimes be the opposite in practise. To be sure what versions of macOS are interoperable we can check with Apple’s Xsan controllers and clients compatibility chart and Xsan versions included in macOS for the rules and exceptions. Check the included version of Xsan on your Mac with the cvversions command
File System Server: Server Revision 5.3.1 Build 589[63493] Branch Head BuildId D Built for Darwin 17.0 x86_64 Created on Sun Dec 1 19:58:57 PST 2019 Built in /BuildRoot/Library/Caches/com.apple.xbs/Sources/XsanFS/XsanFS-613.50.3/buildinfo
This is from a Mac running macOS 10.13
Host OS Version: Darwin 17.7.0 Darwin Kernel Version 17.7.0: Sun Dec 1 19:19:56 PST 2019; root:xnu-4570.71.63~1/RELEASE_X86_64 x86_64
We see similar results from a newer build below:
File System Server: Server Revision 5.3.1 Build 589[63493] Branch Head BuildId D Built for Darwin 19.0 x86_64 Created on Sun Jul 5 02:42:52 PDT 2020 Built in /AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/XsanFS/XsanFS-630.120.1/buildinfo
This is from a Mac running macOS 10.15.
Host OS Version: Darwin 19.6.0 Darwin Kernel Version 19.6.0: Sun Jul 5 00:43:10 PDT 2020; root:xnu-6153.141.1~9/RELEASE_X86_64 x86_64
Which tells us that the same version of Xsan are included with macOS 10.13 and 10.15 (and indeed is the same from 10.12 to 10.15). So we have situations with Xsan controllers running 10.13 and clients running 10.14 are possible even though macOS versions are a mismatch, the Xsan versions are the same. There are other reasons for keeping things the macOS versions the same: troubleshooting, security, management tools, etc To be safe check with Apple and other members of the Xsan community (on MacAdmins Slack).
Backups are important
Do not run Xsan or any kind of storage in production without backups. Do not do it. If your Xsan controllers die then your storage is gone. Early versions of Xsan (v1 especially) were unstable and the backups lesson can be a hard one to learn. All later versions of Xsan are much better but we still recommend backups if you like your data. Or your clients. (Clients are the people that make that data and pay your bills). I use Archiware P5 to make tape backups, tape archives, nearline copies as well as workstation backups. Archiware is a great company and P5 is a great product. It has saved my life (backups are boring, restores are awesome!).
Xsan Upgrade Preparation
Before attempting a macOS or Xsan upgrade please check your backups. Test your restores. Your SAN volumes should be backed up. I backup of all the san volumes on nearline Thunderbolt raids and a Synology NAS as well as LTO tape backups and archives. That’s the SAN data, the creative files that the clients care about, but what about the Xsan itself? The Xsan controller has configuration files. How do we save those?
/Library/Preferences/Xsan
sudo hdiutil create -srcfolder /Library/Preferences/Xsan/ ~/Desktop/Xsan-prefs-afterMojaveUpdate.dmg
cvgather -f XSAN
I use three methods of preserving the data of the Xsan controller.
- I have Time Machine backups (on an external drive).
- And a Mac mini in target mode as a clone created by Carbon Copy Cloner.
- Lastly, copies of all config files before and after all updates and migrations.
I have restored my Xsan from a bad macOS upgrade using Time Machine. Yes, it works. Believe it. The Mac Mini clone is also ready to take over (hat tip to Alex Narvey for this methodology). Lastly the config files by themselves can help restore the Xsan when all the files appear to be gone because the Xsan has lost its mind. Save them, and save yourself. Use this opportunity to update your documentation.
List all the Time Machine backups:
sudo tmutil listbackups
sudo cvlabel -l > ~/Desktop/cvlabel-xsan.txt
Check check all the backups
Before the actual upgrades backup your data (and test your restores), backup your Xsan controllers (and double check) then download the macOS installers. I use installinstallmacos.py to download my installers). In my recent Xsan controller upgrade from 10.13 in preparation I downloaded the latest 10.14.6 and 10.15.6. I also downloaded the actual Server.app installers from various macOS installs to be ready (in case of network difficulties on site during an upgrade).
Finally! Upgrade the Xsan and macOS
An overview of the upgrade steps as outlined by my esteemed Xsan colleague Bob Kite:
- Clone the controllers.
- Turn off the clients.
- Stop the Volume.
- Run cvfsck on the volume.
- Upgrade the secondary.
- Confirm the primary can see the secondary.
- Upgrade the primary.
- Confirm the primary can see the secondary.
- Check SAN access on both controllers.
What about Server.app?
When you upgrade macOS it will warn you that you have Server.app installed and you might have problems. After the macOS upgrade you’ll need to download and install a new version of Server.app. In my recent upgrades from macOS 10.13 to macOS 10.15 via 10.14 detour I started with Server.app 5.6, then install 5.8 and finally version 5.10.
After the macOS upgrade I would zip up the old Server.app application and put in place the new version which I had already downloaded elsewhere. Of course you get a warning about removing the Server app
Install the new Server app then really start your Xsan upgrade adventure.
Restore your previous Xsan setup.
If everything goes well then you have Xsan setup and working on macOS 10.15.6 Catalina
DAM SAN 