Thunderbolt Xsan in a box. I’ve written about the Accusys T-share in 2020 (and in 2015 when I first found this cool tech). What’s different now? New year, new macOS. And a new challenge: can we build Xsan only using Terminal? No apps. It’s the journey that counts, right? One nerd’s journey to make an Xsan with macOS 11 Big Sur cli. Destination adventure with family fun, next stop a blinking cursor on a command line prompt.
make Xsan
make —Xsan —-bigger
reboot
Sudo make me an Xsan sandwich. I wish it were that easy! Stick around for the two or three commands you do need.
Xsan goes Terminal
Important commands for using Xsan have always been cvadmin and cvlabel (cv is short for centravision the original creators) but more recently xsanctl and slapconfig are important for creating the SAN and the OD (Open Directory) environment. Read the man pages, search the web, read some help documents. This blog is for entertainment and occasional learnings.
Xsan Commands: where are they?
/System/Library/Filesystems/acfs.fs/Contents/bin
cvlabel sncfgremove
cvaffinity cvmkdir sncfgtemplate
cvcp
cvmkfile sncfgtransform
cvdb
cvmkfs sncfgvalidate
cvdbset cvupdatefs sndiskmove
cverror cvversions snfsdefrag
cvfsck fsm snlatency
cvfsck_compat fsmpm snlicense
cvfsdb has_snfs_label snprodalert35chk
cvfsid mount_acfs snquota
cvgather sncfgconvert wingather
cvgather_fsm sncfgedit xsanctl
cvgather_multipath sncfginstall xsand
cvgather_sum sncfgquery xsandaily
Lots of interesting cv (CentraVision) and sn (StorNext) commands in macOS (this list is from 10.15 Catalina). Besides binaries, what else is there? Examples. A ton of example files:
/System/Library/Filesystems/acfs.fs/Contents/examples/
cvlabels.example fsnameservers.example rasexec.example
cvpaths.example fsports.example rvio.example
fsmlist.example nss_cctl.example snfs_metadata_network_filter.json.example
Just the facts. The Xsan basics
If you don’t have a fibre channel switch and fibre channel hardware RAIDs do not worry. You can build a useful Thunderbolt based Xsan with a little bit of effort. Just a little bit of peril It’s not too perilous, don’t worry.
Apple includes Xsan for free in macOS. Xsan is Apple’s fork Quantum’s StorNext SAN software. Want large fast storage made for Final Cut Pro editors, just add Xsan. Download Server.app from the Mac App Store and make your Xsan. Easy peasey. Right?
Why? Why are we doing this? Nothing beats fibre channel or Thunderbolt SAN speed for editing. Network attached storage (NAS) at 1GbE is barely usable. NAS at 10GbE is much better but still has road blocks for editors. Fibre channel or Thunderbolt with a big enough raid behind your SAN then life is great. Xsan can be shared by a small or media sized team of editors, producers and assistants.
Oh, ok. There is one problem. Apple did a major upgrade of Xsan (now version 7!) in macOS 11 Big Sur but apparently they took out the Xsan config in Server.app. (Note: This is what I was told early on and what seemed to be confirmed by Apple’s recent Xsan cli guide. It turns out that Xsan’s disappearance in Server.app to not be totally correct). Xsan is there in Server.app if you upgrade to macOS Big Sur but when you install Server on a clean macOS there is no Xsan visible in the app. Hmm. What do we do? Apple published a very nice handy guide about how to build Xsan in Terminal. So let’s get started. This is fun.
What do we need? 1) Hardware raid. Ok check I have an Accusys T-Share. It’s a raid with Thunderbolt switch built in. 2) Mac. Ok I have a Mac Mini. 3) A network. Some cables, a switch and a DNS server. Ok I have a new raspberry Pi. That’s perfect.
Step 1. Hardware raid. With the Accusys T-Share I just have to plug in some clients with a Thunderbolt 3 cable. Let’s fill the RAID with drives. I picked two different sizes. One group of larger disks for a data LUN (main production storage) and two smaller disks for a raid mirror to be used as metadata storage.
Step 2. A Mac running macOS Big Sur 11.5.2. Download the Accusys Mac installer on your Intel Mac (M1 is not supported with the T-Share yet as of this blog post).
Step 3. The network. Ok. This is the fun part. Let’s set up a DNS server. Ok, how do we do that? Remember that raspberry Pi you bought yourself for Christmas but never opened because you have been so busy and well you know life. Ok just me? Well, that one. Let’s use a raspberry Pi. A small inexpensive Linux computer. Install dns masq. It’s perfect for this.
The raid. Not only a great movie it’s the central part of this production media network for creatives. Once the drives are in the raid we have to make raid sets which become LUNs for Xsan. RAID5/6 for the data LUN and RAID1 (mirror) for the metadata LUN.
Read the label. Using Xsan cvlabel
Normally after we create RAID sets in the hardware raid utility we would open up Server.app and label the LUNs for Xsan use. But since we are now hardcore SAN architects we can use Terminal and the cvlabel the command to do this the hard way. Well, it’s not that hard but it can be intimidating the first few times. It’s much easier to label new LUNs than stare at a broken production SAN that has lost its labels. StorNext fun times. More about in another blog post.
Whether using Server.app in the good old days or cvlabel to label your LUNs now you should all be familiar with the command to list available LUNs. For larger SANs that won’t mount the first thing I’d check is see if the LUNs are all there. You don’t want a SAN to mount if it’s missing an important piece of itself.
cvlabel -l
This command lists available LUNs. It’s handy to know. Do this before trouble arises and you will be a cool dude when trouble happens. It does that occasionally. Prepare for the worst, hope for the best, IT motto.
To create labels for newly created RAID arrays use cvlabel to output a text file of the unlabelled LUNs, make some minor changes then label those LUNs. Create the template files first:
cvlabel -c
Edit the file. I like nano. Maybe you like vim. Or BBEdit. Or text edit. Change the name of LUNs from CVFS_unknown to whatever you like. I like to name LUNs based on the hardware they originate from so that I can find them, remove them, fix them or whatever I need to do for troubleshooting. Trust me. It’s a good idea.
cvlabel ~/Desktop/cvlabel
*WARNING* This program will over-write volume labels on the devices specified in the file "/Users/xavier/Desktop/cvlabel". After execution, the devices will only be usable by the Xsan. You will have to re-partition the devices to use them on a different file system.
Do you want to proceed? (Y / N) ->
Requesting disk rescan .
Congratulations this is the hardest part. You’ve labeled the RAID arrays as usable LUNs for Xsan. Ok, just kidding that’s not the hardest part. Have you ever heard of Open Directory? Do you fear LDAP and DNS? Well, maybe you should. It’s always DNS. Just saying.
DNS (domain name system) is just a fancy word for a list of IP addresses and host names. Using the raspberry Pi with dns masq installed we can populate the list of hosts for the Xsan and then we are golden. Hopefully if we did it right. Turns out we can make mistakes here too. Don’t use “.local” domain names. I did. It was late. I blame being tired. Changing them to “.lan” worked better.
Next up we finally create an Xsan in terminal. Or do we? let’s check the hostname first. It’s always DNS.
scutil —get HostName
CrazyMac.local
scutil --set HostName XsanMac.lan.
And now we make very big Xsan using the Xsan guide example
xsanctl createSan 'VIDEOSAN' --account localadmin --pass 72DERjx1 --user localadmin --cert-auth-name videocert --cert-admin-email administrator@example.com
It was at this point that it started falling apart. It was late. I had messed up my DNS with “.local” and the Xsan wouldn’t go past this basic OD setup. I did what I always do and reach out to my Xsan colleagues and I got some curious feedback. “What do you mean Xsan isn’t in macOS Big Sur Server.app?” Hmm. I don’t see it on a fresh install. On an upgrade from 10.15 Catalina I do. So, uh, Where is it? And then it was revealed. In the View menu. Advanced. Ugh. It’s right there. Almost staring right at me. When I opened the app it said it couldn’t create an Xsan with my “.local”. That was helpful. Fixed that and Xsan with my pre-labeled LUNs was super quick to set up.
I’ll have to play with the cli set up again soon. Because there were some strange formatting it recommended to me when I tried some variations of the xsanctl createSan. I’ll dig into another day when I have more sleep. Ha ha.
There’s a lot of useful commands in macOS Big Sur Xsan which was upgraded to v7. You can check which version of Xsan you have in macOS with the cvversions command.
In Catalina (macOS 10.15.7)
File System Server:
Server Revision 5.3.1 Build 589[63493] Branch Head BuildId D
Built for Darwin 19.0 x86_64
Created on Tue Jun 22 21:08:03 PDT 2021
Built in /AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/XsanFS/XsanFS-630.120.1/buildinfo
In Big Sur (macOS 11.5.2)
File System Server:
Server Revision 7.0.1 Build 589[96634] Branch Head BuildId D
Built for Darwin 20.0 x86_64
Created on Wed Jun 23 00:32:35 PDT 2021
Built in /System/Volumes/Data/SWE/macOS/BuildRoots/d7e177bcf5/Library/Caches/com.apple.xbs/Sources/XsanFS/XsanFS-678.120.3/buildinfo
There’s a lot of cool new binaries in Xsan v7. We will dig into those next post. For now enjoy this and go forth make some Xsan volumes with Thunderbolt or fibre channel storage. It’s fun.
