Thunderbolt Xsan in a box. I’ve written about the Accusys T-share in 2020 (and in 2015 when I first found this cool tech). What’s different now? New year, new macOS. And a new challenge: can we build Xsan only using Terminal? No apps. It’s the journey that counts, right? One nerd’s journey to make an Xsan with macOS 11 Big Sur cli. Destination adventure with family fun, next stop a blinking cursor on a command line prompt.
make —Xsan —-bigger
Sudo make me an Xsan sandwich. I wish it were that easy! Stick around for the two or three commands you do need.
Xsan goes Terminal
Important commands for using Xsan have always been cvadmin and cvlabel (cv is short for centravision the original creators) but more recently xsanctl and slapconfig are important for creating the SAN and the OD (Open Directory) environment. Read the man pages, search the web, read some help documents. This blog is for entertainment and occasional learnings.
Xsan Commands: where are they?
cvaffinity cvmkdir sncfgtemplate
cvdbset cvupdatefs sndiskmove
cverror cvversions snfsdefrag
cvfsck fsm snlatency
cvfsck_compat fsmpm snlicense
cvfsdb has_snfs_label snprodalert35chk
cvfsid mount_acfs snquota
cvgather sncfgconvert wingather
cvgather_fsm sncfgedit xsanctl
cvgather_multipath sncfginstall xsand
cvgather_sum sncfgquery xsandaily
Lots of interesting cv (CentraVision) and sn (StorNext) commands in macOS (this list is from 10.15 Catalina). Besides binaries, what else is there? Examples. A ton of example files:
cvlabels.example fsnameservers.example rasexec.example
cvpaths.example fsports.example rvio.example
fsmlist.example nss_cctl.example snfs_metadata_network_filter.json.example
Just the facts. The Xsan basics
If you don’t have a fibre channel switch and fibre channel hardware RAIDs do not worry. You can build a useful Thunderbolt based Xsan with a little bit of effort. Just a little bit of peril It’s not too perilous, don’t worry.
Apple includes Xsan for free in macOS. Xsan is Apple’s fork Quantum’s StorNext SAN software. Want large fast storage made for Final Cut Pro editors, just add Xsan. Download Server.app from the Mac App Store and make your Xsan. Easy peasey. Right?
Why? Why are we doing this? Nothing beats fibre channel or Thunderbolt SAN speed for editing. Network attached storage (NAS) at 1GbE is barely usable. NAS at 10GbE is much better but still has road blocks for editors. Fibre channel or Thunderbolt with a big enough raid behind your SAN then life is great. Xsan can be shared by a small or media sized team of editors, producers and assistants.
Oh, ok. There is one problem. Apple did a major upgrade of Xsan (now version 7!) in macOS 11 Big Sur but apparently they took out the Xsan config in Server.app. (Note: This is what I was told early on and what seemed to be confirmed by Apple’s recent Xsan cli guide. It turns out that Xsan’s disappearance in Server.app to not be totally correct). Xsan is there in Server.app if you upgrade to macOS Big Sur but when you install Server on a clean macOS there is no Xsan visible in the app. Hmm. What do we do? Apple published a very nice handy guide about how to build Xsan in Terminal. So let’s get started. This is fun.
What do we need? 1) Hardware raid. Ok check I have an Accusys T-Share. It’s a raid with Thunderbolt switch built in. 2) Mac. Ok I have a Mac Mini. 3) A network. Some cables, a switch and a DNS server. Ok I have a new raspberry Pi. That’s perfect.
Step 1. Hardware raid. With the Accusys T-Share I just have to plug in some clients with a Thunderbolt 3 cable. Let’s fill the RAID with drives. I picked two different sizes. One group of larger disks for a data LUN (main production storage) and two smaller disks for a raid mirror to be used as metadata storage.
Step 2. A Mac running macOS Big Sur 11.5.2. Download the Accusys Mac installer on your Intel Mac (M1 is not supported with the T-Share yet as of this blog post).
Step 3. The network. Ok. This is the fun part. Let’s set up a DNS server. Ok, how do we do that? Remember that raspberry Pi you bought yourself for Christmas but never opened because you have been so busy and well you know life. Ok just me? Well, that one. Let’s use a raspberry Pi. A small inexpensive Linux computer. Install dns masq. It’s perfect for this.
The raid. Not only a great movie it’s the central part of this production media network for creatives. Once the drives are in the raid we have to make raid sets which become LUNs for Xsan. RAID5/6 for the data LUN and RAID1 (mirror) for the metadata LUN.
Read the label. Using Xsan cvlabel
Normally after we create RAID sets in the hardware raid utility we would open up Server.app and label the LUNs for Xsan use. But since we are now hardcore SAN architects we can use Terminal and the cvlabel the command to do this the hard way. Well, it’s not that hard but it can be intimidating the first few times. It’s much easier to label new LUNs than stare at a broken production SAN that has lost its labels. StorNext fun times. More about in another blog post.
Whether using Server.app in the good old days or cvlabel to label your LUNs now you should all be familiar with the command to list available LUNs. For larger SANs that won’t mount the first thing I’d check is see if the LUNs are all there. You don’t want a SAN to mount if it’s missing an important piece of itself.
This command lists available LUNs. It’s handy to know. Do this before trouble arises and you will be a cool dude when trouble happens. It does that occasionally. Prepare for the worst, hope for the best, IT motto.
To create labels for newly created RAID arrays use cvlabel to output a text file of the unlabelled LUNs, make some minor changes then label those LUNs. Create the template files first:
Edit the file. I like nano. Maybe you like vim. Or BBEdit. Or text edit. Change the name of LUNs from CVFS_unknown to whatever you like. I like to name LUNs based on the hardware they originate from so that I can find them, remove them, fix them or whatever I need to do for troubleshooting. Trust me. It’s a good idea.
*WARNING* This program will over-write volume labels on the devices specified in the file "/Users/xavier/Desktop/cvlabel". After execution, the devices will only be usable by the Xsan. You will have to re-partition the devices to use them on a different file system.
Do you want to proceed? (Y / N) ->
Requesting disk rescan .
Congratulations this is the hardest part. You’ve labeled the RAID arrays as usable LUNs for Xsan. Ok, just kidding that’s not the hardest part. Have you ever heard of Open Directory? Do you fear LDAP and DNS? Well, maybe you should. It’s always DNS. Just saying.
DNS (domain name system) is just a fancy word for a list of IP addresses and host names. Using the raspberry Pi with dns masq installed we can populate the list of hosts for the Xsan and then we are golden. Hopefully if we did it right. Turns out we can make mistakes here too. Don’t use “.local” domain names. I did. It was late. I blame being tired. Changing them to “.lan” worked better.
Next up we finally create an Xsan in terminal. Or do we? let’s check the hostname first. It’s always DNS.
scutil —get HostName
scutil --set HostName XsanMac.lan.
And now we make very big Xsan using the Xsan guide example
xsanctl createSan 'VIDEOSAN' --account localadmin --pass 72DERjx1 --user localadmin --cert-auth-name videocert --cert-admin-email firstname.lastname@example.org
It was at this point that it started falling apart. It was late. I had messed up my DNS with “.local” and the Xsan wouldn’t go past this basic OD setup. I did what I always do and reach out to my Xsan colleagues and I got some curious feedback. “What do you mean Xsan isn’t in macOS Big Sur Server.app?” Hmm. I don’t see it on a fresh install. On an upgrade from 10.15 Catalina I do. So, uh, Where is it? And then it was revealed. In the View menu. Advanced. Ugh. It’s right there. Almost staring right at me. When I opened the app it said it couldn’t create an Xsan with my “.local”. That was helpful. Fixed that and Xsan with my pre-labeled LUNs was super quick to set up.
I’ll have to play with the cli set up again soon. Because there were some strange formatting it recommended to me when I tried some variations of the xsanctl createSan. I’ll dig into another day when I have more sleep. Ha ha.
There’s a lot of useful commands in macOS Big Sur Xsan which was upgraded to v7. You can check which version of Xsan you have in macOS with the cvversions command.
In Catalina (macOS 10.15.7)
File System Server:
Server Revision 5.3.1 Build 589 Branch Head BuildId D
Built for Darwin 19.0 x86_64
Created on Tue Jun 22 21:08:03 PDT 2021
Built in /AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/XsanFS/XsanFS-630.120.1/buildinfo
In Big Sur (macOS 11.5.2)
File System Server:
Server Revision 7.0.1 Build 589 Branch Head BuildId D
Built for Darwin 20.0 x86_64
Created on Wed Jun 23 00:32:35 PDT 2021
Built in /System/Volumes/Data/SWE/macOS/BuildRoots/d7e177bcf5/Library/Caches/com.apple.xbs/Sources/XsanFS/XsanFS-678.120.3/buildinfo
There’s a lot of cool new binaries in Xsan v7. We will dig into those next post. For now enjoy this and go forth make some Xsan volumes with Thunderbolt or fibre channel storage. It’s fun.
I have a Accusys A12T3 setup with one raid 0 and two Macs connected to it. Do you know if I need to have a separate server running Xsan and DNSmasq like you’ve outlined above? If yes, Xsan is not available anymore so what would be an alternative?
My accusys is not stable and will unmount from one computer and not be able to remount or it needs disk warrior to fix the directory before it’ll load again.
Xsan is still available and built in to every Mac. You do need Xsan (or StorNext) for example to have two client mounted simultaneously. The other option is to connect one Mac directly and share via Ethernet.
Thanks for getting back to me! I feel a bit of hope. Ok, so I need to install xsan server running for the accusys. Do I need to get a third computer or raspberry pi to run as the server with DNSmasq? Or can I run that off one of the current computers I have already? I’ve seen some guides on installing DNSmasq using home-brew. Is this the right direction?
I recommend a Mac mini as a server but it could be any computer connected to the Accusys. I ran dns masq on a raspberry pi as a fun project but you could install it anywhere.
Another blog post by a colleague details some of the steps to creating a SAN here: https://www.rskgroup.org/xsan/create-a-new-xsan-on-macos-monterey/
Thanks for pointing me in the right direction. I figured a mini would be good. I will leap into this project and learn a lot on the way. Wish me luck and thanks for being so responsive.
Sorry, final question, does the Mac mini have to have thunderbolt 3 for it to work as a server or can I connect it another way?
Yes. The rules for being a server or a client for Xsan are: 1) connected by Thunderbolt (or fibre channel) and 2) connected to metadata network (a small private Ethernet network for Xsan)
Does the thunderbolt
Have to be 4? Accusys says you must use 4 for it to work. Want to check if I can get away with an adapter from thunderbolt 2 and an older Mac mini.
An adapter should be fine. I’ve used Thunderbolt 2 to 3 adapters.
I’ve configured a Mac mini server using dnsmasq and that seems to be working. Xsan has been installed but I’m stuck at the final step of creating a volume which is so frustrating! This is been my nonstop project for the last 3 days and I’ve learnt a lot but there’s so much more.
Anyway I can entice you to do a screen share on my server to get it over the last step? I’ve been referring heavily to your colleagues blog to get me to this stage.
What errors are you getting? What does it say when you try to create a volume?
if my metadata and journal LUN is called metadataAndJournalLUN and the data LUN is called dataLUN
What’s the command to create a volume called AccusysSAN?
I setup the metadata and journal LUN as a raid 1 and it’s the first LUN on the accusys with the second LUN being the data. I want the metadata and journal LUN on the first volume like you did. This is where I went wrong.
I got the volume mounted!!! Final problem is my DNSmasq server doesn’t seem to allow my clients to access the internet?
2 more challenges. I’m trying to load the backup media onto the Xsan and carbon copy cleaner says there’s some read write errors. I can still run the restore though but concerning perhaps? Second challenge is when I’m trying to load the profile for the SAN onto the client I get the error “Session can’t be opened because daemon refused the connection.”
Can the clients see the Xsan controller? Does it have the same network info?
In the etc/hosts on the server I have the IP’s and the hosts setup to match. However when I run the host command this is what I get back
snowy@PSmacbook ~ % host PSserver.lan
Host PSserver.lan not found: 3(NXDOMAIN)
When I run that command on the server it resolves correctly?
admin@PSserver ~ % host PSserver.lan
PSserver.lan has address 192.168.1.78
On the client side I set the IP address manually and made sure it matches what I have the the etc/hosts file. Subnet and router I left alone. In the DNS tab I added the host server IP to DNS servers and added google DNS servers as the second and third. For search domains I put lan. For setting up DNSmasq I followed https://www.rskgroup.org/xsan/create-a-new-xsan-on-macos-monterey/.
Do you have any suggestions for optimizing speed on the Xsan? I’m getting extremely slow speeds from the 2 disk raid 0 I have in there currently. My journal and metadata are on a 3tb raid 1 in the same tower. Also I have to run the command sudo xsanctl mount XsanVolume for it to show up on my client. I’m getting read write speeds of max 80 MB/s but that takes along time to tick up from 5 MB/s.
Potentially the ethernet is being used for internet and Xsan and that’s probably why it’s slow. Now I have to figure out how to split them up.
For best performance it is recommended to have a private Xsan metadata network. Clients talks to controller through this non bridged network.
Any write ups on how to setup a private one?
In the /etc/resolv.conf on the server I also added
I also modified /usr/local/etc/dnsmasq/dnsmasq.conf
I was able to reboot dnsmasq and voila! It worked and I got my SAN to show up on my client. I think I have to go celebrate now!
So I found the command xsanctl changeIP oldIP newIP to make the metadata network private but I only have one MDC so I put the same IP for new and old IP. The DNS Server/MDC works now (that’s the IP I entered) at the proper speeds but my clients are not resolving when I type host .
Host PSserver.lan not found: 3(NXDOMAIN)
This is what I get back now so obviously they aren’t connected? Apple says the have to be connected to the same public and private networks (clients and MDC to work). Can my wifi be the public network or do I need two ethernet ports for each one and two DNS servers? I thought I had it figured out but I’m lost again. As a last ditch I ran this
snowy@PSmacbook ~ % sudo xsanctl changeIP 192.168.1.78 192.168.1.78
2022-07-14 20:49:34.303 xsanctl[3207:366726] Pushing update change, ‘4C92301C-4D1D-4D9B-A1BF-3519D1406835’
xsanctl: error opening connection: SnAdmin Connect: (Re)connect failed to FSMPM at ‘127.0.0.1’ – The File System Services on 127.0.0.1 may be stopped.