I don’t get High — Sierra!

Friends don’t let friends install macOS High Sierra in production. Don’t get High, Sierra.

macOS 10.13 was released on Sep 25, 2017, and almost two months later with only one point release update, it’s still too new for production. Download it on a test machine or two or more, test it with your apps and systems, file bug reports and radars, but for the love of all that is Python and Monty! don’t run it on your production Xsan. Well, at least not yet. Wait until next year. Or as long as you can. Or until the new iMac Pro is released with 10.13 pre-installed or wait until they ship the new Final Cut Pro X 10.4 that may or may not require macOS High Sierra.

With that out of the way, I’ve just upgraded the production Xsan to … macOS Sierra. Yes, macOS 10.12.6 is stable and it’s a good time to install last year’s macOS release. Time to say good bye to macOS el Capitan 10.11.6, we hardly knew ya. Besides guaranteed security updates, stability and the annoying newness of a changed macOS, what else is there? In Xsan v5 they introduced a new “ignore permissions” checkbox for your Xsan volumes. Looking forward to that feature in production. No more Munki onDemand nopkg scripts to run chmod. No more tech support requests for folders, files, FCP X projects that won’t open because someone else used it, owns it, touched it. We’ll see how that pans out. I’ll let you know.

Upgrading Xsan to v5

Step 1. Back up your data

You’re doing this, right? I’m using Archiware P5 Backup to backup the current projects to LTO tape. I’m using Archiware P5 sync to sync the current Xsan volumes to Thunderbolt RAIDs, and using Archiware P5 Archive (and Archive app) to archive completed projects to the LTO project archive. That’s all I need to do, right?

Step 2. Back up your servers

Don’t forget the servers running your SAN! I use Apple’s Time Machine to backup my Mac Mini Xsan controllers. External USB3 drive. I also use another Mac Mini in target disk mode with Carbon Copy Cloner to clone the server nightly. (Hat tip to Alex Narvey, a real Canadian hero). And of course I grab the Xsan config with hdiutil and all the logs with cvgather. Because, why not?! For Archiware P5 backup server I also have a python scripts to backup everything, another scripts to export a readable list of tapes, and BackupMinder to rotate the backups. Add some rsync scripts and you’re golden.

 

Step 3. Upgrade the OS

Unmount the Xsan volume on your clients or shut them down, disconnect the fibre channel. Do something like that. Stop your volume. Download the macOS Sierra installer from the App Store. Double click upgrade. Wait. Or use Munki. I loaded in the macOS 10.12.6 installer app into Munki and set it up as an optional install to make this portion of the upgrade much quicker and cleaner.

In my case after the OS was upgraded I checked the App Store app for any Apple updates (you can also use Munki’s Managed Software Center to check) and of course there were some security updates. In this case the security upgrade hung on a slow network connection and the server crashed. Server down! I had to restore from Time Machine backup to the point where I just upgraded the server. It took some extra time  but it worked (can’t wait for next year’s mature APFS / Time Machine and restoring from snapshots instead).

Step 4. Upgrade Server

After macOS is upgraded you’ll need to upgrade the Server.app or just upgrade the services used by Server (even those not used by Server get upgraded).

Step 5. Upgrade the Xsan

Bur first we have to restore the Xsan config. Don’t panic! It may invoke bad memories of data loss and restoring from backups. Xsan PTSD is real.

Restore-previous-Xsan.png

Step 6. Upgrade the rest

Next you have to upgrade the Xsan volumes.

Xsan-volume-needs-upgrade

New version of Xsan, ch-ch-changes! Ignore permissions check box will remount the xsan with the “no-owners” flag. Let’s test this out.

 

Upgrade the OS and Server app on the backup controller. Upgrade the OS on the clients using Munki or App Store if you like doing it the hard way. Ha Ha.

Step 7. Enjoy

Plug those Thunderbolt to Fibre adapters back in, mount those Xsan volumes and be happy.

Step 8. Wait for the complaints

The next day the editors walked in and went straight to work with Final Cut Pro X. No one noticed anything. Xsan upgraded. Workstation macOS upgraded. Everything appeared to be the same and just worked. Thankless task but well worth it.

 

Reference: Apple’s iBook guide here

 

 

My Thunderbolt Nightmare

It was a dark and stormy night of cables and capacitors when suddenly I heard the door knocking, or was something falling of a shelf? I was in a cramped server room, if you’d call it that, and I was day dreaming, sorry, night terrorizing, of days gone past when I worked in nice big well ventilated server rooms with proper enterprise gear. Oh wait, did I really dream that? Did it really happen? Maybe it was less well ventilated and there were cables strewn about the tall 42U shelves and sometimes we found a Mac hidden underneath spaghetti. Sometimes. I vaguely remember the long shiny metal servers, they talked to me, they sang, a whiny pitch of whale song. Dream on, dream on.

Now. Today. Apple Music on my iPhone plays every single Arcade Fire album in a long playlist, in order. And I follow the white rabbit of Thunderbolt cables. This is my thunderbolt nightmare. Dead drive in a Thunderbolt Promise Pegasus unit, web ticket filed for registered hardware. Legacy. That’s the word they used. Where’s Marshall McLuhan when you need a proper redux of the shit storm you’re in? Thunderbolt 3 uses USB-C and everything is possible. Can’t wait to step into that confusing identity crisis. OK, back to the present day when I stared at the red blinking drive, a replacement drive from not long ago dead again, sitting in the last row of a now legacy Pegasus R6 unit. RAID 5, the most dangerous kind, this is what stood between me and uncertainty. The worst kind of RAID. Well, not as bad RAID 0. Raid nothing. Raid 5 is one bad drive away from a bad day. Backups? Hmm, I got those, I got plenty of those, but I don’t want to be tested today. No, not today. Not this bloody day.

I open the Pegasus utility and the GUI wants an update. Hmm, that’s not in autopkg, I think. Why is out of date? Munki let me down. I start to drift, to side shift into adding newer better recipes to autopkg, to tweaking my Munki repo, to what sessions would be awesome at the next MacDevOps:YVR conference. Gee whiz, I love open source, and everyone in the Mac Admins community…. Snap out of! I slap myself in the face. I was hallucinating. Stay on task. I update the Pegasus utility. I stare at the critical reports from one of the three R6 units attached to this Mac Mini server. Did I say server? But it’s so small, so little. It works. It’s magical, kinda neat. Until you stare too close at the back. The Thunderbolt cables go from the Mac Mini to the first Pegasus unit to the SANlink fibre channel adapter to the LTO 6 tape library to the next Pegasus utility to the second SANlink adapter to a third and final Pegasus RAID unit. What’s is going on? Where does this cable go? Let me just follow it to the next jumping off point. My brain slows, the lack of oxygen in this cold machine room start to affect my thinking. I lose my way.

I download the report for the Pegasus unit. I had to unlock a pretty neat lock icon and click on the save report. I upload it to the web support and add it to the ticket. Tech support gets backs to me in a day and said all is good, and to carry on. I can’t. The drive is dead. What are they not seeing? It’s right in front of me. I download the report again. Again the same response. Fine. It’s time to stop messing around and pop open Terminal. Loading up promiseutil I check out the options and switches and get into an argument with myself about the currently valid optionals of letters and numbers that are required. I check my notes, online knowledge base, and try again. It’s broken. It doesn’t work. Stumbling around the command line typing imprecisely incorrect statements gets nowhere fast. I realize that there’s no way for the cli utility to properly change its focus to the broken unit with the busted drive. Both the GUI and the binary are stuck on the one R6 unit and won’t see what’s in front of my face.

I call tech support. This is humiliating. This was supposed to be easy. Drive dead, drive reported, drive ordered, drive replaced, then no one the wiser. Data saved, not dead. Backups not tested. Not today. No, not today. Tech support treats me like the imaginary newbie IT people sometimes treat everyone with. He repeats his instructions to me. He is polite. Download the report. I can’t. It won’t work. Unplug the unit. Plug it into something else. I can’t. The cables. The Thunderbolt cables are everywhere. It’s magical, and daisy-chained, and stuck. “Can I remote in and see?” he asks, hoping to resolve this quickly. Sorry. That’s impossible. Even if I thought it was a good idea. I remind him that I have a dead drive. That’s why I called. I want to get a replacement drive. “Sorry sir that legacy unit is not under support most likely,” I know that. I realize that now. That I wasted my time. It happens sometimes. The truth is staring at you. You need a mirror to see. “You need to order compatible drive from the compatibility list.” I am a well spring of emotions. I thank him. I am nice. He was polite. But now I know what I need to do. Oh wait, what? Order a drive now! Order two.

McLuhan never had a chance to evaluate Thunderbolt storage technology but the insane genius and simplicity of Thunderbolt reduced expensive enterprise fibre channel storage to the dust bin. Magical SAN for video editing with a Mac Mini and Thunderbolt RAIDs. Cheap enough to buy with a departmental credit card, fewer meetings to attend, more films to shoot and edit. Backup, archive, repeat. McLuhan would have no doubt reminded me that the tetrad of technology would have flipped Thunderbolt on its head, Fibre channel never went into a dust bin, but was firmly relegated to well cooled storage room, and long ago legacy drives in the enterprise units are humming a long while the cursing wind and emotions swell over the Thunderbolt mountain. Fibre channel just became cool again. Retro smart.

Best of 2015: Archiware P5 Archive app

Announced late in 2015 the Archiware P5 Archive app is a revolution for editors who want to control the archive and restore process. No longer the job of the IT Admin, editors can select files or folders on their SAN volume (or anywhere) and send them to the tape archive.

The Archive app is a brilliantly simple app that allows the right-click services action in OS X, or in another words a it’s a GUI app that presents a contextual menu that knows to how to the talk to your P5 Archive server. When the files are safely on tape the original files on the filesystems are replaced with stub files that can be used to start the restore process.

Requirements: Archiware P5 server with the Archive module setup with an Archive plan. Add to that the P5 Archive App which is installed on the clients.

Note: At the moment all archiving goes over the LAN by default, so if you have a fast SAN then you set up the P5 Archive app client settings as “localhost” instead of their actual client name. That means that when it goes to archive the file, the server knows that the files exist on the SAN at a known path (which is the same on the client and the server).

And now for some detailed steps and screenshots.

  1. Archiving completed projects

Choose the completed project folder and right-click. Select “Archive to P5”.

Note 1: If you want to restore files choose the folder that was archived and right-click. Choose “Restore from P5”.

Note 2: Restoring individual files that have been archived is possible by double-clicking the files with the “.p5a” extension, but it will be much faster to select an entire folder to restore than many individual files.

Note 3: For either archive or restore to work the P5 Archive app needs to be installed.

Note 4: To avoid having a services sub menu keep the contextual-menu items to four.

Right-click folder to archive

2. Archiving app status

When you are archiving or restoring files the Archive app will show you the status of your request. It will also show you the status of other jobs running on the P5 server. This is to let you know why perhaps your archive or restore is taking a long time (it’s possibly waiting for access to the tape drive and it currently busy backing up or archiving something else).

P5 Archive app Running jobs status

The P5 Archive app offers you three operations “cancel job”, “list items” and “get report”. The last two are great when you want to examine a completed job, for example. If you want to find out what files were archived in the particular job choose “list items”.

3. Restoring files

Archived files will have either one of or both of, 1) a”.p5a” file extension and 2) a P5 Archive app icon.

Folders and FCP X project bundles (which are folders) do not get the “.p5a” extension, but FCP X projects have the the icon.

p5a-icon.png

Note 1: Files can also be restored by the admin through the P5 web interface. They can be restored in place or to any other location that is required.

Note 2: On the P5 server jobs that are sent to archive or restored from tape show up as “cli job” with the tapes in use.  Actual files or folders involved need to be noted from the P5 Archive app not the P5 web admin console. Otherwise checking the P5 web restore tab will files actually archived (that can be restored).

That’s enough for the quick overview of this great new app. One of the best things in 2015.

For more information on Archiware’s new P5 Archive app check out their website:

P5 Archive app

2015 in review

The WordPress.com stats helper monkeys prepared a 2015 annual report for this blog, but nobody wants to read no stinkin’ reports so let me just sum it all up: Xsan, Munki, Thunderbolt, Archives. Or is that all one word? Thunderbolt Xsan Munki Archives! That’s better.

Here’s an excerpt from the report that no one will read:

The concert hall at the Sydney Opera House holds 2,700 people. This blog was viewed about 20,000 times in 2015. If it were a concert at Sydney Opera House, it would take about 7 sold-out performances for that many people to see it.

Surprisingly, or not, that opera would be about Xsan. Yes, Apple’s Xsan is still alive, and Apple even added new features with OS X 10.11 El Capitan. I’m still building Xsan shared storage SANs and upgrading old ones to new versions. That was one of the good news stories of 2015 for me.

You can build an Xsan with one or two Mac Minis and add your storage of choice. That used to mean more often than not the fibre channel storage from Promise. A great choice for larger deployments, the x30 Vtraks are solid.

But the real shocker for me in 2015 was stumbling upon the Accusys Thunderbolt SAN RAID, the A16T2-Share. For more than half off the price of a similar fibre channel storage RAID here’s a magical box powered by unicorns that has four (4) Thunderbolt connections. Plug one Thunderbolt cable into that Mac Mini, format the raid, setup Apple’s Xsan, and then plug the other three (3) Thunderbolt cables into iMacs, Mac Pro, MacBook Pros or any Xsan clients. Wow. Awesome.

Suddenly we have a game changer. An affordable SAN storage RAID for real block-level storage. Now more than ever we can afford to have true collaborative workflows for video editors and anyone in the creative. If you need to work together with fast connections to a shared pool then building an Xsan got much more attractive.

Disclaimer: I got a chance to test the Accusys A16T2-Share. And I would be crazy to recommend something without testing it thoroughly. This was used for several weeks by video editors in production. It was much faster than our 4GB fibre channel storage, of course, but it was also faster than our 8GB FC storage. Speed tests showed we got close to 1GB/sec, and even when it was 97% full we got 700MB/sec. Sa-weet.

I look forward to seeing what Accusys bring to NAB in 2016. What new box will they show up with? I hope for more than 4 client ports and faster Thunderbolt 3. Only 82 more sleeps till we all find out.

Apple’s Xsan and Accusys Thunderbolt storage A16T2-Share were big stars of 2015, but what else stood out? The two other bright shiny lights were Archiware’s new P5 Archive app, and Vidispine’s VidiXplore cloud based MAM. More on those in posts to follow. Both of these products have transformed workflows for editors. Stay tuned!

Thunderbolt SAN talk at Mac Admin meet up

Big thanks to Ross at Ping Identity for organizing and Jamf for sponsoring the Mac Admin meetup on September 9, 2015.

We filled the tiny meeting room and we will have to expand to the larger conference room (or theatre) next time. It was a well attended meetup with much discussion of the earlier day’s Apple announcements, new OS X “El Capitan” and iOS 9 changes and how this affects management products like Casper which have had to move the binary because of the new SIP implementation in OS X.

I opened up the meet up with a presentation on Storage, SANs and the new Accusys Thunderbolt SAN A16T2-Share product.

The goal of my presentation was to give a quick overview of SAN technology as I’ve seen it change over the last 10 years: from Fibre Channel, to iSCSI to PCIe and Thunderbolt based. The last change to Thunderbolt based SANs is the most interesting for small video production workgroups or anyone that likes working on small scale shared projects but needs a decent bandwidth at an affordable price. Block level storage (SANs) is straight forward storage tech for users and applications to interact with without having to negotiate network protocols (AFP, SMB, or NFS). It’s never been quite that affordable until now.

Having built a lot of Fibre Channel based SANs for media and entertainment companies and post-production editors in corporate environments I know how awesome and fast and solid these SANs are. Lots of editors and clients can hit a large SAN and it won’t blink. Thirty or Sixty users is not unusual. But not everyone believes in fibre channel or the idea of pulling fibre cables. It is surprisingly a large stumbling block to building large SANs, “no, we don’t want fiber cables”. True, sometimes clients have objected to gigabit Ethernet too, but that’s another story.

I found that iSCSI, especially with the DDP units I’ve set up, has been a great alternative to fibre channel. Not fiber cables to pull. Just use the CAT6 cables already in place. Great Ethernet based SANs using 1 x or 2 x CAT6 cables per client, or even 10G. Works well. Very well indeed. It’s been great for smaller (and larger) clients who want a great Ethernet iSCSI SAN solution without needing fibre channel cables, switches, HBAs, Thunderbolt adapters, etc.

That’s why when I stumbled across the Accusys Thunderbolt storage I was kinda really excited. No fibre channel to Thunderbolt adapters. Just use Thunderbolt cables. Brilliant! Finally a solution for small workgroups. And there’s so many video groups sprouting out of every corporate office, or boutique VFX or post-production shops that have been struggling with small NAS solutions that were not meant for video production. Now you can get that SAN that you’ve wanted, you can really get that block-level storage at an affordable price. Instead of working locally and copying raw footage and finished products  back and forth across slow network links they can work in a small video group with high speed storage. Sa-weet. (Can you tell I’m excited?).

I’ll include the presentation PDF here as a link if anyone is interested. I’ve added a link at the end from Accusys on how to build an Xsan with the A16T2-Share. Yes, Xsan from Apple still exists and is bundled with the OS for free. Building a SAN is pretty easy and everyone can do it. Don’t forget your backups though.

Lastly, anyone interested in attending any meetings for the upcoming MacDevOps:YVR (June 16-17, 2016) drop me a note. I added the email in the presentation document.

MatX_SAN_Accusys-Thunderbolt_2015