My Thunderbolt Nightmare

It was a dark and stormy night of cables and capacitors when suddenly I heard the door knocking, or was something falling of a shelf? I was in a cramped server room, if you’d call it that, and I was day dreaming, sorry, night terrorizing, of days gone past when I worked in nice big well ventilated server rooms with proper enterprise gear. Oh wait, did I really dream that? Did it really happen? Maybe it was less well ventilated and there were cables strewn about the tall 42U shelves and sometimes we found a Mac hidden underneath spaghetti. Sometimes. I vaguely remember the long shiny metal servers, they talked to me, they sang, a whiny pitch of whale song. Dream on, dream on.

Now. Today. Apple Music on my iPhone plays every single Arcade Fire album in a long playlist, in order. And I follow the white rabbit of Thunderbolt cables. This is my thunderbolt nightmare. Dead drive in a Thunderbolt Promise Pegasus unit, web ticket filed for registered hardware. Legacy. That’s the word they used. Where’s Marshall McLuhan when you need a proper redux of the shit storm you’re in? Thunderbolt 3 uses USB-C and everything is possible. Can’t wait to step into that confusing identity crisis. OK, back to the present day when I stared at the red blinking drive, a replacement drive from not long ago dead again, sitting in the last row of a now legacy Pegasus R6 unit. RAID 5, the most dangerous kind, this is what stood between me and uncertainty. The worst kind of RAID. Well, not as bad RAID 0. Raid nothing. Raid 5 is one bad drive away from a bad day. Backups? Hmm, I got those, I got plenty of those, but I don’t want to be tested today. No, not today. Not this bloody day.

I open the Pegasus utility and the GUI wants an update. Hmm, that’s not in autopkg, I think. Why is out of date? Munki let me down. I start to drift, to side shift into adding newer better recipes to autopkg, to tweaking my Munki repo, to what sessions would be awesome at the next MacDevOps:YVR conference. Gee whiz, I love open source, and everyone in the Mac Admins community…. Snap out of! I slap myself in the face. I was hallucinating. Stay on task. I update the Pegasus utility. I stare at the critical reports from one of the three R6 units attached to this Mac Mini server. Did I say server? But it’s so small, so little. It works. It’s magical, kinda neat. Until you stare too close at the back. The Thunderbolt cables go from the Mac Mini to the first Pegasus unit to the SANlink fibre channel adapter to the LTO 6 tape library to the next Pegasus utility to the second SANlink adapter to a third and final Pegasus RAID unit. What’s is going on? Where does this cable go? Let me just follow it to the next jumping off point. My brain slows, the lack of oxygen in this cold machine room start to affect my thinking. I lose my way.

I download the report for the Pegasus unit. I had to unlock a pretty neat lock icon and click on the save report. I upload it to the web support and add it to the ticket. Tech support gets backs to me in a day and said all is good, and to carry on. I can’t. The drive is dead. What are they not seeing? It’s right in front of me. I download the report again. Again the same response. Fine. It’s time to stop messing around and pop open Terminal. Loading up promiseutil I check out the options and switches and get into an argument with myself about the currently valid optionals of letters and numbers that are required. I check my notes, online knowledge base, and try again. It’s broken. It doesn’t work. Stumbling around the command line typing imprecisely incorrect statements gets nowhere fast. I realize that there’s no way for the cli utility to properly change its focus to the broken unit with the busted drive. Both the GUI and the binary are stuck on the one R6 unit and won’t see what’s in front of my face.

I call tech support. This is humiliating. This was supposed to be easy. Drive dead, drive reported, drive ordered, drive replaced, then no one the wiser. Data saved, not dead. Backups not tested. Not today. No, not today. Tech support treats me like the imaginary newbie IT people sometimes treat everyone with. He repeats his instructions to me. He is polite. Download the report. I can’t. It won’t work. Unplug the unit. Plug it into something else. I can’t. The cables. The Thunderbolt cables are everywhere. It’s magical, and daisy-chained, and stuck. “Can I remote in and see?” he asks, hoping to resolve this quickly. Sorry. That’s impossible. Even if I thought it was a good idea. I remind him that I have a dead drive. That’s why I called. I want to get a replacement drive. “Sorry sir that legacy unit is not under support most likely,” I know that. I realize that now. That I wasted my time. It happens sometimes. The truth is staring at you. You need a mirror to see. “You need to order compatible drive from the compatibility list.” I am a well spring of emotions. I thank him. I am nice. He was polite. But now I know what I need to do. Oh wait, what? Order a drive now! Order two.

McLuhan never had a chance to evaluate Thunderbolt storage technology but the insane genius and simplicity of Thunderbolt reduced expensive enterprise fibre channel storage to the dust bin. Magical SAN for video editing with a Mac Mini and Thunderbolt RAIDs. Cheap enough to buy with a departmental credit card, fewer meetings to attend, more films to shoot and edit. Backup, archive, repeat. McLuhan would have no doubt reminded me that the tetrad of technology would have flipped Thunderbolt on its head, Fibre channel never went into a dust bin, but was firmly relegated to well cooled storage room, and long ago legacy drives in the enterprise units are humming a long while the cursing wind and emotions swell over the Thunderbolt mountain. Fibre channel just became cool again. Retro smart.

Archiware P5 and Synology NAS.

Update: As of version 5.4.3 there is an official P5 add-on package for Synology NAS

Archiware P5 available for Synology

Note: The P5 app for Synology NAS first debuted with P5 v.5.3.3

On the Archiware P5 new-features page there’s a blurb about the Synology NAS integration:

From Version 5.3.3, Archiware P5 supports Synology NAS devices without restrictions.  

Synology NAS can serve as a data source or target for P5 Synchronize, P5 Backup and P5 Archive. The Archiware P5 application can now be installed on the Synology NAS itself.

Thanks to the snapshot capability of the DSM platform, powerful enterprise Synology NAS devices can also be used as repository for Backup2Go. This setup opens the possibility of introducing a professional data security solution at an affordable price point.

Let’s look in closer detail how to install Archiware P5 on a new Synology NAS.

For this post I have a new Synology 1515+ NAS, installed with five 6TB hard drives (It is very easy to install hard drives. No tools required). Note: I’ve purchased the NAS with my own money and was not paid to write this article.

At the time of this blog post the latest Synology DSM release is 6.1 and Archiware P5 is at version 5.4.2.

Step 1. Download Synology package from Archiware.com/download

Download Archiware P5 for Synology

 

awpst542spk

Requirements are DSM 5.2+ and Intel x86 64-Bit CPU only. (i.e. Atom but not Marvell).

Step 2. Find and Log into your NAS

Find your new NAS with the Synology Assistant app or use this handy website link:

Find your NAS

I had no luck with the app (it found my existing NAS, but not the new one). Using the website I was able to quickly locate the new NAS that I need to log into and setup. Very nice feature.

synology-1515-setup-welcome2crop

Step 3. Install the new DSM

Install or update new software. You will be prompted to go through the initial setup to prepare your new NAS.

synology-1515-install-diskstation-manager2

Step 4. Set up a new volume

Chose the Btrfs or ext4 filesystem. Btrfs supports snapshots, replication, and much more.

synology-1515-btrfs-setup

Step 5. Monitor the volume setup

Verifying the hard disks will take a moment. Take a break here.

synology-1515-storage-manager

Step 6. Open Package Center

packagecenter

Step 7. Install manually

Install Archiware P5 by selecting the “install manually” option to upload the awpst542.spk downloaded file from archiware.com

synology-1515-archiware-p5-package-center-upload

Step 8. Agree to continue.

Load the Synology P5 installer by agreeing to continue with this “unknown” publisher.

synology-1515-archiware-p5-package-center-unknown

Sep 9. Agree to trust the installer

synology-1515-archiware-p5-package-center-license

Step 10. Confirm the Install

synology-1515-archiware-p5-package-center-confirm-install

Step 11. P5 is now running on the Synology NAS.

Hooray! P5 is now installed. Select the app to examine the details.

synology-1515-archiware-p5-package-center-installed

synology-1515-archiware-p5-package-center2

 

Step 12. Examine the option to stop or uninstall the P5 application

synology-1515-archiware-p5-package-center-stop-uninstall

Step 13. Login to the P5 server running on NAS

To login to P5 open a new tab. Pay attention to the port number: “20,000” (vs 8000 on other platforms such as Solaris, Linux, OSX etc).

synology-1515-archiware-p5-port

 

Step 14. Set up your NAS as a client on another Server

To test the new Synology 1515+ NAS I then set up the NAS as a client on another P5 server, and set up a P5 Sync job to copy data from server with a ZFS based filesystem to the Synology NAS with a btrfs volume.

Testing: Set up the new client in P5 with a name and IP address, then set up a new sync job with source and destination. Start now. Watch the bits fly through the ether. Be happy.

Step 15. Other things to configure

To make your new NAS is working smoothly don’t forget to set up the email notifications, and set up some AFP, SMB, or NFS shares as required.

Take some time to explore the Package Center app and see what other great applications are offered on the Synology NAS.

Synology makes a great low-cost NAS appliance. For SMB or production setups I would recommend two or more (for redundancy, hot or cold spares, disaster recovery, offsite backups/replication). With P5 installed you can Sync your server data to a NAS for onsite or offsite backups, backup your NAS to tape, or use the NAS for your client workstation backups using Backup2Go. Using the new Btrfs filesystem provides many of the same advances as ZFS, including snapshots and replication, over traditional filesystems such as ext4 and hfs which sadly lack these features.

Conclusion:

The Synology NAS is a great experience. Adding Archiware P5 is a recommended way to include this NAS as part of any good backup, archive or DR (disaster recovery) scenario. Two thumbs up. Way up.

References:

Archiware P5 new features

Synology DSM

Move over El Capitan, hello Yosemite!

With all this talk about El Capitan, Apple’s as of yet unreleased version 10.11 of OS X, and its wondrous new features in Xsan, I think it might be time to upgrade to last year’s breakthrough version of OS X, Yosemite. Sure, you might be excited by the press releases for the built-in DLC in El Capitan but seriously sane folks stay 1-year behind the bleeding nose upgrades provided by Apple. So if OS X 10.11 is all the rage before its released it must be time to seriously consider upgrading that working Xsan running OS X 10.8 or OS X 10.9.

In my case, I upgraded a working Xsan running on Mac Minis and OS X 10.8.5. Here are some screenshots from the process. As always think worked better than I could have expected, and it is a much easier process that one expects. But stay sharp kids, danger lurks when you wake the dreamer…. Upgrading a SAN is serious business and doing anything like this without proper backups is taking your life in your own hands. In my case, full disk backups on Promise Pegasus RAIDs and full tape backups using Archiware P5.

Download the Yosemite installer form the App Store. Install. Download the new Server.app from the App Store. Install. Now upgrade your Xsan. That’s it. You’re done. No surprises, aren’t you happy? Ha ha. I’m kidding. The fun is just getting started.

If you’re actually following along, this isn’t a step by step recipe. Go to Apple’s site and read this Kbase and check out the migration guide.

Restore Xsan

Restore Xsan

Step 1 is to launch the new Server.app, find Xsan Admin. Just kidding, it isn’t there. Enable Xsan, and choose to Restore a previous SAN configuration. That wasn’t hard. High five! Actually, we’re not done yet. Set up OD now. Go!

Step 2. Set up your Xsan controller as an Open Directory (OD) master. Does’t matter if it’s joined to another domain, Xsan keeps itself organized in OD, so you need it.

Set up OD

Set up OD

Step 3. Admire your upgraded SAN, “how lovely the flowers do smell…. life is good.”

XSAN LIST

Xsan list

Step 4. Where did my Xsan admin go? Where do I add clients? Where are my clients? Huh? What? Why did I upgrade a perfectly working SAN to this version? Ha ha.

Take it all in, take a good look at what you’ve done to your Xsan. What? Just so the editors could have the latest version of Final Cut Pro (v.10.2.1) which is only compatible with OS X 10.10.4. I see what you’ve done Apple, very clever indeed. Hmm…

Click on the “Save configuration profile” button and download the profile somewhere. Use this to set up the SAN on your clients. Distribute via Profile Manager or install it manually. Up to you. I haven’t gotten it to work with Munki quite yet. Installing it requires the admin password for the Xsan controller. How convenient.

When you client is configured you’ll see a Profile in System Preferences. Remove it and your client is un-configured. No more Xsan.prefpane to list volumes and mount or unmount them. Nope. That would be too easy. Learn to love “xsanctl”, as in “xsanctl mount Xsan”. Read some xsanctl tips in this Kbase

Step 5. Set up a backup Xsan controller. You have one of those, right? In my case, I had a client which I wanted to promote to be a controller.  But first what to do about its status a client of the Xsan?

backup cannot be client

backup cannot be client

Open Server.app, enable Xsan, join current Xsan as a backup controller and set up a replica OD. Confirm, confirm, confirm. Think about what you’re doing, then do it!

confirm OD replica

confirm

Apple wizards are the best wizards, uh, i mean Setup Assistants. No wizards here…. So, you’ve setup a backup Xsan controller, and OD replica, and now look in Server.app. How amazing is that… wait, what? Where’d my Xsan volumes go? Huh? Where are the controllers? Weird. Very strange. Not comforting at all.

Xsan 4 no SAN list crop 122815

The Xsan window eventually shows the volumes and controllers, bur geez, almost gave me a heart attack. It’s not like I never seen Xsan go bad before. Xsan 1 nightmare still haunt me. They do. Backups. Need more backups. Archiware P5 Backups, do it now!

OK, you’ve survived the uncertainty of Xsan upgrades…. But wait more minute… cat the fsnameservers (no, it’s not the name of a band, it’s a command). Check it out. Holy smokes, batman. Xsan 4 by default will set your metatadata network to the public LAN, something that’d would be laughed at years ago, but they do it now by default. Of course, upgrading our SAN kept out metadata network the same. But strangely the Xsan backup controller is set to use the public for metadata when the primary controller is not. WTF.

Change your metadata network. Read the Kbase, and once again wield xsanctl like a boss.