P5 on the Jellyfish: Archiving Gotchas

TL;DR

Using Archiware P5 to Archive files to tapes is awesome, but watch out for little things you might miss, such as the path to the files and backing up your Archive Db.

P5 Archive on the Jellyfish

Using P5 Archive with the Lumaforge Jellyfish is a great way to preserve your digital archives. See this post for how to set up P5 on the Jellyfish

Using Archiware P5 for archiving makes sense. You want your completed projects and original camera footage on LTO tape. But how do you do archives? There are several different ways, and there be gotchas.

P5 Archive vs P5 Archive app

Using P5 Archive to manually archive completed projects to LTO tape is a process of logging into the server via a web browser and selecting the the project folder you want to archive to tape.

The completed project folder could be on the storage visible to the server or it could be storage the client sees. And that can make a difference. Where the storage is mounted is different on a Mac vs Linux. Its’ the difference between “/Volumes” and “/mnt”.

The same Jellyfish storage, either SMB or NFS, when seen on a Mac is mounted by default at “/Volumes” (this can be changed but for most people leave it at the default). But when archiving the storage via a Jellyfish client you will get “/mnt” path.

p5-smb-test2.png

Using the P5 Archive app, which is a Mac only companion application to P5 Archive, to run the archives you will see the storage archived as “/Volumes”.

This first Archiving gotcha is if you’re archiving the Jellyfish storage with the web application of P5 Archive you will have to find your footage and restore from the “/mnt” path vs if you’re archiving from the P5 Archive app which is running from a Mac and will see and store the footage using the “/Volumes” path.

All this to say that using both ways to archive may double up your footage in your archive which may be unintended. And from a restore in the web browser finding your footage may be confusing if you’re used to seeing it mounted in “/Volumes” and you actually find it under “/mnt”.

Note: the reason to use the P5 Archive app is because of the simplicity of right-clicking files in the finder which are on your storage and telling them to archive right then and there. Files are copied to tape then the original files on the storage are replaced with stub files. Right-click again to restore. Simple.

p5-archive-app-job-monitor.png

Backup your Archive!

Don’t forget to backup your archives. Or rather, your archive Db. A more recent addition is the ability to automate the backups on the Archive index, so don’t forget to enable it.

In the managed index section, choose your Archive index.

Set the target client where the backups are going and the backup directory. Choose a time and don’t forget to enable it (check the checkbox and hit apply before closing the windows).

Note: Repeat this setup for each Archive index you want to backup.

Archive Backup db setup3.png

Monitoring your Archive!

Don’t forget to enable email notifications for your P5 server to get your inbox full of status notifications and errors and other important stuff. But if you want to cut down on email notifications or you have multiple P5 servers (many different clients, perhaps), then you might want to check out Watchman Monitoring and the P5 plugin that is built-in). Find out easily when your tape pools are getting low, the tape drives needs to be cleaned, the support maintenance needs renewing etc. All in one dashboard. How convenient!

Maybe everything is going well…

Watchman-P5-info.png

Or maybe not!

Archiware-P5-Jobs-Watchman-tapes-required.png

 

macOS Server is dead. Long live macOS.

Yes, it’s been a hot topic in the MacAdmin community both on Mac Enterprise list (oh no it’s the end of the world!) and MacAdmins Slack (told you it was coming, don’t be surprised).

My professional opinion is: “Don’t panic!”

My MacDevOps conference is all about supporting MacAdmins who have been writing code as infrastructure to manage Macs. And do it while replacing macOS Server in the server room with Linux and other OS.

Xsan is staying in macOS Server so I am happy and that’s my main use for the Mac Mini and macOS Server.

I have other Mac Minis doing file sharing for small work groups and moving that out of Server.app in the last revision was unfortunate (it is in the standard OS and usable there but less manageable). There’s also Synology and QNAP NAS for small workgroup file sharing and so much more And many enterprise storage vendors for larger setups.

Imaging has been dying a slow death for years and has been replaced with a thin or “no imaging” concept supported by tools such as AutoPkg and Munki.

Profile Manager is a demo version of MDM and should not be used to actually manage Macs.

Wikis, DNS and Mail should be hosted on Linux, in VMs, AWS, GCP or anywhere other than macOS Server so no problem.

Overall it might be disconcerting to some. But change is constant. And especially at Apple change comes fast and often. We have to get used to it.

Reference:

Apple Support article

Apple to Deprecate Many macOS Server Services – TidBITS http://tidbits.com/e/17760

Root Me Baby One More Time!

UPDATE: Apple has posted a security update. 2017-001

Root-a-pocalyse. Root down. Root a toot toot. Many funny tweets today about a very serious issue. A bug was discovered in macOS 10.13 that enabled anyone to login with a root account. With no password. Wow. Seriously. Yeah, that’s bad.

Bug discovered by Lemi Orhan Ergin.

I tested by clicking on the lock icon in System Preferences. Normally this requires an admin account. I was able to authenticate with “root” and no password. This actually also set root to no password. You can choose a password here and this makes it for you. How convenient. You can also login to the Mac via the login window. With root. And no password. Crazy.

If your Mac is off it’s safe. Not joking. If your FileVault protected drive is encrypted and your mac is turned off then you’re good. If you Mac is turned on and you’ve logged in at least once (or at least decrypted the drive on boot) then you’re not safe.

What can you do? Change the root password and set the shell to false. Until Apple fixes this. Should be anytime now. Or soon.

dscl . -passwd /Users/root “random or very secure password here”

dscl . -create /Users/root UserShell /usr/bin/false

Read a comprehensive explanation on Rich Trouton’s site:  Der Flounder blog

 

I don’t get High — Sierra!

Friends don’t let friends install macOS High Sierra in production. Don’t get High, Sierra.

macOS 10.13 was released on Sep 25, 2017, and almost two months later with only one point release update, it’s still too new for production. Download it on a test machine or two or more, test it with your apps and systems, file bug reports and radars, but for the love of all that is Python and Monty! don’t run it on your production Xsan. Well, at least not yet. Wait until next year. Or as long as you can. Or until the new iMac Pro is released with 10.13 pre-installed or wait until they ship the new Final Cut Pro X 10.4 that may or may not require macOS High Sierra.

With that out of the way, I’ve just upgraded the production Xsan to … macOS Sierra. Yes, macOS 10.12.6 is stable and it’s a good time to install last year’s macOS release. Time to say good bye to macOS el Capitan 10.11.6, we hardly knew ya. Besides guaranteed security updates, stability and the annoying newness of a changed macOS, what else is there? In Xsan v5 they introduced a new “ignore permissions” checkbox for your Xsan volumes. Looking forward to that feature in production. No more Munki onDemand nopkg scripts to run chmod. No more tech support requests for folders, files, FCP X projects that won’t open because someone else used it, owns it, touched it. We’ll see how that pans out. I’ll let you know.

Upgrading Xsan to v5

Step 1. Back up your data

You’re doing this, right? I’m using Archiware P5 Backup to backup the current projects to LTO tape. I’m using Archiware P5 sync to sync the current Xsan volumes to Thunderbolt RAIDs, and using Archiware P5 Archive (and Archive app) to archive completed projects to the LTO project archive. That’s all I need to do, right?

Step 2. Back up your servers

Don’t forget the servers running your SAN! I use Apple’s Time Machine to backup my Mac Mini Xsan controllers. External USB3 drive. I also use another Mac Mini in target disk mode with Carbon Copy Cloner to clone the server nightly. (Hat tip to Alex Narvey, a real Canadian hero). And of course I grab the Xsan config with hdiutil and all the logs with cvgather. Because, why not?! For Archiware P5 backup server I also have a python scripts to backup everything, another scripts to export a readable list of tapes, and BackupMinder to rotate the backups. Add some rsync scripts and you’re golden.

 

Step 3. Upgrade the OS

Unmount the Xsan volume on your clients or shut them down, disconnect the fibre channel. Do something like that. Stop your volume. Download the macOS Sierra installer from the App Store. Double click upgrade. Wait. Or use Munki. I loaded in the macOS 10.12.6 installer app into Munki and set it up as an optional install to make this portion of the upgrade much quicker and cleaner.

In my case after the OS was upgraded I checked the App Store app for any Apple updates (you can also use Munki’s Managed Software Center to check) and of course there were some security updates. In this case the security upgrade hung on a slow network connection and the server crashed. Server down! I had to restore from Time Machine backup to the point where I just upgraded the server. It took some extra time  but it worked (can’t wait for next year’s mature APFS / Time Machine and restoring from snapshots instead).

Step 4. Upgrade Server

After macOS is upgraded you’ll need to upgrade the Server.app or just upgrade the services used by Server (even those not used by Server get upgraded).

Step 5. Upgrade the Xsan

Bur first we have to restore the Xsan config. Don’t panic! It may invoke bad memories of data loss and restoring from backups. Xsan PTSD is real.

Restore-previous-Xsan.png

Step 6. Upgrade the rest

Next you have to upgrade the Xsan volumes.

Xsan-volume-needs-upgrade

New version of Xsan, ch-ch-changes! Ignore permissions check box will remount the xsan with the “no-owners” flag. Let’s test this out.

 

Upgrade the OS and Server app on the backup controller. Upgrade the OS on the clients using Munki or App Store if you like doing it the hard way. Ha Ha.

Step 7. Enjoy

Plug those Thunderbolt to Fibre adapters back in, mount those Xsan volumes and be happy.

Step 8. Wait for the complaints

The next day the editors walked in and went straight to work with Final Cut Pro X. No one noticed anything. Xsan upgraded. Workstation macOS upgraded. Everything appeared to be the same and just worked. Thankless task but well worth it.

 

Reference: Apple’s iBook guide here

 

 

Be a NoMAD!

 

NoMAD stands for “no more AD” and has nothing to do with a nomadic lifestyle, nomads, ronin or other wandering IT professionals. Sorry.

NoMAD allows you to stop binding Macs to a corporate domain and instead get your kerberos tickets as needed. Connect to those file shares, change your password, and other fun tasks, without being stuck on the domain and constantly resetting your keychain from the insanity of password retention policies.

NoMAD-intro

Using Autopkg and Autopkgr to feed trusted apps into your Munki repo you can easily deploy NoMAD to your fleet of Macs.

And for bonus points you can add your preference settings as “updates for” NoMAD in Munki. One such add on is a setting for an auto mounting sharepoint.

Name your file: “menu.nomad.shares.plist” and open up your favourite text editor.

<?xml version=”1.0″ encoding=”UTF-8″?>
<!DOCTYPE plist PUBLIC “-//Apple//DTD PLIST 1.0//EN” “http://www.apple.com/DTDs/PropertyList-1.0.dtd”&gt;
<plist version=”1.0″>
<dict>
<key>Shares</key>
<array>
<dict>
<key>AutoMount</key>
<true/>
<key>ConnectedOnly</key>
<true/>
<key>Groups</key>
<array/>
<key>LocalMount</key>
<string></string>
<key>Name</key>
<string>Corp_Share</string>
<key>Options</key>
<array/>
<key>URL</key>
<string>smb://winserver5000/Corp_Share</string>
</dict>
</array>
<key>Version</key>
<string>1</string>
</dict>
</plist>

Create a package with munkipkg and add this to Munki. Set the package as an update for Munki and as your NoMAD agent gets installed your updates for NoMAD go with it.

More tips and tricks in the future.

 

My Thunderbolt Nightmare

It was a dark and stormy night of cables and capacitors when suddenly I heard the door knocking, or was something falling of a shelf? I was in a cramped server room, if you’d call it that, and I was day dreaming, sorry, night terrorizing, of days gone past when I worked in nice big well ventilated server rooms with proper enterprise gear. Oh wait, did I really dream that? Did it really happen? Maybe it was less well ventilated and there were cables strewn about the tall 42U shelves and sometimes we found a Mac hidden underneath spaghetti. Sometimes. I vaguely remember the long shiny metal servers, they talked to me, they sang, a whiny pitch of whale song. Dream on, dream on.

Now. Today. Apple Music on my iPhone plays every single Arcade Fire album in a long playlist, in order. And I follow the white rabbit of Thunderbolt cables. This is my thunderbolt nightmare. Dead drive in a Thunderbolt Promise Pegasus unit, web ticket filed for registered hardware. Legacy. That’s the word they used. Where’s Marshall McLuhan when you need a proper redux of the shit storm you’re in? Thunderbolt 3 uses USB-C and everything is possible. Can’t wait to step into that confusing identity crisis. OK, back to the present day when I stared at the red blinking drive, a replacement drive from not long ago dead again, sitting in the last row of a now legacy Pegasus R6 unit. RAID 5, the most dangerous kind, this is what stood between me and uncertainty. The worst kind of RAID. Well, not as bad RAID 0. Raid nothing. Raid 5 is one bad drive away from a bad day. Backups? Hmm, I got those, I got plenty of those, but I don’t want to be tested today. No, not today. Not this bloody day.

I open the Pegasus utility and the GUI wants an update. Hmm, that’s not in autopkg, I think. Why is out of date? Munki let me down. I start to drift, to side shift into adding newer better recipes to autopkg, to tweaking my Munki repo, to what sessions would be awesome at the next MacDevOps:YVR conference. Gee whiz, I love open source, and everyone in the Mac Admins community…. Snap out of! I slap myself in the face. I was hallucinating. Stay on task. I update the Pegasus utility. I stare at the critical reports from one of the three R6 units attached to this Mac Mini server. Did I say server? But it’s so small, so little. It works. It’s magical, kinda neat. Until you stare too close at the back. The Thunderbolt cables go from the Mac Mini to the first Pegasus unit to the SANlink fibre channel adapter to the LTO 6 tape library to the next Pegasus utility to the second SANlink adapter to a third and final Pegasus RAID unit. What’s is going on? Where does this cable go? Let me just follow it to the next jumping off point. My brain slows, the lack of oxygen in this cold machine room start to affect my thinking. I lose my way.

I download the report for the Pegasus unit. I had to unlock a pretty neat lock icon and click on the save report. I upload it to the web support and add it to the ticket. Tech support gets backs to me in a day and said all is good, and to carry on. I can’t. The drive is dead. What are they not seeing? It’s right in front of me. I download the report again. Again the same response. Fine. It’s time to stop messing around and pop open Terminal. Loading up promiseutil I check out the options and switches and get into an argument with myself about the currently valid optionals of letters and numbers that are required. I check my notes, online knowledge base, and try again. It’s broken. It doesn’t work. Stumbling around the command line typing imprecisely incorrect statements gets nowhere fast. I realize that there’s no way for the cli utility to properly change its focus to the broken unit with the busted drive. Both the GUI and the binary are stuck on the one R6 unit and won’t see what’s in front of my face.

I call tech support. This is humiliating. This was supposed to be easy. Drive dead, drive reported, drive ordered, drive replaced, then no one the wiser. Data saved, not dead. Backups not tested. Not today. No, not today. Tech support treats me like the imaginary newbie IT people sometimes treat everyone with. He repeats his instructions to me. He is polite. Download the report. I can’t. It won’t work. Unplug the unit. Plug it into something else. I can’t. The cables. The Thunderbolt cables are everywhere. It’s magical, and daisy-chained, and stuck. “Can I remote in and see?” he asks, hoping to resolve this quickly. Sorry. That’s impossible. Even if I thought it was a good idea. I remind him that I have a dead drive. That’s why I called. I want to get a replacement drive. “Sorry sir that legacy unit is not under support most likely,” I know that. I realize that now. That I wasted my time. It happens sometimes. The truth is staring at you. You need a mirror to see. “You need to order compatible drive from the compatibility list.” I am a well spring of emotions. I thank him. I am nice. He was polite. But now I know what I need to do. Oh wait, what? Order a drive now! Order two.

McLuhan never had a chance to evaluate Thunderbolt storage technology but the insane genius and simplicity of Thunderbolt reduced expensive enterprise fibre channel storage to the dust bin. Magical SAN for video editing with a Mac Mini and Thunderbolt RAIDs. Cheap enough to buy with a departmental credit card, fewer meetings to attend, more films to shoot and edit. Backup, archive, repeat. McLuhan would have no doubt reminded me that the tetrad of technology would have flipped Thunderbolt on its head, Fibre channel never went into a dust bin, but was firmly relegated to well cooled storage room, and long ago legacy drives in the enterprise units are humming a long while the cursing wind and emotions swell over the Thunderbolt mountain. Fibre channel just became cool again. Retro smart.

Screen recording and other tricks

QuickTime has a neat little trick that some may not know about, it can record your screen.

QT-ScreenRecord

Use it to record a how-to video how to navigate System Preferences, or how to use Final Cut Pro, or record a MacDevOps:YVR talk.

The first two MacDevOps:YVR conferences needed to be converted to a suitable format for YouTube and using QuickTime screen recording + Soundflower is the way I chose to do it.

Note: Soundflower is needed to redirect the audio to QuickTime. Screenrecording with QuickTime does not capture the audio without Soundflower.

MacDevOps Screen recording steps

  1. Install Soundflower (Soundflower-2.0b2.dmg)

https://github.com/mattingalls/Soundflower/releases/tag/2.0b2

  1. Set audio output to SoundFlower 2chSoundFlower
  2. Set QT screen recording to Soundflower audio QT-ScreenRecord-SoundFlower-2ch
  3. Start screen recording (select screen area)
  4. Play website audio / video (Safari / Other )
  5. Stop both. Edit and trim QT video as needed
  6. Upload to YouTube
  7. Tag video (mdoyvr, yvr, MacDevOps, MacAdmin, MacIT), put in proper playlist
  8. Publish

Addendum:

I own Rogue Amoeba’s excellent Audio Hijack application and have used this app for audio capture (podcast interviews, etc), but I couldn’t get it to work in this case. It might have also required their Rogue Amoeba’s Loopback app which I did not own. Since I’ve used Soundflower previously I used it here in this case.