Xsan Upgrade and Big Sur Prep. Hello Catalina!

Big Sur summer testing time!

Summer time is beta testing time. A new macOS beta cycle with Big Sur is upon us. Test early, and test often. With all the excitement of Big Sur in the air, it’s time to look at Catalina.

Our day to day production Xsan systems do not run beta software, not even the latest version of macOS, they only run tested and safe versions of macOS. I always recommend being a revision behind the latest. Until now that meant macOS 10.14 (Mojave). With the imminent release of macOS Big Sur (is it 10.16 or macOS 11?) then it’s time to move from 10.14.6 Mojave to 10.15.6 Catalina. It must be safe now, right? 

Background

Xsan is Apple’s based Storage Area Network (SAN) software licensed from Quantum (see StorNext), and since macOS 10.7 aka Lion it has been included with macOS for free (it was $1,000 per client previously!).

Ethernet vs Fibre Channel vs Thunderbolt

A SAN is not the same as a NAS (Network attached storage) or DAS (direct attached storage). A NAS or other network based storage is often 10GbE and can be quite fast and capable. I will often use Synology NAS with 10GbE for a nearline archive (a second copy of tape archive) but can also use it as a primary storage with enough cache. Lumaforge’s Jellyfish is another example of network based storage.

Xsan storage is usually fibre channel based and even old 4GB storage is fast because … fibre channel protocol (FCP) is fast and the data frames are sent in order unlike TCP. It is more common to see 8GB or 16Gb fibre channel storage these days (though 32GB is starting to appear). And while fibre channel is typically what you use for Xsan you can also use shared Thunderbolt based storage like the Accusys A16T3-Share. I have tested a Thunderbolt 2 version of this hardware with Xsan and it works very well. I’m hoping to test a newer Thunderbolt 3 version soon. Stay tuned.

Xsan vs macOS Versions

We’ve discussed all the things that the Xsan is not and now what is it? Xsan is often created from multiple fibre channel RAID storage units but the data is entirely dependent on the Xsan controller that creates the volume. The Xsan controller is typically a Mac Mini but can be any Mac with Server.app (from Apple’s App Store). The existence of any defined Xsan volumes depends on the sanity of its SAN metadata controllers. If the SAN controllers die and the configuration files go with it then your data is gone.  POOF! I’ve always said that Xsan is a shared hallucination, and all the dreamers should dream the same dream. To make sure of this we always recommend running the same version of macOS on the Mac clients as well as the servers (the Xsan controllers). And while the Xsan controllers should be the same or at a higher macOS version level it can sometimes be the opposite in practise. To be sure what versions of macOS are interoperable we can check with Apple’s Xsan controllers and clients compatibility chart and Xsan versions included in macOS for the rules and exceptions. Check the included version of Xsan on your Mac with the cvversions command

File System Server:
  Server  Revision 5.3.1 Build 589[63493] Branch Head BuildId D
   Built for Darwin 17.0 x86_64
   Created on Sun Dec  1 19:58:57 PST 2019
   Built in /BuildRoot/Library/Caches/com.apple.xbs/Sources/XsanFS/XsanFS-613.50.3/buildinfo

This is from a Mac running macOS 10.13

Host OS Version:
 Darwin 17.7.0 Darwin Kernel Version 17.7.0: Sun Dec  1 19:19:56 PST 2019; root:xnu-4570.71.63~1/RELEASE_X86_64 x86_64

We see similar results from a newer build below:

File System Server:
  Server  Revision 5.3.1 Build 589[63493] Branch Head BuildId D
   Built for Darwin 19.0 x86_64
   Created on Sun Jul  5 02:42:52 PDT 2020
   Built in /AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/XsanFS/XsanFS-630.120.1/buildinfo

This is from a Mac running macOS 10.15.

Host OS Version:
 Darwin 19.6.0 Darwin Kernel Version 19.6.0: Sun Jul  5 00:43:10 PDT 2020; root:xnu-6153.141.1~9/RELEASE_X86_64 x86_64

Which tells us that the same version of Xsan are included with macOS 10.13 and 10.15 (and indeed is the same from 10.12 to 10.15). So we have situations with Xsan controllers running 10.13 and clients running 10.14 are possible even though macOS versions are a mismatch, the Xsan versions are the same. There are other reasons for keeping things the macOS versions the same: troubleshooting, security, management tools, etc  To be safe check with Apple and other members of the Xsan community (on MacAdmins Slack).

Backups are important

Do not run Xsan or any kind of storage in production without backups. Do not do it. If your Xsan controllers die then your storage is gone. Early versions of Xsan (v1 especially) were unstable and the backups lesson can be a hard one to learn. All later versions of Xsan are much better but we still recommend backups if you like your data. Or your clients. (Clients are the people that make that data and pay your bills). I use Archiware P5 to make tape backups, tape archives, nearline copies as well as workstation backups. Archiware is a great company and P5 is a great product. It has saved my life (backups are boring, restores are awesome!).

P5-Restore-FCPX.png

Xsan Upgrade Preparation

When you upgrade macOS it will warn you that you have Server.app installed and you might have problems. After the macOS upgrade you’ll need to download and install a new version of Server.app. In my recent upgrades from macOS 10.13 to macOS 10.15 via 10.14 detour I started with Server.app 5.6, then install 5.8 and finally version 5.10.

After the macOS upgrade I would zip up the old Server.app application and put in place the new version which I had already downloaded elsewhere. Of course you get a warning about removing the Server app

 

Xsan-ServerApp-ZipRemovalDetected.png

Install the new Server app then really start your Xsan upgrade adventure.

Serverapp-setup.png

Restore your previous Xsan setup.

This slideshow requires JavaScript.

If everything goes well then you have Xsan setup and working on macOS 10.15.6 Catalina

Xsan-Catalina-Upgrade-Success

Automate those apps. Get some robot love 🤖 ❤️!

If only one person needs an application then I think about using Munki to deploy that app. If more than one person should have it then Munki is definitely the way to automate app deployment. And really, if you’re going to take the time to download an app from a website, mount a disk image or un-pack a ZIP archive, run an installer, type an admin password, close that installer … then for the love of all that is good just put the app into your Munki repo and be done with it. Automate it.

Using Munki to solve problems makes sense. Automation helps everyone in this case. But if you’re putting in one off applications into your Munki repo more often than you need to, you need to get those apps into Autopkg. Using Autopkg recipes to download the latest apps and put them into your Munki repo automatically is an automation love fest, but if your apps don’t have recipes what are you going to do? Manually add your apps to Munki? No way. We need a robot 🤖❤️. Recipe robot, that is.

Using Recipe Robot we can build Autopkg recipes for most apps then add the recipes to the Autopkg community to enjoy. Everyone wins.

I recently created recipes for two important apps in my media workflow: Kyno and Hedge. I’ll show an example of this workflow using Recipe Robot and Munki Admin to demonstrate the workflow.

Step 1. Feed the robot.

Drag and and drop the app you want to create your Autopkg recipes.

RecipeRobot-FeedMe

Step 2. Watch the robot do it’s work

RecipeRobot-start

Step 3. Robot is done. Recipes made.

RecipeRobot-Done

Various type of recipes can be made. I chose download and munki because those are what I am using to automate adding apps to my Munki repo. But there are other options: jss, Filewave, or “install” for example.

reciperobot-options.jpg

Step 4. Run those Recipes

You can use your recipes locally with Autopkg. Run them in Terminal or use Autopkgr , a very nice GUI app for automating the collection and scheduling of recipes. Note: Autopkg and Munki can all be run via cli (command line interface) but for this demo we are showing the GUI apps that are there provided by outstanding members of the community. Many Thanks to them and the contributors to their projects.

Autopkgr-notification

Autopkgr app can send notifications in macOS, emails, or post to your Slack group.

Step 5. See the recipes, Use them wisely

MunkiAdmin-Recently ChangedPKGS

Here is an example of newly imported Kyno and Hedge apps in our Munki repo (via Munki Admin GUI).

MunkiAdmin-Description

Add a display name, choose which catalogs the apps will reside in, and check that the description will help explain what the app is.

References:

Elliot Jordan – Autopkg talk at MacDevOps:YVR

https://youtu.be/Q_cvgGtJ71M

Elliot Jordan – Recipe Robot talk at MacDevOps:YVR

https://youtu.be/DgjO1mfMHtI

 

Compressor Tips and Tricks

 

Issue: Stuck job in Apple’s Compressor app.

Resolution: Remove the historical jobs in your local home folder.

~/Library/Application Support/Compressor/History/V4

Compressor-History2

Note: to get to your home folder hold down the OPTION key and select the Go menu in the Finder.

Compressor is the best sidekick to Apple’s Final Cut Pro X and it gets used a lot. But occasionally something goes awry. It’s software running on a computer. So we troubleshoot. What looked like a stuck running job was mostly leftover evidence of an old job. The Apple support document I found didn’t mention this tip but instead talked about zipping up your settings folder which has all your custom compressor settings for things like YouTube outputs or anything custom. Didn’t seem useful to me to remove but this historical stuff, don’t need it and POOF this solved the issues. It’s not always this easy but something you just take the win and go with it.

Reference:

Resolve an issue in Compressor: Learn how to isolate, troubleshoot, and fix issues in Compressor.

https://support.apple.com/en-ca/HT203476

Zoom in on Privacy and Security

Recent attention on video conferencing app Zoom and security exploits brings attention to the various Privacy and Security settings on your Mac. Currently macOS 10.14.5 Mojave defines microphone and camera settings which should be verified periodically if they’re not being managed by MDM (mobile device management) and even in those case, just to verify.

Zoom update

If you’ve ever had Zoom installed you must launch it and then update it manually, unless you have Munki or other patching solution to manage your Mac.

 

Zoom Enable camera access

If you want Zoom to have access to your camera (useful for video conferencing) then enable it or leave it disabled until the moment you actually need it.

Privacy-Camera-OFF-Settings.pngMaybe this is a good time to review what apps have previously been granted access and disable them or not after you review the situation.

Privacy-MIC2-Settings.png

Check your microphone access as well. What apps are in your list?

Further research:

Check out Objective See’s excellent security tools such as Oversight to protect yourself from unwanted access to your camera.

Also check out this past talk at MacDevOps:YVR 2018 by Kolide’s Zach Wasserman about osquery and at the 11min mark where he talks about another app BlueJeans and how to investigate it with osquery.

The MacDevOps:YVR videos from past talks contain many security related talks as well as other awesome troubleshooting tech talks.

 

 

Use Munki to install a screensaver

Use munki-pkg to package up stuff and make your life easier when managing Macs using munki. Here is an example of installing a screensaver.

Why use munki-pkg? How else do you install stuff using munki, run scripts, and version your testing buildings all in one easy to use application? This is all possible with munki-pkg.

Munki-pkg makes package (PKG) installers, Munki likes pkg installers. Munki will also install apps, run scripts, install profiles, and do many things but packages are useful because we can put files in specific places, such as the main computer level screensaver folder, then run a script to set it as a default.

Download munki-pkg and create a working project folder.

Step 1.

Create the folders you need and place your files (payloads) in the right places.

munkipkg-payload.png

Step 2.

Create your post install script if you need one. Example: setting the screensaver you just installed as the default.

#!/bin/sh

defaults -currentHost write com.apple.screensaver moduleDict -dict moduleName Brooklyn path /System/Library/Screen\ Savers/Brooklyn.saver/ type 0

 

munkipkg-postinstall.png

Step 3. Build your package

Run munki-pkg on the command line and build your package. If you make changes then version up in your build-info.plist and build again.

munkipkg-build.png

 

 

PostLab ❤️ Hedge

Great news everyone. Hedge has acquired PostLab and with this news the main developer Jasper Siegers joins the Hedge team. Great things will come of this collaboration. The brilliant idea of using version control for FCPX project sharing simplifies the whole process that has usually been left to huge media asset management systems that control data and projects. This is project sharing simplified and it’s about to get a lot more awesome with more developer time and a company to support it.

I am equally excited about Jasper coming to MacDevOps:YVR in June to talk about how and why he developed this app using GitLab and Docker and how version control for editors is useful and necessary.

Hedge is a small company in the Netherlands that makes the Hedge software to securely copy camera cards. I’ve incorporated this software with my Final Cut Pro X clients to secure their original footage on location as well as back in the office. Workflow post explains more.

I first wrote about PostLab last year

Some recent articles published about the news:

Fcp.co

Hedge announcement

Automate it! Hedge API example apps

Quick post to talk about some fun I’ve been having with the new Hedge API.

Background: Hedge is an awesome app for securely copying Camera Cards to multiple destinations to ensure data integrity and safety of the original footage. Hedge is one part of a workflow I build for my clients. Hedge is the first step in ensuring an easy and convenient transition from the cameras to the SAN to the tape archive powered by Archiware P5.

TL;DR

Using AppleScript and Automator I have built some apps to quickly set the file naming and data integrity preferences as we want them to be.  And also quickly change them to something else depending on the needed workflow.

API or Clickety click click

Copying camera cards copies is what we use Hedge for. Certain preferences like logs or receipts are great to have to ensure the copy succeeded. Also file naming conventions are good to have. Set and forget, right? But what if you did forget? Or if you’re new and don’t know the convention or you don’t read documentation. What do we do? One way to solve this is build an app that launches Hedge and sets the correct preferences. And if we want to copy USB sticks or something else then we can launch another app that prepares Hedge with a different set of preferences.  For extra points we just ask the user what they want with a nice dialog box and just do that.

hedge

Automator

Automator is awesome. Create workflows, apps, or services amongst many other things. For more info on Automator check out Apple’s official docs or this unofficial website of resources.

Automator.png

For this quick testing I used two AppleScripts with different preferences and settings defined. One for camera card copying and another for USB sticks that need different preferences set. The fancy automator app just calls the needed AppleScript. Make two apps and you have two different workflows without having to explain to users which prefs get set for what, or how the file naming should go to be consistent.

Example 1: Cam Card script (snippet)Hedge-Automator.png

Example 2: USB card copy (snippet)

Hedge-ApplesriptUSB.png

 

Automator can do many things. Call AppleScripts, Run shell scripts, pop-up dialog boxes etc and this is just a simple example of building single purpose apps to set Hedge via its new API. Very cool and so many possibilities.

AppleScript

What if we could just build one app which asked the user what they wanted to do? We can do that!

AppleScript-DialogQuit.png

Choose “Cam” and the appropriate preferences are set and file naming conventions applied.

Hedge-CamCard-Prefs.png

Hedge-CamCard-FileNaming.png

Choose “USB” and a different set of preferences are set. Magic.

Hedge-USBCard-Prefs.png

Hedge-USBCard-FileNaming.png

How do we do this? This piece of AppleScript chains an action to a response or button choice. Run a script or choose an action. The possibilities are endless. And thanks to the Hedge API we can set preferences on or off, and set destinations or many other things. We can do them programmatically and we can ensure they are set correctly. Fun times!

Hedge-API-Script-Quit.png

AppleScript vs POSIX:

I updated my AppleScript code with the POSIX path of the scripts it wants to load. It’s a major improvement! I had packaged up my scripts and my Hedge Setup app with munkipkg then deployed through Munki but when I demoed it — nothing worked…. because the start up disk drive was named something else. The fix: set a variable to be the POSIX path (Unix path in AppleScript friendly format).

Scripting and App Building

I hope that helped. We can do a lot of the same things with Python and in my testing I was working with a script written in python3 but since that’s not shipping on Macs in the current version of macOS by default (not yet!) then AppleScript was the quickest way to get this done. This is not restricted to AppleScript. Using Automator and your favourite scripting language you can build apps for your clients, co-workers, friends and family.

A note about the Hedge API:

There are two major calls I use in my scripts “setDestination” and “setPreferences”

The “setDestination” call looks like this:

open ‘hedge://actions?json=[{“setDestination”:{“path”:”/Volumes/LaCie/Testing/Test1″}},{“token”:”1234567890123345555″}]’

Note: the token is generated for you when you have a Pro license.

The “the setPreferences” call uses plist keys.

Note: I’ll have more say about using the actual Hedge API after it is officially announced.

No NetBoot, No problem: installr and bootstrappr

It’s 2019, and NetBoot is almost dead. All new Macs have T2 chips. Sent from the future to protect us from …. ourselves? No more NetBoot, no problem!!

When NetBoot first appeared and I was able to boot entire labs of Macs across the network I was amazed and overjoyed. It was awesome. Spinning globe, spinning…

Netboot-GlobeSpin.jpg

But in the years since I’ve moved on to no-imaging. Using Munki to manage software means no more imaging, just install Munki and a small config change to point to the Munki server, thereafter the software that should be there goes on, and what’s not supposed to be there goes away. Simple. Just install one package, well, maybe two, then you’re good.

Well, what if you want to streamline or automate these things? What if these are new Macs which don’t have users configured? What if we could do all this from recovery mode? Hmm… Enter bootstrappr and installr!!

bootstrappr

This awesome project allows to add packages to install in one step while booted in recovery mode. Plug in a USB stick with the bootstrapr script to run the package install magic or mount a disk image over http. Create a DMG with the included script make_dmg.sh. And now this is the best part: in recovery mode open the Terminal app from Utilities and type:

hdiutil mount http://server/yourDMG.dmg

Then:

/Volumes/bootstrap/run

When it’s done you can Reboot the Mac and you’ll have a set up customized to your liking with Munki installed and configured with custom settings.

installr

The installr script works in the same way but adds the macOS installer to the party. You can also mount the DMG over http and re-image a Mac and then add your custom packages. It’s awesome. Truly amazing.

One note: Added packages in Installr must be in a special format. From the installr site: startosinstall requires that all additional packages be Distribution-style packages (typically built with productbuild) and not component-style packages (typically built with pkgbuild)

productbuild --package component.pkg --version x.y --identifier com.example.component distribution.pkg

In one of my first tests with installr and pycreateuserpkg I was caught up by this, even though it is properly mentioned in the read me. Packages that work in Bootstrappr or munki directly don’t necessarily work when called by the macOS installer (startoinstall). Armin Briegel was helpful in the MacAdmins Slack and reminded me of this. Thanks Armin and thanks everyone on the MacAdmins Slack.

Many Thanks to Greg Neagle for creating these tools and Munki. Looking forward to hearing him speak at the next MacDevOps:YVR conference June 12-14, 2019. Greg will be speaking about his efforts to port some parts of Munki from Python to Swift. More info on the conference and speakers here: https://mdoyvr.com/speakers/

Also a shout out to Graham Gilbert who has worked on Imagr (MDOYVR talk), over the years, an imaging and automation tool which was also an inspiration (along with bootstrappr and installr) to Tim Perfit and his MDS project.

Update: corrected the names of installr and bootstrappr in the title because… autocorrect.

 

Best of 2018: the conferences

Part of a series of blog post on the “Best of 2018”

Part 2: the conferences

There was NAB in April and the FCPX Creative Summit in November (see more in my previous best of 2018 post here)

New conferences were the theme this year and I’ll start with one I couldn’t attend but really wanted to:

Query conf

⁃ all about osquery

⁃ Great group of people got together in San Francisco to discuss security with open source project originally from Facebook and now a verifiable industry (Kolide, uptycs etc)

⁃ Videos of the talks were posted on their site.

⁃ I had to miss this conference last year because it was just before MacDevOps: YVR and I could not be I two places at once.

Objective by the Sea

⁃ Awesome location. Maui. First time conference and my first time in Hawaii and it was spectacularly beautiful.

⁃ Great people. What a wonderfully diverse group of IT, Security experts and vendors

⁃ Patrick Wardle love fest. Everyone love Patrick and his Objective See free security tools. Look forward to his future projects with Digita Security

⁃ Looking forward to seeing how this conference evolves in the future. Especially if they end up doing more than one a year in different locations. Mahalo to the organizers for putting this together. As a fellow conference organizer I know it is not easy. It takes lots of love and Patrick and his team have lots of love to give. Thanks everyone!

MacDevOps:YVR

Disclaimer: I am the organizer

In 2018 we had our fourth annual conference and it was an amazing group of speakers and attendees. Many people took up my Quick Talk challenge. It is my firm belief that everyone has solved a problem and has knowledge they can share. I love it when people step up, literally step up on stage, and present a story, a solution to a problem, a tech problem they solved. We cheer them on.

What is MacDevOps:YVR?

Just the facts:

⁃ June 2019 will be the 5th annual conference

⁃ Inspired by DevOpsDays held everywhere all over the world this is an inspirational conference to bring together the creators of open source Mac projects and those in IT that use them.

⁃ Bringing a diverse group from around the world to learn about participating in software projects to manage Mac, the aim is to dispel the fear around version control (git), cloud (AWS, GCP, Azure) and various programming languages (Python, Swift, PHP etc)

⁃ Learn about what’s new in various open source projects we depend on: Munki, MunkiReport, Crypt, etc

⁃ Be inspired to share your our own solutions to problems with Quick Talks

⁃ 2019 will be partly security focused with a diverse group of security talks

⁃ Diversity and Inclusion will be front and centre of our IT panel. We are bringing in an amazing group to discuss.

⁃ Hack night. Working on gathering a team of MunkiReport contributors together to help organize some collaborative hacking and programming.

– Workshop. Learn how to make munkireport plugins. MunkiReport server is in PHP but the plugins are written in anything : bash, Python etc

Speakers for MacDevOps:YVR 2019 are on the MDO website.

June 12-14, 2018 join us in Vancouver, Canada for the firth annual MacDevOps:YVR conference.

Many thanks to the crew that that helps me organize this every year and makes the live event as good as it is. Without you I could not make it happen. Also many thanks to my awesome sponsors for helping us pay for bringing in speakers from around the world and paying for the event. Last but not least, thank you for all those that have attended and spoken at this event in the past. I love you all. You are amazing!

Resources

Use Git/GitHub to contribute to these community resources:

⁃ List of all conference videos Conference videos

⁃ MacAdmins Podcast community calendar GitHub repo

– Charles Edge has compiled a list of conferences and it is a great long list. Thanks Charles!

Best of 2018: FCPX and iMac Pro

Part of a series of blog post on the “Best of 2018”

Part 1: the iMac Pro and FCPX

The year started off with the new iMac Pro and Final Cut Pro X 10.4. Both new hardware and software were released in December 2017. New awesome hardware and software to start of 2018.

FCPX and the iMac Pro have proven themselves to be a great combination that has been amazing for FCPX editors everywhere. The new colour grading tools and other enhancements were warmly received in FCP X 10.4. The power of the iMac Pros was not exaggerated. Excellent pro hardware.

FCPX works great on a MacBook Pro and internal storage, with Apple’s Xsan and fibre channel or with Lumaforge Jellyfish 10GbE over NFS. I worked with all different setups in 2018 and happy to report that editors kept editing and left the storage and backup worries to me (and I didn’t worry since I’ve got Archiware P5 watching my back).

Working with the Jellyfish I installed the P5 Linux agent to backup and archive to tape. Getting the Jellyfish to back up to my P5 server running on a Mac Mini couldn’t have been easier. Through the year I worked with Archiware to make improvements in the P5 Archive app so that my editor clients can archive and restore more easily on their own. Works well and look forward to working more closely with both companies to help make awesome setups for FCPX editors and creative professionals everywhere.

NAB and FCPX

The week before NAB 2018, Apple announced a new version of Final Cut Pro X with support for closed captions, and the brand new ProRes RAW codec.

NAB in April is always a busy month with announcements from all companies in the media production and media asset management world and Apple’s public talk at NAB showing off new features so soon after their last major release was unexpected but very warmly received.

Of course there was one more major event in the 2018, in November there was the FCPX Creative Summit.

I attended this year and it was awesome. Apple released a brand new version with 3rd party integration in the form of extensions. This is huge. This will be amazing for FCPX editors who want to stay in FCPX and do their editing work but integrate with other apps.

What was the FCPX creative summit?

⁃ rendez-vous in Cupertino with Final Cut Pro editors, studio owners, plugin authors, creative apps vendors

⁃ Visit to Apple HQ. With Apple Pro Apps engineers, QA, managers and everyone involved.

⁃ In depth discussion of the next version of FCPX extensions which allow third party integration deep into the app for example: Frame IO for review and approve or Keyflow Pro or Cat DV media asset management apps.

⁃ Great team of people organizing. This event had multiple tracks and lots of great sessions for everyone. Well done. Enjoyed it immensely. Everyone using Final Cut Pro or involved in this creative universe should be there.

2018 was great year for pro hardware and software. The iMac Pro and the constant stream of FCPX updates kept us grinning from ear to ear. Great stuff. Awesome year.

Next up: best conferences of 2018