Tag Archives: OS X

Xsan Upgrade and Big Sur Prep. Hello Catalina!

Posted on by
5

Big Sur summer testing time!

Summer time is beta testing time. A new macOS beta cycle with Big Sur is upon us. Test early, and test often. With all the excitement of Big Sur in the air, it’s time to look at Catalina.

Our day to day production Xsan systems do not run beta software, not even the latest version of macOS, they only run tested and safe versions of macOS. I always recommend being a revision behind the latest. Until now that meant macOS 10.14 (Mojave). With the imminent release of macOS Big Sur (is it 10.16 or macOS 11?) then it’s time to move from 10.14.6 Mojave to 10.15.6 Catalina. It must be safe now, right? 

Background

Xsan is Apple’s based Storage Area Network (SAN) software licensed from Quantum (see StorNext), and since macOS 10.7 aka Lion it has been included with macOS for free (it was $1,000 per client previously!).

Ethernet vs Fibre Channel vs Thunderbolt

A SAN is not the same as a NAS (Network attached storage) or DAS (direct attached storage). A NAS or other network based storage is often 10GbE and can be quite fast and capable. I will often use Synology NAS with 10GbE for a nearline archive (a second copy of tape archive) but can also use it as a primary storage with enough cache. Lumaforge’s Jellyfish is another example of network based storage.

Xsan storage is usually fibre channel based and even old 4GB storage is fast because … fibre channel protocol (FCP) is fast and the data frames are sent in order unlike TCP. It is more common to see 8GB or 16Gb fibre channel storage these days (though 32GB is starting to appear). And while fibre channel is typically what you use for Xsan you can also use shared Thunderbolt based storage like the Accusys A16T3-Share. I have tested a Thunderbolt 2 version of this hardware with Xsan and it works very well. I’m hoping to test a newer Thunderbolt 3 version soon. Stay tuned.

Xsan vs macOS Versions

We’ve discussed all the things that the Xsan is not and now what is it? Xsan is often created from multiple fibre channel RAID storage units but the data is entirely dependent on the Xsan controller that creates the volume. The Xsan controller is typically a Mac Mini but can be any Mac with Server.app (from Apple’s App Store). The existence of any defined Xsan volumes depends on the sanity of its SAN metadata controllers. If the SAN controllers die and the configuration files go with it then your data is gone.  POOF! I’ve always said that Xsan is a shared hallucination, and all the dreamers should dream the same dream. To make sure of this we always recommend running the same version of macOS on the Mac clients as well as the servers (the Xsan controllers). And while the Xsan controllers should be the same or at a higher macOS version level it can sometimes be the opposite in practise. To be sure what versions of macOS are interoperable we can check with Apple’s Xsan controllers and clients compatibility chart and Xsan versions included in macOS for the rules and exceptions. Check the included version of Xsan on your Mac with the cvversions command

File System Server:
  Server  Revision 5.3.1 Build 589[63493] Branch Head BuildId D
   Built for Darwin 17.0 x86_64
   Created on Sun Dec  1 19:58:57 PST 2019
   Built in /BuildRoot/Library/Caches/com.apple.xbs/Sources/XsanFS/XsanFS-613.50.3/buildinfo

This is from a Mac running macOS 10.13

Host OS Version:
 Darwin 17.7.0 Darwin Kernel Version 17.7.0: Sun Dec  1 19:19:56 PST 2019; root:xnu-4570.71.63~1/RELEASE_X86_64 x86_64

We see similar results from a newer build below:

File System Server:
  Server  Revision 5.3.1 Build 589[63493] Branch Head BuildId D
   Built for Darwin 19.0 x86_64
   Created on Sun Jul  5 02:42:52 PDT 2020
   Built in /AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/XsanFS/XsanFS-630.120.1/buildinfo

This is from a Mac running macOS 10.15.

Host OS Version:
 Darwin 19.6.0 Darwin Kernel Version 19.6.0: Sun Jul  5 00:43:10 PDT 2020; root:xnu-6153.141.1~9/RELEASE_X86_64 x86_64

Which tells us that the same version of Xsan are included with macOS 10.13 and 10.15 (and indeed is the same from 10.12 to 10.15). So we have situations with Xsan controllers running 10.13 and clients running 10.14 are possible even though macOS versions are a mismatch, the Xsan versions are the same. There are other reasons for keeping things the macOS versions the same: troubleshooting, security, management tools, etc  To be safe check with Apple and other members of the Xsan community (on MacAdmins Slack).

Backups are important

Do not run Xsan or any kind of storage in production without backups. Do not do it. If your Xsan controllers die then your storage is gone. Early versions of Xsan (v1 especially) were unstable and the backups lesson can be a hard one to learn. All later versions of Xsan are much better but we still recommend backups if you like your data. Or your clients. (Clients are the people that make that data and pay your bills). I use Archiware P5 to make tape backups, tape archives, nearline copies as well as workstation backups. Archiware is a great company and P5 is a great product. It has saved my life (backups are boring, restores are awesome!).

P5-Restore-FCPX.png

Xsan Upgrade Preparation

When you upgrade macOS it will warn you that you have Server.app installed and you might have problems. After the macOS upgrade you’ll need to download and install a new version of Server.app. In my recent upgrades from macOS 10.13 to macOS 10.15 via 10.14 detour I started with Server.app 5.6, then install 5.8 and finally version 5.10.

After the macOS upgrade I would zip up the old Server.app application and put in place the new version which I had already downloaded elsewhere. Of course you get a warning about removing the Server app

 

Xsan-ServerApp-ZipRemovalDetected.png

Install the new Server app then really start your Xsan upgrade adventure.

Serverapp-setup.png

Restore your previous Xsan setup.

This slideshow requires JavaScript.

If everything goes well then you have Xsan setup and working on macOS 10.15.6 Catalina

Xsan-Catalina-Upgrade-Success

Automate those apps. Get some robot love 🤖 ❤️!

Posted on by
1

If only one person needs an application then I think about using Munki to deploy that app. If more than one person should have it then Munki is definitely the way to automate app deployment. And really, if you’re going to take the time to download an app from a website, mount a disk image or un-pack a ZIP archive, run an installer, type an admin password, close that installer … then for the love of all that is good just put the app into your Munki repo and be done with it. Automate it.

Using Munki to solve problems makes sense. Automation helps everyone in this case. But if you’re putting in one off applications into your Munki repo more often than you need to, you need to get those apps into Autopkg. Using Autopkg recipes to download the latest apps and put them into your Munki repo automatically is an automation love fest, but if your apps don’t have recipes what are you going to do? Manually add your apps to Munki? No way. We need a robot 🤖❤️. Recipe robot, that is.

Using Recipe Robot we can build Autopkg recipes for most apps then add the recipes to the Autopkg community to enjoy. Everyone wins.

I recently created recipes for two important apps in my media workflow: Kyno and Hedge. I’ll show an example of this workflow using Recipe Robot and Munki Admin to demonstrate the workflow.

Step 1. Feed the robot.

Drag and and drop the app you want to create your Autopkg recipes.

RecipeRobot-FeedMe

Step 2. Watch the robot do it’s work

RecipeRobot-start

Step 3. Robot is done. Recipes made.

RecipeRobot-Done

Various type of recipes can be made. I chose download and munki because those are what I am using to automate adding apps to my Munki repo. But there are other options: jss, Filewave, or “install” for example.

reciperobot-options.jpg

Step 4. Run those Recipes

You can use your recipes locally with Autopkg. Run them in Terminal or use Autopkgr , a very nice GUI app for automating the collection and scheduling of recipes. Note: Autopkg and Munki can all be run via cli (command line interface) but for this demo we are showing the GUI apps that are there provided by outstanding members of the community. Many Thanks to them and the contributors to their projects.

Autopkgr-notification

Autopkgr app can send notifications in macOS, emails, or post to your Slack group.

Step 5. See the recipes, Use them wisely

MunkiAdmin-Recently ChangedPKGS

Here is an example of newly imported Kyno and Hedge apps in our Munki repo (via Munki Admin GUI).

MunkiAdmin-Description

Add a display name, choose which catalogs the apps will reside in, and check that the description will help explain what the app is.

References:

Elliot Jordan – Autopkg talk at MacDevOps:YVR

https://youtu.be/Q_cvgGtJ71M

Elliot Jordan – Recipe Robot talk at MacDevOps:YVR

https://youtu.be/DgjO1mfMHtI

 

The case of the strange disappearing drive space

Posted on by
Reply

Recently I was asked to look at a 4TB drive that was only showing less than 2TB available…. No problem, I said, this is easy to fix. Famous last words.

Just open up Disk Utility and resize the partition, or reformat the disk, right? Easy Peasey. Well, it took some troubleshooting to time to figure out and a trip to Terminal was required to solve this weird case, plus I learned a new command along the way. Fun.

The Problem:

Buying a 4TB hard drive then putting it into your external drive case for backups should be simple,  but what if instead you got a nasty surprise and it showed up as less than 2TB?

Troubleshooting the issue:

4TB drives were presented to me and when I loaded them into an external SATA dock then showed as 4TB drives with a partitioned volume of less than 2TB.

I tried to delete the phantom partition, and I tried resize the volume to use the empty space in Disk Utility.app but it refused to budge. This needed a trip to Terminal.

man diskutil

Using “man” or “info” commands you can find out more about almost any particular command. Maybe some useful options or arguments would be listed or at least some examples would help.

NAME

     diskutil -- modify, verify and repair local disks

SYNOPSIS

     diskutil [quiet] verb [options]

DESCRIPTION

     diskutil manipulates the structure of local disks. 

 

To find out more about what we’re faced with let’s ask diskutil what it sees:

diskutil list
/dev/disk2 (external, physical):

   #:                       TYPE NAME           SIZE       IDENTIFIER

   0:      GUID_partition_scheme               *4.0 TB     disk2

   1:                    EFI                   209.7 MB   disk2s1

   2:        Apple_HFS Backup                  1.8 TB     disk2s2

Looking through the man page the “resizeVolume” command caught my eye. Also the “limits” option seemed interesting. How

diskutil resizeVolume disk2s2 limits

Resize limits for partition disk2s2 Backup:

  Current partition size on map:         1.8 TB (1801419800576 Bytes)

  Minimum (constrained by file usage):   846.4 MB (846426112 Bytes)

  Recommended minimum (if used for macOS):26.8 GB (26843545600 Bytes)

  Maximum (constrained by map space):   4.0 TB (4000442028032 Bytes)

The Answer:

Reading through the man page revealed that the best way, and new to me, was to resize the partition to use all available space with “R”. Of course, so intuitive.

sudo diskutil resizeVolume disk2s2 R

I did get some errors. But repairing the disk fixed those issues. And I was able to resize the disk in Terminal with diskutil where Disk Utility.app had failed.

sudo diskutil resizeVolume disk2s2 R

Resizing to full size (fit to fill)

Started partitioning on disk2s2 Backup

Verifying the disk

Verifying file system

Volume was successfully unmounted

Performing fsck_hfs -fn -x /dev/rdisk2s2

Checking Journaled HFS Plus volume

Checking extents overflow file

Checking catalog file

Checking multi-linked files

Checking catalog hierarchy

Checking extended attributes file

Checking volume bitmap

Checking volume information

The volume Backup appears to be OK

File system check exit code is 0

Restoring the original state found as mounted

Resizing

Modifying partition map

Growing file system

Finished partitioning on disk2s2 Backup

/dev/disk2 (external, physical):

   #:                       TYPE NAME          SIZE       IDENTIFIER

   0:      GUID_partition_scheme              *4.0 TB     disk2

   1:                        EFI             209.7 MB   disk2s1

   2:                  Apple_HFS Backup      4.0 TB     disk2s2

And lastly, the issue may have been caused by the old drive dock which refused to see the 4TB volumes even when correctly resized. A newer drive dock was required.

Zoom in on Privacy and Security

Posted on by
1

Recent attention on video conferencing app Zoom and security exploits brings attention to the various Privacy and Security settings on your Mac. Currently macOS 10.14.5 Mojave defines microphone and camera settings which should be verified periodically if they’re not being managed by MDM (mobile device management) and even in those case, just to verify.

Zoom update

If you’ve ever had Zoom installed you must launch it and then update it manually, unless you have Munki or other patching solution to manage your Mac.

 

Zoom Enable camera access

If you want Zoom to have access to your camera (useful for video conferencing) then enable it or leave it disabled until the moment you actually need it.

Privacy-Camera-OFF-Settings.pngMaybe this is a good time to review what apps have previously been granted access and disable them or not after you review the situation.

Privacy-MIC2-Settings.png

Check your microphone access as well. What apps are in your list?

Further research:

Check out Objective See’s excellent security tools such as Oversight to protect yourself from unwanted access to your camera.

Also check out this past talk at MacDevOps:YVR 2018 by Kolide’s Zach Wasserman about osquery and at the 11min mark where he talks about another app BlueJeans and how to investigate it with osquery.

The MacDevOps:YVR videos from past talks contain many security related talks as well as other awesome troubleshooting tech talks.

 

 

Use Munki to install a screensaver

Posted on by
Reply

Use munki-pkg to package up stuff and make your life easier when managing Macs using munki. Here is an example of installing a screensaver.

Why use munki-pkg? How else do you install stuff using munki, run scripts, and version your testing buildings all in one easy to use application? This is all possible with munki-pkg.

Munki-pkg makes package (PKG) installers, Munki likes pkg installers. Munki will also install apps, run scripts, install profiles, and do many things but packages are useful because we can put files in specific places, such as the main computer level screensaver folder, then run a script to set it as a default.

Download munki-pkg and create a working project folder.

Step 1.

Create the folders you need and place your files (payloads) in the right places.

munkipkg-payload.png

Step 2.

Create your post install script if you need one. Example: setting the screensaver you just installed as the default.

#!/bin/sh

defaults -currentHost write com.apple.screensaver moduleDict -dict moduleName Brooklyn path /System/Library/Screen\ Savers/Brooklyn.saver/ type 0

 

munkipkg-postinstall.png

Step 3. Build your package

Run munki-pkg on the command line and build your package. If you make changes then version up in your build-info.plist and build again.

munkipkg-build.png

 

 

Reset Printer Queue

Posted on by
Reply

TIL (thing I learned)

Had a user upgrade to macOS 10.14.1 and no printers showed up anymore.

So using my Google fu I found some posts (see one below) which described a novel way to reset the print queue on macOS. An old trick apparently. Learn something new everyday.

A quick trip to a terminal and it worked! The existing printers returned to System Preferences and printing resumed.

$ cancel -a

Reference

Blocking minor major macOS upgrades

Posted on by
1

Continuing our theme of welcoming our new macOS overlords, uh, I mean, blocking major macOS upgrades such as macOS 10.14 Mojave with AppBlock we shall examine some other methods of stopping the freight train known as Apple upgrades.

1) A smart person on the MacAdmins Slack posted a useful command to tell macOS not to download major upgrades.

In their testing, running:

`software update –ignore macOSInstallerNotification_GM`

blocks the installation of the Mojave notification package (at /Library/Bundles/OSXNotification.bundle).

However if it already installed, then it’s too late. They pushed out this command prior to that package being distributed by Apple, and they could subsequently see in install.log that the update is being found by softwareupdated but not being installed.

2) If you missed the chance to tell the Mac not to download major macOS upgrades then Rick Heil on his blog has detailed a way using munki to delete the bundle that triggers the macOS upgrade installer.

3) App Block

If your users are intent or their computers are all hell bent on downloading the install app then block it with App block detailed in my previously mentioned blog post

4) Warning

In an effort to get an early warning when users are about to upgrade I use Watchman Monitoring to send me an alert email when a Mac starts downloading the Install macOS app. Sometimes it’s enough of a warning to send an email to a user to ask them whether it is a good idea to upgrade at this time. If storage or software needed for production or backups aren’t qualified or tested thoroughly beforehand then upgrading in the early waves can be less than ideal and frought with peril.

In other interesting and related news, Victor (MicroMDM) was spelunking into the MDM Protocol for what prompts Macs like iOS devices to download major updates. Great post here

If you have any better ways to block macOS upgrades or want to contribute some great solutions let me know. Cheers

 

 

 

 

To install macOS Mojave, or not to?

Posted on by
8

InstallMojave

Just the other day macOS Mojave was released and now the armies of Macs armed only with the AppStore are silently downloading the installer and ready to upgrade. You can’t hurry too fast to be on the bleeding edge, hurry faster!

Just in case you don’t want everyone to install macOS 10.14.0 (dot zero!) in the first week of its release here’s a way to slow down the upgrade hordes using Erik Berglund’s AppBlocker script. Erik Berglund is also the author of ProfileCreator (for creating profiles) and the author of many other great scripts.

Note: for true binary whitelisting check out Google’s Santa project and Upvote (and Moroz and Zentral, two other Santa sync servers).

Step 1. Get it

Clone or download the AppBlocker project from GitHub

AppleBlockerProject.png

Step 2. Do it

Edit the AppBlocker.py script with the Bundle Identifier of your app to block, in this case for the Mojave installer from the AppStore it is:

com.apple.InstallAssistant.Mojave

You can also edit the alert message, and the icon that is shown, as well as decide if the blocked app should be deleted or not. The script is easy to edit in BBEdit, or nano (in Terminal). Use whatever your favorite text editor is to make the necessary changes.

# List of all blocked bundle identifiers. Can use regexes.
blockedBundleIdentifiers = ['com.apple.InstallAssistant.Mojave']

# Whether the blocked application should be deleted if launched
deleteBlockedApplication = False

# Whether the user should be alerted that the launched applicaion was blocked
alertUser = True

# Message displayed to the user when application is blocked
alertMessage = "The application \"{appname}\" has been blocked by IT"
alertInformativeText = "Contact your administrator for more information"

# Use a custom Icon for the alert. If none is defined here, the Python rocketship will be shown.
alertIconPath = "/System/Library/CoreServices/CoreTypes.bundle/Contents/Resources/Actions.icns"

UPDATED NOTE:

To determine the Bundle identifier of other applications you can use osascript

osascript -e 'id of app "iTunes"'
com.apple.iTunes

If you want to block more than one app use a comma separated list in the AppBlocker.py script:

['com.apple.InstallAssistant.Mojave','com.apple.iTunes']

 

Step 3. Run it

Put the script where you want to run it. The default location as defined in the launchd plist included with the app is “/usr/local/bin”. Put the launchd.plist in “/Library/LaunchDaemons/” and start up your launchd to block your apps!

launchctl load /Library/LaunchDaemons/com.github.erikberglund.AppBlocker.plist

Step 4. Automate it

For bonus points we automate! Bundle it all up in a package with munkipkg, then distribute it with Munki to all your clients.

Using munkipkg is easy. Create the folder using munkipkg

./munkipkg --create AppBlocker

munkipkg: Created new package project at AppBlocker

Then you fill the payload folders with those items you downloaded from the AppBlocker project. LauchD plist in the LaunchDaemons folder and AppBlocker.py in the “usr local bin” (create each nested folder).

AppBlocker-Munkipkg3.png

And finally create a post install script (no “.sh”) with the launchctl action to start your plist.

AppBlocker-Munkipkg4.png

Last but not least add this package to your Munki repo as an unattended managed install  that everyone gets. Of course, only do this after testing your package locally somewhere to verify that it works properly. Remember the saying: “You may not test very often, but when you do it’s always in production.” Be very careful with your testing but always automate all the things.

Updated after the initial blog post to explain how to add more than one app to block, and how to use osascript to determine the bundle identifier.

 

 

 

P5 on the Jellyfish: Archiving Gotchas

Posted on by
1

TL;DR

Using Archiware P5 to Archive files to tapes is awesome, but watch out for little things you might miss, such as the path to the files and backing up your Archive Db.

P5 Archive on the Jellyfish

Using P5 Archive with the Lumaforge Jellyfish is a great way to preserve your digital archives. See this post for how to set up P5 on the Jellyfish

Using Archiware P5 for archiving makes sense. You want your completed projects and original camera footage on LTO tape. But how do you do archives? There are several different ways, and there be gotchas.

P5 Archive vs P5 Archive app

Using P5 Archive to manually archive completed projects to LTO tape is a process of logging into the server via a web browser and selecting the the project folder you want to archive to tape.

The completed project folder could be on the storage visible to the server or it could be storage the client sees. And that can make a difference. Where the storage is mounted is different on a Mac vs Linux. Its’ the difference between “/Volumes” and “/mnt”.

The same Jellyfish storage, either SMB or NFS, when seen on a Mac is mounted by default at “/Volumes” (this can be changed but for most people leave it at the default). But when archiving the storage via a Jellyfish client you will get “/mnt” path.

p5-smb-test2.png

Using the P5 Archive app, which is a Mac only companion application to P5 Archive, to run the archives you will see the storage archived as “/Volumes”.

This first Archiving gotcha is if you’re archiving the Jellyfish storage with the web application of P5 Archive you will have to find your footage and restore from the “/mnt” path vs if you’re archiving from the P5 Archive app which is running from a Mac and will see and store the footage using the “/Volumes” path.

All this to say that using both ways to archive may double up your footage in your archive which may be unintended. And from a restore in the web browser finding your footage may be confusing if you’re used to seeing it mounted in “/Volumes” and you actually find it under “/mnt”.

Note: the reason to use the P5 Archive app is because of the simplicity of right-clicking files in the finder which are on your storage and telling them to archive right then and there. Files are copied to tape then the original files on the storage are replaced with stub files. Right-click again to restore. Simple.

p5-archive-app-job-monitor.png

Backup your Archive!

Don’t forget to backup your archives. Or rather, your archive Db. A more recent addition is the ability to automate the backups on the Archive index, so don’t forget to enable it.

In the managed index section, choose your Archive index.

Set the target client where the backups are going and the backup directory. Choose a time and don’t forget to enable it (check the checkbox and hit apply before closing the windows).

Note: Repeat this setup for each Archive index you want to backup.

Archive Backup db setup3.png

Monitoring your Archive!

Don’t forget to enable email notifications for your P5 server to get your inbox full of status notifications and errors and other important stuff. But if you want to cut down on email notifications or you have multiple P5 servers (many different clients, perhaps), then you might want to check out Watchman Monitoring and the P5 plugin that is built-in). Find out easily when your tape pools are getting low, the tape drives needs to be cleaned, the support maintenance needs renewing etc. All in one dashboard. How convenient!

Maybe everything is going well…

Watchman-P5-info.png

Or maybe not!

Archiware-P5-Jobs-Watchman-tapes-required.png

 

macOS Server is dead. Long live macOS.

Posted on by
4

Yes, it’s been a hot topic in the MacAdmin community both on Mac Enterprise list (oh no it’s the end of the world!) and MacAdmins Slack (told you it was coming, don’t be surprised).

My professional opinion is: “Don’t panic!”

My MacDevOps conference is all about supporting MacAdmins who have been writing code as infrastructure to manage Macs. And do it while replacing macOS Server in the server room with Linux and other OS.

Xsan is staying in macOS Server so I am happy and that’s my main use for the Mac Mini and macOS Server.

I have other Mac Minis doing file sharing for small work groups and moving that out of Server.app in the last revision was unfortunate (it is in the standard OS and usable there but less manageable). There’s also Synology and QNAP NAS for small workgroup file sharing and so much more And many enterprise storage vendors for larger setups.

Imaging has been dying a slow death for years and has been replaced with a thin or “no imaging” concept supported by tools such as AutoPkg and Munki.

Profile Manager is a demo version of MDM and should not be used to actually manage Macs.

Wikis, DNS and Mail should be hosted on Linux, in VMs, AWS, GCP or anywhere other than macOS Server so no problem.

Overall it might be disconcerting to some. But change is constant. And especially at Apple change comes fast and often. We have to get used to it.

Reference:

Apple Support article

Apple to Deprecate Many macOS Server Services – TidBITS http://tidbits.com/e/17760