The case of the strange disappearing drive space

Recently I was asked to look at a 4TB drive that was only showing less than 2TB available…. No problem, I said, this is easy to fix. Famous last words.

Just open up Disk Utility and resize the partition, or reformat the disk, right? Easy Peasey. Well, it took some troubleshooting to time to figure out and a trip to Terminal was required to solve this weird case, plus I learned a new command along the way. Fun.

The Problem:

Buying a 4TB hard drive then putting it into your external drive case for backups should be simple,  but what if instead you got a nasty surprise and it showed up as less than 2TB?

Troubleshooting the issue:

4TB drives were presented to me and when I loaded them into an external SATA dock then showed as 4TB drives with a partitioned volume of less than 2TB.

I tried to delete the phantom partition, and I tried resize the volume to use the empty space in Disk Utility.app but it refused to budge. This needed a trip to Terminal.

man diskutil

Using “man” or “info” commands you can find out more about almost any particular command. Maybe some useful options or arguments would be listed or at least some examples would help.

NAME

     diskutil -- modify, verify and repair local disks

SYNOPSIS

     diskutil [quiet] verb [options]

DESCRIPTION

     diskutil manipulates the structure of local disks.  

 

To find out more about what we’re faced with let’s ask diskutil what it sees:

diskutil list
/dev/disk2 (external, physical):

   #:                       TYPE NAME           SIZE       IDENTIFIER

   0:      GUID_partition_scheme               *4.0 TB     disk2

   1:                    EFI                   209.7 MB   disk2s1

   2:        Apple_HFS Backup                  1.8 TB     disk2s2

Looking through the man page the “resizeVolume” command caught my eye. Also the “limits” option seemed interesting. How

diskutil resizeVolume disk2s2 limits

Resize limits for partition disk2s2 Backup:

  Current partition size on map:         1.8 TB (1801419800576 Bytes)

  Minimum (constrained by file usage):   846.4 MB (846426112 Bytes)

  Recommended minimum (if used for macOS):26.8 GB (26843545600 Bytes)

  Maximum (constrained by map space):   4.0 TB (4000442028032 Bytes)

The Answer:

Reading through the man page revealed that the best way, and new to me, was to resize the partition to use all available space with “R”. Of course, so intuitive.

sudo diskutil resizeVolume disk2s2 R

I did get some errors. But repairing the disk fixed those issues. And I was able to resize the disk in Terminal with diskutil where Disk Utility.app had failed.

sudo diskutil resizeVolume disk2s2 R

Resizing to full size (fit to fill)

Started partitioning on disk2s2 Backup

Verifying the disk

Verifying file system

Volume was successfully unmounted

Performing fsck_hfs -fn -x /dev/rdisk2s2

Checking Journaled HFS Plus volume

Checking extents overflow file

Checking catalog file

Checking multi-linked files

Checking catalog hierarchy

Checking extended attributes file

Checking volume bitmap

Checking volume information

The volume Backup appears to be OK

File system check exit code is 0

Restoring the original state found as mounted

Resizing

Modifying partition map

Growing file system

Finished partitioning on disk2s2 Backup

/dev/disk2 (external, physical):

   #:                       TYPE NAME          SIZE       IDENTIFIER

   0:      GUID_partition_scheme              *4.0 TB     disk2

   1:                        EFI             209.7 MB   disk2s1

   2:                  Apple_HFS Backup      4.0 TB     disk2s2

And lastly, the issue may have been caused by the old drive dock which refused to see the 4TB volumes even when correctly resized. A newer drive dock was required.

Automate it! Hedge API example apps

Quick post to talk about some fun I’ve been having with the new Hedge API.

Background: Hedge is an awesome app for securely copying Camera Cards to multiple destinations to ensure data integrity and safety of the original footage. Hedge is one part of a workflow I build for my clients. Hedge is the first step in ensuring an easy and convenient transition from the cameras to the SAN to the tape archive powered by Archiware P5.

TL;DR

Using AppleScript and Automator I have built some apps to quickly set the file naming and data integrity preferences as we want them to be.  And also quickly change them to something else depending on the needed workflow.

API or Clickety click click

Copying camera cards copies is what we use Hedge for. Certain preferences like logs or receipts are great to have to ensure the copy succeeded. Also file naming conventions are good to have. Set and forget, right? But what if you did forget? Or if you’re new and don’t know the convention or you don’t read documentation. What do we do? One way to solve this is build an app that launches Hedge and sets the correct preferences. And if we want to copy USB sticks or something else then we can launch another app that prepares Hedge with a different set of preferences.  For extra points we just ask the user what they want with a nice dialog box and just do that.

hedge

Automator

Automator is awesome. Create workflows, apps, or services amongst many other things. For more info on Automator check out Apple’s official docs or this unofficial website of resources.

Automator.png

For this quick testing I used two AppleScripts with different preferences and settings defined. One for camera card copying and another for USB sticks that need different preferences set. The fancy automator app just calls the needed AppleScript. Make two apps and you have two different workflows without having to explain to users which prefs get set for what, or how the file naming should go to be consistent.

Example 1: Cam Card script (snippet)Hedge-Automator.png

Example 2: USB card copy (snippet)

Hedge-ApplesriptUSB.png

 

Automator can do many things. Call AppleScripts, Run shell scripts, pop-up dialog boxes etc and this is just a simple example of building single purpose apps to set Hedge via its new API. Very cool and so many possibilities.

AppleScript

What if we could just build one app which asked the user what they wanted to do? We can do that!

AppleScript-DialogQuit.png

Choose “Cam” and the appropriate preferences are set and file naming conventions applied.

Hedge-CamCard-Prefs.png

Hedge-CamCard-FileNaming.png

Choose “USB” and a different set of preferences are set. Magic.

Hedge-USBCard-Prefs.png

Hedge-USBCard-FileNaming.png

How do we do this? This piece of AppleScript chains an action to a response or button choice. Run a script or choose an action. The possibilities are endless. And thanks to the Hedge API we can set preferences on or off, and set destinations or many other things. We can do them programmatically and we can ensure they are set correctly. Fun times!

Hedge-API-Script-Quit.png

AppleScript vs POSIX:

I updated my AppleScript code with the POSIX path of the scripts it wants to load. It’s a major improvement! I had packaged up my scripts and my Hedge Setup app with munkipkg then deployed through Munki but when I demoed it — nothing worked…. because the start up disk drive was named something else. The fix: set a variable to be the POSIX path (Unix path in AppleScript friendly format).

Scripting and App Building

I hope that helped. We can do a lot of the same things with Python and in my testing I was working with a script written in python3 but since that’s not shipping on Macs in the current version of macOS by default (not yet!) then AppleScript was the quickest way to get this done. This is not restricted to AppleScript. Using Automator and your favourite scripting language you can build apps for your clients, co-workers, friends and family.

A note about the Hedge API:

There are two major calls I use in my scripts “setDestination” and “setPreferences”

The “setDestination” call looks like this:

open ‘hedge://actions?json=[{“setDestination”:{“path”:”/Volumes/LaCie/Testing/Test1″}},{“token”:”1234567890123345555″}]’

Note: the token is generated for you when you have a Pro license.

The “the setPreferences” call uses plist keys.

Note: I’ll have more say about using the actual Hedge API after it is officially announced.

No NetBoot, No problem: installr and bootstrappr

It’s 2019, and NetBoot is almost dead. All new Macs have T2 chips. Sent from the future to protect us from …. ourselves? No more NetBoot, no problem!!

When NetBoot first appeared and I was able to boot entire labs of Macs across the network I was amazed and overjoyed. It was awesome. Spinning globe, spinning…

Netboot-GlobeSpin.jpg

But in the years since I’ve moved on to no-imaging. Using Munki to manage software means no more imaging, just install Munki and a small config change to point to the Munki server, thereafter the software that should be there goes on, and what’s not supposed to be there goes away. Simple. Just install one package, well, maybe two, then you’re good.

Well, what if you want to streamline or automate these things? What if these are new Macs which don’t have users configured? What if we could do all this from recovery mode? Hmm… Enter bootstrappr and installr!!

bootstrappr

This awesome project allows to add packages to install in one step while booted in recovery mode. Plug in a USB stick with the bootstrapr script to run the package install magic or mount a disk image over http. Create a DMG with the included script make_dmg.sh. And now this is the best part: in recovery mode open the Terminal app from Utilities and type:

hdiutil mount http://server/yourDMG.dmg

Then:

/Volumes/bootstrap/run

When it’s done you can Reboot the Mac and you’ll have a set up customized to your liking with Munki installed and configured with custom settings.

installr

The installr script works in the same way but adds the macOS installer to the party. You can also mount the DMG over http and re-image a Mac and then add your custom packages. It’s awesome. Truly amazing.

One note: Added packages in Installr must be in a special format. From the installr site: startosinstall requires that all additional packages be Distribution-style packages (typically built with productbuild) and not component-style packages (typically built with pkgbuild)

productbuild --package component.pkg --version x.y --identifier com.example.component distribution.pkg

In one of my first tests with installr and pycreateuserpkg I was caught up by this, even though it is properly mentioned in the read me. Packages that work in Bootstrappr or munki directly don’t necessarily work when called by the macOS installer (startoinstall). Armin Briegel was helpful in the MacAdmins Slack and reminded me of this. Thanks Armin and thanks everyone on the MacAdmins Slack.

Many Thanks to Greg Neagle for creating these tools and Munki. Looking forward to hearing him speak at the next MacDevOps:YVR conference June 12-14, 2019. Greg will be speaking about his efforts to port some parts of Munki from Python to Swift. More info on the conference and speakers here: https://mdoyvr.com/speakers/

Also a shout out to Graham Gilbert who has worked on Imagr (MDOYVR talk), over the years, an imaging and automation tool which was also an inspiration (along with bootstrappr and installr) to Tim Perfit and his MDS project.

Update: corrected the names of installr and bootstrappr in the title because… autocorrect.

 

Reset Printer Queue

TIL (thing I learned)

Had a user upgrade to macOS 10.14.1 and no printers showed up anymore.

So using my Google fu I found some posts (see one below) which described a novel way to reset the print queue on macOS. An old trick apparently. Learn something new everyday.

A quick trip to a terminal and it worked! The existing printers returned to System Preferences and printing resumed.

$ cancel -a

Reference

Blocking minor major macOS upgrades

Continuing our theme of welcoming our new macOS overlords, uh, I mean, blocking major macOS upgrades such as macOS 10.14 Mojave with AppBlock we shall examine some other methods of stopping the freight train known as Apple upgrades.

1) A smart person on the MacAdmins Slack posted a useful command to tell macOS not to download major upgrades.

In their testing, running:

`software update –ignore macOSInstallerNotification_GM`

blocks the installation of the Mojave notification package (at /Library/Bundles/OSXNotification.bundle).

However if it already installed, then it’s too late. They pushed out this command prior to that package being distributed by Apple, and they could subsequently see in install.log that the update is being found by softwareupdated but not being installed.

2) If you missed the chance to tell the Mac not to download major macOS upgrades then Rick Heil on his blog has detailed a way using munki to delete the bundle that triggers the macOS upgrade installer.

3) App Block

If your users are intent or their computers are all hell bent on downloading the install app then block it with App block detailed in my previously mentioned blog post

4) Warning

In an effort to get an early warning when users are about to upgrade I use Watchman Monitoring to send me an alert email when a Mac starts downloading the Install macOS app. Sometimes it’s enough of a warning to send an email to a user to ask them whether it is a good idea to upgrade at this time. If storage or software needed for production or backups aren’t qualified or tested thoroughly beforehand then upgrading in the early waves can be less than ideal and frought with peril.

In other interesting and related news, Victor (MicroMDM) was spelunking into the MDM Protocol for what prompts Macs like iOS devices to download major updates. Great post here

If you have any better ways to block macOS upgrades or want to contribute some great solutions let me know. Cheers

 

 

 

 

To install macOS Mojave, or not to?

InstallMojave

Just the other day macOS Mojave was released and now the armies of Macs armed only with the AppStore are silently downloading the installer and ready to upgrade. You can’t hurry too fast to be on the bleeding edge, hurry faster!

Just in case you don’t want everyone to install macOS 10.14.0 (dot zero!) in the first week of its release here’s a way to slow down the upgrade hordes using Erik Berglund’s AppBlocker script. Erik Berglund is also the author of ProfileCreator (for creating profiles) and the author of many other great scripts.

Note: for true binary whitelisting check out Google’s Santa project and Upvote (and Moroz and Zentral, two other Santa sync servers).

Step 1. Get it

Clone or download the AppBlocker project from GitHub

AppleBlockerProject.png

Step 2. Do it

Edit the AppBlocker.py script with the Bundle Identifier of your app to block, in this case for the Mojave installer from the AppStore it is:

com.apple.InstallAssistant.Mojave

You can also edit the alert message, and the icon that is shown, as well as decide if the blocked app should be deleted or not. The script is easy to edit in BBEdit, or nano (in Terminal). Use whatever your favorite text editor is to make the necessary changes.

# List of all blocked bundle identifiers. Can use regexes.
blockedBundleIdentifiers = ['com.apple.InstallAssistant.Mojave']

# Whether the blocked application should be deleted if launched
deleteBlockedApplication = False

# Whether the user should be alerted that the launched applicaion was blocked
alertUser = True

# Message displayed to the user when application is blocked
alertMessage = "The application \"{appname}\" has been blocked by IT"
alertInformativeText = "Contact your administrator for more information"

# Use a custom Icon for the alert. If none is defined here, the Python rocketship will be shown.
alertIconPath = "/System/Library/CoreServices/CoreTypes.bundle/Contents/Resources/Actions.icns"

UPDATED NOTE:

To determine the Bundle identifier of other applications you can use osascript

osascript -e 'id of app "iTunes"'
com.apple.iTunes

If you want to block more than one app use a comma separated list in the AppBlocker.py script:

['com.apple.InstallAssistant.Mojave','com.apple.iTunes']

 

Step 3. Run it

Put the script where you want to run it. The default location as defined in the launchd plist included with the app is “/usr/local/bin”. Put the launchd.plist in “/Library/LaunchDaemons/” and start up your launchd to block your apps!

launchctl load /Library/LaunchDaemons/com.github.erikberglund.AppBlocker.plist

Step 4. Automate it

For bonus points we automate! Bundle it all up in a package with munkipkg, then distribute it with Munki to all your clients.

Using munkipkg is easy. Create the folder using munkipkg

./munkipkg --create AppBlocker

munkipkg: Created new package project at AppBlocker

Then you fill the payload folders with those items you downloaded from the AppBlocker project. LauchD plist in the LaunchDaemons folder and AppBlocker.py in the “usr local bin” (create each nested folder).

AppBlocker-Munkipkg3.png

And finally create a post install script (no “.sh”) with the launchctl action to start your plist.

AppBlocker-Munkipkg4.png

Last but not least add this package to your Munki repo as an unattended managed install  that everyone gets. Of course, only do this after testing your package locally somewhere to verify that it works properly. Remember the saying: “You may not test very often, but when you do it’s always in production.” Be very careful with your testing but always automate all the things.

Updated after the initial blog post to explain how to add more than one app to block, and how to use osascript to determine the bundle identifier.

 

 

 

Root Me Baby One More Time!

UPDATE: Apple has posted a security update. 2017-001

Root-a-pocalyse. Root down. Root a toot toot. Many funny tweets today about a very serious issue. A bug was discovered in macOS 10.13 that enabled anyone to login with a root account. With no password. Wow. Seriously. Yeah, that’s bad.

Bug discovered by Lemi Orhan Ergin.

I tested by clicking on the lock icon in System Preferences. Normally this requires an admin account. I was able to authenticate with “root” and no password. This actually also set root to no password. You can choose a password here and this makes it for you. How convenient. You can also login to the Mac via the login window. With root. And no password. Crazy.

If your Mac is off it’s safe. Not joking. If your FileVault protected drive is encrypted and your mac is turned off then you’re good. If you Mac is turned on and you’ve logged in at least once (or at least decrypted the drive on boot) then you’re not safe.

What can you do? Change the root password and set the shell to false. Until Apple fixes this. Should be anytime now. Or soon.

dscl . -passwd /Users/root “random or very secure password here”

dscl . -create /Users/root UserShell /usr/bin/false

Read a comprehensive explanation on Rich Trouton’s site:  Der Flounder blog