A blog about archives, storage and everything

Tag: sysadmin

No NetBoot, No problem: installr and bootstrappr
It’s 2019, and NetBoot is almost dead. All new Macs have T2 chips. Sent from the future to protect us from …. ourselves? No more NetBoot, no problem!!

When NetBoot first appeared and I was able to boot entire labs of Macs across the network I was amazed and overjoyed. It was awesome. Spinning globe, spinning…

But in the years since I’ve moved on to no-imaging. Using Munki to manage software means no more imaging, just install Munki and a small config change to point to the Munki server, thereafter the software that should be there goes on, and what’s not supposed to be there goes away. Simple. Just install one package, well, maybe two, then you’re good.

Well, what if you want to streamline or automate these things? What if these are new Macs which don’t have users configured? What if we could do all this from recovery mode? Hmm… Enter bootstrappr and installr!!

bootstrappr

This awesome project allows to add packages to install in one step while booted in recovery mode. Plug in a USB stick with the bootstrapr script to run the package install magic or mount a disk image over http. Create a DMG with the included script make_dmg.sh. And now this is the best part: in recovery mode open the Terminal app from Utilities and type:
```
hdiutil mount http://server/yourDMG.dmg
```
Then:
```
/Volumes/bootstrap/run
```
When it’s done you can Reboot the Mac and you’ll have a set up customized to your liking with Munki installed and configured with custom settings.

installr

The installr script works in the same way but adds the macOS installer to the party. You can also mount the DMG over http and re-image a Mac and then add your custom packages. It’s awesome. Truly amazing.

One note: Added packages in Installr must be in a special format. From the installr site: startosinstall requires that all additional packages be Distribution-style packages (typically built with productbuild) and not component-style packages (typically built with pkgbuild)
```
productbuild --package component.pkg --version x.y --identifier com.example.component distribution.pkg
```
In one of my first tests with installr and pycreateuserpkg I was caught up by this, even though it is properly mentioned in the read me. Packages that work in Bootstrappr or munki directly don’t necessarily work when called by the macOS installer (startoinstall). Armin Briegel was helpful in the MacAdmins Slack and reminded me of this. Thanks Armin and thanks everyone on the MacAdmins Slack.

Many Thanks to Greg Neagle for creating these tools and Munki. Looking forward to hearing him speak at the next MacDevOps:YVR conference June 12-14, 2019. Greg will be speaking about his efforts to port some parts of Munki from Python to Swift. More info on the conference and speakers here: https://mdoyvr.com/speakers/

Also a shout out to Graham Gilbert who has worked on Imagr (MDOYVR talk), over the years, an imaging and automation tool which was also an inspiration (along with bootstrappr and installr) to Tim Perfit and his MDS project.

Update: corrected the names of installr and bootstrappr in the title because… autocorrect.
February 28, 2019
Reset Printer Queue
TIL (thing I learned)

Had a user upgrade to macOS 10.14.1 and no printers showed up anymore.

So using my Google fu I found some posts (see one below) which described a novel way to reset the print queue on macOS. An old trick apparently. Learn something new everyday.

A quick trip to a terminal and it worked! The existing printers returned to System Preferences and printing resumed.
```
$ cancel -a
```
Reference
November 25, 2018
To install macOS Mojave, or not to?
Just the other day macOS Mojave was released and now the armies of Macs armed only with the AppStore are silently downloading the installer and ready to upgrade. You can’t hurry too fast to be on the bleeding edge, hurry faster!

Just in case you don’t want everyone to install macOS 10.14.0 (dot zero!) in the first week of its release here’s a way to slow down the upgrade hordes using Erik Berglund’s AppBlocker script. Erik Berglund is also the author of ProfileCreator (for creating profiles) and the author of many other great scripts.

Note: for true binary whitelisting check out Google’s Santa project and Upvote (and Moroz and Zentral, two other Santa sync servers).

Step 1. Get it

Clone or download the AppBlocker project from GitHub

Step 2. Do it

Edit the AppBlocker.py script with the Bundle Identifier of your app to block, in this case for the Mojave installer from the AppStore it is:
```
com.apple.InstallAssistant.Mojave
```
You can also edit the alert message, and the icon that is shown, as well as decide if the blocked app should be deleted or not. The script is easy to edit in BBEdit, or nano (in Terminal). Use whatever your favorite text editor is to make the necessary changes.
```
# List of all blocked bundle identifiers. Can use regexes.
blockedBundleIdentifiers = ['com.apple.InstallAssistant.Mojave']

# Whether the blocked application should be deleted if launched
deleteBlockedApplication = False

# Whether the user should be alerted that the launched applicaion was blocked
alertUser = True

# Message displayed to the user when application is blocked
alertMessage = "The application \"{appname}\" has been blocked by IT"
alertInformativeText = "Contact your administrator for more information"

# Use a custom Icon for the alert. If none is defined here, the Python rocketship will be shown.
alertIconPath = "/System/Library/CoreServices/CoreTypes.bundle/Contents/Resources/Actions.icns"
```
UPDATED NOTE:

To determine the Bundle identifier of other applications you can use osascript
```
osascript -e 'id of app "iTunes"'
```
```
com.apple.iTunes
```
If you want to block more than one app use a comma separated list in the AppBlocker.py script:
```
['com.apple.InstallAssistant.Mojave','com.apple.iTunes']
```
Step 3. Run it

Put the script where you want to run it. The default location as defined in the launchd plist included with the app is “/usr/local/bin”. Put the launchd.plist in “/Library/LaunchDaemons/” and start up your launchd to block your apps!
```
launchctl load /Library/LaunchDaemons/com.github.erikberglund.AppBlocker.plist
```
Step 4. Automate it

For bonus points we automate! Bundle it all up in a package with munkipkg, then distribute it with Munki to all your clients.

Using munkipkg is easy. Create the folder using munkipkg
```
./munkipkg --create AppBlocker

munkipkg: Created new package project at AppBlocker
```
Then you fill the payload folders with those items you downloaded from the AppBlocker project. LauchD plist in the LaunchDaemons folder and AppBlocker.py in the “usr local bin” (create each nested folder).

And finally create a post install script (no “.sh”) with the launchctl action to start your plist.

Last but not least add this package to your Munki repo as an unattended managed install that everyone gets. Of course, only do this after testing your package locally somewhere to verify that it works properly. Remember the saying: “You may not test very often, but when you do it’s always in production.” Be very careful with your testing but always automate all the things.

Updated after the initial blog post to explain how to add more than one app to block, and how to use osascript to determine the bundle identifier.
September 26, 2018
Root Me Baby One More Time!

UPDATE: Apple has posted a security update. 2017-001

Root-a-pocalyse. Root down. Root a toot toot. Many funny tweets today about a very serious issue. A bug was discovered in macOS 10.13 that enabled anyone to login with a root account. With no password. Wow. Seriously. Yeah, that’s bad.

Bug discovered by Lemi Orhan Ergin.

I tested by clicking on the lock icon in System Preferences. Normally this requires an admin account. I was able to authenticate with “root” and no password. This actually also set root to no password. You can choose a password here and this makes it for you. How convenient. You can also login to the Mac via the login window. With root. And no password. Crazy.

If your Mac is off it’s safe. Not joking. If your FileVault protected drive is encrypted and your mac is turned off then you’re good. If you Mac is turned on and you’ve logged in at least once (or at least decrypted the drive on boot) then you’re not safe.

What can you do? Change the root password and set the shell to false. Until Apple fixes this. Should be anytime now. Or soon.

dscl . -passwd /Users/root “random or very secure password here”

dscl . -create /Users/root UserShell /usr/bin/false

Read a comprehensive explanation on Rich Trouton’s site: Der Flounder blog

November 28, 2017
Archiware P5 and Synology NAS.

Update: As of version 5.4.3 there is an official P5 add-on package for Synology NAS

Archiware P5 available for Synology

Note: The P5 app for Synology NAS first debuted with P5 v.5.3.3

On the Archiware P5 new-features page there’s a blurb about the Synology NAS integration:

From Version 5.3.3, Archiware P5 supports Synology NAS devices without restrictions.

Synology NAS can serve as a data source or target for P5 Synchronize, P5 Backup and P5 Archive. The Archiware P5 application can now be installed on the Synology NAS itself.

Thanks to the snapshot capability of the DSM platform, powerful enterprise Synology NAS devices can also be used as repository for Backup2Go. This setup opens the possibility of introducing a professional data security solution at an affordable price point.

Let’s look in closer detail how to install Archiware P5 on a new Synology NAS.

For this post I have a new Synology 1515+ NAS, installed with five 6TB hard drives (It is very easy to install hard drives. No tools required). Note: I’ve purchased the NAS with my own money and was not paid to write this article.

At the time of this blog post the latest Synology DSM release is 6.1 and Archiware P5 is at version 5.4.2.

Step 1. Download Synology package from Archiware.com/download

Download Archiware P5 for Synology

Requirements are DSM 5.2+ and Intel x86 64-Bit CPU only. (i.e. Atom but not Marvell).

Step 2. Find and Log into your NAS

Find your new NAS with the Synology Assistant app or use this handy website link:

Find your NAS

I had no luck with the app (it found my existing NAS, but not the new one). Using the website I was able to quickly locate the new NAS that I need to log into and setup. Very nice feature.

Step 3. Install the new DSM

Install or update new software. You will be prompted to go through the initial setup to prepare your new NAS.

Step 4. Set up a new volume

Chose the Btrfs or ext4 filesystem. Btrfs supports snapshots, replication, and much more.

Step 5. Monitor the volume setup

Verifying the hard disks will take a moment. Take a break here.

Step 6. Open Package Center

Step 7. Install manually

Install Archiware P5 by selecting the “install manually” option to upload the awpst542.spk downloaded file from archiware.com

Step 8. Agree to continue.

Load the Synology P5 installer by agreeing to continue with this “unknown” publisher.

Sep 9. Agree to trust the installer

Step 10. Confirm the Install

Step 11. P5 is now running on the Synology NAS.

Hooray! P5 is now installed. Select the app to examine the details.

Step 12. Examine the option to stop or uninstall the P5 application

Step 13. Login to the P5 server running on NAS

To login to P5 open a new tab. Pay attention to the port number: “20,000” (vs 8000 on other platforms such as Solaris, Linux, OSX etc).

Step 14. Set up your NAS as a client on another Server

To test the new Synology 1515+ NAS I then set up the NAS as a client on another P5 server, and set up a P5 Sync job to copy data from server with a ZFS based filesystem to the Synology NAS with a btrfs volume.

Testing: Set up the new client in P5 with a name and IP address, then set up a new sync job with source and destination. Start now. Watch the bits fly through the ether. Be happy.

Step 15. Other things to configure

To make your new NAS is working smoothly don’t forget to set up the email notifications, and set up some AFP, SMB, or NFS shares as required.

Take some time to explore the Package Center app and see what other great applications are offered on the Synology NAS.

Synology makes a great low-cost NAS appliance. For SMB or production setups I would recommend two or more (for redundancy, hot or cold spares, disaster recovery, offsite backups/replication). With P5 installed you can Sync your server data to a NAS for onsite or offsite backups, backup your NAS to tape, or use the NAS for your client workstation backups using Backup2Go. Using the new Btrfs filesystem provides many of the same advances as ZFS, including snapshots and replication, over traditional filesystems such as ext4 and hfs which sadly lack these features.

Conclusion:

The Synology NAS is a great experience. Adding Archiware P5 is a recommended way to include this NAS as part of any good backup, archive or DR (disaster recovery) scenario. Two thumbs up. Way up.

References:

Archiware P5 new features

Synology DSM

March 4, 2017
MacDevOps:YVR 2017

We’ve had incredible feedback from the last two events and it was so much fun we’ve decided to do it again. Join us on June 5-6th in Vancouver, BC, Canada. Early bird tickets are on sale now.

As a conference we like to gather to discuss Open Source solutions to manage Macs in the enterprise and everywhere else. This year we focus on the new APFS filesystem and what that means for all of us. How do we manage macOS if it is becoming more closed and like iOS? They’ll be talks on what is MDM?, Is imaging dead?, managing Macs with various open source tools, and how to leverage the cloud.

Join us for the technical talks by speakers from Google, Facebook, Dropbox, Airbnb, Square, Uber and many more. Or hang out in the break room and the hallway track. You’ll meet the awesome community members that make up the MacDevOps family. We are all here to share what we know, and to learn from others.

For more information go to our website:

MacDevOps:YVR website

A limited number of early bird tickets are on sale now at Eventbrite:

Get your early bird ticket now!

February 3, 2017
Hello macOS Sierra, bye bye El Cap
We welcome the beautiful and wonderful macOS Sierra (10.12) and say good bye to the old and weary El Capitan (OS X 10.11.6)…. Wait, not so fast. Slow it down. Just a tad bit.

While Watchman is alerting me to users downloading, then installing the newest Apple macOS (née OS X, Mac OS X), we must be ready. Ready to troubleshoot issues with apps that developers haven’t tested thoroughly for a new OS that appears to be the same, but changes everything under the hood.

How do we test? In a VM of course.

What do we need:
1. VMWare Fusion
2. Greg et. al. createOSXinstallPkg
3. Rich Trouton’s disable setup assistant payload free packages
4. Mager Valp’s Create User Pkg
5. Greg et. al. Munki (latest release)
6. add your own packages, such as a munki kicksart (set repo url, client identifier, etc)
7. UPDATE: we can’t forget Rich Trouton’s First Boot Generator App
What are we doing?

createOSXinstallPkg was created to turn Apple’s App Store Install macOS Sierra.app or previous Install OS X versions into nice Apple installer packages to upgrade in place using Munki (or other deployment tools). The new trick added recently is to create a new Fake Install.app with our packages to use install in VMWare Fusion instead of on a real Mac.

UPDATED STEPS! Note: I’d forgotten about First Boot Generator
1. Download your installer app of choice (Install macOS sierra)
2. Download createOSX installer
3. Prepare your custom packages, or gathers ones your want to add to the installer
4. Organize your installers into folders like this: 00, 01, 02, etc
5. Launch First Boot Generator App and transmogrify that folders of packages
6. Run createOSX installer with the fake app option if you want to test a VM, or without if you want to build a package
7. Run createOSX as many times as you want with different OS X installers, and the same first boot package. Test diff OS installers with your customer PKGs.
Note: use the “–make-fake-app” option to prep for VMWare Fusion, omit it for a pkg

Note2: Here’s some examples using createOSXinstallPkg and various OSX installers
```
createOSXinstallPkg sudo ./createOSXinstallPkg --source /Volumes/SSD/Install\ macOS\ Sierra.app --make-fake-app --pkg ~/bin/PKG_BUILD/FirstBoot_staging/First\ Boot\ Package\ Install.pkg --output /Volumes/Updates/Builds

createOSXinstallPkg sudo ./createOSXinstallPkg --source /Volumes/Updates/Builds/Install\ OS\ X\ El\ Capitan.app --pkg ~/bin/PKG_BUILD/FirstBoot_staging/First\ Boot\ Package\ Install.pkg --output /Volumes/Updates/Builds
```
Note: If you get a message that your custom pkg you want to add is not a Flat package then use productbuild to repackage it.

Example:

➜ productbuild –package SetMunkiRepo.pkg SetMunkiRepo_flat.pkg

Reference: See Greg’s post on Managing OS X for more info make VMWare images using this method. And also Rich Trouton’s Der Flounder blog post on First Book Generator App
September 21, 2016
Packaging and deploying software
I am about to send an email to a software vendor asking them to please consider shipping their apps in a deployable Apple PKG format and I wanted to ask if anyone has some boilerplate text, excellent blog entry or list of arguments I can use. I could have posted in the MacAdmins slack, tweeted or posted a lovely photo on Instagram, but instead I sent an email to the MacEnterprise mail list.

Hat tip to Rick Heil on the MacEnterprise for pointing me to this post on AFP548 by Gary Larizza in June 2010.

“This one is an oldie but a goodie. It hits all my pain points, such as not assuming GUI interaction and minimizing pre/post scripts.”

https://www.afp548.com/2010/06/03/the-commandments-of-packaging-in-os-x/

Gary outlines his thesis in six rules:
1. Do not assume that your package will be installed interactively via the GUI or on the currently booted volume.
2. Unnecessary actions are unnecessary.
3. Licensing should have the option to be managed by Systems Administrators.
4. Use pre/post-install scripts only when necessary
5. Be true to the Operating System
6. Naming Conventions are Necessary and Helpful
All software vendors should aspire to follow these rules.

We should always send feedback to software vendors explaining carefully why their Mac OS X installers are not optimal for deployment when they are custom apps (e.g. InstallAnywhere) and not in Apple package format (i.e. PKG). Also, if the installers (as well as the app) require the legacy Java 6 then this seems to be a security risk and it is our duty to provide feedback if we hope to improve the situation in the future.

Another great source of information is Der Flounder, Rich Trouton’s blog, is worth perusing because of Rich’s excellent documentation and many excellent posts, including this one about re-packaging app:

“Using AutoPkg to build installer packages from installer applications” from May 24, 2016.

Reference: Re-packing using Auto PKG

As Rich succinctly puts it: “One of the challenges Mac admins have to deal with are Mac application installers which don’t follow one of the following models: Drag-and-drop installation or Package installation”.

Greg’s managingosx blog has many articles on packaging and I thank you for taking the time to write all those posts. We benefit greatly from all the work of everyone in this community. Greg has spoken at many conferences and given great packaging workshops.

Reference: packaging blog posts on Managing OSX
My personal preference for software deployment is to use Munki to deploy apps and not have to deploy app manually. While Mac sysadmins may use difference software for deployment, I think we are all in agreement in not wanting to do this manually when it can be automated. I don’t have a large IT team, and simple solutions based with Munki are best for me. Hence my desire for vendors to use the Apple PKG format where possible.
August 17, 2016
Watchman Monitoring + Archiware P5
I’ve been a little busy lately. I’m working on some scripts for Watchman Monitoring that alert when Archiware P5 needs attention. It’s really much more exciting than it sounds. 🙂

Archiware P5 plugin (included with Watchman Client 6.6.0)

UPDATE: The Archiware P5 plugin is now included with the Watchman Monitoring client version 6.6.0

Use the link above to read up about Watchman Monitoring and the Archiware P5 plugin.

This plugin is now part of Watchman Monitoring thanks to Allen and his team! Of course, big thanks to a lot of help from Python magician and MacDevOps:YVR colleague, Wade Robson. I couldn’t have finished this plugin without his help. Merci, mon ami. (Early help to get started with this project is thanks to Scott Neal, automation expert and programming wizard. Thank you so much Scott, and thanks for the tasty Portland beer!).

Watchman Monitoring is a group of plugins that will warn when drives are failing, computers have restarted unexpectedly or backups are not running. All reporting goes to a beautiful web interface in the cloud which can keep a history of plugin issues. Watchman allows for integration with ticket systems and multiple users including clients and IT staff that can keep track of what’s up with their workstations, and servers.

Watchman Monitoring helps me keep tabs of major issues at all my clients before they become disasters. I even use it in discovery for new clients to see what issues exist but are ignored or unknown.

Since I set up a lot of SAN storage for my clients, and I use Archiware P5 for backups and archives I realized I needed to write a plugin for Watchman Monitoring that alerts me to issues. Instead of remoting in with VPN to each and every client every day to check on backups the only alternative is to automate it. These scripts watch the LTO tape drives and emails when they need cleaning, or warns when running jobs need tapes, if workstations haven’t backed up in a while or if tape pools need more tapes. And in Beta 2 we’ve added a check to see if the P5 maintenance support needs to be renewed to give you time to renew it before it expires. As well as better alerts for issues with running jobs, and lots of bug fixes.

We have it working on Mac servers running Archiware P5 and the next step is Linux, and the Unix family. Later on, Watchman will port it to Windows. The scripts are written in Python which is great for portability (except to Windows. Ha ha). And the P5 Watchman plugins should eventually run everywhere that Archiware P5 runs (OS X, Linux, FreeBSD, Solaris and Windows).

The best part of writing plugins for Watchman Monitoring is the great help that Allen and the whole team at Watchman have given us been throughout our development of these Archiware P5 plugins. And of course everyone at Archiware and Mike at PVT have been super helpful in explaining the use of the nsdchat cli for Archiware P5, even going so far as to add some features we needed to nsdchat when we explained how useful they’d be for this project. Mille mercis. Vielen danke.

Using GitHub to check code in, document business logic, write code, build a wiki and then track issues that need bug fixes or enhancement requests has been an adventure. It all starts with an problem that you want to be alerted for. It’s easy enough to add custom plugins to Watchman Monitoring you just need some ideas, a programmer (or two) and some time for testing, debugging, more testing and time. Did I mention you need lots of time? Ha ha

And now for a sneak peak of the Archiware P5 beta 2 plugins for Watchman Monitoring.
1. Watchman nicely lists the new warnings and expirations for quickly getting to the issues you need to see.
2. Expirations are tracked with Watchman. In this case we note the date when the maintenance for Archiware P5 needs to be renewed. Don’t want to miss that!
3. Server info is good to know. Uptime, port used, and what exactly is licensed.
4. The LTO tape drive is the heart of any tape library, and alerting when it needs cleaning is very important.
5. Other plugins watch the tape pools, running and completed jobs, as well as Backup2Go (workstation backup).
June 2, 2016
Thoughts on Documentation: What are we afraid of?

People are afraid of documentation… But mostly people just hate it. They don’t like it. They don’t want it. It shouldn’t exist. Fingers in ears. I can’t hear you.

This is about primal fear. And hate. I hate hate. But these are real emotions. Let’s deal with it. What is the reality? Why is documentation is ignored, abandoned, or resisted at all?

As a Sysadmin perhaps you don’t care about documentation, that is, sharing information with others (co workers / bosses), you want to keep it to yourself. But you care very much about building systems. But there’s perhaps no attempt to explain any of this to anyone else. Who else is there really? No one cares. No one is around that would understand if you explained it.

Lesson # 1 – Document for yourself.

Paranoia makes us set up redundant systems for backups. Layers upon layers. Custom scripts and disparate apps. Where was this explained? Documented? Nowhere. Bin dir. Maybe.

If you could replace all that now with one app that did it all then you would. Time is valuable. Easier to monitor. Easier for someone else to monitor and take over.

Lesson # 2 – document for your replacement (job change, bus hit)

Do it continuously. Automate. Or set up systems that work automatically.

Lesson # 3. DevOps.

Integrate systems. IT systems manage computer but maybe they also built Inventory. Automatically. Alert Systems report continuously. Living systems report on the state of everything. Documentation is easier when it is current and relevant.

Lesson # 4. Sustainability

Commercial vs OpenSource. Support vs excellent team, talent retention and documentation. Pro/Con. If your custom solution is not well documented that can be a big problem. If you code is not shared, peer-reviewed, or supported by anyone that could be an issue. If it makes sense to switch to commercial software that is supported then do it. If an OpenSource project or code is supported by a larger community perhaps that makes sense.

Lesson # 5. Improve. Grow. Get better.

Discovery and Documentation lead to suggestions for improvement. Make changes. Code and disparate systems that struggle to be documented make us think about how to replace them or better balance the risks vs cost.

Lesson # 6. Human problems don’t always tech solutions.

Code doesn’t fix broken workflows. Meetings are with people. Talking through systems helps people understand pain points. Don’t forget people want to do their job, meet deadlines, do stuff.

Let’s make their world and our world better.

Love not hate. Peace.

Documentation-MatX

November 8, 2015