A blog about archives, storage and everything

Blog

  • MacDevOps Manifesto

    I was explaining Munki (and autopkg) to some colleagues when I hit on the idea of the MacDevOps manifesto.

    Munki and friends (apps used to augment and extend Munki) are helpful automation tools. Setting up automation systems take time and must be maintained and grown but they pay big dividends.  Freeing us to do Dev work or other tasks they automate and iterate and repeat and build our systems in the way we want.

    No more 100 machines built in a hundred different ways (unless we want to). But now we can check at a glance in MunkiReport to verify that indeed the latest Adobe Flash patch is installed. That may make our lives better. Especially if we need to satisfy corporate IT or our bosses that we are up to date and patched as required.

    The MacDevOps Manifesto Part 1: Munki and friends

    Munki is at its core free software created by Greg Neagle at Disney Animation and used worldwide in many different ways but essentially to distribute apps and run scripts on client workstations. There are many ways to customize it and if fits many different workflows. The MacDevOps:YVR conference I ran last June turned out to be a Munki love-in and showed me the many awesome and varied ways organizations are using it.

    With AutoPkg, another free Mac open source project, Munki can get the latest updates to any software that it has recipes for and by extension install them on clients immediately. This fits the workflow of having Flash, Java and web browsers (Chrome or FireFox) updated as soon as possible for security patches. Exploits on the Mac are coming from these entry points and if you need to use these apps or plugins then having the latest versions helps. For this feature alone I use Munki. In a few months you will see that Munki with AutoPkg has downloaded dozens of versions of each app and keeping up with this takes time away from other tasks. Automation of simple tasks frees up our time so we can focus on other things. That is MacDevOps.

    I also use Munki for installation of any app that is needed everywhere. If I have to download or install one app for one client workstation I put it in Munki and it is ready for installation anywhere with a simple click by the user in a self service portal or automatically by choosing managed installs. Of course if there is an app you don’t want installed (flash or Skype or messenger, etc) add it to Munki and mark it as managed uninstall. Done.

    Scripts and files and config Profiles (replacement for mcx, managed preference settings for OS X) can be imported and used to configure workstations to make deployment easy and flexible. Put everything in Munki and then you don’t have to use golden master builds anymore. Buy a new Mac and install the Munki client. Done.

    Add to this Munki Report which gives an excellent dashboard for what is installed and a total inventory of your client Macs. Very useful info which will let you know if you 15 different versions of flash or Photoshop or any app you choose to look for.

    Last but least I always install Watchman Monitoring which reports to a secure cloud (web portal) to automatically monitor for bad drives, Ram, backups not running etc. It’s a great 50ft overview of all your installs and it can alert you immediately when a machine is having issues that you need to deal with (drives 90% full or Xsan volume not mounted, etc).

    I find this combination of Munki and Watchman great for helping me manage my clients and I want to share these ideas about MacDevOps inspired ways of automating systems with everyone. Jump in and get involved with all these projects. You’ll be writing recipes for AutoPkg and sharing cool Munki tips and tricks with all your friends. And maybe like me you will start writing plugins for Watchman to monitor your favourite apps (I’m working on Archiware P5 backup and archive monitoring scripts).

    Good luck to everyone and hope to see you at the next MacDevOps:YVR conference in June 2016. If you can’t make it go to your nearest Mac Dev / IT conference or start your own meet up somewhere local.

  • Move over El Capitan, hello Yosemite!

    With all this talk about El Capitan, Apple’s as of yet unreleased version 10.11 of OS X, and its wondrous new features in Xsan, I think it might be time to upgrade to last year’s breakthrough version of OS X, Yosemite. Sure, you might be excited by the press releases for the built-in DLC in El Capitan but seriously sane folks stay 1-year behind the bleeding nose upgrades provided by Apple. So if OS X 10.11 is all the rage before its released it must be time to seriously consider upgrading that working Xsan running OS X 10.8 or OS X 10.9.

    In my case, I upgraded a working Xsan running on Mac Minis and OS X 10.8.5. Here are some screenshots from the process. As always think worked better than I could have expected, and it is a much easier process that one expects. But stay sharp kids, danger lurks when you wake the dreamer…. Upgrading a SAN is serious business and doing anything like this without proper backups is taking your life in your own hands. In my case, full disk backups on Promise Pegasus RAIDs and full tape backups using Archiware P5.

    Download the Yosemite installer form the App Store. Install. Download the new Server.app from the App Store. Install. Now upgrade your Xsan. That’s it. You’re done. No surprises, aren’t you happy? Ha ha. I’m kidding. The fun is just getting started.

    If you’re actually following along, this isn’t a step by step recipe. Go to Apple’s site and read this Kbase and check out the migration guide.

    Restore Xsan
    Restore Xsan

    Step 1 is to launch the new Server.app, find Xsan Admin. Just kidding, it isn’t there. Enable Xsan, and choose to Restore a previous SAN configuration. That wasn’t hard. High five! Actually, we’re not done yet. Set up OD now. Go!

    Step 2. Set up your Xsan controller as an Open Directory (OD) master. Does’t matter if it’s joined to another domain, Xsan keeps itself organized in OD, so you need it.

    Set up OD
    Set up OD

    Step 3. Admire your upgraded SAN, “how lovely the flowers do smell…. life is good.”

    XSAN LIST
    Xsan list

    Step 4. Where did my Xsan admin go? Where do I add clients? Where are my clients? Huh? What? Why did I upgrade a perfectly working SAN to this version? Ha ha.

    Take it all in, take a good look at what you’ve done to your Xsan. What? Just so the editors could have the latest version of Final Cut Pro (v.10.2.1) which is only compatible with OS X 10.10.4. I see what you’ve done Apple, very clever indeed. Hmm…

    Click on the “Save configuration profile” button and download the profile somewhere. Use this to set up the SAN on your clients. Distribute via Profile Manager or install it manually. Up to you. I haven’t gotten it to work with Munki quite yet. Installing it requires the admin password for the Xsan controller. How convenient.

    When you client is configured you’ll see a Profile in System Preferences. Remove it and your client is un-configured. No more Xsan.prefpane to list volumes and mount or unmount them. Nope. That would be too easy. Learn to love “xsanctl”, as in “xsanctl mount Xsan”. Read some xsanctl tips in this Kbase

    Step 5. Set up a backup Xsan controller. You have one of those, right? In my case, I had a client which I wanted to promote to be a controller.  But first what to do about its status a client of the Xsan?

    backup cannot be client
    backup cannot be client

    Open Server.app, enable Xsan, join current Xsan as a backup controller and set up a replica OD. Confirm, confirm, confirm. Think about what you’re doing, then do it!

    confirm OD replica
    confirm

    Apple wizards are the best wizards, uh, i mean Setup Assistants. No wizards here…. So, you’ve setup a backup Xsan controller, and OD replica, and now look in Server.app. How amazing is that… wait, what? Where’d my Xsan volumes go? Huh? Where are the controllers? Weird. Very strange. Not comforting at all.

    Xsan 4 no SAN list crop 122815

    The Xsan window eventually shows the volumes and controllers, bur geez, almost gave me a heart attack. It’s not like I never seen Xsan go bad before. Xsan 1 nightmare still haunt me. They do. Backups. Need more backups. Archiware P5 Backups, do it now!

    OK, you’ve survived the uncertainty of Xsan upgrades…. But wait more minute… cat the fsnameservers (no, it’s not the name of a band, it’s a command). Check it out. Holy smokes, batman. Xsan 4 by default will set your metatadata network to the public LAN, something that’d would be laughed at years ago, but they do it now by default. Of course, upgrading our SAN kept out metadata network the same. But strangely the Xsan backup controller is set to use the public for metadata when the primary controller is not. WTF.

    Change your metadata network. Read the Kbase, and once again wield xsanctl like a boss.

  • Affordable Shared Storage: Accusys A16T2-Share Thunderbolt SAN

    Accusys A16T2-Share Thunderbolt SAN storage
    Accusys A16T2-Share Thunderbolt SAN storage

    I’ve mentioned the Accusys A16T2-Share Thunderbolt SAN storage before. I first encountered it at NAB 2015 back in April. It was truly a magical find. A 16-drive RAID unit with 64TB raw storage and ready to be part of your Apple Xsan using Thunderbolt connectors. No Thunderbolt to fibre channel adapters necessary, nor is a fibre channel switch required. Just plugin a Mac Mini as your Xsan controller, and there’s room for 3 other clients to plugin with Thunderbolt. Pretty neat.

    It’s not for everyone. There’s a limit to the length of Thunderbolt optical cables. And there’s only 3 clients possible using one last Thunderbolt connection for an Xsan controller. But this could work well for small work groups. The magical 4-seat SAN setup is finally here. Bob Kite, master SAN builder, would be happy.

    I’ve just gotten word that this unit is available for sale now, as it was recently certified by both Apple and Intel. I believe the street price is approximately $9900 USD, but don’t quote me on that. Information is still emerging on this new product. I do hope to get my hands on one to do some real world application testing. Including testing resilience to failure (drive removals, raid rebuilds, etc).

    If the reliability of the Accusys hardware is good, and backed up with solid support then this could be a great product.
    Of course, it kinda goes without saying that all storage, including Shared SAN storage, needs solid backups. My preference where possible is a second tier disk (NAS/SAN/DAS) and tape. My preferred vendor is Archiware which makes the very excellent product P5.
    With over 10 years of experience with Xsan and setting up storage systems I have learned to always setup excellent backups. Restoring files when a RAID or SAN fails is crucial. Using an a RAID, such as the Accusys Thunderbolt A16T2-Share, as a SAN would be come with a recommendation from me to all my clients to have a secondary Thunderbolt RAID unit of the same size to sync it daily and to use P5 Backup with LTO 6 tape drives (preferably in a tape library). You never can be too paranoid with a client’s data. That’s what my clients pay me for. To plan for failure. I love setting them with excellent SAN storage, but I must counsel them to build also an excellent backups system. Better safe than sorry.
    The price point on this A16T2-Share Thunderbolt RAID is attractive, and when it is released many clients in small shops may very well consider it. At approximately $10K US for 64TB of shared storage with free Xsan 4 it is a pretty sweet deal. Add another $5-10K for a large backup RAID drive, and another $5K for a tape library and $2K for Archiware P5 Backup and the cost adds up. But having an affordable price for the main SAN storage makes this a very real possibility for some clients who have been struggling with editing video on NAS like QNAP, Synology or Drobo.

  • Munki discussion groups

    As posted on the munki-dev list by Greg Neagle and posted https://github.com/munki/munki/wiki/Discussion-Group, a list of discussion groups related to Munki:

    Discussion group for the development of Munki is here: http://groups.google.com/group/munki-dev

    Other related discussion groups:
    General Munki discussion: https://groups.google.com/forum/#!forum/munki-discuss
    MunkiAdmin: https://groups.google.com/forum/#!forum/munkiadmin
    MunkiReport: https://groups.google.com/forum/#!forum/munkireport
    MunkiWebAdmin: https://groups.google.com/forum/#!forum/munki-web-admin
    Sal: https://groups.google.com/forum/#!forum/sal-discuss
    Simian: https://groups.google.com/forum/#!forum/simian-discuss

  • Mac IT training

    I’ve been asked about Mac IT sysadmin training ideas, and I thought perhaps sharing some initial ideas might help other aspiring Mac IT sysadmins.

    It’s hard to recommend a training site or specific place for Mac IT training these days. It used to be that you could sign up for a Unix SysAdmin or bash scripting class and that would be most helpful. And those both are helpful but Mac IT specific technologies are discussed at Mac IT conferences such as MacTech in LA (and periodically in other cities), PSU Mac Admins (Penn State), MacIT (San Jose), MacSysAdmin in Sweden, MacDeploy in Calgary and MacDevOps:YVR in Vancouver. As well as Apple sponsored camps for ACNs (Apple Consultants). Some of these conferences post their video online for free (PSU, MacSysAdmin and MacDevOps) so these are great resources to learn from.

    Online learning: there are resources such as mail lists which have lots of helpful Mac Admins: mac-enterprise (MACENTERPRISE@LISTS.PSU.EDU), munki-dev (munki-dev@googlegroups.com), munki-discussion (munki-discussion@googlegroups.com), autopkg (autopkg-discuss@googlegroups.com) and the Slack MacAdmins channel (http://macadmins.org) and perhaps lots of other places I’m forgetting.

    The most important is local meetups, and right now we’re not hosting anything in Vancouver. But I do want to start some local meetups here to encourage fellow Mac Admins to help each other out learning MacDevOps skills and getting into the greater Mac IT sysadmin community. There are great meetups in many cities across North America and the world. Finding out where they are means looking at afp548.com for info, or check out Watchman Monitoring’s new Mac Meetup site: http://www.macmeetups.com/find/, or check on Jamf Nation (https://jamfnation.jamfsoftware.com/index.html). Lots of options.
    Here in Vancouver we are also contemplating some more in depth workshops for the next MacDevOps:YVR conference and perhaps sooner than that. How to get started with Mac IT technologies and how to build a beautiful Mac Admins community by being a part of it, and sharing what you know with others.
    Anything you all want to learn specifically? What do you want to know about? Let me know.

  • Accusys Thunderbolt shared storage

    I saw something wonderful and new at NAB today. The Accusys ExaSAN Thunderbolt shared storage shouldn’t exist but there it is. With 16 x 4TB drives and 4 Thunderbolt ports out the back to connect 4 clients (one of which could be a Mac mini acting as a Xsan 4 controller). That leaves 3 spots for Mac clients for a small editing SAN or post-production workflow. Prices to be revealed soon, I hope. Very interesting for small setups.

    AccusysExaSAN_TB - 1 AccusysExaSAN_TB - 2 AccusysExaSAN_TB - 3 AccusysExaSAN_TB - 4

  • Munki tricks: Import Adobe CC apps

    In our ongoing quest to use Munki to manage all software, one eventually gets to the realization that Adobe software must be distributed as well. How we do this?

    With Adobe CC Team you can use the excellent CCP (creative cloud packaging) tool to make packages with the settings you want (users can or can’t update, importantly).

    Once you have all these packages what do you do? Grab Tim Sutton’s “munkiimport_cc_installers.py” script and scan your folder with all your newly created package and you’re on your way.

    https://github.com/timsutton/aamporter/blob/master/scripts/munkiimport_cc_installers.py

    Example:

    $ sudo ./munkiimport_cc_installers.py /tmp/CC/ –subdirectory “apps/Adobe/CC/2014” –developer “Adobe” –category “Media”
    Password:
    Making disk image containing AE-CC2014_Install.pkg…
    created: /tmp/munki-3aetVZ/AE-CC2014_Install.dmg
    Disk image created at: /tmp/munki-3aetVZ/AE-CC2014_Install.dmg
    Making disk image containing AE-CC2014_Uninstall.pkg…
    created: /tmp/munki-3aetVZ/AE-CC2014_Uninstall.dmg
    Disk image created at: /tmp/munki-3aetVZ/AE-CC2014_Uninstall.dmg
    Copying AE-CC2014_Install.dmg to /Users/Shared/munki_repo/pkgs/apps/Adobe/CC/2014/AE-CC2014_Install-13.0.0.dmg…
    Copying AE-CC2014_Uninstall.dmg to /Users/Shared/munki_repo/pkgs/apps/Adobe/CC/2014/AE-CC2014_Uninstall-13.0.0.dmg…
    Saving pkginfo to /Users/Shared/munki_repo/pkgsinfo/apps/Adobe/CC/2014/AE-CC2014-13.0.0…
    Making disk image containing Pho-CC2014_Install.pkg…
    created: /tmp/munki-a005sR/Pho-CC2014_Install.dmg
    Disk image created at: /tmp/munki-a005sR/Pho-CC2014_Install.dmg
    Making disk image containing Pho-CC2014_Uninstall.pkg…
    created: /tmp/munki-a005sR/Pho-CC2014_Uninstall.dmg
    Disk image created at: /tmp/munki-a005sR/Pho-CC2014_Uninstall.dmg
    Copying Pho-CC2014_Install.dmg to /Users/Shared/munki_repo/pkgs/apps/Adobe/CC/2014/Pho-CC2014_Install-15.0.dmg…
    Copying Pho-CC2014_Uninstall.dmg to /Users/Shared/munki_repo/pkgs/apps/Adobe/CC/2014/Pho-CC2014_Uninstall-15.0.dmg…
    Saving pkginfo to /Users/Shared/munki_repo/pkgsinfo/apps/Adobe/CC/2014/Pho-CC2014-15.0…

  • Umask fixes in Yosemite aka OS X 10.10.3 and shared storage

    Finally!

    Yes, Apple has restored the ability to set a user and system umask in OS X 10.10.3. This is a huge fix for users of shared storage. Xsan and all SANs where users want to be able to share files, projects and all things without using ACLs or any LDAP directory. This is great. I am jumping up and down. So happy. So many people wanted this. Anyone using shared storage have been demanding this since the upgrade to Yosemite. 10.10.3 is out today and we will be happy.

    Reference: https://support.apple.com/en-us/HT201684

    tl;dr

    sudo launchctl config user umask nnn

    and

    sudo launchctl config system umask nnn

  • Munki: Part 3 aka Setting up MunkiReport-PHP to monitor your Munki Setup

    This is Part 3 in our series on getting started with Munki. Part 1 covered the basic installation of Munki and the elusive Part 2 covers using Munki Admin and Munki’s Managed Software Centre. Part 3 covers the after you’ve setup Munki, now what part of the deployment. Munki is installed. Software is downloaded via AutoPkg and manifest contain catalog and clients have manifests. But is it working? Do all the Macs have the latest Flash plugin? Do they really? We will cover basic setup of MunkiReport-PHP to show  easy it can be to get going.

     Step 1. Download munkireport-php, download ZIP
    Note: It’s a good idea to read through the setup notes on the site
    Step 2. Rename folder to “report”
    Rename the the expanded folder to whatever you like, I shorten it to “report”
    Step 3. Drop in Munki_repo folder
    To get started quickly drop this folder into your munki repo site folder (which is presumably accessible via the web for client access to Munki).
    rename the munkireport folder
    rename the munkireport folder
    Step 4. Change perms of app/db folder
    Your set up will fail is the app/db folder is not accessible. Make it writable.
    Note: Do not make your site accessible to the outside Internet if you’re not confident in your security model. This is for inside your LAN testing. Be careful and mindful of security concerns. ‘Nough said
    MunkiReport app-db-sqlite
    MunkiReport app-db-sqlite

    If you get this error you haven’t changed the permissions it requires.

    Error
    MunkiReport errors
    Step 5. Rename default_config to config.php
    rename the default config php file
    rename the default config php file
    Step 6. Enable php for apache
    Enable PHP in Apache or other web server service you’re using. Example below is using Server.app
    Enable PHP server.app
    Enable PHP server.app
    Step 7. Create user (and hash)
    Load up your MunkiReport-PHP site and create a user and hash.
    Step 8. Add hash to config.php
    hash-crop
    Step 9. Download MunkiReport.plist using curl
    Download MunkiReport.plist using curl. Note: use an IP address accessible to your Munki clients, i.e. not ‘localhost’
    curl
    Step 10. Add MunkiReport.plist to pkgsinfo
    Add MunkiReport.plist to pkgsinfo
    MunkiReport pkgsinfo
    MunkiReport pkgsinfo
    Step 11. Import in munki repo
    Use munkiimport or MunkiAdmin to import your MunkiReport.plist to Munki
    Step 12. Add to client manifest
    Add MunkiReport.plist to client manifest using Munki Admin or Munki cli tools
    MunkiReport plist installs in MunkiAdmin
    MunkiReport plist installs in MunkiAdmin
    Step 13. Add apps to monitor in config.php
    Use the apps_to_track model. See also Rsaeks blog post.
    MunkiReport apps to track
    MunkiReport apps to track
    Step 14. Login and check your App Versions report
    MunkiReport app versions
    MunkiReport app versions

    Step 15. Explore Munki Report

    MunkiReport-Dashboard
    MunkiReport-Dashboard

  • MacDevOps:YVR

    Date: June 19, 2015

    http://www.macdevops.ca/

    A new kind of conference for Mac IT professionals looking to get into DevOps. You’ll hear some about new automation tools, and get a chance to try new things in the computer lab. Join us! Registration limited to 75.

    The cost is $99. Food is included on the day of the conference including a light breakfast and lunch. Register here.

    Call for Submissions!

    MacDevOpsYVR is seeking presenters from across the Pacific Northwest and beyond to participate in this one-day conference for all things Mac!

    If you have an idea for a specific talk, workshop or panel related to deploying Macs in enterprise, corporate or educational environments, we want to hear from you.

    > SUBMIT A PROPOSAL <

    Deadline for Submissions: March 31, 2015.

    Share your experience and join your peers at this one day, all day conference in beautiful Vancouver, BC.

    Topics of Interest:

    • Puppet, Chef and other automation from Desktop to Cloud and back
    • Software deployment with Munki and AutoPkg: the app ecosystem surrounding it
    • Cool tools: demo of awesome Mac Admin projects from GitHub
    • DevOps: How to adopt Automation and Best practices in IT operations
    • Dev skills: workshops on Ruby, Git, Python, Javascript for Mac Admins
    • MDM: Profiles and Mac configuration management in the cloud