Author: Mat X

  • Thoughts on Documentation: What are we afraid of?

    People are afraid of documentation… But mostly people just hate it. They don’t like it. They don’t want it. It shouldn’t exist. Fingers in ears. I can’t hear you.

    This is about primal fear. And hate. I hate hate. But these are real emotions. Let’s deal with it. What is the reality? Why is documentation is ignored, abandoned, or resisted at all?

    As a Sysadmin perhaps you don’t care about documentation, that is, sharing information with others (co workers / bosses), you want to keep it to yourself. But you care very much about building systems. But there’s perhaps no attempt to explain any of this to anyone else. Who else is there really? No one cares. No one is around that would understand if you explained it.

    Lesson # 1 – Document for yourself.

    Paranoia makes us set up redundant systems for backups. Layers upon layers. Custom scripts and disparate apps. Where was this explained? Documented? Nowhere. Bin dir. Maybe.

    If you could replace all that now with one app that did it all then you would. Time is valuable. Easier to monitor. Easier for someone else to monitor and take over.

    Lesson # 2 – document for your replacement (job change, bus hit)

    Do it continuously. Automate. Or set up systems that work automatically.

    Lesson # 3. DevOps.

    Integrate systems. IT systems manage computer but maybe they also built Inventory. Automatically. Alert Systems report continuously. Living systems report on the state of everything. Documentation is easier when it is current and relevant.

    Lesson # 4. Sustainability

    Commercial vs OpenSource. Support vs excellent team, talent retention and documentation. Pro/Con. If your custom solution is not well documented that can be a big problem. If you code is not shared, peer-reviewed, or supported by anyone that could be an issue. If it makes sense to switch to commercial software that is supported then do it. If an OpenSource project or code is supported by a larger community perhaps that makes sense.

    Lesson # 5. Improve. Grow. Get better.

    Discovery and Documentation lead to suggestions for improvement. Make changes. Code and disparate systems that struggle to be documented make us think about how to replace them or better balance the risks vs cost.

    Lesson # 6. Human problems don’t always tech solutions.

    Code doesn’t fix broken workflows. Meetings are with people. Talking through systems helps people understand pain points. Don’t forget people want to do their job, meet deadlines, do stuff.

    Let’s make their world and our world better.

    Love not hate. Peace.

    Documentation-MatX

  • Thunderbolt SAN talk at Mac Admin meet up

    Big thanks to Ross at Ping Identity for organizing and Jamf for sponsoring the Mac Admin meetup on September 9, 2015.

    We filled the tiny meeting room and we will have to expand to the larger conference room (or theatre) next time. It was a well attended meetup with much discussion of the earlier day’s Apple announcements, new OS X “El Capitan” and iOS 9 changes and how this affects management products like Casper which have had to move the binary because of the new SIP implementation in OS X.

    I opened up the meet up with a presentation on Storage, SANs and the new Accusys Thunderbolt SAN A16T2-Share product.

    The goal of my presentation was to give a quick overview of SAN technology as I’ve seen it change over the last 10 years: from Fibre Channel, to iSCSI to PCIe and Thunderbolt based. The last change to Thunderbolt based SANs is the most interesting for small video production workgroups or anyone that likes working on small scale shared projects but needs a decent bandwidth at an affordable price. Block level storage (SANs) is straight forward storage tech for users and applications to interact with without having to negotiate network protocols (AFP, SMB, or NFS). It’s never been quite that affordable until now.

    Having built a lot of Fibre Channel based SANs for media and entertainment companies and post-production editors in corporate environments I know how awesome and fast and solid these SANs are. Lots of editors and clients can hit a large SAN and it won’t blink. Thirty or Sixty users is not unusual. But not everyone believes in fibre channel or the idea of pulling fibre cables. It is surprisingly a large stumbling block to building large SANs, “no, we don’t want fiber cables”. True, sometimes clients have objected to gigabit Ethernet too, but that’s another story.

    I found that iSCSI, especially with the DDP units I’ve set up, has been a great alternative to fibre channel. Not fiber cables to pull. Just use the CAT6 cables already in place. Great Ethernet based SANs using 1 x or 2 x CAT6 cables per client, or even 10G. Works well. Very well indeed. It’s been great for smaller (and larger) clients who want a great Ethernet iSCSI SAN solution without needing fibre channel cables, switches, HBAs, Thunderbolt adapters, etc.

    That’s why when I stumbled across the Accusys Thunderbolt storage I was kinda really excited. No fibre channel to Thunderbolt adapters. Just use Thunderbolt cables. Brilliant! Finally a solution for small workgroups. And there’s so many video groups sprouting out of every corporate office, or boutique VFX or post-production shops that have been struggling with small NAS solutions that were not meant for video production. Now you can get that SAN that you’ve wanted, you can really get that block-level storage at an affordable price. Instead of working locally and copying raw footage and finished products  back and forth across slow network links they can work in a small video group with high speed storage. Sa-weet. (Can you tell I’m excited?).

    I’ll include the presentation PDF here as a link if anyone is interested. I’ve added a link at the end from Accusys on how to build an Xsan with the A16T2-Share. Yes, Xsan from Apple still exists and is bundled with the OS for free. Building a SAN is pretty easy and everyone can do it. Don’t forget your backups though.

    Lastly, anyone interested in attending any meetings for the upcoming MacDevOps:YVR (June 16-17, 2016) drop me a note. I added the email in the presentation document.

    MatX_SAN_Accusys-Thunderbolt_2015

  • MacDevOps Manifesto

    I was explaining Munki (and autopkg) to some colleagues when I hit on the idea of the MacDevOps manifesto.

    Munki and friends (apps used to augment and extend Munki) are helpful automation tools. Setting up automation systems take time and must be maintained and grown but they pay big dividends.  Freeing us to do Dev work or other tasks they automate and iterate and repeat and build our systems in the way we want.

    No more 100 machines built in a hundred different ways (unless we want to). But now we can check at a glance in MunkiReport to verify that indeed the latest Adobe Flash patch is installed. That may make our lives better. Especially if we need to satisfy corporate IT or our bosses that we are up to date and patched as required.

    The MacDevOps Manifesto Part 1: Munki and friends

    Munki is at its core free software created by Greg Neagle at Disney Animation and used worldwide in many different ways but essentially to distribute apps and run scripts on client workstations. There are many ways to customize it and if fits many different workflows. The MacDevOps:YVR conference I ran last June turned out to be a Munki love-in and showed me the many awesome and varied ways organizations are using it.

    With AutoPkg, another free Mac open source project, Munki can get the latest updates to any software that it has recipes for and by extension install them on clients immediately. This fits the workflow of having Flash, Java and web browsers (Chrome or FireFox) updated as soon as possible for security patches. Exploits on the Mac are coming from these entry points and if you need to use these apps or plugins then having the latest versions helps. For this feature alone I use Munki. In a few months you will see that Munki with AutoPkg has downloaded dozens of versions of each app and keeping up with this takes time away from other tasks. Automation of simple tasks frees up our time so we can focus on other things. That is MacDevOps.

    I also use Munki for installation of any app that is needed everywhere. If I have to download or install one app for one client workstation I put it in Munki and it is ready for installation anywhere with a simple click by the user in a self service portal or automatically by choosing managed installs. Of course if there is an app you don’t want installed (flash or Skype or messenger, etc) add it to Munki and mark it as managed uninstall. Done.

    Scripts and files and config Profiles (replacement for mcx, managed preference settings for OS X) can be imported and used to configure workstations to make deployment easy and flexible. Put everything in Munki and then you don’t have to use golden master builds anymore. Buy a new Mac and install the Munki client. Done.

    Add to this Munki Report which gives an excellent dashboard for what is installed and a total inventory of your client Macs. Very useful info which will let you know if you 15 different versions of flash or Photoshop or any app you choose to look for.

    Last but least I always install Watchman Monitoring which reports to a secure cloud (web portal) to automatically monitor for bad drives, Ram, backups not running etc. It’s a great 50ft overview of all your installs and it can alert you immediately when a machine is having issues that you need to deal with (drives 90% full or Xsan volume not mounted, etc).

    I find this combination of Munki and Watchman great for helping me manage my clients and I want to share these ideas about MacDevOps inspired ways of automating systems with everyone. Jump in and get involved with all these projects. You’ll be writing recipes for AutoPkg and sharing cool Munki tips and tricks with all your friends. And maybe like me you will start writing plugins for Watchman to monitor your favourite apps (I’m working on Archiware P5 backup and archive monitoring scripts).

    Good luck to everyone and hope to see you at the next MacDevOps:YVR conference in June 2016. If you can’t make it go to your nearest Mac Dev / IT conference or start your own meet up somewhere local.

  • Move over El Capitan, hello Yosemite!

    With all this talk about El Capitan, Apple’s as of yet unreleased version 10.11 of OS X, and its wondrous new features in Xsan, I think it might be time to upgrade to last year’s breakthrough version of OS X, Yosemite. Sure, you might be excited by the press releases for the built-in DLC in El Capitan but seriously sane folks stay 1-year behind the bleeding nose upgrades provided by Apple. So if OS X 10.11 is all the rage before its released it must be time to seriously consider upgrading that working Xsan running OS X 10.8 or OS X 10.9.

    In my case, I upgraded a working Xsan running on Mac Minis and OS X 10.8.5. Here are some screenshots from the process. As always think worked better than I could have expected, and it is a much easier process that one expects. But stay sharp kids, danger lurks when you wake the dreamer…. Upgrading a SAN is serious business and doing anything like this without proper backups is taking your life in your own hands. In my case, full disk backups on Promise Pegasus RAIDs and full tape backups using Archiware P5.

    Download the Yosemite installer form the App Store. Install. Download the new Server.app from the App Store. Install. Now upgrade your Xsan. That’s it. You’re done. No surprises, aren’t you happy? Ha ha. I’m kidding. The fun is just getting started.

    If you’re actually following along, this isn’t a step by step recipe. Go to Apple’s site and read this Kbase and check out the migration guide.

    Restore Xsan
    Restore Xsan

    Step 1 is to launch the new Server.app, find Xsan Admin. Just kidding, it isn’t there. Enable Xsan, and choose to Restore a previous SAN configuration. That wasn’t hard. High five! Actually, we’re not done yet. Set up OD now. Go!

    Step 2. Set up your Xsan controller as an Open Directory (OD) master. Does’t matter if it’s joined to another domain, Xsan keeps itself organized in OD, so you need it.

    Set up OD
    Set up OD

    Step 3. Admire your upgraded SAN, “how lovely the flowers do smell…. life is good.”

    XSAN LIST
    Xsan list

    Step 4. Where did my Xsan admin go? Where do I add clients? Where are my clients? Huh? What? Why did I upgrade a perfectly working SAN to this version? Ha ha.

    Take it all in, take a good look at what you’ve done to your Xsan. What? Just so the editors could have the latest version of Final Cut Pro (v.10.2.1) which is only compatible with OS X 10.10.4. I see what you’ve done Apple, very clever indeed. Hmm…

    Click on the “Save configuration profile” button and download the profile somewhere. Use this to set up the SAN on your clients. Distribute via Profile Manager or install it manually. Up to you. I haven’t gotten it to work with Munki quite yet. Installing it requires the admin password for the Xsan controller. How convenient.

    When you client is configured you’ll see a Profile in System Preferences. Remove it and your client is un-configured. No more Xsan.prefpane to list volumes and mount or unmount them. Nope. That would be too easy. Learn to love “xsanctl”, as in “xsanctl mount Xsan”. Read some xsanctl tips in this Kbase

    Step 5. Set up a backup Xsan controller. You have one of those, right? In my case, I had a client which I wanted to promote to be a controller.  But first what to do about its status a client of the Xsan?

    backup cannot be client
    backup cannot be client

    Open Server.app, enable Xsan, join current Xsan as a backup controller and set up a replica OD. Confirm, confirm, confirm. Think about what you’re doing, then do it!

    confirm OD replica
    confirm

    Apple wizards are the best wizards, uh, i mean Setup Assistants. No wizards here…. So, you’ve setup a backup Xsan controller, and OD replica, and now look in Server.app. How amazing is that… wait, what? Where’d my Xsan volumes go? Huh? Where are the controllers? Weird. Very strange. Not comforting at all.

    Xsan 4 no SAN list crop 122815

    The Xsan window eventually shows the volumes and controllers, bur geez, almost gave me a heart attack. It’s not like I never seen Xsan go bad before. Xsan 1 nightmare still haunt me. They do. Backups. Need more backups. Archiware P5 Backups, do it now!

    OK, you’ve survived the uncertainty of Xsan upgrades…. But wait more minute… cat the fsnameservers (no, it’s not the name of a band, it’s a command). Check it out. Holy smokes, batman. Xsan 4 by default will set your metatadata network to the public LAN, something that’d would be laughed at years ago, but they do it now by default. Of course, upgrading our SAN kept out metadata network the same. But strangely the Xsan backup controller is set to use the public for metadata when the primary controller is not. WTF.

    Change your metadata network. Read the Kbase, and once again wield xsanctl like a boss.

  • Affordable Shared Storage: Accusys A16T2-Share Thunderbolt SAN

    Accusys A16T2-Share Thunderbolt SAN storage
    Accusys A16T2-Share Thunderbolt SAN storage

    I’ve mentioned the Accusys A16T2-Share Thunderbolt SAN storage before. I first encountered it at NAB 2015 back in April. It was truly a magical find. A 16-drive RAID unit with 64TB raw storage and ready to be part of your Apple Xsan using Thunderbolt connectors. No Thunderbolt to fibre channel adapters necessary, nor is a fibre channel switch required. Just plugin a Mac Mini as your Xsan controller, and there’s room for 3 other clients to plugin with Thunderbolt. Pretty neat.

    It’s not for everyone. There’s a limit to the length of Thunderbolt optical cables. And there’s only 3 clients possible using one last Thunderbolt connection for an Xsan controller. But this could work well for small work groups. The magical 4-seat SAN setup is finally here. Bob Kite, master SAN builder, would be happy.

    I’ve just gotten word that this unit is available for sale now, as it was recently certified by both Apple and Intel. I believe the street price is approximately $9900 USD, but don’t quote me on that. Information is still emerging on this new product. I do hope to get my hands on one to do some real world application testing. Including testing resilience to failure (drive removals, raid rebuilds, etc).

    If the reliability of the Accusys hardware is good, and backed up with solid support then this could be a great product.
    Of course, it kinda goes without saying that all storage, including Shared SAN storage, needs solid backups. My preference where possible is a second tier disk (NAS/SAN/DAS) and tape. My preferred vendor is Archiware which makes the very excellent product P5.
    With over 10 years of experience with Xsan and setting up storage systems I have learned to always setup excellent backups. Restoring files when a RAID or SAN fails is crucial. Using an a RAID, such as the Accusys Thunderbolt A16T2-Share, as a SAN would be come with a recommendation from me to all my clients to have a secondary Thunderbolt RAID unit of the same size to sync it daily and to use P5 Backup with LTO 6 tape drives (preferably in a tape library). You never can be too paranoid with a client’s data. That’s what my clients pay me for. To plan for failure. I love setting them with excellent SAN storage, but I must counsel them to build also an excellent backups system. Better safe than sorry.
    The price point on this A16T2-Share Thunderbolt RAID is attractive, and when it is released many clients in small shops may very well consider it. At approximately $10K US for 64TB of shared storage with free Xsan 4 it is a pretty sweet deal. Add another $5-10K for a large backup RAID drive, and another $5K for a tape library and $2K for Archiware P5 Backup and the cost adds up. But having an affordable price for the main SAN storage makes this a very real possibility for some clients who have been struggling with editing video on NAS like QNAP, Synology or Drobo.
  • Munki discussion groups

    As posted on the munki-dev list by Greg Neagle and posted https://github.com/munki/munki/wiki/Discussion-Group, a list of discussion groups related to Munki:

    Discussion group for the development of Munki is here: http://groups.google.com/group/munki-dev

    Other related discussion groups:
    General Munki discussion: https://groups.google.com/forum/#!forum/munki-discuss
    MunkiAdmin: https://groups.google.com/forum/#!forum/munkiadmin
    MunkiReport: https://groups.google.com/forum/#!forum/munkireport
    MunkiWebAdmin: https://groups.google.com/forum/#!forum/munki-web-admin
    Sal: https://groups.google.com/forum/#!forum/sal-discuss
    Simian: https://groups.google.com/forum/#!forum/simian-discuss

  • Mac IT training

    I’ve been asked about Mac IT sysadmin training ideas, and I thought perhaps sharing some initial ideas might help other aspiring Mac IT sysadmins.

    It’s hard to recommend a training site or specific place for Mac IT training these days. It used to be that you could sign up for a Unix SysAdmin or bash scripting class and that would be most helpful. And those both are helpful but Mac IT specific technologies are discussed at Mac IT conferences such as MacTech in LA (and periodically in other cities), PSU Mac Admins (Penn State), MacIT (San Jose), MacSysAdmin in Sweden, MacDeploy in Calgary and MacDevOps:YVR in Vancouver. As well as Apple sponsored camps for ACNs (Apple Consultants). Some of these conferences post their video online for free (PSU, MacSysAdmin and MacDevOps) so these are great resources to learn from.

    Online learning: there are resources such as mail lists which have lots of helpful Mac Admins: mac-enterprise (MACENTERPRISE@LISTS.PSU.EDU), munki-dev (munki-dev@googlegroups.com), munki-discussion (munki-discussion@googlegroups.com), autopkg (autopkg-discuss@googlegroups.com) and the Slack MacAdmins channel (http://macadmins.org) and perhaps lots of other places I’m forgetting.

    The most important is local meetups, and right now we’re not hosting anything in Vancouver. But I do want to start some local meetups here to encourage fellow Mac Admins to help each other out learning MacDevOps skills and getting into the greater Mac IT sysadmin community. There are great meetups in many cities across North America and the world. Finding out where they are means looking at afp548.com for info, or check out Watchman Monitoring’s new Mac Meetup site: http://www.macmeetups.com/find/, or check on Jamf Nation (https://jamfnation.jamfsoftware.com/index.html). Lots of options.
    Here in Vancouver we are also contemplating some more in depth workshops for the next MacDevOps:YVR conference and perhaps sooner than that. How to get started with Mac IT technologies and how to build a beautiful Mac Admins community by being a part of it, and sharing what you know with others.
    Anything you all want to learn specifically? What do you want to know about? Let me know.
  • Accusys Thunderbolt shared storage

    I saw something wonderful and new at NAB today. The Accusys ExaSAN Thunderbolt shared storage shouldn’t exist but there it is. With 16 x 4TB drives and 4 Thunderbolt ports out the back to connect 4 clients (one of which could be a Mac mini acting as a Xsan 4 controller). That leaves 3 spots for Mac clients for a small editing SAN or post-production workflow. Prices to be revealed soon, I hope. Very interesting for small setups.

    AccusysExaSAN_TB - 1 AccusysExaSAN_TB - 2 AccusysExaSAN_TB - 3 AccusysExaSAN_TB - 4

  • Munki tricks: Import Adobe CC apps

    In our ongoing quest to use Munki to manage all software, one eventually gets to the realization that Adobe software must be distributed as well. How we do this?

    With Adobe CC Team you can use the excellent CCP (creative cloud packaging) tool to make packages with the settings you want (users can or can’t update, importantly).

    Once you have all these packages what do you do? Grab Tim Sutton’s “munkiimport_cc_installers.py” script and scan your folder with all your newly created package and you’re on your way.

    https://github.com/timsutton/aamporter/blob/master/scripts/munkiimport_cc_installers.py

    Example:

    $ sudo ./munkiimport_cc_installers.py /tmp/CC/ –subdirectory “apps/Adobe/CC/2014” –developer “Adobe” –category “Media”
    Password:
    Making disk image containing AE-CC2014_Install.pkg…
    created: /tmp/munki-3aetVZ/AE-CC2014_Install.dmg
    Disk image created at: /tmp/munki-3aetVZ/AE-CC2014_Install.dmg
    Making disk image containing AE-CC2014_Uninstall.pkg…
    created: /tmp/munki-3aetVZ/AE-CC2014_Uninstall.dmg
    Disk image created at: /tmp/munki-3aetVZ/AE-CC2014_Uninstall.dmg
    Copying AE-CC2014_Install.dmg to /Users/Shared/munki_repo/pkgs/apps/Adobe/CC/2014/AE-CC2014_Install-13.0.0.dmg…
    Copying AE-CC2014_Uninstall.dmg to /Users/Shared/munki_repo/pkgs/apps/Adobe/CC/2014/AE-CC2014_Uninstall-13.0.0.dmg…
    Saving pkginfo to /Users/Shared/munki_repo/pkgsinfo/apps/Adobe/CC/2014/AE-CC2014-13.0.0…
    Making disk image containing Pho-CC2014_Install.pkg…
    created: /tmp/munki-a005sR/Pho-CC2014_Install.dmg
    Disk image created at: /tmp/munki-a005sR/Pho-CC2014_Install.dmg
    Making disk image containing Pho-CC2014_Uninstall.pkg…
    created: /tmp/munki-a005sR/Pho-CC2014_Uninstall.dmg
    Disk image created at: /tmp/munki-a005sR/Pho-CC2014_Uninstall.dmg
    Copying Pho-CC2014_Install.dmg to /Users/Shared/munki_repo/pkgs/apps/Adobe/CC/2014/Pho-CC2014_Install-15.0.dmg…
    Copying Pho-CC2014_Uninstall.dmg to /Users/Shared/munki_repo/pkgs/apps/Adobe/CC/2014/Pho-CC2014_Uninstall-15.0.dmg…
    Saving pkginfo to /Users/Shared/munki_repo/pkgsinfo/apps/Adobe/CC/2014/Pho-CC2014-15.0…

  • Umask fixes in Yosemite aka OS X 10.10.3 and shared storage

    Finally!

    Yes, Apple has restored the ability to set a user and system umask in OS X 10.10.3. This is a huge fix for users of shared storage. Xsan and all SANs where users want to be able to share files, projects and all things without using ACLs or any LDAP directory. This is great. I am jumping up and down. So happy. So many people wanted this. Anyone using shared storage have been demanding this since the upgrade to Yosemite. 10.10.3 is out today and we will be happy.

    Reference: https://support.apple.com/en-us/HT201684

    tl;dr

    sudo launchctl config user umask nnn

    and

    sudo launchctl config system umask nnn