A blog about archives, storage and everything

Tag: Mac

Watchman Monitoring + Archiware P5
I’ve been a little busy lately. I’m working on some scripts for Watchman Monitoring that alert when Archiware P5 needs attention. It’s really much more exciting than it sounds. 🙂

Archiware P5 plugin (included with Watchman Client 6.6.0)

UPDATE: The Archiware P5 plugin is now included with the Watchman Monitoring client version 6.6.0

Use the link above to read up about Watchman Monitoring and the Archiware P5 plugin.

This plugin is now part of Watchman Monitoring thanks to Allen and his team! Of course, big thanks to a lot of help from Python magician and MacDevOps:YVR colleague, Wade Robson. I couldn’t have finished this plugin without his help. Merci, mon ami. (Early help to get started with this project is thanks to Scott Neal, automation expert and programming wizard. Thank you so much Scott, and thanks for the tasty Portland beer!).

Watchman Monitoring is a group of plugins that will warn when drives are failing, computers have restarted unexpectedly or backups are not running. All reporting goes to a beautiful web interface in the cloud which can keep a history of plugin issues. Watchman allows for integration with ticket systems and multiple users including clients and IT staff that can keep track of what’s up with their workstations, and servers.

Watchman Monitoring helps me keep tabs of major issues at all my clients before they become disasters. I even use it in discovery for new clients to see what issues exist but are ignored or unknown.

Since I set up a lot of SAN storage for my clients, and I use Archiware P5 for backups and archives I realized I needed to write a plugin for Watchman Monitoring that alerts me to issues. Instead of remoting in with VPN to each and every client every day to check on backups the only alternative is to automate it. These scripts watch the LTO tape drives and emails when they need cleaning, or warns when running jobs need tapes, if workstations haven’t backed up in a while or if tape pools need more tapes. And in Beta 2 we’ve added a check to see if the P5 maintenance support needs to be renewed to give you time to renew it before it expires. As well as better alerts for issues with running jobs, and lots of bug fixes.

We have it working on Mac servers running Archiware P5 and the next step is Linux, and the Unix family. Later on, Watchman will port it to Windows. The scripts are written in Python which is great for portability (except to Windows. Ha ha). And the P5 Watchman plugins should eventually run everywhere that Archiware P5 runs (OS X, Linux, FreeBSD, Solaris and Windows).

The best part of writing plugins for Watchman Monitoring is the great help that Allen and the whole team at Watchman have given us been throughout our development of these Archiware P5 plugins. And of course everyone at Archiware and Mike at PVT have been super helpful in explaining the use of the nsdchat cli for Archiware P5, even going so far as to add some features we needed to nsdchat when we explained how useful they’d be for this project. Mille mercis. Vielen danke.

Using GitHub to check code in, document business logic, write code, build a wiki and then track issues that need bug fixes or enhancement requests has been an adventure. It all starts with an problem that you want to be alerted for. It’s easy enough to add custom plugins to Watchman Monitoring you just need some ideas, a programmer (or two) and some time for testing, debugging, more testing and time. Did I mention you need lots of time? Ha ha

And now for a sneak peak of the Archiware P5 beta 2 plugins for Watchman Monitoring.
1. Watchman nicely lists the new warnings and expirations for quickly getting to the issues you need to see.
2. Expirations are tracked with Watchman. In this case we note the date when the maintenance for Archiware P5 needs to be renewed. Don’t want to miss that!
3. Server info is good to know. Uptime, port used, and what exactly is licensed.
4. The LTO tape drive is the heart of any tape library, and alerting when it needs cleaning is very important.
5. Other plugins watch the tape pools, running and completed jobs, as well as Backup2Go (workstation backup).
June 2, 2016
Best of 2015: Archiware P5 Archive app
Announced late in 2015 the Archiware P5 Archive app is a revolution for editors who want to control the archive and restore process. No longer the job of the IT Admin, editors can select files or folders on their SAN volume (or anywhere) and send them to the tape archive.

The Archive app is a brilliantly simple app that allows the right-click services action in OS X, or in another words a it’s a GUI app that presents a contextual menu that knows to how to the talk to your P5 Archive server. When the files are safely on tape the original files on the filesystems are replaced with stub files that can be used to start the restore process.

Requirements: Archiware P5 server with the Archive module setup with an Archive plan. Add to that the P5 Archive App which is installed on the clients.

Note: At the moment all archiving goes over the LAN by default, so if you have a fast SAN then you set up the P5 Archive app client settings as “localhost” instead of their actual client name. That means that when it goes to archive the file, the server knows that the files exist on the SAN at a known path (which is the same on the client and the server).

And now for some detailed steps and screenshots.
1. Archiving completed projects
Choose the completed project folder and right-click. Select “Archive to P5”.

Note 1: If you want to restore files choose the folder that was archived and right-click. Choose “Restore from P5”.

Note 2: Restoring individual files that have been archived is possible by double-clicking the files with the “.p5a” extension, but it will be much faster to select an entire folder to restore than many individual files.

Note 3: For either archive or restore to work the P5 Archive app needs to be installed.

Note 4: To avoid having a services sub menu keep the contextual-menu items to four.

2. Archiving app status

When you are archiving or restoring files the Archive app will show you the status of your request. It will also show you the status of other jobs running on the P5 server. This is to let you know why perhaps your archive or restore is taking a long time (it’s possibly waiting for access to the tape drive and it currently busy backing up or archiving something else).

The P5 Archive app offers you three operations “cancel job”, “list items” and “get report”. The last two are great when you want to examine a completed job, for example. If you want to find out what files were archived in the particular job choose “list items”.

3. Restoring files

Archived files will have either one of or both of, 1) a”.p5a” file extension and 2) a P5 Archive app icon.

Folders and FCP X project bundles (which are folders) do not get the “.p5a” extension, but FCP X projects have the the icon.

Note 1: Files can also be restored by the admin through the P5 web interface. They can be restored in place or to any other location that is required.

Note 2: On the P5 server jobs that are sent to archive or restored from tape show up as “cli job” with the tapes in use. Actual files or folders involved need to be noted from the P5 Archive app not the P5 web admin console. Otherwise checking the P5 web restore tab will files actually archived (that can be restored).

That’s enough for the quick overview of this great new app. One of the best things in 2015.

For more information on Archiware’s new P5 Archive app check out their website:

P5 Archive app
January 29, 2016
2015 in review

The WordPress.com stats helper monkeys prepared a 2015 annual report for this blog, but nobody wants to read no stinkin’ reports so let me just sum it all up: Xsan, Munki, Thunderbolt, Archives. Or is that all one word? Thunderbolt Xsan Munki Archives! That’s better.

Here’s an excerpt from the report that no one will read:

The concert hall at the Sydney Opera House holds 2,700 people. This blog was viewed about 20,000 times in 2015. If it were a concert at Sydney Opera House, it would take about 7 sold-out performances for that many people to see it.

Surprisingly, or not, that opera would be about Xsan. Yes, Apple’s Xsan is still alive, and Apple even added new features with OS X 10.11 El Capitan. I’m still building Xsan shared storage SANs and upgrading old ones to new versions. That was one of the good news stories of 2015 for me.

You can build an Xsan with one or two Mac Minis and add your storage of choice. That used to mean more often than not the fibre channel storage from Promise. A great choice for larger deployments, the x30 Vtraks are solid.

But the real shocker for me in 2015 was stumbling upon the Accusys Thunderbolt SAN RAID, the A16T2-Share. For more than half off the price of a similar fibre channel storage RAID here’s a magical box powered by unicorns that has four (4) Thunderbolt connections. Plug one Thunderbolt cable into that Mac Mini, format the raid, setup Apple’s Xsan, and then plug the other three (3) Thunderbolt cables into iMacs, Mac Pro, MacBook Pros or any Xsan clients. Wow. Awesome.

Suddenly we have a game changer. An affordable SAN storage RAID for real block-level storage. Now more than ever we can afford to have true collaborative workflows for video editors and anyone in the creative. If you need to work together with fast connections to a shared pool then building an Xsan got much more attractive.

Disclaimer: I got a chance to test the Accusys A16T2-Share. And I would be crazy to recommend something without testing it thoroughly. This was used for several weeks by video editors in production. It was much faster than our 4GB fibre channel storage, of course, but it was also faster than our 8GB FC storage. Speed tests showed we got close to 1GB/sec, and even when it was 97% full we got 700MB/sec. Sa-weet.

I look forward to seeing what Accusys bring to NAB in 2016. What new box will they show up with? I hope for more than 4 client ports and faster Thunderbolt 3. Only 82 more sleeps till we all find out.

Apple’s Xsan and Accusys Thunderbolt storage A16T2-Share were big stars of 2015, but what else stood out? The two other bright shiny lights were Archiware’s new P5 Archive app, and Vidispine’s VidiXplore cloud based MAM. More on those in posts to follow. Both of these products have transformed workflows for editors. Stay tuned!

January 28, 2016
Mac IT training

I’ve been asked about Mac IT sysadmin training ideas, and I thought perhaps sharing some initial ideas might help other aspiring Mac IT sysadmins.

It’s hard to recommend a training site or specific place for Mac IT training these days. It used to be that you could sign up for a Unix SysAdmin or bash scripting class and that would be most helpful. And those both are helpful but Mac IT specific technologies are discussed at Mac IT conferences such as MacTech in LA (and periodically in other cities), PSU Mac Admins (Penn State), MacIT (San Jose), MacSysAdmin in Sweden, MacDeploy in Calgary and MacDevOps:YVR in Vancouver. As well as Apple sponsored camps for ACNs (Apple Consultants). Some of these conferences post their video online for free (PSU, MacSysAdmin and MacDevOps) so these are great resources to learn from.

Online learning: there are resources such as mail lists which have lots of helpful Mac Admins: mac-enterprise (MACENTERPRISE@LISTS.PSU.EDU), munki-dev (munki-dev@googlegroups.com), munki-discussion (munki-discussion@googlegroups.com), autopkg (autopkg-discuss@googlegroups.com) and the Slack MacAdmins channel (http://macadmins.org) and perhaps lots of other places I’m forgetting.

The most important is local meetups, and right now we’re not hosting anything in Vancouver. But I do want to start some local meetups here to encourage fellow Mac Admins to help each other out learning MacDevOps skills and getting into the greater Mac IT sysadmin community. There are great meetups in many cities across North America and the world. Finding out where they are means looking at afp548.com for info, or check out Watchman Monitoring’s new Mac Meetup site: http://www.macmeetups.com/find/, or check on Jamf Nation (https://jamfnation.jamfsoftware.com/index.html). Lots of options.

Here in Vancouver we are also contemplating some more in depth workshops for the next MacDevOps:YVR conference and perhaps sooner than that. How to get started with Mac IT technologies and how to build a beautiful Mac Admins community by being a part of it, and sharing what you know with others.

Anything you all want to learn specifically? What do you want to know about? Let me know.

July 15, 2015
Umask fixes in Yosemite aka OS X 10.10.3 and shared storage
Finally!

Yes, Apple has restored the ability to set a user and system umask in OS X 10.10.3. This is a huge fix for users of shared storage. Xsan and all SANs where users want to be able to share files, projects and all things without using ACLs or any LDAP directory. This is great. I am jumping up and down. So happy. So many people wanted this. Anyone using shared storage have been demanding this since the upgrade to Yosemite. 10.10.3 is out today and we will be happy.

Reference: https://support.apple.com/en-us/HT201684

tl;dr
```
sudo launchctl config user umask nnn
```
and
```
sudo launchctl config system umask nnn
```
April 8, 2015
MacDevOps:YVR
Date: June 19, 2015

http://www.macdevops.ca/

A new kind of conference for Mac IT professionals looking to get into DevOps. You’ll hear some about new automation tools, and get a chance to try new things in the computer lab. Join us! Registration limited to 75.

The cost is $99. Food is included on the day of the conference including a light breakfast and lunch. Register here.

Call for Submissions!

MacDevOpsYVR is seeking presenters from across the Pacific Northwest and beyond to participate in this one-day conference for all things Mac!

If you have an idea for a specific talk, workshop or panel related to deploying Macs in enterprise, corporate or educational environments, we want to hear from you.

> SUBMIT A PROPOSAL <

Deadline for Submissions: March 31, 2015.

Share your experience and join your peers at this one day, all day conference in beautiful Vancouver, BC.

Topics of Interest:
- Puppet, Chef and other automation from Desktop to Cloud and back
- Software deployment with Munki and AutoPkg: the app ecosystem surrounding it
- Cool tools: demo of awesome Mac Admin projects from GitHub
- DevOps: How to adopt Automation and Best practices in IT operations
- Dev skills: workshops on Ruby, Git, Python, Javascript for Mac Admins
- MDM: Profiles and Mac configuration management in the cloud
February 23, 2015
Configuration Profiles and Identity payloads

Pretty sweet. It was a great gathering of IT pros in the deployment session. Great feedback and info sharing.

November 18, 2014
Using Munki and AutoPkg to automate Mac software deployment (Part 1)

Recently Munki v2.01 was released and now more than ever with the help of other apps it is easier to automate software deployment. With help with AutoPkg (and AutoPkgr) you can quickly set up a Munki server to deliver software to all your Macs. In the time it takes to download one new app and update each of your client workstations you could instead put it in your Munki repo and have it ready to deploy to everyone.

Munki allows you to automate software deployment. When you have more than one or two Macs to ensure that they are up to date with security, Flash, Java or other app updates you being to realize that an automated system can save you time and maybe even your sanity. You don’t backup manually, of course, you automate it. When it’s important and you want it done right, then some planning ahead of time and automation will make your life much easier.

If you have not yet set up a Munki server then follow along as I walk you through setting Munki 2.01 with AutoPkgr 1.1 in part 1 of this blog post of Munki and AutoPkg. In part 2 I will go into further detail of how to use MunkiAdmin (Mac app) and Mandrill (a node.js web server) to edit and maintain your Munki set up. Pros and cons of each method will be touched upon. Using the command line in the past was required but I will show you how some really good apps and web services can help you maintain your automated software deployment workflow.

Note: Munki requires only a web server to deploy software, while traditionally the munki tools ran on a Mac. You can put your software repo on any web server. I will show you the set up on a Mac for the purposes of this blog post.

Steps to a basic Munki server set up on a Mac running 10.8, 10.9, or 10.10:

1. Install latest Munki tools (v.2.01 at the time I write this), restart

muni tools 2.01 pkg

2. Install AutoPKGr (v.1.1 at the time I write this)

Install AutoPkg, and Git using AutoPkgr.

Install autopkg and git using autopkgr

3. Set your Munki repo to some folder (for example, /Users/Shared/munki_repo)

Munki repo

4. Set up web services on OS X by manually editing httpd.conf document root to your Munki repo or with Server.app, setting your munki_repo as where you store your site files.

Server.app Website document root munki repo

6. Add recipes to AutoPKGr and choose apps. Set a schedule for AutoPkgr.

Configure AutoPkgr

7. Check for apps manually the first time, and let AutoPKG download them to your repo

Configure AutoPkgr schedule

8. Check your repo for a manifests folder, and if it is not there, create it

Munki repo manifests

9. Download icon importer, move to /usr/local/munki folder, run against your repo

curl -O https://munki.googlecode.com/git-history/Munki2/code/client/iconimporter

mv iconimporter /usr/local/munki/iconimporter.py

sudo chmod +x /usr/local/munki/iconimporter.py

cd /usr/local/munki ; sudo ./iconimporter.py /Users/Shared/munki_repo/

iconimporter munki repo

Next, go to the icons folder in your repo, pick a fav icon and rename if necessary (some have more than one icon with name with “_1, _2, etc”).

10. Open MunkiAdmin and add packages to catalogs as needed, edit package info (add developer and category info, descriptions etc as needed), then create a client manifest.

11. Choose apps to install for clients (choose from installs, optional installs, uninstalls)

12. Set client id and repoURL on actual clients.

sudo defaults write /Library/Preferences/ManagedInstalls ClientIdentifier “test-client”

sudo defaults write /Library/Preferences/ManagedInstalls SoftwareRepoURL “http://ip.addr.ess”

Done. Your munki server is set up and ready for clients to connect. Next up, in part 2, we will look at Munki’s client facing app, the Managed Software Center. We will also look at how to use Munki Admin (Mac app) and Mandrill (a node.js web server) to edit and maintain your Munki set up. Pros and cons of each method will be touched upon. Using the command line with Munki was required in the past but the Munki ecosystem has grown and there are some really good apps and web services can help you maintain your automated software deployment workflow.

Further Reading:

1. What’s new in Munki 2 (Links to apps in the Munki ecosystem)

2. Munki 2 Demonstations setup (basic walkthrough setup)

November 3, 2014
MacTech 2014 LA

MacTech conf 2014

MacTech conference in LA, Nov 5-7, 2014. Grand unification theory of Mac geekdom. IT N3rdz united to share knowledge and make the Mac IT world a better place.

October 14, 2014
MacTech 2014 Seattle

Talked about “backups and storage” at the MacTech bootcamp 2014 in Seattle. Case studies of media storage and project archives to focus the discussion around backup technology best practise.

February 16, 2014