A blog about archives, storage and everything

Tag: IT

Screen recording and other tricks
QuickTime has a neat little trick that some may not know about, it can record your screen.

Use it to record a how-to video how to navigate System Preferences, or how to use Final Cut Pro, or record a MacDevOps:YVR talk.

The first two MacDevOps:YVR conferences needed to be converted to a suitable format for YouTube and using QuickTime screen recording + Soundflower is the way I chose to do it.

Note: Soundflower is needed to redirect the audio to QuickTime. Screenrecording with QuickTime does not capture the audio without Soundflower.

MacDevOps Screen recording steps
1. Install Soundflower (Soundflower-2.0b2.dmg)
https://github.com/mattingalls/Soundflower/releases/tag/2.0b2
1. Set audio output to SoundFlower 2ch
2. Set QT screen recording to Soundflower audio
3. Start screen recording (select screen area)
4. Play website audio / video (Safari / Other )
5. Stop both. Edit and trim QT video as needed
6. Upload to YouTube
7. Tag video (mdoyvr, yvr, MacDevOps, MacAdmin, MacIT), put in proper playlist
8. Publish
Addendum:

I own Rogue Amoeba’s excellent Audio Hijack application and have used this app for audio capture (podcast interviews, etc), but I couldn’t get it to work in this case. It might have also required their Rogue Amoeba’s Loopback app which I did not own. Since I’ve used Soundflower previously I used it here in this case.
August 1, 2017
Archiware P5 and Synology NAS.

Update: As of version 5.4.3 there is an official P5 add-on package for Synology NAS

Archiware P5 available for Synology

Note: The P5 app for Synology NAS first debuted with P5 v.5.3.3

On the Archiware P5 new-features page there’s a blurb about the Synology NAS integration:

From Version 5.3.3, Archiware P5 supports Synology NAS devices without restrictions.

Synology NAS can serve as a data source or target for P5 Synchronize, P5 Backup and P5 Archive. The Archiware P5 application can now be installed on the Synology NAS itself.

Thanks to the snapshot capability of the DSM platform, powerful enterprise Synology NAS devices can also be used as repository for Backup2Go. This setup opens the possibility of introducing a professional data security solution at an affordable price point.

Let’s look in closer detail how to install Archiware P5 on a new Synology NAS.

For this post I have a new Synology 1515+ NAS, installed with five 6TB hard drives (It is very easy to install hard drives. No tools required). Note: I’ve purchased the NAS with my own money and was not paid to write this article.

At the time of this blog post the latest Synology DSM release is 6.1 and Archiware P5 is at version 5.4.2.

Step 1. Download Synology package from Archiware.com/download

Download Archiware P5 for Synology

Requirements are DSM 5.2+ and Intel x86 64-Bit CPU only. (i.e. Atom but not Marvell).

Step 2. Find and Log into your NAS

Find your new NAS with the Synology Assistant app or use this handy website link:

Find your NAS

I had no luck with the app (it found my existing NAS, but not the new one). Using the website I was able to quickly locate the new NAS that I need to log into and setup. Very nice feature.

Step 3. Install the new DSM

Install or update new software. You will be prompted to go through the initial setup to prepare your new NAS.

Step 4. Set up a new volume

Chose the Btrfs or ext4 filesystem. Btrfs supports snapshots, replication, and much more.

Step 5. Monitor the volume setup

Verifying the hard disks will take a moment. Take a break here.

Step 6. Open Package Center

Step 7. Install manually

Install Archiware P5 by selecting the “install manually” option to upload the awpst542.spk downloaded file from archiware.com

Step 8. Agree to continue.

Load the Synology P5 installer by agreeing to continue with this “unknown” publisher.

Sep 9. Agree to trust the installer

Step 10. Confirm the Install

Step 11. P5 is now running on the Synology NAS.

Hooray! P5 is now installed. Select the app to examine the details.

Step 12. Examine the option to stop or uninstall the P5 application

Step 13. Login to the P5 server running on NAS

To login to P5 open a new tab. Pay attention to the port number: “20,000” (vs 8000 on other platforms such as Solaris, Linux, OSX etc).

Step 14. Set up your NAS as a client on another Server

To test the new Synology 1515+ NAS I then set up the NAS as a client on another P5 server, and set up a P5 Sync job to copy data from server with a ZFS based filesystem to the Synology NAS with a btrfs volume.

Testing: Set up the new client in P5 with a name and IP address, then set up a new sync job with source and destination. Start now. Watch the bits fly through the ether. Be happy.

Step 15. Other things to configure

To make your new NAS is working smoothly don’t forget to set up the email notifications, and set up some AFP, SMB, or NFS shares as required.

Take some time to explore the Package Center app and see what other great applications are offered on the Synology NAS.

Synology makes a great low-cost NAS appliance. For SMB or production setups I would recommend two or more (for redundancy, hot or cold spares, disaster recovery, offsite backups/replication). With P5 installed you can Sync your server data to a NAS for onsite or offsite backups, backup your NAS to tape, or use the NAS for your client workstation backups using Backup2Go. Using the new Btrfs filesystem provides many of the same advances as ZFS, including snapshots and replication, over traditional filesystems such as ext4 and hfs which sadly lack these features.

Conclusion:

The Synology NAS is a great experience. Adding Archiware P5 is a recommended way to include this NAS as part of any good backup, archive or DR (disaster recovery) scenario. Two thumbs up. Way up.

References:

Archiware P5 new features

Synology DSM

March 4, 2017
MacDevOps:YVR 2017

We’ve had incredible feedback from the last two events and it was so much fun we’ve decided to do it again. Join us on June 5-6th in Vancouver, BC, Canada. Early bird tickets are on sale now.

As a conference we like to gather to discuss Open Source solutions to manage Macs in the enterprise and everywhere else. This year we focus on the new APFS filesystem and what that means for all of us. How do we manage macOS if it is becoming more closed and like iOS? They’ll be talks on what is MDM?, Is imaging dead?, managing Macs with various open source tools, and how to leverage the cloud.

Join us for the technical talks by speakers from Google, Facebook, Dropbox, Airbnb, Square, Uber and many more. Or hang out in the break room and the hallway track. You’ll meet the awesome community members that make up the MacDevOps family. We are all here to share what we know, and to learn from others.

For more information go to our website:

MacDevOps:YVR website

A limited number of early bird tickets are on sale now at Eventbrite:

Get your early bird ticket now!

February 3, 2017
Troubleshooting Autopkg and AutoPkgr (part 1 of 5,432)
I love Autopkg and Autopkgr. They feed Munki and they keep me fed.

Sometimes Autopkg gives an error that doesn’t make sense since you don’t have enough info. Like this one:

That’s no way to make friends. Nope.

If even I understood all that… which is saying a lot. It doesn’t tell us what to do, or where to go to fix it.

Git makes sense, but maybe not in the context of Autopkgr errors. It wants us to “Git add or rm” (remove) offending items, but what does it have to do with what we’re doing at this moment? Hmm. Ok, we know that autopkgr uses autopkg which uses git but that still leaves us in the dark about what’s going on.

Drop down in terminal and poke at autopkg. That always helps.
```
bash-3.2$ autopkg

Usage: autopkg <verb> <options>, where <verb> is one of the following:

    help             (Display this help)

    info             (Get info about configuration or a recipe)

    install          (Run one or more install recipes. Example: autopkg install Firefox -- equivalent to: autopkg run Firefox.install)

    list-processors  (List available core Processors)

    list-recipes     (List recipes available locally)

    make-override    (Make a recipe override)

    processor-info   (Get information about a specific processor)

    repo-add         (Add one or more recipe repo from a URL)

    repo-delete      (Delete a recipe repo)

    repo-list        (List installed recipe repos)

    repo-update      (Update one or more recipe repos)

    run              (Run one or more recipes)

    search           (Search for recipes on GitHub.)

    version          (Print the current version of autopkg)

autopkg <verb> --help for more help for that verb
```
Looking at all that we notice that “repo-update” is most likely the autopkg command that gets activated when Autopkgr gui “update repos now” button gets clicked.

Running autopkg with repo-update option gets us a better error message.
```
Attempting git pull for /Users/awesome/Library/AutoPkg/RecipeRepos/

com.github.autopkg.wardsparadox-recipes...

ERROR: Pull is not possible because you have unmerged files.

Please, fix them up in the work tree, and then use 'git add/rm <file>'

as appropriate to mark resolution and make a commit.
```
So, at least we know now what is causing that error that Autopkgr showed us. Quick fix:
```
autopkg repo-delete https://github.com/autopkg/wardsparadox-recipes.git
```
And then we go on and pretend like nothing happened and continue on with our day, amirate? Maybe we go to the Mac Admins Slack autopkg channel and ask our colleagues, or post on the autopkg mail-list. Or we write a blog post.

More information:

The Autopkgr read me has troubleshooting tips

In the archives:

I first wrote about troubleshooting Autopkgr 2 years ago
September 29, 2016
Hello macOS Sierra, bye bye El Cap
We welcome the beautiful and wonderful macOS Sierra (10.12) and say good bye to the old and weary El Capitan (OS X 10.11.6)…. Wait, not so fast. Slow it down. Just a tad bit.

While Watchman is alerting me to users downloading, then installing the newest Apple macOS (née OS X, Mac OS X), we must be ready. Ready to troubleshoot issues with apps that developers haven’t tested thoroughly for a new OS that appears to be the same, but changes everything under the hood.

How do we test? In a VM of course.

What do we need:
1. VMWare Fusion
2. Greg et. al. createOSXinstallPkg
3. Rich Trouton’s disable setup assistant payload free packages
4. Mager Valp’s Create User Pkg
5. Greg et. al. Munki (latest release)
6. add your own packages, such as a munki kicksart (set repo url, client identifier, etc)
7. UPDATE: we can’t forget Rich Trouton’s First Boot Generator App
What are we doing?

createOSXinstallPkg was created to turn Apple’s App Store Install macOS Sierra.app or previous Install OS X versions into nice Apple installer packages to upgrade in place using Munki (or other deployment tools). The new trick added recently is to create a new Fake Install.app with our packages to use install in VMWare Fusion instead of on a real Mac.

UPDATED STEPS! Note: I’d forgotten about First Boot Generator
1. Download your installer app of choice (Install macOS sierra)
2. Download createOSX installer
3. Prepare your custom packages, or gathers ones your want to add to the installer
4. Organize your installers into folders like this: 00, 01, 02, etc
5. Launch First Boot Generator App and transmogrify that folders of packages
6. Run createOSX installer with the fake app option if you want to test a VM, or without if you want to build a package
7. Run createOSX as many times as you want with different OS X installers, and the same first boot package. Test diff OS installers with your customer PKGs.
Note: use the “–make-fake-app” option to prep for VMWare Fusion, omit it for a pkg

Note2: Here’s some examples using createOSXinstallPkg and various OSX installers
```
createOSXinstallPkg sudo ./createOSXinstallPkg --source /Volumes/SSD/Install\ macOS\ Sierra.app --make-fake-app --pkg ~/bin/PKG_BUILD/FirstBoot_staging/First\ Boot\ Package\ Install.pkg --output /Volumes/Updates/Builds

createOSXinstallPkg sudo ./createOSXinstallPkg --source /Volumes/Updates/Builds/Install\ OS\ X\ El\ Capitan.app --pkg ~/bin/PKG_BUILD/FirstBoot_staging/First\ Boot\ Package\ Install.pkg --output /Volumes/Updates/Builds
```
Note: If you get a message that your custom pkg you want to add is not a Flat package then use productbuild to repackage it.

Example:

➜ productbuild –package SetMunkiRepo.pkg SetMunkiRepo_flat.pkg

Reference: See Greg’s post on Managing OS X for more info make VMWare images using this method. And also Rich Trouton’s Der Flounder blog post on First Book Generator App
September 21, 2016
Packaging and deploying software
I am about to send an email to a software vendor asking them to please consider shipping their apps in a deployable Apple PKG format and I wanted to ask if anyone has some boilerplate text, excellent blog entry or list of arguments I can use. I could have posted in the MacAdmins slack, tweeted or posted a lovely photo on Instagram, but instead I sent an email to the MacEnterprise mail list.

Hat tip to Rick Heil on the MacEnterprise for pointing me to this post on AFP548 by Gary Larizza in June 2010.

“This one is an oldie but a goodie. It hits all my pain points, such as not assuming GUI interaction and minimizing pre/post scripts.”

https://www.afp548.com/2010/06/03/the-commandments-of-packaging-in-os-x/

Gary outlines his thesis in six rules:
1. Do not assume that your package will be installed interactively via the GUI or on the currently booted volume.
2. Unnecessary actions are unnecessary.
3. Licensing should have the option to be managed by Systems Administrators.
4. Use pre/post-install scripts only when necessary
5. Be true to the Operating System
6. Naming Conventions are Necessary and Helpful
All software vendors should aspire to follow these rules.

We should always send feedback to software vendors explaining carefully why their Mac OS X installers are not optimal for deployment when they are custom apps (e.g. InstallAnywhere) and not in Apple package format (i.e. PKG). Also, if the installers (as well as the app) require the legacy Java 6 then this seems to be a security risk and it is our duty to provide feedback if we hope to improve the situation in the future.

Another great source of information is Der Flounder, Rich Trouton’s blog, is worth perusing because of Rich’s excellent documentation and many excellent posts, including this one about re-packaging app:

“Using AutoPkg to build installer packages from installer applications” from May 24, 2016.

Reference: Re-packing using Auto PKG

As Rich succinctly puts it: “One of the challenges Mac admins have to deal with are Mac application installers which don’t follow one of the following models: Drag-and-drop installation or Package installation”.

Greg’s managingosx blog has many articles on packaging and I thank you for taking the time to write all those posts. We benefit greatly from all the work of everyone in this community. Greg has spoken at many conferences and given great packaging workshops.

Reference: packaging blog posts on Managing OSX
My personal preference for software deployment is to use Munki to deploy apps and not have to deploy app manually. While Mac sysadmins may use difference software for deployment, I think we are all in agreement in not wanting to do this manually when it can be automated. I don’t have a large IT team, and simple solutions based with Munki are best for me. Hence my desire for vendors to use the Apple PKG format where possible.
August 17, 2016
Camera Archives
For editing clients with a proper SAN this is the setup I like to use a watch folder on the SAN that sends to tape the camera archives automatically on a timed interval.

This requires
1. proper SAN
2. a watch folder setup with Archiware P5 archive
3. camera archives, created in FCP X from the camera cards
Note: you can also use Adobe Prelude, Shotput or Resolve to create verified copies of camera cards. Use what you trust and works for you. The idea is not to copy by hand and avoid the perils of corrupt files.

Bonus: multiple drives to enable multi-streaming and parallelizing of your data. Why not makes cloned tapes copies and stream lots of data to all four (yes, four !) drives. See the illustration below.
August 3, 2016
Watchman Monitoring + Archiware P5
I’ve been a little busy lately. I’m working on some scripts for Watchman Monitoring that alert when Archiware P5 needs attention. It’s really much more exciting than it sounds. 🙂

Archiware P5 plugin (included with Watchman Client 6.6.0)

UPDATE: The Archiware P5 plugin is now included with the Watchman Monitoring client version 6.6.0

Use the link above to read up about Watchman Monitoring and the Archiware P5 plugin.

This plugin is now part of Watchman Monitoring thanks to Allen and his team! Of course, big thanks to a lot of help from Python magician and MacDevOps:YVR colleague, Wade Robson. I couldn’t have finished this plugin without his help. Merci, mon ami. (Early help to get started with this project is thanks to Scott Neal, automation expert and programming wizard. Thank you so much Scott, and thanks for the tasty Portland beer!).

Watchman Monitoring is a group of plugins that will warn when drives are failing, computers have restarted unexpectedly or backups are not running. All reporting goes to a beautiful web interface in the cloud which can keep a history of plugin issues. Watchman allows for integration with ticket systems and multiple users including clients and IT staff that can keep track of what’s up with their workstations, and servers.

Watchman Monitoring helps me keep tabs of major issues at all my clients before they become disasters. I even use it in discovery for new clients to see what issues exist but are ignored or unknown.

Since I set up a lot of SAN storage for my clients, and I use Archiware P5 for backups and archives I realized I needed to write a plugin for Watchman Monitoring that alerts me to issues. Instead of remoting in with VPN to each and every client every day to check on backups the only alternative is to automate it. These scripts watch the LTO tape drives and emails when they need cleaning, or warns when running jobs need tapes, if workstations haven’t backed up in a while or if tape pools need more tapes. And in Beta 2 we’ve added a check to see if the P5 maintenance support needs to be renewed to give you time to renew it before it expires. As well as better alerts for issues with running jobs, and lots of bug fixes.

We have it working on Mac servers running Archiware P5 and the next step is Linux, and the Unix family. Later on, Watchman will port it to Windows. The scripts are written in Python which is great for portability (except to Windows. Ha ha). And the P5 Watchman plugins should eventually run everywhere that Archiware P5 runs (OS X, Linux, FreeBSD, Solaris and Windows).

The best part of writing plugins for Watchman Monitoring is the great help that Allen and the whole team at Watchman have given us been throughout our development of these Archiware P5 plugins. And of course everyone at Archiware and Mike at PVT have been super helpful in explaining the use of the nsdchat cli for Archiware P5, even going so far as to add some features we needed to nsdchat when we explained how useful they’d be for this project. Mille mercis. Vielen danke.

Using GitHub to check code in, document business logic, write code, build a wiki and then track issues that need bug fixes or enhancement requests has been an adventure. It all starts with an problem that you want to be alerted for. It’s easy enough to add custom plugins to Watchman Monitoring you just need some ideas, a programmer (or two) and some time for testing, debugging, more testing and time. Did I mention you need lots of time? Ha ha

And now for a sneak peak of the Archiware P5 beta 2 plugins for Watchman Monitoring.
1. Watchman nicely lists the new warnings and expirations for quickly getting to the issues you need to see.
2. Expirations are tracked with Watchman. In this case we note the date when the maintenance for Archiware P5 needs to be renewed. Don’t want to miss that!
3. Server info is good to know. Uptime, port used, and what exactly is licensed.
4. The LTO tape drive is the heart of any tape library, and alerting when it needs cleaning is very important.
5. Other plugins watch the tape pools, running and completed jobs, as well as Backup2Go (workstation backup).
June 2, 2016
2015 in review

The WordPress.com stats helper monkeys prepared a 2015 annual report for this blog, but nobody wants to read no stinkin’ reports so let me just sum it all up: Xsan, Munki, Thunderbolt, Archives. Or is that all one word? Thunderbolt Xsan Munki Archives! That’s better.

Here’s an excerpt from the report that no one will read:

The concert hall at the Sydney Opera House holds 2,700 people. This blog was viewed about 20,000 times in 2015. If it were a concert at Sydney Opera House, it would take about 7 sold-out performances for that many people to see it.

Surprisingly, or not, that opera would be about Xsan. Yes, Apple’s Xsan is still alive, and Apple even added new features with OS X 10.11 El Capitan. I’m still building Xsan shared storage SANs and upgrading old ones to new versions. That was one of the good news stories of 2015 for me.

You can build an Xsan with one or two Mac Minis and add your storage of choice. That used to mean more often than not the fibre channel storage from Promise. A great choice for larger deployments, the x30 Vtraks are solid.

But the real shocker for me in 2015 was stumbling upon the Accusys Thunderbolt SAN RAID, the A16T2-Share. For more than half off the price of a similar fibre channel storage RAID here’s a magical box powered by unicorns that has four (4) Thunderbolt connections. Plug one Thunderbolt cable into that Mac Mini, format the raid, setup Apple’s Xsan, and then plug the other three (3) Thunderbolt cables into iMacs, Mac Pro, MacBook Pros or any Xsan clients. Wow. Awesome.

Suddenly we have a game changer. An affordable SAN storage RAID for real block-level storage. Now more than ever we can afford to have true collaborative workflows for video editors and anyone in the creative. If you need to work together with fast connections to a shared pool then building an Xsan got much more attractive.

Disclaimer: I got a chance to test the Accusys A16T2-Share. And I would be crazy to recommend something without testing it thoroughly. This was used for several weeks by video editors in production. It was much faster than our 4GB fibre channel storage, of course, but it was also faster than our 8GB FC storage. Speed tests showed we got close to 1GB/sec, and even when it was 97% full we got 700MB/sec. Sa-weet.

I look forward to seeing what Accusys bring to NAB in 2016. What new box will they show up with? I hope for more than 4 client ports and faster Thunderbolt 3. Only 82 more sleeps till we all find out.

Apple’s Xsan and Accusys Thunderbolt storage A16T2-Share were big stars of 2015, but what else stood out? The two other bright shiny lights were Archiware’s new P5 Archive app, and Vidispine’s VidiXplore cloud based MAM. More on those in posts to follow. Both of these products have transformed workflows for editors. Stay tuned!

January 28, 2016
Thoughts on Documentation: What are we afraid of?

People are afraid of documentation… But mostly people just hate it. They don’t like it. They don’t want it. It shouldn’t exist. Fingers in ears. I can’t hear you.

This is about primal fear. And hate. I hate hate. But these are real emotions. Let’s deal with it. What is the reality? Why is documentation is ignored, abandoned, or resisted at all?

As a Sysadmin perhaps you don’t care about documentation, that is, sharing information with others (co workers / bosses), you want to keep it to yourself. But you care very much about building systems. But there’s perhaps no attempt to explain any of this to anyone else. Who else is there really? No one cares. No one is around that would understand if you explained it.

Lesson # 1 – Document for yourself.

Paranoia makes us set up redundant systems for backups. Layers upon layers. Custom scripts and disparate apps. Where was this explained? Documented? Nowhere. Bin dir. Maybe.

If you could replace all that now with one app that did it all then you would. Time is valuable. Easier to monitor. Easier for someone else to monitor and take over.

Lesson # 2 – document for your replacement (job change, bus hit)

Do it continuously. Automate. Or set up systems that work automatically.

Lesson # 3. DevOps.

Integrate systems. IT systems manage computer but maybe they also built Inventory. Automatically. Alert Systems report continuously. Living systems report on the state of everything. Documentation is easier when it is current and relevant.

Lesson # 4. Sustainability

Commercial vs OpenSource. Support vs excellent team, talent retention and documentation. Pro/Con. If your custom solution is not well documented that can be a big problem. If you code is not shared, peer-reviewed, or supported by anyone that could be an issue. If it makes sense to switch to commercial software that is supported then do it. If an OpenSource project or code is supported by a larger community perhaps that makes sense.

Lesson # 5. Improve. Grow. Get better.

Discovery and Documentation lead to suggestions for improvement. Make changes. Code and disparate systems that struggle to be documented make us think about how to replace them or better balance the risks vs cost.

Lesson # 6. Human problems don’t always tech solutions.

Code doesn’t fix broken workflows. Meetings are with people. Talking through systems helps people understand pain points. Don’t forget people want to do their job, meet deadlines, do stuff.

Let’s make their world and our world better.

Love not hate. Peace.

Documentation-MatX

November 8, 2015