A blog about archives, storage and everything

Author: Mat X

Screen recording and other tricks
QuickTime has a neat little trick that some may not know about, it can record your screen.

Use it to record a how-to video how to navigate System Preferences, or how to use Final Cut Pro, or record a MacDevOps:YVR talk.

The first two MacDevOps:YVR conferences needed to be converted to a suitable format for YouTube and using QuickTime screen recording + Soundflower is the way I chose to do it.

Note: Soundflower is needed to redirect the audio to QuickTime. Screenrecording with QuickTime does not capture the audio without Soundflower.

MacDevOps Screen recording steps
1. Install Soundflower (Soundflower-2.0b2.dmg)
https://github.com/mattingalls/Soundflower/releases/tag/2.0b2
1. Set audio output to SoundFlower 2ch
2. Set QT screen recording to Soundflower audio
3. Start screen recording (select screen area)
4. Play website audio / video (Safari / Other )
5. Stop both. Edit and trim QT video as needed
6. Upload to YouTube
7. Tag video (mdoyvr, yvr, MacDevOps, MacAdmin, MacIT), put in proper playlist
8. Publish
Addendum:

I own Rogue Amoeba’s excellent Audio Hijack application and have used this app for audio capture (podcast interviews, etc), but I couldn’t get it to work in this case. It might have also required their Rogue Amoeba’s Loopback app which I did not own. Since I’ve used Soundflower previously I used it here in this case.
August 1, 2017
Archiware P5 and Synology NAS.

Update: As of version 5.4.3 there is an official P5 add-on package for Synology NAS

Archiware P5 available for Synology

Note: The P5 app for Synology NAS first debuted with P5 v.5.3.3

On the Archiware P5 new-features page there’s a blurb about the Synology NAS integration:

From Version 5.3.3, Archiware P5 supports Synology NAS devices without restrictions.

Synology NAS can serve as a data source or target for P5 Synchronize, P5 Backup and P5 Archive. The Archiware P5 application can now be installed on the Synology NAS itself.

Thanks to the snapshot capability of the DSM platform, powerful enterprise Synology NAS devices can also be used as repository for Backup2Go. This setup opens the possibility of introducing a professional data security solution at an affordable price point.

Let’s look in closer detail how to install Archiware P5 on a new Synology NAS.

For this post I have a new Synology 1515+ NAS, installed with five 6TB hard drives (It is very easy to install hard drives. No tools required). Note: I’ve purchased the NAS with my own money and was not paid to write this article.

At the time of this blog post the latest Synology DSM release is 6.1 and Archiware P5 is at version 5.4.2.

Step 1. Download Synology package from Archiware.com/download

Download Archiware P5 for Synology

Requirements are DSM 5.2+ and Intel x86 64-Bit CPU only. (i.e. Atom but not Marvell).

Step 2. Find and Log into your NAS

Find your new NAS with the Synology Assistant app or use this handy website link:

Find your NAS

I had no luck with the app (it found my existing NAS, but not the new one). Using the website I was able to quickly locate the new NAS that I need to log into and setup. Very nice feature.

Step 3. Install the new DSM

Install or update new software. You will be prompted to go through the initial setup to prepare your new NAS.

Step 4. Set up a new volume

Chose the Btrfs or ext4 filesystem. Btrfs supports snapshots, replication, and much more.

Step 5. Monitor the volume setup

Verifying the hard disks will take a moment. Take a break here.

Step 6. Open Package Center

Step 7. Install manually

Install Archiware P5 by selecting the “install manually” option to upload the awpst542.spk downloaded file from archiware.com

Step 8. Agree to continue.

Load the Synology P5 installer by agreeing to continue with this “unknown” publisher.

Sep 9. Agree to trust the installer

Step 10. Confirm the Install

Step 11. P5 is now running on the Synology NAS.

Hooray! P5 is now installed. Select the app to examine the details.

Step 12. Examine the option to stop or uninstall the P5 application

Step 13. Login to the P5 server running on NAS

To login to P5 open a new tab. Pay attention to the port number: “20,000” (vs 8000 on other platforms such as Solaris, Linux, OSX etc).

Step 14. Set up your NAS as a client on another Server

To test the new Synology 1515+ NAS I then set up the NAS as a client on another P5 server, and set up a P5 Sync job to copy data from server with a ZFS based filesystem to the Synology NAS with a btrfs volume.

Testing: Set up the new client in P5 with a name and IP address, then set up a new sync job with source and destination. Start now. Watch the bits fly through the ether. Be happy.

Step 15. Other things to configure

To make your new NAS is working smoothly don’t forget to set up the email notifications, and set up some AFP, SMB, or NFS shares as required.

Take some time to explore the Package Center app and see what other great applications are offered on the Synology NAS.

Synology makes a great low-cost NAS appliance. For SMB or production setups I would recommend two or more (for redundancy, hot or cold spares, disaster recovery, offsite backups/replication). With P5 installed you can Sync your server data to a NAS for onsite or offsite backups, backup your NAS to tape, or use the NAS for your client workstation backups using Backup2Go. Using the new Btrfs filesystem provides many of the same advances as ZFS, including snapshots and replication, over traditional filesystems such as ext4 and hfs which sadly lack these features.

Conclusion:

The Synology NAS is a great experience. Adding Archiware P5 is a recommended way to include this NAS as part of any good backup, archive or DR (disaster recovery) scenario. Two thumbs up. Way up.

References:

Archiware P5 new features

Synology DSM

March 4, 2017
MacDevOps:YVR 2017

We’ve had incredible feedback from the last two events and it was so much fun we’ve decided to do it again. Join us on June 5-6th in Vancouver, BC, Canada. Early bird tickets are on sale now.

As a conference we like to gather to discuss Open Source solutions to manage Macs in the enterprise and everywhere else. This year we focus on the new APFS filesystem and what that means for all of us. How do we manage macOS if it is becoming more closed and like iOS? They’ll be talks on what is MDM?, Is imaging dead?, managing Macs with various open source tools, and how to leverage the cloud.

Join us for the technical talks by speakers from Google, Facebook, Dropbox, Airbnb, Square, Uber and many more. Or hang out in the break room and the hallway track. You’ll meet the awesome community members that make up the MacDevOps family. We are all here to share what we know, and to learn from others.

For more information go to our website:

MacDevOps:YVR website

A limited number of early bird tickets are on sale now at Eventbrite:

Get your early bird ticket now!

February 3, 2017
Troubleshooting Autopkg and AutoPkgr (part 1 of 5,432)
I love Autopkg and Autopkgr. They feed Munki and they keep me fed.

Sometimes Autopkg gives an error that doesn’t make sense since you don’t have enough info. Like this one:

That’s no way to make friends. Nope.

If even I understood all that… which is saying a lot. It doesn’t tell us what to do, or where to go to fix it.

Git makes sense, but maybe not in the context of Autopkgr errors. It wants us to “Git add or rm” (remove) offending items, but what does it have to do with what we’re doing at this moment? Hmm. Ok, we know that autopkgr uses autopkg which uses git but that still leaves us in the dark about what’s going on.

Drop down in terminal and poke at autopkg. That always helps.
```
bash-3.2$ autopkg

Usage: autopkg <verb> <options>, where <verb> is one of the following:

    help             (Display this help)

    info             (Get info about configuration or a recipe)

    install          (Run one or more install recipes. Example: autopkg install Firefox -- equivalent to: autopkg run Firefox.install)

    list-processors  (List available core Processors)

    list-recipes     (List recipes available locally)

    make-override    (Make a recipe override)

    processor-info   (Get information about a specific processor)

    repo-add         (Add one or more recipe repo from a URL)

    repo-delete      (Delete a recipe repo)

    repo-list        (List installed recipe repos)

    repo-update      (Update one or more recipe repos)

    run              (Run one or more recipes)

    search           (Search for recipes on GitHub.)

    version          (Print the current version of autopkg)

autopkg <verb> --help for more help for that verb
```
Looking at all that we notice that “repo-update” is most likely the autopkg command that gets activated when Autopkgr gui “update repos now” button gets clicked.

Running autopkg with repo-update option gets us a better error message.
```
Attempting git pull for /Users/awesome/Library/AutoPkg/RecipeRepos/

com.github.autopkg.wardsparadox-recipes...

ERROR: Pull is not possible because you have unmerged files.

Please, fix them up in the work tree, and then use 'git add/rm <file>'

as appropriate to mark resolution and make a commit.
```
So, at least we know now what is causing that error that Autopkgr showed us. Quick fix:
```
autopkg repo-delete https://github.com/autopkg/wardsparadox-recipes.git
```
And then we go on and pretend like nothing happened and continue on with our day, amirate? Maybe we go to the Mac Admins Slack autopkg channel and ask our colleagues, or post on the autopkg mail-list. Or we write a blog post.

More information:

The Autopkgr read me has troubleshooting tips

In the archives:

I first wrote about troubleshooting Autopkgr 2 years ago
September 29, 2016
Hello macOS Sierra, bye bye El Cap
We welcome the beautiful and wonderful macOS Sierra (10.12) and say good bye to the old and weary El Capitan (OS X 10.11.6)…. Wait, not so fast. Slow it down. Just a tad bit.

While Watchman is alerting me to users downloading, then installing the newest Apple macOS (née OS X, Mac OS X), we must be ready. Ready to troubleshoot issues with apps that developers haven’t tested thoroughly for a new OS that appears to be the same, but changes everything under the hood.

How do we test? In a VM of course.

What do we need:
1. VMWare Fusion
2. Greg et. al. createOSXinstallPkg
3. Rich Trouton’s disable setup assistant payload free packages
4. Mager Valp’s Create User Pkg
5. Greg et. al. Munki (latest release)
6. add your own packages, such as a munki kicksart (set repo url, client identifier, etc)
7. UPDATE: we can’t forget Rich Trouton’s First Boot Generator App
What are we doing?

createOSXinstallPkg was created to turn Apple’s App Store Install macOS Sierra.app or previous Install OS X versions into nice Apple installer packages to upgrade in place using Munki (or other deployment tools). The new trick added recently is to create a new Fake Install.app with our packages to use install in VMWare Fusion instead of on a real Mac.

UPDATED STEPS! Note: I’d forgotten about First Boot Generator
1. Download your installer app of choice (Install macOS sierra)
2. Download createOSX installer
3. Prepare your custom packages, or gathers ones your want to add to the installer
4. Organize your installers into folders like this: 00, 01, 02, etc
5. Launch First Boot Generator App and transmogrify that folders of packages
6. Run createOSX installer with the fake app option if you want to test a VM, or without if you want to build a package
7. Run createOSX as many times as you want with different OS X installers, and the same first boot package. Test diff OS installers with your customer PKGs.
Note: use the “–make-fake-app” option to prep for VMWare Fusion, omit it for a pkg

Note2: Here’s some examples using createOSXinstallPkg and various OSX installers
```
createOSXinstallPkg sudo ./createOSXinstallPkg --source /Volumes/SSD/Install\ macOS\ Sierra.app --make-fake-app --pkg ~/bin/PKG_BUILD/FirstBoot_staging/First\ Boot\ Package\ Install.pkg --output /Volumes/Updates/Builds

createOSXinstallPkg sudo ./createOSXinstallPkg --source /Volumes/Updates/Builds/Install\ OS\ X\ El\ Capitan.app --pkg ~/bin/PKG_BUILD/FirstBoot_staging/First\ Boot\ Package\ Install.pkg --output /Volumes/Updates/Builds
```
Note: If you get a message that your custom pkg you want to add is not a Flat package then use productbuild to repackage it.

Example:

➜ productbuild –package SetMunkiRepo.pkg SetMunkiRepo_flat.pkg

Reference: See Greg’s post on Managing OS X for more info make VMWare images using this method. And also Rich Trouton’s Der Flounder blog post on First Book Generator App
September 21, 2016
Packaging and deploying software
I am about to send an email to a software vendor asking them to please consider shipping their apps in a deployable Apple PKG format and I wanted to ask if anyone has some boilerplate text, excellent blog entry or list of arguments I can use. I could have posted in the MacAdmins slack, tweeted or posted a lovely photo on Instagram, but instead I sent an email to the MacEnterprise mail list.

Hat tip to Rick Heil on the MacEnterprise for pointing me to this post on AFP548 by Gary Larizza in June 2010.

“This one is an oldie but a goodie. It hits all my pain points, such as not assuming GUI interaction and minimizing pre/post scripts.”

https://www.afp548.com/2010/06/03/the-commandments-of-packaging-in-os-x/

Gary outlines his thesis in six rules:
1. Do not assume that your package will be installed interactively via the GUI or on the currently booted volume.
2. Unnecessary actions are unnecessary.
3. Licensing should have the option to be managed by Systems Administrators.
4. Use pre/post-install scripts only when necessary
5. Be true to the Operating System
6. Naming Conventions are Necessary and Helpful
All software vendors should aspire to follow these rules.

We should always send feedback to software vendors explaining carefully why their Mac OS X installers are not optimal for deployment when they are custom apps (e.g. InstallAnywhere) and not in Apple package format (i.e. PKG). Also, if the installers (as well as the app) require the legacy Java 6 then this seems to be a security risk and it is our duty to provide feedback if we hope to improve the situation in the future.

Another great source of information is Der Flounder, Rich Trouton’s blog, is worth perusing because of Rich’s excellent documentation and many excellent posts, including this one about re-packaging app:

“Using AutoPkg to build installer packages from installer applications” from May 24, 2016.

Reference: Re-packing using Auto PKG

As Rich succinctly puts it: “One of the challenges Mac admins have to deal with are Mac application installers which don’t follow one of the following models: Drag-and-drop installation or Package installation”.

Greg’s managingosx blog has many articles on packaging and I thank you for taking the time to write all those posts. We benefit greatly from all the work of everyone in this community. Greg has spoken at many conferences and given great packaging workshops.

Reference: packaging blog posts on Managing OSX
My personal preference for software deployment is to use Munki to deploy apps and not have to deploy app manually. While Mac sysadmins may use difference software for deployment, I think we are all in agreement in not wanting to do this manually when it can be automated. I don’t have a large IT team, and simple solutions based with Munki are best for me. Hence my desire for vendors to use the Apple PKG format where possible.
August 17, 2016
Camera Archives
For editing clients with a proper SAN this is the setup I like to use a watch folder on the SAN that sends to tape the camera archives automatically on a timed interval.

This requires
1. proper SAN
2. a watch folder setup with Archiware P5 archive
3. camera archives, created in FCP X from the camera cards
Note: you can also use Adobe Prelude, Shotput or Resolve to create verified copies of camera cards. Use what you trust and works for you. The idea is not to copy by hand and avoid the perils of corrupt files.

Bonus: multiple drives to enable multi-streaming and parallelizing of your data. Why not makes cloned tapes copies and stream lots of data to all four (yes, four !) drives. See the illustration below.
August 3, 2016
Watchman Monitoring + Archiware P5
I’ve been a little busy lately. I’m working on some scripts for Watchman Monitoring that alert when Archiware P5 needs attention. It’s really much more exciting than it sounds. 🙂

Archiware P5 plugin (included with Watchman Client 6.6.0)

UPDATE: The Archiware P5 plugin is now included with the Watchman Monitoring client version 6.6.0

Use the link above to read up about Watchman Monitoring and the Archiware P5 plugin.

This plugin is now part of Watchman Monitoring thanks to Allen and his team! Of course, big thanks to a lot of help from Python magician and MacDevOps:YVR colleague, Wade Robson. I couldn’t have finished this plugin without his help. Merci, mon ami. (Early help to get started with this project is thanks to Scott Neal, automation expert and programming wizard. Thank you so much Scott, and thanks for the tasty Portland beer!).

Watchman Monitoring is a group of plugins that will warn when drives are failing, computers have restarted unexpectedly or backups are not running. All reporting goes to a beautiful web interface in the cloud which can keep a history of plugin issues. Watchman allows for integration with ticket systems and multiple users including clients and IT staff that can keep track of what’s up with their workstations, and servers.

Watchman Monitoring helps me keep tabs of major issues at all my clients before they become disasters. I even use it in discovery for new clients to see what issues exist but are ignored or unknown.

Since I set up a lot of SAN storage for my clients, and I use Archiware P5 for backups and archives I realized I needed to write a plugin for Watchman Monitoring that alerts me to issues. Instead of remoting in with VPN to each and every client every day to check on backups the only alternative is to automate it. These scripts watch the LTO tape drives and emails when they need cleaning, or warns when running jobs need tapes, if workstations haven’t backed up in a while or if tape pools need more tapes. And in Beta 2 we’ve added a check to see if the P5 maintenance support needs to be renewed to give you time to renew it before it expires. As well as better alerts for issues with running jobs, and lots of bug fixes.

We have it working on Mac servers running Archiware P5 and the next step is Linux, and the Unix family. Later on, Watchman will port it to Windows. The scripts are written in Python which is great for portability (except to Windows. Ha ha). And the P5 Watchman plugins should eventually run everywhere that Archiware P5 runs (OS X, Linux, FreeBSD, Solaris and Windows).

The best part of writing plugins for Watchman Monitoring is the great help that Allen and the whole team at Watchman have given us been throughout our development of these Archiware P5 plugins. And of course everyone at Archiware and Mike at PVT have been super helpful in explaining the use of the nsdchat cli for Archiware P5, even going so far as to add some features we needed to nsdchat when we explained how useful they’d be for this project. Mille mercis. Vielen danke.

Using GitHub to check code in, document business logic, write code, build a wiki and then track issues that need bug fixes or enhancement requests has been an adventure. It all starts with an problem that you want to be alerted for. It’s easy enough to add custom plugins to Watchman Monitoring you just need some ideas, a programmer (or two) and some time for testing, debugging, more testing and time. Did I mention you need lots of time? Ha ha

And now for a sneak peak of the Archiware P5 beta 2 plugins for Watchman Monitoring.
1. Watchman nicely lists the new warnings and expirations for quickly getting to the issues you need to see.
2. Expirations are tracked with Watchman. In this case we note the date when the maintenance for Archiware P5 needs to be renewed. Don’t want to miss that!
3. Server info is good to know. Uptime, port used, and what exactly is licensed.
4. The LTO tape drive is the heart of any tape library, and alerting when it needs cleaning is very important.
5. Other plugins watch the tape pools, running and completed jobs, as well as Backup2Go (workstation backup).
June 2, 2016
Best of 2015: VidiXplore

This is another great product of 2015 and when I found it, I thought VidiXplore proves that Media Asset Management could be done better and simpler. At the end of December 2015 they released version 1.0 with new some tricks, including some changes to make a migration from Final Cut Server a reality. Time to move some clients!!

To sum up VidiXplore, I’ll quote my tweet from Dec 21, 2015:

Finally the perfect solution for simple asset management! Keep proxies in the cloud, originals local. Search + share!

Working with video editors, animation and visual effects studios, I’ve come to realize that media asset management (MAM) systems can be complicated and painful. Changing workflow, oh no! Building a better pipeline is not easy, nor is it always welcome. Well, hello from the other side, we found the solution, or at least part of it.

With VidiXplore you have 3 steps:

Step 1. Manage your videos by keeping all the originals local. Use your own storage. Use your folder structure. Use your vids as you would normally. Don’t pay for cloud storage.

Step 2. Proxies (thumbnail vids) go into the cloud to be viewed by you and your team.

Step 3. Organize your videos and photos into collections, batch edit by adding tags to add metadata, search for particular assets and share them with colleagues and external clients.

That’s it. You’re already ahead of the game. We skipped right over step 4 which was “have a lot of meetings to debate proper metadata” and step 5 which was “convince everyone at the company to adopt a different workflow.”

With VidiXplore you switch to a monthly payment model, that is true. You don’t own the cloud platform, but what you gain is that you don’t pay a lot of money upfront to set up a large server (or many large server), nor do you need render farm for video transcoding nor for the databases you need to keep track of it all. Pay monthly. That’s the way for a lot of smaller companies. Lower up front starting cost. No extreme capital outlay in the beginning.

Honestly, VidiXplore is a refreshing and easy way for so many people to use asset management now, so why not try it? Harder to say that with a large system setup that costs a lot of money to set up, only to find that no one wants to use it. That’s not what anyone wants.

And now for something completely different…

Let’s take a quick look at VidiXplore. If you’ve installed the VidiXploreAgent-1.0 agent then you’ll have a nice “V” icon in your menu bar (Mac) or system tray (Windows). Use this to open the VidiXplore agent.

In the VidiXplore menu you can access the settings where can you set whether certain file types get a Cloud Copy uploaded by default or how many concurrent jobs can run at once.

When the agent is open you’ll see folders you’ve configured for media, and an option to go the website of your particular instance of VidiXplore.

For the first web login you’ll see an intro screen which allows you to upload additional media to VidiXplore (that is, in addition to any particular media folders you’ve configured in the agent) and the option to connect cloud storage such as S3, Azure or Dropbox. Lastly, there are also the installers for the agent.

Once logged into your website you see a basic layout with options to sort by files or collections, to specify all files or particular types, sort by location and bulk actions.

If we want we can sort the results and see only collections:

Or we can sort by files which have a cloud copy:

So many options to sort, search and find what we’re looking for. Of course, we want to also edit the metadata but not so much that we require weekly meetings to decide on the 500 important and required metadata fields. Just use tags. Of course, meetings are good, and so is process, but it is so quick to select bulk actions and add a tag to a group of objects. Done!

“Finally the perfect solution for simple asset management! Keep proxies in the cloud, originals local. Search + share!”

There’s so much more you can do with VidiXplore, and I’ll go into more detail in another blog post, but this was just a highlight for my best of 2015.

Check out their website for more information:

VidiXplore.com

January 29, 2016
Best of 2015: Archiware P5 Archive app
Announced late in 2015 the Archiware P5 Archive app is a revolution for editors who want to control the archive and restore process. No longer the job of the IT Admin, editors can select files or folders on their SAN volume (or anywhere) and send them to the tape archive.

The Archive app is a brilliantly simple app that allows the right-click services action in OS X, or in another words a it’s a GUI app that presents a contextual menu that knows to how to the talk to your P5 Archive server. When the files are safely on tape the original files on the filesystems are replaced with stub files that can be used to start the restore process.

Requirements: Archiware P5 server with the Archive module setup with an Archive plan. Add to that the P5 Archive App which is installed on the clients.

Note: At the moment all archiving goes over the LAN by default, so if you have a fast SAN then you set up the P5 Archive app client settings as “localhost” instead of their actual client name. That means that when it goes to archive the file, the server knows that the files exist on the SAN at a known path (which is the same on the client and the server).

And now for some detailed steps and screenshots.
1. Archiving completed projects
Choose the completed project folder and right-click. Select “Archive to P5”.

Note 1: If you want to restore files choose the folder that was archived and right-click. Choose “Restore from P5”.

Note 2: Restoring individual files that have been archived is possible by double-clicking the files with the “.p5a” extension, but it will be much faster to select an entire folder to restore than many individual files.

Note 3: For either archive or restore to work the P5 Archive app needs to be installed.

Note 4: To avoid having a services sub menu keep the contextual-menu items to four.

2. Archiving app status

When you are archiving or restoring files the Archive app will show you the status of your request. It will also show you the status of other jobs running on the P5 server. This is to let you know why perhaps your archive or restore is taking a long time (it’s possibly waiting for access to the tape drive and it currently busy backing up or archiving something else).

The P5 Archive app offers you three operations “cancel job”, “list items” and “get report”. The last two are great when you want to examine a completed job, for example. If you want to find out what files were archived in the particular job choose “list items”.

3. Restoring files

Archived files will have either one of or both of, 1) a”.p5a” file extension and 2) a P5 Archive app icon.

Folders and FCP X project bundles (which are folders) do not get the “.p5a” extension, but FCP X projects have the the icon.

Note 1: Files can also be restored by the admin through the P5 web interface. They can be restored in place or to any other location that is required.

Note 2: On the P5 server jobs that are sent to archive or restored from tape show up as “cli job” with the tapes in use. Actual files or folders involved need to be noted from the P5 Archive app not the P5 web admin console. Otherwise checking the P5 web restore tab will files actually archived (that can be restored).

That’s enough for the quick overview of this great new app. One of the best things in 2015.

For more information on Archiware’s new P5 Archive app check out their website:

P5 Archive app
January 29, 2016