A blog about archives, storage and everything

Author: Mat X

  • MacDevOps:YVR 2017

    wr

    We’ve had incredible feedback from the last two events and it was so much fun we’ve decided to do it again. Join us on June 5-6th in Vancouver, BC, Canada. Early bird tickets are on sale now.

    As a conference we like to gather to discuss Open Source solutions to manage Macs in the enterprise and everywhere else. This year we focus on the new APFS filesystem and what that means for all of us. How do we manage macOS if it is becoming more closed and like iOS? They’ll be talks on what is MDM?, Is imaging dead?, managing Macs with various open source tools, and how to leverage the cloud.

    Join us for the technical talks by speakers from Google, Facebook, Dropbox, Airbnb, Square, Uber and many more. Or hang out in the break room and the hallway track. You’ll meet the awesome community members that make up the MacDevOps family. We are all here to share what we know, and to learn from others.

    For more information go to our website:

    MacDevOps:YVR website

    A limited number of early bird tickets are on sale now at Eventbrite:

    Get your early bird ticket now!

  • Troubleshooting Autopkg and AutoPkgr (part 1 of 5,432)

    I love Autopkg and Autopkgr. They feed Munki and they keep me fed.

    Sometimes Autopkg gives an error that doesn’t make sense since you don’t have enough info. Like this one:

    autopkgr-work-tree

    That’s no way to make friends. Nope.

    If even I understood all that… which is saying a lot. It doesn’t tell us what to do, or where to go to fix it.

    Git makes sense, but maybe not in the context of Autopkgr errors. It wants us to “Git add or rm” (remove) offending items, but what does it have to do with what we’re doing at this moment? Hmm. Ok, we know  that autopkgr uses autopkg which uses git but that still leaves us in the dark about what’s going on.

    Drop down in terminal and poke at autopkg. That always helps.

    bash-3.2$ autopkg

Usage: autopkg <verb> <options>, where <verb> is one of the following:

    help             (Display this help)

    info             (Get info about configuration or a recipe)

    install          (Run one or more install recipes. Example: autopkg install Firefox -- equivalent to: autopkg run Firefox.install)

    list-processors  (List available core Processors)

    list-recipes     (List recipes available locally)

    make-override    (Make a recipe override)

    processor-info   (Get information about a specific processor)

    repo-add         (Add one or more recipe repo from a URL)

    repo-delete      (Delete a recipe repo)

    repo-list        (List installed recipe repos)

    repo-update      (Update one or more recipe repos)

    run              (Run one or more recipes)

    search           (Search for recipes on GitHub.)

    version          (Print the current version of autopkg)

autopkg <verb> --help for more help for that verb

    Looking at all that we notice that “repo-update” is most likely the autopkg command that gets activated when Autopkgr gui “update repos now” button gets clicked.

    screen-shot-2016-09-29-at-10-26-20-am

    Running autopkg with repo-update option gets us a better error message.

    Attempting git pull for /Users/awesome/Library/AutoPkg/RecipeRepos/

com.github.autopkg.wardsparadox-recipes...

ERROR: Pull is not possible because you have unmerged files.

Please, fix them up in the work tree, and then use 'git add/rm <file>'

as appropriate to mark resolution and make a commit.

    So, at least we know now what is causing that error that Autopkgr showed us. Quick fix:

    autopkg repo-delete https://github.com/autopkg/wardsparadox-recipes.git

    And then we go on and pretend like nothing happened and continue on with our day, amirate? Maybe we go to the Mac Admins Slack autopkg channel and ask our colleagues, or  post on the autopkg mail-list. Or we write a blog post.

    More information:

    The Autopkgr read me has troubleshooting tips

    In the archives:

    I first wrote about troubleshooting Autopkgr 2 years ago

     

  • Hello macOS Sierra, bye bye El Cap

    We welcome the beautiful and wonderful macOS Sierra (10.12) and say good bye to the old and weary El Capitan (OS X 10.11.6)…. Wait, not so fast. Slow it down. Just a tad bit.

    While Watchman is alerting me to users downloading, then installing the newest Apple macOS (née OS X, Mac OS X), we must be ready. Ready to troubleshoot issues with apps that developers haven’t tested thoroughly for a new OS that appears to be the same, but changes everything under the hood.

    How do we test? In a VM of course.

    What do we need:

    1. VMWare Fusion
    2. Greg et. al. createOSXinstallPkg
    3. Rich Trouton’s disable setup assistant payload free packages
    4. Mager Valp’s Create User Pkg
    5. Greg et. al. Munki (latest release)
    6. add your own packages, such as a munki kicksart (set repo url, client identifier, etc)
    7. UPDATE: we can’t forget Rich Trouton’s First Boot Generator App

    What are we doing?

    createOSXinstallPkg was created to turn Apple’s App Store Install macOS Sierra.app or previous Install OS X versions into nice Apple installer packages to upgrade in place using Munki (or other deployment tools). The new trick added recently is to create a new Fake Install.app with our packages to use install in VMWare Fusion instead of on a real Mac.

    UPDATED STEPS! Note: I’d forgotten about First Boot Generator

    1. Download your installer app of choice (Install macOS sierra)
    2. Download createOSX installer
    3. Prepare your custom packages, or gathers ones your want to add to the installer
    4. Organize your installers into folders like this: 00, 01, 02, etc
    5. Launch First Boot Generator App and transmogrify that folders of packages
    6. Run createOSX installer with the fake app option if you want to test a VM, or without if you want to build a package
    7. Run createOSX as many times as you want with different OS X installers, and the same first boot package. Test diff OS installers with your customer PKGs.

    Note: use the “–make-fake-app” option to prep for VMWare Fusion, omit it for a pkg

    Note2: Here’s some examples using createOSXinstallPkg and various OSX installers

    createOSXinstallPkg sudo ./createOSXinstallPkg --source /Volumes/SSD/Install\ macOS\ Sierra.app --make-fake-app --pkg ~/bin/PKG_BUILD/FirstBoot_staging/First\ Boot\ Package\ Install.pkg --output /Volumes/Updates/Builds

createOSXinstallPkg sudo ./createOSXinstallPkg --source /Volumes/Updates/Builds/Install\ OS\ X\ El\ Capitan.app --pkg ~/bin/PKG_BUILD/FirstBoot_staging/First\ Boot\ Package\ Install.pkg --output /Volumes/Updates/Builds

     

    firstbootgeneratorapp

    firstbootpackages

    Note: If you get a message that your custom pkg you want to add is not a Flat package then use productbuild to repackage it.

    Example:

    ➜  productbuild –package SetMunkiRepo.pkg SetMunkiRepo_flat.pkg

     

    Reference: See Greg’s post on Managing OS X for more info make VMWare images using this method. And also Rich Trouton’s Der Flounder blog post on First Book Generator App

     

     

  • Packaging and deploying software

    I am about to send an email to a software vendor asking them to please consider shipping their apps in a deployable Apple PKG format and I wanted to ask if anyone has some boilerplate text, excellent blog entry or list of arguments I can use. I could have posted in the MacAdmins slack, tweeted or posted a lovely photo on Instagram, but instead I sent an email to the MacEnterprise mail list.

    Hat tip to Rick Heil on the MacEnterprise for pointing me to this post on AFP548 by Gary Larizza in June 2010.

    “This one is an oldie but a goodie. It hits all my pain points, such as not assuming GUI interaction and minimizing pre/post scripts.”

    https://www.afp548.com/2010/06/03/the-commandments-of-packaging-in-os-x/

    Gary outlines his thesis in six rules:

    1. Do not assume that your package will be installed interactively via the GUI or on the currently booted volume.
    2. Unnecessary actions are unnecessary.
    3. Licensing should have the option to be managed by Systems Administrators.
    4. Use pre/post-install scripts only when necessary
    5. Be true to the Operating System
    6. Naming Conventions are Necessary and Helpful

    All software vendors should aspire to follow these rules.

    We should always send feedback to software vendors explaining carefully why their Mac OS X installers are not optimal for deployment when they are custom apps (e.g. InstallAnywhere) and not in Apple package format (i.e. PKG). Also, if the installers (as well as the app) require the legacy Java 6 then this seems to be a security risk and it is our duty to provide feedback if we hope to improve the situation in the future.

    Another great source of information is Der Flounder, Rich Trouton’s blog, is worth perusing because of Rich’s excellent documentation and many excellent posts, including this one about re-packaging app:

    “Using AutoPkg to build installer packages from installer applications” from May 24, 2016.

    Reference: Re-packing using Auto PKG

    As Rich succinctly puts it: “One of the challenges Mac admins have to deal with are Mac application installers which don’t follow one of the following models: Drag-and-drop installation or Package installation”.

    Greg’s managingosx blog has many articles on packaging and I thank you for taking the time to write all those posts. We benefit greatly from all the work of everyone in this community. Greg has spoken at many conferences and given great packaging workshops.

    Reference: packaging blog posts on Managing OSX
    My personal preference for software deployment is to use Munki to deploy apps and not have to deploy app manually. While Mac sysadmins may use difference software for deployment, I think we are all in agreement in not wanting to do this manually when it can be automated. I don’t have a large IT team, and simple solutions based with Munki are best for me. Hence my desire for vendors to use the Apple PKG format where possible.

     

     

  • Camera Archives

    For editing clients with a proper SAN this is the setup I like to use a watch folder on the SAN that sends to tape the camera archives automatically on a timed interval.

    This requires

    1. proper SAN
    2. a watch folder setup with Archiware P5 archive
    3. camera archives, created in FCP X from the camera cards

    Note: you can also use Adobe Prelude, Shotput or Resolve to create verified copies of camera cards. Use what you trust and works for you. The idea is not to copy by hand and avoid the perils of corrupt files.

    Bonus: multiple drives to enable multi-streaming and parallelizing of your data. Why not makes cloned tapes copies and stream lots of data to all four (yes, four !) drives. See the illustration below.

     

     

    Screen Shot 2016-08-03 at 10.35.40 AM

  • Watchman Monitoring + Archiware P5

    I’ve been a little busy lately. I’m working on some scripts for Watchman Monitoring that alert when Archiware P5 needs attention. It’s really much more exciting than it sounds. 🙂

    WatchmanArchiwareP5

    Archiware P5 plugin (included with Watchman Client 6.6.0)

    UPDATE: The Archiware P5 plugin is now included with the Watchman Monitoring client version 6.6.0

    Use the link above to read up about Watchman Monitoring and the Archiware P5 plugin.

    This plugin is now part of Watchman Monitoring thanks to Allen and his team! Of course, big thanks to a lot of help from Python magician and MacDevOps:YVR colleague, Wade Robson. I couldn’t have finished this plugin without his help. Merci, mon ami. (Early help to get started with this project is thanks to Scott Neal, automation expert and programming wizard. Thank you so much Scott, and thanks for the tasty Portland beer!).

    Watchman Monitoring is a group of plugins that will warn when drives are failing, computers have restarted unexpectedly or backups are not running. All reporting goes to a beautiful web interface in the cloud which can keep a history of plugin issues. Watchman allows for integration with ticket systems and multiple users including clients and IT staff that can keep track of what’s up with their workstations, and servers.

    Watchman Monitoring helps me keep tabs of major issues at all my clients before they become disasters. I even use it in discovery for new clients to see what issues exist but are ignored or unknown.

    Since I set up a lot of SAN storage for my clients, and I use Archiware P5 for backups and archives I realized I needed to write a plugin for Watchman Monitoring that alerts me to issues. Instead of remoting in with VPN to each and every client every day to check on backups the only alternative is to automate it. These scripts watch the LTO tape drives and emails when they need cleaning, or warns when running jobs need tapes, if workstations haven’t backed up in a while or if tape pools need more tapes. And in Beta 2 we’ve added a check to see if the P5 maintenance support needs to be renewed to give you time to renew it before it expires. As well as better alerts for issues with running jobs, and lots of bug fixes.

    We have it working on Mac servers running Archiware P5 and the next step is Linux, and the Unix family. Later on, Watchman will port it to Windows. The scripts are written in Python which is great for portability (except to Windows. Ha ha). And the P5 Watchman plugins should eventually run everywhere that Archiware P5 runs (OS X, Linux, FreeBSD, Solaris and Windows).

    The best part of writing plugins for Watchman Monitoring is the great help that Allen and the whole team at Watchman have given us been throughout our development of these Archiware P5 plugins. And of course everyone at Archiware and Mike at PVT have been super helpful in explaining the use of the nsdchat cli for Archiware P5, even going so far as to add some features we needed to nsdchat when we explained how useful they’d be for this project. Mille mercis. Vielen danke.

    Using GitHub to check code in, document business logic, write code, build a wiki and then track issues that need bug fixes or enhancement requests has been an adventure. It all starts with an problem that you want to be alerted for. It’s easy enough to add custom plugins to Watchman Monitoring you just need some ideas, a programmer (or two) and some time for testing, debugging, more testing and time. Did I mention you need lots of time? Ha ha

    And now for a sneak peak of the Archiware P5 beta 2 plugins for Watchman Monitoring.

    1. Watchman nicely lists the new warnings and expirations for quickly getting to the issues you need to see.             Watchman Monitoring Archiware P5 warnings expiration X
    2. Expirations are tracked with Watchman. In this case we note the date when the maintenance for Archiware P5 needs to be renewed. Don’t want to miss that! Watchman Monitoring Archiware P5 Expirations plugin Xpng
    3. Server info is good to know. Uptime, port used, and what exactly is licensed.         Watchman Monitoring Archiware P5 Info plugin X
    4. The LTO tape drive is the heart of any tape library, and alerting when it needs cleaning is very important.                                               Watchman Monitoring Archiware P5 Devices plugin X
    5. Other plugins watch the tape pools, running and completed jobs, as well as Backup2Go (workstation backup).

    Watchman Monitoring Archiware P5 B2Go plugin X

    Watchman Monitoring Archiware P5 Pools plugin X

    Watchman Monitoring Archiware P5 Jobs plugin X

  • Best of 2015: VidiXplore

    This is another great product of 2015 and  when I found it, I thought VidiXplore proves that Media Asset Management could be done better and simpler. At the end of December 2015 they released version 1.0 with new some tricks, including some changes to make a migration from Final Cut Server a reality. Time to move some clients!!

    To sum up VidiXplore, I’ll quote my tweet from Dec 21, 2015:

    Finally the perfect solution for simple asset management! Keep proxies in the cloud, originals local. Search + share!

    Working with video editors, animation and visual effects studios, I’ve come to realize that media asset management (MAM) systems can be complicated and painful. Changing workflow, oh no! Building a better pipeline is not easy, nor is it always welcome. Well, hello from the other side, we found the solution, or at least part of it.

    With VidiXplore you have 3 steps:

    Step 1. Manage your videos by keeping all the originals local. Use your own storage. Use your folder structure. Use your vids as you would normally. Don’t pay for cloud storage.

    Step 2. Proxies (thumbnail vids) go into the cloud to be viewed by you and your team.

    Step 3. Organize your videos and photos into collections, batch edit by adding tags to add metadata, search for particular assets and share them with colleagues and external clients.

    That’s it. You’re already ahead of the game. We skipped right over step 4 which was “have a lot of meetings to debate proper metadata” and step 5 which was “convince everyone at the company to adopt a different workflow.”

    With VidiXplore you switch to a monthly payment model, that is true. You don’t own the cloud platform, but what you gain is that you don’t pay a lot of money upfront to set up a large server (or many large server), nor do you need render farm for video transcoding nor for the databases you need to keep track of it all. Pay monthly. That’s the way for a lot of smaller companies. Lower up front starting cost. No extreme capital outlay in the beginning.

    Honestly, VidiXplore is a refreshing and easy way for so many people to use asset management now, so why not try it? Harder to say that with a large system setup that costs a lot of money to set up, only to find that no one wants to use it. That’s not what anyone wants.

    And now for something completely different…

    Let’s take a quick look at VidiXplore. If you’ve installed the VidiXploreAgent-1.0 agent then you’ll have a nice “V” icon in your menu bar (Mac) or system tray (Windows). Use this to open the VidiXplore agent.

    Vx Menu agent Open

    In the VidiXplore menu you can access the settings where can you set whether certain file types get a Cloud Copy uploaded by default or how many concurrent jobs can run at once.

    Vx agent prefs settings cloud copy

    When the agent is open you’ll see folders you’ve configured for media, and an option to go the website of your particular instance of VidiXplore.

    Agent

    For the first web login you’ll see an intro screen which allows you to upload additional media to VidiXplore (that is, in addition to any particular media folders you’ve configured in the agent) and the option to connect cloud storage such as S3, Azure or Dropbox. Lastly, there are also the installers for the agent.

    Welcome Screen Shot 2016-01-29 at 11.43.20 AM

    Once logged into your website you see a basic layout with options to sort by files or collections, to specify all files or particular types, sort by location and bulk actions.

    Collections Screen Shot

    If we want we can sort the results and see only collections:

    Sort collections

    Or we can sort by files which have a cloud copy:

    Sort cloud copy images

    So many options to sort, search and find what we’re looking for. Of course, we want to also edit the metadata but not so much that we require weekly meetings to decide on the 500 important and required metadata fields. Just use tags. Of course, meetings are good, and so is process, but it is so quick to select bulk actions and add a tag to a group of objects. Done!

     

    add tags bulk action MacDevOps

    “Finally the perfect solution for simple asset management! Keep proxies in the cloud, originals local. Search + share!” 

    There’s so much more you can do with VidiXplore, and I’ll go into more detail in another blog post, but this was just a highlight for my best of 2015.

    Check out their website for more information:

    VidiXplore.com

  • Best of 2015: Archiware P5 Archive app

    Announced late in 2015 the Archiware P5 Archive app is a revolution for editors who want to control the archive and restore process. No longer the job of the IT Admin, editors can select files or folders on their SAN volume (or anywhere) and send them to the tape archive.

    The Archive app is a brilliantly simple app that allows the right-click services action in OS X, or in another words a it’s a GUI app that presents a contextual menu that knows to how to the talk to your P5 Archive server. When the files are safely on tape the original files on the filesystems are replaced with stub files that can be used to start the restore process.

    Requirements: Archiware P5 server with the Archive module setup with an Archive plan. Add to that the P5 Archive App which is installed on the clients.

    Note: At the moment all archiving goes over the LAN by default, so if you have a fast SAN then you set up the P5 Archive app client settings as “localhost” instead of their actual client name. That means that when it goes to archive the file, the server knows that the files exist on the SAN at a known path (which is the same on the client and the server).

    And now for some detailed steps and screenshots.

    1. Archiving completed projects

    Choose the completed project folder and right-click. Select “Archive to P5”.

    Note 1: If you want to restore files choose the folder that was archived and right-click. Choose “Restore from P5”.

    Note 2: Restoring individual files that have been archived is possible by double-clicking the files with the “.p5a” extension, but it will be much faster to select an entire folder to restore than many individual files.

    Note 3: For either archive or restore to work the P5 Archive app needs to be installed.

    Note 4: To avoid having a services sub menu keep the contextual-menu items to four.

    Right-click folder to archive

    2. Archiving app status

    When you are archiving or restoring files the Archive app will show you the status of your request. It will also show you the status of other jobs running on the P5 server. This is to let you know why perhaps your archive or restore is taking a long time (it’s possibly waiting for access to the tape drive and it currently busy backing up or archiving something else).

    P5 Archive app Running jobs status

    The P5 Archive app offers you three operations “cancel job”, “list items” and “get report”. The last two are great when you want to examine a completed job, for example. If you want to find out what files were archived in the particular job choose “list items”.

    3. Restoring files

    Archived files will have either one of or both of, 1) a”.p5a” file extension and 2) a P5 Archive app icon.

    Folders and FCP X project bundles (which are folders) do not get the “.p5a” extension, but FCP X projects have the the icon.

    p5a-icon.png

    Note 1: Files can also be restored by the admin through the P5 web interface. They can be restored in place or to any other location that is required.

    Note 2: On the P5 server jobs that are sent to archive or restored from tape show up as “cli job” with the tapes in use.  Actual files or folders involved need to be noted from the P5 Archive app not the P5 web admin console. Otherwise checking the P5 web restore tab will files actually archived (that can be restored).

    That’s enough for the quick overview of this great new app. One of the best things in 2015.

    For more information on Archiware’s new P5 Archive app check out their website:

    P5 Archive app

  • 2015 in review

    The WordPress.com stats helper monkeys prepared a 2015 annual report for this blog, but nobody wants to read no stinkin’ reports so let me just sum it all up: Xsan, Munki, Thunderbolt, Archives. Or is that all one word? Thunderbolt Xsan Munki Archives! That’s better.

    Here’s an excerpt from the report that no one will read:

    The concert hall at the Sydney Opera House holds 2,700 people. This blog was viewed about 20,000 times in 2015. If it were a concert at Sydney Opera House, it would take about 7 sold-out performances for that many people to see it.

    Surprisingly, or not, that opera would be about Xsan. Yes, Apple’s Xsan is still alive, and Apple even added new features with OS X 10.11 El Capitan. I’m still building Xsan shared storage SANs and upgrading old ones to new versions. That was one of the good news stories of 2015 for me.

    You can build an Xsan with one or two Mac Minis and add your storage of choice. That used to mean more often than not the fibre channel storage from Promise. A great choice for larger deployments, the x30 Vtraks are solid.

    But the real shocker for me in 2015 was stumbling upon the Accusys Thunderbolt SAN RAID, the A16T2-Share. For more than half off the price of a similar fibre channel storage RAID here’s a magical box powered by unicorns that has four (4) Thunderbolt connections. Plug one Thunderbolt cable into that Mac Mini, format the raid, setup Apple’s Xsan, and then plug the other three (3) Thunderbolt cables into iMacs, Mac Pro, MacBook Pros or any Xsan clients. Wow. Awesome.

    Suddenly we have a game changer. An affordable SAN storage RAID for real block-level storage. Now more than ever we can afford to have true collaborative workflows for video editors and anyone in the creative. If you need to work together with fast connections to a shared pool then building an Xsan got much more attractive.

    Disclaimer: I got a chance to test the Accusys A16T2-Share. And I would be crazy to recommend something without testing it thoroughly. This was used for several weeks by video editors in production. It was much faster than our 4GB fibre channel storage, of course, but it was also faster than our 8GB FC storage. Speed tests showed we got close to 1GB/sec, and even when it was 97% full we got 700MB/sec. Sa-weet.

    I look forward to seeing what Accusys bring to NAB in 2016. What new box will they show up with? I hope for more than 4 client ports and faster Thunderbolt 3. Only 82 more sleeps till we all find out.

    Apple’s Xsan and Accusys Thunderbolt storage A16T2-Share were big stars of 2015, but what else stood out? The two other bright shiny lights were Archiware’s new P5 Archive app, and Vidispine’s VidiXplore cloud based MAM. More on those in posts to follow. Both of these products have transformed workflows for editors. Stay tuned!

  • Hands on with Imagr

    At the recent MacTech conference in Los Angeles I got a chance to sit in a workshop led by Graham Gilbert walking us through his open source imaging tool, Imagr.

    This was a perfect follow-up to last year’s awesome demo by Pepijn Bruienne at last year’s MacTech where he demoed his BSDPy netboot replacement running in a Docker container net booting and imaging a new VM in VMWare. Amazing live netboot demo with bonus points for writing your own netboot replacement in Python, stuffing it into a Docker container!

    This year, Graham Gilbert led us through setting up BSDPy Docker container, getting the link to VMware working and using his Imagr tool to image a new VM instance of OS X. Fun stuff.

    Here are some screenshots:

    1. VMWare booting up looking for NetBoot services
    VMWare booting up
    VMWare booting up

    2. The lovely NetBoot globe spinning

    Netboot globe
    NetBoot

    3. Progress!

    Booting up
    Booting up

    4. Image NetBoot image booted

    Netboot image booted, but there’s an issue with the plist I built by hand. Some of the keys and strings got mixed up when copying from the whiteboard. Thanks to Rich Trouton who was sitting next to me who helped me diff his plist with mine to find how I’d messed it up. Easy to fix, slightly tricky to find. Luckily you only have to edit this plist to do initial set up.

    Image NetBoot image booted
    Image NetBoot image booted

    5. Imagr start up

    Imagr start up
    Imagr start up

    6. Imagr starting, password first

    Image password
    Image password

    7. Imagr restoring OS X image

    Imagr restoring OS X image
    Imagr restoring OS X image

    8. Imgr completed workflow

    Imgr completed workflow
    Imgr completed workflow

    9. Shutting docker down

    docker down
    docker down

    Reference:

    Graham Gilbert’s blog post with slides of the workshop.

    http://grahamgilbert.com/blog/2015/11/12/mactech-2015-hands-on-with-imagr/

    Pepijn Bruienne’s blog, Enterprise Mac

    http://enterprisemac.bruienne.com