Munki makes MDM manageable

How to deploy applications using munki and simplemdm

You want to deploy apps to Macs but you also want to keep them up to date, fear not, we have a way. If you are using SimpleMDM for Mac management but hate the way MDMs deploy applications then listen up it’s easy(*) to set up Munki and use the power Autopkg to deploy and update all your apps. Note: SimpleMDM also offers a short list of curated apps to deploy without any extra setup but these instructions are for those who want to choose the apps they want to deploy. If that’s you then read on.

Managed Software Centre is the AppStore for all your apps you want your Macs to have

SimpleMDM: The basics

Macs are enrolled into SimpleMDM, then assigned to Groups. Groups have profiles assigned to them to enforce and escrow FileVault or set other policies. Simple enough, right?

Ok, what about apps?

SimpleMDM Category setting for a Munki’s Managed Software Centre

When you have apps in your Catalog you can assign a Munki category to the applications to make it show up in a nice group using Managed Software Centre (the client facing app).

With Apps in your Catalog you can manage them with Assignment Groups which are created as Munki (or not-Munki aka Standard). Next select Managed or Self-Serve, two concepts which make sense to Munki admins. One set of apps is required and will be installed without asking, and the other group is presented to the end user to choose as needed (they’re optional).

API key options. Allow Munki plugin access

API key

How do we get applications we want into SimpleMDM? Two ways exist. Import them manually and deploy via MDM or setup up Autopkg. For this we need the API key. Note: Only the munki plugin permissions are needed. Put the key into the Autopkgr.app SimpleMDM integration or set them as an environment variable and use autopkg in Terminal.

Autopkgr app choose autopkg recipes to use

Select recipes using Autopkgr (Linde Group) from the curated list of recipes created by IT Admins around the world or create your own recipes. What used to be a painstakingly difficult process by hand is now much easier with Recipe Robot by Elliot Jordan to help fish out the AppCast / Sparkle / Download URLs and transform into a nice autopkg recipe to be used by Munki (and ingested into SimpleMDM).

autopkg run -v Postlab.munki.recipe  -k MUNKI_REPO_PLUGIN="SimpleMDMRepo" -k MUNKI_REPO="" -k extract_icon=True
MunkiImporter
Using API key provided by environment variable.
MunkiImporter: Using repo lib: MunkiLib
MunkiImporter:         plugin: SimpleMDMRepo
Managed Software Centre notification

Managed Software Centre

Once Macs are enrolled and added to a SimpleMDM Group with the Munki assignment then the Managed Software Centre app will allow users to use the Self-Serve portal to install optional apps. Managed apps will install invisibly in the background.

The beauty of this integration is that Munki is awesome and works well. It is battle tested by many companies and organizations around the world. Using autopkg and its recipes to check for updates allows for a seamless automation of new apps into your catalog and then onto your fleet. Updated Macs are happy Macs.

Reference:

SimpleMDM Munki integration blog post

A Mesh VPN For Everyone

How to use Tailscale (wireguard based) mesh VPN to connect everything

What is Tailscale? It’s a mesh VPN based on the wireguard open source project. It’s a secure network to connect your own devices no matter where they are.

Tailscale is free to use with one account and up to 100 devices, which is enough to see how well this can work to connect up servers, storage and desktops. They have paid plans for teams and enterprise.

Tailscale macOS app icon

macOS and iOS

To start, download Tailscale on your Mac or iPhone then find your IP address. Once you are signed in and have your IP address you can connect easily between devices. For example, on your iPhone open the Tailscale app and see your installed devices. Click on your Mac and the IP will be copied into the clip board. Use this to connect with app such as Secure shellfish for SSH or VNC viewer for remote login. When you’re in the same network it’s impressive, but when you’re on a different network, separated far away, It’s magic.

The real test for me was to install Tailscale on some backup servers I manage to make it more secure and more convenient to access them. I had used a variety of remote control for business services and well, Tailscale is easier, quicker and much more awesome. All the other software I tried was much less awesome.

After using Tailscale mainly for remote control, I tested Tailscale to securely connect my remote Macs to my own MunkiReport server. I use Munki and MunkiReport to manage Macs and having Tailscale allows me to securely connect endpoints to the server without opening up ports on my router. MunkiReport allows me to detect malware (with DetectX plugin) or check on backup jobs with Archiware P5 backup software (using a module I wrote) or a multitude of other diagnostics such as disk space free, apps installed, and all kinds of great hardware and software metrics. So much reporting. And MunkiReport doesn’t need Munki specifically, so if Tailscale is installed for remote control why not report on everything else.

DSM Package Center: Tailscale (and Archiware P5) app on Synology NAS

Synology DSM

Having Tailscale installed in all the Synology NAS I manage in various physical locations allows me to securely connect to all of these NAS from anywhere. With remote work using a NAS is a great way to sync data between locations. Synology has a lot of great built-in tools to make this happen and a very robust quick connect feature combined with ddns, and let’s encrypt certificates to support it. After setting up a few to sync one location to another I was constantly getting notifications of IPs being blocked on my firewall. I had to open a port on my firewall to let in the ssh / rsync traffic through and despite a strong set of firewall rules with a geo block there were still connection failures and password attempts. Using Tailscale I can now have a P5 server set up on one Synology NAS connecting to the Tailscale IP of other remote units and it can easily backup, sync or archive the data from the various locations.

To install the Synology Tailscale package check out this GitHub page. Download the app then side load it (manual install in package center). To enable it you will have to have ssh on, a user with permission to use it, and one command to type.

sudo tailscale up

In one case I didn’t have SSH enabled on the remote unit so I remoted into a Mac on the same network, enabled an admin user, turned on ssh with a time limit on the account, and then logged in. Once the above command is run you will get a link to a website to authenticate the device with your account.

Linux (CentOS)

I have also installed and tested Tailscale on a Linux (CentOS) storage server. In my case a Jellyfish which has a ZFS volume shared over direct 10GbE for Final Cut Pro video editors using nfs or smb. the Jellyfish works well on premise, but wouldn’t it be nice to capture camera cards to the remote storage server via Tailscale? Oh yes it would. And what about playing back some of the video files via VLC on your iPhone! Or Files.app! Yes, to all the above. All made possible with Tailscale. And a huge shout out for their great documentation. Installing Tailscale on CentOS was super simple. Add a yum repo, install, tailscale, and then bring the service up. Couldn’t be easier.

sudo yum-config-manager --add-repo https://pkgs.tailscale.com/stable/centos/7/tailscale.repo

sudo yum install tailscale

Shared Devices

A small, but very exciting, feature was added part way through my testing of Tailscale which made it infinitely more awesome, shared devices. The concept is you are authenticated to your devices and can see in the Tailscale app all the IPs to connect to, but what if you could share one device (computer, server, NAS) with another person? Well, now you can. In the Tailscale admin console choose a device and send a share link to someone, they then will see this devices in their device list as shared. Home users can set up Tailscale to access all their own devices, but now can also choose to share access with a device in particular. For example, if you create an account, open a service (file sharing) and send a share link then the other person will login with the account you create and access the one thing you want them to. Maybe it’s a smb share to drop files. Works great for video collaboration and other kinds of teams.

There’s a whole lot more you can do with Tailscale (and wireguard) mesh VPN but I hope this gives you all some ideas to start with.

2021: Thunderbolt Shared Storage Report

It’s 2021 and what is the state of Thunderbolt Shared Storage?

Thunderbolt Shared Storage is a RAID which you connect to with Thunderbolt (and Ethernet) which can be shared with other workstations. It’s a Thunderbolt SAN. Shared by Thunderbolt.

I’ve long been a fan of Apple’s Xsan and other SAN products that use Fibre Channel (or iSCSI) to connect clients to super fast block storage. It acts like a fast direct attached RAID but you can share it with others. The sharing part is crucial to collaborative workflows. We used XSAN when I was in VFX and I now use XSAN for post production workflows. Editors like to edit, have large video files, and often work in teams. All those camera files aren’t getting any smaller. So you need a lot of storage that can be shared by a team of editors, colourists, motion graphic artists, producers, etc

What you don’t often want is a complicated network infrastructure or a server room with fibre channel switches and fibre channel RAIDs and assorted other equipment. You don’t want that. You’d have to call me and pay me to set up your storage, backup and archive workflow. While it is always recommended to work with a trusted contractor it can be expensive. For small teams a shared Thunderbolt storage SAN can be quick to set up, doesn’t take much room and can easily connect 4-8 editors. Thunderbolt 3 passive copper cables extend up to 2m but optical Thunderbolt cables are available up to 60m.

I want to review one such example of shared Thunderbolt storage that stands out, the Symply Workspace. It’s a RAID, but it’s a SAN too. It is storage you plug into with Thunderbolt, but with an extra ethernet cable you make it a SAN. Inside the storage it has Quantum StorNext which runs most of the world’s expensive enterprise SANs, but it’s in the Symply Workspace and it works with Apple’s Xsan client software (included free with macOS since macOS Lion 10.7 !!). So basically, it is enterprise storage in a Thunderbolt box. One more thing, add a 5 seat license to a simple but powerful media asset management (MAM) tool axle.ai to organize your assets, add keywords and access from anywhere. Almost too good to be true, so I had to test it and see.

I received a Symply Workspace to test with. I like testing storage. My clients always need more storage. I keep telling them to not fill it up, but they do. That’s why we have LTO, nearline and cloud archive with Archiware P5 but that’s another story. For now, let’s test this storage: how easy is it to setup, how awesome is it to edit with, and what is a MAM good for anyway? Let’s find out.

It starts with a box. You open the box. You take the drives out of the box. Two trays of drives. Then there’s the RAID itself. Put the drives in the RAID. No screwdriver or tools necessary, just slide and click. Next step read all documentation online, ha ha, just kidding, don’t read anything keep going. Ok, kidding a bit. There’s an info card in the box with a website link to help explain the setup. But I didn’t read it all, uh, I know Xsan, I can do this, ok, what’s the link? now back to building.

The tricky part is plugging in a few cables, which will be improved in the shipping production version with stenciled labels of where to plug in what. There’s three cables after the power cable. One Thunderbolt to a Mac, that’s easy. One ethernet to your local network, no problem. And lastly one more ethernet to your new SAN production network (aka metadata). Ok, what’s that? Add a 5-port switch and plug in all clients and the storage to this network for SAN private metadata. It really is a SAN. Like Apple’s Xsan or Quantum StorNext, you need a data pipe (40GbE Thunderbolt 3 in this case) to transfer the data, and a metadata network, to talk about the data (1GbE ethernet to our extra switch). The public network will be used to talk to Axle MAM or for re-sharing out the volume to non-Thunderbolt connected Macs.

Once the Symply Workspace is wired up and powered on you’ll be able to reach the unit via a local bonjour name in your web browser (http://symplyworkspace.local:8088) and from there a simple web interface allows you to monitor the status of the unit, download drivers to configure a Mac or Windows client, restart or shutdown, contact support or start troubleshooting if needed. It’s a great tool to do the few things you need to.

So with everyone working remotely how do we connect to our shared storage? I’m glad you asked. There are a lot of good options. My favourite is Tailscale a mesh VPN (based on the open source Wireguard project) which you would install on a locally connected Mac then you can remote in and share a screen or the storage. There’s also the Axle 5-seat license included with the Symply Workspace which can scan your storage, make proxies and serve it all via a website which can also be accessed from anywhere.

To be continued…

Update: Added a clarification that there is an included 5 seat license to Axle media asset management in the Symply Workspace

Do you know where your files are?

Trying to solve the problem of finding production media files across many storage platforms.

Map of storage locations created with Scaple https://www.literatureandlatte.com/scapple/overview

In media production environments you work with high speed and high capacity storage. It can be network based NAS, fibre channel SAN or Thunderbolt DAS. There’s always some backup RAIDs, individual source footage drives, file servers and even network appliances.

Glossary of Storage:

  • SAN – Storage Area Network (typically with Fibre Channel, also with Thunderbolt or iSCSI)
  • NAS – Network Attached Storage (popular vendors include Synology and QNAP)
  • DAS – Direct Attached Storage (hardware or software raid directly attached to a client or server)
  • LTO – Linear Tape Open (tape standard for backups/archives. Current gen8 holds 12TB per tape)
  • Cloud – Other people’s servers and storage. Hosted in a data center (AWS, GCP, Azure, & more)

The variety of file types is astounding: original camera footage, Final Cut Pro projects, stock footage purchased, b-roll shot from other projects, sound effects, music etc. How do you find anything? Files can be in many places, across many different kinds of storage. The question is how do you know where they are?

In the before times…

Before the great pandemic of 2020 I only heard one complaint with finding files on Apple Xsan storage: “Why does spotlight not work on my Mac?” Searching the Xsan volumes was hit and miss. When Spotlight worked it was fast and immediate. And when it didn’t, well…. Not so much. To help finding files we started using one called EasyFind from DevonTechnologies. It was free, and easy to use, but it was not fast enough for ad hoc video searches. Nor could it search across all storage areas at once. A new year, and new solutions required. 

EasyFind. Freeware for finding files. Easy to use but not fast.

One major issue with EasyFind was not being able to refine the search easily for audio and video assets. It seems skewed toward developers and while it allows you to include or exclude some file types globally in settings but it does not allow you to refine the results while searching. This makes it difficult to find what you need when you need it.

Searching for sound effects in EasyFind reveals “.h” source files. Only one file is relevant in these results.

It’s 2021 and a few things have changed over the last year: you may have a lot more places to search (SAN and NAS) and more importantly everyone is working remotely. It has become a lot more challenging to see your files, let alone search through all the storage locations and find what you need. FoxTrot Search to the rescue.

FoxTrot Search Pro. Not freeware but worth every penny. Indexes all locations so that search is fast!!

Editors and other creatives need something that will search through all their different storage places and quickly tell them where a certain file might be. It might be stock footage, a drone shot, sound effect or an old logo. But where exactly is it? What projects was it used in? What was it called?

For those who like organizing their files into logical folders, then the filesystem is your friend. Using The Finder has been the way to find things. But now there’s a lot of folders and a lot of file types in those folders. Choosing what to index in these folders help enormously. You can exclude certain subfolders on a designated storage or kinds of files that are not needed. Do not index mail or chat messages or even source code files if you know the files you want are movies and audio files. Narrow the scope where you can. Find files fast.

Choose what to index in FoxTrot Search and what not to by kind (file type) or ignore by subfolder.

You might think I’d suggest a full enterprise Media Asset Management (MAM) system at this point. And well, in the old days I would have. Apple’s old Final Cut Server worked perfectly well with classic Final Cut Pro (v6 and v7) but required rigid workflows and ingest habits. Same issue with a lot of more expensive bespoke enterprise systems. They catalog assets but at a great cost. Some clients avoid these for the cost, and others for the workflow restrictions. I’ll talk about some newer options later but for now editors just want to find the files in their own folder structure on their production storage. Is that too much to ask?

Using FoxTrot Search Pro and editor can easily search across multiple indicies (each index is a unique storage or separate folder location) than an admin has already set up and created. No waiting for indexing in real time, search now, with results instantly show in app. Narrow down and refine your search results easily. Don’t want to see Mail messages, or images, only audio files? Easy. Then further define only the file type you want. It works.

FoxTrot Search can refine searches by kind and extension to find the relevant files you actually want.

One of the recommendations to my clients with large storage (SAN or NAS) is to have secondary nearline archive as well as backups. Either a Thunderbolt RAID array like the Accusys Gamma Carry or a desktop QNAP or rack mounted Synology NAS. Using Synology is great for many reasons, for example, it includes a nice web login, and it can search for files. But to be completely honest, the built-in search only works sometimes. A common issue with search on the Synology is the corruption of the search index. It seems to happen some times for some storage locations, but all the time somewhere. Rebuilding this does not seem to fix it for long if at all. This was the major reason for switching to FoxTrot Search. I want something that works every time. Across all storage volumes: SAN, and NAS.

Synology NAS corrupt search index.

To be fair, I had to rebuild an index or two in FoxTrot Search as well, but it worked. Every. Single. Time. Rebuilding the Synology search indexes never seemed to fix the issues. With FoxTrot Search it’d warn me there was a lot (A LOT!) of items to index and it will take time but that’s why I’d do it over an evening and rebuild automatically them after hours or every weekend. FoxTrot Search would also warn me of problem files that took too long to index and could then be added to exclusion lists automatically.

When Spotlight stops working, your Synology search index is corrupted again or just want a great search tool for active storage locations, I recommend FoxTrot Search. It searches across all storage and provides access to the files where possible. And it’s fast. Really fast. Did I mention that yet?

FoxTrot Search Admin and Server apps

One word about the setup and the various FoxTrot Search apps. They have a personal edition for searching your local workstation and storage. The major difference with the Professional version is having multiple indices. They also have a server and a per seat license. I tested the Pro version with a few indices first and then set up the server to share these generated search index files with editors. I ended up making more indices after testing. Which so many storage locations I initially tried to make fewer indices but with so many thousands or in some cases millions of files it was better to make an index for each separate storage location (and per specific folder in some cases). It also helps when narrowing down a search, the editor can specify specific locations easily (as well specifying file type, language or file extension).

FoxTrot Search Server app. Define set port to connect to the server.

Once we tested FoxTrot on the local network we needed to make it work with the VPN. And now! The server version of FoxTrot Search allows you to set a fixed port for the server which we could open up on a firewall. In my initial testing I could connect to the server but not to each index of the various storage locations. This was frustrating. So close…. And I will admit here one criticism of FoxTrot Search is the documentation. It doesn’t really exist. They have a user forum and release notes but in my initial setup of the server I didn’t understand exactly what was needed. This could be solved with better documentation. I did resolve this issue with a few emails to the developers who explained to me that each index required its own port and therefore I needed to open more ports on the firewall. Ok, good to know. To confirm that these ports were or were not accessible in my testing I used “nc” in Terminal to scan open ports.

Scanning for specific open ports with nc binary. Port scanning is only one its many useful skills. Read the man pages for more exciting stuff it can do.

So FoxTrot Search is great for search across all active storage and is super helpful for seeing previews of video and audio files you may to use as an editor, but it can help find so much more. If you keep production documents, spreadsheets, PDFs, text files it will search through all them too. So your script or production run sheet is available to help you find what you need.

LTO (tape archives)

In my “where are my files” graphic at the top I show various storage locations including LTO and the cloud. The completed projects always go to LTO (tape) archives. How do we search those with FoxTrot Search? I use Archiware P5 which has a web server that is very easy to search with and restore any files via web login but to make things more fun why not have FoxTrot Search index the archive inventory? Of course it can. There’s a cli command in P5 to export the inventory of every file archived and this is a searchable tab separated (TSV) file. I’ve spent a lot of time with this because I’ve been working on a separate tool to analyse these inventory files (with sometimes millions of items in them) to see historical patterns and predict future trends. More on this specific tool later.

Xsan is archived to LTO (tape) with P5 and to the cloud. Postlab Drive with proxies (smaller version of original media) don’t need to be backed up but other creative production files do. The cycle continues.

The Cloud

New to many is the cloud. Proxies for editors in my recommended workflows get stored on a cloud drive like Postlab, FoxTrot Search can search that too. So many places for originals or backup copies to be. FoxTrot Search should be indexing all of them for you.

In a future blog post I will discuss new media asset management (MAM) systems and what’s changed over the years, but for now if you need a tool search across all active storage then take a look at FoxTrot Search.

Thunderbolt Storage in the field: Part 1 – The Gamma Carry test

I’m testing the new Accusys Gamma Carry thunderbolt storage in a series of blog posts with real life situations. Filming on location and editing with a remote team requires a combination of good workflow, great apps, and excellent storage. The Gamma Carry is small sturdy box meant for on set and on location editing and camera off loading. It also survives going back and forth between your office and your home office, and wherever you need to go. Disclaimer: I was not paid to write this review and this blog post is an ongoing field test with storage required by every day editors making films using local thunderbolt storage and the cloud. This is their story.

In the beginning….

The start of every creative editorial project is choosing the tools needed for the job, and working as team to make it happen. Making a film requires the hard work of everyone, not the least the IT / Tech who supports the crew and prepares the gear. On set the DIT (digital imaging technician) copies camera cards to multiple hard drives and backup devices before going back to the office to copy them to LTO (archive tape storage), but before the DIT can do their job the IT / Tech has to set up the RAID and design the best and safest backup and archive workflow.

Gamma Carry Overview

The Gamma Carry is an 8 drive external RAID box and it really small and sturdily built. There is a solid metal handle built into the top case and the drives carriers where you put the drives are metal and very solid. This entire unit is built to be solid and protect the drives it contains. The drive tray have a pin to lock the track in place when inserted as well as a thumb screw to lock them in. All drives must be attached with provided screws which is not as convenient as some others with quick plastic tray mounts, but this metal cage for your drive is solid and feels safe for a RAID that will probably be transported everywhere. Keep the data safe!

There are other little touches which are well thought out. The blue red green blinky lights are perfect for any Christmas holiday party and tell you important information about the state of your RAID at a glance but you with one button push turn off all the blinky lights. And keep working in the dark editorial suite not bothered by anything blinking. There’s a mute button and an extra port on the back with 60W to charge your laptop via the RAID. Very useful.

Gamma Carry The Setup

The set up of the Gamma Carry software wise is identical to the Accusys A12 T-Share which I set up recently and once you have the Accusys Mac installer then you have the RaidGuardX app. And the usual IT caveats apply, the software is not signed so you will have to right click on it to open it up. Hopefully they will sign this and notarize their software to make this easier for end users. Also there is a requirement for JRE to run RaidGuardX which means downloading and installed Java on your Mac. Also not optimal but only necessary for the RAID setup.

RaifGuardX detects the Gamma Carry

The set up of the Gamma Carry is the same as the Accusys A12 T-Share and a raid array you built with RaidGuardX will be recognized in the Gamma Carry. That could be a good thing, or not. I did have some excitement when it recognized the array I had built previously (I used the same drives form the Xsan setup). And this was the next step to resolve, because tp create a new array, or delete the array then create a new one I have to do one more thing. Since they were used with an Xsan I had a LUN label that identified them as such and had to remove this LUN label before proceeding. Occasionally we see this issue when re-using drives that had once made up a RAID which was part of an Xsan.

Delete the existing RAID array…. button grayed out.

Xsan to the resuce.

To see the RAID arrays avaialable when building an Xsan you can use the cvlabel command to list them. You can also use it to remove this label. WARNING: Do not do this when connect to an Xsan or Stornext storage network. Unless you know what you are doing. You are warned. This is dangerous. Removing a LUN label can bring down the entire SAN. That’s it. Now you know.

sudo cvlabel -l                                      

/dev/rdisk3 [ACCUSYS Gamma Carry      366] acfs-EFI "accusys"Sectors: 46881814495. Sector Size: 512.  Maximum sectors: 46881814495. Stripebreadth: 0.


sudo cvlabel -u "accusys"           

*WARNING* This program will remove the volume label from the
          device specified (accusys).

          After execution, the devices will not be usable by the
          Xsan. You will have to relabel the
          device to use it on the Xsan.


Do you want to proceed? (Y / N) -> y
Requesting disk rescan .% 
                                                                                     sudo cvlabel -l                          

/dev/rdisk3 [ACCUSYS Gamma Carry      366] unknown  Sectors: 46881814495. Sector Size: 512.  Stripebreadth: 0.

So all is good again now we can create new RAID arrays now that the Xsan LUN label was removed. Back to work! Once the drive is set up in Disk Utility as a new volume then you’re ready to go. In this case I added one more drive and created a 5 drive RAID5 set and formatted as HFS+. In my testing this was fast enough and would be faster if all filled with drives or SSDs. There are variations of this hardware with SSD ports instead of 1 or 2 drive bays to allow quicker ingest of SSDs which have camera footage on them.

Speed test of a 5 drive RAID5. Fast enough!

I then set up 48 hours of drive copies via Hedge for some testing of the RAID hardware. Thanks to Hedge Connect I get notified when the large copy finished. In this I was copying Thunderbolt 2 and 3 external hard drives (thunderbolt but hard drive and not that fast) to the Gamma Carry to be copied. The source drives are slow. Copying from SSDs would be way faster.

And of course Hedge found some minor warnings with these old drives. People don’t like LTO archival tape and want to have stacks of hard drives, but that data on those hard drives won’t last forever. Keep your important data in three places on two different kinds of media (tape, cloud, drives etc). This test of mine was to copy off old drives and use the Gamma to re-organize and re-sort for a new edit project. Now to back up to the cloud and set up Postlab projects with postlab drive for the proxies. In the mean time the editors can get the original footage when I carry over the Gamma Carry thunderbolt RAID.

This is part 1 and after some more testing I will publish some real world tests and experiences.

TCC troubleshooting

Download Howard Oakley’s Taccy app

Read Howard Oakley’s blog post on Catalina and privacy protection

Read Apple’s profile reference doc with respect to Privacy Preferences Policy Control payload

Read Rich Trouton’s guide to creating privacy pref policy profiles

This snippet (from MacAdmins slack) shows tcc in the logs if that is the issue:

log stream –debug –predicate ‘subsystem == “com.apple.TCC” AND eventMessage BEGINSWITH “AttributionChain”‘

Drop it! MunkiReport Db hacking

SQLite3 Db hacking for MunkiReport

Making modules for MunkiReport is easier than ever. Seriously.

please make:module

It’s easier than ever to make modules for MunkiReport (check out the recent MDOYVR MunkiReport workshop) and since the heavy lifting is done you can concentrate on the business logic (what makes sense) and the commands to execute or the scripts to run (python, shellI, etc). Worry about actionable data and less about the tables and views.

If you testing in production (which you should never do, always test is a testing environment) then you may happen to change a module (tables and fields etc) but keep the module name the same. This will confuse your database and you will need to erase it from the db to continue. In SQL speak this is “Drop table”. (You could also delete the munkireport Db and start again, but this is for those crazy enough to test in production and may want to keep the other data).

Sqlite commands for Munkireport

  1. Maintenance mode (current way)
sudo ./please down
Application is now in maintenance mode.

Old way –> sudo touch /Users/Shared/MunkiReport/munkireport-php/storage/framework/down

 2. Edit MunkiReport db

sudo /usr/bin/sqlite3 /path/to/MR/app/db/db.sqlite

 3. Exit maintenance mode (current way):

sudo ./please up
MunkiReport is now live.

Old way –> sudo rm /Users/Shared/MunkiReport/munkireport-php/storage/framework/down

Note: if you forget to get out of maintenance mode then clients can’t check in

“ERROR: Server error: MunkiReport is in maintenance mode, try again later.”

4. Migrate Db

sudo ./please migrate

Please use please migrate (or migrate db in web admin) if making changes to a module or else. That is, if you’re crazy to do this in production.

Server An error occurred while processing: \fancy_module_processor
Server Error: SQLSTATE[HY000]: General error: 1 no such table: fancy_module (SQL: select * from "fancy_module" where ("serial_number" = D09TP1QLH1K3) limit 1)

5. SQL hacking:

If you’re testing in prod and change a module’s fields but keep the same module name this will confuse your database and you will need to erase the entire db and start again or just erase this module from the db to continue. In SQL speak this is called “Drop table”.

A. List tables

For tables, the type field will always be ‘table’ and the name field will be the name of the table. So to get a list of all tables in the database, use the following SELECT command:

SELECT name FROM sqlite_master
WHERE type=’table’
ORDER BY name;

B. Drop table

Go through the list of tables and confirm the one you want to drop. Then do it. You’re backed up anyway, right? I mean, the data will come back when the clients check in again. So don’t worry.

Drop table to remove “fancy module” table from Db

DROP TABLE fancy_module;

3. Exit

.quit

REFERNCE:

MunkiReport WIKI – https://github.com/munkireport/munkireport-php/wiki

Jon Crain’s module making blog series – https://joncra.in/2018/11/30/creating-munkireport-modules.html

SQLite FAQ – https://www.sqlite.org/faq.html#q5

Notarize it!

Apple’s notarization service allows Apple to verify apps distributed outside of the App Store system. If you make your own apps to distribute to customers, clients, family or friends then you will have to notarize them by submitting them to Apple. This avoids painful dialog boxes in macOS 10.15 Catalina that prevent your app from launching by default.

NotarizeYourApps-Apple-Oct2019

Notarization The Hard Way

I’d been putting off notarizing my apps created for my clients for three reasons,

1)  it isn’t a strict necessity because most users are on macOS 10.14 Mojave,

2) I use Munki to distribute and install software which bypasses the requirements, and

3) I’m lazy

But it is only a matter of time before this would be a strict requirement and necessity. Also the relaxed requirements for notarization of apps was about to change again in February 2020 and I said this is the moment to do something. What now? Check with Rich Trouton and his blog Der Flounder.

codesign –force –options runtime –deep –sign “Developer ID Application: Name (#H7373736)” “/Applications/Cool-App.app/”

Rich Trouton is the modern major general of documentation and a super awesome dude. His blog Der Flounder has documented this process and now it was time to revisit this. Step by step recipes well explained with comments. What’s not to love?! Well, I didn’t get far because I missed some ingredients. Signing the app failed. I couldn’t notarize it without signing it. Hmm…

error: The specified item could not be found in the keychain.

It didn’t work the hard way, so let’s try it another way.

Notarization Made Easy

A very awesome app from Late Night Software called SD Notary can help make this process go smoothly. Their app detects if you have the right cert to run this process. Something which I thought I had, but did not.

Certificates, Identifiers & Profiles

It’s no accident I got tripped up in the same place with the cli and with the SD Notary app to notarize my app. I was missing the correct certificate. When I tried to codesign as a first step that’s when I got an error that I puzzled over for a minute.

The SD Notary app stopped me also at the first step because it said it couldn’t find a Developer Signing ID. And that with the command line error finally made me realize I’d missed something. And here I thought creating the app specific password was the hard part. (It wasn’t hard, but you have to look in the right place!).

A quick run back to the Apple developer site and a trip to the “certs identifier and profiles” section to create a new “Developer ID Application” cert (I had the installer one previously) solved that. I also had some trouble creating an app specific password, mostly because I was looking in the wrong place (in my dev account, not my apple ID account) but that got sorted.

Notarize-Apple-CreateNewCertificate

The “Developer ID Application” is what I needed.. Of course to get this I need to generate a cert signing request. There’s always a few steps. But once these are done then you’re good to go.

Once the proper Application type cert is in place, and the app specific password then you’re able to notarize via cli or an app like SD Notary. I tested this in Terminal:

xcrun altool --notarize-app --primary-bundle-id "com.apple.automator.Cool-App" --username "memyself@email.com" --password "really-cool-passw0rd" --file "/Applications/Cool-App.app.zip" 

No errors uploading '/Applications/Cool-App.app.zip'.
RequestUUID = 12345f-567e-476f-a229-6789cef906b

And in less than 3 minutes I received an email declaring it done. “Your Mac software was successfully notarized.

Then I went back to SD Notary and tried again. It was also successful and after selecting the app the entire process of signing, zipping, submitting to Apple, then stapling was done seamlessly.

SDNotary-Stapling

Hope that makes sense to someone. And the next time I notarize an app I will be able to do it seamlessly thanks to the help of everyone who has provided documentation and cool apps. Cheers.

References:

SD Notary app — Notarizing made easy

Rich Trouton’s Der Flounder blog

Apple dev docs

Howard Oakley’s Eclectic blog

FCP7 to FCPX

If you used classic Final Cut Pro 7 for years then eventually moved to FCPX now what do you do when you want to restore an old project? Read on…

FCP7 to FCPX

In the beginning we set up an older iMac that had been sitting around and already had macOS 10.12 and Final Cut Pro 7 (and even an early version of FCPX). We used this iMac to open up old FCP7 projects from our projects archive which were restored from LTO tape archive created by Archiware P5.

This process of restoring from tape archive back to the SAN then copying to an external drive to attach to this older iMac to convert worked but was cumbersome and not convenient. Opening old projects in FCP7 and then exporting out the XML was easy. Using SendToX to convert to FCPX XML was also easy. But getting the project to this old Mac off the main network was a drag.

Retroactive app

Use Retroactive app to install Final Cut Pro 7

Then one day I heard of this project that allowed to install iTunes on macOS 10.15 (Catalina) which only had the new Music app. Weird flex, but OK. Reading further it also allowed FCP7 to be installed on macOS 10.14 (Mojave)! Now this was a useful revelation. The app is called Retroactive and it would be very useful to us. Now FCP7 could be installed on the same Mac as FCPX. It would then have access to the network and the SAN where do all our editing and where we restore archive from LTO archive. Awesome.

The best part was that we moved from a dedicated old iMac running macOS 10.12 to a newer iMac Pro with macOS 10.14 on the Xsan and can run FCP7 thanks to that new app that makes it work. And then FCP7 to XML to sendtoX to FCPX is not too bad.

We also used Kyno to drill down into all the restored projects to identify en masse all the restored footage that it incompatible with FCPX. Renamed and then reconverted. All is well again for now. Archive restored, FCP7 projects converted to FCPX. Yeah, happy times.

Kyno batch rename dialog box

Kyno batch rename

Kyno FCPX incompatible files reanaming converting

Kyno convert and transcode

We had one minor snag in the process. Some of the restored projects didn’t use FCP7 they used early versions of FCPX with their events and projects folders separated (not the current library structure). Latest version of FCPX 10.4 did not know what to do with these projects that were also some times stored on sparse disk images (oh how the Xsan did not like these projects at the time). There was a menu for a while to convert these projects but it was now gone. What to do?

Back to the old iMac and we used FCPX 10.2.3 to convert these projects from 10.0 version to 10.2.3 library which can then be converted to the latest FCPX 10.4 format. Almost easier to convert FCP7 to FCPX in one shot but it worked and we were happily editing old projects in the latest version of FCPX.

fcpx 10.2.3 dialog box to update projects

fcpx 10.23. update projects and events dialog

Editing old projects in new FCPX

We have a way to restore old projects from LTO tape thanks to Archiware P5, a way to identify, rename and bulk convert old footage in an easy fashion thanks to Kyno and now also a way to convert FCP7 with SendToX and Retroactive to make it more seamless.

Hope this helps anyone else if want to do the same thing good luck.

Backup and Archive setups for media professionals

I recently did an online presentation on my Archiware P5 backups and archive workflows for media professionals.  

I use Archiware P5 to backup and archive media productions to tape. In this workflow I describe how original material, work in progress and conpleted projects are backed up daily and archived as needed.

This presentation goes into some detail of what two different storage setups look like and what similarities exist in my Xsan (fibre channel) and Jellyfish (10GbE) backup and archive workflows.

Despite the storage differences, the common workflow is that they all use the P5 Archive App to allow the Final Cut Pro X editors to archive completed projects. The simple way to do this is by right-clicking in the Finder and they will go to tape and get replaced by stub files (small files with the same name and negligible file size). When the project or any of the footage or assets needs to be restored it is another right click to restore. No IT people need to be involved. It’s magic. Almost.

Useful resources:

– Case Study: WorkSafe BC

Using P5 and the P5 Archive App

https://p5.archiware.com/solutions/worksafe-bc-hs-video-production

– The P5 Desktop Edition

Aimed at small teams and single users: P5 Archive and P5 Backup to a single LTO tape drive.

https://p5.archiware.com/desktop-edition

– P5 and Xsan

https://p5.archiware.com/solutions/xsan

– P5 and Lumaforge Jellyfish

https://p5.archiware.com/solutions/lumaforge

– Watchman Monitoring

https://www.watchmanmonitoring.com

– MunkiReport

https://github.com/munkireport/

– MacAdmins Slack:

https://macadmins.herokuapp.com

– MacDevOps:YVR Conference

https://mdoyvr.com