The case of the strange disappearing drive space

Recently I was asked to look at a 4TB drive that was only showing less than 2TB available…. No problem, I said, this is easy to fix. Famous last words.

Just open up Disk Utility and resize the partition, or reformat the disk, right? Easy Peasey. Well, it took some troubleshooting to time to figure out and a trip to Terminal was required to solve this weird case, plus I learned a new command along the way. Fun.

The Problem:

Buying a 4TB hard drive then putting it into your external drive case for backups should be simple,  but what if instead you got a nasty surprise and it showed up as less than 2TB?

Troubleshooting the issue:

4TB drives were presented to me and when I loaded them into an external SATA dock then showed as 4TB drives with a partitioned volume of less than 2TB.

I tried to delete the phantom partition, and I tried resize the volume to use the empty space in Disk Utility.app but it refused to budge. This needed a trip to Terminal.

man diskutil

Using “man” or “info” commands you can find out more about almost any particular command. Maybe some useful options or arguments would be listed or at least some examples would help.

NAME

     diskutil -- modify, verify and repair local disks

SYNOPSIS

     diskutil [quiet] verb [options]

DESCRIPTION

     diskutil manipulates the structure of local disks.  

 

To find out more about what we’re faced with let’s ask diskutil what it sees:

diskutil list
/dev/disk2 (external, physical):

   #:                       TYPE NAME           SIZE       IDENTIFIER

   0:      GUID_partition_scheme               *4.0 TB     disk2

   1:                    EFI                   209.7 MB   disk2s1

   2:        Apple_HFS Backup                  1.8 TB     disk2s2

Looking through the man page the “resizeVolume” command caught my eye. Also the “limits” option seemed interesting. How

diskutil resizeVolume disk2s2 limits

Resize limits for partition disk2s2 Backup:

  Current partition size on map:         1.8 TB (1801419800576 Bytes)

  Minimum (constrained by file usage):   846.4 MB (846426112 Bytes)

  Recommended minimum (if used for macOS):26.8 GB (26843545600 Bytes)

  Maximum (constrained by map space):   4.0 TB (4000442028032 Bytes)

The Answer:

Reading through the man page revealed that the best way, and new to me, was to resize the partition to use all available space with “R”. Of course, so intuitive.

sudo diskutil resizeVolume disk2s2 R

I did get some errors. But repairing the disk fixed those issues. And I was able to resize the disk in Terminal with diskutil where Disk Utility.app had failed.

sudo diskutil resizeVolume disk2s2 R

Resizing to full size (fit to fill)

Started partitioning on disk2s2 Backup

Verifying the disk

Verifying file system

Volume was successfully unmounted

Performing fsck_hfs -fn -x /dev/rdisk2s2

Checking Journaled HFS Plus volume

Checking extents overflow file

Checking catalog file

Checking multi-linked files

Checking catalog hierarchy

Checking extended attributes file

Checking volume bitmap

Checking volume information

The volume Backup appears to be OK

File system check exit code is 0

Restoring the original state found as mounted

Resizing

Modifying partition map

Growing file system

Finished partitioning on disk2s2 Backup

/dev/disk2 (external, physical):

   #:                       TYPE NAME          SIZE       IDENTIFIER

   0:      GUID_partition_scheme              *4.0 TB     disk2

   1:                        EFI             209.7 MB   disk2s1

   2:                  Apple_HFS Backup      4.0 TB     disk2s2

And lastly, the issue may have been caused by the old drive dock which refused to see the 4TB volumes even when correctly resized. A newer drive dock was required.

Automate it! Hedge API example apps

Quick post to talk about some fun I’ve been having with the new Hedge API.

Background: Hedge is an awesome app for securely copying Camera Cards to multiple destinations to ensure data integrity and safety of the original footage. Hedge is one part of a workflow I build for my clients. Hedge is the first step in ensuring an easy and convenient transition from the cameras to the SAN to the tape archive powered by Archiware P5.

TL;DR

Using AppleScript and Automator I have built some apps to quickly set the file naming and data integrity preferences as we want them to be.  And also quickly change them to something else depending on the needed workflow.

API or Clickety click click

Copying camera cards copies is what we use Hedge for. Certain preferences like logs or receipts are great to have to ensure the copy succeeded. Also file naming conventions are good to have. Set and forget, right? But what if you did forget? Or if you’re new and don’t know the convention or you don’t read documentation. What do we do? One way to solve this is build an app that launches Hedge and sets the correct preferences. And if we want to copy USB sticks or something else then we can launch another app that prepares Hedge with a different set of preferences.  For extra points we just ask the user what they want with a nice dialog box and just do that.

hedge

Automator

Automator is awesome. Create workflows, apps, or services amongst many other things. For more info on Automator check out Apple’s official docs or this unofficial website of resources.

Automator.png

For this quick testing I used two AppleScripts with different preferences and settings defined. One for camera card copying and another for USB sticks that need different preferences set. The fancy automator app just calls the needed AppleScript. Make two apps and you have two different workflows without having to explain to users which prefs get set for what, or how the file naming should go to be consistent.

Example 1: Cam Card script (snippet)Hedge-Automator.png

Example 2: USB card copy (snippet)

Hedge-ApplesriptUSB.png

 

Automator can do many things. Call AppleScripts, Run shell scripts, pop-up dialog boxes etc and this is just a simple example of building single purpose apps to set Hedge via its new API. Very cool and so many possibilities.

AppleScript

What if we could just build one app which asked the user what they wanted to do? We can do that!

AppleScript-DialogQuit.png

Choose “Cam” and the appropriate preferences are set and file naming conventions applied.

Hedge-CamCard-Prefs.png

Hedge-CamCard-FileNaming.png

Choose “USB” and a different set of preferences are set. Magic.

Hedge-USBCard-Prefs.png

Hedge-USBCard-FileNaming.png

How do we do this? This piece of AppleScript chains an action to a response or button choice. Run a script or choose an action. The possibilities are endless. And thanks to the Hedge API we can set preferences on or off, and set destinations or many other things. We can do them programmatically and we can ensure they are set correctly. Fun times!

Hedge-API-Script-Quit.png

AppleScript vs POSIX:

I updated my AppleScript code with the POSIX path of the scripts it wants to load. It’s a major improvement! I had packaged up my scripts and my Hedge Setup app with munkipkg then deployed through Munki but when I demoed it — nothing worked…. because the start up disk drive was named something else. The fix: set a variable to be the POSIX path (Unix path in AppleScript friendly format).

Scripting and App Building

I hope that helped. We can do a lot of the same things with Python and in my testing I was working with a script written in python3 but since that’s not shipping on Macs in the current version of macOS by default (not yet!) then AppleScript was the quickest way to get this done. This is not restricted to AppleScript. Using Automator and your favourite scripting language you can build apps for your clients, co-workers, friends and family.

A note about the Hedge API:

There are two major calls I use in my scripts “setDestination” and “setPreferences”

The “setDestination” call looks like this:

open ‘hedge://actions?json=[{“setDestination”:{“path”:”/Volumes/LaCie/Testing/Test1″}},{“token”:”1234567890123345555″}]’

Note: the token is generated for you when you have a Pro license.

The “the setPreferences” call uses plist keys.

Note: I’ll have more say about using the actual Hedge API after it is officially announced.

Updating the P5 client on the Jellyfish

You’ve successfully installed Archiware’s P5 backup and archiving software on your backup server following my previous blog posts and after it has run smoothly for a while you decided to upgrade the version of P5 on your server, but how do you do this on the Lumaforge Jellyfish storage? I’m glad you asked.

There are a couple of ways to update your P5 agent, and I will show you the built in way in Archiware’s P5 software. Surprisingly after many years of using P5 I have never used this method before. I’ve been using Munki for years to upgrade all software on my Mac clients including P5 and on Linux and Solaris servers I’ve just done it by hand. Install over top of the previous version and voila upgrade! But what if you didn’t want to ssh in as root and just install over top, what if there was a better way? I present to you the official “Update client” dialog box. It’s nice.

Update-p5-jellyish-1

Updating client software assumes you’ve set up clients in the P5 server clients section, This is needed when you want to use these server agents to designate their attached storage as a backup, archive or sync source. And also, this assumes you’ve updated your server.

P5 client update Jellyfish 2 Screen Shot 2018-08-06 at 4.40.03 PM

During the update process there are some nice dialog boxes to let you know what is happening.

P5 client update Jellyfish 4 Screen Shot 2018-08-06 at 4.44.34 PM

And afterwards you can test your client with a Ping test.

P5 client update Jellyfish 3 Screen Shot 2018-08-06 at 4.44.27 PM

Success! Looks like we’ve updated our client successfully. How wonderful. And no need to mess about in Terminal with a root shell. No telling what kind of trouble we could get into with those elevated privileges…. much safer this way.

Thanks Archiware for making this great software. I depend on it every day.

 

 

P5 on the Jellyfish: Archiving Gotchas

TL;DR

Using Archiware P5 to Archive files to tapes is awesome, but watch out for little things you might miss, such as the path to the files and backing up your Archive Db.

P5 Archive on the Jellyfish

Using P5 Archive with the Lumaforge Jellyfish is a great way to preserve your digital archives. See this post for how to set up P5 on the Jellyfish

Using Archiware P5 for archiving makes sense. You want your completed projects and original camera footage on LTO tape. But how do you do archives? There are several different ways, and there be gotchas.

P5 Archive vs P5 Archive app

Using P5 Archive to manually archive completed projects to LTO tape is a process of logging into the server via a web browser and selecting the the project folder you want to archive to tape.

The completed project folder could be on the storage visible to the server or it could be storage the client sees. And that can make a difference. Where the storage is mounted is different on a Mac vs Linux. Its’ the difference between “/Volumes” and “/mnt”.

The same Jellyfish storage, either SMB or NFS, when seen on a Mac is mounted by default at “/Volumes” (this can be changed but for most people leave it at the default). But when archiving the storage via a Jellyfish client you will get “/mnt” path.

p5-smb-test2.png

Using the P5 Archive app, which is a Mac only companion application to P5 Archive, to run the archives you will see the storage archived as “/Volumes”.

This first Archiving gotcha is if you’re archiving the Jellyfish storage with the web application of P5 Archive you will have to find your footage and restore from the “/mnt” path vs if you’re archiving from the P5 Archive app which is running from a Mac and will see and store the footage using the “/Volumes” path.

All this to say that using both ways to archive may double up your footage in your archive which may be unintended. And from a restore in the web browser finding your footage may be confusing if you’re used to seeing it mounted in “/Volumes” and you actually find it under “/mnt”.

Note: the reason to use the P5 Archive app is because of the simplicity of right-clicking files in the finder which are on your storage and telling them to archive right then and there. Files are copied to tape then the original files on the storage are replaced with stub files. Right-click again to restore. Simple.

p5-archive-app-job-monitor.png

Backup your Archive!

Don’t forget to backup your archives. Or rather, your archive Db. A more recent addition is the ability to automate the backups on the Archive index, so don’t forget to enable it.

In the managed index section, choose your Archive index.

Set the target client where the backups are going and the backup directory. Choose a time and don’t forget to enable it (check the checkbox and hit apply before closing the windows).

Note: Repeat this setup for each Archive index you want to backup.

Archive Backup db setup3.png

Monitoring your Archive!

Don’t forget to enable email notifications for your P5 server to get your inbox full of status notifications and errors and other important stuff. But if you want to cut down on email notifications or you have multiple P5 servers (many different clients, perhaps), then you might want to check out Watchman Monitoring and the P5 plugin that is built-in). Find out easily when your tape pools are getting low, the tape drives needs to be cleaned, the support maintenance needs renewing etc. All in one dashboard. How convenient!

Maybe everything is going well…

Watchman-P5-info.png

Or maybe not!

Archiware-P5-Jobs-Watchman-tapes-required.png

 

Install P5 on the Jellyfish

TL;DR

You can easily install Archiware P5 backup and archive software on a Lumaforge Jellyfish storage server. Once you’ve done that you can backup to tape or disk or the cloud directly or through another P5 server. Backups are good. Archive are good. Restores are better.

P5 install on the Jellyfish (Linux) How-To:

Note: Thank you to Lumaforge’s CTO Eric Altman who gave me some basic instructions to get me going.

Step One: Download the latest Linux P5 rpm file 

http://p5.archiware.com/download

p5-Linux-rpm.png

Copy the downloaded rpm file to the root folder of your SMB or NFS file share.

 

Step Two: Install the rpm file

Open Terminal and ssh into your Jellyfish. Login as root or as another appropriate user.

yum localinstall /mnt/Primary/ShareSMB/awpst554.rpm

 

Step Three: Browse to server on port 8000 to test that the server is up

e.g. https://jellyfish:8000

Or in Terminal and ssh into your Jellyfish and ping your P5 server

cd /usr/local/aw 

./ping-server

Pinging PresStore application servers...

  lexxsrv pid: 4840 (server is running)

  lexxsrv url: http://127.0.1.1:8000/login 

Pinged 1 from 1 application servers.

 

Step Four: Decide if the Jellyfish storage will be a P5 client or a server.

Note: If configuring the Jellyfish storage as the main P5 server you may wish to set up a user that only has access to the shared volumes.

For my set up the Jellyfish storage is going to act as a P5 client to a main P5 server on a Mac mini (yes, they are useful for something). The Mac mini is this case is the P5 server and is attached to theOverland tape library via a Promise SANlink2 Thunderbolt Fibre Channel adapter.

NEOs-T24-large-new.jpg

macmini-ports.png

 

Step Five: Set up the Jellyfish storage as a P5 client

Log into your P5 server and add the Jellyfish by the IP known to the P5 server. In this case the P5 server is connected via 1GB to the Jellyfish in Port 1.

P5 clients jellyfish setup1.png

Note: You could also choose to plug into the Jellyfish via a 10GB port, but in my setup these 10GB ports are reserved for the edit stations. You should choose what’s appropriate for your setup.

P5 clients jellyfish setup2.png

Resource utilization of P5 on the server is low, topping off generally at 1GB of RAM at peak usage. While this does technically take resources from ZFS caching, the impact should be super minimal.

In my observations the CPU never spiked too high while both serving NFS and SMB mount points to multiple Final Cut Pro X workstations even with backups or archive jobs going to tape at the same time.

jellyfish-cpu-resources-graph.png

More Jellyfish P5

See the follow up post on Archiving gotchas with the Jellyfish here

 

macOS High Sierra vs Server.app

Upgrading to macOS High Sierra is akin to walking on the bridge of peril. Too perilous!

I don’t recommend macOS 10.13.x for production, but it is necessary to test and for this reason back in September I did upgrade my test Mac. Of course, when the installer detects server it will give you a warning about it not being compatible and you’ll have to download a compatible version from the App Store. Be warned!

ThisVersionOfServerNoLongerSupported2

Which is no big deal as long you are warned and have backups and maybe you can download the compatible version from the App Store. Trying to launch the old version will get you a warning to go to the App Store and be quick about it.

ThisVersionOfServerNoLongerSupported

Some people are reporting that the macOS installer is erasing their Server.app and refusing to upgrade their Server with the macOS 10.13 compatible version (v.5.4).

In that case, restore from Time Machine or other backups and start again?

I don’t get High — Sierra!

Friends don’t let friends install macOS High Sierra in production. Don’t get High, Sierra.

macOS 10.13 was released on Sep 25, 2017, and almost two months later with only one point release update, it’s still too new for production. Download it on a test machine or two or more, test it with your apps and systems, file bug reports and radars, but for the love of all that is Python and Monty! don’t run it on your production Xsan. Well, at least not yet. Wait until next year. Or as long as you can. Or until the new iMac Pro is released with 10.13 pre-installed or wait until they ship the new Final Cut Pro X 10.4 that may or may not require macOS High Sierra.

With that out of the way, I’ve just upgraded the production Xsan to … macOS Sierra. Yes, macOS 10.12.6 is stable and it’s a good time to install last year’s macOS release. Time to say good bye to macOS el Capitan 10.11.6, we hardly knew ya. Besides guaranteed security updates, stability and the annoying newness of a changed macOS, what else is there? In Xsan v5 they introduced a new “ignore permissions” checkbox for your Xsan volumes. Looking forward to that feature in production. No more Munki onDemand nopkg scripts to run chmod. No more tech support requests for folders, files, FCP X projects that won’t open because someone else used it, owns it, touched it. We’ll see how that pans out. I’ll let you know.

Upgrading Xsan to v5

Step 1. Back up your data

You’re doing this, right? I’m using Archiware P5 Backup to backup the current projects to LTO tape. I’m using Archiware P5 sync to sync the current Xsan volumes to Thunderbolt RAIDs, and using Archiware P5 Archive (and Archive app) to archive completed projects to the LTO project archive. That’s all I need to do, right?

Step 2. Back up your servers

Don’t forget the servers running your SAN! I use Apple’s Time Machine to backup my Mac Mini Xsan controllers. External USB3 drive. I also use another Mac Mini in target disk mode with Carbon Copy Cloner to clone the server nightly. (Hat tip to Alex Narvey, a real Canadian hero). And of course I grab the Xsan config with hdiutil and all the logs with cvgather. Because, why not?! For Archiware P5 backup server I also have a python scripts to backup everything, another scripts to export a readable list of tapes, and BackupMinder to rotate the backups. Add some rsync scripts and you’re golden.

 

Step 3. Upgrade the OS

Unmount the Xsan volume on your clients or shut them down, disconnect the fibre channel. Do something like that. Stop your volume. Download the macOS Sierra installer from the App Store. Double click upgrade. Wait. Or use Munki. I loaded in the macOS 10.12.6 installer app into Munki and set it up as an optional install to make this portion of the upgrade much quicker and cleaner.

In my case after the OS was upgraded I checked the App Store app for any Apple updates (you can also use Munki’s Managed Software Center to check) and of course there were some security updates. In this case the security upgrade hung on a slow network connection and the server crashed. Server down! I had to restore from Time Machine backup to the point where I just upgraded the server. It took some extra time  but it worked (can’t wait for next year’s mature APFS / Time Machine and restoring from snapshots instead).

Step 4. Upgrade Server

After macOS is upgraded you’ll need to upgrade the Server.app or just upgrade the services used by Server (even those not used by Server get upgraded).

Step 5. Upgrade the Xsan

Bur first we have to restore the Xsan config. Don’t panic! It may invoke bad memories of data loss and restoring from backups. Xsan PTSD is real.

Restore-previous-Xsan.png

Step 6. Upgrade the rest

Next you have to upgrade the Xsan volumes.

Xsan-volume-needs-upgrade

New version of Xsan, ch-ch-changes! Ignore permissions check box will remount the xsan with the “no-owners” flag. Let’s test this out.

 

Upgrade the OS and Server app on the backup controller. Upgrade the OS on the clients using Munki or App Store if you like doing it the hard way. Ha Ha.

Step 7. Enjoy

Plug those Thunderbolt to Fibre adapters back in, mount those Xsan volumes and be happy.

Step 8. Wait for the complaints

The next day the editors walked in and went straight to work with Final Cut Pro X. No one noticed anything. Xsan upgraded. Workstation macOS upgraded. Everything appeared to be the same and just worked. Thankless task but well worth it.

 

Reference: Apple’s iBook guide here