Trying to solve the problem of finding production media files across many storage platforms.

In media production environments you work with high speed and high capacity storage. It can be network based NAS, fibre channel SAN or Thunderbolt DAS. There’s always some backup RAIDs, individual source footage drives, file servers and even network appliances.

Glossary of Storage:

• SAN – Storage Area Network (typically with Fibre Channel, also with Thunderbolt or iSCSI)
• NAS – Network Attached Storage (popular vendors include Synology and QNAP)
• DAS – Direct Attached Storage (hardware or software raid directly attached to a client or server)
• LTO – Linear Tape Open (tape standard for backups/archives. Current gen8 holds 12TB per tape)
• Cloud – Other people’s servers and storage. Hosted in a data center (AWS, GCP, Azure, & more)

The variety of file types is astounding: original camera footage, Final Cut Pro projects, stock footage purchased, b-roll shot from other projects, sound effects, music etc. How do you find anything? Files can be in many places, across many different kinds of storage. The question is how do you know where they are?

In the before times…

Before the great pandemic of 2020 I only heard one complaint with finding files on Apple Xsan storage: “Why does spotlight not work on my Mac?” Searching the Xsan volumes was hit and miss. When Spotlight worked it was fast and immediate. And when it didn’t, well…. Not so much. To help finding files we started using one called EasyFind from DevonTechnologies. It was free, and easy to use, but it was not fast enough for ad hoc video searches. Nor could it search across all storage areas at once. A new year, and new solutions required. 

One major issue with EasyFind was not being able to refine the search easily for audio and video assets. It seems skewed toward developers and while it allows you to include or exclude some file types globally in settings but it does not allow you to refine the results while searching. This makes it difficult to find what you need when you need it.

It’s 2021 and a few things have changed over the last year: you may have a lot more places to search (SAN and NAS) and more importantly everyone is working remotely. It has become a lot more challenging to see your files, let alone search through all the storage locations and find what you need. FoxTrot Search to the rescue.

Editors and other creatives need something that will search through all their different storage places and quickly tell them where a certain file might be. It might be stock footage, a drone shot, sound effect or an old logo. But where exactly is it? What projects was it used in? What was it called?

For those who like organizing their files into logical folders, then the filesystem is your friend. Using The Finder has been the way to find things. But now there’s a lot of folders and a lot of file types in those folders. Choosing what to index in these folders help enormously. You can exclude certain subfolders on a designated storage or kinds of files that are not needed. Do not index mail or chat messages or even source code files if you know the files you want are movies and audio files. Narrow the scope where you can. Find files fast.

You might think I’d suggest a full enterprise Media Asset Management (MAM) system at this point. And well, in the old days I would have. Apple’s old Final Cut Server worked perfectly well with classic Final Cut Pro (v6 and v7) but required rigid workflows and ingest habits. Same issue with a lot of more expensive bespoke enterprise systems. They catalog assets but at a great cost. Some clients avoid these for the cost, and others for the workflow restrictions. I’ll talk about some newer options later but for now editors just want to find the files in their own folder structure on their production storage. Is that too much to ask?

Using FoxTrot Search Pro and editor can easily search across multiple indicies (each index is a unique storage or separate folder location) than an admin has already set up and created. No waiting for indexing in real time, search now, with results instantly show in app. Narrow down and refine your search results easily. Don’t want to see Mail messages, or images, only audio files? Easy. Then further define only the file type you want. It works.

One of the recommendations to my clients with large storage (SAN or NAS) is to have secondary nearline archive as well as backups. Either a Thunderbolt RAID array like the Accusys Gamma Carry or a desktop QNAP or rack mounted Synology NAS. Using Synology is great for many reasons, for example, it includes a nice web login, and it can search for files. But to be completely honest, the built-in search only works sometimes. A common issue with search on the Synology is the corruption of the search index. It seems to happen some times for some storage locations, but all the time somewhere. Rebuilding this does not seem to fix it for long if at all. This was the major reason for switching to FoxTrot Search. I want something that works every time. Across all storage volumes: SAN, and NAS.

To be fair, I had to rebuild an index or two in FoxTrot Search as well, but it worked. Every. Single. Time. Rebuilding the Synology search indexes never seemed to fix the issues. With FoxTrot Search it’d warn me there was a lot (A LOT!) of items to index and it will take time but that’s why I’d do it over an evening and rebuild automatically them after hours or every weekend. FoxTrot Search would also warn me of problem files that took too long to index and could then be added to exclusion lists automatically.

When Spotlight stops working, your Synology search index is corrupted again or just want a great search tool for active storage locations, I recommend FoxTrot Search. It searches across all storage and provides access to the files where possible. And it’s fast. Really fast. Did I mention that yet?

One word about the setup and the various FoxTrot Search apps. They have a personal edition for searching your local workstation and storage. The major difference with the Professional version is having multiple indices. They also have a server and a per seat license. I tested the Pro version with a few indices first and then set up the server to share these generated search index files with editors. I ended up making more indices after testing. Which so many storage locations I initially tried to make fewer indices but with so many thousands or in some cases millions of files it was better to make an index for each separate storage location (and per specific folder in some cases). It also helps when narrowing down a search, the editor can specify specific locations easily (as well specifying file type, language or file extension).

Once we tested FoxTrot on the local network we needed to make it work with the VPN. And now! The server version of FoxTrot Search allows you to set a fixed port for the server which we could open up on a firewall. In my initial testing I could connect to the server but not to each index of the various storage locations. This was frustrating. So close…. And I will admit here one criticism of FoxTrot Search is the documentation. It doesn’t really exist. They have a user forum and release notes but in my initial setup of the server I didn’t understand exactly what was needed. This could be solved with better documentation. I did resolve this issue with a few emails to the developers who explained to me that each index required its own port and therefore I needed to open more ports on the firewall. Ok, good to know. To confirm that these ports were or were not accessible in my testing I used “nc” in Terminal to scan open ports.

So FoxTrot Search is great for search across all active storage and is super helpful for seeing previews of video and audio files you may to use as an editor, but it can help find so much more. If you keep production documents, spreadsheets, PDFs, text files it will search through all them too. So your script or production run sheet is available to help you find what you need.

LTO (tape archives)

In my “where are my files” graphic at the top I show various storage locations including LTO and the cloud. The completed projects always go to LTO (tape) archives. How do we search those with FoxTrot Search? I use Archiware P5 which has a web server that is very easy to search with and restore any files via web login but to make things more fun why not have FoxTrot Search index the archive inventory? Of course it can. There’s a cli command in P5 to export the inventory of every file archived and this is a searchable tab separated (TSV) file. I’ve spent a lot of time with this because I’ve been working on a separate tool to analyse these inventory files (with sometimes millions of items in them) to see historical patterns and predict future trends. More on this specific tool later.

The Cloud

New to many is the cloud. Proxies for editors in my recommended workflows get stored on a cloud drive like Postlab, FoxTrot Search can search that too. So many places for originals or backup copies to be. FoxTrot Search should be indexing all of them for you.

In a future blog post I will discuss new media asset management (MAM) systems and what’s changed over the years, but for now if you need a tool search across all active storage then take a look at FoxTrot Search.