HuskHoard Active Archive for Dummies: Building an "Infinite" File Server

Have you ever wished your computer had a hard drive that simply never filled up? What if you could store 100 Terabytes of video files, backups, and documents, but only pay for a 2 Terabyte high-speed NVMe drive?

Welcome to the world of the Active Archive.

In traditional storage, a file is either "alive" on your fast server or "dead" in cold storage (like a dusty USB drive or an offsite cloud backup). An Active Archive (powered by HuskHoard) bridges this gap. It creates the illusion of infinite, instantly accessible high-speed storage while quietly doing the heavy lifting behind the scenes.

Here is the step-by-step guide to exactly how this magic works.

Step 1: The "Hot Tier" (Where the Magic Starts)

In a typical setup, HuskHoard runs as a standard file server. Users in your office or home connect to it over the network (usually via SMB/Samba) just like any other shared folder.

We call this shared folder the Hot Tier. It usually lives on a lightning-fast NVMe SSD. When a user creates, copies, or saves a file to this network share, it saves directly to this ultra-fast drive. To the user, everything feels completely normal and incredibly fast.

Step 2: The Invisible Watcher (fanotify)

Behind the scenes, the HuskHoard daemon acts as an invisible security guard known as the Interceptor. It uses a Linux kernel feature called fanotify to watch everything that happens in the Hot Tier.

When a user saves or modifies a file, the Interceptor notices. It notes the exact time the file was last touched and logs it into a lightweight tracking database. It does this instantly, with zero slowdown to the user. Every time a file is accessed or modified, its "last touched" timer resets.

Step 3: The Janitor and the Age Limit

Let’s say you finish a video project. You stop opening the files, and they sit untouched on the fast NVMe drive.

Enter The Janitor. The Janitor is a scheduled background worker that constantly checks the tracking database. It is looking for files that have exceeded their configured "Age Limit" (for example, max_age_days = 30).

If a file hasn't been touched in 30 days, the Janitor decides it’s taking up precious high-speed space for no reason and queues it up for cold storage.

(Note: You can also configure "Immediate Archive" rules. If you drop a massive .iso or .zip file into the server, the Janitor can instantly archive it without waiting 30 days!)

Step 4: The Magic Trick (File Stubbing)

When the Janitor archives a file, it copies the data to your deep storage—this could be a physical LTO Tape drive, a massive cheap Hard Drive, or even Cloud Storage via Rclone.

But here is where the "Active" part of the Active Archive kicks in. HuskHoard does not delete the file from your Hot Tier. Instead, it performs a magic trick called Stubbing (or "Hole Punching").

The system hollows out the inside of the file on the SSD, freeing up 100% of the disk space, but it leaves the "shell" or "stub" of the file perfectly intact.

  • If a user opens the folder, the file icon is still there.
  • If an application asks how big the file is, the system lies and reports the original massive size.
  • The file permissions, creation dates, and metadata remain identical.

To everyone and everything, the file is still there. But it’s currently consuming zero bytes on your fast SSD.

Step 5: The Recall (Bringing Files Back to Life)

What happens if, six months later, a user double-clicks that old, hollowed-out stub?

The Interceptor is still watching! The millisecond the user attempts to open the file, the Interceptor pauses the user's application, quickly locates the data on the Tape, Disk, or Cloud, and streams it back into the hollow shell on the SSD.

Once the file is filled back up, the Interceptor unpauses the application. The file opens normally. The user might notice a slight delay (a few seconds to fetch from the cloud, or up to a minute if a robotic tape drive needs to spin up), but it happens automatically. No IT support tickets required. The 30-day countdown timer starts all over again.

(Smart Bypass:* If a user’s app opens an old file just to completely overwrite it, the Interceptor is smart enough to skip the recall process entirely, saving time and bandwidth!)*

Step 6: Taking Out the Trash (Deletions & Garbage Collection)

What happens when a user presses "Delete" on a file?

  1. Soft Deletes: The file (or stub) is instantly deleted from the fast Hot Tier SSD. However, the data remains safe on your cold storage.
  2. The Prune Cycle: HuskHoard runs a regular reconciliation scan. It realizes the file is missing from the SSD and flags it in the database with a deleted_at timestamp.
  3. Retention Period: The file enters a purgatory phase based on your retention_days setting (e.g., 30 days). This is the ultimate "Oops!" protection.
  4. Permanent Purge: Once the 30 days pass, the system permanently removes the file's record from the database.
  5. Garbage Collection (Repacking): Eventually, your cold storage tapes or disks will be full of these deleted "ghost" files. You can run a "Repack" command, which acts like a defragmenter. It copies only the active files to a new tape, permanently reclaiming the space used by the dead files.

Superpowers: Other Built-in HuskHoard Features

Because HuskHoard was built to handle massive enterprise data, it comes packed with a few extra superpowers out of the box:

1. N-Way Replication (Never Lose a File)

When the Janitor archives a file, it doesn't have to put it in just one place. You can configure HuskHoard to write to multiple places simultaneously. For instance, it can write one copy to a local external Hard Drive (Primary), and instantly write a second copy to an Amazon S3 Cloud Bucket (Replication). If your office burns down, your data is already safe in the cloud.

2. Time-Machine Versioning

Ransomware encrypts your files? An editor accidentally saves over a masterpiece? No problem. Every time a file is modified and re-archived, HuskHoard saves it as a new Version. By default, it keeps the 3 most recent versions of every file safely locked away in cold storage. You can manually roll back to any historic version at any time.

3. Emergency Spillover

Sometimes, users dump way too much data onto the server at once, threatening to fill up the fast NVMe drive completely. HuskHoard has a "High-Water Mark" setting (e.g., 80%). If the Hot Tier hits 80% capacity, the Janitor drops the 30-day wait limit and goes into overdrive, aggressively hollowing out the oldest files to guarantee the SSD never crashes from being 100% full.

4. Direct HTTP Streaming (StreamGate)

If you want to watch a massive archived movie via VLC or Plex, recalling a 50GB file back to the SSD might be a waste of time and space. HuskHoard includes a built-in "StreamGate" gateway. It can intercept web traffic and stream video files directly from a Tape or Cloud straight to the media player on your TV, bypassing the SSD entirely!

5. Bit-Rot Protection (The Scrubber)

Data degrades over time. HuskHoard generates a unique cryptographic fingerprint (BLAKE3) for every file it archives. You can run the "Scrubber" command anytime, which will quietly read through your cold storage drives and check every file against its fingerprint. If a hard drive starts failing and flipping bits, HuskHoard will catch it long before you try to open the file.

Conclusion

An Active Archive takes the stress out of storage management. By pairing a small, affordable, blazing-fast SSD with deep, cheap, massive cold storage, HuskHoard gives you the best of both worlds. The users get speed and the illusion of infinite space, while you get cost savings and enterprise-grade data protection.