The backup was “running.” It had been running for nine hours.

A client’s nightly Kopia job pointed at a user’s documents folder. Normally it finished in twenty minutes. This morning it was still chewing: no error, no progress, just a cursor blinking next to a snapshot that had hashed maybe a few hundred files and then gone silent.

No CPU. No disk I/O to speak of. No network throughput. Just a process sitting there, perfectly calm, doing absolutely nothing while claiming to be busy.

That’s the worst kind of hang. A crash leaves a body. This left a process that would happily wait until the heat death of the universe.

The investigation

First instinct: blame the repository. Maybe the backend was wedged, maybe a lock was stuck. Nope. A fresh kopia snapshot create against a different folder ran clean in seconds.

So it wasn’t Kopia. It was this folder.

I watched the process with Resource Monitor and caught it: every time the scan touched a particular file, the thread parked in a wait state. Not reading. Not erroring. Waiting on the filesystem to hand back bytes that never came.

Then I actually looked at the folder.

┌────────────────────────────────────────────────┐
│  C:\Users\user\Documents  (OneDrive-synced)     │
├──────────────┬─────────────────────────────────┤
│  Q1-report   │  ✓  green check  (on this PC)    │
│  budget.xlsx │  ✓  green check  (on this PC)    │
│  archive\    │  ☁  cloud icon   (online-only)   │ ◄── placeholder
│  photos\     │  ☁  cloud icon   (online-only)   │ ◄── placeholder
└──────────────┴─────────────────────────────────┘
        the cloud-icon files have no bytes on disk

Half the tree had little cloud icons instead of green checks. OneDrive Files On-Demand. Those files looked real — name, size, timestamp — but the actual content lived in the cloud. On disk they were placeholders.

The “aha”

OneDrive’s on-demand placeholders are implemented with the Windows cloud files filter driver, cldflt. When any process opens a placeholder and tries to read it, cldflt intercepts the read and hydrates the file — downloads it from the cloud, transparently, before returning a single byte.

A backup tool is the most naive reader on earth. It opens every file and reads it end to end. So Kopia walks into a folder full of placeholders and triggers a hydration storm: thousands of files getting pulled down on demand, throttled, retried, and in some cases just… stalling. The read call blocks. The thread parks. Forever.

The backup wasn’t broken. It was politely waiting for the entire cloud to fall back onto the disk, one file at a time, through a sync client that had no idea it was under attack.

The fix

You don’t want to read the live folder. You want to read a snapshot of it — a frozen, point-in-time view that the cloud filter doesn’t get to intercept. That’s exactly what a Volume Shadow Copy gives you. The VSS snapshot presents the volume’s state as committed bytes, with no on-demand hydration path.

Kopia has hooks for precisely this: run an action before it walks the source root, and another after. Create the shadow before, tear it down after.

# before-snapshot: create a shadow copy and expose it via symlink
$vol = (vssadmin create shadow /for=C: | Select-String 'Shadow Copy Volume Name').ToString().Split(' ')[-1]
$env:KOPIA_SNAPSHOT_PATH = $vol
cmd /c "mklink /d C:\kopia-shadow $vol\"

# after-snapshot: clean up the link and drop the shadow
cmd /c "rmdir C:\kopia-shadow"
vssadmin delete shadows /for=C: /oldest /quiet

Then point the source at the shadow instead of the live path and wire the hooks in:

kopia snapshot create C:\kopia-shadow\Users\user\Documents \
  --before-snapshot-root-action="powershell -File C:\scripts\vss-before.ps1" \
  --after-snapshot-root-action="powershell -File C:\scripts\vss-after.ps1"

The next run finished in eighteen minutes. Every placeholder read off the shadow returned instantly, because the snapshot view doesn’t route through cldflt. No hydration, no stalls, no nine-hour ghost.

Why it happened

OneDrive Files On-Demand exists so users don’t sync 400GB of vacation photos onto a 256GB laptop. Great feature for humans. Terrible surprise for any tool that assumes “the file is on this disk.”

Kopia did nothing wrong. It read files. The trap is that on a cloud-synced volume, “open and read” is a network operation in disguise, gated by a filter driver that will block your thread while it phones home. Backups that walk thousands of placeholders turn into a self-inflicted denial of service.

Snapshotting through VSS sidesteps the whole mess, and as a bonus you get a consistent point-in-time copy instead of backing up a folder that’s mutating under you.

Takeaways

• Cloud on-demand files are placeholders, not files. OneDrive, Dropbox Smart Sync, and friends all use a filter driver (cldflt on Windows) that hydrates on read.
• A naive reader hangs on hydration. Backups, indexers, and AV scanners that open every file will trigger downloads — and stall when the cloud throttles or times out.
• Back up from a VSS shadow, not the live tree. The snapshot view bypasses the cloud filter and returns committed bytes instantly.
• Use Kopia’s before/after root-action hooks to create and tear down the shadow around each run — no manual babysitting.
• A backup at “0% for hours” with no I/O isn’t slow, it’s blocked. Watch the thread state; a parked read on a cloud volume is the tell.

The ticket was boring. Cleanup. A client wanted some stale files gone from their primary share. I deleted them. Coffee. Done.

Twenty minutes later the same nine files vanished from the backup node too.

That’s the part that made my stomach drop. The backup node existed precisely so that “oops, deleted it on the primary” was a non-event. It was supposed to be the safety net. Instead it watched me cut the trapeze and then helpfully cut its own rope to match.

The investigation

First instinct: ransomware, a rogue cron, a fat-fingered teammate. All wrong. The truth was dumber and worse.

I’d inherited this setup and never read the sync config closely. I assumed “backup node” meant one-way. It didn’t.

grep -i 'mode\|direction\|propagat' /etc/syncthing/config.xml

<folder id="primary-share" type="sendreceive" ...>

sendreceive. Bidirectional. The backup node wasn’t a backup at all — it was a mirror. And a mirror does exactly one job: make the other side look like this side. Including the absences.

I deleted nine files on the primary. The sync engine saw nine deletions, decided they were authoritative changes, and faithfully replicated them downstream. No malice. No bug. It did exactly what I configured it to do.

┌──────────────┐   delete 9 files    ┌──────────────┐
│   PRIMARY    │ ──────────────────► │  "BACKUP"    │
│  (source)    │   sendreceive       │  node        │
└──────────────┘   propagates ✗      └──────────────┘
       │                                    │
       ▼                                    ▼
   files gone                          files gone too
                                       ▓ safety net: GONE

The “aha” wasn’t a clever diagnostic. It was reading one attribute and feeling like an idiot. Sync is not backup. A sync engine has no concept of “retain this.” Its entire worldview is converge. Deletion is just another change to converge on.

The fix

Recover first, redesign second. The deleted files came back from a daily snapshot that — by luck, not design — lived off the sync path. Then I changed the node from a mirror into an actual backup.

# 1) flip the backup node to PULL-ONLY: it receives, never pushes
#    Syncthing: set this folder to "Receive Only"
#    config.xml -> type="receiveonly"

# 2) route deletions to a versioned trash instead of hard-deleting
#    <versioning type="trashcan"> with a retention window
#    (deletes land in .stversions/, not the void)

# 3) shorten the sync interval to bound the blast radius
#    rescanIntervalS: 3600 -> 300   # 5 min, not 1 hour

# 4) confirm the daily snapshot job runs OUTSIDE the synced tree
systemctl list-timers | grep snapshot
ls -la /var/snapshots/   # snapshots here, NOT under the synced folder

After the flip, the verification was the satisfying part. Delete a test file on the primary, watch the backup node not care:

# on primary
touch /srv/share/CANARY.txt
rm  /srv/share/CANARY.txt

# on backup node, a few minutes later
ls /srv/backup/share/CANARY.txt
# CANARY.txt is still there ✓  (delete did not propagate)

Receive-only means upstream deletions are treated as conflicts to ignore, not commands to obey. The node now keeps things the primary throws away. That’s the whole point of a backup.

Why it happened

Because “sync” and “backup” use the same verbs — copy, mirror, replicate — people treat them as the same noun. They aren’t.

Sync optimizes for agreement: every node looks identical, fast. Backup optimizes for survival: keep a copy even when the source loses it. Those goals conflict the instant you delete something. A mirror resolves the conflict by deleting too. A backup resolves it by holding on.

I’d labeled a mirror “backup” and trusted the label. The config said sendreceive the whole time. I just never looked.

Takeaways

• Sync is not backup. Bidirectional sync replicates deletions perfectly — that’s a feature, and it will eat your data.
• A backup node must be one-way. Receive-only / pull-only: it accepts changes but never pushes or propagates deletes upstream-to-down.
• Route deletions to trash, not the void. A versioned trashcan with retention turns “gone forever” into “gone for now.”
• Keep snapshots off the sync path. If your point-in-time recovery lives inside the synced tree, the sync can delete your recovery too.
• Read the config before you trust the label. grep the one attribute that defines behavior — sendreceive vs receiveonly is the whole story.

The dashboard was green. That’s the part that still bugs me.

A customer calls because a workstation died and they need yesterday’s file back. Easy day. I pull up the backup server, find the machine, and go to restore. Except there’s nothing to restore. Last successful backup: never. Not “stale.” Not “failed.” Never.

But the agent was online. Heartbeat OK. Service running OK. Sitting there reporting in like a good little soldier. It just hadn’t backed up a damn thing the entire time it had been installed.

The investigation

First instinct: blame the agent. So I remoted in and read the client logs. The agent was happily completing runs. No errors. It thought it was doing its job.

So now I’ve got an agent that says “I backed up” and a server that says “this machine has never backed up.” Both swear they’re telling the truth. When two systems disagree about reality, the disagreement is almost always about identity.

I checked what name the agent was registered under on the server. Then I checked the machine’s actual OS hostname.

They didn’t match.

┌──────────────────────────┐        ┌──────────────────────────┐
│      THE WORKSTATION      │        │      BACKUP SERVER        │
│                           │        │                           │
│  hostname:  WS-FRONTDESK  │        │  client:  frontdesk-pc    │
│  agent runs OK            │        │  keyed by name ───┐       │
│  "backup complete" OK     │        │                   ▼       │
└───────────┬──────────────┘        │   ┌───────────────────┐   │
            │  reports as            │   │ frontdesk-pc      │   │
            │  WS-FRONTDESK ─────────┼──►│ (PHANTOM CLIENT)  │   │
            │                        │   │ last backup: none │   │
            ▼                        │   └───────────────────┘   │
   bytes go ►►► nowhere              │   WS-FRONTDESK: not found │
└──────────────────────────┘        └──────────────────────────┘

The agent identified itself by the machine’s real hostname. The server had the client filed under a different, hand-typed name from install day. So the agent checked in as WS-FRONTDESK, the server looked for a client called WS-FRONTDESK, found nobody, and the actual data went into the void. Meanwhile the frontdesk-pc entry someone created manually just sat there forever, eternally “waiting for first backup,” looking close enough to green that nobody clocked it.

The “aha”

Backup identity is the hostname. Not a label you pick. Not a friendly name. The hostname.

The server keys everything — auth, storage path, history — off the client name. If that name doesn’t equal what the OS reports as hostname, the agent and server are two ships passing in the night, both convinced they’re docked.

The fix

Stop guessing. Ask the machine its own name, on the box itself:

# On the workstation — what does the OS actually call itself?
hostname

# Verify what the agent will report
[System.Net.Dns]::GetHostName()

That gave me WS-FRONTDESK. The server had frontdesk-pc. So I deleted the phantom client, re-registered the machine under its real hostname, and pushed the server’s preconfigured installer — the one that bakes the correct identity in instead of trusting a human to type it right:

# Remove the dead phantom entry, register the machine by its true hostname,
# then build the preconfigured client installer the server signs.
backup-admin client remove   --name "frontdesk-pc"
backup-admin client add      --name "WS-FRONTDESK"
backup-admin installer build --client "WS-FRONTDESK" --out ./install-WS-FRONTDESK.exe

Ran the signed installer on the box, kicked off a manual run, and watched the server. First full backup completed. Restore point exists. The phantom is gone.

(The other valid fix is to rename the machine to match the registered name — but renaming a production workstation is a bigger blast radius than fixing a backup entry. Match the agent to the host, not the host to a typo.)

Why it happened

Someone registered the client by hand and typed a “nicer” name than the ugly auto-generated hostname. The installer was generic, so identity was never enforced — it trusted the typed name. Two sources of truth, no reconciliation, and a monitoring view that treated “client exists and pings” as “client is protected.” Existence isn’t protection. A backup you can’t restore is just a process eating CPU.

Takeaways

• Backup identity is the hostname. The registered client name must equal hostname on the machine — verify it, don’t assume it.
• A green agent proves nothing. Online does not mean backing up. The only real green is a recent, restorable recovery point.
• Use the server’s preconfigured installer. Hand-typed client names are how phantom clients are born. Bake identity in.
• Alert on silence. Add a “no successful backup in N hours” check so an empty backup history screams instead of sitting there looking calm.
• When two systems disagree about reality, suspect identity first. Auth, naming, and keys cause more “impossible” failures than disk or network ever will.

A client kept their notes vault in git. Years of markdown. Meeting notes, runbooks, half-finished ideas — the kind of stuff you don’t think about until it’s gone.

They wanted to stop tracking one big folder. Reasonable. It didn’t belong in version control.

So they ran the obvious command:

git rm -r attachments/

Git chewed on it for a second and printed a wall of rm 'attachments/...' lines. Looked fine. Then git status showed something that made my stomach drop.

It wasn’t just the attachments/ folder.

deleted:    notes/2024/standup.md
deleted:    notes/2024/oncall.md
deleted:    runbooks/dns.md
... 30,206 more ...

The working tree was empty. ls came back nearly bare. 30,209 markdown files were gone from disk. Not staged-for-deletion-someday. Gone. Off the filesystem.

The investigation

The panic version of this story ends with a restore from last night’s backup and a lost afternoon. The calm version starts with one question: where does git actually delete from?

People think git rm means “stop tracking this file.” It doesn’t. git rm removes a file from two places at once: the index (staging area) and the working tree — your actual disk.

┌──────────────┐   ┌──────────────┐   ┌──────────────┐
│ Working Tree │   │    Index     │   │   HEAD/Repo  │
│  (your disk) │   │  (staging)   │   │ (last commit)│
└──────┬───────┘   └──────┬───────┘   └──────┬───────┘
       │                  │                  │
  git rm ──► ✗ DELETES ──►│ ✗ stages delete  │ ✓ STILL HERE
       │                  │                  │
  git rm --cached ─────►  │ ✗ untracks only  │ ✓ STILL HERE
       ▼                  ▼                  ▼
   file gone          marked rm          recoverable

That last column is the whole ballgame. git rm touches the working tree and the index, but it does not touch your last commit until you actually commit the deletion.

Nobody had committed yet.

The files were sitting safe in HEAD the entire time, even as the disk looked like a crime scene.

The aha

If HEAD still has every file, recovery is just “copy them back out of the last commit.” No backup tape, no remote clone, no heroics.

The fix

First, confirm the damage scope. Trust nothing, count everything:

git status --porcelain | grep -c '^D'
# 30209

30,209 staged deletions. Matches the missing files exactly. Good — nothing weird, just one bad command.

Now restore the working tree and reset the index straight from HEAD:

git restore --staged --worktree .
# or, on older git:
git checkout HEAD -- .

Verify the files actually came back on disk before believing it:

find . -name '*.md' | wc -l
# 30209

All present. git status clean. If HEAD had been corrupt, the same files were still in the remote — git fetch && git restore --source=origin/main --worktree . would have done it from there.

Then we did the thing they meant to do — untrack without deleting:

git rm -r --cached attachments/
echo 'attachments/' >> .gitignore
git commit -m "Stop tracking attachments/ (keep on disk)"

--cached removes the folder from the index only. The disk is never touched. That one flag is the entire difference between a config tidy-up and a disaster.

Why it happened

git rm carries a default that doesn’t match most people’s mental model. The intent was “git, please forget this exists.” Git heard “delete this from the index and the disk,” because that’s what it does. The -r made it recursive, and the glob pulled in far more than expected.

There was no malice and no bad command syntax. It did exactly what it’s documented to do. The gap was understanding that the working tree is a target, not a bystander.

Takeaways

• git rm deletes from your disk. Use git rm --cached to untrack a file while keeping it on the filesystem — that’s almost always what you actually want.
• Count before, count after. find . -name '*.md' | wc -l on both sides of any bulk git op. A number that drops by 30,209 is not a surprise you want at commit time.
• Test on ONE file first. Run the destructive command against a single path, check git status and ls, then scale up. One file lost is a shrug; thirty thousand is a bad day.
• Don’t run destructive git in a live data directory without a backup. A notes vault is data, not just a repo. Treat it like one.
• Know your three trees. Working tree, index, HEAD. Recovery is trivial when you know which one still has your files — and panic when you don’t.

If ransomware hits or files get deleted, they sync that too—straight into the void.

☁️ Backup Rule 101: Always keep at least one offline or immutable copy of your data.

💾 True backup = a separate system that keeps previous versions safe from sync errors or hacks.

#TechTips #BackupStrategy #CyberSecurity #DataProtection