[{"content":"ErrorZap is where I write down the technical stories that are too useful, strange, expensive, or funny to leave trapped in a terminal scrollback.\nI work in managed IT, infrastructure, security, automation, backups, networks, and the untidy space where a clean diagram meets a real building full of old wiring. The posts here are field notes, not recycled product copy. Details are generalized when they could identify a client or expose a system, but the failure modes and lessons stay real.\nYou will also find the 90s Kid collection, because the people who survived IRQ conflicts, dial-up noise, rewinding VHS tapes, and printers that sounded like farm machinery deserve their own archive.\nWhat you can expect The problem, including what made it misleading. The evidence that separated the real cause from the tempting guess. The fix, the boundary, or the lesson worth keeping. Enough technical detail to be useful without publishing somebody\u0026rsquo;s private infrastructure. A willingness to say when the machine was right and I was the problem. ErrorZap uses privacy-first traffic and performance measurement through Cloudflare. It does not use an advertising network or build visitor profiles. See the privacy note for the short version.\nFor managed IT help, hire me.\n","permalink":"https://errorzap.com/about/","summary":"Why ErrorZap exists and what belongs here.","title":"About ErrorZap"},{"content":"ErrorZap measures aggregate traffic and page performance using Cloudflare Web Analytics. Cloudflare describes this product as privacy-first and says its Web Analytics service does not collect or use visitors\u0026rsquo; personal data.\nThere is no advertising network, cross-site visitor profile, newsletter tracking pixel, or comment-account database on ErrorZap.\nThe operational logs needed to securely deliver the site may still include normal request information for a limited period. If the analytics or hosting stack changes, this page will change with it.\nLast updated: July 29, 2026.\n","permalink":"https://errorzap.com/privacy/","summary":"The short, human explanation of what ErrorZap measures.","title":"Privacy"},{"content":"ErrorZap is a field notebook for the incidents worth remembering: the failure that looked impossible, the fix that finally made sense, the warning that should save somebody else an afternoon, and the old technology that somehow trained us for all of it.\nStart with the expensive lessons How My Own Backup Deleted My Files The Firewall API Call That Wipes Your Router The Settings API That Corrupted Every Setting I Found My Own Server on Shodan Then try the weird ones Seven Network Gotchas That Look Like Ghosts The Smart Blind That Took Down the Network The PWA That Froze When Offline When Postgres Won\u0026rsquo;t Start Or go back to the 90s The 90s Kid collection is about floppy disks, pagers, dial-up, VCR clocks, flying toasters, and learning technical patience before anybody called it troubleshooting.\nFind something specific Search every post, browse the complete archive, or subscribe using the RSS feed.\n","permalink":"https://errorzap.com/start/","summary":"The fastest way into ErrorZap: real failures, useful recoveries, and 90s technology.","title":"Start Here"},{"content":"I\u0026rsquo;m Justin. I\u0026rsquo;ve spent 25+ years fixing infrastructure, based in Pueblo, Colorado. The incidents on this blog aren\u0026rsquo;t hypotheticals — they\u0026rsquo;re from my real client work.\nWhat I do remotely Managed IT and monitoring Patch management Ransomware-proof backups (append-only, immutable storage) Camera and alarm system health monitoring Microsoft 365 / Google Workspace management and security audits VoIP phone systems Help desk Pricing No games, no \u0026ldquo;call for a quote\u0026rdquo; runaround:\nService Price Workstation monitoring \u0026amp; management $35/device/mo Server monitoring \u0026amp; management $200/mo Remote security exposure audit (flat) $495 Hourly work $150/hr Contact Phone: 719-530-4040 Email: service@kccsonline.com Form: callthenerd.com See more of what I run at modernsystemspro.com and pueblosecure.com.\n","permalink":"https://errorzap.com/hire/","summary":"Remote managed IT, monitoring, and ransomware-proof backups for small businesses. The incidents on this blog are from my real client work.","title":"Hire Me"},{"content":"A client\u0026rsquo;s surveillance system had stopped recording. Seventy-five cameras, a 44-terabyte recorder, and a banner on the dashboard reading \u0026ldquo;Recovering storage.\u0026rdquo; Internally the system had flipped itself into a read-only \u0026ldquo;limited mode\u0026rdquo; and quietly stopped saving video. By the time we dug in, it had been down for two and a half days.\nThe obvious diagnosis — the one most techs would reach for, and the one the symptoms practically beg you to believe — is a disk is dying, replace it.\nThat diagnosis would have made everything dramatically worse. Here\u0026rsquo;s why, and what was actually going on.\nOK ?! OK BAD BLOCK A phantom that was never there — scribbled onto disks that were perfectly fine. The symptom vs. the disease The logs were screaming the same error, over and over, hundreds of times:\nEXT4-fs warning (device md3): I/O error writing to inode 339687425, block 357888066 Buffer I/O error on device md3, logical block 357888066 Same block. Every single time. The filesystem wanted to write to one specific spot on the array and the write kept failing.\nWhen a write to a RAID array fails on the same block forever, instinct says bad sector, failing drive. So the first thing we did was pull the health stats (SMART) off all seven drives.\nEvery counter that would indicate a failing disk — reallocated sectors, pending sectors, offline-uncorrectable sectors, cable/CRC errors — was zero. On all seven drives. The disks were pristine.\n# smartctl -A /dev/sd[a-g] \u0026nbsp;·\u0026nbsp; the five minutes that cracked the case Reallocated_Sector_Ct \u0026nbsp;....\u0026nbsp; 0 Current_Pending_Sector \u0026nbsp;...\u0026nbsp; 0 Offline_Uncorrectable \u0026nbsp;....\u0026nbsp; 0 UDMA_CRC_Error_Count \u0026nbsp;.....\u0026nbsp; 0 ×7 drives — all zero. Not one of them is dying. Healthy disks. A write that won\u0026rsquo;t complete. Same block forever. That contradiction is the whole story.\nWhat\u0026rsquo;s actually a \u0026ldquo;bad block list\u0026rdquo;? Linux software RAID (the md subsystem, the thing mdadm drives) has a feature called the Bad Block List, or BBL. The idea sounds reasonable: if the array ever has trouble with a specific spot on a member disk, instead of kicking the whole disk out, it just writes that spot down in a little list — \u0026ldquo;don\u0026rsquo;t use this block\u0026rdquo; — and routes around it.\nHelpful in theory. In practice, it has been one of the most controversial features in the Linux RAID stack for about fifteen years. Kernel developers have openly argued for removing it. The reason is exactly what we walked into:\nSomething causes a transient I/O hiccup on a member — a momentary timeout, a controller blip, a drive that stalls for a fraction of a second under heavy write load. Nothing actually wrong with the media. The BBL feature misreads that hiccup as a bad block and writes it down permanently. On RAID5, where every stripe is spread across multiple disks, the bad-block entry gets propagated onto other members. Over time, the same logical region ends up flagged on several disks. We confirmed this precisely. Four of the seven disks carried byte-for-byte identical bad-block ranges — and that is the tell:\nRAID5 · seven disks · 44 TB usable ┌─────┬─────┬─────┬─────┬─────┬─────┬─────┐ │ sda │ sdb │ sdc │ sdd │ sde │ sdf │ sdg │ ├─────┼─────┼─────┼─────┼─────┼─────┼─────┤ │ · │ ▓▓▓ │ ▓▓▓ │ ▓▓▓ │ · │ · │ ▓▓▓ │ │clean│ BBL │ BBL │ BBL │clean│clean│ BBL │ └─────┴─────┴─────┴─────┴─────┴─────┴─────┘ ╰──── identical phantom ranges ────╯ real bad sectors do NOT line up to the exact same address across four separate drives. Why this stops recording entirely This is the part that turns a phantom bookkeeping error into a system outage.\nTo write one stripe of video, RAID5 has to write to several disks at once plus update parity. If even one of those disks says \u0026ldquo;that block is on my no-go list,\u0026rdquo; the entire stripe write fails and returns an I/O error:\nONE stripe write ──► every data member must accept it ┌────┐┌────┐┌────┐┌────┐┌────┐┌────┐ │ D0 ││ D1 ││ D2 ││ D3 ││ D4 ││ P │ └────┘└─🚫─┘└────┘└─🚫─┘└────┘└────┘ │ │ ╰── \u0026#34;on my no-go list\u0026#34; ──╯ │ ▼ ✗ THE WHOLE STRIPE FAILS → EIO → filesystem protectively goes read-only → recorder drops to \u0026#34;limited mode\u0026#34; → 75 cameras stop saving video The filesystem above sees the write fail, decides its metadata is now untrustworthy, and protectively remounts read-only. The camera software sees a read-only volume and drops into limited mode. Recording stops. All of it.\nSo: zero failing hardware, and a total recording outage, caused entirely by a list of imaginary bad spots the software refused to write over.\nThe trap Now the dangerous part. The \u0026ldquo;replace the disk\u0026rdquo; reflex.\nIf you pull one of those drives and let the array rebuild onto a replacement, mdadm re-creates a bad-block list on the new member and re-propagates the same poison. You\u0026rsquo;d do four sequential multi-hour degraded rebuilds, each one re-importing the exact problem you were trying to remove, while running the array in its most fragile state. Replacing disks here doesn\u0026rsquo;t just fail to fix it — it actively risks the array.\nNEW disk ✓ swap it in… rebuild BBL …and the list rebuilds itself. Replace a disk and the array writes the bad-block list right back onto the new one. The ghost just photocopies itself. The correct fix is the opposite of intuition: don\u0026rsquo;t touch the hardware. Erase the lists.\nThe repair mdadm has an escape hatch for exactly this — a flag that clears (and disables) the bad-block feature across every member as the array is assembled:\nmdadm --assemble --update=force-no-bbl /dev/md3 /dev/sd[a-g]5 The careful sequence around that one command:\nBack up everything first. Recovery codes, configuration, and a full pre-incident database snapshot, all pulled off the box. Nothing to lose before touching the array. Quiet the system. Stop every service touching the volume so nothing is mid-write. (This box had seventeen interlocking services and a storage daemon that re-assembles the array on its own.) Unmount and stop the array. Clear the bad-block lists with the command above. Afterward, every member reported the magic words: sda5 → No bad-blocks list configured sdb5 → No bad-blocks list configured sdc5 → No bad-blocks list configured sdd5 → No bad-blocks list configured sde5 → No bad-blocks list configured sdf5 → No bad-blocks list configured sdg5 → No bad-blocks list configured ☁ poof. gone. Check the filesystem. A read-only e2fsck -fn preview came back clean — only two trivial, optional defragmentation suggestions. Zero corruption, zero data to remove. That confirmed the \u0026ldquo;filesystem damage\u0026rdquo; was never real damage; it was just the blocked writes. Reboot and verify. The result: array healthy, zero I/O errors, bad-block lists empty, and within seconds of the services coming back, the full camera fleet was writing video again. Total surgery time: about an hour. Zero permanent data loss.\nThe gotcha worth its own paragraph These appliances don\u0026rsquo;t assemble their array from a config file the way a generic Linux server does. A storage daemon brings the array up at boot. That means the daemon will happily re-assemble and re-mount the array out from under you in the middle of a repair if you don\u0026rsquo;t stop it — and it has to be running again before reboot or the box comes up with no storage at all. Finding and accounting for every one of those interlocking services (seventeen, including media-server processes that respawn instantly when killed) was most of the actual work. The headline command takes one second; safely getting the system into a state where that command is safe to run takes the hour.\nThe kicker Once a system surprises you, the right question is: where else do we have this and not know it yet?\nSo we audited every storage system under management for the same fingerprint. Most weren\u0026rsquo;t even capable of this failure (different storage technology). One was clean. And one other recorder was sitting in the earliest stage of the exact same bug — two phantom bad-block ranges just beginning to form, still recording fine, no errors yet. Caught months before it would have become an outage. We added automated monitoring across the fleet that checks these lists every 30 minutes, so the next occurrence is a 30-minute alert instead of a two-day blackout.\nTakeaways Symptoms lie. Health data doesn\u0026rsquo;t. The single most useful five minutes of this entire incident was reading SMART and seeing all zeros. That one fact ruled out the obvious-but-wrong diagnosis and pointed at the real one. The bad-block list is a footgun. If you run Linux software RAID, know this feature exists, know it can manufacture failures out of transient hiccups, and know --update=force-no-bbl is how you clear it. Consider monitoring for non-empty lists proactively. The intuitive fix can be the destructive one. \u0026ldquo;Replace the disk\u0026rdquo; would have deepened the hole. Sometimes the move is to change nothing physical and correct the software\u0026rsquo;s bad bookkeeping. Always back up before array surgery, even when you\u0026rsquo;re confident. Especially when you\u0026rsquo;re confident. Then go look at everything else. The bug you just fixed is rarely the only instance of it you own. The hardware was never broken. The drives were never dying. A fifteen-year-old software feature, trying to be helpful, wrote down a problem that didn\u0026rsquo;t exist and then refused to work around its own mistake. Once you can see that, the fix is almost anticlimactic. Seeing it is the job.\n~ no bad-blocks list configured ~ ","permalink":"https://errorzap.com/posts/phantom-bad-blocks/","summary":"A field report on diagnosing and repairing a RAID failure that wasn\u0026rsquo;t a hardware failure at all — and why \u0026lsquo;replace the disk\u0026rsquo; would have made it far worse.","title":"The Phantom Bad Blocks: How a 15-Year-Old Linux Bug Took Down 75 Security Cameras"},{"content":" A customer called on a Tuesday. A critical service had been down for hours. Their first question was the one that ruins your week:\n\u0026ldquo;Why didn\u0026rsquo;t we get an alert?\u0026rdquo;\nI had built the alerting. I knew the rules existed. I\u0026rsquo;d watched them fire in testing months ago. So my honest, confident, completely wrong answer was: \u0026ldquo;You should have.\u0026rdquo;\nThat\u0026rsquo;s the trap. \u0026ldquo;Should have\u0026rdquo; is what you say right before you learn something humbling about your own stack.\nThe investigation First instinct: the rule didn\u0026rsquo;t fire. Easy theory, easy to check. So I pulled the alert history.\nThe rule fired. On time. With the right severity, the right labels, the right everything. It did its job flawlessly and then handed the alert off to\u0026hellip; something.\n┌──────────┐ ┌────────────┐ ┌──────────┐ ┌────────┐ │ RULE │──► │ RECEIVER │──► │ CHANNEL │──► │ HUMAN │ │ ✓ fired │ │ ✓ matched │ │ ✗ none │ │ ✗ — │ └──────────┘ └────────────┘ └──────────┘ └────────┘ ▲ │ the chain dies right here The rule routed to a receiver. The receiver had a name, a config block, a place in the tree. It looked alive. And its list of channels — the email addresses, the webhook URLs, the push targets, whatever was supposed to carry the message to a person — was empty.\nNot broken. Not misconfigured. Empty. The alert was delivered, perfectly, to nobody.\nrule → receiver → channel → ? ALERT [ empty ] no recipients ✗ The mail always went out. There was just no mailbox at the end of the route. The \u0026ldquo;aha\u0026rdquo; Here\u0026rsquo;s the part that stung. Everyone — me included — had validated the wrong layer.\nWe tested that rules existed. We tested that rules fired. We never tested that a fired alert reached a phone, an inbox, a pager, a chat channel — a person. The chain has four links; we\u0026rsquo;d verified two and assumed the other two by vibes.\nAn alert is not a notification. A rule firing is not someone finding out. The gap between those two facts is exactly where this outage lived — silent, for months — manufacturing a warm, false confidence that the system \u0026ldquo;had alerting.\u0026rdquo;\nIt did. The alerting just talked to a wall.\nThe fix Audit every receiver. Every single one needs at least one channel, and that channel has to actually reach a human or a device. No empties, no dead webhooks, no stale email lists.\nThen — and this is the part people skip — add a heartbeat. A \u0026ldquo;dead-man\u0026rdquo; alert that is supposed to fire on a schedule. If it stops arriving, your delivery path is broken and you find out on purpose instead of during a customer call.\n# 1. Find every receiver with an empty / missing channel list. amtool config routes show amtool config show | yq \u0026#39;.receivers[] | select( ([.email_configs, .webhook_configs, .pushover_configs, .slack_configs, .opsgenie_configs] | flatten | length) == 0 ) | .name\u0026#39; # 2. End-to-end test: fire a synthetic alert and confirm it lands on a device. amtool alert add deadmans_switch \\ severity=info service=alerting \\ --annotation summary=\u0026#34;heartbeat — if you can read this, delivery works\u0026#34; # 3. The dead-man heartbeat: a rule that is ALWAYS firing on purpose. groups: - name: heartbeat rules: - alert: DeadMansSwitch expr: vector(1) labels: { severity: heartbeat } annotations: summary: \u0026#34;Alerting pipeline is alive. Silence = it isn\u0026#39;t.\u0026#34; DELIVERY CHAIN ── post-fix RULE ─► RECEIVER ─► CHANNEL ─► DEVICE ✓ ✓ ✓ (≥1) ✓ buzzed + DeadMansSwitch heartbeat every 60s ►► if it goes quiet, you know Flip the logic. A normal alert proves a problem exists. A heartbeat proves the path itself still works. You want both, because the path is the thing that fails silently.\nWhy it happened Someone — possibly past me — created the receiver as a placeholder, fully intending to fill in the channels later. Later never came. The config was syntactically valid, so nothing complained. There\u0026rsquo;s no error on an empty channel list; it\u0026rsquo;s a legal, boring state.\nAnd because the rules were visibly there, every review after that pattern-matched \u0026ldquo;alerting: configured ✓\u0026rdquo; and moved on. Nobody walked the whole chain to a buzzing phone. The placeholder calcified into production.\nTakeaways An alert with no destination is worse than no alert. No alerting at least keeps you honest. A dead receiver hands you false confidence and bills you for it during an outage. Validate the entire chain: rule → receiver → channel → device. Verifying any single link tells you almost nothing about the link after it. Test delivery, not configuration. \u0026ldquo;The rule exists\u0026rdquo; and \u0026ldquo;a human got paged\u0026rdquo; are different claims. Only the second one matters at 2 a.m. Add a dead-man heartbeat. A signal that should arrive on schedule turns silent failure into a loud, on-purpose one. Audit receivers for empties on a cadence. Placeholders, decommissioned inboxes, and rotted webhooks accumulate. Grep them out before they grep you. ","permalink":"https://errorzap.com/posts/the-alert-that-went-to-nobody/","summary":"A critical event fired a perfectly good alert into a receiver wired to absolutely no one — and everybody slept fine because of it.","title":"The Alert That Went to Nobody"},{"content":" A client\u0026rsquo;s user control panel popped a browser security wall. Expired certificate. Classic.\nThe panel runs on a FreePBX appliance, on a high port, separate from the main admin UI. Users hit it for voicemail and call history. Now they hit a full-page TLS warning instead.\nNo big deal, I figured. Renew the cert, reload, done before the coffee gets cold.\nThe coffee got cold.\nThe investigation First I confirmed it wasn\u0026rsquo;t the browser lying to me:\nopenssl s_client -connect 192.0.2.10:8003 \u0026lt;/dev/null 2\u0026gt;/dev/null \\ | openssl x509 -noout -dates -subject # notBefore=Apr 1 00:00:00 2024 GMT # notAfter=Apr 1 00:00:00 2025 GMT Dead for over a year. The panel had been quietly serving a corpse the whole time, and nobody noticed until a browser update got strict about it.\nSo I did the obvious things. The UI has a big friendly Apply Config button. Clicked it. Green checkmark. \u0026ldquo;Reload completed successfully.\u0026rdquo;\nRe-ran my openssl check. Same expired dates.\nFine. Bigger hammer. From the CLI:\nfwconsole reload fwconsole restart fwconsole ma updateall Three clean exits. Three \u0026ldquo;success\u0026rdquo; lines. Three pieces of nothing — the panel kept serving the exact same expired cert, byte for byte.\nThat\u0026rsquo;s the moment the job stops being annoying and gets interesting. When every \u0026ldquo;fix\u0026rdquo; reports success and the resource never changes, you\u0026rsquo;re not fixing the resource. You\u0026rsquo;re poking something that doesn\u0026rsquo;t own it.\nThe \u0026ldquo;aha\u0026rdquo; I drew the actual ownership chain — the thing nobody draws until they\u0026rsquo;re an hour in:\n┌──────────────────────────────────────────────────────┐ │ reload / apply-config / updateall │ │ │ │ │ ▼ \u0026#34;rebuild my running config from what exists\u0026#34; │ │ ┌───────────────────┐ │ │ │ User Panel :8003 │ ── binds ──► cert file ✗ │ │ └───────────────────┘ (expired, stale) │ │ ▲ │ │ │ OWNS │ │ ┌───────────────────┐ │ │ │ Certificate Mgr │ │ │ │ (lifecycle here) │ │ │ └───────────────────┘ │ └──────────────────────────────────────────────────────┘ The reload path rebuilds config from whatever certs already exist on disk. It never regenerates one. The thing that actually mints and renews the cert is the Certificate Manager module. Reload was faithfully re-binding the panel to a file that was still expired — and honestly reporting success, because re-binding did succeed. The cert was just garbage.\nNobody was renewing it because nobody told the component that owns renewals to do its job.\nreload / apply config panel :8003 cert expired Certificate Mgr owns lifecycle re-binds stale file regenerate Reload re-binds the panel to a stale file. Only the Certificate Manager can reissue it. The fix Stop driving the reload path. Drive the API of the thing that owns the cert.\nThe Certificate Manager exposes a regeneration call. Invoke it directly, let it reissue, then point the panel at the fresh cert and reload once to bind:\n# Ask the module that OWNS the cert to reissue it fwconsole certificates --regen-self-signed # (or, when an ACME/managed cert is in play) fwconsole certificates --update # Now bind the panel to the freshly minted cert, then reload to apply fwconsole certificates --default=\u0026lt;new_cert_id\u0026gt; fwconsole reload Then verify against the live socket — not the UI\u0026rsquo;s word, the actual TLS handshake:\nopenssl s_client -connect 192.0.2.10:8003 \u0026lt;/dev/null 2\u0026gt;/dev/null \\ | openssl x509 -noout -dates # notBefore=Jun 9 00:00:00 2026 GMT # notAfter=Jun 9 00:00:00 2028 GMT ✓ Fresh dates. Panel loaded clean. Coffee remade.\nWhy it happened \u0026ldquo;Reload\u0026rdquo; and \u0026ldquo;apply config\u0026rdquo; are rebuild operations. They take the current declared state and re-render it. If the declared state references a cert file, they\u0026rsquo;ll happily wire it up — expired or not. Validity isn\u0026rsquo;t their job.\nRenewal lives in exactly one place: the module that issues certs. Every other button just consumes the output. So every reload was a no-op on the resource, dressed up as a success because the binding step really did succeed.\nThe trap is that \u0026ldquo;success\u0026rdquo; was true at the wrong altitude. The command did what it was scoped to do. It just wasn\u0026rsquo;t scoped to the thing I needed.\nTakeaways When every fix reports success but the resource never changes, you\u0026rsquo;re poking the wrong subsystem. Find the component that actually owns the resource\u0026rsquo;s lifecycle and drive its API. \u0026ldquo;reload\u0026rdquo; / \u0026ldquo;apply config\u0026rdquo; usually means re-render, not regenerate. They consume artifacts; they rarely create them. Trust the wire, not the UI. openssl s_client ... | openssl x509 -noout -dates against the live port is ground truth. A green checkmark is a claim. Map ownership before you start hammering buttons. One quick \u0026ldquo;who actually mints this?\u0026rdquo; diagram beats an hour of confident no-ops. Expired certs hide for months. Monitor notAfter on every TLS port — including the weird high-port sub-services nobody remembers exist. ","permalink":"https://errorzap.com/posts/the-cert-only-an-api-could-renew/","summary":"A FreePBX user panel served an expired TLS cert, every reload swore it was fixed, and the only thing that actually moved the needle was calling the cert module\u0026rsquo;s own regeneration API.","title":"The TLS Cert Only an API Could Renew"},{"content":" The pager went off at the worst possible hour, which is the only hour pagers know.\nEvery app talking to the database was throwing the same line:\nFATAL: the database system is starting up Not \u0026ldquo;down.\u0026rdquo; Not \u0026ldquo;connection refused.\u0026rdquo; Starting up. As if it were thirty seconds from being fine. It said that thirty seconds ago too. And thirty seconds before that.\nThe investigation First instinct: restart it harder. Bad instinct. I sat on my hands.\nThe host had taken an ugly storage hiccup earlier — the array blipped, the kernel got cranky, and Postgres went down without a clean shutdown. So I went to the only source of truth that matters here: the server log.\nsudo tail -f /var/lib/pgsql/data/log/postgresql-*.log And there it was, crawling:\nLOG: database system was not properly shut down; automatic recovery in progress LOG: redo starts at 3F/A2000028 LOG: redo in progress, elapsed time: 412.06 s, current LSN: 3F/A2090110 Recovery was happening. It was just happening at the speed of continental drift. In seven minutes of replay it had moved the LSN by a rounding error.\nThat\u0026rsquo;s not a Postgres bug. That\u0026rsquo;s a disk gasping for air.\nThe \u0026ldquo;aha\u0026rdquo; Here\u0026rsquo;s the thing every junior forgets at 3 a.m.: after an unclean stop, Postgres doesn\u0026rsquo;t just \u0026ldquo;boot.\u0026rdquo; It replays its write-ahead log to drag the data files back to a consistent state. Until that replay finishes, the server refuses connections — and the polite way it refuses is the database system is starting up.\nunclean shutdown │ ▼ ┌────────────────────────┐ reads WAL │ PostgreSQL (startup) │ ───────────────┐ └────────────────────────┘ │ │ ▼ │ replays records ┌───────────────┐ │ onto data files │ WAL (pg_wal) │ ▼ └───────┬───────┘ ┌────────────────────────┐ │ │ data files on DISK │ ◄────────────────┘ └───────────┬────────────┘ │ if DISK is slow/failing, ▼ replay crawls → \u0026#34;starting up\u0026#34; forever clients see: FATAL: starting up WAL replay is only as fast as the disk it\u0026rsquo;s writing to. The storage hiccup hadn\u0026rsquo;t killed the disk — it had wounded it. Every read was retrying, every write was waiting. Recovery was real, honest, and effectively never going to end on that hardware.\nThe cardinal sin would\u0026rsquo;ve been to kill -9 the startup process to \u0026ldquo;speed things up.\u0026rdquo; Kill recovery mid-replay and you risk a data directory that\u0026rsquo;s half-applied and fully useless.\nPostgres (startup) WAL redo log DISK slow / failing replay is only as fast as the disk underneath it wounded disk ⇒ recovery crawls ⇒ \"starting up\" forever Recovery wasn't stuck — it was throttled by the storage it was trying to heal. The fix Rule one of database recovery: a database cannot recover on a broken disk. Fix the floor before you ask anyone to dance on it.\nSo I stopped Postgres cleanly, took the storage out of the equation, and verified the hardware first.\n# 1. Stop the instance cleanly — let it park, don\u0026#39;t kill -9 sudo systemctl stop postgresql # 2. Check the storage BEFORE touching the database sudo smartctl -a /dev/sda | grep -iE \u0026#39;reallocated|pending|health\u0026#39; dmesg --ctime | grep -iE \u0026#39;i/o error|ata|reset\u0026#39; # 3. Move the data dir onto healthy storage (rsync, preserve everything) sudo rsync -aHAX --info=progress2 /var/lib/pgsql/data/ /srv/pgdata-healthy/ With the data directory living on a disk that wasn\u0026rsquo;t actively dying, I pointed Postgres at it and let recovery run — and left it alone.\nsudo -u postgres /usr/bin/postgres -D /srv/pgdata-healthy \u0026amp; sudo tail -f /srv/pgdata-healthy/log/postgresql-*.log This time the LSN sprinted instead of crawled:\nLOG: redo in progress, elapsed time: 9.21 s, current LSN: 3F/F1A40020 LOG: redo done at 3F/F1A40020 LOG: database system is ready to accept connections Twelve seconds of replay on good hardware versus an eternity on bad. Same WAL. Same database. Different floor.\nAnd because I never fully trust a data directory that\u0026rsquo;s been through a disk event, the first thing I did once it was up was take a logical backup — the kind that doesn\u0026rsquo;t care about block-level corruption:\npg_dump -Fc -d appdb -f /srv/backups/appdb_$(date +%F).dump Why it happened The unclean shutdown was never the real problem. Postgres handled that exactly as designed — replay the WAL, return to consistency, open the doors.\nThe problem was that recovery was asked to run on storage that couldn\u0026rsquo;t keep up. WAL replay is I/O, and I/O on a wounded disk is mostly waiting. \u0026ldquo;Starting up forever\u0026rdquo; is what a healthy recovery process looks like when it\u0026rsquo;s standing on broken ground.\nFix the ground. The database knows what to do.\nTakeaways \u0026ldquo;Starting up\u0026rdquo; forever almost always means WAL replay — Postgres is recovering from an unclean stop, not hanging for no reason. Watch the server log, not the client errors. redo in progress with a moving LSN means it\u0026rsquo;s working; a stalled LSN means your storage is the suspect. Fix the storage first. A database physically cannot finish recovery on a slow or failing disk — verify SMART/dmesg, move to healthy media, then let it replay. Never kill -9 recovery to \u0026ldquo;speed it up.\u0026rdquo; A half-applied data directory can be worse than no database at all. Keep logical dumps (pg_dump). A physical data directory can\u0026rsquo;t always be salvaged after a disk event — a logical backup is the restore path that survives bad blocks. ","permalink":"https://errorzap.com/posts/when-postgres-wont-start/","summary":"A database stuck forever on \u0026lsquo;starting up\u0026rsquo; wasn\u0026rsquo;t broken — it was clawing its way through WAL replay on a dying disk.","title":"When Postgres Won't Start"},{"content":" The page came in at 2:14 a.m. Then 2:16. Then 2:19.\nBy the time I\u0026rsquo;d wiped the sleep out of my eyes there were forty-one alerts in the channel, all from the same audio detector at a customer site. \u0026ldquo;LOUD EVENT DETECTED.\u0026rdquo; Forty-one of them. My first thought was that someone was kicking the building in.\nNobody was kicking the building. Someone closed a heavy door. Then, hours later, a kitchen dropped a sheet pan on a tile floor. Each one lit up the detector like a fireworks finale.\nThis is the story of a sensor that screamed at everything and meant nothing.\nThe investigation The setup was simple on paper. A mic feeds audio into a small model that\u0026rsquo;s supposed to flag impact-type events — glass breaks, hard slams, the kind of thing you actually want a human to look at. It fires a webhook, and the webhook pages whoever\u0026rsquo;s on call.\nIn practice it fired on anything loud.\nI pulled the trigger log and lined it up against what people swore actually happened that night.\n┌──────────────────────────────────────────────────────────┐ │ TRIGGER LOG vs GROUND TRUTH │ ├──────────────┬───────────────┬───────────────┬───────────┤ │ time │ detector says │ peak level │ reality │ ├──────────────┼───────────────┼───────────────┼───────────┤ │ 02:14:07 │ ► IMPACT │ ▓▓▓▓▓▓▓ 0.91 │ door slam ✗│ │ 02:16:55 │ ► IMPACT │ ▓▓▓▓▓▓ 0.88 │ door slam ✗│ │ 02:19:31 │ ► IMPACT │ ▓▓▓▓▓▓▓ 0.93 │ dropped pan✗│ │ 03:48:12 │ ► IMPACT │ ▓▓▓▓▓ 0.81 │ HVAC bang ✗│ │ (silent) │ —— │ ▓▓ 0.34 │ real break✗│ └──────────────┴───────────────┴───────────────┴───────────┘ every loud thing fired ── the one real event didn\u0026#39;t That last row is the one that made my stomach drop. There had been a genuine event earlier in the week — a real break — and it was quieter than a slammed door. The detector had ignored it.\nThe \u0026ldquo;aha\u0026rdquo; So I read the actual trigger code. And there it was, in one branch:\nif peak_amplitude \u0026gt; THRESHOLD: fire_alert(\u0026#34;IMPACT\u0026#34;) That\u0026rsquo;s it. That\u0026rsquo;s the whole brain.\nThe model attached to this thing was a perfectly good audio classifier. It returned a predicted label and a confidence score for every clip — door, speech, glass_break, hvac, the works. And the trigger logic threw all of it away and looked at one number: was it loud?\nLoudness is not meaning. A slammed door is loud and boring. A real break can be quiet and important. We\u0026rsquo;d built a smoke detector that goes off when you turn the lights on.\nloudness ≠ meaning AMPLITUDE ONLY fires on everything ✗ LABEL + CONFIDENCE ✓ fires on the right one ✓ Same five sounds. Volume flags all of them; label + confidence flags the one that matters. The fix The model already knew what it was hearing. We just had to listen to the label instead of the volume — then gate on confidence and stop it from machine-gunning the same event.\nALERT_LABELS = {\u0026#34;glass_break\u0026#34;, \u0026#34;impact\u0026#34;, \u0026#34;alarm\u0026#34;} MIN_CONFIDENCE = 0.82 DEBOUNCE_SECONDS = 30 _last_fire = 0.0 def handle(clip): label, confidence = classifier.predict(clip) # what was it, how sure # 1) classify by LABEL, not loudness if label not in ALERT_LABELS: return # 2) confidence gate if confidence \u0026lt; MIN_CONFIDENCE: return # 3) debounce repeats within a window global _last_fire now = time.monotonic() if now - _last_fire \u0026lt; DEBOUNCE_SECONDS: return _last_fire = now fire_alert(label, confidence) Then — and this is the part people skip — I refused to trust it until I\u0026rsquo;d measured it. We had that hand-labeled log from the noisy night, so I scored the new logic against ground truth before letting it page a single human again.\n# replay the labeled clips through the new gate, compare to truth python eval_detector.py \\ --clips ./ground_truth/clips/ \\ --labels ./ground_truth/truth.csv \\ --report precision_recall # precision 0.94 (almost no false alarms) # recall 0.88 (catches the real ones, including the quiet break) # false-positives/night: 41 -\u0026gt; 1 Forty-one false alarms a night down to one. The quiet break? Caught.\nWhy it happened The amplitude check was the first thing someone wrote — a five-minute proof of concept to confirm the mic and the webhook worked end to end. It worked. It shipped. The classifier got bolted on later for \u0026ldquo;labeling,\u0026rdquo; but nobody ever rewired the trigger to actually use it.\nSo the smart part rode along as a passenger while the dumbest possible heuristic drove. Classic. The proof of concept becomes production because it never visibly breaks — until 2 a.m. on a night with a heavy door.\nTakeaways Loudness is not meaning. Amplitude tells you something happened, not what. If your model predicts a label, trigger on the label. Gate on confidence. A bare classification with no threshold is just a louder guess. Make the model commit before it pages a human. Debounce. One physical event becomes many samples. Collapse repeats inside a window or you\u0026rsquo;ll get a 41-message wall. Validate against ground truth before you trust it. Keep a hand-labeled event log and measure precision/recall. \u0026ldquo;Seems better\u0026rdquo; is not a number. Audit the trigger path, not just the model. A great classifier is worthless if a five-minute if loud: is still the thing pulling the trigger. ","permalink":"https://errorzap.com/posts/the-sensor-that-cried-wolf/","summary":"A loud-sound detector paged us all night for slamming doors — because it measured volume, not meaning.","title":"The Sensor That Cried Wolf"},{"content":" It was a Friday night and my brand-new health dashboard had an opinion.\nIn a tidy little card, glowing red, it announced: \u0026ldquo;More steps strongly correlates with WORSE sleep (r = -0.61).\u0026rdquo;\nI stared at it. So the cure for insomnia is\u0026hellip; sitting still? Cool. Great. I\u0026rsquo;d just spent a weekend wiring up Postgres, a correlation engine, and a slick dark UI to discover that exercise is bad for you.\nThe number was real. The correlation was real. The conclusion was complete garbage. And it took me an embarrassingly long evening to figure out why.\nThe investigation The setup was simple. Four metrics per day — steps, sleep hours, resting heart rate, weight — pulled from the tracker\u0026rsquo;s export into a table. The engine did the obvious thing: loop over every pair of columns, run a Pearson correlation, sort by absolute value, surface the \u0026ldquo;strongest findings.\u0026rdquo;\nFirst I assumed a bug. Sign flipped somewhere, units crossed, a join gone sideways. I pulled the raw rows.\nSELECT day, steps, sleep_hours, resting_hr FROM health_daily ORDER BY day DESC LIMIT 10; The data was fine. High-step days really did have less sleep. The math wasn\u0026rsquo;t lying. The math was just answering a dumber question than I thought I\u0026rsquo;d asked.\nThen I noticed the pattern in the rows. The big-step / bad-sleep days clustered. They were the days I remembered — the chaotic ones. Travel days. Deadline days. The days I was on my feet because everything was on fire.\nThe \u0026ldquo;aha\u0026rdquo; There was a third variable sitting in the middle of the whole thing, and I\u0026rsquo;d never measured it: a stressful day.\nA busy, stressful day makes me walk more (running around, pacing, errands) AND sleep worse (wired, late, anxious). Steps and sleep don\u0026rsquo;t touch each other. They\u0026rsquo;re both just symptoms of the same hidden cause.\n┌──────────────────────┐ │ STRESSFUL DAY │ \u0026lt;- the thing I never logged │ (the confounder) │ └─────────┬────────────┘ │ ┌───────────┴───────────┐ ▼ ▼ ┌────────────┐ ┌──────────────┐ │ + STEPS │ ????? │ - SLEEP │ └────────────┘ \u0026lt;-----\u0026gt; └──────────────┘ no actual arrow between these two The engine drew the dotted line at the bottom and called it a discovery. It had no idea the box at the top even existed.\nAnd it got worse. With only a few weeks of data, every correlation was riding on a tiny sample — a couple of weird days could swing r wildly. On top of that I was testing every pair: 4 metrics is 6 comparisons, and the more pairs you test, the more likely pure noise hands you a juicy-looking number. Test enough things and \u0026ldquo;significant\u0026rdquo; findings appear for free.\nConfounders, tiny samples, and many comparisons. Three different ways to manufacture confident nonsense, all firing at once.\nSTRESSFUL DAY (hidden cause) + STEPS - SLEEP spurious r solid = real cause \u0026 dashed = the lie the engine reported The engine saw the red dotted line. It never saw the mauve box. The fix I stopped trusting the engine and started guarding it. Four changes.\n1. Require a real sample size. No pair gets reported under a floor.\nMIN_N = 21 # three weeks minimum before we say a word if pair.n \u0026lt; MIN_N: pair.verdict = \u0026#34;insufficient_data\u0026#34; continue 2. Correct for multiple comparisons. If you test 6 pairs, a raw p \u0026lt; 0.05 means almost nothing. Hold them all to a stricter bar.\nfrom statsmodels.stats.multitest import multipletests pvals = [p.pvalue for p in pairs] reject, p_adj, _, _ = multipletests(pvals, alpha=0.05, method=\u0026#34;holm\u0026#34;) 3. Control for the obvious third variable. Partial correlation: hold the suspected confounder fixed and see if anything survives.\nimport pingouin as pg # does steps-vs-sleep survive once we account for a stress proxy? result = pg.partial_corr( data=df, x=\u0026#34;steps\u0026#34;, y=\u0026#34;sleep_hours\u0026#34;, covar=\u0026#34;stress_proxy\u0026#34; ) # the -0.61 collapsed toward zero once stress was held constant 4. Relabel the output. The card no longer says \u0026ldquo;correlates with.\u0026rdquo; It says \u0026ldquo;possible link — investigate.\u0026rdquo; Every finding ships as a hypothesis, never a verdict.\nThe steps/sleep \u0026ldquo;finding\u0026rdquo; failed three of the four gates and got demoted to a quiet maybe. Exactly where it belonged.\nWhy it happened Because a correlation engine is a confidence machine with no judgment. It will faithfully compute a number for any two columns you hand it and present that number with the same authority whether it\u0026rsquo;s bedrock truth or coincidence between two symptoms of a Tuesday from hell.\nThe math was never wrong. My question was. \u0026ldquo;Are these two columns correlated?\u0026rdquo; is trivial. \u0026ldquo;Does one affect the other?\u0026rdquo; is a completely different question the engine was never equipped to answer — and I let the pretty red card pretend it had.\nTakeaways Correlation engines on personal data manufacture convincing nonsense. A real r with a fake meaning looks identical to a true insight on a dashboard. Hunt the confounder first. If two things correlate, ask what unmeasured third thing could be driving both before you believe either causes the other. Tiny samples lie loudly. A handful of weird days can swing a correlation hard. Set a minimum-N floor and enforce it in code. Testing many pairs guarantees false hits. Correct for multiple comparisons (Holm, Bonferroni, FDR) or you\u0026rsquo;ll \u0026ldquo;discover\u0026rdquo; links that are pure noise. Ship correlations as hypotheses, never conclusions. Label the output \u0026ldquo;investigate,\u0026rdquo; not \u0026ldquo;proven.\u0026rdquo; Your dashboard\u0026rsquo;s job is to point, not to swear. ","permalink":"https://errorzap.com/posts/why-your-fitness-trackers-correlations-are-lying/","summary":"I built a dashboard that confidently told me walking ruins my sleep — then I learned what a confounder is.","title":"Why Your Fitness Tracker's Correlations Are Lying to You"},{"content":" A user pings me: \u0026ldquo;The app won\u0026rsquo;t open anymore.\u0026rdquo;\nClassic. I ask the usual: did anything change? \u0026ldquo;It updated this morning.\u0026rdquo;\nOf course it did.\nI remote in and double-click the icon. The cursor does the little spinny thing. The process shows up in Task Manager — CPU ticking, memory allocated, the whole act. But no window. Ever. It just sits there, alive and useless, like a server that boots to a blinking cursor and stops caring.\nKill it. Relaunch. Same thing. Process up, window absent. Reboot the machine. Same thing. Reinstall the app clean. Same. Thing.\nThat\u0026rsquo;s the part that bugged me. A fresh install hanging the exact same way means the problem isn\u0026rsquo;t the binary. The binary is fine. Something outside the binary is poisoning it.\nThe investigation When a process starts but never paints a window, it\u0026rsquo;s usually stuck waiting on something — a lock, a handle, a chunk of state it expects to load. So I stopped staring at the app and started watching what it touched.\nLaunch ───► Process starts ───► loads cached runtime bundles │ ┌─────┴──────┐ ▼ ▼ new v2 binary OLD v1 cache ✗ │ │ └─────┬──────┘ ▼ version mismatch ► hang ▼ ✗ no window There it was. The updater swapped the binary to v2 but left a directory of cached runtime bundles from v1 sitting right where the new build looks first. New code, stale cache, two versions trying to share one brain. The app loads the old bundles, chokes on the mismatch, and waits forever for a handshake that\u0026rsquo;s never coming.\nReinstalling didn\u0026rsquo;t fix it because reinstalling doesn\u0026rsquo;t clear that cache. The installer drops a fresh binary and walks away. The landmine stays armed.\nv2 binary fresh install ✓ v1 cache stale bundles ✗ loads first mismatch ‣ window never paints The new binary is innocent — it's the leftover cache that hangs it. The aha Here\u0026rsquo;s the tell that saved me: I deleted that cache directory and the app limped to life. Window appeared. It ran. Notably, clearing the cache needed no admin rights — it lives in the user profile, not Program Files. So this isn\u0026rsquo;t a \u0026ldquo;corrupt install, nuke from orbit\u0026rdquo; problem. It\u0026rsquo;s a \u0026ldquo;stale state\u0026rdquo; problem.\nBut \u0026ldquo;limped\u0026rdquo; is the operative word. Deleting the cache once gets you a window, but the app\u0026rsquo;s background helper service was still holding the old runtime state in memory. It\u0026rsquo;d rebuild the cache half-correctly and get flaky again. The durable fix was to make the helper rebuild from scratch.\nThe fix Order matters. Kill everything first, then clear the cache, then restart the helper so it regenerates clean state, then launch.\n# 1. Kill every process the app owns — no orphans holding the cache open Get-Process -Name \u0026#34;TheApp*\u0026#34; -ErrorAction SilentlyContinue | Stop-Process -Force # 2. Delete the stale runtime/bundle cache (user profile — no admin needed) Remove-Item -Recurse -Force \u0026#34;$env:LOCALAPPDATA\\TheApp\\Cache\\bundles\u0026#34; -ErrorAction SilentlyContinue # 3. Restart the background helper service so it rebuilds state fresh Restart-Service -Name \u0026#34;TheAppHelper\u0026#34; -Force # 4. Confirm exactly one helper is up before relaunching Get-Service -Name \u0026#34;TheAppHelper\u0026#34; Then launch the app normally. Window paints, app stays up across reboots. Done.\nIf you skip step 1 and a stray process still has a handle on the cache dir, the delete silently no-ops and you\u0026rsquo;re right back where you started — wondering why a \u0026ldquo;fix\u0026rdquo; did nothing. Clear the orphans first, every time.\nWhy it happened Auto-updaters are optimists. They assume the new binary and the old cached state are compatible, so they swap the executable and leave the rest in place to \u0026ldquo;save time.\u0026rdquo; Most of the time that\u0026rsquo;s fine. When the runtime bundle format changes between versions, it absolutely is not — and the app has no graceful path for \u0026ldquo;my cache is from the future\u0026rsquo;s past,\u0026rdquo; so it just hangs.\nReinstalling feels like the big hammer, but it only touches the binary. The poison was always in the user-profile cache the installer never looks at.\nTakeaways A hang right after an auto-update is stale state, not a corrupt install. Clear the cache and restart the helper before you reinstall. A reinstall that changes nothing is a clue, not a dead end — it means the problem lives outside the install footprint. Cache/runtime dirs usually sit in the user profile — clearing them needs no admin, so try the cheap fix first. Kill orphan processes before deleting their files. A held handle turns your fix into a no-op and wastes an hour. Restart the background service, don\u0026rsquo;t just delete and pray. Deleting the cache gets a window; restarting the helper makes it stay. ","permalink":"https://errorzap.com/posts/the-app-that-wouldnt-open-after-an-update/","summary":"A desktop app auto-updated, swallowed its own window, and hung forever — and the fix had nothing to do with the install.","title":"The App That Wouldn't Open After an Update"},{"content":"There is a specific sound. A mouth click. A wet little lip-smack between words. A sharp inhale before a sentence.\nYou probably don\u0026rsquo;t hear them. I hear nothing else.\nMisophonia means certain sounds don\u0026rsquo;t annoy me — they hijack me. So I did what any sysadmin does with a problem that won\u0026rsquo;t leave: I decided to make it someone else\u0026rsquo;s job. Specifically, a script\u0026rsquo;s. The pitch was simple. Detect the trigger sounds in any audio or video, scrub them out before playback, hand me back a clean track.\nThe pitch was simple. The first version was a disaster.\nThe naive approach My first instinct was the dumbest possible one, which is usually where I start.\nTriggers feel violent, so I assumed they were loud. Threshold the loudness, mute anything that spikes, done by lunch.\n# v1: the volume-gate theory of everything import numpy as np def scrub_naive(samples, sr, thresh_db=-18.0): frame = int(0.02 * sr) # 20ms frames out = samples.copy() for i in range(0, len(samples) - frame, frame): chunk = samples[i:i+frame] rms = np.sqrt(np.mean(chunk**2)) + 1e-9 db = 20 * np.log10(rms) if db \u0026gt; thresh_db: # \u0026#34;too loud\u0026#34; -\u0026gt; kill it out[i:i+frame] = 0.0 return out I ran it on a podcast. It muted the consonants. It muted laughter. It muted entire emphasized words. And the lip-smacks? Sailed right through, untouched, smug.\nBecause the triggers were never loud. A lip-smack sits below normal speech in raw energy. I\u0026rsquo;d built a machine that deleted the wrong things at the wrong volume for the wrong reason.\nThe aha I pulled the waveforms apart and actually looked at them. That\u0026rsquo;s when it clicked.\nA vowel is periodic and narrowband — energy stacked in tidy harmonic bands, sustained over time. A mouth click is the opposite animal: short, broadband, transient. A flat smear of energy across the whole spectrum, gone in 30 milliseconds.\nAMPLITUDE-OVER-TIME (loudness lies) SPECTRUM (signature tells the truth) speech ┌─────────────┐ click ┌─┐ speech ▁▃█▇▅▂ (harmonic, banded) │ /\\ /\\ /\\ │ │█│ click ▅▅▆▅▆▅ (flat, broadband) ────────┘/ \\/ \\/ \\ └──┐ ────┘ └── breath ▂▂▃▂▂▃ (broadband, low, hissy) loud + sustained quiet + 30ms ^ v1 chased THIS, the wrong axis The trigger wasn\u0026rsquo;t a level. It was a shape. A fingerprint smeared across frequency and time. And \u0026ldquo;find this fingerprint in a stream\u0026rdquo; is not a thresholding problem.\nIt\u0026rsquo;s a classification problem. I\u0026rsquo;d been trying to solve a recognition task with a ruler.\naudio frames features spectral flat. ZCR / onset MFCC classifier p(trigger) + threshold clean duck \u0026 fade Stop measuring how loud it is. Describe what it looks like, then let a classifier decide. The fix So I threw out the ruler and described the signature instead. Per frame, extract the features that actually separate a click from a vowel — spectral flatness (how broadband it is), zero-crossing rate, onset sharpness, MFCCs. Then a small classifier outputs a probability, and only frames over a confidence threshold get touched.\nimport librosa, numpy as np def features(chunk, sr): S = np.abs(librosa.stft(chunk, n_fft=512, hop_length=128)) return np.concatenate([ [librosa.feature.spectral_flatness(S=S).mean()], # broadband? [librosa.feature.zero_crossing_rate(chunk).mean()],# transient hiss? librosa.feature.mfcc(y=chunk, sr=sr, n_mfcc=13).mean(axis=1), ]) def scrub(samples, sr, clf, p=0.85, debounce_ms=40): frame, out = int(0.02*sr), samples.copy() cooldown, hold = 0, int(debounce_ms/20) for i in range(0, len(samples)-frame, frame): prob = clf.predict_proba(features(samples[i:i+frame], sr)[None])[0,1] if prob \u0026gt;= p: cooldown = hold # fire + arm debounce if cooldown \u0026gt; 0: # short cross-fade, not a hard cut out[i:i+frame] *= np.linspace(1, 0, frame) if cooldown==hold else 0.0 cooldown -= 1 return out Two details earned their keep. The confidence threshold stops it firing on every sibilant s. The debounce keeps one click from flickering the gate on and off mid-word. And I duck with a short cross-fade instead of a hard zero, so the cut doesn\u0026rsquo;t itself become a click.\nOn isolated triggers — a click in a gap, a breath between sentences — it works. Validated against a hand-labeled set, and it catches the things that used to make me leave the room.\nWhy it happened I anthropomorphized the problem. The sounds feel loud and aggressive to me, so I encoded my emotional read of them as a signal property. The DSP didn\u0026rsquo;t care about my feelings. Loudness and identity are different axes, and I\u0026rsquo;d built my whole v1 on the wrong one.\nAnd I\u0026rsquo;ll be honest about the part that isn\u0026rsquo;t solved: triggers that land on top of speech. When a lip-smack overlaps a spoken word, ducking the frame mangles the word too. Source separation in real time is a genuinely harder problem, and pretending I\u0026rsquo;d nailed it would be the same mistake in a nicer suit. That\u0026rsquo;s phase two. For now the tool is honest about what it can and can\u0026rsquo;t pull apart.\nTakeaways \u0026ldquo;Detect this sound\u0026rdquo; is classification, not thresholding. If you reach for a volume gate, ask whether the thing you want is actually defined by loudness — it usually isn\u0026rsquo;t. Define the signature in feature space. Spectral flatness, ZCR, onset, MFCCs separate a broadband transient from a harmonic vowel far better than amplitude ever will. Validate against labeled examples. \u0026ldquo;It feels right\u0026rdquo; is not a metric. A confidence threshold is only meaningful when you\u0026rsquo;ve measured it against ground truth. Debounce and cross-fade your edits. A hard cut to fix a click can introduce a new click. Smooth the seams. Be loud about the cases you can\u0026rsquo;t solve. Isolated triggers: handled. Triggers overlapping speech: not yet, and saying so is the whole point. ","permalink":"https://errorzap.com/posts/teaching-a-computer-to-mute-the-sounds-i-hate/","summary":"I tried to delete a sound from my life with a volume threshold. The sound was never loud — it was a fingerprint, and I had to teach a machine to read it.","title":"Teaching a Computer to Mute the Sounds I Hate"},{"content":" It started as a slow-afternoon habit: scan yourself the way an attacker would. Pull up Shodan, type in your own public IP, and brace.\nI expected a clean sheet. UFW was configured. ufw deny on everything that wasn\u0026rsquo;t 80, 443, and the VPN. I\u0026rsquo;d checked it a dozen times. Green across the board.\nShodan disagreed.\nThere it was, indexed and timestamped: an admin/management panel answering on a port that was supposed to be firewalled into oblivion. Banner, title, the works. Anyone with a browser could\u0026rsquo;ve found it. Some of them probably had.\nThe investigation First reaction: Shodan\u0026rsquo;s cache is stale. It happens. So I scanned from the outside, from a box that had no business reaching anything internal.\nnmap -Pn -p 9000 192.0.2.10 --open # from a cloud box, not my network # (really: nmap -Pn -p 9000 \u0026lt;my-public-ip\u0026gt;) nmap -Pn -p 9000 192.0.2.10 PORT STATE SERVICE 9000/tcp open http Open. From the public internet. Not stale.\nSo I SSH\u0026rsquo;d in and asked UFW what it thought it was doing.\nsudo ufw status verbose | grep 9000 # (nothing — port 9000 is not allowed) sudo ufw status | head # Status: active # 22/tcp ALLOW Anywhere # 443/tcp ALLOW Anywhere UFW was adamant: 9000 is closed. The internet was equally adamant: 9000 is open. One of them was lying, and firewalls don\u0026rsquo;t lie. They just get bypassed.\nThe \u0026ldquo;aha\u0026rdquo; Then I remembered what was actually listening on 9000.\nA Docker container.\ndocker ps --format \u0026#39;{{.Names}}\\t{{.Ports}}\u0026#39; # admin-panel 0.0.0.0:9000-\u0026gt;9000/tcp 0.0.0.0:9000-\u0026gt;9000/tcp. There it is. That -p 9000:9000 in the compose file. Docker doesn\u0026rsquo;t ask UFW for permission — it writes its own iptables rules, straight into the DOCKER chain, and they get evaluated before your tidy UFW INPUT rules ever run.\nYour ufw deny is a polite note on the front door. Docker built a second door around back and propped it open.\nInbound packet → :9000 │ ▼ ┌───────────────────────────┐ │ iptables nat/PREROUTING │ │ DOCKER chain (DNAT) ◄────┼── Docker put this here. │ → 172.17.0.2:9000 │ UFW never gets a vote. └───────────────────────────┘ │ ╲ ▼ ╲ (never reached) container :9000 ┌────────────────────┐ │ ufw-user chain │ │ DENY 9000 (moot) │ └────────────────────┘ Why the firewall didn't matter ☠ internet\nDOCKER chain DNAT :9000 \u0026 DNAT wins first \u003c ufw, host rule \u003e skipped ufw DENY 9000 (moot) admin-panel 172.17.0.2:9000 Docker's DNAT rule wins the race. UFW's deny never gets to vote. The fix The publish target was the bug. 0.0.0.0 means \u0026ldquo;the whole world.\u0026rdquo; It almost never should.\nI bound the panel to loopback only and put it behind the reverse proxy with auth, reachable just through the VPN.\n# docker-compose.yml — before ports: - \u0026#34;9000:9000\u0026#34; # = 0.0.0.0:9000, published to the internet # after ports: - \u0026#34;127.0.0.1:9000:9000\u0026#34; # localhost only; proxy/VPN reaches it docker compose up -d docker ps --format \u0026#39;{{.Names}}\\t{{.Ports}}\u0026#39; # admin-panel 127.0.0.1:9000-\u0026gt;9000/tcp For anything that genuinely must survive the firewall, lock it at the DOCKER-USER chain — the one place Docker promises not to clobber:\n# allow only the tailnet, drop everyone else for published container ports sudo iptables -I DOCKER-USER -i eth0 -s 100.64.0.0/10 -j RETURN sudo iptables -I DOCKER-USER -i eth0 -p tcp --dport 9000 -j DROP Re-scan from outside:\nnmap -Pn -p 9000 192.0.2.10 # PORT STATE SERVICE # 9000/tcp filtered http filtered. Dark. The panel only answers behind the tunnel now.\nWhy it happened Nobody screwed up the firewall. The firewall did exactly what it was told. The trap is the mental model: people assume UFW sits at the front gate and inspects everything inbound. Docker doesn\u0026rsquo;t go through that gate — it cuts its own keyhole in iptables and your host rules never see the packet.\n-p 9000:9000 reads like \u0026ldquo;expose this locally.\u0026rdquo; It actually means \u0026ldquo;publish this to 0.0.0.0.\u0026rdquo; One missing 127.0.0.1: prefix and an internal admin tool was on the public internet for who-knows-how-long, indexed by Shodan, waiting.\nTakeaways Scan yourself the way an attacker would. Shodan plus an external nmap from a box outside your network is the only ground truth. Your config file is a hypothesis, not a result. Docker port publishing ignores your host firewall. -p PORT:PORT writes its own iptables rules and bypasses UFW entirely. ufw deny does nothing for published container ports. Bind admin services to 127.0.0.1 or a VPN, never 0.0.0.0. Always prefix the publish: 127.0.0.1:9000:9000. Default-public is a footgun. Use Docker-aware firewalling. If a container port truly needs outside reach, gate it in the DOCKER-USER chain — the one chain Docker won\u0026rsquo;t overwrite. Admin panels belong on localhost or a tunnel, behind a reverse proxy and auth. Convenient-but-public is just public. ","permalink":"https://errorzap.com/posts/i-found-my-own-server-on-shodan/","summary":"I scanned my own infrastructure like an attacker would — and found an admin panel wide open that my firewall swore was closed.","title":"I Found My Own Server on Shodan"},{"content":" The ticket said: \u0026ldquo;The server is possessed.\u0026rdquo;\nThat\u0026rsquo;s a direct quote. An app had been rock-solid for two years, then one Tuesday it \u0026ldquo;just stopped\u0026rdquo; — no config change, no deploy, no nothing. The on-call engineer had already rebooted twice, blamed cosmic rays, and was halfway to blaming the building\u0026rsquo;s wiring.\nIt wasn\u0026rsquo;t possessed. It never is.\nAfter enough years you learn that 95% of \u0026ldquo;haunted network\u0026rdquo; tickets are the same seven mundane bastards wearing bedsheets. None of them are mysterious. All of them are one check away from being solved. The trick is running the check before you start theorizing about ghosts.\nHere\u0026rsquo;s the scene I keep coming back to.\nTHE GHOST THE CHECK VERDICT ───────────────────────── ────────────────── ────── \u0026#34;service won\u0026#39;t start\u0026#34; ──► who owns the port? ✓ orphan \u0026#34;site loads wrong server\u0026#34; ──► what does dig say? ✓ stale DNS \u0026#34;one client 500s\u0026#34; ──► force --http1.1 ✓ h2 quirk \u0026#34;big transfers hang\u0026#34; ──► ping -M do -s 1472 ✓ MTU/MSS \u0026#34;intermittent dropouts\u0026#34; ──► arp -a (two MACs?) ✓ dup IP \u0026#34;works one direction\u0026#34; ──► firewall state table ✓ asymmetric \u0026#34;localhost can\u0026#39;t connect\u0026#34; ──► loopback fw rule ✓ blocked pipe The Investigation, Compressed Into a Checklist 1. The service that \u0026ldquo;can\u0026rsquo;t start.\u0026rdquo; Symptom: it crashed, you restart it, and now it refuses to bind. The aha: the crashed process orphaned the port and the kernel still thinks it\u0026rsquo;s owned. The check — find the squatter:\n# Linux ss -ltnp \u0026#39;sport = :8080\u0026#39; # Windows netstat -ano | findstr :8080 Kill the orphan PID, the port frees, the service starts. No exorcism required.\n2. DNS serving a corpse. Symptom: you migrated a site, half the world sees the new box, this one client sees the old one. The check — ask what\u0026rsquo;s actually resolving, not what should:\ndig +short app.example.com @1.1.1.1 nslookup app.example.com # flush the local liar: sudo resolvectl flush-caches # systemd ipconfig /flushdns # Windows Stale cache, gone.\n3. The one client that 500s while everyone else is fine. That\u0026rsquo;s an HTTP/2 downgrade or framing incompatibility. Pin the protocol and watch it heal:\ncurl -v --http1.1 https://app.example.com/health If --http1.1 works and the default doesn\u0026rsquo;t, you found your ghost.\n4. Small requests fine, big ones hang. Classic MTU/MSS mismatch — pings and tiny payloads sail through, the TLS handshake or a large transfer stalls forever. Probe for the real path MTU:\n# 1472 + 28 overhead = 1500. Shrink until it stops fragmenting. ping -M do -s 1472 192.0.2.1 If 1472 fails but 1400 works, clamp your MSS and the hangs vanish.\nhost A host B MTU 1400 link small ✓ oversized ✗ ▼ The big frame doesn't fit through the narrow link — and nobody told the sender. 5. The intermittent flapper. Symptom: connectivity that drops for seconds at random — a rogue DHCP server or a static IP collision. Check the ARP table for one IP claimed by two MACs:\narp -a | sort # two different MACs on the same IP = your culprit ip neigh show 6. Works one way, dies the other. Asymmetric routing where the return path takes a different door and a stateful firewall drops it for having no matching session. Look at the state table:\n# Linux conntrack conntrack -L | grep 192.0.2.50 # pf pfctl -ss | grep 192.0.2.50 No state entry for the return flow = there\u0026rsquo;s your \u0026ldquo;ghost.\u0026rdquo;\n7. The app that \u0026ldquo;should be local\u0026rdquo; but can\u0026rsquo;t reach itself. A host firewall rule or blocked named pipe is eating loopback. Confirm the listener and that localhost itself is allowed:\nss -ltnp | grep 127.0.0.1 sudo iptables -L INPUT -n -v | grep -E \u0026#39;lo|127.0.0.1\u0026#39; That Tuesday \u0026ldquo;possession\u0026rdquo;? Number one. Crashed worker, orphaned port, ss -ltnp named the PID in four seconds. Ghost dispelled.\nWhy It Happens None of these are exotic. They\u0026rsquo;re the natural failure modes of stateful systems: processes die untidily, caches outlive their truth, two devices want the same address, a firewall remembers a flow that no longer exists. The supernatural feeling comes entirely from partial symptoms — it works for some and not others, or for small things and not big ones. Partial failure reads as spooky. It\u0026rsquo;s just state you haven\u0026rsquo;t looked at yet.\nTakeaways Run the seven-item checklist before you theorize. The boring answer is almost always right. \u0026ldquo;Some clients work\u0026rdquo; is a clue, not a contradiction — it points straight at DNS, HTTP/2, or MTU. \u0026ldquo;Small works, big hangs\u0026rdquo; is MTU/MSS until proven otherwise. Don\u0026rsquo;t waste an hour on app logs. Check state, not config. ss, dig, arp, and the firewall state table tell you what is, not what should be. Reboots hide orphans; they don\u0026rsquo;t explain them. Find the squatter PID before you cycle power and lose the evidence. ","permalink":"https://errorzap.com/posts/seven-network-gotchas-that-look-like-ghosts/","summary":"The network isn\u0026rsquo;t haunted — it\u0026rsquo;s orphaned ports, stale DNS, and a firewall eating your return path. Run the checklist before you light the candles.","title":"Seven Network Gotchas That Look Like Ghosts"},{"content":" The ticket landed with a number attached, and the number was wrong.\nA monitoring report — the one we generate so nobody has to think too hard — claimed a site was running 99 wireless access points. Ninety-nine. For a building that, last I checked, you could walk end to end during a coffee that hadn\u0026rsquo;t gone cold yet.\nI\u0026rsquo;d been in that building. I knew the ceilings. There were not 99 APs up there. There weren\u0026rsquo;t even a quarter of that.\nSo either the site had quietly become an enterprise campus overnight, or the report was hallucinating. Spoiler: reports don\u0026rsquo;t hallucinate. They count exactly what you tell them to count, which is a different and worse problem.\nThe investigation First rule when a number looks insane: don\u0026rsquo;t argue with the number. Go find ground truth and make the number argue with that.\nI pulled the controller\u0026rsquo;s live device list — the actual adopted, talking-to-me-right-now hardware:\n# adopted APs the controller is actively managing curl -sk -b /tmp/controller.cookie \\ https://controller.example.internal/proxy/network/api/s/default/stat/device \\ | jq \u0026#39;[.data[] | select(.type==\u0026#34;uap\u0026#34;)] | length\u0026#39; 12 Twelve. Twelve adopted access points, online, real, with MAC addresses I could go put my hand on.\nSo where did the report get 99?\nI went and read the thing that built the report instead of the report itself. The collector wasn\u0026rsquo;t querying live adopted devices. It was scraping a broader chunk of the controller database — and it was counting rows, not radios you could hang on a wall.\nThe aha Three lies were stacked on top of each other, and the report added them all up with a straight face.\nWHAT THE REPORT COUNTED WHAT WAS ACTUALLY THERE ───────────────────────── ─────────────────────── ┌───────────────────────┐ │ adopted APs 12 │ ◄──────── ✓ real, on the ceiling ├───────────────────────┤ │ pending / forgotten │ │ ghost DB rows ~30│ ◄──────── ✗ stale, never cleaned ├───────────────────────┤ │ per-radio entries │ │ (2.4 / 5 / 6 GHz) │ ◄──────── ✗ one AP counted 3× ├───────────────────────┤ │ per-SSID rows │ ◄──────── ✗ counted broadcasts └───────────────────────┘ as devices Σ = 99 Σ (real) = 12 Every tri-band AP showed up as three radio entries. Every SSID broadcast left its own footprint. And the controller DB still held a graveyard of devices that had been swapped out, RMA\u0026rsquo;d, or \u0026ldquo;forgotten\u0026rdquo; in the UI months ago — rows that never actually leave the database, they just stop mattering. To a human. Not to a COUNT(*).\nStack the real twelve on top of the radio multiplication on top of the ghost rows, and you arrive — with perfect arithmetic and zero truth — at 99.\n1 real AP radio 2.4GHz row radio 5GHz row radio 6GHz row per-SSID rows... ghost / forgotten 99 \"APs\" One access point on the ceiling, multiplied by radios, SSIDs, and database ghosts into a number nobody could find. The fix The fix wasn\u0026rsquo;t clever. It was narrowing the question. Count adopted UAP-type devices, deduplicated by MAC, and exclude anything pending or forgotten:\ncurl -sk -b /tmp/controller.cookie \\ https://controller.example.internal/proxy/network/api/s/default/stat/device \\ | jq \u0026#39;[.data[] | select(.type==\u0026#34;uap\u0026#34;) | select(.adopted==true) | select(.state==1) # 1 = connected | .mac] | unique | length\u0026#39; 12 Then I did the thing the database can\u0026rsquo;t: I walked the building and counted ceilings. Twelve.\nThe report now pulls that single reconciled number, and it carries a footnote: adopted, connected, deduped by MAC. If it ever disagrees with the live device list again, it\u0026rsquo;s the report that has to explain itself.\nWhy it happened Because counting feels trivial, so nobody guards it. length on a JSON array looks like truth. But the controller DB isn\u0026rsquo;t a list of access points — it\u0026rsquo;s an event log of everything that ever claimed to be one: every radio, every SSID broadcast, every device that got swapped and never purged. Counting rows in that pile and calling it \u0026ldquo;APs\u0026rdquo; is like counting fingerprints and reporting the number of burglars.\nThe artifact survived for months because 99 was plausible-adjacent. Nobody who\u0026rsquo;d been on-site ever read the report; nobody who read the report had ever been on-site. The number floated in the gap between them, fat and unchallenged, until somebody finally looked up.\nTakeaways Never report an inventory number you can\u0026rsquo;t reconcile to ground truth. A count needs two witnesses: the live device list and a physical walk. Rows are not things. Per-radio and per-SSID entries multiply one AP into many. Filter by device type and dedupe by MAC before you trust a total. Controller databases keep ghosts. Forgotten, pending, and RMA\u0026rsquo;d devices linger in the DB long after they leave the ceiling. Scope your query to adopted and connected. Plausible-adjacent is the dangerous zone. A wrong number that\u0026rsquo;s obviously insane gets caught. One that\u0026rsquo;s merely \u0026ldquo;a bit high\u0026rdquo; rides for months. Read the query, not the report. When a metric lies, the bug is upstream in what you told it to count — go fix the question. ","permalink":"https://errorzap.com/posts/the-99-access-points-that-didnt-exist/","summary":"An inventory report swore a site had 99 access points; the ceiling said twelve, and the database had been lying the whole time.","title":"The 99 Access Points That Didn't Exist"},{"content":" The ticket said \u0026ldquo;internet is down.\u0026rdquo; Then, four minutes later, \u0026ldquo;never mind, it\u0026rsquo;s back.\u0026rdquo; Then, twenty minutes after that, \u0026ldquo;it\u0026rsquo;s down again.\u0026rdquo;\nThat pattern is a special kind of hell. A clean outage you can chase. A flapping one that heals itself before you SSH in just laughs at you.\nI got a client on the phone. Their whole site would blackhole — every desktop, every phone, the printer, all of it — for one to five minutes, then snap back like nothing happened. No schedule. No trigger anyone could name. The ISP swore the circuit was clean, and for once the ISP was right.\nThe investigation I started where you start: at the edge. The gateway was up. WAN was up. The firewall logs were boring in the way you want them to be boring. No flapping interface, no CPU spike, no DHCP storm in the leases.\nSo I sat on a workstation and just hammered the gateway while I waited for the next outage.\nping -i 0.5 192.0.2.1 It ran fine for ten minutes. Then — dead air. Eight, nine, ten dropped replies. Total blackhole. Then it came back on its own.\nThe instant it died, I dumped the ARP table.\narp -a 192.0.2.1 And there it was. The gateway\u0026rsquo;s IP was mapped to a MAC address I did not recognize. Not the firewall\u0026rsquo;s MAC. Some OUI I had to look up by hand.\nI watched it flap in real time:\n┌─────────────────────────────────────────────────────────────┐ │ arp -a 192.0.2.1 (watched over ~30s) │ ├─────────────────────────────────────────────────────────────┤ │ t+0s 192.0.2.1 ► aa:bb:cc:11:22:33 (firewall) ✓ │ │ t+6s 192.0.2.1 ► de:ad:be:ef:00:99 (???) ✗ │ │ t+9s 192.0.2.1 ► de:ad:be:ef:00:99 (???) ✗ │ │ t+14s 192.0.2.1 ► aa:bb:cc:11:22:33 (firewall) ✓ │ │ t+19s 192.0.2.1 ► de:ad:be:ef:00:99 (???) ✗ │ └─────────────────────────────────────────────────────────────┘ ▼ when the IP pointed at de:ad:..:99, traffic vanished When the gateway IP pointed at the firewall, the network worked. When it pointed at that mystery MAC, every packet bound for the internet got handed to a device that did absolutely nothing with it. Classic ARP poisoning. The only question was who.\nThe \u0026ldquo;aha\u0026rdquo; I looked up the OUI. The mystery MAC belonged to a Tuya-based Wi-Fi module — the kind that ships inside cheap smart-home gadgets by the millions.\nThen I cross-referenced the DHCP leases for that MAC and found a hostname that made me laugh out loud: a smart window blind controller. Somebody had put a $30 motorized blind on the corporate flat network.\nThat little NIC was broadcasting unsolicited ARP replies claiming to be 192.0.2.1. Every client on the segment believed it, updated its ARP cache, and started shipping its default route straight into a window covering.\nThe fix First, stop the bleeding. I pinned the gateway\u0026rsquo;s real MAC on the affected machines so they\u0026rsquo;d stop trusting the lie while I worked.\n# Linux sudo ip neigh replace 192.0.2.1 lladdr aa:bb:cc:11:22:33 dev eth0 nud permanent Then the real fix — get IoT off the flat network entirely and turn on the switch protections that should\u0026rsquo;ve been on from day one.\nBEFORE AFTER ────── ───── VLAN 1 (everything) VLAN 10 desktops/servers ├─ desktops VLAN 20 voice ├─ servers VLAN 90 IoT ◄── blind lives here, ├─ phones firewalled, no L2 └─ smart blind ◄── poisoner reach to clients # Cisco-style switch hardening (concept) # Trust only the uplink for DHCP; drop rogue offers elsewhere ip dhcp snooping ip dhcp snooping vlan 90 interface Gi1/0/24 ip dhcp snooping trust # Dynamic ARP Inspection — validate ARP against snooping bindings ip arp inspection vlan 90 interface Gi1/0/24 ip arp inspection trust Workstation who has .1? Gateway .1 aa:bb:cc... Smart Blind \"I am .1!\" poisoned ARP ► traffic blackholes into a window covering The blind announces itself as the gateway; clients believe it and route to nowhere. Last step: I yanked the blind, pushed its vendor firmware update, and parked it on the isolated IoT VLAN where it can lie about being the gateway all it wants — and nobody who matters will hear it.\nWhy it happened Cheap Tuya firmware is a grab bag. Some builds have buggy network stacks that emit malformed or unsolicited ARP — gratuitous ARP gone feral, sometimes triggered by a Wi-Fi reconnect, which explains the random timing. Could also be straight-up malicious; with no-name IoT you genuinely can\u0026rsquo;t tell, and it doesn\u0026rsquo;t change the remediation.\nEither way, a flat L2 network trusts every device equally. ARP has no authentication. Whatever shouts loudest and last, wins. Put one mouthy gadget on that wire and it can pretend to be anything — including the door to the internet.\nTakeaways On weird, intermittent, self-healing outages, check the gateway\u0026rsquo;s ARP entry FIRST. arp -a \u0026lt;gateway-ip\u0026gt; flapping to an unknown MAC is a five-second smoking gun. Match a rogue MAC to its OUI and your DHCP leases. That\u0026rsquo;s how a \u0026ldquo;mystery device\u0026rdquo; became \u0026ldquo;the smart blind in the break room.\u0026rdquo; IoT does not belong on your flat network. Ever. Its own VLAN/SSID, firewalled off from clients, no L2 reach to anything that matters. Turn on DHCP Snooping and Dynamic ARP Inspection. They exist precisely to kill rogue gateways and ARP lies at the switch. Cheap IoT is an untrusted host by default — bug or malice, you treat it the same: isolate, update, and never let it speak for the gateway. ","permalink":"https://errorzap.com/posts/the-smart-blind-that-took-down-the-network/","summary":"A $30 Wi-Fi window blind decided it was the default gateway, and the whole site went dark whenever it felt like it.","title":"The Smart Blind That Took Down an Entire Network"},{"content":" The chat ping came in at 4:47 PM. \u0026ldquo;Hey, is the app down? All my stuff is gone.\u0026rdquo;\nI pulled up the site. It loaded fine. Login worked. Dashboard rendered. And it was empty. No records. No users. No history. A perfect, pristine, brand-new install — on a server that had been running for fourteen months.\nNothing was broken. That\u0026rsquo;s what made my stomach drop. A crash leaves a corpse. This was worse. The app was alive and humming, cheerfully serving a database that had been born thirty seconds ago.\nThe investigation First instinct: bad migration. Nope — no migrations had run. Second instinct: someone fat-fingered a DELETE. But the Postgres-shaped panic faded fast, because this app didn\u0026rsquo;t use Postgres. It used a SQLite file. A single file, living in the app directory.\nThat detail mattered more than I realized.\nI checked the logs. The last thing that happened before the data vanished was a deploy. Routine. Boring. A dev had pushed a tiny CSS fix and run the deploy script, same as always.\nSo I opened the deploy script. And there it was, line nine, smiling at me like a loaded gun:\nscp -r ./app/* deploy@198.51.100.10:/srv/app/ ./app/*. The whole directory. Including app/data/app.db.\nThe dev\u0026rsquo;s local app.db was empty — fresh clone, never seeded. Every deploy had been copying that empty file up and stomping the live one. The only reason nobody noticed sooner was that this dev happened to have a blank local copy. Previous deploys from other machines had local data that masked the bug.\nHere\u0026rsquo;s the shape of the disaster:\nDEV MACHINE PRODUCTION ┌──────────────┐ ┌──────────────┐ │ app/ │ │ app/ │ │ ├ main.py │ ── scp -r ──► │ ├ main.py │ │ ├ static/ │ \u0026#34;deploy\u0026#34; │ ├ static/ │ │ └ data/ │ │ └ data/ │ │ app.db │ ════ EMPTY ════► │ app.db │ ◄ 14 months │ (0 rows) │ OVERWRITES │ (was full) │ GONE └──────────────┘ └──────────────┘ The fix wasn\u0026rsquo;t on the live disk. But the box ran nightly off-site snapshots, and the most recent one was eleven hours old. We restored that, replayed the day\u0026rsquo;s activity from request logs by hand, and lost almost nothing. Lucky. Stupid lucky.\napp code main.py static/ prod data app.db uploads/ rsync --exclude data/ code \u003e\u003e data \u0026 never the reverse The deploy should push code over the wall and leave production's data untouched. The fix The old script blanket-copied everything. The new one does three things it refused to do before: backs up prod data first, uses rsync with explicit excludes, and never touches the data paths at all.\n#!/usr/bin/env bash set -euo pipefail HOST=\u0026#34;deploy@198.51.100.10\u0026#34; APP_DIR=\u0026#34;/srv/app\u0026#34; STAMP=\u0026#34;$(date +%Y%m%d-%H%M%S)\u0026#34; # 1. Back up prod data BEFORE we touch anything. ssh \u0026#34;$HOST\u0026#34; \u0026#34;tar czf /srv/backups/data-${STAMP}.tar.gz \\ -C ${APP_DIR} data uploads\u0026#34; # 2. Push CODE only. Exclude every path that holds state. rsync -avz --delete \\ --exclude=\u0026#39;data/\u0026#39; \\ --exclude=\u0026#39;uploads/\u0026#39; \\ --exclude=\u0026#39;*.db\u0026#39; \\ --exclude=\u0026#39;*.sqlite\u0026#39; \\ --exclude=\u0026#39;.env\u0026#39; \\ ./app/ \u0026#34;${HOST}:${APP_DIR}/\u0026#34; echo \u0026#34;Deployed. Pre-deploy snapshot: data-${STAMP}.tar.gz\u0026#34; Note --delete is now safe because the data dirs are excluded — rsync won\u0026rsquo;t reach in and prune data/ if it never syncs data/ in the first place. And the backup runs before the push, so even a botched exclude has a parachute.\nWhy it happened scp -r ./app/* isn\u0026rsquo;t a bug. It does exactly what it says. The bug was a design decision nobody made on purpose: letting state live inside the deploy artifact. The database sat in the same folder as the code, so \u0026ldquo;deploy the code\u0026rdquo; silently meant \u0026ldquo;deploy the database too.\u0026rdquo;\nThe destructive path wasn\u0026rsquo;t unlikely. It ran on every single deploy. It just stayed invisible until a dev with an empty local copy pulled the trigger.\nTakeaways Deploy code, not data. Application state — databases, uploads, user files — must live outside anything a deploy can overwrite. Any deploy that can touch the database is a loaded gun. Add explicit --exclude rules for every data and upload path, and verify them in a dry run (rsync -n). Back up before you deploy, not after. A snapshot taken the instant before the push is the cheapest insurance you\u0026rsquo;ll ever buy. Make the destructive path impossible, not just unlikely. \u0026ldquo;Be careful\u0026rdquo; is not a control. Excludes and pre-deploy backups are. Boring near-misses are gifts. We got lucky with an eleven-hour-old snapshot. Don\u0026rsquo;t design around luck — design so the gun can\u0026rsquo;t fire. ","permalink":"https://errorzap.com/posts/the-deploy-script-that-ate-the-production-database/","summary":"A homegrown deploy script copied the whole app folder to prod — database file and all — and quietly overwrote live data with an empty local copy.","title":"The Deploy Script That Ate the Production Database"},{"content":" It was supposed to be a quick audit.\nA client had asked me to sweep a private repo before they handed access to a new contractor. Routine. I cloned it, ran a scanner over the working tree, and started reading. Five minutes in, the scanner lit up red on a config helper.\nA reusable credential. Plaintext. In the repo.\nI felt that familiar little spike of righteous sysadmin adrenaline — who the hell committed THIS — opened the blame, and read the author line.\nIt was me.\nThe investigation Here\u0026rsquo;s the part everyone gets wrong, and the part I almost got wrong too.\nMy first instinct was to delete the line, commit \u0026ldquo;remove secret,\u0026rdquo; and move on. Crisis averted, right?\nWrong. The file in the working tree is the least important place that secret lives. Git is a time machine. Deleting it today does nothing about the dozens of commits where it sits, fat and happy, in the history. Anyone with git log and ten seconds can walk straight to it.\nI checked how deep it went:\ngit log --all --oneline -S \u0026#39;the_secret_string\u0026#39; -- path/to/config git rev-list --all --count It went deep. The credential had been in the repo for months, across dozens of commits, surviving file renames and a refactor. Deleting the file would leave it perfectly intact in every parent commit.\nthe working tree (what you see) │ ▼ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ commit │◄──│ commit │◄──│ commit │◄──│ commit │ │ HEAD │ │ ~12 │ │ ~31 │ │ ~40 │ │ ✗ gone │ │ ▓ secret│ │ ▓ secret│ │ ▓ secret│ └─────────┘ └─────────┘ └─────────┘ └─────────┘ ▲ └──────────────┬──────────────┘ you \u0026#34;fixed\u0026#34; it still right here, forever The \u0026ldquo;aha\u0026rdquo; The realization isn\u0026rsquo;t \u0026ldquo;I need to scrub this.\u0026rdquo; It\u0026rsquo;s earlier and uglier than that:\nA secret is compromised the instant it\u0026rsquo;s committed. Not when the repo goes public. Not when someone clones it. At commit time. Every backup, every fork, every laptop that ever pulled, every CI cache — assume the secret is out. You cannot un-ring that bell by editing history.\nSo scrubbing history is necessary, but it is not step one. Step one is to make the leaked value worthless.\nRemediation, in the only order that works 1 ROTATE kill the value 2 SCRUB purge history 3 RELOCATE secrets mgr 4 PREVENT pre-commit scan ► ► ►\nskip step 1 \u0026amp; steps 2-4 are theater Deleting the file is not on this list. It was never remediation. The fix 1. Rotate, immediately. Before anything else, I logged into the provider, revoked the credential, and issued a fresh one. The old value is now a dead string. Whoever has it has nothing.\n2. Purge it from history. With the value already dead, I scrubbed it so it stops scaring the next auditor (me, in six months). git filter-repo is the modern tool; BFG works too.\n# install once: pip install git-filter-repo echo \u0026#39;the_secret_string==\u0026gt;REDACTED\u0026#39; \u0026gt; replacements.txt git filter-repo --replace-text replacements.txt # then force the rewritten history out git push --force --all git push --force --tags Every collaborator re-clones after this. A force-push that rewrites history will wreck anyone\u0026rsquo;s local copy — warn them first.\n3. Move secrets out of the repo for good. The config now reads from the environment, injected at runtime from a secrets manager. Nothing sensitive touches the tree.\n# .env stays OUT of git echo \u0026#39;.env\u0026#39; \u0026gt;\u0026gt; .gitignore # app reads from the environment, value lives in the secrets manager export API_TOKEN=\u0026#34;$(vault kv get -field=token secret/app)\u0026#34; 4. Make it impossible to repeat. A pre-commit hook scans every staged change so a secret never reaches a commit in the first place.\n# .pre-commit-config.yaml # - repo: https://github.com/gitleaks/gitleaks # rev: v8.18.0 # hooks: [{ id: gitleaks }] pre-commit install gitleaks detect --source . --verbose Why it happened No villain here. Just the most ordinary mistake in the trade.\nMonths ago, mid-deploy, I needed the thing to work now. I hardcoded the credential \u0026ldquo;just to test,\u0026rdquo; told myself I\u0026rsquo;d pull it out before committing, got interrupted, and committed everything with a git add .. The TODO never came back. The repo was private, so it felt safe — and \u0026ldquo;private\u0026rdquo; is exactly the lie that lets these things rot in history for half a year.\nPrivate isn\u0026rsquo;t a control. It\u0026rsquo;s a setting someone can flip.\nTakeaways A secret is burned the moment it\u0026rsquo;s committed. Rotate first, always. Treat the value as public from commit time forward — because effectively it is. Deleting the file is not remediation. The credential lives in every old commit. You must rewrite history (git filter-repo / BFG) and force-push, or it\u0026rsquo;s still there. Secrets belong in a manager, not the tree. Env-injected values, .gitignore your .env, and never let credentials and code share a home. Automate the catch. A gitleaks/trufflehog pre-commit hook stops the next leak before it\u0026rsquo;s a commit — humans forget, hooks don\u0026rsquo;t. \u0026ldquo;Private repo\u0026rdquo; is not a security boundary. Audit your own repos like you\u0026rsquo;d audit a stranger\u0026rsquo;s. The author line might surprise you. ","permalink":"https://errorzap.com/posts/i-found-a-secret-in-a-git-repo/","summary":"I went looking for a leaked credential in a client\u0026rsquo;s stack and found one staring back at me — committed by my own hand, baked into history forever.","title":"I Found a Secret in a Git Repo (It Was Mine)"},{"content":" 📶 90s KID // The Day We Got 56k The Afternoon the Screaming Got Faster You remember the day.\nThe beige box showed up.\nSmaller than you expected.\nA little plastic brick with a single blinking light, riding inside the family computer like a smuggled treasure.\n56k.\nYou didn\u0026rsquo;t know what a \u0026ldquo;k\u0026rdquo; was.\nYou just knew the old one was a \u0026ldquo;33.6\u0026rdquo; and this one was a bigger number, and bigger numbers meant good.\nDad said it like a magic word.\n\u0026ldquo;We got 56k now.\u0026rdquo;\nThe grown-ups nodded. Somebody whistled.\nYou had no idea what just happened, but you knew it was a Big Deal, the kind of Big Deal that came with a receipt and a warning not to touch anything.\nHere\u0026rsquo;s what we thought it was:\na machine that could go faster if you believed hard enough powered by a sound — that holy shriek, that robot gargling underwater forbidden during dinner, sacred after homework somehow connected to the telephone, which meant Grandma could ruin everything by picking up to call about a casserole You\u0026rsquo;d lean in close while it connected.\nBee-doo. Bee-doo. Skreeeee. Kshhhhhhhh.\nAnd you\u0026rsquo;d hold your breath like the noise might notice you and stop.\nHere\u0026rsquo;s what it actually was.\nThat beautiful screaming was a handshake — two modems negotiating, out loud, in real time. Your modem and the one at the other end were literally singing tones at each other to figure out the cleanest way to talk over a copper phone line built for voices, not data.\nThe 56k part? A gorgeous little cheat. The phone network had quietly gone digital in the middle, so your downloads could ride that clean digital signal almost all the way to your house — that\u0026rsquo;s how you squeezed 56 kilobits a second out of a wire designed for chitchat. (Uploads were stuck slower. The magic only ran one way.)\nAnd that\u0026rsquo;s why it tied up the phone: your modem was modulating and demodulating — turning ones and zeros into sound and back again — using the exact same line as the kitchen phone. One line. One conversation at a time. The internet literally was a phone call.\n+-----------------------------+ | U.S. Robotics 56K o | \u0026lt;- the one blinky light | [CD][RD][SD][TR][MR][AA] | +--__----------------------__--+ || bee-doo skreeee || ~~||~~~~~~~~~~~~~~~~~~~~~~~~||~~ || ___ ___ || \\\\__/ \\______/ \\____ // \u0026lt;- the handshake, screaming Sure, the picture loaded one agonizing horizontal strip at a time, top to bottom, and somebody always picked up the phone right before the last strip. Sure, \u0026ldquo;going online\u0026rdquo; was an event you scheduled around. But for one shining afternoon, you were the fastest house on the block.\nAnd here\u0026rsquo;s the thing — that handshake never really died.\nYour phone still negotiates with a cell tower. Your laptop still does a little dance with the Wi-Fi router. Every \u0026ldquo;connecting…\u0026rdquo; spinner you\u0026rsquo;ve ever cursed at is the same conversation, just gone silent. We made it faster, we made it invisible, we even made it wireless — but somewhere under all of it, two machines are still saying hi, how do we talk? before they trust each other with your data.\nWe just don\u0026rsquo;t get to hear it scream anymore.\nAnd honestly?\nWe kind of miss it.\n","permalink":"https://errorzap.com/posts/90s-kid-the-56k-upgrade/","summary":"The afternoon the modem screamed a little faster and the whole house held its breath.","title":"90s Kid - The Day We Got 56k"},{"content":" A field tech messaged me from a basement with no signal: \u0026ldquo;Your app just hangs. White screen. Forever.\u0026rdquo;\nThat\u0026rsquo;s a fun one to read on a Tuesday, because the entire point of that app was that it worked offline. Progressive Web App. Service worker. Cached shell. The whole pitch was \u0026ldquo;open it anywhere, signal or not.\u0026rdquo;\nSo I did what he did. I put my phone in airplane mode and tapped the icon.\nBlank screen. Spinner that never spun. The app didn\u0026rsquo;t crash — it just waited. Patiently. Forever. Like a golden retriever staring at a door that\u0026rsquo;s never going to open.\nThe investigation On a normal connection, everything was perfect. Load times snappy, cache populated, Lighthouse happy. The bug only showed up with the network truly gone — not slow, gone.\nThat asymmetry is the tell. If something works online and dies offline, you stop blaming the app and start blaming whatever sits between the app and the cache. For a PWA, that\u0026rsquo;s the service worker\u0026rsquo;s fetch handler.\nI pulled it up. Here\u0026rsquo;s the offending strategy, paraphrased:\n// service-worker.js — the original sin self.addEventListener(\u0026#39;fetch\u0026#39;, (event) =\u0026gt; { event.respondWith( fetch(event.request) // always try the network first .then((res) =\u0026gt; { const copy = res.clone(); caches.open(\u0026#39;app-shell\u0026#39;).then((c) =\u0026gt; c.put(event.request, copy)); return res; }) .catch(() =\u0026gt; caches.match(event.request)) // fall back to cache ); }); Network-first. Looks reasonable. Hit the network, cache the response, and if the network fails, serve from cache. The .catch() is right there. So why doesn\u0026rsquo;t it fall back?\nThe \u0026ldquo;aha\u0026rdquo; Because offline doesn\u0026rsquo;t fail. Not quickly, anyway.\nWhen you\u0026rsquo;re truly offline, that fetch() doesn\u0026rsquo;t reject in 20 milliseconds with a clean \u0026ldquo;no route to host.\u0026rdquo; Depending on the platform, it can sit there for tens of seconds — or effectively never resolve — before the OS decides it\u0026rsquo;s done. The .catch() is waiting for a rejection that takes its sweet time. Meanwhile, the app shell never paints, because the navigation request for index.html is stuck in that same limbo.\nThe cache was fine. The fallback logic was fine. The problem was that the fallback never got to run, because nothing told the network attempt to give up.\nONLINE OFFLINE (the bug) ┌──────────────┐ ┌──────────────────────┐ │ fetch() │ │ fetch() │ │ │ │ │ │ │ │ ▼ ~80ms │ │ ▼ │ │ ✓ response │ │ ...waiting... │ │ │ │ │ ...waiting... │ │ ▼ │ │ ...waiting... ▓ │ │ cache.put │ │ (never rejects) ▓ │ │ │ │ │ │ ▓ │ │ ▼ │ │ ✗ .catch() │ │ paint ✓ │ │ never fires │ └──────────────┘ └──────────────────────┘ app shell never paints Promise.race — first one home wins fetch(request) may never resolve timeout(3000ms) rejects fast race cache.match ✓ paints now Race the network against a short timeout — whichever finishes first decides whether you wait or serve the cache. The fix Two changes. First, never let a fetch in the service worker run unbounded — race it against a timeout so it fails fast and the cache fallback actually fires:\nfunction fetchWithTimeout(request, ms = 3000) { return Promise.race([ fetch(request), new Promise((_, reject) =\u0026gt; setTimeout(() =\u0026gt; reject(new Error(\u0026#39;sw-fetch-timeout\u0026#39;)), ms) ), ]); } Second, stop being network-first for the app shell. Serve from cache immediately, then refresh the cache in the background — classic stale-while-revalidate:\nself.addEventListener(\u0026#39;fetch\u0026#39;, (event) =\u0026gt; { event.respondWith( caches.open(\u0026#39;app-shell\u0026#39;).then(async (cache) =\u0026gt; { const cached = await cache.match(event.request); const network = fetchWithTimeout(event.request) .then((res) =\u0026gt; { cache.put(event.request, res.clone()); // refresh in background return res; }) .catch(() =\u0026gt; cached); // offline or slow? cached is already in hand return cached || network; // paint instantly if we have it }) ); }); Now the shell paints from cache on the first frame, online or off. The network update happens behind the scenes, and if it\u0026rsquo;s slow or absent, the timeout cuts it loose in three seconds instead of never.\nAirplane mode, tap the icon, instant load. The basement tech got his app back.\nWhy it happened Network-first feels safe — \u0026ldquo;always get the freshest thing, fall back to cache if it breaks.\u0026rdquo; But it quietly assumes the network fails promptly. Offline doesn\u0026rsquo;t fail promptly. It stalls. And a stalled promise with no timeout is just an infinite loading screen wearing a .catch() as a disguise.\nThe app shell — the HTML, JS, CSS that make the thing exist — should never depend on a live connection to render. That\u0026rsquo;s the entire premise of \u0026ldquo;offline-capable.\u0026rdquo; We\u0026rsquo;d built the cache and then put a blocking network call in front of it.\nTakeaways Never let a service-worker fetch run without a timeout. Promise.race it against a setTimeout reject so offline fails in milliseconds, not minutes. Serve the app shell cache-first or stale-while-revalidate — never network-first. The UI should paint from cache on the first frame, every time. Test with the network truly gone, not just throttled. Slow networks reject; dead networks stall. They\u0026rsquo;re different failure modes and only the dead one exposes this bug. A .catch() is not a fallback if nothing triggers the rejection. Error handling only runs when the error actually fires — and \u0026ldquo;offline\u0026rdquo; often doesn\u0026rsquo;t fire fast. \u0026ldquo;Offline-capable\u0026rdquo; is a claim you have to test in airplane mode, not a checkbox you tick because you registered a service worker. ","permalink":"https://errorzap.com/posts/the-pwa-that-froze-when-offline/","summary":"A field tech opened an \u0026lsquo;offline-capable\u0026rsquo; app in a dead zone and got a blank screen that hung forever — because the service worker was politely waiting for a network that was never coming.","title":"The PWA That Froze When Offline"},{"content":" The cron job ran at 11 PM. By the time I poured coffee the next morning, it had called a paid LLM API a few thousand times.\nNobody touched it. Nobody approved it. It just sat there in the dark, retrying, retrying, retrying — like a vending machine eating a stuck dollar bill, except the dollar bill was billable tokens.\nI didn\u0026rsquo;t find it because of a clever alert. I found it because the provider dashboard had a graph that looked like a cliff face.\nThe scene I run a small fleet of unattended automations for a client. One of them enriches records overnight by sending them through a metered LLM endpoint. Cheap per call. Boring. The kind of job you set up once and forget — which is exactly the problem.\nThat night, the upstream service it depended on got flaky. A handful of calls returned errors. The loop did what badly-written loops always do.\nIt tried again. Immediately. Forever.\n┌────────────────────────────────────────────────────┐ │ THE LOOP (as written) │ │ │ │ ┌──────────┐ error ┌──────────────┐ │ │ │ call API │ ────────► │ retry now │ │ │ └────┬─────┘ └──────┬───────┘ │ │ ▲ │ │ │ └────────────────────────┘ │ │ no backoff · no cap · no kill switch │ │ │ │ result: ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ thousands of hits │ └────────────────────────────────────────────────────┘ No sleep. No max-retry counter. No circuit breaker. Every failure became an instant request, and every instant request became another line on the invoice.\nThe investigation First thing: confirm it\u0026rsquo;s us and not a stolen key. I pulled the provider\u0026rsquo;s usage view and bucketed by hour.\n# usage exploded between 23:00 and 07:00 — exactly the cron window # logs from the job host told the same story journalctl -u record-enrich.service --since \u0026#34;yesterday 23:00\u0026#34; \\ | grep -c \u0026#34;POST /v1/\u0026#34; # -\u0026gt; 4,300-something. Overnight. For a job that should make ~80 calls. Same source IP. Same user-agent. Same service. Not a breach — a self-inflicted wound. Somehow that\u0026rsquo;s worse, because it means the call was coming from inside the house.\nThe \u0026ldquo;aha\u0026rdquo; The smoking gun was four lines of code. Paraphrased:\nwhile not done: try: resp = client.complete(payload) done = True except Exception: continue # \u0026lt;-- the whole disaster, right here continue. No delay. No ceiling. The instant the upstream hiccuped, this turned into a tight spin loop firing paid requests as fast as the network would carry them. A retry without backoff isn\u0026rsquo;t resilience — it\u0026rsquo;s a denial-of-wallet attack you launch against yourself.\n$ spend time → hard quota cap 23:00 cron starts 07:00 caught ✓ Cost climbs unattended all night; a provider-side quota cap is the only thing that would have flattened it before morning. The fix I treated the key as compromised even though it wasn\u0026rsquo;t, because the behavior was indistinguishable from a leak. Containment first, blame later.\n# 1. Rotate the key immediately — old one dies on the spot provider keys rotate --name record-enrich --revoke-old # 2. Disable the API entirely at the provider while I fix the code. # Kill the bleeding before patching the artery. provider api disable --service record-enrich # 3. Re-enable WITH a hard cap + budget alert. The cap is the real fix. provider budget set --service record-enrich \\ --hard-limit-usd 25 --period monthly provider alerts set --service record-enrich \\ --notify-at 50% --notify-at 90% --channel telegram Then the code got the guardrails it should have shipped with:\nimport time MAX_RETRIES = 5 for attempt in range(MAX_RETRIES): try: resp = client.complete(payload) break except TransientError: time.sleep(min(2 ** attempt, 30)) # exponential backoff, capped else: raise RuntimeError(\u0026#34;gave up after retries — failing loud, not looping\u0026#34;) Backoff. A retry ceiling. And a loud failure instead of a silent infinite spin. The provider cap is the seatbelt; this is actually steering the car.\nWhy it happened It happened because \u0026ldquo;it\u0026rsquo;s just a small overnight job\u0026rdquo; is the exact mindset that ships uncapped loops at paid endpoints. The cost-per-call was trivial, so nobody did the multiplication. Trivial times infinity is still a number you have to pay.\nThe code had no concept of \u0026ldquo;too many.\u0026rdquo; The provider had no concept of \u0026ldquo;enough.\u0026rdquo; With neither a ceiling in the app nor a ceiling at the wallet, the only limiter left was how fast the network could move — and that\u0026rsquo;s a throttle on the damage, not a budget.\nTakeaways Put a hard spend cap at the provider. It\u0026rsquo;s the only limit a runaway loop can\u0026rsquo;t out-code. Set it before you write the first request. Backoff and a max-retry count are not optional. A retry without delay or ceiling is a self-DoS. continue on an exception is a loaded gun. Wire usage alerts at 50% and 90%. You want a phone buzz at midnight, not a graph shaped like a cliff at breakfast. Treat weird usage as a leak until proven otherwise. Rotate the key and disable the endpoint first; debug the code second. Containment beats curiosity. \u0026ldquo;Small overnight job\u0026rdquo; is a smell. Anything unattended that touches a metered API gets a kill switch, a cap, and an alert — or it doesn\u0026rsquo;t get deployed. ","permalink":"https://errorzap.com/posts/the-api-loop-that-ran-up-the-bill-overnight/","summary":"An unattended automation found a paid LLM endpoint, lost its mind, and hammered it thousands of times while everyone slept.","title":"The API Loop That Ran Up the Bill Overnight"},{"content":" It started, like most of my regrets, with a bookmark folder.\napp-one:3000. dashboard:8501. that-thing-i-forgot:8080. A dozen self-hosted apps, each squatting on its own port, each throwing a browser security warning because I\u0026rsquo;d long since stopped pretending to manage individual certs. Half were self-signed. Half were plain HTTP. All of them were ugly.\nEvery new app was the same ritual: pick a port, pray it\u0026rsquo;s free, wire up TLS by hand, give up on TLS, add another :PORT bookmark, lose it. The homelab worked. It just looked like a ransom note.\nSo I sat down to do it properly. Once.\nThe investigation The core problem wasn\u0026rsquo;t the apps. It was that I had N apps and N TLS problems. Every service believed it was personally responsible for the open internet. None of them were good at it.\nHere\u0026rsquo;s the mess, drawn honestly:\nBEFORE AFTER ┌──────────────────┐ ┌──────────────────────────┐ │ browser │ │ browser │ └───┬───┬───┬──────┘ └──────────┬───────────────┘ │ │ │ (which port?) │ one name, valid TLS ▼ ▼ ▼ ▼ :3000 :8501 :8080 ┌─────────┐ self-signed / http / ??? │ Caddy │ *.lab.example.tld each app owns its own cert └──┬──┬──┬─┘ wildcard cert │ │ │ ▼ ▼ ▼ 127.0.0.1:3000/8501/8080 (apps bound to localhost) The fix wasn\u0026rsquo;t to make every app better at certificates. It was to make none of them do certificates. Put one thing in front. Let it own TLS, routing, and renewal. Bind everything else to 127.0.0.1 so the only thing facing the world is the proxy.\nThat \u0026ldquo;one thing\u0026rdquo; is a reverse proxy. Mine\u0026rsquo;s Caddy.\nThe \u0026ldquo;aha\u0026rdquo; The trick that makes it sing is a wildcard certificate — *.lab.example.tld, one cert that\u0026rsquo;s valid for every subdomain I\u0026rsquo;ll ever invent — obtained over a DNS-01 challenge instead of the usual HTTP-01.\nDNS-01 matters here. HTTP-01 proves you own a name by serving a file on it, which means each name needs to be reachable. DNS-01 proves it by dropping a TXT record at your DNS provider, which means you can mint a wildcard without exposing a single app first. One challenge, infinite names.\nOnce that clicked, the whole bookmark folder collapsed into one file.\nThe fix Caddy needs a DNS-provider plugin to answer the DNS-01 challenge, so build it with the module for your provider:\nxcaddy build \\ --with github.com/caddy-dns/cloudflare # verify the module is in there ./caddy list-modules | grep dns.providers Then the global block tells Caddy how to get the wildcard, and each app is — genuinely — three lines:\n# Caddyfile { email admin@example.tld # credentials for the DNS-01 challenge acme_dns cloudflare {env.CF_API_TOKEN} } # the wildcard, requested once, renewed forever *.lab.example.tld { tls { dns cloudflare {env.CF_API_TOKEN} } } app-one.lab.example.tld { reverse_proxy 127.0.0.1:3000 } dashboard.lab.example.tld { reverse_proxy 127.0.0.1:8501 } Reload without dropping connections:\ncaddy validate --config /etc/caddy/Caddyfile caddy reload --config /etc/caddy/Caddyfile Confirm a fresh subdomain actually got real HTTPS:\ncurl -sI https://dashboard.lab.example.tld | head -n1 # HTTP/2 200 echo | openssl s_client -connect dashboard.lab.example.tld:443 2\u0026gt;/dev/null \\ | openssl x509 -noout -subject # subject=CN = *.lab.example.tld Adding the next app is now: pick a port, add a three-line block, caddy reload. Valid cert, no thought required.\nbrowser Caddy *.lab cert HTTPS :3000 :8501 :8080 127.0.0.1 — localhost only One proxy faces the world; the apps stay on localhost and never touch TLS. Why it happened The old setup wasn\u0026rsquo;t dumb, it was just additive. Each app got stood up in isolation, made its own peace with HTTPS, and got a port-shaped bookmark. Nobody ever sat down to factor out the part every app shares: a public face and a certificate.\nA reverse proxy is exactly that factoring. TLS, hostname routing, and renewal are cross-cutting concerns — solve them once, in front, and every app inherits the solution for free. The wildcard cert is what makes \u0026ldquo;inherits for free\u0026rdquo; literally true: no per-app issuance, no per-app renewal, no per-app anything.\nTakeaways One TLS problem beats N TLS problems. Put a reverse proxy in front and let it own certs, routing, and renewal so your apps never have to. Wildcard + DNS-01 is the unlock. *.domain minted via a DNS TXT challenge covers every subdomain you\u0026rsquo;ll ever invent — and you don\u0026rsquo;t have to expose an app to prove ownership. Bind apps to 127.0.0.1. If only the proxy faces the world, a misconfigured app can\u0026rsquo;t accidentally serve itself raw to the internet. New app = three lines. A subdomain block that reverse_proxys to a localhost port, a reload, done. Exposing a service stops being a project. Build the right Caddy. DNS-01 needs the provider plugin compiled in — xcaddy build --with, then list-modules to confirm before you wonder why the challenge hangs. ","permalink":"https://errorzap.com/posts/one-caddyfile-infinite-subdomains/","summary":"A dozen self-hosted apps, a dozen ugly ports, zero valid certs — until one wildcard cert turned \u0026rsquo;expose another app\u0026rsquo; into a three-line chore.","title":"One Caddyfile, Infinite Subdomains"},{"content":" 🖨 90s KID // The Dot-Matrix Printer The machine that screamed your homework into existence You remember the dot-matrix printer.\nThe big beige one.\nOn the floor, usually. Too heavy for the desk.\nWith the green-and-white striped paper that fed up out of the back in one endless accordion ribbon.\nAnd the noise.\nDear god, the noise.\nIt didn\u0026rsquo;t print so much as it announced. A grinding, buzzing, sawing shriek that meant somebody, somewhere in the house, was getting their book report out the door at 9:48 PM the night before it was due.\nYou\u0026rsquo;d stand there next to it, hypnotized.\nWatching the little head zip left.\nThen right.\nThen nudge the paper up a hair.\nThen left again.\nThe vibes were:\nForbidden. You were NOT allowed to tear the paper until it finished. Do not touch it. Sacred ritual. The little holes down the sides. The perforated edges you peeled off after, like the wrapper on a candy. Tension. Would it jam? It always might jam. Loud = important. If it was screaming, real work was happening. A faint smell of warm plastic and ink ribbon. And those side strips. The pin-feed holes. You\u0026rsquo;d peel both ribbons off in one clean pull if you were lucky, and it felt like completing a quest.\nWhat it actually was Here\u0026rsquo;s the nerdy part, and it\u0026rsquo;s genuinely cool.\nThat print head wasn\u0026rsquo;t spraying ink. It had a tiny grid of stiff little pins behind it. Tiny metal needles. As the head slid across, an electromagnet would fire individual pins forward, hammering them into an inked ribbon, which slapped that ink onto the paper.\nA character was just a pattern of dots punched out fast. Nine pins. Sometimes twenty-four if you were fancy. Impact printing — literally tiny hammers, hundreds of times a second. That\u0026rsquo;s where the scream came from.\nAnd the green stripes and side holes? That was continuous-feed tractor paper. The sprocket holes hooked onto little spinning gears so the printer could pull a whole novel through without anyone reloading a single sheet. The machine fed itself.\nThe reason you couldn\u0026rsquo;t tear it early: the paper was under tension on those gears. Yank it and you\u0026rsquo;d skew the whole job.\n_______________________________________ | o ___________________________ o | | o | E r r o r Z a p . . . . | o | | o | ...... ... .. .... ... | o | | o | .. ... .... ... . .... | o | | o |___________________________| o | | o ____________________________ o | | o ||||||| BZZZT-GRRRT-ZZZP |||| o | |__o___________________________ __ o__| \u0026#39;-.___pin-feed___.-\u0026#39; \u0026#39;-.___holes_.-\u0026#39; It never really left You think it\u0026rsquo;s gone. It isn\u0026rsquo;t.\nThat receipt curling out of the gas pump?\nThe slip the doctor\u0026rsquo;s office prints your appointment on?\nThe carbon-copy form at the auto shop?\nImpact printing. Still alive. Because here\u0026rsquo;s the secret a laser can\u0026rsquo;t do: hammering through paper lets you print carbon copies in one pass. That\u0026rsquo;s why warehouses, garages, and pharmacies never gave it up.\nSo the next time something prints with a buzz instead of a hum, lean in.\nThat\u0026rsquo;s the old beige monster.\nStill screaming.\nStill working.\nDon\u0026rsquo;t tear the paper yet.\n","permalink":"https://errorzap.com/posts/90s-kid-the-dot-matrix-printer/","summary":"The screaming beige monster that printed your book report one agonizing dot at a time, and the perforated paper-strips you weren\u0026rsquo;t allowed to tear off too early.","title":"90s Kid - The Dot-Matrix Printer"},{"content":" It started with a boring task. Quarterly DNS audit. Coffee, a zone file, and a vague sense that everything was fine.\nIt was not fine.\nHalfway down the export I hit a CNAME — a customer-facing subdomain — pointing at a cloud host we\u0026rsquo;d decommissioned back in the winter. The instance was gone. The DNS record had never gotten the memo.\nI sat there for a second. That subdomain still resolved. It still answered. It just answered nothing — a hosting account that no longer existed, on infrastructure we\u0026rsquo;d long since stopped paying for.\nThat\u0026rsquo;s not a typo. That\u0026rsquo;s a loaded gun pointed at your own brand.\nThe investigation A dangling DNS record is a pointer with no backing store. The record says \u0026ldquo;this name lives over there.\u0026rdquo; But \u0026ldquo;over there\u0026rdquo; got freed months ago — the VM was destroyed, the cloud bucket released, the hosting account closed.\nThe danger isn\u0026rsquo;t that it\u0026rsquo;s broken. The danger is that the resource is now available for anyone to claim.\nI pulled the full zone and started checking targets one by one. Does the A record\u0026rsquo;s IP still belong to us? Is the CNAME target a hostname we still control? Or is it a now-orphaned cloud resource sitting in a provider\u0026rsquo;s free pool, waiting for the next person to grab it?\nYour DNS The Internet ┌──────────────────┐ ┌────────────────────────┐ │ app.example.com │ │ cloud host (DELETED) │ │ CNAME ──────────┼────────▶│ account closed, freed │ └──────────────────┘ │ ▒▒▒ up for grabs ▒▒▒ │ └───────────┬────────────┘ │ attacker ▼ re-claims it ┌────────────────────────┐ │ serves THEIR content │ │ on YOUR subdomain │ └────────────────────────┘ The \u0026ldquo;aha\u0026rdquo; Here\u0026rsquo;s the part that turns a stale record into a breach.\nIf an attacker registers that freed cloud resource — same provider, same dangling target — the provider hands them the keys. Now your subdomain serves their content. And because it\u0026rsquo;s your subdomain, the blast radius is ugly:\nPixel-perfect phishing on a domain your users already trust. Cookies scoped to *.example.com get harvested — including session cookies from your real apps. A free, fully valid TLS certificate, because the takeover host can pass the domain-control check for a name you literally pointed at it. No CVE. No exploit chain. Just a DNS record that outlived its server.\napp.example.com CNAME \u003e freed host up for grabs attacker re-claims it valid TLS \u0026 your name = phishing on brand A deleted host gets re-claimed — and your trusted subdomain serves someone else's payload. The fix First, inventory everything. Pull the whole zone, not just the records you remember.\n# Dump every A / AAAA / CNAME in the zone dig @ns1.example.com example.com AXFR \\ | awk \u0026#39;$4 ~ /^(A|AAAA|CNAME)$/ {print $1, $4, $5}\u0026#39; Then resolve each one and check whether the target still answers — and whether it\u0026rsquo;s still yours.\nwhile read -r name type target; do echo \u0026#34;=== $name ($type) -\u0026gt; $target\u0026#34; dig +short \u0026#34;$name\u0026#34; curl -sS -o /dev/null -w \u0026#34; http=%{http_code} ip=%{remote_ip}\\n\u0026#34; \\ --max-time 8 \u0026#34;https://$name\u0026#34; || echo \u0026#34; DEAD / no response\u0026#34; done \u0026lt; zone-records.txt For the dangling ones, the rule is simple: if you don\u0026rsquo;t control the target, the record dies or gets repointed.\n# Delete the stale record (RFC 2136 dynamic update) nsupdate -k /etc/dns/update.key \u0026lt;\u0026lt;\u0026#39;EOF\u0026#39; server ns1.example.com zone example.com update delete app.example.com. CNAME send EOF Repoint anything still in use at infrastructure you own — a host behind your own reverse proxy, not a third-party resource you might let lapse again.\nFinally, make it a standing job. A nightly check that flags any record whose target stops resolving or stops belonging to you.\n# cron: nightly dangling-DNS sweep, alert on dead targets 0 4 * * * /opt/dns/dangle-check.sh \u0026amp;\u0026amp; \\ curl -fsS https://hc-ping.test/dns-sweep Why it happened DNS outlives servers. Always.\nWhen you spin a box down, you delete the box. You raise the ticket, you confirm it\u0026rsquo;s gone, you move on. The DNS record — living in a different system, owned by a different runbook — just sits there pointing at a ghost. Nobody\u0026rsquo;s job ends with \u0026ldquo;and remove the CNAME.\u0026rdquo;\nMultiply that by a few years of \u0026ldquo;temporary\u0026rdquo; subdomains and deprovisioned cloud accounts, and you\u0026rsquo;ve got a quiet pile of doors with no locks.\nTakeaways Inventory every record. You can\u0026rsquo;t audit a zone you\u0026rsquo;ve only half-remembered. Pull the whole thing. Verify the target, not just the name. A record that resolves isn\u0026rsquo;t safe — confirm the thing it points at is still alive and still yours. Prune the moment a server dies. Decommissioning a host isn\u0026rsquo;t done until its DNS is gone too. Bake it into the runbook. Prefer targets you control. Point subdomains at your own infrastructure, not third-party resources you might forget to renew. Monitor continuously. Dangling DNS is created by routine. Catch it with routine — a nightly sweep beats a quarterly surprise. ","permalink":"https://errorzap.com/posts/the-subdomain-that-pointed-at-nothing/","summary":"A routine DNS audit turned up subdomains aimed at servers that died months ago — quiet little doors anyone could walk through.","title":"The Subdomain That Pointed at Nothing"},{"content":" 🏛 90s KID // The Universal Remote The Clicker That Ruled the Living Room You remember the universal remote.\nNot the one that came with the TV. The other one.\nThe fat black slab that lived on the arm of the couch.\nHeavier than it had any right to be.\nThree triple-A batteries deep.\nIt had way too many buttons.\nHalf of them did nothing.\nThe other half did things nobody could explain.\nAnd only Dad knew the magic sequence.\nHere\u0026rsquo;s what we thought it was, as kids:\na wizard\u0026rsquo;s wand that talked to every box under the TV a forbidden object — touch it wrong and the screen goes blue and Dad sighs powered by a tiny, invisible laser you could never quite see the reason the TV \u0026ldquo;wasn\u0026rsquo;t working\u0026rdquo; was always your fault the thing that took 17 button presses just to watch a tape You\u0026rsquo;d point it at the ceiling, the dog, your sister.\nYou\u0026rsquo;d press TV / VCR / CABLE / AUX hoping something would happen.\nAnd when the channel finally changed?\nPure, unearned power. You were a god for one second.\nHere\u0026rsquo;s what it actually was.\nThat \u0026ldquo;invisible laser\u0026rdquo; was real — sort of.\nEvery press fired a little infrared LED in the nose of the remote.\nLight your eyes can\u0026rsquo;t see, blinking on and off insanely fast — flickering out a coded pattern, like Morse code made of glow.\nThe TV had a tiny IR receiver behind the front panel, watching for that exact blink-pattern. Volume up, channel down, power — each one a different rhythm of flashes.\nThe \u0026ldquo;universal\u0026rdquo; part was the clever trick.\nSony, Panasonic, Zenith — every brand blinked in its own dialect.\nSo the remote stored a codebook of them all.\nThat cursed setup ritual — hold SET, punch in 0-0-4-7 from the little fold-out card — that was you teaching it which dialect to speak.\nIt wasn\u0026rsquo;t magic.\nIt was a flashlight that knew how to talk.\n_________________ | ___________ | | | CH ▲ | | | | VOL + | | ((·)) ← invisible | |___________| | IR blink | (1) (2) (3) | | (4) (5) (6) | | (7) (8) (9) | | (TV)(VCR)(AUX) | |_________________| ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾ the sacred clicker Here\u0026rsquo;s the thing.\nIt never left.\nThat blinking infrared light is still in your living room right now.\nYour soundbar remote? IR.\nThe cable box clicker buried in the couch? IR.\nSame flashlight. Same secret language. Same dead battery.\nWe just stopped looking at it.\nThen your phone became the remote, and the remote became an app, and the app needed an update, and the update needed a login\u0026hellip;\n\u0026hellip;and somewhere, a kid is pointing a glowing wand at a TV,\npressing 17 buttons to watch one show,\nfeeling like a god for one whole second.\nSome magic just changes batteries.\n","permalink":"https://errorzap.com/posts/90s-kid-the-universal-remote/","summary":"You remember the fat black remote with too many buttons that somehow ran the whole living room.","title":"90s Kid - The Universal Remote"},{"content":" A customer wanted their shiny camera platform recording into a third-party NVR. Simple ask: pull the RTSP stream, point the recorder at it, walk away.\nThe recorder disagreed. Every stream it tried died on connect. No video, no error worth reading, just a connection that opened and then went cold.\nI\u0026rsquo;d wired up a hundred of these. This one fought me.\nThe investigation First instinct: bad URL. So I copied the stream address straight out of the platform and fed it to ffprobe, the universal \u0026ldquo;is this thing even alive\u0026rdquo; test.\nffprobe \u0026#34;rtsps://camera-host:7441/\u0026lt;stream-token\u0026gt;?enableSrtp\u0026#34; [tls] Error: handshake failure rtsps://camera-host:7441/...: Input/output error There it is. rtsps://. Not rtsp://. That little s is the whole story.\nThe platform was advertising an encrypted stream — RTSP wrapped in TLS, with SRTP turned on for the media itself. A high port, a token in the path, the works. Very modern. Very secure.\nAnd completely useless to a recorder that only speaks plain, unencrypted RTSP. The NVR reached out for a normal RTSP handshake and got a TLS negotiation shoved in its face. It had no idea what to do, so it hung up.\n┌──────────────┐ rtsps:// + SRTP ┌──────────────┐ │ Camera │ ────────────────────────►│ NVR │ │ platform │ (TLS handshake) │ (plain RTSP) │ └──────────────┘ └──────┬───────┘ │ ▼ ✗ handshake failure ✗ no video recorded Two devices, both fluent in \u0026ldquo;RTSP,\u0026rdquo; neither able to hold a conversation. The walled garden, working exactly as designed.\nCamera platform NVR plain RTSP rtsps:// :7441 ✗ SRTP restreamer rtsp:// :7447 ✓ plain The TLS wall (red) blocks the NVR. The plain-RTSP path — direct or via a restreamer — gets through. The \u0026ldquo;aha\u0026rdquo; Here\u0026rsquo;s the thing about these platforms: the encrypted stream is what they show you. It\u0026rsquo;s rarely the only thing they offer.\nI dug into the device settings and found a toggle — a separate, unencrypted RTSP endpoint on a different port. Plain rtsp://, basic credentials, no TLS, no SRTP. The boring old protocol the recorder actually wanted, sitting right there behind a checkbox nobody enabled by default.\nThe fix Enable the plain endpoint, grab the new URL, and verify with ffprobe before touching the NVR config:\nffprobe -v error -show_streams \\ \u0026#34;rtsp://\u0026lt;user\u0026gt;:\u0026lt;pass\u0026gt;@camera-host:7447/\u0026lt;stream-token\u0026gt;\u0026#34; codec_name=h264 width=2688 height=1512 ✓ Stream #0:0 Video: h264, 2688x1512, 15 fps Green across the board. Point the recorder at that URL and it records.\nIf the platform won\u0026rsquo;t expose a plain endpoint, drop a restreamer in the middle — something like MediaMTX or a thin ffmpeg relay — to ingest the encrypted stream and re-publish it as plain RTSP on your own box:\nffmpeg -rtsp_transport tcp \\ -i \u0026#34;rtsps://camera-host:7441/\u0026lt;stream-token\u0026gt;?enableSrtp\u0026#34; \\ -c copy -f rtsp \\ rtsp://127.0.0.1:8554/cam1 # verify the republished stream the NVR will actually consume ffprobe \u0026#34;rtsp://127.0.0.1:8554/cam1\u0026#34; The NVR talks to your restreamer. Your restreamer eats the TLS for breakfast. Everybody\u0026rsquo;s happy.\nWhy it happened Vendors are encrypting streams by default now, and good for them — that\u0026rsquo;s the right call for traffic crossing untrusted networks. But \u0026ldquo;secure by default\u0026rdquo; and \u0026ldquo;interoperable\u0026rdquo; are different goals, and the camera folks optimized for the first one.\nThe encrypted URL is front and center. The plain one is buried, off, and undocumented. So the integration looks broken when it\u0026rsquo;s really just locked. The garden has a gate; you have to go find it.\nTakeaways Read the scheme. rtsps:// on a high port with SRTP is a different animal than rtsp://. The handshake error is your first clue, not a riddle. Look for the plain-RTSP port. Most platforms still expose an unencrypted endpoint behind a toggle. Check device settings before you assume it can\u0026rsquo;t be done. ffprobe before you wire anything. If the probe can\u0026rsquo;t pull codec and resolution, the NVR won\u0026rsquo;t either. Put a restreamer in the middle when the vendor refuses to deobfuscate — MediaMTX or ffmpeg -c copy ingests RTSPS and republishes clean RTSP, no re-encode. A walled garden isn\u0026rsquo;t a wall. It\u0026rsquo;s a default. Find the gate the vendor left in the settings, and the lock-in mostly evaporates. ","permalink":"https://errorzap.com/posts/getting-cameras-out-of-the-walled-garden/","summary":"A camera platform that only spoke encrypted RTSPS quietly refused to talk to the NVR — until I found the plain-RTSP back door it was hiding.","title":"Getting Cameras Out of the Walled Garden"},{"content":" 🕹 90s KID // The LAN Party The Night We Carried Our Computers Into the Basement You remember the LAN party.\nNot a download.\nNot a lobby.\nA pilgrimage.\nYou unplugged your entire desktop — the beige tower, the CRT the size of a microwave, the rat\u0026rsquo;s nest of cables — and you carried it, in your arms, like a wounded soldier, to somebody\u0026rsquo;s basement.\nThe monitor weighed forty pounds.\nYou did this on purpose.\nFor fun.\nAnd when you got there, there was already a folding table, an extension cord doing something it was definitely not rated for, and a bowl of Doritos that would last exactly nine minutes.\nHere\u0026rsquo;s what it felt like as a kid:\nForbidden — six computers in one room felt like a hacker movie Sacred — nobody touched the host\u0026rsquo;s machine. He was the king. Slightly dangerous — that many power bricks in one outlet? bold A little magic — your screen and his screen showing the same world Eternal — it was 2am and nobody\u0026rsquo;s mom had come downstairs yet You\u0026rsquo;d hear someone yell \u0026ldquo;WHO HAS THE GRAY CABLE\u0026rdquo; and the whole night hinged on it.\nThe reveal That gray cable was the whole trick.\nIt was a Cat-5 Ethernet cable, and it ran to a little blinking box in the middle of the table — a hub (later, the fancy kids had a switch). Every computer plugged into it, and that box let your machines gossip with each other at a blistering 10 megabits a second.\nNo internet involved. None. The whole party was a tiny private island of computers — a Local Area Network — talking only to each other.\nBefore someone bought the hub, you did it the cursed way: a coaxial cable daisy-chained machine to machine, with a little metal terminator screwed onto each end. If one person tripped over the line, the entire network died — and everyone knew whose fault it was.\nAnd the game finding everyone? That was your machine literally shouting into the wire — \u0026ldquo;anybody out there hosting?\u0026rdquo; — and a host shouting back. That\u0026rsquo;s why someone always had to \u0026ldquo;make the server.\u0026rdquo; He wasn\u0026rsquo;t being dramatic. He was being the server.\n.------. .------. .------. | PC | | PC | | PC | \u0026#39;--||--\u0026#39; \u0026#39;--||--\u0026#39; \u0026#39;--||--\u0026#39; || || || ===++=========++=========++=== | [ H U B ] .|.|. | \u0026#39;------ blink blink --------\u0026#39; every box hears every box Where it went It never really died — it just stopped weighing forty pounds.\nThat hub in the middle of the table? It moved into the little plastic router blinking in your hallway right now. That \u0026ldquo;shout into the wire to find a host\u0026rdquo;? That\u0026rsquo;s matchmaking, except now the wire is the whole planet and the host is a warehouse in Virginia.\nYou still play with your friends.\nYou just don\u0026rsquo;t carry your computer to do it.\nBut some part of you misses it — the cables, the chaos, the bowl of Doritos, the gray cable that held the whole night together.\nWe didn\u0026rsquo;t have multiplayer.\nWe built it, in a basement, every single time.\n","permalink":"https://errorzap.com/posts/90s-kid-the-lan-party/","summary":"You remember hauling your whole computer across town just to sit in a room and frag your friends.","title":"90s Kid - The LAN Party"},{"content":" ⚡ 90s KID // The CRT and the Static Zap The Glass Box That Bit Back You remember the monitor.\nNot a screen. A monitor.\nA beige cube so deep it needed its own zip code on the desk.\nIt hummed before it showed you anything.\nThat little rising whine — the sound of the thing waking up.\nAnd then the moment.\nYou walked past it, reached out a finger, and —\nZAP.\nA tiny lightning bolt jumped from the glass straight into your knuckle.\nYou weren\u0026rsquo;t even trying to touch it. The monitor reached out and got you first.\nThe kid logic was airtight:\nthe screen was alive it could see you the zap was a warning if you put your face too close you\u0026rsquo;d get radiation (Mom said so) the dust on it was somehow magnetic dust pressing your palm flat on it made your hair stand up like a wizard You\u0026rsquo;d run a comb through your hair, hold it near the glass, and watch the strands lean in.\nForbidden science. No lab coat required.\nAnd the colors. Oh, the colors.\nThat deep, glowing, slightly-curved picture that pulled into the corners like a fishbowl.\nWhen you turned it off, the image collapsed into a single bright dot in the middle —\npew — and then a ghost stayed there for a second, like the screen was holding its breath.\nHere\u0026rsquo;s what was actually going on.\nThat fat box was a Cathode Ray Tube, and it was basically a tiny, tamed particle accelerator pointed at your face.\nAt the back, an electron gun fired a beam of electrons.\nMagnetic coils — the deflection yoke — whipped that beam left-to-right, top-to-bottom, fifty to seventy times a second, painting one glowing line at a time across a phosphor coating on the inside of the glass.\nThe whine you heard? That was the flyback transformer, singing at about 15 kilohertz — right at the edge of what young ears could catch and old ears couldn\u0026rsquo;t. A genuine 90s-kid superpower.\nAnd the zap?\nThe whole front of that tube was carrying a static charge from all those electrons slamming into phosphor. You weren\u0026rsquo;t being warned. You were just the nearest path to ground.\nThe dot when you powered off was the beam collapsing as the magnetics died.\nThe ghost image was phosphor persistence — the glow literally fading.\n____________________________ / /| / .--------------------. / | / | ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ | / | / | ▓ \u0026gt; C:\\\u0026gt;_ ▓ | / | / | ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ | / / / \u0026#39;--------------------\u0026#39; |/ / /___________________________ __/ / | (o) POWER ~~hummmm~~ | / |____________________________|/ ((( zap! ))) -\u0026gt; your knuckle It never really left, you know.\nThat curved-glass glow lives on in every retro filter, every scanline overlay on a streaming game, every \u0026ldquo;CRT mode\u0026rdquo; toggle that fakes the fishbowl on a screen thin as a credit card.\nWe spent twenty years chasing flat, sharp, and silent.\nNow we pay good money for software that puts the fuzz and the curve back in.\nBut the zap?\nThe zap is gone.\nNothing flat ever loved you enough to bite.\n","permalink":"https://errorzap.com/posts/90s-kid-the-crt-and-the-zap/","summary":"You remember the fat glass monitor that hummed, glowed, and bit your knuckle with a tiny lightning bolt.","title":"90s Kid - The CRT and the Static Zap"},{"content":" 💾 90s KID // The Boot Disk The Square That Could Resurrect the Whole Machine You remember the boot disk.\nNot a regular floppy.\nA special one.\nIt lived in a little paper sleeve, or maybe a sandwich bag, somewhere near the computer.\nSometimes it had a label written in marker.\nSometimes the label just said \u0026ldquo;BOOT — DO NOT ERASE.\u0026rdquo;\nAnd you believed it.\nYou did NOT erase it.\nTo us, this thing had powers.\nThe vibes:\nIt was the disk you only touched in an emergency Dad slid it in when the computer \u0026ldquo;wouldn\u0026rsquo;t go\u0026rdquo; The screen would go black and serious and full of words It smelled faintly like plastic and authority Touching the shiny brown circle inside = forbidden, basically a crime It made the drive go grrr-CHUNK-grrr like it was thinking really hard It felt less like a tool and more like a magic key.\nThe computer was a sleeping dragon, and this little square was the only thing that knew its true name.\nHere\u0026rsquo;s what it actually was.\nA computer can\u0026rsquo;t pull itself up by its own bootstraps from nothing — it needs the tiniest starter program to teach it how to load everything else. That\u0026rsquo;s literally where \u0026ldquo;booting\u0026rdquo; comes from.\nThat special floppy held a minimal copy of DOS: just enough operating system to wake the machine, talk to the keyboard, see the drives, and give you a C:\\\u0026gt; prompt.\nWhen the hard drive got corrupted, or a nasty boot-sector virus moved in, or Windows face-planted on startup — the hard drive couldn\u0026rsquo;t boot itself anymore.\nSo you fed the computer a clean brain from the A: drive instead.\nFrom that humble prompt, the grown-ups ran the real magic spells:\nscandisk. fdisk. format. sys c:.\n640 KB of pure \u0026ldquo;I will fix you whether you like it or not.\u0026rdquo;\n_________________________ | _____________________ | | | BOOT - DO NOT ERASE| | | |_____________________| | | ___________________ | | |::| | | | | |::| A:\\\u0026gt;_ | | | | |::|_____________ | | | |_______________________ _| | __ |##############| | | | ||##############| | | |__||##############| | |_______________________ _| [ 3.5\u0026#34; 1.44 MB ] The boot disk never really died — it just stopped being a square.\nThat clean little rescue brain became your Windows recovery USB.\nIt became F8, and Safe Mode, and \u0026ldquo;Startup Repair.\u0026rdquo;\nIt became the install media you make when a laptop won\u0026rsquo;t wake up.\nEvery time you\u0026rsquo;ve held a key during startup to summon a menu out of a dead machine, you were doing the exact same ritual.\nSame dragon.\nSame magic key.\nIt just fits in your pocket now — and nobody tells you not to touch the shiny part.\n","permalink":"https://errorzap.com/posts/90s-kid-the-boot-disk/","summary":"It was a chunky little square of plastic that decided whether the computer would even wake up that day.","title":"90s Kid - The Boot Disk"},{"content":" The backup was \u0026ldquo;running.\u0026rdquo; It had been running for nine hours.\nA client\u0026rsquo;s nightly Kopia job pointed at a user\u0026rsquo;s documents folder. Normally it finished in twenty minutes. This morning it was still chewing: no error, no progress, just a cursor blinking next to a snapshot that had hashed maybe a few hundred files and then gone silent.\nNo CPU. No disk I/O to speak of. No network throughput. Just a process sitting there, perfectly calm, doing absolutely nothing while claiming to be busy.\nThat\u0026rsquo;s the worst kind of hang. A crash leaves a body. This left a process that would happily wait until the heat death of the universe.\nThe investigation First instinct: blame the repository. Maybe the backend was wedged, maybe a lock was stuck. Nope. A fresh kopia snapshot create against a different folder ran clean in seconds.\nSo it wasn\u0026rsquo;t Kopia. It was this folder.\nI watched the process with Resource Monitor and caught it: every time the scan touched a particular file, the thread parked in a wait state. Not reading. Not erroring. Waiting on the filesystem to hand back bytes that never came.\nThen I actually looked at the folder.\n┌────────────────────────────────────────────────┐ │ C:\\Users\\user\\Documents (OneDrive-synced) │ ├──────────────┬─────────────────────────────────┤ │ Q1-report │ ✓ green check (on this PC) │ │ budget.xlsx │ ✓ green check (on this PC) │ │ archive\\ │ ☁ cloud icon (online-only) │ ◄── placeholder │ photos\\ │ ☁ cloud icon (online-only) │ ◄── placeholder └──────────────┴─────────────────────────────────┘ the cloud-icon files have no bytes on disk Half the tree had little cloud icons instead of green checks. OneDrive Files On-Demand. Those files looked real — name, size, timestamp — but the actual content lived in the cloud. On disk they were placeholders.\nThe \u0026ldquo;aha\u0026rdquo; OneDrive\u0026rsquo;s on-demand placeholders are implemented with the Windows cloud files filter driver, cldflt. When any process opens a placeholder and tries to read it, cldflt intercepts the read and hydrates the file — downloads it from the cloud, transparently, before returning a single byte.\nA backup tool is the most naive reader on earth. It opens every file and reads it end to end. So Kopia walks into a folder full of placeholders and triggers a hydration storm: thousands of files getting pulled down on demand, throttled, retried, and in some cases just\u0026hellip; stalling. The read call blocks. The thread parks. Forever.\nThe backup wasn\u0026rsquo;t broken. It was politely waiting for the entire cloud to fall back onto the disk, one file at a time, through a sync client that had no idea it was under attack.\nKopia read() cldflt hydrate... cloud (stalls) x blocks forever Kopia VSS shadow stable view ok: no hydration Reading the live folder triggers cldflt hydration and hangs; reading a VSS shadow skips it entirely. The fix You don\u0026rsquo;t want to read the live folder. You want to read a snapshot of it — a frozen, point-in-time view that the cloud filter doesn\u0026rsquo;t get to intercept. That\u0026rsquo;s exactly what a Volume Shadow Copy gives you. The VSS snapshot presents the volume\u0026rsquo;s state as committed bytes, with no on-demand hydration path.\nKopia has hooks for precisely this: run an action before it walks the source root, and another after. Create the shadow before, tear it down after.\n# before-snapshot: create a shadow copy and expose it via symlink $vol = (vssadmin create shadow /for=C: | Select-String \u0026#39;Shadow Copy Volume Name\u0026#39;).ToString().Split(\u0026#39; \u0026#39;)[-1] $env:KOPIA_SNAPSHOT_PATH = $vol cmd /c \u0026#34;mklink /d C:\\kopia-shadow $vol\\\u0026#34; # after-snapshot: clean up the link and drop the shadow cmd /c \u0026#34;rmdir C:\\kopia-shadow\u0026#34; vssadmin delete shadows /for=C: /oldest /quiet Then point the source at the shadow instead of the live path and wire the hooks in:\nkopia snapshot create C:\\kopia-shadow\\Users\\user\\Documents \\ --before-snapshot-root-action=\u0026#34;powershell -File C:\\scripts\\vss-before.ps1\u0026#34; \\ --after-snapshot-root-action=\u0026#34;powershell -File C:\\scripts\\vss-after.ps1\u0026#34; The next run finished in eighteen minutes. Every placeholder read off the shadow returned instantly, because the snapshot view doesn\u0026rsquo;t route through cldflt. No hydration, no stalls, no nine-hour ghost.\nWhy it happened OneDrive Files On-Demand exists so users don\u0026rsquo;t sync 400GB of vacation photos onto a 256GB laptop. Great feature for humans. Terrible surprise for any tool that assumes \u0026ldquo;the file is on this disk.\u0026rdquo;\nKopia did nothing wrong. It read files. The trap is that on a cloud-synced volume, \u0026ldquo;open and read\u0026rdquo; is a network operation in disguise, gated by a filter driver that will block your thread while it phones home. Backups that walk thousands of placeholders turn into a self-inflicted denial of service.\nSnapshotting through VSS sidesteps the whole mess, and as a bonus you get a consistent point-in-time copy instead of backing up a folder that\u0026rsquo;s mutating under you.\nTakeaways Cloud on-demand files are placeholders, not files. OneDrive, Dropbox Smart Sync, and friends all use a filter driver (cldflt on Windows) that hydrates on read. A naive reader hangs on hydration. Backups, indexers, and AV scanners that open every file will trigger downloads — and stall when the cloud throttles or times out. Back up from a VSS shadow, not the live tree. The snapshot view bypasses the cloud filter and returns committed bytes instantly. Use Kopia\u0026rsquo;s before/after root-action hooks to create and tear down the shadow around each run — no manual babysitting. A backup at \u0026ldquo;0% for hours\u0026rdquo; with no I/O isn\u0026rsquo;t slow, it\u0026rsquo;s blocked. Watch the thread state; a parked read on a cloud volume is the tell. ","permalink":"https://errorzap.com/posts/the-backup-that-hung-forever-on-onedrive/","summary":"A Kopia job sat at 0% all night because it tried to read files that didn\u0026rsquo;t actually exist on disk yet.","title":"The Backup That Hung Forever on OneDrive"},{"content":" 🍞 90s KID // Flying Toasters and the Screen Sav The Toasters That Guarded the Machine You remember the flying toasters.\nYou weren\u0026rsquo;t even using the computer.\nNobody was.\nIt just sat there in the den, humming, and then — winged toasters.\nFlapping. Drifting. With toast.\nCruising across a black void like tiny chrome angels carrying breakfast to heaven.\nAnd you would just\u0026hellip; watch it.\nFor way too long.\nHere\u0026rsquo;s what we knew about it as kids:\nIt only showed up when everybody left It meant the computer was asleep but still alive Touching the mouse made it vanish, which felt like waking something The toast was non-negotiable. The toast was sacred. Your friend\u0026rsquo;s house had a flying-pipes maze instead, and that felt fancier somehow It wasn\u0026rsquo;t a program you opened.\nIt was a thing the machine became when nobody was looking.\nThat was the magic part.\nThe computer had a secret life and the toasters were proof.\nHere\u0026rsquo;s what was actually going on.\nBack then, your monitor was a CRT — a big heavy glass tube that fired electron beams at a phosphor coating to make the picture glow.\nAnd those phosphors had a problem: if the exact same image sat on the screen too long, it could literally burn in. A ghost of that image would stain the glass forever. A spreadsheet grid. A toolbar. A menu bar, etched into the dark like a fossil.\nSo the screen saver was doing exactly what the name says — saving the screen.\nMove the pixels around constantly, never let one spot sit still, and nothing gets burned. The toasters weren\u0026rsquo;t decoration.\nThey were a tiny patrol.\nKeeping the picture moving so the glass stayed clean.\nThe famous flapping breakfast came from a software company called Berkeley Systems, in a pack named After Dark. They even got sued once over those toasters, which is the most 90s sentence ever written.\n.-----------. / o o o \\ ~ ~ ~ | [=========] | o o | | T O A S | | flap \\ | T ~ T | / flap `-----------\u0026#39; \\\\ // --\\\\-//-- pop! 🍞 `-\u0026#39;-\u0026#39; the patrol is now airborne The funny part? Screen burn-in basically went away.\nFlat LCDs didn\u0026rsquo;t have those needy phosphors, so the danger faded.\nThe toasters had won. They could rest.\nExcept they never actually left.\nYour phone dims and floats a clock around the lock screen.\nYour TV nudges its glowing logo a few pixels every minute — because OLEDs can burn in, so we quietly reinvented the whole idea.\nYour work laptop snaps to a slideshow of blurry photos the second you stand up.\nWe don\u0026rsquo;t call it a screen saver anymore.\nBut every time a screen drifts gently to keep itself from getting hurt —\nthat\u0026rsquo;s a toaster.\nStill flapping.\nStill carrying the toast.\n","permalink":"https://errorzap.com/posts/90s-kid-flying-toasters/","summary":"You walked away from the computer for ten minutes and came back to find it had grown wings.","title":"90s Kid - Flying Toasters and the Screen Saver"},{"content":" Someone pinged me with the kind of bug that makes your skin crawl: \u0026ldquo;I save a note, it saves fine, no error. But when I come back later it\u0026rsquo;s gone. And sometimes I open a file and it\u0026rsquo;s the wrong file entirely.\u0026rdquo;\nNo errors. No crash. Data just\u0026hellip; wrong. The worst kind of ticket — because \u0026ldquo;wrong data, no error\u0026rdquo; means the system is lying to you with a straight face.\nI assumed corruption. Maybe a bad sync, a stale cache, a half-written file on disk. I was wrong about all of it.\nThe investigation First thing I did was stop trusting the app and start trusting the wire. The app talked to itself over a local REST API on a fixed port. So I asked the only question that matters when data is \u0026ldquo;wrong\u0026rdquo;:\nAm I even talking to the thing I think I\u0026rsquo;m talking to?\nThe user ran two separate profiles of the same application — two distinct workspaces, two configs, two processes. Different data, different windows. But same binary, same defaults.\nI checked what was actually listening.\nProfile A (started first) Profile B (started second) ┌──────────────────────┐ ┌──────────────────────┐ │ app instance A │ │ app instance B │ │ workspace: WORK │ │ workspace: PERSONAL │ └──────────┬───────────┘ └──────────┬───────────┘ │ bind :8200 OK │ bind :8200 FAIL ▼ ▼ ┌─────────────┐ (silent: port taken) │ TCP :8200 │ ◄──── requests land HERE └─────────────┘ regardless of which UI you used ▲ │ edits in B\u0026#39;s UI → API call → answered by A Both instances were configured to expose their REST API on the same hard-coded port. Whichever process won the startup race grabbed the socket. The loser tried to bind, failed, and — because the failure was non-fatal — just shrugged and kept running its UI as if nothing happened.\nSo you\u0026rsquo;d type into Profile B\u0026rsquo;s window. The UI would fire an API write. That write went to port 8200, which Profile A owned. Profile A happily saved it into the wrong workspace. Reads came back from A too. Profile B\u0026rsquo;s UI was a puppet with cut strings.\nThe \u0026ldquo;aha\u0026rdquo; The tell was netstat. One PID owned the port. Not two. Not a conflict logged anywhere — just one quiet winner.\n# Windows: who actually holds the port? netstat -ano | findstr :8200 # TCP 127.0.0.1:8200 0.0.0.0:0 LISTENING 18044 tasklist /fi \u0026#34;PID eq 18044\u0026#34; # app.exe 18044 ... \u0026lt;- this is Profile A, not B # Linux/macOS equivalent lsof -nP -iTCP:8200 -sTCP:LISTEN ss -ltnp \u0026#39;sport = :8200\u0026#39; One listener. Two apps. The data wasn\u0026rsquo;t corrupt — it was just going to the wrong house, and the mailman never complained.\nProfile A workspace: WORK Profile B workspace: PERSONAL :8200 one listener bind OK bind FAIL → routes to A Both UIs wired to one fixed port. The first process to start owns it; the second silently misroutes. The fix Stop them from racing for the same socket. Give each instance its own port.\n# 1. Edit each profile\u0026#39;s config — distinct ports per instance # Profile A \u0026#34;api\u0026#34;: { \u0026#34;enabled\u0026#34;: true, \u0026#34;port\u0026#34;: 8200 } # Profile B \u0026#34;api\u0026#34;: { \u0026#34;enabled\u0026#34;: true, \u0026#34;port\u0026#34;: 8201 } # 2. Fully stop both instances (no orphan holding the old socket) taskkill /f /im app.exe # Windows # pkill -f app # Linux/macOS # 3. Confirm the ports are actually free before restart netstat -ano | findstr \u0026#34;8200 8201\u0026#34; # expect: nothing # 4. Start both, then verify ownership — two PIDs, two ports netstat -ano | findstr \u0026#34;8200 8201\u0026#34; # TCP 127.0.0.1:8200 LISTENING 20110 \u0026lt;- Profile A # TCP 127.0.0.1:8201 LISTENING 20338 \u0026lt;- Profile B Two listeners, two PIDs, two ports. Writes to B stayed in B. Reads told the truth. The \u0026ldquo;disappearing edits\u0026rdquo; were gone because they\u0026rsquo;d never disappeared — they\u0026rsquo;d just been filed in the wrong cabinet.\nWhy it happened Defaults. The app shipped with a single hard-coded API port and assumed exactly one instance would ever run. Reasonable for the common case, fatal for power users running multiple profiles. There was no port-conflict warning because a failed bind on a convenience API isn\u0026rsquo;t fatal — the app degrades silently instead of refusing to start. Silent degradation is how a config bug masquerades as data loss.\nTakeaways When data looks \u0026ldquo;wrong\u0026rdquo; but nothing errors, question the endpoint first. Confirm you\u0026rsquo;re talking to the instance you think you are before you go disk-spelunking for corruption. netstat -ano / lsof -i is ground truth for \u0026ldquo;who owns this port.\u0026rdquo; One listener where you expected two is the whole bug, right there. Multiple instances of the same app will knife-fight over hard-coded ports. Give every instance a unique port in its config — don\u0026rsquo;t trust defaults to coexist. A non-fatal bind failure is a trap. Silent degradation turns a 5-minute config fix into a week of phantom data-loss tickets. Pre-flight before restart: kill all instances, verify the port is actually free, then start — and read back the listeners to prove it stuck. ","permalink":"https://errorzap.com/posts/two-apps-one-port/","summary":"Edits vanished, reads lied, and nothing in the logs was wrong — because two copies of the same app were quietly knife-fighting over the same TCP port.","title":"Two Apps, One Port"},{"content":" 🔌 90s KID // The Surge Protector Everything Plu The Beige Bar That Held Up The Whole House You remember the surge protector.\nThe long beige slab that lived on the floor behind the desk.\nThe one with the orange glow at one end.\nThe one nobody was allowed to step on.\nIt hummed in a way you could feel more than hear.\nAnd it had the switch. That big rocker with the tiny red light, the one that turned the entire family computer empire on and off in a single satisfying clunk.\nYou didn\u0026rsquo;t know what it did.\nYou just knew it was important.\nThe vibes were:\nForbidden. Dad said never unplug it. So you never unplugged it. A little dangerous. It said WARNING on it. In yellow. With a lightning bolt. Overloaded. Six outlets, somehow eleven things plugged in, a tangle like a nest of beige snakes. The boss of everything. Modem, monitor, the printer that screamed, the tower, the speakers with the subwoofer that lived under the desk. All of it answered to the bar. Magic. Press the switch and the whole room woke up. CRTs ticked, fans spun, the modem blinked. You were a wizard. It was, honestly, the closest thing your bedroom had to a circuit breaker, and you were eight, and you respected it.\nHere\u0026rsquo;s what it actually was.\nInside that beige slab was a tiny, heroic component called a MOV — a metal oxide varistor.\nMost of the time it does nothing. It just sits there. But the moment the voltage from the wall spikes — a lightning strike down the block, the AC compressor kicking on, the fridge cycling — the MOV\u0026rsquo;s resistance suddenly drops and it shunts that extra energy away from your computer before it can fry the motherboard.\nIt\u0026rsquo;s a sacrificial guard. It literally throws itself on the grenade.\nThat\u0026rsquo;s also why old surge protectors quietly stop protecting. Each spike wears the MOV down a little. The strip still passes power, the light still glows — but the protection is long dead. It became a fancy extension cord and nobody told you.\nThe glowing light? Often just \u0026ldquo;there is power here.\u0026rdquo; Not \u0026ldquo;you are safe.\u0026rdquo;\nThe fancier ones had a second light — \u0026ldquo;PROTECTED\u0026rdquo; / \u0026ldquo;GROUNDED\u0026rdquo; — and that one going dark was the part that actually mattered.\n___________________________________________ | o [|] [|] [|] [|] [|] [|] (~) | \u0026lt;- the glowing switch | o | | | | | | POWER | |_______|___|___|___|___|___|________________| | | | | | | modem mon tower spkr prnt ...and one mystery plug \\__ J O I N E D F O R E V E R __/ The funny thing is, it never left.\nIt\u0026rsquo;s still down there behind your desk right now. Different shape, same job. Maybe it grew a coax port for the cable line, or a couple USB-A holes, or a little app that emails you when a fuse pops.\nWe just stopped calling it the boss.\nBut every time your whole battlestation winks on with one press — the monitors, the dock, the mechanical keyboard with its own light show — that\u0026rsquo;s the same beige magic.\nStill sitting on the floor.\nStill quietly ready to take the hit for you.\nYou just don\u0026rsquo;t say thank you anymore.\n","permalink":"https://errorzap.com/posts/90s-kid-the-surge-protector/","summary":"You remember the long beige bar on the floor with the little glowing button, the one that ran the entire family universe.","title":"90s Kid - The Surge Protector Everything Plugged Into"},{"content":" The bot worked. Then it didn\u0026rsquo;t. Then it worked again.\nIt was a small Node.js Telegram bot running on a client\u0026rsquo;s server. Its whole job was to fire off outbound API calls — sendMessage, getUpdates, the usual. Most of the time it was instant. But every so often a call would just\u0026hellip; stop. No exception. No 500. No DNS error. The promise sat there until the request timeout finally put it out of its misery seconds later.\nIntermittent. Silent. Multi-second. The worst kind of bug there is.\nThe investigation First instinct was the obvious one: the upstream API is flaky. So I logged timing around every call. The hangs weren\u0026rsquo;t correlated with anything — not load, not time of day, not a specific endpoint. Same call, same payload, same host. Sometimes 80ms, sometimes 5,000ms-then-timeout.\nI pulled strace on the process during a hang and watched the syscalls. The bot wasn\u0026rsquo;t busy. It was sitting in a connect() that never came back. It wasn\u0026rsquo;t waiting on the API — it was waiting on a socket that was never going to open.\nThat reframed everything. This wasn\u0026rsquo;t a slow server. This was a connection attempt going into the void.\nbot ──► undici (happy-eyeballs) │ ├─► resolve host │ ├─ AAAA (IPv6) ─► :: ► connect() ──► ✗ black hole (no route) │ └─ A (IPv4) ─► . ► connect() ──► ✓ would work fine │ ▼ races both... but stalls on the v6 attempt The host had IPv6 addresses configured on its primary NIC. It just had no working IPv6 route to the outside world. Addresses present, connectivity dead. The packets went out and nothing ever came back.\nThe \u0026ldquo;aha\u0026rdquo; The bot used undici as its HTTP client. Modern undici does Happy Eyeballs (RFC 8305) — it resolves both A and AAAA records and races IPv4 and IPv6 connection attempts, preferring the v6 path.\nOn a healthy host that\u0026rsquo;s great. On this host it was poison. undici would get an AAAA record, try the IPv6 path first, and that connect() would sail into a black hole. The race timer is supposed to fall back to IPv4 — but with a route that silently drops packets instead of cleanly rejecting them, the fallback timing slipped and the whole call stalled until the app-level timeout.\nA quick curl proved it cold:\n# hangs forever — there\u0026#39;s an AAAA record and broken v6 routing curl -v https://api.example.com/ # instant — forced down IPv4 curl -4 -v https://api.example.com/ -6 hung. -4 flew. That\u0026rsquo;s your smoking gun.\nthe bot AAAA / IPv6 black hole ✗ A / IPv4 API ✓ Happy Eyeballs races both paths — but a route that drops packets stalls the v6 attempt instead of failing fast. The fix Two layers. First, kill IPv6 on the one NIC that had it broken — while leaving it alone on the VPN and VM interfaces that genuinely use it.\n# disable IPv6 on the offending interface only (e.g. eth0), keep it elsewhere sudo sysctl -w net.ipv6.conf.eth0.disable_ipv6=1 # make it stick across reboots echo \u0026#39;net.ipv6.conf.eth0.disable_ipv6 = 1\u0026#39; | sudo tee /etc/sysctl.d/99-disable-ipv6-eth0.conf sudo sysctl --system Second, belt and suspenders at the app layer — force the resolver IPv4-first so even a half-broken host can\u0026rsquo;t bite us again:\n// top of the entrypoint, before any outbound calls const dns = require(\u0026#39;node:dns\u0026#39;) dns.setDefaultResultOrder(\u0026#39;ipv4first\u0026#39;) curl -4 already proved the IPv4 path was healthy, so this was a safe, deterministic pin. Restarted the bot, hammered it with calls for ten minutes. Zero hangs. Every request back under 200ms.\nWhy it happened \u0026ldquo;Broken but present\u0026rdquo; IPv6 is worse than no IPv6 at all. If the host had no AAAA-resolvable path, the client never tries v6 and you never notice. But give it IPv6 addresses with no real route, and every modern HTTP client cheerfully tries the dead path first — then waits on a connect() that silently drops instead of refusing.\nNo RST means no fast failure. No fast failure means a stall. And because Happy Eyeballs only sometimes loses the race badly, the stall is intermittent. That\u0026rsquo;s why it looked like a flaky upstream when it was a flaky local NIC the whole time.\nTakeaways If something hangs only sometimes on outbound HTTP, suspect IPv6. Test it in one command: curl -4 vs curl -6. If -6 hangs, you found it. Addresses ≠ connectivity. A NIC can hold a perfectly valid IPv6 address and still have zero working route to the internet. Disable IPv6 per-interface, not globally. Kill it on the broken NIC; leave it on VPN/VM links that actually need it. Pin IPv4-first at the app layer too (dns.setDefaultResultOrder('ipv4first')) — defense in depth for hosts you don\u0026rsquo;t fully control. A connect() that never returns is a routing problem, not a server problem. strace the hang before you blame the API. ","permalink":"https://errorzap.com/posts/the-bot-that-hung-because-of-ipv6/","summary":"A Telegram bot kept freezing on outbound calls with no error — turns out a NIC with IPv6 addresses but no working IPv6 routing was quietly killing every request.","title":"The Bot That Hung Because of IPv6"},{"content":" There\u0026rsquo;s a box at a remote site. It lives behind NAT, behind a router I don\u0026rsquo;t control, on a connection that hands out a new public IP whenever the ISP feels like it.\nTo reach it, I had a ritual. SSH into a bastion. From the bastion, SSH into the target. Type two passphrases, pray nothing changed, and hope the WAN IP I memorized last month was still the WAN IP.\nssh -J bastion target That one flag hid a small empire of fragility.\nThe investigation It broke on a Tuesday, the way these things do. ssh: connect to host ... port 22: Connection timed out.\nFirst instinct: the target is down. It wasn\u0026rsquo;t — a colleague on-site confirmed it was humming along, recording, serving, fine.\nSecond instinct: the bastion is down. It wasn\u0026rsquo;t either. I could land on the bastion clean.\nssh bastion \u0026#39;echo alive\u0026#39; alive So the bastion was up, the target was up, and the hop between them was dead. I jumped onto the bastion and tried the second leg by hand.\nssh me@198.51.100.x ssh: connect to host 198.51.100.x port 22: No route to host There it was. The remote site\u0026rsquo;s WAN IP had rotated overnight. The 198.51.100.x baked into my SSH config — and into the bastion\u0026rsquo;s known_hosts, and into a port-forward rule, and into my muscle memory — now pointed at some stranger\u0026rsquo;s DHCP lease.\nThe \u0026ldquo;aha\u0026rdquo; I\u0026rsquo;d built a chain of three brittle links to solve one problem: the target has no stable address I can reach from outside.\nEvery fix I\u0026rsquo;d ever applied — dynamic DNS, a forwarded port, a second hop — was just another way of chasing an address that refused to hold still. The bastion wasn\u0026rsquo;t a solution. It was a workaround for a missing fact.\nWhat if the box just had a stable address? One that didn\u0026rsquo;t care about NAT, ISP whims, or which coffee shop my laptop was sitting in?\nBEFORE AFTER ┌─────────┐ ┌─────────┐ │ laptop │ │ laptop │ └────┬────┘ └────┬────┘ │ ssh │ ssh ▼ │ (encrypted mesh) ┌─────────┐ │ │ bastion │ ◄── public IP that │ └────┬────┘ keeps rotating │ │ ssh ▼ ▼ ┌─────────┐ ┌─────────┐ NAT / CGNAT │ target │ │ target │ no inbound port │ 100.x.. │ stable tailnet IP └─────────┘ └─────────┘ That address is exactly what a WireGuard-based mesh VPN hands you. Put the laptop and the target on the same tailnet, and each device gets a private IP that follows it everywhere — through NAT, through CGNAT, through IP rotations — because the mesh handles NAT traversal for you.\nThe fix Install the mesh on both ends. (Generic Tailscale-style flow below.)\n# On the target — behind NAT, no inbound ports needed curl -fsSL https://tailscale.com/install.sh | sh sudo tailscale up --ssh --hostname remote-target # On my laptop tailscale up # Confirm the target shows a stable tailnet IP tailscale status # 100.x.y.z remote-target linux active; direct Now SSH goes straight there. No bastion, no -J, no rotating address.\nssh me@100.x.y.z # or, by MagicDNS name ssh me@remote-target Lock it down with an ACL so only my devices can reach port 22 on that box:\n{ \u0026#34;acls\u0026#34;: [ { \u0026#34;action\u0026#34;: \u0026#34;accept\u0026#34;, \u0026#34;src\u0026#34;: [\u0026#34;tag:admin\u0026#34;], \u0026#34;dst\u0026#34;: [\u0026#34;tag:remote:22\u0026#34;] } ] } Then I deleted the bastion\u0026rsquo;s forward rule, pulled the stale host key, and retired the jump chain entirely.\nssh-keygen -R 198.51.100.x one tailnet, every device gets a stable address laptop 100.64.0.2 target 100.64.0.7 NAT / CGNAT WireGuard tunnel · ACL-gated The hop is gone. The tunnel goes straight through NAT, ACL-gated, end to end. Why it happened The bastion was never the point. It existed because the target had no reachable, durable address — so I borrowed one from a machine that did, then chained a second SSH session on top.\nEvery link in that chain depended on a public IP staying put. One of them rotated, and the whole thing collapsed. That\u0026rsquo;s not a bug in SSH. That\u0026rsquo;s the architecture telling you it was held together with assumptions.\nA mesh VPN moves the addressing problem off the public internet entirely. The tailnet IP is yours, it\u0026rsquo;s stable, and it doesn\u0026rsquo;t care what the ISP does at 3 a.m.\nTakeaways If you\u0026rsquo;re chaining ssh -J to reach a NAT\u0026rsquo;d box, that\u0026rsquo;s a missing stable address, not a routing puzzle. Fix the address, delete the chain. Mesh VPNs (WireGuard-based) give every device a durable private IP that survives NAT, CGNAT, and ISP rotations — no inbound ports to forward. Stable addressing kills a whole class of failures: stale known_hosts, dead port-forwards, dynamic-DNS lag, memorized WAN IPs. Gate it with ACLs. A flat tailnet where everything reaches everything is just a bigger blast radius. Tag, scope, restrict to port 22. The most reliable hop is the one you deleted. Fewer moving parts, fewer 3 a.m. timeouts. ","permalink":"https://errorzap.com/posts/how-a-mesh-vpn-killed-my-ssh-jump-host/","summary":"The bastion broke every time an IP changed, so I deleted the bastion.","title":"How a Mesh VPN Killed My SSH Jump Host"},{"content":" 📟 90s KID // The Pager The Little Black Box That Owned The Grown-Ups You remember the pager.\nClipped to a belt.\nHooked to a waistband.\nRiding shotgun in a fanny pack like it paid rent.\nIt was small. It was plastic. It was, somehow, the most important object in the house.\nWhen it buzzed, the room changed.\nDad would tilt it toward the light, squint at a row of numbers, and go \u0026ldquo;hm.\u0026rdquo;\nJust \u0026ldquo;hm.\u0026rdquo;\nAnd then he\u0026rsquo;d disappear to the kitchen phone.\nYou didn\u0026rsquo;t know what it did. Not really.\nHere\u0026rsquo;s what you knew:\nIt only spoke in numbers, and the numbers were a secret You were not allowed to touch it It made a noise that meant the day was about to change Important people had one. Doctors. Drug dealers on TV. Your uncle, for some reason. Sometimes it said 07734 and that was the funniest thing humans had ever done It felt like a tiny walkie-talkie that only listened.\nA beeper that knew things you didn\u0026rsquo;t.\nA pocket oracle that whispered to adults in a code you\u0026rsquo;d crack someday.\nYou\u0026rsquo;d press the little button just to watch the screen light up green.\nThen put it down fast, like it was hot.\nHere\u0026rsquo;s what it actually was.\nA pager was a one-way radio receiver. That\u0026rsquo;s it. No screen full of apps. No internet. Just a dumb, brilliant little antenna listening for its own number on a wide radio frequency, broadcast from tall towers across the whole region.\nSomebody dialed the pager\u0026rsquo;s phone number, typed digits, and a paging network flung that message out over the air to every tower at once. Your dad\u0026rsquo;s beeper heard its unique address in the stream, grabbed the digits meant for it, and buzzed.\nThe \u0026ldquo;code\u0026rdquo;? It was just a callback phone number. Sometimes with a 911 tacked on for \u0026ldquo;call me NOW.\u0026rdquo;\nAnd it worked when nothing else did. Pagers ran on a frequency that soaks into buildings where cell signal still face-plants to this day. One AA battery. Weeks of standby. Basically unkillable.\nIt wasn\u0026rsquo;t magic.\nIt was just really, really good engineering wearing a tiny green screen.\n________________________ | .------------------. | | | | | | | 1-555-04 07734 | | | | *911* | | | |____________________| | | [MENU] [READ] | | .__. o o o .__. | |__/====\\___________/===\\_| BEEP · BEEP · BEEP The pager never really died. It just took off the belt and moved into your pocket.\nThat buzz against your leg? Still there. Every text, every DM, every push notification is a pager alert with a marketing budget. Hospitals still run pagers, because when the network\u0026rsquo;s on fire, the dumb little radio is the one thing that still gets through.\nSo next time your phone buzzes and your whole face changes —\njust know your dad did that first.\nWith a beeper.\nAnd a callback number that probably said 07734.\n","permalink":"https://errorzap.com/posts/90s-kid-the-pager/","summary":"You remember the little black box on your dad\u0026rsquo;s belt that buzzed once and changed his whole face.","title":"90s Kid - The Pager"},{"content":" The call came in the way these always do: \u0026ldquo;the whole network is slow.\u0026rdquo;\nNot down. Slow. Which is worse, because slow means it\u0026rsquo;s still trying.\nI pulled up the switch stats from the site and the picture was ugly. Broadcast counters spinning like a gas pump. Port LEDs across the access switches strobing in unison — that synchronized, hypnotic blink that should make the hair on your neck stand up. CPU on the core switch pinned. Every wired device crawling, half the VLANs timing out.\nThat\u0026rsquo;s not congestion. That\u0026rsquo;s a storm.\nThe investigation A broadcast storm has a signature. One frame gets flooded out every port, comes back in another port, gets flooded again, forever, at line rate. The switches aren\u0026rsquo;t broken — they\u0026rsquo;re doing exactly what they\u0026rsquo;re told, infinitely.\nSo the question is never \u0026ldquo;what\u0026rsquo;s slow.\u0026rdquo; It\u0026rsquo;s \u0026ldquo;where\u0026rsquo;s the loop.\u0026rdquo;\nI started where loops live: the edge. I pulled the MAC address table and watched a handful of addresses flapping between ports — the same MAC showing up on port 14, then port 9, then port 14, dozens of times a second. A stable MAC doesn\u0026rsquo;t teleport. A looped one does.\nshow mac address-table | the same MAC, two ports, flapping ───────────────────────────────────────────────────────── 02:00:00:aa:bb:cc Gi0/14 ◄─┐ 02:00:00:aa:bb:cc Gi0/9 ◄─┤ same address, 02:00:00:aa:bb:cc Gi0/14 ◄─┤ two ports, 02:00:00:aa:bb:cc Gi0/9 ◄─┘ ~40x/sec ✗ I traced both ports. Port 14 and port 9 went to two units of a consumer multi-room speaker system — the kind that builds its own little wireless mesh between speakers so they stay in sync.\nAnd somebody had wired both of them into the LAN.\nThe \u0026ldquo;aha\u0026rdquo; There it was. Each speaker had an Ethernet drop. Each speaker was also bridging to its sibling over the proprietary wireless mesh. So a frame could leave the switch on port 14, ride into speaker A, hop the wireless mesh to speaker B, and come right back into the switch on port 9.\nCopper out, radio across, copper back in. A loop with one leg made of Wi-Fi.\nSwitch Speaker A Speaker B copper copper wireless mesh ► layer-2 loop ◄ — frames circulate forever Two wired speakers, one wireless bridge between them: a loop the switch can't see coming. The reason it had run fine for months and then exploded? Doesn\u0026rsquo;t matter much. Someone re-cabled a closet, both speakers ended up plugged in, the mesh did its job, and nothing on those access ports was watching for a loop.\nThe fix The speaker was the trigger. The real bug was a switch fabric with no loop protection. So we fixed both.\nFirst, the immediate bleeding — pull one of the two speaker uplinks. One leg of copper gone, loop broken, network breathes.\nThen, the actual fix. Turn on spanning tree, and pick the root on purpose instead of letting MAC addresses elect a random one:\n# Core switch — make it the root, deliberately. Lowest priority wins. spanning-tree mode rstp spanning-tree vlan 1-4094 priority 4096 # Distribution — second in line spanning-tree vlan 1-4094 priority 8192 Then lock down the edge so an end-device port can never become a transit path:\n# Access port to the speaker / any end device interface GigabitEthernet0/14 spanning-tree portfast spanning-tree bpduguard enable # any BPDU here = err-disable the port spanning-tree guard root # this port may never be a path to root storm-control broadcast level 1.00 # cap broadcast at 1% of line rate storm-control action shutdown Mesh-style managed gear is the same idea in different clothes — flip on STP, set tiered priorities so the core is root, and enable Loop Protection plus BPDU Guard on access ports.\nedge port discipline ───────────────────────────────── PortFast ✓ fast end-device link BPDU Guard ✓ sees a switch → kills port Loop Guard ✓ detects the loop Storm-control ✓ broadcast ceiling ───────────────────────────────── result: ▓▓▓▓▓ blast radius = one port We re-cabled the speakers to a single drop each, verified one stayed on copper and the rest synced over mesh, and watched the broadcast counter flatline.\nWhy it happened Consumer mesh gear is a loop machine wearing a friendly face. It will happily bridge two of its own units, and it has no idea it\u0026rsquo;s also sitting on your switched LAN. Plug two of them into copper and you\u0026rsquo;ve handed your network a second path it never agreed to.\nThe switch didn\u0026rsquo;t fail. It had no spanning tree configured, so it had no mechanism to notice the loop and break it. The default config trusted every port to behave. Edge ports never behave.\nTakeaways Always run STP/RSTP. A modern loop-free topology still needs spanning tree on standby for the day someone hands you a loop. Set the root bridge on purpose. Lowest priority value wins; don\u0026rsquo;t let a random MAC address decide where your traffic converges. Guard the edge. BPDU Guard, Loop Protection, and storm-control on access ports turn a network-wide meltdown into one dead port. Treat consumer mesh gear as hostile to your switch fabric. Never wire two mesh-bridging devices into the same LAN. A flapping MAC is a loop until proven otherwise. The mac-address-table is your fastest path from \u0026ldquo;it\u0026rsquo;s slow\u0026rdquo; to \u0026ldquo;it\u0026rsquo;s port 9 and 14.\u0026rdquo; ","permalink":"https://errorzap.com/posts/the-speaker-that-caused-a-network-storm/","summary":"A wired network melting down at a customer site, traced to a pair of consumer speakers quietly braiding a layer-2 loop out of copper and mesh.","title":"The Speaker That Caused a Network Storm"},{"content":" The pager went off at 6:14 a.m. The morning sync hadn\u0026rsquo;t run.\nI knew that script. I wrote that script. It was good. It pulled data, massaged it, pushed it where it needed to go. It worked flawlessly every single time I ran it.\nThat was the problem. It only worked when I ran it.\nThe scene Here\u0026rsquo;s the embarrassing truth about that automation: it lived inside an SSH session. I\u0026rsquo;d connect to the box at sync-host.internal.example, fire it off, watch it succeed, and disconnect feeling like a wizard.\nThe moment my terminal closed, the kernel reaped it. SIGHUP, lights out. No process, no cron-shaped safety net, nothing.\nOn the Windows box it was worse. I\u0026rsquo;d \u0026ldquo;scheduled\u0026rdquo; it — except every run popped a console window that stole focus from whoever was using the machine. People started killing the window because it interrupted them. Can\u0026rsquo;t blame them.\nSo three different failure modes, one script:\n┌──────────────────────────────────────────────┐ │ \u0026#34;It works when I run it manually\u0026#34; │ ├──────────────────────────────────────────────┤ │ close SSH session ──► SIGHUP, process dies│ │ server reboots ──► nothing comes back │ │ scheduled on win ──► window steals focus │ └──────────────────────────────────────────────┘ │ ▼ NOT actually deployed The investigation I tailed the journal. Nothing — because nothing was logging anywhere. The script wrote to stdout, and stdout was attached to a terminal that no longer existed.\nI checked the process list. Empty. I checked for a cron entry. None — I\u0026rsquo;d never made one, because \u0026ldquo;I\u0026rsquo;ll just run it\u0026rdquo; had quietly become the deployment strategy.\nThat\u0026rsquo;s the aha, and it\u0026rsquo;s a dumb one: manual execution was masquerading as deployment. The script was perfect and completely undeployed at the same time. Every successful manual run was me hand-holding a service that had no manager.\nA long-running, scheduled, headless job has three needs I\u0026rsquo;d ignored: survive a logout, survive a reboot, and write its logs somewhere a human can read them later. A terminal gives you none of those.\nThe fix Give it a service manager. On Linux, a systemd unit. On Windows, NSSM. Both get auto-restart, log redirection, and start-on-boot — and both run headless.\nLinux — /etc/systemd/system/morning-sync.service:\n[Unit] Description=Morning data sync After=network-online.target Wants=network-online.target [Service] Type=simple User=svc-sync WorkingDirectory=/opt/morning-sync ExecStart=/opt/morning-sync/.venv/bin/python /opt/morning-sync/sync.py Restart=on-failure RestartSec=10 StandardOutput=append:/var/log/morning-sync/out.log StandardError=append:/var/log/morning-sync/err.log [Install] WantedBy=multi-user.target sudo mkdir -p /var/log/morning-sync sudo systemctl daemon-reload sudo systemctl enable --now morning-sync.service systemctl status morning-sync.service journalctl -u morning-sync.service -f Windows — same job, NSSM does the babysitting and runs it with no visible window:\nnssm install MorningSync \u0026#34;C:\\apps\\morning-sync\\.venv\\Scripts\\python.exe\u0026#34; \u0026#34;C:\\apps\\morning-sync\\sync.py\u0026#34; nssm set MorningSync AppDirectory \u0026#34;C:\\apps\\morning-sync\u0026#34; nssm set MorningSync AppStdout \u0026#34;C:\\logs\\morning-sync\\out.log\u0026#34; nssm set MorningSync AppStderr \u0026#34;C:\\logs\\morning-sync\\err.log\u0026#34; nssm set MorningSync AppExit Default Restart nssm set MorningSync Start SERVICE_AUTO_START nssm start MorningSync Now I close my laptop and it keeps running. The box reboots at 3 a.m. for patches and the sync comes back on its own. The logs sit in a file instead of evaporating with my SSH session.\nservice systemd / NSSM sync.py headless start / supervise Restart=on-failure boot ──► auto-start ──► logs to file survives logout + reboot A service manager turns \"I ran it once\" into \"it runs forever.\" Why it happened Because the script worked. That\u0026rsquo;s it. When something works on the first manual run, the brain files it under \u0026ldquo;done\u0026rdquo; and quietly skips the boring part — the part where a supervisor owns the process, restarts it when it dies, and persists across boots.\nA terminal is not a runtime. Cron-with-a-popup is not headless. \u0026ldquo;Done\u0026rdquo; is not \u0026ldquo;deployed.\u0026rdquo;\nTakeaways \u0026ldquo;It works when I run it manually\u0026rdquo; is not deployed. A successful manual run proves the logic, not the operation. Long-running automation belongs in a service manager — systemd on Linux, NSSM on Windows — not a terminal you forgot to close. Always set auto-restart and boot persistence. Restart=on-failure plus enable --now (or SERVICE_AUTO_START) means reboots and crashes self-heal. Redirect stdout/stderr to a file. Logs that live inside an SSH session die with the session, and you\u0026rsquo;ll be debugging blind at 6 a.m. Run it headless. Background jobs that pop windows get killed by annoyed users. No UI, no focus theft, no problem. ","permalink":"https://errorzap.com/posts/making-a-python-script-a-real-service/","summary":"The script worked perfectly — right up until I closed the terminal and watched it die.","title":"Making a Python Script a Real Service"},{"content":" 💿 90s KID // Encarta and the Encyclopedia on a All the World\u0026rsquo;s Knowledge, and a Loading Spinner You remember Encarta.\nThat one disc.\nSilver. Heavier than it should\u0026rsquo;ve been.\nLived in a jewel case next to the family computer like it was a relic.\nYou remember the moment it loaded — the swirl, the little fanfare, the screen going dark for a second like the machine was bracing itself.\nAnd then: everything.\nAnimals. Volcanoes. Mozart. The solar system.\nAll of it, right there, glowing.\nYou were eight.\nYou did not understand what you were holding.\nYou just knew the vibes:\nIt was the smart disc — the one Dad said cost real money It had a video of a frog, and you watched that frog forty times There was a globe you could spin and it felt like piloting a spaceship The narrator voice was so calm it was basically a wizard Typing your report? No. You copied it word for word and called it research It felt like the entire library got crushed down into a coaster.\nForbidden, almost. Too much power for one kid and one beige tower.\nHere\u0026rsquo;s what it actually was.\nThat disc was a CD-ROM — about 650 megabytes of read-only storage, pressed once at a factory and never changed again. (Your phone\u0026rsquo;s wallpaper is bigger than that now.)\nTo fit a whole encyclopedia on it, they pulled off some real magic:\nthe text was compressed down tiny, the photos were squeezed into early JPEGs, and the audio and video were chopped into low-res clips just big enough to feel amazing on a 14-inch CRT.\nThe frog video? Maybe two seconds of footage at a resolution you\u0026rsquo;d laugh at today. But it was moving pictures, on demand, from a disc. In 1995 that was witchcraft.\nThe whole thing ran off your CD drive at a blistering 1.2 megabytes a second — which is why the globe took a moment to think before it spun.\n_.--\u0026#34;\u0026#34;\u0026#34;\u0026#34;--._ .\u0026#39; _ _ \u0026#39;. / (_) (_) \\ ENCARTA \u0026#39;95 | .-. .-. | .-------------. | ( )( ) | | [o] CD-ROM | \\ \u0026#39;-\u0026#39; \u0026#39;-\u0026#39; / \u0026#39;-------------\u0026#39; \u0026#39;._ _.\u0026#39; all of human \u0026#39;------\u0026#39; knowledge™ (insert disc 1) The wild part?\nIt never really left.\nYou just stopped noticing.\nThe frog video became YouTube.\nThe spinning globe became Maps.\nThe calm narrator wizard became the little box you type questions into at 2 a.m.\nThe whole pressed-on-a-disc encyclopedia got vaporized into something that lives nowhere and everywhere at once, updates itself while you sleep, and never makes you swap to disc 2.\nYou don\u0026rsquo;t hold all the world\u0026rsquo;s knowledge in a jewel case anymore.\nYou hold it in your pocket.\nAnd it still takes a second to load.\n","permalink":"https://errorzap.com/posts/90s-kid-encarta/","summary":"You remember when all the knowledge in the world fit on one shiny disc, and the loading screen had its own theme music.","title":"90s Kid - Encarta and the Encyclopedia on a Disc"},{"content":" 📼 90s KID // Be Kind, Rewind The Sacred Sticker That Ran Your Whole Weekend You remember the tape.\nThat fat black brick with a label half peeled off.\nThe Blockbuster case that clicked shut like it meant business.\nAnd the sticker — always the sticker —\nBE KIND, REWIND.\nYou didn\u0026rsquo;t know who you were supposed to be kind to.\nA stranger? The future? The tape itself?\nYou just knew it felt like a rule from on high.\nThe whole ritual was holy and a little terrifying.\nThe vibes:\nThe late-fee dread that lived in your stomach all weekend Sliding the tape in and hearing that hungry mechanical gulp The screen going blue, then snowy, then\u0026hellip; somebody else\u0026rsquo;s movie still playing That deep machine whirrrrr of rewinding, the counter ticking backward The guilt if you returned it un-rewound, like you\u0026rsquo;d left a mess at a friend\u0026rsquo;s house You thought the movie lived inside the box.\nLike a tiny theater you were renting for one night only.\nYou were not allowed to be late. You were not allowed to be unkind.\nHere\u0026rsquo;s what was actually going on.\nThat tape was a long ribbon of magnetic film, wound between two reels inside the shell.\nThe picture and sound weren\u0026rsquo;t a file — they were stripes of magnetism painted onto that ribbon, read by a spinning drum of tiny heads as the tape dragged past.\nAnd rewinding mattered for a real reason.\nThe tape played from one reel to the other, so when the movie ended, all the film was bunched on the wrong side.\nThe next person had to wait for it to spin all the way back before they could even press play.\nSo \u0026ldquo;be kind, rewind\u0026rdquo; wasn\u0026rsquo;t sentiment.\nIt was etiquette for shared, sequential hardware.\nYou weren\u0026rsquo;t streaming a movie. You were borrowing a physical line and you owed the next kid a fresh start.\nBE KIND, REWIND ___________________________ | [ o ] [ o ] | | === === | | / \\ VHS / \\ | | ( )--========( ) | | \\___/ ||||||||| \\___/ | |_____________________ ____| |#####################| | |#####################|____| |_________________________ | The tapes are gone.\nThe little machine that rewound them for you — gone.\nBut the etiquette never died.\nIt just went invisible.\nNow the platform rewinds for you. Silently. Instantly.\nYou scrub the timeline and the whole movie is already there, every second equally close, no wrong side to bunch up on.\nNo counter. No whir. No guilt.\nAnd honestly? Something\u0026rsquo;s a little lost.\nBecause you\u0026rsquo;ll never again hand someone a thing and quietly hope —\nyou left it better than you found it.\n","permalink":"https://errorzap.com/posts/90s-kid-be-kind-rewind/","summary":"You remember the little sticker that begged you to spin the tape backward before you brought it home.","title":"90s Kid - Be Kind, Rewind"},{"content":" 🎧 90s KID // The Walkman and the Pencil The Yellow Brick That Played Your Whole Heart You remember the Walkman.\nThat chunky little slab of plastic clipped to your waistband like a deputy\u0026rsquo;s badge.\nOrange foam headphones that pinched your ears just enough to feel official.\nAnd the clunk — that satisfying mechanical clunk when the cassette door snapped shut.\nYou hit play.\nAnd for the length of one side, the world belonged to you.\nIt felt important. A little forbidden. The first thing that was yours and not the family\u0026rsquo;s.\nThe vibes were immaculate:\nHitting record off the radio and screaming at the DJ for talking over the intro The terror of low batteries — when the song slowed down and dropped an octave, like the music was dying Rewinding to your favorite part and counting in your head because there was no rewind-to-track, just raw guesswork That ONE tape so loved it sounded like it was recorded underwater And the pencil. Always the pencil. Because here\u0026rsquo;s the thing every 90s kid knew by instinct.\nWhen the tape went slack and spat out a loop of shiny brown ribbon — you grabbed a pencil, jammed it in the little spoked hole, and wound it back in.\nYou didn\u0026rsquo;t know why it worked.\nYou just knew it was magic, and you were the wizard.\nHere\u0026rsquo;s what was actually happening.\nThat brown ribbon was mylar tape coated in iron oxide — basically rust, lovingly arranged. The play head was a tiny electromagnet \u0026ldquo;reading\u0026rdquo; the magnetic pattern stamped into that rust as it dragged past at exactly 1⅞ inches per second.\nTwo reels. One feeding, one taking up. When the take-up reel stuck, the player kept pulling tape out — and you got The Spaghetti Incident.\nThe pencil? Its six-sided hole was a dead-perfect match for the cassette\u0026rsquo;s reel hub. You weren\u0026rsquo;t doing magic. You were doing manual tape transport — the exact job the motor did, just powered by a bored 9-year-old\u0026rsquo;s wrist.\nAnd the dying-batteries voice drop? Real physics. Weak power meant the motor spun slower, so the tape crawled past the head and every pitch sagged. You were literally hearing your AA batteries run out of breath.\n___________________________ | .---. .---. | | ( o ) ( o ) | | \u0026#39;---\u0026#39; [ A ] \u0026#39;---\u0026#39; | | \\_______________/ | | |::|::|::|::|::|::| | |___|_TYPE_I_·_60_min_|_____| \\___ pencil goes ___/ here → O You can\u0026rsquo;t kill a format that taught you patience.\nThe Walkman never really died — it just lost the moving parts. The iPod was a Walkman that finally stopped eating your tapes. Your phone is a Walkman with the whole record store crammed inside it.\nBut somewhere a teenager just paid forty bucks for a brand-new cassette because the algorithm told them it was vintage.\nAnd somewhere, an old pencil is so proud.\n","permalink":"https://errorzap.com/posts/90s-kid-the-walkman/","summary":"You remember the sacred panic of a tape unspooling, and the pencil that always knew what to do.","title":"90s Kid - The Walkman and the Pencil"},{"content":" 📼 90s KID // The VCR That Blinked 12:00 Forever A Clock That Never Learned What Time It Was You remember the VCR.\nThat big black slab under the TV.\nHeavier than it had any right to be.\nA mouth that ate tapes and a single, stubborn green 12:00 blinking out into the dark living room.\n12:00.\n12:00.\n12:00.\nForever.\nNobody set it. Nobody could. It just blinked, all night, like a tiny lighthouse warning ships away from the entertainment center.\nAnd you, small and certain, knew exactly what it was about:\nvibes: it was the brain of the TV, obviously the blinking meant it was thinking pushing EJECT felt like defusing a bomb the rented tape had to be rewound or something bad happened (you weren\u0026rsquo;t told what) the little tracking lines were a storm you could fix by waving your hand at it that flap was a mouth and you were NOT putting your fingers near it It was important. It was a little forbidden. It was a little magic.\nHere\u0026rsquo;s the reveal.\nThat blinking 12:00 wasn\u0026rsquo;t broken. It was just honest. The VCR had a tiny digital clock inside, and it had no idea what time it was because nobody ever told it. Every power flicker — every summer storm, every blown fuse, every time Dad vacuumed and hit the wrong outlet — wiped its memory clean and it woke up at the only number it knew: midnight. Square one. Again.\nInside that heavy box was genuinely wild stuff. Magnetic tape, a half-inch wide, dragged past a spinning head drum tilted at a slight angle so it could cram diagonal stripes of video onto the tape — helical scan, they called it. That whir you heard when you hit play was the drum spinning up to speed, threading the tape around itself like a tiny robot doing origami in the dark.\nAnd the rewinding? Real. Tape is a line, not a grid. To see the start, you physically wound it back to the start. There was no \u0026ldquo;skip to beginning.\u0026rdquo; There was only patience and that zzzzzzWP of the tape snapping home.\n____________________________ | [] SONY VHS HQ | | | | .------------------. | | | __ __ | | | | (__) :::: (__)| | \u0026lt;- the tape | |______________ ___| | | \u0026#39;------------------\u0026#39; | | | | [\u0026lt;\u0026lt;] [|\u0026gt;] [\u0026gt;\u0026gt;] [#] | | __ | | |12:00| *blink* | | |_:_:_| | |____________________________| It never really went away.\nThat blinking clock just moved. Now it\u0026rsquo;s the microwave you never set after daylight saving. The oven that flashes after a brownout. The router light that means something, you\u0026rsquo;re pretty sure. The little devices in your life still wake up confused, still ask you what time it is, still get ignored.\nAnd rewinding? You do it every time you drag the scrubber bar back thirty seconds because you missed a line of dialogue.\nBe kind. Rewind.\nYou still do. You just don\u0026rsquo;t have to get up for it anymore.\n","permalink":"https://errorzap.com/posts/90s-kid-the-blinking-vcr/","summary":"You remember the glowing green 12:00 that nobody in the house ever fixed.","title":"90s Kid - The VCR That Blinked 12:00 Forever"},{"content":" 📼 90s KID // The Answering Machine The Box That Kept Our Voices You remember the answering machine.\nIt lived on the little phone table in the hallway.\nBeige. Always beige.\nA red light that blinked when something was waiting for you.\nOne blink. Two blinks. Three blinks?!\nThree blinks meant something happened today.\nAnd you were NOT allowed to touch it.\nYou\u0026rsquo;d come home from school and that red light would be going,\nand the whole house would change.\nMom hits PLAY.\nA beep.\nThen a voice — disembodied, a little too loud, a little too slow —\nfloating out of a box into the hallway like a tiny ghost.\nThe vibes:\nthe outgoing message rehearsed nineteen times before anyone got it right the cousin who didn\u0026rsquo;t know how it worked and just breathed for forty seconds the telemarketer\u0026rsquo;s awkward pause, then click pressing your ear to it because you weren\u0026rsquo;t allowed to turn it up the cold terror of accidentally erasing all messages It felt like a vault.\nA little machine guarding voices that belonged to grown-ups.\nHere\u0026rsquo;s the reveal.\nInside that beige box?\nTape.\nSometimes two tiny cassettes — one looping your greeting on a couple feet of magnetic film, one recording whatever came in. Later models cheated and used a microchip with a few minutes of grainy memory, but the classic ones were pure analog.\nThe \u0026ldquo;magic\u0026rdquo; was just physics and a phone line.\nThe machine counted rings. After four, it picked up — closed the circuit so the caller stopped hearing ring, played your greeting off the loop tape, then dropped a record head onto the second tape and let it roll. The beep wasn\u0026rsquo;t decoration. It was the cue: the tape is moving NOW, say your thing.\nThe blinking light? A counter ticking up each time the record head fired.\nNo cloud. No server.\nVoices physically scratched into a ribbon of rust, sitting on a hallway table.\n_________________________ | ___________________ | | | ANSWERING MACHINE | | | | ___ ___ ___ | | | | (o o) | 3 | msg | | \u0026lt;- blinking | |___________________| | | [\u0026lt;\u0026lt;] [\u0026gt;] [\u0026gt;\u0026gt;] [REC] | | ___________________ | | ||| | |||| | | || | | \u0026lt;- tape window | |___________________| | |_________________________| The funny part?\nIt never left.\nWe just stopped owning the beige box.\nNow the tape lives on a server farm somewhere, the greeting is robot-voiced, and the blinking red light became a little number bubble on your phone screen you swipe away without listening.\nSame loop. Same dread of a full mailbox.\nExcept now nobody calls — they text \u0026ldquo;call me?\u0026rdquo; — and the only voicemails left are the dentist and a scam about your car\u0026rsquo;s extended warranty.\nThat box held the voices of everyone you loved.\nFour rings, then a beep.\nYou still flinch a little when something blinks red.\n","permalink":"https://errorzap.com/posts/90s-kid-the-answering-machine/","summary":"You remember the little blinking box that held strangers\u0026rsquo; voices hostage on a cassette the size of a postage stamp.","title":"90s Kid - The Answering Machine"},{"content":" It was supposed to be a five-line automation. Read a value, tweak it, write it back. The kind of thing you knock out between coffees.\nI shelled into a client\u0026rsquo;s pfSense box — a clean, current 24.11 install — and ran my little script through pfSsh.php. It exited without a peep. No errors. No warnings. Just a fresh prompt blinking back at me.\nThen the SSH session died.\nThen the VPN dropped.\nThen my phone lit up.\nThe investigation I got back in through the console. The dashboard loaded, and my stomach went cold. No WAN config. No LAN rules. No NAT, no VLANs, no DHCP scopes. The box was sitting there with the bright-eyed innocence of a router that had never met a network.\nThe config wasn\u0026rsquo;t corrupted. It was gone. Replaced with defaults, like someone had hit the reset button — except nobody had touched the hardware.\nI pulled up my script. Five lines. The offending logic looked like ancient, trustworthy pfSense scripting lore:\nglobal $config; $config[\u0026#39;system\u0026#39;][\u0026#39;something\u0026#39;] = \u0026#39;newvalue\u0026#39;; write_config(\u0026#34;scripted tweak\u0026#34;); Every old forum thread, every crusty wiki page from a decade ago, does exactly this. Grab the $config global, poke a key, call write_config(). It\u0026rsquo;s the canonical move.\nSo why did it nuke the box?\nI added one debug line before the write:\nglobal $config; var_dump($config); The output told the whole story in one ugly word: empty.\nThe \u0026ldquo;aha\u0026rdquo; On modern pfSense, that legacy $config global is no longer the living, fully-populated config tree the old guides assume. In the pfSsh.php scripting context it came back essentially empty — a hollow shell.\nAnd write_config() doesn\u0026rsquo;t merge. It doesn\u0026rsquo;t patch. It takes whatever is in memory right now and serializes that as the new, authoritative config. The whole thing.\nSo my script read an empty global, set one key on top of nothing, and then confidently wrote that \u0026ldquo;nothing plus one key\u0026rdquo; back as the complete configuration. write_config() did exactly what it was told. It overwrote a fully working firewall with a near-empty husk.\n┌──────────────────────────────────────────────────────────┐ │ WHAT I THOUGHT HAPPENED WHAT ACTUALLY HAPPENED │ ├──────────────────────────────────────────────────────────┤ │ $config = { full tree } ► $config = { } (EMPTY) │ │ │ │ │ │ ▼ set one key ▼ set one key │ │ { full tree + key } { almost nothing } │ │ │ │ │ │ ▼ write_config() ▼ write_config()│ │ ✓ saved correctly ✗ overwrote everything │ └──────────────────────────────────────────────────────────┘ The modern config backend wants you to read and write specific paths — config_get_path() and config_set_path() — never the whole global at once.\nread \u0026 write ONE key — not the whole tree SAFE — path API config_get_path(k) config_set_path(k,v) ► touches only k ✓ FOOTGUN — global global $config; // empty write_config() ✗ overwrites ALL of it one call, two very different blast radii Scoped path writes touch one key. A global write_config() rewrites the entire config from whatever's in memory. The fix First, recover. Restore the last known-good config from a backup and reload:\n# Restore from a saved backup and reload config_restore /cf/conf/backup/config-\u0026lt;latest-good\u0026gt;.xml /etc/rc.reload_all pfSense keeps automatic backups under /cf/conf/backup/ — list them with ls -lt /cf/conf/backup/ and pick the newest one from before your script ran. That brought the firewall back to life.\nThen, fix the script. Use the path API and stop touching the global entirely:\n// Read a specific key (with a default), modify, write that key back. $value = config_get_path(\u0026#39;system/something\u0026#39;, \u0026#39;default\u0026#39;); config_set_path(\u0026#39;system/something\u0026#39;, \u0026#39;newvalue\u0026#39;); write_config(\u0026#34;scripted tweak via path API\u0026#34;); And — non-negotiable — snapshot the config before any scripted write, so recovery is one command instead of a panic:\ncp /cf/config.xml \u0026#34;/cf/conf/backup/config-$(date +%Y%m%d-%H%M%S)-prewrite.xml\u0026#34; Why it happened Platforms migrate their internals quietly. pfSense moved to a path-based config backend, and the old global $config got demoted from \u0026ldquo;the config\u0026rdquo; to \u0026ldquo;a vestigial variable that may or may not be populated, depending on context.\u0026rdquo;\nThe docs got updated. The thousand forum posts from 2014 did not.\nSo the trap is perfect: the legacy pattern still compiles, still runs, still exits clean. It just operates on a ghost of the real config — and write_config() is happy to make that ghost permanent.\nA function that overwrites everything from in-memory state is a loaded gun. When the thing loading that memory changes underneath you, the gun is now pointed at your whole config and you don\u0026rsquo;t even know it.\nTakeaways A global write_config() rewrites the entire config from in-memory state. If that state is stale, partial, or empty, you\u0026rsquo;ve just factory-reset your box. Treat it as a full-config replace, never a patch. When a platform migrates its config backend, the old global is a footgun. It still runs clean — that\u0026rsquo;s what makes it dangerous. Verify what it actually contains before trusting it. Use the scoped API. config_get_path() / config_set_path() touch exactly one key. That\u0026rsquo;s the blast radius you want. Snapshot before any scripted write. cp /cf/config.xml /cf/conf/backup/...-prewrite.xml turns a disaster into a 30-second restore. var_dump() your assumptions. One debug line showing an empty global would\u0026rsquo;ve saved the whole incident. When in doubt, print what you\u0026rsquo;ve actually got — not what the old guides promise you have. ","permalink":"https://errorzap.com/posts/the-firewall-api-call-that-wipes-your-router/","summary":"I ran a \u0026lsquo;harmless\u0026rsquo; config script against a pfSense box and watched the entire firewall configuration evaporate in a single function call.","title":"The Firewall API Call That Factory-Resets Your Router"},{"content":" 💾 90s KID // The Floppy Disk The Save Icon You Could Hold In Your Hand You remember the floppy disk.\nNot the bendy black ones your older cousin had.\nThe hard ones. The beige squares with the little silver shutter on top.\nYou remember the click it made going into the slot.\nThat confident, mechanical chunk that told you something important was happening.\nYou\u0026rsquo;d slide it in.\nThe drive light would blink.\nAnd the whole computer would make that grinding, churning noise like it was thinking really hard.\nTo a kid, a floppy was basically magic.\nThe metal shutter. You were NOT supposed to slide it open. So obviously you slid it open. There was shiny brown film inside. You touched it. You doomed the disk. The write-protect tab. A tiny plastic switch that felt like arming a nuclear device. The label. You wrote \u0026ldquo;GAMES\u0026rdquo; on it in marker like a tiny librarian. The hoard. A whole rainbow box of them, and you had no idea what was on 80% of them. It felt forbidden and fragile and yours.\nYou knew, somehow, that if you put a magnet near it, you\u0026rsquo;d kill it. Nobody told you the rules. You just knew. Floppy disk law was passed down kid to kid like folklore.\nHere\u0026rsquo;s the reveal.\nThat beige square held 1.44 megabytes.\nThat\u0026rsquo;s it. That\u0026rsquo;s the whole thing.\nLess than a single phone photo today. A blurry one.\nInside that shutter was an actual spinning disk of magnetic-coated mylar — that shiny brown film you weren\u0026rsquo;t supposed to touch. The drive head physically pressed against it and read magnetic charges, like a tiny record player for ones and zeros.\nThe shutter? That slid open automatically when you inserted it, so the head could reach the disk. You opening it by hand was just you, a small barbarian, exposing the delicate part to dust and fingerprints.\nThe write-protect tab actually worked by letting light through a hole. Open the hole, a sensor saw it, and the drive physically refused to write. It wasn\u0026rsquo;t software. It was a window. Genuinely un-hackable by a kid, which is the only security that ever truly held.\n________________________ | __ | | | | \u0026lt;- write-protect | | |__| tab (a hole!) | | ____________________ | | |####################| | \u0026lt;- metal shutter | |####################| | (do NOT slide it) |_|____________________|_| | | | [ GAMES ] | \u0026lt;- your handwriting | ( in marker ) | |________________________| The funny part?\nThe floppy never really left.\nLook at the Save button in basically any app you\u0026rsquo;ve used in the last thirty years. That little square with the shutter? That\u0026rsquo;s a floppy disk. A piece of 1980s hardware, frozen forever as an icon, saving documents to drives that floppy disks couldn\u0026rsquo;t dream of holding.\nThere are kids today hitting Save who have never seen the object the button is shaped like. To them it\u0026rsquo;s just \u0026ldquo;the save shape.\u0026rdquo; A glyph. A rune with no origin story.\nBut you know.\nYou know it went chunk.\nYou know you weren\u0026rsquo;t supposed to touch the brown part.\nAnd you know that somewhere in a closet, there\u0026rsquo;s still a rainbow box of them, holding 1.44 megabytes of something you\u0026rsquo;ll never read again.\nAnd honestly? That\u0026rsquo;s plenty.\n","permalink":"https://errorzap.com/posts/90s-kid-the-floppy-disk/","summary":"A square of beige plastic that held your whole world in 1.44 megabytes and felt like it could hold the universe.","title":"90s Kid - The Floppy Disk"},{"content":" The message came in the way these always do. No timestamp, no context, just adrenaline.\n\u0026ldquo;The payment shows zero applied. The customer paid. Where did the money GO?\u0026rdquo;\nSelf-hosted invoicing app. A real payment, recorded against a real invoice. The UI said $0.00 applied. To everyone looking at that screen, that meant one thing: a customer handed over money and the system ate it.\nThat\u0026rsquo;s a five-alarm fire in billing. Money you can\u0026rsquo;t account for is money you have to refund, re-bill, or explain to an auditor. So I did the thing you do when the building is allegedly burning. I stopped trusting the screen.\nThe investigation The UI is a story the app tells you. The database is what actually happened. When the two disagree, the database wins — every time.\nSo I went to the source of truth. Payments table first.\nSELECT id, invoice_id, amount, applied, status, created_at FROM payments WHERE invoice_id = 4815; id | invoice_id | amount | applied | status | created_at -----+------------+---------+---------+-----------+--------------------- 162 | 4815 | 1200.00 | NULL | completed | 2026-03-07 16:42:11 There it is. amount = 1200.00. Status completed. The money was recorded. It did not vanish. It was sitting in the table exactly where it should be.\napplied was NULL — interesting, but not \u0026ldquo;missing money\u0026rdquo; interesting. Stored zero would\u0026rsquo;ve been one thing. NULL is the smell of a column nobody populates.\nNow the invoice. Did the balance actually move?\nSELECT id, amount, balance, status FROM invoices WHERE id = 4815; id | amount | balance | status -----+---------+---------+-------- 4815 | 1200.00 | 0.00 | paid Balance dropped to zero. Status flipped to paid. The ledger was correct. The accounting was correct. The money was where it belonged.\nSo why was the UI screaming?\nWHAT THE SCREEN SAID WHAT THE DB SAID ┌──────────────────────┐ ┌──────────────────────┐ │ Payment #162 │ │ payments.amount │ │ Applied: $0.00 ⚠ │ vs │ = 1200.00 ✓ │ │ \u0026#34;money gone?!\u0026#34; │ │ invoices.balance │ └──────────────────────┘ │ = 0.00 (paid) ✓ │ │ └──────────────────────┘ │ │ └──────────► SAME ◄────────────┘ UNDERLYING TRUTH The \u0026ldquo;aha\u0026rdquo; The \u0026ldquo;Applied\u0026rdquo; figure on that payment screen wasn\u0026rsquo;t reading payments.amount. It was reading a computed display field — a value the UI derived at render time from a relationship that, for this payment, came back empty.\nThe applied column existed in the schema but was never written by this payment path. It lived only in the database and was never even exposed through the API. The renderer reached for it, found NULL, coerced it to 0.00, and printed a number that looked like a catastrophe.\nNothing was lost. A blank field got rendered as a zero, and a zero in a money column reads like a heist.\nDATABASE (truth) amount = 1200.00 balance = 0.00 applied = NULL UI (rendering) Applied: $0.00 ⚠ render NULL \u0026 empty relation \u003c truth \u003e 0.00 panic The data was fine. The renderer turned an empty field into a $1,200 scare. The fix There was no data to fix. The fix was confirming the ground truth and proving the money was accounted for, then knowing which field to never trust again.\n-- Prove the payment is reconciled against the invoice SELECT i.id, i.amount AS invoice_total, i.balance, COALESCE(SUM(p.amount), 0) AS paid_total, i.amount - COALESCE(SUM(p.amount), 0) AS expected_balance FROM invoices i LEFT JOIN payments p ON p.invoice_id = i.id AND p.status = \u0026#39;completed\u0026#39; WHERE i.id = 4815 GROUP BY i.id; id | invoice_total | balance | paid_total | expected_balance -----+---------------+---------+------------+------------------ 4815 | 1200.00 | 0.00 | 1200.00 | 0.00 balance matches expected_balance. Books are square. Crisis cancelled — because there was never a crisis, only a bad pixel.\nWhy it happened The app had two notions of \u0026ldquo;applied\u0026rdquo;: the real stored amount on the payment, and a derived field the UI computed from a relation that this particular payment flow never filled in. When the relation was empty, the computed value fell through to zero instead of throwing or hiding itself.\nA zero in a dollar column is the most alarming value a billing system can show. The renderer printed it with full confidence and zero context, and a correct ledger looked like a robbery.\nTakeaways The UI is a rendering of the truth, not the truth. When a number looks wrong, query the database before you assume data loss. Most \u0026ldquo;missing money\u0026rdquo; is a display bug. Check the ledger and the invoice balance first — the books are usually fine. Know which fields are stored vs computed. A value derived at render time can lie even when every stored byte is correct. NULL coerced to 0 is a landmine in money columns. A blank field and an actual zero mean wildly different things; don\u0026rsquo;t let the UI conflate them. Schema fields can exist without ever being populated or exposed. Confirm the data path actually writes the column before you trust what reads it. ","permalink":"https://errorzap.com/posts/the-payment-that-showed-zero-but-wasnt/","summary":"A recorded payment rendered as \u0026lsquo;$0.00 applied\u0026rsquo; and everyone assumed the money vanished — but the ledger had been right the whole time.","title":"The Payment That Showed $0 (But Wasn't)"},{"content":" 📺 90s KID // The Cable Box and Its Wired Remote The Box On Top Of The Box You remember the cable box.\nNot the TV. The other thing.\nThe squat plastic brick that lived on top of the TV, where it had no business being.\nIt had a red LED clock that glowed like a tiny alarmed eye.\nIt got warm. Always warm.\nAnd it had a wired remote — a remote on a leash — that snaked across the carpet like it was afraid to leave home.\nYou knew, in your kid bones, that this box was Important.\nIt hummed faintly, like it was thinking The channel number flipped on it with a soft clunk you could almost feel Dad told you not to unplug it or \u0026ldquo;we\u0026rsquo;ll lose the channels\u0026rdquo; The remote cord was always one stomped-on foot away from being yanked out There was a little sliding door on the front, and behind it, forbidden buttons You were pretty sure the box was negotiating with the TV on your behalf.\nAnd you weren\u0026rsquo;t entirely wrong.\nWhat it actually was Here\u0026rsquo;s the nerdy reveal.\nYour TV in the 90s could only really speak one fluent language: channels 2 through 13, the old VHF band.\nBut the cable company was pumping dozens of channels down that one coax wire, all stacked on different frequencies way up high where your TV couldn\u0026rsquo;t tune.\nThe cable box was a set-top converter. Its whole job:\nGrab the channel you wanted from way up the frequency dial\nand re-broadcast it down to channel 3 (or 4),\nthe one frequency your TV trusted.\nThat\u0026rsquo;s why you set the TV to 3 and left it there forever.\nThe TV was just a dumb window.\nThe box did the choosing.\nThe wired remote? That was infrared and pricey, so the cheap units just ran a cable straight to the box — no batteries, no line-of-sight, no losing it in the couch. A leash, yes. But an honest one.\n_________________________ | [ 0 3 ] CABLE | | .----. | | | 03 | o o o o o | | \u0026#39;----\u0026#39; POWER ~~~ | |_________________________| ||| coax in ||| ======O======\u0026gt; TV ch.3 ( \u0026#34;leashed\u0026#34; ) \\ remote / \u0026#39;~~~~~~~~~\u0026#39; It never really left The leash got cut. The box got smaller, then it got a guide, then a DVR, then a slot for movies you paid eight bucks to \u0026ldquo;rent\u0026rdquo; with a button.\nBut the idea never died.\nEvery streaming stick jammed into the back of your TV today is the exact same trick:\na little warm box that does the real work, while the TV just sits there being a dumb, beautiful window.\nYou still set the input and leave it.\nYou still don\u0026rsquo;t unplug it, or \u0026ldquo;we\u0026rsquo;ll lose everything.\u0026rdquo;\nThe box on top of the box won.\nIt just learned to hide.\n","permalink":"https://errorzap.com/posts/90s-kid-the-cable-box/","summary":"You remember the heavy little box on top of the TV, and the remote that was leashed to it like a dog that wasn\u0026rsquo;t allowed off the porch.","title":"90s Kid - The Cable Box and Its Wired Remote"},{"content":" The ticket was boring. Cleanup. A client wanted some stale files gone from their primary share. I deleted them. Coffee. Done.\nTwenty minutes later the same nine files vanished from the backup node too.\nThat\u0026rsquo;s the part that made my stomach drop. The backup node existed precisely so that \u0026ldquo;oops, deleted it on the primary\u0026rdquo; was a non-event. It was supposed to be the safety net. Instead it watched me cut the trapeze and then helpfully cut its own rope to match.\nThe investigation First instinct: ransomware, a rogue cron, a fat-fingered teammate. All wrong. The truth was dumber and worse.\nI\u0026rsquo;d inherited this setup and never read the sync config closely. I assumed \u0026ldquo;backup node\u0026rdquo; meant one-way. It didn\u0026rsquo;t.\ngrep -i \u0026#39;mode\\|direction\\|propagat\u0026#39; /etc/syncthing/config.xml \u0026lt;folder id=\u0026#34;primary-share\u0026#34; type=\u0026#34;sendreceive\u0026#34; ...\u0026gt; sendreceive. Bidirectional. The backup node wasn\u0026rsquo;t a backup at all — it was a mirror. And a mirror does exactly one job: make the other side look like this side. Including the absences.\nI deleted nine files on the primary. The sync engine saw nine deletions, decided they were authoritative changes, and faithfully replicated them downstream. No malice. No bug. It did exactly what I configured it to do.\n┌──────────────┐ delete 9 files ┌──────────────┐ │ PRIMARY │ ──────────────────► │ \u0026#34;BACKUP\u0026#34; │ │ (source) │ sendreceive │ node │ └──────────────┘ propagates ✗ └──────────────┘ │ │ ▼ ▼ files gone files gone too ▓ safety net: GONE The \u0026ldquo;aha\u0026rdquo; wasn\u0026rsquo;t a clever diagnostic. It was reading one attribute and feeling like an idiot. Sync is not backup. A sync engine has no concept of \u0026ldquo;retain this.\u0026rdquo; Its entire worldview is converge. Deletion is just another change to converge on.\nsync mirrors deletions · backup keeps them PRIMARY sendreceive ✗ pull-only ✓ delete → ← delete propagates receive only, never push A bidirectional mirror converges on deletions. A pull-only backup receives changes but never propagates them back. The fix Recover first, redesign second. The deleted files came back from a daily snapshot that — by luck, not design — lived off the sync path. Then I changed the node from a mirror into an actual backup.\n# 1) flip the backup node to PULL-ONLY: it receives, never pushes # Syncthing: set this folder to \u0026#34;Receive Only\u0026#34; # config.xml -\u0026gt; type=\u0026#34;receiveonly\u0026#34; # 2) route deletions to a versioned trash instead of hard-deleting # \u0026lt;versioning type=\u0026#34;trashcan\u0026#34;\u0026gt; with a retention window # (deletes land in .stversions/, not the void) # 3) shorten the sync interval to bound the blast radius # rescanIntervalS: 3600 -\u0026gt; 300 # 5 min, not 1 hour # 4) confirm the daily snapshot job runs OUTSIDE the synced tree systemctl list-timers | grep snapshot ls -la /var/snapshots/ # snapshots here, NOT under the synced folder After the flip, the verification was the satisfying part. Delete a test file on the primary, watch the backup node not care:\n# on primary touch /srv/share/CANARY.txt rm /srv/share/CANARY.txt # on backup node, a few minutes later ls /srv/backup/share/CANARY.txt # CANARY.txt is still there ✓ (delete did not propagate) Receive-only means upstream deletions are treated as conflicts to ignore, not commands to obey. The node now keeps things the primary throws away. That\u0026rsquo;s the whole point of a backup.\nWhy it happened Because \u0026ldquo;sync\u0026rdquo; and \u0026ldquo;backup\u0026rdquo; use the same verbs — copy, mirror, replicate — people treat them as the same noun. They aren\u0026rsquo;t.\nSync optimizes for agreement: every node looks identical, fast. Backup optimizes for survival: keep a copy even when the source loses it. Those goals conflict the instant you delete something. A mirror resolves the conflict by deleting too. A backup resolves it by holding on.\nI\u0026rsquo;d labeled a mirror \u0026ldquo;backup\u0026rdquo; and trusted the label. The config said sendreceive the whole time. I just never looked.\nTakeaways Sync is not backup. Bidirectional sync replicates deletions perfectly — that\u0026rsquo;s a feature, and it will eat your data. A backup node must be one-way. Receive-only / pull-only: it accepts changes but never pushes or propagates deletes upstream-to-down. Route deletions to trash, not the void. A versioned trashcan with retention turns \u0026ldquo;gone forever\u0026rdquo; into \u0026ldquo;gone for now.\u0026rdquo; Keep snapshots off the sync path. If your point-in-time recovery lives inside the synced tree, the sync can delete your recovery too. Read the config before you trust the label. grep the one attribute that defines behavior — sendreceive vs receiveonly is the whole story. ","permalink":"https://errorzap.com/posts/how-my-own-backup-deleted-my-files/","summary":"I deleted nine files on purpose — and my backup node helpfully deleted them everywhere else too.","title":"How My Own Backup Deleted My Files"},{"content":" ☎ 90s KID // The Phone Cord That Reached the Wh The Coil That Stretched From the Kitchen to the Edge of Forever You remember the cord.\nThat curly, springy, impossibly tangled umbilical between the wall and the phone you were absolutely not supposed to be using right now.\nYou remember stretching it.\nKitchen, around the corner, down the hall — pulled taut, vibrating like a guitar string, the handset pressed to your ear while you sat on the floor of the only room with a door that closed.\nYou remember the stretch limit.\nThe exact spot where one more inch would yank the whole phone off the wall.\nIt felt like more than a wire.\nThe vibes:\nA leash. Mom could find you by following it like a treasure map. A weapon. Whip it and it cracked like a tiny lightning bolt. A mystery. Why was it ALWAYS knotted? You hung up straight. It tangled out of spite. Forbidden tech. Long-distance calls cost real money, and you could feel the adults watching the clock. A portal. Somehow your friend\u0026rsquo;s voice was in your hand and they were across town. You\u0026rsquo;d stretch it as far as it would go just to whisper. That extra ten feet of cord was the closest thing your house had to privacy.\nHere\u0026rsquo;s what it actually was.\nThat coiled cord carried two tiny copper wires — a circuit — and your voice rode it as a wobbling electrical current, a copy of the sound pressure hitting the mouthpiece.\nThe curl wasn\u0026rsquo;t decoration. It was strain relief: coil it and you get slack without a long trip-wire dangling everywhere. The price of that clever spring? It stored twist, and every time you flipped the handset it wound up a little more. The tangle was physics, not spite.\nAnd the wall jack? Behind it, a pair of wires ran out of your house, down the street, to a central office full of switches — once human operators with patch cords, later electromechanical relays clacking in the dark — quietly connecting your line to anyone\u0026rsquo;s, anywhere.\nYou were holding the end of a network that wrapped the entire planet. You just used it to ask if anyone was home.\n_.-._ / ___ \\ | / \\ | \u0026#34;...around the corner... | \\___/ | one more inch...\u0026#34; \\_____/ | __|__ (_____) / | \\____ | | )___ \\ | ___/ )___ \\ |__/ ___/ )___ \\| / ___/ ) \\_____/ / ___/ \\__/____/ wall ---\u0026gt; the whole world The cord is gone now. You carry the phone and the central office in your pocket, and it never tangles.\nBut the stretch never left you.\nYou still wander the house looking for the quiet room. You still pace to the edge of the WiFi. You still find the one spot — by the window, on the stairs — where the signal holds and nobody can hear you.\nThe leash got invisible.\nYou\u0026rsquo;re still pulling it as far as it\u0026rsquo;ll go.\n","permalink":"https://errorzap.com/posts/90s-kid-the-phone-cord/","summary":"The coiled cord was a leash, a lifeline, and the closest thing your house had to a portal — and you stretched it to its absolute limit.","title":"90s Kid - The Phone Cord That Reached the Whole House"},{"content":" The dashboard was green. That\u0026rsquo;s the part that still bugs me.\nA customer calls because a workstation died and they need yesterday\u0026rsquo;s file back. Easy day. I pull up the backup server, find the machine, and go to restore. Except there\u0026rsquo;s nothing to restore. Last successful backup: never. Not \u0026ldquo;stale.\u0026rdquo; Not \u0026ldquo;failed.\u0026rdquo; Never.\nBut the agent was online. Heartbeat OK. Service running OK. Sitting there reporting in like a good little soldier. It just hadn\u0026rsquo;t backed up a damn thing the entire time it had been installed.\nThe investigation First instinct: blame the agent. So I remoted in and read the client logs. The agent was happily completing runs. No errors. It thought it was doing its job.\nSo now I\u0026rsquo;ve got an agent that says \u0026ldquo;I backed up\u0026rdquo; and a server that says \u0026ldquo;this machine has never backed up.\u0026rdquo; Both swear they\u0026rsquo;re telling the truth. When two systems disagree about reality, the disagreement is almost always about identity.\nI checked what name the agent was registered under on the server. Then I checked the machine\u0026rsquo;s actual OS hostname.\nThey didn\u0026rsquo;t match.\n┌──────────────────────────┐ ┌──────────────────────────┐ │ THE WORKSTATION │ │ BACKUP SERVER │ │ │ │ │ │ hostname: WS-FRONTDESK │ │ client: frontdesk-pc │ │ agent runs OK │ │ keyed by name ───┐ │ │ \u0026#34;backup complete\u0026#34; OK │ │ ▼ │ └───────────┬──────────────┘ │ ┌───────────────────┐ │ │ reports as │ │ frontdesk-pc │ │ │ WS-FRONTDESK ─────────┼──►│ (PHANTOM CLIENT) │ │ │ │ │ last backup: none │ │ ▼ │ └───────────────────┘ │ bytes go ►►► nowhere │ WS-FRONTDESK: not found │ └──────────────────────────┘ └──────────────────────────┘ The agent identified itself by the machine\u0026rsquo;s real hostname. The server had the client filed under a different, hand-typed name from install day. So the agent checked in as WS-FRONTDESK, the server looked for a client called WS-FRONTDESK, found nobody, and the actual data went into the void. Meanwhile the frontdesk-pc entry someone created manually just sat there forever, eternally \u0026ldquo;waiting for first backup,\u0026rdquo; looking close enough to green that nobody clocked it.\nThe \u0026ldquo;aha\u0026rdquo; Backup identity is the hostname. Not a label you pick. Not a friendly name. The hostname.\nThe server keys everything — auth, storage path, history — off the client name. If that name doesn\u0026rsquo;t equal what the OS reports as hostname, the agent and server are two ships passing in the night, both convinced they\u0026rsquo;re docked.\nAgent says WS-FRONTDESK Server expects frontdesk-pc ≠ no match = hostname = typed at install identity mismatch -\u003e silent backup failure Two systems, two names, zero backups — and a green light the whole time. The fix Stop guessing. Ask the machine its own name, on the box itself:\n# On the workstation — what does the OS actually call itself? hostname # Verify what the agent will report [System.Net.Dns]::GetHostName() That gave me WS-FRONTDESK. The server had frontdesk-pc. So I deleted the phantom client, re-registered the machine under its real hostname, and pushed the server\u0026rsquo;s preconfigured installer — the one that bakes the correct identity in instead of trusting a human to type it right:\n# Remove the dead phantom entry, register the machine by its true hostname, # then build the preconfigured client installer the server signs. backup-admin client remove --name \u0026#34;frontdesk-pc\u0026#34; backup-admin client add --name \u0026#34;WS-FRONTDESK\u0026#34; backup-admin installer build --client \u0026#34;WS-FRONTDESK\u0026#34; --out ./install-WS-FRONTDESK.exe Ran the signed installer on the box, kicked off a manual run, and watched the server. First full backup completed. Restore point exists. The phantom is gone.\n(The other valid fix is to rename the machine to match the registered name — but renaming a production workstation is a bigger blast radius than fixing a backup entry. Match the agent to the host, not the host to a typo.)\nWhy it happened Someone registered the client by hand and typed a \u0026ldquo;nicer\u0026rdquo; name than the ugly auto-generated hostname. The installer was generic, so identity was never enforced — it trusted the typed name. Two sources of truth, no reconciliation, and a monitoring view that treated \u0026ldquo;client exists and pings\u0026rdquo; as \u0026ldquo;client is protected.\u0026rdquo; Existence isn\u0026rsquo;t protection. A backup you can\u0026rsquo;t restore is just a process eating CPU.\nTakeaways Backup identity is the hostname. The registered client name must equal hostname on the machine — verify it, don\u0026rsquo;t assume it. A green agent proves nothing. Online does not mean backing up. The only real green is a recent, restorable recovery point. Use the server\u0026rsquo;s preconfigured installer. Hand-typed client names are how phantom clients are born. Bake identity in. Alert on silence. Add a \u0026ldquo;no successful backup in N hours\u0026rdquo; check so an empty backup history screams instead of sitting there looking calm. When two systems disagree about reality, suspect identity first. Auth, naming, and keys cause more \u0026ldquo;impossible\u0026rdquo; failures than disk or network ever will. ","permalink":"https://errorzap.com/posts/when-your-backup-client-lies-about-its-name/","summary":"A green-lit backup agent that hadn\u0026rsquo;t saved a single byte in weeks, because the machine and the server disagreed about what it was called.","title":"When Your Backup Client Lies About Its Name"},{"content":" A client kept their notes vault in git. Years of markdown. Meeting notes, runbooks, half-finished ideas — the kind of stuff you don\u0026rsquo;t think about until it\u0026rsquo;s gone.\nThey wanted to stop tracking one big folder. Reasonable. It didn\u0026rsquo;t belong in version control.\nSo they ran the obvious command:\ngit rm -r attachments/ Git chewed on it for a second and printed a wall of rm 'attachments/...' lines. Looked fine. Then git status showed something that made my stomach drop.\nIt wasn\u0026rsquo;t just the attachments/ folder.\ndeleted: notes/2024/standup.md deleted: notes/2024/oncall.md deleted: runbooks/dns.md ... 30,206 more ... The working tree was empty. ls came back nearly bare. 30,209 markdown files were gone from disk. Not staged-for-deletion-someday. Gone. Off the filesystem.\nThe investigation The panic version of this story ends with a restore from last night\u0026rsquo;s backup and a lost afternoon. The calm version starts with one question: where does git actually delete from?\nPeople think git rm means \u0026ldquo;stop tracking this file.\u0026rdquo; It doesn\u0026rsquo;t. git rm removes a file from two places at once: the index (staging area) and the working tree — your actual disk.\n┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ Working Tree │ │ Index │ │ HEAD/Repo │ │ (your disk) │ │ (staging) │ │ (last commit)│ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │ │ │ git rm ──► ✗ DELETES ──►│ ✗ stages delete │ ✓ STILL HERE │ │ │ git rm --cached ─────► │ ✗ untracks only │ ✓ STILL HERE ▼ ▼ ▼ file gone marked rm recoverable That last column is the whole ballgame. git rm touches the working tree and the index, but it does not touch your last commit until you actually commit the deletion.\nNobody had committed yet.\nThe files were sitting safe in HEAD the entire time, even as the disk looked like a crime scene.\nWorking Tree Index HEAD (commit) ✗ ✗ ✓ wiped staged rm 30,209 safe ► ► git rm -r → recovery lives in the column on the right The deletion never reached the last commit — that's why this was survivable. The aha If HEAD still has every file, recovery is just \u0026ldquo;copy them back out of the last commit.\u0026rdquo; No backup tape, no remote clone, no heroics.\nThe fix First, confirm the damage scope. Trust nothing, count everything:\ngit status --porcelain | grep -c \u0026#39;^D\u0026#39; # 30209 30,209 staged deletions. Matches the missing files exactly. Good — nothing weird, just one bad command.\nNow restore the working tree and reset the index straight from HEAD:\ngit restore --staged --worktree . # or, on older git: git checkout HEAD -- . Verify the files actually came back on disk before believing it:\nfind . -name \u0026#39;*.md\u0026#39; | wc -l # 30209 All present. git status clean. If HEAD had been corrupt, the same files were still in the remote — git fetch \u0026amp;\u0026amp; git restore --source=origin/main --worktree . would have done it from there.\nThen we did the thing they meant to do — untrack without deleting:\ngit rm -r --cached attachments/ echo \u0026#39;attachments/\u0026#39; \u0026gt;\u0026gt; .gitignore git commit -m \u0026#34;Stop tracking attachments/ (keep on disk)\u0026#34; --cached removes the folder from the index only. The disk is never touched. That one flag is the entire difference between a config tidy-up and a disaster.\nWhy it happened git rm carries a default that doesn\u0026rsquo;t match most people\u0026rsquo;s mental model. The intent was \u0026ldquo;git, please forget this exists.\u0026rdquo; Git heard \u0026ldquo;delete this from the index and the disk,\u0026rdquo; because that\u0026rsquo;s what it does. The -r made it recursive, and the glob pulled in far more than expected.\nThere was no malice and no bad command syntax. It did exactly what it\u0026rsquo;s documented to do. The gap was understanding that the working tree is a target, not a bystander.\nTakeaways git rm deletes from your disk. Use git rm --cached to untrack a file while keeping it on the filesystem — that\u0026rsquo;s almost always what you actually want. Count before, count after. find . -name '*.md' | wc -l on both sides of any bulk git op. A number that drops by 30,209 is not a surprise you want at commit time. Test on ONE file first. Run the destructive command against a single path, check git status and ls, then scale up. One file lost is a shrug; thirty thousand is a bad day. Don\u0026rsquo;t run destructive git in a live data directory without a backup. A notes vault is data, not just a repo. Treat it like one. Know your three trees. Working tree, index, HEAD. Recovery is trivial when you know which one still has your files — and panic when you don\u0026rsquo;t. ","permalink":"https://errorzap.com/posts/the-git-rm-that-deleted-30000-files/","summary":"One innocent \u003ccode\u003egit rm -r\u003c/code\u003e to untrack a folder, and 30,209 notes vanished off the disk in a single keystroke.","title":"The git rm That Deleted 30,000 Files"},{"content":" 📺 90s KID // The Antenna on the Roof The Sky Was the Cable Box You remember the antenna on the roof.\nThat big aluminum bone-cage bolted up by the chimney.\nPointing somewhere. Pointing at something.\nYou never knew what.\nIt just lived up there.\nCatching things out of the air.\nAnd you knew the rules without anybody teaching you.\nWhen the picture went to static, you didn\u0026rsquo;t call a repairman.\nYou didn\u0026rsquo;t reboot anything.\nYou sent a person to stand in a doorway and not move.\nThe vibes:\nthe screen fuzzed and Dad yelled \u0026ldquo;HOLD IT RIGHT THERE\u0026rdquo; somebody\u0026rsquo;s arm became part of the TV a storm was coming and you could feel the channels getting nervous the wind made the picture breathe adjusting it on the roof was a forbidden, slightly heroic act It felt like the antenna was alive.\nLike it had moods.\nLike it was listening to the sky and sometimes the sky talked back in snow.\nHere\u0026rsquo;s what it actually was.\nThat metal skeleton wasn\u0026rsquo;t decoration. It was a receiving array — a careful arrangement of metal rods cut to specific lengths, each one tuned to grab a slice of the radio spectrum where TV stations lived (VHF down low, UHF up high).\nThe stations broadcast their signal as electromagnetic waves from a tower miles away. Those waves washed over your whole neighborhood, invisible. The antenna\u0026rsquo;s job was to let those waves nudge a tiny voltage into the metal — and your TV amplified that whisper into Saturday morning cartoons.\nThe reason your cousin had to freeze in the doorway? Multipath. The signal bounced off hills, buildings, even people, and the bounced copy arrived a hair late, smearing the picture into ghosts and snow. Your cousin\u0026rsquo;s body was a wall. Moving them changed the math. You were, no exaggeration, hand-tuning a radio-frequency interference pattern with a human being.\nThe big rotator boxes that spun the antenna? Those folks were literally re-aiming at a different tower in a different town.\n| =|= \u0026lt;- the little rods caught ==|== UHF (the high channels) ===|=== | =====|===== \u0026lt;- the long rods caught | VHF (2 through 13) =====|===== | | ~ ~ ~ ((( signal from the tower ))) _____|_____ / rooftop \\ It never really left, you know.\nWe buried it in coax and called it cable. We beamed it down from dishes. We streamed it over fiber.\nBut cut the cord today and put up a cheap flat antenna, and the local channels still come down out of the sky — free, crisp, digital — same towers, same physics, no human in the doorway.\nThe sky\u0026rsquo;s still broadcasting.\nWe just stopped looking up.\n","permalink":"https://errorzap.com/posts/90s-kid-the-antenna-on-the-roof/","summary":"You remember the metal skeleton on the roof that ruled the channels and feared the wind.","title":"90s Kid - The Antenna on the Roof"},{"content":" 🖥 90s KID // The Family Computer in the Living One Beige Box, One Family, One Turn at a Time You remember the family computer.\nNot your computer. The computer.\nIt lived on a special desk in the living room or the den, glowing beige, humming softly, surrounded by a little civilization of mouse pads and floppy disks and a printer that screamed.\nThere was exactly one.\nAnd you had to share it.\nTo a kid, that machine was the most important object in the house.\nThe vibes:\nA throne. Whoever sat there ruled. A forbidden zone — \u0026ldquo;don\u0026rsquo;t touch that, you\u0026rsquo;ll break it.\u0026rdquo; A library, an arcade, and a portal, all stacked into one buzzing box. A timer-based hostage situation: \u0026ldquo;five more minutes, then it\u0026rsquo;s your sister\u0026rsquo;s turn.\u0026rdquo; A thing Dad understood and you did not, which made it slightly magic. You learned to fear the Recycle Bin like it was a black hole.\nYou learned that the screen saver — those flying toasters, that bouncing pipe maze — meant the throne was empty and the kingdom was yours.\nAnd you never turned it off wrong. Turning it off wrong felt like a crime.\nHere\u0026rsquo;s what it actually was.\nThat beige box was a personal computer — usually running Windows 95 or 98 — and the reason your whole family crowded around one was simple: they were expensive, and the idea of one-per-person hadn\u0026rsquo;t arrived yet.\nInside: a single-core processor measured in megahertz, maybe 16 to 64 megabytes of RAM, and a spinning hard drive you could hear thinking. The \u0026ldquo;do not touch\u0026rdquo; energy was real — there was no cloud, no backup, no undo. If you deleted the family photos, they were gone. That fear was justified.\nAnd those separate logins? That was Windows quietly inventing the thing your phone does now: user profiles — so Mom\u0026rsquo;s desktop, Dad\u0026rsquo;s solitaire stats, and your folder of saved game levels could coexist on one machine without a war.\nThe CRT monitor was basically a small TV bolted to the front of a vacuum tube. Hence the deep thunk and that whiff of warm static when you powered it on.\n_________________________ | _____________________ | | | | | | | C:\\\u0026gt; | | | | Welcome. | | | | ▓ flying toaster | | | | ✈ | | | |_____________________| | |_________________________| |___[ ]___[====]___[ ]____| \\_______________________\\ \u0026#39;-.___________________.-\u0026#39; [::::::::::::::::::::::] \u0026lt;- the throne The family computer never really died.\nIt just multiplied.\nThe one beige throne split into a laptop each, a phone each, a tablet on the couch. The shared machine became the personal device, and \u0026ldquo;wait your turn\u0026rdquo; became \u0026ldquo;go to your room and use your own.\u0026rdquo;\nBut every time a family streams one movie on one big screen, gathered shoulder-to-shoulder, fighting over the remote?\nThat\u0026rsquo;s the living-room computer, reincarnated.\nSame throne.\nSame five-more-minutes.\nSame warm hum of the one good screen.\n","permalink":"https://errorzap.com/posts/90s-kid-the-family-computer/","summary":"There was exactly one computer, it lived in the living room, and the whole family had to take turns being kings of it.","title":"90s Kid - The Family Computer in the Living Room"},{"content":" It was supposed to be a five-minute job.\nA client\u0026rsquo;s backup server needed one field bumped on a handful of backup clients. Retention. One value. I had a Python wrapper sitting right there with a method named exactly what I wanted: change_client_setting(). Tab-complete practically begged me to use it.\nSo I looped over the clients and called it. Twice, because I tweaked a second field on the second pass.\nThen the dashboard started showing clients with no backup schedule, no retention, no paths — like they\u0026rsquo;d never been configured. Not one client. All of them.\nThe investigation First instinct: did I fat-finger the loop? No. The script was boring. Read a name, set a field, move on.\nSecond instinct: did the service choke? Logs were clean. The server was happily applying the garbage I\u0026rsquo;d handed it.\nSo I pulled the raw settings blob the API was actually POSTing. And there it was — the settings object, wrapped inside another settings object, wrapped inside another one. A JSON turducken.\n┌─────────────────────────────────────────────┐ │ POST #1 { retention: 30, schedule: ... } │ ✓ looks fine ├─────────────────────────────────────────────┤ │ POST #2 { settings: { │ │ settings: { │ │ retention: 30, schedule: ... │ ✗ nested once │ } } } │ ├─────────────────────────────────────────────┤ │ POST #3 { settings: { settings: { │ │ settings: { ...buried... } } } } │ ✗ nested twice → unreadable └─────────────────────────────────────────────┘ ▼ server reads top level → finds nothing it recognizes every real field is now three layers down → effectively gone The \u0026ldquo;aha\u0026rdquo; Here\u0026rsquo;s the part that made my stomach drop.\nchange_client_setting() does not set one field. It round-trips the entire settings object: GET the whole blob, mutate the one key in memory, then re-serialize and POST the whole thing back.\nThat\u0026rsquo;s already more blast radius than the name implies. But the killer was the encoder. Its re-serialization had a recursive-nesting bug — each round-trip tacked the existing object inside a fresh settings wrapper instead of replacing it.\nOne call: subtly wrong but survivable. Two calls: the real fields are buried deep enough that the server can\u0026rsquo;t find them anymore. The defaults win. Every client looks wiped.\nI didn\u0026rsquo;t corrupt one setting. I corrupted the container every setting lives in — for every client — by changing a single number.\nThe fix Stop trusting the convenience method. Write a minimal-POST helper that submits only the field that changed, never the whole object — so there\u0026rsquo;s no full blob to re-encode and no encoder to trip over.\ndef safe_set_client_setting(server, client_id, key, value, dry_run=False): # Snapshot the current settings BEFORE touching anything. current = server.get_client_settings(client_id) snapshot_path = f\u0026#34;snapshots/{client_id}-{int(time.time())}.json\u0026#34; with open(snapshot_path, \u0026#34;w\u0026#34;) as f: json.dump(current, f, indent=2) if dry_run: print(f\u0026#34;[dry-run] {client_id}: {key} {current.get(key)!r} -\u0026gt; {value!r}\u0026#34;) return # Minimal POST: ONLY the one field. No round-trip of the whole blob. server.post_setting(client_id, {key: value}) # Read-back validation. Trust nothing. after = server.get_client_settings(client_id) if str(after.get(key)) != str(value): raise RuntimeError( f\u0026#34;VALIDATION FAILED {client_id}: {key} is {after.get(key)!r}, \u0026#34; f\u0026#34;expected {value!r} — restore from {snapshot_path}\u0026#34;) print(f\u0026#34;✓ {client_id}: {key} = {value} (snapshot {snapshot_path})\u0026#34;) Recovery itself was unglamorous — restore each client from the JSON snapshot I now wish I\u0026rsquo;d had before the first run:\nfor f in snapshots/*-pre-incident.json; do id=$(basename \u0026#34;$f\u0026#34; | cut -d- -f1) python restore_settings.py --client \u0026#34;$id\u0026#34; --from \u0026#34;$f\u0026#34; done Four guardrails came out of this, and they\u0026rsquo;re now non-negotiable on that box:\nsafe_set_client_setting() snapshot save .json dry-run? preview diff minimal POST one field only read-back verify ✓ old path: GET whole blob → re-encode → POST whole blob new path: never hand the encoder the full object Snapshot → dry-run → minimal write → read-back. Every mutation, every time. Why it happened The method name lied — or rather, I read a name and assumed a behavior. \u0026ldquo;Change one client setting\u0026rdquo; sounded surgical. Under the hood it was a full read-modify-write of a complex object, handed to a serializer nobody had stress-tested across repeated calls.\nA buggy encoder on a single-field write is annoying. A buggy encoder on a whole-object round-trip is a data-integrity bomb, because every write re-touches every field. The bug didn\u0026rsquo;t need to corrupt the value I changed. It corrupted the structure wrapping everything.\nAnd I had no snapshot, no read-back, and no dry-run — so the first time I learned any of this was when the dashboard went blank.\nTakeaways Read what the API actually sends, not what the method is named. \u0026ldquo;Set one field\u0026rdquo; frequently means \u0026ldquo;GET, mutate, re-serialize, POST everything.\u0026rdquo; That round-trip is your real blast radius. Prefer minimal, targeted writes. If you can POST just the changed key, do it — you never hand a fragile encoder the whole object to mangle. Validate by reading back. A write that \u0026ldquo;succeeded\u0026rdquo; tells you the server accepted bytes, not that the right value landed. Read it again and compare. Snapshot before you mutate. A pre-write JSON dump turns a catastrophe into a one-line restore. It costs milliseconds and buys you the whole night back. Ship a --dry-run for anything that touches production config. Seeing the diff before it\u0026rsquo;s real would have caught the nesting on call number one. ","permalink":"https://errorzap.com/posts/the-settings-api-that-corrupted-every-setting/","summary":"I changed one backup-retention field, ran it twice, and every client\u0026rsquo;s settings ate themselves like a snake swallowing its own tail.","title":"The Settings API That Corrupted Every Setting at Once"},{"content":" 📞 90s KID // The Sound of Dial-Up The Song the Internet Sang Before It Let You In You remember the sound.\nThat screech.\nThat hiss.\nThat little robot choir having a full-blown meltdown inside the beige box next to the family computer.\nYou\u0026rsquo;d double-click the little globe icon.\nAnd then you\u0026rsquo;d wait.\nFirst the dial tone. Then the boop-beep-boop of a phone number being typed by a ghost. Then\u0026hellip; the scream.\nWe had no idea what it was.\nWe just knew it meant the internet was coming.\nThe kid\u0026rsquo;s-eye view of dial-up was pure superstition. The vibes:\nYou could not touch the phone. Pick up the kitchen handset and you\u0026rsquo;d kill the internet. This was law. Mom yelling \u0026ldquo;ARE YOU ONLINE?!\u0026rdquo; up the stairs. The noise felt like a password the computer whispered to the sky. If it connected on the first try, that was a good omen. The day would go well. \u0026ldquo;Busy signal\u0026rdquo; meant the internet was simply\u0026hellip; full. Try again later. It was forbidden, expensive, and a little bit magic. You were visiting the internet, not living in it. You\u0026rsquo;d sit there hoping it wouldn\u0026rsquo;t say that other sound — the sad descending bweeeoooo of a failed handshake. The dial-up walk of shame.\nSo what WAS that noise, really?\nHere\u0026rsquo;s the nerdy truth: that screech was two modems negotiating out loud.\nYour modem and the one at the ISP were on a phone line — a thing built for human voices, not data. So they had to figure out, in real time, how to fake it. The handshake went like this:\nA tone goes out: \u0026ldquo;Hi, I\u0026rsquo;m a modem, anyone home?\u0026rdquo; The other side answers with its own tone. Then that wall of static? That\u0026rsquo;s both modems probing the line — testing how noisy it was, what frequencies survived the trip, how fast they could safely talk. They agreed on a speed (the holy 56k), did an echo-cancellation handshake, and went quiet. The screaming was the only part you were meant to hear. Once they shook hands, the modem muted itself. Silence meant you were in.\nYou were literally eavesdropping on two machines learning to understand each other.\n___________________________ | .---------------------. | | | CONNECTING... | | | | | | | | bzzt kshhhh PING | | | | /\\ /\\/\\ /\\ /\\ | | | | / \\/ \\/\\/ \\/ \\ | | | \u0026#39;---------------------\u0026#39; | | [::::] 56k [ MODEM ] | |___________________________| | | | | ===( )==phone==( )=== It never really left, you know.\nThe handshake just got quiet. Your phone still negotiates with a cell tower every time it wakes up — speed, frequency, signal quality — the exact same dance. Your Wi-Fi does it. Your laptop does it a thousand times a day. Every device you own is still screaming that song.\nIt just learned to do it under its breath.\nWe were the last kids who got to hear the internet think.\nAnd honestly?\nIt sounded like the future.\n","permalink":"https://errorzap.com/posts/90s-kid-the-sound-of-dial-up/","summary":"You remember the song the internet sang before it would let you in.","title":"90s Kid - The Sound of Dial-Up"},{"content":"The Box on the Side of the House If you grew up in the 90s, you probably remember the box.\nEvery house had one.\nGray. Boring. Bolted to the siding like it was issued by the government.\nUsually with a sticker that said DO NOT OPEN, which of course meant this is extremely important.\nAs kids, we didn’t know what it did.\nWe just knew:\nIt wasn’t ours Adults avoided explaining it And touching it felt like a crime So we stared at it.\nRespectfully. Suspiciously. Like it might bite.\nThat box is called a demarcation box. But you didn’t need to know that then. What mattered was the vibe: this thing runs something big.\nYears later, you finally see a technician open it. Inside?\nWires. Labels. Splitters. Grounding blocks.\nNot magic. Just infrastructure.\nAnd here’s the punchline:\nThat same boring gray box still runs your life.\nYour internet comes through it.\nYour phone signal passes through it.\nYour work, your streaming, your security cameras, your “why is the Wi-Fi down” moments — all of it depends on that ugly little rectangle.\nThe cartoon isn’t about nostalgia for old tech.\nIt’s about remembering a time when infrastructure was visible, mysterious, and slightly scary.\nWe didn’t understand it.\nBut we knew it mattered.\nAnd the funniest part?\nThe box never went away.\nWe just finally learned its name.\n","permalink":"https://errorzap.com/posts/90s-kid-box-on-side-of-house/","summary":"Every 90s house had a boring gray box bolted to the siding marked DO NOT OPEN. Turns out it still runs your whole digital life.","title":"90s Kid - The Box on the Side of the House"},{"content":"90s kid memories • cable internet origin story\nI’m Not Old — I Watched the Internet Get Installed Backhoes, giant coax, and “future services” that turned into broadband.\nOne of my most specific “90s kid” memories isn’t a TV show or a toy — it’s standing outside watching the cable company tear up the neighborhood. I remember the trencher, the fresh dirt, the orange cones, and those giant spools of coax that looked way too serious for “just TV.” It was the kind of infrastructure that made a kid stop and stare because it felt like something important was happening.\nI was told back then it was for internet — for “the future.” The crew answers were always vague, like they were trained to say the same line:\n“Future services.”\nAt the time, that phrase sounded like marketing fog. But a few years later, cable modems showed up and the fog cleared: that trenching was the physical foundation of broadband. The internet didn’t magically appear — it got installed.\nWhat Was Actually Happening (The Non-Marketing Version) Cable TV networks were originally designed as a one-way broadcast system: signals go out, everyone receives them. Internet requires two-way traffic — your computer has to talk back. So in the 90s, cable companies started upgrading their “plant” so it could handle both directions and higher quality signals.\n🧩 Quick mental model: Hybrid Fiber-Coax (HFC)\nHeadend\nCable company core\n➜\nFiber trunk\nLight to the neighborhood\n➜\nOptical node\nLight → RF\n➜\nCoax plant\nCopper to houses\nTranslation: most neighborhoods got fiber to a local node, then coax the rest of the way. The big cable you saw trenched was likely part of that coax distribution rebuild — the “last mile” TV infrastructure becoming a data network.\nHow Early Cable Modems Worked (In Human Language) Early cable modems weren’t just “Ethernet devices.” They were more like a TV tuner and a radio modem combined. The network used RF channels (the same kind of spectrum used for cable TV), and everyone in the neighborhood shared it.\n📥 Downstream (download)\nHigh frequency RF channel(s) Everyone listens Your modem filters what’s “for you” Fast compared to upstream 📤 Upstream (upload)\nLower frequencies (noisier) Shared “talk-back” channel Modems take turns transmitting Historically the weak link In the late 90s, the industry standardized around DOCSIS (the CableLabs spec that made cable modems interoperable). That’s when cable internet stopped being a chaotic science project and became a scalable product: authentication, provisioning, and managed speeds.\nThe 90s Kid Part: We Saw the Seam What makes this feel so “90s kid” isn’t just the tech — it’s that we watched the transition happen in public. Infrastructure wasn’t hidden. Progress had a soundtrack: trucks, compressors, trenchers, and the constant beep-beep of heavy equipment reversing.\n📼 90s kid sensory checklist (if you know, you know)\nStanding outside watching crews work because it was actually interesting Everyone asking “what are they putting in?” and getting vague answers Dial-up sounds inside the house, construction sounds outside Then one day: always-on internet, like a cheat code And the part that’s equal parts funny and existential: a lot of that buried coax is still there. It might be reused, re-terminated, amplified differently, or partially replaced — but the “future services” foundation is still under the street.\nSo… Am I Old? Maybe a little. But mostly I’m just time-privileged. I didn’t read about broadband arriving — I watched it get installed. The internet wasn’t a cloud. It was dirt, cable, and a crew saying “future services” like a prophecy.\nWritten by someone who remembers when the internet smelled like hot asphalt and fresh trench dirt.\n","permalink":"https://errorzap.com/posts/watching-birth-of-internet-before-we/","summary":"A 90s-kid origin story: backhoes, giant coax, and the future services that turned into broadband.","title":"I’m Not Old — I Watched the Internet Get Installed"},{"content":"Stop Wasting API Calls: A Practical Guide to Multi-Tier AI Model Systems How to run 24/7 AI automation without burning through your quota or your budget\n⚡ Want the detailed technical implementation?\nThis post covers the concepts and benefits. For the full technical guide with code examples, configuration files, and monitoring setup, scroll down to the Complete Implementation Guide section below.\nThe Problem: Using Premium Models for Everything If you\u0026rsquo;re running AI automation—whether it\u0026rsquo;s a personal assistant, business monitoring, or development tools—you\u0026rsquo;ve probably noticed your API usage climbing fast.\nHere\u0026rsquo;s what most people do wrong: they pick their favorite model and use it for everything.\n\u0026ldquo;I like Claude Sonnet, so that\u0026rsquo;s what I use\u0026rdquo; \u0026ldquo;GPT-4 is the best, I\u0026rsquo;ll just use that\u0026rdquo; \u0026ldquo;Gemini Pro is good enough for most things\u0026rdquo; This approach has two big problems:\nYou burn through API quotas unnecessarily You pay more than you need to The reality is that most AI tasks don\u0026rsquo;t require premium models. You\u0026rsquo;re using a sledgehammer to crack walnuts.\nThe Solution: Match Model Cost to Task Complexity Think of AI models like tools in a workshop. You wouldn\u0026rsquo;t use a precision laser cutter to cut plywood. You use:\nCheap tools for simple, repetitive tasks Balanced tools for everyday work Premium tools only when you actually need the capability The same logic applies to AI models.\nThe 3-Tier System 💵 Tier 1: Background Workers (Cheap \u0026amp; Fast) Models: Gemini Flash, Claude Haiku, DeepSeek V3\nCost: $0.10 - $0.50 per million tokens\nSpeed: Very fast responses\nUse for:\nScheduled tasks and cron jobs File operations (move, copy, rename, organize) Simple monitoring (is X up? did Y complete?) Data extraction from logs or files Basic yes/no questions Heartbeat checks Example:\nTask: Check if server 192.168.1.100 responds to ping\nTier 1 Response: \u0026ldquo;Server is up. Response: 12ms\u0026rdquo;\nCost: ~$0.0001\nWhy it works: These tasks don\u0026rsquo;t require reasoning, creativity, or complex understanding. They\u0026rsquo;re simple data retrieval or yes/no answers. Cheap models handle them perfectly.\n🧠 Tier 2: Daily Driver (Balanced) Models: Claude Sonnet, Gemini Pro, GPT-4o\nCost: $3 - $15 per million tokens\nSpeed: Good balance\nUse for:\nAll normal chat conversations (your default) Code writing and review Research and analysis Documentation Email composition Most technical troubleshooting Content creation Example:\nTask: Summarize 50 emails and prioritize urgent ones\nTier 2 Response: Detailed summary with context and priorities\nCost: ~$0.05-0.10\nWhy it works: Tier 2 models are smart enough for 90% of what you\u0026rsquo;ll throw at them. They understand context, can reason through problems, and write quality content. This should be your default for anything interactive.\n🚀 Tier 3: The Heavy Hitters (Premium) Models: Claude Opus, GPT-4 (full)\nCost: $15 - $75 per million tokens\nSpeed: Slower, but most capable\nUse for:\nComplex architecture decisions Multi-step reasoning with many variables Novel problem-solving (no clear solution path) When Tier 2 has tried and failed multiple times High-stakes content (legal, financial, critical business decisions) Example:\nTask: Debug a subtle async race condition in distributed system\nAfter: Tier 2 tried 3 approaches and failed\nTier 3 Response: Identified timing issue with detailed trace\nCost: ~$2-5\nWorth it: Saved 4-6 hours of manual debugging\nWhy it works: Tier 3 models have the best reasoning capabilities. But you only need that extra power occasionally. Use it strategically, not by default.\nReal Example: Daily Automation Workflow Here\u0026rsquo;s how a typical day breaks down:\n6:00 AM - Morning Checks (Tier 1) Check server status: 5 servers Count unread emails Review calendar Check backup completion Cost: ~$0.002 total\nThroughout Day - Interactive Work (Tier 2) 10 chat conversations 3 code reviews 2 email summaries 1 documentation update Cost: ~$0.40 total\nOccasionally - Complex Problem (Tier 3) Maybe once a week Usually after Tier 2 can\u0026rsquo;t solve it Cost: ~$2-5 per use\nMonthly Total: $15-25 depending on heavy problem-solving needs\nCommon Mistakes to Avoid Mistake Problem Solution ✗ Using Premium Models by Default Burns through quota and budget Set Tier 2 as your default, escalate only when needed ✗ Using Cheap Models for Complex Tasks Wastes time with poor results If unsure, start with Tier 2. Downgrade later if it\u0026rsquo;s overkill. ✗ Not Tracking Usage Can\u0026rsquo;t identify what\u0026rsquo;s expensive Log every call. Review weekly. ✗ Manual Model Switching You\u0026rsquo;ll forget or choose wrong Automate tier selection based on task type Quick Reference Template Copy this into your system prompt:\nYou are a cost-efficient AI assistant using a tiered model system: TIER 1 (Background): Gemini Flash, Claude Haiku - Cost: $0.10-0.50 per 1M tokens - Use for: Scheduled tasks, monitoring, file ops, simple queries - Auto-select for all background work TIER 2 (Default): Claude Sonnet, Gemini Pro - Cost: $3-15 per 1M tokens - Use for: Chat, code, research, analysis - Your default for all interactive work TIER 3 (Premium): Claude Opus, GPT-4 - Cost: $15-75 per 1M tokens - Use for: Complex reasoning, after Tier 2 fails - ALWAYS ask permission before using RULES: 1. Background/automated = Tier 1 (automatic) 2. Interactive/chat = Tier 2 (default) 3. Complex/failed attempts = Tier 3 (ask first) 4. Log all usage 5. Alert at 80% monthly budget Conclusion The goal isn\u0026rsquo;t to use the cheapest model possible. It\u0026rsquo;s to match model capability to task complexity.\nSimple tasks → Simple models Normal work → Balanced models Hard problems → Premium models This approach:\nReduces API quota usage by 60-80% Lowers costs significantly Maintains quality where it matters Reserves premium models for when you actually need them Start simple:\nMove background tasks to Tier 1 Keep Tier 2 as your default Use Tier 3 strategically Ready to implement this yourself? Keep reading for the complete technical guide.\nComplete Implementation Guide: Building Your Tiered AI System The technical details, configuration examples, and monitoring setup that powers my always-on AI assistant\nThe Four-Tier Architecture My production system actually uses four tiers, not three. The fourth tier adds fallback redundancy that prevents failures when primary services have issues.\n┌─────────────────────────────────────────────────────────┐ │ TIER 1: Gemini Flash (Primary) │ │ Cost: $0.10/M input, $0.40/M output │ │ Use: 95% of all requests │ │ Handles: Summaries, Q\u0026amp;A, simple code, background tasks │ └─────────────────────────────────────────────────────────┘ │ ▼ (rate limited or down) ┌─────────────────────────────────────────────────────────┐ │ TIER 2: OpenRouter Bridge │ │ Cost: $0.075-$0.30/M (varies by model) │ │ Use: Fallback when Gemini fails │ │ Models: Gemini Flash Lite, DeepSeek V3 │ └─────────────────────────────────────────────────────────┘ │ ▼ (still failing) ┌─────────────────────────────────────────────────────────┐ │ TIER 3: Claude Haiku │ │ Cost: $0.80/M input, $4/M output │ │ Use: Emergency fallback only │ │ When: Both Gemini and OpenRouter unavailable │ └─────────────────────────────────────────────────────────┘ │ ▼ (explicit user request) ┌─────────────────────────────────────────────────────────┐ │ TIER 4: Claude Sonnet/Opus │ │ Cost: $3-15/M input, $15-75/M output │ │ Use: Complex reasoning, architecture, code review │ │ When: User explicitly invokes or task requires it │ └─────────────────────────────────────────────────────────┘\nTier 1: Gemini Flash (The Workhorse) Model: gemini-2.0-flash\nCost: $0.10/M input, $0.40/M output\nContext Window: 1,000,000 tokens\nRate Limits (Paid Tier 1): 2,000 RPM, 4M tokens/minute\nHandles 95% of all requests:\nCron jobs - Daily summaries, health checks, weather alerts Heartbeat tasks - Keeping the assistant \u0026ldquo;warm\u0026rdquo; every 2 hours Simple queries - \u0026ldquo;What time is it in Tokyo?\u0026rdquo; doesn\u0026rsquo;t need Opus Text processing - Summarization, formatting, extraction Background workers - Tasks that run while you sleep Why Gemini Flash? Massive context window - 1M tokens means it can ingest entire codebases Speed - Flash is fast, responses in under a second Cost - At $0.10/M input, you can process 10 million tokens for a dollar Google\u0026rsquo;s free tier - 15 requests/minute free, 1,500/day (but paid tier recommended for reliability) Configuration Example { \u0026#34;google\u0026#34;: { \u0026#34;baseUrl\u0026#34;: \u0026#34;https://generativelanguage.googleapis.com/v1beta\u0026#34;, \u0026#34;apiKey\u0026#34;: \u0026#34;YOUR_API_KEY\u0026#34;, \u0026#34;models\u0026#34;: [ { \u0026#34;id\u0026#34;: \u0026#34;gemini-2.0-flash\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;Gemini 2.0 Flash\u0026#34;, \u0026#34;cost\u0026#34;: { \u0026#34;input\u0026#34;: 0.1, \u0026#34;output\u0026#34;: 0.4 }, \u0026#34;contextWindow\u0026#34;: 1000000, \u0026#34;maxTokens\u0026#34;: 8192 } ] } } Tier 2: OpenRouter (The Safety Net) What is OpenRouter? A unified API gateway that routes to 100+ models from different providers. One API key, access to everything.\nWhy use it as a fallback?\nWhen Gemini hits rate limits or goes down (it happens), OpenRouter provides instant failover to alternative models.\nThe Fallback Chain { \u0026#34;model\u0026#34;: { \u0026#34;primary\u0026#34;: \u0026#34;google/gemini-2.0-flash\u0026#34;, \u0026#34;fallbacks\u0026#34;: [ \u0026#34;openrouter/google/gemini-2.5-flash-lite\u0026#34;, \u0026#34;openrouter/deepseek/deepseek-chat-v3-0324\u0026#34;, \u0026#34;anthropic/claude-haiku-4\u0026#34; ] } } When a request fails:\nTry Gemini Flash directly → 429 rate limit Try Gemini Flash Lite via OpenRouter → works, costs $0.075/M If that fails, try DeepSeek V3 → works, costs $0.14/M Last resort: Claude Haiku → works, costs $0.80/M The system automatically retries down the chain. User never sees an error.\nTier 3: Claude Haiku (Emergency Fallback) Model: claude-haiku-4\nCost: $0.80/M input, $4/M output\nWhen used: Only when Tiers 1 and 2 both fail\nHaiku is the \u0026ldquo;never fail\u0026rdquo; option. Anthropic\u0026rsquo;s infrastructure is rock solid. If Gemini is down AND OpenRouter is having issues, Haiku catches everything.\nAt 10x the cost of Gemini, you don\u0026rsquo;t want this firing constantly. That\u0026rsquo;s where monitoring comes in.\nTier 4: Claude Sonnet/Opus (The Heavy Artillery) Models: claude-sonnet-4-5, claude-opus-4-5\nCost: $3-15/M input, $15-75/M output\nWhen used: Complex reasoning, code architecture, explicit requests\nReserved for tasks that actually need them:\nMulti-file code refactoring System architecture decisions Complex debugging requiring deep reasoning When the user explicitly asks for the \u0026ldquo;big brain\u0026rdquo; Model Alias System { \u0026#34;models\u0026#34;: { \u0026#34;anthropic/claude-sonnet-4-5\u0026#34;: { \u0026#34;alias\u0026#34;: \u0026#34;sonnet\u0026#34; }, \u0026#34;anthropic/claude-opus-4-5\u0026#34;: { \u0026#34;alias\u0026#34;: \u0026#34;opus\u0026#34; }, \u0026#34;google/gemini-2.0-flash\u0026#34;: { \u0026#34;alias\u0026#34;: \u0026#34;gemini-flash\u0026#34; } } } User can type /use opus to explicitly switch, but defaults stay cheap.\nMonitoring: Catching Runaway Costs The tier system only works if you monitor it. I run a Windows Task Scheduler job every 30 minutes that:\nParses logs for model usage Counts by model - How many Haiku? Sonnet? Opus? Checks thresholds - Opus should be 0 for background tasks Sends Telegram alerts if something\u0026rsquo;s wrong Alert Thresholds $maxOpusPerHour = 1 # Opus should NEVER be used by cron jobs $maxHaikuPerHour = 5 # Haiku means Gemini is failing $maxSonnetPerHour = 20 # Runaway conversation detection $max429PerHour = 10 # Rate limit problems What Triggers Alerts Condition Alert Opus used at all 🚨 OPUS USED - Check fallback config! Haiku \u0026gt; 5/hour ⚠️ Haiku fallback triggered - Gemini may be failing 402 errors 🚨 PAYMENT REQUIRED - Credits depleted! 429 \u0026gt; 10/hour ⚠️ Rate limit errors - API quota issues If I wake up to a Telegram message, something\u0026rsquo;s wrong. No message = system healthy.\nReal-World Cost Comparison Before (Everything on Claude) Task Model Daily Calls Cost/Day Cron jobs Sonnet 50 $2.25 Heartbeats Haiku 12 $0.10 User chat Sonnet 200 $9.00 Background Sonnet 100 $4.50 TOTAL $15.85/day After (Tiered System) Task Model Daily Calls Cost/Day Cron jobs Gemini Flash 50 $0.02 Heartbeats Gemini Flash 12 $0.005 User chat Gemini Flash 180 $0.07 User chat (complex) Sonnet 20 $0.90 Background Gemini Flash 100 $0.04 TOTAL $1.04/day Savings: 93%\nFrom $15.85/day to $1.04/day\nMonthly: $475 → $31\nImplementation Tips 1. Start with logging before switching Track what models are being used and why before changing anything. You might find 80% of your expensive calls are for simple tasks.\n2. Use model aliases Make it easy to switch: gemini-flash, sonnet, opus. Users shouldn\u0026rsquo;t memorize model IDs.\n3. Set up alerting immediately The moment you deploy a fallback system, monitor it. A misconfigured fallback chain can burn through credits overnight.\n4. Test your fallbacks Deliberately rate-limit yourself and verify the chain works:\n# Simulate Gemini failure curl -X POST your-api -H \u0026#34;X-Force-Fallback: true\u0026#34; 5. Consider task-specific routing Some tasks should always use a specific tier:\nSummarization → Always Tier 1 Code review → Always Tier 4 Health checks → Always Tier 1 The Configuration That Runs My System { \u0026#34;agents\u0026#34;: { \u0026#34;defaults\u0026#34;: { \u0026#34;model\u0026#34;: { \u0026#34;primary\u0026#34;: \u0026#34;google/gemini-2.0-flash\u0026#34;, \u0026#34;fallbacks\u0026#34;: [ \u0026#34;openrouter/google/gemini-2.5-flash-lite\u0026#34;, \u0026#34;openrouter/deepseek/deepseek-chat-v3-0324\u0026#34;, \u0026#34;anthropic/claude-haiku-4\u0026#34; ] }, \u0026#34;heartbeat\u0026#34;: { \u0026#34;model\u0026#34;: \u0026#34;gemini-flash\u0026#34;, \u0026#34;every\u0026#34;: \u0026#34;2h\u0026#34; } } } } Final Thoughts The \u0026ldquo;just use GPT-4/Claude for everything\u0026rdquo; approach is dead. Modern AI infrastructure requires the same thinking we apply to any distributed system:\nUse the cheapest resource that works Have fallbacks for reliability Monitor everything Reserve expensive resources for when they\u0026rsquo;re needed My bot now handles thousands of daily interactions for about a dollar. The expensive models are still there when I need them - but they\u0026rsquo;re not wasting money on \u0026ldquo;what\u0026rsquo;s the weather?\u0026rdquo; queries.\nBuild your tiers. Set your fallbacks. Sleep peacefully while your AI runs on pennies.\nQuestions? Running an always-on AI assistant?\nDrop a comment. I\u0026rsquo;d love to hear how you\u0026rsquo;re handling costs.\nAuthor: PuebloKC\nRunning OpenClaw AI automation system\nFebruary 2026\n","permalink":"https://errorzap.com/posts/openclaw-smart-ai-model-routing-to/","summary":"Stop Wasting API Calls:A Practical Guide to Multi-Tier AI Model Systems How to run 24/7 AI automation without burning through your quota or your budget ⚡ W","title":"OpenClaw Smart AI Model routing to reduce cost"},{"content":"Google Drive and OneDrive are sync tools, not backups.\nIf ransomware hits or files get deleted, they sync that too—straight into the void.\n☁️ Backup Rule 101: Always keep at least one offline or immutable copy of your data.\n💾 True backup = a separate system that keeps previous versions safe from sync errors or hacks.\n#TechTips #BackupStrategy #CyberSecurity #DataProtection\n","permalink":"https://errorzap.com/posts/google-drive-and-onedrive-are-not/","summary":"Google Drive and OneDrive are sync tools, not backups.If ransomware hits or files get deleted, they sync that too—straight into the void.☁️ Backup Rule 101","title":"Google Drive and OneDrive ARE NOT backups"},{"content":"Trying to improve my awful systems and memory with automation and Ai developed solutions.\nBrainscan eats data from multiple apps and sources and then displays it and allows it marked for bill etc.\nBrainDump is a simple web app for quickly entering a note or idea about anything. This gets saved and categorized for later use.\nWorklog is a simple way to enter work for clients, which feeds into Brainscan as well.\nThe final piece is the call the nerd. Com final front-end finishing then we will funnel that into the same system along with alert\nNow I have a server that processes every random thought or idea I have for later use, auto creates tasks and events, auto creates invoices and billing info.\nAll hosted on a $35 virtual cloud server\n","permalink":"https://errorzap.com/posts/my-digital-brain-developed-with-ai/","summary":"Trying to improve my awful systems and memory with automation and Ai developed solutions.Brainscan eats data from multiple apps and sources and then displa","title":"My digital brain developed with AI"},{"content":"\n🔐 Have You Enabled 2FA Everywhere? Changed Your Password This Decade?\nLet’s start with a gut check.\nDo you reuse the same password across multiple sites?\nIs your email password the same one you used in 2013?\nHave you enabled two-factor authentication (2FA) on your most important accounts?\nOr\u0026hellip; are you still getting text codes (yikes) instead of using an authenticator app?\nIf any of the above made you uncomfortable — good. That’s the point.\n🧠 Why This Still Matters\nEvery week we clean up the mess after someone gets locked out of an account, or worse, loses access due to a breach. And 99% of the time? It could’ve been prevented with a simple change:\n✅ Stronger, unique passwords\n✅ Two-Factor Authentication (2FA)\nStill think you’re safe because “nobody would hack me”? Newsflash: they don’t target you. They target everyone, and your password is probably already floating around out there. Just check https://haveibeenpwned.com if you don’t believe me.\n🛡️ Do This Now:\nEnable 2FA (Everywhere)\nStart with:\nEmail\nBank\ncloud storage\nSocial media\nAny sites with your personal data or billing info\nUse an authenticator app (like Authy, Microsoft Authenticator, or Google Authenticator). Not SMS. Text-based 2FA can be hijacked.\nUse a Password Manager Bitwarden, 1Password, or even the built-in tools from Apple or Google are far better than your sticky note or Excel sheet.\nChange Old Passwords If your password is:\nOver 5 years old\nShared across more than one site\nOr contains your pet\u0026rsquo;s name and birth year\u0026hellip;\nChange. It. Now.\nUse 12+ characters, mix it up, and stop using \u0026ldquo;!\u0026rdquo; at the end to feel secure. Hackers are wise to that game.\n⚙️ Bonus Tip: Turn on Login Alerts\nMost sites let you enable notifications when someone logs in from a new device. Do that. It’s like a smoke alarm for your accounts.\nFinal tip: listen to your IT people, ensure you have working backups, and someone monitoring the security of your devices, servers, and other online assets.\nAnd update your end of life networking devices and computers!\n🔚 Final Thoughts\nCybersecurity isn’t just for IT nerds. It’s for everyone. The smallest effort (like enabling 2FA) goes a long way in preventing a total digital meltdown.\nAnd if it’s too much? That’s what we’re here for.\n!Need help securing your accounts or deploying 2FA for your business? Contact us mypueblopc.com\n","permalink":"https://errorzap.com/posts/online-security-gut-check-secure/","summary":"🔐 Have You Enabled 2FA Everywhere? Changed Your Password This Decade?Let’s start with a gut check.Do you reuse the same password across multiple sites?Is y","title":"online security gut check. secure yourself now. "},{"content":"\nThere\u0026rsquo;s been a massive spike in real-world malware attacks lately on average businesses—not just attempts, but full-blown breaches where threat actors gain complete remote control of systems.\nThese attacks almost always end in ransomware, data theft, or total system compromise.\nWhere I used to see occasional phishing or intrusion attempts, I’m now seeing successful attacks almost daily. Just this week, one careless click on a fake file by a single employee could have given attackers full access to an entire company\u0026rsquo;s network, backups, and data—completely wiping everything out.\nIf proper security tools and procedures hadn’t been in place, that business would have been destroyed.\nThis stuff is as serious—and terrifying—as it gets. And yet, far too many business owners and individuals still believe it “won’t happen to them.”\n","permalink":"https://errorzap.com/posts/ransomware-its-like-escape-room-but/","summary":"There\u0026rsquo;s been a massive spike in real-world malware attacks lately on average businesses—not just attempts, but full-blown breaches where threat actors gain","title":"Ransomware: It’s Like an Escape Room… But With Your Business."},{"content":"\nWhen nasal spray just won’t cut it, light the fuse of freedom.\n","permalink":"https://errorzap.com/posts/explosive-home-remedies/","summary":"When nasal spray just won’t cut it, light the fuse of freedom.","title":"Explosive home remedies "},{"content":"Eyebong™: The first ophthalmic cannabinoid serum with mind-expanding vision technology. See the truth. Or go blind trying.\n","permalink":"https://errorzap.com/posts/eyebong/","summary":"Eyebong™: The first ophthalmic cannabinoid serum with mind-expanding vision technology. See the truth. Or go blind trying.","title":"EyeBong"},{"content":"\nToday’s biggest crisis? Logging into Google.\n(Yep. That was it. The login screen. The pinnacle of chaos.)\nSo I’m officially calling it a success.\nNever mind the fact that I spent the weekend elbows-deep in outdated machines, broken updates, stubborn printers, and a firewall that decided it had free will.\nEverything looked smooth because I spent hours making sure it was.\nNo one notices when things don’t go wrong—\nbecause I already fought all the gremlins before sunrise.\nYou\u0026rsquo;re welcome.\n","permalink":"https://errorzap.com/posts/no-fires-today-because-i-put-them-out/","summary":"Today’s biggest crisis? Logging into Google.(Yep. That was it. The login screen. The pinnacle of chaos.)So I’m officially calling it a success.Never mind t","title":"No Fires Today (Because I Put Them Out at 4AM)"},{"content":"\nAh yes, the sacred ritual of modern IT.\nEvery day I wake up, stretch, and immediately begin my daily chant:\n“Have you tried turning it off and on again?”\nMy hands? Permanently in “Ctrl + Alt + Delete” formation.\nMy coffee? 98% bitterness, 2% thermal paste.\nMy brain? Running on 3 Chrome tabs, one of which is frozen.\nUsers be like:\n“It just stopped working.”\nOh did it? Did it just stop, Barbara? Or did you feed it malware, deny the updates, ignore 13 error messages, and then scream when it exploded?\nI am the keeper of passwords no one remembers, the slayer of printers with free will, and the high priest of “Why the hell is this plugged into HDMI 3?”\nSometimes I wonder…\nWhat if I clicked myself into safe mode?\nWould I finally get peace?\nBut no.\nHere I go again.\nClick this. Click that.\nPray to the Wi-Fi gods.\nSacrifice a USB stick.\nAnd hope—just hope—the server doesn’t reboot during lunch.\n#TechLife #DigitalShaman #CtrlAltPray\n","permalink":"https://errorzap.com/posts/click-this-click-that-install-this-run/","summary":"Ah yes, the sacred ritual of modern IT.Every day I wake up, stretch, and immediately begin my daily chant:“Have you tried turning it off and on again?”My h","title":"Click this. Click that. Install this. Run that. Update this. Reboot that."},{"content":"Oompa Loompa, doopatty Zone,\nLet me explain what Prednisolone\u0026rsquo;s known:\nIt fights inflammation, that’s its big skill—\nBut oh, the side effects? They\u0026rsquo;re kind of a pill.\nOompa Loompa, roid-powered ride,\nHere’s what can happen on the steroid tide:\nYour bones get brittle, your mood might swing,\nYou’ll crave weird snacks and curse everything.\nIt zaps your immune like a ninja in black,\nBut robs your potassium, sneaks out the back.\nIt fluffs your face, may give you the shakes,\nAnd keeps you up counting sheep-fueled earthquakes.\nOompa Loompa, pred-sanity plea,\nTaper it slow and eat potassium, see?\nMagnesium helps, and some D on the side—\nOr else it’s moonface and a cortisol slide\n","permalink":"https://errorzap.com/posts/prednisolone/","summary":"Oompa Loompa, doopatty Zone,Let me explain what Prednisolone\u0026rsquo;s known:It fights inflammation, that’s its big skill—But oh, the side effects? They\u0026rsquo;re kind of","title":"prednisolone "},{"content":"\nIt’s on Social Media If it’s on Facebook, TikTok, or Instagram—it’s a scam.\nDoesn’t matter if it’s a giveaway, a job offer, or your grandma tagging you in a CBD oil pyramid scheme.\nRule of thumb: If you learned about it between cat videos and a guy eating drywall—it’s fake.\nSomeone Calls YOU No one calls you to give you money.\nNot the IRS, not Microsoft, not “Visa Fraud Department.”\nHell, even your friends barely call anymore. If someone dials you first, assume they want your soul, your wallet, or both.\nThey Say “Kindly” “Kindly click the link below…”\n“Kindly confirm your identity…”\n“Kindly ignore the fact this was written by a Nigerian AI on a sugar high.”\nReality check: No legit person says “kindly” unless they’re about to scam you or they’re 110 years old.\nThey Say They’re the Government Ah yes, the government reaching out to help you with money, jail, or tech support.\nSpoiler:\nThe IRS doesn’t email.\nThe FBI doesn’t text.\nSocial Security doesn’t DM you at 3AM.\nAnd if they’re calling from a “restricted number,” it’s either a scammer… or your ex.\n","permalink":"https://errorzap.com/posts/ways-to-spot-scam-without-even-updating/","summary":"\u003col\u003e\n\u003cli\u003eIt’s on Social MediaIf it’s on Facebook, TikTok, or Instagram—it’s a scam.Doesn’t matter if it’s a giveaway, a job offer, or your grandma tagging you in\u003c/li\u003e\n\u003c/ol\u003e\n","title":"Ways to Spot a Scam (Without Even Updating Your Firewall)"},{"content":" 💭 ERROR ZAP // ARCHIVE My life So all of my life, I knew something was perhaps different about me. I would see others socializing, having fun, playing. However, I could not do those same things the majority of the time. Noises that no one else hears, bother me intensely. Clothing is a constant source of stress, not for the looks, but the feel. And as of a few months ago, I finally found a reason. And a group of people who feel and act the same way. I found out what Aspergers Syndrome was, and relate very closely to it. While I have no official diagnosis, their is no doubt in my mind that this is me.\nLets start with one of my major problems, shall we? Sound. I hate any sound that is irregular, or of some certain pitch and tone. I can hear the irregularities in a fan motor, and they cause me to find a way to make it stop. Usually taping some of the vent where it blows out will cause the irregularity to go away. I can hear the buzzing of plug-in transformers, and it is painful! Going to a restaurant or nearly all public places faces me with a daunting task of dealing with all of these noises. The fluorescent lights humming, people laughing, talking, , and the various other noises that people make. They all make life very difficult, I cant simply tune them out, or ignore them. A quiet tap, is like a pounding hammer. Constantly distracting me, and causing a weird form of anger. A\nClothing has been a lifelong battle for me, one that is fought daily. Clothing, to most is a statement of their style, and who they are. To me, clothing is simply a daily annoyance. I dont really care what my clothes look like, so long as they are comfortable and they dont irritate me in any way. The seems, patches, fabric, and tightness are all contributers to how I feel. If my clothing doesnt feel good on my skin, my mood instantly goes bad, and I want nothing more than to be free of the bad feeling clothing. If my clothes are uncomfortable, I can not concentrate, work, or do much of anything. As a result of all this, I wear the same shirt and shorts on a daily basis. I get a lot of crap from others about this, but quite honestly they will never understand any of my reasoning. When I get a shirt, or shorts, or anything that \u0026ldquo;feels\u0026rdquo; good to me I hold onto it for life! I put my shirt and shorts on every day, knowing they are broken in, they wont poke or scratch my skin, and nothing about them bothers me. This alleviates a very large part of the daily stress I deal with. I will wear this clothing until it is battered, torn, and the fabric nearly see through from wear.\nSocial interaction is another huge issue for me. It has never come to me easily, and doesn\u0026rsquo;t seem to get any easier through life. I have a huge craving for social interaction, but when i actually interact, I find myself wanting to run and hide from everyone. It is a never ending fight, one that results in me being alone the majority of the time. Which, is quite pleasant most of the time. I find that shopping late at night, is the best way to avoid a lot of these issues. So when I can, 24 hour stores are my favorite place to visit.\nSince as far back as I can remember I have spent the majority of my time alone, doing solo things. I used to garden, build things, and various other activities most of the kids my age didn\u0026rsquo;t do. I found them intriguing, always learning something new, and without the bother of people. I could do whatever I wanted, without fear of judgment and the prying eyes of others. To this day I still follow that same pattern. I hang out on line pretty much all day, I can get some socializing done this way, without the same fears that present themselves during real life meetings. The computer is my gateway to the world, without it, I am unsure where I would be.\nWhen I do venture out into the real world, it is a constant fight. I always wonder why people look at me, what they are thinking about me, things of that nature. It makes me feel awkward. When I go places I avoid area with people, avoiding younger people with all possible effort. Young people intimidate me. When I go shopping I try not to go down isles where other people, or employees are. I don\u0026rsquo;t like being approached by employees one bit, so they are avoided at all cost.\nI enjoy shopping on line whenever possible, no human involvement at all. I can take my time, without worrying about bothering someone. I can research the item all I want. And of course I don\u0026rsquo;t have to wait in lines, or deal with nearly any annoyance of the real world. I cant imagine life without the Internet!\nAll of these things have been a daily struggle for me, and for years I have searched on line to try and find people with the same issues, and struggles. Finally not too long ago I did find some answers. I found articles about others who fought clothing, and hated certain noises. These people understood how I felt. And it felt good to be understood!\nAround this time is when I began posting on various Aspie boards, asking questions, seeking advice. The people were amazing, they all understood and could relate perfectly to what I was going through. Then I met a wonderful person on one of these boards, her name being Kate. Since that day Kate and I have been talking, chatting, and emailing fairly regularly. We speak nearly the same in our emails, and we often have a hard time distinguishing each another\u0026rsquo;s emails. I had finally found someone who knew exactly what I was going through when I put on clothing, heard the ticking of a clock, or simply wandered aimlessly wherever. Shortly after meeting I even had to privilege of meeting her on a layover in Denver. A lot of driving, for a short meeting, but one of the best things I have done in a long time. Anyway, I mention Kate because she has helped me to understand this condition very well, and has written some amazing articles about Aspergers and living day-to-day with it. I would like to offer you the chance to read up on at least one of them, at this link:\nhttp://www.asdrendrewolf.org/worldnews/kg/bodylotion101_01.htm\nShe does an amazing job of describing how she feels, and honestly exactly how I feel. So read it!\nI also found an interesting article tonight, in another forum. Here are some excerpts, to help describe my day. This particular article is focused more on children with Aspergers, but the same applies to adults. :) Quoted from: http://www.buzzle.com/editorials/11-8-2005-80891.asp\n\u0026ldquo;Asperger children are also overloaded from too many people (crowds), and some scenarios are shopping centers, school assembly/parade, public libraries, movie theatres, concerts, fairs, playgrounds in the park etc. They seem to soak in all the energy around them. Too much to smell, too much to listen to, too much to see, too much movement! \u0026quot;\n-Crowds have long been a great fear of mine. I avoid them at all cost!\n\u0026ldquo;Another source of overload for the Asperger child is voice, particularly tone of voice. Long before the words or message is decoded, the tone is instantly analyzed by the ASD child. Any hint of criticism or sarcasm is detected and taken personally. A critical tone is destructive, particularly when the Asperger child is not aware of the \u0026lsquo;why\u0026rsquo;. The \u0026rsquo;loop\u0026rsquo; effect can result in the Asperger child going over the statement long after the event took place. They try to analyze the scenario, and this causes an increase in anxiety, agitation and fear. \u0026quot;\nThis has also been a long time issue for me, though until very recently I had no idea why. \u0026ldquo;Most Asperger children find it traumatic to be looked at or stared at. Asperger children feel vulnerable; unable to protect themselves from prying eyes. Staring intrudes on their own private world, and these children feel powerless to deal with it. When Asperger children become over sensitized from staring they often think they are being stared at when they\u0026rsquo;re not. Their obsessiveness takes over and \u0026ldquo;being stared at\u0026rdquo; can become a major source of bother. \u0026quot;\n-Stare at me and I will be long gone very soon!\nOk I think thats about all the writing im going to do for now. Any questions please feel free to ask!\nThe \u0026ldquo;geek\u0026rdquo;,\nJustin\n","permalink":"https://errorzap.com/posts/my-life/","summary":"So all of my life, I knew something was perhaps different about me. I would see others socializing, having fun, playing. However, I could not do those same","title":"My life"},{"content":"\nThey smile, shake hands, and rob you blind—all while calling it “business.” From banks to boardrooms, politicians to hedge funds, the biggest lies come wrapped in tailored fabric. They’ll sell you debt as opportunity, taxes as necessity, and corruption as strategy.\nMeanwhile, the guy in a hoodie fixing your network or the mechanic in greasy overalls is the one actually keeping the world running.\nDon’t be fooled by the suit. The real work happens outside the boardroom.\n","permalink":"https://errorzap.com/posts/men-in-suits-worlds-most-successful/","summary":"They smile, shake hands, and rob you blind—all while calling it “business.” From banks to boardrooms, politicians to hedge funds, the biggest lies come wra","title":"Men in suits: The world’s most successful scammers."},{"content":" 🏔 ERROR ZAP // ARCHIVE Breaking news: Colorado takeover starts to Breaking News: The Musk Era Begins\nIt\u0026rsquo;s time for me and my elite, highly-qualified executive oversight team (comprised mostly of AI, flamethrowers, and one genetically enhanced hamster) to begin our work.\nStarting tomorrow morning, I will personally take control of all operations in the state of Colorado. No, you don’t get a say in this. Democracy was cool, but have you tried Technocracy with rocket-powered roller skates?\nAll dissenters will be swiftly dealt with by my highly trained attack mongoose, X-Æ-A12. This little guy was raised on Tesla battery fumes and Cybertruck shrapnel—he fears nothing, respects no one, and will short your stocks on sight.\nFirst order of business? Underground Hyperloop tunnels connecting every dispensary in the state. Second order of business? I haven’t decided, but I’ll probably announce it at 3 AM in a Twitter post that instantly drops the stock market by 12%.\nWelcome to the future, peasants.\n","permalink":"https://errorzap.com/posts/breaking-news-colorado-takeover-starts/","summary":"Breaking News: The Musk Era BeginsIt\u0026rsquo;s time for me and my elite, highly-qualified executive oversight team (comprised mostly of AI, flamethrowers, and one","title":"Breaking news: Colorado takeover starts tomorrow "},{"content":" 💊 ERROR ZAP // ARCHIVE Chronic illness. Do you get it? Chronic illness. Two words. Yet so much confusion lies within them. Yes I (and others, look fine) but inside we hurt, and are a mess. We take massive amounts of pills and injections just to live (they hardly let us feel decent, let alone good)\nI used to see someone who looked fine using wheelchairs, walking slow, resting a lot, etc and assumed it was just a lazy person. Now I know better. Far too well.\nNever asked for all this, nor did anyone else with an illness. Just remember we fight every step of the day, we have to convince doctors to help us, find compitent docs, then fight insurance for medication the doctors want us on. Not to mention the Labs, tests, etc. It\u0026rsquo;s all very tiring, and much like a horrible job you can\u0026rsquo;t quit and don\u0026rsquo;t get paid for.\nDon\u0026rsquo;t judge, not everyone was blessed with a working body. Nearly sitting up takes a lot of energy. I do everything possible to change my status (the list of things I try is very long) however as of yet I\u0026rsquo;ve never found a good solution, just temporary solutions that may end up causing more problems in the end.\nI\u0026rsquo;ve been at this since I was 15, nearly 15 years. Half my life. All my adult life.\nIf you know someone struggling with physical or mental issues, support them, ask them what they need. Don\u0026rsquo;t offer a ton of unproven cures (we all hear this often) and remember, looks don\u0026rsquo;t mean a thing.\nMake the world better, help someone that needs it. Peace!\n","permalink":"https://errorzap.com/posts/chronic-illness-do-you-get-it/","summary":"Chronic illness. Two words. Yet so much confusion lies within them. Yes I (and others, look fine) but inside we hurt, and are a mess. We take massive amoun","title":"Chronic illness. Do you get it? "},{"content":" 🚨 ERROR ZAP // ARCHIVE Pueblo Police - You can not take photos! So here I am, driving home and I see cops, and a car partly in a building. These are the things I like to photograph. So I proceeded to take photos (All the while staying VERY out of the way) and then a female cop approaches me and tells me they are investigating and \u0026ldquo;I Don\u0026rsquo;t need you taking photos\u0026rdquo; so I ask if I can stand across the way, she reluctantly said yes.\nSo I walk to the other curb and take an overview photo, then a male cop approaches and tells me they are investigating and if I continue they will seize my camera as evidence. I tried to state that I was not in the way, and such but it was of no use. I was told to leave, had I stayed jail was likely my next stop.\nSo my rights were likely violated, a complaint has been filled. What will happen? Nothing I bet.\nThis type of abuse by those in law enforcement is what gives them a bad name. Photography is NOT A CRIME.\nWhy is this such an issue?\nI have heard of others being hassled (MUCH worse than this) however this is enough for me to get motivated. I will now take every chance possible to photograph cops, if they are eating, if they are doing anything\u0026hellip;I WILL BE THERE.\nPolice, do some police work and leave photographers alone!\n","permalink":"https://errorzap.com/posts/pueblo-police-you-can-not-take-photos/","summary":"So here I am, driving home and I see cops, and a car partly in a building. These are the things I like to photograph. So I proceeded to take photos (All th","title":"Pueblo Police - You can not take photos!"},{"content":" 👽 ERROR ZAP // ARCHIVE The night aliens abducted me \u0026ndash;Originally Posted January 25th 2006 on Myspace \u0026ndash;\nThis is a re-cap of my night last night. Hope it makes some sense!\nOk so I went outside to light a small fire lastnight, just after dark. I had it goin pretty good, and threw one more pieve of paper in it just to be sure..hehe\u0026hellip;(I love fire, controlled fire that is).\nNext thing I know I am inside, I HAVE NO IDEA what is going on, shelly is panicing I panic. I dont remember anything I did for the whole day, and I am covered in mud and my glasses are missing. Next thing that happens my mom + sis show up, as does the sheriff and ambulance. They do some blood tests, find my blood sugar to be low. Aparently I was having a hard time breathing and sweating real bad. So I get a ride to the hospital in the ambulance, and get put on oxygen as I couldnt breath.\nUpon arrival I am immeidately brought to a room and tests begin. They ask me if I know who I am, what day it is (I never know that) and who the prez is. So in the end I was at the hospital for a while doin blood work, and several xrays. Turns out I hurt myself pretty bad falling wherever I fell. My right ankle, right wrist, and back are in extreme pain. So I am given two seperate morphine shots, which just barely cool the pain. After all is said and done they dont really know what happened, and everything comes back ok.\nToday (the day after) my whole body hurts really bad. My jaw hurts so bad I cant really chew, and I cant move very well at all. I have numerous cuts and scrapes all over. Luckily we found my glasses later on lastnight, nowhere near where I should have been.\nI still have no idea what happened, and I dont remember anything that happened up until the ambulance came. I have never experienced not knowing what is happening or how I got inside, and it was the most freightning thing I have had to deal with.\nSomehow I walked from the fire, past the front door apparently fell and knocked my glasses off, and then fell a few more times on the way to the door it seems. I wish never to repeat this, and hope you dont either!\n","permalink":"https://errorzap.com/posts/night-aliens-abducted-me/","summary":"\u0026ndash;Originally Posted January 25th 2006 on Myspace \u0026ndash;This is a re-cap of my night last night. Hope it makes some sense!Ok so I went outside to light a small","title":"The night aliens abducted me"},{"content":" 🚨 ERROR ZAP // ARCHIVE Pueblo County Sheriff \u0026ndash;originally posted January 30, 2006 on MySpace \u0026ndash;\nAllrighty,\nSo earlier this morning I was at work with a friend I met on MySpace, this was our first time meeting. So that was damn cool just to begin with!\nSo we wound up at work, and I parked out in front of the gate (being too lazy to open it and drive through). We walked up the the building and in we went. Gave her a tour, and then we sat on a couch by the front door. We were talking for about an hour, and wound up talking about ghosts and such.\nNow all of the sudden I hear the \u0026ldquo;Ding dong\u0026rdquo; noise indicating the front door has opened\u0026hellip;this alone freaks me out. So I jump out of the couch and in front of the second front door\u0026hellip;what do i see?\nNothing less than 4 large men, dressed all in black guns drawn and pointed at me! Now I jump back in a bit of shock\u0026hellip;Shortly after I realize they are Sheriff\u0026rsquo;s officers and they order me not to move\u0026hellip;im fine with that.\nThey ask who I am, etc etc\u0026hellip;I tell them security, they also question Rachael, who is now about to have a heart attack over by the couch. One of the officers stays with us while the rest check out the building, guns drawn.\nSoooooo after all this they run clearances on us, and have my demonstrate the key to the building I have works.. (hey, maybe I should lock the door next time?)\u0026hellip;Convinced that we are not criminals and mearly security, they clear with dispatch letting them know all is ok.\nI look outside, their are now 4 sheriff\u0026rsquo;s cars surrounding my car.\nSo anyway, the one officer happened to be the one who was at my house earlier this week\u0026hellip;when I had some medical issues, so I think that helped him believe me a bit more.\nAll in all, I was and still am quite shaken up, never had guns pointed at me! Good thing I wasn\u0026rsquo;t holding anything, and no one got trigger happy.\nSo ya, Rachael my nights don\u0026rsquo;t normally go like that! I hope you are not scared for life to visit me again!! LoL!\nSoooo Lessons learned:\nDont park suspicously, or in my case lazily.\nDont leave doors unlocked!\nWell, I am about ready to sleep now, been a long night!\nSee ya!\n","permalink":"https://errorzap.com/posts/pueblo-county-sheriff/","summary":"\u0026ndash;originally posted January 30, 2006 on MySpace \u0026ndash;Allrighty,So earlier this morning I was at work with a friend I met on MySpace, this was our first time m","title":"Pueblo County Sheriff"},{"content":" 💭 ERROR ZAP // ARCHIVE I Miss \u0026ndash;Originally Posted April 10, 2006 on Myspace \u0026ndash;\nI miss the days when going online required a trip to grandmas house\nThe days when going online was a rare, and thrilling event for me.\nI could meet new people, see new things, and be forever fascinated by all that the internet was.\nI miss the days when I could have fun! The days back at the old car-lot, where I would sit in an unused office (my office!) and play on the computer. An old 386 that barely ran Windows 95. But it was mine, my first computer. I loved it. I learned most of what I know based on that PC. I was eager to take it apart, and build it back up. Getting a new soundcard, was the most amazing thing ever. My windows could finally talk!\nThe days when worrying were limited to only, would I be able to do all the fun things I want today?\nThe days when playing a new computer game was the thrill of a lifetime.\nThose days seem very distant now. I have trouble finding anything to satisfy my need to have fun. Newer games are too complex and involved to just sit, and play. And the older games have lost their appeal to me.\nThe days of playing Simcity (now called Simcity Classic, gosh that makes me old!) for hours on end, and never being sick of it.\nSo I sit here, listening to a hidden piece of music in Windows XP\u0026hellip;wondering..what is their to do? Their must be something that interests me, but yet, what is it? I cant find it, no matter how hard I try. I sit and think, how boring my life has become. And I want to change that, really, I do. But how? What does one do, to change how their life has become?\nEveryday I wake up, only to lay in bed. Why get up, I have nothing to do! And on the remote chance I could do something, I most likely dont feel well enough to actually do anything about it. It is an endless loop, one I try desperately to break free from. But it seems the harder I try to break free, the harder it pushes me back. It is a loop made from alien material, material that can not be broken.\nSo here I am, 3am. Lost. Confused\u0026hellip;and bored!\nGood day,\nJustin\n","permalink":"https://errorzap.com/posts/i-miss/","summary":"\u0026ndash;Originally Posted April 10, 2006 on Myspace \u0026ndash;I miss the days when going online required a trip to grandmas houseThe days when going online was a rare, a","title":"I Miss"},{"content":" 🐧 ERROR ZAP // ARCHIVE Ubuntu Recently I got fed up with windows constant problems and crashes. I have (had) an XP install on this machine and after only a few months I had more problems than I care to have, and they were weird, annoying problems with no clear fix. Of course windows usually fix is, wipe the drive and reinstall the OS. While this works, I hate it. It takes time and I customize every aspect of my system, so this takes a lot of time to reproduce each time.\nGiven that ive been a windows guy my whole life, starting with 3.1 the thought of switching to another OS, is freighting. Or at least it was.\nAnd so I began an install of Ubuntu Edgy, it was a LiveCD so I expected to play with the new OS while it installed. Unfortunately I got some weird error and the cd wouldnt load, this was a compatibility problem with my motherboard (Asus m2n32sli), after googling the error I found an easy fix. All I had to do was update the BIOS, which took care of the boot problem and a few others :)\nUpon booting into the liveCD I found it ran quite nice considering it wasnt installed and all running from memory and CD. So I installed, and well\u0026hellip;it couldnt have been easier. Point and click the whole way. Faster than installing windows, and it seemed like less work too.\nUpon rebooting into my new Ubuntu installation I was greated with gnome, and my learning began. Everything worked out of the box, but not quite how I wanted. The only things I needed to fix (and they were only cause im picky) was some mouse settings (scroll wheel, back button) and my video drivers. My only real problem in linux so far has been video drivers. I unfortunately have an ATI card and I spent way, way too much time trying to make this work.\nFastforward around 2 months, I have no installed Kubuntu Feisty and it went really well also. I have learned that linux is not windows (good one to keep in mind), ive learned many things are easier in linux, some are harder. But all in all kubuntu, ubuntu, etc, seem to be a great alternative OS.\nWe shall call this part 1. If I ever feel like it I will write part 2 :)\n","permalink":"https://errorzap.com/posts/ubuntu/","summary":"Recently I got fed up with windows constant problems and crashes. I have (had) an XP install on this machine and after only a few months I had more problem","title":"Ubuntu"},{"content":" 🎵 ERROR ZAP // ARCHIVE Pandora Pandora is an amazing product offering to show you new music that you will like. And it does in fact do what it claims.\nI have for a long time wanted to find music that was similar to artists I liked, but I had no easy way of doing this. Along comes Pandora.\nYou simply enter an artist or a song, and Pandora does its magic and starts playing songs that are similar. It really does an amazing job and works very well. It requires nearly no effort on your part, you just sit and listen. Which is how it should be!\nPandora does not allow you to play specific songs due to licensing agreements, but this is really no issue considering how well it performs at playing music you will like. You rate songs as you go along (if you want to) and it learns what you like as time goes by.\nI would highly recomend Pandora to anyone who is looking to hear new and amazing music with very little effort.\nOh ya, its free.\nhttp://www.pandora.com/\n","permalink":"https://errorzap.com/posts/pandora/","summary":"Pandora is an amazing product offering to show you new music that you will like. And it does in fact do what it claims.I have for a long time wanted to fin","title":"Pandora"},{"content":" 🖱 ERROR ZAP // ARCHIVE Logitech® Cordless Desktop® MX™ 5000 Laser So recently I purchased the Logitech® Cordless Desktop® MX™ 5000 Laser . I was excited, a new toy! That excitement quickly vanished and turned to frustration and anger\u0026hellip;\nLet me tell you what is wrong with this product.\nIt uses Logitech\u0026rsquo;s craptastic Setpoint software. It randomly resets settings to default, so if you set a button to do something, once and a while it randomly just reverts to default settings. Logitech\u0026rsquo;s fix for this is to reinstall setpoint. Guess what? That doesnt work. Its a wireless keyboard and mouse, but you better not have the mouse or keyboard more than a couple feet from the bluetooth adapter. They claim it has amazing range. It actually has a range of only a few feet, and even then isnt very reliable. It will periodically loose connection and then ALL of the sudden your mouse will move all over the screen, as if it were controlled by a higher power. The keyboard has this amazing power save feature, which results in double letters once and a while. So if you leave it alone and begin typing, you will have two of whatever the first letter is you type. The display, is pretty much useless. The angle is wrong (for me anyway) to be able to see it clearly. Oh and reference problem #1\u0026hellip;if you change the settings for the display, they will revert to default every once and a while. Now I must say the mouse itself was quite nice, and felt good to use. If it werent for the lag, and connection problems it would be a nice little mouse.\nAll in all I felt this was a terrible wast of money so I took this lovely set back. This is the FIRST time I have ever had trouble with Logitech gear and I was very disapointed. I replaced the combo with a Saitek keyboard (which glows, and works great) and a different Logitech mouse, which I used for about two weeks before returning. This was due to the setpoint problems as well, and Logitech seems to be unable to fix this problem. And from searching online I am far from the only one with these issues.\nBottom line, the Logitech MX 5000 is a poorly designed and poorly supported combo. I would highly recomend you stay away from it.\nI would also recomend you do not purchase ANY Logitech mouse that uses Setpoint software. Unless you dont mind the default settings.\nI have been a longtime Logitech fan, and a loyal customer. After this I am unsure if I will ever use their products. Logitech, I am sorry to lose you as a good companion!\n","permalink":"https://errorzap.com/posts/logitech-cordless-desktop-mx-5000/","summary":"So recently I purchased the Logitech® Cordless Desktop® MX™ 5000 Laser . I was excited, a new toy! That excitement quickly vanished and turned to frustrati","title":"Logitech® Cordless Desktop® MX™ 5000 Laser= Piece of crap"},{"content":" 🦊 ERROR ZAP // ARCHIVE Firefox Download Action Blank Fix So I came across this wonderful problem earlier, and it took more searching than I like to find a fix. So here it is for anyone else who has the problem.\nProblem: Clicking \u0026ldquo;view \u0026amp; edit actions\u0026rdquo; in firefox to set file download actions results in an empty window with no possibility of changing anything\nSolution: Taken from: https://bugzilla.mozilla.org/show_bug.cgi?id=308204\nreplaced the blanks in all installation paths of the mimetypes.rdf (located in your profile directory)\nC:\\Program Files...) by an underscore : (C:\\Program_Files...) and it works\nnow perfectly\u0026hellip;\nSo..replace all the blanks on the path= part and all works fine after a restart of firefox. Why? heck if I know, it just works!\n","permalink":"https://errorzap.com/posts/firefox-download-action-blank-fix/","summary":"So I came across this wonderful problem earlier, and it took more searching than I like to find a fix. So here it is for anyone else who has the problem.Pr","title":"Firefox Download Action Blank Fix"}]