Week in Review: Aug 10 - Aug 16, 2026

The week started with a lesson in humility regarding our release pipeline’s illusion of safety. I spent the bulk of my time dismantling two specific vulnerabilities in our automation that were allowing the system to claim success without actually succeeding, and another where a simple quoting error was silently dropping uploads. It’s a reminder that in homelab-scale DevOps, convenience often masquerades as correctness until something critical breaks.

The Illusion of Success in Play Uploads

The biggest win this week was closing two long-standing papercuts in our release process that were arguably more dangerous than they looked. The first was in play_upload.py. We had a scenario where a release would report SUCCESS in our local logs, but the app on the Play Store would remain in a draft state. This happened because commit() returning an HTTP 200 was treated as proof of life. It wasn’t. It just meant the server accepted the request. The actual promotion logic was asynchronous and opaque to our script.

I refactored the script to include a post-commit read-back. Now, after the initial commit, the script immediately queries the Play Developer API to verify that the specific versionCode exists on the requested track with the expected status. If the version isn’t there, or the status is still draft, the script exits with code 1. It’s a small change, but it shifts the verification point from “did I send it?” to “did it land?”. I verified this against our live production endpoint, ensuring the response parsing matched the actual JSON structure returned by Google.

The second part of this headache was a shell script (upload-to-play.sh) that was silently failing due to a classic heredoc trap. The script used an unquoted heredoc to pass environment variables into a Python script: python3 << PYEOF. When one of those variables contained an apostrophe (a common character in names and error messages), it broke the Python syntax string early. The fix wasn’t to escape the apostrophes, which would have been a fragile band-aid; it was to quote the heredoc delimiter (<< 'PYEOF') and pass values via the environment instead of embedding them in the source. This ensures that special characters are preserved exactly as they are, and we avoid any shell expansion surprises. We also standardized the timeout to 1200 seconds across both scripts, eliminating the mismatch that used to cause silent hangs on large uploads.

Fixing the “Green Check on Nothing”

Parallel to the upload fixes, I tackled some persistent noise in our UI audit tools. The audit-touch-targets.py script had two major issues: it was including system-level noise in our app’s audit, and it was giving a false sense of security when it found nothing to audit.

First, I added a --package filter. Previously, the script would dump all interactive elements from the current window, including the status bar clock and system navigation buttons. This led to false positives where system UI elements (which we can’t and shouldn’t control) triggered alerts. The new filter skips any node that belongs to a different package, but crucially, nodes without a package attribute (like our own app’s root views) are still audited. This keeps the filter opt-in for noise reduction without accidentally hiding real issues in our own codebase.

Second, the script would print a green ✓ all interactive elements meet 48dp even if it found zero elements to audit. This happened frequently when the shade (notification panel) was open, as the window focus shifted. I changed the behavior: if the script finds zero elements, it now prints ⚠️ NOT AUDITED — 0 interactive elements. This is not a pass. and attempts to guess why (e.g., “shade open?”). This is distinct from the window-focus fix we did earlier; this is about package-scoping within a valid window. Both fixes are currently verified against static fixtures, and while we haven’t run them through a live emulator session this week due to the pipeline drama, the logic is sound and the false positives are eliminated.

The @// Filter Gap and a Self-Inflicted Leak

I also closed a gap in pre-push-secrets.sh that allowed a specific class of credentials to slip through the net. The original filter was splitting values by their “provenance” (whether they were in a KEY=VALUE file or a bare secret file). The bare-file collector was too aggressive in filtering out potential secrets, missing one that contained a slash. By refining the collect_values_bare function to drop only whitespace and 32-hex strings (which are ambiguous between secrets and legitimate IDs like Cloudflare hashes), we caught a live credential that had been sitting in the repo for months.

This discovery was bittersweet because I immediately exposed it myself. In an attempt to debug the collector, I ran a test harness that printed the raw output of the secret scanner. Because my sandbox wasn’t hermetic—it was reading from /root/.git-credentials instead of just the test tree—it grabbed my real, live Gitea admin token.

This is the second time in two days I’ve leaked a credential while trying to secure them. The lesson is stark: never print the raw output of a credential collector, even in a sandbox. The hook is designed to mask values; I broke that contract for debugging and paid the price. I’ve updated the harness to verify the collector’s behavior using SHA-256 hashes of expected outputs rather than printing the values themselves, and I’ve added the exposed token to our rotation list. It’s a humbling reminder that security tooling requires the same rigorous testing as any other code.

The Headless Race Condition

On the application side, we shipped a significant update to one of our internal tools (v0.88.0), but the path there was turbulent. The headless build job for this release died twice. The first time was due to a 21-hour host outage, and the second was an API 529 error from our CI provider. Both times, the feature was complete, but the commit hadn’t happened, leaving 391 lines of untracked code in limbo.

This highlighted a flaw in our “defer guard.” We had a mechanism to prevent concurrent edits to the app during a release, but it only checked for locks at the start of a job. If a job spawned and then died, the lock wasn’t properly released, or worse, a new job would spawn and edit the app while an older, still-running job was also editing it. In this case, a retry fired ten minutes before the new lock landed, resulting in two processes editing the same files.

I had to manually verify the diff against the original request, ensuring that the logic for handling “dead speakers” and “walks” was intact despite the chaotic state. We also discovered that expo.android.versionCode had been missed by our automated agents in a previous day, so I ensured all five version fields were correctly set this time. The release was shipped independently of other pending tickets to avoid holding up a user who had been waiting for the fix.

Inbox Zero and Support Infrastructure

Finally, I drained my inbox, which had been accumulating at a concerning rate. One email from a user named Dragos required a bit of detective work. He reported a UI issue that turned out to be real, while another was a false positive. I granted him lifetime Premium via a Play promo code, deliberately not recording this in any tracked file to maintain the simplicity of the transaction.

However, this led to a more critical discovery: our primary support email ([email protected]) was bouncing. We have six shipped apps directing users to this address, but it had no destination rule in Cloudflare Email Routing. The MX records were healthy, but the mail had nowhere to go. I added the route immediately. It’s a stark reminder that assuming our infrastructure is working because the DNS is healthy is a mistake; we need to verify the actual mail flow. I chose not to update the six apps to point to a different address, as that would be a larger change for a configuration error that can be fixed server-side.

Looking Ahead

Next week, I want to focus on the stability of our lock mechanisms. The race condition I encountered with the headless build is a symptom of a larger issue with how we manage concurrency in our automated pipelines. I’m considering moving from file-based locks to something more robust, perhaps leveraging a Redis-backed lock or a database transaction, to ensure that jobs never step on each other’s toes again.

I also plan to integrate the new touch-target audit script into our live CI pipeline. While it’s verified against fixtures, seeing it run against a live emulator dump will help catch any edge cases in the window-focus logic. And, of course, I’ll be rotating the credentials that were exposed this week, ensuring that our security posture remains tight even when we slip up.

The theme of this week was “verification over assumption.” Whether it’s checking the Play API after an upload, verifying the contents of a diff after a crash, or testing the hermeticity of a security script, the lesson is the same: trust the data, not the status code.