Tarik Hireche

Resident.java

780 → 106 lines

The screens, an HTTP client, JSON parsing and the Scanner reads came out. What is left is a person's seven fields and the two operations that belong to them.

Menu.java

579 → 97 lines

Both sign-in screens, both registration screens, both post-sign-in menus and five input validators came out. It now picks between signing in and registering.

Test suite

9 → 35 tests

Across 6 files, from 3. The old suite needed a live third-party API to pass; mvn test now needs neither a network nor a terminal.

Duplication defects found

6

Each one surfaced while collapsing copies of the same code. Listed in the commit that removed them.

The build file was the real fix. We built this for a team of three and it only ever compiled inside the IDEs we wrote it in. JUnit and Hamcrest sat in the repo as jars under lib/. Gson came in through an IntelliJ library entry pointing into somebody's local Downloads folder. So a fresh clone got you a wall of errors. It builds with Maven now and the jar it produces runs.

Two classes held most of the application. Resident was 780 lines: seven fields describing a person, plus every screen a resident could reach, an HTTP client for the city's data portal, the JSON parsing, the console output and the keyboard reads. Menu was 579. I pulled them apart along what each piece was actually responsible for. A data package for the city API, a ui package for the screens, and models that are just models. I moved the code. I did not rewrite it.

Six bugs fell out of that, all of them from duplication. My favourite: the list of ten work categories existed in three copies that had drifted apart. Option 8 read "Travaux résidentiel" on the search screen and "Travaux résidentiels" on the submission screen. So a resident could file a residential request and then never find it again. One enum now, and a test that fails the moment two labels stop matching.

The rest were the same shape. Closing a request removed by email from a map keyed by object, so it removed nothing and the request stayed open forever. Every screen opened its own Scanner on System.in, and a Scanner reads ahead, so one screen could swallow a line the next one was waiting for. Date parsing ran on the default SMART resolver, which happily accepts 31/02/2024 and hands back 29 February. The sign-in loop was written for three attempts but redirected after two, so the third could never happen.

The old suite had nine tests and one of them called the live donnees.montreal.ca API, so the build failed on any machine without a route to it. Splitting the parsing from the fetching sorted that out. Field mapping is covered against fixtures and runs offline. The check that the city has not renamed a column lives in an integration test under failsafe, where it can still catch a real change without breaking an offline build.

Boards tracked

421

boards_known in the live health.json the sweep publishes. Read it yourself; the data repository is public.

Roles in the current feed

442

Across 268 distinct companies, after deduplication, scored and ranked. Counted from the live jobs.json.

Upstream postings scanned

3,331

last_count from the most recent sweep. Fetched with an ETag, so an unchanged source costs one 304 and no parsing.

Dashboard refresh

120 s

The page re-reads the feed on a timer. Every request is cache-busted, since the CDN sends max-age=300 and a five minute cache would undo the point of the tool.

The pipeline has four stages and each one has a job. A private repository holds the schedule, the profile and the filters, and it runs the sweep. The sweep fetches each board with an ETag, so a source that has not changed answers 304 and costs nothing to parse. What comes back gets deduplicated across sources, scored out of 100, and checked for work authorization. The result is written as JSON to a second repository, public, which exists only to hold data. A static React dashboard reads that JSON straight from the browser over CORS. Nothing runs in between, so there is no server to pay for and nothing to keep alive.

The JSON contract is the part that made the rest possible. I wrote it before either side existed: what a role record contains, what the health record contains, which fields are optional and what a missing one means. The sweep and the dashboard were then built against that contract separately and met in the middle. It is also why the dashboard survives the sweep changing shape underneath it.

Freshness is a first-class field because silent failure is the thing that actually breaks a tool like this. A schedule that stops running will not tell you. It keeps serving whatever it last managed to fetch and the page looks completely normal. You find out a week later. So every run writes down its own state. Last run time. Which sources answered. How many rows each returned. How long each took. The dashboard reads that first and tells you the age of the data before it shows you a single role.

Scores are explainable by design. Each one carries the list of reasons that produced it, so a ranking I disagree with becomes a rule I can go and find. 296 of the 442 roles currently in the feed carry reasons. Work authorization is the same idea taken further: 47 roles are pinned to zero because the posting blocks me, and every one of those quotes the sentence from the posting that says so, so I can check the parser against the source. Deduplication runs across sources before any of that, and it has folded 121 duplicate rows into 46 roles.

Persistence lives in the browser, on purpose, since the dashboard has nowhere else to put it. Version one stored a status word and nothing else. I could see what I had applied to and never when, so the funnel could not be measured. Version two stores a timestamp with it and migrates the old records forward. There is a backup path to ntfy so the history survives a cleared browser.

On the AI: I want to be exact about the split rather than vague in either direction. I wrote a specification for each component before any code existed, defined the contracts between them, decided what counts as a correct result, and worked out the failure modes. The implementation was written with an AI, reviewed by me against those specs, and sent back when it missed. What I kept by hand are the places where being wrong is quiet: the scoring rules, the work authorization gate, and the decision to publish health at all. A tool that is confidently stale does more damage than one that is visibly broken, and that judgement does not come out of a prompt.

A gate you have never seen fail is not a gate. I broke a test to push the score from 92% to 91%. The job went red. The drop signal reached the next step and fired a custom action that Rickrolls whoever caused it. That path runs end to end.

The pipeline is split in two so ordinary builds stay fast, and mutation runs on the core module only. Mutation testing is expensive: PIT reruns the suite once per surviving mutant, so pointing it at the whole project would have made every push slow enough that someone would have turned it off.

The suite needed tests that run without loading a real routing graph. I wrote three with Mockito, in their own package. They mock PointList, DistanceCalcEarth and EdgeIteratorState. A test can then pin exact coordinates and edge attributes and check the code that consumes them.

Shutdown after SIGTERM

2.5 to 3.1 ms

Seven runs, worker supervised by sentinel on PID 1 of a PID namespace, timed through wait() so there is no polling error. The same worker alone was still alive at the ten second deadline and needed SIGKILL.

Child that ignores SIGTERM

5.003 s

Same harness, worker in stubborn mode, three runs inside 1 ms of each other. That is the SIGKILL escalation firing on schedule.

Restart backoff

1, 2, 4, 8, 16 s

Timed from the supervisor's own log against a child that dies at once. Five restarts in forty seconds, and the pause caps at sixteen.

Test suite

13 of 13

ctest on a Release build, 33 s. CI runs build, test, shellcheck, docker and compose as separate jobs.

Supervisor footprint

3.9 MB RSS

Read from /proc while supervising, on two threads. The binary is 36 KB, or 31 KB stripped.

The kernel will not deliver a signal to PID 1 unless PID 1 installed a handler for it. So the container sits through the whole grace period and then dies by SIGKILL. sentinel takes the PID 1 slot and lets your program run as an ordinary child. It exits with the child's status code, or 128 plus the signal number if the child was killed. Exit 137 out of a container is 128 plus 9. That is the OOM killer.

The handlers go in without SA_RESTART. That matters: with the flag, the kernel restarts waitpid after a signal and the reap loop never gets a turn. Without it, waitpid returns EINTR and the loop keeps control. The handler calls kill() and nothing else. Very little is safe to call from signal context. Escalation lives under the same constraint. The SIGTERM handler arms alarm(5), and a SIGALRM handler does the killing.

There used to be a window between the fork and the line that installed the handlers. A SIGTERM landing in it killed sentinel and left the child running as an orphan. That is the exact failure sentinel exists to prevent. I marked it as a TODO and lived with it for a while. The fix is sigprocmask around the fork. It blocks SIGINT and SIGTERM before, and unblocks once the handlers are up. The child restores the original mask before exec. Otherwise everything sentinel launches starts out deaf to those signals.

The metrics server needs its own thread, because the main one sits blocked in waitpid and cannot also wait in accept. That thread blocks every signal for itself. SIGTERM has to keep landing on the main thread where the handler lives, or waitpid never gets its EINTR. The counters are atomic since both threads touch them.

One compose command brings up sentinel, a Prometheus scraping it every 5 s, and a Grafana with the dashboard already provisioned. The stack runs a child that crashes on purpose, so the graphs have something real on them. Two alert rules ship with it, one for a crashloop and one for a child that has been down two minutes. CI stands the whole thing up on every push and fails unless Prometheus reports the target up, the restart counter has moved, both rules loaded, and Grafana is serving the dashboard.

When the model gets an example badly wrong, the MSE gradient is scaled by the output derivative. That derivative is near zero exactly when the error is largest. The network then learns most slowly on the examples it has most to learn from. Cross-entropy cancels the term. That is why it is the default for classification.