Run the watcher once a day, but page nobody merely because the page bytes changed. Fetch conditionally, extract only the changelog entries, compute a deterministic structural diff, and send a notification only after the new text passes a semantic near-duplicate check against recent property-management records. Keep the old snapshot until the notification succeeds. That is the decision rule.
TL;DR: treat scraping, change detection, semantic retrieval, and delivery as four separate incident controls. A fetch failure is not "no change," a rearranged navigation bar is not a new release, a high vector score is not proof that two records mean the same thing, and a successful diff is not useful if the notification disappears. The page that should fire is the one for a sustained inability to observe the source, not every ordinary content edit.
This design fits a Node.js service even though the reference implementation below is Go, because the important boundaries are HTTP semantics and persisted records rather than a framework-specific scheduler. The worker can be invoked by a Node.js cron process, a system timer, or a job queue; the same run contract applies: one source, one bounded execution, one durable outcome.
How should Node.js scrape a changelog page daily and notify safely?
Start with the failure modes, because they determine the state machine. If yesterday's fetch returned 200, today's request times out, and the worker records an empty body as the new baseline, tomorrow's successful request looks like a complete rewrite. If a consent banner changes on every request, a raw HTML hash creates daily noise. If notification delivery fails after the baseline advances, the only meaningful change is silently consumed.
Those are different incidents.
A useful run record therefore distinguishes unchanged, changed_and_sent, fetch_failed, parse_failed, and send_failed. Page on repeated inability to fetch or parse, because freshness is now unknown. Route a delivery failure to a retryable operational alert. Do not page on unchanged, and do not make every detected edit urgent; a changelog notification is usually work for daylight hours. Consider the concrete failure chain: a scheduled run starts, the origin stalls until the 45-second worker deadline, a careless implementation replaces the prior snapshot with an empty value, and the next healthy response appears to add every historical notice at once; retrieval then spends its latency budget comparing false additions, the delivery channel emits a wall of duplicates, and the on-call engineer receives a page for damage created inside the watcher. Preserving the accepted snapshot on every non-success path breaks that chain at its first link.
The scrape must also identify itself with a stable User-Agent, honor the site's access policy and terms, use a timeout, and avoid overlapping runs. Conditional requests make that courtesy concrete: send the stored ETag in If-None-Match and the stored modification time in If-Modified-Since. An origin may answer 304 Not Modified, but validators are optimizations, not the durable truth; the extracted canonical content remains the comparison boundary.
What page fired? It should be source_observation_stale, carrying the last successful observation time, consecutive failure count, and source identifier. A dashboard full of request counts will not answer whether the property operations team missed a policy change.
No page for ordinary edits.
Build a stable diff before asking for semantic similarity
HTML is presentation, so hash a canonical representation of the entries rather than the response body. Select the smallest stable container that owns the changelog, remove scripts, styles, navigation, timestamps such as "last viewed," and normalize whitespace. Preserve entry order only when order conveys recency. Store both the raw response and canonical text for a limited retention period so a parser regression can be investigated without immediately fetching a moving source again.
The structural diff is the first gate. Exact equality ends the run cheaply and predictably. Added or modified entries continue to retrieval, where each candidate is compared with a bounded window of recent records such as property notices, maintenance advisories, and lease-policy updates. Retrieval-augmented generation joins retrieved external context to generation; the useful idea here is narrower: retrieval supplies possible prior matches, while deterministic policy decides whether to notify. No generator belongs in the critical path unless the notification genuinely needs a summary.
A vector score is a ranking signal, not a universal meaning threshold. Embedding model, text length, chunking, and corpus all affect the score distribution. Calibrate on labeled pairs from this domain: duplicate wording, materially changed dates, different buildings with similar maintenance language, and unrelated notices that share boilerplate. The choice is explicit. A wider candidate set can improve recall but raises latency and creates more opportunities for false matches; a smaller set returns faster but may miss an older duplicate. Measure both on the same frozen evaluation set before changing the cutoff or candidate count.
This approach has limitations. It is a poor fit for a page that requires an interactive browser challenge, for a source whose terms prohibit automated access, or for a feed where every update must be acted on in seconds; use an authorized feed or publisher-supplied webhook in those cases. Daily polling also cannot prove that it observed a short-lived edit between runs, while semantic suppression can hide a meaningful change when protected fields are incomplete. The trade-off is acceptable only when the team can tolerate that observation interval and can route uncertain matches to review.
The four controls should leave these artifacts:
| Control | Durable input | Success evidence | Failure action |
|---|---|---|---|
| Fetch | URL plus validators | status and response hash | retain baseline; retry |
| Extract | raw response plus parser version | canonical content hash | retain baseline; alert after recurrence |
| Compare | new entries plus retrieval corpus version | candidates, scores, decision | quarantine uncertain cases |
| Notify | decision plus idempotency key | delivery receipt | retry without advancing baseline |
Notice the order. Retrieval quality cannot rescue an unstable extractor, and a low-latency similarity query cannot compensate for losing the event after a delivery timeout.
Implement the run as a transactional handoff
Use a lease so two daily invocations cannot race. Persist a source record containing validators, the accepted canonical hash, snapshot location, last success time, and parser version. Persist notification attempts separately with an idempotency key derived from the source, canonical hash, and destination. The baseline advances only in the same durable transition that records successful delivery, or after an operator explicitly accepts a quarantined change.
The following Go sketch shows the control flow without binding the design to a scraping library, vector database, or messaging provider. A Node.js scheduler can execute this worker as a subprocess and treat its exit status as the run outcome; production implementations should back Store with transactions rather than process memory.
package watcher import ( "context" "crypto/sha256" "encoding/hex" "errors" "time" ) type Snapshot struct { ETag, LastModified, Canonical, Hash string } type Candidate struct { ID string Score float64 } type Fetcher interface { Fetch(context.Context, string, Snapshot) (body []byte, next Snapshot, notModified bool, err error) } type Extractor interface { Canonicalize([]byte) (string, error) } type Retriever interface { Nearest(context.Context, string, int) ([]Candidate, error) } type Notifier interface { Send(context.Context, string, string) error } type Store interface { Load(context.Context, string) (Snapshot, error) CommitDelivered(context.Context, string, Snapshot, string) error } func Run(ctx context.Context, source string, f Fetcher, x Extractor, r Retriever, n Notifier, s Store) error { ctx, cancel := context.WithTimeout(ctx, 45*time.Second) defer cancel() old, err := s.Load(ctx, source) if err != nil { return err } body, next, unchanged, err := f.Fetch(ctx, source, old) if err != nil { return err // Preserve the accepted baseline. } if unchanged { return nil } canonical, err := x.Canonicalize(body) if err != nil || canonical == "" { return errors.New("canonical extraction failed") } sum := sha256.Sum256([]byte(canonical)) next.Canonical, next.Hash = canonical, hex.EncodeToString(sum[:]) if next.Hash == old.Hash { return nil } candidates, err := r.Nearest(ctx, canonical, 8) if err != nil { return err } message := RenderDecision(canonical, candidates) key := source + ":" + next.Hash if err := n.Send(ctx, key, message); err != nil { return err // Retry with the same key; do not consume the change. } return s.CommitDelivered(ctx, source, next, key) } RenderDecision should apply a versioned policy, not improvise from a single score. For example, it can suppress a candidate only when calibrated similarity is above the reviewed threshold and protected fields such as property identifier, effective date, and notice type agree. A record saying "water shutoff on May 12" must not collapse into "water shutoff on May 19" because most tokens match. Put uncertain pairs in a review queue, and include the source excerpt and prior candidate in the notification so a human can verify the claim.
There is a small but important state issue in the sketch: if delivery succeeds and the commit fails, the next run may send again. The stable idempotency key makes that retry harmless only if the notifier enforces idempotency. If it cannot, persist an outbox row transactionally with the new snapshot, let a separate dispatcher deliver it, and mark the row sent after acknowledgment. This is the familiar transactional outbox boundary, and it is preferable to pretending a network call and a database commit are atomic.
Verify retrieval quality without hiding latency
Build a labeled corpus before tuning. For property management, include exact reposts, paraphrased maintenance notices, template-heavy notices for different buildings, date-only changes, unit-specific exceptions, and genuinely unrelated records. Split by time so near-identical records from one incident do not leak across training and evaluation. Report precision and recall at the actual operating threshold, then report p50, p95, and p99 end-to-end latency separately for fetch, extraction, retrieval, and notification.
No invented target helps here. The acceptable false-negative rate depends on what the notice controls, while the latency budget depends on the daily workflow. A legal-policy change may favor recall and human review; a low-impact amenity reminder may favor precision to prevent alert fatigue. Choose the threshold from labeled consequences, not from a pleasing round similarity score.
Fast is secondary.
Test the ugly paths. Replay a 304, a timeout, malformed HTML, a selector returning an empty string, reordered entries, duplicated delivery, retrieval unavailability, and a commit failure after delivery. Save fixtures from pages you are authorized to retain, strip personal data, and pin the parser version in every result. A canary run can execute a new extractor against the same stored responses without changing production state.
For observability, emit one structured run event with a correlation ID and stage durations, plus counters by terminal outcome. Track the age of the last successful canonical observation. That age is closer to the user-visible risk than a green process heartbeat.
Roll back the parser, policy, and corpus independently
Deploy extraction and semantic policy as separate versions. During a parser change, dual-run old and new extraction against stored raw responses and compare canonical hashes. During a threshold or embedding change, shadow the new policy on labeled and recent production candidates, record decisions without notifying, and inspect disagreements. Do not overwrite the old vectors until the new index is complete and queryable.
Rollback is then mechanical: stop the schedule, select the last accepted parser and policy versions, point retrieval at the prior complete corpus, and replay from the last delivered canonical hash. Keep notification idempotency keys through the entire replay window. If the source itself made a legitimate large rewrite, quarantine it for review rather than teaching the extractor to ignore broad changes.
Four independent controls make the watcher boring in the useful sense. The daily schedule is incidental; correctness comes from preserving evidence, separating failures, calibrating semantic decisions on property-specific records, and refusing to advance state before the change is durably delivered.



