Your comment timeline is using the wrong clock

Your release report puts the busiest discussion at the start of the song. Before you rewrite the chart, check which clock it reads.

SoundCloud comments separate posting time (createdAt) from playback position (timestampMs). A comment attached to the opening can have been posted long after release.

Example input for collecting reactions by their position inside a track:

const run_input = { mode: "comments", trackUrls: [{ url: "https://soundcloud.com/user/track" }], commentsSort: "track-timestamp", maxItems: 200, }; 

That order serves a different question from “What are listeners saying recently?” Once collection reaches its row cap, changing the chart's sort can't recover comments outside the collected batch.

The annoyance looks cosmetic. The repair might require recollection.

Read the symptom before changing the collector

The Actor asks SoundCloud to order comments using commentsSort. Its choices are newest, oldest, and track-timestamp; the default is newest.

The source supports different views of the same discussion. Your dashboard needs to choose which question it answers.

These signals can coexist. Fixing the chart field doesn't establish that its underlying sample suits the report.

Keep the submitted order beside the finished rows

A dataset tells you what arrived. It doesn't prove which input selected those rows.

Persist the exact submitted input alongside the run ID and dataset ID. For a historical diagnosis, use that snapshot rather than today's scheduler configuration.

This code waits for collection and loads its dataset. The inline run_input serves as the submitted snapshot throughout the remaining cells.

import { ApifyClient } from "apify-client"; import { z } from "zod"; const client = new ApifyClient({ token: process.env.APIFY_TOKEN, }); const run = await client .actor("nimbryxo-studio/soundcloud-charts-search-and-artist-catalog") .call(run_input); const { items } = await client .dataset(run.defaultDatasetId) .listItems(); const finishedSuccessfully = run.status === "SUCCEEDED"; const rawRowCount = items.length; 

Keep rawRowCount before filtering or validation. The cap applies to dataset rows, so comparing it with a smaller chart-ready subset answers the wrong question.

A finished run can also have an unsuccessful status. Treat that as an operational issue before interpreting its contents as a completed collection.

Preserve missing values instead of inventing moments

Empty fields are omitted from records. Missing playback position therefore needs its own handling.

Replacing an absent timestampMs with zero manufactures a reaction at the opening. Replacing an absent createdAt with scrapedAt manufactures a posting date. Both repairs make the chart look healthier than its evidence.

This validator accepts missing clocks, checks supplied values, and keeps rejected comment rows separate.

const identifier = z.union([ z.string().min(1), z.number().finite(), ]); const instant = z.string().refine( value => Number.isFinite(Date.parse(value)), "Expected a parseable instant", ); const commentSchema = z.object({ recordType: z.literal("comment"), commentId: identifier, trackId: identifier, createdAt: instant.optional(), timestampMs: z.number().finite().nonnegative().optional(), scrapedAt: instant.optional(), }); type Comment = z.infer<typeof commentSchema>; const comments: Comment[] = []; let nonCommentRows = 0; let rejectedComments = 0; for (const item of items) { if ( typeof item !== "object" || item === null || !("recordType" in item) || item.recordType !== "comment" ) { nonCommentRows++; continue; } const parsed = commentSchema.safeParse(item); if (parsed.success) comments.push(parsed.data); else rejectedComments++; } 

A validation rejection means the row doesn't meet this application's boundary requirements. It doesn't, by itself, identify an Actor defect.

The distinction matters during diagnosis: an absent clock is different from a supplied value your application can't parse. Keep both counts visible beside the chart.

Which clock does the chart actually read?

Rows can't reveal your dashboard's mapping. Pass the field its mapper actually uses.

The following detector assumes the report's purpose is recent discussion. Example configuration passes a playback field to expose that mismatch:

function diagnoseRecentDiscussion( timeField: "createdAt" | "timestampMs" | "scrapedAt", ) { return { unsuccessfulRun: !finishedSuccessfully, wrongChartAxis: timeField !== "createdAt", collectionTimeUsed: timeField === "scrapedAt", orderMismatch: run_input.commentsSort !== "newest", possibleCapSaturation: finishedSuccessfully && rawRowCount === run_input.maxItems, missingPostingTime: comments.filter( row => row.createdAt === undefined, ).length, missingPlaybackPosition: comments.filter( row => row.timestampMs === undefined, ).length, rejectedComments, nonCommentRows, }; } const diagnosis = diagnoseRecentDiscussion("timestampMs"); 

orderMismatch identifies configuration unsuitable for this purpose. It doesn't establish malformed records or prove that recent comments are missing.

Likewise, equality with the cap is evidence of possible truncation. The public source might have had exactly that many available rows. The count alone can't settle it.

A lower count isn't a completeness certificate either. You've observed the returned batch, not independently established the size of the public discussion.

Build a projection for each question

Keep posting activity and musical position as separate projections. Each can be useful without pretending to answer the other's question.

comment createdAt -> posting day -> release-response timeline timestampMs -> playback bucket -> moments within the track scrapedAt -> collection time -> ingestion monitoring 

Group playback reactions by track. Pooling raw positions across recordings can make unrelated musical moments look equivalent.

Example projection settings use UTC posting days and a playback bucket width defined locally:

const playbackBucketMs = 1000; function projectComments(field: "createdAt" | "timestampMs") { const buckets = new Map<string, number>(); if (!finishedSuccessfully) return buckets; for (const row of comments) { const time = field === "createdAt" ? row.createdAt === undefined ? undefined : Date.parse(row.createdAt) : row.timestampMs; if (time === undefined) continue; const bucket = field === "createdAt" ? new Date(time).toISOString().slice(0, 10) : Math.floor(time / playbackBucketMs) * playbackBucketMs; const key = JSON.stringify([String(row.trackId), bucket]); buckets.set(key, (buckets.get(key) ?? 0) + 1); } return buckets; } const postingDays = projectComments("createdAt"); const playbackPositions = projectComments("timestampMs"); 

These maps describe the validated, collected sample. Neither map repairs its selection.

For release-response analysis, compare posting days with your own release calendar. Keep ingestion time separately so a late collection doesn't create a false surge on collection day.

For comparisons across tracks of different lengths, use relative playback position. That requires a separate duration lookup keyed by trackId; comment rows don't provide track duration.

Recollect only when the sample is wrong

If the mapping is wrong, rebuilding the projection is enough. If the collection order is wrong for recent discussion, sorting existing rows newest-first changes presentation without changing membership.

The SoundCloud Charts, Search & Artist Catalog Scraper is the worked example here. The reusable failure is a bounded collection whose ordering doesn't match the downstream question.

This corrected input changes selection order while retaining the original scope and ceiling:

const correctedInput = { ...run_input, commentsSort: "newest", }; 

Submit it through the same collection path when the diagnosis warrants another run. Keeping that submission separate from diagnosis helps avoid paid retries whenever someone opens the report.

The ceiling remains a trade-off. maxItems supports values from 1 to 5000, and raising it can collect more rows. It still can't establish that a release window is fully covered.

Recollection incurs event fees plus platform usage. On the FREE tier, emitted results cost $0.00099 each. The start event costs $0.10 per event, with one event per GB of memory and a minimum of one event.

There's another analytical catch: newest-first serves recent-discussion monitoring, but it can't guarantee recovery of an older release window after collection has been delayed. Store successive observations if that history matters, and deduplicate using your comment identity policy.

For ingestion, retain the earliest collection time separately from any later refresh time. A returning comment is still the same historical event.

Before you publish the release report

• Confirm the dashboard reads the clock its question requires.
• Inspect the submitted input snapshot.
• Check run status and raw row count before filtering.
• Keep missing-clock counts separate from validation rejections.
• Rebuild projections locally when selection remains suitable.
• Recollect deliberately when selection must change.

A tidy timeline can still be an answer to the wrong question.