Evaluate with only the data available at decision time

A model makes a decision at 10:00. Today's dataset contains an event timestamped 09:30, so it looks safe to include that event in the model's historical input. But the event did not arrive until 10:20. The model could not have used it at 10:00.

Later corrections create the same problem. A value may describe an old event while containing knowledge introduced after the decision. A chronological train/test split does not repair either issue; the feature values inside each row must also respect what was available then.

This extends the time boundary in preventing evaluation data leakage with a worked reconstruction. Polyaxon can retain the source snapshot, reconstruction policy, and resulting dataset so downstream evaluations use an inspectable historical input.

Keep event time and availability time separate

At minimum, record when an event happened and when its value became available to the system making the decision. Those timestamps answer different questions.

Availability should reflect the actual serving path. A raw ingestion timestamp can be too early if validation, transformation, replication, or indexing delayed use. Record the relevant boundary instead of assuming that a row's presence in an upstream system meant the model could read it.

Feast's point-in-time join documentation makes a related distinction: joining by event timestamp alone can expose backfilled or corrected values. Its current documentation describes an optional created-timestamp filter, with offline-store support requirements. Whatever tool you use, verify that the timestamp represents availability in your system.

Work through a late arrival and a correction

Suppose a model uses an account's recorded usage total at 10:00. These are synthetic event versions, not customer data:

At 10:00, only usage-1 revision 1 is available, so the reconstructed input is 10 units. The late usage-2 row must be excluded even though its event time is earlier. The correction to usage-1 must also be excluded because it became known at 11:00.

Selecting the latest version of every event from today's export and filtering only on event time produces 120 units. That may be the corrected total for the past, but it is not the total the model could have seen then. Preserve both views when useful and name the claim each one supports.

Select available versions before aggregating

Save this as reconstruct-usage.py. It uses only Python's standard library and applies two constraints before choosing the latest eligible revision of each event. All timestamps in the example are timezone-aware UTC.

from datetime import datetime
import json


def timestamp(value):
    result = datetime.fromisoformat(value.replace("Z", "+00:00"))
    if result.tzinfo is None:
        raise ValueError("A timezone is required")
    return result


def available_versions(rows, decision_at):
    cutoff = timestamp(decision_at)
    selected = {}
    seen = set()
    for row in rows:
        key = (row["event_id"], row["revision"])
        if key in seen:
            raise ValueError("Duplicate event revision")
        seen.add(key)
        if timestamp(row["event_at"]) > cutoff:
            continue
        if timestamp(row["available_at"]) > cutoff:
            continue
        prior = selected.get(row["event_id"])
        order = (timestamp(row["available_at"]), row["revision"])
        if prior is None or order > (
            timestamp(prior["available_at"]), prior["revision"]
        ):
            selected[row["event_id"]] = row
    return [selected[key] for key in sorted(selected)]


rows = [
    {"event_id": "usage-1", "revision": 1, "units": 10,
     "event_at": "2026-09-01T09:00:00Z", "available_at": "2026-09-01T09:05:00Z"},
    {"event_id": "usage-2", "revision": 1, "units": 20,
     "event_at": "2026-09-01T09:30:00Z", "available_at": "2026-09-01T10:20:00Z"},
    {"event_id": "usage-1", "revision": 2, "units": 100,
     "event_at": "2026-09-01T09:00:00Z", "available_at": "2026-09-01T11:00:00Z"},
]
decision_at = "2026-09-01T10:00:00Z"
selected = available_versions(rows, decision_at)
report = {
    "decision_at": decision_at,
    "selected_versions": [[r["event_id"], r["revision"]] for r in selected],
    "usage_units": sum(r["units"] for r in selected),
}
print(json.dumps(report, indent=2))

The expected selected version is usage-1 revision 1 and the expected total is 10. The example is source-reviewed and unexecuted. It models additive usage events for one account, with stable event times across revisions and no deletions. It is not a general temporal database implementation.

For production data, apply the account/entity key, feature window, freshness policy, and revision rules explicitly. Support tombstones if events can be deleted. If corrections can change an event's timestamp or entity, define how the historical version is resolved before filtering and aggregating; do not assume this simplified selector covers those semantics.

Preserve the information needed to reconstruct

An append-only version history or equivalent audit log is necessary to recover old values. If a source overwrites 10 with 100 and retains no version history, a timestamp filter cannot reconstruct the lost value. Mark that limitation in the dataset rather than presenting today's corrected view as a historical observation.

Feature transformations also have availability boundaries. The source rows may be eligible while the aggregate refresh completed too late. Decide whether the study reconstructs information theoretically available from eligible records or the exact materialized feature served online. Those are different evaluation contracts.

For RAG, apply the same reasoning to document publication, permission changes, index refreshes, and the context available at the question time. A document written after an incident may explain it perfectly while being unavailable to the original assistant. Preserve historical access rules and index state when the evaluation claims to reproduce that assistant's information boundary.

Track the reconstruction in Polyaxon

Run reconstruction as a data-preparation operation with immutable source inputs. Save the decision manifest, selected record revisions, excluded-record reasons, transformation revision, and resulting feature dataset. Record the timezone, inclusivity rule (available_at <= decision_at here), and the meaning of availability.

Use Polyaxon input references and dataset versioning to connect the output to its producer and source snapshot. Retain the actual files through artifact tracking; a source URI alone does not preserve the historical bytes.

For the tiny example, append the following inside a configured Polyaxon tracking environment to retain the already-created report and its policy. It requires the Polyaxon package, run authentication, and artifact storage:

from pathlib import Path

from polyaxon import tracking
from polyaxon.schemas import V1ArtifactKind

tracking.init()
tracking.log_outputs(
    reconstruction_policy="available-before-decision-v1",
    decision_at=decision_at,
)
tracking.log_metrics(selected_event_count=len(selected), usage_units=report["usage_units"])
path = Path(tracking.get_outputs_path("reconstruction.json"))
path.write_text(json.dumps(report, indent=2) + "\n")
tracking.log_artifact_ref(path=str(path), kind=V1ArtifactKind.FILE)

This continuation reuses variables from the previous script. It records the teaching result, not a complete production audit. A real preparation run should retain the source snapshot and exclusions as well as the selected rows, so a reviewer can explain why each version was included or left out.

Keep labels on their own clock

A later outcome may legitimately become the evaluation label for an earlier prediction. It must not leak into that prediction's input. When simulating model training at a historical date, the labels used for training must also have been available by that training cutoff.

Keep decision time, feature availability, outcome window, and label availability separately. Then apply the intended entity and chronological split rules. Reconstructing inputs correctly does not by itself prevent the same customer, document family, or future label from crossing another evaluation boundary.

The final dataset should let you answer a precise question for each prediction: which source versions could this system use at that time? Keeping that answer with the Polyaxon preparation run turns historical evaluation from an assumption into reviewable evidence.