Athena's pipeline has four layers, in order: ingestion/parsing (per source file type) → normalization (mapping everything into one common transaction/balance model) → detection/reasoning (one module per finding type / check) → output/reporting (structured, source-traceable findings). Each layer's boundary is deliberate: normalization is the single place heterogeneous formats become one shape, so every detection module operates on the same model regardless of whether the data originated in a CSV, an XML auditfile, a spreadsheet, or extracted text from a PDF or DOCX.
[CSV] [XAF XML] [XLSX] [PDF] [DOCX]
| | | | |
v v v v v
+-----------------------------------+
| Ingestion / Parsing (per type) |
+-----------------------------------+
|
v
+-----------------------------------+
| Normalization |
| (common transaction/balance/ |
| confirmation-letter model) |
+-----------------------------------+
|
v
+-----------------------------------+
| Detection / Reasoning |
| (one module per finding type |
| / normal_check) |
+-----------------------------------+
|
v
+-----------------------------------+
| Output / Reporting |
| (structured findings, source- |
| traceable, review-flagged) |
+-----------------------------------+
One parser per source file type, each responsible only for turning its file into an intermediate, format-native representation — no cross-file logic here.
01_...bankmutaties.csv, 08_...lapping_scenario.csv) — delimited (;)
parsing into row records keyed by the header. Straightforward, deterministic,
schema known in advance from 10_data_dictionary.json.02_...auditfile_XAF.xml) — XML parsing of
generalLedgerAccounts and transactions elements. Note the seed data explicitly
states this is a simplified XAF-like format, not validated against the official
XAF 3.2 XSD — the parser should be schema-tolerant (namespace-aware but not
XSD-strict) rather than assuming full XAF compliance.03_...saldibalans.xlsx, 07_...bankreconciliatie...xlsx) — spreadsheet
parsing (sheet → rows → typed cells), reading both literal values and, where present,
formula results (e.g., the reconciliation sheet's SUM(B2:B3) cell) — the computed
value, not the formula text, is what normalization needs.04_...rabobank...pdf, 05_...ing...pdf) — text extraction. In this
training set both bank statement PDFs are text-layer PDFs (not scanned images), so
ordinary PDF text extraction is sufficient to recover the tabular statement lines
(date, description, af/bij amount, running balance) — no OCR was needed to read
either file in this dataset. Production bank statements from real institutions may
arrive as scanned/image PDFs; the ingestion layer should still be designed with an
OCR fallback path for that case, but it is not exercised by this training set.06_...bankverklaring...docx) — text/table extraction from the Word
document body. This training file structures its content as a field/value table
(Bank, Peildatum, IBAN, Saldo, Kredietfaciliteit, Bankgaranties, Zekerheden,
Geblokkeerde rekeningen, Bevoegde personen, Overige verplichtingen) — a table-aware
DOCX parser recovers this directly without needing free-text NLP for this particular
document. A production bank confirmation letter is not guaranteed to keep this
tabular structure, which is why the extraction step downstream is LLM-assisted
rather than a fixed table-column parser (see layer 3).Each parser's output retains a pointer back to its source file name and, where meaningful, a location within that file (row number, XML element, cell reference, page) — this pointer is required by the output layer's source-traceability requirement and must not be dropped during normalization.
Maps every parsed source into one common model, so downstream detection logic never
needs to know which file format a fact came from. Based directly on the fields
documented in 10_data_dictionary.json plus the additional fields observed in the
bank statement PDFs and reconciliation/trial-balance spreadsheets:
journal_id, booking_date, document_date, gl_account,
description, counterparty, debit, credit, currency, user,
manual_entry (bool), source (bank_import | manual), plus
source_file/source_location for traceability. CSV rows, XAF <transaction>
elements, and bank-statement PDF lines all normalize into this shape (a bank
statement line becomes a transaction with source = bank_statement and the bank's
own value date as booking_date, distinct from the GL's booking_date for the same
economic event — this date/source distinction is exactly what kiting detection
depends on).gl_account, account_description, debit, credit, net_balance,
classification (e.g., "Liquide middelen", "Liquide middelen beperkt beschikbaar"),
control_note, source_file. Trial balance rows normalize directly into this shape.customer, invoice,
invoice_date, amount, bank_receipt_date, bank_amount, applied_to_invoice,
posting_date, source_file.Normalization is where format-specific quirks get resolved once instead of in every
detection module — e.g., Dutch decimal/thousands formatting in the DOCX/PDF text
("EUR 152.340,25") is parsed into a single canonical numeric type here, and GL account
IDs are matched against the generalLedgerAccounts/trial-balance account list to
attach a consistent account_description/classification regardless of which file an
account ID appeared in.
One module per finding type / normal_check, each consuming only the normalized model. The critical design split, evidenced directly by this training set, is which detections are computable by rule-based cross-referencing of structured data alone, and which require LLM-assisted extraction from unstructured/semi-structured document content:
Rule-detectable from structured data alone (no LLM inference needed):
booking_date (GL) against the bank's own value date on each side, flag when the
incoming side's bank date exceeds the balance sheet date while the GL booking date
does not.manual_entry = true
and source = manual, then check for an offsetting/corroborating bank_import
transaction or reconciliation-workpaper line; flag when none exists.classification containing "beperkt
beschikbaar" or matching account descriptions like "G-rekening", then confirm against
the BankConfirmation's blocked_accounts — both sides are structured once
normalized, so this is a join, not an inference.customer to the customer who owns applied_to_invoice, and flag connected
chains — this is deterministic graph logic over structured CSV data.normal_checks — IBAN format validation (checksum algorithm), balance
reconciliation arithmetic, confirmation-vs-register set comparison, negative-balance
flagging, and date-window filtering are all deterministic computations over the
normalized model.Requires LLM-assisted extraction from unstructured/semi-structured content:
This split is a first-class design decision, not an implementation detail: the four ground-truth finding types (kiting, unsupported_manual_journal, restricted_cash, lapping) are all reproducible by deterministic rules once the common model exists, and v1 should implement them that way rather than reaching for LLM inference where it is not needed.
Produces findings and check results in one structured, auditable format, directly
modeled on 09_verwachte_controlebevindingen.json's shape:
finding_id, type, severity, source_files (the exact
list of files/documents that produced the flag), a human-readable logic string
explaining the pattern, and a expected_detection-shaped payload of the specific
data points involved (amounts, dates, accounts, IDs) — see scope.md for the exact
shape per finding type.normal_check result carries the check name, a pass/exception/gap status, and
the supporting data used to reach that status (not just a boolean).confirmed: true field, no severity level implying
certainty beyond "flag priority").source_files that produced it, and the output layer cannot manufacture that
traceability after the fact if it was dropped upstream.
Reacties