diff --git a/ondeck/docs/shape-historical-import.md b/ondeck/docs/shape-historical-import.md new file mode 100644 index 0000000..f8ac874 --- /dev/null +++ b/ondeck/docs/shape-historical-import.md @@ -0,0 +1,230 @@ +# SHAPE Historical Data Import + +> **Script:** `scripts/import-shape-historical.ts` +> **Last reviewed:** April 2026 + +--- + +## For the Team + +### What this script does + +This script imports historical SHAPE task completion data from the team's SharePoint Excel files into OnDeck. It is a **one-time migration tool** — not part of the ongoing sync. + +Each team member has a folder on SharePoint at: +``` +Claims/SHAPE Accounts/{MEMBER_FOLDER}/{Client}.xlsx +``` + +The script reads every client Excel file for each advocate, matches it to a client in OnDeck, and records whether each SHAPE task was completed, marked N/A, or left blank. + +### Team member folders + +| SharePoint Folder | Advocate | +|---|---| +| `CHRIS` | Christine Gove | +| `DAWN` | Dawn Boland | +| `Jeanne` | Jeanne Strong | +| `LUKE` | Luke Billman | +| `MIMI` | Mimi Rawlings | + +### Running the script + +**Always do a dry run first.** The dry run reads SharePoint and the database but writes nothing. + +```bash +# Dry run (safe — no changes made) +npx tsx scripts/import-shape-historical.ts + +# Execute (writes to database) +npx tsx scripts/import-shape-historical.ts --execute + +# Save report to a custom location +npx tsx scripts/import-shape-historical.ts --output=/home/user/import-report.txt +``` + +The report is always written to `/tmp/shape-import-report.txt` (or your `--output` path). Check it before running `--execute`. + +### What the report tells you + +After running, the report shows: + +| Stat | Meaning | +|---|---| +| **Files processed** | Excel files successfully downloaded and parsed | +| **Clients matched** | Clients in the Excel that were found in OnDeck | +| **Clients unmatched** | Clients that could not be matched — need manual review | +| **Fuzzy matches** | Clients matched by approximate name — review for correctness | +| **Tasks updated** | Existing tasks whose status was set to Completed or N/A | +| **Tasks assigned** | Tasks that had the advocate linked to them | +| **Tasks created** | Tasks from the Excel that didn't exist in OnDeck yet | +| **Duplicates deleted** | Extra copies of the same task removed from the DB | +| **Ad-hoc created** | "Additional Services" rows and unmatched tasks saved as free-form tasks | +| **Advocates assigned** | Clients that had their Claims Advocate set from this import | + +### What gets skipped + +The script will **not** import a row if: +- The client name in the Excel cannot be matched to any OnDeck client (logged as "unmatched") +- The file is named `archive*`, contains `template`, `master workbook`, or starts with `all shape` — these are intentionally excluded + +Nothing else is filtered. Every task row in every matched client file is processed. + +### Does deduplication remove tasks from Excel? + +**No.** Deduplication only removes extra copies of the *same task* that already exist in the database (e.g. from a previous partial run of the script). It does not skip or discard any rows from the Excel files. + +Specifically: if the database already has two identical tasks for the same client and template within ±5 days of the same due date, the older one is kept and the newer duplicate is deleted. The Excel row is still processed normally against the surviving task. + +### What to do with unmatched clients + +After running the dry run, check the **"Unmatched Clients"** section of the report. For each one: + +1. Check if the client exists in OnDeck under a different spelling +2. If yes — either rename the client in OnDeck to match, or rename the Excel file +3. If no — create the client in OnDeck first, then re-run the import + +--- + +## Technical Reference + +### Architecture overview + +``` +SharePoint (Graph API) + └── Claims/SHAPE Accounts/ + ├── CHRIS/ → Christine Gove (userId: cmkm36yc90021y7vb9783nitl) + ├── DAWN/ → Dawn Boland + ├── Jeanne/ → Jeanne Strong + ├── LUKE/ → Luke Billman + └── MIMI/ → Mimi Rawlings + +Each .xlsx file: + ├── One or more client blocks per sheet + │ ├── Client Name + │ ├── Effective Date (Excel serial number) + │ └── Task rows: [Task Name | Days After Renewal | Date Completed | Notes] + └── "Additional Services Provided" section (ad-hoc tasks) +``` + +### Processing pipeline + +``` +1. fixTemplates() Fix known template name/offset discrepancies in DB +2. loadDbState() Load SHAPE/Shape2 designations, templates, and clients +3. discoverFiles() List Excel files from SharePoint via Graph API +4. processFile() × N For each file: + └── parseExcelFile() Parse XLSX without external libraries (custom ZIP+XML) + └── parseExcelBlocks() Extract client blocks and task rows + └── matchClient() Match Excel client name → OnDeck client + └── processTaskRow() × N For each task row: + └── findMatchingTasks() Look up existing task in DB (±5-day window) + └── deduplication Delete extras if >1 match + └── update or create Set status, completedAt, assign advocate + └── createAdHocTask() × N For "Additional Services" rows +5. printReport() Write summary to stdout and output file +``` + +### XLSX parsing + +The script parses `.xlsx` files without any npm dependencies. `.xlsx` is a ZIP archive containing XML files. The parser: + +1. Finds the End of Central Directory record to locate the ZIP central directory +2. Reads each file entry (stored or deflate-compressed via Node's `zlib.inflateRawSync`) +3. Parses `xl/sharedstrings.xml` to build a string table +4. Identifies the target sheet — prefers `SHAPE2` > `SHAPE` > `CLIENT` tab names, falls back to sheet1 +5. Parses the sheet XML into a `(row, col) → string` grid +6. Scans for `Client Name:` anchors and reads blocks downward + +### Client name matching (3 tiers) + +``` +1. Exact normalized match + - Strip "the", "Inc", "LLC", "Corp", etc., lowercase, collapse whitespace + - Direct map lookup + +2. Contains match + - One normalized name contains the other as a substring + +3. Jaccard token similarity ≥ 0.65 + - Tokenize both names, compute intersection/union ratio + - Only used if tiers 1 and 2 fail +``` + +Fuzzy matches (tiers 2–3) are listed in the report for manual review. + +### Template matching (3 tiers) + +``` +1. Exact normalized name (after alias substitution) +2. Contains match on normalized name +3. Days-offset fallback: approxOffset = -(365 - daysAfterRenewal), ±15-day window +``` + +Known aliases handled (`TEMPLATE_NAME_ALIASES`): +- `request 125 day loss runs` → `request 120 day loss runs` +- `request 89 day loss runs` → `request 90 day loss runs` +- `claim review (six month)` / `claims review` → `claim review` + +### Deduplication logic + +`findMatchingTasks(clientId, templateId, dueDate)`: +- Queries for tasks matching `clientId + templateId` with `dueDate ±5 days` +- Returns the oldest match as the canonical task +- Returns all additional matches as `duplicateIds` → these are hard-deleted + +This only cleans up DB duplicates from previous partial runs. It never suppresses Excel rows. + +### Date completed interpretation + +| Excel cell value | Result | +|---|---| +| Empty / `??` / `?` | `NOT_STARTED` | +| `n/a` / `na` | `NA` (with standard naReason) | +| Numeric Excel serial (40000–60000) | `COMPLETED`, date converted | +| Anything else | `NOT_STARTED` | + +### Task creation rules + +| Situation | Action | +|---|---| +| Task exists in DB, status = `NOT_STARTED`, Excel says completed/NA | Update status, completedAt, assign advocate | +| Task exists in DB, already completed | Assign advocate only (never overwrite completion) | +| Task does not exist, Excel says completed/NA | Create task with completed status, assign advocate | +| Task does not exist, Excel says `NOT_STARTED` | Create task as `NOT_STARTED`, assign advocate | +| Template not matched, Excel says completed/NA | Create ad-hoc task: `[Unmatched Task] {row name}` | +| Template not matched, `NOT_STARTED` | Skip (no DB write) | + +### Advocate assignment + +- Advocate is set at the **client level** (`claimsAdvocateId`) only if the client has none yet +- Existing advocate assignments are **never overwritten** +- Advocate is also linked at the **task level** via `TaskAssignment` upsert + +### Template fixes run on each execution + +Even in dry-run mode, these are applied to the DB on `--execute`: +- `Request 125 day loss runs` (offset -125) → renamed to `Request 120 day loss runs`, offset -120 +- Templates with offset -89 → corrected to -90, names updated accordingly +- Missing `Claim Review` at -185 days (Shape designation) → created if absent + +### Environment variables required + +``` +DATABASE_URL PostgreSQL connection string +AZURE_AD_TENANT_ID Microsoft Entra tenant ID +AZURE_AD_CLIENT_ID App registration client ID +AZURE_AD_CLIENT_SECRET App registration client secret +``` + +The Azure app registration needs `Files.Read.All` (or `Sites.Read.All`) on Microsoft Graph. + +### Re-running safely + +The script is fully idempotent on `--execute`: +- Tasks are updated, not duplicated (±5-day deduplication window) +- Advocate assignments use upsert +- Ad-hoc tasks check for existing `(clientId, title, isAdHoc=true)` before creating +- Template fixes use `updateMany` with specific `where` conditions + +Re-running after adding new clients or fixing name mismatches is safe.