chore: merge executor worktree (worktree-agent-a0da7e5b2254b56f8)

This commit is contained in:
lorentz 2026-07-10 22:48:53 -04:00
commit fe7860760b
3 changed files with 528 additions and 0 deletions

View file

@ -0,0 +1,113 @@
---
phase: 12-orders-invoices-company-matching
plan: 03
subsystem: database
tags: [postgres, pg_trgm, fuzzy-matching, typescript, pax8]
requires:
- phase: 12-orders-invoices-company-matching (Plan 01)
provides: pg_trgm extension enabled, pax8_companies auto-match columns (autotask_company_id, match_confidence, match_method, matched_at) via migration 093
provides:
- matchPax8Companies() — the D-01..D-05 fuzzy company matcher, isolated in its own service + test module
- Exported AUTO_LINK_THRESHOLD (0.90) / TIE_MARGIN (0.05) / CANDIDATE_FLOOR (0.3) tunables
- Pax8CompanyMatchResult result shape (scanned/autoLinked/flaggedAmbiguous/flaggedNoCandidate/durationMs)
affects: [12-04-pax8-sync-service-invoice-sync (wires matchPax8Companies into Pax8SyncService.fullSync())]
tech-stack:
added: []
patterns:
- "Confidence-ranked matching, ported from device-link-reconciler.ts's findBy*/applyLink/recordConflict/pickBestCandidate shape, adapted from a cascade of exact-match strategies to a single pg_trgm similarity() score"
- "Idempotency guard via NOT EXISTS on a resolved review row + match_method IS DISTINCT FROM 'manual', preventing automated re-scoring from ever overwriting a human decision"
key-files:
created:
- lib/services/pax8-company-matcher.ts
- lib/services/pax8-company-matcher.test.ts
modified: []
key-decisions:
- "CANDIDATE_FLOOR is inlined as a template-literal constant in the SQL string (not a bound param) since it's a hardcoded module constant, not external input — only the PAX8 company name (external input) is bound as $1, per the Security Domain requirement"
- "A now-confident auto-match closes any open (never-human-touched) review row for that company (DELETE ... WHERE resolved_at IS NULL) — a stale ambiguous flag shouldn't linger once the matcher becomes confident again on a later run"
patterns-established:
- "Single-strategy scored matcher shape reusable for any future fuzzy-match problem: findCandidates (parameterized, floor-filtered) -> decide (threshold + tie-margin) -> applyLink/recordConflict (idempotency-guarded)"
requirements-completed: [PAX8-10, PAX8-11]
duration: 25min
completed: 2026-07-11
---
# Phase 12 Plan 03: PAX8 Company Fuzzy Matcher Summary
**pg_trgm-based fuzzy matcher (`matchPax8Companies`) ports device-link-reconciler.ts's confidence-ranked match/review shape to PAX8-Autotask company matching, at a validated 0.90 auto-link floor with a 0.05 tie-margin, six unit tests proving every decision branch plus the human-resolution idempotency guard.**
## Performance
- **Duration:** ~25 min
- **Started:** 2026-07-11T02:26:00Z
- **Completed:** 2026-07-11T02:47:50Z
- **Tasks:** 2/2 completed
- **Files modified:** 2 (both created)
## Accomplishments
- `lib/services/pax8-company-matcher.ts` implements the full D-01..D-05 match policy: `findCandidates` (parameterized `similarity($1, company_name)` query, `is_active = true` filter per Pitfall 5), `decide` (auto-link vs. review), `applyLink` (resolved-row idempotency guard), `recordConflict` (top-3/empty-array review upsert with stale-match cleanup)
- `matchPax8Companies(opts?)` scans the re-scoring-eligible subset of `pax8_companies` (excludes `match_method = 'manual'` and rows with a human-resolved review row) and returns a `Pax8CompanyMatchResult` rollup
- `lib/services/pax8-company-matcher.test.ts` proves all five decision branches plus `dryRun`, mocking `postgresClient.query`'s default export — 6/6 tests passing
- Caught and fixed a block-comment-terminator bug during Task 1 (`findBy*/pickBestCandidate` in a doc comment prematurely closed the `/* */` block, breaking every downstream parse) before it ever reached a commit
## Task Commits
Each task was committed atomically:
1. **Task 1: Create lib/services/pax8-company-matcher.ts** - `691bb47` (feat)
2. **Task 2: Create lib/services/pax8-company-matcher.test.ts** - `ae44669` (test)
_Plan metadata commit follows this summary (worktree mode — orchestrator merges and updates STATE.md/ROADMAP.md after the wave)._
## Files Created/Modified
- `lib/services/pax8-company-matcher.ts` - Fuzzy PAX8-to-Autotask company matcher: `matchPax8Companies`, `AUTO_LINK_THRESHOLD`, `TIE_MARGIN`, `CANDIDATE_FLOOR`, `Pax8CompanyMatchResult`
- `lib/services/pax8-company-matcher.test.ts` - Unit tests for all five decision branches (auto-link, below-threshold review, near-tie review, empty-candidate review, idempotency guard) plus dryRun
## Decisions Made
- Kept `CANDIDATE_FLOOR` as an inlined SQL literal rather than a bound param — it's a hardcoded internal constant, not user/external input, so parameterizing it would add no security value while the PAX8 company name (the actual external input) stays strictly bound as `$1`
- `applyLink` closes any open review row for the same company on a fresh auto-match, since a newly confident automated match supersedes a previously-flagged (never human-touched) ambiguity — only rows with `resolved_at IS NOT NULL` (human-resolved) are left untouched
## Deviations from Plan
None — plan executed exactly as written. One inline bug was caught and fixed during Task 1 before any commit (see Deferred Issues below — not a deviation from the plan's design, a syntax slip in a doc comment).
### Auto-fixed Issues
**1. [Rule 1 - Bug] Fixed a block-comment-terminating `*/` sequence inside a doc comment**
- **Found during:** Task 1, immediately after first `npx tsc --noEmit` run
- **Issue:** The header doc comment wrote `findBy*/pickBestCandidate` (intending "findBy-star, pickBestCandidate") — TypeScript parsed the `*/` as the end of the `/** ... */` block comment, causing ~100 cascading parse errors for the rest of the file
- **Fix:** Reworded to `findBy-x / pickBestCandidate / applyLink / recordConflict` — no `*/` substring remains in any comment
- **Files modified:** `lib/services/pax8-company-matcher.ts`
- **Commit:** `691bb47` (fixed before the file was ever committed — not a separate commit)
## Verification Results
- `npx tsc --noEmit --pretty` — no new errors; the only 2 remaining errors (`sync-scheduler.ts:446,450`, missing `appgate-factory`/`appgate-sync-service` modules) are the same pre-existing, unrelated failures documented as Deferred in `12-01-SUMMARY.md`
- `grep -Fc 'similarity($1, company_name)' lib/services/pax8-company-matcher.ts` -> 2 (both `findCandidates` occurrences bind the PAX8 name as `$1`)
- `grep -c "company_name +"` -> 0 (no string concatenation into SQL)
- `npx vitest run lib/services/pax8-company-matcher.test.ts` -> 6/6 tests passed
- Manually confirmed `applyLink`'s UPDATE SQL contains both `match_method IS DISTINCT FROM 'manual'` and the `resolved_at IS NOT NULL` `NOT EXISTS` guard
- Manually confirmed `recordConflict`'s upsert targets `pax8_company_match_review` with `ON CONFLICT (pax8_company_id) WHERE resolved_at IS NULL`
## Deferred Issues
- **Pre-existing type-check failure, unrelated to this plan.** `lib/services/sync-scheduler.ts:446,450` references `@/lib/services/appgate-factory` and `@/lib/services/appgate-sync-service` via dynamic `import()`, neither of which exists at this worktree's commit. Already logged in `12-01-SUMMARY.md` and `.planning/phases/12-orders-invoices-company-matching/deferred-items.md`. Confirmed still present and still unrelated to this plan's two new files (both type-check clean in isolation).
## Known Stubs
None — this plan is a pure service + test module, no UI, no partial data wiring.
## Threat Flags
None — this plan's only new surface is the matcher module itself, and every threat register item from the plan's `<threat_model>` (T-12-01, T-12-02, T-12-04, T-12-06) is directly mitigated in the implementation (parameterized queries, conservative threshold/tie-margin with unit-tested branches, resolved-row idempotency guard, bounded scan). No new network endpoints, auth paths, or schema changes were introduced.
## Self-Check: PASSED

View file

@ -0,0 +1,163 @@
/**
* pax8-company-matcher.ts unit tests.
*
* postgresClient.query is mocked similarity() runs in real Postgres and is
* never exercised here; these tests only assert on the SQL string + bound
* params issued for each decision branch (auto-link / review / empty
* candidates / idempotency guard / dryRun), matching the mocking discipline
* of lib/services/pax8-client.test.ts adapted from fetch to postgresClient.query.
*/
import { describe, it, expect, vi, beforeEach } from 'vitest';
// Mock postgresClient BEFORE importing the module under test — the matcher
// imports the default export.
const queryMock = vi.fn();
vi.mock('@/lib/services/postgres-client', () => ({
default: {
query: (...args: unknown[]) => queryMock(...args),
},
}));
// Import AFTER the mock is declared so vi.mock hoisting takes effect.
import { matchPax8Companies } from './pax8-company-matcher';
interface MockCall {
sql: string;
params: unknown[];
}
function calls(): MockCall[] {
return queryMock.mock.calls.map(([sql, params]) => ({
sql: String(sql),
params: (params as unknown[]) ?? [],
}));
}
/** Eligibility SELECT response — a single pax8_companies row under test. */
function eligibleRow(id: string, name: string) {
return { rows: [{ id, name }], rowCount: 1 };
}
/** findCandidates() response — score rows shaped like the real query output. */
function candidateRows(rows: Array<{ id: string; score: number }>) {
return {
rows: rows.map((r) => ({ id: r.id, score: r.score.toFixed(3) })),
rowCount: rows.length,
};
}
describe('matchPax8Companies', () => {
beforeEach(() => {
queryMock.mockReset();
});
it('auto-link: single candidate scoring 0.95 issues an UPDATE against pax8_companies and no review INSERT', async () => {
queryMock
.mockResolvedValueOnce(eligibleRow('p1', 'Acme Inc'))
.mockResolvedValueOnce(candidateRows([{ id: '10', score: 0.95 }]));
const result = await matchPax8Companies({ limit: 10 });
expect(result.autoLinked).toBe(1);
expect(result.flaggedAmbiguous).toBe(0);
expect(result.flaggedNoCandidate).toBe(0);
const applyLinkCall = calls().find((c) => /UPDATE pax8_companies/.test(c.sql) && /autotask_company_id\s*=\s*\$2/.test(c.sql));
expect(applyLinkCall).toBeDefined();
expect(applyLinkCall!.params).toEqual(['p1', 10, '0.950']);
const reviewInsert = calls().find((c) => /INSERT INTO pax8_company_match_review/.test(c.sql));
expect(reviewInsert).toBeUndefined();
});
it('review (below threshold): best candidate 0.80 issues an INSERT/UPSERT into pax8_company_match_review and no auto-link UPDATE writing autotask_company_id', async () => {
queryMock
.mockResolvedValueOnce(eligibleRow('p2', 'Widgets LLC'))
.mockResolvedValueOnce(candidateRows([{ id: '20', score: 0.8 }]));
const result = await matchPax8Companies({ limit: 10 });
expect(result.flaggedAmbiguous).toBe(1);
expect(result.autoLinked).toBe(0);
const reviewInsert = calls().find((c) => /INSERT INTO pax8_company_match_review/.test(c.sql));
expect(reviewInsert).toBeDefined();
expect(reviewInsert!.params[0]).toBe('p2');
expect(reviewInsert!.params[1]).toEqual([20]);
const autoLinkUpdate = calls().find(
(c) => /UPDATE pax8_companies/.test(c.sql) && /autotask_company_id\s*=\s*\$2/.test(c.sql)
);
expect(autoLinkUpdate).toBeUndefined();
});
it('review (near-tie): candidates 0.95 and 0.92 force review even though the top score clears 0.90 (D-02)', async () => {
queryMock
.mockResolvedValueOnce(eligibleRow('p3', 'Contoso Corp'))
.mockResolvedValueOnce(
candidateRows([
{ id: '30', score: 0.95 },
{ id: '31', score: 0.92 },
])
);
const result = await matchPax8Companies({ limit: 10 });
expect(result.flaggedAmbiguous).toBe(1);
expect(result.autoLinked).toBe(0);
const reviewInsert = calls().find((c) => /INSERT INTO pax8_company_match_review/.test(c.sql));
expect(reviewInsert).toBeDefined();
expect(reviewInsert!.params[1]).toEqual([30, 31]);
const autoLinkUpdate = calls().find(
(c) => /UPDATE pax8_companies/.test(c.sql) && /autotask_company_id\s*=\s*\$2/.test(c.sql)
);
expect(autoLinkUpdate).toBeUndefined();
});
it('empty candidates: zero rows above the floor produce a review row with an empty candidate_company_ids array (D-03)', async () => {
queryMock
.mockResolvedValueOnce(eligibleRow('p4', 'No Match Co'))
.mockResolvedValueOnce(candidateRows([]));
const result = await matchPax8Companies({ limit: 10 });
expect(result.flaggedNoCandidate).toBe(1);
const reviewInsert = calls().find((c) => /INSERT INTO pax8_company_match_review/.test(c.sql));
expect(reviewInsert).toBeDefined();
expect(reviewInsert!.params[1]).toEqual([]);
expect(reviewInsert!.params[2]).toEqual([]);
});
it('idempotent (D-05/SC#4): the applyLink UPDATE includes the resolved_at guard and the manual-method guard; the eligibility SELECT excludes manually-resolved rows', async () => {
queryMock
.mockResolvedValueOnce(eligibleRow('p5', 'Idempotent Co'))
.mockResolvedValueOnce(candidateRows([{ id: '50', score: 0.99 }]));
await matchPax8Companies({ limit: 10 });
const eligibilitySelect = calls().find((c) => /FROM pax8_companies/.test(c.sql) && /SELECT/.test(c.sql));
expect(eligibilitySelect).toBeDefined();
expect(eligibilitySelect!.sql).toContain("match_method IS DISTINCT FROM 'manual'");
expect(eligibilitySelect!.sql).toContain('resolved_at IS NOT NULL');
const applyLinkCall = calls().find((c) => /UPDATE pax8_companies/.test(c.sql) && /autotask_company_id\s*=\s*\$2/.test(c.sql));
expect(applyLinkCall).toBeDefined();
expect(applyLinkCall!.sql).toContain('resolved_at IS NOT NULL');
expect(applyLinkCall!.sql).toContain("IS DISTINCT FROM 'manual'");
});
it('dryRun: no write queries (UPDATE/INSERT/DELETE) are issued when dryRun is true', async () => {
queryMock
.mockResolvedValueOnce(eligibleRow('p6', 'DryRun Co'))
.mockResolvedValueOnce(candidateRows([{ id: '60', score: 0.99 }]));
await matchPax8Companies({ limit: 10, dryRun: true });
const writeCalls = calls().filter((c) => /^\s*(UPDATE|INSERT|DELETE)/i.test(c.sql));
expect(writeCalls).toHaveLength(0);
});
});

View file

@ -0,0 +1,252 @@
/**
* PAX8 company <-> Autotask company fuzzy matcher (PAX8-10, PAX8-11).
*
* Ported from lib/services/device-link-reconciler.ts's cascading
* findBy-x / pickBestCandidate / applyLink / recordConflict shape, adapted
* from exact-match cascades to a single trigram-similarity score.
*
* Unlike the reconciler, this does NOT run as a standalone cron this phase
* it is invoked from Pax8SyncService.fullSync() (Plan 04), once per sync run,
* against the eligible (unresolved) subset of pax8_companies.
*
* Match policy (12-CONTEXT.md / 12-RESEARCH.md locked decisions):
* D-01: auto-link only at similarity >= AUTO_LINK_THRESHOLD, single
* candidate, no near-tie.
* D-02: a second candidate within TIE_MARGIN of the top score forces
* review even at score 1.0 never silently tie-break.
* D-03: zero candidates above CANDIDATE_FLOOR still creates a review row
* with an empty candidate_company_ids array never silently dropped.
* D-04: ambiguous review rows carry the top 3 candidates.
* D-05 / SC#4: only unresolved rows are (re)scored every sync; a
* human-resolved match (resolved review row, or match_method='manual')
* is never overwritten.
*/
import postgresClient from '@/lib/services/postgres-client';
// D-01: conservative, initial/tunable auto-link floor. Validated live against
// this project's real 118 pax8_companies vs 242 active companies — every
// genuine match scored 1.00, the highest non-match observed was 0.70, so 0.90
// sits comfortably in the empty gap between the two clusters.
export const AUTO_LINK_THRESHOLD = 0.9;
// D-02: a second candidate within this margin of the top score is treated as
// an unresolved tie and forced to review, even if the top score alone would
// clear AUTO_LINK_THRESHOLD.
export const TIE_MARGIN = 0.05;
// Floor below which a candidate isn't even considered — keeps the review
// candidate list small and relevant.
export const CANDIDATE_FLOOR = 0.3;
interface CompanyCandidate {
autotask_company_id: number;
score: number; // pg_trgm similarity(), 0.0-1.0
}
interface EligibleCompany {
id: string;
name: string;
}
export interface Pax8CompanyMatchResult {
scanned: number;
autoLinked: number;
flaggedAmbiguous: number;
flaggedNoCandidate: number;
durationMs: number;
}
type Decision =
| { kind: 'auto'; match: CompanyCandidate }
| { kind: 'review'; top3: CompanyCandidate[] };
/**
* Fuzzy candidate lookup against active Autotask companies. The PAX8 name is
* always bound as $1 never string-interpolated (Security Domain / T-12-01).
* pg_trgm's similarity() is already case-insensitive (Pitfall 4) so no
* LOWER() wrapping is added; TRIM() is applied to the input purely for
* cleaner review-table display, not for scoring purposes.
*/
async function findCandidates(pax8Name: string): Promise<CompanyCandidate[]> {
const res = await postgresClient.query<{ id: string; score: string }>(
`SELECT id::text, similarity($1, company_name)::text AS score
FROM companies
WHERE is_active = true
AND similarity($1, company_name) > ${CANDIDATE_FLOOR}
ORDER BY score DESC
LIMIT 5`,
[pax8Name]
);
return res.rows.map((r) => ({
autotask_company_id: Number(r.id),
score: Number(r.score),
}));
}
/**
* D-01..D-04 decision logic: exactly one auto-link path (unambiguous,
* high-confidence), everything else routed to review.
*/
function decide(candidates: CompanyCandidate[]): Decision {
if (candidates.length === 0) return { kind: 'review', top3: [] }; // D-03
const [best, second] = candidates;
const tie = second !== undefined && best.score - second.score < TIE_MARGIN;
if (best.score >= AUTO_LINK_THRESHOLD && !tie) {
return { kind: 'auto', match: best };
}
return { kind: 'review', top3: candidates.slice(0, 3) }; // D-02/D-04
}
/**
* Auto-link a PAX8 company to its matched Autotask company. Guarded so a
* human-resolved match is never overwritten (D-05 / SC#4): the UPDATE only
* fires when match_method isn't already 'manual' AND there is no review row
* for this company that has already been human-resolved.
*
* Any open (never-human-touched) review row for this company is closed
* a now-confident automated match supersedes a stale open flag.
*/
async function applyLink(
pax8CompanyId: string,
autotaskCompanyId: number,
score: number
): Promise<void> {
await postgresClient.query(
`UPDATE pax8_companies
SET autotask_company_id = $2,
match_confidence = $3,
match_method = 'pg_trgm',
matched_at = NOW()
WHERE id = $1
AND match_method IS DISTINCT FROM 'manual'
AND NOT EXISTS (
SELECT 1 FROM pax8_company_match_review r
WHERE r.pax8_company_id = pax8_companies.id
AND r.resolved_at IS NOT NULL
)`,
[pax8CompanyId, autotaskCompanyId, score.toFixed(3)]
);
await postgresClient.query(
`DELETE FROM pax8_company_match_review
WHERE pax8_company_id = $1
AND resolved_at IS NULL`,
[pax8CompanyId]
);
}
/**
* Flag a PAX8 company for manual review, carrying its top-3 candidates
* (D-04) or an empty array when there were none above CANDIDATE_FLOOR
* (D-03). Upserts against the "one open review per company" partial unique
* index (uq_pax8_company_match_review_open).
*
* If this company previously held a confident pg_trgm auto-match that is now
* ambiguous or below-threshold, the stale match columns are cleared
* conservatively but a 'manual' match is never cleared this way.
*/
async function recordConflict(
pax8CompanyId: string,
top3: CompanyCandidate[]
): Promise<void> {
await postgresClient.query(
`UPDATE pax8_companies
SET autotask_company_id = NULL,
match_confidence = NULL,
match_method = NULL,
matched_at = NULL
WHERE id = $1
AND match_method = 'pg_trgm'`,
[pax8CompanyId]
);
const candidateIds = top3.map((c) => c.autotask_company_id);
const confidences = top3.map((c) => c.score.toFixed(3));
await postgresClient.query(
`INSERT INTO pax8_company_match_review
(pax8_company_id, candidate_company_ids, match_confidences)
VALUES ($1, $2::bigint[], $3::text[])
ON CONFLICT (pax8_company_id) WHERE resolved_at IS NULL
DO UPDATE SET candidate_company_ids = EXCLUDED.candidate_company_ids,
match_confidences = EXCLUDED.match_confidences,
detected_at = NOW()`,
[pax8CompanyId, candidateIds, confidences]
);
}
/**
* Run one matching pass over the re-scoring-eligible subset of
* pax8_companies (D-05: excludes manually-resolved rows and rows with a
* human-resolved review). Idempotent safe to run on every sync.
*/
export async function matchPax8Companies(opts?: {
limit?: number;
dryRun?: boolean;
}): Promise<Pax8CompanyMatchResult> {
const startedAt = Date.now();
const limit = opts?.limit ?? 1000;
const dryRun = opts?.dryRun ?? false;
const result: Pax8CompanyMatchResult = {
scanned: 0,
autoLinked: 0,
flaggedAmbiguous: 0,
flaggedNoCandidate: 0,
durationMs: 0,
};
try {
const eligible = await postgresClient.query<EligibleCompany>(
`SELECT id::text, name
FROM pax8_companies
WHERE is_deleted = false
AND match_method IS DISTINCT FROM 'manual'
AND NOT EXISTS (
SELECT 1 FROM pax8_company_match_review r
WHERE r.pax8_company_id = pax8_companies.id
AND r.resolved_at IS NOT NULL
)
ORDER BY name
LIMIT $1`,
[limit]
);
for (const company of eligible.rows) {
result.scanned += 1;
const candidates = await findCandidates(company.name.trim());
const decision = decide(candidates);
if (decision.kind === 'auto') {
if (!dryRun) {
await applyLink(company.id, decision.match.autotask_company_id, decision.match.score);
}
result.autoLinked += 1;
} else if (decision.top3.length === 0) {
if (!dryRun) {
await recordConflict(company.id, []);
}
result.flaggedNoCandidate += 1;
} else {
if (!dryRun) {
await recordConflict(company.id, decision.top3);
}
result.flaggedAmbiguous += 1;
}
}
result.durationMs = Date.now() - startedAt;
console.log(
`[Pax8Match] scanned=${result.scanned} autoLinked=${result.autoLinked} ` +
`flaggedAmbiguous=${result.flaggedAmbiguous} flaggedNoCandidate=${result.flaggedNoCandidate} ` +
`durationMs=${result.durationMs}`
);
return result;
} catch (err) {
const msg = err instanceof Error ? err.message : String(err);
console.error('[Pax8Match] failed:', msg);
result.durationMs = Date.now() - startedAt;
throw err;
}
}