Add comprehensive admin features and multi-system integration

- Add admin dashboard with sync controls and data browser - Implement RMM, Auvik, and Addigy organization mappings - Add chunked ticket sync with progress tracking - Implement entity sync service with rate limiting - Add analytics engine and performance optimizer - Create data browser for all PSA entities - Add navigation components and UI improvements - Implement background processing and sync services - Add comprehensive documentation and migration scripts - Update configuration items with multi-system support - Enhance contact management and purchase history - Add issue type assignment and LLM analyzer - Improve error handling and logging utilities
2025-11-19 14:18:16 -05:00 · 2025-11-19 14:18:16 -05:00 · 6eee14f8af
commit 6eee14f8af
parent e8462ef301
171 changed files with 32671 additions and 621 deletions
--- a/docs/fixes/all-systems-inventory-display.md
+++ b/docs/fixes/all-systems-inventory-display.md
@ -0,0 +1,163 @@
+# All Systems Inventory Display Implementation
+
+## Issue
+The configuration items page was only showing devices from PSA (Autotask) and RMM (Datto RMM), but not displaying devices that exist ONLY in NMS (Auvik) or ARMM (Addigy). This violated the goals document requirement to show "the combination of inventories across all the systems."
+
+## Root Cause
+The comparison logic in `/app/api/rmm-devices/route.ts` was:
+1. Adding RMM devices (matched or RMM-only)
+2. Adding Autotask devices (matched or Autotask-only)
+3. Matching Auvik and Addigy devices to existing Autotask devices
+4. **BUT never adding unmatched Auvik-only or Addigy-only devices**
+
+## Solution
+
+### Backend Changes (`/app/api/rmm-devices/route.ts`)
+
+#### 1. Added Unmatched Device Logic
+After processing PSA and RMM devices, added logic to include devices that exist only in Auvik or Addigy:
+
+```typescript
+// Track which Auvik and Addigy devices have been matched
+const matchedAuvikIds = new Set<string>();
+const matchedAddigyIds = new Set<string>();
+
+comparison.forEach(item => {
+  if (item.auvikDevice?.id) matchedAuvikIds.add(item.auvikDevice.id);
+  if (item.addigyDevice?.agentid) matchedAddigyIds.add(item.addigyDevice.agentid);
+});
+
+// Add unmatched Auvik devices (NMS-only)
+for (const auvikDevice of auvikDevices) {
+  if (!matchedAuvikIds.has(auvikDevice.id)) {
+    // Try to match with Addigy by serial number
+    let addigyMatch = addigyDevices.find(d => 
+      d['Serial Number']?.toLowerCase().trim() === auvikDevice.serialNumber?.toLowerCase().trim() &&
+      !matchedAddigyIds.has(d.agentid)
+    );
+    
+    comparison.push({
+      auvikDevice: auvikDevice,
+      addigyDevice: addigyMatch,
+      status: 'rmm-only'
+    });
+  }
+}
+
+// Add unmatched Addigy devices (ARMM-only)
+for (const addigyDevice of addigyDevices) {
+  if (!matchedAddigyIds.has(addigyDevice.agentid)) {
+    comparison.push({
+      addigyDevice: addigyDevice,
+      status: 'rmm-only'
+    });
+  }
+}
+```
+
+#### 2. Updated Sorting Logic
+Enhanced device name sorting to check all four systems:
+
+```typescript
+const aName = a.autotaskDevice?.referenceTitle || 
+              a.rmmDevice?.hostname || 
+              a.auvikDevice?.deviceName || 
+              a.addigyDevice?.['Device Name'] || 
+              '';
+```
+
+#### 3. Added Stats for All Systems
+Updated response to include counts for all four systems:
+
+```typescript
+stats: {
+  totalRmm: rmmDevices.length,
+  totalAutotask: autotaskDevices.length,
+  totalAuvik: auvikDevices.length,
+  totalAddigy: addigyDevices.length,
+  matched: comparison.filter(c => c.status === 'matched').length,
+  autotaskOnly: comparison.filter(c => c.status === 'autotask-only').length,
+  rmmOnly: comparison.filter(c => c.status === 'rmm-only').length,
+}
+```
+
+### Frontend Changes (`/app/configuration-items/page.tsx`)
+
+#### 1. Updated Device Name Display
+Modified table cells to show device names from any of the four systems:
+
+```typescript
+{item.autotaskDevice?.referenceTitle || 
+ item.rmmDevice?.hostname || 
+ item.auvikDevice?.deviceName ||
+ item.addigyDevice?.['Device Name'] ||
+ 'Unknown Device'}
+```
+
+#### 2. Updated Serial Number Display
+```typescript
+{item.autotaskDevice?.serialNumber || 
+ item.rmmDevice?.serialNumber || 
+ item.auvikDevice?.serialNumber ||
+ item.addigyDevice?.['Serial Number'] ||
+ '-'}
+```
+
+#### 3. Updated IP Address Display
+```typescript
+{item.autotaskDevice?.rmmDeviceAuditIPAddress || 
+ item.rmmDevice?.intIpAddress || 
+ item.auvikDevice?.ipAddresses?.[0] ||
+ item.addigyDevice?.['IP Address'] ||
+ '-'}
+```
+
+#### 4. Updated Stats Display
+Both the compact stats badge and detailed table header now show all four systems:
+
+**Compact:** `PSA: X | RMM: X | NMS: X | ARMM: X`
+
+**Detailed:** Individual counts for PSA, RMM, NMS, ARMM, plus Matched count
+
+## Matching Priority (Per Goals Document)
+
+The system now properly implements the matching priority across all systems:
+
+1. **Reference UID** - PSA has reference to RMM device UID
+2. **Serial Number** - Primary matching across all systems
+3. **MAC Address** - Used for Auvik matching
+4. **IP Address** - Used for RMM matching
+5. **Hostname** - Used with client/site filter
+
+## Device Status Types
+
+- **matched** - Device exists in PSA and at least one other system
+- **autotask-only** - Device exists only in PSA (may have NMS/ARMM matches)
+- **rmm-only** - Device exists in RMM, NMS, or ARMM but not in PSA
+
+## Benefits
+
+1. **Complete Visibility** - All devices from all four systems are now visible
+2. **Better Asset Tracking** - No devices are hidden from view
+3. **Compliance with Goals** - Fully implements the "combination of inventories" requirement
+4. **Cross-System Matching** - Auvik and Addigy devices can match each other even without PSA entry
+5. **Accurate Counts** - Stats show true device counts across all systems
+
+## Files Modified
+
+1. `/app/api/rmm-devices/route.ts` - Backend comparison logic
+2. `/app/configuration-items/page.tsx` - Frontend display logic
+
+## Testing Recommendations
+
+1. Select a company with devices in all four systems
+2. Verify devices appear from PSA, RMM, NMS, and ARMM
+3. Check that stats show correct counts for each system
+4. Verify devices that exist only in Auvik or Addigy are displayed
+5. Confirm matching works across all system combinations
+
+## Related Documentation
+
+- `/docs/configuration-item-goals.md` - Original requirements
+- `/docs/fixes/configuration-items-table-improvements.md` - UI improvements
+- `/docs/fixes/rmm-cache-invalidation-fix.md` - Cache fix
--- a/docs/fixes/auvik-device-filtering.md
+++ b/docs/fixes/auvik-device-filtering.md
@ -0,0 +1,77 @@
+# Auvik (NMS) Device Filtering
+
+## Purpose
+Filter out invalid or placeholder Auvik devices that don't represent actual network equipment with meaningful hostnames.
+
+## Filtering Rules
+
+Auvik devices are excluded from the configuration items display if they meet any of these criteria:
+
+1. **No Device Name**: Devices without a `deviceName` field are excluded
+2. **"Device@" Prefix**: Devices with names starting with "Device@" are excluded
+
+## Implementation
+
+Location: `/app/api/rmm-devices/route.ts`
+
+```typescript
+// Filter Auvik devices to only show those with valid hostnames
+// Exclude devices without deviceName or with names starting with "Device@"
+auvikDevices = auvikDevices.filter(device => {
+  if (!device.deviceName) {
+    return false;
+  }
+  if (device.deviceName.startsWith('Device@')) {
+    return false;
+  }
+  return true;
+});
+```
+
+## Rationale
+
+### "Device@" Prefix
+Auvik uses the "Device@" prefix for devices that it has discovered but hasn't been able to identify with a proper hostname. These are typically:
+- Devices without SNMP configured
+- Devices that don't respond to hostname queries
+- Placeholder entries for IP addresses without proper DNS records
+
+These devices provide little value in the configuration items view since they can't be meaningfully matched to PSA records or other systems.
+
+### Missing Device Name
+Devices without any name at all are incomplete records and should not be displayed.
+
+## Logging
+
+The filtering includes logging to track how many devices are filtered:
+
+```
+Fetched X Auvik devices before filtering
+Filtered to Y Auvik devices with valid hostnames
+```
+
+This helps administrators understand how many devices are being excluded and troubleshoot if legitimate devices are being filtered out.
+
+## Examples
+
+### Excluded Devices:
+- `Device@192.168.1.1`
+- `Device@10.0.0.50`
+- `null` or `undefined` deviceName
+
+### Included Devices:
+- `SWITCH-CORE-01`
+- `FW-MAIN`
+- `AP-OFFICE-2`
+- `srv-dc01.domain.com`
+
+## Impact
+
+This filtering ensures that the configuration items page only shows:
+- Real, identifiable network devices
+- Devices that can be meaningfully matched across systems
+- Devices that provide value to executives/management viewing the inventory
+
+## Related Documentation
+- `/docs/configuration-item-goals.md` - Overall goals for the configuration items module
+- `/docs/fixes/all-systems-inventory-display.md` - Multi-system inventory implementation
--- a/docs/fixes/complete-cache-invalidation.md
+++ b/docs/fixes/complete-cache-invalidation.md
@ -0,0 +1,109 @@
+# Complete Cache Invalidation for All System Mappings
+
+## Issue
+Company 29683395 (TK Plastics) was not showing Addigy (ARMM) devices even though:
+- 3 Addigy org mappings existed in the database
+- The API code was correct to fetch Addigy devices
+- The cache was returning stale data from before the mappings were added
+
+## Root Cause
+The cache key only included RMM mapping count:
+```
+rmm-devices:29683395:active:mappings-2
+```
+
+When Addigy or Auvik mappings were added, the cache key didn't change, so the system continued serving cached data with 0 Addigy/Auvik devices.
+
+## Database Verification
+
+### Company 29683395 Mappings:
+- **RMM Sites**: 2 mappings
+  - TK Plastics (612f6b1f-9228-4e2a-8e63-f0ec5d0b6aaa)
+  - TK Plastics - Kaercher (aba39a7b-d9b3-4eff-88e4-4d0182d478cb)
+
+- **Auvik Tenants**: 1 mapping
+  - tkplastics (1228237269652810493)
+
+- **Addigy Orgs**: 3 mappings
+  - TK Plastics (b194ffe7-c354-4cb0-a5e5-c99876729b4b)
+  - TK - iPads (393f67b0-5449-41b4-a5f2-f72de849c5d5)
+  - TK - Macs (36f89454-6dbf-4c03-8843-3dfc34b9534f)
+
+## Solution
+
+Updated the cache key to include mapping counts for ALL three external systems:
+
+### Before:
+```typescript
+const cacheKey = `rmm-devices:${companyId}:${activeFilter}:mappings-${mappingCount}`;
+```
+
+### After:
+```typescript
+const cacheKey = `rmm-devices:${companyId}:${activeFilter}:rmm-${rmmMappingCount}:auvik-${auvikMappingCount}:addigy-${addigyMappingCount}`;
+```
+
+### Implementation:
+```typescript
+// Get mapping counts to include in cache key (so cache invalidates when mappings change)
+let rmmMappingCount = 0;
+let auvikMappingCount = 0;
+let addigyMappingCount = 0;
+
+if (companyId) {
+  // RMM site mappings
+  const rmmMappingsResult = await pool.query(
+    'SELECT COUNT(*) as count FROM rmm_site_mappings WHERE company_id = $1',
+    [parseInt(companyId)]
+  );
+  rmmMappingCount = parseInt(rmmMappingsResult.rows[0]?.count || '0');
+  
+  // Auvik tenant mappings
+  const auvikMappingsResult = await pool.query(
+    'SELECT COUNT(*) as count FROM auvik_tenant_mappings WHERE autotask_company_id = $1',
+    [parseInt(companyId)]
+  );
+  auvikMappingCount = parseInt(auvikMappingsResult.rows[0]?.count || '0');
+  
+  // Addigy org mappings
+  const addigyMappingsResult = await pool.query(
+    'SELECT COUNT(*) as count FROM addigy_org_mappings WHERE autotask_company_id = $1',
+    [parseInt(companyId)]
+  );
+  addigyMappingCount = parseInt(addigyMappingsResult.rows[0]?.count || '0');
+}
+```
+
+## Cache Key Examples
+
+### For Company 29683395:
+- **Old key**: `rmm-devices:29683395:active:mappings-2`
+- **New key**: `rmm-devices:29683395:active:rmm-2:auvik-1:addigy-3`
+
+### Cache Invalidation Scenarios:
+1. **Add RMM site mapping**: `rmm-2` → `rmm-3` (cache invalidated)
+2. **Add Auvik tenant mapping**: `auvik-1` → `auvik-2` (cache invalidated)
+3. **Add Addigy org mapping**: `addigy-3` → `addigy-4` (cache invalidated)
+4. **Remove any mapping**: Count decreases, cache invalidated
+
+## Benefits
+
+1. **Automatic Cache Invalidation**: Cache automatically expires when ANY system mapping changes
+2. **No Manual Cache Clearing**: No need to manually clear cache or use skipCache parameter
+3. **Accurate Data**: Users always see current device data after mapping changes
+4. **System Consistency**: All three external systems (RMM, NMS, ARMM) are treated equally
+
+## Files Modified
+- `/app/api/rmm-devices/route.ts` - Updated cache key generation logic
+
+## Testing
+After deploying:
+1. The cache key for company 29683395 will change from `rmm-devices:29683395:active:mappings-2` to `rmm-devices:29683395:active:rmm-2:auvik-1:addigy-3`
+2. This will trigger a fresh API call
+3. Addigy devices from all 3 mapped organizations will be fetched and displayed
+4. Future mapping changes will automatically invalidate the cache
+
+## Related Documentation
+- `/docs/fixes/rmm-cache-invalidation-fix.md` - Initial RMM cache fix
+- `/docs/fixes/all-systems-inventory-display.md` - All systems display implementation
+- `/docs/rmm-multi-site-integration.md` - RMM multi-site architecture
--- a/docs/fixes/configuration-items-table-improvements.md
+++ b/docs/fixes/configuration-items-table-improvements.md
@ -0,0 +1,56 @@
+# Configuration Items Table UI Improvements
+
+## Changes Made
+
+### 1. Removed "Match Type" Column
+- Removed the column that displayed how devices were matched (e.g., "RMM UID", "Serial Number", "Hostname", "IP Address")
+- This information was deemed not useful for the primary use case
+
+### 2. Removed ChevronRight (">") Action Column
+- Removed the dedicated button column with the ">" icon
+- Made entire table rows clickable instead to open the device detail modal
+- Improved UX by making the click target larger and more intuitive
+- Preserved ChevronRight icons for collapsible sections (filters and contact groups)
+
+### 3. Made Serial Number Column Narrower
+- Changed from full-width to fixed width: `className="w-32"`
+- Applied to both header and body cells
+- Saves horizontal space for other columns
+
+### 4. Renamed "Auvik" to "NMS"
+- Changed column header from "Auvik" to "NMS" (Network Management System)
+- Aligns with the generic naming convention mentioned in configuration-item-goals.md
+- Makes the interface more vendor-agnostic
+
+### 5. Added "ARMM" Column
+- Added new column for Addigy device matches
+- ARMM = Apple Remote Management & Monitoring
+- Shows green checkmark when Addigy device is matched
+- Shows gray X when no Addigy device is matched
+- Positioned after NMS column
+
+## Table Column Order (Final)
+1. Checkbox (selection)
+2. Status (Matched/AT Only/RMM Only)
+3. Device Name (sortable)
+4. Serial Number (narrower, w-32)
+5. IP Address (sortable)
+6. Contact (sortable, with group toggle)
+7. Type
+8. PSA (checkmark/X)
+9. RMM (checkmark/X)
+10. NMS (checkmark/X) - formerly "Auvik"
+11. ARMM (checkmark/X) - **NEW**
+
+## Files Modified
+- `/app/configuration-items/page.tsx` - Updated table headers and body cells
+
+## User Experience Improvements
+- **Cleaner interface**: Removed unnecessary columns
+- **Better space utilization**: Serial number column is narrower
+- **Improved clickability**: Entire row is now clickable to view details
+- **Complete system visibility**: Now shows all 4 systems (PSA, RMM, NMS, ARMM)
+- **Consistent naming**: Uses generic system names (NMS, ARMM) instead of vendor names
+
+## Related Documentation
+- `/docs/configuration-item-goals.md` - Original requirements for multi-system inventory display
--- a/docs/fixes/rmm-cache-invalidation-fix.md
+++ b/docs/fixes/rmm-cache-invalidation-fix.md
@ -0,0 +1,60 @@
+# RMM Cache Invalidation Fix
+
+## Issue
+Company 29683395 (TK Plastics Company, Inc.) was showing 0 RMM devices on the configuration items page despite having 2 RMM site mappings configured in the database.
+
+## Root Cause
+The `/api/rmm-devices` endpoint was using a cache key that didn't account for changes in RMM site mappings:
+
+**Old cache key format:** `rmm-devices:${companyId}:${activeFilter}`
+
+This meant that when RMM site mappings were added for a company, the cache would continue serving the old data (with 0 RMM devices) until the cache expired (2 minutes).
+
+## Database Verification
+```sql
+-- Company has 2 RMM site mappings
+SELECT * FROM rmm_site_mappings WHERE company_id = 29683395;
+-- Results:
+-- 612f6b1f-9228-4e2a-8e63-f0ec5d0b6aaa | TK Plastics
+-- aba39a7b-d9b3-4eff-88e4-4d0182d478cb | TK Plastics - Kaercher
+```
+
+## Solution
+
+### 1. Updated Cache Key to Include Mapping Count
+Modified `/app/api/rmm-devices/route.ts` to include the number of site mappings in the cache key:
+
+**New cache key format:** `rmm-devices:${companyId}:${activeFilter}:mappings-${mappingCount}`
+
+This ensures that when mappings are added or removed, the cache is automatically invalidated because the key changes.
+
+### 2. Added Force Refresh Capability
+Added a `skipCache` query parameter to allow bypassing the cache entirely when needed:
+- Added `forceRefresh` state to the configuration items page
+- Updated the refresh button to increment `forceRefresh` counter
+- When `forceRefresh > 0`, adds `&skipCache=true` to the API request
+- Made the refresh button show a spinner animation while loading
+
+### 3. Improved Refresh Button UX
+- Refresh button now shows spinning animation while loading
+- Button is disabled during loading to prevent multiple simultaneous requests
+- Button is disabled when no company is selected
+
+## Files Modified
+1. `/app/api/rmm-devices/route.ts` - Updated cache key logic and added skipCache parameter
+2. `/app/configuration-items/page.tsx` - Added force refresh capability and improved refresh button
+
+## Testing
+After deploying these changes:
+1. The cache key will automatically change from `rmm-devices:29683395:active:mappings-0` to `rmm-devices:29683395:active:mappings-2`
+2. This will force a fresh API call that will fetch devices from both mapped RMM sites
+3. Users can also click the refresh button to force bypass the cache
+
+## Expected Behavior
+- Company 29683395 should now show RMM devices from both "TK Plastics" and "TK Plastics - Kaercher" sites
+- Any future changes to RMM site mappings will automatically invalidate the cache
+- Users can manually force a refresh by clicking the refresh button
+
+## Related Documentation
+- `/docs/rmm-multi-site-integration.md` - RMM multi-site support architecture
+- System-retrieved memory about RMM site mappings implementation