lorentz/wulf-pulse
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases Packages Wiki Activity Actions

Add comprehensive admin features and multi-system integration

Browse source 
- Add admin dashboard with sync controls and data browser
- Implement RMM, Auvik, and Addigy organization mappings
- Add chunked ticket sync with progress tracking
- Implement entity sync service with rate limiting
- Add analytics engine and performance optimizer
- Create data browser for all PSA entities
- Add navigation components and UI improvements
- Implement background processing and sync services
- Add comprehensive documentation and migration scripts
- Update configuration items with multi-system support
- Enhance contact management and purchase history
- Add issue type assignment and LLM analyzer
- Improve error handling and logging utilities
This commit is contained in:
root 2025-11-19 14:18:16 -05:00
parent e8462ef301
commit 6eee14f8af
171 changed files with 32671 additions and 621 deletions
Download patch file Download diff file Expand all files Collapse all files

265
tasks/prd-auvik-integration.md Normal file
View file

@ -0,0 +1,265 @@
# PRD: Auvik Network Device Integration
## Introduction/Overview
This feature adds Auvik as a third data source for configuration items, complementing existing Autotask (PSA) and Datto RMM integrations. Auvik specializes in network device monitoring and will provide detailed information about switches, routers, firewalls, and other network infrastructure. The integration will match Auvik devices against existing Autotask configuration items using serial numbers, hostnames, and MAC addresses, displaying the data in a new "Auvik" tab within the configuration item modal.
**Problem Statement:** Network administrators and MSP technicians currently lack visibility into network device details (firmware versions, network interfaces, uptime, etc.) when viewing configuration items. This requires switching between multiple tools to get a complete picture of network infrastructure.
**Goal:** Provide seamless access to Auvik network device data within the existing configuration items interface, enabling users to view comprehensive device information from PSA, RMM, and network monitoring systems in one place.
## Goals
1. **Primary Goal:** Integrate Auvik API to fetch and display network device data for configuration items
2. **Matching Goal:** Achieve high match rates (target: 90%+) for network devices using serial number, hostname, and MAC address matching
3. **UX Goal:** Provide a consistent, intuitive interface that follows the existing PSA/RMM tab pattern
4. **Performance Goal:** Fetch Auvik data on-demand without impacting page load times
5. **Reliability Goal:** Gracefully handle Auvik API failures without breaking existing functionality
## User Stories
1. **As a network administrator**, I want to see Auvik device details (firmware version, uptime, interfaces) when viewing a switch in the configuration items page, so that I don't have to switch to the Auvik dashboard.
2. **As an MSP technician**, I want to quickly identify which configuration items have matching Auvik devices, so that I can verify network monitoring coverage.
3. **As a system administrator**, I want the application to automatically match Auvik devices to configuration items using serial numbers and hostnames, so that I don't have to manually correlate devices across systems.
4. **As a user**, I want the configuration items page to continue working even if Auvik is unavailable, so that temporary API issues don't block my work.
5. **As a multi-tenant MSP**, I want Auvik devices to be correctly associated with the right customer/company, so that I see relevant data for each client.
## Functional Requirements
### FR1: Auvik API Client
1.1. Create an Auvik API client service (`/lib/services/auvik-client.ts`) that handles authentication and API requests
1.2. Use credentials from environment variables: `AUVIK_API_URL`, `AUVIK_API_KEY`, `AUVIK_API_USER`
1.3. Implement Basic Authentication using API user and key
1.4. Support fetching device inventory with filtering by tenant
1.5. Include proper error handling and logging for all API calls
1.6. Implement rate limiting to respect Auvik API limits
### FR2: Device Matching Logic
2.1. Match Auvik devices to Autotask configuration items using the following priority order:
- First: Serial number (exact match, case-insensitive)
- Second: Hostname (exact match, case-insensitive)
- Third: MAC address (exact match, normalized format)
2.2. Return only the first match found (no multiple matches per device)
2.3. Log matching results for debugging purposes
2.4. Handle cases where Auvik devices have multiple MAC addresses (match any)
### FR3: Multi-Tenant Support
3.1. Fetch Auvik tenant list from API
3.2. Map Auvik tenants to Autotask companies using tenant name matching
3.3. Filter Auvik device queries by tenant when viewing a specific company's devices
3.4. Handle cases where tenant mapping cannot be determined (show all devices)
### FR4: API Endpoint
4.1. Create endpoint `/api/auvik/devices` that accepts query parameters:
- `companyId`: Autotask company ID (optional)
- `companyName`: Company name for tenant matching (optional)
4.2. Return array of Auvik devices filtered by tenant if company info provided
4.3. Include device details: name, serial number, IP addresses, MAC addresses, device type, firmware version, manufacturer, model, online status, last seen timestamp
4.4. Return empty array (not error) if Auvik API is unavailable
4.5. Log errors to console but return 200 status with empty data
### FR5: Configuration Item Modal - Auvik Tab
5.1. Add "Auvik" tab to the configuration item modal (`/components/configuration-items/config-item-modal.tsx`)
5.2. Create new component `/components/configuration-items/auvik-tab.tsx` following the pattern of `rmm-tab.tsx`
5.3. Display Auvik device information in organized sections:
- **Basic Information:** Device name, device type, serial number, manufacturer, model
- **Network Information:** IP addresses, MAC addresses, subnet, VLAN
- **Status Information:** Online/offline status, last seen, uptime
- **Firmware Information:** Firmware version, last updated
- **Network Interfaces:** List of interfaces with status and speed
5.4. Show "No Auvik data available" message when no matching device is found
5.5. Display Auvik online/offline status badge in tab header
### FR6: Configuration Items Page - Auvik Indicator
6.1. Add "Auvik" column to the configuration items table (after RMM column)
6.2. Display green checkmark icon when Auvik device is matched
6.3. Display gray X icon when no Auvik match exists
6.4. Update table column count and responsive layout accordingly
### FR7: Configuration Item Detail API Enhancement
7.1. Modify `/api/configuration-items/[id]/route.ts` to fetch matching Auvik device
7.2. Use the same matching logic as FR2 (serial number → hostname → MAC address)
7.3. Include Auvik device data in API response: `{ autotaskDevice, rmmDevice, auvikDevice, companyName }`
7.4. Handle Auvik API failures gracefully (return null for auvikDevice)
### FR8: TypeScript Types
8.1. Create Auvik type definitions in `/lib/types/auvik.ts`:
- `AuvikDevice` interface with all device properties
- `AuvikTenant` interface for tenant information
- `AuvikNetworkInterface` interface for network interface details
8.2. Export types for use across the application
### FR9: Error Handling & Logging
9.1. Log all Auvik API errors to console with descriptive messages
9.2. Log successful device matches with match method (serial/hostname/MAC)
9.3. Log tenant mapping results
9.4. Never throw errors that would break the configuration items page
9.5. Display user-friendly error messages in Auvik tab if data fetch fails
### FR10: Real-Time Data Fetching
10.1. Fetch Auvik data on-demand when configuration items page loads
10.2. Fetch Auvik device details when configuration item modal opens
10.3. Do not cache Auvik data (always fetch fresh data)
10.4. Implement loading states while fetching Auvik data
## Non-Goals (Out of Scope)
1. **Database Sync:** Auvik data will NOT be synced to the PostgreSQL database (real-time only)
2. **Auvik Alerts:** Will not display Auvik alerts or notifications
3. **Auvik Configuration:** Will not allow modifying Auvik device settings from the application
4. **Network Topology:** Will not display Auvik network topology maps
5. **Historical Data:** Will not show historical performance metrics or trends
6. **Bulk Operations:** Will not support bulk actions on Auvik devices
7. **Auvik-Only View:** Will not create a dedicated page for viewing only Auvik devices
8. **Custom Field Mapping:** Will not support custom field mapping between Auvik and Autotask
## Design Considerations
### UI Components
- Follow existing design patterns from PSA and RMM tabs
- Use same Card, Badge, Label components from shadcn/ui
- Maintain consistent spacing, typography, and color scheme
- Use Lucide icons for network-related visuals (Network, Wifi, Router, etc.)
### Tab Layout
```
┌─────────────────────────────────────────────┐
│ PSA Data │ RMM Data │ Auvik Data │
└─────────────────────────────────────────────┘
│ │
│ ┌─────────────────┐ ┌──────────────────┐ │
│ │ Basic Info │ │ Network Info │ │
│ │ - Name │ │ - IP Addresses │ │
│ │ - Type │ │ - MAC Addresses │ │
│ │ - Serial │ │ - Interfaces │ │
│ └─────────────────┘ └──────────────────┘ │
│ │
│ ┌─────────────────┐ ┌──────────────────┐ │
│ │ Status │ │ Firmware │ │
│ │ - Online │ │ - Version │ │
│ │ - Last Seen │ │ - Last Updated │ │
│ └─────────────────┘ └──────────────────┘ │
└─────────────────────────────────────────────┘
```
### Status Badges
- **Online:** Green badge with Wifi icon
- **Offline:** Gray badge with X icon
- **Unknown:** Yellow badge with AlertCircle icon
## Technical Considerations
### API Integration
- Auvik API uses Basic Authentication (username:password encoded in Base64)
- API endpoint: `https://auvikapi.us1.my.auvik.com/v1/` (or region-specific)
- Rate limits: Respect Auvik's rate limiting (typically 1000 requests/hour)
- Pagination: Auvik uses cursor-based pagination for large result sets
### Matching Algorithm
```typescript
function matchAuvikDevice(
autotaskDevice: ConfigurationItem,
auvikDevices: AuvikDevice[]
): AuvikDevice | null {
// Priority 1: Serial number
if (autotaskDevice.serialNumber) {
const match = auvikDevices.find(d =>
d.serialNumber?.toLowerCase() === autotaskDevice.serialNumber?.toLowerCase()
);
if (match) return match;
}
// Priority 2: Hostname
if (autotaskDevice.rmmDeviceAuditHostname) {
const match = auvikDevices.find(d =>
d.deviceName?.toLowerCase() === autotaskDevice.rmmDeviceAuditHostname?.toLowerCase()
);
if (match) return match;
}
// Priority 3: MAC address
if (autotaskDevice.rmmDeviceAuditMacAddress) {
const normalizedMac = normalizeMacAddress(autotaskDevice.rmmDeviceAuditMacAddress);
const match = auvikDevices.find(d =>
d.macAddresses?.some(mac => normalizeMacAddress(mac) === normalizedMac)
);
if (match) return match;
}
return null;
}
```
### Dependencies
- No new npm packages required (use built-in fetch)
- Leverage existing service patterns (`autotask-client.ts`, `datto-rmm-client.ts`)
- Use existing UI components from shadcn/ui
### File Structure
```
/lib/services/auvik-client.ts # Auvik API client
/lib/services/auvik-factory.ts # Singleton factory for client
/lib/types/auvik.ts # TypeScript types
/app/api/auvik/devices/route.ts # API endpoint for device list
/components/configuration-items/auvik-tab.tsx # Auvik tab component
```
### Environment Variables
```bash
AUVIK_API_URL=https://auvikapi.us1.my.auvik.com/v1
AUVIK_API_USER=your-api-user
AUVIK_API_KEY=your-api-key
```
## Success Metrics
1. **Match Rate:** 90%+ of network devices (switches, routers, firewalls) with serial numbers are successfully matched to Auvik devices
2. **Performance:** Auvik data loads within 2 seconds for typical company (< 100 devices)
3. **Reliability:** Configuration items page remains functional even when Auvik API returns errors (100% uptime for core functionality)
4. **Adoption:** Network administrators use Auvik tab for at least 50% of network device views within first month
5. **Error Rate:** Less than 1% of Auvik API calls result in unhandled errors
## Open Questions
1. **Regional API Endpoints:** Should we support multiple Auvik regions (US1, US2, EU, AU) or assume single region?
- *Recommendation:* Make `AUVIK_API_URL` configurable to support any region
2. **Tenant Name Matching:** What if Auvik tenant name doesn't exactly match Autotask company name?
- *Recommendation:* Use fuzzy matching or allow manual tenant-to-company mapping in future iteration
3. **Device Type Filtering:** Should we only show Auvik data for network devices (switches, routers, firewalls) or all device types?
- *Recommendation:* Show for all devices but prioritize network devices in matching
4. **API Key Rotation:** How should we handle API key expiration/rotation?
- *Recommendation:* Log clear error messages when authentication fails, require manual .env update
5. **Multiple Auvik Instances:** Do we need to support multiple Auvik accounts (for MSPs with multiple Auvik instances)?
- *Recommendation:* Out of scope for v1, single Auvik instance only
## Implementation Notes for Developers
### Getting Started
1. Review existing RMM integration (`datto-rmm-client.ts`, `rmm-tab.tsx`) as reference
2. Read Auvik API documentation: https://support.auvik.com/hc/en-us/articles/360031007111
3. Set up Auvik API credentials in `.env` file
4. Test API connectivity using Postman or curl before coding
### Testing Checklist
- [ ] Verify authentication works with provided credentials
- [ ] Test device matching with various scenarios (serial match, hostname match, MAC match, no match)
- [ ] Test with company that has no Auvik tenant
- [ ] Test with Auvik API unavailable (network error)
- [ ] Test with large device lists (100+ devices)
- [ ] Verify UI displays correctly on mobile/tablet/desktop
- [ ] Check that existing PSA/RMM functionality is not affected
### Code Review Focus Areas
- Error handling completeness
- TypeScript type safety
- Consistent code style with existing services
- Proper logging for debugging
- Performance (avoid N+1 queries)

447
tasks/prd-data-chatbot.md Normal file
View file

@ -0,0 +1,447 @@
# PRD: Data Chatbot & Query Interface
## Introduction/Overview
The Data Chatbot is an intelligent query interface that allows internal users to quickly access and analyze synced Autotask data through natural language conversations. Users can ask questions like "Who had the most time entries last week?" or "What customer had the most tickets?" and receive accurate, formatted responses with visualizations and export options.
**Problem it solves:** Currently, accessing specific insights from synced data requires writing SQL queries or navigating through multiple database views. This creates a barrier for non-technical users and slows down decision-making. The chatbot democratizes data access by allowing anyone to query the database using plain English.
**Goal:** Provide a conversational, AI-powered interface that makes synced Autotask data instantly accessible to all internal team members, regardless of technical skill level.
## Goals
1. **Accessibility**: Enable non-technical users to query complex database relationships using natural language
2. **Speed**: Reduce time-to-insight from minutes (manual queries) to seconds (conversational interface)
3. **Accuracy**: Deliver correct results with 95%+ accuracy for common query patterns
4. **Flexibility**: Support both AI-powered natural language and structured query templates
5. **Mobile-First**: Provide PWA experience for on-the-go data access
6. **Actionable Insights**: Present data in multiple formats (tables, charts, exports) for immediate use
## User Stories
### Primary User Stories
1. **As a manager**, I want to ask "Who had the most time entries last week?" so that I can quickly identify top performers without running SQL queries.
2. **As a support lead**, I want to ask "What customer had the most tickets this month?" so that I can proactively reach out to high-volume clients.
3. **As a project manager**, I want to ask "Show me all open tickets for Acme Corp assigned to John" so that I can check project status during client calls.
4. **As an executive**, I want to ask "What's our average ticket resolution time by priority?" so that I can track KPIs without waiting for reports.
5. **As a technician on mobile**, I want to quickly check "My open tickets" while in the field so that I can prioritize my work.
6. **As a billing coordinator**, I want to ask "Show me unbilled time entries from last month" so that I can prepare invoices.
### Secondary User Stories
7. **As a data analyst**, I want to export query results to CSV so that I can perform additional analysis in Excel.
8. **As a team lead**, I want to save frequently-used queries so that I can access them quickly without retyping.
9. **As a user**, I want to see my query history so that I can reference previous insights.
10. **As a user**, I want to choose between AI-powered queries and template-based queries so that I can balance cost and flexibility.
## Functional Requirements
### Core Query Engine
1. The system **must** accept natural language queries in a conversational chat interface.
2. The system **must** support querying all synced entities: tickets, tasks, projects, companies, resources, contacts, contracts, time entries, billing items, configuration items, and their relationships.
3. The system **must** support multi-entity joins (e.g., "tickets with their assigned resources and companies").
4. The system **must** execute queries against the live PostgreSQL database.
5. The system **must** return results within 5 seconds for 95% of queries.
6. The system **must** handle common query patterns:
- Aggregations (count, sum, average, min, max)
- Filtering (by date range, status, assignment, company, etc.)
- Sorting (top N, bottom N, ordered by field)
- Grouping (by company, resource, status, etc.)
- Time-based queries (last week, this month, last 30 days, etc.)
### AI/LLM Integration
7. The system **must** allow users to choose between two query modes:
- **AI Mode**: Uses LLM (OpenAI GPT-4 or Claude) for natural language understanding
- **Template Mode**: Uses predefined query patterns (faster, no API cost)
8. The system **must** convert natural language to SQL queries safely (prevent SQL injection).
9. The system **must** validate generated SQL before execution.
10. The system **must** provide query explanations (e.g., "I'm searching for tickets created in the last 7 days...").
### User Interface - Desktop
11. The system **must** provide a dedicated page at `/data-chat` or similar route.
12. The system **must** display a chat interface with:
- Message history (user queries and bot responses)
- Input field for typing queries
- Send button and Enter key support
- Mode toggle (AI vs Template)
13. The system **must** display results in multiple formats:
- **Table view**: Sortable, paginated data tables
- **Card view**: Visual cards for entity records
- **Chart view**: Bar charts, line charts, pie charts for aggregated data
14. The system **must** provide export options:
- CSV download
- JSON download
- Copy to clipboard
15. The system **must** show loading states during query execution.
16. The system **must** display error messages clearly when queries fail.
### User Interface - Mobile (PWA)
17. The system **must** be responsive and optimized for mobile devices.
18. The system **must** function as a Progressive Web App (PWA):
- Installable to home screen
- Works offline for query history (results require connection)
- Fast loading with service worker caching
19. The system **must** provide a mobile-optimized chat interface:
- Full-screen chat on mobile
- Touch-friendly buttons and inputs
- Swipeable result cards
20. The system **must** support voice input on mobile devices (optional but recommended).
### Query Management
21. The system **must** maintain query history for each user session.
22. The system **must** allow users to save favorite queries with custom names.
23. The system **must** provide quick-access buttons for common queries:
- "My open tickets"
- "Team time entries this week"
- "Top 10 customers by ticket volume"
- "Overdue tickets"
24. The system **must** allow users to edit and re-run previous queries.
### Data Visualization
25. The system **must** automatically suggest appropriate chart types based on query results:
- Bar charts for comparisons (e.g., tickets by company)
- Line charts for time series (e.g., tickets over time)
- Pie charts for distributions (e.g., tickets by status)
26. The system **must** allow users to toggle between table and chart views.
27. The system **must** make charts interactive (hover for details, click to filter).
### Security & Permissions
28. The system **must** require authentication (use existing auth system).
29. The system **must** respect user permissions (if implemented in the future).
30. The system **must** log all queries for audit purposes.
31. The system **must** prevent SQL injection and malicious queries.
32. The system **must** rate-limit queries to prevent abuse (e.g., 60 queries per minute per user).
### Performance & Caching
33. The system **should** cache common query results for 5 minutes.
34. The system **should** implement query result pagination for large datasets (>1000 rows).
35. The system **should** provide query performance metrics (execution time).
## Non-Goals (Out of Scope)
1. **Data Modification**: The chatbot will NOT allow users to insert, update, or delete data. It is read-only.
2. **Real-time Streaming**: The chatbot will NOT provide real-time updates or websocket-based live data feeds.
3. **Advanced Analytics**: Complex statistical analysis, machine learning predictions, or forecasting are out of scope.
4. **External Data Sources**: The chatbot will only query synced Autotask data, not external APIs or services.
5. **Multi-tenant Isolation**: Initial version assumes single organization; multi-tenant support is future work.
6. **Custom Dashboards**: Building and saving custom dashboard layouts is out of scope (separate feature).
7. **Scheduled Reports**: Automated report generation and email delivery is out of scope.
8. **Data Governance**: Advanced role-based access control (RBAC) at the field level is out of scope for v1.
## Design Considerations
### UI/UX Guidelines
- **Chat Interface**: Follow modern chat UI patterns (similar to ChatGPT, Claude, or Slack)
- User messages: Right-aligned, blue background
- Bot responses: Left-aligned, gray background
- Timestamps on messages
- Typing indicator while processing
- **Component Library**: Use existing shadcn/ui components
- `Card` for message bubbles
- `Table` for data tables
- `Button` for actions
- `Select` for mode toggle
- `Tabs` for view switching (table/chart)
- **Icons**: Use Lucide React icons
- `MessageSquare` for chat
- `BarChart3` for charts
- `Download` for exports
- `History` for query history
- `Sparkles` for AI mode
- `List` for template mode
- **Color Scheme**: Follow existing app theme
- Primary: Blue for AI mode
- Secondary: Gray for template mode
- Success: Green for successful queries
- Error: Red for failed queries
### Mobile PWA Requirements
- **Manifest File**: Create `manifest.json` with app metadata
- **Service Worker**: Implement for offline query history
- **Responsive Breakpoints**:
- Mobile: < 768px (single column, full-screen chat)
- Tablet: 768px - 1024px (sidebar + chat)
- Desktop: > 1024px (full layout with panels)
### Example Queries to Support
```
Natural Language Examples:
- "Who had the most time entries last week?"
- "What customer had the most tickets this month?"
- "Show me all open tickets for Acme Corp"
- "What's the average ticket resolution time?"
- "List tickets assigned to John Doe that are overdue"
- "How many projects are currently active?"
- "Show me time entries for Project X in October"
- "Which resources have the highest billable hours?"
- "What are the top 5 issues by ticket count?"
- "Show me all high priority tickets created yesterday"
```
## Technical Considerations
### Architecture
1. **Frontend**: Next.js 14+ with App Router
- New route: `/app/data-chat/page.tsx`
- Components: `/components/data-chat/`
- PWA config: `/public/manifest.json`, service worker
2. **Backend API**: Next.js API routes
- `/api/data-chat/query` - Execute queries
- `/api/data-chat/history` - Get/save query history
- `/api/data-chat/templates` - Get predefined query templates
- `/api/data-chat/export` - Export results
3. **Database**: PostgreSQL (existing)
- Read-only queries via connection pool
- New table: `query_history` for storing user queries
- New table: `saved_queries` for favorite queries
4. **LLM Integration**:
- **Option 1**: OpenAI GPT-4 API (more accurate, costs ~$0.01-0.03 per query)
- **Option 2**: Anthropic Claude API (alternative, similar cost)
- **Fallback**: Template-based queries (free, predefined patterns)
5. **Query Generation**:
- Use LLM to generate SQL from natural language
- Implement SQL sanitization and validation
- Use parameterized queries to prevent injection
- Whitelist allowed tables and columns
6. **Caching**: Redis or in-memory cache for frequent queries
### Dependencies
```json
{
"openai": "^4.0.0", // For AI mode
"@anthropic-ai/sdk": "^0.9.0", // Alternative AI provider
"recharts": "^2.10.0", // For charts
"react-chartjs-2": "^5.2.0", // Alternative charting
"papaparse": "^5.4.0", // CSV export
"sql-formatter": "^15.0.0", // SQL formatting for display
"zod": "^3.22.0" // Query validation
}
```
### Database Schema
```sql
-- Query history table
CREATE TABLE query_history (
id SERIAL PRIMARY KEY,
user_id VARCHAR(255) NOT NULL,
query_text TEXT NOT NULL,
query_mode VARCHAR(20) NOT NULL, -- 'ai' or 'template'
generated_sql TEXT,
result_count INTEGER,
execution_time_ms INTEGER,
success BOOLEAN DEFAULT true,
error_message TEXT,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
-- Saved queries table
CREATE TABLE saved_queries (
id SERIAL PRIMARY KEY,
user_id VARCHAR(255) NOT NULL,
name VARCHAR(255) NOT NULL,
query_text TEXT NOT NULL,
query_mode VARCHAR(20) NOT NULL,
is_favorite BOOLEAN DEFAULT false,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
-- Indexes
CREATE INDEX idx_query_history_user_id ON query_history(user_id);
CREATE INDEX idx_query_history_created_at ON query_history(created_at);
CREATE INDEX idx_saved_queries_user_id ON saved_queries(user_id);
```
### Security Considerations
1. **SQL Injection Prevention**:
- Use parameterized queries exclusively
- Validate and sanitize all LLM-generated SQL
- Whitelist allowed tables and columns
- Block dangerous SQL keywords (DROP, DELETE, UPDATE, INSERT, ALTER, etc.)
2. **Rate Limiting**:
- Implement per-user rate limiting (60 queries/minute)
- Implement per-IP rate limiting for API endpoints
- Consider cost controls for AI mode (e.g., 100 AI queries per user per day)
3. **Authentication**:
- Reuse existing NextAuth.js setup
- Require authenticated session for all data-chat routes
4. **Query Validation**:
- Parse generated SQL with SQL parser library
- Verify only SELECT statements are executed
- Ensure queries only access allowed tables
- Limit result set size (max 10,000 rows)
### Performance Optimization
1. **Query Optimization**:
- Add database indexes for common query patterns
- Implement query result pagination
- Set query timeout (30 seconds max)
2. **Caching Strategy**:
- Cache common queries for 5 minutes
- Cache template query results for 10 minutes
- Invalidate cache on data sync completion
3. **Frontend Optimization**:
- Lazy load chart libraries
- Virtual scrolling for large result tables
- Progressive loading for query history
## Success Metrics
### Primary Metrics
1. **Query Accuracy**: 95%+ of queries return correct results (measured by user feedback)
2. **Response Time**: 95% of queries complete within 5 seconds
3. **Adoption Rate**: 70%+ of internal users try the chatbot within first month
4. **Engagement**: Average 10+ queries per active user per week
### Secondary Metrics
5. **AI vs Template Usage**: Track ratio to optimize cost vs. accuracy
6. **Query Success Rate**: 90%+ of queries execute without errors
7. **Export Usage**: Track how often users export results (indicates value)
8. **Mobile Usage**: 30%+ of queries come from mobile devices
9. **Saved Queries**: Average 3+ saved queries per active user
10. **User Satisfaction**: 4.5+ star rating in feedback surveys
### Monitoring
- Log all queries with execution time and success/failure
- Track LLM API costs and usage patterns
- Monitor database query performance
- Collect user feedback via in-app rating system
- Track error rates and common failure patterns
## Open Questions
1. **LLM Provider**: Should we start with OpenAI GPT-4, Claude, or both? (Recommend: Start with OpenAI, add Claude as fallback)
2. **Cost Management**: What's the acceptable monthly budget for LLM API calls? (Estimate: $100-500/month for 10-20 active users)
3. **User Authentication**: Should we use existing NextAuth.js setup or implement separate auth? (Recommend: Use existing auth)
4. **Query Templates**: What are the top 20 most common queries we should pre-build? (Needs input from team)
5. **Data Freshness**: Should we show when data was last synced? (Recommend: Yes, display "Data as of [timestamp]")
6. **Error Handling**: How should we handle ambiguous queries? (Recommend: Ask clarifying questions or suggest alternatives)
7. **Multi-language Support**: Do we need to support languages other than English? (Defer to v2)
8. **Voice Input**: Is voice input a must-have for mobile or nice-to-have? (Recommend: Nice-to-have for v1)
9. **Collaboration**: Should users be able to share queries with team members? (Defer to v2)
10. **Notifications**: Should users get notified when saved queries have new results? (Defer to v2)
## Implementation Phases
### Phase 1: MVP (2-3 weeks)
- Basic chat interface (desktop only)
- AI mode with OpenAI GPT-4
- Table view for results
- Query history
- Basic error handling
- CSV export
### Phase 2: Enhanced Features (2 weeks)
- Template mode with predefined queries
- Chart visualizations
- Saved queries
- Mobile responsive design
- Query explanations
### Phase 3: PWA & Polish (1-2 weeks)
- PWA implementation
- Mobile optimization
- Performance optimization
- Caching layer
- Advanced error handling
- User feedback system
### Phase 4: Advanced Features (Future)
- Voice input
- Query sharing
- Scheduled queries
- Advanced visualizations
- Multi-language support
- Role-based permissions
## Appendix: Example Query Templates
```typescript
// Common query templates for Template Mode
const queryTemplates = [
{
name: "My Open Tickets",
description: "Show all tickets assigned to me that are not completed",
sql: "SELECT * FROM tickets WHERE assigned_resource_id = $userId AND status != 5 AND is_deleted = false"
},
{
name: "Top Customers by Ticket Volume",
description: "Show customers with most tickets this month",
sql: `SELECT c.name, COUNT(t.id) as ticket_count
FROM companies c
JOIN tickets t ON c.id = t.company_id
WHERE t.create_date >= date_trunc('month', CURRENT_DATE)
GROUP BY c.id, c.name
ORDER BY ticket_count DESC
LIMIT 10`
},
{
name: "Time Entries This Week",
description: "Show all time entries for current week",
sql: `SELECT r.first_name, r.last_name, SUM(te.hours_worked) as total_hours
FROM time_entries te
JOIN resources r ON te.resource_id = r.id
WHERE te.date_worked >= date_trunc('week', CURRENT_DATE)
GROUP BY r.id, r.first_name, r.last_name
ORDER BY total_hours DESC`
},
// Add 17 more common templates...
];
```
---
**Document Version**: 1.0
**Created**: 2024-11-03
**Last Updated**: 2024-11-03
**Status**: Draft - Awaiting Approval

0
tasks/prd-postgres.md Normal file
View file

136
tasks/prd-time-entries-analytics.md Normal file
View file

@ -0,0 +1,136 @@
# Time Entries Analytics PRD
## Introduction/Overview
This feature adds Time Entries data synchronization from Autotask to PostgreSQL and provides advanced analytics capabilities for managers and executives to quickly understand what happened on tickets and tasks. The system will include a collapsible timeline view, AI-powered analysis of work performed, and scoring mechanisms for quality and timeliness of entries.
## Goals
1. Enable rapid analysis of ticket/task activity patterns and work progression
2. Provide AI-powered insights into work quality and productivity patterns
3. Create scoring systems to measure entry quality and timeliness
4. Offer flexible timeline views (hourly, daily, weekly, monthly) with key moment highlighting
5. Support both granular single-ticket analysis and aggregate time period summaries
6. Integrate time entry data with existing entities (tickets, tasks, projects, resources) for enriched analysis
## User Stories
**As a manager, I want to** view a timeline of all activities on a specific ticket so that I can quickly understand the complete work progression and identify bottlenecks.
**As a manager, I want to** see AI-generated insights about work patterns so that I can identify productivity trends and areas for improvement.
**As an executive, I want to** view aggregate time entry summaries for weekly/monthly periods so that I can understand overall team productivity and resource allocation.
**As a manager, I want to** see quality and timeliness scores for time entries so that I can identify which team members need training on proper time tracking.
**As an executive, I want to** filter time entries by activity type (human vs. system) so that I can understand the balance between automated and manual work.
**As a manager, I want to** analyze historical time entry data so that I can compare current performance with past periods and identify trends.
## Functional Requirements
### Data Synchronization
1. The system must synchronize Time Entries data from Autotask API to PostgreSQL database
2. The system must import all historical time entry data for comprehensive analysis
3. The system must maintain real-time synchronization for new time entries
4. The system must store all relevant Time Entry fields including duration, entry date, notes, and associated entities
### Timeline View
5. The system must provide a collapsible timeline interface with multiple time range options (Hour, Day, Week, Month)
6. The system must display time entries chronologically with visual distinction between human and system activities
7. The system must highlight key moments in the timeline (e.g., ticket creation, status changes, resolution)
8. The system must allow users to expand/collapse time periods for detailed or summary views
9. The system must show the length of time worked for each entry with clear visual indicators
### Analysis & Scoring
10. The system must provide AI-powered analysis of work performed using LLM processing
11. The system must calculate and display an "Activity Score" based on entry quality, completeness, and work patterns
12. The system must calculate and display a "Content Score" based on the quality and detail of time entry descriptions
13. The system must calculate and display a "Timeliness Score" based on when entries were made relative to the work performed
14. The system must show individual scores alongside each time entry and aggregate scores for time periods
15. The system must provide analysis for both individual tickets/tasks and aggregate date ranges
### Data Integration & Enrichment
16. The system must integrate time entry data with related tickets, tasks, projects, and resources
17. The system must enrich time entry analysis with data from all existing synchronized tables
18. The system must provide filtering capabilities by resource, project, ticket, task, and activity type
19. The system must support both single-entity analysis and multi-entity comparative analysis
### User Interface
20. The system must provide a dedicated Time Entries Analytics page accessible from the admin dashboard
21. The system must offer both detailed single-ticket views and summary dashboard views
22. The system must include export capabilities for analysis results and reports
23. The system must provide responsive design for desktop and tablet viewing
## Non-Goals (Out of Scope)
1. Direct editing of time entries from the analytics interface (this is a read-only analysis tool)
2. Time entry approval workflows or management features
3. Billing or invoicing functionality based on time entries
4. Mobile application development (focus on web interface)
5. Real-time alerts or notifications based on time entry patterns
6. Integration with external time tracking systems beyond Autotask
## Design Considerations
### Timeline Interface
- Use collapsible accordion-style components for different time periods
- Implement color coding for different activity types (human vs. system)
- Use icons and visual indicators to highlight key moments and milestones
- Provide smooth animations for expanding/collapsing timeline sections
### Scoring Visualization
- Use progress bars or radial indicators for individual scores
- Implement trend charts for score changes over time
- Use heat maps for showing activity density across time periods
- Provide tooltips explaining how scores are calculated
### Analysis Display
- Use card-based layout for AI insights and recommendations
- Implement tabbed interface for different analysis views (timeline, scores, insights)
- Use consistent color scheme with existing admin dashboard
- Ensure accessibility with proper contrast ratios and keyboard navigation
## Technical Considerations
### Database Schema
- Add Time Entries table following existing entity patterns
- Include proper indexing for time-based queries and joins
- Implement foreign key relationships to tickets, tasks, projects, and resources
- Consider partitioning for large time entry datasets
### API Integration
- Extend existing Autotask client to support Time Entries entity
- Implement pagination handling for large historical datasets
- Add error handling for API rate limits and data inconsistencies
- Use existing sync service patterns for data synchronization
### LLM Integration
- Integrate with existing AI/LLM services for work analysis
- Implement caching for AI analysis results to improve performance
- Add queue processing for batch analysis of historical data
- Consider cost optimization for LLM API usage
### Performance
- Implement efficient database queries for timeline generation
- Use caching for frequently accessed aggregate data
- Consider background processing for AI analysis and score calculations
- Optimize for handling large datasets (thousands of time entries)
## Success Metrics
1. **Usage Metrics**: 80% of managers and executives access the Time Entries Analytics feature weekly
2. **Efficiency Metrics**: Reduce time spent analyzing ticket activity by 50% compared to current manual methods
3. **Data Quality**: 25% improvement in time entry quality scores within 3 months of implementation
4. **User Satisfaction**: Achieve 4.5/5 user satisfaction score from target users
5. **Performance**: Timeline views and analysis complete within 3 seconds for typical date ranges
## Open Questions
1. What specific LLM model should be used for work analysis, and what are the cost constraints?
2. Should the AI analysis be configurable by organization or role?
3. What retention period should be set for historical time entry data?
4. Should there be role-based access controls for different levels of analysis?
5. How should the system handle time entries from deleted/archived tickets or resources?
6. What export formats are required for analysis reports (PDF, Excel, CSV)?
7. Should the scoring algorithms be customizable or standardized across all organizations?

108
tasks/tasks-prd-auvik-integration.md Normal file
View file

@ -0,0 +1,108 @@
# Tasks: Auvik Integration
## Relevant Files
- `/lib/types/auvik.ts` - TypeScript type definitions for Auvik API entities (devices, tenants, interfaces)
- `/lib/services/auvik-client.ts` - Auvik API client service for making authenticated requests
- `/lib/services/auvik-factory.ts` - Singleton factory pattern for Auvik client instantiation
- `/app/api/auvik/devices/route.ts` - API endpoint for fetching Auvik devices with tenant filtering
- `/components/configuration-items/auvik-tab.tsx` - React component for displaying Auvik device details in modal
- `/components/configuration-items/config-item-modal.tsx` - Existing modal component (modify to add Auvik tab)
- `/app/configuration-items/page.tsx` - Main configuration items page (modify to add Auvik column)
- `/app/api/configuration-items/[id]/route.ts` - Existing API route (modify to include Auvik device matching)
- `/app/api/rmm-devices/route.ts` - Existing comparison endpoint (modify to include Auvik matching)
### Notes
- Follow existing patterns from Datto RMM integration (`datto-rmm-client.ts`, `rmm-tab.tsx`)
- Use Basic Authentication for Auvik API (username:password in Authorization header)
- Auvik API documentation: https://support.auvik.com/hc/en-us/articles/360031007111
- Test with real Auvik credentials from `.env` file
- Ensure graceful degradation when Auvik API is unavailable
## Tasks
- [ ] 1.0 Create Auvik TypeScript Types and API Client
- [ ] 1.1 Create `/lib/types/auvik.ts` with TypeScript interfaces for AuvikDevice, AuvikTenant, AuvikNetworkInterface, and API response structures
- [ ] 1.2 Define AuvikDevice interface with fields: id, deviceName, serialNumber, macAddresses, ipAddresses, deviceType, manufacturer, model, firmwareVersion, onlineStatus, lastSeenTime, uptime, tenantId, tenantName
- [ ] 1.3 Define AuvikNetworkInterface interface with fields: interfaceName, status, speed, macAddress, ipAddress, vlan
- [ ] 1.4 Create `/lib/services/auvik-client.ts` implementing AuvikClient class with constructor accepting config (apiUrl, apiUser, apiKey)
- [ ] 1.5 Implement `getAuthHeaders()` method that returns Basic Authentication header (Base64 encoded username:password)
- [ ] 1.6 Implement `makeApiCall<T>()` method for generic API requests with error handling and logging
- [ ] 1.7 Implement `getAllDevices()` method to fetch device inventory from `/v1/inventory/device/info` endpoint
- [ ] 1.8 Implement `getDevicesByTenant(tenantId: string)` method with tenant filtering
- [ ] 1.9 Implement `getTenants()` method to fetch tenant list from `/v1/tenants` endpoint
- [ ] 1.10 Add rate limiting logic to respect Auvik API limits (track request count and timestamps)
- [ ] 1.11 Create `/lib/services/auvik-factory.ts` with `getAuvikClient()` singleton factory function
- [ ] 1.12 Load Auvik credentials from environment variables in factory (AUVIK_API_URL, AUVIK_API_USER, AUVIK_API_KEY)
- [ ] 2.0 Implement Auvik API Endpoints
- [ ] 2.1 Create `/app/api/auvik/devices/route.ts` with GET handler
- [ ] 2.2 Accept query parameters: `companyId` (optional), `companyName` (optional)
- [ ] 2.3 If companyName provided, fetch Auvik tenants and find matching tenant by name (case-insensitive, fuzzy match)
- [ ] 2.4 If tenant match found, fetch devices filtered by tenantId; otherwise fetch all devices
- [ ] 2.5 Transform Auvik API response to match AuvikDevice interface structure
- [ ] 2.6 Implement try-catch error handling that logs errors but returns 200 with empty array on failure
- [ ] 2.7 Add console logging for tenant matching results and device counts
- [ ] 2.8 Return JSON response with devices array and optional metadata (tenantId, tenantName)
- [ ] 3.0 Add Auvik Tab to Configuration Item Modal
- [ ] 3.1 Create `/components/configuration-items/auvik-tab.tsx` component accepting `device?: AuvikDevice` prop
- [ ] 3.2 Import required UI components (Card, CardContent, CardHeader, Badge, Label) and icons (Network, Info, Wifi, Shield)
- [ ] 3.3 Implement empty state UI when no device provided (show "No Auvik data available" message with icon)
- [ ] 3.4 Create "Basic Information" section displaying: device name, device type, serial number, manufacturer, model
- [ ] 3.5 Create "Network Information" section displaying: IP addresses (list), MAC addresses (list), primary interface details
- [ ] 3.6 Create "Status Information" section displaying: online/offline badge, last seen timestamp (formatted), uptime (formatted duration)
- [ ] 3.7 Create "Firmware Information" section displaying: firmware version, last updated date
- [ ] 3.8 Create "Network Interfaces" section displaying table/list of interfaces with name, status, speed, MAC address
- [ ] 3.9 Style online status with green badge and Wifi icon, offline with gray badge and X icon
- [ ] 3.10 Use consistent spacing and layout matching existing PSA/RMM tabs (2-column grid on desktop)
- [ ] 3.11 Modify `/components/configuration-items/config-item-modal.tsx` to add Auvik tab to TabsList
- [ ] 3.12 Add TabsTrigger for "Auvik Data" with Network icon and online/offline badge if device exists
- [ ] 3.13 Add TabsContent for "auvik" value rendering AuvikTab component with auvikDevice prop
- [ ] 3.14 Update modal state to include `auvikDevice?: AuvikDevice` in ConfigItemDetail interface
- [ ] 4.0 Add Auvik Column to Configuration Items Table
- [ ] 4.1 Open `/app/configuration-items/page.tsx` and locate the table header row (TableHead components)
- [ ] 4.2 Add new `<TableHead>Auvik</TableHead>` column after the RMM column
- [ ] 4.3 Update colspan values in grouped rows from current value to +1 (account for new column)
- [ ] 4.4 In the table body, add new TableCell after RMM cell for grouped rows
- [ ] 4.5 Display green CheckCircle icon if `item.auvikDevice` exists, gray XCircle if not
- [ ] 4.6 Add same TableCell logic for non-grouped rows (around line 1060+)
- [ ] 4.7 Update DeviceComparison interface to include `auvikDevice?: AuvikDevice` field
- [ ] 4.8 Import AuvikDevice type from `/lib/types/auvik`
- [ ] 5.0 Implement Device Matching Logic
- [ ] 5.1 Modify `/app/api/rmm-devices/route.ts` to fetch Auvik devices at the start of GET handler
- [ ] 5.2 Call `getAuvikClient().getAllDevices()` or `getDevicesByTenant()` if companyName provided
- [ ] 5.3 Wrap Auvik API call in try-catch to handle failures gracefully (continue with empty array)
- [ ] 5.4 Create helper function `matchAuvikDevice(autotaskDevice: ConfigurationItem, auvikDevices: AuvikDevice[]): AuvikDevice | null`
- [ ] 5.5 Implement Priority 1 matching: Compare serial numbers (case-insensitive, trimmed)
- [ ] 5.6 Implement Priority 2 matching: Compare hostnames using `rmmDeviceAuditHostname` or `referenceTitle` (case-insensitive)
- [ ] 5.7 Implement Priority 3 matching: Compare MAC addresses (normalize format, check against all device MACs)
- [ ] 5.8 Create `normalizeMacAddress()` helper function to strip colons/hyphens and lowercase
- [ ] 5.9 In comparison loop, call matchAuvikDevice for each autotaskDevice and add result to comparison object
- [ ] 5.10 Log matching results with match method (serial/hostname/MAC) for debugging
- [ ] 5.11 Modify `/app/api/configuration-items/[id]/route.ts` GET handler to fetch Auvik devices
- [ ] 5.12 Use same matchAuvikDevice logic to find matching device for the single configuration item
- [ ] 5.13 Include auvikDevice in response JSON: `{ autotaskDevice, rmmDevice, auvikDevice, companyName }`
- [ ] 5.14 Add console logging for Auvik device matching in detail endpoint
- [ ] 6.0 Testing and Error Handling
- [ ] 6.1 Test Auvik API authentication with valid credentials (verify 200 response)
- [ ] 6.2 Test with invalid credentials (verify graceful failure, no app crash)
- [ ] 6.3 Test device matching with serial number match (verify correct device returned)
- [ ] 6.4 Test device matching with hostname match (verify fallback works)
- [ ] 6.5 Test device matching with MAC address match (verify normalization works)
- [ ] 6.6 Test device matching with no match (verify null returned, "-" displayed)
- [ ] 6.7 Test with company that has no Auvik tenant (verify all devices returned or empty array)
- [ ] 6.8 Test tenant name matching with exact match and fuzzy match scenarios
- [ ] 6.9 Test configuration items page with Auvik API unavailable (verify page still loads)
- [ ] 6.10 Test modal opening with Auvik device (verify tab displays data correctly)
- [ ] 6.11 Test modal opening without Auvik device (verify empty state message)
- [ ] 6.12 Test table column alignment with new Auvik column (verify no layout issues)
- [ ] 6.13 Verify console logs show appropriate messages for matching, errors, and API calls
- [ ] 6.14 Test with large device list (100+ devices) to verify performance
- [ ] 6.15 Test responsive layout on mobile/tablet (verify Auvik tab and column display correctly)
- [ ] 6.16 Verify TypeScript compilation with no errors
- [ ] 6.17 Test that existing PSA and RMM functionality is not affected by changes

215
tasks/tasks-prd-postgres.md Normal file
View file

@ -0,0 +1,215 @@
# Task List: PostgreSQL Autotask Sync Implementation
Generated from: `prd-postgres.md`
## Relevant Files
### Infrastructure & Configuration
- `docker-compose.yml` - ✅ Added PostgreSQL service, volumes, and health checks
- `.env.local` - ✅ Added PostgreSQL connection environment variables
- `.env.example` - ✅ Created with PostgreSQL configuration documentation
- `migrations/` - ✅ Created directory for database migration files
- `migrations/001_initial_schema.sql` - ✅ Initial database schema with all 13 entity tables, audit fields, foreign keys, and sync_history
- `migrations/002_add_indexes.sql` - ✅ Additional performance indexes for common query patterns
- `migrations/003_fix_resource_type.sql` - ✅ Fix data type mismatches for resources table (resource_type, travel_availability_pct)
### Database & Services
- `lib/services/postgres-client.ts` - ✅ PostgreSQL connection pool with query methods, upsert, bulk operations
- `lib/services/sync-service.ts` - ✅ Core sync orchestration service with full/incremental/entity-specific sync
- `lib/services/entity-sync.ts` - ✅ Entity-specific sync logic for all 13 Autotask entities
- `lib/services/rate-limiter.ts` - ✅ Rate limiting with 10 req/sec throttling and queue
- `lib/types/sync.ts` - ✅ TypeScript types for SyncConfig, SyncStatus, SyncHistory, EntityType enum
- `lib/types/database.ts` - ✅ TypeScript interfaces for all 13 entity tables
- `package.json` - ✅ Added pg and @types/pg dependencies
### API Routes
- `app/api/sync/full/route.ts` - ✅ POST endpoint for full sync
- `app/api/sync/incremental/route.ts` - ✅ POST endpoint for incremental sync
- `app/api/sync/entity/route.ts` - ✅ POST endpoint for entity-specific sync
- `app/api/sync/history/route.ts` - ✅ GET endpoint for sync history
- `app/api/sync/last-sync/route.ts` - ✅ GET endpoint for last sync times
- `app/api/data/companies/route.ts` - ✅ GET endpoint for querying companies from PostgreSQL
- `app/api/data/tickets/route.ts` - ✅ GET endpoint for querying tickets from PostgreSQL
- `app/api/data/tasks/route.ts` - GET endpoint for querying tasks from PostgreSQL (pending)
- `app/api/data/configuration-items/route.ts` - GET endpoint for querying config items from PostgreSQL
- `app/api/data/contacts/route.ts` - GET endpoint for querying contacts from PostgreSQL
- `app/api/data/contracts/route.ts` - GET endpoint for querying contracts from PostgreSQL
- `app/api/data/projects/route.ts` - GET endpoint for querying projects from PostgreSQL
- `app/api/data/resources/route.ts` - GET endpoint for querying resources from PostgreSQL
- `app/api/data/billing-items/route.ts` - GET endpoint for querying billing items from PostgreSQL
### Admin UI Components
- `app/layout.tsx` - ✅ Added Toaster component for toast notifications
- `app/admin/sync/page.tsx` - ✅ Main admin sync page with responsive layout
- `components/admin/SyncControlPanel.tsx` - Sync control buttons and entity selector
- `components/admin/SyncDashboard.tsx` - Sync status and history dashboard
- `components/admin/EntitySelector.tsx` - Checkbox component for entity selection
- `components/admin/SyncProgressBar.tsx` - Real-time progress indicator
- `components/admin/SyncHistoryTable.tsx` - Table displaying past sync operations
- `components/admin/SyncStatusBadge.tsx` - Status indicator badge component
### Utilities & Helpers
- `lib/utils/db-helpers.ts` - ✅ Database utility functions (bulk upsert, soft delete, sync stats, etc.)
- `lib/utils/sync-helpers.ts` - ✅ Sync utility functions (dependency ordering, entity helpers, filters)
- `lib/utils/entity-mapper.ts` - ✅ Map Autotask API responses to PostgreSQL schema format
- `lib/utils/logger.ts` - ✅ Structured logging utility for sync operations
- `lib/utils/api-helpers.ts` - ✅ API utilities for query parameter parsing, pagination, filtering, and error handling
- `lib/types/errors.ts` - ✅ Custom error types and error categorization utilities
### Testing
- `dev/test-postgres-connection.ts` - ✅ Test script for PostgreSQL connection and basic CRUD operations
- `dev/test-rate-limiter.ts` - ✅ Test script for rate limiter functionality with various scenarios
- `dev/test-full-sync.ts` - ✅ Test script for full sync with Autotask API (integration test)
- `dev/test-incremental-sync.ts` - ✅ Test script for incremental sync with modified records detection
- `dev/test-entity-specific-sync.ts` - ✅ Test script for entity-specific sync with dependency ordering
- `dev/ui-interaction-tests.md` - ✅ Comprehensive UI interaction test plan for admin sync interface
- `dev/ERROR_HANDLING_GUIDE.md` - ✅ Comprehensive guide for error handling and logging
### Notes
- Database migrations should be run automatically on PostgreSQL container startup via `/docker-entrypoint-initdb.d`
- Use existing `lib/services/autotask-client.ts` for Autotask API integration
- Use existing `lib/services/cache.ts` and `lib/services/redis-client.ts` for Redis caching
- All TypeScript files should include proper type definitions
- Follow existing project structure and naming conventions
- **Environment Configuration**: Both `.env` and `.env.local` are used:
- `.env` - Loaded by docker-compose for variable substitution in docker-compose.yml
- `.env.local` - Loaded by containers via `env_file` directive and by Next.js at runtime
- Keep both files in sync for PostgreSQL credentials
- **Autotask API Data Types**: The API returns some fields as strings that were initially assumed to be integers/decimals:
- `resource_type` returns "Employee", "Contractor" (not integer IDs)
- `travel_availability_pct` returns "up to 25%", "up to 50%" (not decimal percentages)
- Schema has been updated to accommodate actual API response formats
- **Autotask Query API**: Must use POST method (not GET) for `/query` endpoints with filter in request body
- **Date Range Filtering**: To manage rate limits and sync times, time-based entities (Tickets, Tasks, Projects, Billing Items) are limited by date range (default: 2 years). Users can adjust this in the admin UI from 1 year to "All Time" to sync historical data during off-hours.
## Tasks
- [ ] **1.0 Set up PostgreSQL infrastructure and database schema**
- [x] 1.1 Add PostgreSQL service to `docker-compose.yml` with health checks, volumes, and environment variables
- [x] 1.2 Create `.env.local` entries for PostgreSQL connection (host, port, database, user, password, DATABASE_URL)
- [x] 1.3 Update `.env.example` with PostgreSQL configuration documentation
- [x] 1.4 Create `migrations/` directory for SQL migration files
- [x] 1.5 Create `migrations/001_initial_schema.sql` with all 13 entity tables (companies, tickets, tasks, projects, resources, statuses, issue_types, sub_issue_types, work_types, billing_items, configuration_items, contacts, contracts)
- [x] 1.6 Add audit fields to each table (created_at, updated_at, synced_at, is_deleted, deleted_at)
- [x] 1.7 Create `sync_history` table with all required fields (id, entity_type, sync_type, status, started_at, completed_at, records_added, records_updated, records_deleted, error_message, triggered_by)
- [x] 1.8 Add foreign key constraints between related tables (tickets→companies, tasks→resources, configuration_items→companies, etc.)
- [x] 1.9 Create `migrations/002_add_indexes.sql` with indexes on foreign keys and frequently queried fields (company_id, assigned_resource_id, status, is_deleted)
- [x] 1.10 Test PostgreSQL container startup and migration execution
- [x] **2.0 Implement core sync service and Autotask API integration**
- [x] 2.1 Install required dependencies (`pg`, `@types/pg`) via npm
- [x] 2.2 Create `lib/services/postgres-client.ts` with connection pool setup and basic query methods
- [x] 2.3 Create `lib/types/sync.ts` with TypeScript interfaces for SyncConfig, SyncStatus, SyncHistory, EntityType enum
- [x] 2.4 Create `lib/types/database.ts` with TypeScript interfaces matching all database table schemas
- [x] 2.5 Create `lib/services/rate-limiter.ts` implementing 10 requests/second throttling with queue
- [x] 2.6 Create `lib/utils/entity-mapper.ts` to map Autotask API responses to PostgreSQL schema format
- [x] 2.7 Create `lib/utils/sync-helpers.ts` with dependency ordering function (companies first, then tickets/tasks/etc.)
- [x] 2.8 Create `lib/utils/db-helpers.ts` with upsert, soft delete, and bulk insert functions
- [x] 2.9 Test PostgreSQL connection and basic CRUD operations
- [x] 2.10 Test rate limiter with mock API calls
- [x] **3.0 Build sync operations (full, incremental, entity-specific)**
- [x] 3.1 Create `lib/services/sync-service.ts` with main sync orchestration class
- [x] 3.2 Implement `createSyncHistory()` method to create sync_history record with status 'started'
- [x] 3.3 Implement `updateSyncHistory()` method to update sync progress and status
- [x] 3.4 Create `lib/services/entity-sync.ts` with entity-specific sync methods for each of the 13 entities
- [x] 3.5 Implement `syncCompanies()` - fetch all companies from Autotask, upsert to PostgreSQL
- [x] 3.6 Implement `syncTickets()` - fetch all tickets, handle pagination, upsert with foreign keys
- [x] 3.7 Implement `syncTasks()` - fetch all tasks, handle pagination, upsert with foreign keys
- [x] 3.8 Implement `syncProjects()` - fetch all projects, upsert to PostgreSQL
- [x] 3.9 Implement `syncResources()` - fetch all resources (users), upsert to PostgreSQL
- [x] 3.10 Implement `syncConfigurationItems()` - fetch all config items, upsert with foreign keys
- [x] 3.11 Implement `syncContacts()` - fetch all contacts, upsert with company foreign keys
- [x] 3.12 Implement `syncContracts()` - fetch all contracts, upsert with company foreign keys
- [x] 3.13 Implement `syncBillingItems()` - fetch all billing items, upsert to PostgreSQL
- [x] 3.14 Implement `syncStatuses()` - fetch all status picklist values, upsert to PostgreSQL
- [x] 3.15 Implement `syncIssueTypes()` - fetch all issue type picklist values, upsert to PostgreSQL
- [x] 3.16 Implement `syncSubIssueTypes()` - fetch all sub-issue type picklist values, upsert to PostgreSQL
- [x] 3.17 Implement `syncWorkTypes()` - fetch all work type picklist values, upsert to PostgreSQL
- [x] 3.18 Integrate entity sync methods into main sync orchestration service
- [x] 3.19 Add error handling and logging for each sync operation
- [x] 3.20 Test full sync with small dataset from Autotask
- [x] 3.21 Test incremental sync with modified records
- [x] 3.22 Test entity-specific sync for individual entities
- [x] **4.0 Create admin UI for sync control and monitoring**
- [x] 4.1 Create `app/admin/sync/page.tsx` as main admin sync page layout
- [x] 4.2 Create `components/admin/EntitySelector.tsx` with checkboxes for all 13 entities
- [x] 4.3 Create `components/admin/SyncControlPanel.tsx` with Full Sync, Incremental Sync, and Sync Selected buttons
- [x] 4.4 Add sync mode toggle (full/incremental) to control panel for entity-specific syncs
- [x] 4.5 Create `components/admin/SyncDashboard.tsx` showing last sync time per entity with status badges
- [x] 4.6 Display total records synced (added, updated, deleted) in dashboard cards
- [x] 4.7 Create `components/admin/SyncProgressBar.tsx` showing real-time sync progress (optional for MVP)
- [x] 4.8 Create `components/admin/SyncStatusBadge.tsx` for status indicators (started, in_progress, completed, failed)
- [x] 4.9 Create `components/admin/SyncHistoryTable.tsx` with paginated sync history from sync_history table
- [x] 4.10 Add record count display (total, added, updated, deleted) to history table
- [x] 4.11 Implement auto-refresh (every 5 seconds) for dashboard during active sync
- [x] 4.12 Add confirmation dialog before triggering full sync
- [x] 4.13 Add download logs functionality (export as JSON/CSV)
- [x] 4.14 Style all components using TailwindCSS and shadcn/ui to match existing Pulse design
- [x] 4.15 Test UI responsiveness on desktop and tablet
- [x] 4.16 Test all user interactions (button clicks, entity selection, progress updates)
- [ ] **5.0 Implement API endpoints and data query layer**
- [x] 5.1 Create `app/api/sync/full/route.ts` - POST endpoint accepting optional entities array, returns syncId
- [x] 5.2 Create `app/api/sync/incremental/route.ts` - POST endpoint accepting optional entities array, returns syncId
- [x] 5.3 Create `app/api/sync/entity/route.ts` - POST endpoint to trigger entity-specific sync with entity array in body
- [x] 5.4 Create `app/api/sync/history/route.ts` - GET endpoint with pagination (page, limit) and entity filter
- [x] 5.5 Create `app/api/sync/last-sync/route.ts` - GET endpoint returning last sync timestamp per entity
- [x] 5.6 Create `app/api/data/companies/route.ts` - GET endpoint querying companies from PostgreSQL with pagination
- [x] 5.7 Create `app/api/data/tickets/route.ts` - GET endpoint querying tickets with filters, pagination, includeDeleted option
- [x] 5.8 Create `app/api/data/tasks/route.ts` - GET endpoint querying tasks with filters and pagination
- [x] 5.9 Create `app/api/data/configuration-items/route.ts` - GET endpoint querying config items with company filter
- [x] 5.10 Create `app/api/data/contacts/route.ts` - GET endpoint querying contacts with company filter
- [x] 5.11 Create `app/api/data/contracts/route.ts` - GET endpoint querying contracts with pagination
- [x] 5.12 Create `app/api/data/projects/route.ts` - GET endpoint querying projects with company filter
- [x] 5.13 Create `app/api/data/resources/route.ts` - GET endpoint querying resources (users)
- [x] 5.14 Create `app/api/data/billing-items/route.ts` - GET endpoint querying billing items with company filter
- [x] 5.15 Add query parameter support for all data endpoints (page, limit, includeDeleted, filters, sort, order)
- [x] 5.16 Implement default behavior to exclude soft-deleted records (is_deleted=false)
- [ ] 5.17 Add authentication/authorization checks to all sync and data endpoints
- [ ] 5.18 Test all API endpoints with Postman or similar tool
- [ ] 5.19 Test pagination, filtering, and sorting functionality
- [ ] **6.0 Add error handling, logging, and notifications**
- [ ] 6.1 Add comprehensive error logging to sync service (API errors, database errors, validation errors)
- [ ] 6.2 Log all errors to sync_history table with full error message and stack trace
- [ ] 6.3 Implement exponential backoff for Autotask API 429 (rate limit) responses
- [ ] 6.4 Add error context logging (request details, response details, SQL query context)
- [ ] 6.5 Implement toast notification component for success messages (using shadcn/ui toast)
- [ ] 6.6 Implement toast notification component for error messages with error summary
- [ ] 6.7 Add notification triggers in sync API endpoints (success/failure)
- [ ] 6.8 Store error logs for minimum 90 days (add cleanup job or retention policy)
- [ ] 6.9 Add Redis cache invalidation after successful sync (clear cached entities)
- [ ] 6.10 Implement cache invalidation for affected entities only (not all cache)
- [ ] 6.11 Update Redis cache strategy to use PostgreSQL data when available (fallback to API)
- [ ] 6.12 Add sync cancellation functionality (admin can cancel running sync)
- [ ] 6.13 Ensure partial sync progress is preserved on failure (committed transactions)
- [ ] 6.14 Test error handling with various failure scenarios (network timeout, auth failure, constraint violation)
- [ ] 6.15 Test notification display in UI for success and failure cases
- [ ] **7.0 Testing, documentation, and deployment**
- [ ] 7.1 Write integration tests for sync service with test database
- [ ] 7.2 Write unit tests for rate limiter functionality
- [ ] 7.3 Write unit tests for entity mapper and sync helpers
- [ ] 7.4 Write API endpoint tests for all sync and data routes
- [ ] 7.5 Test full sync with production-like data volume (1000+ records per entity)
- [ ] 7.6 Test incremental sync accuracy (verify only changed records are updated)
- [ ] 7.7 Test entity-specific sync with various entity combinations
- [ ] 7.8 Test soft delete functionality (verify records marked as deleted, not removed)
- [ ] 7.9 Test foreign key relationships and data integrity
- [ ] 7.10 Test rate limiting under high load (verify 10 req/sec limit)
- [ ] 7.11 Test sync cancellation and partial progress preservation
- [ ] 7.12 Perform load testing on PostgreSQL queries (verify <500ms response time)
- [ ] 7.13 Create README documentation for sync feature (setup, usage, troubleshooting)
- [ ] 7.14 Document all API endpoints with request/response examples
- [ ] 7.15 Document database schema and entity relationships (ERD diagram)
- [ ] 7.16 Create runbook for common sync issues and resolutions
- [ ] 7.17 Update main project README with PostgreSQL setup instructions
- [ ] 7.18 Build and test Docker containers locally
- [ ] 7.19 Deploy to staging environment and perform end-to-end testing
- [ ] 7.20 Deploy to production and monitor first sync operation
- [ ] 7.21 Set up monitoring/alerting for sync failures (optional webhook integration)
- [ ] 7.22 Create backup strategy for PostgreSQL data (automated backups)
uisng

77
tasks/tasks-prd-time-entries-analytics.md Normal file
View file

@ -0,0 +1,77 @@
## Relevant Files
- `migrations/006_add_time_entries_table.sql` - Database migration for Time Entries table with proper indexing and foreign keys
- `lib/types/database.ts` - TypeScript interfaces for Time Entries entity
- `lib/types/autotask.ts` - TypeScript interfaces for Autotask Time Entries API response
- `lib/services/autotask-client.ts` - Extended Autotask client with Time Entries API methods
- `lib/services/entity-sync.ts` - Sync service integration for Time Entries
- `lib/utils/entity-mapper.ts` - Entity mapping functions for Time Entries
- `app/api/data/time-entries/route.ts` - API endpoint for fetching Time Entries data
- `app/api/sync/entity/route.ts` - Updated sync endpoint to include Time Entries
- `lib/services/analytics-engine.ts` - Core analytics engine for scoring and analysis
- `lib/services/llm-analyzer.ts` - LLM integration for work pattern analysis
- `lib/utils/scoring-algorithms.ts` - Scoring algorithms for Activity, Content, and Timeliness scores
- `components/analytics/TimelineView.tsx` - Main timeline component with collapsible sections
- `components/analytics/ScoreCard.tsx` - Component for displaying individual and aggregate scores
- `components/analytics/AnalysisPanel.tsx` - Component for AI insights and recommendations
- `components/analytics/TimeEntriesDashboard.tsx` - Main dashboard page component
- `app/admin/analytics/time-entries/page.tsx` - Time Entries analytics page
- `lib/utils/time-helpers.ts` - Utility functions for time-based calculations and formatting
- `lib/utils/chart-helpers.ts` - Utility functions for chart data preparation
- `hooks/use-time-entries.ts` - React hook for Time Entries data fetching and state management
- `hooks/use-analytics.ts` - React hook for analytics calculations and scoring
### Notes
- Unit tests should typically be placed alongside the code files they are testing (e.g., `MyComponent.tsx` and `MyComponent.test.tsx` in the same directory).
- Use `npx jest [optional/path/to/test/file]` to run tests. Running without a path executes all tests found by the Jest configuration.
## Tasks
- [x] 1.0 Database Schema and Data Synchronization Setup
- [x] 1.1 Create Time Entries database migration with all required fields (id, resource_id, ticket_id, task_id, project_id, entry_date, hours_worked, notes, created_date, updated_date, etc.)
- [x] 1.2 Add proper indexing for time-based queries and foreign key relationships to existing tables
- [x] 1.3 Define TypeScript interfaces for Time Entries in database and Autotask API types
- [x] 1.4 Create entity mapping functions to convert Autotask Time Entries data to PostgreSQL schema
- [x] 1.5 Add Time Entries to the sync service configuration and entity types
- [x] 2.0 Time Entries API Integration and Sync Service
- [x] 2.1 Extend Autotask client to support Time Entries API endpoints (query, get by ID, pagination)
- [x] 2.2 Implement pagination handling for large historical Time Entries datasets
- [x] 2.3 Add error handling for API rate limits and data inconsistencies specific to Time Entries
- [x] 2.4 Create API endpoint for fetching Time Entries data with filtering and sorting capabilities
- [x] 2.5 Implement initial historical data import and ongoing real-time synchronization
- [x] 2.6 Add Time Entries sync to the existing sync service and entity sync methods
- [ ] 3.0 Analytics Engine and Scoring System Development
- [x] 3.1 Develop core analytics engine for processing Time Entries data and generating insights
- [x] 3.2 Implement Activity Score calculation based on entry quality, completeness, and work patterns
- [x] 3.3 Implement Content Score calculation based on time entry description quality and detail
- [x] 3.4 Implement Timeliness Score calculation based on entry timing relative to work performed
- [x] 3.5 Create LLM integration service for work pattern analysis and insight generation
- [x] 3.6 Implement caching for AI analysis results to improve performance and reduce API costs
- [x] 3.7 Add background processing for batch analysis of historical Time Entries data
- [x] 3.0 Analytics Engine and Scoring System Development
- [ ] 4.0 Timeline View and User Interface Components
- [x] 4.1 Create collapsible timeline component with multiple time range options (Hour, Day, Week, Month)
- [x] 4.2 Implement visual distinction between human and system activities with color coding
- [x] 4.3 Add key moment highlighting for ticket creation, status changes, and resolution events
- [x] 4.4 Create expandable/collapsible time period sections with smooth animations
- [x] 4.5 Implement time worked indicators and duration displays for each entry
- [x] 4.6 Create Score Card components for displaying individual and aggregate scores
- [x] 4.7 Build Analysis Panel component for AI insights and recommendations
- [x] 4.8 Implement responsive design for desktop and tablet viewing
- [ ] 5.0 Analytics Dashboard and Integration
- [x] 5.1 Create main Time Entries analytics dashboard page accessible from admin menu
- [x] 5.2 Implement filtering capabilities by resource, project, ticket, task, and activity type
- [x] 5.3 Build both detailed single-ticket views and summary dashboard views
- [x] 5.4 Add export capabilities for analysis results and reports (CSV, Excel, PDF)
- [x] 5.5 Integrate Time Entries analytics with existing entities for enriched analysis
- [x] 5.6 Create React hooks for Time Entries data fetching and analytics state management
- [x] 5.7 Add Time Entries analytics to the admin navigation and data browser if applicable
- [x] 5.8 Implement performance optimizations for large datasets and caching strategies
- [x] 5.0 Analytics Dashboard and Integration