📋 Summary
Replace the current broad market scanning approach (4000+ stocks) with a focused watchlist-based system where users define a specific list of stocks to monitor. Current scanner settings should become temporary dashboard filters instead of database-stored configuration.
🎯 Problem Statement
The current stock scanner:
Scans 4000+ stocks from the entire market, causing slow performance (30-60 seconds per scan)
Uses extensive database storage for scanner settings that could be temporary filters
May hit API rate limits due to volume of requests
Provides too much data when users typically focus on specific stocks
💡 Proposed Solution
Phase 1: Watchlist Implementation
Replace scanner_settings table with a simplified user_watchlist table:
Update Scanner Service (services/scanner/main.py):
Modify execute_stock_scan() to only scan watchlist tickers
Remove complex market-wide scanning logic
Focus on specific ticker data retrieval from Alpaca API
Update Settings Page (services/web/templates/settings.html):
Replace current scanner criteria with watchlist management interface
Add/remove tickers functionality
Import/export watchlist capabilities
Phase 2: Dashboard Filter Migration
Convert existing settings to dashboard filters:
Move min_price, max_price, min_volume, min_premarket_gain, min_ai_score to client-side filters
Remove database storage for these temporary filter criteria
Store filter preferences in browser localStorage
Update Dashboard (services/web/templates/dashboard.html):
Enhance existing filter section with migrated scanner settings
Add filter presets (e.g., "Penny Stocks", "High Volume", "AI Favorites")
Maintain current sorting and pagination functionality
🏗️ Implementation Details
Database Changes
API Changes
New endpoints:
GET /api/watchlist - Get current watchlist
POST /api/watchlist - Add ticker to watchlist
DELETE /api/watchlist/{ticker} - Remove ticker from watchlist
Modified endpoints:
POST /api/run_scan - Scan only watchlist tickers
GET /stocks/today - Return only watchlist results
Performance Benefits
Scan time: 30-60 seconds → 5-10 seconds (for 20-50 watchlist stocks)
Database storage: Reduced by removing settings table
API calls: 4000+ requests → 20-50 requests per scan
Memory usage: Significantly reduced data processing
🎨 UI/UX Changes
Settings Page Mockup
Dashboard Filter Enhancement
📁 Files to Modify
Core Files
settings.py - Remove ScannerSettings model
init.py - Add UserWatchlist model
main.py - Update scanning logic
main.py - Update settings routes
settings.html - Watchlist UI
dashboard.html - Enhanced filters
Database Migration
create_tables.py - Update table creation
Create migration script for existing data
🧪 Testing Strategy
Unit Tests: Update test_scanner_api.py
Integration Tests: Verify watchlist scanning with real API
UI Tests: Test watchlist management and filter functionality
Performance Tests: Measure scan time improvements
🚀 Deployment Plan
Phase 1: Backend (1-2 weeks)
Create watchlist model and database migration
Update scanner service to use watchlist
Implement watchlist API endpoints
Test with small watchlist (5-10 stocks)
Phase 2: Frontend (1 week)
Build watchlist management UI
Convert settings to dashboard filters
Add filter presets and localStorage persistence
Update documentation
Phase 3: Migration (3-5 days)
Create data migration script for existing users
Deploy to staging environment
Performance testing and optimization
Production deployment
🔄 Migration Path
For existing users:
Export current scanner settings as default filter values
Create starter watchlist from recent stock_candidates table
Preserve user preferences in localStorage
Provide import functionality for popular stock lists (S&P 500, NASDAQ 100, etc.)
📊 Success Metrics
Scan Performance: < 10 seconds for watchlist of 50 stocks
User Adoption: Watchlist usage > 80% of active users
API Efficiency: 95% reduction in unnecessary API calls
User Satisfaction: Positive feedback on focused results
🏷️ Labels
enhancement performance database scanner-service web-service breaking-change
📋 Acceptance Criteria
Users can add/remove stocks from a personal watchlist
Scanner only processes watchlist stocks (not entire market)
Dashboard filters work on watchlist results
Scan time reduced to < 10 seconds for typical watchlist
Settings page shows watchlist management instead of scanner config
All existing functionality preserved for watchlist stocks
Database migration completed without data loss
Documentation updated with new workflow