iddaai-be/mds/archive/AI_CHANGELOG.md

AI Context: Deployment & Feeder Optimization Changelog

Date: 2026-01-12 Component: Backend / DevOps / Feeder Author: AI Agent (Antigravity)

1. Infrastructure & Deployment (EC2 & GitHub Actions)

🚀 Zero-to-Hero Deployment (deploy-feeder.yml)

• Automated Setup: The pipeline now handles full server provisioning (installing Node.js v20, Git, Docker, PM2) and repository cloning if not present.
• Private Repo Access: Switched to using GH_PAT (Personal Access Token) for git clone and git pull, resolving "Username not found" errors on private repositories.
• Secure Environment Management (Senior Approach):
  • Moved away from insecure/fragile .env manipulation (sed/cp).
  • Implemented dynamic .env generation from GitHub Secrets (DATABASE_URL, REDIS_HOST, JWT_SECRET).
  • Critical Config: DATABASE_URL is configured to localhost:15432 for the Host-based PM2 process to access Dockerized Postgres, while Docker containers use internal networking.

💾 Data Persistence

• Docker Volumes: Switched from named volumes to Bind Mounts:
  • Postgres: ./data/postgres:/var/lib/postgresql/data
  • Redis: ./data/redis:/data
• Result: Data persists directly on the EC2 host file system, surviving container recreation and allowing easier backups.

2. Feeder Service Optimization (feeder.service.ts)

⚡ Performance Tuning (Turbo Mode)

• Concurrency: Increased from 5 to 20 parallel requests.
• Request Delay: Reduced from 500ms to 50ms per batch.
• Throughput: ~5-10x speed improvement for historical data ingestion.

🧠 Enhancements

• Smart Resume: The service checks AppSetting to resume from the last successfully processed date.
• ETA Logging: Added real-time calculation logic:
  • Tracks AvgTimePerDay.
  • Projects RemainingTime based on remaining days.
  • Logs nice status: ⏱️ PROGRESS: [X days done] | Avg/Day: Ys | Remaining: Z days | 🏁 ETA: HH:MM:SS
• Clean Code: Removed unused variables (dayStartTime, totalDaysInRange) for better maintainability.

3. Stability & Persistence Fixes (feeder-persistence.service.ts)

🛡️ Race Condition Handling

• Country Upsert: Wrapped prisma.country.upsert in a try-catch block to silently ignore P2002 (Unique Constraint) errors. This fixes crashes caused by multiple parallel workers trying to create the same country simultaneously.

🧹 Data Deduplication

• Match Officials: Implemented in-memory deduplication (using Set) before insertion.
• Problem: Source data (Mackolik) sometimes lists the same official twice for a match, causing DB constraint failures.
• Fix: name + role combinations are checked, and duplicates are filtered out before hitting the database.

4. Current Architecture Overview

• App Runtime: PM2 (Host) -> Runs npm run feeder:historical.
• Database: Docker (Postgres 16) -> Mapped to Host 15432.
• Cache: Docker (Redis 7) -> Mapped to Host 6379.
• Flow: Feeder (Host) connects to -> localhost:15432 (Postgres) & localhost:6379 (Redis).

---

This document serves as a memory checkpoint for future context. Do not delete.

---

AI Context: Ransomware Saldırısı ve Güvenlik Sıkılaştırması

Date: 2026-01-16 Component: Security / Infrastructure / DevOps Author: AI Agent (Antigravity)

1. Olay Özeti

🚨 Ransomware Saldırısı Tespit Edildi

• Tarih: 15 Ocak 2026
• Sorun: PostgreSQL veritabanı (boilerplate_db) silindi
• Sebep: Port 15432 internete açıktı + default credentials (postgres/postgres)
• Fidye Notu: readme_to_recover database'inde Bitcoin talebi

🔍 Saldırı Vektörü

1. Saldırgan açık 15432 portunu taradı
2. Default postgres/postgres ile giriş yaptı
3. boilerplate_db silindi
4. readme_to_recover fidye notu bırakıldı

2. Uygulanan Güvenlik Önlemleri

A. Veritabanı Güvenliği

Önlem	Eski	Yeni
Kullanıcı	postgres	suggestbet
Şifre	postgres	SuGGesT2026SecuRe
Port 15432	Herkese açık	Sadece localhost

B. AWS Security Group

• ❌ Port 15432 kapatıldı
• ❌ Port 22 kapatıldı (SSH yerine SSM)
• ✅ Port 80/443 açık (Nginx)

C. SSM Session Manager

• SSH yerine AWS SSM kullanılıyor
• IAM Role: EC2-SSM-Role ile AmazonSSMManagedInstanceCore policy
• Port forwarding ile DBeaver bağlantısı (lokal port: 15432)

D. UFW Firewall (EC2)

# Aktif kurallar:
80/tcp      ALLOW
443/tcp     ALLOW
15432        ALLOW 127.0.0.1  # Sadece localhost

E. Docker Güvenliği (docker-compose.yml)

# Portlar sadece localhost'a bind
ports:
  - '127.0.0.1:15432:15432' # Postgres
  - '127.0.0.1:6379:6379' # Redis

F. Redis Güvenliği

• Şifre eklendi: RedisSecure2026

3. Dosya Değişiklikleri

Dosya	Değişiklik
docker-compose.yml	Yeni credentials, localhost-only binding
.env	Yeni DB user/password, Redis password
.github/workflows/deploy-feeder.yml	Database varlık kontrolü, Redis password
mds/SERVER_SECURITY_GUIDE.md	YENİ - Bağlantı kılavuzu

4. Bağlantı Yöntemi

DBeaver Bağlantısı (Mac/Windows)

# Terminal'de SSM port forwarding başlat:
dbconnect  # alias

# DBeaver ayarları:
# Host: localhost
# Port: 15432
# Database: boilerplate_db
# User: suggestbet
# Password: SuGGesT2026SecuRe

5. GitHub Secrets Güncellenmeli

Secret	Değer
DATABASE_URL	postgresql://suggestbet:SuGGesT2026SecuRe@localhost:15432/boilerplate_db?schema=public
REDIS_PASSWORD	RedisSecure2026

6. Kalan İşler

• GitHub Secrets güncelle
• Değişiklikleri push'la
• EC2'de docker compose up -d (yeni config için)
• Feeder'ı yeniden başlat

---

AI Context: Database Sample Export Scripts

Date: 2026-01-16 Component: Developer Tools / AI Context Author: AI Agent (Antigravity)

Amaç

AI asistanların veritabanı yapısını ve içeriğini anlaması için örnek veri export script'leri oluşturuldu.

Dosyalar

Dosya	Platform	Açıklama
scripts/export-db-samples.sh	Mac/Linux	Bash script
scripts/export-db-samples.ps1	Windows	PowerShell script
mds/DATABASE_SAMPLES.md	-	Oluşturulan çıktı dosyası

Kullanım

Mac/Linux

# 1. SSM port forwarding başlat
dbconnect

# 2. Yeni terminal'de script çalıştır
bash scripts/export-db-samples.sh

Windows

# 1. SSM port forwarding başlat
dbconnect

# 2. Yeni PowerShell'de script çalıştır
.\scripts\export-db-samples.ps1

Çıktı

Script şu bilgileri mds/DATABASE_SAMPLES.md dosyasına yazar:

• Tüm tabloların kayıt sayıları
• Her önemli tablodan 5-10 örnek kayıt (JSON formatında)
• Matches, Leagues, Teams, Countries, Predictions, Stats, Odds

Ne Zaman Çalıştırılmalı

• Yeni AI oturumu başlamadan önce
• Veritabanı yapısı değiştiğinde
• Önemli veri değişikliklerinden sonra

---

---

AI Context: V20 Ensemble & Feeder Optimization

Date: 2026-02-08 Component: AI Engine / Data Feeder / Stability Author: AI Agent (Antigravity)

1. V20 Ensemble "Beast" Deployment

• Architecture: Synthesis of 4 engines (Team, Player, Odds, Referee).
• Surprise Detection: Added UpsetEngine to track motivation and position-based risks.
• Enhanced Predictions: Added xG (Expected Goals), Top 5 correct scores, and Smart Value recommendations.

2. Core Stability Patches

• Null-Safety: Exhaustive is not None checks added to ContextEngine, UpsetEngine, and V20EnsemblePredictor to prevent crashes when standings/stats are missing.
• Environment Parity: Patched hardcoded production IPs (13.49.226.80) with localhost across all AI sub-engines via patch-ips.js.

3. Feeder & Data Fetching

• Top Leagues Filter: Implementation of top_leagues.json reduced processing load by ~85% (~160 matches vs 1200+).
• Lineup Coverage: Expanded fetch window (4h pre-match, 3h post-match) ensures 11-man starting lineups (XI) are captured for major leagues.
• Retry Logic: Added 502/Timeout handling in DataFetcherTask for resilient data ingestion.

---

This document serves as a memory checkpoint. For deep technical details, see mds/V20_AI_ENGINE_AND_FEEDER_EVOLUTION.md.