259 lines
9.1 KiB
Markdown
Executable File
259 lines
9.1 KiB
Markdown
Executable File
# AI Context: Deployment & Feeder Optimization Changelog
|
||
|
||
**Date:** 2026-01-12
|
||
**Component:** Backend / DevOps / Feeder
|
||
**Author:** AI Agent (Antigravity)
|
||
|
||
## 1. Infrastructure & Deployment (EC2 & GitHub Actions)
|
||
|
||
### 🚀 Zero-to-Hero Deployment (`deploy-feeder.yml`)
|
||
|
||
- **Automated Setup:** The pipeline now handles full server provisioning (installing Node.js v20, Git, Docker, PM2) and repository cloning if not present.
|
||
- **Private Repo Access:** Switched to using `GH_PAT` (Personal Access Token) for `git clone` and `git pull`, resolving "Username not found" errors on private repositories.
|
||
- **Secure Environment Management (Senior Approach):**
|
||
- Moved away from insecure/fragile `.env` manipulation (sed/cp).
|
||
- Implemented dynamic `.env` generation from **GitHub Secrets** (`DATABASE_URL`, `REDIS_HOST`, `JWT_SECRET`).
|
||
- **Critical Config:** `DATABASE_URL` is configured to `localhost:15432` for the Host-based PM2 process to access Dockerized Postgres, while Docker containers use internal networking.
|
||
|
||
### 💾 Data Persistence
|
||
|
||
- **Docker Volumes:** Switched from named volumes to **Bind Mounts**:
|
||
- Postgres: `./data/postgres:/var/lib/postgresql/data`
|
||
- Redis: `./data/redis:/data`
|
||
- **Result:** Data persists directly on the EC2 host file system, surviving container recreation and allowing easier backups.
|
||
|
||
## 2. Feeder Service Optimization (`feeder.service.ts`)
|
||
|
||
### ⚡ Performance Tuning (Turbo Mode)
|
||
|
||
- **Concurrency:** Increased from `5` to **`20`** parallel requests.
|
||
- **Request Delay:** Reduced from `500ms` to **`50ms`** per batch.
|
||
- **Throughput:** ~5-10x speed improvement for historical data ingestion.
|
||
|
||
### 🧠 Enhancements
|
||
|
||
- **Smart Resume:** The service checks `AppSetting` to resume from the last successfully processed date.
|
||
- **ETA Logging:** Added real-time calculation logic:
|
||
- Tracks `AvgTimePerDay`.
|
||
- Projects `RemainingTime` based on remaining days.
|
||
- Logs nice status: `⏱️ PROGRESS: [X days done] | Avg/Day: Ys | Remaining: Z days | 🏁 ETA: HH:MM:SS`
|
||
- **Clean Code:** Removed unused variables (`dayStartTime`, `totalDaysInRange`) for better maintainability.
|
||
|
||
## 3. Stability & Persistence Fixes (`feeder-persistence.service.ts`)
|
||
|
||
### 🛡️ Race Condition Handling
|
||
|
||
- **Country Upsert:** Wrapped `prisma.country.upsert` in a `try-catch` block to silently ignore `P2002` (Unique Constraint) errors. This fixes crashes caused by multiple parallel workers trying to create the same country simultaneously.
|
||
|
||
### 🧹 Data Deduplication
|
||
|
||
- **Match Officials:** Implemented in-memory deduplication (using `Set`) before insertion.
|
||
- **Problem:** Source data (Mackolik) sometimes lists the same official twice for a match, causing DB constraint failures.
|
||
- **Fix:** `name + role` combinations are checked, and duplicates are filtered out before hitting the database.
|
||
|
||
## 4. Current Architecture Overview
|
||
|
||
- **App Runtime:** `PM2` (Host) -> Runs `npm run feeder:historical`.
|
||
- **Database:** `Docker` (Postgres 16) -> Mapped to Host `15432`.
|
||
- **Cache:** `Docker` (Redis 7) -> Mapped to Host `6379`.
|
||
- **Flow:** Feeder (Host) connects to -> localhost:15432 (Postgres) & localhost:6379 (Redis).
|
||
|
||
---
|
||
|
||
_This document serves as a memory checkpoint for future context. Do not delete._
|
||
|
||
---
|
||
|
||
# AI Context: Ransomware Saldırısı ve Güvenlik Sıkılaştırması
|
||
|
||
**Date:** 2026-01-16
|
||
**Component:** Security / Infrastructure / DevOps
|
||
**Author:** AI Agent (Antigravity)
|
||
|
||
## 1. Olay Özeti
|
||
|
||
### 🚨 Ransomware Saldırısı Tespit Edildi
|
||
|
||
- **Tarih:** 15 Ocak 2026
|
||
- **Sorun:** PostgreSQL veritabanı (`boilerplate_db`) silindi
|
||
- **Sebep:** Port 15432 internete açıktı + default credentials (`postgres/postgres`)
|
||
- **Fidye Notu:** `readme_to_recover` database'inde Bitcoin talebi
|
||
|
||
### 🔍 Saldırı Vektörü
|
||
|
||
1. Saldırgan açık 15432 portunu taradı
|
||
2. Default `postgres/postgres` ile giriş yaptı
|
||
3. `boilerplate_db` silindi
|
||
4. `readme_to_recover` fidye notu bırakıldı
|
||
|
||
## 2. Uygulanan Güvenlik Önlemleri
|
||
|
||
### A. Veritabanı Güvenliği
|
||
|
||
| Önlem | Eski | Yeni |
|
||
| --------- | ------------ | ------------------- |
|
||
| Kullanıcı | `postgres` | `suggestbet` |
|
||
| Şifre | `postgres` | `SuGGesT2026SecuRe` |
|
||
| Port 15432 | Herkese açık | Sadece localhost |
|
||
|
||
### B. AWS Security Group
|
||
|
||
- ❌ Port 15432 **kapatıldı**
|
||
- ❌ Port 22 **kapatıldı** (SSH yerine SSM)
|
||
- ✅ Port 80/443 açık (Nginx)
|
||
|
||
### C. SSM Session Manager
|
||
|
||
- SSH yerine AWS SSM kullanılıyor
|
||
- IAM Role: `EC2-SSM-Role` ile `AmazonSSMManagedInstanceCore` policy
|
||
- Port forwarding ile DBeaver bağlantısı (lokal port: 15432)
|
||
|
||
### D. UFW Firewall (EC2)
|
||
|
||
```bash
|
||
# Aktif kurallar:
|
||
80/tcp ALLOW
|
||
443/tcp ALLOW
|
||
15432 ALLOW 127.0.0.1 # Sadece localhost
|
||
```
|
||
|
||
### E. Docker Güvenliği (`docker-compose.yml`)
|
||
|
||
```yaml
|
||
# Portlar sadece localhost'a bind
|
||
ports:
|
||
- '127.0.0.1:15432:15432' # Postgres
|
||
- '127.0.0.1:6379:6379' # Redis
|
||
```
|
||
|
||
### F. Redis Güvenliği
|
||
|
||
- Şifre eklendi: `RedisSecure2026`
|
||
|
||
## 3. Dosya Değişiklikleri
|
||
|
||
| Dosya | Değişiklik |
|
||
| ------------------------------------- | ---------------------------------------- |
|
||
| `docker-compose.yml` | Yeni credentials, localhost-only binding |
|
||
| `.env` | Yeni DB user/password, Redis password |
|
||
| `.github/workflows/deploy-feeder.yml` | Database varlık kontrolü, Redis password |
|
||
| `mds/SERVER_SECURITY_GUIDE.md` | **YENİ** - Bağlantı kılavuzu |
|
||
|
||
## 4. Bağlantı Yöntemi
|
||
|
||
### DBeaver Bağlantısı (Mac/Windows)
|
||
|
||
```bash
|
||
# Terminal'de SSM port forwarding başlat:
|
||
dbconnect # alias
|
||
|
||
# DBeaver ayarları:
|
||
# Host: localhost
|
||
# Port: 15432
|
||
# Database: boilerplate_db
|
||
# User: suggestbet
|
||
# Password: SuGGesT2026SecuRe
|
||
```
|
||
|
||
## 5. GitHub Secrets Güncellenmeli
|
||
|
||
| Secret | Değer |
|
||
| ---------------- | --------------------------------------------------------------------------------------- |
|
||
| `DATABASE_URL` | `postgresql://suggestbet:SuGGesT2026SecuRe@localhost:15432/boilerplate_db?schema=public` |
|
||
| `REDIS_PASSWORD` | `RedisSecure2026` |
|
||
|
||
## 6. Kalan İşler
|
||
|
||
- [x] GitHub Secrets güncelle
|
||
- [x] Değişiklikleri push'la
|
||
- [ ] EC2'de `docker compose up -d` (yeni config için)
|
||
- [ ] Feeder'ı yeniden başlat
|
||
|
||
---
|
||
|
||
# AI Context: Database Sample Export Scripts
|
||
|
||
**Date:** 2026-01-16
|
||
**Component:** Developer Tools / AI Context
|
||
**Author:** AI Agent (Antigravity)
|
||
|
||
## Amaç
|
||
|
||
AI asistanların veritabanı yapısını ve içeriğini anlaması için örnek veri export script'leri oluşturuldu.
|
||
|
||
## Dosyalar
|
||
|
||
| Dosya | Platform | Açıklama |
|
||
| ------------------------------- | --------- | ------------------------- |
|
||
| `scripts/export-db-samples.sh` | Mac/Linux | Bash script |
|
||
| `scripts/export-db-samples.ps1` | Windows | PowerShell script |
|
||
| `mds/DATABASE_SAMPLES.md` | - | Oluşturulan çıktı dosyası |
|
||
|
||
## Kullanım
|
||
|
||
### Mac/Linux
|
||
|
||
```bash
|
||
# 1. SSM port forwarding başlat
|
||
dbconnect
|
||
|
||
# 2. Yeni terminal'de script çalıştır
|
||
bash scripts/export-db-samples.sh
|
||
```
|
||
|
||
### Windows
|
||
|
||
```powershell
|
||
# 1. SSM port forwarding başlat
|
||
dbconnect
|
||
|
||
# 2. Yeni PowerShell'de script çalıştır
|
||
.\scripts\export-db-samples.ps1
|
||
```
|
||
|
||
## Çıktı
|
||
|
||
Script şu bilgileri `mds/DATABASE_SAMPLES.md` dosyasına yazar:
|
||
|
||
- Tüm tabloların kayıt sayıları
|
||
- Her önemli tablodan 5-10 örnek kayıt (JSON formatında)
|
||
- Matches, Leagues, Teams, Countries, Predictions, Stats, Odds
|
||
|
||
## Ne Zaman Çalıştırılmalı
|
||
|
||
- Yeni AI oturumu başlamadan önce
|
||
- Veritabanı yapısı değiştiğinde
|
||
- Önemli veri değişikliklerinden sonra
|
||
|
||
---
|
||
|
||
---
|
||
|
||
# AI Context: V20 Ensemble & Feeder Optimization
|
||
|
||
**Date:** 2026-02-08
|
||
**Component:** AI Engine / Data Feeder / Stability
|
||
**Author:** AI Agent (Antigravity)
|
||
|
||
## 1. V20 Ensemble "Beast" Deployment
|
||
|
||
- **Architecture:** Synthesis of 4 engines (Team, Player, Odds, Referee).
|
||
- **Surprise Detection:** Added `UpsetEngine` to track motivation and position-based risks.
|
||
- **Enhanced Predictions:** Added xG (Expected Goals), Top 5 correct scores, and Smart Value recommendations.
|
||
|
||
## 2. Core Stability Patches
|
||
|
||
- **Null-Safety:** Exhaustive `is not None` checks added to `ContextEngine`, `UpsetEngine`, and `V20EnsemblePredictor` to prevent crashes when standings/stats are missing.
|
||
- **Environment Parity:** Patched hardcoded production IPs (`13.49.226.80`) with `localhost` across all AI sub-engines via `patch-ips.js`.
|
||
|
||
## 3. Feeder & Data Fetching
|
||
|
||
- **Top Leagues Filter:** Implementation of `top_leagues.json` reduced processing load by ~85% (~160 matches vs 1200+).
|
||
- **Lineup Coverage:** Expanded fetch window (4h pre-match, 3h post-match) ensures 11-man starting lineups (XI) are captured for major leagues.
|
||
- **Retry Logic:** Added 502/Timeout handling in `DataFetcherTask` for resilient data ingestion.
|
||
|
||
---
|
||
|
||
_This document serves as a memory checkpoint. For deep technical details, see [mds/V20_AI_ENGINE_AND_FEEDER_EVOLUTION.md](file:///c:/Users/fahri/Documents/GitHub/Suggest-Bet-BE/mds/V20_AI_ENGINE_AND_FEEDER_EVOLUTION.md)._
|
||
|