feat: add place gate population scripts, map style assets, automated cron sync tasks, and expand dashboard and landing applications

This commit is contained in:
Hamza-Ayed
2026-09-19 21:22:05 +03:00
parent 82b1045f9c
commit ce31d79ba6
76 changed files with 118969 additions and 115428 deletions
+70
View File
@@ -0,0 +1,70 @@
# SIRO Maps — Crontab & Scheduled Jobs Directory
# دليل وجدولة المهام الدورية والتشغيل الآلي لمنصة سيرو
This directory contains the central crontab configuration, installer, and documentation for all scheduled background jobs running on the SIRO Maps server.
---
## 📋 Overview of Scheduled Jobs
| # | Task Name | Schedule | Script / Command | Log File | Description |
|---|-----------|----------|-------------------|----------|-------------|
| 1 | **Bi-Monthly Geospatial Sync** | Every 1st & 15th at 03:00 | `scripts/master_cron_sync.sh` | `logs/cron_places_sync.log` | Syncs OSM & Overture places for JO, IQ, EG, SY. Generates and classifies gates & entrances for complexes. Refreshes `unified_search_index` concurrently and flushes Redis cache. |
| 2 | **Road Network & Routing Sync** | Every 1st, 10th, 20th at 01:00 | `scripts/update-data.sh` | `logs/cron_osm_update.log` | Downloads fresh OSM PBF for Jordan & Syria, updates PostGIS road tables, and rebuilds GraphHopper routing index. |
| 3 | **Daily Database Backup** | Daily at 02:00 | `scripts/backup_db.sh` | `logs/cron_backup.log` | Backs up custom tables (`places_*`, `place_gates`, `users`, `api_keys`, `billing_*`, `telemetry_*`) with 7-day retention. |
| 4 | **Weekly DB Vacuum & Analyze** | Every Sunday at 04:00 | `docker compose exec db vacuumdb` | `logs/cron_vacuum.log` | Reclaims space and updates PostgreSQL query planner statistics for fast spatial and trigram GIN queries. |
| 5 | **Weekly Maintenance & Log Rotation** | Every Sunday at 05:00 | `scripts/cleanup_logs.sh` | `logs/cron_cleanup.log` | Truncates logs exceeding 50MB, deletes logs older than 14 days, and prunes dangling Docker images. |
---
## 🚀 Quick Setup on Any Server / Migration Guide
When setting up a new server or migrating the platform:
```bash
# 1. Navigate to the app directory
cd /home/hamzadoctor/app
# 2. Run the Crontab Installer
bash crontab/install_cron.sh
```
The installer will automatically:
1. Create required directories (`logs/` and `backups/db/`).
2. Make all scripts in `scripts/` executable (`chmod +x`).
3. Load all jobs from `crontab/crontab.txt` into the server's crontab.
---
## 🔍 How to Monitor & Inspect Logs
```bash
# View active crontab entries
crontab -l
# Watch live output of places & gates sync
tail -f /home/hamzadoctor/app/logs/cron_places_sync.log
# Check recent database backups
ls -lh /home/hamzadoctor/app/backups/db/
# Check road network update log
tail -n 100 /home/hamzadoctor/app/logs/cron_osm_update.log
```
---
## ⚡ Manual Execution (Testing)
You can manually trigger any of these jobs at any time without waiting for the cron schedule:
```bash
# Run Bi-Monthly Places & Gates Sync
/bin/bash /home/hamzadoctor/app/scripts/master_cron_sync.sh
# Run Database Backup
/bin/bash /home/hamzadoctor/app/scripts/backup_db.sh
# Run Log & Docker Maintenance
/bin/bash /home/hamzadoctor/app/scripts/cleanup_logs.sh
```
+45
View File
@@ -0,0 +1,45 @@
# ==============================================================================
# SIRO Maps — Sovereign Geospatial Platform Crontab Schedule
# ==============================================================================
# Base Directory: /home/hamzadoctor/app
# Environment: Production
# ==============================================================================
SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
APP_DIR=/home/hamzadoctor/app
# ------------------------------------------------------------------------------
# 1. Bi-Monthly Geospatial Synchronization (Every 1st & 15th at 03:00 AM)
# - Syncs OSM & Overture places for Jordan, Iraq, Egypt, Syria
# - Generates and classifies Gates & Entrances (Hospitals, Malls, Unis, Hotels, Parks)
# - Refreshes Materialized View (unified_search_index) concurrently
# - Flushes Redis search cache
# ------------------------------------------------------------------------------
0 3 1,15 * * /bin/bash /home/hamzadoctor/app/scripts/master_cron_sync.sh >> /home/hamzadoctor/app/logs/cron_places_sync.log 2>&1
# ------------------------------------------------------------------------------
# 2. Road Network & Routing Update (Every 10 days at 01:00 AM)
# - Downloads fresh OSM PBF for Jordan & Syria
# - Rebuilds GraphHopper routing index & clears traffic cache
# ------------------------------------------------------------------------------
0 1 1,10,20 * * /bin/bash /home/hamzadoctor/app/scripts/update-data.sh >> /home/hamzadoctor/app/logs/cron_osm_update.log 2>&1
# ------------------------------------------------------------------------------
# 3. Daily Database Backup (Every Day at 02:00 AM)
# - Exports custom places, place_gates, billing, users & telemetry tables
# - Retains backups for 7 days
# ------------------------------------------------------------------------------
0 2 * * * /bin/bash /home/hamzadoctor/app/scripts/backup_db.sh >> /home/hamzadoctor/app/logs/cron_backup.log 2>&1
# ------------------------------------------------------------------------------
# 4. Weekly PostgreSQL Optimization (Every Sunday at 04:00 AM)
# - Runs VACUUM ANALYZE across heavy spatial tables and indexes
# ------------------------------------------------------------------------------
0 4 * * 0 docker compose -f /home/hamzadoctor/app/docker-compose.yml exec -T db vacuumdb -U mapuser -d mapdb --analyze-in-stages >> /home/hamzadoctor/app/logs/cron_vacuum.log 2>&1
# ------------------------------------------------------------------------------
# 5. Weekly Log Rotation & Docker Maintenance (Every Sunday at 05:00 AM)
# - Truncates logs > 50MB and removes logs older than 14 days
# - Prunes dangling Docker images and builder caches
# ------------------------------------------------------------------------------
0 5 * * 0 /bin/bash /home/hamzadoctor/app/scripts/cleanup_logs.sh >> /home/hamzadoctor/app/logs/cron_cleanup.log 2>&1
+32
View File
@@ -0,0 +1,32 @@
#!/bin/bash
# ==============================================================================
# SIRO Maps — Crontab Installer Script
# ==============================================================================
# Installs all scheduled jobs from crontab.txt into the current user's crontab.
# Ensures all target scripts have executable permissions and directories exist.
# ==============================================================================
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
APP_DIR="$(dirname "$SCRIPT_DIR")"
CRON_FILE="${SCRIPT_DIR}/crontab.txt"
echo "🔧 Installing SIRO Maps Crontab..."
echo "📂 App Directory: $APP_DIR"
echo "📄 Crontab File: $CRON_FILE"
# 1. Ensure required directories exist
mkdir -p "${APP_DIR}/logs"
mkdir -p "${APP_DIR}/backups/db"
# 2. Make all scripts executable
chmod +x "${APP_DIR}/scripts/"*.sh 2>/dev/null || true
# 3. Load crontab
crontab "$CRON_FILE"
echo "✅ Crontab successfully installed!"
echo "📋 Active Crontab Entries:"
echo "------------------------------------------------------------------------------"
crontab -l
echo "------------------------------------------------------------------------------"