Production-grade self-healing observability & ML orchestration pipeline for distributed grids.
A production-grade, containerized Self-Healing Observability Console and Orchestration Pipeline designed for distributed high-performance grid environments (such as the Worldwide LHC Computing Grid - WLCG). The system integrates a machine learning engine running an unsupervised Isolation Forest model on normalized telemetry metrics to detect subtle, slow-burning anomalies (like memory leaks) while ignoring geomagnetic solar storm spikes driven by the NASA Space Weather API. Leveraging an automated closed-loop remediation flow, the control plane coordinates dynamic configuration classification via a Puppet External Node Classifier (ENC) endpoint, triggers asynchronous container recovery playbooks via Ansible, and features a Kubernetes-style Horizontal Pod Autoscaler (HPA) to scale and balance worker nodes dynamically under space weather load surges.
Sub-10ms Isolation Forest anomaly inference execution speed; sub-250ms real-time dashboard state polling update; sub-2s automated Ansible recovery trigger reaction time.
60 FPS responsive animations utilizing Vanilla CSS/JS client-side polling, showing live-updating node cards, active grid jobs, and solar weather multipliers. Structured around modern HTML5 semantic elements (<header>, <main>, <section>).
$0/month. Operates entirely within local Docker-Compose environments, or can be deployed on free-tier virtual machines using open-source infrastructure tools (PostgreSQL, Redis, Ansible, Puppet) with zero maintenance fees.
Architected a Closed-Loop Self-Healing System integrating Scikit-Learn Isolation Forest outlier detection with Django, dynamically triggering Ansible recovery playbooks asynchronously to reboot degraded worker nodes and drop memory leaks back to baseline.
Engineered a NASA DONKI Space Weather Driver to simulate geomagnetic solar storm workload surges, dynamically computing stress multipliers to scale simulated CPU, network, and active job loads across worker agents.
Designed a Telemetry Normalization Pipeline dividing raw metrics (CPU, network, active jobs) by the environmental stress multiplier before ML classifier feeding, reducing space-weather false positives to zero.
Implemented a Kubernetes-style Horizontal Pod Autoscaler (HPA) inside the Django control plane, dynamically provisioning and de-provisioning virtual worker nodes (grid-worker-03 to grid-worker-06) to balance stress workloads.
Exhaustive situation-action-result breakdowns showcasing problem-solving and architectural execution.
During solar storm workload surges simulated via the NASA DONKI API, increased CPU and network usage naturally spikes across all workers, causing the Isolation Forest model to misidentify healthy load peaks as anomalies and trigger catastrophic false-positive container reboots.
Designed a telemetry normalization layer that divides raw CPU usage, network activity, and jobs by the current solar weather stress multiplier before training and classifying telemetry, while retaining baseline metrics on raw RAM usage.
Low-level component relationships, system boundaries, and runtime flows.
The Django control plane polls NASA's Space Weather API (DONKI) to fetch active solar events and compute a dynamic stress multiplier. Worker agents boot up, register themselves via the Puppet ENC catalog endpoint, and begin posting telemetry metrics. The control plane filters and normalizes these metrics before processing them through an unsupervised Isolation Forest model (scikit-learn). When the ML engine flags an anomaly, the node is marked degraded, triggering an asynchronous Ansible simulation (recover_node.yml). In parallel, a Horizontal Pod Autoscaler (HPA) dynamically provisions new workers to distribute heavy space weather loads, all visualized in a real-time glassmorphic dashboard.
Normalizing dynamic environmental variables (such as solar weather loads) at the ingestion layer is critical for training robust unsupervised anomaly detection models, preventing normal system peaks from polluting the training window and triggering cascade failure alerts.
Rendered live in real-time. Direct URL: https://hoodieylya13-grid-sentinel.hf.space
Created a Dynamic Configuration Catalog classifier endpoint simulating Puppet External Node Classifier (ENC) returning custom node role parameters in structured YAML.
Built a Glassmorphic Observability Console in Django and Vanilla CSS/JS featuring real-time cluster status visualizations, active job tracking, and auto-updating event logs with sub-250ms polling latency.
Successfully isolated the latent memory leak on grid-worker-02 while keeping false-positive reboots on healthy worker nodes at exactly zero during high solar storm stress events.
Heavy geomagnetic storms spike the stress multiplier to 3.0x, overloading the fixed two-worker cluster and leading to resource exhaustion, job degradation, and potential cascading node failures.
Implemented a Kubernetes-style Horizontal Pod Autoscaler (HPA) that dynamically provisions and bootstraps new worker nodes (up to six total) when stress exceeds 1.2x, while modifying worker agents to query the active worker count and scale down their load by a factor of 2 / active_workers.
Balanced workloads evenly across the dynamically scaled cluster, keeping average CPU loads within safe operating bounds and preventing cascade failure under maximum solar storm spikes.
A slow-burning, linear memory leak of +4.5% per check-in loop on grid-worker-02 slips past traditional static alert thresholds during low-utilization periods, eventually leading to silent system unresponsiveness and job loss.
Trained an unsupervised Scikit-Learn Isolation Forest model on a sliding window of the last 200 normal telemetry points, dynamically isolating outliers near the root of the tree, backed by a deterministic SRE defense-in-depth safety rail (RAM > 90% or CPU > 95%).
Detected and flagged the memory leak within 12 check-in loops before node crash, triggering the Ansible reboot playbook to restore the container to healthy baseline parameters automatically.
