Reinforcement Learning Lab

Train the reward policy on real historical irradiance (Open-Meteo archive → pvlib digital twin)

Open-Meteo Archive· historical irradiancepvlib· digital twinREINFORCE· policy gradientgymnasium· RL environment

Formulas used

RL reward functionMODEL_LEARNED

reward_t = bonus_t − penalty_t
return G_t = Σ_{k=0}^{T} γ^k × reward_{t+k}
# γ = discount rate (0.99 default)

bonus_t — Surplus reward at hour t (₹/kWh × surplus_kWh)

penalty_t — Shortfall charge at hour t (₹/kWh × deficit_kWh)

REINFORCE policy gradientOFFICIAL_SOURCE · REINFORCE

∇θ J(θ) = E_{τ ~ π_θ}
  [ Σ_{t=0}^{T} ∇θ log π_θ(a_t | s_t) × G_t ]
# θ ← θ + α × ∇θ J

π_θ(a|s) — Policy network (action probabilities given state)

α — Learning rate

G_t — Discounted return from time t

Policy state spaceMODEL_LEARNED

state = {
  hour_of_day,      # 0-23
  ghi_w_m2,         # current irradiance
  cloud_cover_pct,  # cloud conditions
  forecast_conf,    # 0.4–0.99
  current_dev_pct   # |actual − scheduled| / scheduled
}

Policy action spaceUSER_CONFIGURABLE

action = {
  penalty_rate,   # ₹/kWh for under-production
  bonus_rate,     # ₹/kWh for over-production
  discount_rate   # ₹/kWh consumer credit
}
# bounds: penalty [1, 10], bonus [0, 5], discount [0, 3]

Hourly production (digital twin)OFFICIAL_SOURCE · pvlib

Erbs(ghi, zenith) → DNI, DHI
POA = get_total_irradiance(...)
DC = pvwatts_dc(poa, t_cell, pdc0, γ)
AC = pvwatts_inverter(DC, pdc0, η=0.96)
prod_kWh = (AC / 1e6) × capacity_mw × Δt

Confidence scoreFALLBACK_DEFAULT

confidence = clamp(1 − 0.35 × cloud_fraction, 0.4, 0.99)
# Used as a state feature and for hybrid forecast blending

Provenance: MODEL_LEARNED — The RL policy is trained on real historical irradiance data via a pvlib digital twin. Rates are optimized, not declared by a regulator — they are decision-support, not binding tariffs.

Site

Episodes

Years (ERA5)

Days (if 0 yrs)

Real data

Real-data training fetches genuine past weather (set Years ≥ 1 for multi-year ERA5 history), runs pvlib physics to build production/target curves, then optimizes penalty/bonus/discount rates with a REINFORCE policy gradient.

Training Run History

When	Algorithm	Episodes	Data Source	Best Reward	Rates (P/B/D)
No training runs yet — train a policy above.