ADAPTIVE CONSUMER AGENT MODEL ON ELECTRICITY MARKET

V zadnjem obdobju je na srednjenapetostnem (SN) in nizkonapetostnem (NN) distribucijskem omrežju delež razpršenih obnovljivih virov električne energije (OVE) v izjemnem porastu. OVE in njihova spremenljiva ter negotova proizvodnja, še posebej solarnih, fotonapetostnih elektrarn (angl. Photovoltaic, PV) in vetrnih elektrarn, ki je ne moremo napovedati natančno, močno vplivajo na zanesljivost elektroenergetskega omrežja. Operaterji distribucijskih omrežji (DSO) so se pri obratovanju distribucijskega sistema v času koničnih ur odjema tradicionalno ukvarjali s problemi prenizkih napetosti, zadnje čase pa so se zaradi razmaha OVE začeli soočati tudi s previsokimi napetostmi (prenapetosti v koničnih urah proizvodnje). Te težave se tradicionalno rešujejo že v fazi načrtovanja in dimenzioniranja elektroenergetskega sistema ter kasneje z naložbami v ojačitve omrežja. Ukrepi, ki jih predpisujejo tradicionalne metode, so dragi, ponekod pa so zaradi fizičnih omejitev (gosto naseljena mesta, nakupovalna središča itn.) tudi skorajda nemogoči. Potreba po zmanjšanju teh stroškov in po zagotovitvi čim manjših odstopanj med proizvedeno in porabljeno energijo v distribucijskem omrežju vodi do pospešenega razvoja novih ukrepov, s katerimi bi ceneje in učinkoviteje sproti dopolnjevali obratovalne zmogljivosti distribucijskega omrežja ter s tem zagotavljali zanesljivost dobave elektrike odjemalcem. Mednje sodijo tudi programi prožnega odjema (angl. Demand Response Programs), ki so namenjeni spodbujanju odjemalcev električne energije, da spremenijo svoje vzorce aktivne porabe energije. Enote s prožnim odjemom (angl. Demand Response Units), imenujmo jih prožnostne enote (PE), in njihovo vodenje predstavljajo nove možnosti za naprednejše vodenje tehničnih razmer v omrežju kot tudi nove priložnosti za bolj učinkovito trgovanje trgovca in dobavitelja na trgu z elektriko. Dobavitelji lahko nastopijo v vlogi agregatorja in združijo prožno energijo aktivnih odjemalcev PE, kot so lastniki toplotnih črpalk, električnih vozil in drugih PE, v skupinski portfelj, ki se prilagaja glede na povpraševanje in ponudbo. Na popolnoma dereguliranem trgu z električno energijo je obratovanje distribucijskega omrežja regulirana dejavnost, ločena od tržnih operacij. Enote PE so po navadi v zasebni lasti, vendar si njihovi lastniki želijo povečati dobiček, zato se državne spodbude razvijajo v smeri sodelovanja PE v tržnih programih prilagajanja odjema za zagotavljanje prožnosti. Agregatorji so akterji na trgu z elektriko. To funkcijo največkrat opravljajo dobavitelji na trgu na drobno, ki vodijo agregacijske platforme za oblikovanje portfelja prožne energije. Z združevanjem PE v portfelju v več SN in NN omrežjih agregator lahko ponudi prožnostne energijske produkte na veleprodajnih trgih z električno energijo, na trgih z rezervo, ali pa zagotavlja druge sistemske storitve sistemskim operaterjem prenosnega ali distribucijskega omrežja. Primeri teh produktov prožne energije so 15-minutni prožnostni produkti na izravnalnem trgu z elektriko ali enourni produkti na veleprodajnem znotrajdnevnem trgu z elektriko, medtem ko primeri frekvenčnih sistemskih storitev vključujejo zagotavljanje ročne rezerve za povrnitev frekvence (rRPF) ali avtomatske rezerve za povrnitev frekvence (aRPF). S PE lahko zagotavljamo tudi druge vrste sistemskih storitev, kot so regulacija napetosti in tokovne prezasedenosti vodov za DSO ter regulacija izravnavanja faznih neravnovesij (npr. z naprednimi energetskimi pretvorniki). Agregator načrtuje delovanje PE v svojem portfelju v smeri povečanja dobička, ki ga nato deli s sodelujočimi enotami v skladu z dogovorjenim poslovnim modelom. Z vidika družbe je zaželena večja aktivnost odjemalcev in s tem povečana udeležba PE v programih zagotavljanja prožnosti. Tehnični učinek načrtovanja v NN omrežjih dojema kot pozitiven, saj aktivira prožno energijo, ki lahko pomaga potešiti naraščajočo potrebo po prožnosti v elektroenergetskem sistemu. PE se lahko uporabi za znižanje konice odjema (angl. peak demand), povečanje lastne porabe ali za zagotavljanje sistemskih storitev. Če bi enote s prožnim odjemom spremenile svojo porabo v napačnem trenutku, bi takšen vozni red PE lahko negativno vplival na lokalne razmere v omrežju in na obratovalno zanesljivost distribucijskega omrežja, kar bi lahko privedlo do kršitev napetostnih omejitev ali do prezasedenosti vodov. Operater distribucijskih omrežij mora v takšnem primeru posredovati in zagotoviti, da vozni redi običajnega odjema, skupaj z voznimi redi lokalnih OVE in PE, ne povzročijo kršitev obratovalnih omejitev omrežja. DSO si mora tako zagotoviti končni nadzor nad vsemi elementi, ki jih je mogoče nadzorovati v omrežju. Eden izmed učinkovitih načinov, ki omogoča popoln nadzor DSO-ja nad akcijami aktivnih odjemalcev, povezanih z vodenjem PE, je sistem semaforja (angl. Traffic Light System, TLS), ki je bil razvit v okviru doktorske raziskave in je predstavljen v projektu INCREASE. Projekt INCREASE, ki ga je sofinancirala Evropska komisija v okviru Sedmega Okvirnega programa EU za raziskave in tehnološki razvoj (EU OP7), se je osredotočal na vodenje OVE in PE v MV in LV omrežju z namenom zagotavljanja sistemskih storitev za DSO, kot so regulacija napetosti in zagotavljanje rezerve. V okviru projekta so bile proučene tudi različne možnosti agregatorjev za zagotavljanje različnih storitev DSO-jem z novimi naprednimi strategijami vodenja, skupaj z novimi poslovnimi modeli za agregatorje ter lastnike OVE in PE. Sistem semaforja operaterju distribucijskega sistema omogoča, da odobri ali zavrne načrtovane vozne rede PE, ki jih predlagajo agregatorji, če bi njihove akcije privedle do omrežnih težav. Obstajajo še druge rešitve, ki omogočajo, da se omrežne razmere nahajajo znotraj predpisanih meja in pripomorejo k ublažitvi obratovalnih težav v distribucijskem omrežju, kot so: (i) vgradnja regulacijskega distribucijskega transformatorja (angl. On Load Tap Changer, OLTC), (ii) vgradnja naprav FACTS (angl. Flexible Alternating Current Transmission System) ter (iii) tradicionalno ojačenje kablov in električnih vodov. Za ublažitev težav z omrežjem DSO-ji trenutno še nimajo drugih rešitev in sistemov, ki bi jim omogočali obratovanje izključno s pomočjo preverjanja stanj v omrežju in vpliva voznih redov PE nanj [1]. Številne raziskave so bile usmerjene v izboljševanje omrežnih razmer z vodenjem enot PE v smeri optimalnega obratovanja distribucijskega omrežja, vendar predlagane izboljšave niso upoštevale ekonomskega izplena, ki bi ga lahko upravljalci PE ali agregatorji zaslužili z drugačnim načrtovanjem voznih redov. Zato je slednje glavna tema doktorske raziskave. V prihodnje se pričakuje znatno povečanje števila in skupne moči PE in s tem tudi pomena vloge agregatorjev, ki bodo združevali veliko število PE in jih vodili z namenom povečanja učinkovitosti in dobička. Za vodenje velikega števila PE bodo agregatorji potrebovali napredne, popolnoma ali vsaj polavtomatizirane procese načrtovanja voznih redov PE. Kot najprimernejše metode za avtomatizacijo procesa načrtovanja se kažejo metode strojnega učenja in umetne inteligence (angl. Artificial Intelligence, AI), še posebej inteligentnih agentov. Alternativa tem metodam je uporaba linearnih optimizacijskih tehnik, ki teoretično zagotavljajo najboljši rezultat v neomejenem omrežju, a se v primeru vnaprej neznanih omejitev in zavrnitev voznih redov s strani DSO-ja njihov rezultat znatno poslabša. Cilj disertacije je bil razviti metodo načrtovanja voznih redov PE s pomočjo agentnega učenja, ki bi v obremenjenem distribucijskem omrežju dosegala boljši rezultat kot ekonomska optimizacija ob upoštevanju zavrnitev. V disertaciji smo tako poleg koncepta TLS-ja zasnovali tudi koncept trgovanja s prožnostjo. Definirali smo vloge vseh deležnikov in izmenjavo informacij med njimi. Novo razviti koncept zajema vse faze trgovanja, od dolgoročnih rezervacij za zagotavljanje sistemskih storitev do izravnalnega trga. Koncept vključuje tudi preventivno in korektivno delovanje DSO-ja glede na čas dobave energije in izvedbe TLS-ja. V koncept trgovanja smo vključili tudi agenta aktivnega odjemalca za načrtovanje voznih redov, ki za načrtovanje uporablja novo razvito metodo Posplošenega Q-učenja (PQL). S to metodo se agent uči izogibati zavrnitvam voznih redov in stremi k doseganju večjega dobička z upoštevanjem in predvidevanjem omrežnih razmer. Razvili smo dva načina uporabe PQL-a, in sicer enostopenjski in dvostopenjski PQL, ki se razlikujeta po procesu načrtovanja voznih redov ter različnih kriterijskih funkcijah ocenjevanja omrežnih omejitev. Rezultati preizkusa novo razvite metode PQL kažejo, da je v veliki meri mogoče zmanjšati tveganje zavrnitve voznih redov in se s tem izogniti morebitnim kaznim za neuspešno aktivacijo prožnosti. Delovanje in učinkovitost različnih metod za načrtovanje voznih redov PE smo preverili na testnem sistemu, ki temelji na realnem omrežju in meritvah odjema in proizvodnje.In the recent period, the share of distributed renewable sources of electricity (RES) in the medium-voltage (SN) and low-voltage (NN) distribution network is experiencing an exceptional increase. RES and its variable and uncertain production, especially solar photovoltaic (PV) and wind power plants, which cannot be dispatched, have a significant impact on the reliability of the electricity grid. Distribution network operators (DSOs) have traditionally been dealing with low voltage problems in the peak hours of consumption. Due to the expansion of RES, they have recently begun to deal with overvoltages during the peak production hours. These problems have traditionally already been solved in the phase of planning and dimensioning of the power system and during the operational phase by investing in the network reinforcements. Traditional methods are expensive, and in some cases almost impossible to implement due to physical constraints (e.g. in densely populated cities, shopping centres, etc.). The need to reduce these costs and to minimize the deviation between the produced and the consumed energy in the distribution network increases the need for the accelerated development of demand response (DR) programs, designed to encourage consumers to change their patterns of active energy consumption. Demand Response Units (DR) feature adjustable consumption and are also referred to as flexibility units. Their control strategies offer new possibilities for more advanced control and operation of the distribution network and for improving the technical conditions in the network. The new possibilities also present new opportunities for new trading strategies of the trader or supplier in the electricity market. Suppliers can act as an aggregator and combine active customers and their DR units, e.g. owners of the heat pumps, electric vehicles and other DR units into a group that adjusts its consumption as needed. In a completely deregulated electricity market, the operation of the distribution network is a regulated activity, separated from the market operations. The DR units are usually privately owned, but their owners want to increase profits so the state-level incentives are being developed to stimulate their participation in commercial adaptation programs to provide flexibility. Aggregators are the actors in the electricity market, and the role is most often assumed by the suppliers who manage aggregation platforms to create a flexible energy portfolio. By combining DR in a portfolio across several medium voltage and low voltage networks, the aggregators can offer flexible energy products on the electricity wholesale and reserve markets or provide ancillary services to network operators. Examples of these flexible energy products are the 15-minute flexibility products on the balancing market and one-hour products on the intraday wholesale electricity market, while examples of frequency-related system services include the provision of manual frequency restoration reserve (rRPF) or automatic frequency restoration reserve (aRPF). DR can also provide other types of system services such as voltage and current congestion control for DSOs and regulation of phase imbalance (e.g. with advanced inverters). The aggregator plans to operate the DR in its flexible energy portfolio in order to increase its profits, which are then shared with the participating units in accordance with the agreed business model. From the society point of view, it is desirable to increase the activity of customers, thereby increasing their participation in the DR programs and co-operating with flexibility schemes. Traditionally, the technical effect of DR operation in low voltage networks is viewed as positive, since it activates the flexible energy that can help fill the growing need for flexibility in the electricity system. DR can be used to reduce the peak demand, to increase consumer’s own consumption or provide system services. If flexible-load units change their consumption at the wrong time, however, these actions can negatively affect local network conditions and the operational reliability of the distribution network, which would lead to violations of voltage constraints or line congestions. In such cases, the distribution network operator must intervene and ensure that the normal consumption schedules, together with the forecast of local RES and scheduled actions of DR, do not lead to a breach of the operational limitations of the network. The DSO must therefore retain the ultimate control over all the elements that can be controlled on the network. One of the effective ways to ensure the full control of the DSO over the actions of active customers is a traffic light system (TLS), which was developed as part of the doctoral research and is presented in the INCREASE project. The INCREASE project was co-financed by the European Commission within the EU\u27s Seventh Framework Program for EU Research and Technological Development (EU FP7) which focused on the management of RES and DR in the MV and LV networks with the aim of providing system services to the DSOs, such as voltage control and provision of a reserve. The project also examined the various options for providing different services to aggregators with new advanced control strategies, along with new business models for the aggregators and for the owners of RES and DR. TLS allows the DSO to approve or reject the scheduled DR actions proposed by aggregators if their actions lead to network problems. There are other solutions that enable the DSO to keep the network conditions within the prescribed limits and help to alleviate the problems in the distribution network, such as: (i) installation of the On-Load Tap Changer (OLTC), (ii) installation of FACTS devices and (iii) the traditional strengthening of cables and power lines. To alleviate network problems, DSOs currently do not have any other solutions or systems that would allow them to control and operate exclusively only by checking the status of the network and the influence of the proposed DR schedules. A number of studies have focused on improving the network conditions with the control of DR units in the direction of the optimum functioning for the distribution network, but these studies did not take into account the economic opportunity cost for the aggregator and the DR unit owner that could be avoided by different control strategy. Therefore, the latter is the main topic of the doctoral research. In the future, a significant increase in the number and hence in the total power of DR is expected. Along with it, the importance of the role of aggregators which will bring together many DR and control them in order to increase efficiency and profit, is expected to increase significantly. For the control of many DRs, the aggregators will require fully automated or at least semi-automated scheduling processes for DR activations. The most appropriate methods for automating the scheduling process are the machine learning methods and artificial intelligence (AI) methods, including the intelligent agent approach. The alternative to these methods is the use of linear optimization techniques that theoretically provide the best result in an unconstrained network, but in the case of a-priori unknown limitations and rejection of timetables by the DSO, their result is significantly worsened. The aim of the doctoral research was to develop the scheduling method for DR based on intelligent agent control that would achieve a better result than economic optimization with rejections in a constrained network. In addition to the concept of TLS, we have also conceived the new concept of flexibility trading. The concept defines the roles of all stakeholders and the exchange of information between them. The newly developed concept covers all stages of trading, from long-term reservation of flexibility for the provision of system services, to the balancing market. The concept also includes the preventive and corrective actions of the DSO, depending on the time of energy delivery and TLS implementation. We have also included an active scheduling agent in the trading concept, who uses the newly developed method of Generalized Q-learning (PQL) for scheduling DR. By using this method, an agent learns to avoid scheduling rejection and steers toward achieving greater profit by considering / predicting network conditions. Two novel ways of using Generalized Q-Learning method are proposed in the thesis, the one-step- and the two-step PQL, which differ in the scheduling process and the criteria function used for predicting the network constrains. The results of the newly developed PQL method show that it is possible to reduce the risk of DR schedule rejection, thereby avoiding possible penalties for unsuccessful activation of scheduled flexibility. The operation and efficiency of various methods for scheduling DR were checked on a test system based on the real network and measurements of consumption and production