State Estimation Setup

[cavassiGE]

Hi! I would like tips and help with my setup for a state estimation.

I have a low voltage network with one transformer, using the "inner side" of the transformer components. I have reduced the components to just include the clients node, busbars, joints, and different cable types. The client nodes are also the points where real life measurements are being made (real life sensors). These measure the voltage which usually is around 230-240V. They also measure active and reactive power.

If it makes a difference, the graph can be seen in "graph.png".

Right now, I'm building the PowerGridModel object with these components:

input_data[ComponentType.node] = nodes
input_data[ComponentType.line] = lines
input_data[ComponentType.source] = source
input_data[ComponentType.sym_voltage_sensor] = sym_voltage_sensor
input_data[ComponentType.sym_power_sensor] = sym_power_sensor

The source, voltage sensor, and power sensor components are pretty much the same as the documentation. Should I consider adding loads? If so, how many? On which nodes? What values should they have?

I build the object and run the state estimation the same way as the documentation.

If my results could be useful, the most typical values can be seen in results.txt

Either my setup is wrong and I need to reconsider something from it. Or the constants I'm using are not accurate.

I'll also upload a file with the constants I'm using and their values

results.txt

config.py

[mgovers]

Hi @cavassiGE,

Thank you for your questions. Here is my attempt to answer them:

Should I consider adding loads? If so, how many? On which nodes? What values should they have?

Yes, you should add a load (resp. generator) at every location where you expect an energy consumer (resp. producer) - i.e., the same locations in which you would have a load if you were to run a power flow calculation. In your case, if my assumption is correct, that that would be at the red nodes.

However, you do not need to assign p_specified and q_specified, as the power consumption will be calculated from the sensor measurements. The reason you need to specify the loads is so that the Power Grid Model knows where consumption is happening. Without loads, the model thinks there is no consumption happening, so all estimated power flows are zero.

The rest of your setup seems to be OK.

I hope this helps. Please let us know how it goes!

[cavassiGE]

Thank you for your answer!

You are correct, the red nodes are where the voltage and active/reactive power is being measured. So having the loads here makes sense.

I have now added the sym_load components, which naturally changed the output. But the results are still not looking very good. The error tolerance need to be minimum 2e-1 for the estimation to be able to run, which I believe is way higher than wanted.

To give you as much context as possible, I'll share the file where I create the PowerGridModel object, as well as the config which is now slightly updated. I have tried tweaking some of the constants, but I'm afraid my physics knowledge are not good enough to know what a reasonable value is for most of them.

I'll also share the results, but since the data used as input is classified, I've had to round/remove/modify some of the output values. But the overall picture is still the same.

Not sure if these points are interesting, but some things to notice are:

• the extreme outliers (try ctrl+f and search for e+0X)
• all Lines have NaN on loading
• the source values does not look reasonable right?

I've added some TODOs in the code for things that I'm not sure about.

Thank you again!

graph_to_pgm.py

summary.txt

[mgovers]

Hi @cavassiGE,

I see that you specify u_measured in p.u. (per unit/natural units) instead of in Volt (sym_voltage_sensor["u_measured"][i] = float(row["voltage"]) / conf.LV_LN_VOLTAGE). This should probably be simply sym_voltage_sensor["u_measured"][i] = float(row["voltage"])

If that does not help, can you run a couple tests to see what is up?

1. I see that you add quite a lot of voltage sensors (while not necessarily an issue, it may hide other things). Can you please run an experiment in which you only add 1 single voltage sensor?
2. What happens if you change the measured_object_type from MeasuredObjectType.node to MeasuredObjectType.sym_load and also change the measured_object to the respective sym_load ID? This should at the very least make the active and reactive power values in the output sym_loads be correct.

[cavassiGE]

Changing the u_measured to Volts gives me a max deviation of around 3.35. I could raise the error tolerance and provide the new results if you'd like.

1. Single voltage sensor:

using p.u gave me a Max deviation of 110
using Volts gave me a Max deviation of 497
(15 iterations on both)

2. I suppose you mean MeasuredTerminalType because I can't find MeasuredObjectType. Same for .sym_load, I can only find .load.

I tried doing the ID changes without reverting from 1. and got that there were Not enough measurements available for state estimation. The total number of independent power sensors is not enough to make the grid observable.

Then I also tried changing the ID's with all the voltage sensors. with both p.u values and Volt.

Volt gave me a max deviation of 1.7
p.u was able to run and gave me a sym_load output seen here, again, rounded:
sym_load_result.txt

[nitbharambe]

Quick note looking at https://github.com/cavassiGE/pgm-test todos

node u_rated -> Line to line
u_measured for sym_voltage_sensor -> Line to line
u_measured for asym_voltage_sensor -> Line to neutral

[cavassiGE]

Hi @figueroa1395,

I've added you now. Thank you for the help!

[figueroa1395]

Hello @cavassiGE,

I took a quick look at the repo and a very small grid I created with it, and even though EE is not my expertise, after playing around, I noticed that your voltage sensors are set to have a very minuscule u_sigma, making them unrealistically precise. In addition, you have more than one, which means that the entirety of your grid's state is dominated by these voltage sensors, specially if you compare it with the values of the power sensors. Now, when I used ~3V or 1% of their nominal voltage for u_sigma, the results started looking a lot better.

In short, the unrealistic values you have for u_sigma in your voltage sensors are constraining the grid in an unrealistic way, which also overshadows whatever you put in your power sensors. My recommendation is to increase this value to around 1% of the nominal voltage.

Some additional remarks based on TODOs and observations:

• You have set the short circuit power (sk) of the source to 1e7, which seems a bit low. Can you confirm this is intended?
• i_n is not needed to be specified for lines in the input data. However, if you want the line loading to not show up as NaN in the results, this attribute is necessary.
• For a voltage sensor, the voltage angle is not "needed" unless you have physical sensors that include this (phasor voltage measurements). If that is not the case you can leave it empty as is.
• When setting p_sigma and q_sigma for a power sensor, if you make them very small, you'll constrain the system even more (having them as zero means your sensor is perfect - which is unrealistic). If you are not sure of these values, making them on the larger side gives the algorithm more flexibility to find a solution. However, keep in mind setting them as np.inf means that you are effectively deactivating the sensors, since their measurement doesn't add any weight to the statistical determination of the state.

[cavassiGE]

Thank you for this! @figueroa1395

To answer your great feedback:

You are correct that the voltage sensor sigma was way too low. I've now created a function that calculates the 1% of the nominal voltage, which is very close to 3V. I could play around with values around 1%, but for now it seems like a good value to begin with.

def voltage_sigma(voltage_df):
    nominal_voltage = voltage_df["voltage"].median()
    sigma = nominal_voltage * 0.01
    return float(sigma)

• I've changed the sk to 1e10 which is also the default value.

• I'll leave the i_n blanc. Don't think it's relevant.

• Leaving the voltage angle empty as well.

• This point seem to be relevant! I tried changing the values a little bit, and got noticeable changes. What's a reasonable value for these two, and what is considered as "larger side"? Can I calculate a value in the same way I did for the voltage sensor? I assume separating them into "p_sigma" and "q_sigma" gives a more accurate result than using "power_sigma"? Right now I have these values:

sym_power_sensor["p_sigma"][i] = 0.02 * abs(P) + 10.0
sym_power_sensor["q_sigma"][i] = 0.03 * abs(Q) + 10.0

I believe that my results look pretty decent now. I raised the dataset to 200. I only updated the results for the large dataset, but to summarize:

• The node voltages looks good. All around 237V and very consistence.

• All lines are energized. Some p/q values may look a little bit high, I'm not entirely sure whats considered realistic, but these may also depend on the power sensor sigmas.

• The source node changed a lot, from p=754.360953 and q=202.290728 to p=-509.728458 and q=714.442421. I'm not sure what a realistic value is for this.

• Both the voltage sensors and power sensor are now very consistent with values around 0, which I believe is an improvement.

• The sym loads look fairly similar, maybe a bit more consistent.

The result seem to be much more accurate at the least, since the max deviation is 7.98141776559569e-13! If this model looks correct, then I'll have to take a deeper look into my measured data or graph structure since it's built from real life values.

Thank you for your help again, very appreciated!

[figueroa1395]

Hello @cavassiGE,

It's great to hear that the feedback was useful. From what I read, it looks like indeed the results are now consistent and "realistic". Now to some specific points you mentioned:

This point seem to be relevant! I tried changing the values a little bit, and got noticeable changes. What's a reasonable value for these two, and what is considered as "larger side"? Can I calculate a value in the same way I did for the voltage sensor? I assume separating them into "p_sigma" and "q_sigma" gives a more accurate result than using "power_sigma"? Right now I have these values:

It's very hard to give specific values on the power sensors as these aren't as "enforcing" as the voltage sensor and the real uncertainty of these can vary greatly across models. Nevertheless, what I can tell you is that using p_sigma and q_sigma individually, contrary to just power_sigma is indeed more accurate, but also that increasing the sigma values gives the model more change to find a solution (keep in mind that making them too large doesn't ensure you find the optimal solution and other local minimum can arise). For these reasons, my recommendation is that if you are unsure of the uncertainty specification of your sensors, just play around with it and adjust them to what you are trying to simulate. Mentioning specific numbers may be misleading though.

In the case of voltage sensors it was easier to mention the value since the one you had was unrealistically small and I don't expect voltage sensors' uncertainty to vary as much as the power sensors' do.

Maybe one last suggestion, is that you should always take a look at the residuals of the sensors in the output. If they are incredibly small (10^-13 or the likes), it probably means you have an exactly determined system (so what's the point of running state estimation then?), but if they are too large compared to the measured values, then it means something crazy is happening which perhaps is related to the sigma values (this would be your cue to play with them).

I'll leave the i_n blanc. Don't think it's relevant.

That's fine. If you don't care about loading output per line, leave as is. The results would not be altered.

• All lines are energized. Some p/q values may look a little bit high, I'm not entirely sure whats considered realistic, but these may also depend on the power sensor sigmas.
• The source node changed a lot, from p=754.360953 and q=202.290728 to p=-509.728458 and q=714.442421. I'm not sure what a realistic value is for this.
• Both the voltage sensors and power sensor are now very consistent with values around 0, which I believe is an improvement.

I'm not sure about the exact numbers, but having small residuals (unless crazy small, as I stated above) is a good thing and the source node values were expected to change their results a lot based on the big change of the voltage sigmas.

The result seem to be much more accurate at the least, since the max deviation is 7.98141776559569e-13! If this model looks correct, then I'll have to take a deeper look into my measured data or graph structure since it's built from real life values.

Awesome! That sounds good. Let us know if you need any more assistance. We are happy to help.

[cavassiGE]

Hello again! @mgovers @figueroa1395

I'm still getting results that I find somewhat suspicious. There are clear improvements! For example, my residuals are much smaller, the estimated voltage looks accurate, and the max deviation is extremely small (could too small be bad?).

However, I’m seeing values like:

Sources
       id  energized            p           q          i            s        pf
0  999999          1  8436.465716 -837.485256  20.431573  8477.932256  0.995109

and

Lines
        id  energized  loading       p_from      q_from     i_from       s_from         p_to        q_to       i_to         s_to
1  1000001          1      NaN  4989.226750 -304.750448  12.046303  4998.525423 -4967.702278  318.166479  12.046303  4977.880657

which seem quite high to me.

To make things easier to debug, I’ve extracted a minimal dataset. Everything can be find in my repository:

• 6 nodes, 5 edges
• Real measurement data (pq.csv and voltage.csv)
• Real network structure (graph.json)

I've also included an interactive graph (graph.html) generated by the current configuration, which should make it much easier for you to see the values for different components (just hover over a node or edge).

I've been playing around a lot with the parameters in config.py and the current setup is more or less the best I've gotten. The parameters that had the biggest effect on the max deviation and the results were the voltage sigma and the p/q sigma. If there are any fallacies with my model, I do not believe it's those parameters. I may be wrong, I don't really have a background in electrical physics, but I would expect better results if that were the issue, given how many values I've tried.

One important question is whether I am constructing the Voltage and P/Q tables correctly.

The voltage for each connection point is measured on three phases, like this:

Voltage L1 value=239
Voltage L2 value=238
Voltage L3 value=239

I take the mean of these three values and use it as the voltage for that connection point.

For the P and Q values, I have data like this:

P14 L1 value=1.19  
P14 L2 value=1.053  
P14 L3 value=1.104  

P23 L1 value=0  
P23 L2 value=0  
P23 L3 value=0  

Q12 L1 value=0  
Q12 L2 value=0  
Q12 L3 value=0  

Q34 L1 value=0.18  
Q34 L2 value=0.24  
Q34 L3 value=0.253  

First, I sum the current on each power flow. This gives me 4 different values. Then, I add a direction with this dictionary:

PQ_DIRECTION = {
    "P14":  1,
    "P23": -1,
    "Q12":  1,
    "Q34": -1,
}

Then I take the sum for the P's and Q's. I may be getting the direction wrong, but I've tried all the combinations, and the results does not differ too much.

As mentioned earlier, can a too small max deviation mean something bad? I tried setting the error tolerance to 1e-100 to see what it was at, and this was the result: Max deviation: 1.6940658945086007e-21. I know my POWER_SENSOR_SIGMA value for P and Q are really tight, but changing them to other values (ex 0.1, 1, 10, 30, 60, 100) does not really give much better results. There's always some spiked values regardless.

You are still collaborators on my repository, so feel free to test it yourselves if that makes things easier. It's public in any case.

[figueroa1395]

Hello @cavassiGE, we are a bit occupied at the moment, but we expect to take a look at this sometime next week. We'll let you know as soon as we have feedback.