|
This forum is read-only now. Please use Forum 2 for new posts
xml |
No replies possible in the archive |
Author: ferdi
Date: 15-09-2005, 16:29
| The discussion in some other thread about the value of estimated probabilities led me to the following idea. I have allready posted the explanations elsewhere but repeat them here for your convenience.
Imagine a set of M contests between the teams A_j and B_j (1{=j{=M). (For simplicity, we assume that there will be no draws possible, but either a victory of team A_j or team B_j - which is an appropritate model for a knock-out phase like 1st round UEFA cup. However, the following thoughts can easily be extended on estimations of win:draw:loss-probabilities.)
Let there be N wannabe-experts {W_1..W_N}. Each wannabe W_i estimates the probability that team A_j will win as p_ij. The by W_i estimated probability that B_j will win then equals (1 - p_ij).
Now calculate the average estimated probability that team A_j will win as
P_j = sum_over_i (p_ij) / N
Clearly the average estimated probability that B_j will win equals then (1 - P_j).
I have done this calculation for the estimation some of you gave in the thread {b>{a href="https://kassiesa.net/uefa//forum/view.php?topic=20050826141956.xml&PHPSESSID =b736da8025393f5ef72f808ca02820f2">UEFA CUP 1. ROUND predictions{/a>{/b>. There were 28 wannabes {em>(budaig, AnorthosisFC, alexx, bjkman1903, forbsey, krdeluxe, LevskFan, vasyaxxx, Nick, Rox, Shadow, PG, Dneprchemp, Pawlm_18, Koulis, Vlad, max, SPCaesar, Krys, Bruno, Kronsky, kharagh, sb, shaga, cattle-ripper, heat1995, delustef, m4e, naaba){/em>, not all of them giving estimations for all of the 40 matches, and the resulted average estimations are as follows (but see the footnote): {pre> Stuttgart Domzale 92:8 Basel Brijeg 88:12 Hertha APOEL 83:17 Slavia Cork 82:18 CSKA Midtjylland 82:18 Beerschot Marseilles 19:81 Middlesborough Xanthi 81:19 Halmstad Sporting 23:77 Aris Roma 24:76 Bolton Plovdiv 75:25 Hamburg Kopenhagen 74:26 Monaco Willem 74:26 Petach-Tikva Partizan 26:74 Palermo Anorthosis 74:26 Tromso Gatala 27:73 Din. Bukarest Everton 28:72 Lok. Moskva Brann 71:29 Sevilla Mainz 71:29 MyPa Grashoppers 29:71 Shakhtar Debrecen 71:29 Lens Groclin 70:30 Feyenoord Rapid 70:30 Besiktas Malmö 69:31 Teplice Espaniol 32:68 Setubal Genua 32:68 Krylya AZ 34:66 Leverkusen Sofia 66:34 Auxerre Levski 66:34 Viking Austria 39:61 Hibernian Dnipro 41:59 Banik Heerenveen 41:59 Guiamras Wisla 41:59 Litex Genk 42:58 Rennes Osasuna 42:58 Bröndby FC Zürich 56:44 Metalurg PAOK 46:54 Red Star Braga 53:47 Graz Strasbourg 52:48 Valerenga Steaua 49:51 Zenit AEK 50:50 {/pre> After the matches are played, each wannabe will receive points or lose points depending on his own estimations p_ij and the average estimations P_j
If team A_j wins, each W_i will get R_(A_j,i) = k ( log(p_ij) - log(P_j) ) pts.
If team B_j wins, each W_i will get R_(B_j,i) = k ( log(1-p_ij) - log(1-P_j) ) pts.
The points for the M matches will then be added up for each wannabe. (k is an arbitrary normalisation factor.) The more points you get, the more possible is it that you are an expert for the UFEA cup.
Those of you familiar with mathematics will easily understand the approach using logarithms which is based on information theory.
Not only for the none-mathematics I explain the properties of this predictors competition:
1.) You can only win or lose points if you give an estimation {b>different from the average estimation{/b>. If you just repeat the average opinion you do not risk anything, but can't gain points as well.
2.) If you give an estimation different from the average estimation, you can only expect to get positive points {em>in sum{/em> (i.e. summed over the M matches) if your estimations p_ij are {b>generally better{/b> than the average estimations Pj.
3.) The better your estimation {b>matches the true probability of outcome{/b> (which we cannot measure, because we cannot repeat the matches several times), the more points you can expect {em>in sum{/em>.
So to prove as an expert you must be {b>near to the true probability{/b> esp. in cases where there is a {b>difference between the average estimated probability and the true probability{/b>.
(For convenience we take the arbitrary constant k = 10/log(2). This means that a quotient of 2 in {em>( individual wannabe's estimated probability / average estimated probability ){/em> will lead to a gain of 10 points, should the team in question advance to the next round.)
{b>Example:{/b> Average estimation is 40:60 for team A_j. You estimate 80:20 for team A_j.
If team A_j should win, you will gain k (log(0.8)-log(0.4)) = 10.00 pts If team B_j should win, you will lose k (log(0.2)-log(0.6)) = - 15.85 pts
If the true probability of outcome is 80:20, as you expect, then the expectation value for your gain of points is
0.8 * 10.00 pts + 0.2 * (-15.85) pts = 4.83 pts
If the true probability of outcome is 40:60, as is the average expectation, then the expectation value for your loss of points is
0.4 * 10.00 pts + 0.6 * (- 15.85) pts = -5.51 pts
I will evaluate the estimations given in the {em>UEFA CUP 1. ROUND predictions{/em> thread after the 2. leg matches are played and then post the results here.
{em>Footnote:{/em> You may well estimate 100:0, but should only one team advance to the next round that you gave a probability of 0, then you will get -infinity points for that match. This can be regarded as a proof that you have {b>absolutely no clue{/b>, and all your estimations will be excluded from the calculation of the average estimated probabilities. The above probabilities do include the estimations of all contestants and are so far to be regarded as provisional. |
Author: FrancoisD
Date: 15-09-2005, 22:48
| I suggest you add a player to this competition : UEFA Club Ranking ! If team Ta has coef Ca, the probability that Ta wins is Ca/(Ca+Cb).
So we will know if Team Ranking can be a "prediction tool" smarter than some "wannabe-experts". |
Author: oribd
Date: 16-09-2005, 02:29
| Sounds really interesting! Being a third year student in mathematics who hasn't got a clue on information theory, I just wonder why the logs. Is their job to prevent a little number of extreme successes/failures from influencing the overall score too much? |
Author: ferdi
Date: 16-09-2005, 10:22
| The idea is that we feel that the {em>information{/em> about the entrance of independent events is additive, whereas {em>probabilities{/em> for independent events are to be multiplied.
For example, the probability p_c that if you toss three coins they will all show up "head" is p_1 (first coin shows head) X p_2 (second coin shows "head") X p_3 (third coin shows "head").
The information I_c which we gain after we know that all coins show up head (or which we lack before we know it) should be I_1 (first coin shows "head") + I_2 (second coin shows "head") + I_3 (third coin shows "head").
This property of {em>information{/em> can obviously achieved if {em>information{/em> is defined as the {em>logarithm of probability (times an arbitrary constant){/em>.
So if you take the arbtrary constant as -log(2), then the {em>information{/em> about the outcome {em>coin shows up "head"{/em> equals 1, as long as the {em>probability{/em> equals 50%. The {em>information{/em> that {em>three coins show up "head"{/em> then adds up to 1 + 1 + 1 = 3. |
Author: ferdi
Date: 16-09-2005, 11:45
| I pick up FrancoisD's suggestion to compare the pseudo-probabilities derived from the seeding points to the estimated probabilitis of the contestants.
It would be very surprising if the seeding points would give a good estimation of the probabilities of outcome, because those points are not meant to be fine-tuned to reflect the current strenght of teams. However, it could be a hint for anybody who ends up with less points than the "Seeding Table", that he should improve his qualities as football expert.
Here are - for your information - the pseudo-probabilities derived from the seeding points of the teams, compared to the average estimations of the experts: {pre> Average Seeding-Points Stuttgart Domzale 92:8 95:5 Basel Brijeg 88:12 93:7 Hertha APOEL 83:17 89:11 Slavia Cork 82:18 96:4 CSKA Midtjylland 82:18 80:20 Beerschot Marseilles 19:81 22:78 Middlesborough Xanthi 81:19 72:28 Halmstad Sporting 23:77 19:81 Aris Roma 24:76 18:82 Bolton Plovdiv 75:25 77:23 Hamburg Kopenhagen 74:26 61:39 Monaco Willem 74:26 80:20 Petach-Tikva Partizan 26:74 19:81 Palermo Anorthosis 74:26 81:19 Tromso Gatala 27:73 12:88 Din. Bukarest Everton 28:72 37:63 Lok. Moskva Brann 71:29 83:17 Sevilla Mainz 71:29 69:31 MyPa Grashoppers 29:71 10:90 Shakhtar Debrecen 71:29 66:34 Lens Groclin 70:30 71:29 Feyenoord Rapid 70:30 88:12 Besiktas Malmö 69:31 88:12 Teplice Espaniol 32:68 38:62 Setubal Genua 32:68 42:58 Krylya AZ 34:66 19:81 Leverkusen Sofia 66:34 78:22 Auxerre Levski 66:34 74:26 Viking Austria 39:61 32:68 Hibernian Dnipro 41:59 34:66 Banik Heerenveen 41:59 28:72 Guiamras Wisla 41:59 31:69 Litex Genk 42:58 42:58 Rennes Osasuna 42:58 40:60 Bröndby FC Zürich 56:44 74:26 Metalurg PAOK 46:54 22:78 Red Star Braga 53:47 57:43 Graz Strasbourg 52:48 59:41 Valerenga Steaua 49:51 29:71 Zenit AEK 50:50 28:72 {/pre> |
Author: bert.kassies
Date: 16-09-2005, 11:54
| ferdi, nice topic! To my surprise the "Seeding-Points" are actually quite close to the average. Which almost implies that some experts will finish below the seeding. |
Author: FrancoisD
Date: 16-09-2005, 14:46
| I also wish to express thanks to ferdi. I really like the wording "estimated probabilities" better than "predictions". The introduction of information theory as a tool to compare two sets of probabilities is also a great idea.
I just wonder if you can also explain why you use the average value of estimations. If you compare the predictions against a fixed probability scheme like 50-50, you also obtain a ranking. Can you develop the advantage of your method ?
I'm glad you valued my proposal to use UEFA ranking and made the computations. I hope such computations could be of some use if the controversy of the country weight in club rankings reopens. |
Author: ferdi
Date: 16-09-2005, 15:25
Edited by: ferdi at: 16-09-2005, 15:46 | Why do I compare against the average estimation? I think the average estimation is a good gauge to measure your own estimation with.
You may of course say "I have never heard of Slavia Prague nor of Cork City, therefore my personal estimation is 50:50". But practically you would in this case stick to what you pick up from all the others. It is indeed a good strategy to repeat the average opinion if you don't feel that you know it better. But to be regarded as a real expert, you should go beyond this and express your dissent when you feel this is justified - and then turn out to be right in a majority of cases.
We can of course most easily count the information that is included in the average estimations compared to an "all games are 50:50"-set of estimations. Let's call this information {b>I_o{/b>. Thus I_o would be a measure for the amount of expertise that we gain by joining our individual estimations together. Because of the additivity of information, your individual infomation against the "50:50"-set is then the sum of your information against the average estimations plus I_o.
In other words, the ranking against a "50:50"-set would be exactly the same, but every contestant would have I_o points more on his account. This would be the amount of points that is guaranteed to you if you always repeat the average opinion, and should therefore be substracted if you want to measure your individual skill. |
Author: ferdi
Date: 16-09-2005, 16:37
| FrancoisD wrote:
{em>I'm glad you valued my proposal to use UEFA ranking and made the computations. I hope such computations could be of some use if the controversy of the country weight in club rankings reopens.{/em>
Indeed one could recalculate the pseudo-probabilities based on seeding-points with different seeding points due to different country weights (50% or 25%). After the games are played, one could compare which county weights lead to the best set of estimations (measured by the amount of information that is included in the set of pseudo-probabilities that belongs to each country weight). |
Author: Toenne
Date: 16-09-2005, 17:21
| My predictions after the first leg
Stuttgart Domzale 96:4 Basel Brijeg 100:0 Hertha APOEL 97:3 Slavia Cork 95:5 CSKA Midtjylland 90:10 Beerschot Marseilles 21:79 Middlesborough Xanthi 97:3 Halmstad Sporting 1:99 Aris Roma 1:99 Bolton Plovdiv 58:42 Hamburg Kopenhagen 46:54 Monaco Willem 86:24 Petach-Tikva Partizan 0:100 Palermo Anorthosis 59:41 Tromso Galatasaray 46:54 Din. Bukarest Everton 95:5 Lok. Moskva Brann 99:1 Sevilla Mainz 50:50 MyPa Grashoppers 54:46 Shakhtar Debrecen 98:2 Lens Groclin 46:54 Feyenoord Rapid 47:53 Besiktas Malmö 40:60 Teplice Espaniol 20:80 Setubal Genua 24:76 Krylya AZ 54:46 Leverkusen Sofia 47:53 Auxerre Levski 52:48 Viking Austria 46:54 Hibernian Dnipro 36:64 Banik Heerenveen 60:40 Guiamras Wisla 75:25 Litex Genk 37:63 Rennes Osasuna 57:43 Bröndby FC Zürich 70:30 Metalurg PAOK 57:43 Red Star Braga 50:50 Graz Strasbourg 1:99 Valerenga Steaua 0:100 Zenit AEK 48:52 |
Author: oribd
Date: 16-09-2005, 20:54
| ferdi,
I see, thanks. |
Author: ferdi
Date: 29-09-2005, 16:12
Edited by: ferdi at: 29-09-2005, 17:54 | Let me use the time while we are waiting for the results to pick up the idea to compare several sets of estimated probabilities against each other (or for example against a set of 50%-50%-estimations).
As allready pointed out, the logarithm of the probability of the outcome of an event is a good measure for the information we gain if we come to know that the outcome did really take place - or for the lack of information that we suffer before we know about the outcome.
We should also point out that there is a difference between {b>personal probabilities{/b> and {b>true probabilities{/b>. The former depend on the amount of personal knowledge or ignorance about a matter, whereas the latter are a result of true properties such as symmetry etc.
Thus it makes a difference whether I estimate the outcome of a football match as 50:50 because I {em>have no idea{/em> about the real qualities of the teams, or whether I estimate the outcome as 50:50 because I {em>know{/em> that both teams are {em>truely{/em> of equal strength, and therefore the outcome is {em>truely{/em> as uncertain as the toss of a coin.
In this sense we can define our target as to find out how good our personal probabilities do match the true probabilities.
We can tackle this with the concept of mean information (also called {b>entropy{/b>.
If p denotes my personal probability for a victory of team A, then I_A=k*log(p) is the personal information I gain when I find out that team A wins, and I_B=k*log(1-p) is the information I gain when I find out that team B wins.
Now let be P the true probability that team A wins. Then my {em>mean personal information{/em> for this game is {pre> I_mean,pers = k ( P I_A + (1-P) I_B ) = k ( P log(p) + (1-P) log(1-p)) {/pre> {b>This mean personal information has an absolute maximum (or minimum, depending on the choice of k) for p if p=P, i. e. if my personal probability matches the true probability.{/b> This maxmimal value we call the {b>entropy S{/b> of the game in question: {pre> S_A-B = k ( P log(P) + (1-P) log(1-P) ) {/pre>
If we have a set of several matches, than we can denote the sum of the entropy of all matches as the entropy S of the whole set.
Unfortunately we do not know the true probabilities, so we cannot compute the mean personal information nor the entropy directly. But as we have a large set of matches, then we can have good confidence that the sum of the actual information we gain in all the matches will converge against the sum of the mean personal information in those matches.
Moreover, we do not know the entropy of the set. But we know that the nearer our personal probabilities approach the true probabilites the bigger will be our mean personal information. So the mean personal information can be taken as a relative measure of quality of personal probabilities.
For example if the mean personal information value of the averaged estimations is bigger than the mean personal information value of the set of 50-50-estimations, then this means that the averaged estimations are "better" then the 50-50-estimations. |
Author: exile
Date: 29-09-2005, 18:07
| Now all we need to do is to convert %age probabilities (or differences in coefficients) into a margin of victory! ie 80-20, or 4:1 in coefficients - 4 goal margin (or whatever)....
(head explodes) |
Author: ferdi
Date: 30-09-2005, 08:59
Edited by: ferdi at: 30-09-2005, 13:42 | Here are the results of the experts competition.
The contestant Shadow_PG has predicted zero-probability for the following teams which reached the next round:
Tromso Rome Petach-Tikva Zenit
Maybe the prediction of 0% for Rome was a mistake, but that doesen't matter because four times -infinity is the same as three times -infinity. Shadow_PG ends up with {b>-infinity{/b> points. His predictions for all games will not be included in the calculation of the average prediction. The average prediction is therefore finally recalculated as follows (The result of the matches is indicated by an 1 for the first team and by a 2 for the second team): {pre> Stuttgart Domzale 92:8 1 Basel Brijeg 87:13 1 Hertha APOEL 83:17 1 Slavia Cork 81:19 1 CSKA Midtjylland 81:19 1 Beerschot Marseilles 20:80 2 Aris Roma 20:80 2 Middlesborough Xanthi 80:20 1 Halmstad Sporting 24:76 1 Bolton Plovdiv 75:25 1 Hamburg Kopenhagen 74:26 1 Palermo Anorthosis 73:27 1 Monaco Willem 73:28 1 Petach-Tikva Partizan 28:72 1 MyPa Grashoppers 28:72 2 Tromso Gatala 28:72 1 Shakhtar Debrecen 71:29 1 Din. Bukarest Everton 30:70 1 Feyenoord Rapid 70:30 2 Lok. Moskva Brann 70:30 1 Lens Groclin 70:30 1 Sevilla Mainz 69:31 1 Besiktas Malmö 69:31 1 Teplice Espaniol 32:68 2 Leverkusen Sofia 66:34 2 Setubal Genua 34:66 2 Auxerre Levski 65:35 2 Krylya AZ 35:65 2 Viking Austria 39:61 1 Hibernian Dnipro 42:58 2 Banik Heerenveen 42:58 2 Guiamras Wisla 43:57 1 Litex Genk 43:57 1 Rennes Osasuna 43:57 1 Bröndby FC Zürich 55:45 1 Zenit AEK 53:47 1 Red Star Braga 53:47 1 Graz Strasbourg 52:48 2 Metalurg PAOK 49:51 2 Valerenga Steaua 49:51 2 {/pre> Here is the number of points each contestant achieved: {pre> Vlad 27 delustef 20 SPCaesar 13 Krys 7 Dneprchamp 5 sb 3 kharagh 3 m4e 1 cattle-ripper 1 Rox 0 bjkman1903 0 heat1995 -1 krdeluxe -2 Nick -4 budaig -6 Kronsky -6 naaba -11 shaga -16 alexx -20 forbsey -20 Pawlm_18 -23
Seed100 -24 Seed50 -25
Bruno -26
Seed33 -29
LevskFan -31
SeedCR -33 Seed25 -34
vasyaxxx -36 AnorthosisFC -37 max -47
50/50 -66
Koulis -76 {/pre>
All contestants who got a positive number of points indicate that their football knowledge might be better than average. If you achieved a negative number of points, then this suggests that your knowledge maybe worse than the average knowledge. However, there is still the possibility that it was just bad luck this time.
I have also included the number of points that would have been achieved by someone who gave 50:50-prediction for all matches. So if you got less than -66 points, your expertise is really in doubt. ;-)
Also included are the pseudo-probabilities based on team seeding coefficients, under the inclusion of a 25, 33, 50, and 100% country bonus, as well as based on the country ranking itself, without any individuel team points at all. Note that the 50% and 100% country bonus seem to lead to slightly better predictions than do 25% ,33%, or pure country ranking.
Here is a detailed list of the number of points the winner Vlad made in each match. {pre> Average Vlad Result Points Tromso Gatala 28:72 50:50 1 8 Zenit AEK 53:47 80:20 1 6 Petach-Tikva Partizan 28:72 40:60 1 5 Feyenoord Rapid 70:30 60:40 2 4 Auxerre Levski 65:35 55:45 2 4 Din. Bukarest Everton 30:70 35:65 1 2 CSKA Midtjylland 81:19 95:5 1 2 Guiamras Wisla 43:57 50:50 1 2 Metalurg PAOK 49:51 40:60 2 2 Viking Austria 39:61 45:55 1 2 Graz Strasbourg 52:48 45:55 2 2 Lens Groclin 70:30 80:20 1 2 Lok. Moskva Brann 70:30 80:20 1 2 Middlesborough Xanthi 80:20 90:10 1 2 Aris Roma 20:80 10:90 2 2 Beerschot Marseilles 20:80 10:90 2 2 Palermo Anorthosis 73:27 80:20 1 1 Bröndby FC Zürich 55:45 60:40 1 1 Hamburg Kopenhagen 74:26 80:20 1 1 Bolton Plovdiv 75:25 80:20 1 1 Litex Genk 43:57 45:55 1 1 Red Star Braga 53:47 55:45 1 1 Banik Heerenveen 42:58 40:60 2 0 Teplice Espaniol 32:68 30:70 2 0 Valerenga Steaua 49:51 50:50 2 0 Stuttgart Domzale 92:8 90:10 1 0 MyPa Grashoppers 28:72 30:70 2 0 Hertha APOEL 83:17 80:20 1 0 Rennes Osasuna 43:57 40:60 1 -1 Basel Brijeg 87:13 80:20 1 -1 Setubal Genua 34:66 40:60 2 -1 Monaco Willem 73:28 65:35 1 -2 Leverkusen Sofia 66:34 70:30 2 -2 Krylya AZ 35:65 45:55 2 -2 Sevilla Mainz 69:31 60:40 1 -2 Shakhtar Debrecen 71:29 60:40 1 -2 Halmstad Sporting 24:76 20:80 1 -3 Slavia Cork 81:19 65:35 1 -3 Besiktas Malmö 69:31 55:45 1 -3 Hibernian Dnipro 42:58 60:40 2 -5 {/pre>
It may also be interesting to see the result of the second placed {em>delustef{/em>, because he gave only estimations for three matches, and gained 20 points: {pre> Average delustef Result Points Feyenoord Rapid 70:30 60:40 2 4 Din. Bukarest Everton 30:70 50:50 1 8 Valerenga Steaua 49:51 10:90 2 8 {/pre>
Finally, I will give you the detailed results of Koulis. It demonstates quite well what happens if you tend to give estimations which are too extreme. You will gain some additional points if the favorite wins, but you lose far more points when at least some of the outsiders win. {pre> Average Koulis Result Points Aris Roma 20:80 1:99 1 3 Besiktas Malmö 69:31 85:15 1 3 Palermo Anorthosis 73:27 90:10 1 3 Banik Heerenveen 42:58 30:70 2 3 Hertha APOEL 83:17 98:2 1 2 MyPa Grashoppers 28:72 15:85 2 2 Valerenga Steaua 49:51 40:60 2 2 Monaco Willem 73:28 85:15 1 2 Metalurg PAOK 49:51 40:60 2 2 Krylya AZ 35:65 25:75 2 2 CSKA Midtjylland 81:19 92:8 1 2 Setubal Genua 34:66 25:75 2 2 Shakhtar Debrecen 71:29 80:20 1 2 Stuttgart Domzale 92:8 99:1 1 1 Hamburg Kopenhagen 74:26 80:20 1 1 Lens Groclin 70:30 75:25 1 1 Bolton Plovdiv 75:25 80:20 1 1 Middlesborough Xanthi 80:20 85:15 1 1 Slavia Cork 81:19 85:15 1 1 Basel Brijeg 87:13 90:10 1 0 Viking Austria 39:61 40:60 1 0 Teplice Espaniol 32:68 30:70 2 0 Beerschot Marseilles 20:80 20:80 2 0 Zenit AEK 53:47 50:50 1 -1 Litex Genk 43:57 40:60 1 -1 Lok. Moskva Brann 70:30 60:40 1 -2 Guiamras Wisla 43:57 35:65 1 -3 Bröndby FC Zürich 55:45 45:55 1 -3 Rennes Osasuna 43:57 35:65 1 -3 Hibernian Dnipro 42:58 55:45 2 -4 Petach-Tikva Partizan 28:72 20:80 1 -5 Auxerre Levski 65:35 75:25 2 -5 Sevilla Mainz 69:31 50:50 1 -5 Tromso Gatala 28:72 20:80 1 -5 Red Star Braga 53:47 35:65 1 -6 Graz Strasbourg 52:48 70:30 2 -7 Din. Bukarest Everton 30:70 15:85 1 -10 Halmstad Sporting 24:76 10:90 1 -13 Leverkusen Sofia 66:34 90:10 2 -17 Feyenoord Rapid 70:30 95:5 2 -26 {/pre> |
|
|