Ok, I was curious as to how most of my benchmarks of memcpy() (I'm dealing with this as it is the most common routine and most of the rest are just copying the mechanisms there) showed a significant speed increase, but when I linked with MySQL I got very little -if at all- speed gains from that particular function. So I did a custom profile of memcpy() use in MySQL. In every call to memcpy() I wrote the following results to a file:
the result of __builtin_return_address(0) (will explain below)
src word-alignment (src % sizeof(uint32_t))
src quad-word-alignment (src % 16)
dst word-alignment (src % sizeof(uint32_t))
dst quad-word-alignment (src % 16)
length (in bytes)
The actual datafiles can be found here:
http://people.debian.org/~markos/powerpc/mysql-profile/
I ran some simple scripts (munge-stats.pl, provided by a friend of mine who gave me some good advice on optimizing, Pantelis Antoniou) and stats.pl, which is based on the first one but does presents results of a different kind.
I ran the scripts for a sample of 10M (10^6) calls to memcpy. The original set is ~100M, but it's a 2GB text file and I ran out of memory, while processing it
Code:
Same word-alignment stats:
dst4 -> count
0 -> 4180106
1 -> 191
2 -> 50575
3 -> 173
Different word-alignment stats:
dst4 -> count
0 -> 3692531
1 -> 468329
2 -> 979331
3 -> 628764
Same 16b-alignment stats:
dst16 -> count
0 -> 1385930
1 -> 21
2 -> 8136
3 -> 3
4 -> 21
5 -> 9
6 -> 1032
7 -> 3
8 -> 279855
9 -> 17
10 -> 8483
11 -> 31
12 -> 60
13 -> 0
14 -> 165
15 -> 3
Different 16b-alignment stats:
dst16 -> count
0 -> 3458045
1 -> 2697
2 -> 220950
3 -> 1259
4 -> 301838
5 -> 85562
6 -> 441238
7 -> 20405
8 -> 2443586
9 -> 1595
10 -> 258891
memcpy len stats:
len -> count
0 -> 13490
1 -> 230922
2 -> 377697
3 -> 759875
4 -> 359972
5 -> 1264863
6 -> 1461654
7 -> 377148
8 -> 930741
9 -> 167
10 -> 600106
11 -> 305529
12 -> 1018399
13 -> 57
14 -> 90
15 -> 9315
16 -> 1457
17 -> 123
18 -> 107
19 -> 300061
20 -> 69
21 -> 48
22 -> 55
23 -> 29
24 -> 933
25 -> 66
26 -> 42
27 -> 63
28 -> 44
29 -> 20
30 -> 59
31 -> 27
32 -> 1251
33 -> 145
34 -> 8
35 -> 598
36 -> 2023
37 -> 5903
38 -> 39
39 -> 58316
40 -> 754005
41 -> 70912
42 -> 5
43 -> 394837
44 -> 8
45 -> 10
46 -> 10
47 -> 12
48 -> 540
49 -> 3
50 -> 9
51 -> 6
52 -> 4
53 -> 1
55 -> 4
56 -> 151
57 -> 5
58 -> 15
59 -> 1
60 -> 336
61 -> 12
62 -> 535
63 -> 13
64 -> 5512
65 -> 5
66 -> 52965
67 -> 4
68 -> 17968
69 -> 182
70 -> 12
72 -> 3746
73 -> 1
74 -> 4
75 -> 18004
76 -> 4
78 -> 180068
80 -> 1599
81 -> 400000
84 -> 194
88 -> 72
91 -> 2
93 -> 5
94 -> 2
96 -> 271
102 -> 2
104 -> 82
108 -> 180
110 -> 1
112 -> 86
114 -> 1
120 -> 261
123 -> 3
124 -> 2
127 -> 2
128 -> 221
131 -> 1
132 -> 189
135 -> 2
136 -> 95
137 -> 4
138 -> 3
140 -> 12
144 -> 308
146 -> 1
152 -> 145
153 -> 49
156 -> 1868
157 -> 22
160 -> 1162
168 -> 1701
172 -> 2
174 -> 2
179 -> 11
180 -> 4
189 -> 1
191 -> 1
195 -> 4
196 -> 2
197 -> 4
228 -> 1
240 -> 11
243 -> 1
245 -> 1
256 -> 1
263 -> 4
274 -> 4
289 -> 4
318 -> 48
322 -> 27
324 -> 1464
328 -> 959
333 -> 42
336 -> 2405
413 -> 2
480 -> 2
492 -> 1
495 -> 1
512 -> 34
581 -> 2
602 -> 2
663 -> 117
672 -> 4828
680 -> 29
685 -> 4
16384 -> 7
And the results from munge-stats.pl. The fields represent:
return address
number of calls for this address
std deviation, if 0, the number in parenthesis represents the number of calls for this size.
Code:
key nr std-dev
0x101b3e18 1 0 (31) *
0x101b3f1c 1 0 (50) *
0x101ba0cc 1 0 (50) *
0x101e4d0c 1 0 (256) *
0x102151a4 1 0 (36) *
0x10215448 1 0 (4) *
0x102cc250 1 0 (512) *
0x102e3d2c 1 0 (16384) *
0x102f5b04 1 0 (512) *
0x1032e560 1 0 (46) *
0x1032f88c 1 0 (32) *
0x1032fba0 1 0 (32) *
0x103dc158 1 0 (16) *
0x1040a608 1 0 (512) *
0x1040ed14 1 0 (20) *
0x10149890 2 0 (1) *
0x10254298 2 3.60555127546399
0x102ee778 2 0 (512) *
0x1030f1fc 2 2.82842712474619
0x1033e1a0 2 3.60555127546399
0x1014bfd8 3 0 (12) *
0x1030e838 3 2.23606797749979
0x103e03b4 3 0 (3) *
0x103e053c 3 2.23606797749979
0xfea2b64 3 0 (128) *
0x102587d0 4 0 (6) *
0x10411328 4 0 (7) *
0x1041ca54 4 0 (80) *
0x1041cbf4 4 14
0xf883460 4 0 (36) *
0xfeef598 4 0 (15) *
0xff47028 4 0 (7) *
0xff5c7ac 4 0 (180) *
0x1025a2a0 5 3.16227766016838
0x10358358 6 16921.3224365
0x103dcd1c 6 0 (16) *
0x101525a0 8 0 (128) *
0x102576cc 8 4
0xff610f8 10 1.73205080756888
0x1013f3bc 11 0 (240) *
0x10402a88 11 9.4339811320566
0x104302d4 12 67.3424086293325
0xfc0896c 12 0 (140) *
0xfee3af8 12 321.158839205774
0xfc06264 15 0 (128) *
0x103dc0c8 17 4.69041575982343
0x10414ae8 18 2
0xfed775c 19 21.3072757526625
0xfea2ac4 24 0 (128) *
0x10189474 29 105.047608254543
0x10189498 29 0 (0) *
0x1018bbdc 29 4.35889894354067
0x101e1f28 29 5.3851648071345
0x103cc330 29 10.0498756211209
0x103cc4f4 29 0 (512) *
0x103ccbd0 29 0 (680) *
0x103ccbe8 29 5.3851648071345
0x103ccbfc 29 3.87298334620742
0x103ccc14 29 2
0xfeb4594 30 8.94427190999916
0x10431bc8 34 1
0x1043206c 34 4.58257569495584
0x101cef8c 36 4.24264068711928
0x1018a2bc 57 155.692003648229
0x10431bfc 83 3.87298334620742
0x1034f7c8 90 2.82842712474619
0xfc08780 95 0 (128) *
0x103dc238 132 0 (1) *
0x1034ea30 152 0 (7) *
0xff183b0 155 7.48331477354788
0x104111f8 177 12.2474487139159
0x10419794 180 14.142135623731
0x10411640 249 6.40312423743285
0x10443068 311 1
0x10442a38 418 2
0x10416d80 438 0 (8) *
0xfed7a20 557 23.3452350598575
0x10148554 945 8.24621125123532
0x103e06c4 1964 1.73205080756888
0x103d49cc 1967 1.73205080756888
0x103e02d0 1967 18.9208879284245
0x103e0b10 4945 646.069655687372
0x103e0b24 4945 13.3790881602597
0x103e0b58 4945 330.607017469382
0x103e0b68 4945 13.3790881602597
0x103e0b7c 4945 13.3790881602597
0x103e0b90 4945 13.3790881602597
0x103e0a10 11098 10.3923048454133
0x103e0a20 11098 10.3923048454133
0x103e0a30 11098 85.9825563704639
0x103e08e8 13262 10.0498756211209
0x103e0908 13262 87.4356906531881
0x103e091c 13262 10.0498756211209
0x10411f24 77062 39.2683078321437
0x1014d74c 153712 5.09901951359278
0x10432d54 153728 5.09901951359278
0x101344d0 376855 8.88819441731559
0x101a5810 376855 43
0x103d69b0 376855 39.9874980462644
0x100bdc5c 376856 8.88819441731559
0x103d68f8 376882 40
0x10167914 376885 73.5934779718964
0x1041201c 376904 7
0x103e90bc 606919 10.0995049383621
0x103d4ac0 608879 10.1980390271856
0x103d4828 678822 10.4403065089106
0x10147760 754516 13.8924439894498
0x1040ff78 1200011 41.5090351610345
0x10411fa0 3014882 4.69041575982343
So, we see from these results why the altivec version of memcpy() does not really benefit MySQL: the sizes used are too small. Still, I see some other information that might be of benefit and still allow me to get some optimizations:
* Most calls were with aligned addresses. Specifically, the majority of calls had the destination pointer (dst) aligned to either a longword boundary (4b) or quad-word (16b), used by Altivec.
* Also, lots of calls were made with very small sizes, with very few calls reaching up to 16384 (the max size used by memcpy). 99% of the calls were below 100 bytes, and this tells me that I could use that to my benefit, writing optimized code that offers max speed for these cases.
* For the munge-stats.pl results, I will need to explain sth first. As Pantelis told me, when __builtin_return_address is different, this means a new memory allocation. The last column represents the standard deviation for the size used in the call to memcpy(). If std-dev is 0, it means this call was always made with a constant size. For constant allocations, these cases could be handled by gcc itself, inlining faster code that handles copying of preset sizes (eg. small sizes and/or sizes % 16 == 0). This was an important observation and one that might be used to as an useful generic optimization to gcc. This will be something that I will try to look in the future, though, unless someone beats me to it
So my plans is to rewrite the code to take advantage of these results. Namely, I consider doing the following:
* Assume word-alignment and do copy as usual. Use Altivec if size permits.
* If alignment was not on a 4b, then copy the initial remaining bytes, otherwise return.
Now this might or it might not offer any improvement, it remains to be seen. Of course these conclusions apply to most other glibc functions I'm working on (mem*, etc).
Last, I tried to do some profiling with gprof on MySQL to actually measure the time that memcpy takes overall, but in all gprof calls I tried afterwards, I could not get any glibc function to show in the summary. Any gprof-guru around?
So, conclusion? MySQL won't benefit much from vectorization/optimization of the glibc functions. It will benefit, but not much. As people told me, MySQL is mostly I/O bound. Given these results I don't expect more than 10-20% max. It will probably benefit more from aforementioned gcc optimizations, or vectorization of the hashing techniques used in MySQL itself. But this is another topic, which I will discuss in another post.
What does this mean? I need to find some other memory-bound application that will show that the Altivec optimizations in glibc are worth the effort. Any ideas?
Konstantinos
Login
Register
FAQ
Search