aboutsummaryrefslogtreecommitdiff
diff options
context:
space:
mode:
-rw-r--r--benchmarks/benchmarks.md155
-rw-r--r--benchmarks/img/16threads.pngbin0 -> 81629 bytes
-rw-r--r--benchmarks/img/17moves16threads.pngbin119331 -> 0 bytes
-rw-r--r--benchmarks/img/17moves1thread.pngbin112954 -> 110266 bytes
-rw-r--r--benchmarks/img/17moves4threads.pngbin114709 -> 0 bytes
-rw-r--r--benchmarks/img/18moves16threads.pngbin112774 -> 0 bytes
-rw-r--r--benchmarks/img/18moves1thread.pngbin113328 -> 110481 bytes
-rw-r--r--benchmarks/img/18moves4threads.pngbin117736 -> 0 bytes
-rw-r--r--benchmarks/img/19moves16threads.pngbin117654 -> 0 bytes
-rw-r--r--benchmarks/img/19moves1thread.pngbin119075 -> 116134 bytes
-rw-r--r--benchmarks/img/19moves4threads.pngbin119488 -> 0 bytes
-rw-r--r--benchmarks/img/20moves16threads.pngbin121476 -> 0 bytes
-rw-r--r--benchmarks/img/20moves1thread.pngbin112429 -> 108798 bytes
-rw-r--r--benchmarks/img/20moves4threads.pngbin123801 -> 0 bytes
-rw-r--r--benchmarks/img/4threads.pngbin0 -> 88252 bytes
-rw-r--r--benchmarks/plot-benchmarks.py216
-rw-r--r--benchmarks/results_h48_clang.py36
-rw-r--r--benchmarks/tables_16_threads.md27
-rw-r--r--benchmarks/tables_1_thread.md40
-rw-r--r--benchmarks/tables_4_threads.md27
-rw-r--r--benchmarks/tables_all_solutions.md26
21 files changed, 309 insertions, 218 deletions
diff --git a/benchmarks/benchmarks.md b/benchmarks/benchmarks.md
index 768a7ea..3fc2093 100644
--- a/benchmarks/benchmarks.md
+++ b/benchmarks/benchmarks.md
@@ -38,21 +38,6 @@ may be particularly inaccurate, because writing the solution (together
38with some log messages) to standard output may take a significant portion 38with some log messages) to standard output may take a significant portion
39of the time, depending on the terminal emulator used and other factors. 39of the time, depending on the terminal emulator used and other factors.
40 40
41The main test we performed was finding a single optimal solution, and we
42compared the results with vcube. This test was run in a single-thread
43configuration and in two multithread configurations (with 4 and 16
44threads). We also ran a test on finding *all* optimal solutions which,
45as far as I know, is a use case not supported by vcube; this latter test
46was only run on 16 threads.
47
48Since the size of the pruning table used by the solver is of utmost
49importance, we include two statistics: time per cube and time per cube
50*adjusted by table size*. The adjustment we took simply consists in
51multiplying the time per cube by the size of the table. Empirically
52this leads to more even results across the board, as the speed of the
53solvers of the same family seems to scale linearly with the size of the
54pruning table.
55
56All benchmark tests were done on the following configuration: 41All benchmark tests were done on the following configuration:
57 42
58* CPU: AMD Ryzen 7 7700 (8 cores, 16 virtual threads) 43* CPU: AMD Ryzen 7 7700 (8 cores, 16 virtual threads)
@@ -61,9 +46,29 @@ All benchmark tests were done on the following configuration:
61* Operating system: Debian 13 (Linux kernel 6.12.57) 46* Operating system: Debian 13 (Linux kernel 6.12.57)
62* Compiler: GCC 14.2.0 for H48 and Clang 19.1.7 for vcube 47* Compiler: GCC 14.2.0 for H48 and Clang 19.1.7 for vcube
63 48
64## Results (click on each item to expand) 49## Benchmark results
50
51### Single solution, single thread
52
53The first test we performed was finding a single optimal solution, and
54we compared the results with vcube.
65 55
66<details><summary>Single solution, single thread</summary> 56This test was run in a single-thread configuration, because the two
57solvers use very different strategies for multithreading: vcube can only
58parallelize by solving one cube per thread, while H48 is able to take
59advantage of multiple threads even when solving a single cube. Therefore
60we prefer to compare their single-threaded performance only.
61
62Since the size of the pruning table used by the solver is of utmost
63importance, we include two statistics: time per cube and time per cube
64*adjusted by table size*. The adjustment we took simply consists in
65multiplying the time per cube by the size of the table. Empirically this
66leads to more even results across the board, although the speed of the
67solvers of the same family does not scale exactly linearly with the size
68of the pruning table.
69
70<!-- The following details block can be found in benchmarks/single_thread_comparison.md -->
71<details><summary>Results: Single solution, single thread</summary>
67 72
68Time per cube (in seconds, lower is better). 73Time per cube (in seconds, lower is better).
69 74
@@ -103,95 +108,92 @@ Time per cube adjusted for table size (in seconds \* GiB, lower is better).
103<img src="img/20moves1thread.png"> 108<img src="img/20moves1thread.png">
104</details> 109</details>
105 110
106<details><summary>Single solution, 4 threads</summary> 111As we can see, adjusting for table size, H48 is generally faster than
112vcube. The gap between the two solvers is larger for scrambles with
113longer optimal solutions.
114
115### Single solution, multiple threads
116
117The same benchmark as before is repeated using 4 and 16 threads (recall
118that the CPU used for these tests has 8 physical cores and 16 virtual
119threads).
120
121As mentioned above, we don't compare these results to vcube. Instead,
122we compare them with the single-threaded results for H48 and we show
123how far the speedup factor is from a theoretically optimal 4x and 16x.
124
125<!-- The following details block can be found in benchmarks/tables_4_threads.md -->
126<details><summary>Results: Single solution, 4 threads</summary>
107 127
108Time per cube (in seconds, lower is better). 128Time per cube (in seconds, lower is better).
109 129
110| Solver | Size |17 moves|18 moves|19 moves|20 moves|Superflip| 130| Solver | Size |17 moves|18 moves|19 moves|20 moves|Superflip|
111|:---------|:-------|-------:|-------:|-------:|-------:|--------:| 131|:---------|:-------|-------:|-------:|-------:|-------:|--------:|
112|vcube 212 |58.2 GiB| 0.03| 0.27| 1.04| 7.70| (a) |
113|H48 h11 |56.5 GiB| 0.04| 0.14| 0.60| 3.97| 4.19| 132|H48 h11 |56.5 GiB| 0.04| 0.14| 0.60| 3.97| 4.19|
114|vcube 404 |31.8 GiB| 0.07| 0.30| 1.65| 16.17| (a) |
115|H48 h10 |28.3 GiB| 0.05| 0.20| 0.91| 6.65| 9.81| 133|H48 h10 |28.3 GiB| 0.05| 0.20| 0.91| 6.65| 9.81|
116|vcube 308 |21.2 GiB| 0.05| 0.35| 1.78| 16.61| (a) |
117|H48 h9 |14.1 GiB| 0.07| 0.39| 1.74| 12.54| 18.96| 134|H48 h9 |14.1 GiB| 0.07| 0.39| 1.74| 12.54| 18.96|
118|vcube 208 | 7.3 GiB| 0.16| 1.47| 5.86| | (a) |
119|H48 h8 | 7.1 GiB| 0.13| 0.90| 3.71| | | 135|H48 h8 | 7.1 GiB| 0.13| 0.90| 3.71| | |
120|H48 h7 | 3.5 GiB| 0.17| 1.28| 6.12| | | 136|H48 h7 | 3.5 GiB| 0.17| 1.28| 6.12| | |
121|vcube 112 | 2.4 GiB| 0.29| 3.13| 11.95| | (a) |
122|H48 h6 | 1.8 GiB| 0.33| 2.47| 12.22| | | 137|H48 h6 | 1.8 GiB| 0.33| 2.47| 12.22| | |
123 138
124Time per cube adjusted for table size (in seconds \* GiB, lower is better). 139Speed-up factor (higher is better).
125 140
126| Solver | Size |17 moves|18 moves|19 moves|20 moves|Superflip| 141| Solver | Size |17 moves|18 moves|19 moves|20 moves|Superflip|
127|:---------|:-------|-------:|-------:|-------:|-------:|--------:| 142|:---------|:-------|-------:|-------:|-------:|-------:|--------:|
128|vcube 212 |58.2 GiB| 2.03| 15.69| 60.25| 447.97| (a) | 143|H48 h11 |56.5 GiB| 2.55| 3.46| 3.75| 3.96| 3.71|
129|H48 h11 |56.5 GiB| 2.04| 8.17| 33.77| 224.59| 236.92| 144|H48 h10 |28.3 GiB| 3.09| 3.71| 3.67| 3.54| 3.88|
130|vcube 404 |31.8 GiB| 2.32| 9.50| 52.46| 514.50| (a) | 145|H48 h9 |14.1 GiB| 3.25| 3.84| 3.85| | |
131|H48 h10 |28.3 GiB| 1.37| 5.76| 25.83| 187.88| 277.18| 146|H48 h8 | 7.1 GiB| 3.53| 3.72| 3.81| | |
132|vcube 308 |21.2 GiB| 1.02| 7.52| 37.82| 352.36| (a) | 147|H48 h7 | 3.5 GiB| 3.60| 3.78| 3.80| | |
133|H48 h9 |14.1 GiB| 1.03| 5.45| 24.58| 177.17| 267.79| 148|H48 h6 | 1.8 GiB| 3.77| 3.82| 3.79| | |
134|vcube 208 | 7.3 GiB| 1.18| 10.69| 42.63| | (a) | 149<img src="img/4threads.png">
135|H48 h8 | 7.1 GiB| 0.92| 6.39| 26.20| | |
136|H48 h7 | 3.5 GiB| 0.62| 4.53| 21.62| | |
137|vcube 112 | 2.4 GiB| 0.69| 7.59| 28.97| | (a) |
138|H48 h6 | 1.8 GiB| 0.59| 4.37| 21.61| | |
139
140(a) vcube cannot parallelize on a single scramble, the results for the
141superflip are going to be the same as in the single thread case.
142 150
143<img src="img/17moves4threads.png">
144<img src="img/18moves4threads.png">
145<img src="img/19moves4threads.png">
146<img src="img/20moves4threads.png">
147</details> 151</details>
148 152
149<details><summary>Single solution, 16 threads</summary> 153<!-- The following details block can be found in benchmarks/tables_16_threads.md -->
154<details><summary>Results: Single solution, 16 threads</summary>
150 155
151Time per cube (in seconds, lower is better). 156Time per cube (in seconds, lower is better).
152 157
153| Solver | Size |17 moves|18 moves|19 moves|20 moves|Superflip| 158| Solver | Size |17 moves|18 moves|19 moves|20 moves|Superflip|
154|:---------|:-------|-------:|-------:|-------:|-------:|--------:| 159|:---------|:-------|-------:|-------:|-------:|-------:|--------:|
155|vcube 212 |58.2 GiB| 0.02| 0.13| 0.45| 2.84| (a) |
156|H48 h11 |56.5 GiB| 0.02| 0.06| 0.22| 1.33| 1.84| 160|H48 h11 |56.5 GiB| 0.02| 0.06| 0.22| 1.33| 1.84|
157|vcube 404 |31.8 GiB| 0.04| 0.14| 0.65| 6.08| (a) |
158|H48 h10 |28.3 GiB| 0.03| 0.08| 0.33| 2.34| 4.18| 161|H48 h10 |28.3 GiB| 0.03| 0.08| 0.33| 2.34| 4.18|
159|vcube 308 |21.2 GiB| 0.03| 0.19| 0.78| 6.67| (a) |
160|H48 h9 |14.1 GiB| 0.04| 0.15| 0.64| 4.45| 8.09| 162|H48 h9 |14.1 GiB| 0.04| 0.15| 0.64| 4.45| 8.09|
161|vcube 208 | 7.3 GiB| 0.08| 0.79| 2.43| | (a) |
162|H48 h8 | 7.1 GiB| 0.06| 0.34| 1.36| | | 163|H48 h8 | 7.1 GiB| 0.06| 0.34| 1.36| | |
163|H48 h7 | 3.5 GiB| 0.07| 0.47| 2.20| | | 164|H48 h7 | 3.5 GiB| 0.07| 0.47| 2.20| | |
164|vcube 112 | 2.4 GiB| 0.15| 1.63| 5.10| | (a) |
165|H48 h6 | 1.8 GiB| 0.13| 0.91| 4.39| | | 165|H48 h6 | 1.8 GiB| 0.13| 0.91| 4.39| | |
166 166
167Time per cube adjusted for table size (in seconds \* GiB, lower is better). 167Speed-up factor (higher is better).
168 168
169| Solver | Size |17 moves|18 moves|19 moves|20 moves|Superflip| 169| Solver | Size |17 moves|18 moves|19 moves|20 moves|Superflip|
170|:---------|:-------|-------:|-------:|-------:|-------:|--------:| 170|:---------|:-------|-------:|-------:|-------:|-------:|--------:|
171|vcube 212 |58.2 GiB| 0.95| 7.83| 26.04| 165.03| (a) | 171|H48 h11 |56.5 GiB| 3.90| 8.14| 10.39| 11.86| 8.45|
172|H48 h11 |56.5 GiB| 1.34| 3.48| 12.19| 74.93| 103.98| 172|H48 h10 |28.3 GiB| 5.72| 9.00| 10.03| 10.04| 9.09|
173|vcube 404 |31.8 GiB| 1.21| 4.60| 20.76| 193.43| (a) | 173|H48 h9 |14.1 GiB| 6.56| 9.75| 10.53| | |
174|H48 h10 |28.3 GiB| 0.74| 2.37| 9.46| 66.13| 118.23| 174|H48 h8 | 7.1 GiB| 8.08| 9.91| 10.40| | |
175|vcube 308 |21.2 GiB| 0.67| 4.01| 16.48| 141.49| (a) | 175|H48 h7 | 3.5 GiB| 8.59| 10.30| 10.57| | |
176|H48 h9 |14.1 GiB| 0.51| 2.15| 8.98| 62.92| 114.31| 176|H48 h6 | 1.8 GiB| 9.60| 10.39| 10.54| | |
177|vcube 208 | 7.3 GiB| 0.56| 5.78| 17.68| | (a) | 177<img src="img/16threads.png">
178|H48 h8 | 7.1 GiB| 0.40| 2.40| 9.59| | |
179|H48 h7 | 3.5 GiB| 0.26| 1.66| 7.77| | |
180|vcube 112 | 2.4 GiB| 0.35| 3.95| 12.37| | (a) |
181|H48 h6 | 1.8 GiB| 0.23| 1.61| 7.76| | |
182 178
183(a) vcube cannot parallelize on a single scramble, the results for the
184superflip are going to be the same as in the single thread case.
185
186<img src="img/17moves16threads.png">
187<img src="img/18moves16threads.png">
188<img src="img/19moves16threads.png">
189<img src="img/20moves16threads.png">
190</details> 179</details>
191 180
192<details><summary>All solutions, 16 threads</summary> 181We can see that H48 scales pretty well with 4 threads, getting close
182to the 4x theoretical maximum speedup in slower cases (small table or
183long solutions).
184
185In the 16 threads benchmark shows that, although the virtual threads
186help push us beyond the 8x theoretical speedup that would be provided
187by the 8 cores, we are nowhere near a 16x speedup.
188
189### All solutions
193 190
194*Note: vcube does not have an option for finding multiple solutions.* 191Finally, we ran a test on finding *all* optimal solutions which, as
192far as I know, is a use case not supported by vcube. For convenience,
193this test is only run on 16 threads.
194
195<!-- The following details block can be found in benchmarks/all_solutions.md -->
196<details><summary>Results: All solutions (16 threads)</summary>
195 197
196Time per cube (in seconds, lower is better). 198Time per cube (in seconds, lower is better).
197 199
@@ -217,16 +219,6 @@ Time per cube adjusted for table size (in seconds \* GiB, lower is better).
217 219
218</details> 220</details>
219 221
220## Comments on the results
221
222* Adjusting for table size, H48 is generally faster than vcube.
223* The gap between the two solvers is larger for scrambles with
224 longer optimal solutions.
225* On sets of 25 scrambles, H48 performs better than vcube when using
226 multiple threads, compared to their single-threaded performance.
227 However, this advantage will likely disappear (or
228 even invert) if we increase the size of the set.
229
230## Other notes 222## Other notes
231 223
232* To repeat the benchmarks, use `./benchmarks/run-h48-benchmarks.sh`. 224* To repeat the benchmarks, use `./benchmarks/run-h48-benchmarks.sh`.
@@ -249,8 +241,3 @@ Time per cube adjusted for table size (in seconds \* GiB, lower is better).
249* For H48, both GCC and Clang have been tried, with the same options; 241* For H48, both GCC and Clang have been tried, with the same options;
250 the resulting executable was about 10% faster with GCC compared to Clang. 242 the resulting executable was about 10% faster with GCC compared to Clang.
251 vcube only supports compiling with Clang. 243 vcube only supports compiling with Clang.
252* The performance of the H48 solver depends slightly, but measurably, on the
253 alignment of the pruning table in memory. This is not handled by the core
254 library, but by the program that uses it. For these tests, we have used as
255 reference implementation the program in `tools/301_solve_file`, which
256 ensures 64-byte alignment.
diff --git a/benchmarks/img/16threads.png b/benchmarks/img/16threads.png
new file mode 100644
index 0000000..8161758
--- /dev/null
+++ b/benchmarks/img/16threads.png
Binary files differ
diff --git a/benchmarks/img/17moves16threads.png b/benchmarks/img/17moves16threads.png
deleted file mode 100644
index 45f5901..0000000
--- a/benchmarks/img/17moves16threads.png
+++ /dev/null
Binary files differ
diff --git a/benchmarks/img/17moves1thread.png b/benchmarks/img/17moves1thread.png
index 70bcad0..da78107 100644
--- a/benchmarks/img/17moves1thread.png
+++ b/benchmarks/img/17moves1thread.png
Binary files differ
diff --git a/benchmarks/img/17moves4threads.png b/benchmarks/img/17moves4threads.png
deleted file mode 100644
index edfca34..0000000
--- a/benchmarks/img/17moves4threads.png
+++ /dev/null
Binary files differ
diff --git a/benchmarks/img/18moves16threads.png b/benchmarks/img/18moves16threads.png
deleted file mode 100644
index df5210a..0000000
--- a/benchmarks/img/18moves16threads.png
+++ /dev/null
Binary files differ
diff --git a/benchmarks/img/18moves1thread.png b/benchmarks/img/18moves1thread.png
index dafba4e..c9406fa 100644
--- a/benchmarks/img/18moves1thread.png
+++ b/benchmarks/img/18moves1thread.png
Binary files differ
diff --git a/benchmarks/img/18moves4threads.png b/benchmarks/img/18moves4threads.png
deleted file mode 100644
index 9b89bed..0000000
--- a/benchmarks/img/18moves4threads.png
+++ /dev/null
Binary files differ
diff --git a/benchmarks/img/19moves16threads.png b/benchmarks/img/19moves16threads.png
deleted file mode 100644
index 42212a6..0000000
--- a/benchmarks/img/19moves16threads.png
+++ /dev/null
Binary files differ
diff --git a/benchmarks/img/19moves1thread.png b/benchmarks/img/19moves1thread.png
index f51886e..8400584 100644
--- a/benchmarks/img/19moves1thread.png
+++ b/benchmarks/img/19moves1thread.png
Binary files differ
diff --git a/benchmarks/img/19moves4threads.png b/benchmarks/img/19moves4threads.png
deleted file mode 100644
index 44ac692..0000000
--- a/benchmarks/img/19moves4threads.png
+++ /dev/null
Binary files differ
diff --git a/benchmarks/img/20moves16threads.png b/benchmarks/img/20moves16threads.png
deleted file mode 100644
index 82c28b2..0000000
--- a/benchmarks/img/20moves16threads.png
+++ /dev/null
Binary files differ
diff --git a/benchmarks/img/20moves1thread.png b/benchmarks/img/20moves1thread.png
index 1f285d9..879448d 100644
--- a/benchmarks/img/20moves1thread.png
+++ b/benchmarks/img/20moves1thread.png
Binary files differ
diff --git a/benchmarks/img/20moves4threads.png b/benchmarks/img/20moves4threads.png
deleted file mode 100644
index c23fe81..0000000
--- a/benchmarks/img/20moves4threads.png
+++ /dev/null
Binary files differ
diff --git a/benchmarks/img/4threads.png b/benchmarks/img/4threads.png
new file mode 100644
index 0000000..a0cfb2f
--- /dev/null
+++ b/benchmarks/img/4threads.png
Binary files differ
diff --git a/benchmarks/plot-benchmarks.py b/benchmarks/plot-benchmarks.py
index 46031c6..027e0fe 100644
--- a/benchmarks/plot-benchmarks.py
+++ b/benchmarks/plot-benchmarks.py
@@ -1,6 +1,14 @@
1import matplotlib.pyplot as plt 1import matplotlib.pyplot as plt
2import results_h48 2import results_h48
3import results_vcube 3import results_vcube
4from pathlib import Path
5
6benchmarks_dir = Path("benchmarks")
7benchmarks_img_dir = benchmarks_dir / "img"
8benchmarks_single_thread = benchmarks_dir / "tables_1_thread.md"
9benchmarks_4_threads = benchmarks_dir / "tables_4_threads.md"
10benchmarks_16_threads = benchmarks_dir / "tables_16_threads.md"
11benchmarks_all_solutions = benchmarks_dir / "tables_all_solutions.md"
4 12
5# Table sizes in bytes 13# Table sizes in bytes
6 14
@@ -23,7 +31,7 @@ sizes_vcube = {
23 31
24# Printing tables in markdown format 32# Printing tables in markdown format
25 33
26def print_row(solver_name, solver_size, dict, mul_by_size, superflip_star): 34def print_row(f, solver_name, solver_size, dict, mul_by_size, superflip_star):
27 if dict is None: 35 if dict is None:
28 return 36 return
29 solver_gib = solver_size / (2**30) 37 solver_gib = solver_size / (2**30)
@@ -36,104 +44,106 @@ def print_row(solver_name, solver_size, dict, mul_by_size, superflip_star):
36 f"{dict["superflip"]*m:>9.2f}" if "superflip" in dict else s 44 f"{dict["superflip"]*m:>9.2f}" if "superflip" in dict else s
37 ] 45 ]
38 sep = "|" 46 sep = "|"
39 print(sep + sep.join(cols) + sep) 47 f.write(sep + sep.join(cols) + sep + "\n")
40 48
41def print_table(h48, vcube, ms, st): 49def print_table(f, h48, vcube, ms, st):
42 vcube = vcube or {} 50 vcube = vcube or {}
43 print("| Solver | Size |17 moves|18 moves|19 moves|20 moves|Superflip|") 51 f.write("| Solver | Size |17 moves|18 moves|19 moves|20 moves|Superflip|\n")
44 print("|:---------|:-------|-------:|-------:|-------:|-------:|--------:|") 52 f.write("|:---------|:-------|-------:|-------:|-------:|-------:|--------:|\n")
45 print_row("vcube 212", sizes_vcube[212], vcube.get(212), ms, not st) 53 print_row(f, "vcube 212", sizes_vcube[212], vcube.get(212), ms, not st)
46 print_row("H48 h11", sizes_h48[11], h48[11], ms, False) 54 print_row(f, "H48 h11", sizes_h48[11], h48[11], ms, False)
47 print_row("vcube 404", sizes_vcube[404], vcube.get(404), ms, not st) 55 print_row(f, "vcube 404", sizes_vcube[404], vcube.get(404), ms, not st)
48 print_row("H48 h10", sizes_h48[10], h48[10], ms, False) 56 print_row(f, "H48 h10", sizes_h48[10], h48[10], ms, False)
49 print_row("vcube 308", sizes_vcube[308], vcube.get(308), ms, not st) 57 print_row(f, "vcube 308", sizes_vcube[308], vcube.get(308), ms, not st)
50 print_row("H48 h9", sizes_h48[9], h48[9], ms, False) 58 print_row(f, "H48 h9", sizes_h48[9], h48[9], ms, False)
51 print_row("vcube 208", sizes_vcube[208], vcube.get(208), ms, not st) 59 print_row(f, "vcube 208", sizes_vcube[208], vcube.get(208), ms, not st)
52 print_row("H48 h8", sizes_h48[8], h48[8], ms, False) 60 print_row(f, "H48 h8", sizes_h48[8], h48[8], ms, False)
53 print_row("H48 h7", sizes_h48[7], h48[7], ms, False) 61 print_row(f, "H48 h7", sizes_h48[7], h48[7], ms, False)
54 print_row("vcube 112", sizes_vcube[112], vcube.get(112), ms, not st) 62 print_row(f, "vcube 112", sizes_vcube[112], vcube.get(112), ms, not st)
55 print_row("H48 h6", sizes_h48[6], h48[6], ms, False) 63 print_row(f, "H48 h6", sizes_h48[6], h48[6], ms, False)
64
65def print_factor_table(f, slow, fast):
66 ratio = {}
67 for m in [6, 7, 8, 9, 10, 11]:
68 ratio[m] = {}
69 for k in slow[m]:
70 ratio[m][k] = (slow[m][k] / fast[m][k]) * (25 if k != "superflip" else 1)
71 f.write("| Solver | Size |17 moves|18 moves|19 moves|20 moves|Superflip|\n")
72 f.write("|:---------|:-------|-------:|-------:|-------:|-------:|--------:|\n")
73 print_row(f, "H48 h11", sizes_h48[11], ratio[11], False, False)
74 print_row(f, "H48 h10", sizes_h48[10], ratio[10], False, False)
75 print_row(f, "H48 h9", sizes_h48[9], ratio[9], False, False)
76 print_row(f, "H48 h8", sizes_h48[8], ratio[8], False, False)
77 print_row(f, "H48 h7", sizes_h48[7], ratio[7], False, False)
78 print_row(f, "H48 h6", sizes_h48[6], ratio[6], False, False)
56 79
57print("<details><summary>Single solution, single thread</summary>") 80with open(benchmarks_single_thread, "w") as f:
58print() 81 f.write(f"<!-- The following details block can be found in {benchmarks_single_thread} -->\n")
59print("Time per cube (in seconds, lower is better).") 82 f.write("<details><summary>Results: Single solution, single thread</summary>\n")
60print() 83 f.write("\n")
61print_table(results_h48.h48_single_thread, results_vcube.vcube_single_thread, False, True) 84 f.write("Time per cube (in seconds, lower is better).\n")
62print() 85 f.write("\n")
63print("Time per cube adjusted for table size (in seconds \\* GiB, lower is better).") 86 print_table(f, results_h48.h48_single_thread, results_vcube.vcube_single_thread, False, True)
64print() 87 f.write("\n")
65print_table(results_h48.h48_single_thread, results_vcube.vcube_single_thread, True, True) 88 f.write("Time per cube adjusted for table size (in seconds \\* GiB, lower is better).\n")
66print() 89 f.write("\n")
67print( 90 print_table(f, results_h48.h48_single_thread, results_vcube.vcube_single_thread, True, True)
68 '<img src="img/17moves1thread.png">\n' 91 f.write("\n")
69 '<img src="img/18moves1thread.png">\n' 92 f.write(
70 '<img src="img/19moves1thread.png">\n' 93 '<img src="img/17moves1thread.png">\n'
71 '<img src="img/20moves1thread.png">' 94 '<img src="img/18moves1thread.png">\n'
72) 95 '<img src="img/19moves1thread.png">\n'
73print("</details>") 96 '<img src="img/20moves1thread.png">\n'
97 )
98 f.write("</details>\n")
74 99
75print() 100with open(benchmarks_4_threads, "w") as f:
76print("<details><summary>Single solution, 4 threads</summary>") 101 f.write(f"<!-- The following details block can be found in {benchmarks_4_threads} -->\n")
77print() 102 f.write("<details><summary>Results: Single solution, 4 threads</summary>\n")
78print("Time per cube (in seconds, lower is better).") 103 f.write("\n")
79print() 104 f.write("Time per cube (in seconds, lower is better).\n")
80print_table(results_h48.h48_4_threads, results_vcube.vcube_4_threads, False, False) 105 f.write("\n")
81print() 106 print_table(f, results_h48.h48_4_threads, None, False, False)
82print("Time per cube adjusted for table size (in seconds \\* GiB, lower is better).") 107 f.write("\n")
83print() 108 f.write("Speed-up factor (higher is better).\n")
84print_table(results_h48.h48_4_threads, results_vcube.vcube_4_threads, True, False) 109 f.write("\n")
85print() 110 print_factor_table(f, results_h48.h48_single_thread, results_h48.h48_4_threads)
86print("(a) vcube cannot parallelize on a single scramble, the results for the") 111 f.write('<img src="img/4threadsspeedupfactor.png">\n')
87print("superflip are going to be the same as in the single thread case.") 112 f.write("\n")
88print() 113 f.write("</details>\n")
89print(
90 '<img src="img/17moves4threads.png">\n'
91 '<img src="img/18moves4threads.png">\n'
92 '<img src="img/19moves4threads.png">\n'
93 '<img src="img/20moves4threads.png">'
94)
95print("</details>")
96 114
97print() 115with open(benchmarks_16_threads, "w") as f:
98print("<details><summary>Single solution, 16 threads</summary>") 116 f.write(f"<!-- The following details block can be found in {benchmarks_16_threads} -->\n")
99print() 117 f.write("<details><summary>Results: Single solution, 16 threads</summary>\n")
100print("Time per cube (in seconds, lower is better).") 118 f.write("\n")
101print() 119 f.write("Time per cube (in seconds, lower is better).\n")
102print_table(results_h48.h48_16_threads, results_vcube.vcube_16_threads, False, False) 120 f.write("\n")
103print() 121 print_table(f, results_h48.h48_16_threads, None, False, False)
104print("Time per cube adjusted for table size (in seconds \\* GiB, lower is better).") 122 f.write("\n")
105print() 123 f.write("Speed-up factor (higher is better).\n")
106print_table(results_h48.h48_16_threads, results_vcube.vcube_16_threads, True, False) 124 f.write("\n")
107print() 125 print_factor_table(f, results_h48.h48_single_thread, results_h48.h48_16_threads)
108print("(a) vcube cannot parallelize on a single scramble, the results for the") 126 f.write('<img src="img/16threadsspeedupfactor.png">\n')
109print("superflip are going to be the same as in the single thread case.") 127 f.write("\n")
110print() 128 f.write("</details>\n")
111print(
112 '<img src="img/17moves16threads.png">\n'
113 '<img src="img/18moves16threads.png">\n'
114 '<img src="img/19moves16threads.png">\n'
115 '<img src="img/20moves16threads.png">'
116)
117print("</details>")
118 129
119print() 130with open(benchmarks_all_solutions, "w") as f:
120print("<details><summary>All solutions, 16 threads</summary>") 131 f.write(f"<!-- The following details block can be found in {benchmarks_all_solutions} -->\n")
121print() 132 f.write("<details><summary>Results: All solutions, 16 threads</summary>\n")
122print("*Note: vcube does not have an option for finding multiple solutions.*") 133 f.write("\n")
123print() 134 f.write("Time per cube (in seconds, lower is better).\n")
124print("Time per cube (in seconds, lower is better).") 135 f.write("\n")
125print() 136 print_table(f, results_h48.h48_all_solutions, None, False, False)
126print_table(results_h48.h48_all_solutions, None, False, False) 137 f.write("\n")
127print() 138 f.write("Time per cube adjusted for table size (in seconds \\* GiB, lower is better).\n")
128print("Time per cube adjusted for table size (in seconds \\* GiB, lower is better).") 139 f.write("\n")
129print() 140 print_table(f, results_h48.h48_all_solutions, None, True, False)
130print_table(results_h48.h48_all_solutions, None, True, False) 141 f.write("\n")
131print() 142 f.write("</details>\n")
132print("</details>")
133 143
134# Plotting 144# Plotting
135 145
136def plot(title, hd, vd, key): 146def plot_comparison(title, hd, vd, key):
137 d = 1 if key == "superflip" else 25 147 d = 1 if key == "superflip" else 25
138 h48x = [sizes_h48[m]/(2**30) for m in hd.keys() if key in hd[m]] 148 h48x = [sizes_h48[m]/(2**30) for m in hd.keys() if key in hd[m]]
139 vcubex = [sizes_vcube[m]/(2**30) for m in vd.keys() if key in vd[m]] 149 vcubex = [sizes_vcube[m]/(2**30) for m in vd.keys() if key in vd[m]]
@@ -148,17 +158,27 @@ def plot(title, hd, vd, key):
148 plt.plot(vcubex, vcubey, "o--", label = "vcube") 158 plt.plot(vcubex, vcubey, "o--", label = "vcube")
149 plt.legend(loc = "right") 159 plt.legend(loc = "right")
150 filename = title.replace(" ", "").replace(",", "") + ".png" 160 filename = title.replace(" ", "").replace(",", "") + ".png"
151 plt.savefig("benchmarks/img/" + filename, dpi=300) 161 plt.savefig(benchmarks_img_dir / filename, dpi=300)
152 #plt.show() 162 #plt.show()
153 163
154rh, rv = results_h48.h48_single_thread, results_vcube.vcube_single_thread 164rh, rv = results_h48.h48_single_thread, results_vcube.vcube_single_thread
155for m in [17, 18, 19, 20]: 165for m in [17, 18, 19, 20]:
156 plot(f"{m} moves 1 thread", rh, rv, m) 166 plot_comparison(f"{m} moves 1 thread", rh, rv, m)
157 167
158rh, rv = results_h48.h48_4_threads, results_vcube.vcube_4_threads 168def plot_multithread_scatter(title, slow, fast):
159for m in [17, 18, 19, 20]: 169 plt.clf()
160 plot(f"{m} moves 4 threads", rh, rv, m) 170 plt.title(title)
171 plt.xlabel("Moves")
172 plt.ylabel("Speed-up factor")
173 x = [17, 18, 19]
174 plt.xticks(x)
175 for h in [11, 10, 9, 8, 7, 6]:
176 y = [slow[h][i] / fast[h][i] for i in x]
177 plt.scatter(x, y, label=f"H48 h{h}")
178 plt.legend(loc = "right")
179 filename = title.replace(" ", "").replace(",", "") + ".png"
180 plt.savefig(benchmarks_img_dir / filename, dpi=300)
181 plt.show()
161 182
162rh, rv = results_h48.h48_16_threads, results_vcube.vcube_16_threads 183plot_multithread_scatter("4 threads speedup factor", results_h48.h48_single_thread, results_h48.h48_4_threads)
163for m in [17, 18, 19, 20]: 184plot_multithread_scatter("16 threads speedup factor", results_h48.h48_single_thread, results_h48.h48_16_threads)
164 plot(f"{m} moves 16 threads", rh, rv, m)
diff --git a/benchmarks/results_h48_clang.py b/benchmarks/results_h48_clang.py
deleted file mode 100644
index 3fbc553..0000000
--- a/benchmarks/results_h48_clang.py
+++ /dev/null
@@ -1,36 +0,0 @@
1h48_single_thread = {
2 6: {17: 57.7296, 18: 436.0622, 19: 2240.5087},
3 7: {17: 27.8748, 18: 222.3812, 19: 1114.8537},
4 8: {17: 23.6884, 18: 181.7169, 19: 722.1819},
5 9: {17: 10.1115, 18: 72.6935, 19: 328.5458},
6 10: {17: 7.0337, 18: 37.5239, 19: 179.3647, 20: 1267.3059, "superflip": 71.8359},
7 11: {17: 3.9693, 18: 24.2573, 19: 106.8995, 20: 688.5594, "superflip": 32.0008},
8}
9
10h48_4_threads = {
11 6: {17: 16.7353, 18: 129.9696, 19: 663.2930},
12 7: {17: 8.4315, 18: 65.4883, 19: 348.3440},
13 8: {17: 6.9127, 18: 51.8117, 19: 208.8438},
14 9: {17: 3.0985, 18: 21.2920, 19: 95.3467, 20: 711.2927, "superflip": 35.3772},
15 10: {17: 2.0592, 18: 10.6636, 19: 49.9811, 20: 388.0203, "superflip": 18.6665},
16 11: {17: 1.3065, 18: 6.9261, 19: 29.7725, 20: 197.9893, "superflip": 9.1158},
17}
18
19h48_16_threads = {
20 6: {17: 5.6335, 18: 39.2955, 19: 207.5730},
21 7: {17: 2.9328, 18: 21.0462, 19: 104.7917},
22 8: {17: 2.4790, 18: 17.0854, 19: 67.6617},
23 9: {17: 1.1561, 18: 7.0899, 19: 30.5952, 20: 220.9804, "superflip": 13.5599},
24 10: {17: 0.8002, 18: 3.6835, 19: 16.4910, 20: 114.4733, "superflip": 7.6359},
25 11: {17: 0.5475, 18: 2.4681, 19: 9.8224, 20: 60.7893, "superflip": 3.8795},
26}
27
28h48_all_solutions = {
29 6: {17: 18.3998, 18: 223.3861, 19: 2459.7513},
30 7: {17: 9.6264, 18: 114.2752, 19: 1242.9741},
31 8: {17: 7.1793, 18: 81.2145, 19: 783.8113},
32 9: {17: 3.3487, 18: 36.9225, 19: 363.1912},
33 10: {17: 2.0002, 18: 20.4565, 19: 187.5199, 20: 1570.6074, "superflip": 68.2016},
34 11: {17: 1.2833, 18: 12.3616, 19: 105.6184, 20: 872.5879, "superflip": 34.1071},
35}
36
diff --git a/benchmarks/tables_16_threads.md b/benchmarks/tables_16_threads.md
new file mode 100644
index 0000000..e9c0d19
--- /dev/null
+++ b/benchmarks/tables_16_threads.md
@@ -0,0 +1,27 @@
1<!-- The following details block can be found in benchmarks/tables_16_threads.md -->
2<details><summary>Results: Single solution, 16 threads</summary>
3
4Time per cube (in seconds, lower is better).
5
6| Solver | Size |17 moves|18 moves|19 moves|20 moves|Superflip|
7|:---------|:-------|-------:|-------:|-------:|-------:|--------:|
8|H48 h11 |56.5 GiB| 0.02| 0.06| 0.22| 1.33| 1.84|
9|H48 h10 |28.3 GiB| 0.03| 0.08| 0.33| 2.34| 4.18|
10|H48 h9 |14.1 GiB| 0.04| 0.15| 0.64| 4.45| 8.09|
11|H48 h8 | 7.1 GiB| 0.06| 0.34| 1.36| | |
12|H48 h7 | 3.5 GiB| 0.07| 0.47| 2.20| | |
13|H48 h6 | 1.8 GiB| 0.13| 0.91| 4.39| | |
14
15Speed-up factor (higher is better).
16
17| Solver | Size |17 moves|18 moves|19 moves|20 moves|Superflip|
18|:---------|:-------|-------:|-------:|-------:|-------:|--------:|
19|H48 h11 |56.5 GiB| 3.90| 8.14| 10.39| 11.86| 8.45|
20|H48 h10 |28.3 GiB| 5.72| 9.00| 10.03| 10.04| 9.09|
21|H48 h9 |14.1 GiB| 6.56| 9.75| 10.53| | |
22|H48 h8 | 7.1 GiB| 8.08| 9.91| 10.40| | |
23|H48 h7 | 3.5 GiB| 8.59| 10.30| 10.57| | |
24|H48 h6 | 1.8 GiB| 9.60| 10.39| 10.54| | |
25<img src="img/16threads.png">
26
27</details>
diff --git a/benchmarks/tables_1_thread.md b/benchmarks/tables_1_thread.md
new file mode 100644
index 0000000..c0abb50
--- /dev/null
+++ b/benchmarks/tables_1_thread.md
@@ -0,0 +1,40 @@
1<!-- The following details block can be found in benchmarks/tables_1_thread.md -->
2<details><summary>Results: Single solution, single thread</summary>
3
4Time per cube (in seconds, lower is better).
5
6| Solver | Size |17 moves|18 moves|19 moves|20 moves|Superflip|
7|:---------|:-------|-------:|-------:|-------:|-------:|--------:|
8|vcube 212 |58.2 GiB| 0.11| 0.75| 3.43| 27.28| 19.30|
9|H48 h11 |56.5 GiB| 0.09| 0.50| 2.24| 15.73| 15.55|
10|vcube 404 |31.8 GiB| 0.23| 1.24| 6.10| 59.33| 268.26|
11|H48 h10 |28.3 GiB| 0.15| 0.76| 3.36| 23.51| 38.05|
12|vcube 308 |21.2 GiB| 0.17| 1.02| 6.20| 58.70| 604.35|
13|H48 h9 |14.1 GiB| 0.24| 1.48| 6.69| | |
14|vcube 208 | 7.3 GiB| 0.56| 4.36| 20.58| | |
15|H48 h8 | 7.1 GiB| 0.46| 3.36| 14.13| | |
16|H48 h7 | 3.5 GiB| 0.63| 4.85| 23.25| | |
17|vcube 112 | 2.4 GiB| 0.96| 9.29| 40.52| | |
18|H48 h6 | 1.8 GiB| 1.25| 9.45| 46.31| | |
19
20Time per cube adjusted for table size (in seconds \* GiB, lower is better).
21
22| Solver | Size |17 moves|18 moves|19 moves|20 moves|Superflip|
23|:---------|:-------|-------:|-------:|-------:|-------:|--------:|
24|vcube 212 |58.2 GiB| 6.43| 43.80| 199.38| 1587.43| 1122.73|
25|H48 h11 |56.5 GiB| 5.21| 28.28| 126.68| 889.01| 878.77|
26|vcube 404 |31.8 GiB| 7.40| 39.47| 194.01| 1887.94| 8535.87|
27|H48 h10 |28.3 GiB| 4.22| 21.37| 94.86| 664.22| 1075.08|
28|vcube 308 |21.2 GiB| 3.51| 21.71| 131.50| 1245.26| 12819.94|
29|H48 h9 |14.1 GiB| 3.34| 20.97| 94.57| | |
30|vcube 208 | 7.3 GiB| 4.08| 31.74| 149.68| | |
31|H48 h8 | 7.1 GiB| 3.25| 23.75| 99.82| | |
32|H48 h7 | 3.5 GiB| 2.22| 17.12| 82.14| | |
33|vcube 112 | 2.4 GiB| 2.33| 22.53| 98.23| | |
34|H48 h6 | 1.8 GiB| 2.21| 16.70| 81.86| | |
35
36<img src="img/17moves1thread.png">
37<img src="img/18moves1thread.png">
38<img src="img/19moves1thread.png">
39<img src="img/20moves1thread.png">
40</details>
diff --git a/benchmarks/tables_4_threads.md b/benchmarks/tables_4_threads.md
new file mode 100644
index 0000000..9da909c
--- /dev/null
+++ b/benchmarks/tables_4_threads.md
@@ -0,0 +1,27 @@
1<!-- The following details block can be found in benchmarks/tables_4_threads.md -->
2<details><summary>Results: Single solution, 4 threads</summary>
3
4Time per cube (in seconds, lower is better).
5
6| Solver | Size |17 moves|18 moves|19 moves|20 moves|Superflip|
7|:---------|:-------|-------:|-------:|-------:|-------:|--------:|
8|H48 h11 |56.5 GiB| 0.04| 0.14| 0.60| 3.97| 4.19|
9|H48 h10 |28.3 GiB| 0.05| 0.20| 0.91| 6.65| 9.81|
10|H48 h9 |14.1 GiB| 0.07| 0.39| 1.74| 12.54| 18.96|
11|H48 h8 | 7.1 GiB| 0.13| 0.90| 3.71| | |
12|H48 h7 | 3.5 GiB| 0.17| 1.28| 6.12| | |
13|H48 h6 | 1.8 GiB| 0.33| 2.47| 12.22| | |
14
15Speed-up factor (higher is better).
16
17| Solver | Size |17 moves|18 moves|19 moves|20 moves|Superflip|
18|:---------|:-------|-------:|-------:|-------:|-------:|--------:|
19|H48 h11 |56.5 GiB| 2.55| 3.46| 3.75| 3.96| 3.71|
20|H48 h10 |28.3 GiB| 3.09| 3.71| 3.67| 3.54| 3.88|
21|H48 h9 |14.1 GiB| 3.25| 3.84| 3.85| | |
22|H48 h8 | 7.1 GiB| 3.53| 3.72| 3.81| | |
23|H48 h7 | 3.5 GiB| 3.60| 3.78| 3.80| | |
24|H48 h6 | 1.8 GiB| 3.77| 3.82| 3.79| | |
25<img src="img/4threads.png">
26
27</details>
diff --git a/benchmarks/tables_all_solutions.md b/benchmarks/tables_all_solutions.md
new file mode 100644
index 0000000..2959e0b
--- /dev/null
+++ b/benchmarks/tables_all_solutions.md
@@ -0,0 +1,26 @@
1<!-- The following details block can be found in benchmarks/tables_all_solutions.md -->
2<details><summary>Results: All solutions, 16 threads</summary>
3
4Time per cube (in seconds, lower is better).
5
6| Solver | Size |17 moves|18 moves|19 moves|20 moves|Superflip|
7|:---------|:-------|-------:|-------:|-------:|-------:|--------:|
8|H48 h11 |56.5 GiB| 0.04| 0.26| 2.22| 18.96| 16.95|
9|H48 h10 |28.3 GiB| 0.05| 0.42| 3.82| 34.42| 36.80|
10|H48 h9 |14.1 GiB| 0.08| 0.73| 7.28| | |
11|H48 h8 | 7.1 GiB| 0.15| 1.56| 15.41| | |
12|H48 h7 | 3.5 GiB| 0.21| 2.38| 26.52| | |
13|H48 h6 | 1.8 GiB| 0.39| 4.67| 53.00| | |
14
15Time per cube adjusted for table size (in seconds \* GiB, lower is better).
16
17| Solver | Size |17 moves|18 moves|19 moves|20 moves|Superflip|
18|:---------|:-------|-------:|-------:|-------:|-------:|--------:|
19|H48 h11 |56.5 GiB| 2.11| 14.55| 125.21| 1071.23| 957.47|
20|H48 h10 |28.3 GiB| 1.48| 11.96| 107.83| 972.35| 1039.73|
21|H48 h9 |14.1 GiB| 1.08| 10.29| 102.88| | |
22|H48 h8 | 7.1 GiB| 1.03| 10.99| 108.87| | |
23|H48 h7 | 3.5 GiB| 0.74| 8.41| 93.69| | |
24|H48 h6 | 1.8 GiB| 0.70| 8.25| 93.68| | |
25
26</details>

Generated with cgit - Back to sebastiano.tronto.net