Added benchmarks for all solutions

author: Sebastiano Tronto <sebastiano@tronto.net> 2025-06-19 17:20:32 +0200
committer: Sebastiano Tronto <sebastiano@tronto.net> 2025-06-19 17:20:32 +0200
commit: 05bae37d768fca39c478d682545fedeb76f35409 (patch)
tree: 3be0844eaf3b9fd3037920bea807595e1e761764 /benchmarks
parent: da31fbb09fec50c127bd16a99fca81bdd013c40d (diff)
download: nissy-core-05bae37d768fca39c478d682545fedeb76f35409.tar.gz
nissy-core-05bae37d768fca39c478d682545fedeb76f35409.zip
1 files changed, 19 insertions, 67 deletions
diff --git a/benchmarks/benchmarks.md b/benchmarks/benchmarks.md
index abecc85..8170892 100644
--- a/benchmarks/benchmarks.md
+++ b/benchmarks/benchmarks.md
@@ -38,12 +38,12 @@ may be particularly inaccurate, because writing the solution (together
 with some log messages) to standard output may take a significant portion
 of the time, depending on the terminal emulator used and other factors.
-We ran two different tests: finding a single optimal solution and finding
+The main test we performed was finding a single optimal solution, and we
-*all* optimal solution. Both problems involve an exhaustive search at
+compared the results with vcube. This test was run in a single-thread
-depth `n-1`, but the second also involves and exhaustive search at depth
+configuration and in two multithread configurations (with 4 and 16
-`n`. Comparisons with vcube are only done in the first case, because
+threads).  We also ran a test on finding *all* optimal solutions which,
-vcube's current configuration does not allow for finding multiple
+as far as I know, is a use case not supported by vcube; this latter test
-solutions, as far as I am aware.
+was only run on 16 threads.
 Since the size of the pruning table used by the solver is of utmost
 importance, we include two statistics: time per cube and time per cube
@@ -167,81 +167,33 @@ Time per cube adjusted for tables size (in seconds \* GiB, lower is better).
 (a) vcube cannot parallelize on a single scramble, the results for the
 Superflip are going to be the same as in the single thread case.
-## All optimal solutions (TODO)
+## All optimal solutions
 Average time for finding all optimal solutions.
-### Single thread
-Time per cube (in seconds, lower is better).
-|  Solver  |  Size  |17 moves|18 moves|19 moves|20 moves|
-|:---------|:-------|-------:|-------:|-------:|-------:|
-|H48 h11 k2|56.1GiB |        |        |        |        |
-|H48 h10 k2|28.1GiB |        |        |        |        |
-|H48 h9 k2 |14.1GiB |        |        |        |        |
-|H48 h8 k2 | 7.1GiB |        |        |        |        |
-|H48 h7 k2 | 3.6GiB |        |        |        |        |
-|H48 h6 k2 | 1.8GiB |        |        |        |        |
-Time per cube adjusted for tables size (in seconds \* GiB, lower is better).
-|  Solver  |  Size  |17 moves|18 moves|19 moves|20 moves|
-|:---------|:-------|-------:|-------:|-------:|-------:|
-|H48 h11 k2|56.1GiB |        |        |        |        |
-|H48 h10 k2|28.1GiB |        |        |        |        |
-|H48 h9 k2 |14.1GiB |        |        |        |        |
-|H48 h8 k2 | 7.1GiB |        |        |        |        |
-|H48 h7 k2 | 3.6GiB |        |        |        |        |
-|H48 h6 k2 | 1.8GiB |        |        |        |        |
-### Multithread (4 threads)
-Time per cube (in seconds, lower is better).
-|  Solver  |  Size  |17 moves|18 moves|19 moves|20 moves|
-|:---------|:-------|-------:|-------:|-------:|-------:|
-|H48 h11 k2|56.1GiB |        |        |        |        |
-|H48 h10 k2|28.1GiB |        |        |        |        |
-|H48 h9 k2 |14.1GiB |        |        |        |        |
-|H48 h8 k2 | 7.1GiB |        |        |        |        |
-|H48 h7 k2 | 3.6GiB |        |        |        |        |
-|H48 h6 k2 | 1.8GiB |        |        |        |        |
-Time per cube adjusted for tables size (in seconds \* GiB, lower is better).
-|  Solver  |  Size  |17 moves|18 moves|19 moves|20 moves|
-|:---------|:-------|-------:|-------:|-------:|-------:|
-|H48 h11 k2|56.1GiB |        |        |        |        |
-|H48 h10 k2|28.1GiB |        |        |        |        |
-|H48 h9 k2 |14.1GiB |        |        |        |        |
-|H48 h8 k2 | 7.1GiB |        |        |        |        |
-|H48 h7 k2 | 3.6GiB |        |        |        |        |
-|H48 h6 k2 | 1.8GiB |        |        |        |        |
 ### Multithread (16 threads)
 Time per cube (in seconds, lower is better).
 |  Solver  |  Size  |17 moves|18 moves|19 moves|20 moves|
 |:---------|:-------|-------:|-------:|-------:|-------:|
-|H48 h11 k2|56.1GiB |        |        |        |        |
+|H48 h11 k2|56.1GiB |  0.05  |  0.50  |  4.24  | 19.75  |
-|H48 h10 k2|28.1GiB |        |        |        |        |
+|H48 h10 k2|28.1GiB |  0.08  |  0.88  |  6.94  |        |
-|H48 h9 k2 |14.1GiB |        |        |        |        |
+|H48 h9 k2 |14.1GiB |  0.13  |  1.39  | 13.50  |        |
-|H48 h8 k2 | 7.1GiB |        |        |        |        |
+|H48 h8 k2 | 7.1GiB |  0.25  |  2.85  |        |        |
-|H48 h7 k2 | 3.6GiB |        |        |        |        |
+|H48 h7 k2 | 3.6GiB |  0.36  |  4.24  |        |        |
-|H48 h6 k2 | 1.8GiB |        |        |        |        |
+|H48 h6 k2 | 1.8GiB |  0.69  |  8.20  |        |        |
 Time per cube adjusted for tables size (in seconds \* GiB, lower is better).
 |  Solver  |  Size  |17 moves|18 moves|19 moves|20 moves|
 |:---------|:-------|-------:|-------:|-------:|-------:|
-|H48 h11 k2|56.1GiB |        |        |        |        |
+|H48 h11 k2|56.1GiB |  2.81  | 28.05  | 237.86 |1107.98 |
-|H48 h10 k2|28.1GiB |        |        |        |        |
+|H48 h10 k2|28.1GiB |  2.25  | 24.73  | 195.01 |        |
-|H48 h9 k2 |14.1GiB |        |        |        |        |
+|H48 h9 k2 |14.1GiB |  1.83  | 19.60  | 190.35 |        |
-|H48 h8 k2 | 7.1GiB |        |        |        |        |
+|H48 h8 k2 | 7.1GiB |  1.77  | 20.24  |        |        |
-|H48 h7 k2 | 3.6GiB |        |        |        |        |
+|H48 h7 k2 | 3.6GiB |  1.30  | 15.26  |        |        |
-|H48 h6 k2 | 1.8GiB |        |        |        |        |
+|H48 h6 k2 | 1.8GiB |  1.24  | 14.76  |        |        |
 ## Other notes
author	Sebastiano Tronto <sebastiano@tronto.net>	2025-06-19 17:20:32 +0200
committer	Sebastiano Tronto <sebastiano@tronto.net>	2025-06-19 17:20:32 +0200
commit	05bae37d768fca39c478d682545fedeb76f35409 (patch)
tree	3be0844eaf3b9fd3037920bea807595e1e761764 /benchmarks
parent	da31fbb09fec50c127bd16a99fca81bdd013c40d (diff)
download	nissy-core-05bae37d768fca39c478d682545fedeb76f35409.tar.gz nissy-core-05bae37d768fca39c478d682545fedeb76f35409.zip

diff --git a/benchmarks/benchmarks.md b/benchmarks/benchmarks.md index abecc85..8170892 100644 --- a/benchmarks/benchmarks.md +++ b/benchmarks/benchmarks.md
@@ -38,12 +38,12 @@ may be particularly inaccurate, because writing the solution (together
38	with some log messages) to standard output may take a significant portion	38	with some log messages) to standard output may take a significant portion
39	of the time, depending on the terminal emulator used and other factors.	39	of the time, depending on the terminal emulator used and other factors.
40		40
41	We ran two different tests: finding a single optimal solution and finding	41	The main test we performed was finding a single optimal solution, and we
42	all optimal solution. Both problems involve an exhaustive search at	42	compared the results with vcube. This test was run in a single-thread
43	depth `n-1`, but the second also involves and exhaustive search at depth	43	configuration and in two multithread configurations (with 4 and 16
44	`n`. Comparisons with vcube are only done in the first case, because	44	threads). We also ran a test on finding all optimal solutions which,
45	vcube's current configuration does not allow for finding multiple	45	as far as I know, is a use case not supported by vcube; this latter test
46	solutions, as far as I am aware.	46	was only run on 16 threads.
47		47
48	Since the size of the pruning table used by the solver is of utmost	48	Since the size of the pruning table used by the solver is of utmost
49	importance, we include two statistics: time per cube and time per cube	49	importance, we include two statistics: time per cube and time per cube
@@ -167,81 +167,33 @@ Time per cube adjusted for tables size (in seconds \* GiB, lower is better).
167	(a) vcube cannot parallelize on a single scramble, the results for the	167	(a) vcube cannot parallelize on a single scramble, the results for the
168	Superflip are going to be the same as in the single thread case.	168	Superflip are going to be the same as in the single thread case.
169		169
170	## All optimal solutions (TODO)	170	## All optimal solutions
171		171
172	Average time for finding all optimal solutions.	172	Average time for finding all optimal solutions.
173		173
174	### Single thread
175
176	Time per cube (in seconds, lower is better).
177
178	\| Solver \| Size \|17 moves\|18 moves\|19 moves\|20 moves\|
179	\|:---------\|:-------\|-------:\|-------:\|-------:\|-------:\|
180	\|H48 h11 k2\|56.1GiB \| \| \| \| \|
181	\|H48 h10 k2\|28.1GiB \| \| \| \| \|
182	\|H48 h9 k2 \|14.1GiB \| \| \| \| \|
183	\|H48 h8 k2 \| 7.1GiB \| \| \| \| \|
184	\|H48 h7 k2 \| 3.6GiB \| \| \| \| \|
185	\|H48 h6 k2 \| 1.8GiB \| \| \| \| \|
186
187	Time per cube adjusted for tables size (in seconds \* GiB, lower is better).
188
189	\| Solver \| Size \|17 moves\|18 moves\|19 moves\|20 moves\|
190	\|:---------\|:-------\|-------:\|-------:\|-------:\|-------:\|
191	\|H48 h11 k2\|56.1GiB \| \| \| \| \|
192	\|H48 h10 k2\|28.1GiB \| \| \| \| \|
193	\|H48 h9 k2 \|14.1GiB \| \| \| \| \|
194	\|H48 h8 k2 \| 7.1GiB \| \| \| \| \|
195	\|H48 h7 k2 \| 3.6GiB \| \| \| \| \|
196	\|H48 h6 k2 \| 1.8GiB \| \| \| \| \|
197
198	### Multithread (4 threads)
199
200	Time per cube (in seconds, lower is better).
201
202	\| Solver \| Size \|17 moves\|18 moves\|19 moves\|20 moves\|
203	\|:---------\|:-------\|-------:\|-------:\|-------:\|-------:\|
204	\|H48 h11 k2\|56.1GiB \| \| \| \| \|
205	\|H48 h10 k2\|28.1GiB \| \| \| \| \|
206	\|H48 h9 k2 \|14.1GiB \| \| \| \| \|
207	\|H48 h8 k2 \| 7.1GiB \| \| \| \| \|
208	\|H48 h7 k2 \| 3.6GiB \| \| \| \| \|
209	\|H48 h6 k2 \| 1.8GiB \| \| \| \| \|
210
211	Time per cube adjusted for tables size (in seconds \* GiB, lower is better).
212
213	\| Solver \| Size \|17 moves\|18 moves\|19 moves\|20 moves\|
214	\|:---------\|:-------\|-------:\|-------:\|-------:\|-------:\|
215	\|H48 h11 k2\|56.1GiB \| \| \| \| \|
216	\|H48 h10 k2\|28.1GiB \| \| \| \| \|
217	\|H48 h9 k2 \|14.1GiB \| \| \| \| \|
218	\|H48 h8 k2 \| 7.1GiB \| \| \| \| \|
219	\|H48 h7 k2 \| 3.6GiB \| \| \| \| \|
220	\|H48 h6 k2 \| 1.8GiB \| \| \| \| \|
221
222	### Multithread (16 threads)	174	### Multithread (16 threads)
223		175
224	Time per cube (in seconds, lower is better).	176	Time per cube (in seconds, lower is better).
225		177
226	\| Solver \| Size \|17 moves\|18 moves\|19 moves\|20 moves\|	178	\| Solver \| Size \|17 moves\|18 moves\|19 moves\|20 moves\|
227	\|:---------\|:-------\|-------:\|-------:\|-------:\|-------:\|	179	\|:---------\|:-------\|-------:\|-------:\|-------:\|-------:\|
228	\|H48 h11 k2\|56.1GiB \| \| \| \| \|	180	\|H48 h11 k2\|56.1GiB \| 0.05 \| 0.50 \| 4.24 \| 19.75 \|
229	\|H48 h10 k2\|28.1GiB \| \| \| \| \|	181	\|H48 h10 k2\|28.1GiB \| 0.08 \| 0.88 \| 6.94 \| \|
230	\|H48 h9 k2 \|14.1GiB \| \| \| \| \|	182	\|H48 h9 k2 \|14.1GiB \| 0.13 \| 1.39 \| 13.50 \| \|
231	\|H48 h8 k2 \| 7.1GiB \| \| \| \| \|	183	\|H48 h8 k2 \| 7.1GiB \| 0.25 \| 2.85 \| \| \|
232	\|H48 h7 k2 \| 3.6GiB \| \| \| \| \|	184	\|H48 h7 k2 \| 3.6GiB \| 0.36 \| 4.24 \| \| \|
233	\|H48 h6 k2 \| 1.8GiB \| \| \| \| \|	185	\|H48 h6 k2 \| 1.8GiB \| 0.69 \| 8.20 \| \| \|
234		186
235	Time per cube adjusted for tables size (in seconds \* GiB, lower is better).	187	Time per cube adjusted for tables size (in seconds \* GiB, lower is better).
236		188
237	\| Solver \| Size \|17 moves\|18 moves\|19 moves\|20 moves\|	189	\| Solver \| Size \|17 moves\|18 moves\|19 moves\|20 moves\|
238	\|:---------\|:-------\|-------:\|-------:\|-------:\|-------:\|	190	\|:---------\|:-------\|-------:\|-------:\|-------:\|-------:\|
239	\|H48 h11 k2\|56.1GiB \| \| \| \| \|	191	\|H48 h11 k2\|56.1GiB \| 2.81 \| 28.05 \| 237.86 \|1107.98 \|
240	\|H48 h10 k2\|28.1GiB \| \| \| \| \|	192	\|H48 h10 k2\|28.1GiB \| 2.25 \| 24.73 \| 195.01 \| \|
241	\|H48 h9 k2 \|14.1GiB \| \| \| \| \|	193	\|H48 h9 k2 \|14.1GiB \| 1.83 \| 19.60 \| 190.35 \| \|
242	\|H48 h8 k2 \| 7.1GiB \| \| \| \| \|	194	\|H48 h8 k2 \| 7.1GiB \| 1.77 \| 20.24 \| \| \|
243	\|H48 h7 k2 \| 3.6GiB \| \| \| \| \|	195	\|H48 h7 k2 \| 3.6GiB \| 1.30 \| 15.26 \| \| \|
244	\|H48 h6 k2 \| 1.8GiB \| \| \| \| \|	196	\|H48 h6 k2 \| 1.8GiB \| 1.24 \| 14.76 \| \| \|
245		197
246	## Other notes	198	## Other notes
247		199