Each cell is the runs that reached the target, out of the runs made, then the operations a run takes for a 99 in 100 chance of reaching it ("-" if no run did). Every solver in a row has the same budget of operations a run as the emulator's sampling mode. One operation is one coupling read, one cell updated or one move considered. | family | variables | pairs | sweeps a start | target | emulator, sampling, default start | emulator, sampling, hot start | emulator, deterministic | simulated annealing | tabu search | greedy descent | |---|---|---|---|---|---|---|---|---|---|---| | clusters | 8 | 16 | 40 | exact | 100/100, 12,800 | 100/100, 12,800 | 60/100, 64,331 | 100/100, 12,801 | 100/100, 12,809 | 100/100, 12,812 | | clusters | 8 | 16 | 320 | exact | 100/100, 102,400 | 100/100, 102,400 | 60/100, 514,650 | 100/100, 102,401 | 100/100, 102,411 | 100/100, 102,413 | | clusters | 12 | 24 | 40 | exact | 100/100, 19,200 | 97/100, 25,215 | 55/100, 110,730 | 100/100, 19,200 | 100/100, 19,212 | 100/100, 19,219 | | clusters | 12 | 24 | 320 | exact | 100/100, 153,600 | 100/100, 153,600 | 55/100, 885,845 | 100/100, 153,601 | 100/100, 153,613 | 100/100, 153,620 | | clusters | 16 | 32 | 40 | exact | 100/100, 25,600 | 97/100, 33,620 | 4/100, 2,887,961 | 100/100, 25,602 | 100/100, 25,618 | 100/100, 25,625 | | clusters | 16 | 32 | 320 | exact | 100/100, 204,800 | 100/100, 204,800 | 4/100, 23,103,693 | 100/100, 204,801 | 100/100, 204,818 | 100/100, 204,829 | | clusters | 24 | 48 | 40 | best known | 100/100, 38,400 | 94/100, 62,855 | 20/100, 792,487 | 100/100, 38,401 | 100/100, 38,423 | 99/100, 38,434 | | clusters | 24 | 48 | 320 | best known | 100/100, 307,200 | 100/100, 307,200 | 20/100, 6,339,902 | 100/100, 307,202 | 100/100, 307,224 | 100/100, 307,237 | | clusters | 32 | 64 | 40 | best known | 97/100, 67,241 | 73/100, 180,079 | 19/100, 1,118,942 | 99/100, 51,201 | 100/100, 51,232 | 82/100, 137,622 | | clusters | 32 | 64 | 320 | best known | 100/100, 409,600 | 96/100, 586,005 | 19/100, 8,951,539 | 100/100, 409,601 | 100/100, 409,633 | 100/100, 409,650 | | clusters | 48 | 96 | 40 | best known | 96/100, 109,875 | 46/100, 573,977 | 0/100, - | 98/100, 90,409 | 90/100, 153,679 | 62/100, 365,865 | | clusters | 48 | 96 | 320 | best known | 99/100, 614,400 | 100/100, 614,400 | 0/100, - | 99/100, 614,401 | 100/100, 614,448 | 98/100, 723,347 | | clusters | 64 | 128 | 40 | best known | 94/100, 167,614 | 35/100, 1,094,679 | 0/100, - | 97/100, 134,484 | 60/100, 514,966 | 22/100, 1,899,749 | | clusters | 64 | 128 | 320 | best known | 100/100, 819,200 | 93/100, 1,418,648 | 0/100, - | 99/100, 819,201 | 94/100, 1,341,027 | 40/100, 7,386,135 | | clusters | 96 | 192 | 40 | best known | 56/100, 861,596 | 4/100, 17,327,770 | 0/100, - | 86/100, 359,777 | 18/100, 3,566,727 | 3/100, 23,245,467 | | clusters | 96 | 192 | 320 | best known | 98/100, 1,446,523 | 80/100, 3,516,030 | 0/100, - | 99/100, 1,228,802 | 68/100, 4,966,754 | 8/100, 67,874,513 | | clusters | 128 | 256 | 40 | best known | 46/100, 1,530,607 | 0/100, - | 0/100, - | 79/100, 604,330 | 7/100, 13,005,283 | 0/100, - | | clusters | 128 | 256 | 320 | best known | 91/100, 3,133,422 | 67/100, 6,805,596 | 0/100, - | 94/100, 2,681,840 | 26/100, 25,060,253 | 0/100, - | | sparse-glass | 8 | 24 | 40 | exact | 100/100, 17,920 | 100/100, 17,920 | 21/100, 350,092 | 100/100, 17,922 | 100/100, 17,929 | 100/100, 17,937 | | sparse-glass | 8 | 24 | 320 | exact | 100/100, 143,360 | 100/100, 143,360 | 21/100, 2,800,740 | 100/100, 143,362 | 100/100, 143,371 | 100/100, 143,379 | | sparse-glass | 12 | 36 | 40 | exact | 100/100, 26,880 | 100/100, 26,880 | 27/100, 393,335 | 100/100, 26,882 | 100/100, 26,894 | 100/100, 26,905 | | sparse-glass | 12 | 36 | 320 | exact | 100/100, 215,040 | 100/100, 215,040 | 27/100, 3,146,685 | 100/100, 215,042 | 100/100, 215,053 | 100/100, 215,066 | | sparse-glass | 16 | 48 | 40 | exact | 100/100, 35,840 | 90/100, 71,680 | 27/100, 524,447 | 100/100, 35,842 | 100/100, 35,859 | 100/100, 35,872 | | sparse-glass | 16 | 48 | 320 | exact | 100/100, 286,720 | 100/100, 286,720 | 27/100, 4,195,580 | 100/100, 286,722 | 100/100, 286,737 | 100/100, 286,756 | | sparse-glass | 24 | 72 | 40 | best known | 100/100, 53,760 | 68/100, 217,277 | 0/100, - | 100/100, 53,762 | 97/100, 70,640 | 90/100, 107,634 | | sparse-glass | 24 | 72 | 320 | best known | 100/100, 430,080 | 100/100, 430,080 | 0/100, - | 100/100, 430,082 | 100/100, 430,107 | 100/100, 430,128 | | sparse-glass | 32 | 96 | 40 | best known | 100/100, 71,680 | 74/100, 245,048 | 3/100, 10,837,399 | 99/100, 71,682 | 98/100, 84,423 | 96/100, 102,657 | | sparse-glass | 32 | 96 | 320 | best known | 100/100, 573,440 | 100/100, 573,440 | 3/100, 86,699,195 | 100/100, 573,442 | 100/100, 573,473 | 100/100, 573,513 | | sparse-glass | 48 | 144 | 40 | best known | 77/100, 336,909 | 28/100, 1,507,280 | 2/100, 24,508,987 | 82/100, 288,757 | 58/100, 571,075 | 38/100, 1,036,727 | | sparse-glass | 48 | 144 | 320 | best known | 85/100, 2,087,998 | 83/100, 2,235,485 | 2/100, 196,071,898 | 87/100, 1,941,551 | 79/100, 2,538,320 | 91/100, 1,645,239 | | sparse-glass | 64 | 192 | 40 | best known | 68/100, 579,407 | 2/100, 32,678,649 | 0/100, - | 90/100, 286,725 | 28/100, 2,010,616 | 12/100, 5,169,468 | | sparse-glass | 64 | 192 | 320 | best known | 94/100, 1,877,286 | 87/100, 2,588,728 | 0/100, - | 100/100, 1,146,882 | 65/100, 5,031,245 | 42/100, 9,696,787 | | sparse-glass | 96 | 288 | 40 | best known | 37/100, 2,143,332 | 0/100, - | 0/100, - | 76/100, 693,922 | 4/100, 24,270,642 | 1/100, 98,616,355 | | sparse-glass | 96 | 288 | 320 | best known | 87/100, 3,883,092 | 69/100, 6,764,413 | 0/100, - | 93/100, 2,979,166 | 9/100, 84,007,918 | 7/100, 109,179,042 | | sparse-glass | 128 | 384 | 40 | best known | 36/100, 2,958,621 | 0/100, - | 0/100, - | 71/100, 1,066,673 | 5/100, 25,753,943 | 0/100, - | | sparse-glass | 128 | 384 | 320 | best known | 70/100, 8,773,582 | 45/100, 17,668,955 | 0/100, - | 84/100, 5,764,091 | 14/100, 70,040,338 | 0/100, - | | dense-glass | 8 | 28 | 40 | exact | 100/100, 20,480 | 100/100, 20,480 | 88/100, 44,482 | 100/100, 20,482 | 100/100, 20,484 | 100/100, 20,506 | | dense-glass | 8 | 28 | 320 | exact | 100/100, 163,840 | 100/100, 163,840 | 88/100, 355,857 | 100/100, 163,842 | 100/100, 163,848 | 100/100, 163,862 | | dense-glass | 12 | 66 | 40 | exact | 100/100, 46,080 | 100/100, 46,080 | 78/100, 140,150 | 100/100, 46,083 | 100/100, 46,086 | 100/100, 46,126 | | dense-glass | 12 | 66 | 320 | exact | 100/100, 368,640 | 100/100, 368,640 | 78/100, 1,121,206 | 100/100, 368,643 | 100/100, 368,660 | 100/100, 368,687 | | dense-glass | 16 | 120 | 40 | exact | 100/100, 81,920 | 100/100, 81,920 | 39/100, 763,217 | 100/100, 81,927 | 100/100, 81,947 | 100/100, 82,007 | | dense-glass | 16 | 120 | 320 | exact | 100/100, 655,360 | 100/100, 655,360 | 39/100, 6,105,739 | 100/100, 655,367 | 100/100, 655,385 | 100/100, 655,447 | | dense-glass | 24 | 276 | 40 | best known | 100/100, 184,320 | 95/100, 283,344 | 25/100, 2,950,566 | 100/100, 184,332 | 100/100, 184,359 | 100/100, 184,502 | | dense-glass | 24 | 276 | 320 | best known | 100/100, 1,474,560 | 100/100, 1,474,560 | 25/100, 23,604,528 | 100/100, 1,474,569 | 100/100, 1,474,593 | 100/100, 1,474,770 | | dense-glass | 32 | 496 | 40 | best known | 94/100, 536,367 | 53/100, 1,998,645 | 4/100, 36,965,909 | 99/100, 327,697 | 95/100, 503,811 | 100/100, 328,039 | | dense-glass | 32 | 496 | 320 | best known | 98/100, 3,085,916 | 94/100, 4,290,940 | 4/100, 295,727,278 | 100/100, 2,621,451 | 100/100, 2,621,481 | 100/100, 2,621,820 | | dense-glass | 48 | 1,128 | 40 | best known | 94/100, 1,206,826 | 22/100, 13,665,303 | 3/100, 111,470,394 | 100/100, 737,340 | 92/100, 1,344,361 | 100/100, 738,037 | | dense-glass | 48 | 1,128 | 320 | best known | 100/100, 5,898,240 | 96/100, 8,438,473 | 3/100, 891,763,156 | 100/100, 5,898,264 | 100/100, 5,898,330 | 100/100, 5,899,028 | | dense-glass | 64 | 2,016 | 40 | best known | 81/100, 3,634,597 | 4/100, 147,863,639 | 0/100, - | 97/100, 1,721,561 | 54/100, 7,773,386 | 91/100, 2,509,220 | | dense-glass | 64 | 2,016 | 320 | best known | 96/100, 15,001,731 | 90/100, 20,971,520 | 0/100, - | 100/100, 10,485,791 | 87/100, 23,668,679 | 100/100, 10,487,106 | | selection | 8 | 10 | 40 | exact | 100/100, 8,960 | 100/100, 8,960 | 100/100, 8,960 | 100/100, 8,960 | 100/100, 8,967 | 100/100, 8,969 | | selection | 8 | 10 | 320 | exact | 100/100, 71,680 | 100/100, 71,680 | 100/100, 71,680 | 100/100, 71,680 | 100/100, 71,686 | 100/100, 71,687 | | selection | 12 | 10 | 40 | exact | 100/100, 10,240 | 100/100, 10,240 | 99/100, 10,240 | 100/100, 10,240 | 100/100, 10,250 | 100/100, 10,251 | | selection | 12 | 10 | 320 | exact | 100/100, 81,920 | 100/100, 81,920 | 99/100, 81,920 | 100/100, 81,920 | 100/100, 81,930 | 100/100, 81,932 | | selection | 16 | 10 | 40 | exact | 100/100, 11,520 | 100/100, 11,520 | 100/100, 11,520 | 100/100, 11,520 | 100/100, 11,536 | 100/100, 11,535 | | selection | 16 | 10 | 320 | exact | 100/100, 92,160 | 100/100, 92,160 | 100/100, 92,160 | 100/100, 92,160 | 100/100, 92,175 | 100/100, 92,175 | | selection | 24 | 15 | 40 | planted | 100/100, 17,280 | 100/100, 17,280 | 100/100, 17,280 | 100/100, 17,280 | 100/100, 17,309 | 100/100, 17,305 | | selection | 24 | 15 | 320 | planted | 100/100, 138,240 | 100/100, 138,240 | 100/100, 138,240 | 100/100, 138,240 | 100/100, 138,265 | 100/100, 138,264 | | selection | 32 | 28 | 40 | planted | 100/100, 28,160 | 100/100, 28,160 | 100/100, 28,160 | 100/100, 28,161 | 100/100, 28,185 | 100/100, 28,195 | | selection | 32 | 28 | 320 | planted | 100/100, 225,280 | 100/100, 225,280 | 100/100, 225,280 | 100/100, 225,280 | 100/100, 225,311 | 100/100, 225,314 | | selection | 48 | 66 | 40 | planted | 100/100, 57,600 | 100/100, 57,600 | 100/100, 57,600 | 100/100, 57,601 | 100/100, 57,625 | 100/100, 57,671 | | selection | 48 | 66 | 320 | planted | 100/100, 460,800 | 100/100, 460,800 | 100/100, 460,800 | 100/100, 460,800 | 100/100, 460,845 | 100/100, 460,871 | | selection | 64 | 120 | 40 | planted | 100/100, 97,280 | 100/100, 97,280 | 100/100, 97,280 | 100/100, 97,283 | 100/100, 97,344 | 100/100, 97,388 | | selection | 64 | 120 | 320 | planted | 100/100, 778,240 | 100/100, 778,240 | 100/100, 778,240 | 100/100, 778,240 | 100/100, 778,294 | 100/100, 778,362 | | selection | 96 | 276 | 40 | planted | 100/100, 207,360 | 100/100, 207,360 | 100/100, 207,360 | 100/100, 207,404 | 100/100, 207,475 | 100/100, 207,611 | | selection | 96 | 276 | 320 | planted | 100/100, 1,658,880 | 100/100, 1,658,880 | 100/100, 1,658,880 | 100/100, 1,658,880 | 100/100, 1,658,968 | 100/100, 1,659,129 | | selection | 128 | 496 | 40 | planted | 100/100, 358,400 | 100/100, 358,400 | 100/100, 358,400 | 100/100, 358,400 | 100/100, 358,502 | 100/100, 358,865 | | selection | 128 | 496 | 320 | planted | 100/100, 2,867,200 | 100/100, 2,867,200 | 100/100, 2,867,200 | 100/100, 2,867,200 | 100/100, 2,867,325 | 100/100, 2,867,566 | Of 68 rows, the emulator's sampling mode at the default start reached the target more often than every classical solver in 2, tied the best of them in 45, and was behind the best of them in 21. Of 68 rows, the emulator's sampling mode at the hot start, the largest row of couplings reached the target more often than every classical solver in 0, tied the best of them in 38, and was behind the best of them in 30. The default start is the square root of the mean couplings a variable has, and at least 1. On instances the rows do not use, 40 runs at each start, mean energy (and the runs that reached the lowest energy found): - clusters, 32 variables, target -16: 0.5: -16.00 (40); 1: -16.00 (40); 1.5: -16.00 (40); 2 default: -16.00 (40); 3: -16.00 (40); 4: -16.00 (40); 6: -15.93 (37); 8: -15.72 (29); 9 hot: -15.65 (28) - clusters, 64 variables, target -26: 0.5: -25.80 (33); 1: -26.00 (40); 1.5: -25.88 (35); 2 default: -25.68 (28); 3: -25.40 (22); 4: -25.15 (12); 6: -24.48 (6); 8 hot: -23.35 (1) - sparse-glass, 32 variables, target -19: 0.5: -19.00 (40); 1: -18.95 (38); 1.5: -18.90 (36); 2: -18.93 (37); 2.449 default: -18.82 (33); 3: -18.93 (37); 4: -18.90 (36); 6: -18.62 (26); 8: -18.55 (25); 13 hot: -18.18 (15) - sparse-glass, 64 variables, target -51: 0.5: -50.98 (39); 1: -51.00 (40); 1.5: -51.00 (40); 2: -50.98 (39); 2.449 default: -51.00 (40); 3: -51.00 (40); 4: -50.85 (34); 6: -50.50 (26); 8: -50.10 (15); 12 hot: -49.10 (4) - dense-glass, 32 variables, target -128: 0.5: -127.90 (38); 1: -127.95 (39); 1.5: -127.90 (38); 2: -127.95 (39); 3: -128.00 (40); 4: -128.00 (40); 5.568 default: -127.95 (39); 6: -127.95 (39); 8: -128.00 (40); 31 hot: -127.08 (23) - dense-glass, 64 variables, target -362: 0.5: -360.55 (29); 1: -362.00 (40); 1.5: -362.00 (40); 2: -362.00 (40); 3: -362.00 (40); 4: -361.95 (39); 6: -362.00 (40); 7.937 default: -361.90 (38); 8: -362.00 (40); 63 hot: -352.85 (0)