Xiaomi Galaxy Talos Book

Talos Test Scenarios and Results

Read/write test in synchronous mode, using Simple Producer & Simple Consumer

Remarks:

1 client, 8 Talos machines, 8 Hdfs, 8 HBase, Talos and Hdfs/HBase distributed, machine models are all 2U;
Read/write is synchronous, each partition corresponds to one thread;
ThroughPut indicates the client's exit/entry bandwidth;
QPS is the cluster's total processing power, not a single machine's QPS

Scenario 1: Write-only, batch=1

Number of partitions	Batch size	Size of a single message (bytes)	QPS	Latency(ms)	Throughput (MB/s)
1	1	100	531	1.88	0.05
8	1	100	4040	1.98	0.40
16	1	100	7843	2.04	0.78
32	1	100	14883	2.15	1.49
64	1	100	27004	2.37	2.70
128	1	100	43537	2.94	4.35

Scenario 2: Synchronous read/write, batch=1

Number of partitions	Batch size	Message size (bytes)	QPS (read/write)	Latency(ms) (read/write)	ThroughPut (MB/s) (read/write)
1	1	100	599/495	1.67/1.88	0.06/0.05
8	1	100	4484/3960	1.78/2.02	0.45/0.40
16	1	100	8602/7582	1.86/2.11	0.86/0.76
32	1	100	16000/14035	2.00/2.28	1.60/1.40
64	1	100	25910/23021	2.47/2.78	2.59/2.30
128	1	100	25498/24196	5.02/5.29	2.55/2.42

Scenario 3: Write-only, batch=10

Number of partitions	Batch size	Size of a single message (bytes)	QPS	Latency(ms)	Throughput (MB/s)
1	10	100	518	1.93	0.52
8	10	100	3883	2.06	3.88
16	10	100	7547	2.12	7.55
32	10	100	14414	2.22	14.41
64	10	100	26337	2.43	26.34
128	10	100	41157	3.11	41.16

Scenario 4: Synchronous read/write, batch=10

Number of partitions	Batch size	Message size (bytes)	QPS (read/write)	Latency(ms) (read/write)	ThroughPut (MB/s) (read/write)
1	10	100	581/507	1.72/1.97	0.58/0.51
8	10	100	4419/3791	1.81/2.11	4.42/3.80
16	10	100	8421/7373	1.90/2.17	8.42/3.37
32	10	100	15686/13559	2.04/2.36	15.69/13.45
64	10	100	24806/21993	2.58/2.91	24.81/22.00
128	10	100	24288/23021	5.27/5.56	24.29/23.02

Scenario 5: Write-only, batch=100

Number of partitions	Batch size	Size of a single message (bytes)	QPS	Latency(ms)	Throughput (MB/s)
1	100	100	469	2.13	4.69
8	100	100	3493	2.29	34.93
16	100	100	6751	2.37	67.51
32	100	100	12800	2.50	128.00
64	100	100	23188	2.76	231.88
128	100	100	36056	3.55	360.56

Scenario 6: Synchronous read/write, batch=100

Number of partitions	Batch size	Message size (bytes)	QPS (read/write)	Latency(ms) (read/write)	ThroughPut (MB/s) (read/write)
1	100	100	546/458	1.83/2.18	5.46/4.58
8	100	100	4102/3433	1.95/2.33	41.01/34.33
16	100	100	7804/6530	2.05/2.45	78.04/65.30
32	100	100	14545/11985	2.20/2.67	145.45/119.85
64	100	100	22377/19219	2.86/3.33	223.77/192.19
128	100	100	22416/20545	5.81/6.23	224.16/205.45

Scenario 7: Write-only, batch=1000

Number of partitions	Batch size	Size of a single message (bytes)	QPS	Latency(ms)	Throughput (MB/s)
1	1000	100	252	3.96	25.2
8	1000	100	1814	4.41	181.4
16	1000	100	3347	4.78	334.7
32	1000	100	5683	5.63	568.3
64	1000	100	7467	8.57	746.7
128	1000	100	8366	15.30	836.6

Scenario 8: Synchronous read/write, batch=1000

Number of partitions	Batch size	Message size (bytes)	QPS (read/write)	Latency(ms) (read/write)	ThroughPut (MB/s) (read/write)
1	1000	100	310/246	3.22/4.06	31/25
8	1000	100	2110/1709	3.79/4.68	211/170
16	1000	100	3703/3007	4.32/5.32	370/300
32	1000	100	5152/4255	6.21/7.52	515/425
64	1000	100	5526/4765	11.58/13.43	552/476
128	1000	100	5553/4996	23.05/25.62	555/500

Read/write test in asynchronous mode, using High Level API, single Producer & single Consumer

Note: Using High Level API to test the write speed resulted in even better performance than synchronous mode, read performance will be no different than write.