Mi Data Stream Introduction and Architecture

Mi Data Stream primarily relies on Streaming Message Queue Talos as the data bus. The architecture is as follows:

As shown in the above image, the main structure of Mi Data Stream is:

• Data bus: Mi Data Stream uses Talos as the data bus, so all data is transmitted through Talos;
• Source support: All written modules are defined as source, which is used to write data to Talos in real time;
• Sink support: All modules written from Talos are defined as sink, which is used to export data from Talos in real time;
• Streaming computing: After all data is written to Talos, the data can be computed using spark streaming;

The main features are:

• Source/sink complete decoupling: By introducing a data bus, source and sink are completely decoupled and no longer interact with each other;
• Source/sink easy extension: Supports a variety of source/sink types and defines interfaces, very easy to extend;
• Comprehensive data transfer service: Using the source/sink mode, the complexity of system data transfer is reduced from (M*N) to (M+N):
• Complete support for streaming calculation: All data can be processed through spark streaming, producing standard streaming calculation;
• Complete security authentication system: The entire data stream is connected to the Eco Cloud account system, ensuring the data are safe and reliable;
• Complete data quality monitoring: Provides an overall data stream SLA that describes data loss and data latency;
• Complete quality of service monitoring: Provides comprehensive monitoring of data stream links, errors detected in real time will create a warning;
• New interactive user design: Applies new central configuration management, so that data stream configuration is converted into the click operation of the interface;