How the Internet of things uses Big Data: IOT system architecture

IoT System Architecture

Typical architecture of IoT systems consists of the following 3 levels:

• end devices (things, Things) – sensors, sensors, controllers and other peripheral equipment for measuring the necessary indicators and transferring this data to the network via wired or wireless protocols (Serial, RS-485, MODBUS, CAN bus, OPC UA, BLE, WiFi, Bluetooth, LoRaWAN, Sigfox, etc.). Since each “portion” of this information is small in volume, such data is called small (Little Data).
• network gateways and hubs (Network)– routers that connect and connect end devices to the cloud.
• cloud– a remote server in the data center that processes, analyzes and securely stores information. Here small data becomes Big Data when there are multiple information flows with different devices. This is how the Internet of things becomes “intelligent”, as data analysis tools are connected, including using machine learning methods. This allows you to efficiently and remotely control the equipment on which the end devices are installed. For example, if the sensors of the vibration levels shown to exceed the permissible values, it is possible to plan preventive maintenance and to avoid breakage of costly tools.

Image: 3 levels of IoT system architecture

How the Internet of Things Collects Small Data

Sensors and sensors measure the necessary parameters (temperature, pressure, level, vibration, etc.), recording changes in the environment, not its static state. The cost of implementing and using such equipment is falling rapidly, allowing more data to be collected while reducing costs.  Today it is possible to use a variety of types of wired and wireless networks for data collection, and therefore even within the same production uses several types of network connections.

The choice of data transfer protocols depends on the following factors:

• information transfer rate — the amount of data transferred per unit of time;
• power consumption– how long the electronics of the end devices can operate without recharging;
• distance– the maximum distance to transmit data;
• transmission frequency (measured in Hz) available for use.

There are 2 categories of sensors:

• active – emit signals themselves and take their reflections, require more energy;
• passive – only receive signals, which reduces their power consumption.

Most sensors are based on the wave principle — reception of sound, ultrasonic, light and heat waves. But there are devices that measure physical characteristics (inductance, capacitance, pressure, etc.). By combining different types of sensors, you can significantly improve the quality and “intelligence level” of the IoT system.

How the Internet of things works with Big Data

As a rule, there is no direct access to end devices in the industrial IoT, so gateways are used to connect the levels of technological equipment and intelligent information processing and storage systems.

End devices are data sources with low processing power that continuously transmit a lot of information of different formats to the gateway. The sensor of the end device generates an analog signal, which is converted into a digital (discrete) value using the ADC – analog-to-digital Converter. This value is time-stamped and classified (tagged) by the local processor of the end device. Tags can be simple, for example, motion detected, or complex, of several parameters (movement + speed, movement + speed + car, etc.). The more complex the tag, the more powerful the peripheral processor and the power consumption of the end device should be. However, more informative tags allow you to reduce the amount of data transferred to the cloud and the bandwidth of information, and this, in turn, increases the speed of reaction to the event.

The gateway, in turn, sends data to the cloud cluster, where a software IoT platform based on Big Data tools for information processing and mining is deployed.

On the cloud server, data from various peripherals are integrated (summarized by tags), systematized and analyzed using Machine Learning and other artificial intelligence methods. The results of data mining are visualized in the form of graphs, charts, etc., displayed in the dashboards of the user interface of the IoT platform.

However, the Internet of things involves not only the transfer of information from technological objects, but also remote management. Therefore, the feedback from the cloud IoT platform to the peripheral control device of the necessary object, for example, a valve on the pipe, etc. is implemented. For this purpose, a virtual representation of the peripheral device is implemented in the cloud, where the necessary information on changing its state is recorded, and then transmitted to the Executive device of the end equipment. While peripheral, the CPU performs recognition of the tags and the DAC, i.e. reverse digital-to-analog conversion – from a discrete value to an analog form.

The entire IoT system is distributed and scalable, but it is not connected by reliable data channels. Therefore, mechanisms of guaranteed information delivery are used. In particular, if data cannot be transferred from the end device to the cloud or Vice versa, retries are made. To exchange signals between the components of a distributed system, special solutions are used – message brokers, which guarantee the delivery of the necessary data to one or more recipients through a controlled queue.

The most popular message brokers are RabbitMQ, Apache Qpid, Apache ActiveMQ. Also for this purpose, a distributed replicated log of Apache Kafka changes is used, which is perfectly scaled, providing capacity increase with the growth of the number and load from the data sources, as well as the number of applications for their processing (subscribers). Apache NiFi event (message) processing platform or its simplified modification Apache MiNiFi is often used for fast data loading from end devices.