What awaits Big Data in 2020: results of the past decade and future prospects

The main IT trends of the last 10 years

In the first 10 years of the 21st century, Cloud computing and its SaaS / PaaS/IaaS-based solutions have successfully taken their place in the corporate infrastructure landscape. NoSQL-DBMS, BI-systems and containerization (virtualization) technologies are no longer a toy for geeks and are actively used by both large players and small businesses. IP telephony, online ERP and CRM products with predictive Analytics modules are a must-have for any enterprise. Cloud services take the place of local solutions, and it giants open the source codes of their programs, supporting free SOFTWARE and making a profit from commercial consultations and custom development.

The 2010s can rightly be called the era of Big Data, as the most discussed topic in the it world. Together with the Internet of Things, digitalization, artificial intelligence, and machine Learning (ML), Big Data technologies are considered the Foundation of the 4th industrial revolution (Industry 4.0, I4.0).

Recall that the essence of I4. 0 is to combine data, tools and processes from different application areas in order to reduce overall costs, reduce risks, and improve the efficiency of production and other areas of human life using cyber-physical systems. However, any technologies, including Big Data (Hadoop, Kafka, Spark, Cassandra, Nix, HBase, etc.) are just tools for optimizing application activities. Today, when artificial intelligence is actively implemented in life, replacing people in performing routine operations and quickly processing huge amounts of information, the winner is not the one who quickly solves typical tasks, but the one who comes up with new ones.

After analyzing the popularity of the most famous 4.0 technologies using Google Trends, we can conclude that a steady interest in the following it concepts has been formed all over the world over the past 10 years:

• Big data, both at the technology level (Apache Hadoop, Kafka, Spark, HBase, Cassandra and other NoSQL databases, Java, R, Python and other programming languages for developing Big Data applications), and practical examples of using all these tools in application areas (from marketing to the oil and gas industry).
• Data Science, including data analysis, Machine Learning, and other artificial intelligence techniques aimed at extracting business-friendly information from vast amounts of information, pattern recognition, and event prediction.
• Agile – approaches to organizing collaboration that have moved from the it sphere to the category of best project management practices for any field of activity.
• The Internet of Things (IOT), including industrial IoT (IIoT), in which more and more devices are equipped with Wi-Fi modules, 5g networks are launched, drones are flying faster, homes and cities are getting smarter, and every manufacturing company is striving to become a data-driven company.

What to expect from Big Data in the future: forecasts and plans for 2020

In 2020, we will see even more interesting achievements in Data Science: automatic machine learning systems (AutoML), DevOps and DataOps engineering, virtual and augmented reality (VR/AR), as well as the synergistic effect of combining these and other I4.0 technologies. For example, street surveillance cameras will identify criminals in real time, and banks are going to assess the solvency of a potential borrower based on their user behavior in the network (browser request history, social network metrics, web page views, purchases in online stores, etc.) In connection with these and other similar cases of using personal information of citizens for business purposes, we can expect a new avalanche of incidents with personal data: from leaks to huge fines due to non-compliance with GDPR and domestic regulators.

In order not to miss the critical moment of market loss or failure of internal indicators, the Manager must be aware of current trends and be aware of the need for corporate changes in a timely manner. It is the need for changes that causes the emergence of strategic business goals that are translated into specific tactical initiatives, operational processes, software products, and other technical solutions. In particular, BABOK (Business Analysis Body Of Knowledge), a professional business analyst standard from the international Institute of business analysis (IIBA), positions the need as a driving force for changes to improve corporate performance, which will bring additional value to all stakeholders.

Customers, competitors, suppliers, partners, regulators and other environmental factors (economy, politics, ecology, etc.) affect any business, determining its existence on a par with internal indicators. Collection, aggregation and analytical comparison of external and internal data will allow you to understand the need for changes in a timely manner, identifying promising directions for eliminating current defects and achieving future opportunities. Further, strategies are implemented in projects that deliver solutions that bring value to the business. Big Data technologies and Agile approaches are only tools for achieving the set goals, and the key role is set by a competent task statement. And this stage is the most time-consuming and complex in any project. For example, Hadoop, Spark, Kafka and other big data technologies will not solve all the problems with falling profits in a hypermarket, but analytical systems based on Them will help reduce churn rate and reduce theft.