How to build an agile and efficient big data analysis platform for consulting enterprises?


The construction of big data analysis platform facilitates iResearch Consulting Group to enhance its business efficiency by several times: delivery cycle of offline report is shorten from 3 weeks to less than 1 week and software delivery is shorten from six to one month, so as to solve customers' diversified requirements of customization and improve the work efficiency by dozens of times.

I Research

Founded in 2002, initiated and established by Yang Weiqing, iResearch is committed to becoming a company with the best Internet audience rating and consumer insight under China's big data era. At present, iResearch is the largest consulting company within internet industry in China who has served for cumulatively more than 1,000 customers. Adhering to a concept of insight into the power of internet, iResearch is providing customers with professional service such as data products, research consultation in the fields relating to internet of Chinese professional market, etc., which facilitates customers to improve cognitive level of internet industry, profitability and comprehensive competitiveness. Let the power of internet serve China's various industries.

Project Background

The statistics of relevant data such as user source area, route of domain name and the number of page access, time on time, effective visits, bounce rate, return visitor, new visitors, times of return visit,interval of return visit, etc., should be carried out according to time dimension, website collections. Conditions can be dynamically added, and the data obtained by user behavior monitoring should be analyzed to understand users’ behavioral habits more particularly and clearly.

Facing with tens of millions of data every day in addition to different analysis demands of different site customers, flexible dimensional analysis demand put forward higher challenge to the performance of analysis, and the traditional database and Hadoop architecture have been unable to meet the demand of high performance real-time analysis.

Project Content

1.90 days of detailed data about 5 billion pieces of data imported into Yonghong data mart can help to directly customize Dashboard analysis.
2. ETL project: centralized import of historical data and automatic import of incremental data. Data enter the mart after associated and tagged.
3. Customization requirements: capability to customize database functions, step function, inContains function, etc.
4. Front-end display :Dashboard designer, possessing Portal, permission management, task scheduling, monitoring system.

Client's Return

1. Improvement of business efficiency: offline reporting delivery cycle is shortened from 3-4 weeks to less than 1 week.
2. Project sources and income space increase.
3. Transformation of subsequent innovation model, a new SaaS platform built on the basis of agile BI tool to transform to internet application provider of big data services.

Case Details

Achieve big data visualization analysis within one week

—agile BI assisting iResearch Consulting Group to achieve internet big data analysis

Introduction: relative to the traditional analysis method, through complementary of agile BI and Hadoop, business efficiency of iResearch Consulting Group has been increased by several times, offline reporting delivery cycle is shortened from 3-4 weeks to less than 1 week,and software delivery is shorten from six to one month. at present, at the mention of big data, Hadoop will occur to people, which seems to be "mouthpiece" of big data. Admittedly, Hadoop has a huge advantage in cluster scalability and cost, but Hadoop is not applicable to real-time analysis.

Therefore, many enterprises use Hadoop to implement data storage, and then realize high-speed capture and real-time analysis of big data through other tools.

Here, we would like to explain how agile BI and Hadoop complement each other to assist it achieving big data analysis on internet through a real case of iResearch Consulting Group.

Customized projects are inefficient.

Focusing on new economic fields such as Internet media, e-commerce, online games, wireless appreciation, etc., iResearch Consulting Group deeply studies and understands consumer behavior, providing customers in network industry and traditional industry with market research and strategic advisory services as a professional market research institution.

At present, iResearch Consulting Group can provide enterprises with two kinds of customized consulting report services namely both offline reports and software.

However, customization requirements of enterprise customers are very changeable. It takes 3 to 4 weeks as delivery cycle for iResearch to generate an offline report , and half a year for software delivery in addition to high labor cost for the project, extension of the iterative cycle, therefore, iResearch is often afraid to take on too many customized projects.

Through investigation, I have found the real demand of iResearch Consulting Group:

The statistics of relevant data such as user source area, route of domain name and the number of page access, time on time, effective visits, bounce rate, return visitor, new visitors, times of return visit,interval of return visit, etc., should be carried out according to time dimension, website collections. Conditions can be dynamically added, and the data obtained by user behavior monitoring should be analyzed to understand users’ behavioral habits more particularly and clearly.

Therefore, iResearch group urgently needs a more agile and efficient big data analysis tool to improve the efficiency of customized business.

Big data: agile BI PK traditional BI.

When solving the difficulties faced by iResearch Consulting Group, by traditional BI, IT staffs may have to carry out modeling work in advance according to the requirement analysis, set up secondary table or use Cube computation method and collect data in advance, then business personnel can check and see results of analysis report at the front end. Although this approach is very mature, it cannot solve the problem of iResearch Consulting Group.

Firstly, relative dynamic state of report queried by business personnel, analytical dimensions and metrical formula mode have been pre-set when modeling, which cannot be changed.For example, we have to modify the model again if we try to change the pre-set formula of sum or average to variance.

Secondly, when analyzing requirements change, the business personnel cannot directly adjust the report otherwise IT staffs re-set the model or modify the existing analytical model, which takes a long time, at a slow response speed.

Finally, it is painful that even some enterprises with small data volume also have to go through such process and architecture for data analysis. The essential reason of these problems is the weak computing power of the previous technical architecture for mass data resulting in that enterprise users have to carry out data operation and summary in advance through modeling, secondary table and Cube.

iResearch wishes to submit such an analysis to corporate clients, applicable to not only review but also dynamic analysis. For iResearch, data presentation should be a starting point but not a destination. Interactively analyze data as shown and make further investigation to find out problems and their answers, then take measures. The process of interacting with data should be fast enough: if the users have to wait for three or five minutes between each click, they can’t make interactive analysis.

Moreover,colleagues in IT department should be enabled to work out the analysis report directly on the analysis platform. And the workload of the IT department will significantly increase if all analysis report requirements submitted to IT department. At the same time, the implementation and operation of agile BI should be simplified so that the business personnel can use it directly.

At the same time, the analysis report requirements often need to involve changes in the data layer, requiring IT departments to improve the data layer and the business layer. The traditional BI platform will take a month or two to complete sorting the model. Agile BI does not need to be modeled in advance, which can help to flexibly adjust the analysis dimension and report presentation in the analysis process, answer the change of requirements within one day, so as to enhance enterprise's insight and decision-making power. Different from weight modeling, unified view of traditional BI, agile BI is adopted with light weight modeling, N view method, unnecessary to build up secondary table and Cube, by which analysis can be directly carried out after data imported, and the business personnel can real-time adjust analytical dimension and metrical calculation method, whose flexibility greatly increases, truly realizing data speak.

Figure 1: Analysis Diagram of Chinese Mainstream Network TV Client Software.

Since there is such a convenient way, why doesn't the traditional BI use this architecture? That is because the traditional technology architecture without big data technology fails to present analysis results that enterprise customers need a few seconds after users’ clicks in the face of mass data, therefore, we have to summarize data by modeling in advance, so as to ensure the speed when showing the analysis report.

Therefore, the premise of implementing agile Bl is to use the new architecture to process data, which involves the technologies of distributed computation, memory calculation, columnar storage and library calculation, etc.

Agile BI can enable companies with a quick insight into meaning and value of data through a lower cost and shorter launch cycle.

Business efficiency increases by several times.

Further research on the data to be analyzed by iResearch Consulting Group shows that the amount of data that iResearch Consulting Group had to analyze every day was up to several ten millions, and analysis demand of different enterprise customers was different. Therefore, complex and changeable multidimensional analysis demand challenged analysis capability of the analysis tools, but the traditional database and Hadoop architecture were incapable to meet the need for high performance and real-time analysis.

For this reason, iResearch has investigated some well-known products abroad, but had to give up as soon as they got to know the product price and the subsequent service fee. And Most of the domestic BI products in previous generation needed to be modeled before analysis, which failed to cope with changeable requirements of flexible multi-dimensional analysis, with processing capacity of large data volume incapable to meet requirements.

In the end, iResearch has chosen Yonghong z-suite. As soon as iResearch imported the detailed data of three months (about 5 billion bars) into agile BI system, it could show customized analysis reports directly. Compared with original analysis method on the basis of Excel and SQL programming, it has helped iResearch Consulting Group to increase its business efficiency by several times: offline reporting delivery cycle was shortened from 3-4 weeks to less than 1 week, and software delivery was shorten from six to one month.

At the same time, many projects that iResearch Consulting Group was once incapable to properly deliver due to worry about changing requirements are now put in its bag. After adopting the agile BI tool, iResearch Consulting Group can quickly build a prototype in a few days to show customers that any change of requirements can be adjusted within a week. With such trial and error by rapid prototyping, iResearch can undertake many of these projects.

Figure 2: Analysis of Popular Terminal Flow and Users

With the ability to undertake more projects due to the great improvement in business efficiency, iResearch Consulting Group's income space has also increased by several times. Meanwhile, iResearch's customer satisfaction has also been steadily improving.

Not only that, in order to provide more intuitive interactive analysis report, improve enterprise user experience, iResearch Consulting Group has based on agile BI tool to build a new SaaS platform, and iResearch Consulting Group has imported the data that corporate customers "stored by Hadoop architecture” into the data mart through the interface provided by agile BI, and presented results quickly through agile BI. In fact: both Hadoop and agile BI are respectively applicable to different business scenarios and complementary to each other. At present, many enterprises use Hadoop to realize data storage, and then import Hadoop data into highly performed data mart of agile BI based on distributed memory computation for data visualization analysis. Since Hadoop is now widely used within enterprises, Yonghong agile BI product also supports the connection of Hadoop data source.

Reasonably, iResearch has taken advantage of Hadoop architecture that it used to spend on human resources and money, so that their previous investment is not wasted. However, Yonghong z-suite can also be well integrated for companies that don’t have Hadoop architecture.

The sales of SaaS accounts also increase a long-term stable income for iResearch Consulting Group whose previous single business model only relying on independent projects has been changed. Subsidiaries of iResearch group quickly have followed up on the use of agile BI and changes in new patterns. At the same time, due to the structure of SaaS platform, iResearch Consulting Group has changed its own value orientation from the media/consultancy services company to internet application service providers of big data, so as to boost the value of the company in the capital market.

Copyright © 2012-2020 Beijing Yonghong Tech Co., Ltd.
京ICP备12050607号 京公网安备110110802011451号