2018大数据趋势：解放、整合、信任（英文版）_17页-2mb

2018 Big Data Trends Summary

Core Content

Syncsort conducted its fourth annual survey of IT professionals involved in Big Data to analyze the current state and future direction of Big Data adoption in enterprises. The survey involved nearly 200 participants across various industries, highlighting the increasing integration of Hadoop and Spark into enterprise IT landscapes.

Main Points

• Hadoop and Spark Adoption: Over 40% of respondents are in production with Hadoop or Spark, while more than 30% are engaged in proof of concept or pilot programs.
• Data Lake Use Cases: ETL, data blending, and real-time analytics are the most common use cases for data lakes, with 70.8% and 63.5% of respondents selecting these, respectively.
• Business Benefits: Data lakes are helping organizations reduce costs, increase agility, and boost revenue through better insights and analytics.
• Data Freshness: Over 75% of respondents face challenges in keeping data lakes up-to-date, especially with mainframe data.
• Legacy Data Integration: Mainframe and IBM i data play a crucial role in data lakes, with over 97% of mainframe users believing it is valuable to integrate this data for real-time analytics.

Key Challenges

Data Quality Insights

• Data Diversity and Quality: The more diverse the data sources, the greater the need for data quality. Over 60% of respondents said storing enterprise-wide data in the data lake is most critical, with an average of four sources.
• Data Quality as Critical Factor: Those with five or more data sources were four times more likely to name data quality as a critical factor for successful data lake implementation.
• Industry Focus: Financial Services and Insurance industries are more focused on data governance and quality, with nearly 60% of respondents in these sectors prioritizing data quality, compared to 40% in other industries.
• Paradox of Analytics and Quality: Some participants who did not prioritize data quality still showed strong interest in advanced analytics, which is a concern as these insights depend on high-quality data.

Data Governance and Compliance

• Regulatory Compliance: Data governance is expanding to include data lakes, as regulatory compliance becomes a key concern for executives.
• Mainframe Integration: There is a strong need to integrate mainframe data into Hadoop for real-time analytics, with over 90% of organizations with IBM i systems finding it valuable.

Cost and Efficiency

• Cost Savings: Optimizing traditional systems to save costs allows organizations to fund Big Data initiatives, with over 90% of respondents finding this strategy valuable.
• Capacity Management: Over 90% of respondents found some value in capacity management from Data Lake projects.

Future Trends

1. More Enterprise Data in Data Lakes: As business users benefit from early Hadoop deployments, more data will flow into data lakes, breaking down data silos.
2. Improved Data Quality: With increased reliance on data insights, there will be a greater emphasis on improving data quality throughout the data lifecycle.
3. Expanded Data Governance: Regulatory compliance will drive the inclusion of data lakes in data governance frameworks.
4. Fresher Data Lakes: Organizations will implement solutions to ensure data lakes stay up to date, even when integrating from difficult-to-access sources.
5. Stronger Big Data Adoption: Despite changing technologies, the core of Big Data initiatives remains strong, with continued investment in analytics and data management.

Syncsort's Role

Syncsort is a leading provider of Big Iron to Big Data solutions, helping organizations optimize traditional data systems and integrate them with next-generation analytics. Their portfolio includes high availability products for IBM i, cross-platform capacity management, and best-in-class data quality capabilities.

Summary

The 2018 Big Data trends indicate a growing reliance on Hadoop and Spark for data integration and analytics, with data lakes becoming a central component of enterprise data strategies. While there are significant benefits, including cost savings and better insights, challenges around data quality, skills shortage, and data governance remain prominent. The integration of legacy data, particularly from mainframes and IBM i systems, is seen as highly valuable. As the industry moves forward, there is a clear need for improved data quality and governance to support the increasing analytical demands of enterprises.