JAWAHARLAL NEHRU TECHNOLOGICAL UNIVERSITY HYDERABAD MCA BIG DATA ANALYTICS Previous Question Paper (February - 2021 - R17)

Structuring Big Data involves transforming raw, heterogeneous, and high-velocity data into a format suitable for analysis and decision-making. Unlike traditional relational databases that enforce a rigid schema before data ingestion (Schema-on-Write), Big Data systems often employ a Schema-on-Read approach, allowing for flexible storage and later interpretation.

The structuring process typically follows these core architectural phases:

• Data Ingestion and Staging: Raw data from various sources (logs, social media, sensors) is ingested into a data lake in its native format. This preserves the original fidelity of the data without immediate transformation.
• Data Normalization and Cleaning: Data is processed to handle missing values, remove duplicates, and resolve inconsistencies. This step ensures that the data is reliable for downstream analytical models.
• Data Modeling and Partitioning: To optimize query performance, data is partitioned by time, geography, or other relevant keys. Structuring involves organizing data into formats like Parquet, Avro, or ORC, which support columnar storage and compression, significantly reducing I/O overhead.
• Metadata Management: A critical component of structuring is the creation of a data catalog. Metadata provides context, lineage, and definitions for unstructured or semi-structured data, enabling users to discover and interpret the data correctly.
• Layered Architecture (Medallion Architecture): Modern big data systems often structure data into three logical layers: Bronze (raw data), Silver (filtered and cleaned data), and Gold (aggregated, business-ready data). This layering separates raw ingestion from refined analytical outputs.

By leveraging distributed processing frameworks like Apache Spark or Hadoop, organizations can apply these structuring techniques at scale, enabling the conversion of massive, chaotic datasets into actionable insights through structured queries and machine learning models.

Open source analytic tools are essential for processing, analyzing, and visualizing large datasets without the cost of proprietary licensing. Key tools include:

• Apache Hadoop: A foundational framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It scales from single servers to thousands of machines, each offering local computation and storage.
• Apache Spark: An open-source, distributed computing system used for big data workloads. It utilizes in-memory processing to perform data analytics, machine learning, and stream processing significantly faster than disk-based alternatives like MapReduce.
• Apache Hive: A data warehouse software project built on top of Apache Hadoop for providing data summarization, query, and analysis. It uses a SQL-like interface called HiveQL to query data stored in various databases and file systems that integrate with Hadoop.
• Apache Flink: A framework and distributed processing engine for stateful computations over unbounded and bounded data streams. It is designed to run in all common cluster environments and perform computations at in-memory speed and at any scale.
• R Programming Language: A powerful language and environment for statistical computing and graphics. It provides a wide variety of statistical (linear and nonlinear modeling, classical statistical tests, time-series analysis, classification, clustering) and graphical techniques, and is highly extensible.
• Python (with Pandas/Scikit-learn): Python is the most popular language for data science due to its rich ecosystem. Pandas provides high-performance data manipulation structures, while Scikit-learn offers simple and efficient tools for predictive data analysis and machine learning.
• Apache Kafka: A distributed event streaming platform capable of handling trillions of events a day. It is primarily used for building real-time data pipelines and streaming applications, acting as a high-throughput message broker.