JAWAHARLAL NEHRU TECHNOLOGICAL UNIVERSITY HYDERABAD B.Tech III DATA ANALYTICS Previous Question Paper (September - 2021 - R18)

Designing a robust Data Architecture requires balancing technical requirements with organizational realities. The following constraints and influences are critical in shaping the design:

• Data Volume, Velocity, and Variety: The Three Vs of Big Data dictate the scalability requirements. High-velocity streaming data necessitates architectures like Lambda or Kappa, while high-variety (unstructured) data requires Data Lakes or Lakehouses rather than traditional relational schemas.
• Latency Requirements: The architecture must align with business needs for data freshness. Real-time or near real-time processing requires event-driven architectures and stream processing engines, whereas batch processing is sufficient for historical reporting and long-term analytical trends.
• Data Governance and Compliance: Regulations such as GDPR, CCPA, and HIPAA impose strict constraints on data lineage, residency, and privacy. The architecture must incorporate Data Masking, Encryption, and robust Access Control mechanisms to ensure data security and regulatory adherence.
• Cost Constraints: The total cost of ownership (TCO) is a major driver. This includes storage costs, compute resources, and licensing fees. Cloud-native architectures often provide elasticity, allowing organizations to scale resources based on demand, thereby optimizing costs compared to fixed on-premises infrastructure.
• Data Quality and Integrity: The architecture must account for the Garbage In, Garbage Out principle. Designing for Data Quality involves implementing automated validation, cleansing, and transformation pipelines (ETL/ELT) to ensure that downstream analytics are based on reliable, accurate data.
• Integration and Interoperability: Modern organizations rely on a polyglot persistence strategy, using different databases for different workloads (e.g., NoSQL for document storage, RDBMS for transactional integrity). The architecture must facilitate seamless integration between these disparate systems through APIs, message brokers, or data virtualization layers.
• Organizational Culture and Skill Sets: The technical complexity of the architecture must be matched by the team's ability to maintain it. Choosing Cloud-Managed Services over self-hosted solutions is often influenced by the availability of specialized talent and the desire to reduce operational overhead.
• Business Agility and Time-to-Market: The design must support rapid iteration. Decoupled architectures, where storage and compute are separated, allow teams to evolve components independently, enabling faster deployment of new analytical features without disrupting existing production workloads.

Data management in modern analytics relies on the ingestion and processing of diverse data streams. These sources provide the raw material for predictive modeling, real-time monitoring, and decision support systems.

• Sensors: These are physical devices that detect changes in the environment and convert them into electrical signals. In data management, sensors are the foundation of the Internet of Things (IoT). They generate high-velocity, time-series data, such as temperature, pressure, humidity, or vibration levels. Managing this data requires specialized architectures like stream processing to handle the continuous influx of telemetry data.
• Signals: Signals represent information transmitted over time or space, often captured as waveforms. These include audio, video, and electromagnetic waves. In a data management context, signals are often unstructured or semi-structured. Processing them involves Digital Signal Processing (DSP) techniques to filter noise and extract meaningful features before the data is stored in data lakes or time-series databases.
• GPS (Global Positioning System): GPS data provides geospatial coordinates (latitude, longitude, altitude) and temporal stamps. This data is critical for location-based services, fleet management, and logistics. Managing GPS data involves handling spatio-temporal datasets, which require specialized indexing techniques like R-trees or Quadtrees to perform efficient spatial queries, such as proximity searches or geofencing.

The integration of these sources into a unified data management framework typically follows a Lambda Architecture or Kappa Architecture, which allows for both batch processing of historical data and real-time processing of incoming streams from sensors and GPS devices.