BD201Notes
  • Notes
  • Structure
  • Links
  • Topics for writiing task
    • HDFS
    • YARN
    • Hive
    • Kafka
    • Spark
    • ELK
  • Questions
    • Questions with answers
      • 2. HDFS
      • 3. DEVOPS
      • 4. HIVE
      • 7. STREAMING
      • 8. Elastic
    • Questions for screanings
      • junior
  • DevOps
    • DevOps: Practices
    • DevOps: Responsibilities in a Cloud
    • DevOps: CI/CD
    • DevOps: Tools. Docker
    • DevOps: Tools. Jenkins
    • DevOps: Tools. Puppet
    • DevOps: Tools. Ansible
    • DevOps: Tools. Kubernetes
  • Introduction To Big Data
    • Value of Data
    • Types of Data (Structured and Unstructured)
    • Schema on read & schema on write
    • Speed of data – from real time to batched processing
    • Storage & formats of data & compression
  • Hadoop Core
    • Hadoop Ecosystem
    • Hadoop Architecture
    • YARN
      • YARN: Logical and physical projections
      • YARN: Scheduler
      • YARN: Exec Modes
      • YARN: Security
    • MAP REDUCE
      • MAP REDUCE: Hints
  • Data Flow
    • NiFi
      • NiFi. HDP pre-configuration guide
      • StreamSets
  • Spark
    • Links
  • ELK
    • Elasticsearch
      • Elasticsearch: Installation and Configuration
    • Kibana
      • Kibana: Installation
    • Logstash
    • Beats
  • Orchestration & Scheduling
    • Apache Airflow
      • Airflow: Notes
Powered by GitBook
On this page
  • Problem statement
  • Problem 1: Connecting A & B
  • Problem 2: Real-life problem to solve
  • Solution: Dataflow Pipe Anatomy

Was this helpful?

Data Flow

PreviousMAP REDUCE: HintsNextNiFi

Last updated 5 years ago

Was this helpful?

Big Data Flow Based Programming

Problem statement

Let’s go through several real-life problems

Problem 1: Connecting A & B

The problem to bring data from producers to consumers. Producer in the modern world can be anything:

  • application servers

  • database servers

  • manufacturing machines

  • smart home appliances and personal wearables

  • cars and so on.

Consumers on other side are represented by alive people, another machines including storage only as well as AI enabled services

Also connecting A & B is not easy. Moving data is hard because of the number of reasons:

  • Formats

  • Protocols

  • Compliances

  • Schemas

  • Volume/Variety/Velocity of data

  • Ensuring security

  • Ensuring reliability

  • Ensuring availability

  • Additional effort to onboard new producer/consumer

Problem 2: Real-life problem to solve

In the process of migration to microservice architecture and every service exposed data in their own way: some of them exposed data in realtime manner while another provided only batch interface.

Solution: Dataflow Pipe Anatomy

Disparate transport mechanism which can transport, route, transform information, takes care about reliability and availability.

Connecting A & B
Disparate transport mechanism