> For the complete documentation index, see [llms.txt](https://bigdata-2.gitbook.io/bd201notes/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://bigdata-2.gitbook.io/bd201notes/data_flow.md).

# Data Flow

Big Data Flow Based Programming

## Problem statement

Let’s go through several real-life problems

### Problem 1: Connecting A & B

The problem to bring data from producers to consumers. Producer in the modern world can be anything:

* application servers
* database servers
* manufacturing machines
* smart home appliances and personal wearables
* cars and so on.

Consumers on other side are represented by alive people, another machines including storage only as well as AI enabled services

![Connecting A & B](https://3047264112-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LoWAdFshQklZGvmhPe2%2F-LteviIMxzz6Qh0hIvdb%2F-LtevjA2iQlJNhiPOiH6%2FIMG_1.png?generation=1573752076075755\&alt=media)

Also connecting A & B is not easy. Moving data is hard because of the number of reasons:

* Formats
* Protocols
* Compliances
* Schemas
* Volume/Variety/Velocity of data
* Ensuring security
* Ensuring reliability
* Ensuring availability
* Additional effort to onboard new producer/consumer

### Problem 2: Real-life problem to solve

In the process of migration to microservice architecture and every service exposed data in their own way: some of them exposed data in realtime manner while another provided only batch interface.

### Solution: Dataflow Pipe Anatomy

Disparate transport mechanism which can transport, route, transform information, takes care about reliability and availability.

![Disparate transport mechanism](https://3047264112-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LoWAdFshQklZGvmhPe2%2F-LteviIMxzz6Qh0hIvdb%2F-LtevjA8sCpV9oH_U4Zi%2FIMG_2.png?generation=1573752076003627\&alt=media)