Data Flow
Big Data Flow Based Programming
Problem statement
Let’s go through several real-life problems
Problem 1: Connecting A & B
The problem to bring data from producers to consumers. Producer in the modern world can be anything:
application servers
database servers
manufacturing machines
smart home appliances and personal wearables
cars and so on.
Consumers on other side are represented by alive people, another machines including storage only as well as AI enabled services
Also connecting A & B is not easy. Moving data is hard because of the number of reasons:
Formats
Protocols
Compliances
Schemas
Volume/Variety/Velocity of data
Ensuring security
Ensuring reliability
Ensuring availability
Additional effort to onboard new producer/consumer
Problem 2: Real-life problem to solve
In the process of migration to microservice architecture and every service exposed data in their own way: some of them exposed data in realtime manner while another provided only batch interface.
Solution: Dataflow Pipe Anatomy
Disparate transport mechanism which can transport, route, transform information, takes care about reliability and availability.
Last updated
