Apache Flink Architecture | Flink Tutorial

In this video we will discuss flink's architecture and flink's basic flow of its programming model. We'll start with flink architecture. Starting with the first component of the architecture. Flink on its own is the runtime core engine which does stream processing. On top of the engine, Flink has it abstraction APIs. Data set API to consume and process batch data sources and data stream API to consume and process real time streaming data. These two APIs are : programming abstractions and are the foundation for user programs. You will get to know how we use these in programs in the practical lectures. And data set API can be used in Java, scala or Python while data stream API can be used in Java and Scala only.
In addition to these APIs, Flink also has some libraries that provide additional functionality . As we know, Flink allows us to write relational queries to process structural tables. It allows us to process graphs and also make us able to apply machine learning algorithms on our data set.

For each of this task, Flink has built-in libraries. It has table API to query the tables and perform basic operations like join, groupBy etc on it. Table APi can be used in both batch and streaming programs. To run graph algorithms it has Gelly API and FlinkML library for machine learning algorithms where it natively executes iterative processing algorithms of machine learning. Flink also offers interactive data analytics by its zeeplin library. Zeeplin is a visualization tool which allows us to view the status of each step of processing in GUI. Talking about deployment of flink, Flink is able to run in a variety of environments Flink can run as a single machine on our local machine on JVM or it can run in a distributed cluster mode with any number of nodes in it. In cluster also it can be run standalone or it can use any resource manager like Hadoops yarn. Flink apps can even be
deployed on cloud like Google GCE and Amazons EC too. Now in terms of storage,Flink can be integrated to these file systems Local, HDFS and S3

It can also be integarted to databases like MongoDB, Hbase. And last 0bviously with streaming services like Kakfa, Flume, RabbitMQ. So this completes our Flink ecosystem. While learning any Flink app all of these components will be used and they work together to give us the desired output. And yes before we start any app or right of a first flink program, let us understand the flow programming model of flink as how the data enters, processed and output in it.

[abubakarsiddiqurrahmanrock]

Hi, Could you please answer me one of the question that I asked you in UDEMY platform (lecture 41)? Please reply there. I faced some errors while running the code (lecture 41).