Introduction
Data management with Apache NiFi
()
Prerequisites
()
1. Installing and Setting up Apache NiFi
Apache NiFi core concepts
()
Apache NiFi architecture
()
Mac OS: Download Apache NiFi
()
Windows: Download Apache NiFi
()
Mac OS and Windows: Start the NiFi server
()
Mac OS and Windows: Reset password and log into Apache NiFi
()
Explore the NiFi UI
()
2. Building and Running Dataflows
Add and configure processors
()
Run our first dataflow
()
Flow summaries and flow operations
()
3. Splitting and Merging FlowFiles
Split flow files into multiple files
()
Run and observe file splits
()
Merge content into a single file
()
Order files in a connection
()
Download the flow definition
()
The flow.json.fz file
()
4. Integrating NiFi with PostgreSQL
Set up a PostgreSQL table
()
Add get file and put database processors
()
Configure the CSV reader controller service
()
Configure the DBCP connection pool controller service
()
Run and encounter dataflow errors
()
Configure the ReplaceText processor
()
Run the dataflow to insert data into PostgreSQL
()
Configure the run schedule
()
Configure task concurrency
()
5. Executing SQL Queries in NiFi
Configure PutSQL for DML operations
()
Execute SQL commands and convert write formats
()
Convert between data formats
()
Update attributes and writing files
()
Run the flow end-to-end
()
Modularize dataflows with process groups
()
Run a dataflow with a process group
()
6. Integrating NiFi with Amazon S3
Create a PostgreSQL table, Amazon S3 bucket, and an IAM user
()
Read from files and store in a database
()
Configure the PutS3Object processor
()
Set up the AWS credential provider controller service
()
Read from the file system, filter records, and write to Amazon S3
()
7. Using Advanced Features of NiFi Dataflows
Configure backpressure
()
Observe the effects of backpressure
()
Configuring an InvokeHTTP processor
()
Use a funnel in the dataflow
()
Run a dataflow to read from a URL and write to files
()
8. Monitoring and Alerting in NiFi
Set up monitoring for the dataflow
()
Configure alerting for the dataflow
()
Email alerts on dataflow success
()
Email alerts on change in activity status
()
Conclusion
Summary and next steps
()
Ex_Files_Data_Apache_NiFi.zip
(447 KB)