Create a Kinesis data streamįirst, you need to create a Kinesis data stream to receive the streaming data.
The materialized view must be incrementally maintainable. You first need to create an external schema to map to Kinesis Data Streams and then create a materialized view to pull data from the stream. Setting up streaming ingestion in Amazon Redshift is a two-step process. The following diagram illustrates this workflow. Subsequent refreshes read data from the last SEQUENCE_NUMBER of the previous refresh until it reaches parity with the stream data. The very first refresh of the materialized view fetches data from the TRIM_HORIZON of the stream. Each slice consumes data from the allocated shards until the materialized view attains parity with the stream. When the materialized view is refreshed, Amazon Redshift compute nodes allocate each data shard to a compute slice. A materialized view is the landing area for data that is consumed from the stream. This means that you can perform downstream processing and transformations of streaming data using SQL at no additional cost and use your existing BI and analytics tools for real-time analytics.Īmazon Redshift streaming ingestion works by acting as a stream consumer.
The materialized views can also include SQL transforms as part of your ELT (extract, load and transform) pipeline.Īfter you define the materialized views, you can refresh them to query the most recent stream data. You can now connect to and access the data from the stream using SQL and simplify your data pipelines by creating materialized views directly on top of the stream. Solution overviewĪmazon Redshift streaming ingestion allows you to connect to Kinesis Data Streams directly, without the latency and complexity associated with staging the data in Amazon S3 and loading it into the cluster. Now, you can ingest data directly from the data stream.
This usually involved latency in the order of minutes and needed data pipelines on top of the data loaded from the stream. Sources of data can vary, from IoT devices to system telemetry, utility service usage, geolocation of devices, and more.īefore the launch of streaming ingestion, if you wanted to ingest real-time data from Kinesis Data Streams, you needed to stage your data in Amazon S3 and use the COPY command to load your data. Use cases for Amazon Redshift streaming ingestion center around working with data that is generated continually (streamed) and needs to be processed within a short period (latency) of its generation. You also want to enrich your real-time analytics by combining them with other data sources in your data warehouse. We hear from our customers that you want to evolve your analytics from batch to real time, and access your streaming data in your data warehouses with low latency and high throughput. We also discuss the benefits of streaming ingestion and common use cases. In this post, we walk through the steps to create a Kinesis data stream, generate and load streaming data, create a materialized view, and query the stream to visualize the results. Streaming ingestion allows you to achieve low latency in the order of seconds while ingesting hundreds of megabytes of data into your Amazon Redshift cluster. We’re excited to launch Amazon Redshift streaming ingestion for Amazon Kinesis Data Streams, which enables you to ingest data directly from the Kinesis data stream without having to stage the data in Amazon Simple Storage Service (Amazon S3). Tens of thousands of customers use Amazon Redshift to process exabytes of data per day and power analytics workloads such as high-performance business intelligence (BI) reporting, dashboarding applications, data exploration, and real-time analytics. Amazon Redshift offers up to three times better price performance than any other cloud data warehouse. Read the announcement in the AWS News Blog and learn more.Īmazon Redshift is a fast, scalable, secure, and fully managed cloud data warehouse that makes it simple and cost-effective to analyze all your data using standard SQL. August 30, 2023: Amazon Kinesis Data Analytics has been renamed to Amazon Managed Service for Apache Flink.
0 kommentar(er)
