神刀安全网

Amazon Kinesis Agent Update – New Data Preprocessing Features

My colleague Ray Zhu wrote the guest post below to introduce you to some new data preprocessing features for the Amazon Kinesis Agent.

Jeff

;

Amazon Kinesis Agent is a stand-alone Java software application that provides an easy and reliable way to send data to Amazon Kinesis Streams and Amazon Kinesis Firehose . The agent monitors a set of files for new data and then sends it to Kinesis Streams or Kinesis Firehose continuously. It handles file rotation, checkpointing, and retrial upon failures. It also supportsAmazon CloudWatch so that you can closely monitor and troubleshoot the data flow from the agent.

Data Preprocessing with Kinesis Agent

Today we are adding data preprocessing capabilities to the agent so that your data can be well formatted before it is sent to Kinesis Streams or Kinesis Firehose. The agent currently supports the three processing options listed below. Because the agent is open source, you can further develop and extend these processing options.

SINGLELINE– This option converts a multi-line record to a single line record by removing newline characters, and leading and trailing spaces.

CSVTOJSON– This option converts a record from delimiter separated format to JSON format.

LOGTOJSON– This option converts a record from several commonly used log formats to JSON format. Currently supported log formats are Apache Common Log , Apache Combined Log, Apache Error Log, and RFC3164 (syslog).

Analyze Apache Tomcat Access Log in Near Real-Time

Let’s look at an example of analyzing Tomcat access logs in near real-time using Kinesis Agent’s preprocessing feature, Amazon Kinesis Firehose, andAmazon Redshift. Here’s the overall flow:

Amazon Kinesis Agent Update – New Data Preprocessing Features

First I need to create a table in my Redshift cluster to store the Tomcat access log. The following SQL statement is used to create the table:

CREATE TABLE logs( host VARCHAR(40), ident VARCHAR(25), authuser VARCHAR(25), datetime VARCHAR(60), request VARCHAR(2048), response SMALLINT NOT NULL, bytes INTEGER, referer VARCHAR(2048), agent VARCHAR(256));

Then I need to create a Kinesis Firehose delivery stream that continuously delivers data to the Redshift table created above:

Amazon Kinesis Agent Update – New Data Preprocessing Features

Now I’ve set up my Redshift table and Firehose delivery stream. Next I need to install the Kinesis Agent on my Tomcat server to monitor my Tomcat access log files and continuously send the log data to my delivery stream. Here is a screenshot of the raw Tomcat access log:

Amazon Kinesis Agent Update – New Data Preprocessing Features

In the agent configuration, I use the LOGTOJSON processing option to convert raw Tomcat access log data to JSON format before sending the data to my delivery stream. Here’s how I set that up:

{    "cloudwatch.emitMetrics":true,    "flows":[       {          "filePattern":"/data/access.log*",          "deliveryStream":"access_log_stream",          "initialPosition":"START_OF_FILE",          "dataProcessingOptions":[             {                "optionName":"LOGTOJSON",                "logFormat":"COMBINEDAPACHELOG"             }          ]       }    ] }

Everything is set up now and let’s start the agent! After a minute or two, my Tomcat access log data shows up in my S3 bucket and Redshift table. Here is how the data looks like in my S3 bucket. Notice that the raw log data has been nicely formatted as JSON:

Amazon Kinesis Agent Update – New Data Preprocessing Features

Here is how the data looks like in my Redshift table:

Amazon Kinesis Agent Update – New Data Preprocessing Features

I can run SQL queries to analyze my Tomcat access log, or use the Business Intelligence tool of my choice to visualize the data:

Amazon Kinesis Agent Update – New Data Preprocessing Features

It took me less than an hour to set up the whole data pipeline. Now I can analyze and visualize access log data using my favorite Business Intelligence tool, only minutes after the data is generated on my Tomcat server!

Available Now

Kinesis Agent’s data preprocessing feature is available now and you can start using it today – visit the Amazon Kinesis Agent Repository ! To learn more, read Use Agent to Preprocess Data in the Kinesis Firehose Developer Guide .

Ray Zhu , Senior Product Manager

转载本站任何文章请注明:转载至神刀安全网,谢谢神刀安全网 » Amazon Kinesis Agent Update – New Data Preprocessing Features

分享到:更多 ()

评论 抢沙发

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址