Analyze Access Log with Apache Spark


Apache Spark is a powerful tool to process large amounts of data. The docs show many examples to analyze csv-like data, which is already covered by spark csv. But how to analyze more complex data, e.g. an accesslog file? At first sight it looks quite simple because its human readable. But looking more sharp let us see, there is no simple way to use split operations to extract data. This artice shows an different approach to process text files, which easily may be adapted to any other line-based format.

The swiss army knife to analyze textual contents are regular expressions. By using capture groups within regular expressions whole lines of text are matched and split into fragments. A recent extension to regex are named capture groups which allow to analyze lines of text, split them into fields and assign each field a name and a value. The example will demonstrate how this will work.

The experienced user will see information like ip address, timestamp and http request details here. It is table data with columnar information. Simple split using whitespace as separator would break some of these columns, e.g. the date. Also the request string is only the first line of an http-request and contains whitespaces. In some cases a request may contain malicious content, e.g. if an invalid request was sent to web server.

To analyze text we start to create an regex and recognize the contents from left to right:

  • line 1: extract ap address to capture group “host”
  • line 2: skip some data
  • line 3: extract timestamp without braces
  • line 4: extract request content
  • line 5: extract http-status
  • line 6: extract response size

The grok debugger is a useful tool to create and test regular expressions with named capture groups in an interactive way. Just edit pattern or data and see result of extraction. Unfortunately java support for named capture groups is limited, so java 8 can evaluate such expressions but the api is not able to associate group name and value. Scala regex are based on java capabilities so scala is limited, too. In fact the spark shell does not support these types of regex. The good news: a short script will fix this limitation.

In first step some helper code is loaded, a scala class handling named capture groups.

Just for demonstration an example content is defined in variable “line”.

Than a regex matching the example ist defined. In next step the regex is applied to example.

An instance of helper class SmartMatcher is assigned to variable m. Regex is analyzed an field names from named capture groups are computed. Detected fields are returned by m.fieldNames. Also example text is parsed and a list of values is returned. An alternative is parseMap which return a map of key-values (field-value) pairs. This are the basics to process files, so let’s start processing whole files.

  • line 1: read file contents into RDD
  • line 4: create dataset from rdd, dataset is List[String] with an entry per column
  • line 6: create RDD from dataset
  • line 10: show some sample data

For more comfortable work we will convert the rdd to a dataframe. A dataframe requires a schema defintition, so any column is mapped to string-type.

A dataframe is created by combining rdd and schema definition:

Finally a text based web server acces log file is converted to an apache spark dataframe. By simply adapting regex to custom input formats, nearly any text based file can be processed. Building regex is a process of stepwise testing over several iterations.

{ Add a Comment }