{"id":21500,"date":"2017-09-15T09:59:50","date_gmt":"2017-09-15T14:59:50","guid":{"rendered":"https:\/\/blog.planview.com\/elk-stack-for-improved-support\/"},"modified":"2017-09-15T09:59:50","modified_gmt":"2017-09-15T14:59:50","slug":"elk-stack-for-improved-support","status":"publish","type":"post","link":"https:\/\/blog.planview.com\/elk-stack-for-improved-support\/","title":{"rendered":"ELK Stack for Improved Support"},"content":{"rendered":"<figure>\n<div class=\"feature-image\"><img decoding=\"async\" class=\"aligncenter size-full\" src=\"https:\/\/media.planview.com\/tasktop_blog\/wp-content\/uploads\/2017\/09\/kibana-feature.png\" alt=\"ELK Stack for Improved Support\"><\/div>\n<\/figure>\n<p>The ELK stack, composed of <a href=\"https:\/\/www.elastic.co\/products\/elasticsearch\">Elasticsearch<\/a>, <a href=\"https:\/\/www.elastic.co\/products\/logstash\">Logstash<\/a> and <a href=\"https:\/\/www.elastic.co\/products\/kibana\">Kibana<\/a>, is world-class dashboarding for real-time monitoring of server environments, enabling sophisticated analysis and troubleshooting. Could we also leverage this great tooling to our advantage in situations where access to the server environment is an impossibility? Recently while investigating a customer support case, I looked into whether or not we could create a repeatable process to enable analysis of log files provided as part of a support case. Following are details of the approach that we came up with.<\/p>\n<p>The requirements were pretty straight-forward:<\/p>\n<ul>\n<li>Enable analysis of multi-GB logs provided as part of a support request<\/li>\n<li>Use familiar, first-class tooling<\/li>\n<li>Zero-installation usage by anyone on the support or engineering team<\/li>\n<li>Zero maintenance (no infrastructure needed)<\/li>\n<\/ul>\n<p>For this we chose <a href=\"https:\/\/www.elastic.co\/products\">ELK<\/a> and <a href=\"https:\/\/docs.docker.com\/compose\/\">Docker Compose<\/a>, with the idea that anyone could bring up and tear down an environment with very little effort. Rather than monitor logs in real time however, we needed to pull in logs from a folder on the local machine. For this we used <a href=\"https:\/\/www.elastic.co\/products\/beats\/filebeat\">Filebeat<\/a>.<\/p>\n<p>This is the <code>docker-compose.yml<\/code> that we came up with:<\/p>\n<pre><code>elk:\n  image: sebp\/elk\n  ports:\n    - &quot;5601:5601&quot;\n    - &quot;9200:9200&quot;\n    - &quot;5044:5044&quot;\n  volumes:\n    - ${PWD}\/02-beats-input.conf:\/etc\/logstash\/conf.d\/02-beats-input.conf\n    - ${PWD}\/log:\/mnt\/log\nfilebeat:\n  image: docker.elastic.co\/beats\/filebeat:5.5.1\n  links:\n    - &quot;elk:logstash&quot;\n  volumes:\n    - ${PWD}\/filebeat.yml:\/usr\/share\/filebeat\/filebeat.yml\n    - ${PWD}\/log:\/mnt\/log<\/code><\/pre>\n<p>This Docker Compose file brings up two containers: <code>elk<\/code>, which as you might have guessed runs Elasticsearch, Logstash and Kibana, and <code>filebeat<\/code>, a container for reading log files that feeds the elk container with data.<\/p>\n<p>The filebbeat container is the most interesting one: it reads files from a local folder named <code>log<\/code> in the current directory of the Docker <em>host<\/em> machine. With the brilliance of <code>${PWD}<\/code> support in Docker Compose, all we have to do is move support log files into that folder!<\/p>\n<p>The following <code>filebeat.yml<\/code> configuration is needed:<\/p>\n<pre><code>filebeat.prospectors:\n- input_type: log\n  paths:\n    - \/mnt\/log\/*\n  include_lines: [&quot;.*? ERROR &quot;]\n  multiline.pattern: ^s*dddd-dd-dd dd:dd:dd,ddd [\n  multiline.negate: true\n  multiline.match: after\n<\/code><code>processors:\n- add_cloud_metadata:\n<\/code><code>output.logstash:\n  # The Logstash hosts\n  hosts: [&quot;logstash:5044&quot;]<\/code><\/pre>\n<p>This one is configured to handle multi-line log entries (including Java stack traces) where the initial line of each log entry starts with a timestmap. The <code>multiline.pattern<\/code> above may need adjusting to suit your log files.<\/p>\n<p>All that remains to get this working is the beats configuration, <code>02-beats-input.conf<\/code>, which uses a bit of filtering hackery to split up the unstructured log entries into structured data before it&#x2019;s added to Elasticsearch:<\/p>\n<pre><code>input {\n  beats {\n    port =&gt; 5044\n  }\n}\nfilter {\n  grok {\n    match =&gt; {\n      &quot;message&quot; =&gt; &quot;s*(?&lt;entry_date&gt;dddd-dd-dd) (?&lt;entry_time&gt;dd:dd:dd),(?&lt;entry_time_millis&gt;ddd) [(?&lt;thread_id&gt;[^]]+)] (?&lt;severity&gt;[^s]+) (?&lt;category&gt;[^s]+) - (?:(?&lt;error_code&gt;CCRRTT-d+(E|W)):s+)?(?&lt;message_text&gt;.*)&quot;\n    }\n  }\n  mutate {\n    add_field =&gt; {\n      &quot;entry_timestamp&quot; =&gt; &quot;%{entry_date}T%{entry_time}.%{entry_time_millis}Z&quot;\n    }\n    remove_field =&gt; [&quot;entry_date&quot;, &quot;entry_time&quot;, &quot;entry_time_millis&quot;]\n  }\n  mutate {\n    remove_field =&gt; [&quot;message&quot;]\n  }\n  mutate {\n    add_field =&gt; {\n      &quot;message&quot; =&gt; &quot;%{message_text}&quot;\n    }\n    remove_field =&gt; [&quot;message_text&quot;]\n  }\n  grok {\n    match =&gt; {\n      &quot;message&quot; =&gt; &quot;s*(?&lt;message_summary&gt;.*?) Cause Context:.*&quot;\n    }\n  }\n  grok {\n    match =&gt; {\n      &quot;message_summary&quot; =&gt; &quot;s*(?&lt;message_first_sentence&gt;.*?.).*&quot;\n    }\n  }\n}<\/code><\/pre>\n<p>After creating those files I ended up with the following:<\/p>\n<pre><code>.\/\n.\/docker-compose.yml\n.\/logs\/\n.\/02-beats-input.conf\n.\/filebeat.yml<\/code><\/pre>\n<p>With a simple <code>docker-compose up<\/code>, I moved over 56GB of log files into the <code>logs<\/code> folder and grabbed coffee. After a few minutes I was happily analyzing the situation using a Kibana dashboard:<\/p>\n<p><img class=\"lazyload\" alt height=\"606\" data-src=\"https:\/\/media.planview.com\/tasktop_blog\/wp-content\/uploads\/2017\/09\/kibana-dashboard.png\"><\/p>\n<p>In this example, we can see a chart of error codes and distinct messages over time.<\/p>\n<p>To make this process even smoother, we used <a href=\"https:\/\/www.npmjs.com\/package\/elasticdump\">elasticdump<\/a> to export our Kibana dashboards for other support cases.<\/p>\n<p>To export dashboards:<\/p>\n<pre><code>elasticdump --input=http:\/\/localhost:9200\/.kibana --output=$ --type=data &gt; kibana-settings.json<\/code><\/pre>\n<p>To import dashboards:<\/p>\n<pre><code>elasticdump --input=.\/kibana-settings.json --output=http:\/\/localhost:9200\/.kibana --type=data<\/code><\/pre>\n<p>Using ELK for post-mortem analysis of log files is a snap. The approach outlined above makes the process repeatable with trivial steps that anyone can follow, with no need to maintain ELK infrastructure.<\/p>\n<p><em>this article was originally published at <a href=\"http:\/\/greensopinion.com\/2017\/09\/09\/elk-stack-for-improved-support.html\">greensopinion.com<\/a><\/em><\/p>\n","protected":false},"excerpt":{"rendered":"<p>The ELK stack, composed of Elasticsearch, Logstash and Kibana, is world-class dashboarding for real-time monitoring of server environments, enabling sophisticated analysis and troubleshooting. Could we also leverage this great tooling to our advantage in situations where access to the server environment is an impossibility? Recently while investigating a customer support case, I looked into whether&#8230;<\/p>\n","protected":false},"author":226,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_editorskit_title_hidden":false,"_editorskit_reading_time":0,"_editorskit_is_block_options_detached":false,"_editorskit_block_options_position":"{}","footnotes":""},"categories":[9548],"tags":[],"class_list":["post-21500","post","type-post","status-publish","format-standard","hentry","category-engineering-teams"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v26.8 (Yoast SEO v26.8) - https:\/\/yoast.com\/product\/yoast-seo-premium-wordpress\/ -->\n<title>ELK Stack for Improved Support | Tasktop Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/blog.planview.com\/elk-stack-for-improved-support\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"ELK Stack for Improved Support\" \/>\n<meta property=\"og:description\" content=\"The ELK stack, composed of Elasticsearch, Logstash and Kibana, is world-class dashboarding for real-time monitoring of server environments, enabling sophisticated analysis and troubleshooting. Could we also leverage this great tooling to our advantage in situations where access to the server environment is an impossibility? Recently while investigating a customer support case, I looked into whether...\" \/>\n<meta property=\"og:url\" content=\"https:\/\/blog.planview.com\/elk-stack-for-improved-support\/\" \/>\n<meta property=\"og:site_name\" content=\"Planview Blog\" \/>\n<meta property=\"article:publisher\" content=\"http:\/\/www.facebook.com\/pages\/Planview-Inc\/89422974772\" \/>\n<meta property=\"article:published_time\" content=\"2017-09-15T14:59:50+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/media.planview.com\/tasktop_blog\/wp-content\/uploads\/2017\/09\/kibana-feature.png\" \/>\n<meta name=\"author\" content=\"Patrick Anderson\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@Planview\" \/>\n<meta name=\"twitter:site\" content=\"@Planview\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Patrick Anderson\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/blog.planview.com\/elk-stack-for-improved-support\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/blog.planview.com\/elk-stack-for-improved-support\/\"},\"author\":{\"name\":\"Patrick Anderson\",\"@id\":\"https:\/\/blog.planview.com\/#\/schema\/person\/0a937ebcbb167df16ae058729e3a8843\"},\"headline\":\"ELK Stack for Improved Support\",\"datePublished\":\"2017-09-15T14:59:50+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/blog.planview.com\/elk-stack-for-improved-support\/\"},\"wordCount\":462,\"publisher\":{\"@id\":\"https:\/\/blog.planview.com\/#organization\"},\"image\":{\"@id\":\"https:\/\/blog.planview.com\/elk-stack-for-improved-support\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/media.planview.com\/tasktop_blog\/wp-content\/uploads\/2017\/09\/kibana-feature.png\",\"articleSection\":[\"Engineering Teams\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/blog.planview.com\/elk-stack-for-improved-support\/\",\"url\":\"https:\/\/blog.planview.com\/elk-stack-for-improved-support\/\",\"name\":\"ELK Stack for Improved Support | Tasktop Blog\",\"isPartOf\":{\"@id\":\"https:\/\/blog.planview.com\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/blog.planview.com\/elk-stack-for-improved-support\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/blog.planview.com\/elk-stack-for-improved-support\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/media.planview.com\/tasktop_blog\/wp-content\/uploads\/2017\/09\/kibana-feature.png\",\"datePublished\":\"2017-09-15T14:59:50+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/blog.planview.com\/elk-stack-for-improved-support\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/blog.planview.com\/elk-stack-for-improved-support\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/blog.planview.com\/elk-stack-for-improved-support\/#primaryimage\",\"url\":\"https:\/\/media.planview.com\/tasktop_blog\/wp-content\/uploads\/2017\/09\/kibana-feature.png\",\"contentUrl\":\"https:\/\/media.planview.com\/tasktop_blog\/wp-content\/uploads\/2017\/09\/kibana-feature.png\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/blog.planview.com\/elk-stack-for-improved-support\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/blog.planview.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"ELK Stack for Improved Support\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/blog.planview.com\/#website\",\"url\":\"https:\/\/blog.planview.com\/\",\"name\":\"Planview Blog\",\"description\":\"Leading the conversation on digital connected work\",\"publisher\":{\"@id\":\"https:\/\/blog.planview.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/blog.planview.com\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/blog.planview.com\/#organization\",\"name\":\"Planview\",\"url\":\"https:\/\/blog.planview.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/blog.planview.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/blog.planview.com\/wp-content\/uploads\/2015\/06\/planview-logo-black.png\",\"contentUrl\":\"https:\/\/blog.planview.com\/wp-content\/uploads\/2015\/06\/planview-logo-black.png\",\"width\":280,\"height\":66,\"caption\":\"Planview\"},\"image\":{\"@id\":\"https:\/\/blog.planview.com\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"http:\/\/www.facebook.com\/pages\/Planview-Inc\/89422974772\",\"https:\/\/x.com\/Planview\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/blog.planview.com\/#\/schema\/person\/0a937ebcbb167df16ae058729e3a8843\",\"name\":\"Patrick Anderson\",\"description\":\"Patrick is Senior Content Manager at Tasktop and oversees the company's content and thought leadership programs. Outside the office, you\u2019ll find him reading, writing, slapping some bass (poorly), rambling in nature and following his English football (soccer) team, West Ham United.\",\"url\":\"https:\/\/blog.planview.com\/author\/patrick-anderson\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"ELK Stack for Improved Support | Tasktop Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/blog.planview.com\/elk-stack-for-improved-support\/","og_locale":"en_US","og_type":"article","og_title":"ELK Stack for Improved Support","og_description":"The ELK stack, composed of Elasticsearch, Logstash and Kibana, is world-class dashboarding for real-time monitoring of server environments, enabling sophisticated analysis and troubleshooting. Could we also leverage this great tooling to our advantage in situations where access to the server environment is an impossibility? Recently while investigating a customer support case, I looked into whether...","og_url":"https:\/\/blog.planview.com\/elk-stack-for-improved-support\/","og_site_name":"Planview Blog","article_publisher":"http:\/\/www.facebook.com\/pages\/Planview-Inc\/89422974772","article_published_time":"2017-09-15T14:59:50+00:00","og_image":[{"url":"https:\/\/media.planview.com\/tasktop_blog\/wp-content\/uploads\/2017\/09\/kibana-feature.png","type":"","width":"","height":""}],"author":"Patrick Anderson","twitter_card":"summary_large_image","twitter_creator":"@Planview","twitter_site":"@Planview","twitter_misc":{"Written by":"Patrick Anderson","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/blog.planview.com\/elk-stack-for-improved-support\/#article","isPartOf":{"@id":"https:\/\/blog.planview.com\/elk-stack-for-improved-support\/"},"author":{"name":"Patrick Anderson","@id":"https:\/\/blog.planview.com\/#\/schema\/person\/0a937ebcbb167df16ae058729e3a8843"},"headline":"ELK Stack for Improved Support","datePublished":"2017-09-15T14:59:50+00:00","mainEntityOfPage":{"@id":"https:\/\/blog.planview.com\/elk-stack-for-improved-support\/"},"wordCount":462,"publisher":{"@id":"https:\/\/blog.planview.com\/#organization"},"image":{"@id":"https:\/\/blog.planview.com\/elk-stack-for-improved-support\/#primaryimage"},"thumbnailUrl":"https:\/\/media.planview.com\/tasktop_blog\/wp-content\/uploads\/2017\/09\/kibana-feature.png","articleSection":["Engineering Teams"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/blog.planview.com\/elk-stack-for-improved-support\/","url":"https:\/\/blog.planview.com\/elk-stack-for-improved-support\/","name":"ELK Stack for Improved Support | Tasktop Blog","isPartOf":{"@id":"https:\/\/blog.planview.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/blog.planview.com\/elk-stack-for-improved-support\/#primaryimage"},"image":{"@id":"https:\/\/blog.planview.com\/elk-stack-for-improved-support\/#primaryimage"},"thumbnailUrl":"https:\/\/media.planview.com\/tasktop_blog\/wp-content\/uploads\/2017\/09\/kibana-feature.png","datePublished":"2017-09-15T14:59:50+00:00","breadcrumb":{"@id":"https:\/\/blog.planview.com\/elk-stack-for-improved-support\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/blog.planview.com\/elk-stack-for-improved-support\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/blog.planview.com\/elk-stack-for-improved-support\/#primaryimage","url":"https:\/\/media.planview.com\/tasktop_blog\/wp-content\/uploads\/2017\/09\/kibana-feature.png","contentUrl":"https:\/\/media.planview.com\/tasktop_blog\/wp-content\/uploads\/2017\/09\/kibana-feature.png"},{"@type":"BreadcrumbList","@id":"https:\/\/blog.planview.com\/elk-stack-for-improved-support\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/blog.planview.com\/"},{"@type":"ListItem","position":2,"name":"ELK Stack for Improved Support"}]},{"@type":"WebSite","@id":"https:\/\/blog.planview.com\/#website","url":"https:\/\/blog.planview.com\/","name":"Planview Blog","description":"Leading the conversation on digital connected work","publisher":{"@id":"https:\/\/blog.planview.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/blog.planview.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/blog.planview.com\/#organization","name":"Planview","url":"https:\/\/blog.planview.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/blog.planview.com\/#\/schema\/logo\/image\/","url":"https:\/\/blog.planview.com\/wp-content\/uploads\/2015\/06\/planview-logo-black.png","contentUrl":"https:\/\/blog.planview.com\/wp-content\/uploads\/2015\/06\/planview-logo-black.png","width":280,"height":66,"caption":"Planview"},"image":{"@id":"https:\/\/blog.planview.com\/#\/schema\/logo\/image\/"},"sameAs":["http:\/\/www.facebook.com\/pages\/Planview-Inc\/89422974772","https:\/\/x.com\/Planview"]},{"@type":"Person","@id":"https:\/\/blog.planview.com\/#\/schema\/person\/0a937ebcbb167df16ae058729e3a8843","name":"Patrick Anderson","description":"Patrick is Senior Content Manager at Tasktop and oversees the company's content and thought leadership programs. Outside the office, you\u2019ll find him reading, writing, slapping some bass (poorly), rambling in nature and following his English football (soccer) team, West Ham United.","url":"https:\/\/blog.planview.com\/author\/patrick-anderson\/"}]}},"_links":{"self":[{"href":"https:\/\/blog.planview.com\/wp-json\/wp\/v2\/posts\/21500","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.planview.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.planview.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.planview.com\/wp-json\/wp\/v2\/users\/226"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.planview.com\/wp-json\/wp\/v2\/comments?post=21500"}],"version-history":[{"count":0,"href":"https:\/\/blog.planview.com\/wp-json\/wp\/v2\/posts\/21500\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.planview.com\/wp-json\/wp\/v2\/media?parent=21500"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.planview.com\/wp-json\/wp\/v2\/categories?post=21500"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.planview.com\/wp-json\/wp\/v2\/tags?post=21500"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}