elasticsearch + logstash + kibana(ELK)是一套开源日志方案

  • Logstash:负责日志的收集,处理和储存
  • Elasticsearch:负责日志检索和分析
  • Kibana:负责日志的可视化(生成各种chart)

通常,我们会使用tail,grep在服务器上查看日志,但服务器权限并不适合开放给所有人,同时,也不是人人都适合上机进行操作,正如不是每个人都适合操作MySQL一样,ELK能方便的收集,查询日志,避免了开发人员直接上机执行命令带来的风险。

下面是安装步骤:

1.安装JDK

yum install java-1.7.0-openjdk

2.安装elasticsearch2.0

导入证书

rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch

添加elasticsearch源

vim /etc/yum.repo.d/elasticsearch.repo
[elasticsearch-2.x]
name=Elasticsearch repository for 2.x packages
baseurl=https://packages.elastic.co/elasticsearch/2.x/centos
gpgcheck=1
gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1

安装

yum install elasticsearch

 

启动elasticsearch

service elasticsearch start

 

3.安装logstash

wget https://download.elastic.co/logstash/logstash/packages/centos/logstash-2.0.0-1.noarch.rpm
rpm -ivh logstash-2.0.0-1.noarch.rpm

添加logstash配置

vim /etc/logstash/conf.d/nginx-accesss.conf
input {
    file {
        type => "nginx-access"
        path => "/var/log/nginx/access.log"
    }
}

filter {
    if [type] == "nginx-access" {
        grok {
            match => {"message" => "%{COMBINEDAPACHELOG}"}
        }

        geoip {
            source => "clientip"
            target => "geoip"
            database => "/etc/logstash/GeoLiteCity.dat"
            add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
            add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
        }

        mutate {
            convert => [ "[geoip][coordinates]", "float"]
        }
    }
}

output {
    elasticsearch {
        hosts => ["127.0.0.1:9200"]
    }
}

更多grok pattern看这里 https://github.com/elastic/logstash/blob/v1.4.2/patterns/grok-patterns

配置geoip(用于定位访客来源)

wget "https://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz"
gzip -d GeoLiteCity.dat.gz
mv GeoLiteCity.dat /etc/logstash/GeoLiteCity.dat

 

启动logstash

logstash -f /etc/logstash/conf.d/nginx-accesss.conf

4.安装kibana

wget "https://download.elastic.co/kibana/kibana/kibana-4.2.0-linux-x64.tar.gz"
tar -xvzf kibana-4.2.0-linux-x64.tar.gz

启动kibana

./kibana-4.2.0-linux-x64/bin/kibana
kibana-start

kibana-start


 

1.浏览器访问5601端口,创建新的索引模式

kibana-index-pattern

kibana-index-pattern

2.在Discover里可以搜索日志

kibana-demo

kibana-demo

3.在Visualize里查看图表

geo

由于elasticsearch和kibana默认是没有鉴权的,不建议直接开放在公网使用。关于elasticsearch权限控制,有兴趣的可以了解https://www.elastic.co/products/shield

 

 

Post Navigation