Elastic Load DataSet
https://www.elastic.co/guide/en/kibana/current/tutorial-load-dataset.html
Type(Table) : shakespeare.json
{
"line_id": INT,
"play_name": "String",
"speech_number": INT,
"line_number": "String",
"speaker": "String",
"text_entry": "String",
}
Type(Table) : account.json
{
"account_number": INT,
"balance": INT,
"firstname": "String",
"lastname": "String",
"age": INT,
"gender": "M or F",
"address": "String",
"employer": "String",
"email": "String",
"city": "String",
"state": "String"
}
Type(Table) : logs.jsonl
{
"memory": INT,
"geo.coordinates": "geo_point"
"@timestamp": "date"
}
Elastic Mapping
curl -XPUT http://192.168.99.100:9200/shakespeare -d '
{
"mappings" : {
"_default_" : {
"properties" : {
"speaker" : {"type": "string", "index" : "not_analyzed" },
"play_name" : {"type": "string", "index" : "not_analyzed" },
"line_id" : { "type" : "integer" },
"speech_number" : { "type" : "integer" }
}
}
}
}
';
RDBMS에서 컬럼의 Type을 정의한다고 생각하면 된다.
그러나 Error가 발생한다.
{
"error" : {
"root_cause" : [
{
"type" : "security_exception",
"reason" : "missing authentication token for REST request [/shakespeare?pretty]",
"header" : {
"WWW-Authenticate" : "Basic realm=\"security\" charset=\"UTF-8\""
}
}
],
"type" : "security_exception",
"reason" : "missing authentication token for REST request [/shakespeare?pretty]",
"header" : {
"WWW-Authenticate" : "Basic realm=\"security\" charset=\"UTF-8\""
}
},
"status" : 401
}
X-pack security 기능이 Enable 되어 있기 때문이다.
curl -u elatic:changeme -XPUT http://192.168.99.100:9200/shakespeare -d '
{
"mappings" : {
"_default_" : {
"properties" : {
"speaker" : {"type": "string", "index" : "not_analyzed" },
"play_name" : {"type": "string", "index" : "not_analyzed" },
"line_id" : { "type" : "integer" },
"speech_number" : { "type" : "integer" }
}
}
}
}
';
-u username:password 를 추가해서 다시 보내보자.
성공했다는 메시지를 받을 수 있다.
{"acknowledged":true,"shards_acknowledged":true}
생성된 Type을 조회해 보자.
curl -u elastic:changeme -XGET http://192.168.99.100:9200/shakespeare?pretty
{
"shakespeare" : {
"aliases" : { },
"mappings" : {
"_default_" : {
"properties" : {
"line_id" : {
"type" : "integer"
},
"play_name" : {
"type" : "keyword"
},
"speaker" : {
"type" : "keyword"
},
"speech_number" : {
"type" : "integer"
}
}
}
},
"settings" : {
"index" : {
"creation_date" : "1488906534773",
"number_of_shards" : "5",
"number_of_replicas" : "1",
"uuid" : "F56j2JFBQzynRxLK8ZWftA",
"version" : {
"created" : "5020199"
},
"provided_name" : "shakespeare"
}
}
}
}
"mappings" : {
"_default_" : {
"properties" : {
"line_id" : {
"type" : "integer"
},
"play_name" : {
"type" : "keyword"
},
"speaker" : {
"type" : "keyword"
},
"speech_number" : {
"type" : "integer"
}
}
}
}
"mappings"에 "type"이 생성되었다.
Document 를 먼저 생성할 수도 있지만, Date type이 string으로 저장되는 경우가 있기 때문에
우리는 항상 mappings > document 를 생성하는 방향이 올바른 방향이다.
이제 logs.jsonl 도 추가해 보자.
curl -u elastic:changeme -XPUT http://192.168.99.100:9200/logstash-2015.05.18 -d '
{
"mappings": {
"log": {
"properties": {
"geo": {
"properties": {
"coordinates": {
"type": "geo_point"
}
}
}
}
}
}
}
';
curl -u elastic:changeme -XPUT http://192.168.99.100:9200/logstash-2015.05.19 -d '
{
"mappings": {
"log": {
"properties": {
"geo": {
"properties": {
"coordinates": {
"type": "geo_point"
}
}
}
}
}
}
}
';
curl -u elastic:changeme -XPUT http://192.168.99.100:9200/logstash-2015.05.20 -d '
{
"mappings": {
"log": {
"properties": {
"geo": {
"properties": {
"coordinates": {
"type": "geo_point"
}
}
}
}
}
}
}
';
account.json 을 따로 추가하지 않는 이유에 대해선 Document를 먼저 생성할 수 있다는 것을 알려주기 위한 것 같다.
아래 "Elastic Put DataSet" 까지 기다려보자.
Elastic Put DataSet
curl -u elastic:changeme -XPOST '192.168.99.100:9200/bank/account/_bulk?pretty' --data-binary @accounts.json
curl -u elastic:changeme -XPOST '192.168.99.100:9200/shakespeare/_bulk?pretty' --data-binary @shakespeare.json
curl -u elastic:changeme -XPOST '192.168.99.100:9200/_bulk?pretty' --data-binary @logs.jsonl
bulk와 관련된 API : bulk
Elastic Cat Indicates
curl -u elastic:changeme '192.168.99.100:9200/_cat/indices?v'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open .monitoring-es-2-2017.03.07 Bl7koyv6Q7K_zLwauerzbA 1 1 5625 180 3.3mb 3.3mb
yellow open logstash-2015.05.19 4zyD7xiARlOkZSb5mSoWFw 5 1 4624 0 31mb 31mb
yellow open .monitoring-logstash-2-2017.03.07 Pthc7iI2ReCyDoR0pYJJQw 1 1 448 0 207.7kb 207.7kb
yellow open .kibana eA4O9svbThaPZcChFLSnrQ 1 1 1 0 3.2kb 3.2kb
yellow open .monitoring-data-2 iQfk6GdsR--nwoeEzI2MIw 1 1 4 0 55.5kb 55.5kb
yellow open logstash-2015.05.20 9KAfIADiQmWcOwP0YjUJGQ 5 1 4750 0 29.7mb 29.7mb
yellow open .monitoring-kibana-2-2017.03.07 64w96_ZKTciPYJdT5Xw1eg 1 1 452 0 193.8kb 193.8kb
yellow open logstash-2015.05.18 w40ShWMMTYWKAmUIJyQ8QA 5 1 4631 0 28.9mb 28.9mb
yellow open shakespeare F56j2JFBQzynRxLK8ZWftA 5 1 111396 0 28.2mb 28.2mb
yellow open bank dfrni_SeQhyLMzUXoiqvfg 5 1 1000 0 640.4kb 640.4kb