Elasticsearch significant terms on nested objects -

-

for masterthesis using elasticsearch measure significance of sentences, paragraphs , documents rest of index. i've used 3 different indexes enable fast querying. works fine, want evaluate if possibe same nested objects or parent child relationships.

here try set , query significant terms nestd objects:

put /test_nested {     "settings": {        "analysis": {         "filter": {           "german_stop": {             "type":       "stop",             "stopwords":  "_german_"            },           "german_keywords": {             "type":       "keyword_marker",             "keywords":   [""]            },           "german_stemmer": {             "type":       "stemmer",             "language":   "light_german"                 },                 "shingle_bigram": {                     "type":       "shingle",                     "max_shingle_size": 2,                     "min_shingle_size": 2,                     "output_unigrams": false                 },                 "shingle_trigram": {                     "type":       "shingle",                     "max_shingle_size": 3,                     "min_shingle_size": 3,                     "output_unigrams": false                                     }         },         "analyzer": {           "unigram": {             "tokenizer":  "standard",             "filter": [               "lowercase",               "german_stop",               "german_keywords",               "german_normalization",               "german_stemmer"             ]           }         }       }     },     "mappings": {     "document": {       "properties": {         "tags" : {           "type" : "string",           "analyzer" : "unigram",           "index" : "analyzed"         },         "publishdate" : {           "type" : "date"          },         "paragraphs": {           "type": "nested",           "properties": {             "sentences" :{               "type" : "nested",               "properties": {                 "textbody": {                   "type": "string",                   "analyzer" : "unigram",                   "index" : "analyzed",                   "term_vector" : "with_positions_offsets",                 "term_statistics" : true                 }               }             }           }         }       }     }   } }

and 2 test documents: 

put /test_nested/document/1 {   "tags" : "derspiegel, frankfurt",   "publishdate" : "2005-12-11",   "paragraphs" : [     {       "sentences" : [          {"textbody" : "größter anzunehmender einschlag"},         {"textbody": "es gibt ziemlich blöde vorurteile über fußball."},         {"textbody": "eines lautet: der ball ist rund."},         {"textbody": "freitagabend, messehalle 1 in leipzig, die auslosung zur fußballweltmeisterschaft: der ball ist gar nicht rund."}         ]     }   ] }  put /test_nested/document/2 {   "tags" : "derspiegel, frankfurt",   "publishdate" : "2005-12-11",   "paragraphs" : [     {       "sentences" : [          {"textbody" : "dafür aber kann man mit einem ball auch viel mehr anstellen als mit diesen runden, kleinen dingern, die früher aus leder waren und heute aus polyurethan sind."},         {"textbody": "zum beispiel die gigantischste fußball-wm-auslosungsshow aller zeiten zelebrieren."},         {"textbody": "eine show, die zum globalen fußball passt."},         {"textbody": "hauptsache riesig - wen interessiert schon rund?"}         ]     },     {       "sentences" : [         {"textbody" : "mit der verteilung der 32 teams auf ihre acht gruppen bekamen die deutschen damit erstmals auch einen vorgeschmack auf das gewicht und die wucht der wm im nächsten jahr." },         {"textbody" : "mag die nachricht des abends auch gewesen sein, dass deutschland gegen costa rica, polen und ecuador spielt und dass im achtelfinale die engländer drohen, die botschaft des spektakels von leipzig heißt, dass die wm mit einer opulenz über das land kommen wird, von der sich die deutschen bisher noch gar keine rechte vorstellung gemacht haben." },         {"textbody" : "die stimme von 1974 gehörte wolfhard kuhlins, sportchef des hr, und das weltereignis war nach 45 minuten ausgestrahlt, nicht nach 150." },         {"textbody" : "zwar kam auch schon franz beckenbauer zum interview ins studio, aber selbst der kaiser war noch nicht, er mal wurde: zum schwarzen anzug trug er weiße socken." }         ]     }   ] }

unfortunately don't significant terms following query:

get test_nested/document/_search?search_type=count {   "query" : {     "match_all" :{}   },   "aggs" :{       "sentences":{         "nested" :{           "path" : "paragraphs.sentences"         }       },       "aggs" : {              "significant_terms" : {                "chi_square": {},                "field" : "paragraphs.sentences.textbody"             }        }   } }

you had syntax error, basically. seems want:

post test_nested/document/_search?search_type=count {    "query": {       "match_all": {}    },    "aggs": {       "sentences": {          "nested": {             "path": "paragraphs.sentences"          },          "aggs": {             "sentances_sig_terms": {                "significant_terms": {                   "chi_square": {},                   "field": "paragraphs.sentences.textbody"                }             }          }       }    } }

here's code used test it:

http://sense.qbox.io/gist/e53122ea5887bf48a9bab570ad1c63546494026d

very well-written question, way.


Comments

Post a Comment

Popular posts from this blog

javascript - Chart.js (Radar Chart) different scaleLineColor for each scaleLine -

-
Image
i try , create radar chart using chart.js has various colours each scaleline, or coloured between scalelines. wondering if possible? from: to: i have working graph, though there doesn't seem method change individual scale lines. kind regards leigh you can extend radar chart type this, so chart.types.radar.extend({ name: "radaralt", initialize: function (data) { chart.types.radar.prototype.initialize.apply(this, arguments); var originalscaledraw = this.scale.draw; var ctx = this.chart.ctx; this.scale.draw = function () { var linewidth = this.linewidth; // bypasses line drawing in originalscaledraw this.linewidth = linewidth; originalscaledraw.apply(this, arguments); ctx.linewidth = this.linewidth; var scale = this; // draw chart.helpers.each(scale.ylabels, function (label, index) { // color of ...
Read more

apache - Error with PHP mail(): Multiple or malformed newlines found in additional_header -

-
suddenly have started receiving above error without changes having been made script. host 1and1 (i know...) the script still works fine on different server, , suspicion there must have been server config change has lead this, although hosts plead ignorance. there's no information on above error @ in google can find - have ideas? server running apache if helps. had similar problem. came out of blue. no php code changed. what changed: php upgraded 5.5.25-1 5.5.26. a security risk in php mail() function has been fixed , newlines in additional_headers allowed no more. because newlines mean: starts email message (and surely don't want inject newlines through headers followed evil message). what have worked fine, e.g. having newlines after headers or passing whole message additional_headers , function no more. solution : sanitize headers. no multiple newlines in additional_headers argument. these count "multiple or malformed newlines":...
Read more

java - Android – MapFragment overlay button shadow, just like MyLocation button -

-
Image
i'm struggling add shadow custom overlay button on google map. goal make custom button google's mylocation button. here's how app looks now: layout.xml <relativelayout xmlns:android="http://schemas.android.com/apk/res/android" xmlns:tools="http://schemas.android.com/tools" xmlns:map="http://schemas.android.com/apk/res-auto" android:layout_height="match_parent" android:layout_width="match_parent" style="@style/apptheme"> <fragment android:layout_width="match_parent" android:layout_height="match_parent" android:id="@+id/map" tools:context=".mapsactivity" android:name="com.google.android.gms.maps.supportmapfragment" android:layout_alignparentend="true" android:layout_alignparentstart="true" android:layout_alignparentbottom="true" ...
Read more