r - Elastic : make a light count query (vs search query) -

-

i accessing bulk data in elastic through r. analytics purpose need query data relatively long duration (say month). data month approx 4.5 million rows , r goes out of memory.

sample data below (for 1 day):

dt <- as.date("2015-09-01", "%y-%m-%d") frmdt <- strftime(dt,"%y-%m-%d")   todt <- as.date(dt+1)   todt <- strftime(todt,"%y-%m-%d")    connect(es_base="http://xx.yy.zzz.kk")   start_date <- as.integer(as.posixct(frmdt))*1000   end_date <- as.integer(as.posixct(todt))*1000     query <- sprintf('{"query":{"range":{"time":{"gte":"%s","lte":"%s"}}}}',start_date,end_date)   s_list <- elastic::search(index = "organised_2015_09",type = "property_search", body=query ,                      fields = c("trackid", "time"), size=1000000)$hits$hits   length(s_list) [1] 144612

this result 1 day has 144k records , 222 mb. sample list item below:

> s_list[[1]] $`_index` [1] "organised_2015_09"  $`_type` [1] "property_search"  $`_id` [1] "1441122918941"  $`_version` [1] 1  $`_score` [1] 1  $fields $fields$time $fields$time[[1]] [1] 1441122918941   $fields$trackid $fields$trackid[[1]] [1] "fd4b4ce88101e58623ba9e6e31971d1f"

actually summary count of number of items "trackid" , "time" (summarize every day) suffice analytics purpose. hence tried transform count query aggregations. constructed below query:

query < -'{"size" : 0, "query": {     "filtered": {         "query": {             "match_all": {}         },         "filter": {             "range": {                 "time": {                     "gte": 1441045800000,                     "lte": 1443551400000                 }             }         }     } }, "aggs": {     "articles_over_time": {         "date_histogram": {             "field": "time",             "interval": "day",             "time_zone": "+05:30"         },         "aggs": {             "group_by_state": {                 "terms": {                     "field": "trackid",                     "size": 0                 }             }         }     } } }'  response <- elastic::search(index="organised_recent",type="property_search",body=query, search_type="count")

however did not gain in speed or document size. think missing not sure what.


Comments

Post a Comment

Popular posts from this blog

javascript - Chart.js (Radar Chart) different scaleLineColor for each scaleLine -

-
Image
i try , create radar chart using chart.js has various colours each scaleline, or coloured between scalelines. wondering if possible? from: to: i have working graph, though there doesn't seem method change individual scale lines. kind regards leigh you can extend radar chart type this, so chart.types.radar.extend({ name: "radaralt", initialize: function (data) { chart.types.radar.prototype.initialize.apply(this, arguments); var originalscaledraw = this.scale.draw; var ctx = this.chart.ctx; this.scale.draw = function () { var linewidth = this.linewidth; // bypasses line drawing in originalscaledraw this.linewidth = linewidth; originalscaledraw.apply(this, arguments); ctx.linewidth = this.linewidth; var scale = this; // draw chart.helpers.each(scale.ylabels, function (label, index) { // color of ...
Read more

apache - Error with PHP mail(): Multiple or malformed newlines found in additional_header -

-
suddenly have started receiving above error without changes having been made script. host 1and1 (i know...) the script still works fine on different server, , suspicion there must have been server config change has lead this, although hosts plead ignorance. there's no information on above error @ in google can find - have ideas? server running apache if helps. had similar problem. came out of blue. no php code changed. what changed: php upgraded 5.5.25-1 5.5.26. a security risk in php mail() function has been fixed , newlines in additional_headers allowed no more. because newlines mean: starts email message (and surely don't want inject newlines through headers followed evil message). what have worked fine, e.g. having newlines after headers or passing whole message additional_headers , function no more. solution : sanitize headers. no multiple newlines in additional_headers argument. these count "multiple or malformed newlines":...
Read more

java - Android – MapFragment overlay button shadow, just like MyLocation button -

-
Image
i'm struggling add shadow custom overlay button on google map. goal make custom button google's mylocation button. here's how app looks now: layout.xml <relativelayout xmlns:android="http://schemas.android.com/apk/res/android" xmlns:tools="http://schemas.android.com/tools" xmlns:map="http://schemas.android.com/apk/res-auto" android:layout_height="match_parent" android:layout_width="match_parent" style="@style/apptheme"> <fragment android:layout_width="match_parent" android:layout_height="match_parent" android:id="@+id/map" tools:context=".mapsactivity" android:name="com.google.android.gms.maps.supportmapfragment" android:layout_alignparentend="true" android:layout_alignparentstart="true" android:layout_alignparentbottom="true" ...
Read more