java - Spark: Running multiple queries on multiple files, optimization -

-

i using spark 1.5.0.

i have set of files on s3 containing json data in sequence file format, worth around 60gb. have fire around 40 queries on dataset , store results s3.

  1. all queries select statements condition on same field. eg. select a,b,c t event_type='alpha', select x,y,z t event_type='beta' etc.
  2. i using aws emr 5 node cluster 2 core nodes , 2 task nodes.
  3. there fields missing in input. eg. a missing. so, first query, selects a fail. avoid have defined schemas each event_type. so, event_type alpha, schema {"a": "", "b": "", c:"", event_type=""}
  4. based on schemas defined each event, i'm creating dataframe input rdd each event corresponding schema.

i'm using following code:

javapairrdd<longwritable,byteswritable> inputrdd = jsc.sequencefile(bucket, longwritable.class, byteswritable.class); javardd<string> events = inputrdd.map(     new function<tuple2<longwritable,byteswritable>, string>() {         public string call(tuple2<longwritable,byteswritable> tuple) throws jsonexception, unsupportedencodingexception {             string valueasstring = new string(tuple._2.getbytes(), "utf-8");             jsonobject data = new jsonobject(valueasstring);             jsonobject payload = new jsonobject(data.getstring("payload"));             return payload.tostring();         }     } ); events.cache();  (string event_type: events_list) {     string query = //read query s3 file event_type.query      string jsonschemastring = //read schema s3 file event_type.json     list<string> jsonschema = arrays.aslist(jsonschemastring);     javardd<string> jsonschemardd = jsc.parallelize(jsonschema);     dataframe df_schema = sqlcontext.read().option("header", "true").json(jsonschemardd);     structtype schema = df_schema.schema();      dataframe df_query = sqlcontext.read().schema(schema).option("header", "true").json(events);     df_query.registertemptable(tablename);     dataframe df_results = sqlcontext.sql(query);      df_results.write().format("com.databricks.spark.csv").save("s3n://some_location); }

this code inefficient, takes around 6-8 hours run. how can optimize code?

  • should try using hivecontext.

  • i think current code taking multipe passes @ data, not sure though have cached rdd? how can in single pass if so.


Comments

Post a Comment

Popular posts from this blog

javascript - Chart.js (Radar Chart) different scaleLineColor for each scaleLine -

-
Image
i try , create radar chart using chart.js has various colours each scaleline, or coloured between scalelines. wondering if possible? from: to: i have working graph, though there doesn't seem method change individual scale lines. kind regards leigh you can extend radar chart type this, so chart.types.radar.extend({ name: "radaralt", initialize: function (data) { chart.types.radar.prototype.initialize.apply(this, arguments); var originalscaledraw = this.scale.draw; var ctx = this.chart.ctx; this.scale.draw = function () { var linewidth = this.linewidth; // bypasses line drawing in originalscaledraw this.linewidth = linewidth; originalscaledraw.apply(this, arguments); ctx.linewidth = this.linewidth; var scale = this; // draw chart.helpers.each(scale.ylabels, function (label, index) { // color of ...
Read more

apache - Error with PHP mail(): Multiple or malformed newlines found in additional_header -

-
suddenly have started receiving above error without changes having been made script. host 1and1 (i know...) the script still works fine on different server, , suspicion there must have been server config change has lead this, although hosts plead ignorance. there's no information on above error @ in google can find - have ideas? server running apache if helps. had similar problem. came out of blue. no php code changed. what changed: php upgraded 5.5.25-1 5.5.26. a security risk in php mail() function has been fixed , newlines in additional_headers allowed no more. because newlines mean: starts email message (and surely don't want inject newlines through headers followed evil message). what have worked fine, e.g. having newlines after headers or passing whole message additional_headers , function no more. solution : sanitize headers. no multiple newlines in additional_headers argument. these count "multiple or malformed newlines":...
Read more

java - Android – MapFragment overlay button shadow, just like MyLocation button -

-
Image
i'm struggling add shadow custom overlay button on google map. goal make custom button google's mylocation button. here's how app looks now: layout.xml <relativelayout xmlns:android="http://schemas.android.com/apk/res/android" xmlns:tools="http://schemas.android.com/tools" xmlns:map="http://schemas.android.com/apk/res-auto" android:layout_height="match_parent" android:layout_width="match_parent" style="@style/apptheme"> <fragment android:layout_width="match_parent" android:layout_height="match_parent" android:id="@+id/map" tools:context=".mapsactivity" android:name="com.google.android.gms.maps.supportmapfragment" android:layout_alignparentend="true" android:layout_alignparentstart="true" android:layout_alignparentbottom="true" ...
Read more