machine learning - Dealing with unbalanced datasets in Spark MLlib -

-

i'm working on particular binary classification problem highly unbalanced dataset, , wondering if has tried implement specific techniques dealing unbalanced datasets (such smote) in classification problems using spark's mllib.

i'm using mllib's random forest implementation , tried simplest approach of randomly undersampling larger class didn't work expected.

i appreciate feedback regarding experience similar issues.

thanks,

class weight spark ml

as of moment, class weighting random forest algorithm still under development (see here)

but if you're willing try other classifiers - functionality has been added logistic regression.

consider case have 80% positives (label == 1) in dataset, theoretically want "under-sample" positive class. logistic loss objective function should treat negative class (label == 0) higher weight.

here example in scala of generating weight, add new column dataframe each record in dataset: 

def balancedataset(dataset: dataframe): dataframe = {      // re-balancing (weighting) of records used in logistic loss objective function     val numnegatives = dataset.filter(dataset("label") === 0).count     val datasetsize = dataset.count     val balancingratio = (datasetsize - numnegatives).todouble / datasetsize      val calculateweights = udf { d: double =>       if (d == 0.0) {         1 * balancingratio       }       else {         (1 * (1.0 - balancingratio))       }     }      val weighteddataset = dataset.withcolumn("classweightcol", calculateweights(dataset("label")))     weighteddataset   }

then, create classier follow:

new logisticregression().setweightcol("classweightcol").setlabelcol("label").setfeaturescol("features")

for more details, watch here: https://issues.apache.org/jira/browse/spark-9610

- predictive power

a different issue should check - whether features have "predictive power" label you're trying predict. in case after under-sampling still have low precision, maybe has nothing fact dataset imbalanced nature.

i exploratory data analysis - if classifier doesn't better random choice, there risk there no connection between features , class.

  • perform correlation analysis every feature label.
  • generating class specific histograms features (i.e. plotting histograms of data each class, given feature on same axis) can way show if feature discriminates between 2 classes.

overfitting - low error on training set , high error on test set might indication overfit using overly flexible feature set.

bias variance - check whether classifier suffers high bias or high variance problem.

  • training error vs. validation error - graph validation error , training set error, function of training examples (do incremental learning)
    • if lines seem converge same value , close @ end, classifier has high bias. in such case, adding more data won't help. change classifier 1 has higher variance, or lower regularization parameter of current one.
    • if on other hand lines quite far apart, , have low training set error high validation error, classifier has high variance. in case getting more data help. if after getting more data variance still high, can increase regularization parameter.

Comments

Post a Comment

Popular posts from this blog

javascript - Chart.js (Radar Chart) different scaleLineColor for each scaleLine -

-
Image
i try , create radar chart using chart.js has various colours each scaleline, or coloured between scalelines. wondering if possible? from: to: i have working graph, though there doesn't seem method change individual scale lines. kind regards leigh you can extend radar chart type this, so chart.types.radar.extend({ name: "radaralt", initialize: function (data) { chart.types.radar.prototype.initialize.apply(this, arguments); var originalscaledraw = this.scale.draw; var ctx = this.chart.ctx; this.scale.draw = function () { var linewidth = this.linewidth; // bypasses line drawing in originalscaledraw this.linewidth = linewidth; originalscaledraw.apply(this, arguments); ctx.linewidth = this.linewidth; var scale = this; // draw chart.helpers.each(scale.ylabels, function (label, index) { // color of ...
Read more

apache - Error with PHP mail(): Multiple or malformed newlines found in additional_header -

-
suddenly have started receiving above error without changes having been made script. host 1and1 (i know...) the script still works fine on different server, , suspicion there must have been server config change has lead this, although hosts plead ignorance. there's no information on above error @ in google can find - have ideas? server running apache if helps. had similar problem. came out of blue. no php code changed. what changed: php upgraded 5.5.25-1 5.5.26. a security risk in php mail() function has been fixed , newlines in additional_headers allowed no more. because newlines mean: starts email message (and surely don't want inject newlines through headers followed evil message). what have worked fine, e.g. having newlines after headers or passing whole message additional_headers , function no more. solution : sanitize headers. no multiple newlines in additional_headers argument. these count "multiple or malformed newlines":...
Read more

java - Android – MapFragment overlay button shadow, just like MyLocation button -

-
Image
i'm struggling add shadow custom overlay button on google map. goal make custom button google's mylocation button. here's how app looks now: layout.xml <relativelayout xmlns:android="http://schemas.android.com/apk/res/android" xmlns:tools="http://schemas.android.com/tools" xmlns:map="http://schemas.android.com/apk/res-auto" android:layout_height="match_parent" android:layout_width="match_parent" style="@style/apptheme"> <fragment android:layout_width="match_parent" android:layout_height="match_parent" android:id="@+id/map" tools:context=".mapsactivity" android:name="com.google.android.gms.maps.supportmapfragment" android:layout_alignparentend="true" android:layout_alignparentstart="true" android:layout_alignparentbottom="true" ...
Read more