java - Force BufferedWriter to write from BlockingQueue when another tasks finished -

i'm writing simple html parser jsoup. i've got 50 000 links check, thought it's great chance learn abut threads , concurnecy. i've got 8 tasks registered executorservice: 6 of them parse links data stored in arraylists , add blockingqueues. 2 of tasks filewriters based on bufferedwriter. problem when 6 tasks finish prase links, file writers stop write data blockingqueue, lose part of data. i'm pretty newbie in java, if give me hand.... code:

main file:

public static void main(string[] args) {     blockingqueue<arraylist<string>> units = new arrayblockingqueue<arraylist<string>>(50, true);     blockingqueue<arraylist<string>> subjects = new arrayblockingqueue<arraylist<string>>(50, true);     file subjectfile = new file("lekarze.csv");     file unitfile = new file("miejsca.csv");     executorservice executor = executors.newfixedthreadpool(9);     executor.submit(new thread(new filesaver(subjects, subjectfile)));     executor.submit(new thread(new filesaver(units, unitfile)));     for(int = 29323; < 29400; i++){         executor.submit(new parserdocsthread(i, subjects, units, errors));     }      executor.shutdown(); } 

filesaver class:

package parser;  import java.io.bufferedwriter; import java.io.file; import java.io.filewriter; import java.io.ioexception; import java.util.arraylist; import java.util.concurrent.blockingqueue;  public class filesaver implements runnable {     private blockingqueue<arraylist<string>> towrite = null;     private file outputfile = null;     private bufferedwriter writer = null;      public filesaver(blockingqueue<arraylist<string>> queue, file file){        towrite = queue;        outputfile = file;    }      public void run() {        try {           writer = new bufferedwriter(new filewriter(outputfile, true));           while(true){               try{                   save(towrite.take());               } catch(interruptedexception e) {                   e.printstacktrace();               }           }       } catch (ioexception e) {           e.printstacktrace();       }   }    private void save(arraylist<string> data){       string temp ="";       int size = data.size();       for(int = 0; < size; i++){           temp += data.get(i);           if(i != size - 1) temp += '\t';        }       try {           writer.write(temp);           writer.newline();       } catch (ioexception e) {           e.printstacktrace();       }   } } 

in parserdocsthread i'm use put() method add elements blockingqueue.

your consumer threads don't end cleanly because take() call waiting new array, , not closing buffered writer. serviceexecutor gives on waiting these threads finish, , kills them. causing last lines in writer not written out disk.

you should use poll(10, timeunit.seconds) (but appropriate timeout). after timeout, consumers give on producers, , should make sure close buffered writer last of buffer printed out properly.

try (bufferedwriter writer = new bufferedwriter(new filewriter(outputfile, true))) {     while(true){         list<string> data = towrite.poll(10, timeunit.seconds);         if (data == null) {            break;         }         save(data, writer);     } } catch (...) { } 

i've put buffered writer here try-with-resources (so try here automatically close writer) , passed save method, can way, , manually close writer in block if want:

try {     ... } catch(...) { } {     writer.close(); // closes , flushes out remaining lines } 

you may want put in call awaittermination on executor servier (like so: how wait threads finish, using executorservice?) wait time greater poll timeout.