python - How to use scrapy to crawl same url by post different data? -

-

i want crawl website post different page numbers,but data of first page spider finished,i think maybe crawl same url, filtered scrappy.
here code:

class zhejiangcrawl(spider):     name = 'zhejiangcrawl'     root_url= 'http://www.zjsfgkw.cn/execute/creditcompany'     start_page = 1     current_page = start_page     end_page = 24974     post_data = {'pageno': str(current_page), 'pagesize': '5', 'reallyname': '', 'credentialsnumber': '', 'ah': '',                       'zxfy': '', 'startlarq': '','endlarq':''}     headers = header     cookies = cookies      def start_requests(self):         return [formrequest(self.root_url, headers=self.headers, cookies=self.cookies, formdata=self.post_data, dont_filter=true,                         callback=self.parse)]      def parse(self, response):         if self.current_page < self.end_page:             self.current_page += 1             self.post_data['pageno'] = str(self.current_page)             yield [formrequest(self.root_url, headers=self.headers, cookies=self.cookies, dont_filter=true,                            formdata=self.post_data, callback=self.parse)]          jsonstr = json.loads(response.body)         item_dict in jsonstr['informationmodels']:             item = zhejiangcrawlitem()             item['name'] = item_dict['reallyname']             item['cardnum'] = item_dict['credentialsnumber']             item['performance'] = item_dict['zxje']             item['unperformance'] = item_dict['wzxje']             item['gistunit'] = item_dict['zxfy']             item['address'] = item_dict['address']             item['gistid'] = item_dict['zxyj']             item['casecode'] = item_dict['ah']             item['regdate'] = item_dict['larq']             item['exposuredate'] = item_dict['bgrq']             item['gistreason'] = item_dict['zxay']             yield item

how fix it?

if think it's being filtered because of dupefilter, add dont_filter=true formrequests.

also note, there's no reason make lists out of yielded/returned content.


Comments

Post a Comment

Popular posts from this blog

javascript - Chart.js (Radar Chart) different scaleLineColor for each scaleLine -

-
Image
i try , create radar chart using chart.js has various colours each scaleline, or coloured between scalelines. wondering if possible? from: to: i have working graph, though there doesn't seem method change individual scale lines. kind regards leigh you can extend radar chart type this, so chart.types.radar.extend({ name: "radaralt", initialize: function (data) { chart.types.radar.prototype.initialize.apply(this, arguments); var originalscaledraw = this.scale.draw; var ctx = this.chart.ctx; this.scale.draw = function () { var linewidth = this.linewidth; // bypasses line drawing in originalscaledraw this.linewidth = linewidth; originalscaledraw.apply(this, arguments); ctx.linewidth = this.linewidth; var scale = this; // draw chart.helpers.each(scale.ylabels, function (label, index) { // color of ...
Read more

apache - Error with PHP mail(): Multiple or malformed newlines found in additional_header -

-
suddenly have started receiving above error without changes having been made script. host 1and1 (i know...) the script still works fine on different server, , suspicion there must have been server config change has lead this, although hosts plead ignorance. there's no information on above error @ in google can find - have ideas? server running apache if helps. had similar problem. came out of blue. no php code changed. what changed: php upgraded 5.5.25-1 5.5.26. a security risk in php mail() function has been fixed , newlines in additional_headers allowed no more. because newlines mean: starts email message (and surely don't want inject newlines through headers followed evil message). what have worked fine, e.g. having newlines after headers or passing whole message additional_headers , function no more. solution : sanitize headers. no multiple newlines in additional_headers argument. these count "multiple or malformed newlines":...
Read more

java - Android – MapFragment overlay button shadow, just like MyLocation button -

-
Image
i'm struggling add shadow custom overlay button on google map. goal make custom button google's mylocation button. here's how app looks now: layout.xml <relativelayout xmlns:android="http://schemas.android.com/apk/res/android" xmlns:tools="http://schemas.android.com/tools" xmlns:map="http://schemas.android.com/apk/res-auto" android:layout_height="match_parent" android:layout_width="match_parent" style="@style/apptheme"> <fragment android:layout_width="match_parent" android:layout_height="match_parent" android:id="@+id/map" tools:context=".mapsactivity" android:name="com.google.android.gms.maps.supportmapfragment" android:layout_alignparentend="true" android:layout_alignparentstart="true" android:layout_alignparentbottom="true" ...
Read more