python - Scrapy Spider cannot Extract contents of web page using xpath -

-

i have scrapy spider , using xpath selectors extract contents of page,kindly check going wrong

from scrapy.contrib.loader import itemloader scrapy.contrib.spiders import crawlspider,rule scrapy.selector import htmlxpathselector medicalproject.items import medicalprojectitem scrapy.contrib.linkextractors.sgml import sgmllinkextractor scrapy.selector import htmlxpathselector  scrapy import request     class myspider(crawlspider):       name = "medical"       allowed_domains = ["yananow.org"]       start_urls = ["http://yananow.org/query_stories.php"]  rules = (     rule(sgmllinkextractor(allow=[r'display_story.php\?\id\=\d+']),callback='parse_page',follow=true),            )  def parse_items(self, response):     hxs = htmlxpathselector(response)     titles = hxs.xpath('/html/body/div/table/tbody/tr[2]/td/table/tbody/tr/td')     items = []     title in titles:         item = medicalprojectitem()         item["patient_name"] = title.xpath("/html/body/div/table/tbody/tr[2]/td/table/tbody/tr/td/img[1]/text()").extract()         item["stories"] = title.xpath("/html/body/div/table/tbody/tr[2]/td/table/tbody/tr/td/div/font/p/text()").extract()         items.append(item)     return(items)

there lot of issues code here different approach.

i opted against crawlspider have more control on scraping process. grabbing name query page , story detail page.

i tried simplify xpath statements not diving (nested) table structures looking patterns of content. if want extract story ... there must link story.

here comes tested code (with comments):

# -*- coding: utf-8 -*- import scrapy  class myitem(scrapy.item):     name = scrapy.field()     story = scrapy.field()  class myspider(scrapy.spider):      name = 'medical'     allowed_domains = ['yananow.org']     start_urls = ['http://yananow.org/query_stories.php']      def parse(self, response):          rows = response.xpath('//a[contains(@href,"display_story")]')          #loop on links stories         row in rows:             myitem = myitem() # create new item             myitem['name'] = row.xpath('./text()').extract() # assign name link             story_url = response.urljoin(row.xpath('./@href').extract()[0]) # extract url link             request = scrapy.request(url = story_url, callback = self.parse_detail) # create request detail page story             request.meta['myitem'] = myitem # pass item request             yield request      def parse_detail(self, response):         myitem = response.meta['myitem'] # extract item (with name) response         text_raw = response.xpath('//font[@size=3]//text()').extract() # extract story (text)         myitem['story'] = ' '.join(map(unicode.strip, text_raw)) # clean text , assign item         yield myitem # return item

Comments

Post a Comment

Popular posts from this blog

javascript - Chart.js (Radar Chart) different scaleLineColor for each scaleLine -

-
Image
i try , create radar chart using chart.js has various colours each scaleline, or coloured between scalelines. wondering if possible? from: to: i have working graph, though there doesn't seem method change individual scale lines. kind regards leigh you can extend radar chart type this, so chart.types.radar.extend({ name: "radaralt", initialize: function (data) { chart.types.radar.prototype.initialize.apply(this, arguments); var originalscaledraw = this.scale.draw; var ctx = this.chart.ctx; this.scale.draw = function () { var linewidth = this.linewidth; // bypasses line drawing in originalscaledraw this.linewidth = linewidth; originalscaledraw.apply(this, arguments); ctx.linewidth = this.linewidth; var scale = this; // draw chart.helpers.each(scale.ylabels, function (label, index) { // color of ...
Read more

apache - Error with PHP mail(): Multiple or malformed newlines found in additional_header -

-
suddenly have started receiving above error without changes having been made script. host 1and1 (i know...) the script still works fine on different server, , suspicion there must have been server config change has lead this, although hosts plead ignorance. there's no information on above error @ in google can find - have ideas? server running apache if helps. had similar problem. came out of blue. no php code changed. what changed: php upgraded 5.5.25-1 5.5.26. a security risk in php mail() function has been fixed , newlines in additional_headers allowed no more. because newlines mean: starts email message (and surely don't want inject newlines through headers followed evil message). what have worked fine, e.g. having newlines after headers or passing whole message additional_headers , function no more. solution : sanitize headers. no multiple newlines in additional_headers argument. these count "multiple or malformed newlines":...
Read more

java - Android – MapFragment overlay button shadow, just like MyLocation button -

-
Image
i'm struggling add shadow custom overlay button on google map. goal make custom button google's mylocation button. here's how app looks now: layout.xml <relativelayout xmlns:android="http://schemas.android.com/apk/res/android" xmlns:tools="http://schemas.android.com/tools" xmlns:map="http://schemas.android.com/apk/res-auto" android:layout_height="match_parent" android:layout_width="match_parent" style="@style/apptheme"> <fragment android:layout_width="match_parent" android:layout_height="match_parent" android:id="@+id/map" tools:context=".mapsactivity" android:name="com.google.android.gms.maps.supportmapfragment" android:layout_alignparentend="true" android:layout_alignparentstart="true" android:layout_alignparentbottom="true" ...
Read more